Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.03-32-g1b73099a80e6) initialized Starting run at 08:23:39 UTC on 2022-03-28. Successfully read inputs file ... Castro git describe: 22.02-19-ga92ac477d AMReX git describe: 22.03-32-g1b73099a8 Microphysics git describe: 22.03-13-gf58972c7 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.039288969 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.023345275 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.109707975 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.055868421 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.048910413 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.057221622 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.073029642 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.035855855 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.05822032 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051646354 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.047617546 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059807722 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063689412 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.035447216 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.022009448 seconds Ending run at 08:23:40 UTC on 2022-03-28. Run time = 0.843547128 Run time without initialization = 0.719536552 Average number of zones advanced per microsecond: 3.643 Average number of zones advanced per microsecond per rank: 3.643 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8436 ... 0.8436 ... 0.8436 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2231 0.2231 0.2231 26.45% VisMF::Write(FabArray) 11 0.149 0.149 0.149 17.67% MLCellLinOp::applyBC() 4433 0.09363 0.09363 0.09363 11.10% MLPoisson::Fsmooth() 3280 0.05931 0.05931 0.05931 7.03% FabArray::setVal() 1144 0.02591 0.02591 0.02591 3.07% StateData::FillBoundary(geom) 328 0.02323 0.02323 0.02323 2.75% StateDataPhysBCFunct::() 41 0.02216 0.02216 0.02216 2.63% MLCGSolver::bicgstab 82 0.02166 0.02166 0.02166 2.57% MultiFab::Dot() 1114 0.02119 0.02119 0.02119 2.51% FillBoundary_nowait() 4023 0.01781 0.01781 0.01781 2.11% FabArray::ParallelCopy_nowait() 861 0.01595 0.01595 0.01595 1.89% Castro::computeTemp() 63 0.01358 0.01358 0.01358 1.61% MultiFab::LinComb() 1586 0.01223 0.01223 0.01223 1.45% Gravity::fill_multipole_BCs() 11 0.01144 0.01144 0.01144 1.36% MLPoisson::Fapply() 1142 0.01045 0.01045 0.01045 1.24% MLCellLinOp::defineAuxData() 11 0.01007 0.01007 0.01007 1.19% Castro::enforce_min_density() 62 0.01004 0.01004 0.01004 1.19% MLMG::addInterpCorrection() 410 0.00697 0.00697 0.00697 0.83% amrex::average_down 410 0.006453 0.006453 0.006453 0.77% Castro::estTimeStep() 21 0.006065 0.006065 0.006065 0.72% MultiFab::Xpay() 585 0.006036 0.006036 0.006036 0.72% Castro::reset_internal_energy() 63 0.005528 0.005528 0.005528 0.66% FabArray::setDomainBndry() 41 0.005234 0.005234 0.005234 0.62% Castro::expand_state() 10 0.004719 0.004719 0.004719 0.56% Castro::normalize_species() 62 0.004404 0.004404 0.004404 0.52% Amr::checkPoint() 3 0.003921 0.003921 0.003921 0.46% BndryData::define() 11 0.003578 0.003578 0.003578 0.42% Castro::do_advance_ctu() 10 0.003182 0.003182 0.003182 0.38% Amr::writePlotFile() 2 0.003035 0.003035 0.003035 0.36% Castro::construct_new_gravity_source() 10 0.002873 0.002873 0.002873 0.34% Gravity::get_new_grav_vector() 11 0.002706 0.002706 0.002706 0.32% Castro::construct_old_gravity_source() 10 0.002406 0.002406 0.002406 0.29% Castro::enforce_speed_limit() 62 0.002348 0.002348 0.002348 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00225 0.00225 0.00225 0.27% MLCellLinOp::compGrad() 11 0.002043 0.002043 0.002043 0.24% MLMG::ResNormInf() 93 0.001937 0.001937 0.001937 0.23% MultiFab::Saxpy() 20 0.001813 0.001813 0.001813 0.21% Gravity::get_old_grav_vector() 10 0.001752 0.001752 0.001752 0.21% MLMG::oneIter() 82 0.001689 0.001689 0.001689 0.20% MLCellLinOp::setLevelBC() 11 0.001518 0.001518 0.001518 0.18% Gravity::actual_solve_with_mlmg() 11 0.001413 0.001413 0.001413 0.17% MLCellLinOp::prepareForSolve() 11 0.001324 0.001324 0.001324 0.16% FabArray::mult() 43 0.001323 0.001323 0.001323 0.16% MLCellLinOp::smooth() 1640 0.00123 0.00123 0.00123 0.15% Castro::initData() 1 0.001206 0.001206 0.001206 0.14% FabArrayBase::getCPC() 1323 0.0008825 0.0008825 0.0008825 0.10% FabArray::FillBoundary() 4023 0.0008267 0.0008267 0.0008267 0.10% MLMG::prepareForSolve() 11 0.0007635 0.0007635 0.0007635 0.09% FabArrayBase::CPC::define() 454 0.0007199 0.0007199 0.0007199 0.09% FabArrayBase::getFB() 4023 0.0007087 0.0007087 0.0007087 0.08% Gravity::update_max_rhs() 11 0.0005479 0.0005479 0.0005479 0.06% MLCellLinOp::apply() 1142 0.0005397 0.0005397 0.0005397 0.06% MultiFab::Copy() 11 0.0005142 0.0005142 0.0005142 0.06% CGSolver::sxay() 1586 0.0005036 0.0005036 0.0005036 0.06% Gravity::solve_for_phi() 10 0.000464 0.000464 0.000464 0.06% Amr::InitAmr() 1 0.0004357 0.0004357 0.0004357 0.05% MLMG::mgVcycle() 82 0.0003982 0.0003982 0.0003982 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003686 0.0003686 0.0003686 0.04% MultiFab::min() 10 0.0003196 0.0003196 0.0003196 0.04% FabArray::ParallelCopy() 861 0.0002941 0.0002941 0.0002941 0.03% main() 1 0.0002889 0.0002889 0.0002889 0.03% MultiFab::max() 11 0.0002566 0.0002566 0.0002566 0.03% FillPatchIterator::Initialize 41 0.0002504 0.0002504 0.0002504 0.03% Amr::coarseTimeStep() 10 0.0002456 0.0002456 0.0002456 0.03% Castro::construct_new_gravity() 10 0.0002396 0.0002396 0.0002396 0.03% MLCellLinOp::correctionResidual() 492 0.0002394 0.0002394 0.0002394 0.03% Gravity::actual_multilevel_solve() 1 0.0002297 0.0002297 0.0002297 0.03% Amr::timeStep() 10 0.0002068 0.0002068 0.0002068 0.02% MLMG::MLRhsNormInf() 11 0.0001964 0.0001964 0.0001964 0.02% MLCellLinOp::defineBC() 11 0.0001859 0.0001859 0.0001859 0.02% MLLinOp::defineGrids() 11 0.0001755 0.0001755 0.0001755 0.02% Castro::subcycle_advance_ctu() 10 0.0001573 0.0001573 0.0001573 0.02% Amr::defBaseLevel() 1 0.0001497 0.0001497 0.0001497 0.02% MLMG:computeResOfCorrection() 410 0.0001444 0.0001444 0.0001444 0.02% StateData::checkPoint() 12 0.0001371 0.0001371 0.0001371 0.02% Castro::construct_new_source() 50 0.0001175 0.0001175 0.0001175 0.01% MLMG::actualBottomSolve() 82 0.0001148 0.0001148 0.0001148 0.01% Castro::advance() 10 0.0001094 0.0001094 0.0001094 0.01% MLMG::mgVcycle_down::0 82 0.0001002 0.0001002 0.0001002 0.01% FabArrayBase::FB::FB() 56 9.608e-05 9.608e-05 9.608e-05 0.01% MLMG::mgVcycle_down::1 82 9.34e-05 9.34e-05 9.34e-05 0.01% MLMG::mgVcycle_down::4 82 8.769e-05 8.769e-05 8.769e-05 0.01% MLMG::mgVcycle_down::2 82 8.744e-05 8.744e-05 8.744e-05 0.01% MLMG::mgVcycle_down::3 82 8.618e-05 8.618e-05 8.618e-05 0.01% MLMG::solve() 11 8.33e-05 8.33e-05 8.33e-05 0.01% Castro::initialize_advance() 10 8.254e-05 8.254e-05 8.254e-05 0.01% Castro::clean_state() 62 7.465e-05 7.465e-05 7.465e-05 0.01% AmrLevel::checkPoint() 3 7.285e-05 7.285e-05 7.285e-05 0.01% Castro::finalize_advance() 10 7.037e-05 7.037e-05 7.037e-05 0.01% MLMG::mgVcycle_up::4 82 6.663e-05 6.663e-05 6.663e-05 0.01% Castro::initialize_do_advance() 10 5.903e-05 5.903e-05 5.903e-05 0.01% MLMG::mgVcycle_up::0 82 5.836e-05 5.836e-05 5.836e-05 0.01% MLMG::mgVcycle_up::3 82 5.78e-05 5.78e-05 5.78e-05 0.01% MLMG::mgVcycle_up::1 82 5.568e-05 5.568e-05 5.568e-05 0.01% MLMG::mgVcycle_up::2 82 5.483e-05 5.483e-05 5.483e-05 0.01% Castro::post_timestep() 10 5.231e-05 5.231e-05 5.231e-05 0.01% MLCellLinOp::solutionResidual() 93 5.219e-05 5.219e-05 5.219e-05 0.01% StateData::define() 4 4.333e-05 4.333e-05 4.333e-05 0.01% Castro::finalize_do_advance() 10 3.922e-05 3.922e-05 3.922e-05 0.00% Castro::swap_state_time_levels() 10 3.795e-05 3.795e-05 3.795e-05 0.00% MLMG::mgVcycle_bottom 82 3.774e-05 3.774e-05 3.774e-05 0.00% MLMG::computeResidual() 82 3.599e-05 3.599e-05 3.599e-05 0.00% Castro::enforce_consistent_e() 1 3.521e-05 3.521e-05 3.521e-05 0.00% FillPatchSingleLevel 41 3.232e-05 3.232e-05 3.232e-05 0.00% MLLinOp::define() 11 3.025e-05 3.025e-05 3.025e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.824e-05 2.824e-05 2.824e-05 0.00% makeSFC 55 2.744e-05 2.744e-05 2.744e-05 0.00% Amr::writeSmallPlotFile() 1 2.531e-05 2.531e-05 2.531e-05 0.00% MLPoisson::define() 11 2.32e-05 2.32e-05 2.32e-05 0.00% Amr::FinalizeInit() 1 2.102e-05 2.102e-05 2.102e-05 0.00% Castro::construct_old_source() 50 2.097e-05 2.097e-05 2.097e-05 0.00% Castro::do_new_sources() 10 1.865e-05 1.865e-05 1.865e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.726e-05 1.726e-05 1.726e-05 0.00% Castro::do_old_sources() 10 1.723e-05 1.723e-05 1.723e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.678e-05 1.678e-05 1.678e-05 0.00% DistributionMapping::Distribute() 56 1.644e-05 1.644e-05 1.644e-05 0.00% Castro::apply_source_to_state() 20 1.156e-05 1.156e-05 1.156e-05 0.00% Castro::construct_old_gravity() 10 1.108e-05 1.108e-05 1.108e-05 0.00% Gravity::swapTimeLevels() 10 1.052e-05 1.052e-05 1.052e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.006e-05 1.006e-05 1.006e-05 0.00% MLPoisson::prepareForSolve() 11 9.121e-06 9.121e-06 9.121e-06 0.00% Amr::initSubcycle() 1 8.906e-06 8.906e-06 8.906e-06 0.00% MLMG::computeMLResidual() 11 7.408e-06 7.408e-06 7.408e-06 0.00% Amr::InitializeInit() 1 6.549e-06 6.549e-06 6.549e-06 0.00% MLMG::buildFineMask() 11 6.249e-06 6.249e-06 6.249e-06 0.00% Castro::computeNewDt() 9 5.957e-06 5.957e-06 5.957e-06 0.00% MLMG::getGradSolution() 11 5.87e-06 5.87e-06 5.87e-06 0.00% AmrLevel::checkPointPost() 3 5.695e-06 5.695e-06 5.695e-06 0.00% MLMG::MLResNormInf() 11 5.502e-06 5.502e-06 5.502e-06 0.00% Castro::create_source_corrector() 10 5.379e-06 5.379e-06 5.379e-06 0.00% Gravity::set_mass_offset() 11 4.301e-06 4.301e-06 4.301e-06 0.00% Castro::retry_advance_ctu() 10 4.288e-06 4.288e-06 4.288e-06 0.00% Castro::post_init() 1 3.755e-06 3.755e-06 3.755e-06 0.00% Castro::FluxRegFineAdd() 10 3.474e-06 3.474e-06 3.474e-06 0.00% Castro::computeInitialDt() 2 2.838e-06 2.838e-06 2.838e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.629e-06 2.629e-06 2.629e-06 0.00% Amr::init() 1 2.388e-06 2.388e-06 2.388e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.166e-06 2.166e-06 2.166e-06 0.00% AmrLevel::checkPointPre() 3 1.952e-06 1.952e-06 1.952e-06 0.00% Castro::post_regrid() 1 1.213e-06 1.213e-06 1.213e-06 0.00% Amr::initialInit() 1 1.122e-06 1.122e-06 1.122e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8436 0.8436 0.8436 100.00% Amr::coarseTimeStep() 10 0.6973 0.6973 0.6973 82.66% Amr::timeStep() 10 0.6225 0.6225 0.6225 73.79% Castro::advance() 10 0.6158 0.6158 0.6158 73.00% Castro::subcycle_advance_ctu() 10 0.6017 0.6017 0.6017 71.33% Castro::do_advance_ctu() 10 0.6016 0.6016 0.6016 71.31% Gravity::solve_phi_with_mlmg() 11 0.3242 0.3242 0.3242 38.43% Gravity::actual_solve_with_mlmg() 11 0.3125 0.3125 0.3125 37.04% Castro::construct_new_gravity() 10 0.2949 0.2949 0.2949 34.95% MLMG::solve() 11 0.2899 0.2899 0.2899 34.36% Gravity::solve_for_phi() 10 0.2764 0.2764 0.2764 32.76% MLMG::oneIter() 82 0.2739 0.2739 0.2739 32.47% MLMG::mgVcycle() 82 0.2722 0.2722 0.2722 32.27% Castro::construct_ctu_hydro_source() 10 0.2231 0.2231 0.2231 26.45% VisMF::Write(FabArray) 11 0.149 0.149 0.149 17.67% MLCellLinOp::smooth() 1640 0.1444 0.1444 0.1444 17.12% Amr::init() 1 0.1234 0.1234 0.1234 14.63% MLCellLinOp::applyBC() 4433 0.1131 0.1131 0.1131 13.40% Amr::checkPoint() 3 0.1107 0.1107 0.1107 13.12% AmrLevel::checkPoint() 3 0.1068 0.1068 0.1068 12.66% StateData::checkPoint() 12 0.1067 0.1067 0.1067 12.65% MLMG::mgVcycle_bottom 82 0.07739 0.07739 0.07739 9.17% MLMG::actualBottomSolve() 82 0.07736 0.07736 0.07736 9.17% MLCGSolver::bicgstab 82 0.07662 0.07662 0.07662 9.08% Amr::initialInit() 1 0.06067 0.06067 0.06067 7.19% MLPoisson::Fsmooth() 3280 0.05931 0.05931 0.05931 7.03% FillPatchIterator::Initialize 41 0.05506 0.05506 0.05506 6.53% Amr::FinalizeInit() 1 0.05163 0.05163 0.05163 6.12% Castro::post_init() 1 0.05088 0.05088 0.05088 6.03% FillPatchSingleLevel 41 0.04957 0.04957 0.04957 5.88% Gravity::multilevel_solve_for_new_phi() 1 0.04851 0.04851 0.04851 5.75% Gravity::actual_multilevel_solve() 1 0.04849 0.04849 0.04849 5.75% Amr::writePlotFile() 2 0.04549 0.04549 0.04549 5.39% StateDataPhysBCFunct::() 41 0.04539 0.04539 0.04539 5.38% MLCellLinOp::apply() 1142 0.03965 0.03965 0.03965 4.70% MLMG::mgVcycle_down::0 82 0.03849 0.03849 0.03849 4.56% Castro::clean_state() 62 0.03526 0.03526 0.03526 4.18% MLMG::mgVcycle_up::0 82 0.03299 0.03299 0.03299 3.91% FabArray::setVal() 1144 0.02591 0.02591 0.02591 3.07% Castro::initialize_do_advance() 10 0.02351 0.02351 0.02351 2.79% StateData::FillBoundary(geom) 328 0.02323 0.02323 0.02323 2.75% MLCellLinOp::correctionResidual() 492 0.02219 0.02219 0.02219 2.63% MultiFab::Dot() 1114 0.02119 0.02119 0.02119 2.51% Gravity::get_new_grav_vector() 11 0.02046 0.02046 0.02046 2.43% FabArray::FillBoundary() 4023 0.01945 0.01945 0.01945 2.31% MLMG:computeResOfCorrection() 410 0.01924 0.01924 0.01924 2.28% Castro::computeTemp() 63 0.01911 0.01911 0.01911 2.27% FillBoundary_nowait() 4023 0.01862 0.01862 0.01862 2.21% MLMG::mgVcycle_down::1 82 0.01839 0.01839 0.01839 2.18% MLMG::mgVcycle_down::2 82 0.01774 0.01774 0.01774 2.10% Castro::expand_state() 10 0.01774 0.01774 0.01774 2.10% FabArray::ParallelCopy() 861 0.01715 0.01715 0.01715 2.03% MLPoisson::define() 11 0.01707 0.01707 0.01707 2.02% MLMG::mgVcycle_down::3 82 0.01695 0.01695 0.01695 2.01% FabArray::ParallelCopy_nowait() 861 0.01685 0.01685 0.01685 2.00% MLMG::mgVcycle_down::4 82 0.01618 0.01618 0.01618 1.92% Castro::construct_old_gravity() 10 0.01556 0.01556 0.01556 1.84% Gravity::get_old_grav_vector() 10 0.01555 0.01555 0.01555 1.84% Castro::initialize_advance() 10 0.01387 0.01387 0.01387 1.64% MLMG::mgVcycle_up::2 82 0.01385 0.01385 0.01385 1.64% MLMG::mgVcycle_up::1 82 0.01361 0.01361 0.01361 1.61% MLMG::addInterpCorrection() 410 0.01349 0.01349 0.01349 1.60% MLMG::mgVcycle_up::3 82 0.01317 0.01317 0.01317 1.56% MLMG::mgVcycle_up::4 82 0.01306 0.01306 0.01306 1.55% amrex::average_down 410 0.01294 0.01294 0.01294 1.53% MLCGSolver::ParallelAllReduce 1514 0.01281 0.01281 0.01281 1.52% CGSolver::sxay() 1586 0.01273 0.01273 0.01273 1.51% MultiFab::LinComb() 1586 0.01223 0.01223 0.01223 1.45% MLCellLinOp::defineAuxData() 11 0.01175 0.01175 0.01175 1.39% Castro::do_new_sources() 10 0.01158 0.01158 0.01158 1.37% Gravity::fill_multipole_BCs() 11 0.01144 0.01144 0.01144 1.36% MLPoisson::Fapply() 1142 0.01045 0.01045 0.01045 1.24% Castro::enforce_min_density() 62 0.01004 0.01004 0.01004 1.19% Amr::InitializeInit() 1 0.009045 0.009045 0.009045 1.07% Amr::defBaseLevel() 1 0.009038 0.009038 0.009038 1.07% Castro::do_old_sources() 10 0.008063 0.008063 0.008063 0.96% MLCellLinOp::solutionResidual() 93 0.007906 0.007906 0.007906 0.94% MLMG::computeResidual() 82 0.006834 0.006834 0.006834 0.81% Castro::post_timestep() 10 0.006474 0.006474 0.006474 0.77% Castro::estTimeStep() 21 0.006065 0.006065 0.006065 0.72% MultiFab::Xpay() 585 0.006036 0.006036 0.006036 0.72% MLMG::prepareForSolve() 11 0.005762 0.005762 0.005762 0.68% Castro::reset_internal_energy() 63 0.005528 0.005528 0.005528 0.66% FabArray::setDomainBndry() 41 0.005234 0.005234 0.005234 0.62% MLCellLinOp::defineBC() 11 0.005025 0.005025 0.005025 0.60% BndryData::define() 11 0.004839 0.004839 0.004839 0.57% Castro::normalize_species() 62 0.004404 0.004404 0.004404 0.52% Castro::initData() 1 0.003361 0.003361 0.003361 0.40% Castro::construct_new_source() 50 0.00299 0.00299 0.00299 0.35% Castro::construct_new_gravity_source() 10 0.002873 0.002873 0.002873 0.34% Castro::computeNewDt() 9 0.002728 0.002728 0.002728 0.32% MLMG::getGradSolution() 11 0.002617 0.002617 0.002617 0.31% MLCellLinOp::compGrad() 11 0.002611 0.002611 0.002611 0.31% Castro::construct_old_source() 50 0.002427 0.002427 0.002427 0.29% Castro::construct_old_gravity_source() 10 0.002406 0.002406 0.002406 0.29% Castro::enforce_speed_limit() 62 0.002348 0.002348 0.002348 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00225 0.00225 0.00225 0.27% MLMG::ResNormInf() 93 0.001937 0.001937 0.001937 0.23% Castro::apply_source_to_state() 20 0.001824 0.001824 0.001824 0.22% MultiFab::Saxpy() 20 0.001813 0.001813 0.001813 0.21% FabArrayBase::getCPC() 1323 0.001602 0.001602 0.001602 0.19% MLCellLinOp::setLevelBC() 11 0.001518 0.001518 0.001518 0.18% MLPoisson::prepareForSolve() 11 0.001333 0.001333 0.001333 0.16% MLCellLinOp::prepareForSolve() 11 0.001324 0.001324 0.001324 0.16% FabArray::mult() 43 0.001323 0.001323 0.001323 0.16% MLMG::computeMLResidual() 11 0.001115 0.001115 0.001115 0.13% Gravity::swapTimeLevels() 10 0.00103 0.00103 0.00103 0.12% Gravity::update_max_rhs() 11 0.0009473 0.0009473 0.0009473 0.11% FabArrayBase::getFB() 4023 0.0008048 0.0008048 0.0008048 0.10% FabArrayBase::CPC::define() 454 0.0007199 0.0007199 0.0007199 0.09% Castro::computeInitialDt() 2 0.0006931 0.0006931 0.0006931 0.08% MultiFab::Copy() 11 0.0005142 0.0005142 0.0005142 0.06% Castro::post_regrid() 1 0.0005121 0.0005121 0.0005121 0.06% Amr::InitAmr() 1 0.0004446 0.0004446 0.0004446 0.05% MultiFab::min() 10 0.0003196 0.0003196 0.0003196 0.04% MLLinOp::define() 11 0.0002677 0.0002677 0.0002677 0.03% MultiFab::max() 11 0.0002566 0.0002566 0.0002566 0.03% MLMG::MLResNormInf() 11 0.0002542 0.0002542 0.0002542 0.03% MLLinOp::defineGrids() 11 0.0002374 0.0002374 0.0002374 0.03% MLMG::MLRhsNormInf() 11 0.0001964 0.0001964 0.0001964 0.02% FabArrayBase::FB::FB() 56 9.608e-05 9.608e-05 9.608e-05 0.01% Castro::finalize_advance() 10 7.384e-05 7.384e-05 7.384e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.977e-05 5.977e-05 5.977e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.339e-05 5.339e-05 5.339e-05 0.01% StateData::define() 4 4.333e-05 4.333e-05 4.333e-05 0.01% makeSFC 55 4.252e-05 4.252e-05 4.252e-05 0.01% Castro::finalize_do_advance() 10 3.922e-05 3.922e-05 3.922e-05 0.00% Castro::swap_state_time_levels() 10 3.795e-05 3.795e-05 3.795e-05 0.00% Castro::enforce_consistent_e() 1 3.521e-05 3.521e-05 3.521e-05 0.00% Amr::writeSmallPlotFile() 1 2.531e-05 2.531e-05 2.531e-05 0.00% DistributionMapping::Distribute() 56 1.644e-05 1.644e-05 1.644e-05 0.00% Amr::initSubcycle() 1 8.906e-06 8.906e-06 8.906e-06 0.00% MLMG::buildFineMask() 11 6.249e-06 6.249e-06 6.249e-06 0.00% AmrLevel::checkPointPost() 3 5.695e-06 5.695e-06 5.695e-06 0.00% Castro::create_source_corrector() 10 5.379e-06 5.379e-06 5.379e-06 0.00% Gravity::set_mass_offset() 11 4.301e-06 4.301e-06 4.301e-06 0.00% Castro::retry_advance_ctu() 10 4.288e-06 4.288e-06 4.288e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.997e-06 3.997e-06 3.997e-06 0.00% Castro::FluxRegFineAdd() 10 3.474e-06 3.474e-06 3.474e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.166e-06 2.166e-06 2.166e-06 0.00% AmrLevel::checkPointPre() 3 1.952e-06 1.952e-06 1.952e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.03-32-g1b73099a80e6) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.03-32-g1b73099a80e6) initialized Starting run at 08:23:40 UTC on 2022-03-28. Successfully read inputs file ... Castro git describe: 22.02-19-ga92ac477d AMReX git describe: 22.03-32-g1b73099a8 Microphysics git describe: 22.03-13-gf58972c7 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.504541796 Restart time = 0.055250428 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.110315838 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048046042 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.049338668 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.055781279 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062569016 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.060316044 seconds Ending run at 08:23:41 UTC on 2022-03-28. Run time = 0.442522226 Run time without initialization = 0.386714941 Average number of zones advanced per microsecond: 3.389 Average number of zones advanced per microsecond per rank: 3.389 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4426 ... 0.4426 ... 0.4426 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1224 0.1224 0.1224 27.65% VisMF::Read() 3 0.04368 0.04368 0.04368 9.87% Amr::writePlotFile() 1 0.03835 0.03835 0.03835 8.67% MLCellLinOp::applyBC() 1946 0.03735 0.03735 0.03735 8.44% MLPoisson::Fsmooth() 1440 0.02539 0.02539 0.02539 5.74% VisMF::Write(FabArray) 1 0.02205 0.02205 0.02205 4.98% FabArray::setVal() 537 0.01573 0.01573 0.01573 3.55% StateData::FillBoundary(geom) 160 0.01099 0.01099 0.01099 2.48% MLCGSolver::bicgstab 36 0.009185 0.009185 0.009185 2.08% MultiFab::Dot() 484 0.009046 0.009046 0.009046 2.04% StateDataPhysBCFunct::() 20 0.007631 0.007631 0.007631 1.72% FillBoundary_nowait() 1766 0.007053 0.007053 0.007053 1.59% FabArray::ParallelCopy_nowait() 380 0.007015 0.007015 0.007015 1.59% Castro::computeTemp() 30 0.006969 0.006969 0.006969 1.57% Gravity::fill_multipole_BCs() 6 0.00613 0.00613 0.00613 1.38% MLCellLinOp::defineAuxData() 6 0.005376 0.005376 0.005376 1.21% MultiFab::LinComb() 690 0.005206 0.005206 0.005206 1.18% Castro::enforce_min_density() 30 0.004732 0.004732 0.004732 1.07% MLPoisson::Fapply() 500 0.004533 0.004533 0.004533 1.02% Castro::expand_state() 5 0.004253 0.004253 0.004253 0.96% FabArray::setDomainBndry() 20 0.003844 0.003844 0.003844 0.87% Castro::estTimeStep() 10 0.003437 0.003437 0.003437 0.78% Amr::restart() 1 0.00295 0.00295 0.00295 0.67% MLMG::addInterpCorrection() 180 0.002949 0.002949 0.002949 0.67% amrex::average_down 180 0.002754 0.002754 0.002754 0.62% MultiFab::Xpay() 258 0.002622 0.002622 0.002622 0.59% Castro::do_advance_ctu() 5 0.002541 0.002541 0.002541 0.57% Castro::reset_internal_energy() 30 0.002383 0.002383 0.002383 0.54% Castro::normalize_species() 30 0.002183 0.002183 0.002183 0.49% BndryData::define() 6 0.001933 0.001933 0.001933 0.44% MLCellLinOp::compGrad() 6 0.001456 0.001456 0.001456 0.33% Castro::construct_new_gravity_source() 5 0.001413 0.001413 0.001413 0.32% Gravity::get_new_grav_vector() 5 0.00124 0.00124 0.00124 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.00118 0.00118 0.00118 0.27% Castro::enforce_speed_limit() 30 0.001115 0.001115 0.001115 0.25% Castro::construct_old_gravity_source() 5 0.00106 0.00106 0.00106 0.24% MultiFab::Saxpy() 10 0.0009204 0.0009204 0.0009204 0.21% Gravity::get_old_grav_vector() 5 0.0009032 0.0009032 0.0009032 0.20% MLMG::ResNormInf() 42 0.0008581 0.0008581 0.0008581 0.19% MLCellLinOp::setLevelBC() 6 0.0008121 0.0008121 0.0008121 0.18% MLMG::oneIter() 36 0.0007436 0.0007436 0.0007436 0.17% Gravity::actual_solve_with_mlmg() 6 0.0007225 0.0007225 0.0007225 0.16% MLCellLinOp::prepareForSolve() 6 0.0007052 0.0007052 0.0007052 0.16% FabArray::mult() 22 0.0006502 0.0006502 0.0006502 0.15% Gravity::update_max_rhs() 6 0.0005121 0.0005121 0.0005121 0.12% MLCellLinOp::smooth() 720 0.0004861 0.0004861 0.0004861 0.11% FabArrayBase::CPC::define() 244 0.0004447 0.0004447 0.0004447 0.10% MLMG::prepareForSolve() 6 0.0004305 0.0004305 0.0004305 0.10% FabArrayBase::getCPC() 632 0.0004127 0.0004127 0.0004127 0.09% FabArray::FillBoundary() 1766 0.0003957 0.0003957 0.0003957 0.09% Amr::InitAmr() 1 0.0003677 0.0003677 0.0003677 0.08% MultiFab::Copy() 6 0.000367 0.000367 0.000367 0.08% main() 1 0.0003088 0.0003088 0.0003088 0.07% FabArrayBase::getFB() 1766 0.0003063 0.0003063 0.0003063 0.07% Gravity::solve_for_phi() 5 0.0002309 0.0002309 0.0002309 0.05% CGSolver::sxay() 690 0.0002207 0.0002207 0.0002207 0.05% MLCellLinOp::apply() 500 0.0002139 0.0002139 0.0002139 0.05% Gravity::actual_multilevel_solve() 1 0.000209 0.000209 0.000209 0.05% Castro::construct_new_gravity() 5 0.0001801 0.0001801 0.0001801 0.04% MLMG::mgVcycle() 36 0.0001713 0.0001713 0.0001713 0.04% MultiFab::min() 5 0.0001625 0.0001625 0.0001625 0.04% FabArray::ParallelCopy() 380 0.0001408 0.0001408 0.0001408 0.03% MLCGSolver::ParallelAllReduce 659 0.0001399 0.0001399 0.0001399 0.03% Castro::construct_new_source() 25 0.0001334 0.0001334 0.0001334 0.03% MultiFab::max() 6 0.0001316 0.0001316 0.0001316 0.03% FillPatchIterator::Initialize 20 0.0001189 0.0001189 0.0001189 0.03% Amr::coarseTimeStep() 5 0.0001115 0.0001115 0.0001115 0.03% MLMG::MLRhsNormInf() 6 0.0001051 0.0001051 0.0001051 0.02% Amr::timeStep() 5 0.0001043 0.0001043 0.0001043 0.02% MLCellLinOp::defineBC() 6 0.000101 0.000101 0.000101 0.02% MLCellLinOp::correctionResidual() 216 9.594e-05 9.594e-05 9.594e-05 0.02% MLLinOp::defineGrids() 6 9.349e-05 9.349e-05 9.349e-05 0.02% StateData::restartDoit() 4 7.61e-05 7.61e-05 7.61e-05 0.02% Castro::initialize_advance() 5 7.413e-05 7.413e-05 7.413e-05 0.02% AmrLevel::restart() 1 7.33e-05 7.33e-05 7.33e-05 0.02% Castro::subcycle_advance_ctu() 5 6.533e-05 6.533e-05 6.533e-05 0.01% FabArrayBase::FB::FB() 26 6.358e-05 6.358e-05 6.358e-05 0.01% MLMG:computeResOfCorrection() 180 6.31e-05 6.31e-05 6.31e-05 0.01% Castro::post_timestep() 5 5.434e-05 5.434e-05 5.434e-05 0.01% Castro::advance() 5 5.288e-05 5.288e-05 5.288e-05 0.01% Castro::construct_old_source() 25 5.145e-05 5.145e-05 5.145e-05 0.01% MLMG::actualBottomSolve() 36 4.986e-05 4.986e-05 4.986e-05 0.01% MLMG::solve() 6 3.949e-05 3.949e-05 3.949e-05 0.01% MLMG::mgVcycle_down::1 36 3.946e-05 3.946e-05 3.946e-05 0.01% MLMG::mgVcycle_down::0 36 3.943e-05 3.943e-05 3.943e-05 0.01% MLMG::mgVcycle_down::2 36 3.547e-05 3.547e-05 3.547e-05 0.01% Castro::clean_state() 30 3.439e-05 3.439e-05 3.439e-05 0.01% MLMG::mgVcycle_down::4 36 3.408e-05 3.408e-05 3.408e-05 0.01% MLMG::mgVcycle_down::3 36 3.339e-05 3.339e-05 3.339e-05 0.01% Castro::do_new_sources() 5 3.231e-05 3.231e-05 3.231e-05 0.01% Castro::create_source_corrector() 5 3.152e-05 3.152e-05 3.152e-05 0.01% Castro::initialize_do_advance() 5 2.748e-05 2.748e-05 2.748e-05 0.01% MLLinOp::define() 6 2.744e-05 2.744e-05 2.744e-05 0.01% Castro::swap_state_time_levels() 5 2.707e-05 2.707e-05 2.707e-05 0.01% MLMG::mgVcycle_up::4 36 2.699e-05 2.699e-05 2.699e-05 0.01% Castro::post_restart() 1 2.687e-05 2.687e-05 2.687e-05 0.01% Amr::writeSmallPlotFile() 1 2.577e-05 2.577e-05 2.577e-05 0.01% Castro::finalize_advance() 5 2.537e-05 2.537e-05 2.537e-05 0.01% MLMG::mgVcycle_up::3 36 2.429e-05 2.429e-05 2.429e-05 0.01% MLCellLinOp::solutionResidual() 42 2.302e-05 2.302e-05 2.302e-05 0.01% MLMG::mgVcycle_up::0 36 2.15e-05 2.15e-05 2.15e-05 0.00% MLMG::mgVcycle_up::2 36 2.13e-05 2.13e-05 2.13e-05 0.00% MLMG::mgVcycle_up::1 36 2.041e-05 2.041e-05 2.041e-05 0.00% Castro::finalize_do_advance() 5 1.947e-05 1.947e-05 1.947e-05 0.00% MLMG::mgVcycle_bottom 36 1.707e-05 1.707e-05 1.707e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.662e-05 1.662e-05 1.662e-05 0.00% MLMG::computeResidual() 36 1.66e-05 1.66e-05 1.66e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.537e-05 1.537e-05 1.537e-05 0.00% MLPoisson::define() 6 1.489e-05 1.489e-05 1.489e-05 0.00% FillPatchSingleLevel 20 1.469e-05 1.469e-05 1.469e-05 0.00% makeSFC 30 1.438e-05 1.438e-05 1.438e-05 0.00% DistributionMapping::Distribute() 31 9.361e-06 9.361e-06 9.361e-06 0.00% Amr::initSubcycle() 1 8.784e-06 8.784e-06 8.784e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.454e-06 8.454e-06 8.454e-06 0.00% Castro::do_old_sources() 5 8.041e-06 8.041e-06 8.041e-06 0.00% Castro::apply_source_to_state() 10 5.764e-06 5.764e-06 5.764e-06 0.00% Castro::construct_old_gravity() 5 5.6e-06 5.6e-06 5.6e-06 0.00% MLPoisson::prepareForSolve() 6 4.517e-06 4.517e-06 4.517e-06 0.00% Gravity::swapTimeLevels() 5 4.355e-06 4.355e-06 4.355e-06 0.00% MLMG::buildFineMask() 6 3.29e-06 3.29e-06 3.29e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.204e-06 3.204e-06 3.204e-06 0.00% Castro::computeNewDt() 5 3.176e-06 3.176e-06 3.176e-06 0.00% MLMG::getGradSolution() 6 3.087e-06 3.087e-06 3.087e-06 0.00% MLMG::computeMLResidual() 6 2.931e-06 2.931e-06 2.931e-06 0.00% MLMG::MLResNormInf() 6 2.779e-06 2.779e-06 2.779e-06 0.00% Gravity::set_mass_offset() 6 2.688e-06 2.688e-06 2.688e-06 0.00% Castro::retry_advance_ctu() 5 1.619e-06 1.619e-06 1.619e-06 0.00% Castro::FluxRegFineAdd() 5 1.45e-06 1.45e-06 1.45e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.153e-06 1.153e-06 1.153e-06 0.00% Amr::init() 1 1.096e-06 1.096e-06 1.096e-06 0.00% AmrLevel::AmrLevel() 1 1.026e-06 1.026e-06 1.026e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4426 0.4426 0.4426 100.00% Amr::coarseTimeStep() 5 0.3262 0.3262 0.3262 73.70% Amr::timeStep() 5 0.3245 0.3245 0.3245 73.32% Castro::advance() 5 0.3205 0.3205 0.3205 72.43% Castro::subcycle_advance_ctu() 5 0.3099 0.3099 0.3099 70.02% Castro::do_advance_ctu() 5 0.3098 0.3098 0.3098 70.00% Castro::construct_new_gravity() 5 0.144 0.144 0.144 32.53% Gravity::solve_phi_with_mlmg() 6 0.1404 0.1404 0.1404 31.73% Gravity::solve_for_phi() 5 0.1347 0.1347 0.1347 30.44% Gravity::actual_solve_with_mlmg() 6 0.1342 0.1342 0.1342 30.32% Castro::construct_ctu_hydro_source() 5 0.1224 0.1224 0.1224 27.65% MLMG::solve() 6 0.1217 0.1217 0.1217 27.50% MLMG::oneIter() 36 0.1138 0.1138 0.1138 25.72% MLMG::mgVcycle() 36 0.1131 0.1131 0.1131 25.55% Amr::writePlotFile() 1 0.0604 0.0604 0.0604 13.65% MLCellLinOp::smooth() 720 0.05925 0.05925 0.05925 13.39% Amr::init() 1 0.05529 0.05529 0.05529 12.49% Amr::restart() 1 0.05528 0.05528 0.05528 12.49% MLCellLinOp::applyBC() 1946 0.04517 0.04517 0.04517 10.21% AmrLevel::restart() 1 0.04457 0.04457 0.04457 10.07% StateData::restartDoit() 4 0.04449 0.04449 0.04449 10.05% VisMF::Read() 3 0.04368 0.04368 0.04368 9.87% MLMG::mgVcycle_bottom 36 0.03258 0.03258 0.03258 7.36% MLMG::actualBottomSolve() 36 0.03256 0.03256 0.03256 7.36% MLCGSolver::bicgstab 36 0.03224 0.03224 0.03224 7.29% MLPoisson::Fsmooth() 1440 0.02539 0.02539 0.02539 5.74% FillPatchIterator::Initialize 20 0.02466 0.02466 0.02466 5.57% VisMF::Write(FabArray) 1 0.02205 0.02205 0.02205 4.98% FillPatchSingleLevel 20 0.0207 0.0207 0.0207 4.68% StateDataPhysBCFunct::() 20 0.01862 0.01862 0.01862 4.21% Castro::clean_state() 30 0.01742 0.01742 0.01742 3.94% MLCellLinOp::apply() 500 0.01626 0.01626 0.01626 3.67% MLMG::mgVcycle_down::0 36 0.01584 0.01584 0.01584 3.58% FabArray::setVal() 537 0.01573 0.01573 0.01573 3.55% Castro::initialize_do_advance() 5 0.01469 0.01469 0.01469 3.32% MLMG::mgVcycle_up::0 36 0.01347 0.01347 0.01347 3.04% Castro::expand_state() 5 0.01201 0.01201 0.01201 2.71% StateData::FillBoundary(geom) 160 0.01099 0.01099 0.01099 2.48% Castro::initialize_advance() 5 0.01058 0.01058 0.01058 2.39% Castro::computeTemp() 30 0.009352 0.009352 0.009352 2.11% MLPoisson::define() 6 0.009195 0.009195 0.009195 2.08% MLCellLinOp::correctionResidual() 216 0.009185 0.009185 0.009185 2.08% Gravity::get_new_grav_vector() 5 0.009065 0.009065 0.009065 2.05% MultiFab::Dot() 484 0.009046 0.009046 0.009046 2.04% MLMG:computeResOfCorrection() 180 0.007962 0.007962 0.007962 1.80% Castro::construct_old_gravity() 5 0.007883 0.007883 0.007883 1.78% Gravity::get_old_grav_vector() 5 0.007877 0.007877 0.007877 1.78% FabArray::FillBoundary() 1766 0.007818 0.007818 0.007818 1.77% MLMG::mgVcycle_down::1 36 0.007689 0.007689 0.007689 1.74% FabArray::ParallelCopy() 380 0.007577 0.007577 0.007577 1.71% FabArray::ParallelCopy_nowait() 380 0.007436 0.007436 0.007436 1.68% FillBoundary_nowait() 1766 0.007423 0.007423 0.007423 1.68% MLMG::mgVcycle_down::2 36 0.007397 0.007397 0.007397 1.67% MLMG::mgVcycle_down::3 36 0.00701 0.00701 0.00701 1.58% Castro::do_new_sources() 5 0.006699 0.006699 0.006699 1.51% MLMG::mgVcycle_down::4 36 0.006692 0.006692 0.006692 1.51% Castro::post_restart() 1 0.006544 0.006544 0.006544 1.48% MLCellLinOp::defineAuxData() 6 0.006295 0.006295 0.006295 1.42% Gravity::multilevel_solve_for_new_phi() 1 0.006163 0.006163 0.006163 1.39% Gravity::actual_multilevel_solve() 1 0.006146 0.006146 0.006146 1.39% Gravity::fill_multipole_BCs() 6 0.00613 0.00613 0.00613 1.38% MLMG::mgVcycle_up::2 36 0.005745 0.005745 0.005745 1.30% MLMG::addInterpCorrection() 180 0.005706 0.005706 0.005706 1.29% MLMG::mgVcycle_up::1 36 0.005668 0.005668 0.005668 1.28% amrex::average_down 180 0.005512 0.005512 0.005512 1.25% MLCGSolver::ParallelAllReduce 659 0.005472 0.005472 0.005472 1.24% MLMG::mgVcycle_up::3 36 0.005433 0.005433 0.005433 1.23% CGSolver::sxay() 690 0.005427 0.005427 0.005427 1.23% MLMG::mgVcycle_up::4 36 0.005376 0.005376 0.005376 1.21% MultiFab::LinComb() 690 0.005206 0.005206 0.005206 1.18% Castro::enforce_min_density() 30 0.004732 0.004732 0.004732 1.07% MLPoisson::Fapply() 500 0.004533 0.004533 0.004533 1.02% Castro::post_timestep() 5 0.003857 0.003857 0.003857 0.87% FabArray::setDomainBndry() 20 0.003844 0.003844 0.003844 0.87% Castro::do_old_sources() 5 0.003755 0.003755 0.003755 0.85% MLMG::prepareForSolve() 6 0.003495 0.003495 0.003495 0.79% Castro::estTimeStep() 10 0.003437 0.003437 0.003437 0.78% MLCellLinOp::solutionResidual() 42 0.003375 0.003375 0.003375 0.76% MLMG::computeResidual() 36 0.002791 0.002791 0.002791 0.63% MLCellLinOp::defineBC() 6 0.002733 0.002733 0.002733 0.62% BndryData::define() 6 0.002632 0.002632 0.002632 0.59% MultiFab::Xpay() 258 0.002622 0.002622 0.002622 0.59% Castro::reset_internal_energy() 30 0.002383 0.002383 0.002383 0.54% Castro::normalize_species() 30 0.002183 0.002183 0.002183 0.49% MLMG::getGradSolution() 6 0.001745 0.001745 0.001745 0.39% MLCellLinOp::compGrad() 6 0.001742 0.001742 0.001742 0.39% Castro::construct_new_source() 25 0.001546 0.001546 0.001546 0.35% Castro::computeNewDt() 5 0.001533 0.001533 0.001533 0.35% Castro::construct_new_gravity_source() 5 0.001413 0.001413 0.001413 0.32% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.00118 0.00118 0.00118 0.27% Castro::enforce_speed_limit() 30 0.001115 0.001115 0.001115 0.25% Castro::construct_old_source() 25 0.001111 0.001111 0.001111 0.25% Castro::construct_old_gravity_source() 5 0.00106 0.00106 0.00106 0.24% Castro::apply_source_to_state() 10 0.0009261 0.0009261 0.0009261 0.21% MultiFab::Saxpy() 10 0.0009204 0.0009204 0.0009204 0.21% Gravity::swapTimeLevels() 5 0.000873 0.000873 0.000873 0.20% MLMG::ResNormInf() 42 0.0008581 0.0008581 0.0008581 0.19% FabArrayBase::getCPC() 632 0.0008574 0.0008574 0.0008574 0.19% MLCellLinOp::setLevelBC() 6 0.0008121 0.0008121 0.0008121 0.18% Gravity::update_max_rhs() 6 0.0007202 0.0007202 0.0007202 0.16% MLPoisson::prepareForSolve() 6 0.0007097 0.0007097 0.0007097 0.16% MLCellLinOp::prepareForSolve() 6 0.0007052 0.0007052 0.0007052 0.16% FabArray::mult() 22 0.0006502 0.0006502 0.0006502 0.15% MLMG::computeMLResidual() 6 0.0006032 0.0006032 0.0006032 0.14% FabArrayBase::CPC::define() 244 0.0004447 0.0004447 0.0004447 0.10% Amr::InitAmr() 1 0.0003765 0.0003765 0.0003765 0.09% FabArrayBase::getFB() 1766 0.0003699 0.0003699 0.0003699 0.08% MultiFab::Copy() 6 0.000367 0.000367 0.000367 0.08% MultiFab::min() 5 0.0001625 0.0001625 0.0001625 0.04% MLLinOp::define() 6 0.0001525 0.0001525 0.0001525 0.03% MLMG::MLResNormInf() 6 0.0001346 0.0001346 0.0001346 0.03% MultiFab::max() 6 0.0001316 0.0001316 0.0001316 0.03% MLLinOp::defineGrids() 6 0.0001251 0.0001251 0.0001251 0.03% MLMG::MLRhsNormInf() 6 0.0001051 0.0001051 0.0001051 0.02% FabArrayBase::FB::FB() 26 6.358e-05 6.358e-05 6.358e-05 0.01% Castro::create_source_corrector() 5 3.152e-05 3.152e-05 3.152e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.042e-05 3.042e-05 3.042e-05 0.01% Castro::swap_state_time_levels() 5 2.707e-05 2.707e-05 2.707e-05 0.01% Castro::finalize_advance() 5 2.682e-05 2.682e-05 2.682e-05 0.01% Amr::writeSmallPlotFile() 1 2.577e-05 2.577e-05 2.577e-05 0.01% makeSFC 30 2.197e-05 2.197e-05 2.197e-05 0.00% Castro::finalize_do_advance() 5 1.947e-05 1.947e-05 1.947e-05 0.00% DistributionMapping::Distribute() 31 9.361e-06 9.361e-06 9.361e-06 0.00% Amr::initSubcycle() 1 8.784e-06 8.784e-06 8.784e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.981e-06 4.981e-06 4.981e-06 0.00% MLMG::buildFineMask() 6 3.29e-06 3.29e-06 3.29e-06 0.00% Gravity::set_mass_offset() 6 2.688e-06 2.688e-06 2.688e-06 0.00% Castro::retry_advance_ctu() 5 1.619e-06 1.619e-06 1.619e-06 0.00% Castro::FluxRegFineAdd() 5 1.45e-06 1.45e-06 1.45e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.153e-06 1.153e-06 1.153e-06 0.00% AmrLevel::AmrLevel() 1 1.026e-06 1.026e-06 1.026e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.03-32-g1b73099a80e6) finalized