Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-10-g1a5d8917fd49) initialized Starting run at 08:26:05 UTC on 2022-04-08. Successfully read inputs file ... Castro git describe: 22.04-2-gb74783602 AMReX git describe: 22.04-10-g1a5d8917f Microphysics git describe: 22.04-1-gef526ccd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.040704533 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.024669918 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.109765468 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050992231 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.062161613 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.051273663 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.069290958 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.038769615 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.060612647 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049524903 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.059916793 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.067831457 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.071306693 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.038229056 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.0239402 seconds Ending run at 08:26:06 UTC on 2022-04-08. Run time = 0.881832133 Run time without initialization = 0.754078352 Average number of zones advanced per microsecond: 3.476 Average number of zones advanced per microsecond per rank: 3.476 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8819 ... 0.8819 ... 0.8819 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2279 0.2279 0.2279 25.85% VisMF::Write(FabArray) 11 0.159 0.159 0.159 18.03% MLCellLinOp::applyBC() 4433 0.09516 0.09516 0.09516 10.79% MLPoisson::Fsmooth() 3280 0.06089 0.06089 0.06089 6.90% StateDataPhysBCFunct::() 41 0.03447 0.03447 0.03447 3.91% FabArray::setVal() 1144 0.02627 0.02627 0.02627 2.98% StateData::FillBoundary(geom) 328 0.02403 0.02403 0.02403 2.73% MLCGSolver::bicgstab 82 0.02238 0.02238 0.02238 2.54% MultiFab::Dot() 1114 0.02204 0.02204 0.02204 2.50% FillBoundary_nowait() 4023 0.01801 0.01801 0.01801 2.04% FabArray::ParallelCopy_nowait() 861 0.01633 0.01633 0.01633 1.85% Castro::computeTemp() 63 0.01346 0.01346 0.01346 1.53% MultiFab::LinComb() 1586 0.01265 0.01265 0.01265 1.43% Gravity::fill_multipole_BCs() 11 0.01173 0.01173 0.01173 1.33% Castro::enforce_min_density() 62 0.01112 0.01112 0.01112 1.26% MLPoisson::Fapply() 1142 0.01086 0.01086 0.01086 1.23% MLCellLinOp::defineAuxData() 11 0.01047 0.01047 0.01047 1.19% MLMG::addInterpCorrection() 410 0.007162 0.007162 0.007162 0.81% amrex::average_down 410 0.006729 0.006729 0.006729 0.76% MultiFab::Xpay() 585 0.006196 0.006196 0.006196 0.70% FabArray::setDomainBndry() 41 0.005559 0.005559 0.005559 0.63% Castro::reset_internal_energy() 63 0.005439 0.005439 0.005439 0.62% Castro::estTimeStep() 21 0.005235 0.005235 0.005235 0.59% Castro::expand_state() 10 0.004772 0.004772 0.004772 0.54% Castro::normalize_species() 62 0.004484 0.004484 0.004484 0.51% Amr::checkPoint() 3 0.004113 0.004113 0.004113 0.47% BndryData::define() 11 0.003793 0.003793 0.003793 0.43% Castro::enforce_speed_limit() 62 0.003528 0.003528 0.003528 0.40% Castro::do_advance_ctu() 10 0.00326 0.00326 0.00326 0.37% Amr::writePlotFile() 2 0.0032 0.0032 0.0032 0.36% Castro::construct_new_gravity_source() 10 0.00306 0.00306 0.00306 0.35% Gravity::get_new_grav_vector() 11 0.002733 0.002733 0.002733 0.31% Castro::construct_old_gravity_source() 10 0.002392 0.002392 0.002392 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002314 0.002314 0.002314 0.26% MLCellLinOp::compGrad() 11 0.002052 0.002052 0.002052 0.23% MLMG::ResNormInf() 93 0.001965 0.001965 0.001965 0.22% MultiFab::Saxpy() 20 0.001815 0.001815 0.001815 0.21% Gravity::get_old_grav_vector() 10 0.001767 0.001767 0.001767 0.20% MLMG::oneIter() 82 0.001724 0.001724 0.001724 0.20% MLCellLinOp::setLevelBC() 11 0.001562 0.001562 0.001562 0.18% Gravity::actual_solve_with_mlmg() 11 0.001405 0.001405 0.001405 0.16% MLCellLinOp::prepareForSolve() 11 0.001369 0.001369 0.001369 0.16% FabArray::mult() 43 0.001329 0.001329 0.001329 0.15% MLCellLinOp::smooth() 1640 0.001221 0.001221 0.001221 0.14% Castro::initData() 1 0.001186 0.001186 0.001186 0.13% FabArrayBase::getCPC() 1323 0.0008497 0.0008497 0.0008497 0.10% FabArray::FillBoundary() 4023 0.0008451 0.0008451 0.0008451 0.10% MLMG::prepareForSolve() 11 0.0007707 0.0007707 0.0007707 0.09% FabArrayBase::CPC::define() 454 0.0007263 0.0007263 0.0007263 0.08% FabArrayBase::getFB() 4023 0.0006771 0.0006771 0.0006771 0.08% Gravity::update_max_rhs() 11 0.0005592 0.0005592 0.0005592 0.06% MultiFab::Copy() 11 0.0005305 0.0005305 0.0005305 0.06% MLCellLinOp::apply() 1142 0.0005224 0.0005224 0.0005224 0.06% CGSolver::sxay() 1586 0.0004848 0.0004848 0.0004848 0.05% Gravity::solve_for_phi() 10 0.0004584 0.0004584 0.0004584 0.05% Amr::InitAmr() 1 0.0004492 0.0004492 0.0004492 0.05% MLMG::mgVcycle() 82 0.0004256 0.0004256 0.0004256 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003356 0.0003356 0.0003356 0.04% MultiFab::min() 10 0.0003294 0.0003294 0.0003294 0.04% FabArray::ParallelCopy() 861 0.0002938 0.0002938 0.0002938 0.03% main() 1 0.000276 0.000276 0.000276 0.03% Castro::construct_new_gravity() 10 0.0002662 0.0002662 0.0002662 0.03% FillPatchIterator::Initialize 41 0.0002577 0.0002577 0.0002577 0.03% MultiFab::max() 11 0.0002566 0.0002566 0.0002566 0.03% MLCellLinOp::correctionResidual() 492 0.0002463 0.0002463 0.0002463 0.03% Gravity::actual_multilevel_solve() 1 0.0002158 0.0002158 0.0002158 0.02% Amr::coarseTimeStep() 10 0.0002008 0.0002008 0.0002008 0.02% MLMG::MLRhsNormInf() 11 0.0002003 0.0002003 0.0002003 0.02% Amr::timeStep() 10 0.0001837 0.0001837 0.0001837 0.02% MLCellLinOp::defineBC() 11 0.0001813 0.0001813 0.0001813 0.02% MLLinOp::defineGrids() 11 0.0001694 0.0001694 0.0001694 0.02% Castro::subcycle_advance_ctu() 10 0.0001652 0.0001652 0.0001652 0.02% Amr::defBaseLevel() 1 0.0001454 0.0001454 0.0001454 0.02% MLMG:computeResOfCorrection() 410 0.0001318 0.0001318 0.0001318 0.01% StateData::checkPoint() 12 0.0001257 0.0001257 0.0001257 0.01% MLMG::actualBottomSolve() 82 0.0001137 0.0001137 0.0001137 0.01% MLMG::mgVcycle_down::0 82 0.000105 0.000105 0.000105 0.01% Castro::construct_new_source() 50 0.0001007 0.0001007 0.0001007 0.01% MLMG::mgVcycle_down::1 82 0.0001006 0.0001006 0.0001006 0.01% MLMG::mgVcycle_down::2 82 9.21e-05 9.21e-05 9.21e-05 0.01% MLMG::mgVcycle_down::4 82 8.86e-05 8.86e-05 8.86e-05 0.01% MLMG::mgVcycle_down::3 82 8.824e-05 8.824e-05 8.824e-05 0.01% FabArrayBase::FB::FB() 56 8.807e-05 8.807e-05 8.807e-05 0.01% Castro::advance() 10 8.572e-05 8.572e-05 8.572e-05 0.01% MLMG::solve() 11 8.068e-05 8.068e-05 8.068e-05 0.01% Castro::initialize_advance() 10 7.758e-05 7.758e-05 7.758e-05 0.01% Castro::clean_state() 62 7.565e-05 7.565e-05 7.565e-05 0.01% Castro::post_timestep() 10 7.364e-05 7.364e-05 7.364e-05 0.01% AmrLevel::checkPoint() 3 7.008e-05 7.008e-05 7.008e-05 0.01% MLMG::mgVcycle_up::4 82 6.948e-05 6.948e-05 6.948e-05 0.01% MLMG::mgVcycle_up::0 82 5.932e-05 5.932e-05 5.932e-05 0.01% MLMG::mgVcycle_up::1 82 5.84e-05 5.84e-05 5.84e-05 0.01% MLMG::mgVcycle_up::3 82 5.798e-05 5.798e-05 5.798e-05 0.01% Castro::finalize_advance() 10 5.691e-05 5.691e-05 5.691e-05 0.01% Castro::initialize_do_advance() 10 5.634e-05 5.634e-05 5.634e-05 0.01% MLMG::mgVcycle_up::2 82 5.592e-05 5.592e-05 5.592e-05 0.01% MLCellLinOp::solutionResidual() 93 5.573e-05 5.573e-05 5.573e-05 0.01% StateData::define() 4 4.129e-05 4.129e-05 4.129e-05 0.00% Castro::finalize_do_advance() 10 3.847e-05 3.847e-05 3.847e-05 0.00% MLMG::mgVcycle_bottom 82 3.743e-05 3.743e-05 3.743e-05 0.00% MLMG::computeResidual() 82 3.696e-05 3.696e-05 3.696e-05 0.00% Castro::swap_state_time_levels() 10 3.688e-05 3.688e-05 3.688e-05 0.00% Castro::enforce_consistent_e() 1 3.621e-05 3.621e-05 3.621e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.095e-05 3.095e-05 3.095e-05 0.00% MLLinOp::define() 11 2.888e-05 2.888e-05 2.888e-05 0.00% FillPatchSingleLevel 41 2.887e-05 2.887e-05 2.887e-05 0.00% makeSFC 55 2.592e-05 2.592e-05 2.592e-05 0.00% MLPoisson::define() 11 2.408e-05 2.408e-05 2.408e-05 0.00% Amr::writeSmallPlotFile() 1 2.363e-05 2.363e-05 2.363e-05 0.00% Castro::construct_old_source() 50 2.151e-05 2.151e-05 2.151e-05 0.00% Amr::FinalizeInit() 1 2.056e-05 2.056e-05 2.056e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.739e-05 1.739e-05 1.739e-05 0.00% Castro::do_new_sources() 10 1.729e-05 1.729e-05 1.729e-05 0.00% Castro::do_old_sources() 10 1.617e-05 1.617e-05 1.617e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.58e-05 1.58e-05 1.58e-05 0.00% DistributionMapping::Distribute() 56 1.517e-05 1.517e-05 1.517e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.454e-05 1.454e-05 1.454e-05 0.00% Castro::apply_source_to_state() 20 1.193e-05 1.193e-05 1.193e-05 0.00% Castro::construct_old_gravity() 10 1.103e-05 1.103e-05 1.103e-05 0.00% Amr::initSubcycle() 1 9.389e-06 9.389e-06 9.389e-06 0.00% Gravity::swapTimeLevels() 10 9.249e-06 9.249e-06 9.249e-06 0.00% MLPoisson::prepareForSolve() 11 8.543e-06 8.543e-06 8.543e-06 0.00% MLMG::computeMLResidual() 11 7.27e-06 7.27e-06 7.27e-06 0.00% Castro::computeNewDt() 9 6.776e-06 6.776e-06 6.776e-06 0.00% Castro::create_source_corrector() 10 6.304e-06 6.304e-06 6.304e-06 0.00% Amr::InitializeInit() 1 6.238e-06 6.238e-06 6.238e-06 0.00% MLMG::getGradSolution() 11 6.006e-06 6.006e-06 6.006e-06 0.00% MLMG::buildFineMask() 11 5.584e-06 5.584e-06 5.584e-06 0.00% MLMG::MLResNormInf() 11 4.781e-06 4.781e-06 4.781e-06 0.00% AmrLevel::checkPointPost() 3 4.7e-06 4.7e-06 4.7e-06 0.00% Gravity::set_mass_offset() 11 4.44e-06 4.44e-06 4.44e-06 0.00% Castro::post_init() 1 4.149e-06 4.149e-06 4.149e-06 0.00% Castro::retry_advance_ctu() 10 3.666e-06 3.666e-06 3.666e-06 0.00% Castro::FluxRegFineAdd() 10 3.459e-06 3.459e-06 3.459e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.979e-06 2.979e-06 2.979e-06 0.00% Castro::computeInitialDt() 2 2.637e-06 2.637e-06 2.637e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.552e-06 2.552e-06 2.552e-06 0.00% Amr::init() 1 2.456e-06 2.456e-06 2.456e-06 0.00% AmrLevel::checkPointPre() 3 1.998e-06 1.998e-06 1.998e-06 0.00% Castro::post_regrid() 1 1.19e-06 1.19e-06 1.19e-06 0.00% Amr::initialInit() 1 9.62e-07 9.62e-07 9.62e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8818 0.8818 0.8818 100.00% Amr::coarseTimeStep() 10 0.7299 0.7299 0.7299 82.77% Amr::timeStep() 10 0.65 0.65 0.65 73.70% Castro::advance() 10 0.6423 0.6423 0.6423 72.83% Castro::subcycle_advance_ctu() 10 0.6283 0.6283 0.6283 71.25% Castro::do_advance_ctu() 10 0.6282 0.6282 0.6282 71.23% Gravity::solve_phi_with_mlmg() 11 0.3322 0.3322 0.3322 37.67% Gravity::actual_solve_with_mlmg() 11 0.3203 0.3203 0.3203 36.32% Castro::construct_new_gravity() 10 0.3072 0.3072 0.3072 34.83% MLMG::solve() 11 0.297 0.297 0.297 33.67% Gravity::solve_for_phi() 10 0.2834 0.2834 0.2834 32.13% MLMG::oneIter() 82 0.2808 0.2808 0.2808 31.84% MLMG::mgVcycle() 82 0.2791 0.2791 0.2791 31.65% Castro::construct_ctu_hydro_source() 10 0.2279 0.2279 0.2279 25.85% VisMF::Write(FabArray) 11 0.159 0.159 0.159 18.03% MLCellLinOp::smooth() 1640 0.1473 0.1473 0.1473 16.71% Amr::init() 1 0.1272 0.1272 0.1272 14.42% Amr::checkPoint() 3 0.1178 0.1178 0.1178 13.36% MLCellLinOp::applyBC() 4433 0.1148 0.1148 0.1148 13.02% AmrLevel::checkPoint() 3 0.1137 0.1137 0.1137 12.89% StateData::checkPoint() 12 0.1136 0.1136 0.1136 12.88% MLMG::mgVcycle_bottom 82 0.07985 0.07985 0.07985 9.05% MLMG::actualBottomSolve() 82 0.07981 0.07981 0.07981 9.05% MLCGSolver::bicgstab 82 0.07905 0.07905 0.07905 8.96% FillPatchIterator::Initialize 41 0.06853 0.06853 0.06853 7.77% FillPatchSingleLevel 41 0.06271 0.06271 0.06271 7.11% Amr::initialInit() 1 0.06169 0.06169 0.06169 7.00% MLPoisson::Fsmooth() 3280 0.06089 0.06089 0.06089 6.90% StateDataPhysBCFunct::() 41 0.0585 0.0585 0.0585 6.63% Amr::FinalizeInit() 1 0.05273 0.05273 0.05273 5.98% Castro::post_init() 1 0.05197 0.05197 0.05197 5.89% Gravity::multilevel_solve_for_new_phi() 1 0.04957 0.04957 0.04957 5.62% Gravity::actual_multilevel_solve() 1 0.04955 0.04955 0.04955 5.62% Amr::writePlotFile() 2 0.04871 0.04871 0.04871 5.52% MLCellLinOp::apply() 1142 0.04038 0.04038 0.04038 4.58% MLMG::mgVcycle_down::0 82 0.03874 0.03874 0.03874 4.39% Castro::initialize_do_advance() 10 0.03774 0.03774 0.03774 4.28% Castro::clean_state() 62 0.03742 0.03742 0.03742 4.24% MLMG::mgVcycle_up::0 82 0.03326 0.03326 0.03326 3.77% Castro::expand_state() 10 0.03187 0.03187 0.03187 3.61% FabArray::setVal() 1144 0.02627 0.02627 0.02627 2.98% Gravity::get_new_grav_vector() 11 0.02576 0.02576 0.02576 2.92% StateData::FillBoundary(geom) 328 0.02403 0.02403 0.02403 2.73% MLCellLinOp::correctionResidual() 492 0.02277 0.02277 0.02277 2.58% MultiFab::Dot() 1114 0.02204 0.02204 0.02204 2.50% MLMG:computeResOfCorrection() 410 0.01968 0.01968 0.01968 2.23% FabArray::FillBoundary() 4023 0.01962 0.01962 0.01962 2.22% MLMG::mgVcycle_down::1 82 0.01896 0.01896 0.01896 2.15% Castro::computeTemp() 63 0.0189 0.0189 0.0189 2.14% FillBoundary_nowait() 4023 0.01878 0.01878 0.01878 2.13% MLMG::mgVcycle_down::2 82 0.01834 0.01834 0.01834 2.08% MLPoisson::define() 11 0.01771 0.01771 0.01771 2.01% FabArray::ParallelCopy() 861 0.01753 0.01753 0.01753 1.99% MLMG::mgVcycle_down::3 82 0.01752 0.01752 0.01752 1.99% FabArray::ParallelCopy_nowait() 861 0.01723 0.01723 0.01723 1.95% MLMG::mgVcycle_down::4 82 0.01675 0.01675 0.01675 1.90% Castro::construct_old_gravity() 10 0.01593 0.01593 0.01593 1.81% Gravity::get_old_grav_vector() 10 0.01591 0.01591 0.01591 1.80% MLMG::mgVcycle_up::2 82 0.01423 0.01423 0.01423 1.61% MLMG::mgVcycle_up::1 82 0.01405 0.01405 0.01405 1.59% MLMG::addInterpCorrection() 410 0.01382 0.01382 0.01382 1.57% Castro::initialize_advance() 10 0.01381 0.01381 0.01381 1.57% MLMG::mgVcycle_up::3 82 0.01355 0.01355 0.01355 1.54% MLMG::mgVcycle_up::4 82 0.01343 0.01343 0.01343 1.52% amrex::average_down 410 0.01341 0.01341 0.01341 1.52% MLCGSolver::ParallelAllReduce 1514 0.01326 0.01326 0.01326 1.50% CGSolver::sxay() 1586 0.01314 0.01314 0.01314 1.49% MultiFab::LinComb() 1586 0.01265 0.01265 0.01265 1.43% MLCellLinOp::defineAuxData() 11 0.01216 0.01216 0.01216 1.38% Castro::do_new_sources() 10 0.01179 0.01179 0.01179 1.34% Gravity::fill_multipole_BCs() 11 0.01173 0.01173 0.01173 1.33% Castro::enforce_min_density() 62 0.01112 0.01112 0.01112 1.26% MLPoisson::Fapply() 1142 0.01086 0.01086 0.01086 1.23% Amr::InitializeInit() 1 0.008967 0.008967 0.008967 1.02% Amr::defBaseLevel() 1 0.00896 0.00896 0.00896 1.02% Castro::do_old_sources() 10 0.00812 0.00812 0.00812 0.92% MLCellLinOp::solutionResidual() 93 0.007921 0.007921 0.007921 0.90% Castro::post_timestep() 10 0.007508 0.007508 0.007508 0.85% MLMG::computeResidual() 82 0.006854 0.006854 0.006854 0.78% MultiFab::Xpay() 585 0.006196 0.006196 0.006196 0.70% MLMG::prepareForSolve() 11 0.005921 0.005921 0.005921 0.67% FabArray::setDomainBndry() 41 0.005559 0.005559 0.005559 0.63% Castro::reset_internal_energy() 63 0.005439 0.005439 0.005439 0.62% MLCellLinOp::defineBC() 11 0.005273 0.005273 0.005273 0.60% Castro::estTimeStep() 21 0.005235 0.005235 0.005235 0.59% BndryData::define() 11 0.005092 0.005092 0.005092 0.58% Castro::normalize_species() 62 0.004484 0.004484 0.004484 0.51% Castro::enforce_speed_limit() 62 0.003528 0.003528 0.003528 0.40% Castro::initData() 1 0.003307 0.003307 0.003307 0.38% Castro::construct_new_source() 50 0.003161 0.003161 0.003161 0.36% Castro::construct_new_gravity_source() 10 0.00306 0.00306 0.00306 0.35% MLMG::getGradSolution() 11 0.002623 0.002623 0.002623 0.30% MLCellLinOp::compGrad() 11 0.002617 0.002617 0.002617 0.30% Castro::construct_old_source() 50 0.002413 0.002413 0.002413 0.27% Castro::construct_old_gravity_source() 10 0.002392 0.002392 0.002392 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002314 0.002314 0.002314 0.26% Castro::computeNewDt() 9 0.002231 0.002231 0.002231 0.25% MLMG::ResNormInf() 93 0.001965 0.001965 0.001965 0.22% Castro::apply_source_to_state() 20 0.001826 0.001826 0.001826 0.21% MultiFab::Saxpy() 20 0.001815 0.001815 0.001815 0.21% FabArrayBase::getCPC() 1323 0.001576 0.001576 0.001576 0.18% MLCellLinOp::setLevelBC() 11 0.001562 0.001562 0.001562 0.18% MLPoisson::prepareForSolve() 11 0.001377 0.001377 0.001377 0.16% MLCellLinOp::prepareForSolve() 11 0.001369 0.001369 0.001369 0.16% FabArray::mult() 43 0.001329 0.001329 0.001329 0.15% Gravity::swapTimeLevels() 10 0.001146 0.001146 0.001146 0.13% MLMG::computeMLResidual() 11 0.001111 0.001111 0.001111 0.13% Gravity::update_max_rhs() 11 0.0009606 0.0009606 0.0009606 0.11% FabArrayBase::getFB() 4023 0.0007652 0.0007652 0.0007652 0.09% FabArrayBase::CPC::define() 454 0.0007263 0.0007263 0.0007263 0.08% Castro::computeInitialDt() 2 0.0006862 0.0006862 0.0006862 0.08% MultiFab::Copy() 11 0.0005305 0.0005305 0.0005305 0.06% Castro::post_regrid() 1 0.0005069 0.0005069 0.0005069 0.06% Amr::InitAmr() 1 0.0004585 0.0004585 0.0004585 0.05% MultiFab::min() 10 0.0003294 0.0003294 0.0003294 0.04% MultiFab::max() 11 0.0002566 0.0002566 0.0002566 0.03% MLLinOp::define() 11 0.0002564 0.0002564 0.0002564 0.03% MLMG::MLResNormInf() 11 0.0002562 0.0002562 0.0002562 0.03% MLLinOp::defineGrids() 11 0.0002275 0.0002275 0.0002275 0.03% MLMG::MLRhsNormInf() 11 0.0002003 0.0002003 0.0002003 0.02% FabArrayBase::FB::FB() 56 8.807e-05 8.807e-05 8.807e-05 0.01% Castro::finalize_advance() 10 6.036e-05 6.036e-05 6.036e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.583e-05 5.583e-05 5.583e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.557e-05 5.557e-05 5.557e-05 0.01% StateData::define() 4 4.129e-05 4.129e-05 4.129e-05 0.00% makeSFC 55 3.977e-05 3.977e-05 3.977e-05 0.00% Castro::finalize_do_advance() 10 3.847e-05 3.847e-05 3.847e-05 0.00% Castro::swap_state_time_levels() 10 3.688e-05 3.688e-05 3.688e-05 0.00% Castro::enforce_consistent_e() 1 3.621e-05 3.621e-05 3.621e-05 0.00% Amr::writeSmallPlotFile() 1 2.363e-05 2.363e-05 2.363e-05 0.00% DistributionMapping::Distribute() 56 1.517e-05 1.517e-05 1.517e-05 0.00% Amr::initSubcycle() 1 9.389e-06 9.389e-06 9.389e-06 0.00% Castro::create_source_corrector() 10 6.304e-06 6.304e-06 6.304e-06 0.00% MLMG::buildFineMask() 11 5.584e-06 5.584e-06 5.584e-06 0.00% AmrLevel::checkPointPost() 3 4.7e-06 4.7e-06 4.7e-06 0.00% Gravity::set_mass_offset() 11 4.44e-06 4.44e-06 4.44e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.298e-06 4.298e-06 4.298e-06 0.00% Castro::retry_advance_ctu() 10 3.666e-06 3.666e-06 3.666e-06 0.00% Castro::FluxRegFineAdd() 10 3.459e-06 3.459e-06 3.459e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.552e-06 2.552e-06 2.552e-06 0.00% AmrLevel::checkPointPre() 3 1.998e-06 1.998e-06 1.998e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-10-g1a5d8917fd49) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-10-g1a5d8917fd49) initialized Starting run at 08:26:07 UTC on 2022-04-08. Successfully read inputs file ... Castro git describe: 22.04-2-gb74783602 AMReX git describe: 22.04-10-g1a5d8917f Microphysics git describe: 22.04-1-gef526ccd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.509939792 Restart time = 0.055537992 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.111576679 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049326907 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.049257997 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059372633 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.064593061 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024278071 seconds Ending run at 08:26:07 UTC on 2022-04-08. Run time = 0.414877964 Run time without initialization = 0.358737396 Average number of zones advanced per microsecond: 3.654 Average number of zones advanced per microsecond per rank: 3.654 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4149 ... 0.4149 ... 0.4149 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1248 0.1248 0.1248 30.07% VisMF::Read() 3 0.04383 0.04383 0.04383 10.56% MLCellLinOp::applyBC() 1946 0.03792 0.03792 0.03792 9.14% MLPoisson::Fsmooth() 1440 0.02583 0.02583 0.02583 6.23% VisMF::Write(FabArray) 1 0.02295 0.02295 0.02295 5.53% FabArray::setVal() 537 0.01695 0.01695 0.01695 4.09% StateData::FillBoundary(geom) 160 0.01158 0.01158 0.01158 2.79% MLCGSolver::bicgstab 36 0.009513 0.009513 0.009513 2.29% MultiFab::Dot() 484 0.009339 0.009339 0.009339 2.25% Castro::computeTemp() 30 0.007297 0.007297 0.007297 1.76% FillBoundary_nowait() 1766 0.007237 0.007237 0.007237 1.74% FabArray::ParallelCopy_nowait() 380 0.007154 0.007154 0.007154 1.72% StateDataPhysBCFunct::() 20 0.007099 0.007099 0.007099 1.71% Gravity::fill_multipole_BCs() 6 0.006164 0.006164 0.006164 1.49% MLCellLinOp::defineAuxData() 6 0.005582 0.005582 0.005582 1.35% MultiFab::LinComb() 690 0.005422 0.005422 0.005422 1.31% Castro::enforce_min_density() 30 0.00521 0.00521 0.00521 1.26% MLPoisson::Fapply() 500 0.004697 0.004697 0.004697 1.13% Castro::expand_state() 5 0.003928 0.003928 0.003928 0.95% FabArray::setDomainBndry() 20 0.00375 0.00375 0.00375 0.90% Castro::reset_internal_energy() 30 0.003288 0.003288 0.003288 0.79% MLMG::addInterpCorrection() 180 0.003027 0.003027 0.003027 0.73% Amr::restart() 1 0.003012 0.003012 0.003012 0.73% amrex::average_down 180 0.002833 0.002833 0.002833 0.68% MultiFab::Xpay() 258 0.002726 0.002726 0.002726 0.66% Castro::do_advance_ctu() 5 0.002513 0.002513 0.002513 0.61% Castro::estTimeStep() 10 0.002434 0.002434 0.002434 0.59% Castro::normalize_species() 30 0.002257 0.002257 0.002257 0.54% BndryData::define() 6 0.002091 0.002091 0.002091 0.50% Gravity::get_new_grav_vector() 5 0.001738 0.001738 0.001738 0.42% Castro::construct_new_gravity_source() 5 0.001519 0.001519 0.001519 0.37% MLCellLinOp::compGrad() 6 0.001464 0.001464 0.001464 0.35% Amr::writePlotFile() 1 0.001405 0.001405 0.001405 0.34% Castro::construct_old_gravity_source() 5 0.001332 0.001332 0.001332 0.32% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001237 0.001237 0.001237 0.30% Castro::enforce_speed_limit() 30 0.001199 0.001199 0.001199 0.29% MultiFab::Saxpy() 10 0.0009203 0.0009203 0.0009203 0.22% Gravity::get_old_grav_vector() 5 0.0009131 0.0009131 0.0009131 0.22% MLMG::ResNormInf() 42 0.0008663 0.0008663 0.0008663 0.21% MLCellLinOp::setLevelBC() 6 0.0008381 0.0008381 0.0008381 0.20% MLMG::oneIter() 36 0.0007569 0.0007569 0.0007569 0.18% Gravity::actual_solve_with_mlmg() 6 0.0007118 0.0007118 0.0007118 0.17% MLCellLinOp::prepareForSolve() 6 0.0006985 0.0006985 0.0006985 0.17% FabArray::mult() 22 0.0006561 0.0006561 0.0006561 0.16% Gravity::update_max_rhs() 6 0.0005238 0.0005238 0.0005238 0.13% MLCellLinOp::smooth() 720 0.000485 0.000485 0.000485 0.12% FabArrayBase::CPC::define() 244 0.0004528 0.0004528 0.0004528 0.11% MLMG::prepareForSolve() 6 0.0004236 0.0004236 0.0004236 0.10% Amr::InitAmr() 1 0.0004181 0.0004181 0.0004181 0.10% FabArrayBase::getCPC() 632 0.0004156 0.0004156 0.0004156 0.10% FabArray::FillBoundary() 1766 0.0004103 0.0004103 0.0004103 0.10% MultiFab::Copy() 6 0.000381 0.000381 0.000381 0.09% FabArrayBase::getFB() 1766 0.0003281 0.0003281 0.0003281 0.08% main() 1 0.0002663 0.0002663 0.0002663 0.06% Gravity::solve_for_phi() 5 0.0002352 0.0002352 0.0002352 0.06% Gravity::actual_multilevel_solve() 1 0.0002234 0.0002234 0.0002234 0.05% MLCellLinOp::apply() 500 0.0002138 0.0002138 0.0002138 0.05% CGSolver::sxay() 690 0.0002048 0.0002048 0.0002048 0.05% Castro::construct_new_gravity() 5 0.0001979 0.0001979 0.0001979 0.05% MLMG::mgVcycle() 36 0.0001908 0.0001908 0.0001908 0.05% MultiFab::min() 5 0.0001586 0.0001586 0.0001586 0.04% FabArray::ParallelCopy() 380 0.0001453 0.0001453 0.0001453 0.04% Castro::construct_new_source() 25 0.0001407 0.0001407 0.0001407 0.03% MLCGSolver::ParallelAllReduce 659 0.0001351 0.0001351 0.0001351 0.03% MultiFab::max() 6 0.0001338 0.0001338 0.0001338 0.03% FillPatchIterator::Initialize 20 0.0001222 0.0001222 0.0001222 0.03% MLMG::MLRhsNormInf() 6 0.0001076 0.0001076 0.0001076 0.03% Amr::coarseTimeStep() 5 0.0001033 0.0001033 0.0001033 0.02% MLCellLinOp::correctionResidual() 216 9.78e-05 9.78e-05 9.78e-05 0.02% MLCellLinOp::defineBC() 6 9.631e-05 9.631e-05 9.631e-05 0.02% Amr::timeStep() 5 9.18e-05 9.18e-05 9.18e-05 0.02% MLLinOp::defineGrids() 6 9.172e-05 9.172e-05 9.172e-05 0.02% AmrLevel::restart() 1 7.431e-05 7.431e-05 7.431e-05 0.02% Castro::advance() 5 7.309e-05 7.309e-05 7.309e-05 0.02% StateData::restartDoit() 4 7.169e-05 7.169e-05 7.169e-05 0.02% Castro::subcycle_advance_ctu() 5 6.431e-05 6.431e-05 6.431e-05 0.02% MLMG:computeResOfCorrection() 180 6.096e-05 6.096e-05 6.096e-05 0.01% Castro::post_timestep() 5 5.958e-05 5.958e-05 5.958e-05 0.01% FabArrayBase::FB::FB() 26 5.952e-05 5.952e-05 5.952e-05 0.01% Castro::construct_old_source() 25 5.283e-05 5.283e-05 5.283e-05 0.01% MLMG::actualBottomSolve() 36 4.889e-05 4.889e-05 4.889e-05 0.01% Castro::initialize_advance() 5 4.705e-05 4.705e-05 4.705e-05 0.01% Castro::create_source_corrector() 5 4.404e-05 4.404e-05 4.404e-05 0.01% MLMG::mgVcycle_down::0 36 3.869e-05 3.869e-05 3.869e-05 0.01% Castro::do_new_sources() 5 3.837e-05 3.837e-05 3.837e-05 0.01% MLMG::mgVcycle_down::2 36 3.732e-05 3.732e-05 3.732e-05 0.01% MLMG::solve() 6 3.677e-05 3.677e-05 3.677e-05 0.01% MLMG::mgVcycle_down::1 36 3.677e-05 3.677e-05 3.677e-05 0.01% Castro::clean_state() 30 3.632e-05 3.632e-05 3.632e-05 0.01% MLMG::mgVcycle_down::4 36 3.276e-05 3.276e-05 3.276e-05 0.01% MLMG::mgVcycle_down::3 36 3.227e-05 3.227e-05 3.227e-05 0.01% Castro::initialize_do_advance() 5 2.872e-05 2.872e-05 2.872e-05 0.01% MLLinOp::define() 6 2.729e-05 2.729e-05 2.729e-05 0.01% MLMG::mgVcycle_up::4 36 2.696e-05 2.696e-05 2.696e-05 0.01% Castro::swap_state_time_levels() 5 2.536e-05 2.536e-05 2.536e-05 0.01% Amr::writeSmallPlotFile() 1 2.491e-05 2.491e-05 2.491e-05 0.01% Castro::finalize_advance() 5 2.489e-05 2.489e-05 2.489e-05 0.01% Castro::post_restart() 1 2.405e-05 2.405e-05 2.405e-05 0.01% MLCellLinOp::solutionResidual() 42 2.392e-05 2.392e-05 2.392e-05 0.01% MLMG::mgVcycle_up::0 36 2.243e-05 2.243e-05 2.243e-05 0.01% MLMG::mgVcycle_up::3 36 2.128e-05 2.128e-05 2.128e-05 0.01% Castro::finalize_do_advance() 5 2.11e-05 2.11e-05 2.11e-05 0.01% MLMG::mgVcycle_up::2 36 2.037e-05 2.037e-05 2.037e-05 0.00% MLMG::mgVcycle_up::1 36 1.936e-05 1.936e-05 1.936e-05 0.00% MLMG::computeResidual() 36 1.678e-05 1.678e-05 1.678e-05 0.00% MLMG::mgVcycle_bottom 36 1.638e-05 1.638e-05 1.638e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.569e-05 1.569e-05 1.569e-05 0.00% MLPoisson::define() 6 1.554e-05 1.554e-05 1.554e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.507e-05 1.507e-05 1.507e-05 0.00% FillPatchSingleLevel 20 1.452e-05 1.452e-05 1.452e-05 0.00% makeSFC 30 1.361e-05 1.361e-05 1.361e-05 0.00% DistributionMapping::Distribute() 31 1.04e-05 1.04e-05 1.04e-05 0.00% Amr::initSubcycle() 1 9.992e-06 9.992e-06 9.992e-06 0.00% Castro::do_old_sources() 5 8.225e-06 8.225e-06 8.225e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.175e-06 8.175e-06 8.175e-06 0.00% Castro::apply_source_to_state() 10 6.101e-06 6.101e-06 6.101e-06 0.00% Castro::construct_old_gravity() 5 5.829e-06 5.829e-06 5.829e-06 0.00% MLPoisson::prepareForSolve() 6 4.719e-06 4.719e-06 4.719e-06 0.00% Gravity::swapTimeLevels() 5 4.21e-06 4.21e-06 4.21e-06 0.00% Castro::computeNewDt() 5 3.472e-06 3.472e-06 3.472e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.327e-06 3.327e-06 3.327e-06 0.00% MLMG::computeMLResidual() 6 3.233e-06 3.233e-06 3.233e-06 0.00% MLMG::buildFineMask() 6 3.054e-06 3.054e-06 3.054e-06 0.00% MLMG::getGradSolution() 6 3.023e-06 3.023e-06 3.023e-06 0.00% MLMG::MLResNormInf() 6 2.565e-06 2.565e-06 2.565e-06 0.00% Gravity::set_mass_offset() 6 2.486e-06 2.486e-06 2.486e-06 0.00% Castro::retry_advance_ctu() 5 1.973e-06 1.973e-06 1.973e-06 0.00% Castro::FluxRegFineAdd() 5 1.628e-06 1.628e-06 1.628e-06 0.00% Amr::init() 1 1.325e-06 1.325e-06 1.325e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.318e-06 1.318e-06 1.318e-06 0.00% AmrLevel::AmrLevel() 1 1.073e-06 1.073e-06 1.073e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4149 0.4149 0.4149 100.00% Amr::coarseTimeStep() 5 0.3342 0.3342 0.3342 80.55% Amr::timeStep() 5 0.3329 0.3329 0.3329 80.24% Castro::advance() 5 0.3292 0.3292 0.3292 79.35% Castro::subcycle_advance_ctu() 5 0.3184 0.3184 0.3184 76.75% Castro::do_advance_ctu() 5 0.3184 0.3184 0.3184 76.74% Castro::construct_new_gravity() 5 0.1476 0.1476 0.1476 35.57% Gravity::solve_phi_with_mlmg() 6 0.1437 0.1437 0.1437 34.62% Gravity::solve_for_phi() 5 0.138 0.138 0.138 33.27% Gravity::actual_solve_with_mlmg() 6 0.1374 0.1374 0.1374 33.11% Castro::construct_ctu_hydro_source() 5 0.1248 0.1248 0.1248 30.07% MLMG::solve() 6 0.1244 0.1244 0.1244 29.99% MLMG::oneIter() 36 0.1165 0.1165 0.1165 28.07% MLMG::mgVcycle() 36 0.1157 0.1157 0.1157 27.89% MLCellLinOp::smooth() 720 0.06024 0.06024 0.06024 14.52% Amr::init() 1 0.05559 0.05559 0.05559 13.40% Amr::restart() 1 0.05559 0.05559 0.05559 13.40% MLCellLinOp::applyBC() 1946 0.04596 0.04596 0.04596 11.08% AmrLevel::restart() 1 0.04473 0.04473 0.04473 10.78% StateData::restartDoit() 4 0.04465 0.04465 0.04465 10.76% VisMF::Read() 3 0.04383 0.04383 0.04383 10.56% MLMG::mgVcycle_bottom 36 0.03366 0.03366 0.03366 8.11% MLMG::actualBottomSolve() 36 0.03365 0.03365 0.03365 8.11% MLCGSolver::bicgstab 36 0.03332 0.03332 0.03332 8.03% MLPoisson::Fsmooth() 1440 0.02583 0.02583 0.02583 6.23% FillPatchIterator::Initialize 20 0.02465 0.02465 0.02465 5.94% Amr::writePlotFile() 1 0.02436 0.02436 0.02436 5.87% VisMF::Write(FabArray) 1 0.02295 0.02295 0.02295 5.53% FillPatchSingleLevel 20 0.02078 0.02078 0.02078 5.01% Castro::clean_state() 30 0.01929 0.01929 0.01929 4.65% StateDataPhysBCFunct::() 20 0.01868 0.01868 0.01868 4.50% FabArray::setVal() 537 0.01695 0.01695 0.01695 4.09% MLCellLinOp::apply() 500 0.01665 0.01665 0.01665 4.01% MLMG::mgVcycle_down::0 36 0.01609 0.01609 0.01609 3.88% Castro::initialize_do_advance() 5 0.01444 0.01444 0.01444 3.48% MLMG::mgVcycle_up::0 36 0.01367 0.01367 0.01367 3.29% Castro::expand_state() 5 0.01165 0.01165 0.01165 2.81% StateData::FillBoundary(geom) 160 0.01158 0.01158 0.01158 2.79% Castro::initialize_advance() 5 0.01069 0.01069 0.01069 2.58% Castro::computeTemp() 30 0.01059 0.01059 0.01059 2.55% MLPoisson::define() 6 0.009622 0.009622 0.009622 2.32% MLCellLinOp::correctionResidual() 216 0.009463 0.009463 0.009463 2.28% Gravity::get_new_grav_vector() 5 0.009345 0.009345 0.009345 2.25% MultiFab::Dot() 484 0.009339 0.009339 0.009339 2.25% MLMG:computeResOfCorrection() 180 0.008193 0.008193 0.008193 1.97% FabArray::FillBoundary() 1766 0.008035 0.008035 0.008035 1.94% Castro::construct_old_gravity() 5 0.007978 0.007978 0.007978 1.92% Gravity::get_old_grav_vector() 5 0.007972 0.007972 0.007972 1.92% Castro::do_new_sources() 5 0.007855 0.007855 0.007855 1.89% MLMG::mgVcycle_down::1 36 0.007851 0.007851 0.007851 1.89% FabArray::ParallelCopy() 380 0.007718 0.007718 0.007718 1.86% FillBoundary_nowait() 1766 0.007625 0.007625 0.007625 1.84% FabArray::ParallelCopy_nowait() 380 0.007572 0.007572 0.007572 1.83% MLMG::mgVcycle_down::2 36 0.007538 0.007538 0.007538 1.82% MLMG::mgVcycle_down::3 36 0.007168 0.007168 0.007168 1.73% MLMG::mgVcycle_down::4 36 0.006876 0.006876 0.006876 1.66% MLCellLinOp::defineAuxData() 6 0.006528 0.006528 0.006528 1.57% Castro::post_restart() 1 0.00649 0.00649 0.00649 1.56% Gravity::fill_multipole_BCs() 6 0.006164 0.006164 0.006164 1.49% Gravity::multilevel_solve_for_new_phi() 1 0.006109 0.006109 0.006109 1.47% Gravity::actual_multilevel_solve() 1 0.006093 0.006093 0.006093 1.47% MLMG::addInterpCorrection() 180 0.005841 0.005841 0.005841 1.41% MLMG::mgVcycle_up::2 36 0.005831 0.005831 0.005831 1.41% MLMG::mgVcycle_up::1 36 0.005771 0.005771 0.005771 1.39% amrex::average_down 180 0.00565 0.00565 0.00565 1.36% MLCGSolver::ParallelAllReduce 659 0.005639 0.005639 0.005639 1.36% CGSolver::sxay() 690 0.005627 0.005627 0.005627 1.36% MLMG::mgVcycle_up::3 36 0.005548 0.005548 0.005548 1.34% MLMG::mgVcycle_up::4 36 0.005497 0.005497 0.005497 1.32% MultiFab::LinComb() 690 0.005422 0.005422 0.005422 1.31% Castro::enforce_min_density() 30 0.00521 0.00521 0.00521 1.26% MLPoisson::Fapply() 500 0.004697 0.004697 0.004697 1.13% Castro::do_old_sources() 5 0.004427 0.004427 0.004427 1.07% FabArray::setDomainBndry() 20 0.00375 0.00375 0.00375 0.90% Castro::post_timestep() 5 0.003601 0.003601 0.003601 0.87% MLMG::prepareForSolve() 6 0.003517 0.003517 0.003517 0.85% MLCellLinOp::solutionResidual() 42 0.003422 0.003422 0.003422 0.82% Castro::reset_internal_energy() 30 0.003288 0.003288 0.003288 0.79% MLCellLinOp::defineBC() 6 0.002928 0.002928 0.002928 0.71% MLMG::computeResidual() 36 0.00284 0.00284 0.00284 0.68% BndryData::define() 6 0.002831 0.002831 0.002831 0.68% MultiFab::Xpay() 258 0.002726 0.002726 0.002726 0.66% Castro::estTimeStep() 10 0.002434 0.002434 0.002434 0.59% Castro::normalize_species() 30 0.002257 0.002257 0.002257 0.54% MLMG::getGradSolution() 6 0.001761 0.001761 0.001761 0.42% MLCellLinOp::compGrad() 6 0.001758 0.001758 0.001758 0.42% Castro::construct_new_source() 25 0.00166 0.00166 0.00166 0.40% Castro::construct_new_gravity_source() 5 0.001519 0.001519 0.001519 0.37% Castro::construct_old_source() 25 0.001385 0.001385 0.001385 0.33% Castro::construct_old_gravity_source() 5 0.001332 0.001332 0.001332 0.32% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001237 0.001237 0.001237 0.30% Castro::enforce_speed_limit() 30 0.001199 0.001199 0.001199 0.29% Castro::computeNewDt() 5 0.001191 0.001191 0.001191 0.29% Castro::apply_source_to_state() 10 0.0009264 0.0009264 0.0009264 0.22% MultiFab::Saxpy() 10 0.0009203 0.0009203 0.0009203 0.22% Gravity::swapTimeLevels() 5 0.0009021 0.0009021 0.0009021 0.22% FabArrayBase::getCPC() 632 0.0008684 0.0008684 0.0008684 0.21% MLMG::ResNormInf() 42 0.0008663 0.0008663 0.0008663 0.21% MLCellLinOp::setLevelBC() 6 0.0008381 0.0008381 0.0008381 0.20% Gravity::update_max_rhs() 6 0.0007359 0.0007359 0.0007359 0.18% MLPoisson::prepareForSolve() 6 0.0007032 0.0007032 0.0007032 0.17% MLCellLinOp::prepareForSolve() 6 0.0006985 0.0006985 0.0006985 0.17% FabArray::mult() 22 0.0006561 0.0006561 0.0006561 0.16% MLMG::computeMLResidual() 6 0.0006021 0.0006021 0.0006021 0.15% FabArrayBase::CPC::define() 244 0.0004528 0.0004528 0.0004528 0.11% Amr::InitAmr() 1 0.0004281 0.0004281 0.0004281 0.10% FabArrayBase::getFB() 1766 0.0003877 0.0003877 0.0003877 0.09% MultiFab::Copy() 6 0.000381 0.000381 0.000381 0.09% MultiFab::min() 5 0.0001586 0.0001586 0.0001586 0.04% MLLinOp::define() 6 0.0001511 0.0001511 0.0001511 0.04% MLMG::MLResNormInf() 6 0.0001367 0.0001367 0.0001367 0.03% MultiFab::max() 6 0.0001338 0.0001338 0.0001338 0.03% MLLinOp::defineGrids() 6 0.0001238 0.0001238 0.0001238 0.03% MLMG::MLRhsNormInf() 6 0.0001076 0.0001076 0.0001076 0.03% FabArrayBase::FB::FB() 26 5.952e-05 5.952e-05 5.952e-05 0.01% Castro::create_source_corrector() 5 4.404e-05 4.404e-05 4.404e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.076e-05 3.076e-05 3.076e-05 0.01% Castro::finalize_advance() 5 2.652e-05 2.652e-05 2.652e-05 0.01% Castro::swap_state_time_levels() 5 2.536e-05 2.536e-05 2.536e-05 0.01% Amr::writeSmallPlotFile() 1 2.491e-05 2.491e-05 2.491e-05 0.01% makeSFC 30 2.258e-05 2.258e-05 2.258e-05 0.01% Castro::finalize_do_advance() 5 2.11e-05 2.11e-05 2.11e-05 0.01% DistributionMapping::Distribute() 31 1.04e-05 1.04e-05 1.04e-05 0.00% Amr::initSubcycle() 1 9.992e-06 9.992e-06 9.992e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.745e-06 4.745e-06 4.745e-06 0.00% MLMG::buildFineMask() 6 3.054e-06 3.054e-06 3.054e-06 0.00% Gravity::set_mass_offset() 6 2.486e-06 2.486e-06 2.486e-06 0.00% Castro::retry_advance_ctu() 5 1.973e-06 1.973e-06 1.973e-06 0.00% Castro::FluxRegFineAdd() 5 1.628e-06 1.628e-06 1.628e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.318e-06 1.318e-06 1.318e-06 0.00% AmrLevel::AmrLevel() 1 1.073e-06 1.073e-06 1.073e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-10-g1a5d8917fd49) finalized