Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.01-12-g1404c4a9acc6) initialized Starting run at 09:06:05 UTC on 2022-01-07. Successfully read inputs file ... Castro git describe: 21.12-23-g727e39bb6 AMReX git describe: 22.01-12-g1404c4a9a Microphysics git describe: 22.01-14-geeef682d reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.042242023 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025088346 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.136220664 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.073454278 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.072826191 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.076590587 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.08414792 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.039666176 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.088659278 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.080276398 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.082114367 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.087011185 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.092474423 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.073661445 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.025248555 seconds Ending run at 09:06:07 UTC on 2022-01-07. Run time = 1.180913768 Run time without initialization = 1.013041722 Average number of zones advanced per microsecond: 2.588 Average number of zones advanced per microsecond per rank: 2.588 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489334272 TinyProfiler total time across processes [min...avg...max]: 1.181 ... 1.181 ... 1.181 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2411 0.2411 0.2411 20.42% VisMF::Write(FabArray) 11 0.1664 0.1664 0.1664 14.09% MLCellLinOp::applyBC() 4433 0.1192 0.1192 0.1192 10.09% MLPoisson::Fsmooth() 3280 0.07577 0.07577 0.07577 6.42% FillBoundary_nowait() 4023 0.04011 0.04011 0.04011 3.40% MLCGSolver::bicgstab 82 0.03838 0.03838 0.03838 3.25% Amr::checkPoint() 3 0.03603 0.03603 0.03603 3.05% FabArray::setVal() 1140 0.03078 0.03078 0.03078 2.61% MultiFab::Dot() 1114 0.02817 0.02817 0.02817 2.39% StateDataPhysBCFunct::() 41 0.02777 0.02777 0.02777 2.35% StateData::FillBoundary(geom) 328 0.02562 0.02562 0.02562 2.17% MLCellLinOp::smooth() 1640 0.02365 0.02365 0.02365 2.00% FabArray::FillBoundary() 4023 0.02298 0.02298 0.02298 1.95% FabArray::ParallelCopy_nowait() 861 0.0223 0.0223 0.0223 1.89% MultiFab::LinComb() 1586 0.02003 0.02003 0.02003 1.70% MLPoisson::Fapply() 1142 0.01615 0.01615 0.01615 1.37% MLCellLinOp::defineAuxData() 11 0.01414 0.01414 0.01414 1.20% Castro::computeTemp() 63 0.01362 0.01362 0.01362 1.15% Gravity::fill_multipole_BCs() 11 0.01331 0.01331 0.01331 1.13% FabArrayBase::getFB() 4023 0.01193 0.01193 0.01193 1.01% MLMG::addInterpCorrection() 410 0.009971 0.009971 0.009971 0.84% MLCellLinOp::apply() 1142 0.009838 0.009838 0.009838 0.83% Castro::enforce_min_density() 62 0.009558 0.009558 0.009558 0.81% amrex::average_down 410 0.009206 0.009206 0.009206 0.78% CGSolver::sxay() 1586 0.00906 0.00906 0.00906 0.77% MultiFab::Xpay() 585 0.008952 0.008952 0.008952 0.76% MLCGSolver::ParallelAllReduce 1514 0.006224 0.006224 0.006224 0.53% Castro::estTimeStep() 21 0.005889 0.005889 0.005889 0.50% FabArrayBase::getCPC() 1323 0.005771 0.005771 0.005771 0.49% FabArray::setDomainBndry() 41 0.005721 0.005721 0.005721 0.48% Castro::reset_internal_energy() 63 0.005707 0.005707 0.005707 0.48% FabArray::ParallelCopy() 861 0.005087 0.005087 0.005087 0.43% MLMG::actualBottomSolve() 82 0.004815 0.004815 0.004815 0.41% Castro::expand_state() 10 0.004659 0.004659 0.004659 0.39% BndryData::define() 11 0.004467 0.004467 0.004467 0.38% Castro::enforce_speed_limit() 62 0.004365 0.004365 0.004365 0.37% MLCellLinOp::correctionResidual() 492 0.004249 0.004249 0.004249 0.36% Castro::do_advance_ctu() 10 0.003876 0.003876 0.003876 0.33% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.003722 0.003722 0.003722 0.32% Amr::writePlotFile() 2 0.003447 0.003447 0.003447 0.29% Castro::construct_new_gravity_source() 10 0.003224 0.003224 0.003224 0.27% MLMG::mgVcycle() 82 0.003052 0.003052 0.003052 0.26% Castro::normalize_species() 62 0.003017 0.003017 0.003017 0.26% Gravity::get_new_grav_vector() 11 0.002985 0.002985 0.002985 0.25% Castro::construct_old_gravity_source() 10 0.002718 0.002718 0.002718 0.23% MLMG:computeResOfCorrection() 410 0.002352 0.002352 0.002352 0.20% MLMG::ResNormInf() 93 0.002328 0.002328 0.002328 0.20% MLMG::oneIter() 82 0.002307 0.002307 0.002307 0.20% Amr::defBaseLevel() 1 0.002232 0.002232 0.002232 0.19% MLCellLinOp::compGrad() 11 0.002135 0.002135 0.002135 0.18% FabArrayBase::CPC::define() 454 0.001983 0.001983 0.001983 0.17% Gravity::get_old_grav_vector() 10 0.001945 0.001945 0.001945 0.16% MultiFab::Saxpy() 20 0.001929 0.001929 0.001929 0.16% MLCellLinOp::setLevelBC() 11 0.001668 0.001668 0.001668 0.14% MLMG::prepareForSolve() 11 0.001631 0.001631 0.001631 0.14% Gravity::actual_solve_with_mlmg() 11 0.00158 0.00158 0.00158 0.13% FabArray::mult() 43 0.001521 0.001521 0.001521 0.13% MLMG::mgVcycle_down::0 82 0.001455 0.001455 0.001455 0.12% MLMG::mgVcycle_down::1 82 0.001426 0.001426 0.001426 0.12% MLMG::mgVcycle_down::4 82 0.001419 0.001419 0.001419 0.12% MLMG::mgVcycle_down::2 82 0.001407 0.001407 0.001407 0.12% MLMG::mgVcycle_down::3 82 0.001403 0.001403 0.001403 0.12% MLCellLinOp::prepareForSolve() 11 0.001402 0.001402 0.001402 0.12% Castro::initData() 1 0.001379 0.001379 0.001379 0.12% MLMG::solve() 11 0.001017 0.001017 0.001017 0.09% MLMG::mgVcycle_up::3 82 0.0009823 0.0009823 0.0009823 0.08% MLMG::mgVcycle_up::4 82 0.0009666 0.0009666 0.0009666 0.08% MLMG::mgVcycle_up::2 82 0.0009515 0.0009515 0.0009515 0.08% MLMG::mgVcycle_up::0 82 0.0009512 0.0009512 0.0009512 0.08% Castro::clean_state() 62 0.0009461 0.0009461 0.0009461 0.08% MLMG::mgVcycle_up::1 82 0.0009358 0.0009358 0.0009358 0.08% MLCellLinOp::solutionResidual() 93 0.000834 0.000834 0.000834 0.07% Gravity::update_max_rhs() 11 0.0007002 0.0007002 0.0007002 0.06% Gravity::solve_for_phi() 10 0.0006079 0.0006079 0.0006079 0.05% FillPatchIterator::Initialize 41 0.0005899 0.0005899 0.0005899 0.05% MultiFab::Copy() 11 0.0005832 0.0005832 0.0005832 0.05% Amr::InitAmr() 1 0.0004985 0.0004985 0.0004985 0.04% MLMG::mgVcycle_bottom 82 0.000476 0.000476 0.000476 0.04% MLMG::computeResidual() 82 0.0004754 0.0004754 0.0004754 0.04% main() 1 0.000407 0.000407 0.000407 0.03% Castro::initialize_advance() 10 0.0004002 0.0004002 0.0004002 0.03% Amr::coarseTimeStep() 10 0.0003985 0.0003985 0.0003985 0.03% FillPatchSingleLevel 41 0.0003855 0.0003855 0.0003855 0.03% MultiFab::min() 10 0.0003628 0.0003628 0.0003628 0.03% Castro::construct_new_gravity() 10 0.0003551 0.0003551 0.0003551 0.03% makeSFC 55 0.0003289 0.0003289 0.0003289 0.03% Castro::construct_new_source() 50 0.0003148 0.0003148 0.0003148 0.03% MultiFab::max() 11 0.000302 0.000302 0.000302 0.03% Castro::do_new_sources() 10 0.0002917 0.0002917 0.0002917 0.02% Castro::do_old_sources() 10 0.0002906 0.0002906 0.0002906 0.02% Castro::subcycle_advance_ctu() 10 0.0002845 0.0002845 0.0002845 0.02% Amr::timeStep() 10 0.0002798 0.0002798 0.0002798 0.02% MLLinOp::defineGrids() 11 0.0002757 0.0002757 0.0002757 0.02% MLCellLinOp::defineBC() 11 0.0002528 0.0002528 0.0002528 0.02% FabArrayBase::FB::FB() 56 0.0002487 0.0002487 0.0002487 0.02% Castro::advance() 10 0.0002457 0.0002457 0.0002457 0.02% MLMG::MLRhsNormInf() 11 0.0002413 0.0002413 0.0002413 0.02% Gravity::actual_multilevel_solve() 1 0.0002351 0.0002351 0.0002351 0.02% Castro::construct_old_source() 50 0.0002113 0.0002113 0.0002113 0.02% MLLinOp::makeAgglomeratedDMap 11 0.0001963 0.0001963 0.0001963 0.02% StateData::checkPoint() 12 0.0001884 0.0001884 0.0001884 0.02% DistributionMapping::Distribute() 56 0.0001692 0.0001692 0.0001692 0.01% AmrLevel::checkPoint() 3 0.0001596 0.0001596 0.0001596 0.01% MLPoisson::define() 11 0.0001494 0.0001494 0.0001494 0.01% Gravity::solve_phi_with_mlmg() 11 0.0001489 0.0001489 0.0001489 0.01% Castro::finalize_advance() 10 0.0001398 0.0001398 0.0001398 0.01% Castro::initialize_do_advance() 10 0.0001386 0.0001386 0.0001386 0.01% Castro::apply_source_to_state() 20 0.0001215 0.0001215 0.0001215 0.01% Gravity::swapTimeLevels() 10 0.0001184 0.0001184 0.0001184 0.01% Castro::post_timestep() 10 0.0001036 0.0001036 0.0001036 0.01% MLMG::MLResNormInf() 11 8e-05 8e-05 8e-05 0.01% MLLinOp::define() 11 7.354e-05 7.354e-05 7.354e-05 0.01% MLPoisson::prepareForSolve() 11 6.775e-05 6.775e-05 6.775e-05 0.01% MLMG::computeMLResidual() 11 6.692e-05 6.692e-05 6.692e-05 0.01% Castro::swap_state_time_levels() 10 6.563e-05 6.563e-05 6.563e-05 0.01% Castro::construct_old_gravity() 10 6.504e-05 6.504e-05 6.504e-05 0.01% Castro::finalize_do_advance() 10 6.496e-05 6.496e-05 6.496e-05 0.01% MLMG::getGradSolution() 11 6.488e-05 6.488e-05 6.488e-05 0.01% Amr::FinalizeInit() 1 5.804e-05 5.804e-05 5.804e-05 0.00% Castro::computeNewDt() 9 5.429e-05 5.429e-05 5.429e-05 0.00% StateData::define() 4 5.217e-05 5.217e-05 5.217e-05 0.00% Gravity::set_mass_offset() 11 4.992e-05 4.992e-05 4.992e-05 0.00% Castro::enforce_consistent_e() 1 3.867e-05 3.867e-05 3.867e-05 0.00% MLMG::buildFineMask() 11 3.662e-05 3.662e-05 3.662e-05 0.00% Castro::create_source_corrector() 10 3.279e-05 3.279e-05 3.279e-05 0.00% MLLinOp::makeSubCommunicator() 11 3.25e-05 3.25e-05 3.25e-05 0.00% Castro::retry_advance_ctu() 10 3.079e-05 3.079e-05 3.079e-05 0.00% Castro::FluxRegFineAdd() 10 3.033e-05 3.033e-05 3.033e-05 0.00% AmrLevel::AmrLevel(dm) 1 2.487e-05 2.487e-05 2.487e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.283e-05 2.283e-05 2.283e-05 0.00% Castro::post_init() 1 1.746e-05 1.746e-05 1.746e-05 0.00% AmrLevel::checkPointPost() 3 1.713e-05 1.713e-05 1.713e-05 0.00% Castro::computeInitialDt() 2 1.367e-05 1.367e-05 1.367e-05 0.00% Amr::writeSmallPlotFile() 1 1.367e-05 1.367e-05 1.367e-05 0.00% Amr::init() 1 1.312e-05 1.312e-05 1.312e-05 0.00% Amr::InitializeInit() 1 1.175e-05 1.175e-05 1.175e-05 0.00% AmrLevel::checkPointPre() 3 1.1e-05 1.1e-05 1.1e-05 0.00% Amr::initialInit() 1 9.501e-06 9.501e-06 9.501e-06 0.00% Amr::initSubcycle() 1 8.751e-06 8.751e-06 8.751e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 8.416e-06 8.416e-06 8.416e-06 0.00% Castro::post_regrid() 1 6.667e-06 6.667e-06 6.667e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 1.181 1.181 1.181 100.00% Amr::coarseTimeStep() 10 0.9875 0.9875 0.9875 83.62% Amr::timeStep() 10 0.8704 0.8704 0.8704 73.71% Castro::advance() 10 0.8633 0.8633 0.8633 73.10% Castro::subcycle_advance_ctu() 10 0.8487 0.8487 0.8487 71.87% Castro::do_advance_ctu() 10 0.8484 0.8484 0.8484 71.84% Gravity::solve_phi_with_mlmg() 11 0.5755 0.5755 0.5755 48.73% Gravity::actual_solve_with_mlmg() 11 0.5618 0.5618 0.5618 47.57% MLMG::solve() 11 0.5279 0.5279 0.5279 44.70% Castro::construct_new_gravity() 10 0.5175 0.5175 0.5175 43.82% MLMG::oneIter() 82 0.5032 0.5032 0.5032 42.61% MLMG::mgVcycle() 82 0.5009 0.5009 0.5009 42.41% Gravity::solve_for_phi() 10 0.491 0.491 0.491 41.57% MLCellLinOp::smooth() 1640 0.2418 0.2418 0.2418 20.48% Castro::construct_ctu_hydro_source() 10 0.2411 0.2411 0.2411 20.42% MLCellLinOp::applyBC() 4433 0.1944 0.1944 0.1944 16.46% Amr::init() 1 0.1672 0.1672 0.1672 14.16% VisMF::Write(FabArray) 11 0.1664 0.1664 0.1664 14.09% Amr::checkPoint() 3 0.1558 0.1558 0.1558 13.19% MLMG::mgVcycle_bottom 82 0.1525 0.1525 0.1525 12.91% MLMG::actualBottomSolve() 82 0.152 0.152 0.152 12.87% MLCGSolver::bicgstab 82 0.146 0.146 0.146 12.36% AmrLevel::checkPoint() 3 0.1197 0.1197 0.1197 10.14% StateData::checkPoint() 12 0.1195 0.1195 0.1195 10.12% Amr::initialInit() 1 0.09971 0.09971 0.09971 8.44% Amr::FinalizeInit() 1 0.08927 0.08927 0.08927 7.56% Castro::post_init() 1 0.08848 0.08848 0.08848 7.49% Gravity::multilevel_solve_for_new_phi() 1 0.08539 0.08539 0.08539 7.23% Gravity::actual_multilevel_solve() 1 0.08537 0.08537 0.08537 7.23% MLCellLinOp::apply() 1142 0.07725 0.07725 0.07725 6.54% MLPoisson::Fsmooth() 3280 0.07577 0.07577 0.07577 6.42% FabArray::FillBoundary() 4023 0.07528 0.07528 0.07528 6.37% FillPatchIterator::Initialize 41 0.06489 0.06489 0.06489 5.49% FillPatchSingleLevel 41 0.05858 0.05858 0.05858 4.96% MLMG::mgVcycle_down::0 82 0.05505 0.05505 0.05505 4.66% StateDataPhysBCFunct::() 41 0.05338 0.05338 0.05338 4.52% FillBoundary_nowait() 4023 0.05229 0.05229 0.05229 4.43% Amr::writePlotFile() 2 0.05047 0.05047 0.05047 4.27% MLMG::mgVcycle_up::0 82 0.04595 0.04595 0.04595 3.89% MLCellLinOp::correctionResidual() 492 0.04492 0.04492 0.04492 3.80% MLMG:computeResOfCorrection() 410 0.04022 0.04022 0.04022 3.41% Castro::clean_state() 62 0.03639 0.03639 0.03639 3.08% MLMG::mgVcycle_down::1 82 0.03514 0.03514 0.03514 2.98% MLMG::mgVcycle_down::2 82 0.03477 0.03477 0.03477 2.94% MLMG::mgVcycle_down::3 82 0.03409 0.03409 0.03409 2.89% MLMG::mgVcycle_down::4 82 0.03324 0.03324 0.03324 2.81% FabArray::ParallelCopy() 861 0.0313 0.0313 0.0313 2.65% FabArray::setVal() 1140 0.03078 0.03078 0.03078 2.61% CGSolver::sxay() 1586 0.02909 0.02909 0.02909 2.46% Gravity::get_new_grav_vector() 11 0.02902 0.02902 0.02902 2.46% MultiFab::Dot() 1114 0.02817 0.02817 0.02817 2.39% MLPoisson::define() 11 0.02765 0.02765 0.02765 2.34% MLMG::mgVcycle_up::2 82 0.02742 0.02742 0.02742 2.32% MLMG::mgVcycle_up::1 82 0.02712 0.02712 0.02712 2.30% MLMG::mgVcycle_up::3 82 0.0263 0.0263 0.0263 2.23% MLMG::mgVcycle_up::4 82 0.02624 0.02624 0.02624 2.22% FabArray::ParallelCopy_nowait() 861 0.02622 0.02622 0.02622 2.22% StateData::FillBoundary(geom) 328 0.02562 0.02562 0.02562 2.17% Castro::initialize_do_advance() 10 0.02558 0.02558 0.02558 2.17% MLMG::addInterpCorrection() 410 0.02319 0.02319 0.02319 1.96% MLCGSolver::ParallelAllReduce 1514 0.02267 0.02267 0.02267 1.92% amrex::average_down 410 0.02247 0.02247 0.02247 1.90% MultiFab::LinComb() 1586 0.02003 0.02003 0.02003 1.70% MLCellLinOp::defineAuxData() 11 0.01995 0.01995 0.01995 1.69% Castro::expand_state() 10 0.01982 0.01982 0.01982 1.68% Castro::construct_old_gravity() 10 0.01976 0.01976 0.01976 1.67% Gravity::get_old_grav_vector() 10 0.0197 0.0197 0.0197 1.67% Castro::computeTemp() 63 0.01932 0.01932 0.01932 1.64% MLPoisson::Fapply() 1142 0.01615 0.01615 0.01615 1.37% Castro::initialize_advance() 10 0.0142 0.0142 0.0142 1.20% Gravity::fill_multipole_BCs() 11 0.01331 0.01331 0.01331 1.13% Castro::do_new_sources() 10 0.01262 0.01262 0.01262 1.07% MLCellLinOp::solutionResidual() 93 0.01222 0.01222 0.01222 1.03% FabArrayBase::getFB() 4023 0.01218 0.01218 0.01218 1.03% MLMG::computeResidual() 82 0.01109 0.01109 0.01109 0.94% Amr::InitializeInit() 1 0.01044 0.01044 0.01044 0.88% Amr::defBaseLevel() 1 0.01043 0.01043 0.01043 0.88% Castro::enforce_min_density() 62 0.009558 0.009558 0.009558 0.81% Castro::do_old_sources() 10 0.009214 0.009214 0.009214 0.78% MultiFab::Xpay() 585 0.008952 0.008952 0.008952 0.76% MLMG::prepareForSolve() 11 0.008313 0.008313 0.008313 0.70% FabArrayBase::getCPC() 1323 0.007753 0.007753 0.007753 0.66% Castro::post_timestep() 10 0.006845 0.006845 0.006845 0.58% MLCellLinOp::defineBC() 11 0.006471 0.006471 0.006471 0.55% BndryData::define() 11 0.006219 0.006219 0.006219 0.53% Castro::estTimeStep() 21 0.005889 0.005889 0.005889 0.50% FabArray::setDomainBndry() 41 0.005721 0.005721 0.005721 0.48% Castro::reset_internal_energy() 63 0.005707 0.005707 0.005707 0.48% Castro::enforce_speed_limit() 62 0.004365 0.004365 0.004365 0.37% Castro::initData() 1 0.003735 0.003735 0.003735 0.32% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.003722 0.003722 0.003722 0.32% Castro::construct_new_source() 50 0.003539 0.003539 0.003539 0.30% Castro::construct_new_gravity_source() 10 0.003224 0.003224 0.003224 0.27% Castro::normalize_species() 62 0.003017 0.003017 0.003017 0.26% MLMG::getGradSolution() 11 0.002971 0.002971 0.002971 0.25% Castro::construct_old_source() 50 0.002929 0.002929 0.002929 0.25% MLCellLinOp::compGrad() 11 0.002906 0.002906 0.002906 0.25% Castro::construct_old_gravity_source() 10 0.002718 0.002718 0.002718 0.23% Castro::computeNewDt() 9 0.002662 0.002662 0.002662 0.23% MLMG::ResNormInf() 93 0.002328 0.002328 0.002328 0.20% Castro::apply_source_to_state() 20 0.002051 0.002051 0.002051 0.17% FabArrayBase::CPC::define() 454 0.001983 0.001983 0.001983 0.17% MultiFab::Saxpy() 20 0.001929 0.001929 0.001929 0.16% MLMG::computeMLResidual() 11 0.001669 0.001669 0.001669 0.14% MLCellLinOp::setLevelBC() 11 0.001668 0.001668 0.001668 0.14% FabArray::mult() 43 0.001521 0.001521 0.001521 0.13% MLPoisson::prepareForSolve() 11 0.001469 0.001469 0.001469 0.12% MLCellLinOp::prepareForSolve() 11 0.001402 0.001402 0.001402 0.12% Gravity::swapTimeLevels() 10 0.001358 0.001358 0.001358 0.12% Gravity::update_max_rhs() 11 0.0012 0.0012 0.0012 0.10% MLLinOp::define() 11 0.001072 0.001072 0.001072 0.09% MLLinOp::defineGrids() 11 0.0009988 0.0009988 0.0009988 0.08% Castro::computeInitialDt() 2 0.0007169 0.0007169 0.0007169 0.06% MLLinOp::makeAgglomeratedDMap 11 0.0006906 0.0006906 0.0006906 0.06% MultiFab::Copy() 11 0.0005832 0.0005832 0.0005832 0.05% Castro::post_regrid() 1 0.000513 0.000513 0.000513 0.04% Amr::InitAmr() 1 0.0005073 0.0005073 0.0005073 0.04% makeSFC 55 0.0004943 0.0004943 0.0004943 0.04% MLMG::MLResNormInf() 11 0.0003673 0.0003673 0.0003673 0.03% MultiFab::min() 10 0.0003628 0.0003628 0.0003628 0.03% MultiFab::max() 11 0.000302 0.000302 0.000302 0.03% FabArrayBase::FB::FB() 56 0.0002487 0.0002487 0.0002487 0.02% MLMG::MLRhsNormInf() 11 0.0002413 0.0002413 0.0002413 0.02% Castro::finalize_advance() 10 0.0001702 0.0001702 0.0001702 0.01% DistributionMapping::Distribute() 56 0.0001692 0.0001692 0.0001692 0.01% AmrLevel::AmrLevel(dm) 1 7.704e-05 7.704e-05 7.704e-05 0.01% Castro::swap_state_time_levels() 10 6.563e-05 6.563e-05 6.563e-05 0.01% Castro::finalize_do_advance() 10 6.496e-05 6.496e-05 6.496e-05 0.01% StateData::define() 4 5.217e-05 5.217e-05 5.217e-05 0.00% Gravity::set_mass_offset() 11 4.992e-05 4.992e-05 4.992e-05 0.00% Castro::enforce_consistent_e() 1 3.867e-05 3.867e-05 3.867e-05 0.00% MLMG::buildFineMask() 11 3.662e-05 3.662e-05 3.662e-05 0.00% Castro::create_source_corrector() 10 3.279e-05 3.279e-05 3.279e-05 0.00% MLLinOp::makeSubCommunicator() 11 3.25e-05 3.25e-05 3.25e-05 0.00% Castro::retry_advance_ctu() 10 3.079e-05 3.079e-05 3.079e-05 0.00% Castro::FluxRegFineAdd() 10 3.033e-05 3.033e-05 3.033e-05 0.00% AmrLevel::checkPointPost() 3 1.713e-05 1.713e-05 1.713e-05 0.00% Amr::writeSmallPlotFile() 1 1.367e-05 1.367e-05 1.367e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 1.221e-05 1.221e-05 1.221e-05 0.00% AmrLevel::checkPointPre() 3 1.1e-05 1.1e-05 1.1e-05 0.00% Amr::initSubcycle() 1 8.751e-06 8.751e-06 8.751e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10622 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.01-12-g1404c4a9acc6) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.01-12-g1404c4a9acc6) initialized Starting run at 09:06:07 UTC on 2022-01-07. Successfully read inputs file ... Castro git describe: 21.12-23-g727e39bb6 AMReX git describe: 22.01-12-g1404c4a9a Microphysics git describe: 22.01-14-geeef682d reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.650695944 Restart time = 0.059248699 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.146462136 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.076367961 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.081773448 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.087617773 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.096139027 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026898814 seconds Ending run at 09:06:08 UTC on 2022-01-07. Run time = 0.575502573 Run time without initialization = 0.515618245 Average number of zones advanced per microsecond: 2.542 Average number of zones advanced per microsecond per rank: 2.542 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489334272 TinyProfiler total time across processes [min...avg...max]: 0.5756 ... 0.5756 ... 0.5756 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1515 0.1515 0.1515 26.33% MLCellLinOp::applyBC() 1946 0.05668 0.05668 0.05668 9.85% VisMF::Read() 3 0.04487 0.04487 0.04487 7.80% MLPoisson::Fsmooth() 1440 0.03274 0.03274 0.03274 5.69% VisMF::Write(FabArray) 1 0.02537 0.02537 0.02537 4.41% FabArray::setVal() 533 0.0189 0.0189 0.0189 3.28% FillBoundary_nowait() 1766 0.01881 0.01881 0.01881 3.27% MLCGSolver::bicgstab 36 0.01667 0.01667 0.01667 2.90% StateData::FillBoundary(geom) 160 0.01259 0.01259 0.01259 2.19% MultiFab::Dot() 484 0.0123 0.0123 0.0123 2.14% FabArray::ParallelCopy_nowait() 380 0.01051 0.01051 0.01051 1.83% MLCellLinOp::smooth() 720 0.01045 0.01045 0.01045 1.81% FabArray::FillBoundary() 1766 0.01014 0.01014 0.01014 1.76% StateDataPhysBCFunct::() 20 0.009387 0.009387 0.009387 1.63% MultiFab::LinComb() 690 0.008822 0.008822 0.008822 1.53% MLCellLinOp::defineAuxData() 6 0.007578 0.007578 0.007578 1.32% Castro::computeTemp() 30 0.007449 0.007449 0.007449 1.29% Gravity::fill_multipole_BCs() 6 0.007016 0.007016 0.007016 1.22% MLPoisson::Fapply() 500 0.006983 0.006983 0.006983 1.21% FabArrayBase::getFB() 1766 0.005305 0.005305 0.005305 0.92% Castro::enforce_min_density() 30 0.004947 0.004947 0.004947 0.86% Amr::restart() 1 0.004939 0.004939 0.004939 0.86% MLMG::addInterpCorrection() 180 0.004559 0.004559 0.004559 0.79% MLCellLinOp::apply() 500 0.004393 0.004393 0.004393 0.76% Castro::expand_state() 5 0.004299 0.004299 0.004299 0.75% FabArray::setDomainBndry() 20 0.004151 0.004151 0.004151 0.72% amrex::average_down 180 0.004103 0.004103 0.004103 0.71% CGSolver::sxay() 690 0.003972 0.003972 0.003972 0.69% MultiFab::Xpay() 258 0.003924 0.003924 0.003924 0.68% Castro::estTimeStep() 10 0.003675 0.003675 0.003675 0.64% Castro::do_advance_ctu() 5 0.002963 0.002963 0.002963 0.51% FabArrayBase::getCPC() 632 0.002846 0.002846 0.002846 0.49% Castro::reset_internal_energy() 30 0.00283 0.00283 0.00283 0.49% MLCGSolver::ParallelAllReduce 659 0.002718 0.002718 0.002718 0.47% BndryData::define() 6 0.002449 0.002449 0.002449 0.43% FabArray::ParallelCopy() 380 0.002216 0.002216 0.002216 0.38% Gravity::get_new_grav_vector() 5 0.002102 0.002102 0.002102 0.37% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001978 0.001978 0.001978 0.34% MLCellLinOp::correctionResidual() 216 0.001904 0.001904 0.001904 0.33% Castro::construct_new_gravity_source() 5 0.00178 0.00178 0.00178 0.31% Amr::writePlotFile() 1 0.001599 0.001599 0.001599 0.28% MLCellLinOp::compGrad() 6 0.001514 0.001514 0.001514 0.26% Castro::enforce_speed_limit() 30 0.001509 0.001509 0.001509 0.26% Castro::normalize_species() 30 0.001496 0.001496 0.001496 0.26% Castro::construct_old_gravity_source() 5 0.001442 0.001442 0.001442 0.25% MLMG::mgVcycle() 36 0.001377 0.001377 0.001377 0.24% Gravity::get_old_grav_vector() 5 0.001176 0.001176 0.001176 0.20% FabArrayBase::CPC::define() 244 0.00111 0.00111 0.00111 0.19% MLMG::ResNormInf() 42 0.001048 0.001048 0.001048 0.18% MLMG:computeResOfCorrection() 180 0.001046 0.001046 0.001046 0.18% MLMG::oneIter() 36 0.001018 0.001018 0.001018 0.18% MultiFab::Saxpy() 10 0.0009895 0.0009895 0.0009895 0.17% MLCellLinOp::setLevelBC() 6 0.0009246 0.0009246 0.0009246 0.16% MLMG::prepareForSolve() 6 0.0008987 0.0008987 0.0008987 0.16% Gravity::actual_solve_with_mlmg() 6 0.0008788 0.0008788 0.0008788 0.15% MLCellLinOp::prepareForSolve() 6 0.0007961 0.0007961 0.0007961 0.14% FabArray::mult() 22 0.0007502 0.0007502 0.0007502 0.13% Gravity::update_max_rhs() 6 0.0006777 0.0006777 0.0006777 0.12% MLMG::mgVcycle_down::3 36 0.0006488 0.0006488 0.0006488 0.11% MLMG::mgVcycle_down::2 36 0.0006365 0.0006365 0.0006365 0.11% MLMG::mgVcycle_down::4 36 0.0006352 0.0006352 0.0006352 0.11% MLMG::mgVcycle_down::0 36 0.0006263 0.0006263 0.0006263 0.11% MLMG::mgVcycle_down::1 36 0.0006262 0.0006262 0.0006262 0.11% MLMG::actualBottomSolve() 36 0.000556 0.000556 0.000556 0.10% MLMG::solve() 6 0.0004812 0.0004812 0.0004812 0.08% Castro::clean_state() 30 0.0004462 0.0004462 0.0004462 0.08% MLMG::mgVcycle_up::2 36 0.0004412 0.0004412 0.0004412 0.08% MLMG::mgVcycle_up::0 36 0.0004412 0.0004412 0.0004412 0.08% MLMG::mgVcycle_up::1 36 0.0004285 0.0004285 0.0004285 0.07% Amr::InitAmr() 1 0.0004238 0.0004238 0.0004238 0.07% MLMG::mgVcycle_up::4 36 0.0004185 0.0004185 0.0004185 0.07% MLMG::mgVcycle_up::3 36 0.000414 0.000414 0.000414 0.07% MLCellLinOp::solutionResidual() 42 0.0003822 0.0003822 0.0003822 0.07% MultiFab::Copy() 6 0.0003661 0.0003661 0.0003661 0.06% Castro::construct_new_gravity() 5 0.000361 0.000361 0.000361 0.06% main() 1 0.0003193 0.0003193 0.0003193 0.06% FillPatchIterator::Initialize 20 0.0003147 0.0003147 0.0003147 0.05% Gravity::solve_for_phi() 5 0.0003026 0.0003026 0.0003026 0.05% Castro::construct_new_source() 25 0.0002888 0.0002888 0.0002888 0.05% Castro::initialize_advance() 5 0.0002509 0.0002509 0.0002509 0.04% MLMG::mgVcycle_bottom 36 0.0002379 0.0002379 0.0002379 0.04% Gravity::actual_multilevel_solve() 1 0.0002325 0.0002325 0.0002325 0.04% MLMG::computeResidual() 36 0.0002242 0.0002242 0.0002242 0.04% MultiFab::min() 5 0.0001951 0.0001951 0.0001951 0.03% Amr::coarseTimeStep() 5 0.0001926 0.0001926 0.0001926 0.03% Castro::advance() 5 0.0001882 0.0001882 0.0001882 0.03% makeSFC 30 0.0001794 0.0001794 0.0001794 0.03% Castro::construct_old_source() 25 0.0001794 0.0001794 0.0001794 0.03% FillPatchSingleLevel 20 0.0001786 0.0001786 0.0001786 0.03% MultiFab::max() 6 0.0001619 0.0001619 0.0001619 0.03% MLLinOp::defineGrids() 6 0.0001551 0.0001551 0.0001551 0.03% Castro::do_new_sources() 5 0.0001479 0.0001479 0.0001479 0.03% Castro::do_old_sources() 5 0.0001467 0.0001467 0.0001467 0.03% MLCellLinOp::defineBC() 6 0.0001385 0.0001385 0.0001385 0.02% FabArrayBase::FB::FB() 26 0.0001371 0.0001371 0.0001371 0.02% Castro::subcycle_advance_ctu() 5 0.000137 0.000137 0.000137 0.02% Amr::timeStep() 5 0.0001352 0.0001352 0.0001352 0.02% MLMG::MLRhsNormInf() 6 0.0001279 0.0001279 0.0001279 0.02% MLLinOp::makeAgglomeratedDMap 6 0.0001068 0.0001068 0.0001068 0.02% Castro::create_source_corrector() 5 0.0001049 0.0001049 0.0001049 0.02% AmrLevel::restart() 1 9.692e-05 9.692e-05 9.692e-05 0.02% StateData::restartDoit() 4 9.608e-05 9.608e-05 9.608e-05 0.02% Castro::initialize_do_advance() 5 9.567e-05 9.567e-05 9.567e-05 0.02% DistributionMapping::Distribute() 31 9.496e-05 9.496e-05 9.496e-05 0.02% MLPoisson::define() 6 8.588e-05 8.588e-05 8.588e-05 0.01% Gravity::solve_phi_with_mlmg() 6 8.142e-05 8.142e-05 8.142e-05 0.01% Castro::post_timestep() 5 7.212e-05 7.212e-05 7.212e-05 0.01% Castro::finalize_advance() 5 6.483e-05 6.483e-05 6.483e-05 0.01% Castro::apply_source_to_state() 10 6.104e-05 6.104e-05 6.104e-05 0.01% Gravity::swapTimeLevels() 5 5.871e-05 5.871e-05 5.871e-05 0.01% MLLinOp::define() 6 4.379e-05 4.379e-05 4.379e-05 0.01% Castro::swap_state_time_levels() 5 4.042e-05 4.042e-05 4.042e-05 0.01% MLPoisson::prepareForSolve() 6 3.683e-05 3.683e-05 3.683e-05 0.01% Castro::post_restart() 1 3.587e-05 3.587e-05 3.587e-05 0.01% MLMG::computeMLResidual() 6 3.579e-05 3.579e-05 3.579e-05 0.01% MLMG::getGradSolution() 6 3.529e-05 3.529e-05 3.529e-05 0.01% MLMG::MLResNormInf() 6 3.513e-05 3.513e-05 3.513e-05 0.01% Castro::construct_old_gravity() 5 3.333e-05 3.333e-05 3.333e-05 0.01% Castro::finalize_do_advance() 5 3.233e-05 3.233e-05 3.233e-05 0.01% Castro::computeNewDt() 5 3.116e-05 3.116e-05 3.116e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 2.138e-05 2.138e-05 2.138e-05 0.00% MLMG::buildFineMask() 6 1.992e-05 1.992e-05 1.992e-05 0.00% Gravity::set_mass_offset() 6 1.892e-05 1.892e-05 1.892e-05 0.00% MLLinOp::makeSubCommunicator() 6 1.767e-05 1.767e-05 1.767e-05 0.00% Castro::retry_advance_ctu() 5 1.517e-05 1.517e-05 1.517e-05 0.00% Castro::FluxRegFineAdd() 5 1.496e-05 1.496e-05 1.496e-05 0.00% Amr::writeSmallPlotFile() 1 1.242e-05 1.242e-05 1.242e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 8.583e-06 8.583e-06 8.583e-06 0.00% Amr::initSubcycle() 1 8.375e-06 8.375e-06 8.375e-06 0.00% Amr::init() 1 6.5e-06 6.5e-06 6.5e-06 0.00% AmrLevel::AmrLevel() 1 3.667e-06 3.667e-06 3.667e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.5755 0.5755 0.5755 100.00% Amr::coarseTimeStep() 5 0.4885 0.4885 0.4885 84.87% Amr::timeStep() 5 0.4868 0.4868 0.4868 84.58% Castro::advance() 5 0.4841 0.4841 0.4841 84.11% Castro::subcycle_advance_ctu() 5 0.4712 0.4712 0.4712 81.87% Castro::do_advance_ctu() 5 0.471 0.471 0.471 81.84% Castro::construct_new_gravity() 5 0.2675 0.2675 0.2675 46.48% Gravity::solve_phi_with_mlmg() 6 0.2632 0.2632 0.2632 45.73% Gravity::actual_solve_with_mlmg() 6 0.2559 0.2559 0.2559 44.47% Gravity::solve_for_phi() 5 0.2559 0.2559 0.2559 44.46% MLMG::solve() 6 0.2371 0.2371 0.2371 41.20% MLMG::oneIter() 36 0.2245 0.2245 0.2245 39.01% MLMG::mgVcycle() 36 0.2235 0.2235 0.2235 38.83% Castro::construct_ctu_hydro_source() 5 0.1515 0.1515 0.1515 26.33% MLCellLinOp::smooth() 720 0.1099 0.1099 0.1099 19.09% MLCellLinOp::applyBC() 1946 0.09108 0.09108 0.09108 15.82% MLMG::mgVcycle_bottom 36 0.06559 0.06559 0.06559 11.40% MLMG::actualBottomSolve() 36 0.06535 0.06535 0.06535 11.35% MLCGSolver::bicgstab 36 0.06433 0.06433 0.06433 11.18% Amr::init() 1 0.05931 0.05931 0.05931 10.30% Amr::restart() 1 0.0593 0.0593 0.0593 10.30% AmrLevel::restart() 1 0.04601 0.04601 0.04601 7.99% StateData::restartDoit() 4 0.0459 0.0459 0.0459 7.97% VisMF::Read() 3 0.04487 0.04487 0.04487 7.80% MLCellLinOp::apply() 500 0.03528 0.03528 0.03528 6.13% FabArray::FillBoundary() 1766 0.03439 0.03439 0.03439 5.98% MLPoisson::Fsmooth() 1440 0.03274 0.03274 0.03274 5.69% FillPatchIterator::Initialize 20 0.02909 0.02909 0.02909 5.05% Amr::writePlotFile() 1 0.02697 0.02697 0.02697 4.69% VisMF::Write(FabArray) 1 0.02537 0.02537 0.02537 4.41% MLMG::mgVcycle_down::0 36 0.02504 0.02504 0.02504 4.35% FillPatchSingleLevel 20 0.02462 0.02462 0.02462 4.28% FillBoundary_nowait() 1766 0.02425 0.02425 0.02425 4.21% StateDataPhysBCFunct::() 20 0.02197 0.02197 0.02197 3.82% MLMG::mgVcycle_up::0 36 0.02126 0.02126 0.02126 3.69% MLCellLinOp::correctionResidual() 216 0.02031 0.02031 0.02031 3.53% FabArray::setVal() 533 0.0189 0.0189 0.0189 3.28% Castro::clean_state() 30 0.01868 0.01868 0.01868 3.25% MLMG:computeResOfCorrection() 180 0.01817 0.01817 0.01817 3.16% Castro::initialize_do_advance() 5 0.01637 0.01637 0.01637 2.84% MLMG::mgVcycle_down::2 36 0.01592 0.01592 0.01592 2.77% MLMG::mgVcycle_down::1 36 0.01586 0.01586 0.01586 2.75% MLMG::mgVcycle_down::3 36 0.01518 0.01518 0.01518 2.64% MLMG::mgVcycle_down::4 36 0.01501 0.01501 0.01501 2.61% MLPoisson::define() 6 0.015 0.015 0.015 2.61% FabArray::ParallelCopy() 380 0.0145 0.0145 0.0145 2.52% Castro::expand_state() 5 0.01312 0.01312 0.01312 2.28% CGSolver::sxay() 690 0.01279 0.01279 0.01279 2.22% Castro::initialize_advance() 5 0.01265 0.01265 0.01265 2.20% StateData::FillBoundary(geom) 160 0.01259 0.01259 0.01259 2.19% MLMG::mgVcycle_up::2 36 0.01244 0.01244 0.01244 2.16% MultiFab::Dot() 484 0.0123 0.0123 0.0123 2.14% FabArray::ParallelCopy_nowait() 380 0.01229 0.01229 0.01229 2.14% MLMG::mgVcycle_up::1 36 0.0122 0.0122 0.0122 2.12% MLMG::mgVcycle_up::4 36 0.01192 0.01192 0.01192 2.07% MLMG::mgVcycle_up::3 36 0.01171 0.01171 0.01171 2.04% Gravity::get_new_grav_vector() 5 0.01128 0.01128 0.01128 1.96% MLCellLinOp::defineAuxData() 6 0.01075 0.01075 0.01075 1.87% MLMG::addInterpCorrection() 180 0.0106 0.0106 0.0106 1.84% Castro::computeTemp() 30 0.01028 0.01028 0.01028 1.79% amrex::average_down 180 0.01009 0.01009 0.01009 1.75% MLCGSolver::ParallelAllReduce 659 0.009928 0.009928 0.009928 1.73% Castro::construct_old_gravity() 5 0.009676 0.009676 0.009676 1.68% Gravity::get_old_grav_vector() 5 0.009643 0.009643 0.009643 1.68% MultiFab::LinComb() 690 0.008822 0.008822 0.008822 1.53% Castro::post_restart() 1 0.008355 0.008355 0.008355 1.45% Castro::do_new_sources() 5 0.008023 0.008023 0.008023 1.39% Gravity::multilevel_solve_for_new_phi() 1 0.007852 0.007852 0.007852 1.36% Gravity::actual_multilevel_solve() 1 0.007831 0.007831 0.007831 1.36% Gravity::fill_multipole_BCs() 6 0.007016 0.007016 0.007016 1.22% MLPoisson::Fapply() 500 0.006983 0.006983 0.006983 1.21% MLCellLinOp::solutionResidual() 42 0.005784 0.005784 0.005784 1.00% FabArrayBase::getFB() 1766 0.005442 0.005442 0.005442 0.95% MLMG::computeResidual() 36 0.005096 0.005096 0.005096 0.89% Castro::enforce_min_density() 30 0.004947 0.004947 0.004947 0.86% MLMG::prepareForSolve() 6 0.004891 0.004891 0.004891 0.85% Castro::do_old_sources() 5 0.00488 0.00488 0.00488 0.85% FabArray::setDomainBndry() 20 0.004151 0.004151 0.004151 0.72% FabArrayBase::getCPC() 632 0.003956 0.003956 0.003956 0.69% MultiFab::Xpay() 258 0.003924 0.003924 0.003924 0.68% Castro::estTimeStep() 10 0.003675 0.003675 0.003675 0.64% MLCellLinOp::defineBC() 6 0.003564 0.003564 0.003564 0.62% BndryData::define() 6 0.003425 0.003425 0.003425 0.60% Castro::reset_internal_energy() 30 0.00283 0.00283 0.00283 0.49% Castro::post_timestep() 5 0.002576 0.002576 0.002576 0.45% Castro::construct_new_source() 25 0.002069 0.002069 0.002069 0.36% MLMG::getGradSolution() 6 0.00201 0.00201 0.00201 0.35% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001978 0.001978 0.001978 0.34% MLCellLinOp::compGrad() 6 0.001975 0.001975 0.001975 0.34% Castro::construct_new_gravity_source() 5 0.00178 0.00178 0.00178 0.31% Castro::construct_old_source() 25 0.001621 0.001621 0.001621 0.28% Castro::enforce_speed_limit() 30 0.001509 0.001509 0.001509 0.26% Castro::normalize_species() 30 0.001496 0.001496 0.001496 0.26% Castro::computeNewDt() 5 0.001458 0.001458 0.001458 0.25% Castro::construct_old_gravity_source() 5 0.001442 0.001442 0.001442 0.25% FabArrayBase::CPC::define() 244 0.00111 0.00111 0.00111 0.19% Castro::apply_source_to_state() 10 0.001051 0.001051 0.001051 0.18% Gravity::swapTimeLevels() 5 0.001048 0.001048 0.001048 0.18% MLMG::ResNormInf() 42 0.001048 0.001048 0.001048 0.18% MultiFab::Saxpy() 10 0.0009895 0.0009895 0.0009895 0.17% MLMG::computeMLResidual() 6 0.0009479 0.0009479 0.0009479 0.16% Gravity::update_max_rhs() 6 0.0009402 0.0009402 0.0009402 0.16% MLCellLinOp::setLevelBC() 6 0.0009246 0.0009246 0.0009246 0.16% MLPoisson::prepareForSolve() 6 0.0008329 0.0008329 0.0008329 0.14% MLCellLinOp::prepareForSolve() 6 0.0007961 0.0007961 0.0007961 0.14% FabArray::mult() 22 0.0007502 0.0007502 0.0007502 0.13% MLLinOp::define() 6 0.0005938 0.0005938 0.0005938 0.10% MLLinOp::defineGrids() 6 0.00055 0.00055 0.00055 0.10% Amr::InitAmr() 1 0.0004322 0.0004322 0.0004322 0.08% MLLinOp::makeAgglomeratedDMap 6 0.0003772 0.0003772 0.0003772 0.07% MultiFab::Copy() 6 0.0003661 0.0003661 0.0003661 0.06% makeSFC 30 0.0002704 0.0002704 0.0002704 0.05% MultiFab::min() 5 0.0001951 0.0001951 0.0001951 0.03% MLMG::MLResNormInf() 6 0.0001933 0.0001933 0.0001933 0.03% MultiFab::max() 6 0.0001619 0.0001619 0.0001619 0.03% FabArrayBase::FB::FB() 26 0.0001371 0.0001371 0.0001371 0.02% MLMG::MLRhsNormInf() 6 0.0001279 0.0001279 0.0001279 0.02% Castro::create_source_corrector() 5 0.0001049 0.0001049 0.0001049 0.02% DistributionMapping::Distribute() 31 9.496e-05 9.496e-05 9.496e-05 0.02% Castro::finalize_advance() 5 7.979e-05 7.979e-05 7.979e-05 0.01% Castro::swap_state_time_levels() 5 4.042e-05 4.042e-05 4.042e-05 0.01% Castro::finalize_do_advance() 5 3.233e-05 3.233e-05 3.233e-05 0.01% MLMG::buildFineMask() 6 1.992e-05 1.992e-05 1.992e-05 0.00% Gravity::set_mass_offset() 6 1.892e-05 1.892e-05 1.892e-05 0.00% MLLinOp::makeSubCommunicator() 6 1.767e-05 1.767e-05 1.767e-05 0.00% Castro::retry_advance_ctu() 5 1.517e-05 1.517e-05 1.517e-05 0.00% Castro::FluxRegFineAdd() 5 1.496e-05 1.496e-05 1.496e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 1.258e-05 1.258e-05 1.258e-05 0.00% Amr::writeSmallPlotFile() 1 1.242e-05 1.242e-05 1.242e-05 0.00% Amr::initSubcycle() 1 8.375e-06 8.375e-06 8.375e-06 0.00% AmrLevel::AmrLevel() 1 3.667e-06 3.667e-06 3.667e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10622 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.01-12-g1404c4a9acc6) finalized