Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-24-g3ec07681574a) initialized Starting run at 14:21:17 UTC on 2022-10-27. Successfully read inputs file ... Castro git describe: 22.09-3-g41697201c AMReX git describe: 22.10-24-g3ec076815 Microphysics git describe: 22.10-5-g820f45d5 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.052024477 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030023719 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.04812339 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051241894 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.054366013 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.050714753 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.073435077 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.048267261 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.0711751 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.05926575 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.062157967 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060302243 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063940851 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.048236429 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.030306461 seconds Ending run at 14:21:18 UTC on 2022-10-27. Run time = 0.856253643 Run time without initialization = 0.72212505 Average number of zones advanced per microsecond: 3.630 Average number of zones advanced per microsecond per rank: 3.630 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8563 ... 0.8563 ... 0.8563 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2106 0.2106 0.2106 24.59% VisMF::Write(FabArray) 11 0.2009 0.2009 0.2009 23.46% MLCellLinOp::applyBC() 4433 0.07977 0.07977 0.07977 9.32% MLPoisson::Fsmooth() 3280 0.06319 0.06319 0.06319 7.38% MLCGSolver::bicgstab 82 0.02354 0.02354 0.02354 2.75% StateData::FillBoundary(geom) 328 0.0232 0.0232 0.0232 2.71% MultiFab::Dot() 1114 0.02197 0.02197 0.02197 2.57% Castro::normalize_species() 62 0.01575 0.01575 0.01575 1.84% FillBoundary_nowait() 4023 0.01436 0.01436 0.01436 1.68% MultiFab::LinComb() 1586 0.01415 0.01415 0.01415 1.65% FabArray::setVal() 1144 0.01397 0.01397 0.01397 1.63% StateDataPhysBCFunct::() 41 0.01378 0.01378 0.01378 1.61% Castro::computeTemp() 63 0.01324 0.01324 0.01324 1.55% FabArray::ParallelCopy_nowait() 861 0.013 0.013 0.013 1.52% MLPoisson::Fapply() 1142 0.01154 0.01154 0.01154 1.35% MLCellLinOp::defineAuxData() 11 0.01134 0.01134 0.01134 1.32% Castro::enforce_min_density() 62 0.01025 0.01025 0.01025 1.20% Gravity::fill_multipole_BCs() 11 0.008357 0.008357 0.008357 0.98% MLMG::addInterpCorrection() 410 0.007738 0.007738 0.007738 0.90% amrex::average_down 410 0.006807 0.006807 0.006807 0.79% MultiFab::Xpay() 585 0.006479 0.006479 0.006479 0.76% Amr::checkPoint() 3 0.005191 0.005191 0.005191 0.61% Castro::do_advance_ctu() 10 0.004707 0.004707 0.004707 0.55% Castro::estTimeStep() 21 0.004706 0.004706 0.004706 0.55% Castro::reset_internal_energy(MultiFab) 63 0.004038 0.004038 0.004038 0.47% BndryData::define() 11 0.003747 0.003747 0.003747 0.44% Castro::construct_new_gravity_source() 10 0.003301 0.003301 0.003301 0.39% Amr::writePlotFile() 2 0.002817 0.002817 0.002817 0.33% Castro::construct_old_gravity_source() 10 0.002706 0.002706 0.002706 0.32% MLMG::ResNormInf() 93 0.002009 0.002009 0.002009 0.23% Gravity::get_new_grav_vector() 11 0.001926 0.001926 0.001926 0.22% MultiFab::Saxpy() 20 0.001796 0.001796 0.001796 0.21% Castro::expand_state() 10 0.001731 0.001731 0.001731 0.20% Gravity::get_old_grav_vector() 10 0.001722 0.001722 0.001722 0.20% MultiFab::Add() 82 0.001672 0.001672 0.001672 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001638 0.001638 0.001638 0.19% Castro::create_source_corrector() 10 0.001548 0.001548 0.001548 0.18% Castro::reset_internal_energy(Fab) 504 0.001519 0.001519 0.001519 0.18% MLCellLinOp::setLevelBC() 11 0.001506 0.001506 0.001506 0.18% Gravity::actual_solve_with_mlmg() 11 0.001449 0.001449 0.001449 0.17% Castro::enforce_speed_limit() 62 0.001407 0.001407 0.001407 0.16% FabArray::mult() 43 0.001312 0.001312 0.001312 0.15% FabArray::setDomainBndry() 41 0.001276 0.001276 0.001276 0.15% Castro::initData() 1 0.001223 0.001223 0.001223 0.14% MLMG::prepareForSolve() 11 0.001213 0.001213 0.001213 0.14% MultiFab::contains_nan() 20 0.001167 0.001167 0.001167 0.14% MLCellLinOp::prepareForSolve() 11 0.001159 0.001159 0.001159 0.14% MLCellLinOp::smooth() 1640 0.001043 0.001043 0.001043 0.12% MLCellLinOp::compGrad() 11 0.0009234 0.0009234 0.0009234 0.11% FabArray::FillBoundary() 4023 0.0008521 0.0008521 0.0008521 0.10% FabArrayBase::getCPC() 1323 0.000766 0.000766 0.000766 0.09% Castro::subcycle_advance_ctu() 10 0.0006955 0.0006955 0.0006955 0.08% FabArrayBase::CPC::define() 454 0.000667 0.000667 0.000667 0.08% FabArrayBase::getFB() 4023 0.0005927 0.0005927 0.0005927 0.07% Amr::InitAmr() 1 0.0004711 0.0004711 0.0004711 0.06% MLCellLinOp::apply() 1142 0.0004708 0.0004708 0.0004708 0.05% Gravity::solve_for_phi() 10 0.0004441 0.0004441 0.0004441 0.05% Gravity::update_max_rhs() 11 0.0003973 0.0003973 0.0003973 0.05% CGSolver::sxay() 1586 0.0003729 0.0003729 0.0003729 0.04% MultiFab::Copy() 11 0.0003152 0.0003152 0.0003152 0.04% Amr::coarseTimeStep() 10 0.0003133 0.0003133 0.0003133 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002917 0.0002917 0.0002917 0.03% FillPatchIterator::Initialize 41 0.0002846 0.0002846 0.0002846 0.03% FabArray::ParallelCopy() 861 0.000278 0.000278 0.000278 0.03% MLCellLinOp::defineBC() 11 0.0002725 0.0002725 0.0002725 0.03% MultiFab::max() 11 0.000252 0.000252 0.000252 0.03% main() 1 0.0002491 0.0002491 0.0002491 0.03% MLCellLinOp::correctionResidual() 492 0.0002392 0.0002392 0.0002392 0.03% MLMG::mgVcycle() 82 0.0002376 0.0002376 0.0002376 0.03% MLMG::MLRhsNormInf() 11 0.0002166 0.0002166 0.0002166 0.03% Castro::construct_new_gravity() 10 0.0002025 0.0002025 0.0002025 0.02% MLMG:computeResOfCorrection() 410 0.000157 0.000157 0.000157 0.02% Amr::timeStep() 10 0.0001471 0.0001471 0.0001471 0.02% MLLinOp::defineGrids() 11 0.0001341 0.0001341 0.0001341 0.02% StateData::checkPoint() 12 0.0001319 0.0001319 0.0001319 0.02% MLMG::mgVcycle_down::0 82 0.0001176 0.0001176 0.0001176 0.01% Castro::finalize_advance() 10 0.0001049 0.0001049 0.0001049 0.01% MLMG::mgVcycle_down::1 82 0.0001035 0.0001035 0.0001035 0.01% MLMG::mgVcycle_down::2 82 9.886e-05 9.886e-05 9.886e-05 0.01% MLMG::mgVcycle_down::3 82 9.127e-05 9.127e-05 9.127e-05 0.01% MLMG::mgVcycle_down::4 82 8.876e-05 8.876e-05 8.876e-05 0.01% FabArrayBase::FB::FB() 56 8.48e-05 8.48e-05 8.48e-05 0.01% Castro::Castro() 1 8.442e-05 8.442e-05 8.442e-05 0.01% MLMG::actualBottomSolve() 82 8.289e-05 8.289e-05 8.289e-05 0.01% Castro::initialize_advance() 10 8.041e-05 8.041e-05 8.041e-05 0.01% Castro::clean_state() 62 7.934e-05 7.934e-05 7.934e-05 0.01% Castro::advance() 10 7.728e-05 7.728e-05 7.728e-05 0.01% MLMG::mgVcycle_up::4 82 7.418e-05 7.418e-05 7.418e-05 0.01% AmrLevel::checkPoint() 3 7.18e-05 7.18e-05 7.18e-05 0.01% MLMG::solve() 11 7.148e-05 7.148e-05 7.148e-05 0.01% MLMG::mgVcycle_up::0 82 6.227e-05 6.227e-05 6.227e-05 0.01% Castro::initialize_do_advance() 10 6.019e-05 6.019e-05 6.019e-05 0.01% MLMG::oneIter() 82 6.016e-05 6.016e-05 6.016e-05 0.01% MLMG::mgVcycle_up::3 82 5.909e-05 5.909e-05 5.909e-05 0.01% MLMG::mgVcycle_up::1 82 5.785e-05 5.785e-05 5.785e-05 0.01% MLMG::mgVcycle_up::2 82 5.748e-05 5.748e-05 5.748e-05 0.01% MLCellLinOp::solutionResidual() 93 5.596e-05 5.596e-05 5.596e-05 0.01% StateData::define() 4 4.184e-05 4.184e-05 4.184e-05 0.00% MLMG::computeResidual() 82 4.01e-05 4.01e-05 4.01e-05 0.00% Castro::swap_state_time_levels() 10 3.809e-05 3.809e-05 3.809e-05 0.00% Castro::finalize_do_advance() 10 3.323e-05 3.323e-05 3.323e-05 0.00% MLMG::mgVcycle_bottom 82 3.279e-05 3.279e-05 3.279e-05 0.00% Castro::enforce_consistent_e() 1 3.157e-05 3.157e-05 3.157e-05 0.00% Gravity::actual_multilevel_solve() 1 3.072e-05 3.072e-05 3.072e-05 0.00% MLPoisson::define() 11 2.884e-05 2.884e-05 2.884e-05 0.00% makeSFC 55 2.717e-05 2.717e-05 2.717e-05 0.00% FillPatchSingleLevel 41 2.642e-05 2.642e-05 2.642e-05 0.00% Castro::initMFs() 1 2.615e-05 2.615e-05 2.615e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.559e-05 2.559e-05 2.559e-05 0.00% Amr::writeSmallPlotFile() 1 2.476e-05 2.476e-05 2.476e-05 0.00% Castro::buildMetrics() 1 2.342e-05 2.342e-05 2.342e-05 0.00% MLLinOp::define() 11 2.132e-05 2.132e-05 2.132e-05 0.00% Amr::defBaseLevel() 1 2.05e-05 2.05e-05 2.05e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.924e-05 1.924e-05 1.924e-05 0.00% Amr::FinalizeInit() 1 1.89e-05 1.89e-05 1.89e-05 0.00% Castro::construct_new_source() 50 1.723e-05 1.723e-05 1.723e-05 0.00% Castro::construct_old_source() 50 1.703e-05 1.703e-05 1.703e-05 0.00% Castro::do_new_sources() 10 1.697e-05 1.697e-05 1.697e-05 0.00% Castro::do_old_sources() 10 1.6e-05 1.6e-05 1.6e-05 0.00% DistributionMapping::Distribute() 56 1.586e-05 1.586e-05 1.586e-05 0.00% MLMG::computeMLResidual() 11 1.572e-05 1.572e-05 1.572e-05 0.00% Castro::check_for_nan() 20 1.305e-05 1.305e-05 1.305e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.282e-05 1.282e-05 1.282e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.215e-05 1.215e-05 1.215e-05 0.00% Castro::construct_old_gravity() 10 1.089e-05 1.089e-05 1.089e-05 0.00% Castro::apply_source_to_state() 20 1.086e-05 1.086e-05 1.086e-05 0.00% Amr::initSubcycle() 1 8.672e-06 8.672e-06 8.672e-06 0.00% Gravity::swapTimeLevels() 10 8.484e-06 8.484e-06 8.484e-06 0.00% MLPoisson::prepareForSolve() 11 8.252e-06 8.252e-06 8.252e-06 0.00% Castro::post_timestep() 10 7.616e-06 7.616e-06 7.616e-06 0.00% MLMG::getGradSolution() 11 6.796e-06 6.796e-06 6.796e-06 0.00% Castro::computeNewDt() 9 5.904e-06 5.904e-06 5.904e-06 0.00% AmrLevel::checkPointPost() 3 5.789e-06 5.789e-06 5.789e-06 0.00% Amr::InitializeInit() 1 4.925e-06 4.925e-06 4.925e-06 0.00% MLMG::MLResNormInf() 11 3.686e-06 3.686e-06 3.686e-06 0.00% Castro::retry_advance_ctu() 10 3.656e-06 3.656e-06 3.656e-06 0.00% Castro::post_init() 1 3.576e-06 3.576e-06 3.576e-06 0.00% Gravity::set_mass_offset() 11 3.531e-06 3.531e-06 3.531e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.963e-06 2.963e-06 2.963e-06 0.00% Castro::FluxRegCrseInit 10 2.726e-06 2.726e-06 2.726e-06 0.00% Castro::computeInitialDt() 2 2.534e-06 2.534e-06 2.534e-06 0.00% Castro::FluxRegFineAdd() 10 2.512e-06 2.512e-06 2.512e-06 0.00% Amr::init() 1 2.486e-06 2.486e-06 2.486e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.778e-06 1.778e-06 1.778e-06 0.00% AmrLevel::checkPointPre() 3 1.752e-06 1.752e-06 1.752e-06 0.00% Castro::post_regrid() 1 1.214e-06 1.214e-06 1.214e-06 0.00% Amr::initialInit() 1 1.06e-06 1.06e-06 1.06e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8563 0.8563 0.8563 100.00% Amr::coarseTimeStep() 10 0.6916 0.6916 0.6916 80.77% Amr::timeStep() 10 0.5921 0.5921 0.5921 69.15% Castro::advance() 10 0.5846 0.5846 0.5846 68.27% Castro::subcycle_advance_ctu() 10 0.5715 0.5715 0.5715 66.74% Castro::do_advance_ctu() 10 0.5708 0.5708 0.5708 66.66% Gravity::solve_phi_with_mlmg() 11 0.3115 0.3115 0.3115 36.37% Gravity::actual_solve_with_mlmg() 11 0.3029 0.3029 0.3029 35.37% Castro::construct_new_gravity() 10 0.2834 0.2834 0.2834 33.10% MLMG::solve() 11 0.2806 0.2806 0.2806 32.77% Gravity::solve_for_phi() 10 0.268 0.268 0.268 31.30% MLMG::oneIter() 82 0.2659 0.2659 0.2659 31.05% MLMG::mgVcycle() 82 0.2642 0.2642 0.2642 30.85% Castro::construct_ctu_hydro_source() 10 0.2106 0.2106 0.2106 24.59% VisMF::Write(FabArray) 11 0.2009 0.2009 0.2009 23.46% Amr::checkPoint() 3 0.1487 0.1487 0.1487 17.36% AmrLevel::checkPoint() 3 0.1435 0.1435 0.1435 16.75% StateData::checkPoint() 12 0.1434 0.1434 0.1434 16.74% MLCellLinOp::smooth() 1640 0.1353 0.1353 0.1353 15.80% Amr::init() 1 0.1336 0.1336 0.1336 15.60% MLCellLinOp::applyBC() 4433 0.09567 0.09567 0.09567 11.17% MLMG::mgVcycle_bottom 82 0.08081 0.08081 0.08081 9.44% MLMG::actualBottomSolve() 82 0.08078 0.08078 0.08078 9.43% MLCGSolver::bicgstab 82 0.07999 0.07999 0.07999 9.34% MLPoisson::Fsmooth() 3280 0.06319 0.06319 0.06319 7.38% Amr::writePlotFile() 2 0.06045 0.06045 0.06045 7.06% Amr::initialInit() 1 0.05139 0.05139 0.05139 6.00% Amr::FinalizeInit() 1 0.04719 0.04719 0.04719 5.51% Castro::post_init() 1 0.0458 0.0458 0.0458 5.35% Castro::clean_state() 62 0.04545 0.04545 0.04545 5.31% Gravity::multilevel_solve_for_new_phi() 1 0.0439 0.0439 0.0439 5.13% Gravity::actual_multilevel_solve() 1 0.04388 0.04388 0.04388 5.12% FillPatchIterator::Initialize 41 0.04249 0.04249 0.04249 4.96% FillPatchSingleLevel 41 0.04093 0.04093 0.04093 4.78% StateDataPhysBCFunct::() 41 0.03698 0.03698 0.03698 4.32% MLCellLinOp::apply() 1142 0.03608 0.03608 0.03608 4.21% MLMG::mgVcycle_down::0 82 0.03518 0.03518 0.03518 4.11% MLMG::mgVcycle_up::0 82 0.03025 0.03025 0.03025 3.53% StateData::FillBoundary(geom) 328 0.0232 0.0232 0.0232 2.71% MultiFab::Dot() 1114 0.02197 0.02197 0.02197 2.57% MLCellLinOp::correctionResidual() 492 0.0211 0.0211 0.0211 2.46% Castro::initialize_do_advance() 10 0.01952 0.01952 0.01952 2.28% Castro::computeTemp() 63 0.0188 0.0188 0.0188 2.19% MLMG:computeResOfCorrection() 410 0.01824 0.01824 0.01824 2.13% MLPoisson::define() 11 0.01785 0.01785 0.01785 2.08% MLMG::mgVcycle_down::1 82 0.01759 0.01759 0.01759 2.05% MLMG::mgVcycle_down::2 82 0.01714 0.01714 0.01714 2.00% Gravity::get_new_grav_vector() 11 0.01699 0.01699 0.01699 1.98% MLMG::mgVcycle_down::3 82 0.01628 0.01628 0.01628 1.90% FabArray::FillBoundary() 4023 0.01589 0.01589 0.01589 1.86% Castro::normalize_species() 62 0.01575 0.01575 0.01575 1.84% MLMG::mgVcycle_down::4 82 0.01553 0.01553 0.01553 1.81% FillBoundary_nowait() 4023 0.01504 0.01504 0.01504 1.76% CGSolver::sxay() 1586 0.01452 0.01452 0.01452 1.70% Castro::construct_old_gravity() 10 0.01434 0.01434 0.01434 1.67% Gravity::get_old_grav_vector() 10 0.01433 0.01433 0.01433 1.67% MultiFab::LinComb() 1586 0.01415 0.01415 0.01415 1.65% FabArray::ParallelCopy() 861 0.01409 0.01409 0.01409 1.65% FabArray::setVal() 1144 0.01397 0.01397 0.01397 1.63% FabArray::ParallelCopy_nowait() 861 0.01382 0.01382 0.01382 1.61% MLMG::mgVcycle_up::2 82 0.01327 0.01327 0.01327 1.55% MLCGSolver::ParallelAllReduce 1514 0.0131 0.0131 0.0131 1.53% MLMG::mgVcycle_up::1 82 0.01301 0.01301 0.01301 1.52% Castro::initialize_advance() 10 0.0129 0.0129 0.0129 1.51% MLMG::addInterpCorrection() 410 0.01279 0.01279 0.01279 1.49% MLCellLinOp::defineAuxData() 11 0.01266 0.01266 0.01266 1.48% MLMG::mgVcycle_up::3 82 0.01253 0.01253 0.01253 1.46% MLMG::mgVcycle_up::4 82 0.01233 0.01233 0.01233 1.44% Castro::expand_state() 10 0.01224 0.01224 0.01224 1.43% amrex::average_down 410 0.01192 0.01192 0.01192 1.39% MLPoisson::Fapply() 1142 0.01154 0.01154 0.01154 1.35% Castro::do_new_sources() 10 0.01138 0.01138 0.01138 1.33% Castro::do_old_sources() 10 0.01069 0.01069 0.01069 1.25% Castro::enforce_min_density() 62 0.01025 0.01025 0.01025 1.20% Gravity::fill_multipole_BCs() 11 0.008357 0.008357 0.008357 0.98% Castro::post_timestep() 10 0.007361 0.007361 0.007361 0.86% MLCellLinOp::solutionResidual() 93 0.007098 0.007098 0.007098 0.83% MultiFab::Xpay() 585 0.006479 0.006479 0.006479 0.76% MLMG::computeResidual() 82 0.006125 0.006125 0.006125 0.72% Castro::reset_internal_energy(MultiFab) 63 0.005558 0.005558 0.005558 0.65% MLMG::prepareForSolve() 11 0.005286 0.005286 0.005286 0.62% MLCellLinOp::defineBC() 11 0.004953 0.004953 0.004953 0.58% Castro::estTimeStep() 21 0.004706 0.004706 0.004706 0.55% BndryData::define() 11 0.00468 0.00468 0.00468 0.55% Amr::InitializeInit() 1 0.004194 0.004194 0.004194 0.49% Amr::defBaseLevel() 1 0.004189 0.004189 0.004189 0.49% Castro::initData() 1 0.003678 0.003678 0.003678 0.43% Castro::construct_new_source() 50 0.003318 0.003318 0.003318 0.39% Castro::construct_new_gravity_source() 10 0.003301 0.003301 0.003301 0.39% Castro::construct_old_source() 50 0.002723 0.002723 0.002723 0.32% Castro::construct_old_gravity_source() 10 0.002706 0.002706 0.002706 0.32% Castro::computeNewDt() 9 0.002142 0.002142 0.002142 0.25% MLMG::ResNormInf() 93 0.002009 0.002009 0.002009 0.23% Castro::apply_source_to_state() 20 0.001807 0.001807 0.001807 0.21% MultiFab::Saxpy() 20 0.001796 0.001796 0.001796 0.21% MultiFab::Add() 82 0.001672 0.001672 0.001672 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001638 0.001638 0.001638 0.19% Castro::create_source_corrector() 10 0.001548 0.001548 0.001548 0.18% Castro::reset_internal_energy(Fab) 504 0.001519 0.001519 0.001519 0.18% MLCellLinOp::setLevelBC() 11 0.001506 0.001506 0.001506 0.18% FabArrayBase::getCPC() 1323 0.001433 0.001433 0.001433 0.17% MLMG::getGradSolution() 11 0.00143 0.00143 0.00143 0.17% MLCellLinOp::compGrad() 11 0.001423 0.001423 0.001423 0.17% Castro::enforce_speed_limit() 62 0.001407 0.001407 0.001407 0.16% FabArray::mult() 43 0.001312 0.001312 0.001312 0.15% FabArray::setDomainBndry() 41 0.001276 0.001276 0.001276 0.15% Castro::check_for_nan() 20 0.00118 0.00118 0.00118 0.14% MultiFab::contains_nan() 20 0.001167 0.001167 0.001167 0.14% MLPoisson::prepareForSolve() 11 0.001167 0.001167 0.001167 0.14% MLCellLinOp::prepareForSolve() 11 0.001159 0.001159 0.001159 0.14% Castro::post_regrid() 1 0.001137 0.001137 0.001137 0.13% MLMG::computeMLResidual() 11 0.001029 0.001029 0.001029 0.12% Gravity::update_max_rhs() 11 0.0007946 0.0007946 0.0007946 0.09% Castro::computeInitialDt() 2 0.0007123 0.0007123 0.0007123 0.08% FabArrayBase::getFB() 4023 0.0006775 0.0006775 0.0006775 0.08% FabArrayBase::CPC::define() 454 0.000667 0.000667 0.000667 0.08% Amr::InitAmr() 1 0.0004798 0.0004798 0.0004798 0.06% Castro::Castro() 1 0.0004332 0.0004332 0.0004332 0.05% Gravity::swapTimeLevels() 10 0.0004235 0.0004235 0.0004235 0.05% MultiFab::Copy() 11 0.0003152 0.0003152 0.0003152 0.04% MLMG::MLResNormInf() 11 0.0002777 0.0002777 0.0002777 0.03% MultiFab::max() 11 0.000252 0.000252 0.000252 0.03% MLMG::MLRhsNormInf() 11 0.0002166 0.0002166 0.0002166 0.03% MLLinOp::define() 11 0.0002119 0.0002119 0.0002119 0.02% MLLinOp::defineGrids() 11 0.0001906 0.0001906 0.0001906 0.02% Castro::buildMetrics() 1 0.0001597 0.0001597 0.0001597 0.02% Castro::finalize_advance() 10 0.0001102 0.0001102 0.0001102 0.01% FabArrayBase::FB::FB() 56 8.48e-05 8.48e-05 8.48e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.466e-05 5.466e-05 5.466e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.399e-05 5.399e-05 5.399e-05 0.01% makeSFC 55 4.184e-05 4.184e-05 4.184e-05 0.00% StateData::define() 4 4.184e-05 4.184e-05 4.184e-05 0.00% Castro::swap_state_time_levels() 10 3.809e-05 3.809e-05 3.809e-05 0.00% Castro::finalize_do_advance() 10 3.323e-05 3.323e-05 3.323e-05 0.00% Castro::enforce_consistent_e() 1 3.157e-05 3.157e-05 3.157e-05 0.00% Castro::initMFs() 1 2.615e-05 2.615e-05 2.615e-05 0.00% Amr::writeSmallPlotFile() 1 2.476e-05 2.476e-05 2.476e-05 0.00% DistributionMapping::Distribute() 56 1.586e-05 1.586e-05 1.586e-05 0.00% Amr::initSubcycle() 1 8.672e-06 8.672e-06 8.672e-06 0.00% AmrLevel::checkPointPost() 3 5.789e-06 5.789e-06 5.789e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.151e-06 4.151e-06 4.151e-06 0.00% Castro::retry_advance_ctu() 10 3.656e-06 3.656e-06 3.656e-06 0.00% Gravity::set_mass_offset() 11 3.531e-06 3.531e-06 3.531e-06 0.00% Castro::FluxRegCrseInit 10 2.726e-06 2.726e-06 2.726e-06 0.00% Castro::FluxRegFineAdd() 10 2.512e-06 2.512e-06 2.512e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.778e-06 1.778e-06 1.778e-06 0.00% AmrLevel::checkPointPre() 3 1.752e-06 1.752e-06 1.752e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-24-g3ec07681574a) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-24-g3ec07681574a) initialized Starting run at 14:21:19 UTC on 2022-10-27. Successfully read inputs file ... Castro git describe: 22.09-3-g41697201c AMReX git describe: 22.10-24-g3ec076815 Microphysics git describe: 22.10-5-g820f45d5 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.460297631 Restart time = 0.047732289 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.053937273 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.055968899 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.05965623 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060073216 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.077327124 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031806358 seconds Ending run at 14:21:20 UTC on 2022-10-27. Run time = 0.387426161 Run time without initialization = 0.339152377 Average number of zones advanced per microsecond: 3.865 Average number of zones advanced per microsecond per rank: 3.865 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3875 ... 0.3875 ... 0.3875 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1111 0.1111 0.1111 28.66% VisMF::Read() 3 0.03996 0.03996 0.03996 10.31% MLCellLinOp::applyBC() 1946 0.03409 0.03409 0.03409 8.80% VisMF::Write(FabArray) 1 0.03023 0.03023 0.03023 7.80% MLPoisson::Fsmooth() 1440 0.02669 0.02669 0.02669 6.89% StateData::FillBoundary(geom) 160 0.01119 0.01119 0.01119 2.89% MLCGSolver::bicgstab 36 0.009988 0.009988 0.009988 2.58% MultiFab::Dot() 484 0.009253 0.009253 0.009253 2.39% Castro::normalize_species() 30 0.009137 0.009137 0.009137 2.36% Castro::computeTemp() 30 0.007409 0.007409 0.007409 1.91% FabArray::setVal() 537 0.006621 0.006621 0.006621 1.71% FillBoundary_nowait() 1766 0.006203 0.006203 0.006203 1.60% MultiFab::LinComb() 690 0.005996 0.005996 0.005996 1.55% MLCellLinOp::defineAuxData() 6 0.005985 0.005985 0.005985 1.54% FabArray::ParallelCopy_nowait() 380 0.005807 0.005807 0.005807 1.50% StateDataPhysBCFunct::() 20 0.005517 0.005517 0.005517 1.42% MLPoisson::Fapply() 500 0.00495 0.00495 0.00495 1.28% Gravity::fill_multipole_BCs() 6 0.004707 0.004707 0.004707 1.21% Castro::enforce_min_density() 30 0.004605 0.004605 0.004605 1.19% Amr::restart() 1 0.003557 0.003557 0.003557 0.92% MLMG::addInterpCorrection() 180 0.003303 0.003303 0.003303 0.85% Castro::estTimeStep() 10 0.003191 0.003191 0.003191 0.82% amrex::average_down 180 0.002919 0.002919 0.002919 0.75% MultiFab::Xpay() 258 0.002834 0.002834 0.002834 0.73% Castro::do_advance_ctu() 5 0.002343 0.002343 0.002343 0.60% BndryData::define() 6 0.002032 0.002032 0.002032 0.52% Castro::construct_new_gravity_source() 5 0.001796 0.001796 0.001796 0.46% Castro::reset_internal_energy(MultiFab) 30 0.001705 0.001705 0.001705 0.44% Amr::writePlotFile() 1 0.001654 0.001654 0.001654 0.43% Castro::construct_old_gravity_source() 5 0.001522 0.001522 0.001522 0.39% Castro::subcycle_advance_ctu() 5 0.0009378 0.0009378 0.0009378 0.24% Castro::reset_internal_energy(Fab) 240 0.000936 0.000936 0.000936 0.24% Gravity::get_old_grav_vector() 5 0.0009313 0.0009313 0.0009313 0.24% MultiFab::Saxpy() 10 0.0009177 0.0009177 0.0009177 0.24% MLMG::ResNormInf() 42 0.0008853 0.0008853 0.0008853 0.23% Castro::expand_state() 5 0.0008635 0.0008635 0.0008635 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008616 0.0008616 0.0008616 0.22% Gravity::get_new_grav_vector() 5 0.0008546 0.0008546 0.0008546 0.22% Castro::advance() 5 0.0008198 0.0008198 0.0008198 0.21% MLCellLinOp::setLevelBC() 6 0.0007963 0.0007963 0.0007963 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007556 0.0007556 0.0007556 0.20% MultiFab::Add() 36 0.0007201 0.0007201 0.0007201 0.19% MLMG::prepareForSolve() 6 0.0006514 0.0006514 0.0006514 0.17% FabArray::mult() 22 0.0006432 0.0006432 0.0006432 0.17% FabArray::setDomainBndry() 20 0.0006292 0.0006292 0.0006292 0.16% MLCellLinOp::prepareForSolve() 6 0.0006119 0.0006119 0.0006119 0.16% MultiFab::contains_nan() 10 0.0005886 0.0005886 0.0005886 0.15% Castro::enforce_speed_limit() 30 0.000506 0.000506 0.000506 0.13% MLCellLinOp::smooth() 720 0.0004976 0.0004976 0.0004976 0.13% MLCellLinOp::compGrad() 6 0.0004744 0.0004744 0.0004744 0.12% FabArray::FillBoundary() 1766 0.0003957 0.0003957 0.0003957 0.10% Amr::InitAmr() 1 0.000387 0.000387 0.000387 0.10% FabArrayBase::CPC::define() 244 0.0003807 0.0003807 0.0003807 0.10% FabArrayBase::getCPC() 632 0.0003582 0.0003582 0.0003582 0.09% FabArrayBase::getFB() 1766 0.0002595 0.0002595 0.0002595 0.07% main() 1 0.0002426 0.0002426 0.0002426 0.06% Gravity::update_max_rhs() 6 0.0002178 0.0002178 0.0002178 0.06% MLCellLinOp::apply() 500 0.0002056 0.0002056 0.0002056 0.05% Gravity::solve_for_phi() 5 0.0001999 0.0001999 0.0001999 0.05% MultiFab::Copy() 6 0.0001713 0.0001713 0.0001713 0.04% CGSolver::sxay() 690 0.0001711 0.0001711 0.0001711 0.04% Castro::construct_new_gravity() 5 0.0001707 0.0001707 0.0001707 0.04% Amr::coarseTimeStep() 5 0.0001576 0.0001576 0.0001576 0.04% MLCellLinOp::defineBC() 6 0.000145 0.000145 0.000145 0.04% FillPatchIterator::Initialize 20 0.0001411 0.0001411 0.0001411 0.04% MultiFab::max() 6 0.0001319 0.0001319 0.0001319 0.03% MLCGSolver::ParallelAllReduce 659 0.0001214 0.0001214 0.0001214 0.03% FabArray::ParallelCopy() 380 0.0001202 0.0001202 0.0001202 0.03% MLMG::MLRhsNormInf() 6 0.0001119 0.0001119 0.0001119 0.03% MLMG::mgVcycle() 36 0.0001106 0.0001106 0.0001106 0.03% Castro::construct_new_source() 25 0.0001067 0.0001067 0.0001067 0.03% MLCellLinOp::correctionResidual() 216 0.0001048 0.0001048 0.0001048 0.03% Castro::create_source_corrector() 5 9.104e-05 9.104e-05 9.104e-05 0.02% MLLinOp::defineGrids() 6 8.999e-05 8.999e-05 8.999e-05 0.02% Castro::post_timestep() 5 8.015e-05 8.015e-05 8.015e-05 0.02% AmrLevel::restart() 1 7.977e-05 7.977e-05 7.977e-05 0.02% Castro::finalize_advance() 5 7.963e-05 7.963e-05 7.963e-05 0.02% Amr::timeStep() 5 7.7e-05 7.7e-05 7.7e-05 0.02% Castro::initialize_do_advance() 5 7.687e-05 7.687e-05 7.687e-05 0.02% Castro::construct_old_source() 25 7.641e-05 7.641e-05 7.641e-05 0.02% StateData::restartDoit() 4 7.436e-05 7.436e-05 7.436e-05 0.02% MLMG:computeResOfCorrection() 180 7.005e-05 7.005e-05 7.005e-05 0.02% FabArrayBase::FB::FB() 26 6.393e-05 6.393e-05 6.393e-05 0.02% MLMG::mgVcycle_down::0 36 5.154e-05 5.154e-05 5.154e-05 0.01% MLMG::mgVcycle_down::1 36 4.768e-05 4.768e-05 4.768e-05 0.01% MLMG::mgVcycle_down::2 36 4.511e-05 4.511e-05 4.511e-05 0.01% MLMG::mgVcycle_down::4 36 4.373e-05 4.373e-05 4.373e-05 0.01% MLMG::mgVcycle_down::3 36 4.273e-05 4.273e-05 4.273e-05 0.01% Castro::clean_state() 30 4.104e-05 4.104e-05 4.104e-05 0.01% Castro::initialize_advance() 5 3.847e-05 3.847e-05 3.847e-05 0.01% MLMG::actualBottomSolve() 36 3.643e-05 3.643e-05 3.643e-05 0.01% MLMG::mgVcycle_up::4 36 3.478e-05 3.478e-05 3.478e-05 0.01% MLMG::solve() 6 3.4e-05 3.4e-05 3.4e-05 0.01% Castro::buildMetrics() 1 3.185e-05 3.185e-05 3.185e-05 0.01% Gravity::actual_multilevel_solve() 1 2.924e-05 2.924e-05 2.924e-05 0.01% MLMG::mgVcycle_up::0 36 2.901e-05 2.901e-05 2.901e-05 0.01% MLMG::mgVcycle_up::3 36 2.848e-05 2.848e-05 2.848e-05 0.01% Castro::post_restart() 1 2.732e-05 2.732e-05 2.732e-05 0.01% MLMG::oneIter() 36 2.7e-05 2.7e-05 2.7e-05 0.01% MLMG::mgVcycle_up::2 36 2.648e-05 2.648e-05 2.648e-05 0.01% Amr::writeSmallPlotFile() 1 2.629e-05 2.629e-05 2.629e-05 0.01% Castro::initMFs() 1 2.579e-05 2.579e-05 2.579e-05 0.01% Castro::swap_state_time_levels() 5 2.565e-05 2.565e-05 2.565e-05 0.01% MLMG::mgVcycle_up::1 36 2.544e-05 2.544e-05 2.544e-05 0.01% MLCellLinOp::solutionResidual() 42 2.514e-05 2.514e-05 2.514e-05 0.01% MLPoisson::define() 6 2.09e-05 2.09e-05 2.09e-05 0.01% Castro::construct_old_gravity() 5 2.08e-05 2.08e-05 2.08e-05 0.01% Castro::computeNewDt() 5 2.041e-05 2.041e-05 2.041e-05 0.01% MLLinOp::define() 6 1.983e-05 1.983e-05 1.983e-05 0.01% Castro::finalize_do_advance() 5 1.816e-05 1.816e-05 1.816e-05 0.00% MLMG::computeResidual() 36 1.787e-05 1.787e-05 1.787e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.694e-05 1.694e-05 1.694e-05 0.00% MLMG::mgVcycle_bottom 36 1.542e-05 1.542e-05 1.542e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.39e-05 1.39e-05 1.39e-05 0.00% makeSFC 30 1.358e-05 1.358e-05 1.358e-05 0.00% FillPatchSingleLevel 20 1.318e-05 1.318e-05 1.318e-05 0.00% Castro::apply_source_to_state() 10 1.294e-05 1.294e-05 1.294e-05 0.00% Castro::do_old_sources() 5 8.977e-06 8.977e-06 8.977e-06 0.00% Castro::do_new_sources() 5 8.906e-06 8.906e-06 8.906e-06 0.00% DistributionMapping::Distribute() 31 8.673e-06 8.673e-06 8.673e-06 0.00% Amr::initSubcycle() 1 7.85e-06 7.85e-06 7.85e-06 0.00% Castro::check_for_nan() 10 7.572e-06 7.572e-06 7.572e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.569e-06 7.569e-06 7.569e-06 0.00% MLMG::computeMLResidual() 6 4.893e-06 4.893e-06 4.893e-06 0.00% MLPoisson::prepareForSolve() 6 4.619e-06 4.619e-06 4.619e-06 0.00% Gravity::swapTimeLevels() 5 4.284e-06 4.284e-06 4.284e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.549e-06 3.549e-06 3.549e-06 0.00% MLMG::getGradSolution() 6 3.326e-06 3.326e-06 3.326e-06 0.00% Gravity::set_mass_offset() 6 2.337e-06 2.337e-06 2.337e-06 0.00% MLMG::MLResNormInf() 6 2.172e-06 2.172e-06 2.172e-06 0.00% Castro::retry_advance_ctu() 5 1.715e-06 1.715e-06 1.715e-06 0.00% Castro::FluxRegCrseInit 5 1.592e-06 1.592e-06 1.592e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.185e-06 1.185e-06 1.185e-06 0.00% Amr::init() 1 1.11e-06 1.11e-06 1.11e-06 0.00% Castro::FluxRegFineAdd() 5 1.071e-06 1.071e-06 1.071e-06 0.00% AmrLevel::AmrLevel() 1 8.99e-07 8.99e-07 8.99e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3874 0.3874 0.3874 100.00% Amr::coarseTimeStep() 5 0.3071 0.3071 0.3071 79.26% Amr::timeStep() 5 0.3056 0.3056 0.3056 78.88% Castro::advance() 5 0.3007 0.3007 0.3007 77.62% Castro::subcycle_advance_ctu() 5 0.2946 0.2946 0.2946 76.04% Castro::do_advance_ctu() 5 0.2937 0.2937 0.2937 75.80% Castro::construct_new_gravity() 5 0.141 0.141 0.141 36.38% Gravity::solve_phi_with_mlmg() 6 0.137 0.137 0.137 35.35% Gravity::solve_for_phi() 5 0.1335 0.1335 0.1335 34.45% Gravity::actual_solve_with_mlmg() 6 0.1321 0.1321 0.1321 34.11% MLMG::solve() 6 0.1203 0.1203 0.1203 31.04% MLMG::oneIter() 36 0.1132 0.1132 0.1132 29.22% MLMG::mgVcycle() 36 0.1125 0.1125 0.1125 29.03% Castro::construct_ctu_hydro_source() 5 0.1111 0.1111 0.1111 28.66% MLCellLinOp::smooth() 720 0.05756 0.05756 0.05756 14.86% Amr::init() 1 0.04777 0.04777 0.04777 12.33% Amr::restart() 1 0.04777 0.04777 0.04777 12.33% MLCellLinOp::applyBC() 1946 0.04101 0.04101 0.04101 10.59% AmrLevel::restart() 1 0.04017 0.04017 0.04017 10.37% StateData::restartDoit() 4 0.04008 0.04008 0.04008 10.35% VisMF::Read() 3 0.03996 0.03996 0.03996 10.31% MLMG::mgVcycle_bottom 36 0.03422 0.03422 0.03422 8.83% MLMG::actualBottomSolve() 36 0.03421 0.03421 0.03421 8.83% MLCGSolver::bicgstab 36 0.03387 0.03387 0.03387 8.74% Amr::writePlotFile() 1 0.03189 0.03189 0.03189 8.23% VisMF::Write(FabArray) 1 0.03023 0.03023 0.03023 7.80% MLPoisson::Fsmooth() 1440 0.02669 0.02669 0.02669 6.89% Castro::clean_state() 30 0.02434 0.02434 0.02434 6.28% FillPatchIterator::Initialize 20 0.01944 0.01944 0.01944 5.02% FillPatchSingleLevel 20 0.01867 0.01867 0.01867 4.82% StateDataPhysBCFunct::() 20 0.0167 0.0167 0.0167 4.31% MLCellLinOp::apply() 500 0.01552 0.01552 0.01552 4.01% MLMG::mgVcycle_down::0 36 0.01515 0.01515 0.01515 3.91% MLMG::mgVcycle_up::0 36 0.01292 0.01292 0.01292 3.34% StateData::FillBoundary(geom) 160 0.01119 0.01119 0.01119 2.89% Castro::computeTemp() 30 0.01005 0.01005 0.01005 2.59% Castro::initialize_do_advance() 5 0.009888 0.009888 0.009888 2.55% MLPoisson::define() 6 0.009554 0.009554 0.009554 2.47% MultiFab::Dot() 484 0.009253 0.009253 0.009253 2.39% Castro::normalize_species() 30 0.009137 0.009137 0.009137 2.36% MLCellLinOp::correctionResidual() 216 0.009105 0.009105 0.009105 2.35% Castro::do_new_sources() 5 0.007928 0.007928 0.007928 2.05% MLMG:computeResOfCorrection() 180 0.007877 0.007877 0.007877 2.03% MLMG::mgVcycle_down::1 36 0.007533 0.007533 0.007533 1.94% Gravity::get_new_grav_vector() 5 0.007334 0.007334 0.007334 1.89% MLMG::mgVcycle_down::2 36 0.007297 0.007297 0.007297 1.88% Castro::construct_old_gravity() 5 0.007259 0.007259 0.007259 1.87% Gravity::get_old_grav_vector() 5 0.007238 0.007238 0.007238 1.87% MLMG::mgVcycle_down::3 36 0.006933 0.006933 0.006933 1.79% FabArray::FillBoundary() 1766 0.006922 0.006922 0.006922 1.79% MLCellLinOp::defineAuxData() 6 0.006698 0.006698 0.006698 1.73% FabArray::setVal() 537 0.006621 0.006621 0.006621 1.71% MLMG::mgVcycle_down::4 36 0.006619 0.006619 0.006619 1.71% FillBoundary_nowait() 1766 0.006526 0.006526 0.006526 1.68% FabArray::ParallelCopy() 380 0.006296 0.006296 0.006296 1.62% FabArray::ParallelCopy_nowait() 380 0.006176 0.006176 0.006176 1.59% CGSolver::sxay() 690 0.006167 0.006167 0.006167 1.59% MultiFab::LinComb() 690 0.005996 0.005996 0.005996 1.55% Castro::do_old_sources() 5 0.005812 0.005812 0.005812 1.50% MLMG::mgVcycle_up::2 36 0.005603 0.005603 0.005603 1.45% Castro::expand_state() 5 0.005603 0.005603 0.005603 1.45% MLCGSolver::ParallelAllReduce 659 0.005543 0.005543 0.005543 1.43% MLMG::mgVcycle_up::1 36 0.005505 0.005505 0.005505 1.42% MLMG::addInterpCorrection() 180 0.005456 0.005456 0.005456 1.41% MLMG::mgVcycle_up::3 36 0.005321 0.005321 0.005321 1.37% MLMG::mgVcycle_up::4 36 0.005243 0.005243 0.005243 1.35% Castro::initialize_advance() 5 0.005192 0.005192 0.005192 1.34% amrex::average_down 180 0.005105 0.005105 0.005105 1.32% MLPoisson::Fapply() 500 0.00495 0.00495 0.00495 1.28% Castro::post_timestep() 5 0.004822 0.004822 0.004822 1.24% Gravity::fill_multipole_BCs() 6 0.004707 0.004707 0.004707 1.21% Castro::enforce_min_density() 30 0.004605 0.004605 0.004605 1.19% Castro::post_restart() 1 0.003867 0.003867 0.003867 1.00% Gravity::multilevel_solve_for_new_phi() 1 0.003749 0.003749 0.003749 0.97% Gravity::actual_multilevel_solve() 1 0.003733 0.003733 0.003733 0.96% Castro::estTimeStep() 10 0.003191 0.003191 0.003191 0.82% MLCellLinOp::solutionResidual() 42 0.003189 0.003189 0.003189 0.82% MultiFab::Xpay() 258 0.002834 0.002834 0.002834 0.73% MLMG::prepareForSolve() 6 0.002826 0.002826 0.002826 0.73% MLCellLinOp::defineBC() 6 0.002695 0.002695 0.002695 0.70% MLMG::computeResidual() 36 0.002646 0.002646 0.002646 0.68% Castro::reset_internal_energy(MultiFab) 30 0.002641 0.002641 0.002641 0.68% BndryData::define() 6 0.00255 0.00255 0.00255 0.66% Castro::construct_new_source() 25 0.001903 0.001903 0.001903 0.49% Castro::construct_new_gravity_source() 5 0.001796 0.001796 0.001796 0.46% Castro::construct_old_source() 25 0.001598 0.001598 0.001598 0.41% Castro::construct_old_gravity_source() 5 0.001522 0.001522 0.001522 0.39% Castro::computeNewDt() 5 0.001322 0.001322 0.001322 0.34% Castro::reset_internal_energy(Fab) 240 0.000936 0.000936 0.000936 0.24% Castro::apply_source_to_state() 10 0.0009307 0.0009307 0.0009307 0.24% MultiFab::Saxpy() 10 0.0009177 0.0009177 0.0009177 0.24% MLMG::ResNormInf() 42 0.0008853 0.0008853 0.0008853 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008616 0.0008616 0.0008616 0.22% MLCellLinOp::setLevelBC() 6 0.0007963 0.0007963 0.0007963 0.21% MLMG::getGradSolution() 6 0.000754 0.000754 0.000754 0.19% MLCellLinOp::compGrad() 6 0.0007507 0.0007507 0.0007507 0.19% FabArrayBase::getCPC() 632 0.0007389 0.0007389 0.0007389 0.19% MultiFab::Add() 36 0.0007201 0.0007201 0.0007201 0.19% FabArray::mult() 22 0.0006432 0.0006432 0.0006432 0.17% FabArray::setDomainBndry() 20 0.0006292 0.0006292 0.0006292 0.16% MLPoisson::prepareForSolve() 6 0.0006165 0.0006165 0.0006165 0.16% MLCellLinOp::prepareForSolve() 6 0.0006119 0.0006119 0.0006119 0.16% Castro::check_for_nan() 10 0.0005961 0.0005961 0.0005961 0.15% MultiFab::contains_nan() 10 0.0005886 0.0005886 0.0005886 0.15% MLMG::computeMLResidual() 6 0.0005659 0.0005659 0.0005659 0.15% Castro::enforce_speed_limit() 30 0.000506 0.000506 0.000506 0.13% Gravity::update_max_rhs() 6 0.0004271 0.0004271 0.0004271 0.11% Amr::InitAmr() 1 0.0003948 0.0003948 0.0003948 0.10% FabArrayBase::CPC::define() 244 0.0003807 0.0003807 0.0003807 0.10% FabArrayBase::getFB() 1766 0.0003234 0.0003234 0.0003234 0.08% Gravity::swapTimeLevels() 5 0.0002253 0.0002253 0.0002253 0.06% MultiFab::Copy() 6 0.0001713 0.0001713 0.0001713 0.04% Castro::buildMetrics() 1 0.00015 0.00015 0.00015 0.04% MLMG::MLResNormInf() 6 0.0001472 0.0001472 0.0001472 0.04% MLLinOp::define() 6 0.0001394 0.0001394 0.0001394 0.04% MultiFab::max() 6 0.0001319 0.0001319 0.0001319 0.03% MLLinOp::defineGrids() 6 0.0001196 0.0001196 0.0001196 0.03% MLMG::MLRhsNormInf() 6 0.0001119 0.0001119 0.0001119 0.03% Castro::create_source_corrector() 5 9.104e-05 9.104e-05 9.104e-05 0.02% Castro::finalize_advance() 5 8.229e-05 8.229e-05 8.229e-05 0.02% FabArrayBase::FB::FB() 26 6.393e-05 6.393e-05 6.393e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.841e-05 2.841e-05 2.841e-05 0.01% Amr::writeSmallPlotFile() 1 2.629e-05 2.629e-05 2.629e-05 0.01% Castro::initMFs() 1 2.579e-05 2.579e-05 2.579e-05 0.01% Castro::swap_state_time_levels() 5 2.565e-05 2.565e-05 2.565e-05 0.01% makeSFC 30 2.084e-05 2.084e-05 2.084e-05 0.01% Castro::finalize_do_advance() 5 1.816e-05 1.816e-05 1.816e-05 0.00% DistributionMapping::Distribute() 31 8.673e-06 8.673e-06 8.673e-06 0.00% Amr::initSubcycle() 1 7.85e-06 7.85e-06 7.85e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.965e-06 4.965e-06 4.965e-06 0.00% Gravity::set_mass_offset() 6 2.337e-06 2.337e-06 2.337e-06 0.00% Castro::retry_advance_ctu() 5 1.715e-06 1.715e-06 1.715e-06 0.00% Castro::FluxRegCrseInit 5 1.592e-06 1.592e-06 1.592e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.185e-06 1.185e-06 1.185e-06 0.00% Castro::FluxRegFineAdd() 5 1.071e-06 1.071e-06 1.071e-06 0.00% AmrLevel::AmrLevel() 1 8.99e-07 8.99e-07 8.99e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-24-g3ec07681574a) finalized