Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-24-gea7e8a507e77) initialized Starting run at 10:10:06 UTC on 2022-12-30. Successfully read inputs file ... Castro git describe: 22.12-11-g45d01ffe1 AMReX git describe: 22.12-24-gea7e8a507 Microphysics git describe: 22.12-28-gdc4a7e21 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.058175131 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.034362105 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.044770404 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049149545 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.047605855 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.056954359 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.077892512 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.057814624 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.059445945 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.061705832 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.059644541 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.058439922 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.061174837 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.057708433 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033746208 seconds Ending run at 10:10:07 UTC on 2022-12-30. Run time = 0.867349614 Run time without initialization = 0.726667725 Average number of zones advanced per microsecond: 3.607 Average number of zones advanced per microsecond per rank: 3.607 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8674 ... 0.8674 ... 0.8674 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2349 0.2349 0.2349 27.08% Castro::construct_ctu_hydro_source() 10 0.2203 0.2203 0.2203 25.40% MLCellLinOp::applyBC() 4433 0.07502 0.07502 0.07502 8.65% MLPoisson::Fsmooth() 3280 0.03216 0.03216 0.03216 3.71% FillBoundary_nowait() 4023 0.03139 0.03139 0.03139 3.62% StateData::FillBoundary(geom) 328 0.02601 0.02601 0.02601 3.00% amrex::Dot() 1114 0.02029 0.02029 0.02029 2.34% Castro::normalize_species() 62 0.01561 0.01561 0.01561 1.80% amrex::Copy() 1029 0.01479 0.01479 0.01479 1.70% FabArray::norminf() 743 0.01429 0.01429 0.01429 1.65% Castro::computeTemp() 63 0.01342 0.01342 0.01342 1.55% FabArray::ParallelCopy_nowait() 861 0.01312 0.01312 0.01312 1.51% FabArray::setVal() 1144 0.01303 0.01303 0.01303 1.50% StateDataPhysBCFunct::() 41 0.0122 0.0122 0.0122 1.41% MLPoisson::Fapply() 1142 0.01038 0.01038 0.01038 1.20% MLCellLinOp::defineAuxData() 11 0.009642 0.009642 0.009642 1.11% Castro::enforce_min_density() 62 0.009301 0.009301 0.009301 1.07% FabArray::Saxpy() 813 0.008168 0.008168 0.008168 0.94% FabArray::Xpay() 821 0.008118 0.008118 0.008118 0.94% MLMG::addInterpCorrection() 410 0.006558 0.006558 0.006558 0.76% Gravity::fill_multipole_BCs() 11 0.006412 0.006412 0.006412 0.74% amrex::average_down 410 0.005784 0.005784 0.005784 0.67% FabArray::LinComb() 557 0.004508 0.004508 0.004508 0.52% amrex::Add() 164 0.004316 0.004316 0.004316 0.50% Castro::reset_internal_energy(MultiFab) 63 0.00427 0.00427 0.00427 0.49% Castro::estTimeStep() 21 0.004236 0.004236 0.004236 0.49% Amr::checkPoint() 3 0.00404 0.00404 0.00404 0.47% BndryData::define() 11 0.003672 0.003672 0.003672 0.42% Castro::construct_new_gravity_source() 10 0.003341 0.003341 0.003341 0.39% Castro::do_advance_ctu() 10 0.003179 0.003179 0.003179 0.37% Castro::construct_old_gravity_source() 10 0.002674 0.002674 0.002674 0.31% Amr::writePlotFile() 2 0.002397 0.002397 0.002397 0.28% MLCGSolver::bicgstab 82 0.002139 0.002139 0.002139 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001674 0.001674 0.001674 0.19% Castro::enforce_speed_limit() 62 0.001551 0.001551 0.001551 0.18% Castro::reset_internal_energy(Fab) 504 0.001483 0.001483 0.001483 0.17% MLCellLinOp::setLevelBC() 11 0.001374 0.001374 0.001374 0.16% Gravity::actual_solve_with_mlmg() 11 0.001369 0.001369 0.001369 0.16% FabArray::mult() 43 0.001327 0.001327 0.001327 0.15% FabArray::setDomainBndry() 41 0.001297 0.001297 0.001297 0.15% Castro::initData() 1 0.00123 0.00123 0.00123 0.14% MLCellLinOp::smooth() 1640 0.001203 0.001203 0.001203 0.14% MultiFab::contains_nan() 20 0.001179 0.001179 0.001179 0.14% MLCellLinOp::prepareForSolve() 11 0.001114 0.001114 0.001114 0.13% MLCellLinOp::compGrad() 11 0.0009169 0.0009169 0.0009169 0.11% MLMG::prepareForSolve() 11 0.0008883 0.0008883 0.0008883 0.10% FabArray::FillBoundary() 4023 0.0007979 0.0007979 0.0007979 0.09% FabArrayBase::getCPC() 1323 0.0007347 0.0007347 0.0007347 0.08% FabArrayBase::CPC::define() 454 0.000683 0.000683 0.000683 0.08% FabArrayBase::getFB() 4023 0.0006179 0.0006179 0.0006179 0.07% Gravity::get_new_grav_vector() 11 0.0006167 0.0006167 0.0006167 0.07% Gravity::get_old_grav_vector() 10 0.000553 0.000553 0.000553 0.06% Amr::InitAmr() 1 0.0005165 0.0005165 0.0005165 0.06% MLCellLinOp::apply() 1142 0.0004862 0.0004862 0.0004862 0.06% MLMG::mgVcycle() 82 0.0004008 0.0004008 0.0004008 0.05% Amr::coarseTimeStep() 10 0.0003265 0.0003265 0.0003265 0.04% main() 1 0.0002864 0.0002864 0.0002864 0.03% MLCGSolver::ParallelAllReduce 1514 0.0002822 0.0002822 0.0002822 0.03% MultiFab::max() 11 0.0002566 0.0002566 0.0002566 0.03% FabArray::ParallelCopy() 861 0.0002406 0.0002406 0.0002406 0.03% MLCellLinOp::correctionResidual() 492 0.000236 0.000236 0.000236 0.03% MLLinOp::defineGrids() 11 0.0002207 0.0002207 0.0002207 0.03% FillPatchIterator::Initialize 41 0.0002141 0.0002141 0.0002141 0.02% MLCellLinOp::defineBC() 11 0.0002057 0.0002057 0.0002057 0.02% Gravity::solve_for_phi() 10 0.0001744 0.0001744 0.0001744 0.02% Castro::finalize_advance() 10 0.0001568 0.0001568 0.0001568 0.02% Castro::subcycle_advance_ctu() 10 0.0001528 0.0001528 0.0001528 0.02% Amr::timeStep() 10 0.0001507 0.0001507 0.0001507 0.02% StateData::checkPoint() 12 0.0001327 0.0001327 0.0001327 0.02% MLMG:computeResOfCorrection() 410 0.0001147 0.0001147 0.0001147 0.01% Gravity::update_max_rhs() 11 0.0001142 0.0001142 0.0001142 0.01% MLMG::mgVcycle_down::0 82 9.596e-05 9.596e-05 9.596e-05 0.01% Castro::advance() 10 9.359e-05 9.359e-05 9.359e-05 0.01% Castro::clean_state() 62 9.316e-05 9.316e-05 9.316e-05 0.01% MLMG::actualBottomSolve() 82 8.975e-05 8.975e-05 8.975e-05 0.01% MLMG::mgVcycle_down::1 82 8.767e-05 8.767e-05 8.767e-05 0.01% Castro::Castro() 1 8.413e-05 8.413e-05 8.413e-05 0.01% MLMG::mgVcycle_down::2 82 8.403e-05 8.403e-05 8.403e-05 0.01% MLMG::solve() 11 7.99e-05 7.99e-05 7.99e-05 0.01% FabArrayBase::FB::FB() 56 7.988e-05 7.988e-05 7.988e-05 0.01% MLMG::mgVcycle_down::4 82 7.857e-05 7.857e-05 7.857e-05 0.01% MLMG::mgVcycle_down::3 82 7.81e-05 7.81e-05 7.81e-05 0.01% Castro::expand_state() 10 7.379e-05 7.379e-05 7.379e-05 0.01% AmrLevel::checkPoint() 3 7.343e-05 7.343e-05 7.343e-05 0.01% MLMG::mgVcycle_up::4 82 7.003e-05 7.003e-05 7.003e-05 0.01% Castro::initialize_advance() 10 6.593e-05 6.593e-05 6.593e-05 0.01% MLMG::mgVcycle_up::0 82 5.905e-05 5.905e-05 5.905e-05 0.01% MLMG::mgVcycle_up::1 82 5.587e-05 5.587e-05 5.587e-05 0.01% MLCellLinOp::solutionResidual() 93 5.443e-05 5.443e-05 5.443e-05 0.01% MLMG::mgVcycle_up::3 82 5.358e-05 5.358e-05 5.358e-05 0.01% MLMG::mgVcycle_up::2 82 5.337e-05 5.337e-05 5.337e-05 0.01% MLMG::oneIter() 82 5.011e-05 5.011e-05 5.011e-05 0.01% Castro::initialize_do_advance() 10 4.37e-05 4.37e-05 4.37e-05 0.01% Castro::finalize_do_advance() 10 3.986e-05 3.986e-05 3.986e-05 0.00% Castro::swap_state_time_levels() 10 3.418e-05 3.418e-05 3.418e-05 0.00% Castro::enforce_consistent_e() 1 3.334e-05 3.334e-05 3.334e-05 0.00% MLMG::computeResidual() 82 3.312e-05 3.312e-05 3.312e-05 0.00% StateData::define() 4 3.307e-05 3.307e-05 3.307e-05 0.00% MLMG::ResNormInf() 93 3.181e-05 3.181e-05 3.181e-05 0.00% MLMG::mgVcycle_bottom 82 3.131e-05 3.131e-05 3.131e-05 0.00% FillPatchSingleLevel 41 3.009e-05 3.009e-05 3.009e-05 0.00% Amr::FinalizeInit() 1 2.88e-05 2.88e-05 2.88e-05 0.00% makeSFC 55 2.537e-05 2.537e-05 2.537e-05 0.00% Castro::construct_new_gravity() 10 2.528e-05 2.528e-05 2.528e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.469e-05 2.469e-05 2.469e-05 0.00% Amr::writeSmallPlotFile() 1 2.465e-05 2.465e-05 2.465e-05 0.00% Castro::create_source_corrector() 10 2.348e-05 2.348e-05 2.348e-05 0.00% MLPoisson::define() 11 2.037e-05 2.037e-05 2.037e-05 0.00% Castro::construct_old_source() 50 1.913e-05 1.913e-05 1.913e-05 0.00% Castro::initMFs() 1 1.899e-05 1.899e-05 1.899e-05 0.00% Castro::construct_new_source() 50 1.887e-05 1.887e-05 1.887e-05 0.00% Castro::do_new_sources() 10 1.784e-05 1.784e-05 1.784e-05 0.00% Castro::buildMetrics() 1 1.69e-05 1.69e-05 1.69e-05 0.00% DistributionMapping::Distribute() 56 1.543e-05 1.543e-05 1.543e-05 0.00% Castro::do_old_sources() 10 1.538e-05 1.538e-05 1.538e-05 0.00% Amr::defBaseLevel() 1 1.411e-05 1.411e-05 1.411e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.392e-05 1.392e-05 1.392e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.285e-05 1.285e-05 1.285e-05 0.00% Castro::check_for_nan() 20 1.15e-05 1.15e-05 1.15e-05 0.00% Castro::apply_source_to_state() 20 1.126e-05 1.126e-05 1.126e-05 0.00% MLLinOp::define() 11 1.116e-05 1.116e-05 1.116e-05 0.00% Castro::post_init() 1 1.047e-05 1.047e-05 1.047e-05 0.00% Castro::construct_old_gravity() 10 1.033e-05 1.033e-05 1.033e-05 0.00% Gravity::swapTimeLevels() 10 9.184e-06 9.184e-06 9.184e-06 0.00% Castro::post_timestep() 10 9.141e-06 9.141e-06 9.141e-06 0.00% AmrLevel::AmrLevel(dm) 1 8.99e-06 8.99e-06 8.99e-06 0.00% Amr::initSubcycle() 1 8.951e-06 8.951e-06 8.951e-06 0.00% MLPoisson::prepareForSolve() 11 8.918e-06 8.918e-06 8.918e-06 0.00% MLMG::computeMLResidual() 11 8.765e-06 8.765e-06 8.765e-06 0.00% Castro::computeNewDt() 9 7.364e-06 7.364e-06 7.364e-06 0.00% Gravity::actual_multilevel_solve() 1 7.06e-06 7.06e-06 7.06e-06 0.00% MLMG::getGradSolution() 11 5.929e-06 5.929e-06 5.929e-06 0.00% Amr::InitializeInit() 1 5.011e-06 5.011e-06 5.011e-06 0.00% Gravity::set_mass_offset() 11 4.403e-06 4.403e-06 4.403e-06 0.00% AmrLevel::checkPointPost() 3 4.302e-06 4.302e-06 4.302e-06 0.00% MLMG::MLRhsNormInf() 11 3.977e-06 3.977e-06 3.977e-06 0.00% MLMG::MLResNormInf() 11 3.941e-06 3.941e-06 3.941e-06 0.00% Castro::retry_advance_ctu() 10 3.835e-06 3.835e-06 3.835e-06 0.00% Castro::FluxRegCrseInit 10 3.252e-06 3.252e-06 3.252e-06 0.00% Castro::computeInitialDt() 2 3.029e-06 3.029e-06 3.029e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.964e-06 2.964e-06 2.964e-06 0.00% Amr::init() 1 2.418e-06 2.418e-06 2.418e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.39e-06 2.39e-06 2.39e-06 0.00% Castro::FluxRegFineAdd() 10 2.277e-06 2.277e-06 2.277e-06 0.00% AmrLevel::checkPointPre() 3 1.656e-06 1.656e-06 1.656e-06 0.00% Amr::initialInit() 1 1.334e-06 1.334e-06 1.334e-06 0.00% Castro::post_regrid() 1 1.063e-06 1.063e-06 1.063e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8674 0.8674 0.8674 100.00% Amr::coarseTimeStep() 10 0.6927 0.6927 0.6927 79.86% Amr::timeStep() 10 0.5743 0.5743 0.5743 66.21% Castro::advance() 10 0.5667 0.5667 0.5667 65.33% Castro::subcycle_advance_ctu() 10 0.5538 0.5538 0.5538 63.85% Castro::do_advance_ctu() 10 0.5536 0.5536 0.5536 63.83% Gravity::solve_phi_with_mlmg() 11 0.2804 0.2804 0.2804 32.33% Gravity::actual_solve_with_mlmg() 11 0.2735 0.2735 0.2735 31.53% Castro::construct_new_gravity() 10 0.257 0.257 0.257 29.63% MLMG::solve() 11 0.2533 0.2533 0.2533 29.20% Gravity::solve_for_phi() 10 0.2408 0.2408 0.2408 27.76% MLMG::oneIter() 82 0.2391 0.2391 0.2391 27.56% MLMG::mgVcycle() 82 0.2355 0.2355 0.2355 27.15% VisMF::Write(FabArray) 11 0.2349 0.2349 0.2349 27.08% Castro::construct_ctu_hydro_source() 10 0.2203 0.2203 0.2203 25.40% Amr::checkPoint() 3 0.1738 0.1738 0.1738 20.04% AmrLevel::checkPoint() 3 0.1698 0.1698 0.1698 19.58% StateData::checkPoint() 12 0.1697 0.1697 0.1697 19.57% Amr::init() 1 0.14 0.14 0.14 16.14% MLCellLinOp::smooth() 1640 0.1158 0.1158 0.1158 13.36% MLCellLinOp::applyBC() 4433 0.1079 0.1079 0.1079 12.44% MLMG::mgVcycle_bottom 82 0.07302 0.07302 0.07302 8.42% MLMG::actualBottomSolve() 82 0.07298 0.07298 0.07298 8.41% MLCGSolver::bicgstab 82 0.07229 0.07229 0.07229 8.33% Amr::writePlotFile() 2 0.06824 0.06824 0.06824 7.87% Amr::initialInit() 1 0.04735 0.04735 0.04735 5.46% Castro::clean_state() 62 0.04484 0.04484 0.04484 5.17% FillPatchIterator::Initialize 41 0.0438 0.0438 0.0438 5.05% Amr::FinalizeInit() 1 0.04326 0.04326 0.04326 4.99% FillPatchSingleLevel 41 0.04229 0.04229 0.04229 4.88% Castro::post_init() 1 0.04198 0.04198 0.04198 4.84% Gravity::multilevel_solve_for_new_phi() 1 0.04005 0.04005 0.04005 4.62% Gravity::actual_multilevel_solve() 1 0.04004 0.04004 0.04004 4.62% StateDataPhysBCFunct::() 41 0.03821 0.03821 0.03821 4.41% MLCellLinOp::apply() 1142 0.03582 0.03582 0.03582 4.13% MLMG::mgVcycle_down::0 82 0.03366 0.03366 0.03366 3.88% FabArray::FillBoundary() 4023 0.03288 0.03288 0.03288 3.79% MLPoisson::Fsmooth() 3280 0.03216 0.03216 0.03216 3.71% FillBoundary_nowait() 4023 0.03209 0.03209 0.03209 3.70% StateData::FillBoundary(geom) 328 0.02601 0.02601 0.02601 3.00% MLMG::mgVcycle_up::0 82 0.02564 0.02564 0.02564 2.96% MLCellLinOp::correctionResidual() 492 0.02208 0.02208 0.02208 2.55% amrex::Dot() 1114 0.02029 0.02029 0.02029 2.34% MLMG:computeResOfCorrection() 410 0.01948 0.01948 0.01948 2.25% Castro::computeTemp() 63 0.01917 0.01917 0.01917 2.21% Castro::initialize_do_advance() 10 0.01853 0.01853 0.01853 2.14% Gravity::get_new_grav_vector() 11 0.01784 0.01784 0.01784 2.06% MLPoisson::define() 11 0.01612 0.01612 0.01612 1.86% Castro::construct_old_gravity() 10 0.01594 0.01594 0.01594 1.84% Gravity::get_old_grav_vector() 10 0.01593 0.01593 0.01593 1.84% Castro::normalize_species() 62 0.01561 0.01561 0.01561 1.80% MLMG::mgVcycle_down::1 82 0.01556 0.01556 0.01556 1.79% amrex::Copy() 1029 0.01479 0.01479 0.01479 1.70% MLMG::mgVcycle_down::2 82 0.0145 0.0145 0.0145 1.67% FabArray::norminf() 743 0.01429 0.01429 0.01429 1.65% FabArray::ParallelCopy() 861 0.01416 0.01416 0.01416 1.63% MLMG::mgVcycle_down::3 82 0.01414 0.01414 0.01414 1.63% MLMG::mgVcycle_down::4 82 0.01397 0.01397 0.01397 1.61% FabArray::ParallelCopy_nowait() 861 0.01392 0.01392 0.01392 1.60% FabArray::setVal() 1144 0.01303 0.01303 0.01303 1.50% MLCGSolver::ParallelAllReduce 1514 0.01218 0.01218 0.01218 1.40% Castro::initialize_advance() 10 0.01209 0.01209 0.01209 1.39% MLMG::addInterpCorrection() 410 0.01166 0.01166 0.01166 1.34% Castro::do_old_sources() 10 0.01153 0.01153 0.01153 1.33% MLMG::mgVcycle_up::4 82 0.01136 0.01136 0.01136 1.31% MLMG::mgVcycle_up::1 82 0.0113 0.0113 0.0113 1.30% Castro::expand_state() 10 0.01125 0.01125 0.01125 1.30% MLMG::mgVcycle_up::2 82 0.01106 0.01106 0.01106 1.28% MLCellLinOp::defineAuxData() 11 0.01099 0.01099 0.01099 1.27% MLMG::mgVcycle_up::3 82 0.01087 0.01087 0.01087 1.25% amrex::average_down 410 0.0108 0.0108 0.0108 1.24% Castro::do_new_sources() 10 0.01066 0.01066 0.01066 1.23% MLPoisson::Fapply() 1142 0.01038 0.01038 0.01038 1.20% Castro::enforce_min_density() 62 0.009301 0.009301 0.009301 1.07% FabArray::Saxpy() 813 0.008168 0.008168 0.008168 0.94% FabArray::Xpay() 821 0.008118 0.008118 0.008118 0.94% Castro::post_timestep() 10 0.007535 0.007535 0.007535 0.87% MLCellLinOp::solutionResidual() 93 0.007102 0.007102 0.007102 0.82% Gravity::fill_multipole_BCs() 11 0.006646 0.006646 0.006646 0.77% MLMG::computeResidual() 82 0.006131 0.006131 0.006131 0.71% Castro::reset_internal_energy(MultiFab) 63 0.005753 0.005753 0.005753 0.66% MLCellLinOp::defineBC() 11 0.004828 0.004828 0.004828 0.56% MLMG::prepareForSolve() 11 0.004641 0.004641 0.004641 0.54% BndryData::define() 11 0.004623 0.004623 0.004623 0.53% FabArray::LinComb() 557 0.004508 0.004508 0.004508 0.52% amrex::Add() 164 0.004316 0.004316 0.004316 0.50% Castro::estTimeStep() 21 0.004236 0.004236 0.004236 0.49% Amr::InitializeInit() 1 0.004086 0.004086 0.004086 0.47% Amr::defBaseLevel() 1 0.004081 0.004081 0.004081 0.47% Castro::initData() 1 0.00359 0.00359 0.00359 0.41% Castro::construct_new_source() 50 0.003359 0.003359 0.003359 0.39% Castro::construct_new_gravity_source() 10 0.003341 0.003341 0.003341 0.39% Castro::construct_old_source() 50 0.002693 0.002693 0.002693 0.31% Castro::construct_old_gravity_source() 10 0.002674 0.002674 0.002674 0.31% MLMG::ResNormInf() 93 0.002108 0.002108 0.002108 0.24% Castro::apply_source_to_state() 20 0.00182 0.00182 0.00182 0.21% Castro::computeNewDt() 9 0.001719 0.001719 0.001719 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001674 0.001674 0.001674 0.19% Castro::enforce_speed_limit() 62 0.001551 0.001551 0.001551 0.18% Castro::reset_internal_energy(Fab) 504 0.001483 0.001483 0.001483 0.17% FabArrayBase::getCPC() 1323 0.001418 0.001418 0.001418 0.16% MLMG::getGradSolution() 11 0.001385 0.001385 0.001385 0.16% MLCellLinOp::compGrad() 11 0.001379 0.001379 0.001379 0.16% MLCellLinOp::setLevelBC() 11 0.001374 0.001374 0.001374 0.16% FabArray::mult() 43 0.001327 0.001327 0.001327 0.15% FabArray::setDomainBndry() 41 0.001297 0.001297 0.001297 0.15% Castro::check_for_nan() 20 0.001191 0.001191 0.001191 0.14% MultiFab::contains_nan() 20 0.001179 0.001179 0.001179 0.14% MLPoisson::prepareForSolve() 11 0.001123 0.001123 0.001123 0.13% MLCellLinOp::prepareForSolve() 11 0.001114 0.001114 0.001114 0.13% Castro::post_regrid() 1 0.001099 0.001099 0.001099 0.13% MLMG::computeMLResidual() 11 0.001013 0.001013 0.001013 0.12% Castro::computeInitialDt() 2 0.0008617 0.0008617 0.0008617 0.10% Gravity::update_max_rhs() 11 0.0008034 0.0008034 0.0008034 0.09% FabArrayBase::getFB() 4023 0.0006978 0.0006978 0.0006978 0.08% FabArrayBase::CPC::define() 454 0.000683 0.000683 0.000683 0.08% Castro::finalize_advance() 10 0.0006721 0.0006721 0.0006721 0.08% Amr::InitAmr() 1 0.0005255 0.0005255 0.0005255 0.06% Gravity::swapTimeLevels() 10 0.0004338 0.0004338 0.0004338 0.05% Castro::Castro() 1 0.0004306 0.0004306 0.0004306 0.05% MLLinOp::define() 11 0.0002867 0.0002867 0.0002867 0.03% MLMG::MLResNormInf() 11 0.0002826 0.0002826 0.0002826 0.03% MLLinOp::defineGrids() 11 0.0002755 0.0002755 0.0002755 0.03% MultiFab::max() 11 0.0002566 0.0002566 0.0002566 0.03% MLMG::MLRhsNormInf() 11 0.0002186 0.0002186 0.0002186 0.03% Castro::buildMetrics() 1 0.0001559 0.0001559 0.0001559 0.02% FabArrayBase::FB::FB() 56 7.988e-05 7.988e-05 7.988e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.241e-05 5.241e-05 5.241e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.206e-05 4.206e-05 4.206e-05 0.00% Castro::finalize_do_advance() 10 3.986e-05 3.986e-05 3.986e-05 0.00% makeSFC 55 3.956e-05 3.956e-05 3.956e-05 0.00% Castro::swap_state_time_levels() 10 3.418e-05 3.418e-05 3.418e-05 0.00% Castro::enforce_consistent_e() 1 3.334e-05 3.334e-05 3.334e-05 0.00% StateData::define() 4 3.307e-05 3.307e-05 3.307e-05 0.00% Amr::writeSmallPlotFile() 1 2.465e-05 2.465e-05 2.465e-05 0.00% Castro::create_source_corrector() 10 2.348e-05 2.348e-05 2.348e-05 0.00% Castro::initMFs() 1 1.899e-05 1.899e-05 1.899e-05 0.00% DistributionMapping::Distribute() 56 1.543e-05 1.543e-05 1.543e-05 0.00% Amr::initSubcycle() 1 8.951e-06 8.951e-06 8.951e-06 0.00% Gravity::set_mass_offset() 11 4.403e-06 4.403e-06 4.403e-06 0.00% AmrLevel::checkPointPost() 3 4.302e-06 4.302e-06 4.302e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.199e-06 4.199e-06 4.199e-06 0.00% Castro::retry_advance_ctu() 10 3.835e-06 3.835e-06 3.835e-06 0.00% Castro::FluxRegCrseInit 10 3.252e-06 3.252e-06 3.252e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.39e-06 2.39e-06 2.39e-06 0.00% Castro::FluxRegFineAdd() 10 2.277e-06 2.277e-06 2.277e-06 0.00% AmrLevel::checkPointPre() 3 1.656e-06 1.656e-06 1.656e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-24-gea7e8a507e77) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-24-gea7e8a507e77) initialized Starting run at 10:10:08 UTC on 2022-12-30. Successfully read inputs file ... Castro git describe: 22.12-11-g45d01ffe1 AMReX git describe: 22.12-24-gea7e8a507 Microphysics git describe: 22.12-28-gdc4a7e21 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.474873936 Restart time = 0.04632585 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.050785582 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.052253169 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.060116956 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060395281 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.066307414 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032962009 seconds Ending run at 10:10:08 UTC on 2022-12-30. Run time = 0.370189838 Run time without initialization = 0.323278136 Average number of zones advanced per microsecond: 4.054 Average number of zones advanced per microsecond per rank: 4.054 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3702 ... 0.3702 ... 0.3702 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1034 0.1034 0.1034 27.93% VisMF::Read() 3 0.04021 0.04021 0.04021 10.86% MLCellLinOp::applyBC() 1946 0.03287 0.03287 0.03287 8.88% VisMF::Write(FabArray) 1 0.03141 0.03141 0.03141 8.49% MLPoisson::Fsmooth() 1440 0.0142 0.0142 0.0142 3.84% FillBoundary_nowait() 1766 0.01282 0.01282 0.01282 3.46% StateData::FillBoundary(geom) 160 0.01171 0.01171 0.01171 3.16% Castro::normalize_species() 30 0.01011 0.01011 0.01011 2.73% amrex::Dot() 484 0.008837 0.008837 0.008837 2.39% Castro::computeTemp() 30 0.007468 0.007468 0.007468 2.02% amrex::Copy() 463 0.007058 0.007058 0.007058 1.91% FabArray::setVal() 537 0.00627 0.00627 0.00627 1.69% FabArray::norminf() 326 0.006241 0.006241 0.006241 1.69% FabArray::ParallelCopy_nowait() 380 0.006003 0.006003 0.006003 1.62% StateDataPhysBCFunct::() 20 0.005945 0.005945 0.005945 1.61% MLCellLinOp::defineAuxData() 6 0.005255 0.005255 0.005255 1.42% Castro::enforce_min_density() 30 0.00511 0.00511 0.00511 1.38% MLPoisson::Fapply() 500 0.004566 0.004566 0.004566 1.23% FabArray::Saxpy() 355 0.003685 0.003685 0.003685 1.00% FabArray::Xpay() 361 0.00354 0.00354 0.00354 0.96% MLMG::addInterpCorrection() 180 0.002864 0.002864 0.002864 0.77% Gravity::fill_multipole_BCs() 6 0.002667 0.002667 0.002667 0.72% amrex::average_down 180 0.002582 0.002582 0.002582 0.70% Castro::estTimeStep() 10 0.002574 0.002574 0.002574 0.70% Amr::restart() 1 0.002441 0.002441 0.002441 0.66% BndryData::define() 6 0.00204 0.00204 0.00204 0.55% FabArray::LinComb() 242 0.001954 0.001954 0.001954 0.53% Castro::reset_internal_energy(MultiFab) 30 0.001875 0.001875 0.001875 0.51% amrex::Add() 72 0.001842 0.001842 0.001842 0.50% Castro::construct_new_gravity_source() 5 0.001764 0.001764 0.001764 0.48% Castro::do_advance_ctu() 5 0.001493 0.001493 0.001493 0.40% Amr::writePlotFile() 1 0.001374 0.001374 0.001374 0.37% Castro::construct_old_gravity_source() 5 0.00135 0.00135 0.00135 0.36% MLCGSolver::bicgstab 36 0.0009847 0.0009847 0.0009847 0.27% Castro::reset_internal_energy(Fab) 240 0.0009181 0.0009181 0.0009181 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008953 0.0008953 0.0008953 0.24% Castro::post_timestep() 5 0.0008513 0.0008513 0.0008513 0.23% Castro::finalize_advance() 5 0.0008376 0.0008376 0.0008376 0.23% MLCellLinOp::setLevelBC() 6 0.0007582 0.0007582 0.0007582 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007286 0.0007286 0.0007286 0.20% FabArray::mult() 22 0.000656 0.000656 0.000656 0.18% FabArray::setDomainBndry() 20 0.0006462 0.0006462 0.0006462 0.17% MLCellLinOp::prepareForSolve() 6 0.0006159 0.0006159 0.0006159 0.17% MultiFab::contains_nan() 10 0.0005881 0.0005881 0.0005881 0.16% Castro::enforce_speed_limit() 30 0.0005803 0.0005803 0.0005803 0.16% MLCellLinOp::smooth() 720 0.0005471 0.0005471 0.0005471 0.15% MLCellLinOp::compGrad() 6 0.0004888 0.0004888 0.0004888 0.13% MLMG::prepareForSolve() 6 0.0004638 0.0004638 0.0004638 0.13% Amr::InitAmr() 1 0.00041 0.00041 0.00041 0.11% FabArrayBase::CPC::define() 244 0.0004022 0.0004022 0.0004022 0.11% Gravity::get_old_grav_vector() 5 0.0003717 0.0003717 0.0003717 0.10% FabArrayBase::getCPC() 632 0.0003486 0.0003486 0.0003486 0.09% FabArray::FillBoundary() 1766 0.0003472 0.0003472 0.0003472 0.09% Gravity::get_new_grav_vector() 5 0.0002937 0.0002937 0.0002937 0.08% main() 1 0.0002604 0.0002604 0.0002604 0.07% FabArrayBase::getFB() 1766 0.0002566 0.0002566 0.0002566 0.07% Amr::coarseTimeStep() 5 0.0002356 0.0002356 0.0002356 0.06% MLCellLinOp::apply() 500 0.0002074 0.0002074 0.0002074 0.06% MLMG::mgVcycle() 36 0.0001805 0.0001805 0.0001805 0.05% Castro::subcycle_advance_ctu() 5 0.0001604 0.0001604 0.0001604 0.04% MultiFab::max() 6 0.0001351 0.0001351 0.0001351 0.04% MLLinOp::defineGrids() 6 0.0001279 0.0001279 0.0001279 0.03% MLCGSolver::ParallelAllReduce 659 0.000127 0.000127 0.000127 0.03% Amr::timeStep() 5 0.0001207 0.0001207 0.0001207 0.03% MLCellLinOp::defineBC() 6 0.0001139 0.0001139 0.0001139 0.03% Castro::construct_new_source() 25 0.0001108 0.0001108 0.0001108 0.03% Castro::advance() 5 0.0001092 0.0001092 0.0001092 0.03% FabArray::ParallelCopy() 380 0.000109 0.000109 0.000109 0.03% MLCellLinOp::correctionResidual() 216 0.0001053 0.0001053 0.0001053 0.03% FillPatchIterator::Initialize 20 0.0001009 0.0001009 0.0001009 0.03% AmrLevel::restart() 1 7.498e-05 7.498e-05 7.498e-05 0.02% Castro::computeNewDt() 5 7.144e-05 7.144e-05 7.144e-05 0.02% Castro::construct_new_gravity() 5 6.582e-05 6.582e-05 6.582e-05 0.02% Gravity::solve_for_phi() 5 6.285e-05 6.285e-05 6.285e-05 0.02% Gravity::update_max_rhs() 6 5.977e-05 5.977e-05 5.977e-05 0.02% StateData::restartDoit() 4 5.887e-05 5.887e-05 5.887e-05 0.02% FabArrayBase::FB::FB() 26 5.849e-05 5.849e-05 5.849e-05 0.02% MLMG:computeResOfCorrection() 180 5.081e-05 5.081e-05 5.081e-05 0.01% MLMG::mgVcycle_down::0 36 4.66e-05 4.66e-05 4.66e-05 0.01% MLMG::mgVcycle_down::1 36 4.351e-05 4.351e-05 4.351e-05 0.01% Castro::clean_state() 30 4.229e-05 4.229e-05 4.229e-05 0.01% MLMG::actualBottomSolve() 36 4.141e-05 4.141e-05 4.141e-05 0.01% MLMG::mgVcycle_down::4 36 4.004e-05 4.004e-05 4.004e-05 0.01% MLMG::solve() 6 3.984e-05 3.984e-05 3.984e-05 0.01% MLMG::mgVcycle_down::2 36 3.96e-05 3.96e-05 3.96e-05 0.01% Castro::initialize_do_advance() 5 3.936e-05 3.936e-05 3.936e-05 0.01% Castro::expand_state() 5 3.903e-05 3.903e-05 3.903e-05 0.01% MLMG::mgVcycle_down::3 36 3.811e-05 3.811e-05 3.811e-05 0.01% MLMG::mgVcycle_up::4 36 3.156e-05 3.156e-05 3.156e-05 0.01% Castro::initialize_advance() 5 3.097e-05 3.097e-05 3.097e-05 0.01% MLMG::mgVcycle_up::0 36 2.678e-05 2.678e-05 2.678e-05 0.01% Castro::buildMetrics() 1 2.562e-05 2.562e-05 2.562e-05 0.01% Amr::writeSmallPlotFile() 1 2.422e-05 2.422e-05 2.422e-05 0.01% MLMG::mgVcycle_up::3 36 2.413e-05 2.413e-05 2.413e-05 0.01% MLMG::mgVcycle_up::2 36 2.405e-05 2.405e-05 2.405e-05 0.01% MLMG::mgVcycle_up::1 36 2.322e-05 2.322e-05 2.322e-05 0.01% MLCellLinOp::solutionResidual() 42 2.322e-05 2.322e-05 2.322e-05 0.01% MLMG::oneIter() 36 2.312e-05 2.312e-05 2.312e-05 0.01% Castro::post_restart() 1 2.152e-05 2.152e-05 2.152e-05 0.01% Castro::initMFs() 1 2.076e-05 2.076e-05 2.076e-05 0.01% Castro::swap_state_time_levels() 5 2.049e-05 2.049e-05 2.049e-05 0.01% Castro::finalize_do_advance() 5 1.763e-05 1.763e-05 1.763e-05 0.00% MLMG::ResNormInf() 42 1.657e-05 1.657e-05 1.657e-05 0.00% MLMG::mgVcycle_bottom 36 1.502e-05 1.502e-05 1.502e-05 0.00% FillPatchSingleLevel 20 1.436e-05 1.436e-05 1.436e-05 0.00% MLPoisson::define() 6 1.404e-05 1.404e-05 1.404e-05 0.00% makeSFC 30 1.369e-05 1.369e-05 1.369e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.342e-05 1.342e-05 1.342e-05 0.00% MLMG::computeResidual() 36 1.34e-05 1.34e-05 1.34e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.257e-05 1.257e-05 1.257e-05 0.00% Castro::construct_old_source() 25 1.004e-05 1.004e-05 1.004e-05 0.00% Castro::do_new_sources() 5 9.863e-06 9.863e-06 9.863e-06 0.00% DistributionMapping::Distribute() 31 8.628e-06 8.628e-06 8.628e-06 0.00% Castro::do_old_sources() 5 8.516e-06 8.516e-06 8.516e-06 0.00% Amr::initSubcycle() 1 8.318e-06 8.318e-06 8.318e-06 0.00% Castro::apply_source_to_state() 10 7.716e-06 7.716e-06 7.716e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.966e-06 6.966e-06 6.966e-06 0.00% Gravity::actual_multilevel_solve() 1 6.633e-06 6.633e-06 6.633e-06 0.00% Castro::check_for_nan() 10 5.981e-06 5.981e-06 5.981e-06 0.00% Castro::construct_old_gravity() 5 5.474e-06 5.474e-06 5.474e-06 0.00% MLLinOp::define() 6 5.446e-06 5.446e-06 5.446e-06 0.00% MLPoisson::prepareForSolve() 6 4.521e-06 4.521e-06 4.521e-06 0.00% Gravity::swapTimeLevels() 5 4.38e-06 4.38e-06 4.38e-06 0.00% MLMG::computeMLResidual() 6 3.497e-06 3.497e-06 3.497e-06 0.00% Gravity::set_mass_offset() 6 2.971e-06 2.971e-06 2.971e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.904e-06 2.904e-06 2.904e-06 0.00% MLMG::getGradSolution() 6 2.792e-06 2.792e-06 2.792e-06 0.00% Castro::FluxRegCrseInit 5 2.16e-06 2.16e-06 2.16e-06 0.00% MLMG::MLRhsNormInf() 6 2.154e-06 2.154e-06 2.154e-06 0.00% MLMG::MLResNormInf() 6 2.078e-06 2.078e-06 2.078e-06 0.00% Castro::create_source_corrector() 5 2.057e-06 2.057e-06 2.057e-06 0.00% Castro::retry_advance_ctu() 5 1.711e-06 1.711e-06 1.711e-06 0.00% Castro::FluxRegFineAdd() 5 1.554e-06 1.554e-06 1.554e-06 0.00% Amr::init() 1 1.251e-06 1.251e-06 1.251e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.076e-06 1.076e-06 1.076e-06 0.00% AmrLevel::AmrLevel() 1 6.89e-07 6.89e-07 6.89e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3702 0.3702 0.3702 99.99% Amr::coarseTimeStep() 5 0.2901 0.2901 0.2901 78.35% Amr::timeStep() 5 0.2885 0.2885 0.2885 77.93% Castro::advance() 5 0.2824 0.2824 0.2824 76.27% Castro::subcycle_advance_ctu() 5 0.2756 0.2756 0.2756 74.45% Castro::do_advance_ctu() 5 0.2755 0.2755 0.2755 74.40% Castro::construct_new_gravity() 5 0.1297 0.1297 0.1297 35.02% Gravity::solve_phi_with_mlmg() 6 0.1247 0.1247 0.1247 33.68% Gravity::actual_solve_with_mlmg() 6 0.1218 0.1218 0.1218 32.89% Gravity::solve_for_phi() 5 0.1217 0.1217 0.1217 32.87% MLMG::solve() 6 0.1107 0.1107 0.1107 29.90% MLMG::oneIter() 36 0.1038 0.1038 0.1038 28.05% Castro::construct_ctu_hydro_source() 5 0.1034 0.1034 0.1034 27.93% MLMG::mgVcycle() 36 0.1023 0.1023 0.1023 27.64% MLCellLinOp::smooth() 720 0.04998 0.04998 0.04998 13.50% Amr::init() 1 0.04637 0.04637 0.04637 12.52% Amr::restart() 1 0.04637 0.04637 0.04637 12.52% MLCellLinOp::applyBC() 1946 0.04636 0.04636 0.04636 12.52% AmrLevel::restart() 1 0.0404 0.0404 0.0404 10.91% StateData::restartDoit() 4 0.04032 0.04032 0.04032 10.89% VisMF::Read() 3 0.04021 0.04021 0.04021 10.86% Amr::writePlotFile() 1 0.03305 0.03305 0.03305 8.93% MLMG::mgVcycle_bottom 36 0.03195 0.03195 0.03195 8.63% MLMG::actualBottomSolve() 36 0.03193 0.03193 0.03193 8.63% MLCGSolver::bicgstab 36 0.03163 0.03163 0.03163 8.54% VisMF::Write(FabArray) 1 0.03141 0.03141 0.03141 8.49% Castro::clean_state() 30 0.0261 0.0261 0.0261 7.05% FillPatchIterator::Initialize 20 0.0204 0.0204 0.0204 5.51% FillPatchSingleLevel 20 0.01965 0.01965 0.01965 5.31% StateDataPhysBCFunct::() 20 0.01765 0.01765 0.01765 4.77% MLCellLinOp::apply() 500 0.01564 0.01564 0.01564 4.22% MLMG::mgVcycle_down::0 36 0.01433 0.01433 0.01433 3.87% MLPoisson::Fsmooth() 1440 0.0142 0.0142 0.0142 3.84% FabArray::FillBoundary() 1766 0.01349 0.01349 0.01349 3.64% FillBoundary_nowait() 1766 0.01314 0.01314 0.01314 3.55% StateData::FillBoundary(geom) 160 0.01171 0.01171 0.01171 3.16% MLMG::mgVcycle_up::0 36 0.01073 0.01073 0.01073 2.90% Castro::computeTemp() 30 0.01026 0.01026 0.01026 2.77% Castro::normalize_species() 30 0.01011 0.01011 0.01011 2.73% Castro::initialize_do_advance() 5 0.01008 0.01008 0.01008 2.72% MLCellLinOp::correctionResidual() 216 0.009498 0.009498 0.009498 2.57% MLPoisson::define() 6 0.008855 0.008855 0.008855 2.39% amrex::Dot() 484 0.008837 0.008837 0.008837 2.39% MLMG:computeResOfCorrection() 180 0.008344 0.008344 0.008344 2.25% Gravity::get_new_grav_vector() 5 0.007817 0.007817 0.007817 2.11% Castro::do_new_sources() 5 0.007653 0.007653 0.007653 2.07% Castro::construct_old_gravity() 5 0.007428 0.007428 0.007428 2.01% Gravity::get_old_grav_vector() 5 0.007422 0.007422 0.007422 2.00% amrex::Copy() 463 0.007058 0.007058 0.007058 1.91% MLMG::mgVcycle_down::1 36 0.006862 0.006862 0.006862 1.85% Castro::do_old_sources() 5 0.006555 0.006555 0.006555 1.77% FabArray::ParallelCopy() 380 0.006489 0.006489 0.006489 1.75% MLMG::mgVcycle_down::2 36 0.006382 0.006382 0.006382 1.72% FabArray::ParallelCopy_nowait() 380 0.00638 0.00638 0.00638 1.72% FabArray::setVal() 537 0.00627 0.00627 0.00627 1.69% FabArray::norminf() 326 0.006241 0.006241 0.006241 1.69% MLMG::mgVcycle_down::3 36 0.006175 0.006175 0.006175 1.67% MLMG::mgVcycle_down::4 36 0.006161 0.006161 0.006161 1.66% Castro::post_timestep() 5 0.00602 0.00602 0.00602 1.63% MLCellLinOp::defineAuxData() 6 0.00598 0.00598 0.00598 1.62% Castro::expand_state() 5 0.005846 0.005846 0.005846 1.58% Castro::initialize_advance() 5 0.005508 0.005508 0.005508 1.49% MLCGSolver::ParallelAllReduce 659 0.005337 0.005337 0.005337 1.44% MLMG::addInterpCorrection() 180 0.005131 0.005131 0.005131 1.39% Castro::enforce_min_density() 30 0.00511 0.00511 0.00511 1.38% MLMG::mgVcycle_up::4 36 0.005029 0.005029 0.005029 1.36% MLMG::mgVcycle_up::1 36 0.004946 0.004946 0.004946 1.34% MLMG::mgVcycle_up::2 36 0.004843 0.004843 0.004843 1.31% amrex::average_down 180 0.004817 0.004817 0.004817 1.30% MLMG::mgVcycle_up::3 36 0.004731 0.004731 0.004731 1.28% MLPoisson::Fapply() 500 0.004566 0.004566 0.004566 1.23% FabArray::Saxpy() 355 0.003685 0.003685 0.003685 1.00% FabArray::Xpay() 361 0.00354 0.00354 0.00354 0.96% Castro::post_restart() 1 0.003352 0.003352 0.003352 0.91% Gravity::multilevel_solve_for_new_phi() 1 0.003239 0.003239 0.003239 0.87% Gravity::actual_multilevel_solve() 1 0.003225 0.003225 0.003225 0.87% MLCellLinOp::solutionResidual() 42 0.003213 0.003213 0.003213 0.87% Castro::reset_internal_energy(MultiFab) 30 0.002793 0.002793 0.002793 0.75% Gravity::fill_multipole_BCs() 6 0.002792 0.002792 0.002792 0.75% MLCellLinOp::defineBC() 6 0.002699 0.002699 0.002699 0.73% MLMG::computeResidual() 36 0.002668 0.002668 0.002668 0.72% BndryData::define() 6 0.002585 0.002585 0.002585 0.70% Castro::estTimeStep() 10 0.002574 0.002574 0.002574 0.70% MLMG::prepareForSolve() 6 0.002516 0.002516 0.002516 0.68% FabArray::LinComb() 242 0.001954 0.001954 0.001954 0.53% Castro::construct_new_source() 25 0.001875 0.001875 0.001875 0.51% amrex::Add() 72 0.001842 0.001842 0.001842 0.50% Castro::construct_new_gravity_source() 5 0.001764 0.001764 0.001764 0.48% Castro::construct_old_source() 25 0.00136 0.00136 0.00136 0.37% Castro::construct_old_gravity_source() 5 0.00135 0.00135 0.00135 0.36% Castro::computeNewDt() 5 0.001346 0.001346 0.001346 0.36% Castro::finalize_advance() 5 0.001121 0.001121 0.001121 0.30% MLMG::ResNormInf() 42 0.0009341 0.0009341 0.0009341 0.25% Castro::apply_source_to_state() 10 0.0009205 0.0009205 0.0009205 0.25% Castro::reset_internal_energy(Fab) 240 0.0009181 0.0009181 0.0009181 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008953 0.0008953 0.0008953 0.24% MLCellLinOp::setLevelBC() 6 0.0007582 0.0007582 0.0007582 0.20% FabArrayBase::getCPC() 632 0.0007508 0.0007508 0.0007508 0.20% MLMG::getGradSolution() 6 0.0007504 0.0007504 0.0007504 0.20% MLCellLinOp::compGrad() 6 0.0007476 0.0007476 0.0007476 0.20% FabArray::mult() 22 0.000656 0.000656 0.000656 0.18% FabArray::setDomainBndry() 20 0.0006462 0.0006462 0.0006462 0.17% MLPoisson::prepareForSolve() 6 0.0006204 0.0006204 0.0006204 0.17% MLCellLinOp::prepareForSolve() 6 0.0006159 0.0006159 0.0006159 0.17% Castro::check_for_nan() 10 0.000594 0.000594 0.000594 0.16% MultiFab::contains_nan() 10 0.0005881 0.0005881 0.0005881 0.16% Castro::enforce_speed_limit() 30 0.0005803 0.0005803 0.0005803 0.16% MLMG::computeMLResidual() 6 0.0005627 0.0005627 0.0005627 0.15% Gravity::update_max_rhs() 6 0.000461 0.000461 0.000461 0.12% Amr::InitAmr() 1 0.0004183 0.0004183 0.0004183 0.11% FabArrayBase::CPC::define() 244 0.0004022 0.0004022 0.0004022 0.11% FabArrayBase::getFB() 1766 0.000315 0.000315 0.000315 0.09% Gravity::swapTimeLevels() 5 0.0002237 0.0002237 0.0002237 0.06% MLLinOp::define() 6 0.0001624 0.0001624 0.0001624 0.04% MLLinOp::defineGrids() 6 0.000157 0.000157 0.000157 0.04% MLMG::MLResNormInf() 6 0.0001503 0.0001503 0.0001503 0.04% Castro::buildMetrics() 1 0.0001488 0.0001488 0.0001488 0.04% MultiFab::max() 6 0.0001351 0.0001351 0.0001351 0.04% MLMG::MLRhsNormInf() 6 0.0001175 0.0001175 0.0001175 0.03% FabArrayBase::FB::FB() 26 5.849e-05 5.849e-05 5.849e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.799e-05 2.799e-05 2.799e-05 0.01% Amr::writeSmallPlotFile() 1 2.422e-05 2.422e-05 2.422e-05 0.01% makeSFC 30 2.102e-05 2.102e-05 2.102e-05 0.01% Castro::initMFs() 1 2.076e-05 2.076e-05 2.076e-05 0.01% Castro::swap_state_time_levels() 5 2.049e-05 2.049e-05 2.049e-05 0.01% Castro::finalize_do_advance() 5 1.763e-05 1.763e-05 1.763e-05 0.00% DistributionMapping::Distribute() 31 8.628e-06 8.628e-06 8.628e-06 0.00% Amr::initSubcycle() 1 8.318e-06 8.318e-06 8.318e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.201e-06 4.201e-06 4.201e-06 0.00% Gravity::set_mass_offset() 6 2.971e-06 2.971e-06 2.971e-06 0.00% Castro::FluxRegCrseInit 5 2.16e-06 2.16e-06 2.16e-06 0.00% Castro::create_source_corrector() 5 2.057e-06 2.057e-06 2.057e-06 0.00% Castro::retry_advance_ctu() 5 1.711e-06 1.711e-06 1.711e-06 0.00% Castro::FluxRegFineAdd() 5 1.554e-06 1.554e-06 1.554e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.076e-06 1.076e-06 1.076e-06 0.00% AmrLevel::AmrLevel() 1 6.89e-07 6.89e-07 6.89e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-24-gea7e8a507e77) finalized