Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.07-4-gcbdc6580ee3d) initialized Starting run at 08:29:12 UTC on 2022-07-06. Successfully read inputs file ... Castro git describe: 22.06-15-gd68821af9 AMReX git describe: 22.07-4-gcbdc6580e Microphysics git describe: 22.07-6-g94202288 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.041207569 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.023641526 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.046721273 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049430624 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.050802607 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.048001983 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.06080441 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.037910579 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.083402259 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.072655743 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.052885537 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.06096389 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.066205276 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.03821816 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.023380645 seconds Ending run at 08:29:13 UTC on 2022-07-06. Run time = 0.808595928 Run time without initialization = 0.692074386 Average number of zones advanced per microsecond: 3.788 Average number of zones advanced per microsecond per rank: 3.788 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8086 ... 0.8086 ... 0.8086 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2058 0.2058 0.2058 25.45% VisMF::Write(FabArray) 11 0.1575 0.1575 0.1575 19.48% MLCellLinOp::applyBC() 4433 0.08082 0.08082 0.08082 9.99% MLPoisson::Fsmooth() 3280 0.06354 0.06354 0.06354 7.86% StateData::FillBoundary(geom) 328 0.02443 0.02443 0.02443 3.02% MLCGSolver::bicgstab 82 0.02403 0.02403 0.02403 2.97% MultiFab::Dot() 1114 0.02199 0.02199 0.02199 2.72% FillBoundary_nowait() 4023 0.01439 0.01439 0.01439 1.78% MultiFab::LinComb() 1586 0.01429 0.01429 0.01429 1.77% FabArray::setVal() 1144 0.01418 0.01418 0.01418 1.75% Castro::normalize_species() 62 0.01392 0.01392 0.01392 1.72% Castro::computeTemp() 63 0.01373 0.01373 0.01373 1.70% FabArray::ParallelCopy_nowait() 861 0.01311 0.01311 0.01311 1.62% StateDataPhysBCFunct::() 41 0.01213 0.01213 0.01213 1.50% MLCellLinOp::defineAuxData() 11 0.01175 0.01175 0.01175 1.45% MLPoisson::Fapply() 1142 0.01166 0.01166 0.01166 1.44% Castro::enforce_min_density() 62 0.0104 0.0104 0.0104 1.29% Gravity::fill_multipole_BCs() 11 0.008038 0.008038 0.008038 0.99% MLMG::addInterpCorrection() 410 0.007467 0.007467 0.007467 0.92% amrex::average_down 410 0.006884 0.006884 0.006884 0.85% MultiFab::Xpay() 585 0.006621 0.006621 0.006621 0.82% Castro::estTimeStep() 21 0.00571 0.00571 0.00571 0.71% Castro::do_advance_ctu() 10 0.004822 0.004822 0.004822 0.60% Castro::reset_internal_energy(MultiFab) 63 0.00445 0.00445 0.00445 0.55% Amr::checkPoint() 3 0.004384 0.004384 0.004384 0.54% BndryData::define() 11 0.004046 0.004046 0.004046 0.50% Castro::construct_new_gravity_source() 10 0.003201 0.003201 0.003201 0.40% Castro::construct_old_gravity_source() 10 0.002541 0.002541 0.002541 0.31% Amr::writePlotFile() 2 0.002488 0.002488 0.002488 0.31% Gravity::get_new_grav_vector() 11 0.001959 0.001959 0.001959 0.24% MLMG::ResNormInf() 93 0.001939 0.001939 0.001939 0.24% MultiFab::Saxpy() 20 0.001813 0.001813 0.001813 0.22% Gravity::get_old_grav_vector() 10 0.001745 0.001745 0.001745 0.22% Castro::expand_state() 10 0.001733 0.001733 0.001733 0.21% MLMG::oneIter() 82 0.00169 0.00169 0.00169 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001667 0.001667 0.001667 0.21% MLCellLinOp::setLevelBC() 11 0.001603 0.001603 0.001603 0.20% Castro::reset_internal_energy(Fab) 504 0.001497 0.001497 0.001497 0.19% Gravity::actual_solve_with_mlmg() 11 0.001449 0.001449 0.001449 0.18% FabArray::setDomainBndry() 41 0.001338 0.001338 0.001338 0.17% FabArray::mult() 43 0.001322 0.001322 0.001322 0.16% MultiFab::contains_nan() 20 0.001198 0.001198 0.001198 0.15% MLCellLinOp::smooth() 1640 0.001181 0.001181 0.001181 0.15% Castro::enforce_speed_limit() 62 0.001173 0.001173 0.001173 0.15% MLCellLinOp::prepareForSolve() 11 0.001165 0.001165 0.001165 0.14% Castro::initData() 1 0.001159 0.001159 0.001159 0.14% MLMG::prepareForSolve() 11 0.001105 0.001105 0.001105 0.14% MLCellLinOp::compGrad() 11 0.0009313 0.0009313 0.0009313 0.12% FabArrayBase::getCPC() 1323 0.0008478 0.0008478 0.0008478 0.10% FabArray::FillBoundary() 4023 0.0008104 0.0008104 0.0008104 0.10% Castro::subcycle_advance_ctu() 10 0.000787 0.000787 0.000787 0.10% FabArrayBase::CPC::define() 454 0.0007241 0.0007241 0.0007241 0.09% FabArrayBase::getFB() 4023 0.0006517 0.0006517 0.0006517 0.08% MLCellLinOp::apply() 1142 0.0004794 0.0004794 0.0004794 0.06% Gravity::solve_for_phi() 10 0.0004783 0.0004783 0.0004783 0.06% Amr::InitAmr() 1 0.0004444 0.0004444 0.0004444 0.05% CGSolver::sxay() 1586 0.0004195 0.0004195 0.0004195 0.05% Gravity::update_max_rhs() 11 0.0004137 0.0004137 0.0004137 0.05% Amr::coarseTimeStep() 10 0.0004001 0.0004001 0.0004001 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003392 0.0003392 0.0003392 0.04% FillPatchIterator::Initialize 41 0.0003384 0.0003384 0.0003384 0.04% MLCellLinOp::defineBC() 11 0.000312 0.000312 0.000312 0.04% FabArray::ParallelCopy() 861 0.000299 0.000299 0.000299 0.04% Amr::timeStep() 10 0.0002689 0.0002689 0.0002689 0.03% MultiFab::Copy() 11 0.0002661 0.0002661 0.0002661 0.03% main() 1 0.0002657 0.0002657 0.0002657 0.03% MultiFab::max() 11 0.000255 0.000255 0.000255 0.03% MLCellLinOp::correctionResidual() 492 0.000246 0.000246 0.000246 0.03% MLMG::mgVcycle() 82 0.0002166 0.0002166 0.0002166 0.03% Castro::construct_new_gravity() 10 0.0002159 0.0002159 0.0002159 0.03% MLMG::MLRhsNormInf() 11 0.0001986 0.0001986 0.0001986 0.02% MLLinOp::defineGrids() 11 0.0001705 0.0001705 0.0001705 0.02% MLMG:computeResOfCorrection() 410 0.0001364 0.0001364 0.0001364 0.02% StateData::checkPoint() 12 0.0001317 0.0001317 0.0001317 0.02% MLMG::actualBottomSolve() 82 0.000119 0.000119 0.000119 0.01% MLMG::mgVcycle_down::0 82 0.0001065 0.0001065 0.0001065 0.01% FabArrayBase::FB::FB() 56 9.465e-05 9.465e-05 9.465e-05 0.01% Castro::initialize_advance() 10 9.241e-05 9.241e-05 9.241e-05 0.01% MLMG::mgVcycle_down::1 82 9.085e-05 9.085e-05 9.085e-05 0.01% Castro::Castro() 1 8.633e-05 8.633e-05 8.633e-05 0.01% MLMG::mgVcycle_down::2 82 8.608e-05 8.608e-05 8.608e-05 0.01% MLMG::solve() 11 8.43e-05 8.43e-05 8.43e-05 0.01% Castro::advance() 10 8.394e-05 8.394e-05 8.394e-05 0.01% Castro::clean_state() 62 8.376e-05 8.376e-05 8.376e-05 0.01% MLMG::mgVcycle_down::4 82 8.305e-05 8.305e-05 8.305e-05 0.01% Castro::finalize_advance() 10 8.199e-05 8.199e-05 8.199e-05 0.01% MLMG::mgVcycle_down::3 82 8.025e-05 8.025e-05 8.025e-05 0.01% Castro::initialize_do_advance() 10 7.378e-05 7.378e-05 7.378e-05 0.01% AmrLevel::checkPoint() 3 7.296e-05 7.296e-05 7.296e-05 0.01% MLMG::mgVcycle_up::4 82 6.392e-05 6.392e-05 6.392e-05 0.01% MLMG::mgVcycle_up::2 82 5.71e-05 5.71e-05 5.71e-05 0.01% MLCellLinOp::solutionResidual() 93 5.66e-05 5.66e-05 5.66e-05 0.01% MLMG::mgVcycle_up::3 82 5.592e-05 5.592e-05 5.592e-05 0.01% MLMG::mgVcycle_up::0 82 5.518e-05 5.518e-05 5.518e-05 0.01% MLMG::mgVcycle_up::1 82 5.508e-05 5.508e-05 5.508e-05 0.01% StateData::define() 4 4.754e-05 4.754e-05 4.754e-05 0.01% MLMG::mgVcycle_bottom 82 4.468e-05 4.468e-05 4.468e-05 0.01% Castro::construct_new_source() 50 4.31e-05 4.31e-05 4.31e-05 0.01% Castro::swap_state_time_levels() 10 4.2e-05 4.2e-05 4.2e-05 0.01% Castro::finalize_do_advance() 10 4.026e-05 4.026e-05 4.026e-05 0.00% MLMG::computeResidual() 82 3.562e-05 3.562e-05 3.562e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.527e-05 3.527e-05 3.527e-05 0.00% Castro::enforce_consistent_e() 1 3.176e-05 3.176e-05 3.176e-05 0.00% Amr::writeSmallPlotFile() 1 3.147e-05 3.147e-05 3.147e-05 0.00% Gravity::actual_multilevel_solve() 1 3.059e-05 3.059e-05 3.059e-05 0.00% FillPatchSingleLevel 41 3.034e-05 3.034e-05 3.034e-05 0.00% MLLinOp::define() 11 2.758e-05 2.758e-05 2.758e-05 0.00% makeSFC 55 2.749e-05 2.749e-05 2.749e-05 0.00% Castro::initMFs() 1 2.629e-05 2.629e-05 2.629e-05 0.00% MLPoisson::define() 11 2.618e-05 2.618e-05 2.618e-05 0.00% Castro::buildMetrics() 1 2.309e-05 2.309e-05 2.309e-05 0.00% Amr::defBaseLevel() 1 2.191e-05 2.191e-05 2.191e-05 0.00% Amr::FinalizeInit() 1 2.116e-05 2.116e-05 2.116e-05 0.00% Castro::construct_old_source() 50 1.989e-05 1.989e-05 1.989e-05 0.00% Castro::do_new_sources() 10 1.912e-05 1.912e-05 1.912e-05 0.00% Castro::check_for_nan() 20 1.813e-05 1.813e-05 1.813e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.779e-05 1.779e-05 1.779e-05 0.00% Castro::do_old_sources() 10 1.777e-05 1.777e-05 1.777e-05 0.00% DistributionMapping::Distribute() 56 1.563e-05 1.563e-05 1.563e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.518e-05 1.518e-05 1.518e-05 0.00% Castro::apply_source_to_state() 20 1.365e-05 1.365e-05 1.365e-05 0.00% Castro::post_timestep() 10 1.078e-05 1.078e-05 1.078e-05 0.00% Castro::construct_old_gravity() 10 1.075e-05 1.075e-05 1.075e-05 0.00% MLPoisson::prepareForSolve() 11 1.049e-05 1.049e-05 1.049e-05 0.00% Gravity::swapTimeLevels() 10 1.039e-05 1.039e-05 1.039e-05 0.00% Amr::initSubcycle() 1 8.86e-06 8.86e-06 8.86e-06 0.00% MLMG::getGradSolution() 11 7.979e-06 7.979e-06 7.979e-06 0.00% MLMG::computeMLResidual() 11 7.929e-06 7.929e-06 7.929e-06 0.00% MLMG::buildFineMask() 11 6.677e-06 6.677e-06 6.677e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.603e-06 6.603e-06 6.603e-06 0.00% Castro::computeNewDt() 9 6.434e-06 6.434e-06 6.434e-06 0.00% Amr::InitializeInit() 1 6.375e-06 6.375e-06 6.375e-06 0.00% AmrLevel::checkPointPost() 3 5.58e-06 5.58e-06 5.58e-06 0.00% MLMG::MLResNormInf() 11 5.21e-06 5.21e-06 5.21e-06 0.00% Gravity::set_mass_offset() 11 4.787e-06 4.787e-06 4.787e-06 0.00% Castro::retry_advance_ctu() 10 4.726e-06 4.726e-06 4.726e-06 0.00% Castro::create_source_corrector() 10 4.25e-06 4.25e-06 4.25e-06 0.00% Castro::FluxRegCrseInit 10 3.85e-06 3.85e-06 3.85e-06 0.00% Castro::post_init() 1 3.67e-06 3.67e-06 3.67e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.611e-06 3.611e-06 3.611e-06 0.00% Castro::computeInitialDt() 2 3.148e-06 3.148e-06 3.148e-06 0.00% Amr::init() 1 2.416e-06 2.416e-06 2.416e-06 0.00% Castro::FluxRegFineAdd() 10 2.263e-06 2.263e-06 2.263e-06 0.00% AmrLevel::checkPointPre() 3 2.133e-06 2.133e-06 2.133e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.792e-06 1.792e-06 1.792e-06 0.00% Castro::post_regrid() 1 1.196e-06 1.196e-06 1.196e-06 0.00% Amr::initialInit() 1 1.084e-06 1.084e-06 1.084e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8086 0.8086 0.8086 100.00% Amr::coarseTimeStep() 10 0.6685 0.6685 0.6685 82.67% Amr::timeStep() 10 0.5887 0.5887 0.5887 72.81% Castro::advance() 10 0.5815 0.5815 0.5815 71.91% Castro::subcycle_advance_ctu() 10 0.5694 0.5694 0.5694 70.42% Castro::do_advance_ctu() 10 0.5686 0.5686 0.5686 70.32% Gravity::solve_phi_with_mlmg() 11 0.3145 0.3145 0.3145 38.89% Gravity::actual_solve_with_mlmg() 11 0.3062 0.3062 0.3062 37.87% Castro::construct_new_gravity() 10 0.2869 0.2869 0.2869 35.48% MLMG::solve() 11 0.283 0.283 0.283 34.99% Gravity::solve_for_phi() 10 0.2712 0.2712 0.2712 33.54% MLMG::oneIter() 82 0.2682 0.2682 0.2682 33.17% MLMG::mgVcycle() 82 0.2665 0.2665 0.2665 32.96% Castro::construct_ctu_hydro_source() 10 0.2058 0.2058 0.2058 25.45% VisMF::Write(FabArray) 11 0.1575 0.1575 0.1575 19.48% MLCellLinOp::smooth() 1640 0.1368 0.1368 0.1368 16.91% Amr::checkPoint() 3 0.1175 0.1175 0.1175 14.53% Amr::init() 1 0.116 0.116 0.116 14.34% AmrLevel::checkPoint() 3 0.1131 0.1131 0.1131 13.98% StateData::checkPoint() 12 0.113 0.113 0.113 13.97% MLCellLinOp::applyBC() 4433 0.09676 0.09676 0.09676 11.97% MLMG::mgVcycle_bottom 82 0.08168 0.08168 0.08168 10.10% MLMG::actualBottomSolve() 82 0.08164 0.08164 0.08164 10.10% MLCGSolver::bicgstab 82 0.08078 0.08078 0.08078 9.99% MLPoisson::Fsmooth() 3280 0.06354 0.06354 0.06354 7.86% Amr::initialInit() 1 0.05099 0.05099 0.05099 6.31% Amr::writePlotFile() 2 0.04715 0.04715 0.04715 5.83% Amr::FinalizeInit() 1 0.04698 0.04698 0.04698 5.81% Castro::post_init() 1 0.04561 0.04561 0.04561 5.64% Castro::clean_state() 62 0.04443 0.04443 0.04443 5.49% Gravity::multilevel_solve_for_new_phi() 1 0.04375 0.04375 0.04375 5.41% Gravity::actual_multilevel_solve() 1 0.04373 0.04373 0.04373 5.41% FillPatchIterator::Initialize 41 0.04231 0.04231 0.04231 5.23% FillPatchSingleLevel 41 0.04064 0.04064 0.04064 5.03% StateDataPhysBCFunct::() 41 0.03657 0.03657 0.03657 4.52% MLCellLinOp::apply() 1142 0.03637 0.03637 0.03637 4.50% MLMG::mgVcycle_down::0 82 0.03559 0.03559 0.03559 4.40% MLMG::mgVcycle_up::0 82 0.03053 0.03053 0.03053 3.78% StateData::FillBoundary(geom) 328 0.02443 0.02443 0.02443 3.02% MultiFab::Dot() 1114 0.02199 0.02199 0.02199 2.72% MLCellLinOp::correctionResidual() 492 0.02139 0.02139 0.02139 2.64% Castro::computeTemp() 63 0.01968 0.01968 0.01968 2.43% Castro::initialize_do_advance() 10 0.0188 0.0188 0.0188 2.32% MLPoisson::define() 11 0.01873 0.01873 0.01873 2.32% MLMG:computeResOfCorrection() 410 0.01844 0.01844 0.01844 2.28% MLMG::mgVcycle_down::1 82 0.0178 0.0178 0.0178 2.20% MLMG::mgVcycle_down::2 82 0.01728 0.01728 0.01728 2.14% Gravity::get_new_grav_vector() 11 0.01722 0.01722 0.01722 2.13% MLMG::mgVcycle_down::3 82 0.01643 0.01643 0.01643 2.03% FabArray::FillBoundary() 4023 0.01594 0.01594 0.01594 1.97% MLMG::mgVcycle_down::4 82 0.01567 0.01567 0.01567 1.94% FillBoundary_nowait() 4023 0.01513 0.01513 0.01513 1.87% Castro::construct_old_gravity() 10 0.015 0.015 0.015 1.86% Gravity::get_old_grav_vector() 10 0.01499 0.01499 0.01499 1.85% CGSolver::sxay() 1586 0.01471 0.01471 0.01471 1.82% FabArray::ParallelCopy() 861 0.0143 0.0143 0.0143 1.77% MultiFab::LinComb() 1586 0.01429 0.01429 0.01429 1.77% FabArray::setVal() 1144 0.01418 0.01418 0.01418 1.75% FabArray::ParallelCopy_nowait() 861 0.01401 0.01401 0.01401 1.73% Castro::normalize_species() 62 0.01392 0.01392 0.01392 1.72% MLMG::mgVcycle_up::2 82 0.01329 0.01329 0.01329 1.64% MLCellLinOp::defineAuxData() 11 0.01315 0.01315 0.01315 1.63% MLCGSolver::ParallelAllReduce 1514 0.01311 0.01311 0.01311 1.62% MLMG::mgVcycle_up::1 82 0.01301 0.01301 0.01301 1.61% MLMG::mgVcycle_up::3 82 0.01259 0.01259 0.01259 1.56% MLMG::addInterpCorrection() 410 0.01258 0.01258 0.01258 1.56% MLMG::mgVcycle_up::4 82 0.01244 0.01244 0.01244 1.54% amrex::average_down 410 0.01204 0.01204 0.01204 1.49% Castro::do_new_sources() 10 0.0119 0.0119 0.0119 1.47% Castro::initialize_advance() 10 0.01188 0.01188 0.01188 1.47% MLPoisson::Fapply() 1142 0.01166 0.01166 0.01166 1.44% Castro::expand_state() 10 0.01159 0.01159 0.01159 1.43% Castro::enforce_min_density() 62 0.0104 0.0104 0.0104 1.29% Castro::do_old_sources() 10 0.00998 0.00998 0.00998 1.23% Gravity::fill_multipole_BCs() 11 0.008038 0.008038 0.008038 0.99% MLCellLinOp::solutionResidual() 93 0.007281 0.007281 0.007281 0.90% Castro::post_timestep() 10 0.007007 0.007007 0.007007 0.87% MultiFab::Xpay() 585 0.006621 0.006621 0.006621 0.82% MLMG::computeResidual() 82 0.006209 0.006209 0.006209 0.77% Castro::reset_internal_energy(MultiFab) 63 0.005947 0.005947 0.005947 0.74% Castro::estTimeStep() 21 0.00571 0.00571 0.00571 0.71% MLCellLinOp::defineBC() 11 0.005297 0.005297 0.005297 0.66% MLMG::prepareForSolve() 11 0.005201 0.005201 0.005201 0.64% BndryData::define() 11 0.004985 0.004985 0.004985 0.62% Amr::InitializeInit() 1 0.004016 0.004016 0.004016 0.50% Amr::defBaseLevel() 1 0.00401 0.00401 0.00401 0.50% Castro::initData() 1 0.003495 0.003495 0.003495 0.43% Castro::construct_new_source() 50 0.003244 0.003244 0.003244 0.40% Castro::construct_new_gravity_source() 10 0.003201 0.003201 0.003201 0.40% Castro::computeNewDt() 9 0.002619 0.002619 0.002619 0.32% Castro::construct_old_source() 50 0.002561 0.002561 0.002561 0.32% Castro::construct_old_gravity_source() 10 0.002541 0.002541 0.002541 0.31% MLMG::ResNormInf() 93 0.001939 0.001939 0.001939 0.24% Castro::apply_source_to_state() 20 0.001827 0.001827 0.001827 0.23% MultiFab::Saxpy() 20 0.001813 0.001813 0.001813 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001667 0.001667 0.001667 0.21% MLCellLinOp::setLevelBC() 11 0.001603 0.001603 0.001603 0.20% FabArrayBase::getCPC() 1323 0.001572 0.001572 0.001572 0.19% Castro::reset_internal_energy(Fab) 504 0.001497 0.001497 0.001497 0.19% MLMG::getGradSolution() 11 0.001429 0.001429 0.001429 0.18% MLCellLinOp::compGrad() 11 0.001421 0.001421 0.001421 0.18% FabArray::setDomainBndry() 41 0.001338 0.001338 0.001338 0.17% FabArray::mult() 43 0.001322 0.001322 0.001322 0.16% Castro::check_for_nan() 20 0.001216 0.001216 0.001216 0.15% MultiFab::contains_nan() 20 0.001198 0.001198 0.001198 0.15% MLPoisson::prepareForSolve() 11 0.001176 0.001176 0.001176 0.15% Castro::enforce_speed_limit() 62 0.001173 0.001173 0.001173 0.15% MLCellLinOp::prepareForSolve() 11 0.001165 0.001165 0.001165 0.14% MLMG::computeMLResidual() 11 0.001115 0.001115 0.001115 0.14% Castro::post_regrid() 1 0.001111 0.001111 0.001111 0.14% Gravity::update_max_rhs() 11 0.000814 0.000814 0.000814 0.10% FabArrayBase::getFB() 4023 0.0007464 0.0007464 0.0007464 0.09% FabArrayBase::CPC::define() 454 0.0007241 0.0007241 0.0007241 0.09% Castro::computeInitialDt() 2 0.0007166 0.0007166 0.0007166 0.09% Amr::InitAmr() 1 0.0004533 0.0004533 0.0004533 0.06% Gravity::swapTimeLevels() 10 0.0004458 0.0004458 0.0004458 0.06% Castro::Castro() 1 0.0004337 0.0004337 0.0004337 0.05% MultiFab::Copy() 11 0.0002661 0.0002661 0.0002661 0.03% MLMG::MLResNormInf() 11 0.0002599 0.0002599 0.0002599 0.03% MLLinOp::define() 11 0.0002571 0.0002571 0.0002571 0.03% MultiFab::max() 11 0.000255 0.000255 0.000255 0.03% MLLinOp::defineGrids() 11 0.0002295 0.0002295 0.0002295 0.03% MLMG::MLRhsNormInf() 11 0.0001986 0.0001986 0.0001986 0.02% Castro::buildMetrics() 1 0.0001537 0.0001537 0.0001537 0.02% FabArrayBase::FB::FB() 56 9.465e-05 9.465e-05 9.465e-05 0.01% Castro::finalize_advance() 10 8.81e-05 8.81e-05 8.81e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.723e-05 5.723e-05 5.723e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.414e-05 5.414e-05 5.414e-05 0.01% StateData::define() 4 4.754e-05 4.754e-05 4.754e-05 0.01% makeSFC 55 4.205e-05 4.205e-05 4.205e-05 0.01% Castro::swap_state_time_levels() 10 4.2e-05 4.2e-05 4.2e-05 0.01% Castro::finalize_do_advance() 10 4.026e-05 4.026e-05 4.026e-05 0.00% Castro::enforce_consistent_e() 1 3.176e-05 3.176e-05 3.176e-05 0.00% Amr::writeSmallPlotFile() 1 3.147e-05 3.147e-05 3.147e-05 0.00% Castro::initMFs() 1 2.629e-05 2.629e-05 2.629e-05 0.00% DistributionMapping::Distribute() 56 1.563e-05 1.563e-05 1.563e-05 0.00% Amr::initSubcycle() 1 8.86e-06 8.86e-06 8.86e-06 0.00% MLMG::buildFineMask() 11 6.677e-06 6.677e-06 6.677e-06 0.00% AmrLevel::checkPointPost() 3 5.58e-06 5.58e-06 5.58e-06 0.00% Gravity::set_mass_offset() 11 4.787e-06 4.787e-06 4.787e-06 0.00% Castro::retry_advance_ctu() 10 4.726e-06 4.726e-06 4.726e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.686e-06 4.686e-06 4.686e-06 0.00% Castro::create_source_corrector() 10 4.25e-06 4.25e-06 4.25e-06 0.00% Castro::FluxRegCrseInit 10 3.85e-06 3.85e-06 3.85e-06 0.00% Castro::FluxRegFineAdd() 10 2.263e-06 2.263e-06 2.263e-06 0.00% AmrLevel::checkPointPre() 3 2.133e-06 2.133e-06 2.133e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.792e-06 1.792e-06 1.792e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.07-4-gcbdc6580ee3d) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.07-4-gcbdc6580ee3d) initialized Starting run at 08:29:13 UTC on 2022-07-06. Successfully read inputs file ... Castro git describe: 22.06-15-gd68821af9 AMReX git describe: 22.07-4-gcbdc6580e Microphysics git describe: 22.07-6-g94202288 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.410231625 Restart time = 0.045913832 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.050323499 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049790787 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.060539582 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.064524963 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.067420772 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.025210434 seconds Ending run at 08:29:14 UTC on 2022-07-06. Run time = 0.364661377 Run time without initialization = 0.318205697 Average number of zones advanced per microsecond: 4.119 Average number of zones advanced per microsecond per rank: 4.119 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3647 ... 0.3647 ... 0.3647 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0949 0.0949 0.0949 26.01% VisMF::Read() 3 0.03864 0.03864 0.03864 10.59% MLCellLinOp::applyBC() 1946 0.0347 0.0347 0.0347 9.52% MLPoisson::Fsmooth() 1440 0.02693 0.02693 0.02693 7.38% VisMF::Write(FabArray) 1 0.02368 0.02368 0.02368 6.49% StateData::FillBoundary(geom) 160 0.01117 0.01117 0.01117 3.06% MLCGSolver::bicgstab 36 0.01017 0.01017 0.01017 2.79% MultiFab::Dot() 484 0.009327 0.009327 0.009327 2.56% Castro::normalize_species() 30 0.008633 0.008633 0.008633 2.37% Castro::computeTemp() 30 0.007117 0.007117 0.007117 1.95% MLCellLinOp::defineAuxData() 6 0.006774 0.006774 0.006774 1.86% FabArray::setVal() 537 0.006626 0.006626 0.006626 1.82% FillBoundary_nowait() 1766 0.006257 0.006257 0.006257 1.72% MultiFab::LinComb() 690 0.006039 0.006039 0.006039 1.66% Castro::enforce_min_density() 30 0.006001 0.006001 0.006001 1.65% FabArray::ParallelCopy_nowait() 380 0.0059 0.0059 0.0059 1.62% StateDataPhysBCFunct::() 20 0.005248 0.005248 0.005248 1.44% MLPoisson::Fapply() 500 0.00499 0.00499 0.00499 1.37% Gravity::fill_multipole_BCs() 6 0.004201 0.004201 0.004201 1.15% MLMG::addInterpCorrection() 180 0.003205 0.003205 0.003205 0.88% Amr::restart() 1 0.003139 0.003139 0.003139 0.86% amrex::average_down 180 0.002919 0.002919 0.002919 0.80% MultiFab::Xpay() 258 0.002812 0.002812 0.002812 0.77% Castro::estTimeStep() 10 0.002201 0.002201 0.002201 0.60% Castro::do_advance_ctu() 5 0.002153 0.002153 0.002153 0.59% BndryData::define() 6 0.002042 0.002042 0.002042 0.56% Castro::construct_new_gravity_source() 5 0.00177 0.00177 0.00177 0.49% Amr::writePlotFile() 1 0.001612 0.001612 0.001612 0.44% Castro::reset_internal_energy(MultiFab) 30 0.001601 0.001601 0.001601 0.44% Castro::construct_old_gravity_source() 5 0.00153 0.00153 0.00153 0.42% Castro::enforce_speed_limit() 30 0.00127 0.00127 0.00127 0.35% Castro::reset_internal_energy(Fab) 240 0.001069 0.001069 0.001069 0.29% Gravity::get_old_grav_vector() 5 0.0009943 0.0009943 0.0009943 0.27% Castro::subcycle_advance_ctu() 5 0.0009908 0.0009908 0.0009908 0.27% Gravity::get_new_grav_vector() 5 0.0009486 0.0009486 0.0009486 0.26% MultiFab::Saxpy() 10 0.0009189 0.0009189 0.0009189 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009045 0.0009045 0.0009045 0.25% Castro::construct_new_source() 25 0.0009013 0.0009013 0.0009013 0.25% Castro::expand_state() 5 0.0008674 0.0008674 0.0008674 0.24% MLMG::ResNormInf() 42 0.0008568 0.0008568 0.0008568 0.23% MLCellLinOp::setLevelBC() 6 0.0008416 0.0008416 0.0008416 0.23% Gravity::actual_solve_with_mlmg() 6 0.000756 0.000756 0.000756 0.21% MLMG::oneIter() 36 0.0007464 0.0007464 0.0007464 0.20% FabArray::mult() 22 0.0006456 0.0006456 0.0006456 0.18% MLCellLinOp::prepareForSolve() 6 0.0006335 0.0006335 0.0006335 0.17% FabArray::setDomainBndry() 20 0.0006262 0.0006262 0.0006262 0.17% MultiFab::contains_nan() 10 0.0005888 0.0005888 0.0005888 0.16% MLMG::prepareForSolve() 6 0.000565 0.000565 0.000565 0.15% MLCellLinOp::smooth() 720 0.0004986 0.0004986 0.0004986 0.14% MLCellLinOp::compGrad() 6 0.0004843 0.0004843 0.0004843 0.13% FabArrayBase::CPC::define() 244 0.0004244 0.0004244 0.0004244 0.12% FabArrayBase::getCPC() 632 0.0003956 0.0003956 0.0003956 0.11% Amr::InitAmr() 1 0.0003679 0.0003679 0.0003679 0.10% FabArray::FillBoundary() 1766 0.0003575 0.0003575 0.0003575 0.10% FabArrayBase::getFB() 1766 0.0002584 0.0002584 0.0002584 0.07% main() 1 0.0002407 0.0002407 0.0002407 0.07% Gravity::update_max_rhs() 6 0.0002269 0.0002269 0.0002269 0.06% Gravity::solve_for_phi() 5 0.0002254 0.0002254 0.0002254 0.06% MLCellLinOp::apply() 500 0.0002073 0.0002073 0.0002073 0.06% CGSolver::sxay() 690 0.0001753 0.0001753 0.0001753 0.05% Amr::coarseTimeStep() 5 0.0001649 0.0001649 0.0001649 0.05% Castro::create_source_corrector() 5 0.000159 0.000159 0.000159 0.04% Castro::construct_new_gravity() 5 0.0001553 0.0001553 0.0001553 0.04% Castro::advance() 5 0.00015 0.00015 0.00015 0.04% MLCellLinOp::defineBC() 6 0.0001499 0.0001499 0.0001499 0.04% MultiFab::Copy() 6 0.0001383 0.0001383 0.0001383 0.04% FillPatchIterator::Initialize 20 0.0001371 0.0001371 0.0001371 0.04% MultiFab::max() 6 0.0001337 0.0001337 0.0001337 0.04% MLCGSolver::ParallelAllReduce 659 0.0001308 0.0001308 0.0001308 0.04% FabArray::ParallelCopy() 380 0.0001308 0.0001308 0.0001308 0.04% Amr::timeStep() 5 0.0001235 0.0001235 0.0001235 0.03% MLMG::MLRhsNormInf() 6 0.0001041 0.0001041 0.0001041 0.03% MLCellLinOp::correctionResidual() 216 9.874e-05 9.874e-05 9.874e-05 0.03% MLMG::mgVcycle() 36 9.202e-05 9.202e-05 9.202e-05 0.03% Castro::construct_old_source() 25 8.957e-05 8.957e-05 8.957e-05 0.02% MLLinOp::defineGrids() 6 8.119e-05 8.119e-05 8.119e-05 0.02% AmrLevel::restart() 1 7.66e-05 7.66e-05 7.66e-05 0.02% Castro::computeNewDt() 5 7.442e-05 7.442e-05 7.442e-05 0.02% StateData::restartDoit() 4 6.875e-05 6.875e-05 6.875e-05 0.02% Castro::finalize_advance() 5 6.346e-05 6.346e-05 6.346e-05 0.02% MLMG:computeResOfCorrection() 180 5.987e-05 5.987e-05 5.987e-05 0.02% FabArrayBase::FB::FB() 26 5.679e-05 5.679e-05 5.679e-05 0.02% Castro::clean_state() 30 4.81e-05 4.81e-05 4.81e-05 0.01% MLMG::actualBottomSolve() 36 4.741e-05 4.741e-05 4.741e-05 0.01% Castro::initialize_do_advance() 5 4.56e-05 4.56e-05 4.56e-05 0.01% Castro::initialize_advance() 5 4.03e-05 4.03e-05 4.03e-05 0.01% Castro::buildMetrics() 1 3.981e-05 3.981e-05 3.981e-05 0.01% MLMG::mgVcycle_down::0 36 3.882e-05 3.882e-05 3.882e-05 0.01% MLMG::solve() 6 3.874e-05 3.874e-05 3.874e-05 0.01% MLMG::mgVcycle_down::1 36 3.669e-05 3.669e-05 3.669e-05 0.01% MLMG::mgVcycle_down::2 36 3.643e-05 3.643e-05 3.643e-05 0.01% Gravity::actual_multilevel_solve() 1 3.566e-05 3.566e-05 3.566e-05 0.01% MLMG::mgVcycle_down::4 36 3.392e-05 3.392e-05 3.392e-05 0.01% MLMG::mgVcycle_down::3 36 3.341e-05 3.341e-05 3.341e-05 0.01% Castro::post_restart() 1 3.024e-05 3.024e-05 3.024e-05 0.01% MLMG::mgVcycle_up::4 36 2.875e-05 2.875e-05 2.875e-05 0.01% Castro::initMFs() 1 2.859e-05 2.859e-05 2.859e-05 0.01% Amr::writeSmallPlotFile() 1 2.748e-05 2.748e-05 2.748e-05 0.01% Castro::swap_state_time_levels() 5 2.728e-05 2.728e-05 2.728e-05 0.01% MLCellLinOp::solutionResidual() 42 2.471e-05 2.471e-05 2.471e-05 0.01% MLMG::mgVcycle_up::3 36 2.362e-05 2.362e-05 2.362e-05 0.01% MLMG::mgVcycle_up::2 36 2.335e-05 2.335e-05 2.335e-05 0.01% MLMG::mgVcycle_up::0 36 2.24e-05 2.24e-05 2.24e-05 0.01% MLMG::mgVcycle_up::1 36 2.123e-05 2.123e-05 2.123e-05 0.01% Castro::post_timestep() 5 2.122e-05 2.122e-05 2.122e-05 0.01% MLLinOp::define() 6 2.108e-05 2.108e-05 2.108e-05 0.01% Castro::finalize_do_advance() 5 1.887e-05 1.887e-05 1.887e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.746e-05 1.746e-05 1.746e-05 0.00% makeSFC 30 1.662e-05 1.662e-05 1.662e-05 0.00% MLMG::mgVcycle_bottom 36 1.61e-05 1.61e-05 1.61e-05 0.00% MLMG::computeResidual() 36 1.531e-05 1.531e-05 1.531e-05 0.00% MLPoisson::define() 6 1.463e-05 1.463e-05 1.463e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.443e-05 1.443e-05 1.443e-05 0.00% FillPatchSingleLevel 20 1.268e-05 1.268e-05 1.268e-05 0.00% Castro::do_new_sources() 5 1.028e-05 1.028e-05 1.028e-05 0.00% DistributionMapping::Distribute() 31 9.984e-06 9.984e-06 9.984e-06 0.00% Castro::do_old_sources() 5 9.254e-06 9.254e-06 9.254e-06 0.00% Amr::initSubcycle() 1 8.917e-06 8.917e-06 8.917e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.875e-06 7.875e-06 7.875e-06 0.00% Castro::check_for_nan() 10 7.28e-06 7.28e-06 7.28e-06 0.00% Castro::apply_source_to_state() 10 6.238e-06 6.238e-06 6.238e-06 0.00% Castro::construct_old_gravity() 5 5.717e-06 5.717e-06 5.717e-06 0.00% Gravity::swapTimeLevels() 5 5.485e-06 5.485e-06 5.485e-06 0.00% MLPoisson::prepareForSolve() 6 4.632e-06 4.632e-06 4.632e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.307e-06 3.307e-06 3.307e-06 0.00% MLMG::buildFineMask() 6 3.193e-06 3.193e-06 3.193e-06 0.00% MLMG::computeMLResidual() 6 3.106e-06 3.106e-06 3.106e-06 0.00% MLMG::getGradSolution() 6 3.078e-06 3.078e-06 3.078e-06 0.00% MLMG::MLResNormInf() 6 2.355e-06 2.355e-06 2.355e-06 0.00% Gravity::set_mass_offset() 6 2.107e-06 2.107e-06 2.107e-06 0.00% Castro::retry_advance_ctu() 5 1.947e-06 1.947e-06 1.947e-06 0.00% Castro::FluxRegCrseInit 5 1.692e-06 1.692e-06 1.692e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.203e-06 1.203e-06 1.203e-06 0.00% Castro::FluxRegFineAdd() 5 1.179e-06 1.179e-06 1.179e-06 0.00% Amr::init() 1 1.002e-06 1.002e-06 1.002e-06 0.00% AmrLevel::AmrLevel() 1 9.4e-07 9.4e-07 9.4e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3647 0.3647 0.3647 100.00% Amr::coarseTimeStep() 5 0.2928 0.2928 0.2928 80.28% Amr::timeStep() 5 0.2914 0.2914 0.2914 79.91% Castro::advance() 5 0.2878 0.2878 0.2878 78.92% Castro::subcycle_advance_ctu() 5 0.2812 0.2812 0.2812 77.10% Castro::do_advance_ctu() 5 0.2802 0.2802 0.2802 76.83% Castro::construct_new_gravity() 5 0.1425 0.1425 0.1425 39.08% Gravity::solve_phi_with_mlmg() 6 0.1384 0.1384 0.1384 37.94% Gravity::solve_for_phi() 5 0.1349 0.1349 0.1349 37.00% Gravity::actual_solve_with_mlmg() 6 0.134 0.134 0.134 36.75% MLMG::solve() 6 0.1212 0.1212 0.1212 33.25% MLMG::oneIter() 36 0.1142 0.1142 0.1142 31.32% MLMG::mgVcycle() 36 0.1135 0.1135 0.1135 31.12% Castro::construct_ctu_hydro_source() 5 0.09485 0.09485 0.09485 26.01% MLCellLinOp::smooth() 720 0.05827 0.05827 0.05827 15.98% Amr::init() 1 0.04598 0.04598 0.04598 12.61% Amr::restart() 1 0.04598 0.04598 0.04598 12.61% MLCellLinOp::applyBC() 1946 0.04163 0.04163 0.04163 11.42% AmrLevel::restart() 1 0.03884 0.03884 0.03884 10.65% StateData::restartDoit() 4 0.03876 0.03876 0.03876 10.63% VisMF::Read() 3 0.03864 0.03864 0.03864 10.59% MLMG::mgVcycle_bottom 36 0.03463 0.03463 0.03463 9.50% MLMG::actualBottomSolve() 36 0.03462 0.03462 0.03462 9.49% MLCGSolver::bicgstab 36 0.03426 0.03426 0.03426 9.39% MLPoisson::Fsmooth() 1440 0.02693 0.02693 0.02693 7.38% Castro::clean_state() 30 0.02574 0.02574 0.02574 7.06% Amr::writePlotFile() 1 0.0253 0.0253 0.0253 6.94% VisMF::Write(FabArray) 1 0.02368 0.02368 0.02368 6.49% FillPatchIterator::Initialize 20 0.01919 0.01919 0.01919 5.26% FillPatchSingleLevel 20 0.01842 0.01842 0.01842 5.05% StateDataPhysBCFunct::() 20 0.01642 0.01642 0.01642 4.50% MLCellLinOp::apply() 500 0.01572 0.01572 0.01572 4.31% MLMG::mgVcycle_down::0 36 0.01538 0.01538 0.01538 4.22% MLMG::mgVcycle_up::0 36 0.01307 0.01307 0.01307 3.58% StateData::FillBoundary(geom) 160 0.01117 0.01117 0.01117 3.06% Castro::initialize_do_advance() 5 0.01099 0.01099 0.01099 3.01% MLPoisson::define() 6 0.01044 0.01044 0.01044 2.86% Castro::computeTemp() 30 0.009787 0.009787 0.009787 2.68% MultiFab::Dot() 484 0.009327 0.009327 0.009327 2.56% MLCellLinOp::correctionResidual() 216 0.009126 0.009126 0.009126 2.50% Castro::do_new_sources() 5 0.008674 0.008674 0.008674 2.38% Castro::normalize_species() 30 0.008633 0.008633 0.008633 2.37% MLMG:computeResOfCorrection() 180 0.007881 0.007881 0.007881 2.16% MLCellLinOp::defineAuxData() 6 0.007578 0.007578 0.007578 2.08% MLMG::mgVcycle_down::1 36 0.007545 0.007545 0.007545 2.07% Gravity::get_new_grav_vector() 5 0.007412 0.007412 0.007412 2.03% MLMG::mgVcycle_down::2 36 0.007329 0.007329 0.007329 2.01% Castro::construct_old_gravity() 5 0.007238 0.007238 0.007238 1.98% Gravity::get_old_grav_vector() 5 0.007232 0.007232 0.007232 1.98% MLMG::mgVcycle_down::3 36 0.006947 0.006947 0.006947 1.90% FabArray::FillBoundary() 1766 0.00693 0.00693 0.00693 1.90% FabArray::setVal() 537 0.006626 0.006626 0.006626 1.82% MLMG::mgVcycle_down::4 36 0.006621 0.006621 0.006621 1.82% FillBoundary_nowait() 1766 0.006572 0.006572 0.006572 1.80% FabArray::ParallelCopy() 380 0.006427 0.006427 0.006427 1.76% Castro::initialize_advance() 5 0.006415 0.006415 0.006415 1.76% FabArray::ParallelCopy_nowait() 380 0.006296 0.006296 0.006296 1.73% CGSolver::sxay() 690 0.006214 0.006214 0.006214 1.70% MultiFab::LinComb() 690 0.006039 0.006039 0.006039 1.66% Castro::enforce_min_density() 30 0.006001 0.006001 0.006001 1.65% MLMG::mgVcycle_up::2 36 0.005641 0.005641 0.005641 1.55% MLCGSolver::ParallelAllReduce 659 0.005614 0.005614 0.005614 1.54% Castro::do_old_sources() 5 0.005546 0.005546 0.005546 1.52% MLMG::mgVcycle_up::1 36 0.005524 0.005524 0.005524 1.51% MLMG::addInterpCorrection() 180 0.005419 0.005419 0.005419 1.49% MLMG::mgVcycle_up::3 36 0.005386 0.005386 0.005386 1.48% MLMG::mgVcycle_up::4 36 0.005321 0.005321 0.005321 1.46% Castro::expand_state() 5 0.005216 0.005216 0.005216 1.43% amrex::average_down 180 0.005139 0.005139 0.005139 1.41% MLPoisson::Fapply() 500 0.00499 0.00499 0.00499 1.37% Gravity::fill_multipole_BCs() 6 0.004201 0.004201 0.004201 1.15% Castro::post_restart() 1 0.003815 0.003815 0.003815 1.05% Gravity::multilevel_solve_for_new_phi() 1 0.003696 0.003696 0.003696 1.01% Gravity::actual_multilevel_solve() 1 0.003678 0.003678 0.003678 1.01% Castro::post_timestep() 5 0.003496 0.003496 0.003496 0.96% MLCellLinOp::solutionResidual() 42 0.003271 0.003271 0.003271 0.90% MultiFab::Xpay() 258 0.002812 0.002812 0.002812 0.77% MLCellLinOp::defineBC() 6 0.002716 0.002716 0.002716 0.74% MLMG::prepareForSolve() 6 0.002716 0.002716 0.002716 0.74% MLMG::computeResidual() 36 0.00271 0.00271 0.00271 0.74% Castro::construct_new_source() 25 0.002671 0.002671 0.002671 0.73% Castro::reset_internal_energy(MultiFab) 30 0.00267 0.00267 0.00267 0.73% BndryData::define() 6 0.002566 0.002566 0.002566 0.70% Castro::estTimeStep() 10 0.002201 0.002201 0.002201 0.60% Castro::construct_new_gravity_source() 5 0.00177 0.00177 0.00177 0.49% Castro::construct_old_source() 25 0.001619 0.001619 0.001619 0.44% Castro::construct_old_gravity_source() 5 0.00153 0.00153 0.00153 0.42% Castro::enforce_speed_limit() 30 0.00127 0.00127 0.00127 0.35% Castro::computeNewDt() 5 0.001148 0.001148 0.001148 0.31% Castro::reset_internal_energy(Fab) 240 0.001069 0.001069 0.001069 0.29% Castro::apply_source_to_state() 10 0.0009251 0.0009251 0.0009251 0.25% MultiFab::Saxpy() 10 0.0009189 0.0009189 0.0009189 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009045 0.0009045 0.0009045 0.25% MLMG::ResNormInf() 42 0.0008568 0.0008568 0.0008568 0.23% MLCellLinOp::setLevelBC() 6 0.0008416 0.0008416 0.0008416 0.23% FabArrayBase::getCPC() 632 0.00082 0.00082 0.00082 0.22% MLMG::getGradSolution() 6 0.0007515 0.0007515 0.0007515 0.21% MLCellLinOp::compGrad() 6 0.0007485 0.0007485 0.0007485 0.21% FabArray::mult() 22 0.0006456 0.0006456 0.0006456 0.18% MLPoisson::prepareForSolve() 6 0.0006382 0.0006382 0.0006382 0.17% MLCellLinOp::prepareForSolve() 6 0.0006335 0.0006335 0.0006335 0.17% FabArray::setDomainBndry() 20 0.0006262 0.0006262 0.0006262 0.17% Castro::check_for_nan() 10 0.0005961 0.0005961 0.0005961 0.16% MultiFab::contains_nan() 10 0.0005888 0.0005888 0.0005888 0.16% MLMG::computeMLResidual() 6 0.0005799 0.0005799 0.0005799 0.16% Gravity::update_max_rhs() 6 0.0004369 0.0004369 0.0004369 0.12% FabArrayBase::CPC::define() 244 0.0004244 0.0004244 0.0004244 0.12% Amr::InitAmr() 1 0.0003768 0.0003768 0.0003768 0.10% FabArrayBase::getFB() 1766 0.0003152 0.0003152 0.0003152 0.09% Gravity::swapTimeLevels() 5 0.0002241 0.0002241 0.0002241 0.06% Castro::create_source_corrector() 5 0.000159 0.000159 0.000159 0.04% Castro::buildMetrics() 1 0.0001542 0.0001542 0.0001542 0.04% MultiFab::Copy() 6 0.0001383 0.0001383 0.0001383 0.04% MLLinOp::define() 6 0.0001366 0.0001366 0.0001366 0.04% MLMG::MLResNormInf() 6 0.0001353 0.0001353 0.0001353 0.04% MultiFab::max() 6 0.0001337 0.0001337 0.0001337 0.04% MLLinOp::defineGrids() 6 0.0001155 0.0001155 0.0001155 0.03% MLMG::MLRhsNormInf() 6 0.0001041 0.0001041 0.0001041 0.03% Castro::finalize_advance() 5 6.633e-05 6.633e-05 6.633e-05 0.02% FabArrayBase::FB::FB() 26 5.679e-05 5.679e-05 5.679e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 3.31e-05 3.31e-05 3.31e-05 0.01% Castro::initMFs() 1 2.859e-05 2.859e-05 2.859e-05 0.01% Amr::writeSmallPlotFile() 1 2.748e-05 2.748e-05 2.748e-05 0.01% Castro::swap_state_time_levels() 5 2.728e-05 2.728e-05 2.728e-05 0.01% makeSFC 30 2.522e-05 2.522e-05 2.522e-05 0.01% Castro::finalize_do_advance() 5 1.887e-05 1.887e-05 1.887e-05 0.01% DistributionMapping::Distribute() 31 9.984e-06 9.984e-06 9.984e-06 0.00% Amr::initSubcycle() 1 8.917e-06 8.917e-06 8.917e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.692e-06 4.692e-06 4.692e-06 0.00% MLMG::buildFineMask() 6 3.193e-06 3.193e-06 3.193e-06 0.00% Gravity::set_mass_offset() 6 2.107e-06 2.107e-06 2.107e-06 0.00% Castro::retry_advance_ctu() 5 1.947e-06 1.947e-06 1.947e-06 0.00% Castro::FluxRegCrseInit 5 1.692e-06 1.692e-06 1.692e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.203e-06 1.203e-06 1.203e-06 0.00% Castro::FluxRegFineAdd() 5 1.179e-06 1.179e-06 1.179e-06 0.00% AmrLevel::AmrLevel() 1 9.4e-07 9.4e-07 9.4e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.07-4-gcbdc6580ee3d) finalized