Initializing CUDA...
CUDA initialized with 1 device.
AMReX (23.02-50-g32a65ffa8aba) initialized
Starting run at 10:01:59 UTC on 2023-03-01.
Successfully read inputs file ...
Castro git describe: 23.01-47-g97cf48985
AMReX git describe: 23.02-50-g32a65ffa8
Microphysics git describe: 23.02-16-g3e0cd993
reading extern runtime parameters ...
3 Species:
C12 O16 Mg24
Successfully read inputs file ...
INITIAL GRIDS
Level 0 8 grids 262144 cells 100 % of domain
smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32
CHECKPOINT: file = dustcollapse-restart_chk00000
checkPoint() time = 0.05673175 secs.
PLOTFILE: file = dustcollapse-restart_plt00000
Write plotfile time = 0.03243021 seconds
[Level 0 step 1] ADVANCE with dt = 4.541742215e-05
[Level 0 step 1] Advanced 262144 cells
[STEP 1] Coarse TimeStep time: 0.04590672
[STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05
[Level 0 step 2] ADVANCE with dt = 4.768829326e-05
[Level 0 step 2] Advanced 262144 cells
[STEP 2] Coarse TimeStep time: 0.048279034
[STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05
[Level 0 step 3] ADVANCE with dt = 5.007270792e-05
[Level 0 step 3] Advanced 262144 cells
[STEP 3] Coarse TimeStep time: 0.057913507
[STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05
[Level 0 step 4] ADVANCE with dt = 5.257634331e-05
[Level 0 step 4] Advanced 262144 cells
[STEP 4] Coarse TimeStep time: 0.059651203
[STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05
[Level 0 step 5] ADVANCE with dt = 5.520516048e-05
[Level 0 step 5] Advanced 262144 cells
[STEP 5] Coarse TimeStep time: 0.071497568
[STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05
CHECKPOINT: file = dustcollapse-restart_chk00005
checkPoint() time = 0.055596324 secs.
[Level 0 step 6] ADVANCE with dt = 5.79654185e-05
[Level 0 step 6] Advanced 262144 cells
[STEP 6] Coarse TimeStep time: 0.059882162
[STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05
[Level 0 step 7] ADVANCE with dt = 6.086368943e-05
[Level 0 step 7] Advanced 262144 cells
[STEP 7] Coarse TimeStep time: 0.047716008
[STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05
[Level 0 step 8] ADVANCE with dt = 6.39068739e-05
[Level 0 step 8] Advanced 262144 cells
[STEP 8] Coarse TimeStep time: 0.052432366
[STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05
[Level 0 step 9] ADVANCE with dt = 6.71022176e-05
[Level 0 step 9] Advanced 262144 cells
[STEP 9] Coarse TimeStep time: 0.055375551
[STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05
[Level 0 step 10] ADVANCE with dt = 7.045732848e-05
[Level 0 step 10] Advanced 262144 cells
[STEP 10] Coarse TimeStep time: 0.060007136
[STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05
CHECKPOINT: file = dustcollapse-restart_chk00010
checkPoint() time = 0.05566734 secs.
PLOTFILE: file = dustcollapse-restart_plt00010
Write plotfile time = 0.032475875 seconds
Ending run at 10:02:00 UTC on 2023-03-01.
Run time = 0.840012688
Run time without initialization = 0.703070957
Average number of zones advanced per microsecond: 3.729
Average number of zones advanced per microsecond per rank: 3.729
CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472
TinyProfiler total time across processes [min...avg...max]: 0.84 ... 0.84 ... 0.84
--------------------------------------------------------------------------------------------
Name NCalls Excl. Min Excl. Avg Excl. Max Max %
--------------------------------------------------------------------------------------------
VisMF::Write(FabArray) 11 0.2262 0.2262 0.2262 26.92%
Castro::construct_ctu_hydro_source() 10 0.2043 0.2043 0.2043 24.32%
MLCellLinOp::applyBC() 4433 0.07535 0.07535 0.07535 8.97%
FillBoundary_nowait() 4023 0.0319 0.0319 0.0319 3.80%
MLPoisson::Fsmooth() 3280 0.03158 0.03158 0.03158 3.76%
StateData::FillBoundary(geom) 328 0.02324 0.02324 0.02324 2.77%
amrex::Dot() 1114 0.02011 0.02011 0.02011 2.39%
Castro::normalize_species() 62 0.0153 0.0153 0.0153 1.82%
amrex::Copy() 1029 0.01454 0.01454 0.01454 1.73%
Castro::computeTemp() 63 0.01444 0.01444 0.01444 1.72%
StateDataPhysBCFunct::() 41 0.01443 0.01443 0.01443 1.72%
FabArray::norminf() 743 0.01403 0.01403 0.01403 1.67%
FabArray::ParallelCopy_nowait() 861 0.0132 0.0132 0.0132 1.57%
FabArray::setVal() 1144 0.01288 0.01288 0.01288 1.53%
MLPoisson::Fapply() 1142 0.01024 0.01024 0.01024 1.22%
MLCellLinOp::defineAuxData() 11 0.009403 0.009403 0.009403 1.12%
Castro::enforce_min_density() 62 0.008519 0.008519 0.008519 1.01%
FabArray::Saxpy() 813 0.007941 0.007941 0.007941 0.95%
FabArray::Xpay() 821 0.007838 0.007838 0.007838 0.93%
MLMG::addInterpCorrection() 410 0.006741 0.006741 0.006741 0.80%
Gravity::fill_multipole_BCs() 11 0.00634 0.00634 0.00634 0.75%
amrex::average_down 410 0.005923 0.005923 0.005923 0.71%
Castro::estTimeStep() 21 0.004794 0.004794 0.004794 0.57%
FabArray::LinComb() 557 0.004384 0.004384 0.004384 0.52%
amrex::Add() 164 0.004341 0.004341 0.004341 0.52%
Castro::reset_internal_energy(MultiFab) 63 0.004234 0.004234 0.004234 0.50%
Amr::checkPoint() 3 0.003925 0.003925 0.003925 0.47%
BndryData::define() 11 0.003563 0.003563 0.003563 0.42%
Castro::construct_new_gravity_source() 10 0.003281 0.003281 0.003281 0.39%
Castro::do_advance_ctu() 10 0.002612 0.002612 0.002612 0.31%
Castro::construct_old_gravity_source() 10 0.002592 0.002592 0.002592 0.31%
Amr::writePlotFile() 2 0.002316 0.002316 0.002316 0.28%
MLCGSolver::bicgstab 82 0.002234 0.002234 0.002234 0.27%
FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001639 0.001639 0.001639 0.20%
Gravity::actual_solve_with_mlmg() 11 0.001483 0.001483 0.001483 0.18%
Castro::reset_internal_energy(Fab) 504 0.001427 0.001427 0.001427 0.17%
MLCellLinOp::setLevelBC() 11 0.001351 0.001351 0.001351 0.16%
FabArray::mult() 43 0.001328 0.001328 0.001328 0.16%
FabArray::setDomainBndry() 41 0.001287 0.001287 0.001287 0.15%
Castro::initData() 1 0.001244 0.001244 0.001244 0.15%
MultiFab::contains_nan() 20 0.001193 0.001193 0.001193 0.14%
Castro::enforce_speed_limit() 62 0.001155 0.001155 0.001155 0.14%
MLCellLinOp::smooth() 1640 0.001139 0.001139 0.001139 0.14%
MLCellLinOp::prepareForSolve() 11 0.001114 0.001114 0.001114 0.13%
MLCellLinOp::compGrad() 11 0.0008934 0.0008934 0.0008934 0.11%
MLMG::prepareForSolve() 11 0.0008713 0.0008713 0.0008713 0.10%
FabArray::FillBoundary() 4023 0.0008203 0.0008203 0.0008203 0.10%
FabArrayBase::getCPC() 1323 0.0007383 0.0007383 0.0007383 0.09%
FabArrayBase::CPC::define() 454 0.0007091 0.0007091 0.0007091 0.08%
FabArrayBase::getFB() 4023 0.0006353 0.0006353 0.0006353 0.08%
Gravity::get_new_grav_vector() 11 0.0006257 0.0006257 0.0006257 0.07%
Gravity::get_old_grav_vector() 10 0.000555 0.000555 0.000555 0.07%
Amr::InitAmr() 1 0.000495 0.000495 0.000495 0.06%
MLCellLinOp::apply() 1142 0.0004792 0.0004792 0.0004792 0.06%
MLMG::mgVcycle() 82 0.0004063 0.0004063 0.0004063 0.05%
Amr::coarseTimeStep() 10 0.0003673 0.0003673 0.0003673 0.04%
main() 1 0.0003601 0.0003601 0.0003601 0.04%
MLLinOp::defineGrids() 11 0.0003337 0.0003337 0.0003337 0.04%
MLCGSolver::ParallelAllReduce 1514 0.0003017 0.0003017 0.0003017 0.04%
MultiFab::max() 11 0.0002602 0.0002602 0.0002602 0.03%
FabArray::ParallelCopy() 861 0.0002478 0.0002478 0.0002478 0.03%
FillPatchIterator::Initialize 41 0.0002275 0.0002275 0.0002275 0.03%
MLCellLinOp::correctionResidual() 492 0.0002156 0.0002156 0.0002156 0.03%
MLCellLinOp::defineBC() 11 0.0002044 0.0002044 0.0002044 0.02%
Amr::timeStep() 10 0.0001573 0.0001573 0.0001573 0.02%
StateData::checkPoint() 12 0.0001347 0.0001347 0.0001347 0.02%
Castro::subcycle_advance_ctu() 10 0.0001279 0.0001279 0.0001279 0.02%
Gravity::update_max_rhs() 11 0.0001205 0.0001205 0.0001205 0.01%
Gravity::solve_for_phi() 10 0.000117 0.000117 0.000117 0.01%
MLMG:computeResOfCorrection() 410 0.0001095 0.0001095 0.0001095 0.01%
MLMG::mgVcycle_down::0 82 8.956e-05 8.956e-05 8.956e-05 0.01%
MLMG::actualBottomSolve() 82 8.729e-05 8.729e-05 8.729e-05 0.01%
Castro::finalize_advance() 10 8.695e-05 8.695e-05 8.695e-05 0.01%
FabArrayBase::FB::FB() 56 8.579e-05 8.579e-05 8.579e-05 0.01%
MLMG::mgVcycle_down::1 82 8.466e-05 8.466e-05 8.466e-05 0.01%
MLMG::mgVcycle_down::2 82 8.258e-05 8.258e-05 8.258e-05 0.01%
Castro::advance() 10 8.079e-05 8.079e-05 8.079e-05 0.01%
Castro::Castro() 1 8.028e-05 8.028e-05 8.028e-05 0.01%
Castro::expand_state() 10 7.85e-05 7.85e-05 7.85e-05 0.01%
MLMG::mgVcycle_down::3 82 7.842e-05 7.842e-05 7.842e-05 0.01%
MLMG::mgVcycle_down::4 82 7.803e-05 7.803e-05 7.803e-05 0.01%
Castro::clean_state() 62 7.787e-05 7.787e-05 7.787e-05 0.01%
Castro::initialize_advance() 10 7.782e-05 7.782e-05 7.782e-05 0.01%
AmrLevel::checkPoint() 3 7.547e-05 7.547e-05 7.547e-05 0.01%
MLMG::solve() 11 7.516e-05 7.516e-05 7.516e-05 0.01%
MLMG::mgVcycle_up::4 82 6.27e-05 6.27e-05 6.27e-05 0.01%
MLMG::mgVcycle_up::1 82 5.449e-05 5.449e-05 5.449e-05 0.01%
MLMG::mgVcycle_up::2 82 5.359e-05 5.359e-05 5.359e-05 0.01%
MLMG::mgVcycle_up::3 82 5.346e-05 5.346e-05 5.346e-05 0.01%
Castro::initialize_do_advance() 10 5.304e-05 5.304e-05 5.304e-05 0.01%
MLMG::oneIter() 82 5.237e-05 5.237e-05 5.237e-05 0.01%
MLMG::mgVcycle_up::0 82 5.145e-05 5.145e-05 5.145e-05 0.01%
MLCellLinOp::solutionResidual() 93 4.976e-05 4.976e-05 4.976e-05 0.01%
Castro::construct_new_source() 50 4.051e-05 4.051e-05 4.051e-05 0.00%
Castro::finalize_do_advance() 10 3.796e-05 3.796e-05 3.796e-05 0.00%
Castro::swap_state_time_levels() 10 3.582e-05 3.582e-05 3.582e-05 0.00%
MLMG::ResNormInf() 93 3.464e-05 3.464e-05 3.464e-05 0.00%
Castro::enforce_consistent_e() 1 3.285e-05 3.285e-05 3.285e-05 0.00%
MLMG::computeResidual() 82 3.151e-05 3.151e-05 3.151e-05 0.00%
MLMG::mgVcycle_bottom 82 3.081e-05 3.081e-05 3.081e-05 0.00%
FillPatchSingleLevel 41 2.866e-05 2.866e-05 2.866e-05 0.00%
StateData::define() 4 2.848e-05 2.848e-05 2.848e-05 0.00%
makeSFC 55 2.542e-05 2.542e-05 2.542e-05 0.00%
Castro::construct_new_gravity() 10 2.538e-05 2.538e-05 2.538e-05 0.00%
Amr::writeSmallPlotFile() 1 2.371e-05 2.371e-05 2.371e-05 0.00%
Gravity::solve_phi_with_mlmg() 11 2.165e-05 2.165e-05 2.165e-05 0.00%
MLPoisson::define() 11 2.063e-05 2.063e-05 2.063e-05 0.00%
Castro::initMFs() 1 1.992e-05 1.992e-05 1.992e-05 0.00%
Amr::FinalizeInit() 1 1.884e-05 1.884e-05 1.884e-05 0.00%
Castro::buildMetrics() 1 1.871e-05 1.871e-05 1.871e-05 0.00%
Castro::construct_old_source() 50 1.772e-05 1.772e-05 1.772e-05 0.00%
Gravity::multilevel_solve_for_new_phi() 1 1.735e-05 1.735e-05 1.735e-05 0.00%
MLLinOp::makeAgglomeratedDMap 11 1.642e-05 1.642e-05 1.642e-05 0.00%
Castro::do_new_sources() 10 1.625e-05 1.625e-05 1.625e-05 0.00%
MLLinOp::define() 11 1.588e-05 1.588e-05 1.588e-05 0.00%
Castro::do_old_sources() 10 1.517e-05 1.517e-05 1.517e-05 0.00%
AmrLevel::AmrLevel(dm) 1 1.455e-05 1.455e-05 1.455e-05 0.00%
DistributionMapping::Distribute() 56 1.439e-05 1.439e-05 1.439e-05 0.00%
Amr::defBaseLevel() 1 1.361e-05 1.361e-05 1.361e-05 0.00%
Castro::check_for_nan() 20 1.049e-05 1.049e-05 1.049e-05 0.00%
Amr::InitializeInit() 1 1.046e-05 1.046e-05 1.046e-05 0.00%
Castro::apply_source_to_state() 20 1.025e-05 1.025e-05 1.025e-05 0.00%
Castro::post_init() 1 9.773e-06 9.773e-06 9.773e-06 0.00%
Castro::construct_old_gravity() 10 9.608e-06 9.608e-06 9.608e-06 0.00%
Castro::post_timestep() 10 9.518e-06 9.518e-06 9.518e-06 0.00%
Gravity::swapTimeLevels() 10 9.2e-06 9.2e-06 9.2e-06 0.00%
Amr::initSubcycle() 1 8.301e-06 8.301e-06 8.301e-06 0.00%
Gravity::actual_multilevel_solve() 1 8.291e-06 8.291e-06 8.291e-06 0.00%
MLPoisson::prepareForSolve() 11 7.967e-06 7.967e-06 7.967e-06 0.00%
MLMG::computeMLResidual() 11 6.818e-06 6.818e-06 6.818e-06 0.00%
Castro::computeNewDt() 9 6.665e-06 6.665e-06 6.665e-06 0.00%
MLMG::getGradSolution() 11 5.747e-06 5.747e-06 5.747e-06 0.00%
Gravity::set_mass_offset() 11 4.99e-06 4.99e-06 4.99e-06 0.00%
AmrLevel::checkPointPost() 3 4.953e-06 4.953e-06 4.953e-06 0.00%
Castro::create_source_corrector() 10 3.761e-06 3.761e-06 3.761e-06 0.00%
MLMG::MLRhsNormInf() 11 3.532e-06 3.532e-06 3.532e-06 0.00%
MLMG::MLResNormInf() 11 3.461e-06 3.461e-06 3.461e-06 0.00%
Castro::retry_advance_ctu() 10 3.445e-06 3.445e-06 3.445e-06 0.00%
Castro::FluxRegCrseInit 10 2.775e-06 2.775e-06 2.775e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 2.625e-06 2.625e-06 2.625e-06 0.00%
Amr::init() 1 2.57e-06 2.57e-06 2.57e-06 0.00%
MLLinOp::makeSubCommunicator() 11 2.478e-06 2.478e-06 2.478e-06 0.00%
Castro::computeInitialDt() 2 2.439e-06 2.439e-06 2.439e-06 0.00%
Castro::FluxRegFineAdd() 10 2.001e-06 2.001e-06 2.001e-06 0.00%
AmrLevel::checkPointPre() 3 1.93e-06 1.93e-06 1.93e-06 0.00%
Amr::initialInit() 1 1.016e-06 1.016e-06 1.016e-06 0.00%
Castro::post_regrid() 1 1e-06 1e-06 1e-06 0.00%
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Name NCalls Incl. Min Incl. Avg Incl. Max Max %
--------------------------------------------------------------------------------------------
main() 1 0.84 0.84 0.84 100.00%
Amr::coarseTimeStep() 10 0.6704 0.6704 0.6704 79.80%
Amr::timeStep() 10 0.5555 0.5555 0.5555 66.13%
Castro::advance() 10 0.5487 0.5487 0.5487 65.31%
Castro::subcycle_advance_ctu() 10 0.537 0.537 0.537 63.92%
Castro::do_advance_ctu() 10 0.5368 0.5368 0.5368 63.90%
Gravity::solve_phi_with_mlmg() 11 0.2793 0.2793 0.2793 33.25%
Gravity::actual_solve_with_mlmg() 11 0.2725 0.2725 0.2725 32.44%
Castro::construct_new_gravity() 10 0.2567 0.2567 0.2567 30.56%
MLMG::solve() 11 0.2524 0.2524 0.2524 30.05%
Gravity::solve_for_phi() 10 0.2398 0.2398 0.2398 28.55%
MLMG::oneIter() 82 0.2384 0.2384 0.2384 28.37%
MLMG::mgVcycle() 82 0.2347 0.2347 0.2347 27.94%
VisMF::Write(FabArray) 11 0.2262 0.2262 0.2262 26.92%
Castro::construct_ctu_hydro_source() 10 0.2043 0.2043 0.2043 24.32%
Amr::checkPoint() 3 0.1681 0.1681 0.1681 20.01%
AmrLevel::checkPoint() 3 0.1642 0.1642 0.1642 19.55%
StateData::checkPoint() 12 0.1641 0.1641 0.1641 19.54%
Amr::init() 1 0.1363 0.1363 0.1363 16.22%
MLCellLinOp::smooth() 1640 0.1159 0.1159 0.1159 13.80%
MLCellLinOp::applyBC() 4433 0.1088 0.1088 0.1088 12.95%
MLMG::mgVcycle_bottom 82 0.07195 0.07195 0.07195 8.56%
MLMG::actualBottomSolve() 82 0.07192 0.07192 0.07192 8.56%
MLCGSolver::bicgstab 82 0.07124 0.07124 0.07124 8.48%
Amr::writePlotFile() 2 0.06502 0.06502 0.06502 7.74%
Amr::initialInit() 1 0.04697 0.04697 0.04697 5.59%
Castro::clean_state() 62 0.04436 0.04436 0.04436 5.28%
FillPatchIterator::Initialize 41 0.04326 0.04326 0.04326 5.15%
Amr::FinalizeInit() 1 0.04302 0.04302 0.04302 5.12%
Castro::post_init() 1 0.04177 0.04177 0.04177 4.97%
FillPatchSingleLevel 41 0.04174 0.04174 0.04174 4.97%
Gravity::multilevel_solve_for_new_phi() 1 0.03993 0.03993 0.03993 4.75%
Gravity::actual_multilevel_solve() 1 0.03991 0.03991 0.03991 4.75%
StateDataPhysBCFunct::() 41 0.03768 0.03768 0.03768 4.49%
MLCellLinOp::apply() 1142 0.03586 0.03586 0.03586 4.27%
MLMG::mgVcycle_down::0 82 0.03407 0.03407 0.03407 4.06%
FabArray::FillBoundary() 4023 0.03344 0.03344 0.03344 3.98%
FillBoundary_nowait() 4023 0.03262 0.03262 0.03262 3.88%
MLPoisson::Fsmooth() 3280 0.03158 0.03158 0.03158 3.76%
MLMG::mgVcycle_up::0 82 0.02584 0.02584 0.02584 3.08%
StateData::FillBoundary(geom) 328 0.02324 0.02324 0.02324 2.77%
MLCellLinOp::correctionResidual() 492 0.02193 0.02193 0.02193 2.61%
amrex::Dot() 1114 0.02011 0.02011 0.02011 2.39%
Castro::computeTemp() 63 0.0201 0.0201 0.0201 2.39%
MLMG:computeResOfCorrection() 410 0.01934 0.01934 0.01934 2.30%
Castro::initialize_do_advance() 10 0.01925 0.01925 0.01925 2.29%
Gravity::get_new_grav_vector() 11 0.01842 0.01842 0.01842 2.19%
MLPoisson::define() 11 0.01586 0.01586 0.01586 1.89%
MLMG::mgVcycle_down::1 82 0.01553 0.01553 0.01553 1.85%
Castro::normalize_species() 62 0.0153 0.0153 0.0153 1.82%
Castro::construct_old_gravity() 10 0.01455 0.01455 0.01455 1.73%
Gravity::get_old_grav_vector() 10 0.01454 0.01454 0.01454 1.73%
amrex::Copy() 1029 0.01454 0.01454 0.01454 1.73%
MLMG::mgVcycle_down::2 82 0.01445 0.01445 0.01445 1.72%
FabArray::ParallelCopy() 861 0.01428 0.01428 0.01428 1.70%
MLMG::mgVcycle_down::3 82 0.01411 0.01411 0.01411 1.68%
FabArray::ParallelCopy_nowait() 861 0.01403 0.01403 0.01403 1.67%
FabArray::norminf() 743 0.01403 0.01403 0.01403 1.67%
Castro::do_new_sources() 10 0.01402 0.01402 0.01402 1.67%
MLMG::mgVcycle_down::4 82 0.01389 0.01389 0.01389 1.65%
FabArray::setVal() 1144 0.01288 0.01288 0.01288 1.53%
MLCGSolver::ParallelAllReduce 1514 0.01209 0.01209 0.01209 1.44%
MLMG::addInterpCorrection() 410 0.01185 0.01185 0.01185 1.41%
Castro::expand_state() 10 0.01176 0.01176 0.01176 1.40%
MLMG::mgVcycle_up::4 82 0.01132 0.01132 0.01132 1.35%
MLMG::mgVcycle_up::1 82 0.01128 0.01128 0.01128 1.34%
amrex::average_down 410 0.01105 0.01105 0.01105 1.32%
MLMG::mgVcycle_up::2 82 0.01104 0.01104 0.01104 1.31%
Castro::initialize_advance() 10 0.01102 0.01102 0.01102 1.31%
MLMG::mgVcycle_up::3 82 0.01082 0.01082 0.01082 1.29%
MLCellLinOp::defineAuxData() 11 0.01071 0.01071 0.01071 1.28%
MLPoisson::Fapply() 1142 0.01024 0.01024 0.01024 1.22%
Castro::do_old_sources() 10 0.00974 0.00974 0.00974 1.16%
Castro::enforce_min_density() 62 0.008519 0.008519 0.008519 1.01%
FabArray::Saxpy() 813 0.007941 0.007941 0.007941 0.95%
FabArray::Xpay() 821 0.007838 0.007838 0.007838 0.93%
MLCellLinOp::solutionResidual() 93 0.007082 0.007082 0.007082 0.84%
Castro::post_timestep() 10 0.006694 0.006694 0.006694 0.80%
Gravity::fill_multipole_BCs() 11 0.006582 0.006582 0.006582 0.78%
MLMG::computeResidual() 82 0.006119 0.006119 0.006119 0.73%
Castro::reset_internal_energy(MultiFab) 63 0.005661 0.005661 0.005661 0.67%
Castro::estTimeStep() 21 0.004794 0.004794 0.004794 0.57%
MLCellLinOp::defineBC() 11 0.004717 0.004717 0.004717 0.56%
MLMG::prepareForSolve() 11 0.004591 0.004591 0.004591 0.55%
BndryData::define() 11 0.004512 0.004512 0.004512 0.54%
FabArray::LinComb() 557 0.004384 0.004384 0.004384 0.52%
amrex::Add() 164 0.004341 0.004341 0.004341 0.52%
Amr::InitializeInit() 1 0.003946 0.003946 0.003946 0.47%
Amr::defBaseLevel() 1 0.003936 0.003936 0.003936 0.47%
Castro::initData() 1 0.00345 0.00345 0.00345 0.41%
Castro::construct_new_source() 50 0.003322 0.003322 0.003322 0.40%
Castro::construct_new_gravity_source() 10 0.003281 0.003281 0.003281 0.39%
Castro::construct_old_source() 50 0.00261 0.00261 0.00261 0.31%
Castro::construct_old_gravity_source() 10 0.002592 0.002592 0.002592 0.31%
Castro::computeNewDt() 9 0.002415 0.002415 0.002415 0.29%
MLMG::ResNormInf() 93 0.002073 0.002073 0.002073 0.25%
Castro::apply_source_to_state() 20 0.001812 0.001812 0.001812 0.22%
FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001639 0.001639 0.001639 0.20%
FabArrayBase::getCPC() 1323 0.001447 0.001447 0.001447 0.17%
Castro::reset_internal_energy(Fab) 504 0.001427 0.001427 0.001427 0.17%
MLMG::getGradSolution() 11 0.001366 0.001366 0.001366 0.16%
MLCellLinOp::compGrad() 11 0.00136 0.00136 0.00136 0.16%
MLCellLinOp::setLevelBC() 11 0.001351 0.001351 0.001351 0.16%
FabArray::mult() 43 0.001328 0.001328 0.001328 0.16%
FabArray::setDomainBndry() 41 0.001287 0.001287 0.001287 0.15%
Castro::check_for_nan() 20 0.001203 0.001203 0.001203 0.14%
MultiFab::contains_nan() 20 0.001193 0.001193 0.001193 0.14%
Castro::enforce_speed_limit() 62 0.001155 0.001155 0.001155 0.14%
MLPoisson::prepareForSolve() 11 0.001122 0.001122 0.001122 0.13%
MLCellLinOp::prepareForSolve() 11 0.001114 0.001114 0.001114 0.13%
Castro::post_regrid() 1 0.00108 0.00108 0.00108 0.13%
MLMG::computeMLResidual() 11 0.001001 0.001001 0.001001 0.12%
Castro::computeInitialDt() 2 0.0008605 0.0008605 0.0008605 0.10%
Gravity::update_max_rhs() 11 0.0008308 0.0008308 0.0008308 0.10%
FabArrayBase::getFB() 4023 0.0007211 0.0007211 0.0007211 0.09%
FabArrayBase::CPC::define() 454 0.0007091 0.0007091 0.0007091 0.08%
Castro::finalize_advance() 10 0.0006015 0.0006015 0.0006015 0.07%
Amr::InitAmr() 1 0.0005033 0.0005033 0.0005033 0.06%
Gravity::swapTimeLevels() 10 0.0004359 0.0004359 0.0004359 0.05%
Castro::Castro() 1 0.0004254 0.0004254 0.0004254 0.05%
MLLinOp::define() 11 0.0004072 0.0004072 0.0004072 0.05%
MLLinOp::defineGrids() 11 0.0003913 0.0003913 0.0003913 0.05%
MLMG::MLResNormInf() 11 0.0002839 0.0002839 0.0002839 0.03%
MultiFab::max() 11 0.0002602 0.0002602 0.0002602 0.03%
MLMG::MLRhsNormInf() 11 0.0002176 0.0002176 0.0002176 0.03%
Castro::buildMetrics() 1 0.0001581 0.0001581 0.0001581 0.02%
FabArrayBase::FB::FB() 56 8.579e-05 8.579e-05 8.579e-05 0.01%
MLLinOp::makeAgglomeratedDMap 11 5.512e-05 5.512e-05 5.512e-05 0.01%
AmrLevel::AmrLevel(dm) 1 4.303e-05 4.303e-05 4.303e-05 0.01%
makeSFC 55 3.869e-05 3.869e-05 3.869e-05 0.00%
Castro::finalize_do_advance() 10 3.796e-05 3.796e-05 3.796e-05 0.00%
Castro::swap_state_time_levels() 10 3.582e-05 3.582e-05 3.582e-05 0.00%
Castro::enforce_consistent_e() 1 3.285e-05 3.285e-05 3.285e-05 0.00%
StateData::define() 4 2.848e-05 2.848e-05 2.848e-05 0.00%
Amr::writeSmallPlotFile() 1 2.371e-05 2.371e-05 2.371e-05 0.00%
Castro::initMFs() 1 1.992e-05 1.992e-05 1.992e-05 0.00%
DistributionMapping::Distribute() 56 1.439e-05 1.439e-05 1.439e-05 0.00%
Amr::initSubcycle() 1 8.301e-06 8.301e-06 8.301e-06 0.00%
Gravity::set_mass_offset() 11 4.99e-06 4.99e-06 4.99e-06 0.00%
AmrLevel::checkPointPost() 3 4.953e-06 4.953e-06 4.953e-06 0.00%
Castro::create_source_corrector() 10 3.761e-06 3.761e-06 3.761e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 3.746e-06 3.746e-06 3.746e-06 0.00%
Castro::retry_advance_ctu() 10 3.445e-06 3.445e-06 3.445e-06 0.00%
Castro::FluxRegCrseInit 10 2.775e-06 2.775e-06 2.775e-06 0.00%
MLLinOp::makeSubCommunicator() 11 2.478e-06 2.478e-06 2.478e-06 0.00%
Castro::FluxRegFineAdd() 10 2.001e-06 2.001e-06 2.001e-06 0.00%
AmrLevel::checkPointPre() 3 1.93e-06 1.93e-06 1.93e-06 0.00%
--------------------------------------------------------------------------------------------
Device Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Arena::Initialize() 1 1 102 MiB 9042 MiB
Castro::construct_ctu_hydro_source() 2880 2880 111 MiB 692 MiB
Castro::initMFs() 48 48 67 MiB 68 MiB
Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB
StateData::define() 32 32 55 MiB 55 MiB
FillPatchIterator::Initialize 328 328 1096 KiB 39 MiB
Castro::initialize_do_advance() 80 80 24 MiB 39 MiB
ResizeRandomSeed 1 1 30 MiB 30 MiB
Amr::writePlotFile() 16 16 2175 KiB 28 MiB
Castro::initialize_advance() 80 80 15 MiB 23 MiB
Castro::buildMetrics() 32 32 15 MiB 15 MiB
Castro::Castro() 48 48 7611 KiB 14 MiB
MLMG::prepareForSolve() 660 660 3697 KiB 12 MiB
Gravity::get_new_grav_vector() 91 91 227 KiB 10 MiB
Gravity::get_old_grav_vector() 80 80 177 KiB 10 MiB
Gravity::multilevel_solve_for_new_phi() 24 24 7524 KiB 7586 KiB
Gravity::fill_multipole_BCs() 154 154 12 KiB 2053 KiB
Gravity::update_max_rhs() 88 88 1900 B 2048 KiB
Gravity::solve_for_phi() 80 80 583 KiB 2048 KiB
Gravity::actual_multilevel_solve() 8 8 97 KiB 2048 KiB
BndryData::define() 1056 1056 337 KiB 1095 KiB
MLCellLinOp::defineAuxData() 1716 1716 214 KiB 671 KiB
Castro::estTimeStep() 21 21 2327 B 480 KiB
VisMF::Write(FabArray) 656 656 3501 B 320 KiB
Castro::normalize_species() 62 62 5930 B 320 KiB
amrex::average_down 1067 1067 1301 B 257 KiB
MLMG::addInterpCorrection() 1066 1066 1217 B 257 KiB
amrex::Dot() 1360 1360 3516 B 160 KiB
FabArray::norminf() 907 907 2459 B 160 KiB
Castro::do_advance_ctu() 10 10 480 B 160 KiB
MultiFab::max() 11 11 47 B 160 KiB
Castro::initData() 1 1 28 B 160 KiB
FabArray::setVal() 106 106 21 KiB 26 KiB
MultiFab::contains_nan() 20 20 28 B 20 KiB
MLPoisson::Fsmooth() 132 132 3615 B 12 KiB
FabArray::ParallelCopy_nowait() 861 861 47 B 10 KiB
FillBoundary_nowait() 760 760 333 B 9648 B
MLCellLinOp::applyBC() 8866 8866 232 B 9344 B
MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B
amrex::Copy() 100 100 3866 B 6144 B
StateData::FillBoundary(geom) 1992 1992 50 B 2688 B
FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B
MLCGSolver::bicgstab 738 738 123 B 1472 B
MLCellLinOp::defineBC() 66 66 380 B 1248 B
------------------------------------------------------------------------------
Managed Memory Usage:
-----------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
-----------------------------------------------------------------
The_Managed_Arena::Initialize() 1 1 570 B 8192 KiB
-----------------------------------------------------------------
Pinned Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Pinned_Arena::Initialize() 1 1 52 KiB 8192 KiB
VisMF::Write(FabArray) 744 744 626 KiB 3584 KiB
FabArray::setVal() 106 106 21 KiB 26 KiB
MLPoisson::Fsmooth() 132 132 3615 B 12 KiB
FabArray::ParallelCopy_nowait() 861 861 47 B 10 KiB
FillBoundary_nowait() 760 760 333 B 9648 B
MLCellLinOp::applyBC() 4433 4433 230 B 9328 B
MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B
amrex::Copy() 100 100 3866 B 6144 B
Gravity::get_new_grav_vector() 3 3 2901 B 3072 B
Gravity::fill_multipole_BCs() 33 33 4 B 2832 B
StateData::FillBoundary(geom) 1992 1992 51 B 2688 B
FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B
amrex::average_down 83 83 272 B 1296 B
MLMG::addInterpCorrection() 82 82 2 B 1024 B
MLMG::prepareForSolve() 11 11 308 B 1024 B
MLCellLinOp::setLevelBC() 66 66 0 B 768 B
amrex::Dot() 1360 1360 25 B 400 B
FabArray::norminf() 907 907 18 B 272 B
Castro::estTimeStep() 21 21 0 B 32 B
MultiFab::contains_nan() 20 20 0 B 16 B
MultiFab::max() 11 11 0 B 16 B
Castro::normalize_species() 62 62 0 B 16 B
Castro::initData() 1 1 0 B 16 B
Castro::do_advance_ctu() 10 10 0 B 16 B
------------------------------------------------------------------------------
Cpu (amrex_mempool) Memory Usage:
-----------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
-----------------------------------------------------
amrex_mempool_init() 1 1 58 KiB 8192 KiB
-----------------------------------------------------
Unused ParmParse Variables:
[TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2]
[TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2]
Total GPU global memory (MB): 12056
Free GPU global memory (MB): 2459
[The Arena] space allocated (MB): 9042
[The Arena] space used (MB): 0
[The Managed Arena] space allocated (MB): 8
[The Managed Arena] space used (MB): 0
[The Pinned Arena] space allocated (MB): 8
[The Pinned Arena] space used (MB): 0
AMReX (23.02-50-g32a65ffa8aba) finalized
Initializing CUDA...
CUDA initialized with 1 device.
AMReX (23.02-50-g32a65ffa8aba) initialized
Starting run at 10:02:01 UTC on 2023-03-01.
Successfully read inputs file ...
Castro git describe: 23.01-47-g97cf48985
AMReX git describe: 23.02-50-g32a65ffa8
Microphysics git describe: 23.02-16-g3e0cd993
reading extern runtime parameters ...
3 Species:
C12 O16 Mg24
restarting calculation from file: dustcollapse-restart_chk00005
Successfully read inputs file ...
read CPU time: 0.475733228
Restart time = 0.047838669 seconds.
[Level 0 step 6] ADVANCE with dt = 5.79654185e-05
[Level 0 step 6] Advanced 262144 cells
[STEP 6] Coarse TimeStep time: 0.054853592
[STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05
[Level 0 step 7] ADVANCE with dt = 6.086368943e-05
[Level 0 step 7] Advanced 262144 cells
[STEP 7] Coarse TimeStep time: 0.061391471
[STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05
[Level 0 step 8] ADVANCE with dt = 6.39068739e-05
[Level 0 step 8] Advanced 262144 cells
[STEP 8] Coarse TimeStep time: 0.063589835
[STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05
[Level 0 step 9] ADVANCE with dt = 6.71022176e-05
[Level 0 step 9] Advanced 262144 cells
[STEP 9] Coarse TimeStep time: 0.059646298
[STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05
[Level 0 step 10] ADVANCE with dt = 7.045732848e-05
[Level 0 step 10] Advanced 262144 cells
[STEP 10] Coarse TimeStep time: 0.063557745
[STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05
PLOTFILE: file = dustcollapse-restart_plt00010
Write plotfile time = 0.037597739 seconds
Ending run at 10:02:01 UTC on 2023-03-01.
Run time = 0.389663425
Run time without initialization = 0.341221588
Average number of zones advanced per microsecond: 3.841
Average number of zones advanced per microsecond per rank: 3.841
CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472
TinyProfiler total time across processes [min...avg...max]: 0.3897 ... 0.3897 ... 0.3897
--------------------------------------------------------------------------------------------
Name NCalls Excl. Min Excl. Avg Excl. Max Max %
--------------------------------------------------------------------------------------------
Castro::construct_ctu_hydro_source() 5 0.1037 0.1037 0.1037 26.61%
VisMF::Read() 3 0.04136 0.04136 0.04136 10.61%
VisMF::Write(FabArray) 1 0.03599 0.03599 0.03599 9.24%
MLCellLinOp::applyBC() 1946 0.03444 0.03444 0.03444 8.84%
StateData::FillBoundary(geom) 160 0.02218 0.02218 0.02218 5.69%
MLPoisson::Fsmooth() 1440 0.01423 0.01423 0.01423 3.65%
FillBoundary_nowait() 1766 0.01278 0.01278 0.01278 3.28%
Castro::normalize_species() 30 0.01004 0.01004 0.01004 2.58%
amrex::Dot() 484 0.009009 0.009009 0.009009 2.31%
amrex::Copy() 463 0.007056 0.007056 0.007056 1.81%
Castro::enforce_min_density() 30 0.007043 0.007043 0.007043 1.81%
FabArray::setVal() 537 0.00639 0.00639 0.00639 1.64%
FabArray::norminf() 326 0.006302 0.006302 0.006302 1.62%
FabArray::ParallelCopy_nowait() 380 0.006283 0.006283 0.006283 1.61%
Castro::computeTemp() 30 0.00567 0.00567 0.00567 1.45%
MLCellLinOp::defineAuxData() 6 0.005507 0.005507 0.005507 1.41%
MLPoisson::Fapply() 500 0.004559 0.004559 0.004559 1.17%
StateDataPhysBCFunct::() 20 0.003957 0.003957 0.003957 1.02%
FabArray::Saxpy() 355 0.003681 0.003681 0.003681 0.94%
FabArray::Xpay() 361 0.003512 0.003512 0.003512 0.90%
MLMG::addInterpCorrection() 180 0.003087 0.003087 0.003087 0.79%
Castro::estTimeStep() 10 0.002766 0.002766 0.002766 0.71%
amrex::average_down 180 0.002746 0.002746 0.002746 0.70%
Amr::restart() 1 0.002555 0.002555 0.002555 0.66%
Gravity::fill_multipole_BCs() 6 0.00245 0.00245 0.00245 0.63%
BndryData::define() 6 0.002071 0.002071 0.002071 0.53%
FabArray::LinComb() 242 0.001934 0.001934 0.001934 0.50%
Castro::reset_internal_energy(MultiFab) 30 0.001883 0.001883 0.001883 0.48%
amrex::Add() 72 0.001844 0.001844 0.001844 0.47%
Castro::construct_new_gravity_source() 5 0.001718 0.001718 0.001718 0.44%
Castro::do_advance_ctu() 5 0.00147 0.00147 0.00147 0.38%
Amr::writePlotFile() 1 0.001463 0.001463 0.001463 0.38%
Castro::enforce_speed_limit() 30 0.001387 0.001387 0.001387 0.36%
Castro::construct_old_gravity_source() 5 0.001329 0.001329 0.001329 0.34%
MLCGSolver::bicgstab 36 0.001137 0.001137 0.001137 0.29%
Castro::reset_internal_energy(Fab) 240 0.001074 0.001074 0.001074 0.28%
FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009479 0.0009479 0.0009479 0.24%
Castro::post_timestep() 5 0.0008152 0.0008152 0.0008152 0.21%
Gravity::actual_solve_with_mlmg() 6 0.0007956 0.0007956 0.0007956 0.20%
MLCellLinOp::setLevelBC() 6 0.0007421 0.0007421 0.0007421 0.19%
Castro::finalize_advance() 5 0.0007361 0.0007361 0.0007361 0.19%
MLCellLinOp::smooth() 720 0.0007082 0.0007082 0.0007082 0.18%
FabArray::mult() 22 0.0006663 0.0006663 0.0006663 0.17%
MLCellLinOp::prepareForSolve() 6 0.000641 0.000641 0.000641 0.16%
FabArray::setDomainBndry() 20 0.000635 0.000635 0.000635 0.16%
MultiFab::contains_nan() 10 0.0006092 0.0006092 0.0006092 0.16%
MLMG::prepareForSolve() 6 0.0005317 0.0005317 0.0005317 0.14%
MLCellLinOp::compGrad() 6 0.0004818 0.0004818 0.0004818 0.12%
FabArrayBase::CPC::define() 244 0.0004294 0.0004294 0.0004294 0.11%
FabArray::FillBoundary() 1766 0.0004094 0.0004094 0.0004094 0.11%
Amr::InitAmr() 1 0.0003905 0.0003905 0.0003905 0.10%
Gravity::get_old_grav_vector() 5 0.0003852 0.0003852 0.0003852 0.10%
FabArrayBase::getCPC() 632 0.0003749 0.0003749 0.0003749 0.10%
main() 1 0.0003618 0.0003618 0.0003618 0.09%
Gravity::get_new_grav_vector() 5 0.000319 0.000319 0.000319 0.08%
FabArrayBase::getFB() 1766 0.0002602 0.0002602 0.0002602 0.07%
MLCellLinOp::apply() 500 0.0002511 0.0002511 0.0002511 0.06%
Amr::coarseTimeStep() 5 0.0002401 0.0002401 0.0002401 0.06%
MLMG::mgVcycle() 36 0.000207 0.000207 0.000207 0.05%
MLLinOp::defineGrids() 6 0.0001743 0.0001743 0.0001743 0.04%
Castro::construct_new_source() 25 0.0001444 0.0001444 0.0001444 0.04%
MLCGSolver::ParallelAllReduce 659 0.0001418 0.0001418 0.0001418 0.04%
MultiFab::max() 6 0.0001416 0.0001416 0.0001416 0.04%
FabArray::ParallelCopy() 380 0.0001304 0.0001304 0.0001304 0.03%
FillPatchIterator::Initialize 20 0.0001294 0.0001294 0.0001294 0.03%
Castro::advance() 5 0.0001283 0.0001283 0.0001283 0.03%
MLCellLinOp::correctionResidual() 216 0.0001193 0.0001193 0.0001193 0.03%
MLCellLinOp::defineBC() 6 0.0001177 0.0001177 0.0001177 0.03%
Castro::subcycle_advance_ctu() 5 0.0001171 0.0001171 0.0001171 0.03%
Amr::timeStep() 5 9.871e-05 9.871e-05 9.871e-05 0.03%
AmrLevel::restart() 1 8.096e-05 8.096e-05 8.096e-05 0.02%
Gravity::update_max_rhs() 6 7.697e-05 7.697e-05 7.697e-05 0.02%
StateData::restartDoit() 4 6.416e-05 6.416e-05 6.416e-05 0.02%
FabArrayBase::FB::FB() 26 6.337e-05 6.337e-05 6.337e-05 0.02%
Gravity::solve_for_phi() 5 5.922e-05 5.922e-05 5.922e-05 0.02%
Castro::create_source_corrector() 5 5.869e-05 5.869e-05 5.869e-05 0.02%
Castro::construct_new_gravity() 5 5.659e-05 5.659e-05 5.659e-05 0.01%
MLMG:computeResOfCorrection() 180 5.397e-05 5.397e-05 5.397e-05 0.01%
Castro::initialize_advance() 5 4.966e-05 4.966e-05 4.966e-05 0.01%
MLMG::mgVcycle_down::0 36 4.954e-05 4.954e-05 4.954e-05 0.01%
MLMG::mgVcycle_down::1 36 4.948e-05 4.948e-05 4.948e-05 0.01%
Castro::clean_state() 30 4.93e-05 4.93e-05 4.93e-05 0.01%
Castro::expand_state() 5 4.779e-05 4.779e-05 4.779e-05 0.01%
MLMG::mgVcycle_down::2 36 4.689e-05 4.689e-05 4.689e-05 0.01%
Castro::initialize_do_advance() 5 4.652e-05 4.652e-05 4.652e-05 0.01%
MLMG::mgVcycle_down::4 36 4.65e-05 4.65e-05 4.65e-05 0.01%
MLMG::mgVcycle_down::3 36 4.519e-05 4.519e-05 4.519e-05 0.01%
MLMG::actualBottomSolve() 36 4.489e-05 4.489e-05 4.489e-05 0.01%
Castro::computeNewDt() 5 4.21e-05 4.21e-05 4.21e-05 0.01%
Amr::writeSmallPlotFile() 1 4.181e-05 4.181e-05 4.181e-05 0.01%
MLMG::solve() 6 4.038e-05 4.038e-05 4.038e-05 0.01%
MLMG::mgVcycle_up::4 36 3.219e-05 3.219e-05 3.219e-05 0.01%
Castro::buildMetrics() 1 2.958e-05 2.958e-05 2.958e-05 0.01%
MLMG::mgVcycle_up::0 36 2.884e-05 2.884e-05 2.884e-05 0.01%
MLMG::mgVcycle_up::2 36 2.818e-05 2.818e-05 2.818e-05 0.01%
MLMG::mgVcycle_up::3 36 2.807e-05 2.807e-05 2.807e-05 0.01%
MLMG::mgVcycle_up::1 36 2.751e-05 2.751e-05 2.751e-05 0.01%
MLCellLinOp::solutionResidual() 42 2.638e-05 2.638e-05 2.638e-05 0.01%
Castro::swap_state_time_levels() 5 2.613e-05 2.613e-05 2.613e-05 0.01%
Castro::post_restart() 1 2.556e-05 2.556e-05 2.556e-05 0.01%
MLMG::oneIter() 36 2.539e-05 2.539e-05 2.539e-05 0.01%
Castro::initMFs() 1 2.399e-05 2.399e-05 2.399e-05 0.01%
Castro::finalize_do_advance() 5 2.148e-05 2.148e-05 2.148e-05 0.01%
MLMG::ResNormInf() 42 1.954e-05 1.954e-05 1.954e-05 0.01%
FillPatchSingleLevel 20 1.79e-05 1.79e-05 1.79e-05 0.00%
Gravity::multilevel_solve_for_new_phi() 1 1.63e-05 1.63e-05 1.63e-05 0.00%
MLMG::mgVcycle_bottom 36 1.49e-05 1.49e-05 1.49e-05 0.00%
MLMG::computeResidual() 36 1.471e-05 1.471e-05 1.471e-05 0.00%
MLPoisson::define() 6 1.46e-05 1.46e-05 1.46e-05 0.00%
makeSFC 30 1.425e-05 1.425e-05 1.425e-05 0.00%
Gravity::solve_phi_with_mlmg() 6 1.322e-05 1.322e-05 1.322e-05 0.00%
MLLinOp::makeAgglomeratedDMap 6 1.314e-05 1.314e-05 1.314e-05 0.00%
Castro::check_for_nan() 10 1.19e-05 1.19e-05 1.19e-05 0.00%
MLLinOp::define() 6 1.11e-05 1.11e-05 1.11e-05 0.00%
Castro::do_old_sources() 5 1.026e-05 1.026e-05 1.026e-05 0.00%
Castro::do_new_sources() 5 1.008e-05 1.008e-05 1.008e-05 0.00%
Castro::construct_old_source() 25 9.892e-06 9.892e-06 9.892e-06 0.00%
DistributionMapping::Distribute() 31 8.295e-06 8.295e-06 8.295e-06 0.00%
Amr::initSubcycle() 1 8.22e-06 8.22e-06 8.22e-06 0.00%
Gravity::actual_multilevel_solve() 1 8.103e-06 8.103e-06 8.103e-06 0.00%
Castro::construct_old_gravity() 5 6.632e-06 6.632e-06 6.632e-06 0.00%
Castro::apply_source_to_state() 10 6.604e-06 6.604e-06 6.604e-06 0.00%
Gravity::swapTimeLevels() 5 5.84e-06 5.84e-06 5.84e-06 0.00%
MLPoisson::prepareForSolve() 6 4.664e-06 4.664e-06 4.664e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 4.204e-06 4.204e-06 4.204e-06 0.00%
MLMG::computeMLResidual() 6 3.615e-06 3.615e-06 3.615e-06 0.00%
MLMG::getGradSolution() 6 3.195e-06 3.195e-06 3.195e-06 0.00%
Gravity::set_mass_offset() 6 2.806e-06 2.806e-06 2.806e-06 0.00%
MLMG::MLRhsNormInf() 6 2.319e-06 2.319e-06 2.319e-06 0.00%
MLMG::MLResNormInf() 6 2.277e-06 2.277e-06 2.277e-06 0.00%
Castro::FluxRegCrseInit 5 1.697e-06 1.697e-06 1.697e-06 0.00%
Castro::retry_advance_ctu() 5 1.685e-06 1.685e-06 1.685e-06 0.00%
Amr::init() 1 1.485e-06 1.485e-06 1.485e-06 0.00%
Castro::FluxRegFineAdd() 5 1.256e-06 1.256e-06 1.256e-06 0.00%
MLLinOp::makeSubCommunicator() 6 1.068e-06 1.068e-06 1.068e-06 0.00%
AmrLevel::AmrLevel() 1 8.61e-07 8.61e-07 8.61e-07 0.00%
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Name NCalls Incl. Min Incl. Avg Incl. Max Max %
--------------------------------------------------------------------------------------------
main() 1 0.3897 0.3897 0.3897 99.99%
Amr::coarseTimeStep() 5 0.3033 0.3033 0.3033 77.82%
Amr::timeStep() 5 0.3018 0.3018 0.3018 77.45%
Castro::advance() 5 0.2965 0.2965 0.2965 76.09%
Castro::subcycle_advance_ctu() 5 0.2893 0.2893 0.2893 74.24%
Castro::do_advance_ctu() 5 0.2892 0.2892 0.2892 74.21%
Castro::construct_new_gravity() 5 0.1362 0.1362 0.1362 34.96%
Gravity::solve_phi_with_mlmg() 6 0.128 0.128 0.128 32.85%
Gravity::actual_solve_with_mlmg() 6 0.1253 0.1253 0.1253 32.15%
Gravity::solve_for_phi() 5 0.1248 0.1248 0.1248 32.02%
MLMG::solve() 6 0.1137 0.1137 0.1137 29.18%
MLMG::oneIter() 36 0.1067 0.1067 0.1067 27.37%
MLMG::mgVcycle() 36 0.1051 0.1051 0.1051 26.98%
Castro::construct_ctu_hydro_source() 5 0.1037 0.1037 0.1037 26.61%
MLCellLinOp::smooth() 720 0.05121 0.05121 0.05121 13.14%
MLCellLinOp::applyBC() 1946 0.04795 0.04795 0.04795 12.31%
Amr::init() 1 0.0479 0.0479 0.0479 12.29%
Amr::restart() 1 0.0479 0.0479 0.0479 12.29%
AmrLevel::restart() 1 0.04157 0.04157 0.04157 10.67%
StateData::restartDoit() 4 0.04148 0.04148 0.04148 10.64%
VisMF::Read() 3 0.04136 0.04136 0.04136 10.61%
Amr::writePlotFile() 1 0.0377 0.0377 0.0377 9.68%
VisMF::Write(FabArray) 1 0.03599 0.03599 0.03599 9.24%
MLMG::mgVcycle_bottom 36 0.03254 0.03254 0.03254 8.35%
MLMG::actualBottomSolve() 36 0.03252 0.03252 0.03252 8.35%
MLCGSolver::bicgstab 36 0.03221 0.03221 0.03221 8.27%
FillPatchIterator::Initialize 20 0.02893 0.02893 0.02893 7.42%
FillPatchSingleLevel 20 0.02817 0.02817 0.02817 7.23%
Castro::clean_state() 30 0.02714 0.02714 0.02714 6.97%
StateDataPhysBCFunct::() 20 0.02614 0.02614 0.02614 6.71%
StateData::FillBoundary(geom) 160 0.02218 0.02218 0.02218 5.69%
MLCellLinOp::apply() 500 0.01622 0.01622 0.01622 4.16%
MLMG::mgVcycle_down::0 36 0.01447 0.01447 0.01447 3.71%
MLPoisson::Fsmooth() 1440 0.01423 0.01423 0.01423 3.65%
FabArray::FillBoundary() 1766 0.01352 0.01352 0.01352 3.47%
FillBoundary_nowait() 1766 0.01311 0.01311 0.01311 3.36%
Castro::initialize_do_advance() 5 0.0113 0.0113 0.0113 2.90%
Gravity::get_new_grav_vector() 5 0.01128 0.01128 0.01128 2.90%
Castro::construct_old_gravity() 5 0.01121 0.01121 0.01121 2.88%
Gravity::get_old_grav_vector() 5 0.0112 0.0112 0.0112 2.87%
MLMG::mgVcycle_up::0 36 0.011 0.011 0.011 2.82%
Castro::normalize_species() 30 0.01004 0.01004 0.01004 2.58%
MLCellLinOp::correctionResidual() 216 0.009752 0.009752 0.009752 2.50%
MLPoisson::define() 6 0.009291 0.009291 0.009291 2.38%
amrex::Dot() 484 0.009009 0.009009 0.009009 2.31%
Castro::computeTemp() 30 0.008627 0.008627 0.008627 2.21%
MLMG:computeResOfCorrection() 180 0.008561 0.008561 0.008561 2.20%
Castro::do_new_sources() 5 0.0083 0.0083 0.0083 2.13%
amrex::Copy() 463 0.007056 0.007056 0.007056 1.81%
MLMG::mgVcycle_down::1 36 0.007051 0.007051 0.007051 1.81%
Castro::enforce_min_density() 30 0.007043 0.007043 0.007043 1.81%
FabArray::ParallelCopy() 380 0.006806 0.006806 0.006806 1.75%
FabArray::ParallelCopy_nowait() 380 0.006676 0.006676 0.006676 1.71%
MLMG::mgVcycle_down::2 36 0.006649 0.006649 0.006649 1.71%
MLMG::mgVcycle_down::3 36 0.006437 0.006437 0.006437 1.65%
MLMG::mgVcycle_down::4 36 0.006403 0.006403 0.006403 1.64%
FabArray::setVal() 537 0.00639 0.00639 0.00639 1.64%
FabArray::norminf() 326 0.006302 0.006302 0.006302 1.62%
MLCellLinOp::defineAuxData() 6 0.006291 0.006291 0.006291 1.61%
Castro::do_old_sources() 5 0.006205 0.006205 0.006205 1.59%
Castro::initialize_advance() 5 0.00609 0.00609 0.00609 1.56%
Castro::expand_state() 5 0.006058 0.006058 0.006058 1.55%
MLMG::addInterpCorrection() 180 0.005499 0.005499 0.005499 1.41%
MLCGSolver::ParallelAllReduce 659 0.005446 0.005446 0.005446 1.40%
Castro::post_timestep() 5 0.005192 0.005192 0.005192 1.33%
MLMG::mgVcycle_up::4 36 0.00518 0.00518 0.00518 1.33%
MLMG::mgVcycle_up::1 36 0.005157 0.005157 0.005157 1.32%
amrex::average_down 180 0.005127 0.005127 0.005127 1.32%
MLMG::mgVcycle_up::2 36 0.005063 0.005063 0.005063 1.30%
MLMG::mgVcycle_up::3 36 0.004971 0.004971 0.004971 1.28%
MLPoisson::Fapply() 500 0.004559 0.004559 0.004559 1.17%
FabArray::Saxpy() 355 0.003681 0.003681 0.003681 0.94%
Castro::post_restart() 1 0.003591 0.003591 0.003591 0.92%
FabArray::Xpay() 361 0.003512 0.003512 0.003512 0.90%
Gravity::multilevel_solve_for_new_phi() 1 0.003466 0.003466 0.003466 0.89%
Gravity::actual_multilevel_solve() 1 0.003449 0.003449 0.003449 0.89%
MLCellLinOp::solutionResidual() 42 0.003297 0.003297 0.003297 0.85%
Castro::reset_internal_energy(MultiFab) 30 0.002957 0.002957 0.002957 0.76%
Castro::estTimeStep() 10 0.002766 0.002766 0.002766 0.71%
MLCellLinOp::defineBC() 6 0.002764 0.002764 0.002764 0.71%
MLMG::computeResidual() 36 0.002724 0.002724 0.002724 0.70%
BndryData::define() 6 0.002646 0.002646 0.002646 0.68%
MLMG::prepareForSolve() 6 0.002636 0.002636 0.002636 0.68%
Gravity::fill_multipole_BCs() 6 0.002586 0.002586 0.002586 0.66%
FabArray::LinComb() 242 0.001934 0.001934 0.001934 0.50%
Castro::construct_new_source() 25 0.001863 0.001863 0.001863 0.48%
amrex::Add() 72 0.001844 0.001844 0.001844 0.47%
Castro::construct_new_gravity_source() 5 0.001718 0.001718 0.001718 0.44%
Castro::enforce_speed_limit() 30 0.001387 0.001387 0.001387 0.36%
Castro::construct_old_source() 25 0.001339 0.001339 0.001339 0.34%
Castro::construct_old_gravity_source() 5 0.001329 0.001329 0.001329 0.34%
Castro::computeNewDt() 5 0.001204 0.001204 0.001204 0.31%
Castro::reset_internal_energy(Fab) 240 0.001074 0.001074 0.001074 0.28%
Castro::finalize_advance() 5 0.001004 0.001004 0.001004 0.26%
FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009479 0.0009479 0.0009479 0.24%
MLMG::ResNormInf() 42 0.0009375 0.0009375 0.0009375 0.24%
Castro::apply_source_to_state() 10 0.0009271 0.0009271 0.0009271 0.24%
FabArrayBase::getCPC() 632 0.0008043 0.0008043 0.0008043 0.21%
MLMG::getGradSolution() 6 0.0007538 0.0007538 0.0007538 0.19%
MLCellLinOp::compGrad() 6 0.0007506 0.0007506 0.0007506 0.19%
MLCellLinOp::setLevelBC() 6 0.0007421 0.0007421 0.0007421 0.19%
FabArray::mult() 22 0.0006663 0.0006663 0.0006663 0.17%
MLPoisson::prepareForSolve() 6 0.0006456 0.0006456 0.0006456 0.17%
MLCellLinOp::prepareForSolve() 6 0.000641 0.000641 0.000641 0.16%
FabArray::setDomainBndry() 20 0.000635 0.000635 0.000635 0.16%
Castro::check_for_nan() 10 0.0006211 0.0006211 0.0006211 0.16%
MultiFab::contains_nan() 10 0.0006092 0.0006092 0.0006092 0.16%
MLMG::computeMLResidual() 6 0.0005913 0.0005913 0.0005913 0.15%
Gravity::update_max_rhs() 6 0.000489 0.000489 0.000489 0.13%
FabArrayBase::CPC::define() 244 0.0004294 0.0004294 0.0004294 0.11%
Amr::InitAmr() 1 0.0003987 0.0003987 0.0003987 0.10%
FabArrayBase::getFB() 1766 0.0003235 0.0003235 0.0003235 0.08%
Gravity::swapTimeLevels() 5 0.0002446 0.0002446 0.0002446 0.06%
MLLinOp::define() 6 0.000221 0.000221 0.000221 0.06%
MLLinOp::defineGrids() 6 0.0002099 0.0002099 0.0002099 0.05%
Castro::buildMetrics() 1 0.0001622 0.0001622 0.0001622 0.04%
MLMG::MLResNormInf() 6 0.0001545 0.0001545 0.0001545 0.04%
MultiFab::max() 6 0.0001416 0.0001416 0.0001416 0.04%
MLMG::MLRhsNormInf() 6 0.0001197 0.0001197 0.0001197 0.03%
FabArrayBase::FB::FB() 26 6.337e-05 6.337e-05 6.337e-05 0.02%
Castro::create_source_corrector() 5 5.869e-05 5.869e-05 5.869e-05 0.02%
Amr::writeSmallPlotFile() 1 4.181e-05 4.181e-05 4.181e-05 0.01%
MLLinOp::makeAgglomeratedDMap 6 3.454e-05 3.454e-05 3.454e-05 0.01%
Castro::swap_state_time_levels() 5 2.613e-05 2.613e-05 2.613e-05 0.01%
Castro::initMFs() 1 2.399e-05 2.399e-05 2.399e-05 0.01%
Castro::finalize_do_advance() 5 2.148e-05 2.148e-05 2.148e-05 0.01%
makeSFC 30 2.14e-05 2.14e-05 2.14e-05 0.01%
DistributionMapping::Distribute() 31 8.295e-06 8.295e-06 8.295e-06 0.00%
Amr::initSubcycle() 1 8.22e-06 8.22e-06 8.22e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 5.356e-06 5.356e-06 5.356e-06 0.00%
Gravity::set_mass_offset() 6 2.806e-06 2.806e-06 2.806e-06 0.00%
Castro::FluxRegCrseInit 5 1.697e-06 1.697e-06 1.697e-06 0.00%
Castro::retry_advance_ctu() 5 1.685e-06 1.685e-06 1.685e-06 0.00%
Castro::FluxRegFineAdd() 5 1.256e-06 1.256e-06 1.256e-06 0.00%
MLLinOp::makeSubCommunicator() 6 1.068e-06 1.068e-06 1.068e-06 0.00%
AmrLevel::AmrLevel() 1 8.61e-07 8.61e-07 8.61e-07 0.00%
--------------------------------------------------------------------------------------------
Device Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Arena::Initialize() 1 1 223 MiB 9042 MiB
Castro::construct_ctu_hydro_source() 1440 1440 114 MiB 692 MiB
Castro::initMFs() 48 48 60 MiB 68 MiB
StateData::restartDoit() 32 32 53 MiB 55 MiB
Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB
Castro::initialize_do_advance() 40 40 28 MiB 39 MiB
FillPatchIterator::Initialize 160 160 1429 KiB 39 MiB
ResizeRandomSeed 1 1 30 MiB 30 MiB
Amr::writePlotFile() 8 8 2712 KiB 28 MiB
Castro::initialize_advance() 40 40 17 MiB 23 MiB
Castro::buildMetrics() 32 32 14 MiB 15 MiB
Castro::post_restart() 48 48 6698 KiB 14 MiB
MLMG::prepareForSolve() 361 361 3581 KiB 12 MiB
Gravity::get_old_grav_vector() 43 43 295 KiB 10 MiB
Gravity::get_new_grav_vector() 40 40 296 KiB 10 MiB
Gravity::multilevel_solve_for_new_phi() 24 24 6693 KiB 7586 KiB
Gravity::fill_multipole_BCs() 84 84 9229 B 2053 KiB
Gravity::update_max_rhs() 48 48 2392 B 2048 KiB
Gravity::solve_for_phi() 40 40 654 KiB 2048 KiB
Gravity::actual_multilevel_solve() 8 8 18 KiB 2048 KiB
BndryData::define() 576 576 329 KiB 1095 KiB
MLCellLinOp::defineAuxData() 936 936 211 KiB 671 KiB
Castro::estTimeStep() 10 10 2542 B 480 KiB
VisMF::Write(FabArray) 112 112 1320 B 320 KiB
Castro::normalize_species() 30 30 8368 B 320 KiB
amrex::average_down 469 469 1238 B 257 KiB
MLMG::addInterpCorrection() 468 468 1189 B 257 KiB
amrex::Dot() 592 592 3384 B 160 KiB
FabArray::norminf() 398 398 2372 B 160 KiB
Castro::do_advance_ctu() 5 5 517 B 160 KiB
MultiFab::max() 6 6 55 B 160 KiB
FabArray::setVal() 66 66 18 KiB 23 KiB
MultiFab::contains_nan() 10 10 30 B 20 KiB
MLPoisson::Fsmooth() 60 60 3482 B 12 KiB
FabArray::ParallelCopy_nowait() 380 380 50 B 10 KiB
FillBoundary_nowait() 336 336 283 B 9648 B
MLCellLinOp::applyBC() 3892 3892 224 B 9344 B
amrex::Copy() 56 56 5941 B 8464 B
MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B
StateData::FillBoundary(geom) 960 960 41 B 2784 B
FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B
MLCGSolver::bicgstab 324 324 120 B 1472 B
MLCellLinOp::defineBC() 36 36 370 B 1248 B
------------------------------------------------------------------------------
Managed Memory Usage:
------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------
The_Managed_Arena::Initialize() 1 1 1280 B 8192 KiB
------------------------------------------------------------------
Pinned Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Pinned_Arena::Initialize() 1 1 113 KiB 8192 KiB
VisMF::Write(FabArray) 120 120 290 KiB 3584 KiB
VisMF::Read() 24 24 196 KiB 3000 KiB
FabArray::setVal() 66 66 18 KiB 23 KiB
MLPoisson::Fsmooth() 60 60 3482 B 12 KiB
FabArray::ParallelCopy_nowait() 380 380 50 B 10 KiB
FillBoundary_nowait() 336 336 282 B 9648 B
MLCellLinOp::applyBC() 1946 1946 222 B 9328 B
amrex::Copy() 56 56 5941 B 8464 B
MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B
Gravity::get_old_grav_vector() 3 3 2640 B 3072 B
Gravity::fill_multipole_BCs() 18 18 5 B 2832 B
StateData::FillBoundary(geom) 960 960 42 B 2784 B
FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B
MLMG::prepareForSolve() 7 7 536 B 1296 B
amrex::average_down 37 37 222 B 1296 B
MLMG::addInterpCorrection() 36 36 2 B 1024 B
MLCellLinOp::setLevelBC() 36 36 0 B 768 B
amrex::Dot() 592 592 25 B 400 B
FabArray::norminf() 398 398 17 B 272 B
Castro::estTimeStep() 10 10 0 B 32 B
MultiFab::max() 6 6 0 B 16 B
MultiFab::contains_nan() 10 10 0 B 16 B
Castro::do_advance_ctu() 5 5 0 B 16 B
Castro::normalize_species() 30 30 0 B 16 B
------------------------------------------------------------------------------
Cpu (amrex_mempool) Memory Usage:
------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------
amrex_mempool_init() 1 1 126 KiB 8192 KiB
------------------------------------------------------
Unused ParmParse Variables:
[TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2]
[TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2]
Total GPU global memory (MB): 12056
Free GPU global memory (MB): 2459
[The Arena] space allocated (MB): 9042
[The Arena] space used (MB): 0
[The Managed Arena] space allocated (MB): 8
[The Managed Arena] space used (MB): 0
[The Pinned Arena] space allocated (MB): 8
[The Pinned Arena] space used (MB): 0
AMReX (23.02-50-g32a65ffa8aba) finalized