Initializing CUDA...
CUDA initialized with 1 device.
AMReX (23.05-5-g68aa69a11750) initialized
Starting run at 07:40:20 UTC on 2023-05-08.
Successfully read inputs file ...
Castro git describe: 23.05-3-gf4650ffb0
AMReX git describe: 23.05-5-g68aa69a11
Microphysics git describe: 23.05-3-g3a9cad19
reading extern runtime parameters ...
3 Species:
C12 O16 Mg24
Successfully read inputs file ...
INITIAL GRIDS
Level 0 8 grids 262144 cells 100 % of domain
smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32
CHECKPOINT: file = dustcollapse-restart_chk00000
checkPoint() time = 0.057624173 secs.
PLOTFILE: file = dustcollapse-restart_plt00000
Write plotfile time = 0.033111245 seconds
[Level 0 step 1] ADVANCE with dt = 4.541742215e-05
[Level 0 step 1] Advanced 262144 cells
[STEP 1] Coarse TimeStep time: 0.04675864
[STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05
[Level 0 step 2] ADVANCE with dt = 4.768829326e-05
[Level 0 step 2] Advanced 262144 cells
[STEP 2] Coarse TimeStep time: 0.047333237
[STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05
[Level 0 step 3] ADVANCE with dt = 5.007270792e-05
[Level 0 step 3] Advanced 262144 cells
[STEP 3] Coarse TimeStep time: 0.046451304
[STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05
[Level 0 step 4] ADVANCE with dt = 5.257634331e-05
[Level 0 step 4] Advanced 262144 cells
[STEP 4] Coarse TimeStep time: 0.048237582
[STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05
[Level 0 step 5] ADVANCE with dt = 5.520516048e-05
[Level 0 step 5] Advanced 262144 cells
[STEP 5] Coarse TimeStep time: 0.071352653
[STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05
CHECKPOINT: file = dustcollapse-restart_chk00005
checkPoint() time = 0.057685157 secs.
[Level 0 step 6] ADVANCE with dt = 5.79654185e-05
[Level 0 step 6] Advanced 262144 cells
[STEP 6] Coarse TimeStep time: 0.070054673
[STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05
[Level 0 step 7] ADVANCE with dt = 6.086368943e-05
[Level 0 step 7] Advanced 262144 cells
[STEP 7] Coarse TimeStep time: 0.063381112
[STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05
[Level 0 step 8] ADVANCE with dt = 6.39068739e-05
[Level 0 step 8] Advanced 262144 cells
[STEP 8] Coarse TimeStep time: 0.059401556
[STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05
[Level 0 step 9] ADVANCE with dt = 6.71022176e-05
[Level 0 step 9] Advanced 262144 cells
[STEP 9] Coarse TimeStep time: 0.055609851
[STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05
[Level 0 step 10] ADVANCE with dt = 7.045732848e-05
[Level 0 step 10] Advanced 262144 cells
[STEP 10] Coarse TimeStep time: 0.060624714
[STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05
CHECKPOINT: file = dustcollapse-restart_chk00010
checkPoint() time = 0.05781485 secs.
PLOTFILE: file = dustcollapse-restart_plt00010
Write plotfile time = 0.03329302 seconds
Ending run at 07:40:21 UTC on 2023-05-08.
Run time = 0.857747737
Run time without initialization = 0.718628693
Average number of zones advanced per microsecond: 3.648
Average number of zones advanced per microsecond per rank: 3.648
CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472
TinyProfiler total time across processes [min...avg...max]: 0.8578 ... 0.8578 ... 0.8578
--------------------------------------------------------------------------------------------
Name NCalls Excl. Min Excl. Avg Excl. Max Max %
--------------------------------------------------------------------------------------------
VisMF::Write(FabArray) 11 0.2328 0.2328 0.2328 27.14%
Castro::construct_ctu_hydro_source() 10 0.2172 0.2172 0.2172 25.32%
MLCellLinOp::applyBC() 4433 0.07482 0.07482 0.07482 8.72%
FillBoundary_nowait() 4023 0.03149 0.03149 0.03149 3.67%
MLPoisson::Fsmooth() 3280 0.03126 0.03126 0.03126 3.64%
StateData::FillBoundary(geom) 328 0.02412 0.02412 0.02412 2.81%
amrex::Dot() 1114 0.01962 0.01962 0.01962 2.29%
Castro::computeTemp() 63 0.01477 0.01477 0.01477 1.72%
Castro::normalize_species() 62 0.0146 0.0146 0.0146 1.70%
amrex::Copy() 1029 0.01443 0.01443 0.01443 1.68%
FabArray::norminf() 743 0.01392 0.01392 0.01392 1.62%
StateDataPhysBCFunct::() 41 0.01337 0.01337 0.01337 1.56%
FabArray::ParallelCopy_nowait() 861 0.01293 0.01293 0.01293 1.51%
FabArray::setVal() 1144 0.01267 0.01267 0.01267 1.48%
Castro::enforce_min_density() 62 0.01015 0.01015 0.01015 1.18%
MLPoisson::Fapply() 1142 0.009984 0.009984 0.009984 1.16%
MLCellLinOp::defineAuxData() 11 0.009359 0.009359 0.009359 1.09%
FabArray::Saxpy() 813 0.007967 0.007967 0.007967 0.93%
FabArray::Xpay() 821 0.007767 0.007767 0.007767 0.91%
MLMG::addInterpCorrection() 410 0.006457 0.006457 0.006457 0.75%
Gravity::fill_multipole_BCs() 11 0.006398 0.006398 0.006398 0.75%
amrex::average_down 410 0.00577 0.00577 0.00577 0.67%
Castro::estTimeStep() 21 0.005425 0.005425 0.005425 0.63%
FabArray::LinComb() 557 0.004363 0.004363 0.004363 0.51%
amrex::Add() 164 0.004329 0.004329 0.004329 0.50%
Castro::reset_internal_energy(MultiFab) 63 0.004188 0.004188 0.004188 0.49%
Amr::checkPoint() 3 0.003928 0.003928 0.003928 0.46%
BndryData::define() 11 0.003524 0.003524 0.003524 0.41%
Castro::construct_new_gravity_source() 10 0.0033 0.0033 0.0033 0.38%
Castro::construct_old_gravity_source() 10 0.002768 0.002768 0.002768 0.32%
Castro::do_advance_ctu() 10 0.002671 0.002671 0.002671 0.31%
Amr::writePlotFile() 2 0.002354 0.002354 0.002354 0.27%
MLCGSolver::bicgstab 82 0.002194 0.002194 0.002194 0.26%
FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001642 0.001642 0.001642 0.19%
Gravity::actual_solve_with_mlmg() 11 0.001508 0.001508 0.001508 0.18%
Castro::reset_internal_energy(Fab) 504 0.00148 0.00148 0.00148 0.17%
MLCellLinOp::setLevelBC() 11 0.001347 0.001347 0.001347 0.16%
FabArray::mult() 43 0.001304 0.001304 0.001304 0.15%
FabArray::setDomainBndry() 41 0.00129 0.00129 0.00129 0.15%
Castro::initData() 1 0.001232 0.001232 0.001232 0.14%
MultiFab::contains_nan() 20 0.001191 0.001191 0.001191 0.14%
MLCellLinOp::prepareForSolve() 11 0.001102 0.001102 0.001102 0.13%
Castro::enforce_speed_limit() 62 0.001085 0.001085 0.001085 0.13%
MLCellLinOp::smooth() 1640 0.001018 0.001018 0.001018 0.12%
MLCellLinOp::compGrad() 11 0.0008845 0.0008845 0.0008845 0.10%
MLMG::prepareForSolve() 11 0.0008834 0.0008834 0.0008834 0.10%
FabArray::FillBoundary() 4023 0.0007909 0.0007909 0.0007909 0.09%
FabArrayBase::getCPC() 1323 0.0007333 0.0007333 0.0007333 0.09%
FabArrayBase::CPC::define() 454 0.0006697 0.0006697 0.0006697 0.08%
Gravity::get_new_grav_vector() 11 0.0006371 0.0006371 0.0006371 0.07%
FabArrayBase::getFB() 4023 0.0005847 0.0005847 0.0005847 0.07%
Gravity::get_old_grav_vector() 10 0.0005412 0.0005412 0.0005412 0.06%
Amr::InitAmr() 1 0.0004543 0.0004543 0.0004543 0.05%
MLCellLinOp::apply() 1142 0.0004364 0.0004364 0.0004364 0.05%
MLLinOp::defineGrids() 11 0.0003574 0.0003574 0.0003574 0.04%
Amr::coarseTimeStep() 10 0.0003503 0.0003503 0.0003503 0.04%
MLMG::mgVcycle() 82 0.0003454 0.0003454 0.0003454 0.04%
main() 1 0.0002885 0.0002885 0.0002885 0.03%
MLCGSolver::ParallelAllReduce 1514 0.0002835 0.0002835 0.0002835 0.03%
MultiFab::max() 11 0.0002529 0.0002529 0.0002529 0.03%
FabArray::ParallelCopy() 861 0.0002374 0.0002374 0.0002374 0.03%
FillPatchIterator::Initialize 41 0.000226 0.000226 0.000226 0.03%
MLCellLinOp::correctionResidual() 492 0.0002224 0.0002224 0.0002224 0.03%
MLCellLinOp::defineBC() 11 0.000218 0.000218 0.000218 0.03%
Castro::subcycle_advance_ctu() 10 0.0001906 0.0001906 0.0001906 0.02%
Amr::timeStep() 10 0.0001583 0.0001583 0.0001583 0.02%
StateData::checkPoint() 12 0.0001376 0.0001376 0.0001376 0.02%
Castro::create_source_corrector() 10 0.0001249 0.0001249 0.0001249 0.01%
Gravity::update_max_rhs() 11 0.0001232 0.0001232 0.0001232 0.01%
MLMG:computeResOfCorrection() 410 0.0001126 0.0001126 0.0001126 0.01%
Gravity::solve_for_phi() 10 0.0001079 0.0001079 0.0001079 0.01%
MLMG::mgVcycle_down::0 82 8.711e-05 8.711e-05 8.711e-05 0.01%
MLMG::actualBottomSolve() 82 8.592e-05 8.592e-05 8.592e-05 0.01%
Castro::Castro() 1 8.497e-05 8.497e-05 8.497e-05 0.01%
FabArrayBase::FB::FB() 56 8.493e-05 8.493e-05 8.493e-05 0.01%
MLMG::mgVcycle_down::1 82 7.822e-05 7.822e-05 7.822e-05 0.01%
MLMG::mgVcycle_down::2 82 7.764e-05 7.764e-05 7.764e-05 0.01%
Castro::expand_state() 10 7.75e-05 7.75e-05 7.75e-05 0.01%
AmrLevel::checkPoint() 3 7.651e-05 7.651e-05 7.651e-05 0.01%
Castro::initialize_advance() 10 7.442e-05 7.442e-05 7.442e-05 0.01%
MLMG::mgVcycle_down::3 82 7.347e-05 7.347e-05 7.347e-05 0.01%
Castro::clean_state() 62 7.346e-05 7.346e-05 7.346e-05 0.01%
MLMG::mgVcycle_down::4 82 7.342e-05 7.342e-05 7.342e-05 0.01%
MLMG::solve() 11 7.026e-05 7.026e-05 7.026e-05 0.01%
MLMG::mgVcycle_up::4 82 5.998e-05 5.998e-05 5.998e-05 0.01%
MLMG::oneIter() 82 5.911e-05 5.911e-05 5.911e-05 0.01%
Castro::finalize_advance() 10 5.827e-05 5.827e-05 5.827e-05 0.01%
Castro::initialize_do_advance() 10 5.303e-05 5.303e-05 5.303e-05 0.01%
MLMG::mgVcycle_up::2 82 5.166e-05 5.166e-05 5.166e-05 0.01%
MLCellLinOp::solutionResidual() 93 4.911e-05 4.911e-05 4.911e-05 0.01%
MLMG::mgVcycle_up::1 82 4.873e-05 4.873e-05 4.873e-05 0.01%
MLMG::mgVcycle_up::3 82 4.852e-05 4.852e-05 4.852e-05 0.01%
MLMG::mgVcycle_up::0 82 4.707e-05 4.707e-05 4.707e-05 0.01%
Castro::advance() 10 4.143e-05 4.143e-05 4.143e-05 0.00%
Castro::construct_new_source() 50 3.922e-05 3.922e-05 3.922e-05 0.00%
Castro::swap_state_time_levels() 10 3.777e-05 3.777e-05 3.777e-05 0.00%
Castro::finalize_do_advance() 10 3.547e-05 3.547e-05 3.547e-05 0.00%
StateData::define() 4 3.496e-05 3.496e-05 3.496e-05 0.00%
MLMG::ResNormInf() 93 3.479e-05 3.479e-05 3.479e-05 0.00%
Castro::post_timestep() 10 3.468e-05 3.468e-05 3.468e-05 0.00%
Castro::enforce_consistent_e() 1 3.245e-05 3.245e-05 3.245e-05 0.00%
MLMG::mgVcycle_bottom 82 3.026e-05 3.026e-05 3.026e-05 0.00%
FillPatchSingleLevel 41 2.961e-05 2.961e-05 2.961e-05 0.00%
MLMG::computeResidual() 82 2.899e-05 2.899e-05 2.899e-05 0.00%
Castro::construct_new_gravity() 10 2.815e-05 2.815e-05 2.815e-05 0.00%
Amr::writeSmallPlotFile() 1 2.559e-05 2.559e-05 2.559e-05 0.00%
Castro::initMFs() 1 2.455e-05 2.455e-05 2.455e-05 0.00%
makeSFC 55 2.42e-05 2.42e-05 2.42e-05 0.00%
Amr::FinalizeInit() 1 2.179e-05 2.179e-05 2.179e-05 0.00%
MLPoisson::define() 11 2.178e-05 2.178e-05 2.178e-05 0.00%
Gravity::solve_phi_with_mlmg() 11 2.161e-05 2.161e-05 2.161e-05 0.00%
Amr::defBaseLevel() 1 2.017e-05 2.017e-05 2.017e-05 0.00%
Castro::construct_old_source() 50 1.901e-05 1.901e-05 1.901e-05 0.00%
Castro::buildMetrics() 1 1.777e-05 1.777e-05 1.777e-05 0.00%
Gravity::multilevel_solve_for_new_phi() 1 1.7e-05 1.7e-05 1.7e-05 0.00%
Castro::do_new_sources() 10 1.652e-05 1.652e-05 1.652e-05 0.00%
Castro::do_old_sources() 10 1.566e-05 1.566e-05 1.566e-05 0.00%
MLPoisson::prepareForSolve() 11 1.502e-05 1.502e-05 1.502e-05 0.00%
DistributionMapping::Distribute() 56 1.481e-05 1.481e-05 1.481e-05 0.00%
Castro::apply_source_to_state() 20 1.453e-05 1.453e-05 1.453e-05 0.00%
MLMG::computeMLResidual() 11 1.42e-05 1.42e-05 1.42e-05 0.00%
MLLinOp::makeAgglomeratedDMap 11 1.364e-05 1.364e-05 1.364e-05 0.00%
Castro::check_for_nan() 20 1.072e-05 1.072e-05 1.072e-05 0.00%
AmrLevel::AmrLevel(dm) 1 1.028e-05 1.028e-05 1.028e-05 0.00%
MLLinOp::define() 11 1.015e-05 1.015e-05 1.015e-05 0.00%
Castro::construct_old_gravity() 10 9.064e-06 9.064e-06 9.064e-06 0.00%
Gravity::actual_multilevel_solve() 1 8.773e-06 8.773e-06 8.773e-06 0.00%
Amr::initSubcycle() 1 8.754e-06 8.754e-06 8.754e-06 0.00%
Gravity::swapTimeLevels() 10 8.626e-06 8.626e-06 8.626e-06 0.00%
Castro::computeNewDt() 9 6.743e-06 6.743e-06 6.743e-06 0.00%
MLMG::getGradSolution() 11 6.208e-06 6.208e-06 6.208e-06 0.00%
AmrLevel::checkPointPost() 3 5.659e-06 5.659e-06 5.659e-06 0.00%
Amr::InitializeInit() 1 4.948e-06 4.948e-06 4.948e-06 0.00%
Gravity::set_mass_offset() 11 4.358e-06 4.358e-06 4.358e-06 0.00%
MLMG::MLRhsNormInf() 11 3.878e-06 3.878e-06 3.878e-06 0.00%
MLMG::MLResNormInf() 11 3.682e-06 3.682e-06 3.682e-06 0.00%
Castro::retry_advance_ctu() 10 3.648e-06 3.648e-06 3.648e-06 0.00%
Castro::post_init() 1 3.542e-06 3.542e-06 3.542e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 3.188e-06 3.188e-06 3.188e-06 0.00%
Castro::computeInitialDt() 2 2.688e-06 2.688e-06 2.688e-06 0.00%
Castro::FluxRegCrseInit 10 2.505e-06 2.505e-06 2.505e-06 0.00%
Amr::init() 1 2.346e-06 2.346e-06 2.346e-06 0.00%
MLLinOp::makeSubCommunicator() 11 1.933e-06 1.933e-06 1.933e-06 0.00%
Castro::FluxRegFineAdd() 10 1.771e-06 1.771e-06 1.771e-06 0.00%
AmrLevel::checkPointPre() 3 1.573e-06 1.573e-06 1.573e-06 0.00%
Castro::post_regrid() 1 1.357e-06 1.357e-06 1.357e-06 0.00%
Amr::initialInit() 1 1.069e-06 1.069e-06 1.069e-06 0.00%
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Name NCalls Incl. Min Incl. Avg Incl. Max Max %
--------------------------------------------------------------------------------------------
main() 1 0.8578 0.8578 0.8578 100.00%
Amr::coarseTimeStep() 10 0.6851 0.6851 0.6851 79.87%
Amr::timeStep() 10 0.5657 0.5657 0.5657 65.95%
Castro::advance() 10 0.5584 0.5584 0.5584 65.10%
Castro::subcycle_advance_ctu() 10 0.5475 0.5475 0.5475 63.83%
Castro::do_advance_ctu() 10 0.5473 0.5473 0.5473 63.80%
Gravity::solve_phi_with_mlmg() 11 0.2758 0.2758 0.2758 32.15%
Gravity::actual_solve_with_mlmg() 11 0.2689 0.2689 0.2689 31.35%
Castro::construct_new_gravity() 10 0.2519 0.2519 0.2519 29.37%
MLMG::solve() 11 0.2489 0.2489 0.2489 29.02%
Gravity::solve_for_phi() 10 0.2366 0.2366 0.2366 27.58%
MLMG::oneIter() 82 0.235 0.235 0.235 27.39%
VisMF::Write(FabArray) 11 0.2328 0.2328 0.2328 27.14%
MLMG::mgVcycle() 82 0.2313 0.2313 0.2313 26.97%
Castro::construct_ctu_hydro_source() 10 0.2172 0.2172 0.2172 25.32%
Amr::checkPoint() 3 0.1733 0.1733 0.1733 20.20%
AmrLevel::checkPoint() 3 0.1693 0.1693 0.1693 19.74%
StateData::checkPoint() 12 0.1693 0.1693 0.1693 19.73%
Amr::init() 1 0.1385 0.1385 0.1385 16.15%
MLCellLinOp::smooth() 1640 0.1148 0.1148 0.1148 13.38%
MLCellLinOp::applyBC() 4433 0.1078 0.1078 0.1078 12.56%
MLMG::mgVcycle_bottom 82 0.07086 0.07086 0.07086 8.26%
MLMG::actualBottomSolve() 82 0.07083 0.07083 0.07083 8.26%
MLCGSolver::bicgstab 82 0.07015 0.07015 0.07015 8.18%
Amr::writePlotFile() 2 0.06653 0.06653 0.06653 7.76%
Amr::initialInit() 1 0.04765 0.04765 0.04765 5.55%
Castro::clean_state() 62 0.04557 0.04557 0.04557 5.31%
Amr::FinalizeInit() 1 0.04352 0.04352 0.04352 5.07%
FillPatchIterator::Initialize 41 0.04306 0.04306 0.04306 5.02%
Castro::post_init() 1 0.04211 0.04211 0.04211 4.91%
FillPatchSingleLevel 41 0.04154 0.04154 0.04154 4.84%
Gravity::multilevel_solve_for_new_phi() 1 0.03962 0.03962 0.03962 4.62%
Gravity::actual_multilevel_solve() 1 0.0396 0.0396 0.0396 4.62%
StateDataPhysBCFunct::() 41 0.03749 0.03749 0.03749 4.37%
MLCellLinOp::apply() 1142 0.03525 0.03525 0.03525 4.11%
MLMG::mgVcycle_down::0 82 0.03363 0.03363 0.03363 3.92%
FabArray::FillBoundary() 4023 0.03295 0.03295 0.03295 3.84%
FillBoundary_nowait() 4023 0.03216 0.03216 0.03216 3.75%
MLPoisson::Fsmooth() 3280 0.03126 0.03126 0.03126 3.64%
MLMG::mgVcycle_up::0 82 0.02548 0.02548 0.02548 2.97%
StateData::FillBoundary(geom) 328 0.02412 0.02412 0.02412 2.81%
MLCellLinOp::correctionResidual() 492 0.02164 0.02164 0.02164 2.52%
Castro::computeTemp() 63 0.02044 0.02044 0.02044 2.38%
Castro::initialize_do_advance() 10 0.01963 0.01963 0.01963 2.29%
amrex::Dot() 1114 0.01962 0.01962 0.01962 2.29%
MLMG:computeResOfCorrection() 410 0.01909 0.01909 0.01909 2.23%
Gravity::get_new_grav_vector() 11 0.01755 0.01755 0.01755 2.05%
MLPoisson::define() 11 0.01579 0.01579 0.01579 1.84%
MLMG::mgVcycle_down::1 82 0.01527 0.01527 0.01527 1.78%
Castro::construct_old_gravity() 10 0.01507 0.01507 0.01507 1.76%
Gravity::get_old_grav_vector() 10 0.01506 0.01506 0.01506 1.76%
Castro::normalize_species() 62 0.0146 0.0146 0.0146 1.70%
amrex::Copy() 1029 0.01443 0.01443 0.01443 1.68%
MLMG::mgVcycle_down::2 82 0.01425 0.01425 0.01425 1.66%
FabArray::ParallelCopy() 861 0.01397 0.01397 0.01397 1.63%
FabArray::norminf() 743 0.01392 0.01392 0.01392 1.62%
MLMG::mgVcycle_down::3 82 0.01392 0.01392 0.01392 1.62%
MLMG::mgVcycle_down::4 82 0.01383 0.01383 0.01383 1.61%
FabArray::ParallelCopy_nowait() 861 0.01374 0.01374 0.01374 1.60%
Castro::do_new_sources() 10 0.01369 0.01369 0.01369 1.60%
FabArray::setVal() 1144 0.01267 0.01267 0.01267 1.48%
MLCGSolver::ParallelAllReduce 1514 0.01179 0.01179 0.01179 1.37%
Castro::expand_state() 10 0.0116 0.0116 0.0116 1.35%
MLMG::addInterpCorrection() 410 0.01142 0.01142 0.01142 1.33%
MLMG::mgVcycle_up::1 82 0.01116 0.01116 0.01116 1.30%
MLMG::mgVcycle_up::4 82 0.01105 0.01105 0.01105 1.29%
MLMG::mgVcycle_up::2 82 0.01088 0.01088 0.01088 1.27%
Castro::do_old_sources() 10 0.01077 0.01077 0.01077 1.26%
amrex::average_down 410 0.01076 0.01076 0.01076 1.25%
MLMG::mgVcycle_up::3 82 0.01065 0.01065 0.01065 1.24%
MLCellLinOp::defineAuxData() 11 0.01065 0.01065 0.01065 1.24%
Castro::initialize_advance() 10 0.01032 0.01032 0.01032 1.20%
Castro::enforce_min_density() 62 0.01015 0.01015 0.01015 1.18%
MLPoisson::Fapply() 1142 0.009984 0.009984 0.009984 1.16%
FabArray::Saxpy() 813 0.007967 0.007967 0.007967 0.93%
FabArray::Xpay() 821 0.007767 0.007767 0.007767 0.91%
Castro::post_timestep() 10 0.007109 0.007109 0.007109 0.83%
MLCellLinOp::solutionResidual() 93 0.007022 0.007022 0.007022 0.82%
Gravity::fill_multipole_BCs() 11 0.006633 0.006633 0.006633 0.77%
MLMG::computeResidual() 82 0.006068 0.006068 0.006068 0.71%
Castro::reset_internal_energy(MultiFab) 63 0.005668 0.005668 0.005668 0.66%
Castro::estTimeStep() 21 0.005425 0.005425 0.005425 0.63%
MLCellLinOp::defineBC() 11 0.004694 0.004694 0.004694 0.55%
MLMG::prepareForSolve() 11 0.004543 0.004543 0.004543 0.53%
BndryData::define() 11 0.004476 0.004476 0.004476 0.52%
FabArray::LinComb() 557 0.004363 0.004363 0.004363 0.51%
amrex::Add() 164 0.004329 0.004329 0.004329 0.50%
Amr::InitializeInit() 1 0.004127 0.004127 0.004127 0.48%
Amr::defBaseLevel() 1 0.004122 0.004122 0.004122 0.48%
Castro::initData() 1 0.003625 0.003625 0.003625 0.42%
Castro::construct_new_source() 50 0.003339 0.003339 0.003339 0.39%
Castro::construct_new_gravity_source() 10 0.0033 0.0033 0.0033 0.38%
Castro::computeNewDt() 9 0.002805 0.002805 0.002805 0.33%
Castro::construct_old_source() 50 0.002787 0.002787 0.002787 0.32%
Castro::construct_old_gravity_source() 10 0.002768 0.002768 0.002768 0.32%
MLMG::ResNormInf() 93 0.002062 0.002062 0.002062 0.24%
Castro::apply_source_to_state() 20 0.00182 0.00182 0.00182 0.21%
FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001642 0.001642 0.001642 0.19%
Castro::reset_internal_energy(Fab) 504 0.00148 0.00148 0.00148 0.17%
FabArrayBase::getCPC() 1323 0.001403 0.001403 0.001403 0.16%
MLMG::getGradSolution() 11 0.001355 0.001355 0.001355 0.16%
MLCellLinOp::compGrad() 11 0.001349 0.001349 0.001349 0.16%
MLCellLinOp::setLevelBC() 11 0.001347 0.001347 0.001347 0.16%
FabArray::mult() 43 0.001304 0.001304 0.001304 0.15%
FabArray::setDomainBndry() 41 0.00129 0.00129 0.00129 0.15%
Castro::check_for_nan() 20 0.001202 0.001202 0.001202 0.14%
MultiFab::contains_nan() 20 0.001191 0.001191 0.001191 0.14%
Castro::post_regrid() 1 0.001168 0.001168 0.001168 0.14%
MLPoisson::prepareForSolve() 11 0.001117 0.001117 0.001117 0.13%
MLCellLinOp::prepareForSolve() 11 0.001102 0.001102 0.001102 0.13%
Castro::enforce_speed_limit() 62 0.001085 0.001085 0.001085 0.13%
MLMG::computeMLResidual() 11 0.0009973 0.0009973 0.0009973 0.12%
Castro::computeInitialDt() 2 0.0009231 0.0009231 0.0009231 0.11%
Gravity::update_max_rhs() 11 0.0008076 0.0008076 0.0008076 0.09%
FabArrayBase::CPC::define() 454 0.0006697 0.0006697 0.0006697 0.08%
FabArrayBase::getFB() 4023 0.0006696 0.0006696 0.0006696 0.08%
Castro::finalize_advance() 10 0.0005633 0.0005633 0.0005633 0.07%
Amr::InitAmr() 1 0.000463 0.000463 0.000463 0.05%
Castro::Castro() 1 0.0004271 0.0004271 0.0004271 0.05%
Gravity::swapTimeLevels() 10 0.0004219 0.0004219 0.0004219 0.05%
MLLinOp::define() 11 0.000421 0.000421 0.000421 0.05%
MLLinOp::defineGrids() 11 0.0004108 0.0004108 0.0004108 0.05%
MLMG::MLResNormInf() 11 0.0002831 0.0002831 0.0002831 0.03%
MultiFab::max() 11 0.0002529 0.0002529 0.0002529 0.03%
MLMG::MLRhsNormInf() 11 0.0002151 0.0002151 0.0002151 0.03%
Castro::buildMetrics() 1 0.0001589 0.0001589 0.0001589 0.02%
Castro::create_source_corrector() 10 0.0001249 0.0001249 0.0001249 0.01%
FabArrayBase::FB::FB() 56 8.493e-05 8.493e-05 8.493e-05 0.01%
MLLinOp::makeAgglomeratedDMap 11 5.146e-05 5.146e-05 5.146e-05 0.01%
AmrLevel::AmrLevel(dm) 1 4.524e-05 4.524e-05 4.524e-05 0.01%
makeSFC 55 3.781e-05 3.781e-05 3.781e-05 0.00%
Castro::swap_state_time_levels() 10 3.777e-05 3.777e-05 3.777e-05 0.00%
Castro::finalize_do_advance() 10 3.547e-05 3.547e-05 3.547e-05 0.00%
StateData::define() 4 3.496e-05 3.496e-05 3.496e-05 0.00%
Castro::enforce_consistent_e() 1 3.245e-05 3.245e-05 3.245e-05 0.00%
Amr::writeSmallPlotFile() 1 2.559e-05 2.559e-05 2.559e-05 0.00%
Castro::initMFs() 1 2.455e-05 2.455e-05 2.455e-05 0.00%
DistributionMapping::Distribute() 56 1.481e-05 1.481e-05 1.481e-05 0.00%
Amr::initSubcycle() 1 8.754e-06 8.754e-06 8.754e-06 0.00%
AmrLevel::checkPointPost() 3 5.659e-06 5.659e-06 5.659e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 4.379e-06 4.379e-06 4.379e-06 0.00%
Gravity::set_mass_offset() 11 4.358e-06 4.358e-06 4.358e-06 0.00%
Castro::retry_advance_ctu() 10 3.648e-06 3.648e-06 3.648e-06 0.00%
Castro::FluxRegCrseInit 10 2.505e-06 2.505e-06 2.505e-06 0.00%
MLLinOp::makeSubCommunicator() 11 1.933e-06 1.933e-06 1.933e-06 0.00%
Castro::FluxRegFineAdd() 10 1.771e-06 1.771e-06 1.771e-06 0.00%
AmrLevel::checkPointPre() 3 1.573e-06 1.573e-06 1.573e-06 0.00%
--------------------------------------------------------------------------------------------
Unused ParmParse Variables:
[TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2]
[TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2]
Device Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Arena::Initialize() 1 1 90 MiB 9042 MiB
Castro::construct_ctu_hydro_source() 2880 2880 120 MiB 692 MiB
Castro::initMFs() 48 48 67 MiB 68 MiB
Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB
StateData::define() 32 32 55 MiB 55 MiB
FillPatchIterator::Initialize 328 328 1070 KiB 39 MiB
Castro::initialize_do_advance() 80 80 24 MiB 39 MiB
ResizeRandomSeed 1 1 30 MiB 30 MiB
Amr::writePlotFile() 16 16 2180 KiB 28 MiB
Castro::initialize_advance() 80 80 15 MiB 23 MiB
Castro::buildMetrics() 32 32 15 MiB 15 MiB
Castro::Castro() 48 48 7614 KiB 14 MiB
MLMG::prepareForSolve() 660 660 3570 KiB 12 MiB
Gravity::get_new_grav_vector() 91 91 212 KiB 10 MiB
Gravity::get_old_grav_vector() 80 80 179 KiB 10 MiB
Gravity::multilevel_solve_for_new_phi() 24 24 7523 KiB 7586 KiB
Gravity::fill_multipole_BCs() 154 154 12 KiB 2053 KiB
Gravity::update_max_rhs() 88 88 1794 B 2048 KiB
Gravity::solve_for_phi() 80 80 564 KiB 2048 KiB
Gravity::actual_multilevel_solve() 8 8 94 KiB 2048 KiB
BndryData::define() 1056 1056 325 KiB 1095 KiB
MLCellLinOp::defineAuxData() 1716 1716 207 KiB 671 KiB
Castro::estTimeStep() 21 21 2909 B 480 KiB
VisMF::Write(FabArray) 656 656 3426 B 320 KiB
Castro::normalize_species() 62 62 5540 B 320 KiB
amrex::average_down 1067 1067 1253 B 257 KiB
MLMG::addInterpCorrection() 1066 1066 1145 B 257 KiB
amrex::Dot() 1360 1360 3374 B 160 KiB
FabArray::norminf() 907 907 2404 B 160 KiB
Castro::do_advance_ctu() 10 10 463 B 160 KiB
MultiFab::max() 11 11 45 B 160 KiB
Castro::initData() 1 1 26 B 160 KiB
FabArray::setVal() 106 106 21 KiB 26 KiB
MultiFab::contains_nan() 20 20 27 B 20 KiB
MLPoisson::Fsmooth() 132 132 3491 B 12 KiB
FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB
FillBoundary_nowait() 760 760 322 B 9648 B
MLCellLinOp::applyBC() 8866 8866 224 B 9344 B
MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B
amrex::Copy() 100 100 3847 B 6144 B
StateData::FillBoundary(geom) 1992 1992 53 B 3520 B
FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B
MLCGSolver::bicgstab 738 738 119 B 1472 B
MLCellLinOp::defineBC() 66 66 367 B 1248 B
------------------------------------------------------------------------------
Managed Memory Usage:
-----------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
-----------------------------------------------------------------
The_Managed_Arena::Initialize() 1 1 613 B 8192 KiB
-----------------------------------------------------------------
Pinned Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Pinned_Arena::Initialize() 1 1 51 KiB 8192 KiB
VisMF::Write(FabArray) 744 744 628 KiB 3584 KiB
FabArray::setVal() 106 106 21 KiB 26 KiB
MLPoisson::Fsmooth() 132 132 3491 B 12 KiB
FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB
FillBoundary_nowait() 760 760 322 B 9648 B
MLCellLinOp::applyBC() 4433 4433 223 B 9328 B
MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B
amrex::Copy() 100 100 3847 B 6144 B
StateData::FillBoundary(geom) 1992 1992 54 B 3520 B
Gravity::get_new_grav_vector() 3 3 2904 B 3072 B
Gravity::fill_multipole_BCs() 33 33 4 B 2832 B
FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B
amrex::average_down 83 83 272 B 1296 B
MLMG::addInterpCorrection() 82 82 2 B 1024 B
MLMG::prepareForSolve() 11 11 297 B 1024 B
MLCellLinOp::setLevelBC() 66 66 0 B 768 B
amrex::Dot() 1360 1360 24 B 400 B
FabArray::norminf() 907 907 17 B 272 B
Castro::estTimeStep() 21 21 0 B 32 B
MultiFab::contains_nan() 20 20 0 B 16 B
MultiFab::max() 11 11 0 B 16 B
Castro::normalize_species() 62 62 0 B 16 B
Castro::initData() 1 1 0 B 16 B
Castro::do_advance_ctu() 10 10 0 B 16 B
------------------------------------------------------------------------------
Total GPU global memory (MB): 12056
Free GPU global memory (MB): 2424
[The Arena] space allocated (MB): 9042
[The Arena] space used (MB): 0
[The Managed Arena] space allocated (MB): 8
[The Managed Arena] space used (MB): 0
[The Pinned Arena] space allocated (MB): 8
[The Pinned Arena] space used (MB): 0
AMReX (23.05-5-g68aa69a11750) finalized
Initializing CUDA...
CUDA initialized with 1 device.
AMReX (23.05-5-g68aa69a11750) initialized
Starting run at 07:40:22 UTC on 2023-05-08.
Successfully read inputs file ...
Castro git describe: 23.05-3-gf4650ffb0
AMReX git describe: 23.05-5-g68aa69a11
Microphysics git describe: 23.05-3-g3a9cad19
reading extern runtime parameters ...
3 Species:
C12 O16 Mg24
restarting calculation from file: dustcollapse-restart_chk00005
Successfully read inputs file ...
read CPU time: 0.456937269
Restart time = 0.045342051 seconds.
[Level 0 step 6] ADVANCE with dt = 5.79654185e-05
[Level 0 step 6] Advanced 262144 cells
[STEP 6] Coarse TimeStep time: 0.049104683
[STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05
[Level 0 step 7] ADVANCE with dt = 6.086368943e-05
[Level 0 step 7] Advanced 262144 cells
[STEP 7] Coarse TimeStep time: 0.047586171
[STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05
[Level 0 step 8] ADVANCE with dt = 6.39068739e-05
[Level 0 step 8] Advanced 262144 cells
[STEP 8] Coarse TimeStep time: 0.046743164
[STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05
[Level 0 step 9] ADVANCE with dt = 6.71022176e-05
[Level 0 step 9] Advanced 262144 cells
[STEP 9] Coarse TimeStep time: 0.064126611
[STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05
[Level 0 step 10] ADVANCE with dt = 7.045732848e-05
[Level 0 step 10] Advanced 262144 cells
[STEP 10] Coarse TimeStep time: 0.070414062
[STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05
PLOTFILE: file = dustcollapse-restart_plt00010
Write plotfile time = 0.03285914 seconds
Ending run at 07:40:22 UTC on 2023-05-08.
Run time = 0.357149275
Run time without initialization = 0.311237932
Average number of zones advanced per microsecond: 4.211
Average number of zones advanced per microsecond per rank: 4.211
CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472
TinyProfiler total time across processes [min...avg...max]: 0.3572 ... 0.3572 ... 0.3572
--------------------------------------------------------------------------------------------
Name NCalls Excl. Min Excl. Avg Excl. Max Max %
--------------------------------------------------------------------------------------------
Castro::construct_ctu_hydro_source() 5 0.0995 0.0995 0.0995 27.87%
VisMF::Read() 3 0.03906 0.03906 0.03906 10.94%
MLCellLinOp::applyBC() 1946 0.03267 0.03267 0.03267 9.15%
VisMF::Write(FabArray) 1 0.03134 0.03134 0.03134 8.78%
MLPoisson::Fsmooth() 1440 0.01353 0.01353 0.01353 3.79%
FillBoundary_nowait() 1766 0.01319 0.01319 0.01319 3.69%
StateData::FillBoundary(geom) 160 0.01159 0.01159 0.01159 3.25%
amrex::Dot() 484 0.008416 0.008416 0.008416 2.36%
Castro::normalize_species() 30 0.00715 0.00715 0.00715 2.00%
amrex::Copy() 463 0.006787 0.006787 0.006787 1.90%
FabArray::setVal() 537 0.00604 0.00604 0.00604 1.69%
FabArray::norminf() 326 0.005996 0.005996 0.005996 1.68%
FabArray::ParallelCopy_nowait() 380 0.005873 0.005873 0.005873 1.64%
StateDataPhysBCFunct::() 20 0.005864 0.005864 0.005864 1.64%
Castro::enforce_min_density() 30 0.005706 0.005706 0.005706 1.60%
Castro::computeTemp() 30 0.005449 0.005449 0.005449 1.53%
MLCellLinOp::defineAuxData() 6 0.004997 0.004997 0.004997 1.40%
MLPoisson::Fapply() 500 0.004285 0.004285 0.004285 1.20%
FabArray::Saxpy() 355 0.003513 0.003513 0.003513 0.98%
Gravity::fill_multipole_BCs() 6 0.003376 0.003376 0.003376 0.95%
FabArray::Xpay() 361 0.003371 0.003371 0.003371 0.94%
Castro::reset_internal_energy(MultiFab) 30 0.003012 0.003012 0.003012 0.84%
Castro::estTimeStep() 10 0.002909 0.002909 0.002909 0.81%
MLMG::addInterpCorrection() 180 0.0028 0.0028 0.0028 0.78%
Amr::restart() 1 0.002591 0.002591 0.002591 0.73%
amrex::average_down 180 0.002544 0.002544 0.002544 0.71%
BndryData::define() 6 0.001906 0.001906 0.001906 0.53%
FabArray::LinComb() 242 0.001851 0.001851 0.001851 0.52%
amrex::Add() 72 0.001838 0.001838 0.001838 0.51%
Castro::construct_new_gravity_source() 5 0.001705 0.001705 0.001705 0.48%
Castro::construct_old_gravity_source() 5 0.00139 0.00139 0.00139 0.39%
Amr::writePlotFile() 1 0.001354 0.001354 0.001354 0.38%
Castro::do_advance_ctu() 5 0.001334 0.001334 0.001334 0.37%
MLCGSolver::bicgstab 36 0.0009297 0.0009297 0.0009297 0.26%
FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008789 0.0008789 0.0008789 0.25%
Gravity::actual_solve_with_mlmg() 6 0.0007827 0.0007827 0.0007827 0.22%
MLCellLinOp::setLevelBC() 6 0.0007296 0.0007296 0.0007296 0.20%
Castro::enforce_speed_limit() 30 0.0006956 0.0006956 0.0006956 0.19%
Castro::reset_internal_energy(Fab) 240 0.0006953 0.0006953 0.0006953 0.19%
FabArray::mult() 22 0.0006423 0.0006423 0.0006423 0.18%
FabArray::setDomainBndry() 20 0.000623 0.000623 0.000623 0.17%
MultiFab::contains_nan() 10 0.0006 0.0006 0.0006 0.17%
MLCellLinOp::prepareForSolve() 6 0.0005958 0.0005958 0.0005958 0.17%
MLCellLinOp::compGrad() 6 0.0004804 0.0004804 0.0004804 0.13%
MLMG::prepareForSolve() 6 0.0004758 0.0004758 0.0004758 0.13%
MLCellLinOp::smooth() 720 0.0004339 0.0004339 0.0004339 0.12%
FabArrayBase::CPC::define() 244 0.0004124 0.0004124 0.0004124 0.12%
Amr::InitAmr() 1 0.000381 0.000381 0.000381 0.11%
FabArray::FillBoundary() 1766 0.0003458 0.0003458 0.0003458 0.10%
FabArrayBase::getCPC() 632 0.0003406 0.0003406 0.0003406 0.10%
Gravity::get_old_grav_vector() 5 0.0002962 0.0002962 0.0002962 0.08%
main() 1 0.0002887 0.0002887 0.0002887 0.08%
Gravity::get_new_grav_vector() 5 0.0002712 0.0002712 0.0002712 0.08%
FabArrayBase::getFB() 1766 0.0002513 0.0002513 0.0002513 0.07%
MLCellLinOp::apply() 500 0.0001911 0.0001911 0.0001911 0.05%
MLLinOp::defineGrids() 6 0.0001715 0.0001715 0.0001715 0.05%
MLMG::mgVcycle() 36 0.0001623 0.0001623 0.0001623 0.05%
Amr::coarseTimeStep() 5 0.0001593 0.0001593 0.0001593 0.04%
MultiFab::max() 6 0.0001348 0.0001348 0.0001348 0.04%
Castro::create_source_corrector() 5 0.0001207 0.0001207 0.0001207 0.03%
MLCGSolver::ParallelAllReduce 659 0.0001207 0.0001207 0.0001207 0.03%
MLCellLinOp::defineBC() 6 0.0001166 0.0001166 0.0001166 0.03%
FillPatchIterator::Initialize 20 0.000113 0.000113 0.000113 0.03%
FabArray::ParallelCopy() 380 0.0001037 0.0001037 0.0001037 0.03%
MLCellLinOp::correctionResidual() 216 9.191e-05 9.191e-05 9.191e-05 0.03%
Amr::timeStep() 5 8.182e-05 8.182e-05 8.182e-05 0.02%
Castro::subcycle_advance_ctu() 5 8.093e-05 8.093e-05 8.093e-05 0.02%
AmrLevel::restart() 1 7.625e-05 7.625e-05 7.625e-05 0.02%
StateData::restartDoit() 4 6.077e-05 6.077e-05 6.077e-05 0.02%
Gravity::update_max_rhs() 6 5.983e-05 5.983e-05 5.983e-05 0.02%
FabArrayBase::FB::FB() 26 5.825e-05 5.825e-05 5.825e-05 0.02%
Gravity::solve_for_phi() 5 5.005e-05 5.005e-05 5.005e-05 0.01%
MLMG:computeResOfCorrection() 180 4.869e-05 4.869e-05 4.869e-05 0.01%
Castro::expand_state() 5 4.794e-05 4.794e-05 4.794e-05 0.01%
MLMG::actualBottomSolve() 36 3.828e-05 3.828e-05 3.828e-05 0.01%
MLMG::mgVcycle_down::0 36 3.826e-05 3.826e-05 3.826e-05 0.01%
Castro::clean_state() 30 3.739e-05 3.739e-05 3.739e-05 0.01%
MLMG::mgVcycle_down::2 36 3.62e-05 3.62e-05 3.62e-05 0.01%
Castro::construct_new_source() 25 3.526e-05 3.526e-05 3.526e-05 0.01%
Castro::initialize_advance() 5 3.396e-05 3.396e-05 3.396e-05 0.01%
MLMG::mgVcycle_down::1 36 3.301e-05 3.301e-05 3.301e-05 0.01%
MLMG::solve() 6 3.174e-05 3.174e-05 3.174e-05 0.01%
MLMG::mgVcycle_down::4 36 3.101e-05 3.101e-05 3.101e-05 0.01%
MLMG::mgVcycle_down::3 36 2.991e-05 2.991e-05 2.991e-05 0.01%
Castro::finalize_advance() 5 2.818e-05 2.818e-05 2.818e-05 0.01%
MLMG::mgVcycle_up::4 36 2.7e-05 2.7e-05 2.7e-05 0.01%
MLCellLinOp::solutionResidual() 42 2.692e-05 2.692e-05 2.692e-05 0.01%
Castro::initialize_do_advance() 5 2.639e-05 2.639e-05 2.639e-05 0.01%
Castro::buildMetrics() 1 2.574e-05 2.574e-05 2.574e-05 0.01%
Amr::writeSmallPlotFile() 1 2.556e-05 2.556e-05 2.556e-05 0.01%
Castro::initMFs() 1 2.535e-05 2.535e-05 2.535e-05 0.01%
MLMG::oneIter() 36 2.466e-05 2.466e-05 2.466e-05 0.01%
Castro::swap_state_time_levels() 5 2.414e-05 2.414e-05 2.414e-05 0.01%
MLMG::mgVcycle_up::0 36 2.228e-05 2.228e-05 2.228e-05 0.01%
MLMG::mgVcycle_up::3 36 2.082e-05 2.082e-05 2.082e-05 0.01%
MLMG::mgVcycle_up::2 36 2.063e-05 2.063e-05 2.063e-05 0.01%
MLMG::mgVcycle_up::1 36 2.049e-05 2.049e-05 2.049e-05 0.01%
Castro::post_restart() 1 1.924e-05 1.924e-05 1.924e-05 0.01%
Castro::finalize_do_advance() 5 1.796e-05 1.796e-05 1.796e-05 0.01%
MLMG::ResNormInf() 42 1.722e-05 1.722e-05 1.722e-05 0.00%
MLPoisson::define() 6 1.607e-05 1.607e-05 1.607e-05 0.00%
Gravity::multilevel_solve_for_new_phi() 1 1.565e-05 1.565e-05 1.565e-05 0.00%
FillPatchSingleLevel 20 1.435e-05 1.435e-05 1.435e-05 0.00%
Castro::advance() 5 1.377e-05 1.377e-05 1.377e-05 0.00%
MLMG::mgVcycle_bottom 36 1.348e-05 1.348e-05 1.348e-05 0.00%
makeSFC 30 1.346e-05 1.346e-05 1.346e-05 0.00%
MLMG::computeResidual() 36 1.322e-05 1.322e-05 1.322e-05 0.00%
Castro::construct_new_gravity() 5 1.206e-05 1.206e-05 1.206e-05 0.00%
Gravity::solve_phi_with_mlmg() 6 1.187e-05 1.187e-05 1.187e-05 0.00%
MLPoisson::prepareForSolve() 6 1.06e-05 1.06e-05 1.06e-05 0.00%
MLMG::computeMLResidual() 6 1.025e-05 1.025e-05 1.025e-05 0.00%
Castro::construct_old_source() 25 9.594e-06 9.594e-06 9.594e-06 0.00%
DistributionMapping::Distribute() 31 8.843e-06 8.843e-06 8.843e-06 0.00%
Amr::initSubcycle() 1 8.522e-06 8.522e-06 8.522e-06 0.00%
Castro::do_new_sources() 5 7.91e-06 7.91e-06 7.91e-06 0.00%
Gravity::actual_multilevel_solve() 1 7.835e-06 7.835e-06 7.835e-06 0.00%
Castro::do_old_sources() 5 7.462e-06 7.462e-06 7.462e-06 0.00%
MLLinOp::makeAgglomeratedDMap 6 7.108e-06 7.108e-06 7.108e-06 0.00%
MLLinOp::define() 6 5.534e-06 5.534e-06 5.534e-06 0.00%
Castro::apply_source_to_state() 10 5.374e-06 5.374e-06 5.374e-06 0.00%
Castro::check_for_nan() 10 5.243e-06 5.243e-06 5.243e-06 0.00%
Castro::construct_old_gravity() 5 5.042e-06 5.042e-06 5.042e-06 0.00%
Castro::post_timestep() 5 4.247e-06 4.247e-06 4.247e-06 0.00%
Gravity::swapTimeLevels() 5 3.86e-06 3.86e-06 3.86e-06 0.00%
Castro::computeNewDt() 5 3.376e-06 3.376e-06 3.376e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 2.995e-06 2.995e-06 2.995e-06 0.00%
MLMG::getGradSolution() 6 2.88e-06 2.88e-06 2.88e-06 0.00%
Gravity::set_mass_offset() 6 2.171e-06 2.171e-06 2.171e-06 0.00%
MLMG::MLResNormInf() 6 2.118e-06 2.118e-06 2.118e-06 0.00%
MLMG::MLRhsNormInf() 6 2.089e-06 2.089e-06 2.089e-06 0.00%
Castro::retry_advance_ctu() 5 1.651e-06 1.651e-06 1.651e-06 0.00%
Castro::FluxRegCrseInit 5 1.552e-06 1.552e-06 1.552e-06 0.00%
MLLinOp::makeSubCommunicator() 6 1.114e-06 1.114e-06 1.114e-06 0.00%
Castro::FluxRegFineAdd() 5 1.084e-06 1.084e-06 1.084e-06 0.00%
Amr::init() 1 1.027e-06 1.027e-06 1.027e-06 0.00%
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Name NCalls Incl. Min Incl. Avg Incl. Max Max %
--------------------------------------------------------------------------------------------
main() 1 0.3572 0.3572 0.3572 99.99%
Amr::coarseTimeStep() 5 0.2781 0.2781 0.2781 77.87%
Amr::timeStep() 5 0.2758 0.2758 0.2758 77.21%
Castro::advance() 5 0.2722 0.2722 0.2722 76.22%
Castro::subcycle_advance_ctu() 5 0.2656 0.2656 0.2656 74.35%
Castro::do_advance_ctu() 5 0.2655 0.2655 0.2655 74.33%
Castro::construct_new_gravity() 5 0.1269 0.1269 0.1269 35.53%
Gravity::solve_phi_with_mlmg() 6 0.1222 0.1222 0.1222 34.22%
Gravity::solve_for_phi() 5 0.1192 0.1192 0.1192 33.37%
Gravity::actual_solve_with_mlmg() 6 0.1186 0.1186 0.1186 33.20%
MLMG::solve() 6 0.1078 0.1078 0.1078 30.19%
MLMG::oneIter() 36 0.1011 0.1011 0.1011 28.31%
MLMG::mgVcycle() 36 0.09957 0.09957 0.09957 27.88%
Castro::construct_ctu_hydro_source() 5 0.09954 0.09954 0.09954 27.87%
MLCellLinOp::smooth() 720 0.04941 0.04941 0.04941 13.83%
MLCellLinOp::applyBC() 1946 0.04652 0.04652 0.04652 13.03%
Amr::init() 1 0.04538 0.04538 0.04538 12.71%
Amr::restart() 1 0.04538 0.04538 0.04538 12.71%
AmrLevel::restart() 1 0.03925 0.03925 0.03925 10.99%
StateData::restartDoit() 4 0.03917 0.03917 0.03917 10.97%
VisMF::Read() 3 0.03906 0.03906 0.03906 10.94%
Amr::writePlotFile() 1 0.03295 0.03295 0.03295 9.22%
VisMF::Write(FabArray) 1 0.03134 0.03134 0.03134 8.78%
MLMG::mgVcycle_bottom 36 0.03038 0.03038 0.03038 8.51%
MLMG::actualBottomSolve() 36 0.03037 0.03037 0.03037 8.50%
MLCGSolver::bicgstab 36 0.03008 0.03008 0.03008 8.42%
Castro::clean_state() 30 0.02274 0.02274 0.02274 6.37%
FillPatchIterator::Initialize 20 0.02022 0.02022 0.02022 5.66%
FillPatchSingleLevel 20 0.01949 0.01949 0.01949 5.46%
StateDataPhysBCFunct::() 20 0.01746 0.01746 0.01746 4.89%
MLCellLinOp::apply() 500 0.01528 0.01528 0.01528 4.28%
MLMG::mgVcycle_down::0 36 0.01443 0.01443 0.01443 4.04%
FabArray::FillBoundary() 1766 0.01385 0.01385 0.01385 3.88%
MLPoisson::Fsmooth() 1440 0.01353 0.01353 0.01353 3.79%
FillBoundary_nowait() 1766 0.0135 0.0135 0.0135 3.78%
StateData::FillBoundary(geom) 160 0.01159 0.01159 0.01159 3.25%
MLMG::mgVcycle_up::0 36 0.01084 0.01084 0.01084 3.04%
Castro::initialize_do_advance() 5 0.01028 0.01028 0.01028 2.88%
MLCellLinOp::correctionResidual() 216 0.009276 0.009276 0.009276 2.60%
Castro::computeTemp() 30 0.009156 0.009156 0.009156 2.56%
MLPoisson::define() 6 0.008496 0.008496 0.008496 2.38%
amrex::Dot() 484 0.008416 0.008416 0.008416 2.36%
MLMG:computeResOfCorrection() 180 0.008157 0.008157 0.008157 2.28%
Gravity::get_new_grav_vector() 5 0.007609 0.007609 0.007609 2.13%
Castro::normalize_species() 30 0.00715 0.00715 0.00715 2.00%
Castro::do_new_sources() 5 0.007123 0.007123 0.007123 1.99%
Castro::construct_old_gravity() 5 0.0071 0.0071 0.0071 1.99%
Gravity::get_old_grav_vector() 5 0.007095 0.007095 0.007095 1.99%
amrex::Copy() 463 0.006787 0.006787 0.006787 1.90%
MLMG::mgVcycle_down::1 36 0.006618 0.006618 0.006618 1.85%
Castro::initialize_advance() 5 0.00638 0.00638 0.00638 1.79%
FabArray::ParallelCopy() 380 0.006355 0.006355 0.006355 1.78%
FabArray::ParallelCopy_nowait() 380 0.006251 0.006251 0.006251 1.75%
MLMG::mgVcycle_down::2 36 0.0062 0.0062 0.0062 1.74%
Castro::expand_state() 5 0.006108 0.006108 0.006108 1.71%
FabArray::setVal() 537 0.00604 0.00604 0.00604 1.69%
MLMG::mgVcycle_down::3 36 0.006026 0.006026 0.006026 1.69%
FabArray::norminf() 326 0.005996 0.005996 0.005996 1.68%
MLMG::mgVcycle_down::4 36 0.005965 0.005965 0.005965 1.67%
Castro::enforce_min_density() 30 0.005706 0.005706 0.005706 1.60%
MLCellLinOp::defineAuxData() 6 0.005706 0.005706 0.005706 1.60%
MLCGSolver::ParallelAllReduce 659 0.005059 0.005059 0.005059 1.42%
Castro::do_old_sources() 5 0.005 0.005 0.005 1.40%
MLMG::addInterpCorrection() 180 0.004953 0.004953 0.004953 1.39%
MLMG::mgVcycle_up::1 36 0.004832 0.004832 0.004832 1.35%
MLMG::mgVcycle_up::4 36 0.004799 0.004799 0.004799 1.34%
amrex::average_down 180 0.00473 0.00473 0.00473 1.32%
MLMG::mgVcycle_up::2 36 0.004705 0.004705 0.004705 1.32%
MLMG::mgVcycle_up::3 36 0.004612 0.004612 0.004612 1.29%
MLPoisson::Fapply() 500 0.004285 0.004285 0.004285 1.20%
Castro::reset_internal_energy(MultiFab) 30 0.003708 0.003708 0.003708 1.04%
FabArray::Saxpy() 355 0.003513 0.003513 0.003513 0.98%
Gravity::fill_multipole_BCs() 6 0.003506 0.003506 0.003506 0.98%
Castro::post_timestep() 5 0.003461 0.003461 0.003461 0.97%
FabArray::Xpay() 361 0.003371 0.003371 0.003371 0.94%
Castro::post_restart() 1 0.003368 0.003368 0.003368 0.94%
Gravity::multilevel_solve_for_new_phi() 1 0.003263 0.003263 0.003263 0.91%
Gravity::actual_multilevel_solve() 1 0.003247 0.003247 0.003247 0.91%
MLCellLinOp::solutionResidual() 42 0.003163 0.003163 0.003163 0.89%
Castro::estTimeStep() 10 0.002909 0.002909 0.002909 0.81%
MLMG::computeResidual() 36 0.002617 0.002617 0.002617 0.73%
MLCellLinOp::defineBC() 6 0.002568 0.002568 0.002568 0.72%
MLMG::prepareForSolve() 6 0.00246 0.00246 0.00246 0.69%
BndryData::define() 6 0.002451 0.002451 0.002451 0.69%
Castro::computeNewDt() 5 0.002175 0.002175 0.002175 0.61%
FabArray::LinComb() 242 0.001851 0.001851 0.001851 0.52%
amrex::Add() 72 0.001838 0.001838 0.001838 0.51%
Castro::construct_new_source() 25 0.001741 0.001741 0.001741 0.49%
Castro::construct_new_gravity_source() 5 0.001705 0.001705 0.001705 0.48%
Castro::construct_old_source() 25 0.001399 0.001399 0.001399 0.39%
Castro::construct_old_gravity_source() 5 0.00139 0.00139 0.00139 0.39%
Castro::apply_source_to_state() 10 0.000915 0.000915 0.000915 0.26%
MLMG::ResNormInf() 42 0.0009093 0.0009093 0.0009093 0.25%
FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008789 0.0008789 0.0008789 0.25%
FabArrayBase::getCPC() 632 0.000753 0.000753 0.000753 0.21%
MLMG::getGradSolution() 6 0.0007495 0.0007495 0.0007495 0.21%
MLCellLinOp::compGrad() 6 0.0007467 0.0007467 0.0007467 0.21%
MLCellLinOp::setLevelBC() 6 0.0007296 0.0007296 0.0007296 0.20%
Castro::enforce_speed_limit() 30 0.0006956 0.0006956 0.0006956 0.19%
Castro::reset_internal_energy(Fab) 240 0.0006953 0.0006953 0.0006953 0.19%
FabArray::mult() 22 0.0006423 0.0006423 0.0006423 0.18%
FabArray::setDomainBndry() 20 0.000623 0.000623 0.000623 0.17%
MLPoisson::prepareForSolve() 6 0.0006064 0.0006064 0.0006064 0.17%
Castro::check_for_nan() 10 0.0006053 0.0006053 0.0006053 0.17%
MultiFab::contains_nan() 10 0.0006 0.0006 0.0006 0.17%
MLCellLinOp::prepareForSolve() 6 0.0005958 0.0005958 0.0005958 0.17%
MLMG::computeMLResidual() 6 0.0005694 0.0005694 0.0005694 0.16%
Gravity::update_max_rhs() 6 0.0004351 0.0004351 0.0004351 0.12%
FabArrayBase::CPC::define() 244 0.0004124 0.0004124 0.0004124 0.12%
Amr::InitAmr() 1 0.0003896 0.0003896 0.0003896 0.11%
FabArrayBase::getFB() 1766 0.0003095 0.0003095 0.0003095 0.09%
Castro::finalize_advance() 5 0.0002828 0.0002828 0.0002828 0.08%
Gravity::swapTimeLevels() 5 0.0002209 0.0002209 0.0002209 0.06%
MLLinOp::define() 6 0.0002059 0.0002059 0.0002059 0.06%
MLLinOp::defineGrids() 6 0.0002004 0.0002004 0.0002004 0.06%
MLMG::MLResNormInf() 6 0.0001496 0.0001496 0.0001496 0.04%
Castro::buildMetrics() 1 0.0001441 0.0001441 0.0001441 0.04%
MultiFab::max() 6 0.0001348 0.0001348 0.0001348 0.04%
Castro::create_source_corrector() 5 0.0001207 0.0001207 0.0001207 0.03%
MLMG::MLRhsNormInf() 6 0.0001153 0.0001153 0.0001153 0.03%
FabArrayBase::FB::FB() 26 5.825e-05 5.825e-05 5.825e-05 0.02%
MLLinOp::makeAgglomeratedDMap 6 2.781e-05 2.781e-05 2.781e-05 0.01%
Amr::writeSmallPlotFile() 1 2.556e-05 2.556e-05 2.556e-05 0.01%
Castro::initMFs() 1 2.535e-05 2.535e-05 2.535e-05 0.01%
Castro::swap_state_time_levels() 5 2.414e-05 2.414e-05 2.414e-05 0.01%
makeSFC 30 2.07e-05 2.07e-05 2.07e-05 0.01%
Castro::finalize_do_advance() 5 1.796e-05 1.796e-05 1.796e-05 0.01%
DistributionMapping::Distribute() 31 8.843e-06 8.843e-06 8.843e-06 0.00%
Amr::initSubcycle() 1 8.522e-06 8.522e-06 8.522e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 4.599e-06 4.599e-06 4.599e-06 0.00%
Gravity::set_mass_offset() 6 2.171e-06 2.171e-06 2.171e-06 0.00%
Castro::retry_advance_ctu() 5 1.651e-06 1.651e-06 1.651e-06 0.00%
Castro::FluxRegCrseInit 5 1.552e-06 1.552e-06 1.552e-06 0.00%
MLLinOp::makeSubCommunicator() 6 1.114e-06 1.114e-06 1.114e-06 0.00%
Castro::FluxRegFineAdd() 5 1.084e-06 1.084e-06 1.084e-06 0.00%
--------------------------------------------------------------------------------------------
Unused ParmParse Variables:
[TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2]
[TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2]
Device Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Arena::Initialize() 1 1 228 MiB 9042 MiB
Castro::construct_ctu_hydro_source() 1440 1440 131 MiB 692 MiB
Castro::initMFs() 48 48 59 MiB 68 MiB
StateData::restartDoit() 32 32 53 MiB 55 MiB
Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB
Castro::initialize_do_advance() 40 40 28 MiB 39 MiB
FillPatchIterator::Initialize 160 160 1266 KiB 39 MiB
ResizeRandomSeed 1 1 30 MiB 30 MiB
Amr::writePlotFile() 8 8 2581 KiB 28 MiB
Castro::initialize_advance() 40 40 17 MiB 23 MiB
Castro::buildMetrics() 32 32 14 MiB 15 MiB
Castro::post_restart() 48 48 6670 KiB 14 MiB
MLMG::prepareForSolve() 361 361 3707 KiB 12 MiB
Gravity::get_old_grav_vector() 43 43 205 KiB 10 MiB
Gravity::get_new_grav_vector() 40 40 218 KiB 10 MiB
Gravity::multilevel_solve_for_new_phi() 24 24 6665 KiB 7586 KiB
Gravity::fill_multipole_BCs() 84 84 14 KiB 2053 KiB
Gravity::update_max_rhs() 48 48 2352 B 2048 KiB
Gravity::solve_for_phi() 40 40 681 KiB 2048 KiB
Gravity::actual_multilevel_solve() 8 8 18 KiB 2048 KiB
BndryData::define() 576 576 340 KiB 1095 KiB
MLCellLinOp::defineAuxData() 936 936 218 KiB 671 KiB
Castro::estTimeStep() 10 10 2769 B 480 KiB
VisMF::Write(FabArray) 112 112 1345 B 320 KiB
Castro::normalize_species() 30 30 6506 B 320 KiB
amrex::average_down 469 469 1271 B 257 KiB
MLMG::addInterpCorrection() 468 468 1196 B 257 KiB
amrex::Dot() 592 592 3467 B 160 KiB
FabArray::norminf() 398 398 2479 B 160 KiB
Castro::do_advance_ctu() 5 5 576 B 160 KiB
MultiFab::max() 6 6 58 B 160 KiB
FabArray::setVal() 66 66 18 KiB 23 KiB
MultiFab::contains_nan() 10 10 33 B 20 KiB
MLPoisson::Fsmooth() 60 60 3605 B 12 KiB
FabArray::ParallelCopy_nowait() 380 380 54 B 10 KiB
FillBoundary_nowait() 336 336 322 B 9648 B
MLCellLinOp::applyBC() 3892 3892 236 B 9344 B
amrex::Copy() 56 56 5943 B 8464 B
MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B
StateData::FillBoundary(geom) 960 960 55 B 2784 B
FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B
MLCGSolver::bicgstab 324 324 122 B 1472 B
MLCellLinOp::defineBC() 36 36 383 B 1248 B
------------------------------------------------------------------------------
Managed Memory Usage:
------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------
The_Managed_Arena::Initialize() 1 1 1597 B 8192 KiB
------------------------------------------------------------------
Pinned Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Pinned_Arena::Initialize() 1 1 122 KiB 8192 KiB
VisMF::Write(FabArray) 120 120 270 KiB 3584 KiB
VisMF::Read() 24 24 212 KiB 3000 KiB
FabArray::setVal() 66 66 18 KiB 23 KiB
MLPoisson::Fsmooth() 60 60 3605 B 12 KiB
FabArray::ParallelCopy_nowait() 380 380 54 B 10 KiB
FillBoundary_nowait() 336 336 322 B 9648 B
MLCellLinOp::applyBC() 1946 1946 234 B 9328 B
amrex::Copy() 56 56 5943 B 8464 B
MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B
Gravity::get_old_grav_vector() 3 3 2625 B 3072 B
Gravity::fill_multipole_BCs() 18 18 5 B 2832 B
StateData::FillBoundary(geom) 960 960 56 B 2784 B
FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B
MLMG::prepareForSolve() 7 7 546 B 1296 B
amrex::average_down 37 37 221 B 1296 B
MLMG::addInterpCorrection() 36 36 2 B 1024 B
MLCellLinOp::setLevelBC() 36 36 0 B 768 B
amrex::Dot() 592 592 25 B 400 B
FabArray::norminf() 398 398 17 B 272 B
Castro::estTimeStep() 10 10 0 B 32 B
MultiFab::max() 6 6 0 B 16 B
MultiFab::contains_nan() 10 10 0 B 16 B
Castro::do_advance_ctu() 5 5 0 B 16 B
Castro::normalize_species() 30 30 0 B 16 B
------------------------------------------------------------------------------
Total GPU global memory (MB): 12056
Free GPU global memory (MB): 2424
[The Arena] space allocated (MB): 9042
[The Arena] space used (MB): 0
[The Managed Arena] space allocated (MB): 8
[The Managed Arena] space used (MB): 0
[The Pinned Arena] space allocated (MB): 8
[The Pinned Arena] space used (MB): 0
AMReX (23.05-5-g68aa69a11750) finalized