Initializing CUDA...
CUDA initialized with 1 device.
AMReX (23.05-31-g2e1106e246c4) initialized
Starting run at 08:06:14 UTC on 2023-06-01.
Successfully read inputs file ...
Castro git describe: 23.05-30-g283f15d69
AMReX git describe: 23.05-31-g2e1106e24
Microphysics git describe: 23.05-15-g2652c955
reading extern runtime parameters ...
3 Species:
C12 O16 Mg24
Successfully read inputs file ...
INITIAL GRIDS
Level 0 8 grids 262144 cells 100 % of domain
smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32
CHECKPOINT: file = dustcollapse-restart_chk00000
checkPoint() time = 0.058990983 secs.
PLOTFILE: file = dustcollapse-restart_plt00000
Write plotfile time = 0.033798689 seconds
[Level 0 step 1] ADVANCE with dt = 4.541742215e-05
[Level 0 step 1] Advanced 262144 cells
[STEP 1] Coarse TimeStep time: 0.046467461
[STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05
[Level 0 step 2] ADVANCE with dt = 4.768829326e-05
[Level 0 step 2] Advanced 262144 cells
[STEP 2] Coarse TimeStep time: 0.052759025
[STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05
[Level 0 step 3] ADVANCE with dt = 5.007270792e-05
[Level 0 step 3] Advanced 262144 cells
[STEP 3] Coarse TimeStep time: 0.052315349
[STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05
[Level 0 step 4] ADVANCE with dt = 5.257634331e-05
[Level 0 step 4] Advanced 262144 cells
[STEP 4] Coarse TimeStep time: 0.070743705
[STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05
[Level 0 step 5] ADVANCE with dt = 5.520516048e-05
[Level 0 step 5] Advanced 262144 cells
[STEP 5] Coarse TimeStep time: 0.06711848
[STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05
CHECKPOINT: file = dustcollapse-restart_chk00005
checkPoint() time = 0.05846933 secs.
[Level 0 step 6] ADVANCE with dt = 5.79654185e-05
[Level 0 step 6] Advanced 262144 cells
[STEP 6] Coarse TimeStep time: 0.068480091
[STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05
[Level 0 step 7] ADVANCE with dt = 6.086368943e-05
[Level 0 step 7] Advanced 262144 cells
[STEP 7] Coarse TimeStep time: 0.047869379
[STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05
[Level 0 step 8] ADVANCE with dt = 6.39068739e-05
[Level 0 step 8] Advanced 262144 cells
[STEP 8] Coarse TimeStep time: 0.047200776
[STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05
[Level 0 step 9] ADVANCE with dt = 6.71022176e-05
[Level 0 step 9] Advanced 262144 cells
[STEP 9] Coarse TimeStep time: 0.047452515
[STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05
[Level 0 step 10] ADVANCE with dt = 7.045732848e-05
[Level 0 step 10] Advanced 262144 cells
[STEP 10] Coarse TimeStep time: 0.066189161
[STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05
CHECKPOINT: file = dustcollapse-restart_chk00010
checkPoint() time = 0.05798371 secs.
PLOTFILE: file = dustcollapse-restart_plt00010
Write plotfile time = 0.033509817 seconds
Ending run at 08:06:15 UTC on 2023-06-01.
Run time = 0.858925861
Run time without initialization = 0.717183381
Average number of zones advanced per microsecond: 3.655
Average number of zones advanced per microsecond per rank: 3.655
CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472
TinyProfiler total time across processes [min...avg...max]: 0.859 ... 0.859 ... 0.859
--------------------------------------------------------------------------------------------
Name NCalls Excl. Min Excl. Avg Excl. Max Max %
--------------------------------------------------------------------------------------------
VisMF::Write(FabArray) 11 0.2358 0.2358 0.2358 27.45%
Castro::construct_ctu_hydro_source() 10 0.1984 0.1984 0.1984 23.09%
MLCellLinOp::applyBC() 4433 0.07879 0.07879 0.07879 9.17%
MLPoisson::Fsmooth() 3280 0.03292 0.03292 0.03292 3.83%
FillBoundary_nowait() 4023 0.03154 0.03154 0.03154 3.67%
StateData::FillBoundary(geom) 328 0.02617 0.02617 0.02617 3.05%
amrex::Dot() 1114 0.02099 0.02099 0.02099 2.44%
amrex::Copy() 1029 0.01511 0.01511 0.01511 1.76%
Castro::normalize_species() 62 0.01476 0.01476 0.01476 1.72%
Castro::computeTemp() 63 0.01469 0.01469 0.01469 1.71%
FabArray::norminf() 743 0.01467 0.01467 0.01467 1.71%
FabArray::ParallelCopy_nowait() 861 0.01366 0.01366 0.01366 1.59%
FabArray::setVal() 1144 0.01336 0.01336 0.01336 1.56%
StateDataPhysBCFunct::() 41 0.01091 0.01091 0.01091 1.27%
MLPoisson::Fapply() 1142 0.01057 0.01057 0.01057 1.23%
MLCellLinOp::defineAuxData() 11 0.01004 0.01004 0.01004 1.17%
Castro::enforce_min_density() 62 0.009791 0.009791 0.009791 1.14%
FabArray::Saxpy() 813 0.008306 0.008306 0.008306 0.97%
FabArray::Xpay() 821 0.008205 0.008205 0.008205 0.96%
MLMG::addInterpCorrection() 410 0.006821 0.006821 0.006821 0.79%
Gravity::fill_multipole_BCs() 11 0.006696 0.006696 0.006696 0.78%
amrex::average_down 410 0.006106 0.006106 0.006106 0.71%
Castro::estTimeStep() 21 0.005628 0.005628 0.005628 0.66%
FabArray::LinComb() 557 0.00468 0.00468 0.00468 0.54%
amrex::Add() 164 0.004397 0.004397 0.004397 0.51%
Castro::reset_internal_energy(MultiFab) 63 0.004116 0.004116 0.004116 0.48%
Amr::checkPoint() 3 0.004 0.004 0.004 0.47%
BndryData::define() 11 0.003889 0.003889 0.003889 0.45%
Castro::do_advance_ctu() 10 0.00332 0.00332 0.00332 0.39%
Castro::construct_new_gravity_source() 10 0.003313 0.003313 0.003313 0.39%
Castro::construct_old_gravity_source() 10 0.002738 0.002738 0.002738 0.32%
Amr::writePlotFile() 2 0.002448 0.002448 0.002448 0.28%
MLCGSolver::bicgstab 82 0.002272 0.002272 0.002272 0.26%
FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001759 0.001759 0.001759 0.20%
Gravity::actual_solve_with_mlmg() 11 0.001584 0.001584 0.001584 0.18%
Castro::reset_internal_energy(Fab) 504 0.001579 0.001579 0.001579 0.18%
Castro::subcycle_advance_ctu() 10 0.001577 0.001577 0.001577 0.18%
MLCellLinOp::setLevelBC() 11 0.00145 0.00145 0.00145 0.17%
FabArray::setDomainBndry() 41 0.001338 0.001338 0.001338 0.16%
FabArray::mult() 43 0.001331 0.001331 0.001331 0.15%
Castro::initData() 1 0.001252 0.001252 0.001252 0.15%
MultiFab::contains_nan() 20 0.001198 0.001198 0.001198 0.14%
MLCellLinOp::prepareForSolve() 11 0.001189 0.001189 0.001189 0.14%
MLCellLinOp::smooth() 1640 0.001113 0.001113 0.001113 0.13%
Castro::enforce_speed_limit() 62 0.001013 0.001013 0.001013 0.12%
MLCellLinOp::compGrad() 11 0.0009273 0.0009273 0.0009273 0.11%
MLMG::prepareForSolve() 11 0.0009175 0.0009175 0.0009175 0.11%
FabArray::FillBoundary() 4023 0.0008855 0.0008855 0.0008855 0.10%
FabArrayBase::getCPC() 1323 0.0008036 0.0008036 0.0008036 0.09%
FabArrayBase::CPC::define() 454 0.0006974 0.0006974 0.0006974 0.08%
Gravity::get_new_grav_vector() 11 0.0006487 0.0006487 0.0006487 0.08%
FabArrayBase::getFB() 4023 0.0006405 0.0006405 0.0006405 0.07%
Gravity::get_old_grav_vector() 10 0.0005667 0.0005667 0.0005667 0.07%
Amr::InitAmr() 1 0.0005292 0.0005292 0.0005292 0.06%
MLCellLinOp::apply() 1142 0.0004773 0.0004773 0.0004773 0.06%
MLMG::mgVcycle() 82 0.0004104 0.0004104 0.0004104 0.05%
MLLinOp::defineGrids() 11 0.0003495 0.0003495 0.0003495 0.04%
Amr::coarseTimeStep() 10 0.0003469 0.0003469 0.0003469 0.04%
MLCGSolver::ParallelAllReduce 1514 0.0003089 0.0003089 0.0003089 0.04%
main() 1 0.0002906 0.0002906 0.0002906 0.03%
MultiFab::max() 11 0.0002653 0.0002653 0.0002653 0.03%
FabArray::ParallelCopy() 861 0.0002596 0.0002596 0.0002596 0.03%
FillPatchIterator::Initialize 41 0.0002376 0.0002376 0.0002376 0.03%
MLCellLinOp::correctionResidual() 492 0.0002296 0.0002296 0.0002296 0.03%
MLCellLinOp::defineBC() 11 0.0002272 0.0002272 0.0002272 0.03%
Castro::create_source_corrector() 10 0.0001843 0.0001843 0.0001843 0.02%
Amr::timeStep() 10 0.0001659 0.0001659 0.0001659 0.02%
Gravity::update_max_rhs() 11 0.0001441 0.0001441 0.0001441 0.02%
StateData::checkPoint() 12 0.0001384 0.0001384 0.0001384 0.02%
Castro::advance() 10 0.0001187 0.0001187 0.0001187 0.01%
MLMG:computeResOfCorrection() 410 0.000115 0.000115 0.000115 0.01%
Gravity::solve_for_phi() 10 0.0001148 0.0001148 0.0001148 0.01%
Castro::Castro() 1 9.472e-05 9.472e-05 9.472e-05 0.01%
MLMG::mgVcycle_down::0 82 9.397e-05 9.397e-05 9.397e-05 0.01%
MLMG::actualBottomSolve() 82 9.32e-05 9.32e-05 9.32e-05 0.01%
FabArrayBase::FB::FB() 56 9.037e-05 9.037e-05 9.037e-05 0.01%
Castro::initialize_advance() 10 8.485e-05 8.485e-05 8.485e-05 0.01%
MLMG::mgVcycle_down::1 82 8.284e-05 8.284e-05 8.284e-05 0.01%
Castro::expand_state() 10 8.215e-05 8.215e-05 8.215e-05 0.01%
MLMG::solve() 11 8.018e-05 8.018e-05 8.018e-05 0.01%
MLMG::mgVcycle_down::2 82 7.794e-05 7.794e-05 7.794e-05 0.01%
MLMG::mgVcycle_down::3 82 7.736e-05 7.736e-05 7.736e-05 0.01%
AmrLevel::checkPoint() 3 7.447e-05 7.447e-05 7.447e-05 0.01%
Castro::clean_state() 62 7.379e-05 7.379e-05 7.379e-05 0.01%
MLMG::mgVcycle_down::4 82 7.328e-05 7.328e-05 7.328e-05 0.01%
MLMG::mgVcycle_up::2 82 6.502e-05 6.502e-05 6.502e-05 0.01%
Castro::finalize_advance() 10 6.457e-05 6.457e-05 6.457e-05 0.01%
MLMG::mgVcycle_up::4 82 6.357e-05 6.357e-05 6.357e-05 0.01%
MLMG::oneIter() 82 5.97e-05 5.97e-05 5.97e-05 0.01%
MLMG::mgVcycle_up::3 82 5.54e-05 5.54e-05 5.54e-05 0.01%
Castro::initialize_do_advance() 10 5.389e-05 5.389e-05 5.389e-05 0.01%
MLCellLinOp::solutionResidual() 93 5.296e-05 5.296e-05 5.296e-05 0.01%
MLMG::mgVcycle_up::1 82 5.07e-05 5.07e-05 5.07e-05 0.01%
MLMG::mgVcycle_up::0 82 5.034e-05 5.034e-05 5.034e-05 0.01%
Castro::construct_new_source() 50 4.755e-05 4.755e-05 4.755e-05 0.01%
StateData::define() 4 4.574e-05 4.574e-05 4.574e-05 0.01%
Castro::swap_state_time_levels() 10 3.984e-05 3.984e-05 3.984e-05 0.00%
Castro::finalize_do_advance() 10 3.775e-05 3.775e-05 3.775e-05 0.00%
MLMG::ResNormInf() 93 3.746e-05 3.746e-05 3.746e-05 0.00%
MLMG::mgVcycle_bottom 82 3.62e-05 3.62e-05 3.62e-05 0.00%
Castro::enforce_consistent_e() 1 3.292e-05 3.292e-05 3.292e-05 0.00%
MLMG::computeResidual() 82 3.279e-05 3.279e-05 3.279e-05 0.00%
MLPoisson::define() 11 3.142e-05 3.142e-05 3.142e-05 0.00%
Castro::post_timestep() 10 3.116e-05 3.116e-05 3.116e-05 0.00%
FillPatchSingleLevel 41 3.084e-05 3.084e-05 3.084e-05 0.00%
makeSFC 55 2.673e-05 2.673e-05 2.673e-05 0.00%
Castro::initMFs() 1 2.656e-05 2.656e-05 2.656e-05 0.00%
Castro::construct_new_gravity() 10 2.651e-05 2.651e-05 2.651e-05 0.00%
Amr::writeSmallPlotFile() 1 2.534e-05 2.534e-05 2.534e-05 0.00%
Amr::defBaseLevel() 1 2.497e-05 2.497e-05 2.497e-05 0.00%
Gravity::solve_phi_with_mlmg() 11 2.474e-05 2.474e-05 2.474e-05 0.00%
Castro::construct_old_source() 50 2.089e-05 2.089e-05 2.089e-05 0.00%
Castro::buildMetrics() 1 1.95e-05 1.95e-05 1.95e-05 0.00%
Amr::FinalizeInit() 1 1.921e-05 1.921e-05 1.921e-05 0.00%
AmrLevel::AmrLevel(dm) 1 1.843e-05 1.843e-05 1.843e-05 0.00%
Gravity::multilevel_solve_for_new_phi() 1 1.785e-05 1.785e-05 1.785e-05 0.00%
Castro::do_old_sources() 10 1.765e-05 1.765e-05 1.765e-05 0.00%
Castro::do_new_sources() 10 1.708e-05 1.708e-05 1.708e-05 0.00%
MLLinOp::define() 11 1.682e-05 1.682e-05 1.682e-05 0.00%
DistributionMapping::Distribute() 56 1.629e-05 1.629e-05 1.629e-05 0.00%
MLLinOp::makeAgglomeratedDMap 11 1.528e-05 1.528e-05 1.528e-05 0.00%
Castro::check_for_nan() 20 1.193e-05 1.193e-05 1.193e-05 0.00%
Castro::apply_source_to_state() 20 1.124e-05 1.124e-05 1.124e-05 0.00%
Castro::construct_old_gravity() 10 9.593e-06 9.593e-06 9.593e-06 0.00%
MLPoisson::prepareForSolve() 11 9.468e-06 9.468e-06 9.468e-06 0.00%
Gravity::swapTimeLevels() 10 9.304e-06 9.304e-06 9.304e-06 0.00%
Amr::initSubcycle() 1 8.732e-06 8.732e-06 8.732e-06 0.00%
MLMG::computeMLResidual() 11 8.699e-06 8.699e-06 8.699e-06 0.00%
Gravity::actual_multilevel_solve() 1 8.08e-06 8.08e-06 8.08e-06 0.00%
Castro::computeNewDt() 9 7.883e-06 7.883e-06 7.883e-06 0.00%
MLMG::getGradSolution() 11 5.871e-06 5.871e-06 5.871e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 4.77e-06 4.77e-06 4.77e-06 0.00%
Amr::InitializeInit() 1 4.754e-06 4.754e-06 4.754e-06 0.00%
Castro::retry_advance_ctu() 10 4.626e-06 4.626e-06 4.626e-06 0.00%
Gravity::set_mass_offset() 11 4.195e-06 4.195e-06 4.195e-06 0.00%
AmrLevel::checkPointPost() 3 4.182e-06 4.182e-06 4.182e-06 0.00%
MLMG::MLRhsNormInf() 11 3.943e-06 3.943e-06 3.943e-06 0.00%
Castro::post_init() 1 3.72e-06 3.72e-06 3.72e-06 0.00%
MLMG::MLResNormInf() 11 3.689e-06 3.689e-06 3.689e-06 0.00%
Amr::init() 1 3.576e-06 3.576e-06 3.576e-06 0.00%
Castro::FluxRegCrseInit 10 3.307e-06 3.307e-06 3.307e-06 0.00%
Castro::computeInitialDt() 2 2.985e-06 2.985e-06 2.985e-06 0.00%
Castro::FluxRegFineAdd() 10 2.672e-06 2.672e-06 2.672e-06 0.00%
AmrLevel::checkPointPre() 3 2.017e-06 2.017e-06 2.017e-06 0.00%
MLLinOp::makeSubCommunicator() 11 1.922e-06 1.922e-06 1.922e-06 0.00%
Amr::initialInit() 1 1.345e-06 1.345e-06 1.345e-06 0.00%
Castro::post_regrid() 1 1.012e-06 1.012e-06 1.012e-06 0.00%
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Name NCalls Incl. Min Incl. Avg Incl. Max Max %
--------------------------------------------------------------------------------------------
main() 1 0.8589 0.8589 0.8589 100.00%
Amr::coarseTimeStep() 10 0.6835 0.6835 0.6835 79.57%
Amr::timeStep() 10 0.5629 0.5629 0.5629 65.53%
Castro::advance() 10 0.5559 0.5559 0.5559 64.72%
Castro::subcycle_advance_ctu() 10 0.5438 0.5438 0.5438 63.31%
Castro::do_advance_ctu() 10 0.5422 0.5422 0.5422 63.12%
Gravity::solve_phi_with_mlmg() 11 0.2902 0.2902 0.2902 33.79%
Gravity::actual_solve_with_mlmg() 11 0.283 0.283 0.283 32.95%
Castro::construct_new_gravity() 10 0.2658 0.2658 0.2658 30.95%
MLMG::solve() 11 0.2616 0.2616 0.2616 30.45%
Gravity::solve_for_phi() 10 0.2499 0.2499 0.2499 29.10%
MLMG::oneIter() 82 0.2469 0.2469 0.2469 28.75%
MLMG::mgVcycle() 82 0.2432 0.2432 0.2432 28.32%
VisMF::Write(FabArray) 11 0.2358 0.2358 0.2358 27.45%
Castro::construct_ctu_hydro_source() 10 0.1984 0.1984 0.1984 23.09%
Amr::checkPoint() 3 0.1756 0.1756 0.1756 20.44%
AmrLevel::checkPoint() 3 0.1716 0.1716 0.1716 19.97%
StateData::checkPoint() 12 0.1715 0.1715 0.1715 19.96%
Amr::init() 1 0.1411 0.1411 0.1411 16.42%
MLCellLinOp::smooth() 1640 0.1195 0.1195 0.1195 13.91%
MLCellLinOp::applyBC() 4433 0.1119 0.1119 0.1119 13.03%
MLMG::mgVcycle_bottom 82 0.07555 0.07555 0.07555 8.80%
MLMG::actualBottomSolve() 82 0.07551 0.07551 0.07551 8.79%
MLCGSolver::bicgstab 82 0.0748 0.0748 0.0748 8.71%
Amr::writePlotFile() 2 0.06743 0.06743 0.06743 7.85%
Amr::initialInit() 1 0.04816 0.04816 0.04816 5.61%
Castro::clean_state() 62 0.04521 0.04521 0.04521 5.26%
Amr::FinalizeInit() 1 0.04402 0.04402 0.04402 5.13%
FillPatchIterator::Initialize 41 0.04277 0.04277 0.04277 4.98%
Castro::post_init() 1 0.04267 0.04267 0.04267 4.97%
FillPatchSingleLevel 41 0.0412 0.0412 0.0412 4.80%
Gravity::multilevel_solve_for_new_phi() 1 0.04071 0.04071 0.04071 4.74%
Gravity::actual_multilevel_solve() 1 0.04069 0.04069 0.04069 4.74%
MLCellLinOp::apply() 1142 0.03709 0.03709 0.03709 4.32%
StateDataPhysBCFunct::() 41 0.03708 0.03708 0.03708 4.32%
MLMG::mgVcycle_down::0 82 0.03441 0.03441 0.03441 4.01%
FabArray::FillBoundary() 4023 0.03316 0.03316 0.03316 3.86%
MLPoisson::Fsmooth() 3280 0.03292 0.03292 0.03292 3.83%
FillBoundary_nowait() 4023 0.03227 0.03227 0.03227 3.76%
StateData::FillBoundary(geom) 328 0.02617 0.02617 0.02617 3.05%
MLMG::mgVcycle_up::0 82 0.0261 0.0261 0.0261 3.04%
MLCellLinOp::correctionResidual() 492 0.02256 0.02256 0.02256 2.63%
amrex::Dot() 1114 0.02099 0.02099 0.02099 2.44%
Castro::computeTemp() 63 0.02038 0.02038 0.02038 2.37%
MLMG:computeResOfCorrection() 410 0.01986 0.01986 0.01986 2.31%
Castro::initialize_do_advance() 10 0.01816 0.01816 0.01816 2.11%
Gravity::get_new_grav_vector() 11 0.01757 0.01757 0.01757 2.05%
MLPoisson::define() 11 0.01701 0.01701 0.01701 1.98%
MLMG::mgVcycle_down::1 82 0.01615 0.01615 0.01615 1.88%
Castro::construct_old_gravity() 10 0.01531 0.01531 0.01531 1.78%
Gravity::get_old_grav_vector() 10 0.0153 0.0153 0.0153 1.78%
amrex::Copy() 1029 0.01511 0.01511 0.01511 1.76%
MLMG::mgVcycle_down::2 82 0.01508 0.01508 0.01508 1.76%
FabArray::ParallelCopy() 861 0.01479 0.01479 0.01479 1.72%
Castro::normalize_species() 62 0.01476 0.01476 0.01476 1.72%
MLMG::mgVcycle_down::3 82 0.01469 0.01469 0.01469 1.71%
FabArray::norminf() 743 0.01467 0.01467 0.01467 1.71%
MLMG::mgVcycle_down::4 82 0.01454 0.01454 0.01454 1.69%
FabArray::ParallelCopy_nowait() 861 0.01453 0.01453 0.01453 1.69%
Castro::do_new_sources() 10 0.01391 0.01391 0.01391 1.62%
FabArray::setVal() 1144 0.01336 0.01336 0.01336 1.56%
MLCGSolver::ParallelAllReduce 1514 0.01256 0.01256 0.01256 1.46%
MLMG::addInterpCorrection() 410 0.01219 0.01219 0.01219 1.42%
MLMG::mgVcycle_up::4 82 0.01178 0.01178 0.01178 1.37%
MLMG::mgVcycle_up::1 82 0.01174 0.01174 0.01174 1.37%
MLMG::mgVcycle_up::2 82 0.01152 0.01152 0.01152 1.34%
amrex::average_down 410 0.01144 0.01144 0.01144 1.33%
MLCellLinOp::defineAuxData() 11 0.01143 0.01143 0.01143 1.33%
Castro::initialize_advance() 10 0.01137 0.01137 0.01137 1.32%
MLMG::mgVcycle_up::3 82 0.01126 0.01126 0.01126 1.31%
Castro::expand_state() 10 0.01097 0.01097 0.01097 1.28%
MLPoisson::Fapply() 1142 0.01057 0.01057 0.01057 1.23%
Castro::do_old_sources() 10 0.009923 0.009923 0.009923 1.16%
Castro::enforce_min_density() 62 0.009791 0.009791 0.009791 1.14%
FabArray::Saxpy() 813 0.008306 0.008306 0.008306 0.97%
FabArray::Xpay() 821 0.008205 0.008205 0.008205 0.96%
MLCellLinOp::solutionResidual() 93 0.007321 0.007321 0.007321 0.85%
Gravity::fill_multipole_BCs() 11 0.006973 0.006973 0.006973 0.81%
Castro::post_timestep() 10 0.006823 0.006823 0.006823 0.79%
MLMG::computeResidual() 82 0.006311 0.006311 0.006311 0.73%
Castro::reset_internal_energy(MultiFab) 63 0.005695 0.005695 0.005695 0.66%
Castro::estTimeStep() 21 0.005628 0.005628 0.005628 0.66%
MLCellLinOp::defineBC() 11 0.00513 0.00513 0.00513 0.60%
BndryData::define() 11 0.004903 0.004903 0.004903 0.57%
MLMG::prepareForSolve() 11 0.004837 0.004837 0.004837 0.56%
FabArray::LinComb() 557 0.00468 0.00468 0.00468 0.54%
amrex::Add() 164 0.004397 0.004397 0.004397 0.51%
Amr::InitializeInit() 1 0.004132 0.004132 0.004132 0.48%
Amr::defBaseLevel() 1 0.004127 0.004127 0.004127 0.48%
Castro::initData() 1 0.003575 0.003575 0.003575 0.42%
Castro::construct_new_source() 50 0.003361 0.003361 0.003361 0.39%
Castro::construct_new_gravity_source() 10 0.003313 0.003313 0.003313 0.39%
Castro::computeNewDt() 9 0.003043 0.003043 0.003043 0.35%
Castro::construct_old_source() 50 0.002758 0.002758 0.002758 0.32%
Castro::construct_old_gravity_source() 10 0.002738 0.002738 0.002738 0.32%
MLMG::ResNormInf() 93 0.002109 0.002109 0.002109 0.25%
Castro::apply_source_to_state() 20 0.001827 0.001827 0.001827 0.21%
FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001759 0.001759 0.001759 0.20%
Castro::reset_internal_energy(Fab) 504 0.001579 0.001579 0.001579 0.18%
FabArrayBase::getCPC() 1323 0.001501 0.001501 0.001501 0.17%
MLCellLinOp::setLevelBC() 11 0.00145 0.00145 0.00145 0.17%
MLMG::getGradSolution() 11 0.001417 0.001417 0.001417 0.16%
MLCellLinOp::compGrad() 11 0.001411 0.001411 0.001411 0.16%
FabArray::setDomainBndry() 41 0.001338 0.001338 0.001338 0.16%
FabArray::mult() 43 0.001331 0.001331 0.001331 0.15%
Castro::check_for_nan() 20 0.00121 0.00121 0.00121 0.14%
MLPoisson::prepareForSolve() 11 0.001199 0.001199 0.001199 0.14%
MultiFab::contains_nan() 20 0.001198 0.001198 0.001198 0.14%
MLCellLinOp::prepareForSolve() 11 0.001189 0.001189 0.001189 0.14%
Castro::post_regrid() 1 0.001123 0.001123 0.001123 0.13%
MLMG::computeMLResidual() 11 0.001052 0.001052 0.001052 0.12%
Castro::enforce_speed_limit() 62 0.001013 0.001013 0.001013 0.12%
Gravity::update_max_rhs() 11 0.0008912 0.0008912 0.0008912 0.10%
Castro::computeInitialDt() 2 0.0008891 0.0008891 0.0008891 0.10%
FabArrayBase::getFB() 4023 0.0007309 0.0007309 0.0007309 0.09%
FabArrayBase::CPC::define() 454 0.0006974 0.0006974 0.0006974 0.08%
Castro::finalize_advance() 10 0.0006088 0.0006088 0.0006088 0.07%
Amr::InitAmr() 1 0.0005379 0.0005379 0.0005379 0.06%
Castro::Castro() 1 0.0004568 0.0004568 0.0004568 0.05%
Gravity::swapTimeLevels() 10 0.0004552 0.0004552 0.0004552 0.05%
MLLinOp::define() 11 0.0004251 0.0004251 0.0004251 0.05%
MLLinOp::defineGrids() 11 0.0004082 0.0004082 0.0004082 0.05%
MLMG::MLResNormInf() 11 0.0002893 0.0002893 0.0002893 0.03%
MultiFab::max() 11 0.0002653 0.0002653 0.0002653 0.03%
MLMG::MLRhsNormInf() 11 0.000223 0.000223 0.000223 0.03%
Castro::create_source_corrector() 10 0.0001843 0.0001843 0.0001843 0.02%
Castro::buildMetrics() 1 0.0001721 0.0001721 0.0001721 0.02%
FabArrayBase::FB::FB() 56 9.037e-05 9.037e-05 9.037e-05 0.01%
AmrLevel::AmrLevel(dm) 1 6.417e-05 6.417e-05 6.417e-05 0.01%
MLLinOp::makeAgglomeratedDMap 11 5.678e-05 5.678e-05 5.678e-05 0.01%
StateData::define() 4 4.574e-05 4.574e-05 4.574e-05 0.01%
makeSFC 55 4.149e-05 4.149e-05 4.149e-05 0.00%
Castro::swap_state_time_levels() 10 3.984e-05 3.984e-05 3.984e-05 0.00%
Castro::finalize_do_advance() 10 3.775e-05 3.775e-05 3.775e-05 0.00%
Castro::enforce_consistent_e() 1 3.292e-05 3.292e-05 3.292e-05 0.00%
Castro::initMFs() 1 2.656e-05 2.656e-05 2.656e-05 0.00%
Amr::writeSmallPlotFile() 1 2.534e-05 2.534e-05 2.534e-05 0.00%
DistributionMapping::Distribute() 56 1.629e-05 1.629e-05 1.629e-05 0.00%
Amr::initSubcycle() 1 8.732e-06 8.732e-06 8.732e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 6.295e-06 6.295e-06 6.295e-06 0.00%
Castro::retry_advance_ctu() 10 4.626e-06 4.626e-06 4.626e-06 0.00%
Gravity::set_mass_offset() 11 4.195e-06 4.195e-06 4.195e-06 0.00%
AmrLevel::checkPointPost() 3 4.182e-06 4.182e-06 4.182e-06 0.00%
Castro::FluxRegCrseInit 10 3.307e-06 3.307e-06 3.307e-06 0.00%
Castro::FluxRegFineAdd() 10 2.672e-06 2.672e-06 2.672e-06 0.00%
AmrLevel::checkPointPre() 3 2.017e-06 2.017e-06 2.017e-06 0.00%
MLLinOp::makeSubCommunicator() 11 1.922e-06 1.922e-06 1.922e-06 0.00%
--------------------------------------------------------------------------------------------
Unused ParmParse Variables:
[TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2]
[TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2]
Device Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Arena::Initialize() 1 1 95 MiB 9042 MiB
Castro::construct_ctu_hydro_source() 2880 2880 105 MiB 692 MiB
Castro::initMFs() 48 48 67 MiB 68 MiB
Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB
StateData::define() 32 32 55 MiB 55 MiB
FillPatchIterator::Initialize 328 328 1048 KiB 39 MiB
Castro::initialize_do_advance() 80 80 24 MiB 39 MiB
ResizeRandomSeed 1 1 30 MiB 30 MiB
Amr::writePlotFile() 16 16 2204 KiB 28 MiB
Castro::initialize_advance() 80 80 15 MiB 23 MiB
Castro::buildMetrics() 32 32 15 MiB 15 MiB
Castro::Castro() 48 48 7611 KiB 14 MiB
MLMG::prepareForSolve() 660 660 3745 KiB 12 MiB
Gravity::get_new_grav_vector() 91 91 212 KiB 10 MiB
Gravity::get_old_grav_vector() 80 80 182 KiB 10 MiB
Gravity::multilevel_solve_for_new_phi() 24 24 7523 KiB 7586 KiB
Gravity::fill_multipole_BCs() 154 154 12 KiB 2053 KiB
Gravity::update_max_rhs() 88 88 1965 B 2048 KiB
Gravity::solve_for_phi() 80 80 595 KiB 2048 KiB
Gravity::actual_multilevel_solve() 8 8 96 KiB 2048 KiB
BndryData::define() 1056 1056 342 KiB 1095 KiB
MLCellLinOp::defineAuxData() 1716 1716 217 KiB 671 KiB
Castro::estTimeStep() 21 21 2678 B 480 KiB
VisMF::Write(FabArray) 656 656 3503 B 320 KiB
Castro::normalize_species() 62 62 5592 B 320 KiB
amrex::average_down 1067 1067 1321 B 257 KiB
MLMG::addInterpCorrection() 1066 1066 1208 B 257 KiB
amrex::Dot() 1360 1360 3594 B 160 KiB
FabArray::norminf() 907 907 2519 B 160 KiB
Castro::do_advance_ctu() 10 10 584 B 160 KiB
MultiFab::max() 11 11 47 B 160 KiB
Castro::initData() 1 1 28 B 160 KiB
FabArray::setVal() 106 106 21 KiB 26 KiB
MultiFab::contains_nan() 20 20 27 B 20 KiB
MLPoisson::Fsmooth() 132 132 3662 B 12 KiB
FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB
FillBoundary_nowait() 760 760 321 B 9648 B
MLCellLinOp::applyBC() 8866 8866 234 B 9344 B
MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B
amrex::Copy() 100 100 3865 B 6144 B
StateData::FillBoundary(geom) 1992 1992 45 B 2688 B
FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B
MLCGSolver::bicgstab 738 738 126 B 1472 B
MLCellLinOp::defineBC() 66 66 385 B 1248 B
------------------------------------------------------------------------------
Managed Memory Usage:
-----------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
-----------------------------------------------------------------
The_Managed_Arena::Initialize() 1 1 582 B 8192 KiB
-----------------------------------------------------------------
Pinned Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Pinned_Arena::Initialize() 1 1 49 KiB 8192 KiB
VisMF::Write(FabArray) 744 744 639 KiB 3584 KiB
FabArray::setVal() 106 106 21 KiB 26 KiB
MLPoisson::Fsmooth() 132 132 3662 B 12 KiB
FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB
FillBoundary_nowait() 760 760 321 B 9648 B
MLCellLinOp::applyBC() 4433 4433 232 B 9328 B
MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B
amrex::Copy() 100 100 3865 B 6144 B
Gravity::get_new_grav_vector() 3 3 2900 B 3072 B
Gravity::fill_multipole_BCs() 33 33 4 B 2832 B
StateData::FillBoundary(geom) 1992 1992 45 B 2688 B
FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B
amrex::average_down 83 83 272 B 1296 B
MLMG::addInterpCorrection() 82 82 2 B 1024 B
MLMG::prepareForSolve() 11 11 312 B 1024 B
MLCellLinOp::setLevelBC() 66 66 0 B 768 B
amrex::Dot() 1360 1360 26 B 400 B
FabArray::norminf() 907 907 18 B 272 B
Castro::estTimeStep() 21 21 0 B 32 B
MultiFab::contains_nan() 20 20 0 B 16 B
MultiFab::max() 11 11 0 B 16 B
Castro::normalize_species() 62 62 0 B 16 B
Castro::initData() 1 1 0 B 16 B
Castro::do_advance_ctu() 10 10 0 B 16 B
------------------------------------------------------------------------------
Total GPU global memory (MB): 12056
Free GPU global memory (MB): 2424
[The Arena] space allocated (MB): 9042
[The Arena] space used (MB): 0
[The Managed Arena] space allocated (MB): 8
[The Managed Arena] space used (MB): 0
[The Pinned Arena] space allocated (MB): 8
[The Pinned Arena] space used (MB): 0
AMReX (23.05-31-g2e1106e246c4) finalized
Initializing CUDA...
CUDA initialized with 1 device.
AMReX (23.05-31-g2e1106e246c4) initialized
Starting run at 08:06:15 UTC on 2023-06-01.
Successfully read inputs file ...
Castro git describe: 23.05-30-g283f15d69
AMReX git describe: 23.05-31-g2e1106e24
Microphysics git describe: 23.05-15-g2652c955
reading extern runtime parameters ...
3 Species:
C12 O16 Mg24
restarting calculation from file: dustcollapse-restart_chk00005
Successfully read inputs file ...
read CPU time: 0.489627537
Restart time = 0.043065873 seconds.
[Level 0 step 6] ADVANCE with dt = 5.79654185e-05
[Level 0 step 6] Advanced 262144 cells
[STEP 6] Coarse TimeStep time: 0.049219979
[STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05
[Level 0 step 7] ADVANCE with dt = 6.086368943e-05
[Level 0 step 7] Advanced 262144 cells
[STEP 7] Coarse TimeStep time: 0.047861698
[STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05
[Level 0 step 8] ADVANCE with dt = 6.39068739e-05
[Level 0 step 8] Advanced 262144 cells
[STEP 8] Coarse TimeStep time: 0.048411596
[STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05
[Level 0 step 9] ADVANCE with dt = 6.71022176e-05
[Level 0 step 9] Advanced 262144 cells
[STEP 9] Coarse TimeStep time: 0.046771757
[STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05
[Level 0 step 10] ADVANCE with dt = 7.045732848e-05
[Level 0 step 10] Advanced 262144 cells
[STEP 10] Coarse TimeStep time: 0.065047232
[STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05
PLOTFILE: file = dustcollapse-restart_plt00010
Write plotfile time = 0.033148908 seconds
Ending run at 08:06:16 UTC on 2023-06-01.
Run time = 0.334561352
Run time without initialization = 0.290868027
Average number of zones advanced per microsecond: 4.506
Average number of zones advanced per microsecond per rank: 4.506
CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472
TinyProfiler total time across processes [min...avg...max]: 0.3346 ... 0.3346 ... 0.3346
--------------------------------------------------------------------------------------------
Name NCalls Excl. Min Excl. Avg Excl. Max Max %
--------------------------------------------------------------------------------------------
Castro::construct_ctu_hydro_source() 5 0.0777 0.0777 0.0777 23.22%
VisMF::Read() 3 0.03669 0.03669 0.03669 10.97%
MLCellLinOp::applyBC() 1946 0.03347 0.03347 0.03347 10.00%
VisMF::Write(FabArray) 1 0.03152 0.03152 0.03152 9.42%
MLPoisson::Fsmooth() 1440 0.01396 0.01396 0.01396 4.17%
FillBoundary_nowait() 1766 0.01276 0.01276 0.01276 3.81%
StateData::FillBoundary(geom) 160 0.01206 0.01206 0.01206 3.60%
amrex::Dot() 484 0.008778 0.008778 0.008778 2.62%
amrex::Copy() 463 0.00697 0.00697 0.00697 2.08%
Castro::normalize_species() 30 0.006618 0.006618 0.006618 1.98%
FabArray::setVal() 537 0.006271 0.006271 0.006271 1.87%
FabArray::norminf() 326 0.006236 0.006236 0.006236 1.86%
FabArray::ParallelCopy_nowait() 380 0.006071 0.006071 0.006071 1.81%
StateDataPhysBCFunct::() 20 0.005535 0.005535 0.005535 1.65%
Castro::computeTemp() 30 0.005377 0.005377 0.005377 1.61%
MLCellLinOp::defineAuxData() 6 0.005306 0.005306 0.005306 1.59%
MLPoisson::Fapply() 500 0.004481 0.004481 0.004481 1.34%
Castro::enforce_min_density() 30 0.00397 0.00397 0.00397 1.19%
FabArray::Saxpy() 355 0.003668 0.003668 0.003668 1.10%
Gravity::fill_multipole_BCs() 6 0.003608 0.003608 0.003608 1.08%
FabArray::Xpay() 361 0.003542 0.003542 0.003542 1.06%
Castro::estTimeStep() 10 0.003053 0.003053 0.003053 0.91%
MLMG::addInterpCorrection() 180 0.002893 0.002893 0.002893 0.86%
amrex::average_down 180 0.002599 0.002599 0.002599 0.78%
Amr::restart() 1 0.002518 0.002518 0.002518 0.75%
Castro::reset_internal_energy(MultiFab) 30 0.002455 0.002455 0.002455 0.73%
BndryData::define() 6 0.00207 0.00207 0.00207 0.62%
FabArray::LinComb() 242 0.00198 0.00198 0.00198 0.59%
amrex::Add() 72 0.00185 0.00185 0.00185 0.55%
Castro::construct_new_gravity_source() 5 0.001719 0.001719 0.001719 0.51%
Castro::do_advance_ctu() 5 0.001491 0.001491 0.001491 0.45%
Amr::writePlotFile() 1 0.001464 0.001464 0.001464 0.44%
Castro::construct_old_gravity_source() 5 0.001382 0.001382 0.001382 0.41%
MLCGSolver::bicgstab 36 0.0009365 0.0009365 0.0009365 0.28%
FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009145 0.0009145 0.0009145 0.27%
Gravity::actual_solve_with_mlmg() 6 0.0007848 0.0007848 0.0007848 0.23%
MLCellLinOp::setLevelBC() 6 0.000771 0.000771 0.000771 0.23%
Castro::reset_internal_energy(Fab) 240 0.0007225 0.0007225 0.0007225 0.22%
FabArray::setDomainBndry() 20 0.0006587 0.0006587 0.0006587 0.20%
FabArray::mult() 22 0.0006536 0.0006536 0.0006536 0.20%
MLCellLinOp::prepareForSolve() 6 0.0006046 0.0006046 0.0006046 0.18%
MultiFab::contains_nan() 10 0.0005893 0.0005893 0.0005893 0.18%
Castro::enforce_speed_limit() 30 0.0005787 0.0005787 0.0005787 0.17%
MLCellLinOp::compGrad() 6 0.0004897 0.0004897 0.0004897 0.15%
MLMG::prepareForSolve() 6 0.000485 0.000485 0.000485 0.14%
MLCellLinOp::smooth() 720 0.0004633 0.0004633 0.0004633 0.14%
Amr::InitAmr() 1 0.0004255 0.0004255 0.0004255 0.13%
FabArrayBase::CPC::define() 244 0.0004235 0.0004235 0.0004235 0.13%
FabArrayBase::getCPC() 632 0.000368 0.000368 0.000368 0.11%
FabArray::FillBoundary() 1766 0.0003662 0.0003662 0.0003662 0.11%
Gravity::get_old_grav_vector() 5 0.0003097 0.0003097 0.0003097 0.09%
main() 1 0.0002838 0.0002838 0.0002838 0.08%
Gravity::get_new_grav_vector() 5 0.000274 0.000274 0.000274 0.08%
FabArrayBase::getFB() 1766 0.0002572 0.0002572 0.0002572 0.08%
MLCellLinOp::apply() 500 0.0001978 0.0001978 0.0001978 0.06%
MLMG::mgVcycle() 36 0.0001677 0.0001677 0.0001677 0.05%
Amr::coarseTimeStep() 5 0.0001641 0.0001641 0.0001641 0.05%
MultiFab::max() 6 0.0001524 0.0001524 0.0001524 0.05%
MLLinOp::defineGrids() 6 0.0001336 0.0001336 0.0001336 0.04%
MLCGSolver::ParallelAllReduce 659 0.0001258 0.0001258 0.0001258 0.04%
MLCellLinOp::defineBC() 6 0.0001141 0.0001141 0.0001141 0.03%
FillPatchIterator::Initialize 20 0.0001109 0.0001109 0.0001109 0.03%
FabArray::ParallelCopy() 380 0.0001107 0.0001107 0.0001107 0.03%
Castro::subcycle_advance_ctu() 5 0.0001074 0.0001074 0.0001074 0.03%
Castro::create_source_corrector() 5 0.0001029 0.0001029 0.0001029 0.03%
AmrLevel::restart() 1 9.892e-05 9.892e-05 9.892e-05 0.03%
MLCellLinOp::correctionResidual() 216 8.986e-05 8.986e-05 8.986e-05 0.03%
StateData::restartDoit() 4 7.564e-05 7.564e-05 7.564e-05 0.02%
Amr::timeStep() 5 7.536e-05 7.536e-05 7.536e-05 0.02%
Gravity::solve_for_phi() 5 7.027e-05 7.027e-05 7.027e-05 0.02%
Gravity::update_max_rhs() 6 6.046e-05 6.046e-05 6.046e-05 0.02%
FabArrayBase::FB::FB() 26 5.981e-05 5.981e-05 5.981e-05 0.02%
Castro::construct_new_source() 25 5.242e-05 5.242e-05 5.242e-05 0.02%
MLMG::actualBottomSolve() 36 5.048e-05 5.048e-05 5.048e-05 0.02%
MLMG:computeResOfCorrection() 180 4.853e-05 4.853e-05 4.853e-05 0.01%
MLMG::mgVcycle_down::0 36 4.694e-05 4.694e-05 4.694e-05 0.01%
Castro::expand_state() 5 4.184e-05 4.184e-05 4.184e-05 0.01%
Castro::advance() 5 3.54e-05 3.54e-05 3.54e-05 0.01%
Castro::initialize_advance() 5 3.527e-05 3.527e-05 3.527e-05 0.01%
Castro::clean_state() 30 3.52e-05 3.52e-05 3.52e-05 0.01%
MLMG::solve() 6 3.427e-05 3.427e-05 3.427e-05 0.01%
MLMG::mgVcycle_down::1 36 3.366e-05 3.366e-05 3.366e-05 0.01%
Castro::initialize_do_advance() 5 3.328e-05 3.328e-05 3.328e-05 0.01%
MLMG::mgVcycle_down::4 36 3.173e-05 3.173e-05 3.173e-05 0.01%
MLMG::mgVcycle_down::2 36 3.081e-05 3.081e-05 3.081e-05 0.01%
MLMG::mgVcycle_down::3 36 3.059e-05 3.059e-05 3.059e-05 0.01%
MLMG::mgVcycle_up::4 36 2.834e-05 2.834e-05 2.834e-05 0.01%
Castro::finalize_advance() 5 2.829e-05 2.829e-05 2.829e-05 0.01%
Castro::initMFs() 1 2.79e-05 2.79e-05 2.79e-05 0.01%
Castro::buildMetrics() 1 2.748e-05 2.748e-05 2.748e-05 0.01%
Amr::writeSmallPlotFile() 1 2.71e-05 2.71e-05 2.71e-05 0.01%
Castro::swap_state_time_levels() 5 2.414e-05 2.414e-05 2.414e-05 0.01%
MLPoisson::define() 6 2.405e-05 2.405e-05 2.405e-05 0.01%
MLMG::oneIter() 36 2.239e-05 2.239e-05 2.239e-05 0.01%
MLMG::mgVcycle_up::0 36 2.177e-05 2.177e-05 2.177e-05 0.01%
MLCellLinOp::solutionResidual() 42 2.163e-05 2.163e-05 2.163e-05 0.01%
MLMG::mgVcycle_up::3 36 2.14e-05 2.14e-05 2.14e-05 0.01%
MLMG::mgVcycle_up::2 36 2.097e-05 2.097e-05 2.097e-05 0.01%
MLMG::mgVcycle_up::1 36 1.992e-05 1.992e-05 1.992e-05 0.01%
Castro::post_restart() 1 1.973e-05 1.973e-05 1.973e-05 0.01%
Castro::finalize_do_advance() 5 1.96e-05 1.96e-05 1.96e-05 0.01%
MLMG::ResNormInf() 42 1.817e-05 1.817e-05 1.817e-05 0.01%
Gravity::multilevel_solve_for_new_phi() 1 1.613e-05 1.613e-05 1.613e-05 0.00%
MLMG::mgVcycle_bottom 36 1.513e-05 1.513e-05 1.513e-05 0.00%
FillPatchSingleLevel 20 1.445e-05 1.445e-05 1.445e-05 0.00%
makeSFC 30 1.363e-05 1.363e-05 1.363e-05 0.00%
MLMG::computeResidual() 36 1.335e-05 1.335e-05 1.335e-05 0.00%
MLLinOp::define() 6 1.324e-05 1.324e-05 1.324e-05 0.00%
Castro::construct_new_gravity() 5 1.256e-05 1.256e-05 1.256e-05 0.00%
Gravity::solve_phi_with_mlmg() 6 1.244e-05 1.244e-05 1.244e-05 0.00%
Castro::construct_old_source() 25 1.053e-05 1.053e-05 1.053e-05 0.00%
DistributionMapping::Distribute() 31 9.099e-06 9.099e-06 9.099e-06 0.00%
Amr::initSubcycle() 1 8.851e-06 8.851e-06 8.851e-06 0.00%
Castro::do_new_sources() 5 8.552e-06 8.552e-06 8.552e-06 0.00%
Castro::do_old_sources() 5 8.193e-06 8.193e-06 8.193e-06 0.00%
MLLinOp::makeAgglomeratedDMap 6 7.609e-06 7.609e-06 7.609e-06 0.00%
Gravity::actual_multilevel_solve() 1 7.543e-06 7.543e-06 7.543e-06 0.00%
Castro::check_for_nan() 10 6.289e-06 6.289e-06 6.289e-06 0.00%
Castro::apply_source_to_state() 10 5.688e-06 5.688e-06 5.688e-06 0.00%
Castro::construct_old_gravity() 5 5.175e-06 5.175e-06 5.175e-06 0.00%
Castro::post_timestep() 5 4.957e-06 4.957e-06 4.957e-06 0.00%
MLPoisson::prepareForSolve() 6 4.327e-06 4.327e-06 4.327e-06 0.00%
Castro::computeNewDt() 5 3.987e-06 3.987e-06 3.987e-06 0.00%
Gravity::swapTimeLevels() 5 3.928e-06 3.928e-06 3.928e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 3.735e-06 3.735e-06 3.735e-06 0.00%
MLMG::computeMLResidual() 6 3.723e-06 3.723e-06 3.723e-06 0.00%
MLMG::getGradSolution() 6 3.069e-06 3.069e-06 3.069e-06 0.00%
Castro::retry_advance_ctu() 5 2.704e-06 2.704e-06 2.704e-06 0.00%
MLMG::MLResNormInf() 6 2.16e-06 2.16e-06 2.16e-06 0.00%
MLMG::MLRhsNormInf() 6 2.054e-06 2.054e-06 2.054e-06 0.00%
Gravity::set_mass_offset() 6 2.052e-06 2.052e-06 2.052e-06 0.00%
Castro::FluxRegCrseInit 5 1.458e-06 1.458e-06 1.458e-06 0.00%
Amr::init() 1 1.371e-06 1.371e-06 1.371e-06 0.00%
MLLinOp::makeSubCommunicator() 6 1.054e-06 1.054e-06 1.054e-06 0.00%
Castro::FluxRegFineAdd() 5 1.002e-06 1.002e-06 1.002e-06 0.00%
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Name NCalls Incl. Min Incl. Avg Incl. Max Max %
--------------------------------------------------------------------------------------------
main() 1 0.3346 0.3346 0.3346 99.99%
Amr::coarseTimeStep() 5 0.2575 0.2575 0.2575 76.95%
Amr::timeStep() 5 0.2555 0.2555 0.2555 76.37%
Castro::advance() 5 0.2525 0.2525 0.2525 75.47%
Castro::subcycle_advance_ctu() 5 0.2464 0.2464 0.2464 73.64%
Castro::do_advance_ctu() 5 0.2463 0.2463 0.2463 73.60%
Castro::construct_new_gravity() 5 0.1304 0.1304 0.1304 38.97%
Gravity::solve_phi_with_mlmg() 6 0.1258 0.1258 0.1258 37.61%
Gravity::solve_for_phi() 5 0.1227 0.1227 0.1227 36.67%
Gravity::actual_solve_with_mlmg() 6 0.122 0.122 0.122 36.45%
MLMG::solve() 6 0.1107 0.1107 0.1107 33.07%
MLMG::oneIter() 36 0.1038 0.1038 0.1038 31.03%
MLMG::mgVcycle() 36 0.1023 0.1023 0.1023 30.57%
Castro::construct_ctu_hydro_source() 5 0.07769 0.07769 0.07769 23.22%
MLCellLinOp::smooth() 720 0.05004 0.05004 0.05004 14.95%
MLCellLinOp::applyBC() 1946 0.04691 0.04691 0.04691 14.02%
Amr::init() 1 0.04312 0.04312 0.04312 12.89%
Amr::restart() 1 0.04312 0.04312 0.04312 12.89%
AmrLevel::restart() 1 0.03692 0.03692 0.03692 11.04%
StateData::restartDoit() 4 0.03682 0.03682 0.03682 11.00%
VisMF::Read() 3 0.03669 0.03669 0.03669 10.97%
Amr::writePlotFile() 1 0.03324 0.03324 0.03324 9.93%
MLMG::mgVcycle_bottom 36 0.03183 0.03183 0.03183 9.51%
MLMG::actualBottomSolve() 36 0.03182 0.03182 0.03182 9.51%
VisMF::Write(FabArray) 1 0.03152 0.03152 0.03152 9.42%
MLCGSolver::bicgstab 36 0.03149 0.03149 0.03149 9.41%
FillPatchIterator::Initialize 20 0.02042 0.02042 0.02042 6.10%
Castro::clean_state() 30 0.01976 0.01976 0.01976 5.90%
FillPatchSingleLevel 20 0.01965 0.01965 0.01965 5.87%
StateDataPhysBCFunct::() 20 0.01759 0.01759 0.01759 5.26%
MLCellLinOp::apply() 500 0.01572 0.01572 0.01572 4.70%
MLMG::mgVcycle_down::0 36 0.01428 0.01428 0.01428 4.27%
MLPoisson::Fsmooth() 1440 0.01396 0.01396 0.01396 4.17%
FabArray::FillBoundary() 1766 0.01344 0.01344 0.01344 4.02%
FillBoundary_nowait() 1766 0.01307 0.01307 0.01307 3.91%
StateData::FillBoundary(geom) 160 0.01206 0.01206 0.01206 3.60%
MLMG::mgVcycle_up::0 36 0.01079 0.01079 0.01079 3.22%
MLCellLinOp::correctionResidual() 216 0.009511 0.009511 0.009511 2.84%
Castro::initialize_do_advance() 5 0.009465 0.009465 0.009465 2.83%
MLPoisson::define() 6 0.009003 0.009003 0.009003 2.69%
amrex::Dot() 484 0.008778 0.008778 0.008778 2.62%
Castro::computeTemp() 30 0.008554 0.008554 0.008554 2.56%
MLMG:computeResOfCorrection() 180 0.008359 0.008359 0.008359 2.50%
Gravity::get_new_grav_vector() 5 0.0076 0.0076 0.0076 2.27%
Castro::construct_old_gravity() 5 0.007517 0.007517 0.007517 2.25%
Gravity::get_old_grav_vector() 5 0.007512 0.007512 0.007512 2.25%
amrex::Copy() 463 0.00697 0.00697 0.00697 2.08%
MLMG::mgVcycle_down::1 36 0.006824 0.006824 0.006824 2.04%
Castro::do_new_sources() 5 0.006723 0.006723 0.006723 2.01%
Castro::normalize_species() 30 0.006618 0.006618 0.006618 1.98%
FabArray::ParallelCopy() 380 0.006575 0.006575 0.006575 1.96%
FabArray::ParallelCopy_nowait() 380 0.006464 0.006464 0.006464 1.93%
MLMG::mgVcycle_down::2 36 0.00637 0.00637 0.00637 1.90%
FabArray::setVal() 537 0.006271 0.006271 0.006271 1.87%
MLMG::mgVcycle_down::3 36 0.006264 0.006264 0.006264 1.87%
FabArray::norminf() 326 0.006236 0.006236 0.006236 1.86%
MLMG::mgVcycle_down::4 36 0.00622 0.00622 0.00622 1.86%
MLCellLinOp::defineAuxData() 6 0.006048 0.006048 0.006048 1.81%
Castro::expand_state() 5 0.005886 0.005886 0.005886 1.76%
Castro::initialize_advance() 5 0.005792 0.005792 0.005792 1.73%
MLCGSolver::ParallelAllReduce 659 0.005288 0.005288 0.005288 1.58%
MLMG::addInterpCorrection() 180 0.005151 0.005151 0.005151 1.54%
MLMG::mgVcycle_up::4 36 0.00501 0.00501 0.00501 1.50%
MLMG::mgVcycle_up::1 36 0.004946 0.004946 0.004946 1.48%
amrex::average_down 180 0.004878 0.004878 0.004878 1.46%
MLMG::mgVcycle_up::2 36 0.004831 0.004831 0.004831 1.44%
MLMG::mgVcycle_up::3 36 0.004748 0.004748 0.004748 1.42%
Castro::do_old_sources() 5 0.004669 0.004669 0.004669 1.40%
MLPoisson::Fapply() 500 0.004481 0.004481 0.004481 1.34%
Castro::enforce_min_density() 30 0.00397 0.00397 0.00397 1.19%
Gravity::fill_multipole_BCs() 6 0.003742 0.003742 0.003742 1.12%
FabArray::Saxpy() 355 0.003668 0.003668 0.003668 1.10%
FabArray::Xpay() 361 0.003542 0.003542 0.003542 1.06%
Castro::post_restart() 1 0.003503 0.003503 0.003503 1.05%
Gravity::multilevel_solve_for_new_phi() 1 0.003394 0.003394 0.003394 1.01%
Gravity::actual_multilevel_solve() 1 0.003378 0.003378 0.003378 1.01%
MLCellLinOp::solutionResidual() 42 0.003226 0.003226 0.003226 0.96%
Castro::reset_internal_energy(MultiFab) 30 0.003177 0.003177 0.003177 0.95%
Castro::estTimeStep() 10 0.003053 0.003053 0.003053 0.91%
Castro::post_timestep() 5 0.002935 0.002935 0.002935 0.88%
MLCellLinOp::defineBC() 6 0.002755 0.002755 0.002755 0.82%
MLMG::computeResidual() 36 0.002665 0.002665 0.002665 0.80%
BndryData::define() 6 0.00264 0.00264 0.00264 0.79%
MLMG::prepareForSolve() 6 0.002517 0.002517 0.002517 0.75%
FabArray::LinComb() 242 0.00198 0.00198 0.00198 0.59%
amrex::Add() 72 0.00185 0.00185 0.00185 0.55%
Castro::computeNewDt() 5 0.00179 0.00179 0.00179 0.53%
Castro::construct_new_source() 25 0.001772 0.001772 0.001772 0.53%
Castro::construct_new_gravity_source() 5 0.001719 0.001719 0.001719 0.51%
Castro::construct_old_source() 25 0.001392 0.001392 0.001392 0.42%
Castro::construct_old_gravity_source() 5 0.001382 0.001382 0.001382 0.41%
Castro::apply_source_to_state() 10 0.0009204 0.0009204 0.0009204 0.28%
MLMG::ResNormInf() 42 0.0009171 0.0009171 0.0009171 0.27%
FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009145 0.0009145 0.0009145 0.27%
FabArrayBase::getCPC() 632 0.0007915 0.0007915 0.0007915 0.24%
MLCellLinOp::setLevelBC() 6 0.000771 0.000771 0.000771 0.23%
MLMG::getGradSolution() 6 0.0007544 0.0007544 0.0007544 0.23%
MLCellLinOp::compGrad() 6 0.0007513 0.0007513 0.0007513 0.22%
Castro::reset_internal_energy(Fab) 240 0.0007225 0.0007225 0.0007225 0.22%
FabArray::setDomainBndry() 20 0.0006587 0.0006587 0.0006587 0.20%
FabArray::mult() 22 0.0006536 0.0006536 0.0006536 0.20%
MLPoisson::prepareForSolve() 6 0.0006089 0.0006089 0.0006089 0.18%
MLCellLinOp::prepareForSolve() 6 0.0006046 0.0006046 0.0006046 0.18%
Castro::check_for_nan() 10 0.0005956 0.0005956 0.0005956 0.18%
MultiFab::contains_nan() 10 0.0005893 0.0005893 0.0005893 0.18%
Castro::enforce_speed_limit() 30 0.0005787 0.0005787 0.0005787 0.17%
MLMG::computeMLResidual() 6 0.0005776 0.0005776 0.0005776 0.17%
Gravity::update_max_rhs() 6 0.0004656 0.0004656 0.0004656 0.14%
Amr::InitAmr() 1 0.0004343 0.0004343 0.0004343 0.13%
FabArrayBase::CPC::define() 244 0.0004235 0.0004235 0.0004235 0.13%
FabArrayBase::getFB() 1766 0.000317 0.000317 0.000317 0.09%
Castro::finalize_advance() 5 0.0002912 0.0002912 0.0002912 0.09%
Gravity::swapTimeLevels() 5 0.0002263 0.0002263 0.0002263 0.07%
MLLinOp::define() 6 0.0001767 0.0001767 0.0001767 0.05%
MLLinOp::defineGrids() 6 0.0001635 0.0001635 0.0001635 0.05%
MultiFab::max() 6 0.0001524 0.0001524 0.0001524 0.05%
MLMG::MLResNormInf() 6 0.0001509 0.0001509 0.0001509 0.05%
Castro::buildMetrics() 1 0.0001497 0.0001497 0.0001497 0.04%
MLMG::MLRhsNormInf() 6 0.0001162 0.0001162 0.0001162 0.03%
Castro::create_source_corrector() 5 0.0001029 0.0001029 0.0001029 0.03%
FabArrayBase::FB::FB() 26 5.981e-05 5.981e-05 5.981e-05 0.02%
MLLinOp::makeAgglomeratedDMap 6 2.884e-05 2.884e-05 2.884e-05 0.01%
Castro::initMFs() 1 2.79e-05 2.79e-05 2.79e-05 0.01%
Amr::writeSmallPlotFile() 1 2.71e-05 2.71e-05 2.71e-05 0.01%
Castro::swap_state_time_levels() 5 2.414e-05 2.414e-05 2.414e-05 0.01%
makeSFC 30 2.123e-05 2.123e-05 2.123e-05 0.01%
Castro::finalize_do_advance() 5 1.96e-05 1.96e-05 1.96e-05 0.01%
DistributionMapping::Distribute() 31 9.099e-06 9.099e-06 9.099e-06 0.00%
Amr::initSubcycle() 1 8.851e-06 8.851e-06 8.851e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 5.231e-06 5.231e-06 5.231e-06 0.00%
Castro::retry_advance_ctu() 5 2.704e-06 2.704e-06 2.704e-06 0.00%
Gravity::set_mass_offset() 6 2.052e-06 2.052e-06 2.052e-06 0.00%
Castro::FluxRegCrseInit 5 1.458e-06 1.458e-06 1.458e-06 0.00%
MLLinOp::makeSubCommunicator() 6 1.054e-06 1.054e-06 1.054e-06 0.00%
Castro::FluxRegFineAdd() 5 1.002e-06 1.002e-06 1.002e-06 0.00%
--------------------------------------------------------------------------------------------
Unused ParmParse Variables:
[TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2]
[TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2]
Device Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Arena::Initialize() 1 1 237 MiB 9042 MiB
Castro::construct_ctu_hydro_source() 1440 1440 104 MiB 692 MiB
Castro::initMFs() 48 48 59 MiB 68 MiB
StateData::restartDoit() 32 32 53 MiB 55 MiB
Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB
Castro::initialize_do_advance() 40 40 28 MiB 39 MiB
FillPatchIterator::Initialize 160 160 1341 KiB 39 MiB
ResizeRandomSeed 1 1 30 MiB 30 MiB
Amr::writePlotFile() 8 8 2773 KiB 28 MiB
Castro::initialize_advance() 40 40 17 MiB 23 MiB
Castro::buildMetrics() 32 32 13 MiB 15 MiB
Castro::post_restart() 48 48 6661 KiB 14 MiB
MLMG::prepareForSolve() 361 361 4060 KiB 12 MiB
Gravity::get_old_grav_vector() 43 43 232 KiB 10 MiB
Gravity::get_new_grav_vector() 40 40 232 KiB 10 MiB
Gravity::multilevel_solve_for_new_phi() 24 24 6656 KiB 7586 KiB
Gravity::fill_multipole_BCs() 84 84 17 KiB 2053 KiB
Gravity::update_max_rhs() 48 48 2700 B 2048 KiB
Gravity::solve_for_phi() 40 40 749 KiB 2048 KiB
Gravity::actual_multilevel_solve() 8 8 20 KiB 2048 KiB
BndryData::define() 576 576 373 KiB 1095 KiB
MLCellLinOp::defineAuxData() 936 936 239 KiB 671 KiB
Castro::estTimeStep() 10 10 4264 B 480 KiB
VisMF::Write(FabArray) 112 112 1473 B 320 KiB
Castro::normalize_species() 30 30 6425 B 320 KiB
amrex::average_down 469 469 1378 B 257 KiB
MLMG::addInterpCorrection() 468 468 1312 B 257 KiB
amrex::Dot() 592 592 3860 B 160 KiB
FabArray::norminf() 398 398 2749 B 160 KiB
Castro::do_advance_ctu() 5 5 683 B 160 KiB
MultiFab::max() 6 6 70 B 160 KiB
FabArray::setVal() 66 66 18 KiB 23 KiB
MultiFab::contains_nan() 10 10 35 B 20 KiB
MLPoisson::Fsmooth() 60 60 3949 B 12 KiB
FabArray::ParallelCopy_nowait() 380 380 58 B 10 KiB
FillBoundary_nowait() 336 336 330 B 9648 B
MLCellLinOp::applyBC() 3892 3892 258 B 9344 B
amrex::Copy() 56 56 5978 B 8464 B
MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B
StateData::FillBoundary(geom) 960 960 59 B 2880 B
FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B
MLCGSolver::bicgstab 324 324 136 B 1472 B
MLCellLinOp::defineBC() 36 36 419 B 1248 B
------------------------------------------------------------------------------
Managed Memory Usage:
------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------
The_Managed_Arena::Initialize() 1 1 1543 B 8192 KiB
------------------------------------------------------------------
Pinned Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Pinned_Arena::Initialize() 1 1 126 KiB 8192 KiB
VisMF::Write(FabArray) 120 120 290 KiB 3584 KiB
VisMF::Read() 24 24 215 KiB 3000 KiB
FabArray::setVal() 66 66 18 KiB 23 KiB
MLPoisson::Fsmooth() 60 60 3949 B 12 KiB
FabArray::ParallelCopy_nowait() 380 380 58 B 10 KiB
FillBoundary_nowait() 336 336 330 B 9648 B
MLCellLinOp::applyBC() 1946 1946 256 B 9328 B
amrex::Copy() 56 56 5978 B 8464 B
MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B
Gravity::get_old_grav_vector() 3 3 2618 B 3072 B
StateData::FillBoundary(geom) 960 960 59 B 2880 B
Gravity::fill_multipole_BCs() 18 18 6 B 2832 B
FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B
MLMG::prepareForSolve() 7 7 575 B 1296 B
amrex::average_down 37 37 219 B 1296 B
MLMG::addInterpCorrection() 36 36 3 B 1024 B
MLCellLinOp::setLevelBC() 36 36 0 B 768 B
amrex::Dot() 592 592 28 B 400 B
FabArray::norminf() 398 398 20 B 272 B
Castro::estTimeStep() 10 10 0 B 32 B
MultiFab::max() 6 6 0 B 16 B
MultiFab::contains_nan() 10 10 0 B 16 B
Castro::do_advance_ctu() 5 5 0 B 16 B
Castro::normalize_species() 30 30 0 B 16 B
------------------------------------------------------------------------------
Total GPU global memory (MB): 12056
Free GPU global memory (MB): 2424
[The Arena] space allocated (MB): 9042
[The Arena] space used (MB): 0
[The Managed Arena] space allocated (MB): 8
[The Managed Arena] space used (MB): 0
[The Pinned Arena] space allocated (MB): 8
[The Pinned Arena] space used (MB): 0
AMReX (23.05-31-g2e1106e246c4) finalized