Initializing CUDA...
CUDA initialized with 1 device.
AMReX (23.04-9-g1e73fa8ec7c3) initialized
Starting run at 09:01:23 UTC on 2023-04-05.
Successfully read inputs file ...
Castro git describe: 23.04-2-gabeccccb9
AMReX git describe: 23.04-9-g1e73fa8ec
Microphysics git describe: 23.04-3-gdc41ef4e
reading extern runtime parameters ...
3 Species:
C12 O16 Mg24
Successfully read inputs file ...
INITIAL GRIDS
Level 0 8 grids 262144 cells 100 % of domain
smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32
CHECKPOINT: file = dustcollapse-restart_chk00000
checkPoint() time = 0.058429388 secs.
PLOTFILE: file = dustcollapse-restart_plt00000
Write plotfile time = 0.033848947 seconds
[Level 0 step 1] ADVANCE with dt = 4.541742215e-05
[Level 0 step 1] Advanced 262144 cells
[STEP 1] Coarse TimeStep time: 0.048488268
[STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05
[Level 0 step 2] ADVANCE with dt = 4.768829326e-05
[Level 0 step 2] Advanced 262144 cells
[STEP 2] Coarse TimeStep time: 0.050615748
[STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05
[Level 0 step 3] ADVANCE with dt = 5.007270792e-05
[Level 0 step 3] Advanced 262144 cells
[STEP 3] Coarse TimeStep time: 0.062226989
[STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05
[Level 0 step 4] ADVANCE with dt = 5.257634331e-05
[Level 0 step 4] Advanced 262144 cells
[STEP 4] Coarse TimeStep time: 0.066465316
[STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05
[Level 0 step 5] ADVANCE with dt = 5.520516048e-05
[Level 0 step 5] Advanced 262144 cells
[STEP 5] Coarse TimeStep time: 0.067817584
[STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05
CHECKPOINT: file = dustcollapse-restart_chk00005
checkPoint() time = 0.057944473 secs.
[Level 0 step 6] ADVANCE with dt = 5.79654185e-05
[Level 0 step 6] Advanced 262144 cells
[STEP 6] Coarse TimeStep time: 0.07052608
[STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05
[Level 0 step 7] ADVANCE with dt = 6.086368943e-05
[Level 0 step 7] Advanced 262144 cells
[STEP 7] Coarse TimeStep time: 0.056236577
[STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05
[Level 0 step 8] ADVANCE with dt = 6.39068739e-05
[Level 0 step 8] Advanced 262144 cells
[STEP 8] Coarse TimeStep time: 0.056563982
[STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05
[Level 0 step 9] ADVANCE with dt = 6.71022176e-05
[Level 0 step 9] Advanced 262144 cells
[STEP 9] Coarse TimeStep time: 0.059066316
[STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05
[Level 0 step 10] ADVANCE with dt = 7.045732848e-05
[Level 0 step 10] Advanced 262144 cells
[STEP 10] Coarse TimeStep time: 0.06869387
[STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05
CHECKPOINT: file = dustcollapse-restart_chk00010
checkPoint() time = 0.058685127 secs.
PLOTFILE: file = dustcollapse-restart_plt00010
Write plotfile time = 0.033658649 seconds
Ending run at 09:01:24 UTC on 2023-04-05.
Run time = 0.899339749
Run time without initialization = 0.75762449
Average number of zones advanced per microsecond: 3.460
Average number of zones advanced per microsecond per rank: 3.460
CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472
TinyProfiler total time across processes [min...avg...max]: 0.8994 ... 0.8994 ... 0.8994
--------------------------------------------------------------------------------------------
Name NCalls Excl. Min Excl. Avg Excl. Max Max %
--------------------------------------------------------------------------------------------
Castro::construct_ctu_hydro_source() 10 0.2428 0.2428 0.2428 27.00%
VisMF::Write(FabArray) 11 0.2358 0.2358 0.2358 26.22%
MLCellLinOp::applyBC() 4433 0.0768 0.0768 0.0768 8.54%
MLPoisson::Fsmooth() 3280 0.03254 0.03254 0.03254 3.62%
FillBoundary_nowait() 4023 0.02972 0.02972 0.02972 3.30%
StateData::FillBoundary(geom) 328 0.02491 0.02491 0.02491 2.77%
amrex::Dot() 1114 0.02073 0.02073 0.02073 2.31%
Castro::computeTemp() 63 0.01504 0.01504 0.01504 1.67%
amrex::Copy() 1029 0.01486 0.01486 0.01486 1.65%
FabArray::norminf() 743 0.01445 0.01445 0.01445 1.61%
Castro::normalize_species() 62 0.01422 0.01422 0.01422 1.58%
Castro::enforce_min_density() 62 0.01416 0.01416 0.01416 1.57%
FabArray::ParallelCopy_nowait() 861 0.0136 0.0136 0.0136 1.51%
FabArray::setVal() 1144 0.01321 0.01321 0.01321 1.47%
StateDataPhysBCFunct::() 41 0.01263 0.01263 0.01263 1.40%
MLPoisson::Fapply() 1142 0.01038 0.01038 0.01038 1.15%
MLCellLinOp::defineAuxData() 11 0.009839 0.009839 0.009839 1.09%
FabArray::Saxpy() 813 0.008218 0.008218 0.008218 0.91%
FabArray::Xpay() 821 0.008072 0.008072 0.008072 0.90%
MLMG::addInterpCorrection() 410 0.006783 0.006783 0.006783 0.75%
Gravity::fill_multipole_BCs() 11 0.006752 0.006752 0.006752 0.75%
amrex::average_down 410 0.005984 0.005984 0.005984 0.67%
Castro::estTimeStep() 21 0.005063 0.005063 0.005063 0.56%
FabArray::LinComb() 557 0.004529 0.004529 0.004529 0.50%
amrex::Add() 164 0.004337 0.004337 0.004337 0.48%
Castro::reset_internal_energy(MultiFab) 63 0.0043 0.0043 0.0043 0.48%
Amr::checkPoint() 3 0.003942 0.003942 0.003942 0.44%
BndryData::define() 11 0.003758 0.003758 0.003758 0.42%
Castro::do_advance_ctu() 10 0.003685 0.003685 0.003685 0.41%
Castro::construct_new_gravity_source() 10 0.003348 0.003348 0.003348 0.37%
Castro::construct_old_gravity_source() 10 0.002735 0.002735 0.002735 0.30%
Amr::writePlotFile() 2 0.002365 0.002365 0.002365 0.26%
MLCGSolver::bicgstab 82 0.002136 0.002136 0.002136 0.24%
FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001708 0.001708 0.001708 0.19%
Gravity::actual_solve_with_mlmg() 11 0.001471 0.001471 0.001471 0.16%
Castro::reset_internal_energy(Fab) 504 0.001452 0.001452 0.001452 0.16%
MLCellLinOp::setLevelBC() 11 0.001384 0.001384 0.001384 0.15%
Castro::initData() 1 0.00133 0.00133 0.00133 0.15%
FabArray::mult() 43 0.001313 0.001313 0.001313 0.15%
FabArray::setDomainBndry() 41 0.001298 0.001298 0.001298 0.14%
MultiFab::contains_nan() 20 0.001202 0.001202 0.001202 0.13%
MLCellLinOp::prepareForSolve() 11 0.001128 0.001128 0.001128 0.13%
MLCellLinOp::smooth() 1640 0.001086 0.001086 0.001086 0.12%
Castro::enforce_speed_limit() 62 0.0009557 0.0009557 0.0009557 0.11%
MLCellLinOp::compGrad() 11 0.00092 0.00092 0.00092 0.10%
MLMG::prepareForSolve() 11 0.0008802 0.0008802 0.0008802 0.10%
FabArray::FillBoundary() 4023 0.0008699 0.0008699 0.0008699 0.10%
FabArrayBase::getCPC() 1323 0.0007505 0.0007505 0.0007505 0.08%
FabArrayBase::CPC::define() 454 0.0006734 0.0006734 0.0006734 0.07%
Gravity::get_new_grav_vector() 11 0.0006382 0.0006382 0.0006382 0.07%
FabArrayBase::getFB() 4023 0.0006328 0.0006328 0.0006328 0.07%
Amr::InitAmr() 1 0.000548 0.000548 0.000548 0.06%
Gravity::get_old_grav_vector() 10 0.0005453 0.0005453 0.0005453 0.06%
MLCellLinOp::apply() 1142 0.0004661 0.0004661 0.0004661 0.05%
MLLinOp::defineGrids() 11 0.0003959 0.0003959 0.0003959 0.04%
MLMG::mgVcycle() 82 0.0003849 0.0003849 0.0003849 0.04%
Amr::coarseTimeStep() 10 0.0003269 0.0003269 0.0003269 0.04%
main() 1 0.0003078 0.0003078 0.0003078 0.03%
MLCGSolver::ParallelAllReduce 1514 0.0002925 0.0002925 0.0002925 0.03%
FabArray::ParallelCopy() 861 0.0002642 0.0002642 0.0002642 0.03%
MultiFab::max() 11 0.0002563 0.0002563 0.0002563 0.03%
FillPatchIterator::Initialize 41 0.0002519 0.0002519 0.0002519 0.03%
MLCellLinOp::defineBC() 11 0.0002205 0.0002205 0.0002205 0.02%
MLCellLinOp::correctionResidual() 492 0.0002186 0.0002186 0.0002186 0.02%
Castro::create_source_corrector() 10 0.0001937 0.0001937 0.0001937 0.02%
Amr::timeStep() 10 0.0001578 0.0001578 0.0001578 0.02%
Castro::subcycle_advance_ctu() 10 0.0001487 0.0001487 0.0001487 0.02%
StateData::checkPoint() 12 0.0001335 0.0001335 0.0001335 0.01%
Gravity::update_max_rhs() 11 0.0001229 0.0001229 0.0001229 0.01%
MLMG:computeResOfCorrection() 410 0.0001149 0.0001149 0.0001149 0.01%
Gravity::solve_for_phi() 10 0.0001136 0.0001136 0.0001136 0.01%
Castro::Castro() 1 0.0001035 0.0001035 0.0001035 0.01%
MLMG::mgVcycle_down::0 82 9.342e-05 9.342e-05 9.342e-05 0.01%
FabArrayBase::FB::FB() 56 8.99e-05 8.99e-05 8.99e-05 0.01%
MLMG::actualBottomSolve() 82 8.603e-05 8.603e-05 8.603e-05 0.01%
MLMG::mgVcycle_down::1 82 8.402e-05 8.402e-05 8.402e-05 0.01%
MLMG::mgVcycle_down::2 82 8.354e-05 8.354e-05 8.354e-05 0.01%
MLMG::mgVcycle_down::4 82 8.068e-05 8.068e-05 8.068e-05 0.01%
MLMG::mgVcycle_down::3 82 7.739e-05 7.739e-05 7.739e-05 0.01%
AmrLevel::checkPoint() 3 7.671e-05 7.671e-05 7.671e-05 0.01%
Castro::expand_state() 10 7.656e-05 7.656e-05 7.656e-05 0.01%
MLMG::solve() 11 7.265e-05 7.265e-05 7.265e-05 0.01%
Castro::initialize_advance() 10 7.263e-05 7.263e-05 7.263e-05 0.01%
Castro::clean_state() 62 7.076e-05 7.076e-05 7.076e-05 0.01%
Castro::advance() 10 6.701e-05 6.701e-05 6.701e-05 0.01%
Castro::finalize_advance() 10 6.403e-05 6.403e-05 6.403e-05 0.01%
MLMG::mgVcycle_up::4 82 6.184e-05 6.184e-05 6.184e-05 0.01%
MLMG::mgVcycle_up::0 82 5.465e-05 5.465e-05 5.465e-05 0.01%
MLMG::oneIter() 82 5.443e-05 5.443e-05 5.443e-05 0.01%
MLCellLinOp::solutionResidual() 93 5.352e-05 5.352e-05 5.352e-05 0.01%
MLMG::mgVcycle_up::3 82 5.048e-05 5.048e-05 5.048e-05 0.01%
MLMG::mgVcycle_up::1 82 4.984e-05 4.984e-05 4.984e-05 0.01%
Castro::initialize_do_advance() 10 4.964e-05 4.964e-05 4.964e-05 0.01%
MLMG::mgVcycle_up::2 82 4.802e-05 4.802e-05 4.802e-05 0.01%
Amr::writeSmallPlotFile() 1 4.607e-05 4.607e-05 4.607e-05 0.01%
Castro::construct_new_source() 50 4.312e-05 4.312e-05 4.312e-05 0.00%
FillPatchSingleLevel 41 4.238e-05 4.238e-05 4.238e-05 0.00%
Castro::finalize_do_advance() 10 3.753e-05 3.753e-05 3.753e-05 0.00%
Castro::swap_state_time_levels() 10 3.685e-05 3.685e-05 3.685e-05 0.00%
StateData::define() 4 3.598e-05 3.598e-05 3.598e-05 0.00%
MLMG::ResNormInf() 93 3.582e-05 3.582e-05 3.582e-05 0.00%
Castro::enforce_consistent_e() 1 3.492e-05 3.492e-05 3.492e-05 0.00%
MLMG::mgVcycle_bottom 82 3.299e-05 3.299e-05 3.299e-05 0.00%
MLMG::computeResidual() 82 2.988e-05 2.988e-05 2.988e-05 0.00%
Castro::initMFs() 1 2.627e-05 2.627e-05 2.627e-05 0.00%
Castro::construct_new_gravity() 10 2.487e-05 2.487e-05 2.487e-05 0.00%
makeSFC 55 2.349e-05 2.349e-05 2.349e-05 0.00%
Gravity::solve_phi_with_mlmg() 11 2.289e-05 2.289e-05 2.289e-05 0.00%
Amr::FinalizeInit() 1 2.191e-05 2.191e-05 2.191e-05 0.00%
Amr::defBaseLevel() 1 2.188e-05 2.188e-05 2.188e-05 0.00%
MLPoisson::define() 11 2.175e-05 2.175e-05 2.175e-05 0.00%
Castro::buildMetrics() 1 1.934e-05 1.934e-05 1.934e-05 0.00%
Castro::construct_old_source() 50 1.795e-05 1.795e-05 1.795e-05 0.00%
Castro::do_new_sources() 10 1.628e-05 1.628e-05 1.628e-05 0.00%
Gravity::multilevel_solve_for_new_phi() 1 1.612e-05 1.612e-05 1.612e-05 0.00%
Castro::do_old_sources() 10 1.565e-05 1.565e-05 1.565e-05 0.00%
DistributionMapping::Distribute() 56 1.529e-05 1.529e-05 1.529e-05 0.00%
MLLinOp::makeAgglomeratedDMap 11 1.412e-05 1.412e-05 1.412e-05 0.00%
MLPoisson::prepareForSolve() 11 1.399e-05 1.399e-05 1.399e-05 0.00%
MLMG::computeMLResidual() 11 1.315e-05 1.315e-05 1.315e-05 0.00%
MLLinOp::define() 11 1.107e-05 1.107e-05 1.107e-05 0.00%
Castro::check_for_nan() 20 1.092e-05 1.092e-05 1.092e-05 0.00%
AmrLevel::AmrLevel(dm) 1 1.086e-05 1.086e-05 1.086e-05 0.00%
Castro::apply_source_to_state() 20 1.073e-05 1.073e-05 1.073e-05 0.00%
Gravity::swapTimeLevels() 10 9.51e-06 9.51e-06 9.51e-06 0.00%
Amr::initSubcycle() 1 9.319e-06 9.319e-06 9.319e-06 0.00%
Castro::construct_old_gravity() 10 9.208e-06 9.208e-06 9.208e-06 0.00%
Gravity::actual_multilevel_solve() 1 8.859e-06 8.859e-06 8.859e-06 0.00%
Castro::post_timestep() 10 7.937e-06 7.937e-06 7.937e-06 0.00%
Castro::computeNewDt() 9 7.695e-06 7.695e-06 7.695e-06 0.00%
MLMG::getGradSolution() 11 5.753e-06 5.753e-06 5.753e-06 0.00%
Amr::InitializeInit() 1 5.223e-06 5.223e-06 5.223e-06 0.00%
Castro::post_init() 1 4.831e-06 4.831e-06 4.831e-06 0.00%
AmrLevel::checkPointPost() 3 4.707e-06 4.707e-06 4.707e-06 0.00%
MLMG::MLRhsNormInf() 11 3.918e-06 3.918e-06 3.918e-06 0.00%
Castro::retry_advance_ctu() 10 3.799e-06 3.799e-06 3.799e-06 0.00%
Gravity::set_mass_offset() 11 3.776e-06 3.776e-06 3.776e-06 0.00%
MLMG::MLResNormInf() 11 3.616e-06 3.616e-06 3.616e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 3.539e-06 3.539e-06 3.539e-06 0.00%
Castro::computeInitialDt() 2 3.374e-06 3.374e-06 3.374e-06 0.00%
Castro::FluxRegCrseInit 10 2.988e-06 2.988e-06 2.988e-06 0.00%
Amr::init() 1 2.902e-06 2.902e-06 2.902e-06 0.00%
AmrLevel::checkPointPre() 3 2.065e-06 2.065e-06 2.065e-06 0.00%
Castro::FluxRegFineAdd() 10 2.016e-06 2.016e-06 2.016e-06 0.00%
MLLinOp::makeSubCommunicator() 11 1.934e-06 1.934e-06 1.934e-06 0.00%
Castro::post_regrid() 1 1.25e-06 1.25e-06 1.25e-06 0.00%
Amr::initialInit() 1 1.201e-06 1.201e-06 1.201e-06 0.00%
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Name NCalls Incl. Min Incl. Avg Incl. Max Max %
--------------------------------------------------------------------------------------------
main() 1 0.8994 0.8994 0.8994 100.00%
Amr::coarseTimeStep() 10 0.7237 0.7237 0.7237 80.47%
Amr::timeStep() 10 0.6037 0.6037 0.6037 67.12%
Castro::advance() 10 0.5956 0.5956 0.5956 66.23%
Castro::subcycle_advance_ctu() 10 0.5827 0.5827 0.5827 64.79%
Castro::do_advance_ctu() 10 0.5826 0.5826 0.5826 64.78%
Gravity::solve_phi_with_mlmg() 11 0.2835 0.2835 0.2835 31.52%
Gravity::actual_solve_with_mlmg() 11 0.2763 0.2763 0.2763 30.72%
Castro::construct_new_gravity() 10 0.2587 0.2587 0.2587 28.77%
MLMG::solve() 11 0.2554 0.2554 0.2554 28.40%
Gravity::solve_for_phi() 10 0.243 0.243 0.243 27.02%
Castro::construct_ctu_hydro_source() 10 0.2428 0.2428 0.2428 27.00%
MLMG::oneIter() 82 0.2411 0.2411 0.2411 26.81%
MLMG::mgVcycle() 82 0.2375 0.2375 0.2375 26.41%
VisMF::Write(FabArray) 11 0.2358 0.2358 0.2358 26.22%
Amr::checkPoint() 3 0.1752 0.1752 0.1752 19.48%
AmrLevel::checkPoint() 3 0.1712 0.1712 0.1712 19.04%
StateData::checkPoint() 12 0.1712 0.1712 0.1712 19.03%
Amr::init() 1 0.141 0.141 0.141 15.68%
MLCellLinOp::smooth() 1640 0.116 0.116 0.116 12.89%
MLCellLinOp::applyBC() 4433 0.1081 0.1081 0.1081 12.02%
MLMG::mgVcycle_bottom 82 0.07402 0.07402 0.07402 8.23%
MLMG::actualBottomSolve() 82 0.07398 0.07398 0.07398 8.23%
MLCGSolver::bicgstab 82 0.07328 0.07328 0.07328 8.15%
Amr::writePlotFile() 2 0.06764 0.06764 0.06764 7.52%
Castro::clean_state() 62 0.04925 0.04925 0.04925 5.48%
Amr::initialInit() 1 0.0486 0.0486 0.0486 5.40%
Amr::FinalizeInit() 1 0.04425 0.04425 0.04425 4.92%
FillPatchIterator::Initialize 41 0.0432 0.0432 0.0432 4.80%
Castro::post_init() 1 0.0429 0.0429 0.0429 4.77%
FillPatchSingleLevel 41 0.04165 0.04165 0.04165 4.63%
Gravity::multilevel_solve_for_new_phi() 1 0.04091 0.04091 0.04091 4.55%
Gravity::actual_multilevel_solve() 1 0.0409 0.0409 0.0409 4.55%
StateDataPhysBCFunct::() 41 0.03755 0.03755 0.03755 4.17%
MLCellLinOp::apply() 1142 0.03614 0.03614 0.03614 4.02%
MLMG::mgVcycle_down::0 82 0.0331 0.0331 0.0331 3.68%
MLPoisson::Fsmooth() 3280 0.03254 0.03254 0.03254 3.62%
FabArray::FillBoundary() 4023 0.03131 0.03131 0.03131 3.48%
FillBoundary_nowait() 4023 0.03044 0.03044 0.03044 3.38%
MLMG::mgVcycle_up::0 82 0.02507 0.02507 0.02507 2.79%
StateData::FillBoundary(geom) 328 0.02491 0.02491 0.02491 2.77%
MLCellLinOp::correctionResidual() 492 0.02208 0.02208 0.02208 2.46%
Castro::computeTemp() 63 0.02079 0.02079 0.02079 2.31%
amrex::Dot() 1114 0.02073 0.02073 0.02073 2.31%
Castro::initialize_do_advance() 10 0.02026 0.02026 0.02026 2.25%
MLMG:computeResOfCorrection() 410 0.01942 0.01942 0.01942 2.16%
Gravity::get_new_grav_vector() 11 0.01738 0.01738 0.01738 1.93%
MLPoisson::define() 11 0.01663 0.01663 0.01663 1.85%
MLMG::mgVcycle_down::1 82 0.01585 0.01585 0.01585 1.76%
Castro::construct_old_gravity() 10 0.0155 0.0155 0.0155 1.72%
Gravity::get_old_grav_vector() 10 0.0155 0.0155 0.0155 1.72%
amrex::Copy() 1029 0.01486 0.01486 0.01486 1.65%
MLMG::mgVcycle_down::2 82 0.01482 0.01482 0.01482 1.65%
FabArray::ParallelCopy() 861 0.01467 0.01467 0.01467 1.63%
FabArray::norminf() 743 0.01445 0.01445 0.01445 1.61%
MLMG::mgVcycle_down::3 82 0.01443 0.01443 0.01443 1.60%
FabArray::ParallelCopy_nowait() 861 0.0144 0.0144 0.0144 1.60%
MLMG::mgVcycle_down::4 82 0.01434 0.01434 0.01434 1.59%
Castro::normalize_species() 62 0.01422 0.01422 0.01422 1.58%
Castro::enforce_min_density() 62 0.01416 0.01416 0.01416 1.57%
FabArray::setVal() 1144 0.01321 0.01321 0.01321 1.47%
Castro::do_new_sources() 10 0.01264 0.01264 0.01264 1.41%
MLCGSolver::ParallelAllReduce 1514 0.01243 0.01243 0.01243 1.38%
Castro::initialize_advance() 10 0.01224 0.01224 0.01224 1.36%
MLMG::addInterpCorrection() 410 0.01206 0.01206 0.01206 1.34%
MLMG::mgVcycle_up::4 82 0.01162 0.01162 0.01162 1.29%
MLMG::mgVcycle_up::1 82 0.01153 0.01153 0.01153 1.28%
Castro::expand_state() 10 0.01146 0.01146 0.01146 1.27%
amrex::average_down 410 0.01131 0.01131 0.01131 1.26%
MLMG::mgVcycle_up::2 82 0.01128 0.01128 0.01128 1.25%
Castro::do_old_sources() 10 0.01122 0.01122 0.01122 1.25%
MLCellLinOp::defineAuxData() 11 0.01118 0.01118 0.01118 1.24%
MLMG::mgVcycle_up::3 82 0.01106 0.01106 0.01106 1.23%
MLPoisson::Fapply() 1142 0.01038 0.01038 0.01038 1.15%
FabArray::Saxpy() 813 0.008218 0.008218 0.008218 0.91%
FabArray::Xpay() 821 0.008072 0.008072 0.008072 0.90%
Castro::post_timestep() 10 0.007882 0.007882 0.007882 0.88%
MLCellLinOp::solutionResidual() 93 0.007165 0.007165 0.007165 0.80%
Gravity::fill_multipole_BCs() 11 0.006984 0.006984 0.006984 0.78%
MLMG::computeResidual() 82 0.006173 0.006173 0.006173 0.69%
Castro::reset_internal_energy(MultiFab) 63 0.005752 0.005752 0.005752 0.64%
Castro::estTimeStep() 21 0.005063 0.005063 0.005063 0.56%
MLCellLinOp::defineBC() 11 0.004972 0.004972 0.004972 0.55%
BndryData::define() 11 0.004752 0.004752 0.004752 0.53%
MLMG::prepareForSolve() 11 0.004704 0.004704 0.004704 0.52%
FabArray::LinComb() 557 0.004529 0.004529 0.004529 0.50%
Amr::InitializeInit() 1 0.004353 0.004353 0.004353 0.48%
Amr::defBaseLevel() 1 0.004347 0.004347 0.004347 0.48%
amrex::Add() 164 0.004337 0.004337 0.004337 0.48%
Castro::initData() 1 0.003816 0.003816 0.003816 0.42%
Castro::construct_new_source() 50 0.003392 0.003392 0.003392 0.38%
Castro::construct_new_gravity_source() 10 0.003348 0.003348 0.003348 0.37%
Castro::construct_old_source() 50 0.002753 0.002753 0.002753 0.31%
Castro::construct_old_gravity_source() 10 0.002735 0.002735 0.002735 0.30%
Castro::computeNewDt() 9 0.002289 0.002289 0.002289 0.25%
MLMG::ResNormInf() 93 0.00208 0.00208 0.00208 0.23%
Castro::apply_source_to_state() 20 0.001823 0.001823 0.001823 0.20%
FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001708 0.001708 0.001708 0.19%
Castro::reset_internal_energy(Fab) 504 0.001452 0.001452 0.001452 0.16%
FabArrayBase::getCPC() 1323 0.001424 0.001424 0.001424 0.16%
MLMG::getGradSolution() 11 0.001406 0.001406 0.001406 0.16%
MLCellLinOp::compGrad() 11 0.0014 0.0014 0.0014 0.16%
MLCellLinOp::setLevelBC() 11 0.001384 0.001384 0.001384 0.15%
FabArray::mult() 43 0.001313 0.001313 0.001313 0.15%
FabArray::setDomainBndry() 41 0.001298 0.001298 0.001298 0.14%
Castro::check_for_nan() 20 0.001213 0.001213 0.001213 0.13%
MultiFab::contains_nan() 20 0.001202 0.001202 0.001202 0.13%
MLPoisson::prepareForSolve() 11 0.001142 0.001142 0.001142 0.13%
MLCellLinOp::prepareForSolve() 11 0.001128 0.001128 0.001128 0.13%
Castro::post_regrid() 1 0.001103 0.001103 0.001103 0.12%
MLMG::computeMLResidual() 11 0.001035 0.001035 0.001035 0.12%
Castro::enforce_speed_limit() 62 0.0009557 0.0009557 0.0009557 0.11%
Castro::computeInitialDt() 2 0.0009528 0.0009528 0.0009528 0.11%
Gravity::update_max_rhs() 11 0.0008237 0.0008237 0.0008237 0.09%
FabArrayBase::getFB() 4023 0.0007227 0.0007227 0.0007227 0.08%
FabArrayBase::CPC::define() 454 0.0006734 0.0006734 0.0006734 0.07%
Castro::finalize_advance() 10 0.000597 0.000597 0.000597 0.07%
Amr::InitAmr() 1 0.0005573 0.0005573 0.0005573 0.06%
MLLinOp::define() 11 0.0004603 0.0004603 0.0004603 0.05%
Castro::Castro() 1 0.0004578 0.0004578 0.0004578 0.05%
MLLinOp::defineGrids() 11 0.0004492 0.0004492 0.0004492 0.05%
Gravity::swapTimeLevels() 10 0.0004372 0.0004372 0.0004372 0.05%
MLMG::MLResNormInf() 11 0.0002837 0.0002837 0.0002837 0.03%
MultiFab::max() 11 0.0002563 0.0002563 0.0002563 0.03%
MLMG::MLRhsNormInf() 11 0.0002189 0.0002189 0.0002189 0.02%
Castro::create_source_corrector() 10 0.0001937 0.0001937 0.0001937 0.02%
Castro::buildMetrics() 1 0.000163 0.000163 0.000163 0.02%
FabArrayBase::FB::FB() 56 8.99e-05 8.99e-05 8.99e-05 0.01%
MLLinOp::makeAgglomeratedDMap 11 5.137e-05 5.137e-05 5.137e-05 0.01%
AmrLevel::AmrLevel(dm) 1 4.684e-05 4.684e-05 4.684e-05 0.01%
Amr::writeSmallPlotFile() 1 4.607e-05 4.607e-05 4.607e-05 0.01%
Castro::finalize_do_advance() 10 3.753e-05 3.753e-05 3.753e-05 0.00%
makeSFC 55 3.725e-05 3.725e-05 3.725e-05 0.00%
Castro::swap_state_time_levels() 10 3.685e-05 3.685e-05 3.685e-05 0.00%
StateData::define() 4 3.598e-05 3.598e-05 3.598e-05 0.00%
Castro::enforce_consistent_e() 1 3.492e-05 3.492e-05 3.492e-05 0.00%
Castro::initMFs() 1 2.627e-05 2.627e-05 2.627e-05 0.00%
DistributionMapping::Distribute() 56 1.529e-05 1.529e-05 1.529e-05 0.00%
Amr::initSubcycle() 1 9.319e-06 9.319e-06 9.319e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 5.077e-06 5.077e-06 5.077e-06 0.00%
AmrLevel::checkPointPost() 3 4.707e-06 4.707e-06 4.707e-06 0.00%
Castro::retry_advance_ctu() 10 3.799e-06 3.799e-06 3.799e-06 0.00%
Gravity::set_mass_offset() 11 3.776e-06 3.776e-06 3.776e-06 0.00%
Castro::FluxRegCrseInit 10 2.988e-06 2.988e-06 2.988e-06 0.00%
AmrLevel::checkPointPre() 3 2.065e-06 2.065e-06 2.065e-06 0.00%
Castro::FluxRegFineAdd() 10 2.016e-06 2.016e-06 2.016e-06 0.00%
MLLinOp::makeSubCommunicator() 11 1.934e-06 1.934e-06 1.934e-06 0.00%
--------------------------------------------------------------------------------------------
Unused ParmParse Variables:
[TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2]
[TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2]
Device Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Arena::Initialize() 1 1 92 MiB 9042 MiB
Castro::construct_ctu_hydro_source() 2880 2880 126 MiB 692 MiB
Castro::initMFs() 48 48 67 MiB 68 MiB
Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB
StateData::define() 32 32 55 MiB 55 MiB
FillPatchIterator::Initialize 328 328 1019 KiB 39 MiB
Castro::initialize_do_advance() 80 80 25 MiB 39 MiB
ResizeRandomSeed 1 1 30 MiB 30 MiB
Amr::writePlotFile() 16 16 2114 KiB 28 MiB
Castro::initialize_advance() 80 80 15 MiB 23 MiB
Castro::buildMetrics() 32 32 15 MiB 15 MiB
Castro::Castro() 48 48 7612 KiB 14 MiB
MLMG::prepareForSolve() 660 660 3494 KiB 12 MiB
Gravity::get_new_grav_vector() 91 91 200 KiB 10 MiB
Gravity::get_old_grav_vector() 80 80 176 KiB 10 MiB
Gravity::multilevel_solve_for_new_phi() 24 24 7524 KiB 7586 KiB
Gravity::fill_multipole_BCs() 154 154 12 KiB 2053 KiB
Gravity::update_max_rhs() 88 88 1749 B 2048 KiB
Gravity::solve_for_phi() 80 80 552 KiB 2048 KiB
Gravity::actual_multilevel_solve() 8 8 93 KiB 2048 KiB
BndryData::define() 1056 1056 319 KiB 1095 KiB
MLCellLinOp::defineAuxData() 1716 1716 203 KiB 671 KiB
Castro::estTimeStep() 21 21 2684 B 480 KiB
VisMF::Write(FabArray) 656 656 3355 B 320 KiB
Castro::normalize_species() 62 62 5145 B 320 KiB
amrex::average_down 1067 1067 1250 B 257 KiB
MLMG::addInterpCorrection() 1066 1066 1146 B 257 KiB
amrex::Dot() 1360 1360 3395 B 160 KiB
FabArray::norminf() 907 907 2377 B 160 KiB
Castro::do_advance_ctu() 10 10 625 B 160 KiB
MultiFab::max() 11 11 44 B 160 KiB
Castro::initData() 1 1 27 B 160 KiB
FabArray::setVal() 106 106 21 KiB 26 KiB
MultiFab::contains_nan() 20 20 26 B 20 KiB
MLPoisson::Fsmooth() 132 132 3416 B 12 KiB
FabArray::ParallelCopy_nowait() 861 861 44 B 10 KiB
FillBoundary_nowait() 760 760 288 B 9648 B
MLCellLinOp::applyBC() 8866 8866 220 B 9344 B
MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B
amrex::Copy() 100 100 3855 B 6144 B
StateData::FillBoundary(geom) 1992 1992 45 B 2880 B
FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B
MLCGSolver::bicgstab 738 738 118 B 1472 B
MLCellLinOp::defineBC() 66 66 359 B 1248 B
------------------------------------------------------------------------------
Managed Memory Usage:
-----------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
-----------------------------------------------------------------
The_Managed_Arena::Initialize() 1 1 546 B 8192 KiB
-----------------------------------------------------------------
Pinned Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Pinned_Arena::Initialize() 1 1 48 KiB 8192 KiB
VisMF::Write(FabArray) 744 744 613 KiB 3584 KiB
FabArray::setVal() 106 106 21 KiB 26 KiB
MLPoisson::Fsmooth() 132 132 3416 B 12 KiB
FabArray::ParallelCopy_nowait() 861 861 44 B 10 KiB
FillBoundary_nowait() 760 760 288 B 9648 B
MLCellLinOp::applyBC() 4433 4433 219 B 9328 B
MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B
amrex::Copy() 100 100 3855 B 6144 B
Gravity::get_new_grav_vector() 3 3 2907 B 3072 B
StateData::FillBoundary(geom) 1992 1992 45 B 2880 B
Gravity::fill_multipole_BCs() 33 33 4 B 2832 B
FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B
amrex::average_down 83 83 272 B 1296 B
MLMG::addInterpCorrection() 82 82 2 B 1024 B
MLMG::prepareForSolve() 11 11 291 B 1024 B
MLCellLinOp::setLevelBC() 66 66 0 B 768 B
amrex::Dot() 1360 1360 24 B 400 B
FabArray::norminf() 907 907 17 B 272 B
Castro::estTimeStep() 21 21 0 B 32 B
MultiFab::contains_nan() 20 20 0 B 16 B
MultiFab::max() 11 11 0 B 16 B
Castro::normalize_species() 62 62 0 B 16 B
Castro::initData() 1 1 0 B 16 B
Castro::do_advance_ctu() 10 10 0 B 16 B
------------------------------------------------------------------------------
Total GPU global memory (MB): 12056
Free GPU global memory (MB): 2459
[The Arena] space allocated (MB): 9042
[The Arena] space used (MB): 0
[The Managed Arena] space allocated (MB): 8
[The Managed Arena] space used (MB): 0
[The Pinned Arena] space allocated (MB): 8
[The Pinned Arena] space used (MB): 0
AMReX (23.04-9-g1e73fa8ec7c3) finalized
Initializing CUDA...
CUDA initialized with 1 device.
AMReX (23.04-9-g1e73fa8ec7c3) initialized
Starting run at 09:01:25 UTC on 2023-04-05.
Successfully read inputs file ...
Castro git describe: 23.04-2-gabeccccb9
AMReX git describe: 23.04-9-g1e73fa8ec
Microphysics git describe: 23.04-3-gdc41ef4e
reading extern runtime parameters ...
3 Species:
C12 O16 Mg24
restarting calculation from file: dustcollapse-restart_chk00005
Successfully read inputs file ...
read CPU time: 0.495258124
Restart time = 0.046451927 seconds.
[Level 0 step 6] ADVANCE with dt = 5.79654185e-05
[Level 0 step 6] Advanced 262144 cells
[STEP 6] Coarse TimeStep time: 0.049168075
[STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05
[Level 0 step 7] ADVANCE with dt = 6.086368943e-05
[Level 0 step 7] Advanced 262144 cells
[STEP 7] Coarse TimeStep time: 0.048069798
[STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05
[Level 0 step 8] ADVANCE with dt = 6.39068739e-05
[Level 0 step 8] Advanced 262144 cells
[STEP 8] Coarse TimeStep time: 0.047031561
[STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05
[Level 0 step 9] ADVANCE with dt = 6.71022176e-05
[Level 0 step 9] Advanced 262144 cells
[STEP 9] Coarse TimeStep time: 0.059152971
[STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05
[Level 0 step 10] ADVANCE with dt = 7.045732848e-05
[Level 0 step 10] Advanced 262144 cells
[STEP 10] Coarse TimeStep time: 0.061903425
[STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05
PLOTFILE: file = dustcollapse-restart_plt00010
Write plotfile time = 0.033246831 seconds
Ending run at 09:01:25 UTC on 2023-04-05.
Run time = 0.346033682
Run time without initialization = 0.298984095
Average number of zones advanced per microsecond: 4.384
Average number of zones advanced per microsecond per rank: 4.384
CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472
TinyProfiler total time across processes [min...avg...max]: 0.3461 ... 0.3461 ... 0.3461
--------------------------------------------------------------------------------------------
Name NCalls Excl. Min Excl. Avg Excl. Max Max %
--------------------------------------------------------------------------------------------
Castro::construct_ctu_hydro_source() 5 0.0836 0.0836 0.0836 24.16%
VisMF::Read() 3 0.04024 0.04024 0.04024 11.63%
MLCellLinOp::applyBC() 1946 0.03312 0.03312 0.03312 9.57%
VisMF::Write(FabArray) 1 0.03173 0.03173 0.03173 9.17%
MLPoisson::Fsmooth() 1440 0.01404 0.01404 0.01404 4.06%
FillBoundary_nowait() 1766 0.01288 0.01288 0.01288 3.72%
StateData::FillBoundary(geom) 160 0.01239 0.01239 0.01239 3.58%
amrex::Dot() 484 0.008847 0.008847 0.008847 2.56%
amrex::Copy() 463 0.006949 0.006949 0.006949 2.01%
Castro::normalize_species() 30 0.006419 0.006419 0.006419 1.85%
FabArray::setVal() 537 0.006245 0.006245 0.006245 1.80%
FabArray::norminf() 326 0.006221 0.006221 0.006221 1.80%
FabArray::ParallelCopy_nowait() 380 0.006163 0.006163 0.006163 1.78%
Castro::computeTemp() 30 0.006003 0.006003 0.006003 1.73%
Castro::enforce_min_density() 30 0.005839 0.005839 0.005839 1.69%
StateDataPhysBCFunct::() 20 0.005413 0.005413 0.005413 1.56%
MLCellLinOp::defineAuxData() 6 0.00526 0.00526 0.00526 1.52%
MLPoisson::Fapply() 500 0.004429 0.004429 0.004429 1.28%
Gravity::fill_multipole_BCs() 6 0.003729 0.003729 0.003729 1.08%
FabArray::Saxpy() 355 0.003642 0.003642 0.003642 1.05%
FabArray::Xpay() 361 0.003485 0.003485 0.003485 1.01%
MLMG::addInterpCorrection() 180 0.00293 0.00293 0.00293 0.85%
Castro::reset_internal_energy(MultiFab) 30 0.002657 0.002657 0.002657 0.77%
amrex::average_down 180 0.002619 0.002619 0.002619 0.76%
Castro::estTimeStep() 10 0.002495 0.002495 0.002495 0.72%
Amr::restart() 1 0.002484 0.002484 0.002484 0.72%
BndryData::define() 6 0.002058 0.002058 0.002058 0.59%
FabArray::LinComb() 242 0.001915 0.001915 0.001915 0.55%
amrex::Add() 72 0.001855 0.001855 0.001855 0.54%
Castro::construct_new_gravity_source() 5 0.001703 0.001703 0.001703 0.49%
Castro::do_advance_ctu() 5 0.00146 0.00146 0.00146 0.42%
Castro::construct_old_gravity_source() 5 0.00144 0.00144 0.00144 0.42%
Amr::writePlotFile() 1 0.001352 0.001352 0.001352 0.39%
MLCGSolver::bicgstab 36 0.0009412 0.0009412 0.0009412 0.27%
FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009294 0.0009294 0.0009294 0.27%
Gravity::actual_solve_with_mlmg() 6 0.0007694 0.0007694 0.0007694 0.22%
MLCellLinOp::setLevelBC() 6 0.0007504 0.0007504 0.0007504 0.22%
Castro::reset_internal_energy(Fab) 240 0.0006708 0.0006708 0.0006708 0.19%
FabArray::mult() 22 0.0006435 0.0006435 0.0006435 0.19%
FabArray::setDomainBndry() 20 0.0006422 0.0006422 0.0006422 0.19%
Castro::enforce_speed_limit() 30 0.0006287 0.0006287 0.0006287 0.18%
MLCellLinOp::prepareForSolve() 6 0.0006203 0.0006203 0.0006203 0.18%
MultiFab::contains_nan() 10 0.0005922 0.0005922 0.0005922 0.17%
MLCellLinOp::compGrad() 6 0.0004975 0.0004975 0.0004975 0.14%
MLCellLinOp::smooth() 720 0.0004774 0.0004774 0.0004774 0.14%
MLMG::prepareForSolve() 6 0.0004769 0.0004769 0.0004769 0.14%
Amr::InitAmr() 1 0.0004164 0.0004164 0.0004164 0.12%
FabArrayBase::CPC::define() 244 0.0004113 0.0004113 0.0004113 0.12%
FabArray::FillBoundary() 1766 0.0003669 0.0003669 0.0003669 0.11%
FabArrayBase::getCPC() 632 0.0003566 0.0003566 0.0003566 0.10%
Gravity::get_old_grav_vector() 5 0.0003149 0.0003149 0.0003149 0.09%
main() 1 0.0002793 0.0002793 0.0002793 0.08%
Gravity::get_new_grav_vector() 5 0.0002763 0.0002763 0.0002763 0.08%
FabArrayBase::getFB() 1766 0.0002495 0.0002495 0.0002495 0.07%
MLCellLinOp::apply() 500 0.0001992 0.0001992 0.0001992 0.06%
MLLinOp::defineGrids() 6 0.0001921 0.0001921 0.0001921 0.06%
MLMG::mgVcycle() 36 0.0001621 0.0001621 0.0001621 0.05%
Amr::coarseTimeStep() 5 0.0001614 0.0001614 0.0001614 0.05%
MultiFab::max() 6 0.0001368 0.0001368 0.0001368 0.04%
Castro::create_source_corrector() 5 0.000129 0.000129 0.000129 0.04%
MLCGSolver::ParallelAllReduce 659 0.0001257 0.0001257 0.0001257 0.04%
FillPatchIterator::Initialize 20 0.0001255 0.0001255 0.0001255 0.04%
MLCellLinOp::defineBC() 6 0.0001187 0.0001187 0.0001187 0.03%
FabArray::ParallelCopy() 380 0.0001096 0.0001096 0.0001096 0.03%
MLCellLinOp::correctionResidual() 216 9.192e-05 9.192e-05 9.192e-05 0.03%
Castro::subcycle_advance_ctu() 5 7.944e-05 7.944e-05 7.944e-05 0.02%
Amr::timeStep() 5 7.803e-05 7.803e-05 7.803e-05 0.02%
AmrLevel::restart() 1 7.677e-05 7.677e-05 7.677e-05 0.02%
Gravity::update_max_rhs() 6 6.711e-05 6.711e-05 6.711e-05 0.02%
StateData::restartDoit() 4 6.477e-05 6.477e-05 6.477e-05 0.02%
Gravity::solve_for_phi() 5 6.264e-05 6.264e-05 6.264e-05 0.02%
FabArrayBase::FB::FB() 26 5.904e-05 5.904e-05 5.904e-05 0.02%
MLMG:computeResOfCorrection() 180 4.802e-05 4.802e-05 4.802e-05 0.01%
Castro::expand_state() 5 4.631e-05 4.631e-05 4.631e-05 0.01%
MLMG::mgVcycle_down::0 36 4.169e-05 4.169e-05 4.169e-05 0.01%
MLMG::actualBottomSolve() 36 3.853e-05 3.853e-05 3.853e-05 0.01%
MLMG::mgVcycle_down::1 36 3.575e-05 3.575e-05 3.575e-05 0.01%
Castro::clean_state() 30 3.545e-05 3.545e-05 3.545e-05 0.01%
Castro::initialize_advance() 5 3.378e-05 3.378e-05 3.378e-05 0.01%
MLMG::solve() 6 3.37e-05 3.37e-05 3.37e-05 0.01%
Castro::advance() 5 3.322e-05 3.322e-05 3.322e-05 0.01%
MLMG::mgVcycle_down::4 36 3.275e-05 3.275e-05 3.275e-05 0.01%
MLMG::mgVcycle_down::2 36 3.247e-05 3.247e-05 3.247e-05 0.01%
MLMG::mgVcycle_down::3 36 3.15e-05 3.15e-05 3.15e-05 0.01%
Castro::finalize_advance() 5 2.852e-05 2.852e-05 2.852e-05 0.01%
MLMG::mgVcycle_up::4 36 2.835e-05 2.835e-05 2.835e-05 0.01%
Amr::writeSmallPlotFile() 1 2.741e-05 2.741e-05 2.741e-05 0.01%
Castro::initMFs() 1 2.702e-05 2.702e-05 2.702e-05 0.01%
Castro::buildMetrics() 1 2.631e-05 2.631e-05 2.631e-05 0.01%
Castro::initialize_do_advance() 5 2.602e-05 2.602e-05 2.602e-05 0.01%
MLMG::mgVcycle_up::0 36 2.458e-05 2.458e-05 2.458e-05 0.01%
MLMG::oneIter() 36 2.399e-05 2.399e-05 2.399e-05 0.01%
Castro::swap_state_time_levels() 5 2.304e-05 2.304e-05 2.304e-05 0.01%
MLMG::mgVcycle_up::3 36 2.179e-05 2.179e-05 2.179e-05 0.01%
MLCellLinOp::solutionResidual() 42 2.129e-05 2.129e-05 2.129e-05 0.01%
MLMG::mgVcycle_up::2 36 2.121e-05 2.121e-05 2.121e-05 0.01%
MLMG::mgVcycle_up::1 36 2.032e-05 2.032e-05 2.032e-05 0.01%
Castro::post_restart() 1 1.993e-05 1.993e-05 1.993e-05 0.01%
Castro::finalize_do_advance() 5 1.883e-05 1.883e-05 1.883e-05 0.01%
MLMG::ResNormInf() 42 1.728e-05 1.728e-05 1.728e-05 0.00%
Gravity::multilevel_solve_for_new_phi() 1 1.574e-05 1.574e-05 1.574e-05 0.00%
MLMG::mgVcycle_bottom 36 1.44e-05 1.44e-05 1.44e-05 0.00%
MLMG::computeResidual() 36 1.355e-05 1.355e-05 1.355e-05 0.00%
MLPoisson::define() 6 1.348e-05 1.348e-05 1.348e-05 0.00%
FillPatchSingleLevel 20 1.346e-05 1.346e-05 1.346e-05 0.00%
makeSFC 30 1.25e-05 1.25e-05 1.25e-05 0.00%
Castro::construct_new_gravity() 5 1.234e-05 1.234e-05 1.234e-05 0.00%
Gravity::solve_phi_with_mlmg() 6 1.208e-05 1.208e-05 1.208e-05 0.00%
Castro::construct_new_source() 25 1.076e-05 1.076e-05 1.076e-05 0.00%
Castro::do_new_sources() 5 1.061e-05 1.061e-05 1.061e-05 0.00%
MLPoisson::prepareForSolve() 6 9.743e-06 9.743e-06 9.743e-06 0.00%
Castro::construct_old_source() 25 9.225e-06 9.225e-06 9.225e-06 0.00%
MLMG::computeMLResidual() 6 9.224e-06 9.224e-06 9.224e-06 0.00%
DistributionMapping::Distribute() 31 8.98e-06 8.98e-06 8.98e-06 0.00%
Amr::initSubcycle() 1 8.635e-06 8.635e-06 8.635e-06 0.00%
Castro::do_old_sources() 5 8.399e-06 8.399e-06 8.399e-06 0.00%
MLLinOp::makeAgglomeratedDMap 6 7.404e-06 7.404e-06 7.404e-06 0.00%
Gravity::actual_multilevel_solve() 1 7.349e-06 7.349e-06 7.349e-06 0.00%
MLLinOp::define() 6 6.001e-06 6.001e-06 6.001e-06 0.00%
Castro::check_for_nan() 10 5.793e-06 5.793e-06 5.793e-06 0.00%
Castro::construct_old_gravity() 5 5.345e-06 5.345e-06 5.345e-06 0.00%
Castro::apply_source_to_state() 10 5.109e-06 5.109e-06 5.109e-06 0.00%
Castro::post_timestep() 5 4.934e-06 4.934e-06 4.934e-06 0.00%
Gravity::swapTimeLevels() 5 3.846e-06 3.846e-06 3.846e-06 0.00%
Castro::computeNewDt() 5 3.455e-06 3.455e-06 3.455e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 3e-06 3e-06 3e-06 0.00%
MLMG::getGradSolution() 6 2.924e-06 2.924e-06 2.924e-06 0.00%
MLMG::MLRhsNormInf() 6 2.188e-06 2.188e-06 2.188e-06 0.00%
Gravity::set_mass_offset() 6 2.161e-06 2.161e-06 2.161e-06 0.00%
MLMG::MLResNormInf() 6 2.138e-06 2.138e-06 2.138e-06 0.00%
Castro::FluxRegFineAdd() 5 2.128e-06 2.128e-06 2.128e-06 0.00%
Castro::retry_advance_ctu() 5 1.603e-06 1.603e-06 1.603e-06 0.00%
Castro::FluxRegCrseInit 5 1.523e-06 1.523e-06 1.523e-06 0.00%
Amr::init() 1 1.334e-06 1.334e-06 1.334e-06 0.00%
MLLinOp::makeSubCommunicator() 6 1.078e-06 1.078e-06 1.078e-06 0.00%
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Name NCalls Incl. Min Incl. Avg Incl. Max Max %
--------------------------------------------------------------------------------------------
main() 1 0.346 0.346 0.346 100.00%
Amr::coarseTimeStep() 5 0.2655 0.2655 0.2655 76.71%
Amr::timeStep() 5 0.2636 0.2636 0.2636 76.16%
Castro::advance() 5 0.2605 0.2605 0.2605 75.26%
Castro::subcycle_advance_ctu() 5 0.2537 0.2537 0.2537 73.31%
Castro::do_advance_ctu() 5 0.2536 0.2536 0.2536 73.29%
Castro::construct_new_gravity() 5 0.1306 0.1306 0.1306 37.75%
Gravity::solve_phi_with_mlmg() 6 0.1257 0.1257 0.1257 36.33%
Gravity::solve_for_phi() 5 0.1227 0.1227 0.1227 35.45%
Gravity::actual_solve_with_mlmg() 6 0.1217 0.1217 0.1217 35.17%
MLMG::solve() 6 0.1104 0.1104 0.1104 31.91%
MLMG::oneIter() 36 0.1036 0.1036 0.1036 29.94%
MLMG::mgVcycle() 36 0.1021 0.1021 0.1021 29.49%
Castro::construct_ctu_hydro_source() 5 0.08361 0.08361 0.08361 24.16%
MLCellLinOp::smooth() 720 0.04997 0.04997 0.04997 14.44%
MLCellLinOp::applyBC() 1946 0.04667 0.04667 0.04667 13.49%
Amr::init() 1 0.04649 0.04649 0.04649 13.44%
Amr::restart() 1 0.04649 0.04649 0.04649 13.43%
AmrLevel::restart() 1 0.04044 0.04044 0.04044 11.69%
StateData::restartDoit() 4 0.04036 0.04036 0.04036 11.66%
VisMF::Read() 3 0.04024 0.04024 0.04024 11.63%
Amr::writePlotFile() 1 0.03334 0.03334 0.03334 9.63%
VisMF::Write(FabArray) 1 0.03173 0.03173 0.03173 9.17%
MLMG::mgVcycle_bottom 36 0.0316 0.0316 0.0316 9.13%
MLMG::actualBottomSolve() 36 0.03159 0.03159 0.03159 9.13%
MLCGSolver::bicgstab 36 0.03129 0.03129 0.03129 9.04%
Castro::clean_state() 30 0.02225 0.02225 0.02225 6.43%
FillPatchIterator::Initialize 20 0.02062 0.02062 0.02062 5.96%
FillPatchSingleLevel 20 0.01986 0.01986 0.01986 5.74%
StateDataPhysBCFunct::() 20 0.01781 0.01781 0.01781 5.15%
MLCellLinOp::apply() 500 0.01558 0.01558 0.01558 4.50%
MLMG::mgVcycle_down::0 36 0.01429 0.01429 0.01429 4.13%
MLPoisson::Fsmooth() 1440 0.01404 0.01404 0.01404 4.06%
FabArray::FillBoundary() 1766 0.01355 0.01355 0.01355 3.92%
FillBoundary_nowait() 1766 0.01319 0.01319 0.01319 3.81%
StateData::FillBoundary(geom) 160 0.01239 0.01239 0.01239 3.58%
MLMG::mgVcycle_up::0 36 0.01089 0.01089 0.01089 3.15%
Castro::initialize_do_advance() 5 0.01009 0.01009 0.01009 2.92%
MLCellLinOp::correctionResidual() 216 0.009459 0.009459 0.009459 2.73%
Castro::computeTemp() 30 0.009332 0.009332 0.009332 2.70%
MLPoisson::define() 6 0.008996 0.008996 0.008996 2.60%
amrex::Dot() 484 0.008847 0.008847 0.008847 2.56%
MLMG:computeResOfCorrection() 180 0.008315 0.008315 0.008315 2.40%
Gravity::get_new_grav_vector() 5 0.00783 0.00783 0.00783 2.26%
Castro::construct_old_gravity() 5 0.007541 0.007541 0.007541 2.18%
Gravity::get_old_grav_vector() 5 0.007536 0.007536 0.007536 2.18%
Castro::do_new_sources() 5 0.007067 0.007067 0.007067 2.04%
amrex::Copy() 463 0.006949 0.006949 0.006949 2.01%
MLMG::mgVcycle_down::1 36 0.006814 0.006814 0.006814 1.97%
FabArray::ParallelCopy() 380 0.006651 0.006651 0.006651 1.92%
FabArray::ParallelCopy_nowait() 380 0.006541 0.006541 0.006541 1.89%
Castro::initialize_advance() 5 0.006436 0.006436 0.006436 1.86%
Castro::normalize_species() 30 0.006419 0.006419 0.006419 1.85%
MLMG::mgVcycle_down::2 36 0.006337 0.006337 0.006337 1.83%
FabArray::setVal() 537 0.006245 0.006245 0.006245 1.80%
MLMG::mgVcycle_down::3 36 0.006229 0.006229 0.006229 1.80%
FabArray::norminf() 326 0.006221 0.006221 0.006221 1.80%
MLMG::mgVcycle_down::4 36 0.006163 0.006163 0.006163 1.78%
MLCellLinOp::defineAuxData() 6 0.006012 0.006012 0.006012 1.74%
Castro::expand_state() 5 0.005908 0.005908 0.005908 1.71%
Castro::enforce_min_density() 30 0.005839 0.005839 0.005839 1.69%
MLCGSolver::ParallelAllReduce 659 0.005335 0.005335 0.005335 1.54%
MLMG::addInterpCorrection() 180 0.005217 0.005217 0.005217 1.51%
Castro::do_old_sources() 5 0.005082 0.005082 0.005082 1.47%
MLMG::mgVcycle_up::4 36 0.00502 0.00502 0.00502 1.45%
amrex::average_down 180 0.004945 0.004945 0.004945 1.43%
MLMG::mgVcycle_up::1 36 0.004936 0.004936 0.004936 1.43%
MLMG::mgVcycle_up::2 36 0.004844 0.004844 0.004844 1.40%
MLMG::mgVcycle_up::3 36 0.004768 0.004768 0.004768 1.38%
MLPoisson::Fapply() 500 0.004429 0.004429 0.004429 1.28%
Gravity::fill_multipole_BCs() 6 0.003891 0.003891 0.003891 1.12%
FabArray::Saxpy() 355 0.003642 0.003642 0.003642 1.05%
FabArray::Xpay() 361 0.003485 0.003485 0.003485 1.01%
Castro::post_restart() 1 0.003399 0.003399 0.003399 0.98%
Castro::reset_internal_energy(MultiFab) 30 0.003328 0.003328 0.003328 0.96%
Gravity::multilevel_solve_for_new_phi() 1 0.003291 0.003291 0.003291 0.95%
Gravity::actual_multilevel_solve() 1 0.003275 0.003275 0.003275 0.95%
MLCellLinOp::solutionResidual() 42 0.003203 0.003203 0.003203 0.93%
Castro::post_timestep() 5 0.003034 0.003034 0.003034 0.88%
MLCellLinOp::defineBC() 6 0.002744 0.002744 0.002744 0.79%
MLMG::computeResidual() 36 0.002652 0.002652 0.002652 0.77%
BndryData::define() 6 0.002625 0.002625 0.002625 0.76%
MLMG::prepareForSolve() 6 0.002536 0.002536 0.002536 0.73%
Castro::estTimeStep() 10 0.002495 0.002495 0.002495 0.72%
FabArray::LinComb() 242 0.001915 0.001915 0.001915 0.55%
amrex::Add() 72 0.001855 0.001855 0.001855 0.54%
Castro::computeNewDt() 5 0.001742 0.001742 0.001742 0.50%
Castro::construct_new_source() 25 0.001713 0.001713 0.001713 0.50%
Castro::construct_new_gravity_source() 5 0.001703 0.001703 0.001703 0.49%
Castro::construct_old_source() 25 0.001449 0.001449 0.001449 0.42%
Castro::construct_old_gravity_source() 5 0.00144 0.00144 0.00144 0.42%
FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009294 0.0009294 0.0009294 0.27%
Castro::apply_source_to_state() 10 0.0009196 0.0009196 0.0009196 0.27%
MLMG::ResNormInf() 42 0.0009176 0.0009176 0.0009176 0.27%
FabArrayBase::getCPC() 632 0.0007679 0.0007679 0.0007679 0.22%
MLMG::getGradSolution() 6 0.0007651 0.0007651 0.0007651 0.22%
MLCellLinOp::compGrad() 6 0.0007622 0.0007622 0.0007622 0.22%
MLCellLinOp::setLevelBC() 6 0.0007504 0.0007504 0.0007504 0.22%
Castro::reset_internal_energy(Fab) 240 0.0006708 0.0006708 0.0006708 0.19%
FabArray::mult() 22 0.0006435 0.0006435 0.0006435 0.19%
FabArray::setDomainBndry() 20 0.0006422 0.0006422 0.0006422 0.19%
MLPoisson::prepareForSolve() 6 0.00063 0.00063 0.00063 0.18%
Castro::enforce_speed_limit() 30 0.0006287 0.0006287 0.0006287 0.18%
MLCellLinOp::prepareForSolve() 6 0.0006203 0.0006203 0.0006203 0.18%
Castro::check_for_nan() 10 0.000598 0.000598 0.000598 0.17%
MultiFab::contains_nan() 10 0.0005922 0.0005922 0.0005922 0.17%
MLMG::computeMLResidual() 6 0.0005739 0.0005739 0.0005739 0.17%
Gravity::update_max_rhs() 6 0.0004477 0.0004477 0.0004477 0.13%
Amr::InitAmr() 1 0.0004251 0.0004251 0.0004251 0.12%
FabArrayBase::CPC::define() 244 0.0004113 0.0004113 0.0004113 0.12%
FabArrayBase::getFB() 1766 0.0003085 0.0003085 0.0003085 0.09%
Castro::finalize_advance() 5 0.0002859 0.0002859 0.0002859 0.08%
Gravity::swapTimeLevels() 5 0.000228 0.000228 0.000228 0.07%
MLLinOp::define() 6 0.0002267 0.0002267 0.0002267 0.07%
MLLinOp::defineGrids() 6 0.0002207 0.0002207 0.0002207 0.06%
MLMG::MLResNormInf() 6 0.0001505 0.0001505 0.0001505 0.04%
Castro::buildMetrics() 1 0.0001454 0.0001454 0.0001454 0.04%
MultiFab::max() 6 0.0001368 0.0001368 0.0001368 0.04%
Castro::create_source_corrector() 5 0.000129 0.000129 0.000129 0.04%
MLMG::MLRhsNormInf() 6 0.0001173 0.0001173 0.0001173 0.03%
FabArrayBase::FB::FB() 26 5.904e-05 5.904e-05 5.904e-05 0.02%
MLLinOp::makeAgglomeratedDMap 6 2.749e-05 2.749e-05 2.749e-05 0.01%
Amr::writeSmallPlotFile() 1 2.741e-05 2.741e-05 2.741e-05 0.01%
Castro::initMFs() 1 2.702e-05 2.702e-05 2.702e-05 0.01%
Castro::swap_state_time_levels() 5 2.304e-05 2.304e-05 2.304e-05 0.01%
makeSFC 30 2.009e-05 2.009e-05 2.009e-05 0.01%
Castro::finalize_do_advance() 5 1.883e-05 1.883e-05 1.883e-05 0.01%
DistributionMapping::Distribute() 31 8.98e-06 8.98e-06 8.98e-06 0.00%
Amr::initSubcycle() 1 8.635e-06 8.635e-06 8.635e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 4.397e-06 4.397e-06 4.397e-06 0.00%
Gravity::set_mass_offset() 6 2.161e-06 2.161e-06 2.161e-06 0.00%
Castro::FluxRegFineAdd() 5 2.128e-06 2.128e-06 2.128e-06 0.00%
Castro::retry_advance_ctu() 5 1.603e-06 1.603e-06 1.603e-06 0.00%
Castro::FluxRegCrseInit 5 1.523e-06 1.523e-06 1.523e-06 0.00%
MLLinOp::makeSubCommunicator() 6 1.078e-06 1.078e-06 1.078e-06 0.00%
--------------------------------------------------------------------------------------------
Unused ParmParse Variables:
[TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2]
[TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2]
Device Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Arena::Initialize() 1 1 242 MiB 9042 MiB
Castro::construct_ctu_hydro_source() 1440 1440 108 MiB 692 MiB
Castro::initMFs() 48 48 59 MiB 68 MiB
StateData::restartDoit() 32 32 53 MiB 55 MiB
Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB
Castro::initialize_do_advance() 40 40 28 MiB 39 MiB
FillPatchIterator::Initialize 160 160 1301 KiB 39 MiB
ResizeRandomSeed 1 1 30 MiB 30 MiB
Amr::writePlotFile() 8 8 2695 KiB 28 MiB
Castro::initialize_advance() 40 40 17 MiB 23 MiB
Castro::buildMetrics() 32 32 13 MiB 15 MiB
Castro::post_restart() 48 48 6616 KiB 14 MiB
MLMG::prepareForSolve() 361 361 3918 KiB 12 MiB
Gravity::get_old_grav_vector() 43 43 225 KiB 10 MiB
Gravity::get_new_grav_vector() 40 40 231 KiB 10 MiB
Gravity::multilevel_solve_for_new_phi() 24 24 6611 KiB 7586 KiB
Gravity::fill_multipole_BCs() 84 84 17 KiB 2053 KiB
Gravity::update_max_rhs() 48 48 2490 B 2048 KiB
Gravity::solve_for_phi() 40 40 724 KiB 2048 KiB
Gravity::actual_multilevel_solve() 8 8 19 KiB 2048 KiB
BndryData::define() 576 576 360 KiB 1095 KiB
MLCellLinOp::defineAuxData() 936 936 231 KiB 671 KiB
Castro::estTimeStep() 10 10 3352 B 480 KiB
VisMF::Write(FabArray) 112 112 1435 B 320 KiB
Castro::normalize_species() 30 30 6025 B 320 KiB
amrex::average_down 469 469 1336 B 257 KiB
MLMG::addInterpCorrection() 468 468 1286 B 257 KiB
amrex::Dot() 592 592 3757 B 160 KiB
FabArray::norminf() 398 398 2649 B 160 KiB
Castro::do_advance_ctu() 5 5 643 B 160 KiB
MultiFab::max() 6 6 60 B 160 KiB
FabArray::setVal() 66 66 18 KiB 23 KiB
MultiFab::contains_nan() 10 10 34 B 20 KiB
MLPoisson::Fsmooth() 60 60 3811 B 12 KiB
FabArray::ParallelCopy_nowait() 380 380 56 B 10 KiB
FillBoundary_nowait() 336 336 323 B 9648 B
MLCellLinOp::applyBC() 3892 3892 247 B 9344 B
amrex::Copy() 56 56 5931 B 8464 B
MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B
StateData::FillBoundary(geom) 960 960 54 B 2688 B
FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B
MLCGSolver::bicgstab 324 324 131 B 1472 B
MLCellLinOp::defineBC() 36 36 405 B 1248 B
------------------------------------------------------------------------------
Managed Memory Usage:
------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------
The_Managed_Arena::Initialize() 1 1 1474 B 8192 KiB
------------------------------------------------------------------
Pinned Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Pinned_Arena::Initialize() 1 1 125 KiB 8192 KiB
VisMF::Write(FabArray) 120 120 282 KiB 3584 KiB
VisMF::Read() 24 24 226 KiB 3000 KiB
FabArray::setVal() 66 66 18 KiB 23 KiB
MLPoisson::Fsmooth() 60 60 3811 B 12 KiB
FabArray::ParallelCopy_nowait() 380 380 56 B 10 KiB
FillBoundary_nowait() 336 336 323 B 9648 B
MLCellLinOp::applyBC() 1946 1946 246 B 9328 B
amrex::Copy() 56 56 5931 B 8464 B
MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B
Gravity::get_old_grav_vector() 3 3 2603 B 3072 B
Gravity::fill_multipole_BCs() 18 18 5 B 2832 B
StateData::FillBoundary(geom) 960 960 54 B 2688 B
FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B
MLMG::prepareForSolve() 7 7 562 B 1296 B
amrex::average_down 37 37 219 B 1296 B
MLMG::addInterpCorrection() 36 36 2 B 1024 B
MLCellLinOp::setLevelBC() 36 36 0 B 768 B
amrex::Dot() 592 592 27 B 400 B
FabArray::norminf() 398 398 19 B 272 B
Castro::estTimeStep() 10 10 0 B 32 B
MultiFab::max() 6 6 0 B 16 B
MultiFab::contains_nan() 10 10 0 B 16 B
Castro::do_advance_ctu() 5 5 0 B 16 B
Castro::normalize_species() 30 30 0 B 16 B
------------------------------------------------------------------------------
Total GPU global memory (MB): 12056
Free GPU global memory (MB): 2459
[The Arena] space allocated (MB): 9042
[The Arena] space used (MB): 0
[The Managed Arena] space allocated (MB): 8
[The Managed Arena] space used (MB): 0
[The Pinned Arena] space allocated (MB): 8
[The Pinned Arena] space used (MB): 0
AMReX (23.04-9-g1e73fa8ec7c3) finalized