Initializing CUDA...
CUDA initialized with 1 device.
AMReX (23.02-16-g0c444b898f8a) initialized
Starting run at 09:58:18 UTC on 2023-02-09.
Successfully read inputs file ...
Castro git describe: 23.01-29-g09845e315
AMReX git describe: 23.02-16-g0c444b898
Microphysics git describe: 23.02-6-gb79fe5f7
reading extern runtime parameters ...
3 Species:
C12 O16 Mg24
Successfully read inputs file ...
INITIAL GRIDS
Level 0 8 grids 262144 cells 100 % of domain
smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32
CHECKPOINT: file = dustcollapse-restart_chk00000
checkPoint() time = 0.058807407 secs.
PLOTFILE: file = dustcollapse-restart_plt00000
Write plotfile time = 0.033609935 seconds
[Level 0 step 1] ADVANCE with dt = 4.541742215e-05
[Level 0 step 1] Advanced 262144 cells
[STEP 1] Coarse TimeStep time: 0.046581638
[STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05
[Level 0 step 2] ADVANCE with dt = 4.768829326e-05
[Level 0 step 2] Advanced 262144 cells
[STEP 2] Coarse TimeStep time: 0.047909471
[STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05
[Level 0 step 3] ADVANCE with dt = 5.007270792e-05
[Level 0 step 3] Advanced 262144 cells
[STEP 3] Coarse TimeStep time: 0.049283706
[STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05
[Level 0 step 4] ADVANCE with dt = 5.257634331e-05
[Level 0 step 4] Advanced 262144 cells
[STEP 4] Coarse TimeStep time: 0.049994165
[STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05
[Level 0 step 5] ADVANCE with dt = 5.520516048e-05
[Level 0 step 5] Advanced 262144 cells
[STEP 5] Coarse TimeStep time: 0.076300844
[STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05
CHECKPOINT: file = dustcollapse-restart_chk00005
checkPoint() time = 0.058087288 secs.
[Level 0 step 6] ADVANCE with dt = 5.79654185e-05
[Level 0 step 6] Advanced 262144 cells
[STEP 6] Coarse TimeStep time: 0.06399996
[STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05
[Level 0 step 7] ADVANCE with dt = 6.086368943e-05
[Level 0 step 7] Advanced 262144 cells
[STEP 7] Coarse TimeStep time: 0.059827516
[STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05
[Level 0 step 8] ADVANCE with dt = 6.39068739e-05
[Level 0 step 8] Advanced 262144 cells
[STEP 8] Coarse TimeStep time: 0.059707767
[STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05
[Level 0 step 9] ADVANCE with dt = 6.71022176e-05
[Level 0 step 9] Advanced 262144 cells
[STEP 9] Coarse TimeStep time: 0.059444389
[STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05
[Level 0 step 10] ADVANCE with dt = 7.045732848e-05
[Level 0 step 10] Advanced 262144 cells
[STEP 10] Coarse TimeStep time: 0.062645405
[STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05
CHECKPOINT: file = dustcollapse-restart_chk00010
checkPoint() time = 0.058227181 secs.
PLOTFILE: file = dustcollapse-restart_plt00010
Write plotfile time = 0.033721791 seconds
Ending run at 09:58:19 UTC on 2023-02-09.
Run time = 0.868069965
Run time without initialization = 0.72638097
Average number of zones advanced per microsecond: 3.609
Average number of zones advanced per microsecond per rank: 3.609
CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472
TinyProfiler total time across processes [min...avg...max]: 0.8681 ... 0.8681 ... 0.8681
--------------------------------------------------------------------------------------------
Name NCalls Excl. Min Excl. Avg Excl. Max Max %
--------------------------------------------------------------------------------------------
VisMF::Write(FabArray) 11 0.2356 0.2356 0.2356 27.13%
Castro::construct_ctu_hydro_source() 10 0.2139 0.2139 0.2139 24.64%
MLCellLinOp::applyBC() 4433 0.07843 0.07843 0.07843 9.03%
MLPoisson::Fsmooth() 3280 0.03268 0.03268 0.03268 3.76%
FillBoundary_nowait() 4023 0.03184 0.03184 0.03184 3.67%
StateData::FillBoundary(geom) 328 0.02541 0.02541 0.02541 2.93%
amrex::Dot() 1114 0.02094 0.02094 0.02094 2.41%
amrex::Copy() 1029 0.01491 0.01491 0.01491 1.72%
FabArray::norminf() 743 0.01466 0.01466 0.01466 1.69%
Castro::computeTemp() 63 0.01464 0.01464 0.01464 1.69%
Castro::normalize_species() 62 0.01385 0.01385 0.01385 1.60%
FabArray::ParallelCopy_nowait() 861 0.01366 0.01366 0.01366 1.57%
FabArray::setVal() 1144 0.01328 0.01328 0.01328 1.53%
StateDataPhysBCFunct::() 41 0.0125 0.0125 0.0125 1.44%
MLPoisson::Fapply() 1142 0.01052 0.01052 0.01052 1.21%
MLCellLinOp::defineAuxData() 11 0.00978 0.00978 0.00978 1.13%
Castro::enforce_min_density() 62 0.009708 0.009708 0.009708 1.12%
FabArray::Saxpy() 813 0.0082 0.0082 0.0082 0.94%
FabArray::Xpay() 821 0.008153 0.008153 0.008153 0.94%
MLMG::addInterpCorrection() 410 0.006795 0.006795 0.006795 0.78%
Gravity::fill_multipole_BCs() 11 0.006586 0.006586 0.006586 0.76%
amrex::average_down 410 0.006054 0.006054 0.006054 0.70%
FabArray::LinComb() 557 0.004566 0.004566 0.004566 0.53%
amrex::Add() 164 0.004337 0.004337 0.004337 0.50%
Amr::checkPoint() 3 0.004054 0.004054 0.004054 0.47%
Castro::reset_internal_energy(MultiFab) 63 0.004007 0.004007 0.004007 0.46%
Castro::estTimeStep() 21 0.003915 0.003915 0.003915 0.45%
BndryData::define() 11 0.003769 0.003769 0.003769 0.43%
Castro::do_advance_ctu() 10 0.003357 0.003357 0.003357 0.39%
Castro::construct_new_gravity_source() 10 0.003246 0.003246 0.003246 0.37%
Castro::construct_old_gravity_source() 10 0.002663 0.002663 0.002663 0.31%
Amr::writePlotFile() 2 0.00235 0.00235 0.00235 0.27%
MLCGSolver::bicgstab 82 0.002157 0.002157 0.002157 0.25%
FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001724 0.001724 0.001724 0.20%
Castro::reset_internal_energy(Fab) 504 0.001511 0.001511 0.001511 0.17%
Gravity::actual_solve_with_mlmg() 11 0.001506 0.001506 0.001506 0.17%
Castro::enforce_speed_limit() 62 0.001421 0.001421 0.001421 0.16%
MLCellLinOp::setLevelBC() 11 0.001388 0.001388 0.001388 0.16%
FabArray::mult() 43 0.00133 0.00133 0.00133 0.15%
FabArray::setDomainBndry() 41 0.001314 0.001314 0.001314 0.15%
Castro::initData() 1 0.001308 0.001308 0.001308 0.15%
MultiFab::contains_nan() 20 0.001198 0.001198 0.001198 0.14%
MLCellLinOp::prepareForSolve() 11 0.001142 0.001142 0.001142 0.13%
MLCellLinOp::smooth() 1640 0.00107 0.00107 0.00107 0.12%
MLCellLinOp::compGrad() 11 0.0009265 0.0009265 0.0009265 0.11%
MLMG::prepareForSolve() 11 0.0008713 0.0008713 0.0008713 0.10%
FabArray::FillBoundary() 4023 0.0008072 0.0008072 0.0008072 0.09%
FabArrayBase::getCPC() 1323 0.0007524 0.0007524 0.0007524 0.09%
FabArrayBase::CPC::define() 454 0.0006817 0.0006817 0.0006817 0.08%
Gravity::get_new_grav_vector() 11 0.0006246 0.0006246 0.0006246 0.07%
FabArrayBase::getFB() 4023 0.0005841 0.0005841 0.0005841 0.07%
Gravity::get_old_grav_vector() 10 0.0005458 0.0005458 0.0005458 0.06%
Amr::InitAmr() 1 0.000509 0.000509 0.000509 0.06%
MLCellLinOp::apply() 1142 0.0004693 0.0004693 0.0004693 0.05%
MLMG::mgVcycle() 82 0.0004249 0.0004249 0.0004249 0.05%
main() 1 0.0003241 0.0003241 0.0003241 0.04%
Amr::coarseTimeStep() 10 0.0003234 0.0003234 0.0003234 0.04%
MLCGSolver::ParallelAllReduce 1514 0.0002976 0.0002976 0.0002976 0.03%
MultiFab::max() 11 0.0002626 0.0002626 0.0002626 0.03%
FabArray::ParallelCopy() 861 0.0002432 0.0002432 0.0002432 0.03%
FillPatchIterator::Initialize 41 0.0002332 0.0002332 0.0002332 0.03%
MLCellLinOp::correctionResidual() 492 0.0002204 0.0002204 0.0002204 0.03%
MLCellLinOp::defineBC() 11 0.0002133 0.0002133 0.0002133 0.02%
Castro::subcycle_advance_ctu() 10 0.0001741 0.0001741 0.0001741 0.02%
MLLinOp::defineGrids() 11 0.0001601 0.0001601 0.0001601 0.02%
Amr::timeStep() 10 0.0001535 0.0001535 0.0001535 0.02%
Gravity::solve_for_phi() 10 0.000147 0.000147 0.000147 0.02%
StateData::checkPoint() 12 0.0001386 0.0001386 0.0001386 0.02%
Gravity::update_max_rhs() 11 0.0001205 0.0001205 0.0001205 0.01%
MLMG:computeResOfCorrection() 410 0.0001178 0.0001178 0.0001178 0.01%
Castro::finalize_advance() 10 0.0001075 0.0001075 0.0001075 0.01%
MLMG::mgVcycle_down::0 82 9.527e-05 9.527e-05 9.527e-05 0.01%
MLMG::actualBottomSolve() 82 9.417e-05 9.417e-05 9.417e-05 0.01%
Castro::Castro() 1 8.972e-05 8.972e-05 8.972e-05 0.01%
FabArrayBase::FB::FB() 56 8.722e-05 8.722e-05 8.722e-05 0.01%
Castro::expand_state() 10 7.95e-05 7.95e-05 7.95e-05 0.01%
Castro::clean_state() 62 7.919e-05 7.919e-05 7.919e-05 0.01%
MLMG::mgVcycle_down::1 82 7.89e-05 7.89e-05 7.89e-05 0.01%
MLMG::solve() 11 7.709e-05 7.709e-05 7.709e-05 0.01%
MLMG::mgVcycle_down::2 82 7.661e-05 7.661e-05 7.661e-05 0.01%
AmrLevel::checkPoint() 3 7.565e-05 7.565e-05 7.565e-05 0.01%
Castro::initialize_advance() 10 7.169e-05 7.169e-05 7.169e-05 0.01%
MLMG::mgVcycle_down::3 82 7.032e-05 7.032e-05 7.032e-05 0.01%
MLMG::mgVcycle_down::4 82 6.925e-05 6.925e-05 6.925e-05 0.01%
MLMG::mgVcycle_up::4 82 6.124e-05 6.124e-05 6.124e-05 0.01%
Castro::initialize_do_advance() 10 5.329e-05 5.329e-05 5.329e-05 0.01%
MLMG::mgVcycle_up::1 82 5.157e-05 5.157e-05 5.157e-05 0.01%
MLMG::mgVcycle_up::0 82 5.111e-05 5.111e-05 5.111e-05 0.01%
MLMG::oneIter() 82 4.998e-05 4.998e-05 4.998e-05 0.01%
MLMG::mgVcycle_up::3 82 4.992e-05 4.992e-05 4.992e-05 0.01%
MLMG::mgVcycle_up::2 82 4.946e-05 4.946e-05 4.946e-05 0.01%
MLCellLinOp::solutionResidual() 93 4.894e-05 4.894e-05 4.894e-05 0.01%
Castro::advance() 10 4.304e-05 4.304e-05 4.304e-05 0.00%
Castro::swap_state_time_levels() 10 3.842e-05 3.842e-05 3.842e-05 0.00%
Castro::finalize_do_advance() 10 3.739e-05 3.739e-05 3.739e-05 0.00%
MLMG::ResNormInf() 93 3.727e-05 3.727e-05 3.727e-05 0.00%
Castro::enforce_consistent_e() 1 3.42e-05 3.42e-05 3.42e-05 0.00%
MLMG::mgVcycle_bottom 82 3.259e-05 3.259e-05 3.259e-05 0.00%
Gravity::solve_phi_with_mlmg() 11 3.069e-05 3.069e-05 3.069e-05 0.00%
FillPatchSingleLevel 41 2.925e-05 2.925e-05 2.925e-05 0.00%
StateData::define() 4 2.854e-05 2.854e-05 2.854e-05 0.00%
MLMG::computeResidual() 82 2.845e-05 2.845e-05 2.845e-05 0.00%
Amr::writeSmallPlotFile() 1 2.777e-05 2.777e-05 2.777e-05 0.00%
Castro::construct_new_gravity() 10 2.417e-05 2.417e-05 2.417e-05 0.00%
MLPoisson::define() 11 2.289e-05 2.289e-05 2.289e-05 0.00%
makeSFC 55 2.271e-05 2.271e-05 2.271e-05 0.00%
Castro::initMFs() 1 2.123e-05 2.123e-05 2.123e-05 0.00%
Amr::FinalizeInit() 1 2.045e-05 2.045e-05 2.045e-05 0.00%
Castro::construct_old_source() 50 1.86e-05 1.86e-05 1.86e-05 0.00%
Castro::construct_new_source() 50 1.827e-05 1.827e-05 1.827e-05 0.00%
Castro::buildMetrics() 1 1.782e-05 1.782e-05 1.782e-05 0.00%
Gravity::multilevel_solve_for_new_phi() 1 1.745e-05 1.745e-05 1.745e-05 0.00%
Castro::do_new_sources() 10 1.642e-05 1.642e-05 1.642e-05 0.00%
AmrLevel::AmrLevel(dm) 1 1.551e-05 1.551e-05 1.551e-05 0.00%
DistributionMapping::Distribute() 56 1.537e-05 1.537e-05 1.537e-05 0.00%
Castro::do_old_sources() 10 1.531e-05 1.531e-05 1.531e-05 0.00%
MLLinOp::define() 11 1.483e-05 1.483e-05 1.483e-05 0.00%
Amr::defBaseLevel() 1 1.442e-05 1.442e-05 1.442e-05 0.00%
MLLinOp::makeAgglomeratedDMap 11 1.392e-05 1.392e-05 1.392e-05 0.00%
Amr::InitializeInit() 1 1.099e-05 1.099e-05 1.099e-05 0.00%
Castro::post_init() 1 1.086e-05 1.086e-05 1.086e-05 0.00%
Castro::check_for_nan() 20 1.032e-05 1.032e-05 1.032e-05 0.00%
Castro::apply_source_to_state() 20 9.663e-06 9.663e-06 9.663e-06 0.00%
Castro::post_timestep() 10 9.6e-06 9.6e-06 9.6e-06 0.00%
Castro::construct_old_gravity() 10 9.272e-06 9.272e-06 9.272e-06 0.00%
Gravity::swapTimeLevels() 10 8.847e-06 8.847e-06 8.847e-06 0.00%
MLPoisson::prepareForSolve() 11 8.716e-06 8.716e-06 8.716e-06 0.00%
Amr::initSubcycle() 1 8.458e-06 8.458e-06 8.458e-06 0.00%
Gravity::actual_multilevel_solve() 1 8.272e-06 8.272e-06 8.272e-06 0.00%
MLMG::computeMLResidual() 11 7.969e-06 7.969e-06 7.969e-06 0.00%
Castro::computeNewDt() 9 7.206e-06 7.206e-06 7.206e-06 0.00%
MLMG::getGradSolution() 11 5.764e-06 5.764e-06 5.764e-06 0.00%
AmrLevel::checkPointPost() 3 5.391e-06 5.391e-06 5.391e-06 0.00%
Castro::create_source_corrector() 10 4.495e-06 4.495e-06 4.495e-06 0.00%
Gravity::set_mass_offset() 11 4.127e-06 4.127e-06 4.127e-06 0.00%
Castro::retry_advance_ctu() 10 3.985e-06 3.985e-06 3.985e-06 0.00%
MLMG::MLRhsNormInf() 11 3.693e-06 3.693e-06 3.693e-06 0.00%
MLMG::MLResNormInf() 11 3.379e-06 3.379e-06 3.379e-06 0.00%
Castro::FluxRegCrseInit 10 3.202e-06 3.202e-06 3.202e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 2.783e-06 2.783e-06 2.783e-06 0.00%
Castro::computeInitialDt() 2 2.422e-06 2.422e-06 2.422e-06 0.00%
Amr::init() 1 2.206e-06 2.206e-06 2.206e-06 0.00%
Castro::FluxRegFineAdd() 10 2.145e-06 2.145e-06 2.145e-06 0.00%
MLLinOp::makeSubCommunicator() 11 1.972e-06 1.972e-06 1.972e-06 0.00%
AmrLevel::checkPointPre() 3 1.726e-06 1.726e-06 1.726e-06 0.00%
Castro::post_regrid() 1 1.142e-06 1.142e-06 1.142e-06 0.00%
Amr::initialInit() 1 9.45e-07 9.45e-07 9.45e-07 0.00%
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Name NCalls Incl. Min Incl. Avg Incl. Max Max %
--------------------------------------------------------------------------------------------
main() 1 0.8681 0.8681 0.8681 100.00%
Amr::coarseTimeStep() 10 0.6924 0.6924 0.6924 79.76%
Amr::timeStep() 10 0.5735 0.5735 0.5735 66.07%
Castro::advance() 10 0.5674 0.5674 0.5674 65.36%
Castro::subcycle_advance_ctu() 10 0.5552 0.5552 0.5552 63.96%
Castro::do_advance_ctu() 10 0.5551 0.5551 0.5551 63.94%
Gravity::solve_phi_with_mlmg() 11 0.2878 0.2878 0.2878 33.16%
Gravity::actual_solve_with_mlmg() 11 0.2808 0.2808 0.2808 32.34%
Castro::construct_new_gravity() 10 0.2635 0.2635 0.2635 30.35%
MLMG::solve() 11 0.2601 0.2601 0.2601 29.97%
Gravity::solve_for_phi() 10 0.2474 0.2474 0.2474 28.49%
MLMG::oneIter() 82 0.2456 0.2456 0.2456 28.29%
MLMG::mgVcycle() 82 0.242 0.242 0.242 27.88%
VisMF::Write(FabArray) 11 0.2356 0.2356 0.2356 27.13%
Castro::construct_ctu_hydro_source() 10 0.2139 0.2139 0.2139 24.64%
Amr::checkPoint() 3 0.1752 0.1752 0.1752 20.19%
AmrLevel::checkPoint() 3 0.1712 0.1712 0.1712 19.72%
StateData::checkPoint() 12 0.1711 0.1711 0.1711 19.71%
Amr::init() 1 0.141 0.141 0.141 16.25%
MLCellLinOp::smooth() 1640 0.119 0.119 0.119 13.70%
MLCellLinOp::applyBC() 4433 0.1117 0.1117 0.1117 12.87%
MLMG::mgVcycle_bottom 82 0.07485 0.07485 0.07485 8.62%
MLMG::actualBottomSolve() 82 0.07482 0.07482 0.07482 8.62%
MLCGSolver::bicgstab 82 0.07411 0.07411 0.07411 8.54%
Amr::writePlotFile() 2 0.06746 0.06746 0.06746 7.77%
Amr::initialInit() 1 0.0485 0.0485 0.0485 5.59%
Castro::clean_state() 62 0.04441 0.04441 0.04441 5.12%
Amr::FinalizeInit() 1 0.04435 0.04435 0.04435 5.11%
FillPatchIterator::Initialize 41 0.04351 0.04351 0.04351 5.01%
Castro::post_init() 1 0.04305 0.04305 0.04305 4.96%
FillPatchSingleLevel 41 0.04196 0.04196 0.04196 4.83%
Gravity::multilevel_solve_for_new_phi() 1 0.04095 0.04095 0.04095 4.72%
Gravity::actual_multilevel_solve() 1 0.04093 0.04093 0.04093 4.72%
StateDataPhysBCFunct::() 41 0.03791 0.03791 0.03791 4.37%
MLCellLinOp::apply() 1142 0.03702 0.03702 0.03702 4.26%
MLMG::mgVcycle_down::0 82 0.03448 0.03448 0.03448 3.97%
FabArray::FillBoundary() 4023 0.03332 0.03332 0.03332 3.84%
MLPoisson::Fsmooth() 3280 0.03268 0.03268 0.03268 3.76%
FillBoundary_nowait() 4023 0.03251 0.03251 0.03251 3.74%
MLMG::mgVcycle_up::0 82 0.02619 0.02619 0.02619 3.02%
StateData::FillBoundary(geom) 328 0.02541 0.02541 0.02541 2.93%
MLCellLinOp::correctionResidual() 492 0.02256 0.02256 0.02256 2.60%
amrex::Dot() 1114 0.02094 0.02094 0.02094 2.41%
Castro::computeTemp() 63 0.02016 0.02016 0.02016 2.32%
MLMG:computeResOfCorrection() 410 0.01987 0.01987 0.01987 2.29%
Castro::initialize_do_advance() 10 0.01888 0.01888 0.01888 2.17%
Gravity::get_new_grav_vector() 11 0.01789 0.01789 0.01789 2.06%
MLPoisson::define() 11 0.01635 0.01635 0.01635 1.88%
MLMG::mgVcycle_down::1 82 0.01605 0.01605 0.01605 1.85%
Castro::construct_old_gravity() 10 0.01531 0.01531 0.01531 1.76%
Gravity::get_old_grav_vector() 10 0.0153 0.0153 0.0153 1.76%
MLMG::mgVcycle_down::2 82 0.01496 0.01496 0.01496 1.72%
amrex::Copy() 1029 0.01491 0.01491 0.01491 1.72%
FabArray::ParallelCopy() 861 0.01473 0.01473 0.01473 1.70%
FabArray::norminf() 743 0.01466 0.01466 0.01466 1.69%
MLMG::mgVcycle_down::3 82 0.01463 0.01463 0.01463 1.69%
FabArray::ParallelCopy_nowait() 861 0.01449 0.01449 0.01449 1.67%
MLMG::mgVcycle_down::4 82 0.01443 0.01443 0.01443 1.66%
Castro::normalize_species() 62 0.01385 0.01385 0.01385 1.60%
FabArray::setVal() 1144 0.01328 0.01328 0.01328 1.53%
MLCGSolver::ParallelAllReduce 1514 0.01255 0.01255 0.01255 1.45%
Castro::do_new_sources() 10 0.01239 0.01239 0.01239 1.43%
MLMG::addInterpCorrection() 410 0.01216 0.01216 0.01216 1.40%
MLMG::mgVcycle_up::4 82 0.01173 0.01173 0.01173 1.35%
MLMG::mgVcycle_up::1 82 0.01165 0.01165 0.01165 1.34%
Castro::initialize_advance() 10 0.01149 0.01149 0.01149 1.32%
Castro::expand_state() 10 0.01144 0.01144 0.01144 1.32%
MLMG::mgVcycle_up::2 82 0.01142 0.01142 0.01142 1.32%
amrex::average_down 410 0.0114 0.0114 0.0114 1.31%
MLMG::mgVcycle_up::3 82 0.01117 0.01117 0.01117 1.29%
MLCellLinOp::defineAuxData() 11 0.01112 0.01112 0.01112 1.28%
Castro::do_old_sources() 10 0.01083 0.01083 0.01083 1.25%
MLPoisson::Fapply() 1142 0.01052 0.01052 0.01052 1.21%
Castro::enforce_min_density() 62 0.009708 0.009708 0.009708 1.12%
FabArray::Saxpy() 813 0.0082 0.0082 0.0082 0.94%
FabArray::Xpay() 821 0.008153 0.008153 0.008153 0.94%
MLCellLinOp::solutionResidual() 93 0.007312 0.007312 0.007312 0.84%
Gravity::fill_multipole_BCs() 11 0.006827 0.006827 0.006827 0.79%
MLMG::computeResidual() 82 0.006309 0.006309 0.006309 0.73%
Castro::post_timestep() 10 0.005977 0.005977 0.005977 0.69%
Castro::reset_internal_energy(MultiFab) 63 0.005518 0.005518 0.005518 0.64%
MLCellLinOp::defineBC() 11 0.004975 0.004975 0.004975 0.57%
BndryData::define() 11 0.004761 0.004761 0.004761 0.55%
MLMG::prepareForSolve() 11 0.004745 0.004745 0.004745 0.55%
FabArray::LinComb() 557 0.004566 0.004566 0.004566 0.53%
amrex::Add() 164 0.004337 0.004337 0.004337 0.50%
Amr::InitializeInit() 1 0.004149 0.004149 0.004149 0.48%
Amr::defBaseLevel() 1 0.004138 0.004138 0.004138 0.48%
Castro::estTimeStep() 21 0.003915 0.003915 0.003915 0.45%
Castro::initData() 1 0.003622 0.003622 0.003622 0.42%
Castro::construct_new_source() 50 0.003264 0.003264 0.003264 0.38%
Castro::construct_new_gravity_source() 10 0.003246 0.003246 0.003246 0.37%
Castro::construct_old_source() 50 0.002682 0.002682 0.002682 0.31%
Castro::construct_old_gravity_source() 10 0.002663 0.002663 0.002663 0.31%
MLMG::ResNormInf() 93 0.002127 0.002127 0.002127 0.25%
Castro::apply_source_to_state() 20 0.001822 0.001822 0.001822 0.21%
FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001724 0.001724 0.001724 0.20%
Castro::reset_internal_energy(Fab) 504 0.001511 0.001511 0.001511 0.17%
FabArrayBase::getCPC() 1323 0.001434 0.001434 0.001434 0.17%
Castro::computeNewDt() 9 0.001421 0.001421 0.001421 0.16%
Castro::enforce_speed_limit() 62 0.001421 0.001421 0.001421 0.16%
MLMG::getGradSolution() 11 0.00142 0.00142 0.00142 0.16%
MLCellLinOp::compGrad() 11 0.001414 0.001414 0.001414 0.16%
MLCellLinOp::setLevelBC() 11 0.001388 0.001388 0.001388 0.16%
FabArray::mult() 43 0.00133 0.00133 0.00133 0.15%
FabArray::setDomainBndry() 41 0.001314 0.001314 0.001314 0.15%
Castro::check_for_nan() 20 0.001208 0.001208 0.001208 0.14%
MultiFab::contains_nan() 20 0.001198 0.001198 0.001198 0.14%
MLPoisson::prepareForSolve() 11 0.00115 0.00115 0.00115 0.13%
MLCellLinOp::prepareForSolve() 11 0.001142 0.001142 0.001142 0.13%
Castro::post_regrid() 1 0.001122 0.001122 0.001122 0.13%
MLMG::computeMLResidual() 11 0.001039 0.001039 0.001039 0.12%
Castro::computeInitialDt() 2 0.0008849 0.0008849 0.0008849 0.10%
Gravity::update_max_rhs() 11 0.0008397 0.0008397 0.0008397 0.10%
FabArrayBase::CPC::define() 454 0.0006817 0.0006817 0.0006817 0.08%
FabArrayBase::getFB() 4023 0.0006713 0.0006713 0.0006713 0.08%
Castro::finalize_advance() 10 0.0006285 0.0006285 0.0006285 0.07%
Amr::InitAmr() 1 0.0005174 0.0005174 0.0005174 0.06%
Castro::Castro() 1 0.0004536 0.0004536 0.0004536 0.05%
Gravity::swapTimeLevels() 10 0.0004475 0.0004475 0.0004475 0.05%
MLMG::MLResNormInf() 11 0.0002863 0.0002863 0.0002863 0.03%
MultiFab::max() 11 0.0002626 0.0002626 0.0002626 0.03%
MLLinOp::define() 11 0.0002278 0.0002278 0.0002278 0.03%
MLMG::MLRhsNormInf() 11 0.0002245 0.0002245 0.0002245 0.03%
MLLinOp::defineGrids() 11 0.0002129 0.0002129 0.0002129 0.02%
Castro::buildMetrics() 1 0.0001537 0.0001537 0.0001537 0.02%
FabArrayBase::FB::FB() 56 8.722e-05 8.722e-05 8.722e-05 0.01%
MLLinOp::makeAgglomeratedDMap 11 5.082e-05 5.082e-05 5.082e-05 0.01%
AmrLevel::AmrLevel(dm) 1 4.405e-05 4.405e-05 4.405e-05 0.01%
Castro::swap_state_time_levels() 10 3.842e-05 3.842e-05 3.842e-05 0.00%
Castro::finalize_do_advance() 10 3.739e-05 3.739e-05 3.739e-05 0.00%
makeSFC 55 3.69e-05 3.69e-05 3.69e-05 0.00%
Castro::enforce_consistent_e() 1 3.42e-05 3.42e-05 3.42e-05 0.00%
StateData::define() 4 2.854e-05 2.854e-05 2.854e-05 0.00%
Amr::writeSmallPlotFile() 1 2.777e-05 2.777e-05 2.777e-05 0.00%
Castro::initMFs() 1 2.123e-05 2.123e-05 2.123e-05 0.00%
DistributionMapping::Distribute() 56 1.537e-05 1.537e-05 1.537e-05 0.00%
Amr::initSubcycle() 1 8.458e-06 8.458e-06 8.458e-06 0.00%
AmrLevel::checkPointPost() 3 5.391e-06 5.391e-06 5.391e-06 0.00%
Castro::create_source_corrector() 10 4.495e-06 4.495e-06 4.495e-06 0.00%
Gravity::set_mass_offset() 11 4.127e-06 4.127e-06 4.127e-06 0.00%
Castro::retry_advance_ctu() 10 3.985e-06 3.985e-06 3.985e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 3.971e-06 3.971e-06 3.971e-06 0.00%
Castro::FluxRegCrseInit 10 3.202e-06 3.202e-06 3.202e-06 0.00%
Castro::FluxRegFineAdd() 10 2.145e-06 2.145e-06 2.145e-06 0.00%
MLLinOp::makeSubCommunicator() 11 1.972e-06 1.972e-06 1.972e-06 0.00%
AmrLevel::checkPointPre() 3 1.726e-06 1.726e-06 1.726e-06 0.00%
--------------------------------------------------------------------------------------------
Device Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Arena::Initialize() 1 1 99 MiB 9042 MiB
Castro::construct_ctu_hydro_source() 2880 2880 114 MiB 692 MiB
Castro::initMFs() 48 48 67 MiB 68 MiB
Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB
StateData::define() 32 32 55 MiB 55 MiB
FillPatchIterator::Initialize 328 328 1059 KiB 39 MiB
Castro::initialize_do_advance() 80 80 24 MiB 39 MiB
ResizeRandomSeed 1 1 30 MiB 30 MiB
Amr::writePlotFile() 16 16 2183 KiB 28 MiB
Castro::initialize_advance() 80 80 15 MiB 23 MiB
Castro::buildMetrics() 32 32 15 MiB 15 MiB
Castro::Castro() 48 48 7611 KiB 14 MiB
MLMG::prepareForSolve() 660 660 3687 KiB 12 MiB
Gravity::get_new_grav_vector() 91 91 214 KiB 10 MiB
Gravity::get_old_grav_vector() 80 80 180 KiB 10 MiB
Gravity::multilevel_solve_for_new_phi() 24 24 7524 KiB 7586 KiB
Gravity::fill_multipole_BCs() 154 154 12 KiB 2053 KiB
Gravity::update_max_rhs() 88 88 1855 B 2048 KiB
Gravity::solve_for_phi() 80 80 582 KiB 2048 KiB
Gravity::actual_multilevel_solve() 8 8 96 KiB 2048 KiB
BndryData::define() 1056 1056 336 KiB 1095 KiB
MLCellLinOp::defineAuxData() 1716 1716 214 KiB 671 KiB
Castro::estTimeStep() 21 21 2462 B 540 KiB
VisMF::Write(FabArray) 656 656 3901 B 360 KiB
Castro::normalize_species() 62 62 5839 B 360 KiB
amrex::average_down 1067 1067 1304 B 257 KiB
MLMG::addInterpCorrection() 1066 1066 1206 B 257 KiB
amrex::Dot() 1360 1360 3992 B 180 KiB
FabArray::norminf() 907 907 2805 B 180 KiB
Castro::do_advance_ctu() 10 10 679 B 180 KiB
MultiFab::max() 11 11 52 B 180 KiB
Castro::initData() 1 1 31 B 180 KiB
FabArray::setVal() 106 106 21 KiB 26 KiB
MultiFab::contains_nan() 20 20 30 B 22 KiB
MLPoisson::Fsmooth() 132 132 3604 B 12 KiB
FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB
FillBoundary_nowait() 760 760 321 B 9648 B
MLCellLinOp::applyBC() 8866 8866 232 B 9344 B
MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B
amrex::Copy() 100 100 3866 B 6144 B
StateData::FillBoundary(geom) 1992 1992 50 B 2976 B
FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B
MLCGSolver::bicgstab 738 738 124 B 1472 B
MLCellLinOp::defineBC() 66 66 379 B 1248 B
------------------------------------------------------------------------------
Managed Memory Usage:
-----------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
-----------------------------------------------------------------
The_Managed_Arena::Initialize() 1 1 540 B 8192 KiB
-----------------------------------------------------------------
Pinned Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Pinned_Arena::Initialize() 1 1 51 KiB 8192 KiB
VisMF::Write(FabArray) 744 744 635 KiB 3584 KiB
FabArray::setVal() 106 106 21 KiB 26 KiB
MLPoisson::Fsmooth() 132 132 3604 B 12 KiB
FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB
FillBoundary_nowait() 760 760 321 B 9648 B
MLCellLinOp::applyBC() 4433 4433 230 B 9328 B
MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B
amrex::Copy() 100 100 3866 B 6144 B
StateData::FillBoundary(geom) 1992 1992 50 B 3072 B
Gravity::get_new_grav_vector() 3 3 2901 B 3072 B
Gravity::fill_multipole_BCs() 33 33 4 B 2832 B
FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B
amrex::average_down 83 83 272 B 1296 B
MLMG::addInterpCorrection() 82 82 2 B 1024 B
MLMG::prepareForSolve() 11 11 307 B 1024 B
MLCellLinOp::setLevelBC() 66 66 0 B 768 B
amrex::Dot() 1360 1360 25 B 400 B
FabArray::norminf() 907 907 18 B 272 B
Castro::estTimeStep() 21 21 0 B 32 B
MultiFab::contains_nan() 20 20 0 B 16 B
MultiFab::max() 11 11 0 B 16 B
Castro::normalize_species() 62 62 0 B 16 B
Castro::initData() 1 1 0 B 16 B
Castro::do_advance_ctu() 10 10 0 B 16 B
------------------------------------------------------------------------------
Cpu (amrex_mempool) Memory Usage:
-----------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
-----------------------------------------------------
amrex_mempool_init() 1 1 56 KiB 8192 KiB
-----------------------------------------------------
Unused ParmParse Variables:
[TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2]
[TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2]
Total GPU global memory (MB): 12056
Free GPU global memory (MB): 2459
[The Arena] space allocated (MB): 9042
[The Arena] space used (MB): 0
[The Managed Arena] space allocated (MB): 8
[The Managed Arena] space used (MB): 0
[The Pinned Arena] space allocated (MB): 8
[The Pinned Arena] space used (MB): 0
AMReX (23.02-16-g0c444b898f8a) finalized
Initializing CUDA...
CUDA initialized with 1 device.
AMReX (23.02-16-g0c444b898f8a) initialized
Starting run at 09:58:20 UTC on 2023-02-09.
Successfully read inputs file ...
Castro git describe: 23.01-29-g09845e315
AMReX git describe: 23.02-16-g0c444b898
Microphysics git describe: 23.02-6-gb79fe5f7
reading extern runtime parameters ...
3 Species:
C12 O16 Mg24
restarting calculation from file: dustcollapse-restart_chk00005
Successfully read inputs file ...
read CPU time: 0.469850322
Restart time = 0.047263365 seconds.
[Level 0 step 6] ADVANCE with dt = 5.79654185e-05
[Level 0 step 6] Advanced 262144 cells
[STEP 6] Coarse TimeStep time: 0.049050396
[STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05
[Level 0 step 7] ADVANCE with dt = 6.086368943e-05
[Level 0 step 7] Advanced 262144 cells
[STEP 7] Coarse TimeStep time: 0.049440267
[STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05
[Level 0 step 8] ADVANCE with dt = 6.39068739e-05
[Level 0 step 8] Advanced 262144 cells
[STEP 8] Coarse TimeStep time: 0.059346138
[STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05
[Level 0 step 9] ADVANCE with dt = 6.71022176e-05
[Level 0 step 9] Advanced 262144 cells
[STEP 9] Coarse TimeStep time: 0.056766495
[STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05
[Level 0 step 10] ADVANCE with dt = 7.045732848e-05
[Level 0 step 10] Advanced 262144 cells
[STEP 10] Coarse TimeStep time: 0.061652229
[STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638]
STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05
PLOTFILE: file = dustcollapse-restart_plt00010
Write plotfile time = 0.033373193 seconds
Ending run at 09:58:20 UTC on 2023-02-09.
Run time = 0.358027955
Run time without initialization = 0.310096068
Average number of zones advanced per microsecond: 4.227
Average number of zones advanced per microsecond per rank: 4.227
CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472
TinyProfiler total time across processes [min...avg...max]: 0.3581 ... 0.3581 ... 0.3581
--------------------------------------------------------------------------------------------
Name NCalls Excl. Min Excl. Avg Excl. Max Max %
--------------------------------------------------------------------------------------------
Castro::construct_ctu_hydro_source() 5 0.0921 0.0921 0.0921 25.72%
VisMF::Read() 3 0.04101 0.04101 0.04101 11.45%
MLCellLinOp::applyBC() 1946 0.03329 0.03329 0.03329 9.30%
VisMF::Write(FabArray) 1 0.03177 0.03177 0.03177 8.87%
MLPoisson::Fsmooth() 1440 0.01401 0.01401 0.01401 3.91%
FillBoundary_nowait() 1766 0.01275 0.01275 0.01275 3.56%
StateData::FillBoundary(geom) 160 0.01196 0.01196 0.01196 3.34%
Castro::normalize_species() 30 0.00979 0.00979 0.00979 2.73%
amrex::Dot() 484 0.008887 0.008887 0.008887 2.48%
amrex::Copy() 463 0.006947 0.006947 0.006947 1.94%
FabArray::norminf() 326 0.006317 0.006317 0.006317 1.76%
FabArray::setVal() 537 0.006238 0.006238 0.006238 1.74%
FabArray::ParallelCopy_nowait() 380 0.006073 0.006073 0.006073 1.70%
Castro::enforce_min_density() 30 0.005946 0.005946 0.005946 1.66%
Castro::computeTemp() 30 0.005945 0.005945 0.005945 1.66%
StateDataPhysBCFunct::() 20 0.005752 0.005752 0.005752 1.61%
MLCellLinOp::defineAuxData() 6 0.005219 0.005219 0.005219 1.46%
MLPoisson::Fapply() 500 0.004495 0.004495 0.004495 1.26%
FabArray::Saxpy() 355 0.003645 0.003645 0.003645 1.02%
FabArray::Xpay() 361 0.003503 0.003503 0.003503 0.98%
MLMG::addInterpCorrection() 180 0.00289 0.00289 0.00289 0.81%
amrex::average_down 180 0.002589 0.002589 0.002589 0.72%
Gravity::fill_multipole_BCs() 6 0.002529 0.002529 0.002529 0.71%
Amr::restart() 1 0.002513 0.002513 0.002513 0.70%
Castro::estTimeStep() 10 0.002285 0.002285 0.002285 0.64%
BndryData::define() 6 0.002031 0.002031 0.002031 0.57%
FabArray::LinComb() 242 0.001952 0.001952 0.001952 0.55%
Castro::reset_internal_energy(MultiFab) 30 0.001935 0.001935 0.001935 0.54%
amrex::Add() 72 0.001844 0.001844 0.001844 0.51%
Castro::construct_new_gravity_source() 5 0.001747 0.001747 0.001747 0.49%
Castro::do_advance_ctu() 5 0.001696 0.001696 0.001696 0.47%
Amr::writePlotFile() 1 0.001431 0.001431 0.001431 0.40%
Castro::construct_old_gravity_source() 5 0.001347 0.001347 0.001347 0.38%
MLCGSolver::bicgstab 36 0.0009727 0.0009727 0.0009727 0.27%
Castro::reset_internal_energy(Fab) 240 0.0009137 0.0009137 0.0009137 0.26%
FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009049 0.0009049 0.0009049 0.25%
Gravity::actual_solve_with_mlmg() 6 0.000781 0.000781 0.000781 0.22%
Castro::finalize_advance() 5 0.0007654 0.0007654 0.0007654 0.21%
MLCellLinOp::setLevelBC() 6 0.000748 0.000748 0.000748 0.21%
FabArray::mult() 22 0.0006524 0.0006524 0.0006524 0.18%
FabArray::setDomainBndry() 20 0.000641 0.000641 0.000641 0.18%
MLCellLinOp::prepareForSolve() 6 0.0006035 0.0006035 0.0006035 0.17%
MultiFab::contains_nan() 10 0.000596 0.000596 0.000596 0.17%
Castro::enforce_speed_limit() 30 0.0005931 0.0005931 0.0005931 0.17%
MLCellLinOp::smooth() 720 0.0004909 0.0004909 0.0004909 0.14%
MLCellLinOp::compGrad() 6 0.000487 0.000487 0.000487 0.14%
MLMG::prepareForSolve() 6 0.0004657 0.0004657 0.0004657 0.13%
Amr::InitAmr() 1 0.0004482 0.0004482 0.0004482 0.13%
FabArrayBase::CPC::define() 244 0.0003945 0.0003945 0.0003945 0.11%
FabArray::FillBoundary() 1766 0.0003683 0.0003683 0.0003683 0.10%
Gravity::get_old_grav_vector() 5 0.0003493 0.0003493 0.0003493 0.10%
FabArrayBase::getCPC() 632 0.0003481 0.0003481 0.0003481 0.10%
main() 1 0.0003066 0.0003066 0.0003066 0.09%
Gravity::get_new_grav_vector() 5 0.0002735 0.0002735 0.0002735 0.08%
FabArrayBase::getFB() 1766 0.0002388 0.0002388 0.0002388 0.07%
Amr::coarseTimeStep() 5 0.0002255 0.0002255 0.0002255 0.06%
MLCellLinOp::apply() 500 0.0002016 0.0002016 0.0002016 0.06%
MLMG::mgVcycle() 36 0.0001664 0.0001664 0.0001664 0.05%
Castro::construct_new_source() 25 0.0001495 0.0001495 0.0001495 0.04%
MultiFab::max() 6 0.0001366 0.0001366 0.0001366 0.04%
MLCGSolver::ParallelAllReduce 659 0.0001274 0.0001274 0.0001274 0.04%
FillPatchIterator::Initialize 20 0.0001157 0.0001157 0.0001157 0.03%
FabArray::ParallelCopy() 380 0.0001131 0.0001131 0.0001131 0.03%
MLCellLinOp::defineBC() 6 0.0001117 0.0001117 0.0001117 0.03%
Castro::subcycle_advance_ctu() 5 0.0001013 0.0001013 0.0001013 0.03%
MLLinOp::defineGrids() 6 9.21e-05 9.21e-05 9.21e-05 0.03%
MLCellLinOp::correctionResidual() 216 8.948e-05 8.948e-05 8.948e-05 0.02%
Amr::timeStep() 5 7.492e-05 7.492e-05 7.492e-05 0.02%
AmrLevel::restart() 1 7.468e-05 7.468e-05 7.468e-05 0.02%
Gravity::solve_for_phi() 5 7.013e-05 7.013e-05 7.013e-05 0.02%
Gravity::update_max_rhs() 6 5.851e-05 5.851e-05 5.851e-05 0.02%
Castro::advance() 5 5.796e-05 5.796e-05 5.796e-05 0.02%
StateData::restartDoit() 4 5.788e-05 5.788e-05 5.788e-05 0.02%
FabArrayBase::FB::FB() 26 5.701e-05 5.701e-05 5.701e-05 0.02%
Castro::create_source_corrector() 5 5.632e-05 5.632e-05 5.632e-05 0.02%
Castro::post_timestep() 5 5.506e-05 5.506e-05 5.506e-05 0.02%
Castro::construct_new_gravity() 5 5.14e-05 5.14e-05 5.14e-05 0.01%
MLMG:computeResOfCorrection() 180 4.828e-05 4.828e-05 4.828e-05 0.01%
MLMG::actualBottomSolve() 36 4.298e-05 4.298e-05 4.298e-05 0.01%
Castro::clean_state() 30 4.211e-05 4.211e-05 4.211e-05 0.01%
Castro::initialize_do_advance() 5 4.193e-05 4.193e-05 4.193e-05 0.01%
Castro::expand_state() 5 4.136e-05 4.136e-05 4.136e-05 0.01%
MLMG::mgVcycle_down::0 36 3.961e-05 3.961e-05 3.961e-05 0.01%
MLMG::mgVcycle_down::1 36 3.797e-05 3.797e-05 3.797e-05 0.01%
MLMG::solve() 6 3.537e-05 3.537e-05 3.537e-05 0.01%
MLMG::mgVcycle_down::2 36 3.491e-05 3.491e-05 3.491e-05 0.01%
MLMG::mgVcycle_down::4 36 3.44e-05 3.44e-05 3.44e-05 0.01%
Castro::initialize_advance() 5 3.408e-05 3.408e-05 3.408e-05 0.01%
MLMG::mgVcycle_down::3 36 3.266e-05 3.266e-05 3.266e-05 0.01%
Castro::buildMetrics() 1 3.252e-05 3.252e-05 3.252e-05 0.01%
MLMG::mgVcycle_up::4 36 2.858e-05 2.858e-05 2.858e-05 0.01%
Amr::writeSmallPlotFile() 1 2.645e-05 2.645e-05 2.645e-05 0.01%
Castro::post_restart() 1 2.505e-05 2.505e-05 2.505e-05 0.01%
Castro::swap_state_time_levels() 5 2.396e-05 2.396e-05 2.396e-05 0.01%
Castro::initMFs() 1 2.383e-05 2.383e-05 2.383e-05 0.01%
MLMG::mgVcycle_up::0 36 2.34e-05 2.34e-05 2.34e-05 0.01%
MLMG::mgVcycle_up::3 36 2.296e-05 2.296e-05 2.296e-05 0.01%
MLMG::mgVcycle_up::2 36 2.25e-05 2.25e-05 2.25e-05 0.01%
MLMG::oneIter() 36 2.249e-05 2.249e-05 2.249e-05 0.01%
MLMG::mgVcycle_up::1 36 2.206e-05 2.206e-05 2.206e-05 0.01%
MLCellLinOp::solutionResidual() 42 2.089e-05 2.089e-05 2.089e-05 0.01%
Castro::finalize_do_advance() 5 1.918e-05 1.918e-05 1.918e-05 0.01%
MLMG::ResNormInf() 42 1.818e-05 1.818e-05 1.818e-05 0.01%
Gravity::multilevel_solve_for_new_phi() 1 1.582e-05 1.582e-05 1.582e-05 0.00%
MLMG::mgVcycle_bottom 36 1.446e-05 1.446e-05 1.446e-05 0.00%
FillPatchSingleLevel 20 1.424e-05 1.424e-05 1.424e-05 0.00%
MLPoisson::define() 6 1.391e-05 1.391e-05 1.391e-05 0.00%
makeSFC 30 1.339e-05 1.339e-05 1.339e-05 0.00%
MLMG::computeResidual() 36 1.327e-05 1.327e-05 1.327e-05 0.00%
Gravity::solve_phi_with_mlmg() 6 1.141e-05 1.141e-05 1.141e-05 0.00%
MLLinOp::define() 6 1.043e-05 1.043e-05 1.043e-05 0.00%
Castro::do_new_sources() 5 9.861e-06 9.861e-06 9.861e-06 0.00%
Castro::construct_old_source() 25 8.753e-06 8.753e-06 8.753e-06 0.00%
Amr::initSubcycle() 1 8.72e-06 8.72e-06 8.72e-06 0.00%
Castro::do_old_sources() 5 8.65e-06 8.65e-06 8.65e-06 0.00%
DistributionMapping::Distribute() 31 8.43e-06 8.43e-06 8.43e-06 0.00%
Gravity::actual_multilevel_solve() 1 7.914e-06 7.914e-06 7.914e-06 0.00%
MLLinOp::makeAgglomeratedDMap 6 6.922e-06 6.922e-06 6.922e-06 0.00%
Castro::construct_old_gravity() 5 5.328e-06 5.328e-06 5.328e-06 0.00%
Castro::check_for_nan() 10 5.267e-06 5.267e-06 5.267e-06 0.00%
Castro::apply_source_to_state() 10 5.107e-06 5.107e-06 5.107e-06 0.00%
MLPoisson::prepareForSolve() 6 4.201e-06 4.201e-06 4.201e-06 0.00%
Gravity::swapTimeLevels() 5 4.115e-06 4.115e-06 4.115e-06 0.00%
Castro::computeNewDt() 5 3.961e-06 3.961e-06 3.961e-06 0.00%
MLMG::computeMLResidual() 6 3.671e-06 3.671e-06 3.671e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 3.057e-06 3.057e-06 3.057e-06 0.00%
MLMG::getGradSolution() 6 2.998e-06 2.998e-06 2.998e-06 0.00%
Gravity::set_mass_offset() 6 2.747e-06 2.747e-06 2.747e-06 0.00%
MLMG::MLResNormInf() 6 2.124e-06 2.124e-06 2.124e-06 0.00%
MLMG::MLRhsNormInf() 6 2.043e-06 2.043e-06 2.043e-06 0.00%
Castro::FluxRegCrseInit 5 1.565e-06 1.565e-06 1.565e-06 0.00%
Castro::retry_advance_ctu() 5 1.463e-06 1.463e-06 1.463e-06 0.00%
Castro::FluxRegFineAdd() 5 1.264e-06 1.264e-06 1.264e-06 0.00%
MLLinOp::makeSubCommunicator() 6 1.07e-06 1.07e-06 1.07e-06 0.00%
Amr::init() 1 1.07e-06 1.07e-06 1.07e-06 0.00%
AmrLevel::AmrLevel() 1 6.62e-07 6.62e-07 6.62e-07 0.00%
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Name NCalls Incl. Min Incl. Avg Incl. Max Max %
--------------------------------------------------------------------------------------------
main() 1 0.358 0.358 0.358 100.00%
Amr::coarseTimeStep() 5 0.2765 0.2765 0.2765 77.22%
Amr::timeStep() 5 0.275 0.275 0.275 76.80%
Castro::advance() 5 0.2698 0.2698 0.2698 75.35%
Castro::subcycle_advance_ctu() 5 0.2633 0.2633 0.2633 73.53%
Castro::do_advance_ctu() 5 0.2632 0.2632 0.2632 73.50%
Castro::construct_new_gravity() 5 0.1294 0.1294 0.1294 36.14%
Gravity::solve_phi_with_mlmg() 6 0.1245 0.1245 0.1245 34.76%
Gravity::actual_solve_with_mlmg() 6 0.1217 0.1217 0.1217 33.99%
Gravity::solve_for_phi() 5 0.1214 0.1214 0.1214 33.91%
MLMG::solve() 6 0.1106 0.1106 0.1106 30.90%
MLMG::oneIter() 36 0.1038 0.1038 0.1038 28.98%
MLMG::mgVcycle() 36 0.1023 0.1023 0.1023 28.56%
Castro::construct_ctu_hydro_source() 5 0.0921 0.0921 0.0921 25.72%
MLCellLinOp::smooth() 720 0.04993 0.04993 0.04993 13.94%
Amr::init() 1 0.04732 0.04732 0.04732 13.22%
Amr::restart() 1 0.04732 0.04732 0.04732 13.22%
MLCellLinOp::applyBC() 1946 0.0467 0.0467 0.0467 13.04%
AmrLevel::restart() 1 0.0412 0.0412 0.0412 11.51%
StateData::restartDoit() 4 0.04112 0.04112 0.04112 11.48%
VisMF::Read() 3 0.04101 0.04101 0.04101 11.45%
Amr::writePlotFile() 1 0.03346 0.03346 0.03346 9.34%
MLMG::mgVcycle_bottom 36 0.03195 0.03195 0.03195 8.92%
MLMG::actualBottomSolve() 36 0.03194 0.03194 0.03194 8.92%
VisMF::Write(FabArray) 1 0.03177 0.03177 0.03177 8.87%
MLCGSolver::bicgstab 36 0.03163 0.03163 0.03163 8.83%
Castro::clean_state() 30 0.02517 0.02517 0.02517 7.03%
FillPatchIterator::Initialize 20 0.02048 0.02048 0.02048 5.72%
FillPatchSingleLevel 20 0.01972 0.01972 0.01972 5.51%
StateDataPhysBCFunct::() 20 0.01771 0.01771 0.01771 4.95%
MLCellLinOp::apply() 500 0.01571 0.01571 0.01571 4.39%
MLMG::mgVcycle_down::0 36 0.01424 0.01424 0.01424 3.98%
MLPoisson::Fsmooth() 1440 0.01401 0.01401 0.01401 3.91%
FabArray::FillBoundary() 1766 0.01342 0.01342 0.01342 3.75%
FillBoundary_nowait() 1766 0.01305 0.01305 0.01305 3.64%
StateData::FillBoundary(geom) 160 0.01196 0.01196 0.01196 3.34%
MLMG::mgVcycle_up::0 36 0.01078 0.01078 0.01078 3.01%
Castro::initialize_do_advance() 5 0.009837 0.009837 0.009837 2.75%
Castro::normalize_species() 30 0.00979 0.00979 0.00979 2.73%
MLCellLinOp::correctionResidual() 216 0.009481 0.009481 0.009481 2.65%
amrex::Dot() 484 0.008887 0.008887 0.008887 2.48%
Castro::computeTemp() 30 0.008794 0.008794 0.008794 2.46%
MLPoisson::define() 6 0.008784 0.008784 0.008784 2.45%
MLMG:computeResOfCorrection() 180 0.008315 0.008315 0.008315 2.32%
Gravity::get_new_grav_vector() 5 0.007856 0.007856 0.007856 2.19%
Castro::construct_old_gravity() 5 0.007682 0.007682 0.007682 2.15%
Gravity::get_old_grav_vector() 5 0.007676 0.007676 0.007676 2.14%
Castro::do_new_sources() 5 0.007617 0.007617 0.007617 2.13%
amrex::Copy() 463 0.006947 0.006947 0.006947 1.94%
MLMG::mgVcycle_down::1 36 0.006823 0.006823 0.006823 1.91%
FabArray::ParallelCopy() 380 0.006557 0.006557 0.006557 1.83%
FabArray::ParallelCopy_nowait() 380 0.006444 0.006444 0.006444 1.80%
Castro::do_old_sources() 5 0.006357 0.006357 0.006357 1.78%
MLMG::mgVcycle_down::2 36 0.006339 0.006339 0.006339 1.77%
FabArray::norminf() 326 0.006317 0.006317 0.006317 1.76%
MLMG::mgVcycle_down::3 36 0.00624 0.00624 0.00624 1.74%
FabArray::setVal() 537 0.006238 0.006238 0.006238 1.74%
MLMG::mgVcycle_down::4 36 0.006142 0.006142 0.006142 1.72%
MLCellLinOp::defineAuxData() 6 0.005948 0.005948 0.005948 1.66%
Castro::enforce_min_density() 30 0.005946 0.005946 0.005946 1.66%
Castro::expand_state() 5 0.00553 0.00553 0.00553 1.54%
Castro::initialize_advance() 5 0.005426 0.005426 0.005426 1.52%
MLCGSolver::ParallelAllReduce 659 0.005364 0.005364 0.005364 1.50%
MLMG::addInterpCorrection() 180 0.00518 0.00518 0.00518 1.45%
Castro::post_timestep() 5 0.005128 0.005128 0.005128 1.43%
MLMG::mgVcycle_up::4 36 0.004978 0.004978 0.004978 1.39%
MLMG::mgVcycle_up::1 36 0.004971 0.004971 0.004971 1.39%
MLMG::mgVcycle_up::2 36 0.004862 0.004862 0.004862 1.36%
amrex::average_down 180 0.004857 0.004857 0.004857 1.36%
MLMG::mgVcycle_up::3 36 0.004768 0.004768 0.004768 1.33%
MLPoisson::Fapply() 500 0.004495 0.004495 0.004495 1.26%
FabArray::Saxpy() 355 0.003645 0.003645 0.003645 1.02%
FabArray::Xpay() 361 0.003503 0.003503 0.003503 0.98%
Castro::post_restart() 1 0.003431 0.003431 0.003431 0.96%
Gravity::multilevel_solve_for_new_phi() 1 0.00331 0.00331 0.00331 0.92%
Gravity::actual_multilevel_solve() 1 0.003294 0.003294 0.003294 0.92%
MLCellLinOp::solutionResidual() 42 0.003238 0.003238 0.003238 0.90%
Castro::reset_internal_energy(MultiFab) 30 0.002849 0.002849 0.002849 0.80%
MLCellLinOp::defineBC() 6 0.002691 0.002691 0.002691 0.75%
MLMG::computeResidual() 36 0.00269 0.00269 0.00269 0.75%
Gravity::fill_multipole_BCs() 6 0.002654 0.002654 0.002654 0.74%
BndryData::define() 6 0.00258 0.00258 0.00258 0.72%
MLMG::prepareForSolve() 6 0.002498 0.002498 0.002498 0.70%
Castro::estTimeStep() 10 0.002285 0.002285 0.002285 0.64%
FabArray::LinComb() 242 0.001952 0.001952 0.001952 0.55%
Castro::construct_new_source() 25 0.001897 0.001897 0.001897 0.53%
amrex::Add() 72 0.001844 0.001844 0.001844 0.51%
Castro::construct_new_gravity_source() 5 0.001747 0.001747 0.001747 0.49%
Castro::construct_old_source() 25 0.001356 0.001356 0.001356 0.38%
Castro::construct_old_gravity_source() 5 0.001347 0.001347 0.001347 0.38%
Castro::computeNewDt() 5 0.001249 0.001249 0.001249 0.35%
Castro::finalize_advance() 5 0.001025 0.001025 0.001025 0.29%
MLMG::ResNormInf() 42 0.0009436 0.0009436 0.0009436 0.26%
Castro::apply_source_to_state() 10 0.0009245 0.0009245 0.0009245 0.26%
Castro::reset_internal_energy(Fab) 240 0.0009137 0.0009137 0.0009137 0.26%
FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009049 0.0009049 0.0009049 0.25%
MLMG::getGradSolution() 6 0.0007517 0.0007517 0.0007517 0.21%
MLCellLinOp::compGrad() 6 0.0007487 0.0007487 0.0007487 0.21%
MLCellLinOp::setLevelBC() 6 0.000748 0.000748 0.000748 0.21%
FabArrayBase::getCPC() 632 0.0007426 0.0007426 0.0007426 0.21%
FabArray::mult() 22 0.0006524 0.0006524 0.0006524 0.18%
FabArray::setDomainBndry() 20 0.000641 0.000641 0.000641 0.18%
MLPoisson::prepareForSolve() 6 0.0006077 0.0006077 0.0006077 0.17%
MLCellLinOp::prepareForSolve() 6 0.0006035 0.0006035 0.0006035 0.17%
Castro::check_for_nan() 10 0.0006013 0.0006013 0.0006013 0.17%
MultiFab::contains_nan() 10 0.000596 0.000596 0.000596 0.17%
Castro::enforce_speed_limit() 30 0.0005931 0.0005931 0.0005931 0.17%
MLMG::computeMLResidual() 6 0.0005654 0.0005654 0.0005654 0.16%
Amr::InitAmr() 1 0.0004569 0.0004569 0.0004569 0.13%
Gravity::update_max_rhs() 6 0.000453 0.000453 0.000453 0.13%
FabArrayBase::CPC::define() 244 0.0003945 0.0003945 0.0003945 0.11%
FabArrayBase::getFB() 1766 0.0002958 0.0002958 0.0002958 0.08%
Gravity::swapTimeLevels() 5 0.0002262 0.0002262 0.0002262 0.06%
Castro::buildMetrics() 1 0.000152 0.000152 0.000152 0.04%
MLMG::MLResNormInf() 6 0.0001516 0.0001516 0.0001516 0.04%
MultiFab::max() 6 0.0001366 0.0001366 0.0001366 0.04%
MLLinOp::define() 6 0.000131 0.000131 0.000131 0.04%
MLLinOp::defineGrids() 6 0.0001206 0.0001206 0.0001206 0.03%
MLMG::MLRhsNormInf() 6 0.0001174 0.0001174 0.0001174 0.03%
FabArrayBase::FB::FB() 26 5.701e-05 5.701e-05 5.701e-05 0.02%
Castro::create_source_corrector() 5 5.632e-05 5.632e-05 5.632e-05 0.02%
MLLinOp::makeAgglomeratedDMap 6 2.74e-05 2.74e-05 2.74e-05 0.01%
Amr::writeSmallPlotFile() 1 2.645e-05 2.645e-05 2.645e-05 0.01%
Castro::swap_state_time_levels() 5 2.396e-05 2.396e-05 2.396e-05 0.01%
Castro::initMFs() 1 2.383e-05 2.383e-05 2.383e-05 0.01%
makeSFC 30 2.048e-05 2.048e-05 2.048e-05 0.01%
Castro::finalize_do_advance() 5 1.918e-05 1.918e-05 1.918e-05 0.01%
Amr::initSubcycle() 1 8.72e-06 8.72e-06 8.72e-06 0.00%
DistributionMapping::Distribute() 31 8.43e-06 8.43e-06 8.43e-06 0.00%
DistributionMapping::SFCProcessorMapDoIt() 1 4.402e-06 4.402e-06 4.402e-06 0.00%
Gravity::set_mass_offset() 6 2.747e-06 2.747e-06 2.747e-06 0.00%
Castro::FluxRegCrseInit 5 1.565e-06 1.565e-06 1.565e-06 0.00%
Castro::retry_advance_ctu() 5 1.463e-06 1.463e-06 1.463e-06 0.00%
Castro::FluxRegFineAdd() 5 1.264e-06 1.264e-06 1.264e-06 0.00%
MLLinOp::makeSubCommunicator() 6 1.07e-06 1.07e-06 1.07e-06 0.00%
AmrLevel::AmrLevel() 1 6.62e-07 6.62e-07 6.62e-07 0.00%
--------------------------------------------------------------------------------------------
Device Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Arena::Initialize() 1 1 242 MiB 9042 MiB
Castro::construct_ctu_hydro_source() 1440 1440 112 MiB 692 MiB
Castro::initMFs() 48 48 59 MiB 68 MiB
StateData::restartDoit() 32 32 53 MiB 55 MiB
Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB
Castro::initialize_do_advance() 40 40 28 MiB 39 MiB
FillPatchIterator::Initialize 160 160 1226 KiB 39 MiB
ResizeRandomSeed 1 1 30 MiB 30 MiB
Amr::writePlotFile() 8 8 2613 KiB 28 MiB
Castro::initialize_advance() 40 40 17 MiB 23 MiB
Castro::buildMetrics() 32 32 13 MiB 15 MiB
Castro::post_restart() 48 48 6631 KiB 14 MiB
MLMG::prepareForSolve() 361 361 3793 KiB 12 MiB
Gravity::get_old_grav_vector() 43 43 220 KiB 10 MiB
Gravity::get_new_grav_vector() 40 40 224 KiB 10 MiB
Gravity::multilevel_solve_for_new_phi() 24 24 6625 KiB 7586 KiB
Gravity::fill_multipole_BCs() 84 84 10 KiB 2053 KiB
Gravity::update_max_rhs() 48 48 2455 B 2048 KiB
Gravity::solve_for_phi() 40 40 692 KiB 2048 KiB
Gravity::actual_multilevel_solve() 8 8 18 KiB 2048 KiB
BndryData::define() 576 576 348 KiB 1095 KiB
MLCellLinOp::defineAuxData() 936 936 223 KiB 671 KiB
Castro::estTimeStep() 10 10 3309 B 540 KiB
VisMF::Write(FabArray) 112 112 1592 B 360 KiB
Castro::normalize_species() 30 30 9 KiB 360 KiB
amrex::average_down 469 469 1293 B 257 KiB
MLMG::addInterpCorrection() 468 468 1239 B 257 KiB
amrex::Dot() 592 592 4111 B 180 KiB
FabArray::norminf() 398 398 2930 B 180 KiB
Castro::do_advance_ctu() 5 5 761 B 180 KiB
MultiFab::max() 6 6 66 B 180 KiB
FabArray::setVal() 66 66 18 KiB 23 KiB
MultiFab::contains_nan() 10 10 37 B 22 KiB
MLPoisson::Fsmooth() 60 60 3690 B 12 KiB
FabArray::ParallelCopy_nowait() 380 380 54 B 10 KiB
FillBoundary_nowait() 336 336 309 B 9648 B
MLCellLinOp::applyBC() 3892 3892 239 B 9344 B
amrex::Copy() 56 56 5925 B 8464 B
MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B
StateData::FillBoundary(geom) 960 960 52 B 2880 B
FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B
MLCGSolver::bicgstab 324 324 128 B 1472 B
MLCellLinOp::defineBC() 36 36 392 B 1248 B
------------------------------------------------------------------------------
Managed Memory Usage:
------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------
The_Managed_Arena::Initialize() 1 1 1312 B 8192 KiB
------------------------------------------------------------------
Pinned Memory Usage:
------------------------------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------------------------------
The_Pinned_Arena::Initialize() 1 1 126 KiB 8192 KiB
VisMF::Write(FabArray) 120 120 273 KiB 3584 KiB
VisMF::Read() 24 24 213 KiB 3000 KiB
FabArray::setVal() 66 66 18 KiB 23 KiB
MLPoisson::Fsmooth() 60 60 3690 B 12 KiB
FabArray::ParallelCopy_nowait() 380 380 54 B 10 KiB
FillBoundary_nowait() 336 336 308 B 9648 B
MLCellLinOp::applyBC() 1946 1946 238 B 9328 B
amrex::Copy() 56 56 5925 B 8464 B
MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B
Gravity::get_old_grav_vector() 3 3 2609 B 3072 B
StateData::FillBoundary(geom) 960 960 52 B 2880 B
Gravity::fill_multipole_BCs() 18 18 5 B 2832 B
FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B
MLMG::prepareForSolve() 7 7 552 B 1296 B
amrex::average_down 37 37 220 B 1296 B
MLMG::addInterpCorrection() 36 36 2 B 1024 B
MLCellLinOp::setLevelBC() 36 36 0 B 768 B
amrex::Dot() 592 592 26 B 400 B
FabArray::norminf() 398 398 18 B 272 B
Castro::estTimeStep() 10 10 0 B 32 B
MultiFab::max() 6 6 0 B 16 B
MultiFab::contains_nan() 10 10 0 B 16 B
Castro::do_advance_ctu() 5 5 0 B 16 B
Castro::normalize_species() 30 30 0 B 16 B
------------------------------------------------------------------------------
Cpu (amrex_mempool) Memory Usage:
------------------------------------------------------
Name Nalloc Nfree AvgMem MaxMem
------------------------------------------------------
amrex_mempool_init() 1 1 138 KiB 8192 KiB
------------------------------------------------------
Unused ParmParse Variables:
[TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2]
[TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2]
Total GPU global memory (MB): 12056
Free GPU global memory (MB): 2459
[The Arena] space allocated (MB): 9042
[The Arena] space used (MB): 0
[The Managed Arena] space allocated (MB): 8
[The Managed Arena] space used (MB): 0
[The Pinned Arena] space allocated (MB): 8
[The Pinned Arena] space used (MB): 0
AMReX (23.02-16-g0c444b898f8a) finalized