Initializing CUDA... CUDA initialized with 1 device. AMReX (23.08-4-g67b8a5b04b7f) initialized Starting run at 07:38:40 UTC on 2023-08-03. Successfully read inputs file ... Castro git describe: 23.08 AMReX git describe: 23.08-4-g67b8a5b04 Microphysics git describe: 23.08 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.05730705 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.032817626 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.04580898 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049576147 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.068422006 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.073313142 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.052944277 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.056604181 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.065212306 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.065251904 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.072364408 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.058253939 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.048382258 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.056368765 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032655572 seconds Ending run at 07:38:40 UTC on 2023-08-03. Run time = 0.8836254 Run time without initialization = 0.745785828 Average number of zones advanced per microsecond: 3.515 Average number of zones advanced per microsecond per rank: 3.515 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8837 ... 0.8837 ... 0.8837 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2414 0.2414 0.2414 27.32% VisMF::Write(FabArray) 11 0.229 0.229 0.229 25.91% MLCellLinOp::applyBC() 4433 0.07472 0.07472 0.07472 8.46% MLPoisson::Fsmooth() 3280 0.03129 0.03129 0.03129 3.54% FillBoundary_nowait() 4023 0.03086 0.03086 0.03086 3.49% StateData::FillBoundary(geom) 328 0.0235 0.0235 0.0235 2.66% amrex::Dot() 1114 0.01975 0.01975 0.01975 2.24% Castro::normalize_species() 62 0.01828 0.01828 0.01828 2.07% amrex::Copy() 1029 0.01451 0.01451 0.01451 1.64% Castro::computeTemp() 63 0.01405 0.01405 0.01405 1.59% FabArray::norminf() 743 0.01383 0.01383 0.01383 1.56% FabArray::ParallelCopy_nowait() 861 0.01308 0.01308 0.01308 1.48% StateDataPhysBCFunct::() 41 0.013 0.013 0.013 1.47% FabArray::setVal() 1144 0.01271 0.01271 0.01271 1.44% Castro::enforce_min_density() 62 0.01149 0.01149 0.01149 1.30% MLPoisson::Fapply() 1142 0.009963 0.009963 0.009963 1.13% MLCellLinOp::defineAuxData() 11 0.009246 0.009246 0.009246 1.05% FabArray::Saxpy() 813 0.007844 0.007844 0.007844 0.89% FabArray::Xpay() 821 0.007837 0.007837 0.007837 0.89% MLMG::addInterpCorrection() 410 0.006604 0.006604 0.006604 0.75% Gravity::fill_multipole_BCs() 11 0.006432 0.006432 0.006432 0.73% amrex::average_down 410 0.005839 0.005839 0.005839 0.66% Castro::estTimeStep() 21 0.005459 0.005459 0.005459 0.62% Castro::reset_internal_energy(MultiFab) 63 0.005 0.005 0.005 0.57% FabArray::LinComb() 557 0.004364 0.004364 0.004364 0.49% amrex::Add() 164 0.004294 0.004294 0.004294 0.49% Amr::checkPoint() 3 0.003933 0.003933 0.003933 0.45% Castro::enforce_speed_limit() 62 0.003548 0.003548 0.003548 0.40% BndryData::define() 11 0.003516 0.003516 0.003516 0.40% Castro::construct_new_gravity_source() 10 0.003093 0.003093 0.003093 0.35% Castro::construct_old_gravity_source() 10 0.00269 0.00269 0.00269 0.30% Amr::writePlotFile() 2 0.002366 0.002366 0.002366 0.27% MLCGSolver::bicgstab 82 0.002173 0.002173 0.002173 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001622 0.001622 0.001622 0.18% Castro::reset_internal_energy(Fab) 504 0.001547 0.001547 0.001547 0.18% Gravity::actual_solve_with_mlmg() 11 0.001477 0.001477 0.001477 0.17% MLCellLinOp::setLevelBC() 11 0.001361 0.001361 0.001361 0.15% FabArray::mult() 43 0.001295 0.001295 0.001295 0.15% FabArray::setDomainBndry() 41 0.00127 0.00127 0.00127 0.14% check_for_negative_density() 10 0.001256 0.001256 0.001256 0.14% Castro::initData() 1 0.001247 0.001247 0.001247 0.14% MultiFab::contains_nan() 20 0.001194 0.001194 0.001194 0.14% MLCellLinOp::smooth() 1640 0.001191 0.001191 0.001191 0.13% MLCellLinOp::prepareForSolve() 11 0.001086 0.001086 0.001086 0.12% MLCellLinOp::compGrad() 11 0.0009 0.0009 0.0009 0.10% MLMG::prepareForSolve() 11 0.0008665 0.0008665 0.0008665 0.10% FabArray::FillBoundary() 4023 0.0008624 0.0008624 0.0008624 0.10% FabArrayBase::getCPC() 1323 0.0007714 0.0007714 0.0007714 0.09% FabArrayBase::getFB() 4023 0.0007012 0.0007012 0.0007012 0.08% FabArrayBase::CPC::define() 454 0.000665 0.000665 0.000665 0.08% Gravity::get_new_grav_vector() 11 0.0005381 0.0005381 0.0005381 0.06% MLCellLinOp::apply() 1142 0.0004896 0.0004896 0.0004896 0.06% Gravity::get_old_grav_vector() 10 0.0004682 0.0004682 0.0004682 0.05% Amr::InitAmr() 1 0.0004579 0.0004579 0.0004579 0.05% MLLinOp::defineGrids() 11 0.0003549 0.0003549 0.0003549 0.04% Amr::coarseTimeStep() 10 0.0003487 0.0003487 0.0003487 0.04% MLMG::mgVcycle() 82 0.0003422 0.0003422 0.0003422 0.04% AmrLevel::FillPatch() 41 0.000336 0.000336 0.000336 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002974 0.0002974 0.0002974 0.03% main() 1 0.000283 0.000283 0.000283 0.03% MultiFab::max() 11 0.0002542 0.0002542 0.0002542 0.03% FabArray::ParallelCopy() 861 0.0002429 0.0002429 0.0002429 0.03% Castro::subcycle_advance_ctu() 10 0.0002327 0.0002327 0.0002327 0.03% MLCellLinOp::correctionResidual() 492 0.0002218 0.0002218 0.0002218 0.03% MLCellLinOp::defineBC() 11 0.0002164 0.0002164 0.0002164 0.02% FillPatchIterator::Initialize 41 0.0002018 0.0002018 0.0002018 0.02% Castro::create_source_corrector() 10 0.0001745 0.0001745 0.0001745 0.02% Amr::timeStep() 10 0.0001556 0.0001556 0.0001556 0.02% StateData::checkPoint() 12 0.0001378 0.0001378 0.0001378 0.02% Gravity::update_max_rhs() 11 0.0001353 0.0001353 0.0001353 0.02% MLMG:computeResOfCorrection() 410 0.0001311 0.0001311 0.0001311 0.01% Gravity::solve_for_phi() 10 0.0001138 0.0001138 0.0001138 0.01% Castro::advance() 10 9.298e-05 9.298e-05 9.298e-05 0.01% FabArrayBase::FB::FB() 56 8.928e-05 8.928e-05 8.928e-05 0.01% MLMG::actualBottomSolve() 82 8.821e-05 8.821e-05 8.821e-05 0.01% MLMG::mgVcycle_down::1 82 8.801e-05 8.801e-05 8.801e-05 0.01% MLMG::mgVcycle_down::2 82 8.75e-05 8.75e-05 8.75e-05 0.01% MLMG::mgVcycle_down::0 82 8.737e-05 8.737e-05 8.737e-05 0.01% Castro::Castro() 1 8.665e-05 8.665e-05 8.665e-05 0.01% MLMG::mgVcycle_down::4 82 8.619e-05 8.619e-05 8.619e-05 0.01% MLMG::mgVcycle_down::3 82 8.418e-05 8.418e-05 8.418e-05 0.01% AmrLevel::checkPoint() 3 7.616e-05 7.616e-05 7.616e-05 0.01% MLMG::solve() 11 7.52e-05 7.52e-05 7.52e-05 0.01% Castro::initialize_advance() 10 7.409e-05 7.409e-05 7.409e-05 0.01% Castro::clean_state() 62 6.616e-05 6.616e-05 6.616e-05 0.01% MLMG::mgVcycle_up::4 82 6.249e-05 6.249e-05 6.249e-05 0.01% Castro::finalize_advance() 10 5.813e-05 5.813e-05 5.813e-05 0.01% Castro::initialize_do_advance() 10 5.762e-05 5.762e-05 5.762e-05 0.01% MLMG::mgVcycle_up::0 82 5.492e-05 5.492e-05 5.492e-05 0.01% MLMG::oneIter() 82 5.194e-05 5.194e-05 5.194e-05 0.01% MLMG::mgVcycle_up::1 82 5.125e-05 5.125e-05 5.125e-05 0.01% MLMG::mgVcycle_up::3 82 5.112e-05 5.112e-05 5.112e-05 0.01% MLMG::mgVcycle_up::2 82 4.973e-05 4.973e-05 4.973e-05 0.01% FillPatchIterator::FillFromLevel0() 41 4.786e-05 4.786e-05 4.786e-05 0.01% MLCellLinOp::solutionResidual() 93 4.549e-05 4.549e-05 4.549e-05 0.01% Castro::do_advance_ctu() 10 4.476e-05 4.476e-05 4.476e-05 0.01% Castro::do_new_sources() 10 4.376e-05 4.376e-05 4.376e-05 0.00% Castro::finalize_do_advance() 10 3.908e-05 3.908e-05 3.908e-05 0.00% Castro::post_timestep() 10 3.899e-05 3.899e-05 3.899e-05 0.00% Castro::swap_state_time_levels() 10 3.899e-05 3.899e-05 3.899e-05 0.00% FillPatchSingleLevel 41 3.637e-05 3.637e-05 3.637e-05 0.00% Castro::construct_new_source() 50 3.589e-05 3.589e-05 3.589e-05 0.00% StateData::define() 4 3.494e-05 3.494e-05 3.494e-05 0.00% MLMG::mgVcycle_bottom 82 3.288e-05 3.288e-05 3.288e-05 0.00% Castro::enforce_consistent_e() 1 3.242e-05 3.242e-05 3.242e-05 0.00% MLMG::ResNormInf() 93 3.163e-05 3.163e-05 3.163e-05 0.00% MLMG::computeResidual() 82 3.096e-05 3.096e-05 3.096e-05 0.00% MLPoisson::define() 11 2.733e-05 2.733e-05 2.733e-05 0.00% Castro::construct_old_source() 50 2.556e-05 2.556e-05 2.556e-05 0.00% Amr::writeSmallPlotFile() 1 2.51e-05 2.51e-05 2.51e-05 0.00% makeSFC 55 2.386e-05 2.386e-05 2.386e-05 0.00% Castro::construct_new_gravity() 10 2.334e-05 2.334e-05 2.334e-05 0.00% Castro::initMFs() 1 2.33e-05 2.33e-05 2.33e-05 0.00% Castro::do_old_sources() 10 2.099e-05 2.099e-05 2.099e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.058e-05 2.058e-05 2.058e-05 0.00% Amr::FinalizeInit() 1 2.033e-05 2.033e-05 2.033e-05 0.00% Castro::buildMetrics() 1 1.733e-05 1.733e-05 1.733e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.662e-05 1.662e-05 1.662e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.634e-05 1.634e-05 1.634e-05 0.00% MLLinOp::define() 11 1.632e-05 1.632e-05 1.632e-05 0.00% DistributionMapping::Distribute() 56 1.422e-05 1.422e-05 1.422e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.375e-05 1.375e-05 1.375e-05 0.00% Amr::defBaseLevel() 1 1.187e-05 1.187e-05 1.187e-05 0.00% Castro::check_for_nan() 20 1.076e-05 1.076e-05 1.076e-05 0.00% Castro::apply_source_to_state() 20 1.025e-05 1.025e-05 1.025e-05 0.00% Castro::construct_old_gravity() 10 1.003e-05 1.003e-05 1.003e-05 0.00% MLPoisson::prepareForSolve() 11 9.206e-06 9.206e-06 9.206e-06 0.00% Gravity::actual_multilevel_solve() 1 8.94e-06 8.94e-06 8.94e-06 0.00% Gravity::swapTimeLevels() 10 8.738e-06 8.738e-06 8.738e-06 0.00% Amr::initSubcycle() 1 8.635e-06 8.635e-06 8.635e-06 0.00% MLMG::computeMLResidual() 11 7.458e-06 7.458e-06 7.458e-06 0.00% Castro::computeNewDt() 9 6.341e-06 6.341e-06 6.341e-06 0.00% MLMG::getGradSolution() 11 5.651e-06 5.651e-06 5.651e-06 0.00% Castro::expand_state() 10 5.514e-06 5.514e-06 5.514e-06 0.00% AmrLevel::checkPointPost() 3 5.13e-06 5.13e-06 5.13e-06 0.00% Amr::InitializeInit() 1 4.518e-06 4.518e-06 4.518e-06 0.00% Gravity::set_mass_offset() 11 4.096e-06 4.096e-06 4.096e-06 0.00% Castro::retry_advance_ctu() 10 3.94e-06 3.94e-06 3.94e-06 0.00% Castro::post_init() 1 3.893e-06 3.893e-06 3.893e-06 0.00% MLMG::MLRhsNormInf() 11 3.759e-06 3.759e-06 3.759e-06 0.00% MLMG::MLResNormInf() 11 3.474e-06 3.474e-06 3.474e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.989e-06 2.989e-06 2.989e-06 0.00% Castro::FluxRegCrseInit 10 2.724e-06 2.724e-06 2.724e-06 0.00% Amr::init() 1 2.422e-06 2.422e-06 2.422e-06 0.00% Castro::computeInitialDt() 2 2.31e-06 2.31e-06 2.31e-06 0.00% Castro::FluxRegFineAdd() 10 1.96e-06 1.96e-06 1.96e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.85e-06 1.85e-06 1.85e-06 0.00% AmrLevel::checkPointPre() 3 1.739e-06 1.739e-06 1.739e-06 0.00% Amr::initialInit() 1 1.096e-06 1.096e-06 1.096e-06 0.00% Castro::post_regrid() 1 1.045e-06 1.045e-06 1.045e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8836 0.8836 0.8836 100.00% Amr::coarseTimeStep() 10 0.7129 0.7129 0.7129 80.68% Amr::timeStep() 10 0.5963 0.5963 0.5963 67.49% Castro::advance() 10 0.585 0.585 0.585 66.20% Castro::subcycle_advance_ctu() 10 0.5714 0.5714 0.5714 64.66% Castro::do_advance_ctu() 10 0.5711 0.5711 0.5711 64.63% Gravity::solve_phi_with_mlmg() 11 0.2758 0.2758 0.2758 31.22% Gravity::actual_solve_with_mlmg() 11 0.269 0.269 0.269 30.44% Castro::construct_new_gravity() 10 0.2518 0.2518 0.2518 28.50% Castro::construct_ctu_hydro_source() 10 0.2503 0.2503 0.2503 28.32% MLMG::solve() 11 0.2491 0.2491 0.2491 28.19% Gravity::solve_for_phi() 10 0.2366 0.2366 0.2366 26.77% MLMG::oneIter() 82 0.2352 0.2352 0.2352 26.61% MLMG::mgVcycle() 82 0.2316 0.2316 0.2316 26.21% VisMF::Write(FabArray) 11 0.229 0.229 0.229 25.91% Amr::checkPoint() 3 0.1704 0.1704 0.1704 19.28% AmrLevel::checkPoint() 3 0.1665 0.1665 0.1665 18.84% StateData::checkPoint() 12 0.1664 0.1664 0.1664 18.83% Amr::init() 1 0.1372 0.1372 0.1372 15.53% MLCellLinOp::smooth() 1640 0.1144 0.1144 0.1144 12.95% MLCellLinOp::applyBC() 4433 0.1072 0.1072 0.1072 12.14% MLMG::mgVcycle_bottom 82 0.07098 0.07098 0.07098 8.03% MLMG::actualBottomSolve() 82 0.07094 0.07094 0.07094 8.03% MLCGSolver::bicgstab 82 0.07027 0.07027 0.07027 7.95% Amr::writePlotFile() 2 0.06559 0.06559 0.06559 7.42% Castro::clean_state() 62 0.05317 0.05317 0.05317 6.02% Amr::initialInit() 1 0.04701 0.04701 0.04701 5.32% AmrLevel::FillPatch() 41 0.0462 0.0462 0.0462 5.23% Amr::FinalizeInit() 1 0.04291 0.04291 0.04291 4.86% FillPatchIterator::Initialize 41 0.04207 0.04207 0.04207 4.76% Castro::post_init() 1 0.04162 0.04162 0.04162 4.71% FillPatchIterator::FillFromLevel0() 41 0.0406 0.0406 0.0406 4.59% FillPatchSingleLevel 41 0.04055 0.04055 0.04055 4.59% Gravity::multilevel_solve_for_new_phi() 1 0.03971 0.03971 0.03971 4.49% Gravity::actual_multilevel_solve() 1 0.0397 0.0397 0.0397 4.49% StateDataPhysBCFunct::() 41 0.0365 0.0365 0.0365 4.13% MLCellLinOp::apply() 1142 0.0353 0.0353 0.0353 3.99% MLMG::mgVcycle_down::0 82 0.03327 0.03327 0.03327 3.77% FabArray::FillBoundary() 4023 0.03252 0.03252 0.03252 3.68% FillBoundary_nowait() 4023 0.03165 0.03165 0.03165 3.58% MLPoisson::Fsmooth() 3280 0.03129 0.03129 0.03129 3.54% MLMG::mgVcycle_up::0 82 0.02534 0.02534 0.02534 2.87% StateData::FillBoundary(geom) 328 0.0235 0.0235 0.0235 2.66% MLCellLinOp::correctionResidual() 492 0.02166 0.02166 0.02166 2.45% Castro::initialize_do_advance() 10 0.02119 0.02119 0.02119 2.40% Castro::computeTemp() 63 0.0206 0.0206 0.0206 2.33% amrex::Dot() 1114 0.01975 0.01975 0.01975 2.24% MLMG:computeResOfCorrection() 410 0.01912 0.01912 0.01912 2.16% Castro::do_old_sources() 10 0.01888 0.01888 0.01888 2.14% Castro::normalize_species() 62 0.01828 0.01828 0.01828 2.07% Gravity::get_new_grav_vector() 11 0.0169 0.0169 0.0169 1.91% MLPoisson::define() 11 0.01566 0.01566 0.01566 1.77% MLMG::mgVcycle_down::1 82 0.0154 0.0154 0.0154 1.74% amrex::Copy() 1029 0.01451 0.01451 0.01451 1.64% MLMG::mgVcycle_down::2 82 0.01436 0.01436 0.01436 1.62% Castro::construct_old_gravity() 10 0.01435 0.01435 0.01435 1.62% Gravity::get_old_grav_vector() 10 0.01434 0.01434 0.01434 1.62% FabArray::ParallelCopy() 861 0.01414 0.01414 0.01414 1.60% MLMG::mgVcycle_down::3 82 0.01402 0.01402 0.01402 1.59% FabArray::ParallelCopy_nowait() 861 0.0139 0.0139 0.0139 1.57% MLMG::mgVcycle_down::4 82 0.01385 0.01385 0.01385 1.57% FabArray::norminf() 743 0.01383 0.01383 0.01383 1.56% Castro::initialize_advance() 10 0.0129 0.0129 0.0129 1.46% FabArray::setVal() 1144 0.01271 0.01271 0.01271 1.44% MLCGSolver::ParallelAllReduce 1514 0.01187 0.01187 0.01187 1.34% Castro::expand_state() 10 0.01185 0.01185 0.01185 1.34% MLMG::addInterpCorrection() 410 0.01164 0.01164 0.01164 1.32% Castro::enforce_min_density() 62 0.01149 0.01149 0.01149 1.30% Castro::do_new_sources() 10 0.01148 0.01148 0.01148 1.30% Castro::post_timestep() 10 0.01123 0.01123 0.01123 1.27% MLMG::mgVcycle_up::1 82 0.0112 0.0112 0.0112 1.27% MLMG::mgVcycle_up::4 82 0.01118 0.01118 0.01118 1.27% amrex::average_down 410 0.01093 0.01093 0.01093 1.24% MLMG::mgVcycle_up::2 82 0.01092 0.01092 0.01092 1.24% MLMG::mgVcycle_up::3 82 0.01072 0.01072 0.01072 1.21% MLCellLinOp::defineAuxData() 11 0.01054 0.01054 0.01054 1.19% MLPoisson::Fapply() 1142 0.009963 0.009963 0.009963 1.13% FabArray::Saxpy() 813 0.007844 0.007844 0.007844 0.89% FabArray::Xpay() 821 0.007837 0.007837 0.007837 0.89% MLCellLinOp::solutionResidual() 93 0.006965 0.006965 0.006965 0.79% Gravity::fill_multipole_BCs() 11 0.006682 0.006682 0.006682 0.76% Castro::reset_internal_energy(MultiFab) 63 0.006546 0.006546 0.006546 0.74% MLMG::computeResidual() 82 0.006006 0.006006 0.006006 0.68% Castro::estTimeStep() 21 0.005459 0.005459 0.005459 0.62% MLCellLinOp::defineBC() 11 0.004671 0.004671 0.004671 0.53% MLMG::prepareForSolve() 11 0.004521 0.004521 0.004521 0.51% BndryData::define() 11 0.004455 0.004455 0.004455 0.50% FabArray::LinComb() 557 0.004364 0.004364 0.004364 0.49% amrex::Add() 164 0.004294 0.004294 0.004294 0.49% Amr::InitializeInit() 1 0.004094 0.004094 0.004094 0.46% Amr::defBaseLevel() 1 0.00409 0.00409 0.00409 0.46% Castro::initData() 1 0.003592 0.003592 0.003592 0.41% Castro::enforce_speed_limit() 62 0.003548 0.003548 0.003548 0.40% Castro::construct_new_source() 50 0.003129 0.003129 0.003129 0.35% Castro::construct_new_gravity_source() 10 0.003093 0.003093 0.003093 0.35% Castro::construct_old_source() 50 0.002715 0.002715 0.002715 0.31% Castro::construct_old_gravity_source() 10 0.00269 0.00269 0.00269 0.30% Castro::computeNewDt() 9 0.002481 0.002481 0.002481 0.28% Castro::finalize_do_advance() 10 0.002186 0.002186 0.002186 0.25% MLMG::ResNormInf() 93 0.002082 0.002082 0.002082 0.24% Castro::apply_source_to_state() 20 0.001807 0.001807 0.001807 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001622 0.001622 0.001622 0.18% Castro::reset_internal_energy(Fab) 504 0.001547 0.001547 0.001547 0.18% FabArrayBase::getCPC() 1323 0.001436 0.001436 0.001436 0.16% MLMG::getGradSolution() 11 0.001374 0.001374 0.001374 0.16% MLCellLinOp::compGrad() 11 0.001368 0.001368 0.001368 0.15% MLCellLinOp::setLevelBC() 11 0.001361 0.001361 0.001361 0.15% FabArray::mult() 43 0.001295 0.001295 0.001295 0.15% FabArray::setDomainBndry() 41 0.00127 0.00127 0.00127 0.14% check_for_negative_density() 10 0.001256 0.001256 0.001256 0.14% Castro::check_for_nan() 20 0.001205 0.001205 0.001205 0.14% MultiFab::contains_nan() 20 0.001194 0.001194 0.001194 0.14% Castro::post_regrid() 1 0.001111 0.001111 0.001111 0.13% MLPoisson::prepareForSolve() 11 0.001095 0.001095 0.001095 0.12% MLCellLinOp::prepareForSolve() 11 0.001086 0.001086 0.001086 0.12% MLMG::computeMLResidual() 11 0.0009973 0.0009973 0.0009973 0.11% Castro::computeInitialDt() 2 0.0008399 0.0008399 0.0008399 0.10% Gravity::update_max_rhs() 11 0.0008188 0.0008188 0.0008188 0.09% FabArrayBase::getFB() 4023 0.0007904 0.0007904 0.0007904 0.09% FabArrayBase::CPC::define() 454 0.000665 0.000665 0.000665 0.08% Castro::finalize_advance() 10 0.0005766 0.0005766 0.0005766 0.07% Amr::InitAmr() 1 0.0004665 0.0004665 0.0004665 0.05% Castro::Castro() 1 0.0004305 0.0004305 0.0004305 0.05% Gravity::swapTimeLevels() 10 0.0004267 0.0004267 0.0004267 0.05% MLLinOp::define() 11 0.0004237 0.0004237 0.0004237 0.05% MLLinOp::defineGrids() 11 0.0004074 0.0004074 0.0004074 0.05% MLMG::MLResNormInf() 11 0.0002836 0.0002836 0.0002836 0.03% MultiFab::max() 11 0.0002542 0.0002542 0.0002542 0.03% MLMG::MLRhsNormInf() 11 0.0002161 0.0002161 0.0002161 0.02% Castro::create_source_corrector() 10 0.0001745 0.0001745 0.0001745 0.02% Castro::buildMetrics() 1 0.0001516 0.0001516 0.0001516 0.02% FabArrayBase::FB::FB() 56 8.928e-05 8.928e-05 8.928e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.156e-05 5.156e-05 5.156e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.07e-05 5.07e-05 5.07e-05 0.01% Castro::swap_state_time_levels() 10 3.899e-05 3.899e-05 3.899e-05 0.00% makeSFC 55 3.695e-05 3.695e-05 3.695e-05 0.00% StateData::define() 4 3.494e-05 3.494e-05 3.494e-05 0.00% Castro::enforce_consistent_e() 1 3.242e-05 3.242e-05 3.242e-05 0.00% Amr::writeSmallPlotFile() 1 2.51e-05 2.51e-05 2.51e-05 0.00% Castro::initMFs() 1 2.33e-05 2.33e-05 2.33e-05 0.00% DistributionMapping::Distribute() 56 1.422e-05 1.422e-05 1.422e-05 0.00% Amr::initSubcycle() 1 8.635e-06 8.635e-06 8.635e-06 0.00% AmrLevel::checkPointPost() 3 5.13e-06 5.13e-06 5.13e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.125e-06 4.125e-06 4.125e-06 0.00% Gravity::set_mass_offset() 11 4.096e-06 4.096e-06 4.096e-06 0.00% Castro::retry_advance_ctu() 10 3.94e-06 3.94e-06 3.94e-06 0.00% Castro::FluxRegCrseInit 10 2.724e-06 2.724e-06 2.724e-06 0.00% Castro::FluxRegFineAdd() 10 1.96e-06 1.96e-06 1.96e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.85e-06 1.85e-06 1.85e-06 0.00% AmrLevel::checkPointPre() 3 1.739e-06 1.739e-06 1.739e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 90 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 131 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1037 KiB 39 MiB Castro::initialize_do_advance() 80 80 25 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 2085 KiB 28 MiB Castro::initialize_advance() 80 80 15 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7612 KiB 14 MiB MLMG::prepareForSolve() 660 660 3468 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 198 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 166 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7526 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 12 KiB 2053 KiB Gravity::update_max_rhs() 88 88 1773 B 2048 KiB Gravity::solve_for_phi() 80 80 547 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 91 KiB 2048 KiB BndryData::define() 1056 1056 316 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 201 KiB 671 KiB Castro::estTimeStep() 21 21 3014 B 480 KiB VisMF::Write(FabArray) 656 656 3330 B 320 KiB Castro::normalize_species() 62 62 6740 B 320 KiB amrex::average_down 1067 1067 1237 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1136 B 257 KiB amrex::Dot() 1360 1360 3295 B 160 KiB FabArray::norminf() 907 907 2315 B 160 KiB check_for_negative_density() 10 10 224 B 160 KiB Castro::initData() 1 1 26 B 160 KiB MultiFab::max() 11 11 44 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 26 B 20 KiB MLPoisson::Fsmooth() 132 132 3391 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 44 B 10 KiB FillBoundary_nowait() 760 760 306 B 9648 B MLCellLinOp::applyBC() 8866 8866 218 B 9344 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3854 B 6144 B StateData::FillBoundary(geom) 1992 1992 50 B 2976 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 738 738 115 B 1472 B MLCellLinOp::defineBC() 66 66 356 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 614 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 49 KiB 8192 KiB VisMF::Write(FabArray) 744 744 597 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3391 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 44 B 10 KiB FillBoundary_nowait() 760 760 306 B 9648 B MLCellLinOp::applyBC() 4433 4433 216 B 9328 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3854 B 6144 B Gravity::get_new_grav_vector() 3 3 2909 B 3072 B StateData::FillBoundary(geom) 1992 1992 50 B 2976 B Gravity::fill_multipole_BCs() 33 33 3 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 83 83 272 B 1296 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 288 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 24 B 400 B FabArray::norminf() 907 907 17 B 272 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2422 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.08-4-g67b8a5b04b7f) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.08-4-g67b8a5b04b7f) initialized Starting run at 07:38:41 UTC on 2023-08-03. Successfully read inputs file ... Castro git describe: 23.08 AMReX git describe: 23.08-4-g67b8a5b04 Microphysics git describe: 23.08 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.484528018 Restart time = 0.045911511 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.050499776 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048643854 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.063711136 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.078030873 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.071863094 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032593524 seconds Ending run at 07:38:42 UTC on 2023-08-03. Run time = 0.392232284 Run time without initialization = 0.345757781 Average number of zones advanced per microsecond: 3.791 Average number of zones advanced per microsecond per rank: 3.791 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3923 ... 0.3923 ... 0.3923 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1253 0.1253 0.1253 31.95% VisMF::Read() 3 0.03977 0.03977 0.03977 10.14% MLCellLinOp::applyBC() 1946 0.03238 0.03238 0.03238 8.26% VisMF::Write(FabArray) 1 0.031 0.031 0.031 7.90% MLPoisson::Fsmooth() 1440 0.01354 0.01354 0.01354 3.45% StateDataPhysBCFunct::() 20 0.01276 0.01276 0.01276 3.25% FillBoundary_nowait() 1766 0.01274 0.01274 0.01274 3.25% StateData::FillBoundary(geom) 160 0.01055 0.01055 0.01055 2.69% Castro::normalize_species() 30 0.009076 0.009076 0.009076 2.31% amrex::Dot() 484 0.0085 0.0085 0.0085 2.17% amrex::Copy() 463 0.006827 0.006827 0.006827 1.74% Castro::computeTemp() 30 0.006261 0.006261 0.006261 1.60% FabArray::setVal() 537 0.006076 0.006076 0.006076 1.55% FabArray::norminf() 326 0.005997 0.005997 0.005997 1.53% FabArray::ParallelCopy_nowait() 380 0.005955 0.005955 0.005955 1.52% Castro::enforce_min_density() 30 0.005445 0.005445 0.005445 1.39% MLCellLinOp::defineAuxData() 6 0.00501 0.00501 0.00501 1.28% MLPoisson::Fapply() 500 0.004291 0.004291 0.004291 1.09% FabArray::Saxpy() 355 0.003502 0.003502 0.003502 0.89% Castro::estTimeStep() 10 0.003452 0.003452 0.003452 0.88% FabArray::Xpay() 361 0.003411 0.003411 0.003411 0.87% Gravity::fill_multipole_BCs() 6 0.003357 0.003357 0.003357 0.86% MLMG::addInterpCorrection() 180 0.002871 0.002871 0.002871 0.73% amrex::average_down 180 0.002531 0.002531 0.002531 0.65% Amr::restart() 1 0.00249 0.00249 0.00249 0.63% Castro::reset_internal_energy(MultiFab) 30 0.002054 0.002054 0.002054 0.52% BndryData::define() 6 0.00195 0.00195 0.00195 0.50% FabArray::LinComb() 242 0.001848 0.001848 0.001848 0.47% amrex::Add() 72 0.001832 0.001832 0.001832 0.47% Castro::construct_new_gravity_source() 5 0.001604 0.001604 0.001604 0.41% Castro::construct_old_gravity_source() 5 0.00146 0.00146 0.00146 0.37% Castro::reset_internal_energy(Fab) 240 0.00143 0.00143 0.00143 0.36% Amr::writePlotFile() 1 0.001421 0.001421 0.001421 0.36% Castro::subcycle_advance_ctu() 5 0.001153 0.001153 0.001153 0.29% MLCGSolver::bicgstab 36 0.0009203 0.0009203 0.0009203 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008757 0.0008757 0.0008757 0.22% Gravity::actual_solve_with_mlmg() 6 0.000777 0.000777 0.000777 0.20% MLCellLinOp::setLevelBC() 6 0.0007282 0.0007282 0.0007282 0.19% check_for_negative_density() 5 0.0006998 0.0006998 0.0006998 0.18% FabArray::mult() 22 0.0006424 0.0006424 0.0006424 0.16% FabArray::setDomainBndry() 20 0.0006308 0.0006308 0.0006308 0.16% MLCellLinOp::prepareForSolve() 6 0.0005887 0.0005887 0.0005887 0.15% MultiFab::contains_nan() 10 0.0005867 0.0005867 0.0005867 0.15% MLCellLinOp::smooth() 720 0.0004887 0.0004887 0.0004887 0.12% MLCellLinOp::compGrad() 6 0.0004834 0.0004834 0.0004834 0.12% MLMG::prepareForSolve() 6 0.000459 0.000459 0.000459 0.12% Castro::enforce_speed_limit() 30 0.0004456 0.0004456 0.0004456 0.11% FabArrayBase::CPC::define() 244 0.0004108 0.0004108 0.0004108 0.10% Amr::InitAmr() 1 0.0003839 0.0003839 0.0003839 0.10% FabArray::FillBoundary() 1766 0.0003682 0.0003682 0.0003682 0.09% FabArrayBase::getCPC() 632 0.0003573 0.0003573 0.0003573 0.09% main() 1 0.0002714 0.0002714 0.0002714 0.07% FabArrayBase::getFB() 1766 0.0002693 0.0002693 0.0002693 0.07% Gravity::get_old_grav_vector() 5 0.0002665 0.0002665 0.0002665 0.07% Gravity::get_new_grav_vector() 5 0.0002356 0.0002356 0.0002356 0.06% MLCellLinOp::apply() 500 0.0001979 0.0001979 0.0001979 0.05% MLLinOp::defineGrids() 6 0.0001796 0.0001796 0.0001796 0.05% Amr::coarseTimeStep() 5 0.0001787 0.0001787 0.0001787 0.05% AmrLevel::FillPatch() 20 0.0001663 0.0001663 0.0001663 0.04% MLMG::mgVcycle() 36 0.000148 0.000148 0.000148 0.04% MultiFab::max() 6 0.0001418 0.0001418 0.0001418 0.04% MLCGSolver::ParallelAllReduce 659 0.0001267 0.0001267 0.0001267 0.03% MLCellLinOp::defineBC() 6 0.000113 0.000113 0.000113 0.03% FabArray::ParallelCopy() 380 0.0001037 0.0001037 0.0001037 0.03% FillPatchIterator::Initialize 20 0.0001009 0.0001009 0.0001009 0.03% Amr::timeStep() 5 9.965e-05 9.965e-05 9.965e-05 0.03% Castro::do_new_sources() 5 9.573e-05 9.573e-05 9.573e-05 0.02% MLCellLinOp::correctionResidual() 216 9.377e-05 9.377e-05 9.377e-05 0.02% Castro::create_source_corrector() 5 8.888e-05 8.888e-05 8.888e-05 0.02% Castro::construct_new_source() 25 7.749e-05 7.749e-05 7.749e-05 0.02% AmrLevel::restart() 1 7.549e-05 7.549e-05 7.549e-05 0.02% Castro::finalize_do_advance() 5 6.922e-05 6.922e-05 6.922e-05 0.02% Gravity::update_max_rhs() 6 6.707e-05 6.707e-05 6.707e-05 0.02% StateData::restartDoit() 4 6.432e-05 6.432e-05 6.432e-05 0.02% Castro::post_timestep() 5 5.985e-05 5.985e-05 5.985e-05 0.02% FabArrayBase::FB::FB() 26 5.959e-05 5.959e-05 5.959e-05 0.02% Castro::advance() 5 5.848e-05 5.848e-05 5.848e-05 0.01% MLMG:computeResOfCorrection() 180 5.637e-05 5.637e-05 5.637e-05 0.01% Gravity::solve_for_phi() 5 5.104e-05 5.104e-05 5.104e-05 0.01% Castro::finalize_advance() 5 4.262e-05 4.262e-05 4.262e-05 0.01% MLMG::actualBottomSolve() 36 3.932e-05 3.932e-05 3.932e-05 0.01% MLMG::mgVcycle_down::0 36 3.898e-05 3.898e-05 3.898e-05 0.01% MLMG::mgVcycle_down::1 36 3.824e-05 3.824e-05 3.824e-05 0.01% MLMG::mgVcycle_down::4 36 3.647e-05 3.647e-05 3.647e-05 0.01% MLMG::mgVcycle_down::2 36 3.617e-05 3.617e-05 3.617e-05 0.01% MLMG::mgVcycle_down::3 36 3.548e-05 3.548e-05 3.548e-05 0.01% MLMG::solve() 6 3.411e-05 3.411e-05 3.411e-05 0.01% Castro::initialize_advance() 5 3.382e-05 3.382e-05 3.382e-05 0.01% Castro::clean_state() 30 3.196e-05 3.196e-05 3.196e-05 0.01% Castro::initialize_do_advance() 5 3.012e-05 3.012e-05 3.012e-05 0.01% MLMG::mgVcycle_up::4 36 2.858e-05 2.858e-05 2.858e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.831e-05 2.831e-05 2.831e-05 0.01% Castro::buildMetrics() 1 2.782e-05 2.782e-05 2.782e-05 0.01% Amr::writeSmallPlotFile() 1 2.596e-05 2.596e-05 2.596e-05 0.01% Castro::do_advance_ctu() 5 2.472e-05 2.472e-05 2.472e-05 0.01% Castro::initMFs() 1 2.468e-05 2.468e-05 2.468e-05 0.01% Castro::swap_state_time_levels() 5 2.412e-05 2.412e-05 2.412e-05 0.01% MLMG::mgVcycle_up::3 36 2.361e-05 2.361e-05 2.361e-05 0.01% MLMG::mgVcycle_up::0 36 2.353e-05 2.353e-05 2.353e-05 0.01% MLMG::mgVcycle_up::2 36 2.305e-05 2.305e-05 2.305e-05 0.01% MLMG::oneIter() 36 2.246e-05 2.246e-05 2.246e-05 0.01% MLMG::mgVcycle_up::1 36 2.241e-05 2.241e-05 2.241e-05 0.01% MLPoisson::define() 6 2.175e-05 2.175e-05 2.175e-05 0.01% Castro::post_restart() 1 2.098e-05 2.098e-05 2.098e-05 0.01% MLCellLinOp::solutionResidual() 42 2.035e-05 2.035e-05 2.035e-05 0.01% MLMG::ResNormInf() 42 1.724e-05 1.724e-05 1.724e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.58e-05 1.58e-05 1.58e-05 0.00% MLMG::computeResidual() 36 1.467e-05 1.467e-05 1.467e-05 0.00% MLMG::mgVcycle_bottom 36 1.454e-05 1.454e-05 1.454e-05 0.00% FillPatchSingleLevel 20 1.427e-05 1.427e-05 1.427e-05 0.00% makeSFC 30 1.326e-05 1.326e-05 1.326e-05 0.00% Castro::construct_new_gravity() 5 1.289e-05 1.289e-05 1.289e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.151e-05 1.151e-05 1.151e-05 0.00% MLLinOp::define() 6 1.117e-05 1.117e-05 1.117e-05 0.00% Castro::construct_old_source() 25 1.111e-05 1.111e-05 1.111e-05 0.00% Castro::do_old_sources() 5 1.064e-05 1.064e-05 1.064e-05 0.00% Amr::initSubcycle() 1 8.484e-06 8.484e-06 8.484e-06 0.00% DistributionMapping::Distribute() 31 8.338e-06 8.338e-06 8.338e-06 0.00% Gravity::actual_multilevel_solve() 1 7.758e-06 7.758e-06 7.758e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.261e-06 7.261e-06 7.261e-06 0.00% Castro::check_for_nan() 10 5.632e-06 5.632e-06 5.632e-06 0.00% Castro::apply_source_to_state() 10 5.591e-06 5.591e-06 5.591e-06 0.00% Castro::construct_old_gravity() 5 5.543e-06 5.543e-06 5.543e-06 0.00% Castro::FluxRegFineAdd() 5 4.755e-06 4.755e-06 4.755e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.372e-06 4.372e-06 4.372e-06 0.00% MLPoisson::prepareForSolve() 6 4.274e-06 4.274e-06 4.274e-06 0.00% MLMG::computeMLResidual() 6 3.782e-06 3.782e-06 3.782e-06 0.00% Gravity::swapTimeLevels() 5 3.672e-06 3.672e-06 3.672e-06 0.00% Castro::retry_advance_ctu() 5 3.185e-06 3.185e-06 3.185e-06 0.00% Castro::computeNewDt() 5 3.03e-06 3.03e-06 3.03e-06 0.00% MLMG::getGradSolution() 6 2.893e-06 2.893e-06 2.893e-06 0.00% Castro::expand_state() 5 2.641e-06 2.641e-06 2.641e-06 0.00% Castro::FluxRegCrseInit 5 2.454e-06 2.454e-06 2.454e-06 0.00% Gravity::set_mass_offset() 6 2.33e-06 2.33e-06 2.33e-06 0.00% MLMG::MLResNormInf() 6 2.322e-06 2.322e-06 2.322e-06 0.00% MLMG::MLRhsNormInf() 6 2.292e-06 2.292e-06 2.292e-06 0.00% Amr::init() 1 9.93e-07 9.93e-07 9.93e-07 0.00% MLLinOp::makeSubCommunicator() 6 9.9e-07 9.9e-07 9.9e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3922 0.3922 0.3922 100.00% Amr::coarseTimeStep() 5 0.3129 0.3129 0.3129 79.77% Amr::timeStep() 5 0.3102 0.3102 0.3102 79.08% Castro::advance() 5 0.306 0.306 0.306 78.02% Castro::subcycle_advance_ctu() 5 0.2998 0.2998 0.2998 76.43% Castro::do_advance_ctu() 5 0.2986 0.2986 0.2986 76.13% Castro::construct_ctu_hydro_source() 5 0.1306 0.1306 0.1306 33.30% Castro::construct_new_gravity() 5 0.1293 0.1293 0.1293 32.97% Gravity::solve_phi_with_mlmg() 6 0.1219 0.1219 0.1219 31.06% Gravity::solve_for_phi() 5 0.1189 0.1189 0.1189 30.30% Gravity::actual_solve_with_mlmg() 6 0.1183 0.1183 0.1183 30.15% MLMG::solve() 6 0.1074 0.1074 0.1074 27.39% MLMG::oneIter() 36 0.1008 0.1008 0.1008 25.69% MLMG::mgVcycle() 36 0.09923 0.09923 0.09923 25.30% MLCellLinOp::smooth() 720 0.04884 0.04884 0.04884 12.45% Amr::init() 1 0.04595 0.04595 0.04595 11.72% Amr::restart() 1 0.04595 0.04595 0.04595 11.71% MLCellLinOp::applyBC() 1946 0.04582 0.04582 0.04582 11.68% AmrLevel::restart() 1 0.03996 0.03996 0.03996 10.19% StateData::restartDoit() 4 0.03988 0.03988 0.03988 10.17% VisMF::Read() 3 0.03977 0.03977 0.03977 10.14% Amr::writePlotFile() 1 0.03268 0.03268 0.03268 8.33% VisMF::Write(FabArray) 1 0.031 0.031 0.031 7.90% MLMG::mgVcycle_bottom 36 0.03046 0.03046 0.03046 7.77% MLMG::actualBottomSolve() 36 0.03045 0.03045 0.03045 7.76% MLCGSolver::bicgstab 36 0.03016 0.03016 0.03016 7.69% AmrLevel::FillPatch() 20 0.0282 0.0282 0.0282 7.19% FillPatchIterator::Initialize 20 0.02615 0.02615 0.02615 6.67% FillPatchIterator::FillFromLevel0() 20 0.02542 0.02542 0.02542 6.48% FillPatchSingleLevel 20 0.02539 0.02539 0.02539 6.47% Castro::clean_state() 30 0.02474 0.02474 0.02474 6.31% StateDataPhysBCFunct::() 20 0.0233 0.0233 0.0233 5.94% MLCellLinOp::apply() 500 0.01524 0.01524 0.01524 3.88% MLMG::mgVcycle_down::0 36 0.01402 0.01402 0.01402 3.58% MLPoisson::Fsmooth() 1440 0.01354 0.01354 0.01354 3.45% FabArray::FillBoundary() 1766 0.01344 0.01344 0.01344 3.43% FillBoundary_nowait() 1766 0.01307 0.01307 0.01307 3.33% MLMG::mgVcycle_up::0 36 0.0108 0.0108 0.0108 2.75% Castro::initialize_do_advance() 5 0.01059 0.01059 0.01059 2.70% StateData::FillBoundary(geom) 160 0.01055 0.01055 0.01055 2.69% Castro::do_old_sources() 5 0.0104 0.0104 0.0104 2.65% Gravity::get_new_grav_vector() 5 0.01036 0.01036 0.01036 2.64% Castro::computeTemp() 30 0.009745 0.009745 0.009745 2.48% Castro::construct_old_gravity() 5 0.009545 0.009545 0.009545 2.43% Gravity::get_old_grav_vector() 5 0.009539 0.009539 0.009539 2.43% MLCellLinOp::correctionResidual() 216 0.00929 0.00929 0.00929 2.37% Castro::normalize_species() 30 0.009076 0.009076 0.009076 2.31% MLPoisson::define() 6 0.008572 0.008572 0.008572 2.19% amrex::Dot() 484 0.0085 0.0085 0.0085 2.17% MLMG:computeResOfCorrection() 180 0.008185 0.008185 0.008185 2.09% amrex::Copy() 463 0.006827 0.006827 0.006827 1.74% Castro::do_new_sources() 5 0.006658 0.006658 0.006658 1.70% MLMG::mgVcycle_down::1 36 0.006636 0.006636 0.006636 1.69% FabArray::ParallelCopy() 380 0.006445 0.006445 0.006445 1.64% FabArray::ParallelCopy_nowait() 380 0.006342 0.006342 0.006342 1.62% MLMG::mgVcycle_down::2 36 0.006201 0.006201 0.006201 1.58% FabArray::setVal() 537 0.006076 0.006076 0.006076 1.55% MLMG::mgVcycle_down::3 36 0.006047 0.006047 0.006047 1.54% FabArray::norminf() 326 0.005997 0.005997 0.005997 1.53% MLMG::mgVcycle_down::4 36 0.005963 0.005963 0.005963 1.52% Castro::initialize_advance() 5 0.005877 0.005877 0.005877 1.50% Castro::expand_state() 5 0.005822 0.005822 0.005822 1.48% MLCellLinOp::defineAuxData() 6 0.005721 0.005721 0.005721 1.46% Castro::enforce_min_density() 30 0.005445 0.005445 0.005445 1.39% MLCGSolver::ParallelAllReduce 659 0.005127 0.005127 0.005127 1.31% MLMG::addInterpCorrection() 180 0.00504 0.00504 0.00504 1.28% MLMG::mgVcycle_up::4 36 0.004815 0.004815 0.004815 1.23% MLMG::mgVcycle_up::1 36 0.004808 0.004808 0.004808 1.23% amrex::average_down 180 0.004732 0.004732 0.004732 1.21% MLMG::mgVcycle_up::2 36 0.004706 0.004706 0.004706 1.20% MLMG::mgVcycle_up::3 36 0.004622 0.004622 0.004622 1.18% MLPoisson::Fapply() 500 0.004291 0.004291 0.004291 1.09% Castro::post_timestep() 5 0.004051 0.004051 0.004051 1.03% FabArray::Saxpy() 355 0.003502 0.003502 0.003502 0.89% Castro::reset_internal_energy(MultiFab) 30 0.003483 0.003483 0.003483 0.89% Gravity::fill_multipole_BCs() 6 0.003479 0.003479 0.003479 0.89% Castro::estTimeStep() 10 0.003452 0.003452 0.003452 0.88% FabArray::Xpay() 361 0.003411 0.003411 0.003411 0.87% Castro::post_restart() 1 0.003327 0.003327 0.003327 0.85% Gravity::multilevel_solve_for_new_phi() 1 0.003213 0.003213 0.003213 0.82% Gravity::actual_multilevel_solve() 1 0.003197 0.003197 0.003197 0.82% MLCellLinOp::solutionResidual() 42 0.003145 0.003145 0.003145 0.80% MLCellLinOp::defineBC() 6 0.002609 0.002609 0.002609 0.67% MLMG::computeResidual() 36 0.002605 0.002605 0.002605 0.66% Castro::computeNewDt() 5 0.00254 0.00254 0.00254 0.65% BndryData::define() 6 0.002496 0.002496 0.002496 0.64% MLMG::prepareForSolve() 6 0.00243 0.00243 0.00243 0.62% FabArray::LinComb() 242 0.001848 0.001848 0.001848 0.47% amrex::Add() 72 0.001832 0.001832 0.001832 0.47% Castro::construct_new_source() 25 0.001681 0.001681 0.001681 0.43% Castro::construct_new_gravity_source() 5 0.001604 0.001604 0.001604 0.41% Castro::construct_old_source() 25 0.001471 0.001471 0.001471 0.38% Castro::construct_old_gravity_source() 5 0.00146 0.00146 0.00146 0.37% Castro::reset_internal_energy(Fab) 240 0.00143 0.00143 0.00143 0.36% Castro::finalize_do_advance() 5 0.000984 0.000984 0.000984 0.25% MLMG::ResNormInf() 42 0.000924 0.000924 0.000924 0.24% Castro::apply_source_to_state() 10 0.0009175 0.0009175 0.0009175 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008757 0.0008757 0.0008757 0.22% FabArrayBase::getCPC() 632 0.0007681 0.0007681 0.0007681 0.20% MLMG::getGradSolution() 6 0.000747 0.000747 0.000747 0.19% MLCellLinOp::compGrad() 6 0.0007441 0.0007441 0.0007441 0.19% MLCellLinOp::setLevelBC() 6 0.0007282 0.0007282 0.0007282 0.19% check_for_negative_density() 5 0.0006998 0.0006998 0.0006998 0.18% FabArray::mult() 22 0.0006424 0.0006424 0.0006424 0.16% FabArray::setDomainBndry() 20 0.0006308 0.0006308 0.0006308 0.16% MLPoisson::prepareForSolve() 6 0.000593 0.000593 0.000593 0.15% Castro::check_for_nan() 10 0.0005923 0.0005923 0.0005923 0.15% MLCellLinOp::prepareForSolve() 6 0.0005887 0.0005887 0.0005887 0.15% MultiFab::contains_nan() 10 0.0005867 0.0005867 0.0005867 0.15% MLMG::computeMLResidual() 6 0.0005584 0.0005584 0.0005584 0.14% Gravity::update_max_rhs() 6 0.0004486 0.0004486 0.0004486 0.11% Castro::enforce_speed_limit() 30 0.0004456 0.0004456 0.0004456 0.11% FabArrayBase::CPC::define() 244 0.0004108 0.0004108 0.0004108 0.10% Amr::InitAmr() 1 0.0003924 0.0003924 0.0003924 0.10% FabArrayBase::getFB() 1766 0.0003289 0.0003289 0.0003289 0.08% Castro::finalize_advance() 5 0.0003181 0.0003181 0.0003181 0.08% Gravity::swapTimeLevels() 5 0.0002214 0.0002214 0.0002214 0.06% MLLinOp::define() 6 0.0002194 0.0002194 0.0002194 0.06% MLLinOp::defineGrids() 6 0.0002082 0.0002082 0.0002082 0.05% MLMG::MLResNormInf() 6 0.0001502 0.0001502 0.0001502 0.04% Castro::buildMetrics() 1 0.0001475 0.0001475 0.0001475 0.04% MultiFab::max() 6 0.0001418 0.0001418 0.0001418 0.04% MLMG::MLRhsNormInf() 6 0.0001196 0.0001196 0.0001196 0.03% Castro::create_source_corrector() 5 8.888e-05 8.888e-05 8.888e-05 0.02% FabArrayBase::FB::FB() 26 5.959e-05 5.959e-05 5.959e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.764e-05 2.764e-05 2.764e-05 0.01% Amr::writeSmallPlotFile() 1 2.596e-05 2.596e-05 2.596e-05 0.01% Castro::initMFs() 1 2.468e-05 2.468e-05 2.468e-05 0.01% Castro::swap_state_time_levels() 5 2.412e-05 2.412e-05 2.412e-05 0.01% makeSFC 30 2.038e-05 2.038e-05 2.038e-05 0.01% Amr::initSubcycle() 1 8.484e-06 8.484e-06 8.484e-06 0.00% DistributionMapping::Distribute() 31 8.338e-06 8.338e-06 8.338e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.584e-06 5.584e-06 5.584e-06 0.00% Castro::FluxRegFineAdd() 5 4.755e-06 4.755e-06 4.755e-06 0.00% Castro::retry_advance_ctu() 5 3.185e-06 3.185e-06 3.185e-06 0.00% Castro::FluxRegCrseInit 5 2.454e-06 2.454e-06 2.454e-06 0.00% Gravity::set_mass_offset() 6 2.33e-06 2.33e-06 2.33e-06 0.00% MLLinOp::makeSubCommunicator() 6 9.9e-07 9.9e-07 9.9e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 203 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 149 MiB 692 MiB Castro::initMFs() 48 48 60 MiB 68 MiB Castro::swap_state_time_levels() 32 32 49 MiB 55 MiB StateData::restartDoit() 32 32 53 MiB 55 MiB FillPatchIterator::Initialize 160 160 1323 KiB 39 MiB Castro::initialize_do_advance() 40 40 29 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 2330 KiB 28 MiB Castro::initialize_advance() 40 40 18 MiB 23 MiB Castro::buildMetrics() 32 32 14 MiB 15 MiB Castro::post_restart() 48 48 6740 KiB 14 MiB MLMG::prepareForSolve() 361 361 3364 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 251 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 270 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6735 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 14 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2210 B 2048 KiB Gravity::solve_for_phi() 40 40 619 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 16 KiB 2048 KiB BndryData::define() 576 576 309 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 198 KiB 671 KiB Castro::estTimeStep() 10 10 4171 B 480 KiB VisMF::Write(FabArray) 112 112 1229 B 320 KiB Castro::normalize_species() 30 30 7530 B 320 KiB amrex::average_down 469 469 1181 B 257 KiB MLMG::addInterpCorrection() 468 468 1110 B 257 KiB amrex::Dot() 592 592 3191 B 160 KiB FabArray::norminf() 398 398 2261 B 160 KiB check_for_negative_density() 5 5 282 B 160 KiB MultiFab::max() 6 6 55 B 160 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MultiFab::contains_nan() 10 10 29 B 20 KiB MLPoisson::Fsmooth() 60 60 3272 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 49 B 10 KiB FillBoundary_nowait() 336 336 282 B 9648 B MLCellLinOp::applyBC() 3892 3892 216 B 9344 B amrex::Copy() 56 56 5957 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B StateData::FillBoundary(geom) 960 960 101 B 4096 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLCGSolver::bicgstab 324 324 111 B 1472 B MLCellLinOp::defineBC() 36 36 347 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1378 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 111 KiB 8192 KiB VisMF::Write(FabArray) 120 120 242 KiB 3584 KiB VisMF::Read() 24 24 197 KiB 3000 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3272 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 49 B 10 KiB FillBoundary_nowait() 336 336 282 B 9648 B MLCellLinOp::applyBC() 1946 1946 215 B 9328 B amrex::Copy() 56 56 5957 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B StateData::FillBoundary(geom) 960 960 101 B 4096 B Gravity::get_old_grav_vector() 3 3 2663 B 3072 B Gravity::fill_multipole_BCs() 18 18 4 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::prepareForSolve() 7 7 520 B 1296 B amrex::average_down 37 37 224 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 23 B 400 B FabArray::norminf() 398 398 16 B 272 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2422 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.08-4-g67b8a5b04b7f) finalized