Initializing CUDA... CUDA initialized with 1 device. AMReX (23.04-9-g1e73fa8ec7c3) initialized Starting run at 07:37:28 UTC on 2023-04-06. Successfully read inputs file ... Castro git describe: 23.04-2-gabeccccb9 AMReX git describe: 23.04-9-g1e73fa8ec Microphysics git describe: 23.04-3-gdc41ef4e reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.056265202 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.032309466 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.045720024 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049747856 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.048182783 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.04757888 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.068238301 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.05567904 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.068323393 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047805155 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.047441687 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.04638843 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.06143563 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.055750502 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032451106 seconds Ending run at 07:37:29 UTC on 2023-04-06. Run time = 0.808780557 Run time without initialization = 0.67538018 Average number of zones advanced per microsecond: 3.881 Average number of zones advanced per microsecond per rank: 3.881 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8088 ... 0.8088 ... 0.8088 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2256 0.2256 0.2256 27.89% Castro::construct_ctu_hydro_source() 10 0.1776 0.1776 0.1776 21.96% MLCellLinOp::applyBC() 4379 0.07375 0.07375 0.07375 9.12% MLPoisson::Fsmooth() 3240 0.03114 0.03114 0.03114 3.85% FillBoundary_nowait() 3974 0.03047 0.03047 0.03047 3.77% StateData::FillBoundary(geom) 328 0.02368 0.02368 0.02368 2.93% amrex::Dot() 1100 0.0196 0.0196 0.0196 2.42% StateDataPhysBCFunct::() 41 0.01608 0.01608 0.01608 1.99% Castro::normalize_species() 62 0.01515 0.01515 0.01515 1.87% amrex::Copy() 1019 0.0144 0.0144 0.0144 1.78% Castro::computeTemp() 63 0.01417 0.01417 0.01417 1.75% FabArray::norminf() 734 0.01374 0.01374 0.01374 1.70% FabArray::ParallelCopy_nowait() 851 0.01296 0.01296 0.01296 1.60% FabArray::setVal() 1135 0.0127 0.0127 0.0127 1.57% MLPoisson::Fapply() 1128 0.009939 0.009939 0.009939 1.23% MLCellLinOp::defineAuxData() 11 0.009304 0.009304 0.009304 1.15% FabArray::Saxpy() 803 0.00787 0.00787 0.00787 0.97% FabArray::Xpay() 811 0.007751 0.007751 0.007751 0.96% Castro::enforce_min_density() 62 0.007218 0.007218 0.007218 0.89% Gravity::fill_multipole_BCs() 11 0.006602 0.006602 0.006602 0.82% MLMG::addInterpCorrection() 405 0.006575 0.006575 0.006575 0.81% Castro::estTimeStep() 21 0.006064 0.006064 0.006064 0.75% amrex::average_down 405 0.00577 0.00577 0.00577 0.71% Castro::reset_internal_energy(MultiFab) 63 0.004323 0.004323 0.004323 0.53% FabArray::LinComb() 550 0.004301 0.004301 0.004301 0.53% amrex::Add() 162 0.004268 0.004268 0.004268 0.53% Amr::checkPoint() 3 0.00399 0.00399 0.00399 0.49% BndryData::define() 11 0.003503 0.003503 0.003503 0.43% Castro::construct_new_gravity_source() 10 0.003398 0.003398 0.003398 0.42% Castro::construct_old_gravity_source() 10 0.002855 0.002855 0.002855 0.35% Amr::writePlotFile() 2 0.002395 0.002395 0.002395 0.30% Castro::do_advance_ctu() 10 0.002351 0.002351 0.002351 0.29% MLCGSolver::bicgstab 81 0.00211 0.00211 0.00211 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001613 0.001613 0.001613 0.20% Gravity::actual_solve_with_mlmg() 11 0.00148 0.00148 0.00148 0.18% Castro::reset_internal_energy(Fab) 504 0.001417 0.001417 0.001417 0.18% MLCellLinOp::setLevelBC() 11 0.001364 0.001364 0.001364 0.17% FabArray::mult() 43 0.001311 0.001311 0.001311 0.16% FabArray::setDomainBndry() 41 0.00128 0.00128 0.00128 0.16% Castro::initData() 1 0.001244 0.001244 0.001244 0.15% MultiFab::contains_nan() 20 0.001195 0.001195 0.001195 0.15% MLCellLinOp::prepareForSolve() 11 0.001099 0.001099 0.001099 0.14% MLCellLinOp::smooth() 1620 0.001085 0.001085 0.001085 0.13% Castro::enforce_speed_limit() 62 0.001036 0.001036 0.001036 0.13% MLCellLinOp::compGrad() 11 0.0009047 0.0009047 0.0009047 0.11% MLMG::prepareForSolve() 11 0.0008836 0.0008836 0.0008836 0.11% FabArray::FillBoundary() 3974 0.0008325 0.0008325 0.0008325 0.10% FabArrayBase::getCPC() 1313 0.0007408 0.0007408 0.0007408 0.09% FabArrayBase::CPC::define() 454 0.0006769 0.0006769 0.0006769 0.08% Gravity::get_new_grav_vector() 11 0.00063 0.00063 0.00063 0.08% FabArrayBase::getFB() 3974 0.0005961 0.0005961 0.0005961 0.07% Gravity::get_old_grav_vector() 10 0.0005413 0.0005413 0.0005413 0.07% Amr::InitAmr() 1 0.0004849 0.0004849 0.0004849 0.06% MLCellLinOp::apply() 1128 0.0004462 0.0004462 0.0004462 0.06% MLMG::mgVcycle() 81 0.0004017 0.0004017 0.0004017 0.05% MLLinOp::defineGrids() 11 0.0003571 0.0003571 0.0003571 0.04% Amr::coarseTimeStep() 10 0.000354 0.000354 0.000354 0.04% main() 1 0.0002997 0.0002997 0.0002997 0.04% MLCGSolver::ParallelAllReduce 1495 0.0002909 0.0002909 0.0002909 0.04% MultiFab::max() 11 0.0002563 0.0002563 0.0002563 0.03% FillPatchIterator::Initialize 41 0.0002451 0.0002451 0.0002451 0.03% FabArray::ParallelCopy() 851 0.0002348 0.0002348 0.0002348 0.03% Castro::subcycle_advance_ctu() 10 0.000217 0.000217 0.000217 0.03% MLCellLinOp::defineBC() 11 0.0002166 0.0002166 0.0002166 0.03% MLCellLinOp::correctionResidual() 486 0.0002161 0.0002161 0.0002161 0.03% Amr::timeStep() 10 0.0001631 0.0001631 0.0001631 0.02% StateData::checkPoint() 12 0.0001429 0.0001429 0.0001429 0.02% Castro::create_source_corrector() 10 0.0001422 0.0001422 0.0001422 0.02% Gravity::update_max_rhs() 11 0.0001345 0.0001345 0.0001345 0.02% Gravity::solve_for_phi() 10 0.0001102 0.0001102 0.0001102 0.01% MLMG:computeResOfCorrection() 405 0.0001084 0.0001084 0.0001084 0.01% Castro::Castro() 1 9.976e-05 9.976e-05 9.976e-05 0.01% MLMG::mgVcycle_down::0 81 8.964e-05 8.964e-05 8.964e-05 0.01% FabArrayBase::FB::FB() 56 8.897e-05 8.897e-05 8.897e-05 0.01% MLMG::actualBottomSolve() 81 8.863e-05 8.863e-05 8.863e-05 0.01% MLMG::mgVcycle_down::1 81 8.194e-05 8.194e-05 8.194e-05 0.01% Castro::expand_state() 10 7.872e-05 7.872e-05 7.872e-05 0.01% Castro::initialize_advance() 10 7.87e-05 7.87e-05 7.87e-05 0.01% MLMG::mgVcycle_down::2 81 7.815e-05 7.815e-05 7.815e-05 0.01% AmrLevel::checkPoint() 3 7.793e-05 7.793e-05 7.793e-05 0.01% MLMG::mgVcycle_down::4 81 7.395e-05 7.395e-05 7.395e-05 0.01% MLMG::mgVcycle_down::3 81 7.362e-05 7.362e-05 7.362e-05 0.01% Castro::clean_state() 62 7.289e-05 7.289e-05 7.289e-05 0.01% MLMG::solve() 11 7.15e-05 7.15e-05 7.15e-05 0.01% Castro::finalize_advance() 10 5.923e-05 5.923e-05 5.923e-05 0.01% MLMG::mgVcycle_up::4 81 5.92e-05 5.92e-05 5.92e-05 0.01% MLMG::oneIter() 81 5.263e-05 5.263e-05 5.263e-05 0.01% MLMG::mgVcycle_up::0 81 5.169e-05 5.169e-05 5.169e-05 0.01% Castro::initialize_do_advance() 10 5.12e-05 5.12e-05 5.12e-05 0.01% MLMG::mgVcycle_up::1 81 4.883e-05 4.883e-05 4.883e-05 0.01% MLMG::mgVcycle_up::3 81 4.834e-05 4.834e-05 4.834e-05 0.01% MLMG::mgVcycle_up::2 81 4.679e-05 4.679e-05 4.679e-05 0.01% MLCellLinOp::solutionResidual() 92 4.654e-05 4.654e-05 4.654e-05 0.01% Castro::construct_new_source() 50 4.107e-05 4.107e-05 4.107e-05 0.01% Castro::advance() 10 4.099e-05 4.099e-05 4.099e-05 0.01% Castro::swap_state_time_levels() 10 4.094e-05 4.094e-05 4.094e-05 0.01% StateData::define() 4 3.839e-05 3.839e-05 3.839e-05 0.00% Castro::finalize_do_advance() 10 3.588e-05 3.588e-05 3.588e-05 0.00% MLMG::ResNormInf() 92 3.435e-05 3.435e-05 3.435e-05 0.00% Castro::enforce_consistent_e() 1 3.294e-05 3.294e-05 3.294e-05 0.00% MLMG::mgVcycle_bottom 81 3.169e-05 3.169e-05 3.169e-05 0.00% Castro::post_timestep() 10 3.141e-05 3.141e-05 3.141e-05 0.00% FillPatchSingleLevel 41 2.975e-05 2.975e-05 2.975e-05 0.00% MLMG::computeResidual() 81 2.816e-05 2.816e-05 2.816e-05 0.00% Castro::construct_new_gravity() 10 2.483e-05 2.483e-05 2.483e-05 0.00% Amr::writeSmallPlotFile() 1 2.482e-05 2.482e-05 2.482e-05 0.00% Castro::initMFs() 1 2.45e-05 2.45e-05 2.45e-05 0.00% MLPoisson::define() 11 2.361e-05 2.361e-05 2.361e-05 0.00% Amr::defBaseLevel() 1 2.335e-05 2.335e-05 2.335e-05 0.00% makeSFC 55 2.311e-05 2.311e-05 2.311e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.167e-05 2.167e-05 2.167e-05 0.00% Amr::FinalizeInit() 1 2.034e-05 2.034e-05 2.034e-05 0.00% Castro::buildMetrics() 1 1.836e-05 1.836e-05 1.836e-05 0.00% Castro::construct_old_source() 50 1.741e-05 1.741e-05 1.741e-05 0.00% Castro::do_new_sources() 10 1.675e-05 1.675e-05 1.675e-05 0.00% Castro::do_old_sources() 10 1.644e-05 1.644e-05 1.644e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.581e-05 1.581e-05 1.581e-05 0.00% DistributionMapping::Distribute() 56 1.548e-05 1.548e-05 1.548e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.525e-05 1.525e-05 1.525e-05 0.00% MLPoisson::prepareForSolve() 11 1.487e-05 1.487e-05 1.487e-05 0.00% MLMG::computeMLResidual() 11 1.453e-05 1.453e-05 1.453e-05 0.00% MLLinOp::define() 11 1.188e-05 1.188e-05 1.188e-05 0.00% Castro::check_for_nan() 20 1.084e-05 1.084e-05 1.084e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.076e-05 1.076e-05 1.076e-05 0.00% Castro::apply_source_to_state() 20 1.033e-05 1.033e-05 1.033e-05 0.00% Castro::construct_old_gravity() 10 9.42e-06 9.42e-06 9.42e-06 0.00% Amr::initSubcycle() 1 9.381e-06 9.381e-06 9.381e-06 0.00% Gravity::swapTimeLevels() 10 9.093e-06 9.093e-06 9.093e-06 0.00% Gravity::actual_multilevel_solve() 1 8.508e-06 8.508e-06 8.508e-06 0.00% Castro::computeNewDt() 9 6.641e-06 6.641e-06 6.641e-06 0.00% MLMG::getGradSolution() 11 5.53e-06 5.53e-06 5.53e-06 0.00% Amr::InitializeInit() 1 4.827e-06 4.827e-06 4.827e-06 0.00% AmrLevel::checkPointPost() 3 4.573e-06 4.573e-06 4.573e-06 0.00% Castro::post_init() 1 4.101e-06 4.101e-06 4.101e-06 0.00% Gravity::set_mass_offset() 11 4.018e-06 4.018e-06 4.018e-06 0.00% Castro::computeInitialDt() 2 3.877e-06 3.877e-06 3.877e-06 0.00% MLMG::MLRhsNormInf() 11 3.79e-06 3.79e-06 3.79e-06 0.00% MLMG::MLResNormInf() 11 3.648e-06 3.648e-06 3.648e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.464e-06 3.464e-06 3.464e-06 0.00% Castro::retry_advance_ctu() 10 3.435e-06 3.435e-06 3.435e-06 0.00% Castro::FluxRegCrseInit 10 2.667e-06 2.667e-06 2.667e-06 0.00% Amr::init() 1 2.397e-06 2.397e-06 2.397e-06 0.00% Castro::FluxRegFineAdd() 10 2.046e-06 2.046e-06 2.046e-06 0.00% AmrLevel::checkPointPre() 3 1.796e-06 1.796e-06 1.796e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.786e-06 1.786e-06 1.786e-06 0.00% Amr::initialInit() 1 1.437e-06 1.437e-06 1.437e-06 0.00% Castro::post_regrid() 1 8.49e-07 8.49e-07 8.49e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8088 0.8088 0.8088 100.00% Amr::coarseTimeStep() 10 0.6427 0.6427 0.6427 79.46% Amr::timeStep() 10 0.5269 0.5269 0.5269 65.15% Castro::advance() 10 0.5199 0.5199 0.5199 64.28% Castro::subcycle_advance_ctu() 10 0.5088 0.5088 0.5088 62.91% Castro::do_advance_ctu() 10 0.5086 0.5086 0.5086 62.88% Gravity::solve_phi_with_mlmg() 11 0.2734 0.2734 0.2734 33.80% Gravity::actual_solve_with_mlmg() 11 0.2663 0.2663 0.2663 32.93% Castro::construct_new_gravity() 10 0.2526 0.2526 0.2526 31.23% MLMG::solve() 11 0.2464 0.2464 0.2464 30.46% Gravity::solve_for_phi() 10 0.2371 0.2371 0.2371 29.32% MLMG::oneIter() 81 0.2325 0.2325 0.2325 28.75% MLMG::mgVcycle() 81 0.2289 0.2289 0.2289 28.30% VisMF::Write(FabArray) 11 0.2256 0.2256 0.2256 27.89% Castro::construct_ctu_hydro_source() 10 0.1776 0.1776 0.1776 21.96% Amr::checkPoint() 3 0.1678 0.1678 0.1678 20.75% AmrLevel::checkPoint() 3 0.1638 0.1638 0.1638 20.25% StateData::checkPoint() 12 0.1637 0.1637 0.1637 20.25% Amr::init() 1 0.1328 0.1328 0.1328 16.41% MLCellLinOp::smooth() 1620 0.113 0.113 0.113 13.97% MLCellLinOp::applyBC() 4379 0.1057 0.1057 0.1057 13.07% MLMG::mgVcycle_bottom 81 0.07027 0.07027 0.07027 8.69% MLMG::actualBottomSolve() 81 0.07024 0.07024 0.07024 8.68% MLCGSolver::bicgstab 81 0.06957 0.06957 0.06957 8.60% Amr::writePlotFile() 2 0.06488 0.06488 0.06488 8.02% FillPatchIterator::Initialize 41 0.04537 0.04537 0.04537 5.61% Amr::initialInit() 1 0.04407 0.04407 0.04407 5.45% FillPatchSingleLevel 41 0.04384 0.04384 0.04384 5.42% Castro::clean_state() 62 0.0425 0.0425 0.0425 5.25% Amr::FinalizeInit() 1 0.03994 0.03994 0.03994 4.94% StateDataPhysBCFunct::() 41 0.03976 0.03976 0.03976 4.92% Castro::post_init() 1 0.03853 0.03853 0.03853 4.76% Gravity::multilevel_solve_for_new_phi() 1 0.03668 0.03668 0.03668 4.54% Gravity::actual_multilevel_solve() 1 0.03667 0.03667 0.03667 4.53% MLCellLinOp::apply() 1128 0.03489 0.03489 0.03489 4.31% MLMG::mgVcycle_down::0 81 0.03299 0.03299 0.03299 4.08% FabArray::FillBoundary() 3974 0.03199 0.03199 0.03199 3.95% FillBoundary_nowait() 3974 0.03115 0.03115 0.03115 3.85% MLPoisson::Fsmooth() 3240 0.03114 0.03114 0.03114 3.85% MLMG::mgVcycle_up::0 81 0.02499 0.02499 0.02499 3.09% StateData::FillBoundary(geom) 328 0.02368 0.02368 0.02368 2.93% Castro::initialize_do_advance() 10 0.02208 0.02208 0.02208 2.73% MLCellLinOp::correctionResidual() 486 0.02141 0.02141 0.02141 2.65% Castro::computeTemp() 63 0.01991 0.01991 0.01991 2.46% amrex::Dot() 1100 0.0196 0.0196 0.0196 2.42% MLMG:computeResOfCorrection() 405 0.01886 0.01886 0.01886 2.33% Gravity::get_new_grav_vector() 11 0.01702 0.01702 0.01702 2.10% MLPoisson::define() 11 0.01569 0.01569 0.01569 1.94% MLMG::mgVcycle_down::1 81 0.01519 0.01519 0.01519 1.88% Castro::normalize_species() 62 0.01515 0.01515 0.01515 1.87% Castro::construct_old_gravity() 10 0.0149 0.0149 0.0149 1.84% Gravity::get_old_grav_vector() 10 0.01489 0.01489 0.01489 1.84% Castro::expand_state() 10 0.01449 0.01449 0.01449 1.79% amrex::Copy() 1019 0.0144 0.0144 0.0144 1.78% MLMG::mgVcycle_down::2 81 0.01415 0.01415 0.01415 1.75% FabArray::ParallelCopy() 851 0.014 0.014 0.014 1.73% MLMG::mgVcycle_down::3 81 0.01378 0.01378 0.01378 1.70% FabArray::ParallelCopy_nowait() 851 0.01376 0.01376 0.01376 1.70% FabArray::norminf() 734 0.01374 0.01374 0.01374 1.70% MLMG::mgVcycle_down::4 81 0.01361 0.01361 0.01361 1.68% FabArray::setVal() 1135 0.0127 0.0127 0.0127 1.57% Castro::do_new_sources() 10 0.01263 0.01263 0.01263 1.56% MLCGSolver::ParallelAllReduce 1495 0.01178 0.01178 0.01178 1.46% MLMG::addInterpCorrection() 405 0.01153 0.01153 0.01153 1.43% MLMG::mgVcycle_up::4 81 0.01111 0.01111 0.01111 1.37% MLMG::mgVcycle_up::1 81 0.01103 0.01103 0.01103 1.36% MLMG::mgVcycle_up::2 81 0.01078 0.01078 0.01078 1.33% amrex::average_down 405 0.01076 0.01076 0.01076 1.33% Castro::do_old_sources() 10 0.01065 0.01065 0.01065 1.32% MLMG::mgVcycle_up::3 81 0.01061 0.01061 0.01061 1.31% MLCellLinOp::defineAuxData() 11 0.01058 0.01058 0.01058 1.31% Castro::initialize_advance() 10 0.01044 0.01044 0.01044 1.29% MLPoisson::Fapply() 1128 0.009939 0.009939 0.009939 1.23% FabArray::Saxpy() 803 0.00787 0.00787 0.00787 0.97% FabArray::Xpay() 811 0.007751 0.007751 0.007751 0.96% Castro::enforce_min_density() 62 0.007218 0.007218 0.007218 0.89% MLCellLinOp::solutionResidual() 92 0.006966 0.006966 0.006966 0.86% Castro::post_timestep() 10 0.00687 0.00687 0.00687 0.85% Gravity::fill_multipole_BCs() 11 0.006848 0.006848 0.006848 0.85% Castro::estTimeStep() 21 0.006064 0.006064 0.006064 0.75% MLMG::computeResidual() 81 0.005997 0.005997 0.005997 0.74% Castro::reset_internal_energy(MultiFab) 63 0.00574 0.00574 0.00574 0.71% MLCellLinOp::defineBC() 11 0.004662 0.004662 0.004662 0.58% MLMG::prepareForSolve() 11 0.004566 0.004566 0.004566 0.56% BndryData::define() 11 0.004446 0.004446 0.004446 0.55% FabArray::LinComb() 550 0.004301 0.004301 0.004301 0.53% amrex::Add() 162 0.004268 0.004268 0.004268 0.53% Amr::InitializeInit() 1 0.004131 0.004131 0.004131 0.51% Amr::defBaseLevel() 1 0.004126 0.004126 0.004126 0.51% Castro::initData() 1 0.003601 0.003601 0.003601 0.45% Castro::construct_new_source() 50 0.003439 0.003439 0.003439 0.43% Castro::construct_new_gravity_source() 10 0.003398 0.003398 0.003398 0.42% Castro::computeNewDt() 9 0.003192 0.003192 0.003192 0.39% Castro::construct_old_source() 50 0.002872 0.002872 0.002872 0.36% Castro::construct_old_gravity_source() 10 0.002855 0.002855 0.002855 0.35% MLMG::ResNormInf() 92 0.002036 0.002036 0.002036 0.25% Castro::apply_source_to_state() 20 0.001815 0.001815 0.001815 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001613 0.001613 0.001613 0.20% FabArrayBase::getCPC() 1313 0.001418 0.001418 0.001418 0.18% Castro::reset_internal_energy(Fab) 504 0.001417 0.001417 0.001417 0.18% MLMG::getGradSolution() 11 0.001378 0.001378 0.001378 0.17% MLCellLinOp::compGrad() 11 0.001372 0.001372 0.001372 0.17% MLCellLinOp::setLevelBC() 11 0.001364 0.001364 0.001364 0.17% FabArray::mult() 43 0.001311 0.001311 0.001311 0.16% FabArray::setDomainBndry() 41 0.00128 0.00128 0.00128 0.16% Castro::check_for_nan() 20 0.001206 0.001206 0.001206 0.15% MultiFab::contains_nan() 20 0.001195 0.001195 0.001195 0.15% Castro::post_regrid() 1 0.001154 0.001154 0.001154 0.14% MLPoisson::prepareForSolve() 11 0.001114 0.001114 0.001114 0.14% MLCellLinOp::prepareForSolve() 11 0.001099 0.001099 0.001099 0.14% Castro::enforce_speed_limit() 62 0.001036 0.001036 0.001036 0.13% MLMG::computeMLResidual() 11 0.001011 0.001011 0.001011 0.12% Castro::computeInitialDt() 2 0.0009652 0.0009652 0.0009652 0.12% Gravity::update_max_rhs() 11 0.0008389 0.0008389 0.0008389 0.10% FabArrayBase::getFB() 3974 0.0006851 0.0006851 0.0006851 0.08% FabArrayBase::CPC::define() 454 0.0006769 0.0006769 0.0006769 0.08% Castro::finalize_advance() 10 0.000567 0.000567 0.000567 0.07% Amr::InitAmr() 1 0.0004943 0.0004943 0.0004943 0.06% Castro::Castro() 1 0.0004469 0.0004469 0.0004469 0.06% Gravity::swapTimeLevels() 10 0.00043 0.00043 0.00043 0.05% MLLinOp::define() 11 0.0004227 0.0004227 0.0004227 0.05% MLLinOp::defineGrids() 11 0.0004108 0.0004108 0.0004108 0.05% MLMG::MLResNormInf() 11 0.0002835 0.0002835 0.0002835 0.04% MultiFab::max() 11 0.0002563 0.0002563 0.0002563 0.03% MLMG::MLRhsNormInf() 11 0.0002167 0.0002167 0.0002167 0.03% Castro::buildMetrics() 1 0.0001623 0.0001623 0.0001623 0.02% Castro::create_source_corrector() 10 0.0001422 0.0001422 0.0001422 0.02% FabArrayBase::FB::FB() 56 8.897e-05 8.897e-05 8.897e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.196e-05 5.196e-05 5.196e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.914e-05 4.914e-05 4.914e-05 0.01% Castro::swap_state_time_levels() 10 4.094e-05 4.094e-05 4.094e-05 0.01% StateData::define() 4 3.839e-05 3.839e-05 3.839e-05 0.00% makeSFC 55 3.671e-05 3.671e-05 3.671e-05 0.00% Castro::finalize_do_advance() 10 3.588e-05 3.588e-05 3.588e-05 0.00% Castro::enforce_consistent_e() 1 3.294e-05 3.294e-05 3.294e-05 0.00% Amr::writeSmallPlotFile() 1 2.482e-05 2.482e-05 2.482e-05 0.00% Castro::initMFs() 1 2.45e-05 2.45e-05 2.45e-05 0.00% DistributionMapping::Distribute() 56 1.548e-05 1.548e-05 1.548e-05 0.00% Amr::initSubcycle() 1 9.381e-06 9.381e-06 9.381e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.341e-06 5.341e-06 5.341e-06 0.00% AmrLevel::checkPointPost() 3 4.573e-06 4.573e-06 4.573e-06 0.00% Gravity::set_mass_offset() 11 4.018e-06 4.018e-06 4.018e-06 0.00% Castro::retry_advance_ctu() 10 3.435e-06 3.435e-06 3.435e-06 0.00% Castro::FluxRegCrseInit 10 2.667e-06 2.667e-06 2.667e-06 0.00% Castro::FluxRegFineAdd() 10 2.046e-06 2.046e-06 2.046e-06 0.00% AmrLevel::checkPointPre() 3 1.796e-06 1.796e-06 1.796e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.786e-06 1.786e-06 1.786e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 102 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 102 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1272 KiB 39 MiB Castro::initialize_do_advance() 80 80 24 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 2252 KiB 28 MiB Castro::initialize_advance() 80 80 15 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7614 KiB 14 MiB MLMG::prepareForSolve() 660 660 3748 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 218 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 188 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7519 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 13 KiB 2053 KiB Gravity::update_max_rhs() 88 88 1977 B 2048 KiB Gravity::solve_for_phi() 80 80 599 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 92 KiB 2048 KiB BndryData::define() 1056 1056 342 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 217 KiB 671 KiB Castro::estTimeStep() 21 21 2874 B 480 KiB VisMF::Write(FabArray) 656 656 3650 B 320 KiB Castro::normalize_species() 62 62 6098 B 320 KiB amrex::average_down 1054 1054 1315 B 257 KiB MLMG::addInterpCorrection() 1053 1053 1223 B 257 KiB amrex::Dot() 1343 1343 3573 B 160 KiB FabArray::norminf() 896 896 2513 B 160 KiB Castro::do_advance_ctu() 10 10 431 B 160 KiB MultiFab::max() 11 11 49 B 160 KiB Castro::initData() 1 1 29 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 29 B 20 KiB MLPoisson::Fsmooth() 132 132 3663 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 48 B 10 KiB FillBoundary_nowait() 751 751 330 B 9648 B MLCellLinOp::applyBC() 8758 8758 237 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3855 B 6144 B StateData::FillBoundary(geom) 1992 1992 64 B 2976 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 729 729 125 B 1472 B MLCellLinOp::defineBC() 66 66 385 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 627 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 53 KiB 8192 KiB VisMF::Write(FabArray) 744 744 646 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3663 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 48 B 10 KiB FillBoundary_nowait() 751 751 329 B 9648 B MLCellLinOp::applyBC() 4379 4379 235 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3855 B 6144 B Gravity::get_new_grav_vector() 3 3 2905 B 3072 B StateData::FillBoundary(geom) 1992 1992 65 B 2976 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 82 82 272 B 1296 B MLMG::addInterpCorrection() 81 81 2 B 1024 B MLMG::prepareForSolve() 11 11 312 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1343 1343 26 B 400 B FabArray::norminf() 896 896 18 B 272 B Castro::estTimeStep() 21 21 0 B 32 B MultiFab::contains_nan() 20 20 0 B 16 B MultiFab::max() 11 11 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B Castro::do_advance_ctu() 10 10 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.04-9-g1e73fa8ec7c3) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.04-9-g1e73fa8ec7c3) initialized Starting run at 07:37:30 UTC on 2023-04-06. Successfully read inputs file ... Castro git describe: 23.04-2-gabeccccb9 AMReX git describe: 23.04-9-g1e73fa8ec Microphysics git describe: 23.04-3-gdc41ef4e reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.448536218 Restart time = 0.046034649 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.049627977 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047941029 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058520323 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.064570878 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.067208202 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032653601 seconds Ending run at 07:37:30 UTC on 2023-04-06. Run time = 0.367576169 Run time without initialization = 0.320939406 Average number of zones advanced per microsecond: 4.084 Average number of zones advanced per microsecond per rank: 4.084 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3676 ... 0.3676 ... 0.3676 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1037 0.1037 0.1037 28.22% VisMF::Read() 3 0.03981 0.03981 0.03981 10.83% MLCellLinOp::applyBC() 1946 0.03221 0.03221 0.03221 8.76% VisMF::Write(FabArray) 1 0.03102 0.03102 0.03102 8.44% MLPoisson::Fsmooth() 1440 0.01371 0.01371 0.01371 3.73% FillBoundary_nowait() 1766 0.01328 0.01328 0.01328 3.61% StateData::FillBoundary(geom) 160 0.01156 0.01156 0.01156 3.15% Castro::normalize_species() 30 0.009025 0.009025 0.009025 2.46% amrex::Dot() 484 0.008453 0.008453 0.008453 2.30% Castro::enforce_min_density() 30 0.006861 0.006861 0.006861 1.87% amrex::Copy() 463 0.006791 0.006791 0.006791 1.85% StateDataPhysBCFunct::() 20 0.006116 0.006116 0.006116 1.66% FabArray::setVal() 537 0.006085 0.006085 0.006085 1.66% FabArray::norminf() 326 0.00597 0.00597 0.00597 1.62% FabArray::ParallelCopy_nowait() 380 0.005917 0.005917 0.005917 1.61% Castro::computeTemp() 30 0.005771 0.005771 0.005771 1.57% MLCellLinOp::defineAuxData() 6 0.005122 0.005122 0.005122 1.39% MLPoisson::Fapply() 500 0.004289 0.004289 0.004289 1.17% Castro::estTimeStep() 10 0.004015 0.004015 0.004015 1.09% FabArray::Saxpy() 355 0.003531 0.003531 0.003531 0.96% FabArray::Xpay() 361 0.0034 0.0034 0.0034 0.93% MLMG::addInterpCorrection() 180 0.00288 0.00288 0.00288 0.78% Gravity::fill_multipole_BCs() 6 0.002766 0.002766 0.002766 0.75% amrex::average_down 180 0.002523 0.002523 0.002523 0.69% Amr::restart() 1 0.002513 0.002513 0.002513 0.68% Castro::do_advance_ctu() 5 0.002422 0.002422 0.002422 0.66% Castro::reset_internal_energy(MultiFab) 30 0.002281 0.002281 0.002281 0.62% BndryData::define() 6 0.001931 0.001931 0.001931 0.53% FabArray::LinComb() 242 0.001846 0.001846 0.001846 0.50% amrex::Add() 72 0.001835 0.001835 0.001835 0.50% Castro::construct_new_gravity_source() 5 0.001802 0.001802 0.001802 0.49% Castro::construct_old_gravity_source() 5 0.001499 0.001499 0.001499 0.41% Amr::writePlotFile() 1 0.001468 0.001468 0.001468 0.40% Castro::enforce_speed_limit() 30 0.001335 0.001335 0.001335 0.36% MLCGSolver::bicgstab 36 0.0009272 0.0009272 0.0009272 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008935 0.0008935 0.0008935 0.24% Gravity::actual_solve_with_mlmg() 6 0.0007679 0.0007679 0.0007679 0.21% MLCellLinOp::setLevelBC() 6 0.0007228 0.0007228 0.0007228 0.20% Castro::reset_internal_energy(Fab) 240 0.000669 0.000669 0.000669 0.18% FabArray::mult() 22 0.0006406 0.0006406 0.0006406 0.17% FabArray::setDomainBndry() 20 0.0006313 0.0006313 0.0006313 0.17% MLCellLinOp::prepareForSolve() 6 0.0005906 0.0005906 0.0005906 0.16% MultiFab::contains_nan() 10 0.0005893 0.0005893 0.0005893 0.16% MLCellLinOp::compGrad() 6 0.0004859 0.0004859 0.0004859 0.13% MLMG::prepareForSolve() 6 0.0004797 0.0004797 0.0004797 0.13% MLCellLinOp::smooth() 720 0.0004642 0.0004642 0.0004642 0.13% FabArrayBase::CPC::define() 244 0.0004141 0.0004141 0.0004141 0.11% Amr::InitAmr() 1 0.0003954 0.0003954 0.0003954 0.11% FabArray::FillBoundary() 1766 0.0003522 0.0003522 0.0003522 0.10% FabArrayBase::getCPC() 632 0.0003496 0.0003496 0.0003496 0.10% Gravity::get_old_grav_vector() 5 0.0002966 0.0002966 0.0002966 0.08% main() 1 0.0002824 0.0002824 0.0002824 0.08% Gravity::get_new_grav_vector() 5 0.0002717 0.0002717 0.0002717 0.07% FabArrayBase::getFB() 1766 0.0002519 0.0002519 0.0002519 0.07% MLCellLinOp::apply() 500 0.0001892 0.0001892 0.0001892 0.05% Amr::coarseTimeStep() 5 0.0001778 0.0001778 0.0001778 0.05% MLMG::mgVcycle() 36 0.0001668 0.0001668 0.0001668 0.05% Castro::construct_new_source() 25 0.0001509 0.0001509 0.0001509 0.04% MultiFab::max() 6 0.0001356 0.0001356 0.0001356 0.04% MLLinOp::defineGrids() 6 0.0001259 0.0001259 0.0001259 0.03% MLCGSolver::ParallelAllReduce 659 0.0001201 0.0001201 0.0001201 0.03% MLCellLinOp::defineBC() 6 0.0001146 0.0001146 0.0001146 0.03% Castro::create_source_corrector() 5 0.0001121 0.0001121 0.0001121 0.03% FillPatchIterator::Initialize 20 0.0001118 0.0001118 0.0001118 0.03% FabArray::ParallelCopy() 380 0.0001091 0.0001091 0.0001091 0.03% MLCellLinOp::correctionResidual() 216 9.193e-05 9.193e-05 9.193e-05 0.03% Castro::initialize_do_advance() 5 8.94e-05 8.94e-05 8.94e-05 0.02% Castro::post_timestep() 5 8.639e-05 8.639e-05 8.639e-05 0.02% Amr::timeStep() 5 8.177e-05 8.177e-05 8.177e-05 0.02% AmrLevel::restart() 1 7.557e-05 7.557e-05 7.557e-05 0.02% StateData::restartDoit() 4 6.116e-05 6.116e-05 6.116e-05 0.02% Gravity::update_max_rhs() 6 5.996e-05 5.996e-05 5.996e-05 0.02% FabArrayBase::FB::FB() 26 5.975e-05 5.975e-05 5.975e-05 0.02% Castro::construct_old_gravity() 5 5.864e-05 5.864e-05 5.864e-05 0.02% Gravity::solve_for_phi() 5 5.665e-05 5.665e-05 5.665e-05 0.02% Castro::subcycle_advance_ctu() 5 5.023e-05 5.023e-05 5.023e-05 0.01% Castro::expand_state() 5 4.83e-05 4.83e-05 4.83e-05 0.01% MLMG:computeResOfCorrection() 180 4.672e-05 4.672e-05 4.672e-05 0.01% MLMG::mgVcycle_down::0 36 4.015e-05 4.015e-05 4.015e-05 0.01% MLMG::solve() 6 3.918e-05 3.918e-05 3.918e-05 0.01% MLMG::actualBottomSolve() 36 3.899e-05 3.899e-05 3.899e-05 0.01% MLMG::mgVcycle_down::1 36 3.665e-05 3.665e-05 3.665e-05 0.01% Castro::clean_state() 30 3.577e-05 3.577e-05 3.577e-05 0.01% Castro::initialize_advance() 5 3.432e-05 3.432e-05 3.432e-05 0.01% MLMG::mgVcycle_down::2 36 3.355e-05 3.355e-05 3.355e-05 0.01% MLMG::mgVcycle_down::4 36 3.336e-05 3.336e-05 3.336e-05 0.01% MLMG::mgVcycle_down::3 36 3.259e-05 3.259e-05 3.259e-05 0.01% Castro::construct_old_source() 25 3.149e-05 3.149e-05 3.149e-05 0.01% Castro::finalize_advance() 5 2.869e-05 2.869e-05 2.869e-05 0.01% MLMG::mgVcycle_up::4 36 2.798e-05 2.798e-05 2.798e-05 0.01% Castro::buildMetrics() 1 2.682e-05 2.682e-05 2.682e-05 0.01% Amr::writeSmallPlotFile() 1 2.573e-05 2.573e-05 2.573e-05 0.01% Castro::initMFs() 1 2.569e-05 2.569e-05 2.569e-05 0.01% MLMG::mgVcycle_up::0 36 2.446e-05 2.446e-05 2.446e-05 0.01% Castro::swap_state_time_levels() 5 2.388e-05 2.388e-05 2.388e-05 0.01% MLMG::oneIter() 36 2.343e-05 2.343e-05 2.343e-05 0.01% MLCellLinOp::solutionResidual() 42 2.13e-05 2.13e-05 2.13e-05 0.01% MLMG::mgVcycle_up::3 36 2.12e-05 2.12e-05 2.12e-05 0.01% Castro::post_restart() 1 2.112e-05 2.112e-05 2.112e-05 0.01% MLMG::mgVcycle_up::2 36 2.079e-05 2.079e-05 2.079e-05 0.01% MLMG::mgVcycle_up::1 36 1.992e-05 1.992e-05 1.992e-05 0.01% Castro::finalize_do_advance() 5 1.962e-05 1.962e-05 1.962e-05 0.01% MLMG::ResNormInf() 42 1.745e-05 1.745e-05 1.745e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.514e-05 1.514e-05 1.514e-05 0.00% MLMG::mgVcycle_bottom 36 1.497e-05 1.497e-05 1.497e-05 0.00% MLPoisson::define() 6 1.424e-05 1.424e-05 1.424e-05 0.00% Castro::advance() 5 1.33e-05 1.33e-05 1.33e-05 0.00% MLMG::computeResidual() 36 1.313e-05 1.313e-05 1.313e-05 0.00% FillPatchSingleLevel 20 1.306e-05 1.306e-05 1.306e-05 0.00% Castro::construct_new_gravity() 5 1.258e-05 1.258e-05 1.258e-05 0.00% makeSFC 30 1.248e-05 1.248e-05 1.248e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.193e-05 1.193e-05 1.193e-05 0.00% MLPoisson::prepareForSolve() 6 1.087e-05 1.087e-05 1.087e-05 0.00% Castro::do_new_sources() 5 9.963e-06 9.963e-06 9.963e-06 0.00% MLMG::computeMLResidual() 6 9.589e-06 9.589e-06 9.589e-06 0.00% DistributionMapping::Distribute() 31 9.277e-06 9.277e-06 9.277e-06 0.00% Amr::initSubcycle() 1 9.16e-06 9.16e-06 9.16e-06 0.00% Castro::do_old_sources() 5 8.752e-06 8.752e-06 8.752e-06 0.00% Gravity::actual_multilevel_solve() 1 7.651e-06 7.651e-06 7.651e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.25e-06 7.25e-06 7.25e-06 0.00% MLLinOp::define() 6 5.99e-06 5.99e-06 5.99e-06 0.00% Castro::check_for_nan() 10 5.987e-06 5.987e-06 5.987e-06 0.00% Castro::apply_source_to_state() 10 5.394e-06 5.394e-06 5.394e-06 0.00% Gravity::swapTimeLevels() 5 4.587e-06 4.587e-06 4.587e-06 0.00% Castro::computeNewDt() 5 3.297e-06 3.297e-06 3.297e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.278e-06 3.278e-06 3.278e-06 0.00% MLMG::getGradSolution() 6 2.884e-06 2.884e-06 2.884e-06 0.00% MLMG::MLResNormInf() 6 2.167e-06 2.167e-06 2.167e-06 0.00% Gravity::set_mass_offset() 6 2.163e-06 2.163e-06 2.163e-06 0.00% MLMG::MLRhsNormInf() 6 2.156e-06 2.156e-06 2.156e-06 0.00% Castro::retry_advance_ctu() 5 1.672e-06 1.672e-06 1.672e-06 0.00% Castro::FluxRegCrseInit 5 1.498e-06 1.498e-06 1.498e-06 0.00% Castro::FluxRegFineAdd() 5 1.138e-06 1.138e-06 1.138e-06 0.00% MLLinOp::makeSubCommunicator() 6 9.92e-07 9.92e-07 9.92e-07 0.00% Amr::init() 1 9.55e-07 9.55e-07 9.55e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3676 0.3676 0.3676 100.00% Amr::coarseTimeStep() 5 0.288 0.288 0.288 78.36% Amr::timeStep() 5 0.2857 0.2857 0.2857 77.71% Castro::advance() 5 0.2807 0.2807 0.2807 76.35% Castro::subcycle_advance_ctu() 5 0.2743 0.2743 0.2743 74.62% Castro::do_advance_ctu() 5 0.2743 0.2743 0.2743 74.61% Castro::construct_new_gravity() 5 0.1264 0.1264 0.1264 34.38% Gravity::solve_phi_with_mlmg() 6 0.1218 0.1218 0.1218 33.12% Gravity::actual_solve_with_mlmg() 6 0.1187 0.1187 0.1187 32.30% Gravity::solve_for_phi() 5 0.1187 0.1187 0.1187 32.29% MLMG::solve() 6 0.1079 0.1079 0.1079 29.35% Castro::construct_ctu_hydro_source() 5 0.1037 0.1037 0.1037 28.22% MLMG::oneIter() 36 0.1012 0.1012 0.1012 27.52% MLMG::mgVcycle() 36 0.09961 0.09961 0.09961 27.10% MLCellLinOp::smooth() 720 0.04929 0.04929 0.04929 13.41% MLCellLinOp::applyBC() 1946 0.04615 0.04615 0.04615 12.55% Amr::init() 1 0.0461 0.0461 0.0461 12.54% Amr::restart() 1 0.0461 0.0461 0.0461 12.54% AmrLevel::restart() 1 0.04001 0.04001 0.04001 10.88% StateData::restartDoit() 4 0.03993 0.03993 0.03993 10.86% VisMF::Read() 3 0.03981 0.03981 0.03981 10.83% Amr::writePlotFile() 1 0.03274 0.03274 0.03274 8.91% VisMF::Write(FabArray) 1 0.03102 0.03102 0.03102 8.44% MLMG::mgVcycle_bottom 36 0.03037 0.03037 0.03037 8.26% MLMG::actualBottomSolve() 36 0.03036 0.03036 0.03036 8.26% MLCGSolver::bicgstab 36 0.03007 0.03007 0.03007 8.18% Castro::clean_state() 30 0.02598 0.02598 0.02598 7.07% FillPatchIterator::Initialize 20 0.02045 0.02045 0.02045 5.56% FillPatchSingleLevel 20 0.0197 0.0197 0.0197 5.36% StateDataPhysBCFunct::() 20 0.01768 0.01768 0.01768 4.81% MLCellLinOp::apply() 500 0.01525 0.01525 0.01525 4.15% MLMG::mgVcycle_down::0 36 0.01439 0.01439 0.01439 3.91% FabArray::FillBoundary() 1766 0.01394 0.01394 0.01394 3.79% MLPoisson::Fsmooth() 1440 0.01371 0.01371 0.01371 3.73% FillBoundary_nowait() 1766 0.01359 0.01359 0.01359 3.70% StateData::FillBoundary(geom) 160 0.01156 0.01156 0.01156 3.15% MLMG::mgVcycle_up::0 36 0.01089 0.01089 0.01089 2.96% Castro::initialize_do_advance() 5 0.01028 0.01028 0.01028 2.80% MLCellLinOp::correctionResidual() 216 0.009295 0.009295 0.009295 2.53% Castro::normalize_species() 30 0.009025 0.009025 0.009025 2.46% Castro::computeTemp() 30 0.008722 0.008722 0.008722 2.37% MLPoisson::define() 6 0.008619 0.008619 0.008619 2.34% amrex::Dot() 484 0.008453 0.008453 0.008453 2.30% MLMG:computeResOfCorrection() 180 0.008182 0.008182 0.008182 2.23% Castro::do_new_sources() 5 0.007791 0.007791 0.007791 2.12% Gravity::get_new_grav_vector() 5 0.007583 0.007583 0.007583 2.06% Castro::construct_old_gravity() 5 0.007376 0.007376 0.007376 2.01% Gravity::get_old_grav_vector() 5 0.007318 0.007318 0.007318 1.99% Castro::enforce_min_density() 30 0.006861 0.006861 0.006861 1.87% amrex::Copy() 463 0.006791 0.006791 0.006791 1.85% MLMG::mgVcycle_down::1 36 0.006625 0.006625 0.006625 1.80% FabArray::ParallelCopy() 380 0.006407 0.006407 0.006407 1.74% Castro::do_old_sources() 5 0.006318 0.006318 0.006318 1.72% FabArray::ParallelCopy_nowait() 380 0.006298 0.006298 0.006298 1.71% MLMG::mgVcycle_down::2 36 0.006177 0.006177 0.006177 1.68% FabArray::setVal() 537 0.006085 0.006085 0.006085 1.66% Castro::initialize_advance() 5 0.006052 0.006052 0.006052 1.65% MLMG::mgVcycle_down::3 36 0.006024 0.006024 0.006024 1.64% FabArray::norminf() 326 0.00597 0.00597 0.00597 1.62% MLMG::mgVcycle_down::4 36 0.005958 0.005958 0.005958 1.62% Castro::expand_state() 5 0.005902 0.005902 0.005902 1.61% MLCellLinOp::defineAuxData() 6 0.00585 0.00585 0.00585 1.59% MLCGSolver::ParallelAllReduce 659 0.005096 0.005096 0.005096 1.39% MLMG::addInterpCorrection() 180 0.005067 0.005067 0.005067 1.38% Castro::post_timestep() 5 0.004927 0.004927 0.004927 1.34% MLMG::mgVcycle_up::4 36 0.004848 0.004848 0.004848 1.32% MLMG::mgVcycle_up::1 36 0.004804 0.004804 0.004804 1.31% amrex::average_down 180 0.004731 0.004731 0.004731 1.29% MLMG::mgVcycle_up::2 36 0.004729 0.004729 0.004729 1.29% MLMG::mgVcycle_up::3 36 0.00463 0.00463 0.00463 1.26% MLPoisson::Fapply() 500 0.004289 0.004289 0.004289 1.17% Castro::estTimeStep() 10 0.004015 0.004015 0.004015 1.09% FabArray::Saxpy() 355 0.003531 0.003531 0.003531 0.96% Castro::post_restart() 1 0.003401 0.003401 0.003401 0.93% FabArray::Xpay() 361 0.0034 0.0034 0.0034 0.93% Gravity::multilevel_solve_for_new_phi() 1 0.003293 0.003293 0.003293 0.90% Gravity::actual_multilevel_solve() 1 0.003278 0.003278 0.003278 0.89% MLCellLinOp::solutionResidual() 42 0.00317 0.00317 0.00317 0.86% Castro::reset_internal_energy(MultiFab) 30 0.00295 0.00295 0.00295 0.80% Gravity::fill_multipole_BCs() 6 0.002903 0.002903 0.002903 0.79% MLMG::computeResidual() 36 0.002623 0.002623 0.002623 0.71% MLCellLinOp::defineBC() 6 0.002595 0.002595 0.002595 0.71% BndryData::define() 6 0.00248 0.00248 0.00248 0.67% MLMG::prepareForSolve() 6 0.002464 0.002464 0.002464 0.67% Castro::computeNewDt() 5 0.002187 0.002187 0.002187 0.59% Castro::construct_new_source() 25 0.001953 0.001953 0.001953 0.53% FabArray::LinComb() 242 0.001846 0.001846 0.001846 0.50% amrex::Add() 72 0.001835 0.001835 0.001835 0.50% Castro::construct_new_gravity_source() 5 0.001802 0.001802 0.001802 0.49% Castro::construct_old_source() 25 0.001531 0.001531 0.001531 0.42% Castro::construct_old_gravity_source() 5 0.001499 0.001499 0.001499 0.41% Castro::enforce_speed_limit() 30 0.001335 0.001335 0.001335 0.36% Castro::apply_source_to_state() 10 0.0009184 0.0009184 0.0009184 0.25% MLMG::ResNormInf() 42 0.0009052 0.0009052 0.0009052 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008935 0.0008935 0.0008935 0.24% FabArrayBase::getCPC() 632 0.0007637 0.0007637 0.0007637 0.21% MLMG::getGradSolution() 6 0.000748 0.000748 0.000748 0.20% MLCellLinOp::compGrad() 6 0.0007451 0.0007451 0.0007451 0.20% MLCellLinOp::setLevelBC() 6 0.0007228 0.0007228 0.0007228 0.20% Castro::reset_internal_energy(Fab) 240 0.000669 0.000669 0.000669 0.18% FabArray::mult() 22 0.0006406 0.0006406 0.0006406 0.17% FabArray::setDomainBndry() 20 0.0006313 0.0006313 0.0006313 0.17% MLPoisson::prepareForSolve() 6 0.0006015 0.0006015 0.0006015 0.16% Castro::check_for_nan() 10 0.0005953 0.0005953 0.0005953 0.16% MLCellLinOp::prepareForSolve() 6 0.0005906 0.0005906 0.0005906 0.16% MultiFab::contains_nan() 10 0.0005893 0.0005893 0.0005893 0.16% MLMG::computeMLResidual() 6 0.0005702 0.0005702 0.0005702 0.16% Gravity::update_max_rhs() 6 0.0004393 0.0004393 0.0004393 0.12% FabArrayBase::CPC::define() 244 0.0004141 0.0004141 0.0004141 0.11% Amr::InitAmr() 1 0.0004046 0.0004046 0.0004046 0.11% FabArrayBase::getFB() 1766 0.0003116 0.0003116 0.0003116 0.08% Castro::finalize_advance() 5 0.0002871 0.0002871 0.0002871 0.08% Gravity::swapTimeLevels() 5 0.0002235 0.0002235 0.0002235 0.06% MLLinOp::define() 6 0.0001604 0.0001604 0.0001604 0.04% MLLinOp::defineGrids() 6 0.0001544 0.0001544 0.0001544 0.04% MLMG::MLResNormInf() 6 0.0001498 0.0001498 0.0001498 0.04% Castro::buildMetrics() 1 0.0001475 0.0001475 0.0001475 0.04% MultiFab::max() 6 0.0001356 0.0001356 0.0001356 0.04% MLMG::MLRhsNormInf() 6 0.0001157 0.0001157 0.0001157 0.03% Castro::create_source_corrector() 5 0.0001121 0.0001121 0.0001121 0.03% FabArrayBase::FB::FB() 26 5.975e-05 5.975e-05 5.975e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.753e-05 2.753e-05 2.753e-05 0.01% Amr::writeSmallPlotFile() 1 2.573e-05 2.573e-05 2.573e-05 0.01% Castro::initMFs() 1 2.569e-05 2.569e-05 2.569e-05 0.01% Castro::swap_state_time_levels() 5 2.388e-05 2.388e-05 2.388e-05 0.01% makeSFC 30 2.028e-05 2.028e-05 2.028e-05 0.01% Castro::finalize_do_advance() 5 1.962e-05 1.962e-05 1.962e-05 0.01% DistributionMapping::Distribute() 31 9.277e-06 9.277e-06 9.277e-06 0.00% Amr::initSubcycle() 1 9.16e-06 9.16e-06 9.16e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.746e-06 4.746e-06 4.746e-06 0.00% Gravity::set_mass_offset() 6 2.163e-06 2.163e-06 2.163e-06 0.00% Castro::retry_advance_ctu() 5 1.672e-06 1.672e-06 1.672e-06 0.00% Castro::FluxRegCrseInit 5 1.498e-06 1.498e-06 1.498e-06 0.00% Castro::FluxRegFineAdd() 5 1.138e-06 1.138e-06 1.138e-06 0.00% MLLinOp::makeSubCommunicator() 6 9.92e-07 9.92e-07 9.92e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 226 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 128 MiB 692 MiB Castro::initMFs() 48 48 59 MiB 68 MiB StateData::restartDoit() 32 32 53 MiB 55 MiB Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB Castro::initialize_do_advance() 40 40 29 MiB 39 MiB FillPatchIterator::Initialize 160 160 1228 KiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 2488 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 14 MiB 15 MiB Castro::post_restart() 48 48 6682 KiB 14 MiB MLMG::prepareForSolve() 361 361 3603 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 206 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 211 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6677 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 11 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2305 B 2048 KiB Gravity::solve_for_phi() 40 40 659 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 18 KiB 2048 KiB BndryData::define() 576 576 331 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 212 KiB 671 KiB Castro::estTimeStep() 10 10 4085 B 480 KiB VisMF::Write(FabArray) 112 112 1329 B 320 KiB Castro::normalize_species() 30 30 7986 B 320 KiB amrex::average_down 469 469 1247 B 257 KiB MLMG::addInterpCorrection() 468 468 1180 B 257 KiB amrex::Dot() 592 592 3381 B 160 KiB FabArray::norminf() 398 398 2399 B 160 KiB Castro::do_advance_ctu() 5 5 596 B 160 KiB MultiFab::max() 6 6 56 B 160 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MultiFab::contains_nan() 10 10 31 B 20 KiB MLPoisson::Fsmooth() 60 60 3505 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 52 B 10 KiB FillBoundary_nowait() 336 336 315 B 9648 B MLCellLinOp::applyBC() 3892 3892 228 B 9344 B amrex::Copy() 56 56 5933 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B StateData::FillBoundary(geom) 960 960 57 B 2976 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLCGSolver::bicgstab 324 324 118 B 1472 B MLCellLinOp::defineBC() 36 36 372 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1453 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 118 KiB 8192 KiB VisMF::Write(FabArray) 120 120 259 KiB 3584 KiB VisMF::Read() 24 24 209 KiB 3000 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3505 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 52 B 10 KiB FillBoundary_nowait() 336 336 315 B 9648 B MLCellLinOp::applyBC() 1946 1946 226 B 9328 B amrex::Copy() 56 56 5933 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B Gravity::get_old_grav_vector() 3 3 2632 B 3072 B StateData::FillBoundary(geom) 960 960 57 B 2976 B Gravity::fill_multipole_BCs() 18 18 5 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::prepareForSolve() 7 7 538 B 1296 B amrex::average_down 37 37 221 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 24 B 400 B FabArray::norminf() 398 398 17 B 272 B Castro::estTimeStep() 10 10 0 B 32 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::do_advance_ctu() 5 5 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.04-9-g1e73fa8ec7c3) finalized