Initializing AMReX (24.04-21-g96f6d669d1bb)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.04-21-g96f6d669d1bb) initialized Starting run at 09:19:28 UTC on 2024-04-17. Successfully read inputs file ... Castro git describe: 24.04-15-g5cacc9476 AMReX git describe: 24.04-21-g96f6d669d Microphysics git describe: 24.04-9-g4ad225b7 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.045705455 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025242033 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.069383234 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.082089834 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.084987127 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.058872115 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.057727916 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.045738186 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.078192304 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.080994488 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.06600122 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059752309 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.078790153 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.080280758 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.025809017 seconds Ending run at 09:19:29 UTC on 2024-04-17. Run time = 0.995864289 Run time without initialization = 0.869309658 Average number of zones advanced per microsecond: 3.016 Average number of zones advanced per microsecond per rank: 3.016 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.9959 ... 0.9959 ... 0.9959 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.3275 0.3275 0.3275 32.89% VisMF::Write(FabArray) 11 0.1775 0.1775 0.1775 17.82% MLCellLinOp::applyBC() 4351 0.08137 0.08137 0.08137 8.17% Amr::checkPoint() 3 0.0427 0.0427 0.0427 4.29% MLPoisson::Fsmooth() 3280 0.03399 0.03399 0.03399 3.41% StateData::FillBoundary(geom) 328 0.03386 0.03386 0.03386 3.40% FillBoundary_nowait() 3941 0.03137 0.03137 0.03137 3.15% Castro::normalize_species() 62 0.02208 0.02208 0.02208 2.22% amrex::Dot() 1114 0.02172 0.02172 0.02172 2.18% FabArray::norminf() 1061 0.02011 0.02011 0.02011 2.02% Castro::computeTemp() 63 0.01593 0.01593 0.01593 1.60% FabArray::ParallelCopy_nowait() 861 0.01383 0.01383 0.01383 1.39% FabArray::setVal() 1062 0.01363 0.01363 0.01363 1.37% FabArray::Saxpy() 1370 0.01336 0.01336 0.01336 1.34% StateDataPhysBCFunct::() 41 0.01252 0.01252 0.01252 1.26% amrex::Copy() 472 0.01101 0.01101 0.01101 1.11% MLPoisson::Fapply() 1060 0.01046 0.01046 0.01046 1.05% Castro::enforce_min_density() 62 0.01044 0.01044 0.01044 1.05% MLCellLinOp::defineAuxData() 11 0.01032 0.01032 0.01032 1.04% Gravity::fill_multipole_BCs() 11 0.009758 0.009758 0.009758 0.98% FabArray::Xpay() 739 0.007932 0.007932 0.007932 0.80% MLMG::addInterpCorrection() 410 0.007096 0.007096 0.007096 0.71% amrex::average_down 410 0.006249 0.006249 0.006249 0.63% Castro::estTimeStep() 21 0.005708 0.005708 0.005708 0.57% Castro::reset_internal_energy(MultiFab) 63 0.004915 0.004915 0.004915 0.49% BndryData::define() 11 0.004037 0.004037 0.004037 0.41% amrex::Add() 82 0.003612 0.003612 0.003612 0.36% Castro::construct_new_gravity_source() 10 0.003606 0.003606 0.003606 0.36% Castro::construct_old_gravity_source() 10 0.002929 0.002929 0.002929 0.29% Castro::enforce_speed_limit() 62 0.002233 0.002233 0.002233 0.22% Amr::writePlotFile() 2 0.002123 0.002123 0.002123 0.21% check_for_negative_density() 10 0.002084 0.002084 0.002084 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001812 0.001812 0.001812 0.18% Castro::reset_internal_energy(Fab) 504 0.001772 0.001772 0.001772 0.18% MLCellLinOp::setLevelBC() 11 0.001629 0.001629 0.001629 0.16% Castro::initData() 1 0.001565 0.001565 0.001565 0.16% MLCGSolver::bicgstab 82 0.001554 0.001554 0.001554 0.16% Gravity::actual_solve_with_mlmg() 11 0.001544 0.001544 0.001544 0.15% FabArray::setDomainBndry() 41 0.001401 0.001401 0.001401 0.14% FabArray::mult() 43 0.001394 0.001394 0.001394 0.14% MLCellLinOp::prepareForSolve() 11 0.001316 0.001316 0.001316 0.13% MultiFab::contains_nan() 20 0.00128 0.00128 0.00128 0.13% MLCellLinOp::smooth() 1640 0.001109 0.001109 0.001109 0.11% MLCellLinOp::compGrad() 11 0.00108 0.00108 0.00108 0.11% MLMG::prepareForSolve() 11 0.0009393 0.0009393 0.0009393 0.09% FabArrayBase::getCPC() 1323 0.0007745 0.0007745 0.0007745 0.08% FabArray::FillBoundary() 3941 0.000746 0.000746 0.000746 0.07% Gravity::get_new_grav_vector() 11 0.0006119 0.0006119 0.0006119 0.06% Gravity::get_old_grav_vector() 10 0.0004889 0.0004889 0.0004889 0.05% AmrLevel::FillPatch() 41 0.0004191 0.0004191 0.0004191 0.04% MLCellLinOp::apply() 1060 0.000407 0.000407 0.000407 0.04% Amr::coarseTimeStep() 10 0.0003594 0.0003594 0.0003594 0.04% main() 1 0.0003095 0.0003095 0.0003095 0.03% MLCGSolver::ParallelAllReduce 1832 0.0003083 0.0003083 0.0003083 0.03% MLCellLinOp::defineBC() 11 0.0002592 0.0002592 0.0002592 0.03% FabArray::ParallelCopy() 861 0.0002507 0.0002507 0.0002507 0.03% FillPatchIterator::Initialize 41 0.0002176 0.0002176 0.0002176 0.02% Castro::construct_new_source() 50 0.0001996 0.0001996 0.0001996 0.02% MLMG::mgVcycle() 82 0.0001888 0.0001888 0.0001888 0.02% Castro::subcycle_advance_ctu() 10 0.0001844 0.0001844 0.0001844 0.02% MLCellLinOp::correctionResidual() 410 0.0001586 0.0001586 0.0001586 0.02% Amr::timeStep() 10 0.0001522 0.0001522 0.0001522 0.02% MLMG:computeResOfCorrection() 410 0.0001134 0.0001134 0.0001134 0.01% StateData::checkPoint() 12 0.0001122 0.0001122 0.0001122 0.01% Gravity::solve_for_phi() 10 0.0001022 0.0001022 0.0001022 0.01% MLMG::actualBottomSolve() 82 8.378e-05 8.378e-05 8.378e-05 0.01% MLMG::mgVcycle_down::0 82 7.98e-05 7.98e-05 7.98e-05 0.01% Castro::advance() 10 7.687e-05 7.687e-05 7.687e-05 0.01% Castro::initialize_advance() 10 7.644e-05 7.644e-05 7.644e-05 0.01% MLMG::mgVcycle_down::1 82 7.465e-05 7.465e-05 7.465e-05 0.01% MLMG::mgVcycle_down::2 82 7.284e-05 7.284e-05 7.284e-05 0.01% MLMG::mgVcycle_down::4 82 6.879e-05 6.879e-05 6.879e-05 0.01% MLMG::solve() 11 6.875e-05 6.875e-05 6.875e-05 0.01% Castro::clean_state() 62 6.83e-05 6.83e-05 6.83e-05 0.01% MLMG::mgVcycle_down::3 82 6.664e-05 6.664e-05 6.664e-05 0.01% AmrLevel::checkPoint() 3 6.255e-05 6.255e-05 6.255e-05 0.01% Castro::initialize_do_advance() 10 5.903e-05 5.903e-05 5.903e-05 0.01% MLMG::oneIter() 82 5.578e-05 5.578e-05 5.578e-05 0.01% FillPatchIterator::FillFromLevel0() 41 5.426e-05 5.426e-05 5.426e-05 0.01% Castro::finalize_do_advance() 10 4.872e-05 4.872e-05 4.872e-05 0.00% MLMG::mgVcycle_up::0 82 4.843e-05 4.843e-05 4.843e-05 0.00% MLMG::mgVcycle_up::4 82 4.813e-05 4.813e-05 4.813e-05 0.00% Castro::do_advance_ctu() 10 4.75e-05 4.75e-05 4.75e-05 0.00% MLMG::mgVcycle_up::1 82 4.446e-05 4.446e-05 4.446e-05 0.00% MLCellLinOp::solutionResidual() 93 4.416e-05 4.416e-05 4.416e-05 0.00% MLMG::mgVcycle_up::3 82 4.355e-05 4.355e-05 4.355e-05 0.00% MLMG::mgVcycle_up::2 82 4.295e-05 4.295e-05 4.295e-05 0.00% FillPatchSingleLevel 41 3.807e-05 3.807e-05 3.807e-05 0.00% MLMG::computeResidual() 82 3.672e-05 3.672e-05 3.672e-05 0.00% MLMG::ResNormInf() 93 3.228e-05 3.228e-05 3.228e-05 0.00% Amr::defBaseLevel() 1 3.091e-05 3.091e-05 3.091e-05 0.00% MLMG::mgVcycle_bottom 82 3.064e-05 3.064e-05 3.064e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.725e-05 2.725e-05 2.725e-05 0.00% Castro::construct_new_gravity() 10 2.574e-05 2.574e-05 2.574e-05 0.00% Castro::do_new_sources() 10 2.482e-05 2.482e-05 2.482e-05 0.00% MLPoisson::define() 11 2.387e-05 2.387e-05 2.387e-05 0.00% Castro::do_old_sources() 10 1.984e-05 1.984e-05 1.984e-05 0.00% Amr::FinalizeInit() 1 1.899e-05 1.899e-05 1.899e-05 0.00% Castro::construct_old_source() 50 1.889e-05 1.889e-05 1.889e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.672e-05 1.672e-05 1.672e-05 0.00% Gravity::actual_multilevel_solve() 1 1.438e-05 1.438e-05 1.438e-05 0.00% Castro::check_for_nan() 20 1.179e-05 1.179e-05 1.179e-05 0.00% Castro::apply_source_to_state() 20 1.167e-05 1.167e-05 1.167e-05 0.00% MLMG::computeMLResidual() 11 1.069e-05 1.069e-05 1.069e-05 0.00% Castro::construct_old_gravity() 10 9.781e-06 9.781e-06 9.781e-06 0.00% Castro::post_timestep() 10 9.349e-06 9.349e-06 9.349e-06 0.00% Amr::InitializeInit() 1 6.398e-06 6.398e-06 6.398e-06 0.00% Castro::expand_state() 10 6.287e-06 6.287e-06 6.287e-06 0.00% Castro::computeNewDt() 9 6.027e-06 6.027e-06 6.027e-06 0.00% MLPoisson::prepareForSolve() 11 5.917e-06 5.917e-06 5.917e-06 0.00% MLMG::getGradSolution() 11 5.703e-06 5.703e-06 5.703e-06 0.00% Castro::post_init() 1 4.886e-06 4.886e-06 4.886e-06 0.00% Amr::init() 1 2.5e-06 2.5e-06 2.5e-06 0.00% Amr::initialInit() 1 1.089e-06 1.089e-06 1.089e-06 0.00% Other 4753 0.003313 0.003313 0.003313 0.33% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9959 0.9959 0.9959 100.00% Amr::coarseTimeStep() 10 0.8432 0.8432 0.8432 84.67% Amr::timeStep() 10 0.7138 0.7138 0.7138 71.67% Castro::advance() 10 0.7036 0.7036 0.7036 70.65% Castro::subcycle_advance_ctu() 10 0.6901 0.6901 0.6901 69.29% Castro::do_advance_ctu() 10 0.6899 0.6899 0.6899 69.27% Castro::construct_ctu_hydro_source() 10 0.3393 0.3393 0.3393 34.07% Gravity::solve_phi_with_mlmg() 11 0.2976 0.2976 0.2976 29.88% Gravity::actual_solve_with_mlmg() 11 0.2873 0.2873 0.2873 28.85% Castro::construct_new_gravity() 10 0.2759 0.2759 0.2759 27.70% MLMG::solve() 11 0.2652 0.2652 0.2652 26.63% Gravity::solve_for_phi() 10 0.2526 0.2526 0.2526 25.37% MLMG::oneIter() 82 0.2497 0.2497 0.2497 25.07% MLMG::mgVcycle() 82 0.246 0.246 0.246 24.71% VisMF::Write(FabArray) 11 0.1775 0.1775 0.1775 17.82% Amr::checkPoint() 3 0.1718 0.1718 0.1718 17.26% AmrLevel::checkPoint() 3 0.1291 0.1291 0.1291 12.97% StateData::checkPoint() 12 0.1291 0.1291 0.1291 12.96% Amr::init() 1 0.1258 0.1258 0.1258 12.63% MLCellLinOp::smooth() 1640 0.123 0.123 0.123 12.35% MLCellLinOp::applyBC() 4351 0.1142 0.1142 0.1142 11.46% MLMG::mgVcycle_bottom 82 0.07363 0.07363 0.07363 7.39% MLMG::actualBottomSolve() 82 0.07359 0.07359 0.07359 7.39% MLCGSolver::bicgstab 82 0.07277 0.07277 0.07277 7.31% AmrLevel::FillPatch() 41 0.05658 0.05658 0.05658 5.68% Castro::clean_state() 62 0.05626 0.05626 0.05626 5.65% Amr::initialInit() 1 0.05475 0.05475 0.05475 5.50% FillPatchIterator::Initialize 41 0.05221 0.05221 0.05221 5.24% Amr::writePlotFile() 2 0.05119 0.05119 0.05119 5.14% FillPatchIterator::FillFromLevel0() 41 0.05059 0.05059 0.05059 5.08% FillPatchSingleLevel 41 0.05054 0.05054 0.05054 5.07% Amr::FinalizeInit() 1 0.04932 0.04932 0.04932 4.95% Castro::post_init() 1 0.04789 0.04789 0.04789 4.81% StateDataPhysBCFunct::() 41 0.04638 0.04638 0.04638 4.66% Gravity::multilevel_solve_for_new_phi() 1 0.04538 0.04538 0.04538 4.56% Gravity::actual_multilevel_solve() 1 0.04537 0.04537 0.04537 4.56% MLCellLinOp::apply() 1060 0.03662 0.03662 0.03662 3.68% MLMG::mgVcycle_down::0 82 0.03515 0.03515 0.03515 3.53% MLPoisson::Fsmooth() 3280 0.03399 0.03399 0.03399 3.41% StateData::FillBoundary(geom) 328 0.03386 0.03386 0.03386 3.40% FabArray::FillBoundary() 3941 0.03281 0.03281 0.03281 3.29% FillBoundary_nowait() 3941 0.03207 0.03207 0.03207 3.22% MLMG::mgVcycle_up::0 82 0.02663 0.02663 0.02663 2.67% Gravity::get_new_grav_vector() 11 0.02534 0.02534 0.02534 2.54% Castro::computeTemp() 63 0.02262 0.02262 0.02262 2.27% Castro::normalize_species() 62 0.02208 0.02208 0.02208 2.22% amrex::Dot() 1114 0.02172 0.02172 0.02172 2.18% Castro::initialize_do_advance() 10 0.02142 0.02142 0.02142 2.15% MLMG:computeResOfCorrection() 410 0.02061 0.02061 0.02061 2.07% MLCellLinOp::correctionResidual() 410 0.0205 0.0205 0.0205 2.06% FabArray::norminf() 1061 0.02011 0.02011 0.02011 2.02% Castro::do_old_sources() 10 0.01937 0.01937 0.01937 1.95% MLPoisson::define() 11 0.01733 0.01733 0.01733 1.74% MLMG::mgVcycle_down::1 82 0.01679 0.01679 0.01679 1.69% Castro::construct_old_gravity() 10 0.01595 0.01595 0.01595 1.60% Gravity::get_old_grav_vector() 10 0.01594 0.01594 0.01594 1.60% MLMG::mgVcycle_down::2 82 0.01562 0.01562 0.01562 1.57% MLMG::mgVcycle_down::3 82 0.01523 0.01523 0.01523 1.53% MLMG::mgVcycle_down::4 82 0.01512 0.01512 0.01512 1.52% FabArray::ParallelCopy() 861 0.01488 0.01488 0.01488 1.49% FabArray::ParallelCopy_nowait() 861 0.01462 0.01462 0.01462 1.47% Castro::do_new_sources() 10 0.0144 0.0144 0.0144 1.45% FabArray::setVal() 1062 0.01363 0.01363 0.01363 1.37% FabArray::Saxpy() 1370 0.01336 0.01336 0.01336 1.34% MLCGSolver::ParallelAllReduce 1832 0.01299 0.01299 0.01299 1.30% Castro::initialize_advance() 10 0.01286 0.01286 0.01286 1.29% MLMG::addInterpCorrection() 410 0.01244 0.01244 0.01244 1.25% Castro::expand_state() 10 0.01239 0.01239 0.01239 1.24% MLMG::mgVcycle_up::1 82 0.01213 0.01213 0.01213 1.22% MLMG::mgVcycle_up::4 82 0.01204 0.01204 0.01204 1.21% MLMG::mgVcycle_up::2 82 0.01185 0.01185 0.01185 1.19% MLCellLinOp::defineAuxData() 11 0.01174 0.01174 0.01174 1.18% MLMG::mgVcycle_up::3 82 0.01167 0.01167 0.01167 1.17% amrex::average_down 410 0.01166 0.01166 0.01166 1.17% amrex::Copy() 472 0.01101 0.01101 0.01101 1.11% MLPoisson::Fapply() 1060 0.01046 0.01046 0.01046 1.05% Castro::enforce_min_density() 62 0.01044 0.01044 0.01044 1.05% Castro::post_timestep() 10 0.009998 0.009998 0.009998 1.00% Gravity::fill_multipole_BCs() 11 0.009997 0.009997 0.009997 1.00% FabArray::Xpay() 739 0.007932 0.007932 0.007932 0.80% MLCellLinOp::solutionResidual() 93 0.007822 0.007822 0.007822 0.79% Castro::reset_internal_energy(MultiFab) 63 0.006688 0.006688 0.006688 0.67% MLMG::computeResidual() 82 0.006564 0.006564 0.006564 0.66% Castro::estTimeStep() 21 0.005708 0.005708 0.005708 0.57% Amr::InitializeInit() 1 0.005428 0.005428 0.005428 0.54% Amr::defBaseLevel() 1 0.005421 0.005421 0.005421 0.54% MLCellLinOp::defineBC() 11 0.005326 0.005326 0.005326 0.53% MLMG::prepareForSolve() 11 0.005173 0.005173 0.005173 0.52% BndryData::define() 11 0.005066 0.005066 0.005066 0.51% Castro::initData() 1 0.004721 0.004721 0.004721 0.47% Castro::construct_new_source() 50 0.003806 0.003806 0.003806 0.38% amrex::Add() 82 0.003612 0.003612 0.003612 0.36% Castro::construct_new_gravity_source() 10 0.003606 0.003606 0.003606 0.36% Castro::construct_old_source() 50 0.002948 0.002948 0.002948 0.30% Castro::construct_old_gravity_source() 10 0.002929 0.002929 0.002929 0.29% Castro::finalize_do_advance() 10 0.002498 0.002498 0.002498 0.25% Castro::computeNewDt() 9 0.002295 0.002295 0.002295 0.23% Castro::enforce_speed_limit() 62 0.002233 0.002233 0.002233 0.22% MLMG::ResNormInf() 93 0.002182 0.002182 0.002182 0.22% check_for_negative_density() 10 0.002084 0.002084 0.002084 0.21% Castro::apply_source_to_state() 20 0.001885 0.001885 0.001885 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001812 0.001812 0.001812 0.18% Castro::reset_internal_energy(Fab) 504 0.001772 0.001772 0.001772 0.18% MLCellLinOp::setLevelBC() 11 0.001629 0.001629 0.001629 0.16% MLMG::getGradSolution() 11 0.001597 0.001597 0.001597 0.16% MLCellLinOp::compGrad() 11 0.001592 0.001592 0.001592 0.16% FabArrayBase::getCPC() 1323 0.00143 0.00143 0.00143 0.14% FabArray::setDomainBndry() 41 0.001401 0.001401 0.001401 0.14% FabArray::mult() 43 0.001394 0.001394 0.001394 0.14% MLPoisson::prepareForSolve() 11 0.001322 0.001322 0.001322 0.13% MLCellLinOp::prepareForSolve() 11 0.001316 0.001316 0.001316 0.13% MLMG::computeMLResidual() 11 0.001306 0.001306 0.001306 0.13% Castro::check_for_nan() 20 0.001292 0.001292 0.001292 0.13% MultiFab::contains_nan() 20 0.00128 0.00128 0.00128 0.13% Other 4753 0.009176 0.009176 0.009176 0.92% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 5266 KiB 9037 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1043 KiB 39 MiB Castro::initialize_do_advance() 80 80 27 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1434 KiB 28 MiB Castro::initialize_advance() 80 80 16 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7617 KiB 14 MiB MLMG::prepareForSolve() 660 660 3276 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 263 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 163 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7520 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 16 KiB 2053 KiB Gravity::update_max_rhs() 88 88 1960 B 2048 KiB Gravity::solve_for_phi() 80 80 519 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 93 KiB 2048 KiB BndryData::define() 1056 1056 299 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 190 KiB 671 KiB Castro::estTimeStep() 21 21 2793 B 480 KiB VisMF::Write(FabArray) 656 656 3106 B 320 KiB Castro::normalize_species() 62 62 7227 B 320 KiB amrex::average_down 1067 1067 1527 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1079 B 257 KiB amrex::Dot() 1360 1360 3226 B 160 KiB FabArray::norminf() 1143 1143 3123 B 160 KiB check_for_negative_density() 10 10 336 B 160 KiB Castro::initData() 1 1 47 B 160 KiB MultiFab::max() 11 11 53 B 160 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MultiFab::contains_nan() 20 20 25 B 20 KiB MLPoisson::Fsmooth() 132 132 3198 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 40 B 10 KiB FillBoundary_nowait() 760 760 274 B 9648 B MLCellLinOp::applyBC() 8702 8702 204 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3905 B 6144 B StateData::FillBoundary(geom) 1992 1992 39 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 337 B 1248 B MLCGSolver::bicgstab 410 410 88 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 526 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 37 KiB 8192 KiB VisMF::Write(FabArray) 744 744 390 KiB 3584 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MLPoisson::Fsmooth() 132 132 3198 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 40 B 10 KiB FillBoundary_nowait() 760 760 274 B 9648 B MLCellLinOp::applyBC() 4351 4351 202 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3905 B 6144 B Gravity::get_new_grav_vector() 3 3 2905 B 3072 B StateData::FillBoundary(geom) 1992 1992 40 B 3024 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B amrex::average_down 83 83 617 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 273 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 23 B 400 B FabArray::norminf() 1143 1143 9 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2105 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.04-21-g96f6d669d1bb) finalized Initializing AMReX (24.04-21-g96f6d669d1bb)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.04-21-g96f6d669d1bb) initialized Starting run at 09:19:29 UTC on 2024-04-17. Successfully read inputs file ... Castro git describe: 24.04-15-g5cacc9476 AMReX git describe: 24.04-21-g96f6d669d Microphysics git describe: 24.04-9-g4ad225b7 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.525403466 Restart time = 0.07180087 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.070020208 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050243866 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.075571144 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.075401988 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.055858274 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.027921057 seconds Ending run at 09:19:30 UTC on 2024-04-17. Run time = 0.427876742 Run time without initialization = 0.355442491 Average number of zones advanced per microsecond: 3.688 Average number of zones advanced per microsecond per rank: 3.688 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.4279 ... 0.4279 ... 0.4279 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1303 0.1303 0.1303 30.44% VisMF::Read() 3 0.06052 0.06052 0.06052 14.14% MLCellLinOp::applyBC() 1910 0.03609 0.03609 0.03609 8.43% VisMF::Write(FabArray) 1 0.0252 0.0252 0.0252 5.89% MLPoisson::Fsmooth() 1440 0.01501 0.01501 0.01501 3.51% StateData::FillBoundary(geom) 160 0.01317 0.01317 0.01317 3.08% FillBoundary_nowait() 1730 0.01302 0.01302 0.01302 3.04% Castro::normalize_species() 30 0.009779 0.009779 0.009779 2.29% amrex::Dot() 484 0.009371 0.009371 0.009371 2.19% FabArray::norminf() 465 0.008763 0.008763 0.008763 2.05% Castro::computeTemp() 30 0.008231 0.008231 0.008231 1.92% Castro::enforce_min_density() 30 0.0071 0.0071 0.0071 1.66% FabArray::setVal() 501 0.006747 0.006747 0.006747 1.58% FabArray::ParallelCopy_nowait() 380 0.006295 0.006295 0.006295 1.47% FabArray::Saxpy() 597 0.005994 0.005994 0.005994 1.40% MLCellLinOp::defineAuxData() 6 0.005766 0.005766 0.005766 1.35% amrex::Copy() 221 0.00553 0.00553 0.00553 1.29% StateDataPhysBCFunct::() 20 0.005451 0.005451 0.005451 1.27% Gravity::fill_multipole_BCs() 6 0.005427 0.005427 0.005427 1.27% Amr::restart() 1 0.004858 0.004858 0.004858 1.14% MLPoisson::Fapply() 464 0.004532 0.004532 0.004532 1.06% FabArray::Xpay() 325 0.003559 0.003559 0.003559 0.83% MLMG::addInterpCorrection() 180 0.003142 0.003142 0.003142 0.73% amrex::average_down 180 0.002819 0.002819 0.002819 0.66% Castro::estTimeStep() 10 0.00272 0.00272 0.00272 0.64% Amr::writePlotFile() 1 0.002547 0.002547 0.002547 0.60% BndryData::define() 6 0.00224 0.00224 0.00224 0.52% Castro::reset_internal_energy(MultiFab) 30 0.002043 0.002043 0.002043 0.48% Castro::construct_new_gravity_source() 5 0.001808 0.001808 0.001808 0.42% Castro::construct_old_gravity_source() 5 0.001555 0.001555 0.001555 0.36% amrex::Add() 36 0.001548 0.001548 0.001548 0.36% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000999 0.000999 0.000999 0.23% check_for_negative_density() 5 0.000942 0.000942 0.000942 0.22% MLCellLinOp::setLevelBC() 6 0.0008934 0.0008934 0.0008934 0.21% Gravity::actual_solve_with_mlmg() 6 0.0008352 0.0008352 0.0008352 0.20% Castro::reset_internal_energy(Fab) 240 0.0008248 0.0008248 0.0008248 0.19% MLCellLinOp::prepareForSolve() 6 0.0008084 0.0008084 0.0008084 0.19% Castro::enforce_speed_limit() 30 0.0007258 0.0007258 0.0007258 0.17% MLCGSolver::bicgstab 36 0.0007154 0.0007154 0.0007154 0.17% FabArray::setDomainBndry() 20 0.0007075 0.0007075 0.0007075 0.17% FabArray::mult() 22 0.0006936 0.0006936 0.0006936 0.16% MLMG::prepareForSolve() 6 0.0006389 0.0006389 0.0006389 0.15% MLCellLinOp::compGrad() 6 0.0006069 0.0006069 0.0006069 0.14% MLCellLinOp::smooth() 720 0.00051 0.00051 0.00051 0.12% FabArrayBase::getCPC() 632 0.0003947 0.0003947 0.0003947 0.09% FabArray::FillBoundary() 1730 0.0003339 0.0003339 0.0003339 0.08% Gravity::get_old_grav_vector() 5 0.0003228 0.0003228 0.0003228 0.08% Gravity::get_new_grav_vector() 5 0.0002628 0.0002628 0.0002628 0.06% main() 1 0.0002531 0.0002531 0.0002531 0.06% AmrLevel::FillPatch() 20 0.0002007 0.0002007 0.0002007 0.05% MLCellLinOp::apply() 464 0.0001854 0.0001854 0.0001854 0.04% Amr::coarseTimeStep() 5 0.0001785 0.0001785 0.0001785 0.04% MLCellLinOp::defineBC() 6 0.000159 0.000159 0.000159 0.04% MLCGSolver::ParallelAllReduce 798 0.0001329 0.0001329 0.0001329 0.03% FabArray::ParallelCopy() 380 0.0001114 0.0001114 0.0001114 0.03% FillPatchIterator::Initialize 20 0.0001031 0.0001031 0.0001031 0.02% Castro::subcycle_advance_ctu() 5 9.326e-05 9.326e-05 9.326e-05 0.02% MLMG::mgVcycle() 36 8.335e-05 8.335e-05 8.335e-05 0.02% Amr::timeStep() 5 8.179e-05 8.179e-05 8.179e-05 0.02% MLCellLinOp::correctionResidual() 180 7.525e-05 7.525e-05 7.525e-05 0.02% AmrLevel::restart() 1 7.317e-05 7.317e-05 7.317e-05 0.02% Castro::initialize_do_advance() 5 6.811e-05 6.811e-05 6.811e-05 0.02% StateData::restartDoit() 4 6.538e-05 6.538e-05 6.538e-05 0.02% Gravity::update_max_rhs() 6 6.325e-05 6.325e-05 6.325e-05 0.01% Castro::post_timestep() 5 6.214e-05 6.214e-05 6.214e-05 0.01% Gravity::solve_for_phi() 5 5.102e-05 5.102e-05 5.102e-05 0.01% MLMG:computeResOfCorrection() 180 5.038e-05 5.038e-05 5.038e-05 0.01% Castro::do_advance_ctu() 5 4.658e-05 4.658e-05 4.658e-05 0.01% Castro::construct_new_source() 25 4.551e-05 4.551e-05 4.551e-05 0.01% Castro::finalize_do_advance() 5 4.295e-05 4.295e-05 4.295e-05 0.01% MLMG::actualBottomSolve() 36 3.889e-05 3.889e-05 3.889e-05 0.01% MLMG::mgVcycle_down::0 36 3.719e-05 3.719e-05 3.719e-05 0.01% Castro::initialize_advance() 5 3.532e-05 3.532e-05 3.532e-05 0.01% Castro::advance() 5 3.529e-05 3.529e-05 3.529e-05 0.01% Castro::clean_state() 30 3.524e-05 3.524e-05 3.524e-05 0.01% MLMG::mgVcycle_down::1 36 3.436e-05 3.436e-05 3.436e-05 0.01% MLMG::solve() 6 3.346e-05 3.346e-05 3.346e-05 0.01% MLMG::mgVcycle_down::2 36 3.242e-05 3.242e-05 3.242e-05 0.01% MLMG::mgVcycle_down::4 36 3.13e-05 3.13e-05 3.13e-05 0.01% MLMG::mgVcycle_down::3 36 2.999e-05 2.999e-05 2.999e-05 0.01% Castro::do_new_sources() 5 2.94e-05 2.94e-05 2.94e-05 0.01% Castro::construct_old_source() 25 2.8e-05 2.8e-05 2.8e-05 0.01% MLMG::oneIter() 36 2.543e-05 2.543e-05 2.543e-05 0.01% MLMG::mgVcycle_up::4 36 2.448e-05 2.448e-05 2.448e-05 0.01% Castro::post_restart() 1 2.184e-05 2.184e-05 2.184e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.178e-05 2.178e-05 2.178e-05 0.01% MLCellLinOp::solutionResidual() 42 2.17e-05 2.17e-05 2.17e-05 0.01% MLMG::mgVcycle_up::3 36 2.129e-05 2.129e-05 2.129e-05 0.00% MLMG::mgVcycle_up::0 36 2.074e-05 2.074e-05 2.074e-05 0.00% MLMG::mgVcycle_up::2 36 1.918e-05 1.918e-05 1.918e-05 0.00% MLMG::mgVcycle_up::1 36 1.837e-05 1.837e-05 1.837e-05 0.00% MLPoisson::define() 6 1.751e-05 1.751e-05 1.751e-05 0.00% Gravity::actual_multilevel_solve() 1 1.68e-05 1.68e-05 1.68e-05 0.00% FillPatchSingleLevel 20 1.663e-05 1.663e-05 1.663e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.655e-05 1.655e-05 1.655e-05 0.00% MLMG::ResNormInf() 42 1.605e-05 1.605e-05 1.605e-05 0.00% MLMG::computeMLResidual() 6 1.483e-05 1.483e-05 1.483e-05 0.00% MLMG::computeResidual() 36 1.43e-05 1.43e-05 1.43e-05 0.00% MLMG::mgVcycle_bottom 36 1.4e-05 1.4e-05 1.4e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.384e-05 1.384e-05 1.384e-05 0.00% Castro::construct_new_gravity() 5 1.241e-05 1.241e-05 1.241e-05 0.00% Castro::do_old_sources() 5 1.001e-05 1.001e-05 1.001e-05 0.00% Castro::expand_state() 5 8.476e-06 8.476e-06 8.476e-06 0.00% Castro::check_for_nan() 10 6.679e-06 6.679e-06 6.679e-06 0.00% Castro::apply_source_to_state() 10 5.82e-06 5.82e-06 5.82e-06 0.00% Castro::construct_old_gravity() 5 5.264e-06 5.264e-06 5.264e-06 0.00% MLPoisson::prepareForSolve() 6 3.746e-06 3.746e-06 3.746e-06 0.00% Castro::computeNewDt() 5 3.44e-06 3.44e-06 3.44e-06 0.00% MLMG::getGradSolution() 6 3.309e-06 3.309e-06 3.309e-06 0.00% Amr::init() 1 7.7e-07 7.7e-07 7.7e-07 0.00% Other 2160 0.00255 0.00255 0.00255 0.60% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4279 0.4279 0.4279 99.99% Amr::coarseTimeStep() 5 0.3273 0.3273 0.3273 76.48% Amr::timeStep() 5 0.3254 0.3254 0.3254 76.05% Castro::advance() 5 0.32 0.32 0.32 74.77% Castro::subcycle_advance_ctu() 5 0.3123 0.3123 0.3123 72.99% Castro::do_advance_ctu() 5 0.3122 0.3122 0.3122 72.97% Castro::construct_new_gravity() 5 0.1379 0.1379 0.1379 32.23% Castro::construct_ctu_hydro_source() 5 0.1362 0.1362 0.1362 31.83% Gravity::solve_phi_with_mlmg() 6 0.1351 0.1351 0.1351 31.56% Gravity::solve_for_phi() 5 0.1298 0.1298 0.1298 30.33% Gravity::actual_solve_with_mlmg() 6 0.1294 0.1294 0.1294 30.24% MLMG::solve() 6 0.117 0.117 0.117 27.34% MLMG::oneIter() 36 0.109 0.109 0.109 25.47% MLMG::mgVcycle() 36 0.1074 0.1074 0.1074 25.11% Amr::init() 1 0.07184 0.07184 0.07184 16.79% Amr::restart() 1 0.07184 0.07184 0.07184 16.79% AmrLevel::restart() 1 0.06089 0.06089 0.06089 14.23% StateData::restartDoit() 4 0.06081 0.06081 0.06081 14.21% VisMF::Read() 3 0.06052 0.06052 0.06052 14.14% MLCellLinOp::smooth() 720 0.05349 0.05349 0.05349 12.50% MLCellLinOp::applyBC() 1910 0.04977 0.04977 0.04977 11.63% MLMG::mgVcycle_bottom 36 0.0321 0.0321 0.0321 7.50% MLMG::actualBottomSolve() 36 0.03209 0.03209 0.03209 7.50% MLCGSolver::bicgstab 36 0.03172 0.03172 0.03172 7.41% Castro::clean_state() 30 0.02874 0.02874 0.02874 6.72% Amr::writePlotFile() 1 0.02801 0.02801 0.02801 6.55% VisMF::Write(FabArray) 1 0.0252 0.0252 0.0252 5.89% AmrLevel::FillPatch() 20 0.02364 0.02364 0.02364 5.53% FillPatchIterator::Initialize 20 0.02152 0.02152 0.02152 5.03% FillPatchIterator::FillFromLevel0() 20 0.02071 0.02071 0.02071 4.84% FillPatchSingleLevel 20 0.02069 0.02069 0.02069 4.84% StateDataPhysBCFunct::() 20 0.01863 0.01863 0.01863 4.35% MLCellLinOp::apply() 464 0.01622 0.01622 0.01622 3.79% MLMG::mgVcycle_down::0 36 0.01532 0.01532 0.01532 3.58% MLPoisson::Fsmooth() 1440 0.01501 0.01501 0.01501 3.51% FabArray::FillBoundary() 1730 0.01368 0.01368 0.01368 3.20% FillBoundary_nowait() 1730 0.01335 0.01335 0.01335 3.12% StateData::FillBoundary(geom) 160 0.01317 0.01317 0.01317 3.08% Castro::initialize_do_advance() 5 0.01196 0.01196 0.01196 2.79% MLMG::mgVcycle_up::0 36 0.01132 0.01132 0.01132 2.64% Castro::computeTemp() 30 0.0111 0.0111 0.0111 2.59% Castro::do_old_sources() 5 0.01007 0.01007 0.01007 2.35% Castro::normalize_species() 30 0.009779 0.009779 0.009779 2.29% MLPoisson::define() 6 0.009759 0.009759 0.009759 2.28% amrex::Dot() 484 0.009371 0.009371 0.009371 2.19% MLMG:computeResOfCorrection() 180 0.009016 0.009016 0.009016 2.11% MLCellLinOp::correctionResidual() 180 0.008965 0.008965 0.008965 2.10% FabArray::norminf() 465 0.008763 0.008763 0.008763 2.05% Gravity::get_new_grav_vector() 5 0.008014 0.008014 0.008014 1.87% Castro::construct_old_gravity() 5 0.007884 0.007884 0.007884 1.84% Gravity::get_old_grav_vector() 5 0.007879 0.007879 0.007879 1.84% MLMG::mgVcycle_down::1 36 0.007514 0.007514 0.007514 1.76% Castro::initialize_advance() 5 0.007293 0.007293 0.007293 1.70% Castro::enforce_min_density() 30 0.0071 0.0071 0.0071 1.66% MLMG::mgVcycle_down::2 36 0.006861 0.006861 0.006861 1.60% FabArray::ParallelCopy() 380 0.00682 0.00682 0.00682 1.59% FabArray::setVal() 501 0.006747 0.006747 0.006747 1.58% FabArray::ParallelCopy_nowait() 380 0.006709 0.006709 0.006709 1.57% MLMG::mgVcycle_down::3 36 0.006705 0.006705 0.006705 1.57% MLMG::mgVcycle_down::4 36 0.006662 0.006662 0.006662 1.56% MLCellLinOp::defineAuxData() 6 0.006605 0.006605 0.006605 1.54% Castro::do_new_sources() 5 0.00659 0.00659 0.00659 1.54% Castro::expand_state() 5 0.006309 0.006309 0.006309 1.47% FabArray::Saxpy() 597 0.005994 0.005994 0.005994 1.40% Castro::post_restart() 1 0.005911 0.005911 0.005911 1.38% MLCGSolver::ParallelAllReduce 798 0.005654 0.005654 0.005654 1.32% Gravity::fill_multipole_BCs() 6 0.005553 0.005553 0.005553 1.30% amrex::Copy() 221 0.00553 0.00553 0.00553 1.29% Gravity::multilevel_solve_for_new_phi() 1 0.005529 0.005529 0.005529 1.29% Gravity::actual_multilevel_solve() 1 0.005513 0.005513 0.005513 1.29% MLMG::addInterpCorrection() 180 0.0055 0.0055 0.0055 1.29% Castro::post_timestep() 5 0.005376 0.005376 0.005376 1.26% MLMG::mgVcycle_up::4 36 0.005367 0.005367 0.005367 1.25% MLMG::mgVcycle_up::1 36 0.005251 0.005251 0.005251 1.23% amrex::average_down 180 0.005233 0.005233 0.005233 1.22% MLMG::mgVcycle_up::2 36 0.005135 0.005135 0.005135 1.20% MLMG::mgVcycle_up::3 36 0.00512 0.00512 0.00512 1.20% MLPoisson::Fapply() 464 0.004532 0.004532 0.004532 1.06% MLCellLinOp::solutionResidual() 42 0.003697 0.003697 0.003697 0.86% FabArray::Xpay() 325 0.003559 0.003559 0.003559 0.83% MLMG::prepareForSolve() 6 0.003101 0.003101 0.003101 0.72% MLCellLinOp::defineBC() 6 0.002997 0.002997 0.002997 0.70% MLMG::computeResidual() 36 0.002872 0.002872 0.002872 0.67% Castro::reset_internal_energy(MultiFab) 30 0.002868 0.002868 0.002868 0.67% BndryData::define() 6 0.002838 0.002838 0.002838 0.66% Castro::estTimeStep() 10 0.00272 0.00272 0.00272 0.64% Castro::construct_new_source() 25 0.001853 0.001853 0.001853 0.43% Castro::construct_new_gravity_source() 5 0.001808 0.001808 0.001808 0.42% Castro::computeNewDt() 5 0.001672 0.001672 0.001672 0.39% Castro::construct_old_source() 25 0.001583 0.001583 0.001583 0.37% Castro::construct_old_gravity_source() 5 0.001555 0.001555 0.001555 0.36% amrex::Add() 36 0.001548 0.001548 0.001548 0.36% Castro::finalize_do_advance() 5 0.001095 0.001095 0.001095 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000999 0.000999 0.000999 0.23% MLMG::ResNormInf() 42 0.0009912 0.0009912 0.0009912 0.23% Castro::apply_source_to_state() 10 0.0009671 0.0009671 0.0009671 0.23% check_for_negative_density() 5 0.000942 0.000942 0.000942 0.22% MLMG::getGradSolution() 6 0.0009033 0.0009033 0.0009033 0.21% MLCellLinOp::compGrad() 6 0.0009 0.0009 0.0009 0.21% MLCellLinOp::setLevelBC() 6 0.0008934 0.0008934 0.0008934 0.21% MLMG::computeMLResidual() 6 0.0008542 0.0008542 0.0008542 0.20% FabArrayBase::getCPC() 632 0.0008515 0.0008515 0.0008515 0.20% Castro::reset_internal_energy(Fab) 240 0.0008248 0.0008248 0.0008248 0.19% MLPoisson::prepareForSolve() 6 0.0008121 0.0008121 0.0008121 0.19% MLCellLinOp::prepareForSolve() 6 0.0008084 0.0008084 0.0008084 0.19% Gravity::update_max_rhs() 6 0.000733 0.000733 0.000733 0.17% Castro::enforce_speed_limit() 30 0.0007258 0.0007258 0.0007258 0.17% FabArray::setDomainBndry() 20 0.0007075 0.0007075 0.0007075 0.17% FabArray::mult() 22 0.0006936 0.0006936 0.0006936 0.16% Castro::check_for_nan() 10 0.0006673 0.0006673 0.0006673 0.16% Other 2160 0.003786 0.003786 0.003786 0.88% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 12 MiB 9037 MiB Castro::initMFs() 48 48 57 MiB 68 MiB Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB StateData::restartDoit() 32 32 52 MiB 55 MiB FillPatchIterator::Initialize 160 160 1107 KiB 39 MiB Castro::initialize_do_advance() 40 40 28 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1742 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6403 KiB 14 MiB MLMG::prepareForSolve() 361 361 3353 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 190 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 191 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6389 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 21 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3411 B 2048 KiB Gravity::solve_for_phi() 40 40 620 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 26 KiB 2048 KiB BndryData::define() 576 576 309 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 199 KiB 671 KiB Castro::estTimeStep() 10 10 2996 B 480 KiB VisMF::Write(FabArray) 112 112 1324 B 320 KiB Castro::normalize_species() 30 30 7444 B 320 KiB amrex::average_down 469 469 1420 B 257 KiB MLMG::addInterpCorrection() 468 468 1095 B 257 KiB amrex::Dot() 592 592 3231 B 160 KiB FabArray::norminf() 501 501 3166 B 160 KiB check_for_negative_density() 5 5 353 B 160 KiB MultiFab::max() 6 6 75 B 160 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MultiFab::contains_nan() 10 10 30 B 20 KiB MLPoisson::Fsmooth() 60 60 3245 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 46 B 10 KiB FillBoundary_nowait() 336 336 262 B 9648 B MLCellLinOp::applyBC() 3820 3820 210 B 9344 B amrex::Copy() 56 56 5843 B 8816 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B StateData::FillBoundary(geom) 960 960 44 B 2640 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLCellLinOp::defineBC() 36 36 347 B 1248 B MLCGSolver::bicgstab 180 180 89 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1188 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 87 KiB 8192 KiB VisMF::Write(FabArray) 120 120 164 KiB 3584 KiB VisMF::Read() 24 24 214 KiB 3000 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MLPoisson::Fsmooth() 60 60 3245 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 46 B 10 KiB FillBoundary_nowait() 336 336 262 B 9648 B MLCellLinOp::applyBC() 1910 1910 208 B 9328 B amrex::Copy() 56 56 5843 B 8816 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B Gravity::get_old_grav_vector() 3 3 2510 B 3072 B Gravity::fill_multipole_BCs() 18 18 6 B 2832 B StateData::FillBoundary(geom) 960 960 45 B 2688 B MLMG::prepareForSolve() 7 7 799 B 1648 B amrex::average_down 37 37 458 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 23 B 400 B FabArray::norminf() 501 501 9 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2105 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.04-21-g96f6d669d1bb) finalized