Initializing AMReX (24.05-29-g74ab0719f697)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.05-29-g74ab0719f697) initialized Starting run at 09:27:07 UTC on 2024-05-24. Successfully read inputs file ... Castro git describe: 24.05-13-gf7309fc71 AMReX git describe: 24.05-29-g74ab0719f Microphysics git describe: 24.05-12-g0dd7c3c7 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.04538565 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.02522536 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.070974831 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051448281 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.075546838 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.082007979 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.058449286 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.045175951 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.066263808 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.067466781 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.066306139 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.077358043 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.078104819 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.043337506 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024924069 seconds Ending run at 09:27:08 UTC on 2024-05-24. Run time = 0.930591882 Run time without initialization = 0.808127087 Average number of zones advanced per microsecond: 3.244 Average number of zones advanced per microsecond per rank: 3.244 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.9306 ... 0.9306 ... 0.9306 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.3005 0.3005 0.3005 32.29% VisMF::Write(FabArray) 11 0.1754 0.1754 0.1754 18.85% MLCellLinOp::applyBC() 4298 0.08106 0.08106 0.08106 8.71% MLPoisson::Fsmooth() 3240 0.03387 0.03387 0.03387 3.64% FillBoundary_nowait() 3893 0.03092 0.03092 0.03092 3.32% StateData::FillBoundary(geom) 328 0.02607 0.02607 0.02607 2.80% amrex::Dot() 1100 0.02166 0.02166 0.02166 2.33% Castro::normalize_species() 62 0.02089 0.02089 0.02089 2.24% FabArray::norminf() 1048 0.02011 0.02011 0.02011 2.16% Castro::computeTemp() 63 0.01664 0.01664 0.01664 1.79% FabArray::ParallelCopy_nowait() 851 0.01376 0.01376 0.01376 1.48% FabArray::setVal() 1054 0.01364 0.01364 0.01364 1.47% FabArray::Saxpy() 1353 0.01323 0.01323 0.01323 1.42% Castro::enforce_min_density() 62 0.01274 0.01274 0.01274 1.37% StateDataPhysBCFunct::() 41 0.01256 0.01256 0.01256 1.35% amrex::Copy() 469 0.011 0.011 0.011 1.18% MLPoisson::Fapply() 1047 0.01047 0.01047 0.01047 1.12% MLCellLinOp::defineAuxData() 11 0.01037 0.01037 0.01037 1.11% Gravity::fill_multipole_BCs() 11 0.009728 0.009728 0.009728 1.05% MLCellLinOp::prepareForSolve() 11 0.008415 0.008415 0.008415 0.90% FabArray::Xpay() 730 0.007888 0.007888 0.007888 0.85% MLMG::addInterpCorrection() 405 0.007028 0.007028 0.007028 0.76% Castro::estTimeStep() 21 0.006227 0.006227 0.006227 0.67% amrex::average_down 405 0.0062 0.0062 0.0062 0.67% Amr::checkPoint() 3 0.006023 0.006023 0.006023 0.65% Castro::reset_internal_energy(MultiFab) 63 0.004841 0.004841 0.004841 0.52% BndryData::define() 11 0.004083 0.004083 0.004083 0.44% amrex::Add() 81 0.003594 0.003594 0.003594 0.39% Castro::construct_new_gravity_source() 10 0.003432 0.003432 0.003432 0.37% Castro::construct_old_gravity_source() 10 0.002857 0.002857 0.002857 0.31% Castro::enforce_speed_limit() 62 0.002707 0.002707 0.002707 0.29% Amr::writePlotFile() 2 0.002178 0.002178 0.002178 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001822 0.001822 0.001822 0.20% check_for_negative_density() 10 0.00182 0.00182 0.00182 0.20% Castro::reset_internal_energy(Fab) 504 0.001749 0.001749 0.001749 0.19% MLCellLinOp::setLevelBC() 11 0.001635 0.001635 0.001635 0.18% Gravity::actual_solve_with_mlmg() 11 0.001552 0.001552 0.001552 0.17% MLCGSolver::bicgstab 81 0.001539 0.001539 0.001539 0.17% Castro::initData() 1 0.001508 0.001508 0.001508 0.16% FabArray::setDomainBndry() 41 0.00141 0.00141 0.00141 0.15% FabArray::mult() 43 0.001382 0.001382 0.001382 0.15% MultiFab::contains_nan() 20 0.001283 0.001283 0.001283 0.14% MLCellLinOp::compGrad() 11 0.001086 0.001086 0.001086 0.12% MLCellLinOp::smooth() 1620 0.001039 0.001039 0.001039 0.11% MLMG::prepareForSolve() 11 0.0009949 0.0009949 0.0009949 0.11% FabArrayBase::getCPC() 1313 0.0007966 0.0007966 0.0007966 0.09% FabArray::FillBoundary() 3893 0.0007838 0.0007838 0.0007838 0.08% Gravity::get_new_grav_vector() 11 0.000609 0.000609 0.000609 0.07% Gravity::get_old_grav_vector() 10 0.000479 0.000479 0.000479 0.05% MLCellLinOp::apply() 1047 0.0004199 0.0004199 0.0004199 0.05% AmrLevel::FillPatch() 41 0.0004122 0.0004122 0.0004122 0.04% Amr::coarseTimeStep() 10 0.0003954 0.0003954 0.0003954 0.04% main() 1 0.0003203 0.0003203 0.0003203 0.03% MLCGSolver::ParallelAllReduce 1809 0.0003123 0.0003123 0.0003123 0.03% MLCellLinOp::defineBC() 11 0.0002628 0.0002628 0.0002628 0.03% FabArray::ParallelCopy() 851 0.0002511 0.0002511 0.0002511 0.03% FillPatchIterator::Initialize 41 0.0002148 0.0002148 0.0002148 0.02% Castro::subcycle_advance_ctu() 10 0.0002141 0.0002141 0.0002141 0.02% MLMG::mgVcycle() 81 0.0002016 0.0002016 0.0002016 0.02% Amr::timeStep() 10 0.0001688 0.0001688 0.0001688 0.02% MLCellLinOp::correctionResidual() 405 0.0001601 0.0001601 0.0001601 0.02% MLMG:computeResOfCorrection() 405 0.0001151 0.0001151 0.0001151 0.01% StateData::checkPoint() 12 0.0001104 0.0001104 0.0001104 0.01% Gravity::solve_for_phi() 10 0.0001032 0.0001032 0.0001032 0.01% Castro::advance() 10 8.397e-05 8.397e-05 8.397e-05 0.01% MLMG::actualBottomSolve() 81 8.045e-05 8.045e-05 8.045e-05 0.01% Castro::initialize_advance() 10 8.035e-05 8.035e-05 8.035e-05 0.01% MLMG::mgVcycle_down::0 81 7.608e-05 7.608e-05 7.608e-05 0.01% Castro::clean_state() 62 7.511e-05 7.511e-05 7.511e-05 0.01% MLMG::solve() 11 7.12e-05 7.12e-05 7.12e-05 0.01% MLMG::mgVcycle_down::1 81 7.055e-05 7.055e-05 7.055e-05 0.01% MLMG::mgVcycle_down::2 81 6.691e-05 6.691e-05 6.691e-05 0.01% MLMG::mgVcycle_down::4 81 6.29e-05 6.29e-05 6.29e-05 0.01% AmrLevel::checkPoint() 3 6.091e-05 6.091e-05 6.091e-05 0.01% MLMG::mgVcycle_down::3 81 6.043e-05 6.043e-05 6.043e-05 0.01% Castro::initialize_do_advance() 10 6.011e-05 6.011e-05 6.011e-05 0.01% MLMG::oneIter() 81 5.778e-05 5.778e-05 5.778e-05 0.01% MLMG::mgVcycle_up::4 81 5.34e-05 5.34e-05 5.34e-05 0.01% Castro::do_new_sources() 10 4.928e-05 4.928e-05 4.928e-05 0.01% MLMG::mgVcycle_up::0 81 4.825e-05 4.825e-05 4.825e-05 0.01% Castro::finalize_do_advance() 10 4.807e-05 4.807e-05 4.807e-05 0.01% Castro::do_advance_ctu() 10 4.669e-05 4.669e-05 4.669e-05 0.01% MLMG::mgVcycle_up::3 81 4.666e-05 4.666e-05 4.666e-05 0.01% FillPatchIterator::FillFromLevel0() 41 4.661e-05 4.661e-05 4.661e-05 0.01% MLCellLinOp::solutionResidual() 92 4.619e-05 4.619e-05 4.619e-05 0.00% MLMG::mgVcycle_up::1 81 4.553e-05 4.553e-05 4.553e-05 0.00% MLMG::mgVcycle_up::2 81 4.477e-05 4.477e-05 4.477e-05 0.00% Castro::post_timestep() 10 4.264e-05 4.264e-05 4.264e-05 0.00% Castro::construct_new_source() 50 4.26e-05 4.26e-05 4.26e-05 0.00% Amr::defBaseLevel() 1 3.658e-05 3.658e-05 3.658e-05 0.00% FillPatchSingleLevel 41 3.467e-05 3.467e-05 3.467e-05 0.00% MLMG::computeResidual() 81 3.356e-05 3.356e-05 3.356e-05 0.00% MLMG::mgVcycle_bottom 81 3.271e-05 3.271e-05 3.271e-05 0.00% MLMG::ResNormInf() 92 2.988e-05 2.988e-05 2.988e-05 0.00% Castro::construct_new_gravity() 10 2.906e-05 2.906e-05 2.906e-05 0.00% MLPoisson::define() 11 2.475e-05 2.475e-05 2.475e-05 0.00% Amr::FinalizeInit() 1 2.341e-05 2.341e-05 2.341e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.317e-05 2.317e-05 2.317e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.152e-05 2.152e-05 2.152e-05 0.00% Castro::do_old_sources() 10 1.946e-05 1.946e-05 1.946e-05 0.00% Castro::construct_old_source() 50 1.844e-05 1.844e-05 1.844e-05 0.00% Castro::apply_source_to_state() 20 1.318e-05 1.318e-05 1.318e-05 0.00% Castro::check_for_nan() 20 1.158e-05 1.158e-05 1.158e-05 0.00% Castro::construct_old_gravity() 10 1.141e-05 1.141e-05 1.141e-05 0.00% MLMG::computeMLResidual() 11 1.064e-05 1.064e-05 1.064e-05 0.00% Castro::computeNewDt() 9 8.22e-06 8.22e-06 8.22e-06 0.00% MLPoisson::prepareForSolve() 11 8.1e-06 8.1e-06 8.1e-06 0.00% Gravity::actual_multilevel_solve() 1 7.986e-06 7.986e-06 7.986e-06 0.00% Amr::InitializeInit() 1 6.503e-06 6.503e-06 6.503e-06 0.00% Castro::expand_state() 10 6.156e-06 6.156e-06 6.156e-06 0.00% MLMG::getGradSolution() 11 6.153e-06 6.153e-06 6.153e-06 0.00% Castro::post_init() 1 4.466e-06 4.466e-06 4.466e-06 0.00% Amr::init() 1 2.679e-06 2.679e-06 2.679e-06 0.00% Amr::initialInit() 1 1.141e-06 1.141e-06 1.141e-06 0.00% Other 4705 0.003403 0.003403 0.003403 0.37% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9306 0.9306 0.9306 100.00% Amr::coarseTimeStep() 10 0.7829 0.7829 0.7829 84.13% Amr::timeStep() 10 0.6905 0.6905 0.6905 74.19% Castro::advance() 10 0.6789 0.6789 0.6789 72.95% Castro::subcycle_advance_ctu() 10 0.6645 0.6645 0.6645 71.40% Castro::do_advance_ctu() 10 0.6643 0.6643 0.6643 71.38% Castro::construct_ctu_hydro_source() 10 0.3115 0.3115 0.3115 33.47% Gravity::solve_phi_with_mlmg() 11 0.3037 0.3037 0.3037 32.63% Gravity::actual_solve_with_mlmg() 11 0.2935 0.2935 0.2935 31.53% Castro::construct_new_gravity() 10 0.2782 0.2782 0.2782 29.89% MLMG::solve() 11 0.2712 0.2712 0.2712 29.14% Gravity::solve_for_phi() 10 0.262 0.262 0.262 28.15% MLMG::oneIter() 81 0.2484 0.2484 0.2484 26.70% MLMG::mgVcycle() 81 0.2448 0.2448 0.2448 26.30% VisMF::Write(FabArray) 11 0.1754 0.1754 0.1754 18.85% Amr::checkPoint() 3 0.134 0.134 0.134 14.40% AmrLevel::checkPoint() 3 0.128 0.128 0.128 13.76% StateData::checkPoint() 12 0.128 0.128 0.128 13.75% MLCellLinOp::smooth() 1620 0.1221 0.1221 0.1221 13.12% Amr::init() 1 0.1217 0.1217 0.1217 13.08% MLCellLinOp::applyBC() 4298 0.1135 0.1135 0.1135 12.19% MLMG::mgVcycle_bottom 81 0.07339 0.07339 0.07339 7.89% MLMG::actualBottomSolve() 81 0.07336 0.07336 0.07336 7.88% MLCGSolver::bicgstab 81 0.07255 0.07255 0.07255 7.80% Castro::clean_state() 62 0.05876 0.05876 0.05876 6.31% Amr::initialInit() 1 0.05095 0.05095 0.05095 5.47% Amr::writePlotFile() 2 0.05031 0.05031 0.05031 5.41% AmrLevel::FillPatch() 41 0.04873 0.04873 0.04873 5.24% Amr::FinalizeInit() 1 0.04598 0.04598 0.04598 4.94% Castro::post_init() 1 0.04469 0.04469 0.04469 4.80% FillPatchIterator::Initialize 41 0.04442 0.04442 0.04442 4.77% FillPatchIterator::FillFromLevel0() 41 0.0428 0.0428 0.0428 4.60% FillPatchSingleLevel 41 0.04275 0.04275 0.04275 4.59% Gravity::multilevel_solve_for_new_phi() 1 0.04213 0.04213 0.04213 4.53% Gravity::actual_multilevel_solve() 1 0.0421 0.0421 0.0421 4.52% StateDataPhysBCFunct::() 41 0.03863 0.03863 0.03863 4.15% MLCellLinOp::apply() 1047 0.03661 0.03661 0.03661 3.93% MLMG::mgVcycle_down::0 81 0.03476 0.03476 0.03476 3.74% MLPoisson::Fsmooth() 3240 0.03387 0.03387 0.03387 3.64% FabArray::FillBoundary() 3893 0.03241 0.03241 0.03241 3.48% FillBoundary_nowait() 3893 0.03163 0.03163 0.03163 3.40% MLMG::mgVcycle_up::0 81 0.02633 0.02633 0.02633 2.83% StateData::FillBoundary(geom) 328 0.02607 0.02607 0.02607 2.80% Castro::initialize_do_advance() 10 0.02418 0.02418 0.02418 2.60% Castro::computeTemp() 63 0.02323 0.02323 0.02323 2.50% amrex::Dot() 1100 0.02166 0.02166 0.02166 2.33% Castro::normalize_species() 62 0.02089 0.02089 0.02089 2.24% MLMG:computeResOfCorrection() 405 0.02056 0.02056 0.02056 2.21% MLCellLinOp::correctionResidual() 405 0.02044 0.02044 0.02044 2.20% FabArray::norminf() 1048 0.02011 0.02011 0.02011 2.16% Castro::do_old_sources() 10 0.01926 0.01926 0.01926 2.07% Gravity::get_new_grav_vector() 11 0.01833 0.01833 0.01833 1.97% MLPoisson::define() 11 0.01745 0.01745 0.01745 1.88% MLMG::mgVcycle_down::1 81 0.01672 0.01672 0.01672 1.80% MLMG::mgVcycle_down::2 81 0.01556 0.01556 0.01556 1.67% Castro::construct_old_gravity() 10 0.01534 0.01534 0.01534 1.65% Gravity::get_old_grav_vector() 10 0.01532 0.01532 0.01532 1.65% MLMG::mgVcycle_down::3 81 0.01516 0.01516 0.01516 1.63% MLMG::mgVcycle_down::4 81 0.01506 0.01506 0.01506 1.62% FabArray::ParallelCopy() 851 0.01482 0.01482 0.01482 1.59% FabArray::ParallelCopy_nowait() 851 0.01457 0.01457 0.01457 1.57% Castro::initialize_advance() 10 0.01374 0.01374 0.01374 1.48% FabArray::setVal() 1054 0.01364 0.01364 0.01364 1.47% FabArray::Saxpy() 1353 0.01323 0.01323 0.01323 1.42% MLCGSolver::ParallelAllReduce 1809 0.01299 0.01299 0.01299 1.40% Castro::enforce_min_density() 62 0.01274 0.01274 0.01274 1.37% Castro::do_new_sources() 10 0.01244 0.01244 0.01244 1.34% MLMG::prepareForSolve() 11 0.0124 0.0124 0.0124 1.33% MLMG::addInterpCorrection() 405 0.01238 0.01238 0.01238 1.33% Castro::expand_state() 10 0.01232 0.01232 0.01232 1.32% MLMG::mgVcycle_up::1 81 0.01212 0.01212 0.01212 1.30% MLMG::mgVcycle_up::4 81 0.01202 0.01202 0.01202 1.29% MLMG::mgVcycle_up::2 81 0.01185 0.01185 0.01185 1.27% MLCellLinOp::defineAuxData() 11 0.01182 0.01182 0.01182 1.27% MLMG::mgVcycle_up::3 81 0.01161 0.01161 0.01161 1.25% amrex::average_down 405 0.01158 0.01158 0.01158 1.24% Castro::post_timestep() 10 0.01138 0.01138 0.01138 1.22% amrex::Copy() 469 0.011 0.011 0.011 1.18% MLPoisson::Fapply() 1047 0.01047 0.01047 0.01047 1.12% Gravity::fill_multipole_BCs() 11 0.009965 0.009965 0.009965 1.07% MLPoisson::prepareForSolve() 11 0.008424 0.008424 0.008424 0.91% MLCellLinOp::prepareForSolve() 11 0.008415 0.008415 0.008415 0.90% FabArray::Xpay() 730 0.007888 0.007888 0.007888 0.85% MLCellLinOp::solutionResidual() 92 0.00785 0.00785 0.00785 0.84% Castro::reset_internal_energy(MultiFab) 63 0.006591 0.006591 0.006591 0.71% MLMG::computeResidual() 81 0.006545 0.006545 0.006545 0.70% Castro::estTimeStep() 21 0.006227 0.006227 0.006227 0.67% MLCellLinOp::defineBC() 11 0.005378 0.005378 0.005378 0.58% BndryData::define() 11 0.005115 0.005115 0.005115 0.55% Amr::InitializeInit() 1 0.004969 0.004969 0.004969 0.53% Amr::defBaseLevel() 1 0.004963 0.004963 0.004963 0.53% Castro::initData() 1 0.004308 0.004308 0.004308 0.46% amrex::Add() 81 0.003594 0.003594 0.003594 0.39% Castro::construct_new_source() 50 0.003475 0.003475 0.003475 0.37% Castro::construct_new_gravity_source() 10 0.003432 0.003432 0.003432 0.37% Castro::construct_old_source() 50 0.002875 0.002875 0.002875 0.31% Castro::construct_old_gravity_source() 10 0.002857 0.002857 0.002857 0.31% Castro::computeNewDt() 9 0.002733 0.002733 0.002733 0.29% Castro::enforce_speed_limit() 62 0.002707 0.002707 0.002707 0.29% Castro::finalize_do_advance() 10 0.00241 0.00241 0.00241 0.26% MLMG::ResNormInf() 92 0.002176 0.002176 0.002176 0.23% Castro::apply_source_to_state() 20 0.001877 0.001877 0.001877 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001822 0.001822 0.001822 0.20% check_for_negative_density() 10 0.00182 0.00182 0.00182 0.20% Castro::reset_internal_energy(Fab) 504 0.001749 0.001749 0.001749 0.19% MLCellLinOp::setLevelBC() 11 0.001635 0.001635 0.001635 0.18% MLMG::getGradSolution() 11 0.001609 0.001609 0.001609 0.17% MLCellLinOp::compGrad() 11 0.001603 0.001603 0.001603 0.17% FabArrayBase::getCPC() 1313 0.001468 0.001468 0.001468 0.16% FabArray::setDomainBndry() 41 0.00141 0.00141 0.00141 0.15% FabArray::mult() 43 0.001382 0.001382 0.001382 0.15% MLMG::computeMLResidual() 11 0.001349 0.001349 0.001349 0.14% Castro::check_for_nan() 20 0.001295 0.001295 0.001295 0.14% MultiFab::contains_nan() 20 0.001283 0.001283 0.001283 0.14% Other 4705 0.009024 0.009024 0.009024 0.97% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 5510 KiB 9037 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1024 KiB 39 MiB Castro::initialize_do_advance() 80 80 27 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1507 KiB 28 MiB Castro::initialize_advance() 80 80 17 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7614 KiB 14 MiB MLMG::prepareForSolve() 660 660 3489 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 204 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 168 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7519 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 18 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2171 B 2048 KiB Gravity::solve_for_phi() 80 80 575 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 92 KiB 2048 KiB BndryData::define() 1056 1056 327 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 208 KiB 671 KiB Castro::estTimeStep() 21 21 3261 B 480 KiB VisMF::Write(FabArray) 656 656 3325 B 320 KiB Castro::normalize_species() 62 62 7313 B 320 KiB amrex::average_down 1054 1054 1583 B 257 KiB MLMG::addInterpCorrection() 1053 1053 1146 B 257 KiB amrex::Dot() 1343 1343 3439 B 160 KiB FabArray::norminf() 1129 1129 3342 B 160 KiB check_for_negative_density() 10 10 314 B 160 KiB Castro::initData() 1 1 50 B 160 KiB MultiFab::max() 11 11 56 B 160 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MultiFab::contains_nan() 20 20 27 B 20 KiB MLPoisson::Fsmooth() 132 132 3405 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 43 B 10 KiB FillBoundary_nowait() 751 751 289 B 9648 B MLCellLinOp::applyBC() 8596 8596 217 B 9344 B MLCellLinOp::prepareForSolve() 66 66 10 B 7792 B amrex::Copy() 100 100 3948 B 6144 B StateData::FillBoundary(geom) 1992 1992 41 B 4064 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 369 B 1248 B MLCGSolver::bicgstab 405 405 94 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 542 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 40 KiB 8192 KiB VisMF::Write(FabArray) 744 744 412 KiB 3584 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MLPoisson::Fsmooth() 132 132 3405 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 43 B 10 KiB FillBoundary_nowait() 751 751 289 B 9648 B MLCellLinOp::applyBC() 4298 4298 215 B 9328 B MLCellLinOp::prepareForSolve() 66 66 10 B 7792 B amrex::Copy() 100 100 3948 B 6144 B StateData::FillBoundary(geom) 1992 1992 41 B 4064 B Gravity::get_new_grav_vector() 3 3 2905 B 3072 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B amrex::average_down 82 82 617 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLMG::addInterpCorrection() 81 81 2 B 1024 B MLMG::prepareForSolve() 11 11 290 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1343 1343 25 B 400 B FabArray::norminf() 1129 1129 9 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2105 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.05-29-g74ab0719f697) finalized Initializing AMReX (24.05-29-g74ab0719f697)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.05-29-g74ab0719f697) initialized Starting run at 09:27:08 UTC on 2024-05-24. Successfully read inputs file ... Castro git describe: 24.05-13-gf7309fc71 AMReX git describe: 24.05-29-g74ab0719f Microphysics git describe: 24.05-12-g0dd7c3c7 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.506117759 Restart time = 0.072557223 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.074431209 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.060671901 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.074106425 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.075836074 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.054282544 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.027543503 seconds Ending run at 09:27:09 UTC on 2024-05-24. Run time = 0.44054765 Run time without initialization = 0.367336296 Average number of zones advanced per microsecond: 3.568 Average number of zones advanced per microsecond per rank: 3.568 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.4406 ... 0.4406 ... 0.4406 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1407 0.1407 0.1407 31.93% VisMF::Read() 3 0.06127 0.06127 0.06127 13.91% MLCellLinOp::applyBC() 1910 0.03585 0.03585 0.03585 8.14% VisMF::Write(FabArray) 1 0.02478 0.02478 0.02478 5.62% MLPoisson::Fsmooth() 1440 0.01504 0.01504 0.01504 3.41% StateData::FillBoundary(geom) 160 0.01323 0.01323 0.01323 3.00% FillBoundary_nowait() 1730 0.01302 0.01302 0.01302 2.95% Castro::normalize_species() 30 0.01052 0.01052 0.01052 2.39% amrex::Dot() 484 0.009322 0.009322 0.009322 2.12% FabArray::norminf() 465 0.008748 0.008748 0.008748 1.99% Castro::computeTemp() 30 0.008093 0.008093 0.008093 1.84% Castro::enforce_min_density() 30 0.006977 0.006977 0.006977 1.58% FabArray::setVal() 501 0.006694 0.006694 0.006694 1.52% FabArray::ParallelCopy_nowait() 380 0.006237 0.006237 0.006237 1.42% FabArray::Saxpy() 597 0.005954 0.005954 0.005954 1.35% StateDataPhysBCFunct::() 20 0.005855 0.005855 0.005855 1.33% MLCellLinOp::defineAuxData() 6 0.005688 0.005688 0.005688 1.29% amrex::Copy() 221 0.005501 0.005501 0.005501 1.25% Gravity::fill_multipole_BCs() 6 0.00548 0.00548 0.00548 1.24% Amr::restart() 1 0.004894 0.004894 0.004894 1.11% MLPoisson::Fapply() 464 0.004565 0.004565 0.004565 1.04% FabArray::Xpay() 325 0.003548 0.003548 0.003548 0.81% Castro::estTimeStep() 10 0.003523 0.003523 0.003523 0.80% MLMG::addInterpCorrection() 180 0.003144 0.003144 0.003144 0.71% amrex::average_down 180 0.002776 0.002776 0.002776 0.63% Amr::writePlotFile() 1 0.002601 0.002601 0.002601 0.59% BndryData::define() 6 0.002247 0.002247 0.002247 0.51% Castro::reset_internal_energy(MultiFab) 30 0.002156 0.002156 0.002156 0.49% Castro::construct_new_gravity_source() 5 0.001888 0.001888 0.001888 0.43% Castro::construct_old_gravity_source() 5 0.001565 0.001565 0.001565 0.36% amrex::Add() 36 0.001549 0.001549 0.001549 0.35% Castro::enforce_speed_limit() 30 0.001187 0.001187 0.001187 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009873 0.0009873 0.0009873 0.22% check_for_negative_density() 5 0.0009155 0.0009155 0.0009155 0.21% MLCellLinOp::setLevelBC() 6 0.0008962 0.0008962 0.0008962 0.20% Castro::reset_internal_energy(Fab) 240 0.0008409 0.0008409 0.0008409 0.19% Gravity::actual_solve_with_mlmg() 6 0.0008404 0.0008404 0.0008404 0.19% MLCellLinOp::prepareForSolve() 6 0.0007606 0.0007606 0.0007606 0.17% FabArray::setDomainBndry() 20 0.0007126 0.0007126 0.0007126 0.16% MLCGSolver::bicgstab 36 0.0007064 0.0007064 0.0007064 0.16% FabArray::mult() 22 0.0006938 0.0006938 0.0006938 0.16% MLCellLinOp::compGrad() 6 0.0006068 0.0006068 0.0006068 0.14% MLMG::prepareForSolve() 6 0.0005477 0.0005477 0.0005477 0.12% MLCellLinOp::smooth() 720 0.0004791 0.0004791 0.0004791 0.11% FabArrayBase::getCPC() 632 0.0003771 0.0003771 0.0003771 0.09% Gravity::get_old_grav_vector() 5 0.0003514 0.0003514 0.0003514 0.08% FabArray::FillBoundary() 1730 0.0003236 0.0003236 0.0003236 0.07% Gravity::get_new_grav_vector() 5 0.0003082 0.0003082 0.0003082 0.07% main() 1 0.0002644 0.0002644 0.0002644 0.06% AmrLevel::FillPatch() 20 0.0002054 0.0002054 0.0002054 0.05% Amr::coarseTimeStep() 5 0.0001924 0.0001924 0.0001924 0.04% MLCellLinOp::apply() 464 0.0001855 0.0001855 0.0001855 0.04% MLCellLinOp::defineBC() 6 0.000145 0.000145 0.000145 0.03% MLCGSolver::ParallelAllReduce 798 0.0001384 0.0001384 0.0001384 0.03% Castro::subcycle_advance_ctu() 5 0.0001378 0.0001378 0.0001378 0.03% FabArray::ParallelCopy() 380 0.0001107 0.0001107 0.0001107 0.03% FillPatchIterator::Initialize 20 0.0001042 0.0001042 0.0001042 0.02% MLMG::mgVcycle() 36 9.489e-05 9.489e-05 9.489e-05 0.02% Castro::advance() 5 9.319e-05 9.319e-05 9.319e-05 0.02% Amr::timeStep() 5 8.791e-05 8.791e-05 8.791e-05 0.02% AmrLevel::restart() 1 7.24e-05 7.24e-05 7.24e-05 0.02% MLCellLinOp::correctionResidual() 180 6.878e-05 6.878e-05 6.878e-05 0.02% Castro::do_advance_ctu() 5 6.775e-05 6.775e-05 6.775e-05 0.02% Castro::initialize_do_advance() 5 6.76e-05 6.76e-05 6.76e-05 0.02% StateData::restartDoit() 4 6.418e-05 6.418e-05 6.418e-05 0.01% Gravity::update_max_rhs() 6 6.047e-05 6.047e-05 6.047e-05 0.01% Castro::construct_old_source() 25 5.187e-05 5.187e-05 5.187e-05 0.01% Gravity::solve_for_phi() 5 5.012e-05 5.012e-05 5.012e-05 0.01% MLMG:computeResOfCorrection() 180 4.659e-05 4.659e-05 4.659e-05 0.01% Castro::finalize_do_advance() 5 4.242e-05 4.242e-05 4.242e-05 0.01% MLMG::actualBottomSolve() 36 3.56e-05 3.56e-05 3.56e-05 0.01% Castro::initialize_advance() 5 3.549e-05 3.549e-05 3.549e-05 0.01% MLMG::mgVcycle_down::0 36 3.531e-05 3.531e-05 3.531e-05 0.01% Castro::do_new_sources() 5 3.404e-05 3.404e-05 3.404e-05 0.01% MLMG::solve() 6 3.394e-05 3.394e-05 3.394e-05 0.01% Castro::clean_state() 30 3.313e-05 3.313e-05 3.313e-05 0.01% MLMG::mgVcycle_down::1 36 3.167e-05 3.167e-05 3.167e-05 0.01% MLMG::mgVcycle_down::2 36 3.032e-05 3.032e-05 3.032e-05 0.01% Castro::construct_new_source() 25 2.882e-05 2.882e-05 2.882e-05 0.01% MLMG::mgVcycle_down::4 36 2.83e-05 2.83e-05 2.83e-05 0.01% MLMG::mgVcycle_down::3 36 2.719e-05 2.719e-05 2.719e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 2.611e-05 2.611e-05 2.611e-05 0.01% MLMG::mgVcycle_up::4 36 2.585e-05 2.585e-05 2.585e-05 0.01% MLMG::oneIter() 36 2.573e-05 2.573e-05 2.573e-05 0.01% MLMG::mgVcycle_up::3 36 2.317e-05 2.317e-05 2.317e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.182e-05 2.182e-05 2.182e-05 0.00% MLMG::mgVcycle_up::0 36 2.179e-05 2.179e-05 2.179e-05 0.00% Castro::post_restart() 1 2.139e-05 2.139e-05 2.139e-05 0.00% MLMG::mgVcycle_up::2 36 2.104e-05 2.104e-05 2.104e-05 0.00% MLCellLinOp::solutionResidual() 42 2.102e-05 2.102e-05 2.102e-05 0.00% MLMG::mgVcycle_up::1 36 2.099e-05 2.099e-05 2.099e-05 0.00% MLPoisson::define() 6 1.818e-05 1.818e-05 1.818e-05 0.00% MLMG::ResNormInf() 42 1.605e-05 1.605e-05 1.605e-05 0.00% FillPatchSingleLevel 20 1.562e-05 1.562e-05 1.562e-05 0.00% MLMG::mgVcycle_bottom 36 1.494e-05 1.494e-05 1.494e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.363e-05 1.363e-05 1.363e-05 0.00% MLMG::computeResidual() 36 1.35e-05 1.35e-05 1.35e-05 0.00% Castro::construct_new_gravity() 5 1.231e-05 1.231e-05 1.231e-05 0.00% Castro::do_old_sources() 5 1.061e-05 1.061e-05 1.061e-05 0.00% Gravity::actual_multilevel_solve() 1 8.783e-06 8.783e-06 8.783e-06 0.00% Castro::expand_state() 5 8.108e-06 8.108e-06 8.108e-06 0.00% MLMG::getGradSolution() 6 7.931e-06 7.931e-06 7.931e-06 0.00% Castro::apply_source_to_state() 10 5.89e-06 5.89e-06 5.89e-06 0.00% Castro::check_for_nan() 10 5.869e-06 5.869e-06 5.869e-06 0.00% Castro::construct_old_gravity() 5 5.648e-06 5.648e-06 5.648e-06 0.00% Castro::post_timestep() 5 4.827e-06 4.827e-06 4.827e-06 0.00% MLMG::computeMLResidual() 6 4.167e-06 4.167e-06 4.167e-06 0.00% Castro::computeNewDt() 5 3.696e-06 3.696e-06 3.696e-06 0.00% MLPoisson::prepareForSolve() 6 3.509e-06 3.509e-06 3.509e-06 0.00% Amr::init() 1 7.55e-07 7.55e-07 7.55e-07 0.00% Other 2160 0.002522 0.002522 0.002522 0.57% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4406 0.4406 0.4406 99.99% Amr::coarseTimeStep() 5 0.3395 0.3395 0.3395 77.06% Amr::timeStep() 5 0.3373 0.3373 0.3373 76.56% Castro::advance() 5 0.3312 0.3312 0.3312 75.17% Castro::subcycle_advance_ctu() 5 0.324 0.324 0.324 73.53% Castro::do_advance_ctu() 5 0.3238 0.3238 0.3238 73.50% Castro::construct_ctu_hydro_source() 5 0.1462 0.1462 0.1462 33.19% Castro::construct_new_gravity() 5 0.1372 0.1372 0.1372 31.15% Gravity::solve_phi_with_mlmg() 6 0.1343 0.1343 0.1343 30.48% Gravity::solve_for_phi() 5 0.129 0.129 0.129 29.28% Gravity::actual_solve_with_mlmg() 6 0.1286 0.1286 0.1286 29.18% MLMG::solve() 6 0.1163 0.1163 0.1163 26.39% MLMG::oneIter() 36 0.1085 0.1085 0.1085 24.63% MLMG::mgVcycle() 36 0.1069 0.1069 0.1069 24.27% Amr::init() 1 0.0726 0.0726 0.0726 16.48% Amr::restart() 1 0.0726 0.0726 0.0726 16.48% AmrLevel::restart() 1 0.06163 0.06163 0.06163 13.99% StateData::restartDoit() 4 0.06156 0.06156 0.06156 13.97% VisMF::Read() 3 0.06127 0.06127 0.06127 13.91% MLCellLinOp::smooth() 720 0.05333 0.05333 0.05333 12.10% MLCellLinOp::applyBC() 1910 0.04951 0.04951 0.04951 11.24% MLMG::mgVcycle_bottom 36 0.03192 0.03192 0.03192 7.25% MLMG::actualBottomSolve() 36 0.03191 0.03191 0.03191 7.24% MLCGSolver::bicgstab 36 0.03154 0.03154 0.03154 7.16% Castro::clean_state() 30 0.0298 0.0298 0.0298 6.76% Amr::writePlotFile() 1 0.02765 0.02765 0.02765 6.28% VisMF::Write(FabArray) 1 0.02478 0.02478 0.02478 5.62% AmrLevel::FillPatch() 20 0.0241 0.0241 0.0241 5.47% FillPatchIterator::Initialize 20 0.02198 0.02198 0.02198 4.99% FillPatchIterator::FillFromLevel0() 20 0.02116 0.02116 0.02116 4.80% FillPatchSingleLevel 20 0.02114 0.02114 0.02114 4.80% StateDataPhysBCFunct::() 20 0.01908 0.01908 0.01908 4.33% MLCellLinOp::apply() 464 0.01614 0.01614 0.01614 3.66% MLMG::mgVcycle_down::0 36 0.01515 0.01515 0.01515 3.44% MLPoisson::Fsmooth() 1440 0.01504 0.01504 0.01504 3.41% FabArray::FillBoundary() 1730 0.01366 0.01366 0.01366 3.10% FillBoundary_nowait() 1730 0.01334 0.01334 0.01334 3.03% StateData::FillBoundary(geom) 160 0.01323 0.01323 0.01323 3.00% Castro::initialize_do_advance() 5 0.01178 0.01178 0.01178 2.67% MLMG::mgVcycle_up::0 36 0.01134 0.01134 0.01134 2.57% Castro::computeTemp() 30 0.01109 0.01109 0.01109 2.52% Castro::normalize_species() 30 0.01052 0.01052 0.01052 2.39% Castro::do_old_sources() 5 0.01024 0.01024 0.01024 2.32% MLPoisson::define() 6 0.009619 0.009619 0.009619 2.18% amrex::Dot() 484 0.009322 0.009322 0.009322 2.12% MLMG:computeResOfCorrection() 180 0.008959 0.008959 0.008959 2.03% MLCellLinOp::correctionResidual() 180 0.008912 0.008912 0.008912 2.02% FabArray::norminf() 465 0.008748 0.008748 0.008748 1.99% Castro::do_new_sources() 5 0.008297 0.008297 0.008297 1.88% Gravity::get_new_grav_vector() 5 0.008082 0.008082 0.008082 1.83% Castro::construct_old_gravity() 5 0.007936 0.007936 0.007936 1.80% Gravity::get_old_grav_vector() 5 0.00793 0.00793 0.00793 1.80% MLMG::mgVcycle_down::1 36 0.007423 0.007423 0.007423 1.68% Castro::enforce_min_density() 30 0.006977 0.006977 0.006977 1.58% MLMG::mgVcycle_down::2 36 0.006827 0.006827 0.006827 1.55% Castro::initialize_advance() 5 0.006813 0.006813 0.006813 1.55% FabArray::ParallelCopy() 380 0.006745 0.006745 0.006745 1.53% FabArray::setVal() 501 0.006694 0.006694 0.006694 1.52% MLMG::mgVcycle_down::3 36 0.006668 0.006668 0.006668 1.51% FabArray::ParallelCopy_nowait() 380 0.006635 0.006635 0.006635 1.51% MLMG::mgVcycle_down::4 36 0.006605 0.006605 0.006605 1.50% Castro::expand_state() 5 0.006602 0.006602 0.006602 1.50% MLCellLinOp::defineAuxData() 6 0.006476 0.006476 0.006476 1.47% Castro::post_timestep() 5 0.00607 0.00607 0.00607 1.38% FabArray::Saxpy() 597 0.005954 0.005954 0.005954 1.35% Castro::post_restart() 1 0.00589 0.00589 0.00589 1.34% MLCGSolver::ParallelAllReduce 798 0.005624 0.005624 0.005624 1.28% Gravity::fill_multipole_BCs() 6 0.005608 0.005608 0.005608 1.27% Gravity::multilevel_solve_for_new_phi() 1 0.005518 0.005518 0.005518 1.25% amrex::Copy() 221 0.005501 0.005501 0.005501 1.25% MLMG::addInterpCorrection() 180 0.005492 0.005492 0.005492 1.25% Gravity::actual_multilevel_solve() 1 0.005492 0.005492 0.005492 1.25% MLMG::mgVcycle_up::4 36 0.005352 0.005352 0.005352 1.21% MLMG::mgVcycle_up::1 36 0.005298 0.005298 0.005298 1.20% MLMG::mgVcycle_up::2 36 0.005166 0.005166 0.005166 1.17% amrex::average_down 180 0.005131 0.005131 0.005131 1.16% MLMG::mgVcycle_up::3 36 0.005098 0.005098 0.005098 1.16% MLPoisson::Fapply() 464 0.004565 0.004565 0.004565 1.04% MLCellLinOp::solutionResidual() 42 0.003702 0.003702 0.003702 0.84% FabArray::Xpay() 325 0.003548 0.003548 0.003548 0.81% Castro::estTimeStep() 10 0.003523 0.003523 0.003523 0.80% Castro::reset_internal_energy(MultiFab) 30 0.002997 0.002997 0.002997 0.68% MLCellLinOp::defineBC() 6 0.002985 0.002985 0.002985 0.68% MLMG::prepareForSolve() 6 0.002898 0.002898 0.002898 0.66% MLMG::computeResidual() 36 0.002884 0.002884 0.002884 0.65% BndryData::define() 6 0.00284 0.00284 0.00284 0.64% Castro::computeNewDt() 5 0.001997 0.001997 0.001997 0.45% Castro::construct_new_source() 25 0.001917 0.001917 0.001917 0.44% Castro::construct_new_gravity_source() 5 0.001888 0.001888 0.001888 0.43% Castro::construct_old_source() 25 0.001616 0.001616 0.001616 0.37% Castro::finalize_do_advance() 5 0.001572 0.001572 0.001572 0.36% Castro::construct_old_gravity_source() 5 0.001565 0.001565 0.001565 0.36% amrex::Add() 36 0.001549 0.001549 0.001549 0.35% Castro::enforce_speed_limit() 30 0.001187 0.001187 0.001187 0.27% MLMG::ResNormInf() 42 0.0009973 0.0009973 0.0009973 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009873 0.0009873 0.0009873 0.22% Castro::apply_source_to_state() 10 0.0009734 0.0009734 0.0009734 0.22% MLMG::getGradSolution() 6 0.0009159 0.0009159 0.0009159 0.21% check_for_negative_density() 5 0.0009155 0.0009155 0.0009155 0.21% MLCellLinOp::compGrad() 6 0.0009079 0.0009079 0.0009079 0.21% MLCellLinOp::setLevelBC() 6 0.0008962 0.0008962 0.0008962 0.20% Castro::reset_internal_energy(Fab) 240 0.0008409 0.0008409 0.0008409 0.19% MLMG::computeMLResidual() 6 0.0008359 0.0008359 0.0008359 0.19% FabArrayBase::getCPC() 632 0.0007909 0.0007909 0.0007909 0.18% MLPoisson::prepareForSolve() 6 0.0007641 0.0007641 0.0007641 0.17% MLCellLinOp::prepareForSolve() 6 0.0007606 0.0007606 0.0007606 0.17% Gravity::update_max_rhs() 6 0.0007222 0.0007222 0.0007222 0.16% FabArray::setDomainBndry() 20 0.0007126 0.0007126 0.0007126 0.16% FabArray::mult() 22 0.0006938 0.0006938 0.0006938 0.16% Castro::check_for_nan() 10 0.0006712 0.0006712 0.0006712 0.15% Other 2160 0.003749 0.003749 0.003749 0.85% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 12 MiB 9037 MiB Castro::initMFs() 48 48 57 MiB 68 MiB Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB StateData::restartDoit() 32 32 52 MiB 55 MiB FillPatchIterator::Initialize 160 160 1109 KiB 39 MiB Castro::initialize_do_advance() 40 40 28 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1664 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6422 KiB 14 MiB MLMG::prepareForSolve() 361 361 3241 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 186 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 187 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6409 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 20 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3271 B 2048 KiB Gravity::solve_for_phi() 40 40 598 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 25 KiB 2048 KiB BndryData::define() 576 576 299 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 192 KiB 671 KiB Castro::estTimeStep() 10 10 3838 B 480 KiB VisMF::Write(FabArray) 112 112 1262 B 320 KiB Castro::normalize_species() 30 30 7774 B 320 KiB amrex::average_down 469 469 1371 B 257 KiB MLMG::addInterpCorrection() 468 468 1069 B 257 KiB amrex::Dot() 592 592 3126 B 160 KiB FabArray::norminf() 501 501 3071 B 160 KiB check_for_negative_density() 5 5 333 B 160 KiB MultiFab::max() 6 6 74 B 160 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MultiFab::contains_nan() 10 10 30 B 20 KiB MLPoisson::Fsmooth() 60 60 3139 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 44 B 10 KiB FillBoundary_nowait() 336 336 255 B 9648 B MLCellLinOp::applyBC() 3820 3820 203 B 9344 B amrex::Copy() 56 56 5819 B 8816 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B StateData::FillBoundary(geom) 960 960 47 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCellLinOp::defineBC() 36 36 335 B 1248 B MLCGSolver::bicgstab 180 180 86 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1183 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 83 KiB 8192 KiB VisMF::Write(FabArray) 120 120 157 KiB 3584 KiB VisMF::Read() 24 24 210 KiB 3000 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MLPoisson::Fsmooth() 60 60 3139 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 44 B 10 KiB FillBoundary_nowait() 336 336 254 B 9648 B MLCellLinOp::applyBC() 1910 1910 202 B 9328 B amrex::Copy() 56 56 5819 B 8816 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B Gravity::get_old_grav_vector() 3 3 2517 B 3072 B StateData::FillBoundary(geom) 960 960 48 B 3024 B Gravity::fill_multipole_BCs() 18 18 6 B 2832 B MLMG::prepareForSolve() 7 7 791 B 1648 B amrex::average_down 37 37 455 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 22 B 400 B FabArray::norminf() 501 501 8 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2105 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.05-29-g74ab0719f697) finalized