Initializing AMReX (24.07-15-gcca5aade6929)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.07-15-gcca5aade6929) initialized Starting run at 09:03:49 UTC on 2024-07-10. Successfully read inputs file ... Castro git describe: 24.07-9-gdd89b8efb AMReX git describe: 24.07-15-gcca5aade6 Microphysics git describe: 24.07-10-g0b1a2310 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.050084524 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.027919673 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.062602269 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.0575236 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.060484667 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.075695131 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.083954895 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.06548728 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.053718928 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.061366139 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.064891409 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.064146218 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.059950921 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.047590221 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.027585684 seconds Ending run at 09:03:50 UTC on 2024-07-10. Run time = 0.915376124 Run time without initialization = 0.785649436 Average number of zones advanced per microsecond: 3.337 Average number of zones advanced per microsecond per rank: 3.337 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.9154 ... 0.9154 ... 0.9154 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2397 0.2397 0.2397 26.18% VisMF::Write(FabArray) 11 0.1938 0.1938 0.1938 21.17% MLCellLinOp::applyBC() 4298 0.08396 0.08396 0.08396 9.17% MLPoisson::Fsmooth() 3240 0.03488 0.03488 0.03488 3.81% StateData::FillBoundary(geom) 328 0.03439 0.03439 0.03439 3.76% FillBoundary_nowait() 3893 0.03331 0.03331 0.03331 3.64% Amr::checkPoint() 3 0.02253 0.02253 0.02253 2.46% amrex::Dot() 1100 0.02243 0.02243 0.02243 2.45% Castro::reset_internal_energy(MultiFab) 63 0.02203 0.02203 0.02203 2.41% FabArray::norminf() 1048 0.02063 0.02063 0.02063 2.25% Castro::computeTemp() 63 0.02037 0.02037 0.02037 2.23% FabArray::ParallelCopy_nowait() 851 0.01413 0.01413 0.01413 1.54% FabArray::setVal() 1054 0.01391 0.01391 0.01391 1.52% FabArray::Saxpy() 1353 0.01348 0.01348 0.01348 1.47% amrex::Copy() 469 0.01119 0.01119 0.01119 1.22% MLPoisson::Fapply() 1047 0.01076 0.01076 0.01076 1.18% MLCellLinOp::defineAuxData() 11 0.01064 0.01064 0.01064 1.16% StateDataPhysBCFunct::() 41 0.01054 0.01054 0.01054 1.15% Castro::normalize_species() 62 0.009602 0.009602 0.009602 1.05% Gravity::fill_multipole_BCs() 11 0.008243 0.008243 0.008243 0.90% FabArray::Xpay() 730 0.008095 0.008095 0.008095 0.88% MLMG::addInterpCorrection() 405 0.007193 0.007193 0.007193 0.79% Castro::enforce_min_density() 62 0.007012 0.007012 0.007012 0.77% amrex::average_down 405 0.006432 0.006432 0.006432 0.70% Castro::estTimeStep() 21 0.005792 0.005792 0.005792 0.63% BndryData::define() 11 0.004196 0.004196 0.004196 0.46% amrex::Add() 81 0.003744 0.003744 0.003744 0.41% Castro::construct_new_gravity_source() 10 0.003711 0.003711 0.003711 0.41% Castro::construct_old_gravity_source() 10 0.00308 0.00308 0.00308 0.34% Castro::reset_internal_energy(Fab) 504 0.002747 0.002747 0.002747 0.30% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001859 0.001859 0.001859 0.20% Amr::writePlotFile() 2 0.001829 0.001829 0.001829 0.20% MLCellLinOp::setLevelBC() 11 0.001669 0.001669 0.001669 0.18% MLCGSolver::bicgstab 81 0.001636 0.001636 0.001636 0.18% Castro::initData() 1 0.001605 0.001605 0.001605 0.18% Gravity::actual_solve_with_mlmg() 11 0.001534 0.001534 0.001534 0.17% FabArray::setDomainBndry() 41 0.001463 0.001463 0.001463 0.16% FabArray::mult() 43 0.001428 0.001428 0.001428 0.16% MLCellLinOp::prepareForSolve() 11 0.00137 0.00137 0.00137 0.15% MultiFab::contains_nan() 20 0.001329 0.001329 0.001329 0.15% check_for_negative_density() 10 0.001287 0.001287 0.001287 0.14% MLCellLinOp::compGrad() 11 0.001107 0.001107 0.001107 0.12% MLCellLinOp::smooth() 1620 0.001089 0.001089 0.001089 0.12% MLMG::prepareForSolve() 11 0.0009748 0.0009748 0.0009748 0.11% FabArrayBase::getCPC() 1313 0.0007891 0.0007891 0.0007891 0.09% FabArray::FillBoundary() 3893 0.0007689 0.0007689 0.0007689 0.08% Castro::subcycle_advance_ctu() 10 0.0007634 0.0007634 0.0007634 0.08% Gravity::get_new_grav_vector() 11 0.0006325 0.0006325 0.0006325 0.07% Gravity::get_old_grav_vector() 10 0.0004985 0.0004985 0.0004985 0.05% AmrLevel::FillPatch() 41 0.0004119 0.0004119 0.0004119 0.04% MLCellLinOp::apply() 1047 0.0004097 0.0004097 0.0004097 0.04% Amr::coarseTimeStep() 10 0.0003459 0.0003459 0.0003459 0.04% MLCGSolver::ParallelAllReduce 1809 0.0003203 0.0003203 0.0003203 0.03% main() 1 0.0002995 0.0002995 0.0002995 0.03% MLCellLinOp::defineBC() 11 0.0002704 0.0002704 0.0002704 0.03% Castro::construct_new_source() 50 0.0002537 0.0002537 0.0002537 0.03% FabArray::ParallelCopy() 851 0.0002488 0.0002488 0.0002488 0.03% FillPatchIterator::Initialize 41 0.0002133 0.0002133 0.0002133 0.02% MLMG::mgVcycle() 81 0.0002032 0.0002032 0.0002032 0.02% MLCellLinOp::correctionResidual() 405 0.0001694 0.0001694 0.0001694 0.02% Castro::advance() 10 0.0001668 0.0001668 0.0001668 0.02% Amr::timeStep() 10 0.00016 0.00016 0.00016 0.02% Gravity::solve_for_phi() 10 0.0001567 0.0001567 0.0001567 0.02% Castro::do_advance_ctu() 10 0.0001533 0.0001533 0.0001533 0.02% Castro::finalize_do_advance() 10 0.0001402 0.0001402 0.0001402 0.02% MLMG:computeResOfCorrection() 405 0.0001189 0.0001189 0.0001189 0.01% StateData::checkPoint() 12 0.0001114 0.0001114 0.0001114 0.01% Castro::do_new_sources() 10 0.0001061 0.0001061 0.0001061 0.01% MLMG::actualBottomSolve() 81 9.551e-05 9.551e-05 9.551e-05 0.01% MLMG::mgVcycle_down::0 81 8.426e-05 8.426e-05 8.426e-05 0.01% Castro::initialize_advance() 10 7.791e-05 7.791e-05 7.791e-05 0.01% MLMG::mgVcycle_down::2 81 7.498e-05 7.498e-05 7.498e-05 0.01% MLMG::mgVcycle_down::4 81 7.32e-05 7.32e-05 7.32e-05 0.01% MLMG::mgVcycle_down::1 81 7.312e-05 7.312e-05 7.312e-05 0.01% MLMG::solve() 11 7.3e-05 7.3e-05 7.3e-05 0.01% Castro::construct_old_source() 50 7.15e-05 7.15e-05 7.15e-05 0.01% MLMG::mgVcycle_down::3 81 6.779e-05 6.779e-05 6.779e-05 0.01% Castro::clean_state() 62 6.554e-05 6.554e-05 6.554e-05 0.01% AmrLevel::checkPoint() 3 6.361e-05 6.361e-05 6.361e-05 0.01% Castro::initialize_do_advance() 10 5.88e-05 5.88e-05 5.88e-05 0.01% MLMG::oneIter() 81 5.391e-05 5.391e-05 5.391e-05 0.01% MLMG::mgVcycle_up::4 81 5.374e-05 5.374e-05 5.374e-05 0.01% FillPatchIterator::FillFromLevel0() 41 4.973e-05 4.973e-05 4.973e-05 0.01% MLMG::mgVcycle_up::0 81 4.97e-05 4.97e-05 4.97e-05 0.01% MLMG::mgVcycle_up::3 81 4.913e-05 4.913e-05 4.913e-05 0.01% MLMG::mgVcycle_up::1 81 4.799e-05 4.799e-05 4.799e-05 0.01% MLMG::mgVcycle_up::2 81 4.697e-05 4.697e-05 4.697e-05 0.01% MLCellLinOp::solutionResidual() 92 4.358e-05 4.358e-05 4.358e-05 0.00% FillPatchSingleLevel 41 3.749e-05 3.749e-05 3.749e-05 0.00% MLMG::mgVcycle_bottom 81 3.208e-05 3.208e-05 3.208e-05 0.00% MLMG::ResNormInf() 92 3.181e-05 3.181e-05 3.181e-05 0.00% MLMG::computeResidual() 81 3.119e-05 3.119e-05 3.119e-05 0.00% Amr::defBaseLevel() 1 3.092e-05 3.092e-05 3.092e-05 0.00% Castro::construct_new_gravity() 10 2.783e-05 2.783e-05 2.783e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.782e-05 2.782e-05 2.782e-05 0.00% Castro::do_old_sources() 10 2.614e-05 2.614e-05 2.614e-05 0.00% MLPoisson::define() 11 1.997e-05 1.997e-05 1.997e-05 0.00% Amr::FinalizeInit() 1 1.827e-05 1.827e-05 1.827e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.673e-05 1.673e-05 1.673e-05 0.00% Castro::apply_source_to_state() 20 1.314e-05 1.314e-05 1.314e-05 0.00% MLPoisson::prepareForSolve() 11 1.308e-05 1.308e-05 1.308e-05 0.00% Castro::check_for_nan() 20 1.225e-05 1.225e-05 1.225e-05 0.00% Castro::construct_old_gravity() 10 1.08e-05 1.08e-05 1.08e-05 0.00% MLMG::computeMLResidual() 11 8.942e-06 8.942e-06 8.942e-06 0.00% Gravity::actual_multilevel_solve() 1 8.728e-06 8.728e-06 8.728e-06 0.00% Castro::post_init() 1 8.701e-06 8.701e-06 8.701e-06 0.00% Castro::post_timestep() 10 8.388e-06 8.388e-06 8.388e-06 0.00% Castro::computeNewDt() 9 7.8e-06 7.8e-06 7.8e-06 0.00% Amr::InitializeInit() 1 6.595e-06 6.595e-06 6.595e-06 0.00% Castro::expand_state() 10 5.967e-06 5.967e-06 5.967e-06 0.00% MLMG::getGradSolution() 11 5.788e-06 5.788e-06 5.788e-06 0.00% Amr::init() 1 2.493e-06 2.493e-06 2.493e-06 0.00% Amr::initialInit() 1 1.309e-06 1.309e-06 1.309e-06 0.00% Other 4767 0.003276 0.003276 0.003276 0.36% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9154 0.9154 0.9154 100.00% Amr::coarseTimeStep() 10 0.7578 0.7578 0.7578 82.79% Amr::timeStep() 10 0.6417 0.6417 0.6417 70.10% Castro::advance() 10 0.6321 0.6321 0.6321 69.05% Castro::subcycle_advance_ctu() 10 0.6187 0.6187 0.6187 67.58% Castro::do_advance_ctu() 10 0.6179 0.6179 0.6179 67.50% Gravity::solve_phi_with_mlmg() 11 0.3052 0.3052 0.3052 33.35% Gravity::actual_solve_with_mlmg() 11 0.2965 0.2965 0.2965 32.39% Castro::construct_new_gravity() 10 0.2866 0.2866 0.2866 31.30% MLMG::solve() 11 0.2738 0.2738 0.2738 29.91% Gravity::solve_for_phi() 10 0.263 0.263 0.263 28.73% MLMG::oneIter() 81 0.2578 0.2578 0.2578 28.16% MLMG::mgVcycle() 81 0.254 0.254 0.254 27.74% Castro::construct_ctu_hydro_source() 10 0.2517 0.2517 0.2517 27.49% VisMF::Write(FabArray) 11 0.1938 0.1938 0.1938 21.17% Amr::checkPoint() 3 0.1633 0.1633 0.1633 17.84% AmrLevel::checkPoint() 3 0.1408 0.1408 0.1408 15.38% StateData::checkPoint() 12 0.1407 0.1407 0.1407 15.37% Amr::init() 1 0.1291 0.1291 0.1291 14.10% MLCellLinOp::smooth() 1620 0.1274 0.1274 0.1274 13.92% MLCellLinOp::applyBC() 4298 0.1188 0.1188 0.1188 12.98% MLMG::mgVcycle_bottom 81 0.07565 0.07565 0.07565 8.26% MLMG::actualBottomSolve() 81 0.07562 0.07562 0.07562 8.26% MLCGSolver::bicgstab 81 0.07477 0.07477 0.07477 8.17% Castro::clean_state() 62 0.06109 0.06109 0.06109 6.67% Amr::writePlotFile() 2 0.05563 0.05563 0.05563 6.08% AmrLevel::FillPatch() 41 0.05519 0.05519 0.05519 6.03% Amr::initialInit() 1 0.05096 0.05096 0.05096 5.57% FillPatchIterator::Initialize 41 0.05082 0.05082 0.05082 5.55% FillPatchIterator::FillFromLevel0() 41 0.04915 0.04915 0.04915 5.37% FillPatchSingleLevel 41 0.0491 0.0491 0.0491 5.36% Amr::FinalizeInit() 1 0.04604 0.04604 0.04604 5.03% Castro::computeTemp() 63 0.04515 0.04515 0.04515 4.93% Castro::post_init() 1 0.04514 0.04514 0.04514 4.93% StateDataPhysBCFunct::() 41 0.04493 0.04493 0.04493 4.91% Gravity::multilevel_solve_for_new_phi() 1 0.04272 0.04272 0.04272 4.67% Gravity::actual_multilevel_solve() 1 0.0427 0.0427 0.0427 4.66% MLCellLinOp::apply() 1047 0.038 0.038 0.038 4.15% MLMG::mgVcycle_down::0 81 0.03658 0.03658 0.03658 4.00% MLPoisson::Fsmooth() 3240 0.03488 0.03488 0.03488 3.81% FabArray::FillBoundary() 3893 0.03482 0.03482 0.03482 3.80% StateData::FillBoundary(geom) 328 0.03439 0.03439 0.03439 3.76% FillBoundary_nowait() 3893 0.03405 0.03405 0.03405 3.72% MLMG::mgVcycle_up::0 81 0.02764 0.02764 0.02764 3.02% Gravity::get_new_grav_vector() 11 0.02551 0.02551 0.02551 2.79% Castro::reset_internal_energy(MultiFab) 63 0.02478 0.02478 0.02478 2.71% Castro::initialize_do_advance() 10 0.02318 0.02318 0.02318 2.53% amrex::Dot() 1100 0.02243 0.02243 0.02243 2.45% MLMG:computeResOfCorrection() 405 0.02141 0.02141 0.02141 2.34% MLCellLinOp::correctionResidual() 405 0.02129 0.02129 0.02129 2.33% FabArray::norminf() 1048 0.02063 0.02063 0.02063 2.25% Castro::do_old_sources() 10 0.0199 0.0199 0.0199 2.17% MLPoisson::define() 11 0.01789 0.01789 0.01789 1.95% MLMG::mgVcycle_down::1 81 0.01728 0.01728 0.01728 1.89% Castro::do_new_sources() 10 0.0165 0.0165 0.0165 1.80% MLMG::mgVcycle_down::2 81 0.0161 0.0161 0.0161 1.76% Castro::construct_old_gravity() 10 0.01597 0.01597 0.01597 1.74% Gravity::get_old_grav_vector() 10 0.01596 0.01596 0.01596 1.74% MLMG::mgVcycle_down::3 81 0.01572 0.01572 0.01572 1.72% MLMG::mgVcycle_down::4 81 0.0156 0.0156 0.0156 1.70% FabArray::ParallelCopy() 851 0.01519 0.01519 0.01519 1.66% FabArray::ParallelCopy_nowait() 851 0.01494 0.01494 0.01494 1.63% FabArray::setVal() 1054 0.01391 0.01391 0.01391 1.52% MLCGSolver::ParallelAllReduce 1809 0.01349 0.01349 0.01349 1.47% FabArray::Saxpy() 1353 0.01348 0.01348 0.01348 1.47% MLMG::addInterpCorrection() 405 0.01271 0.01271 0.01271 1.39% Castro::initialize_advance() 10 0.01262 0.01262 0.01262 1.38% MLMG::mgVcycle_up::1 81 0.01252 0.01252 0.01252 1.37% MLMG::mgVcycle_up::4 81 0.01244 0.01244 0.01244 1.36% MLMG::mgVcycle_up::2 81 0.01223 0.01223 0.01223 1.34% MLCellLinOp::defineAuxData() 11 0.01209 0.01209 0.01209 1.32% MLMG::mgVcycle_up::3 81 0.01201 0.01201 0.01201 1.31% amrex::average_down 405 0.01198 0.01198 0.01198 1.31% amrex::Copy() 469 0.01119 0.01119 0.01119 1.22% Castro::expand_state() 10 0.01115 0.01115 0.01115 1.22% MLPoisson::Fapply() 1047 0.01076 0.01076 0.01076 1.18% Castro::normalize_species() 62 0.009602 0.009602 0.009602 1.05% Castro::post_timestep() 10 0.009397 0.009397 0.009397 1.03% Gravity::fill_multipole_BCs() 11 0.008482 0.008482 0.008482 0.93% FabArray::Xpay() 730 0.008095 0.008095 0.008095 0.88% MLCellLinOp::solutionResidual() 92 0.008029 0.008029 0.008029 0.88% Castro::enforce_min_density() 62 0.007012 0.007012 0.007012 0.77% MLMG::computeResidual() 81 0.006719 0.006719 0.006719 0.73% Castro::estTimeStep() 21 0.005792 0.005792 0.005792 0.63% MLCellLinOp::defineBC() 11 0.005528 0.005528 0.005528 0.60% MLMG::prepareForSolve() 11 0.005374 0.005374 0.005374 0.59% BndryData::define() 11 0.005257 0.005257 0.005257 0.57% Amr::InitializeInit() 1 0.004916 0.004916 0.004916 0.54% Amr::defBaseLevel() 1 0.004909 0.004909 0.004909 0.54% Castro::initData() 1 0.004197 0.004197 0.004197 0.46% Castro::construct_new_source() 50 0.003965 0.003965 0.003965 0.43% amrex::Add() 81 0.003744 0.003744 0.003744 0.41% Castro::construct_new_gravity_source() 10 0.003711 0.003711 0.003711 0.41% Castro::construct_old_source() 50 0.003151 0.003151 0.003151 0.34% Castro::construct_old_gravity_source() 10 0.00308 0.00308 0.00308 0.34% Castro::finalize_do_advance() 10 0.003024 0.003024 0.003024 0.33% Castro::reset_internal_energy(Fab) 504 0.002747 0.002747 0.002747 0.30% MLMG::ResNormInf() 92 0.002243 0.002243 0.002243 0.25% Castro::computeNewDt() 9 0.00199 0.00199 0.00199 0.22% Castro::apply_source_to_state() 20 0.001897 0.001897 0.001897 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001859 0.001859 0.001859 0.20% MLCellLinOp::setLevelBC() 11 0.001669 0.001669 0.001669 0.18% MLMG::getGradSolution() 11 0.001643 0.001643 0.001643 0.18% MLCellLinOp::compGrad() 11 0.001637 0.001637 0.001637 0.18% FabArrayBase::getCPC() 1313 0.001471 0.001471 0.001471 0.16% FabArray::setDomainBndry() 41 0.001463 0.001463 0.001463 0.16% FabArray::mult() 43 0.001428 0.001428 0.001428 0.16% MLPoisson::prepareForSolve() 11 0.001383 0.001383 0.001383 0.15% MLCellLinOp::prepareForSolve() 11 0.00137 0.00137 0.00137 0.15% MLMG::computeMLResidual() 11 0.00135 0.00135 0.00135 0.15% Castro::check_for_nan() 20 0.001341 0.001341 0.001341 0.15% MultiFab::contains_nan() 20 0.001329 0.001329 0.001329 0.15% check_for_negative_density() 10 0.001287 0.001287 0.001287 0.14% Other 4767 0.00865 0.00865 0.00865 0.94% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 6088 KiB 9037 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1075 KiB 39 MiB Castro::initialize_do_advance() 80 80 26 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1710 KiB 28 MiB Castro::initialize_advance() 80 80 16 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7613 KiB 14 MiB MLMG::prepareForSolve() 660 660 3679 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 288 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 178 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7524 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 15 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2207 B 2048 KiB Gravity::solve_for_phi() 80 80 587 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 95 KiB 2048 KiB BndryData::define() 1056 1056 336 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 214 KiB 671 KiB Castro::estTimeStep() 21 21 3071 B 480 KiB VisMF::Write(FabArray) 656 656 3511 B 320 KiB Castro::normalize_species() 62 62 3398 B 320 KiB amrex::average_down 1054 1054 1635 B 257 KiB MLMG::addInterpCorrection() 1053 1053 1193 B 257 KiB amrex::Dot() 1343 1343 3617 B 160 KiB FabArray::norminf() 1129 1129 3487 B 160 KiB check_for_negative_density() 10 10 223 B 160 KiB Castro::initData() 1 1 53 B 160 KiB MultiFab::max() 11 11 59 B 160 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MultiFab::contains_nan() 20 20 28 B 20 KiB MLPoisson::Fsmooth() 132 132 3591 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 44 B 10 KiB FillBoundary_nowait() 751 751 318 B 9648 B MLCellLinOp::applyBC() 8596 8596 229 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3925 B 6144 B StateData::FillBoundary(geom) 1992 1992 42 B 2688 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 379 B 1248 B MLCGSolver::bicgstab 405 405 98 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 638 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 41 KiB 8192 KiB VisMF::Write(FabArray) 744 744 473 KiB 3584 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MLPoisson::Fsmooth() 132 132 3591 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 44 B 10 KiB FillBoundary_nowait() 751 751 318 B 9648 B MLCellLinOp::applyBC() 4298 4298 227 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3925 B 6144 B Gravity::get_new_grav_vector() 3 3 2903 B 3072 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B StateData::FillBoundary(geom) 1992 1992 43 B 2688 B amrex::average_down 82 82 618 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLMG::addInterpCorrection() 81 81 2 B 1024 B MLMG::prepareForSolve() 11 11 306 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1343 1343 26 B 400 B FabArray::norminf() 1129 1129 10 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2109 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.07-15-gcca5aade6929) finalized Initializing AMReX (24.07-15-gcca5aade6929)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.07-15-gcca5aade6929) initialized Starting run at 09:03:50 UTC on 2024-07-10. Successfully read inputs file ... Castro git describe: 24.07-9-gdd89b8efb AMReX git describe: 24.07-15-gcca5aade6 Microphysics git describe: 24.07-10-g0b1a2310 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.535490501 Restart time = 0.075207585 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.070308499 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050022294 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.061474128 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060740444 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.081920083 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.028987236 seconds Ending run at 09:03:51 UTC on 2024-07-10. Run time = 0.429683764 Run time without initialization = 0.35386835 Average number of zones advanced per microsecond: 3.704 Average number of zones advanced per microsecond per rank: 3.704 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.4298 ... 0.4298 ... 0.4298 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1223 0.1223 0.1223 28.45% VisMF::Read() 3 0.06318 0.06318 0.06318 14.70% MLCellLinOp::applyBC() 1910 0.03871 0.03871 0.03871 9.01% VisMF::Write(FabArray) 1 0.02649 0.02649 0.02649 6.16% MLPoisson::Fsmooth() 1440 0.01618 0.01618 0.01618 3.76% FillBoundary_nowait() 1730 0.01453 0.01453 0.01453 3.38% StateData::FillBoundary(geom) 160 0.01303 0.01303 0.01303 3.03% amrex::Dot() 484 0.01 0.01 0.01 2.33% FabArray::norminf() 465 0.009379 0.009379 0.009379 2.18% Castro::reset_internal_energy(MultiFab) 30 0.008457 0.008457 0.008457 1.97% FabArray::setVal() 501 0.007073 0.007073 0.007073 1.65% FabArray::ParallelCopy_nowait() 380 0.00659 0.00659 0.00659 1.53% Castro::computeTemp() 30 0.006547 0.006547 0.006547 1.52% FabArray::Saxpy() 597 0.006308 0.006308 0.006308 1.47% MLCellLinOp::defineAuxData() 6 0.006036 0.006036 0.006036 1.40% amrex::Copy() 221 0.005741 0.005741 0.005741 1.34% StateDataPhysBCFunct::() 20 0.00542 0.00542 0.00542 1.26% Amr::restart() 1 0.005326 0.005326 0.005326 1.24% Castro::normalize_species() 30 0.005182 0.005182 0.005182 1.21% Gravity::fill_multipole_BCs() 6 0.0051 0.0051 0.0051 1.19% MLPoisson::Fapply() 464 0.004898 0.004898 0.004898 1.14% FabArray::Xpay() 325 0.003761 0.003761 0.003761 0.88% Castro::enforce_min_density() 30 0.003745 0.003745 0.003745 0.87% MLMG::addInterpCorrection() 180 0.003387 0.003387 0.003387 0.79% amrex::average_down 180 0.003029 0.003029 0.003029 0.70% Castro::estTimeStep() 10 0.00269 0.00269 0.00269 0.63% BndryData::define() 6 0.002373 0.002373 0.002373 0.55% Amr::writePlotFile() 1 0.002314 0.002314 0.002314 0.54% Castro::construct_new_gravity_source() 5 0.001918 0.001918 0.001918 0.45% amrex::Add() 36 0.001649 0.001649 0.001649 0.38% Castro::construct_old_gravity_source() 5 0.001528 0.001528 0.001528 0.36% Castro::reset_internal_energy(Fab) 240 0.001196 0.001196 0.001196 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001055 0.001055 0.001055 0.25% MLCellLinOp::setLevelBC() 6 0.0009494 0.0009494 0.0009494 0.22% Gravity::actual_solve_with_mlmg() 6 0.0008871 0.0008871 0.0008871 0.21% MLCellLinOp::prepareForSolve() 6 0.0008058 0.0008058 0.0008058 0.19% MLCGSolver::bicgstab 36 0.0007986 0.0007986 0.0007986 0.19% FabArray::setDomainBndry() 20 0.0007424 0.0007424 0.0007424 0.17% FabArray::mult() 22 0.0007191 0.0007191 0.0007191 0.17% check_for_negative_density() 5 0.0007029 0.0007029 0.0007029 0.16% MLCellLinOp::compGrad() 6 0.0006395 0.0006395 0.0006395 0.15% MLMG::prepareForSolve() 6 0.000588 0.000588 0.000588 0.14% MLCellLinOp::smooth() 720 0.0005211 0.0005211 0.0005211 0.12% FabArrayBase::getCPC() 632 0.0003943 0.0003943 0.0003943 0.09% FabArray::FillBoundary() 1730 0.0003573 0.0003573 0.0003573 0.08% Gravity::get_old_grav_vector() 5 0.0003487 0.0003487 0.0003487 0.08% main() 1 0.0002855 0.0002855 0.0002855 0.07% Gravity::get_new_grav_vector() 5 0.0002388 0.0002388 0.0002388 0.06% AmrLevel::FillPatch() 20 0.0002119 0.0002119 0.0002119 0.05% MLCellLinOp::apply() 464 0.0002045 0.0002045 0.0002045 0.05% Amr::coarseTimeStep() 5 0.0001588 0.0001588 0.0001588 0.04% MLCellLinOp::defineBC() 6 0.0001569 0.0001569 0.0001569 0.04% MLCGSolver::ParallelAllReduce 798 0.0001514 0.0001514 0.0001514 0.04% Castro::subcycle_advance_ctu() 5 0.0001309 0.0001309 0.0001309 0.03% FabArray::ParallelCopy() 380 0.000116 0.000116 0.000116 0.03% FillPatchIterator::Initialize 20 0.0001109 0.0001109 0.0001109 0.03% Castro::initialize_do_advance() 5 9.67e-05 9.67e-05 9.67e-05 0.02% MLMG::mgVcycle() 36 9.174e-05 9.174e-05 9.174e-05 0.02% Amr::timeStep() 5 8.998e-05 8.998e-05 8.998e-05 0.02% StateData::restartDoit() 4 8.684e-05 8.684e-05 8.684e-05 0.02% MLCellLinOp::correctionResidual() 180 8.359e-05 8.359e-05 8.359e-05 0.02% Castro::advance() 5 7.428e-05 7.428e-05 7.428e-05 0.02% Castro::construct_old_source() 25 7.392e-05 7.392e-05 7.392e-05 0.02% AmrLevel::restart() 1 7.118e-05 7.118e-05 7.118e-05 0.02% Gravity::update_max_rhs() 6 7.091e-05 7.091e-05 7.091e-05 0.02% MLMG:computeResOfCorrection() 180 5.559e-05 5.559e-05 5.559e-05 0.01% Castro::do_advance_ctu() 5 5.503e-05 5.503e-05 5.503e-05 0.01% Gravity::solve_for_phi() 5 5.257e-05 5.257e-05 5.257e-05 0.01% Castro::finalize_do_advance() 5 4.78e-05 4.78e-05 4.78e-05 0.01% MLMG::mgVcycle_down::0 36 4.527e-05 4.527e-05 4.527e-05 0.01% MLMG::actualBottomSolve() 36 4.265e-05 4.265e-05 4.265e-05 0.01% MLMG::mgVcycle_down::1 36 4.144e-05 4.144e-05 4.144e-05 0.01% MLMG::mgVcycle_down::2 36 3.797e-05 3.797e-05 3.797e-05 0.01% Castro::initialize_advance() 5 3.7e-05 3.7e-05 3.7e-05 0.01% MLMG::solve() 6 3.697e-05 3.697e-05 3.697e-05 0.01% Castro::clean_state() 30 3.383e-05 3.383e-05 3.383e-05 0.01% MLMG::mgVcycle_down::4 36 3.243e-05 3.243e-05 3.243e-05 0.01% MLMG::mgVcycle_down::3 36 3.166e-05 3.166e-05 3.166e-05 0.01% Castro::construct_new_source() 25 3.034e-05 3.034e-05 3.034e-05 0.01% MLMG::mgVcycle_up::4 36 2.846e-05 2.846e-05 2.846e-05 0.01% Castro::construct_old_gravity() 5 2.732e-05 2.732e-05 2.732e-05 0.01% MLMG::oneIter() 36 2.706e-05 2.706e-05 2.706e-05 0.01% Castro::post_restart() 1 2.642e-05 2.642e-05 2.642e-05 0.01% MLMG::mgVcycle_up::3 36 2.447e-05 2.447e-05 2.447e-05 0.01% MLCellLinOp::solutionResidual() 42 2.282e-05 2.282e-05 2.282e-05 0.01% MLMG::mgVcycle_up::2 36 2.273e-05 2.273e-05 2.273e-05 0.01% MLMG::mgVcycle_up::0 36 2.267e-05 2.267e-05 2.267e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.226e-05 2.226e-05 2.226e-05 0.01% MLMG::mgVcycle_up::1 36 2.145e-05 2.145e-05 2.145e-05 0.00% MLMG::ResNormInf() 42 1.838e-05 1.838e-05 1.838e-05 0.00% Castro::construct_new_gravity() 5 1.837e-05 1.837e-05 1.837e-05 0.00% FillPatchSingleLevel 20 1.751e-05 1.751e-05 1.751e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.626e-05 1.626e-05 1.626e-05 0.00% MLMG::computeResidual() 36 1.528e-05 1.528e-05 1.528e-05 0.00% MLMG::mgVcycle_bottom 36 1.511e-05 1.511e-05 1.511e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.411e-05 1.411e-05 1.411e-05 0.00% MLPoisson::define() 6 1.346e-05 1.346e-05 1.346e-05 0.00% Castro::do_old_sources() 5 1.154e-05 1.154e-05 1.154e-05 0.00% Castro::do_new_sources() 5 9.965e-06 9.965e-06 9.965e-06 0.00% MLPoisson::prepareForSolve() 6 9.536e-06 9.536e-06 9.536e-06 0.00% Castro::expand_state() 5 9.279e-06 9.279e-06 9.279e-06 0.00% Gravity::actual_multilevel_solve() 1 8.682e-06 8.682e-06 8.682e-06 0.00% Castro::check_for_nan() 10 6.371e-06 6.371e-06 6.371e-06 0.00% Castro::apply_source_to_state() 10 5.84e-06 5.84e-06 5.84e-06 0.00% Castro::post_timestep() 5 5.543e-06 5.543e-06 5.543e-06 0.00% MLMG::computeMLResidual() 6 4.212e-06 4.212e-06 4.212e-06 0.00% Castro::computeNewDt() 5 4.074e-06 4.074e-06 4.074e-06 0.00% MLMG::getGradSolution() 6 3.239e-06 3.239e-06 3.239e-06 0.00% Amr::init() 1 8.09e-07 8.09e-07 8.09e-07 0.00% Other 2190 0.002605 0.002605 0.002605 0.61% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4297 0.4297 0.4297 99.98% Amr::coarseTimeStep() 5 0.3246 0.3246 0.3246 75.53% Amr::timeStep() 5 0.3228 0.3228 0.3228 75.10% Castro::advance() 5 0.3171 0.3171 0.3171 73.79% Castro::subcycle_advance_ctu() 5 0.3111 0.3111 0.3111 72.38% Castro::do_advance_ctu() 5 0.3109 0.3109 0.3109 72.35% Castro::construct_new_gravity() 5 0.1467 0.1467 0.1467 34.14% Gravity::solve_phi_with_mlmg() 6 0.1441 0.1441 0.1441 33.52% Gravity::actual_solve_with_mlmg() 6 0.1387 0.1387 0.1387 32.27% Gravity::solve_for_phi() 5 0.1385 0.1385 0.1385 32.23% Castro::construct_ctu_hydro_source() 5 0.128 0.128 0.128 29.78% MLMG::solve() 6 0.1257 0.1257 0.1257 29.25% MLMG::oneIter() 36 0.1174 0.1174 0.1174 27.31% MLMG::mgVcycle() 36 0.1157 0.1157 0.1157 26.92% Amr::init() 1 0.07527 0.07527 0.07527 17.51% Amr::restart() 1 0.07527 0.07527 0.07527 17.51% AmrLevel::restart() 1 0.06357 0.06357 0.06357 14.79% StateData::restartDoit() 4 0.06349 0.06349 0.06349 14.77% VisMF::Read() 3 0.06318 0.06318 0.06318 14.70% MLCellLinOp::smooth() 720 0.05797 0.05797 0.05797 13.49% MLCellLinOp::applyBC() 1910 0.05395 0.05395 0.05395 12.55% MLMG::mgVcycle_bottom 36 0.03431 0.03431 0.03431 7.98% MLMG::actualBottomSolve() 36 0.0343 0.0343 0.0343 7.98% MLCGSolver::bicgstab 36 0.0339 0.0339 0.0339 7.89% Amr::writePlotFile() 1 0.02907 0.02907 0.02907 6.76% VisMF::Write(FabArray) 1 0.02649 0.02649 0.02649 6.16% Castro::clean_state() 30 0.02534 0.02534 0.02534 5.90% AmrLevel::FillPatch() 20 0.02359 0.02359 0.02359 5.49% FillPatchIterator::Initialize 20 0.02144 0.02144 0.02144 4.99% FillPatchIterator::FillFromLevel0() 20 0.02059 0.02059 0.02059 4.79% FillPatchSingleLevel 20 0.02057 0.02057 0.02057 4.79% StateDataPhysBCFunct::() 20 0.01845 0.01845 0.01845 4.29% MLCellLinOp::apply() 464 0.01747 0.01747 0.01747 4.06% MLMG::mgVcycle_down::0 36 0.01656 0.01656 0.01656 3.85% Castro::computeTemp() 30 0.0162 0.0162 0.0162 3.77% MLPoisson::Fsmooth() 1440 0.01618 0.01618 0.01618 3.76% FabArray::FillBoundary() 1730 0.01524 0.01524 0.01524 3.55% FillBoundary_nowait() 1730 0.01488 0.01488 0.01488 3.46% StateData::FillBoundary(geom) 160 0.01303 0.01303 0.01303 3.03% MLMG::mgVcycle_up::0 36 0.01244 0.01244 0.01244 2.89% Castro::initialize_do_advance() 5 0.01057 0.01057 0.01057 2.46% MLPoisson::define() 6 0.01019 0.01019 0.01019 2.37% amrex::Dot() 484 0.01 0.01 0.01 2.33% MLMG:computeResOfCorrection() 180 0.009726 0.009726 0.009726 2.26% MLCellLinOp::correctionResidual() 180 0.00967 0.00967 0.00967 2.25% Castro::reset_internal_energy(MultiFab) 30 0.009653 0.009653 0.009653 2.25% FabArray::norminf() 465 0.009379 0.009379 0.009379 2.18% Castro::do_old_sources() 5 0.009066 0.009066 0.009066 2.11% Gravity::get_new_grav_vector() 5 0.008063 0.008063 0.008063 1.88% MLMG::mgVcycle_down::1 36 0.008059 0.008059 0.008059 1.88% Castro::construct_old_gravity() 5 0.007832 0.007832 0.007832 1.82% Gravity::get_old_grav_vector() 5 0.007804 0.007804 0.007804 1.82% MLMG::mgVcycle_down::2 36 0.007376 0.007376 0.007376 1.72% Castro::do_new_sources() 5 0.007226 0.007226 0.007226 1.68% MLMG::mgVcycle_down::3 36 0.007165 0.007165 0.007165 1.67% FabArray::ParallelCopy() 380 0.007124 0.007124 0.007124 1.66% MLMG::mgVcycle_down::4 36 0.007087 0.007087 0.007087 1.65% FabArray::setVal() 501 0.007073 0.007073 0.007073 1.65% FabArray::ParallelCopy_nowait() 380 0.007008 0.007008 0.007008 1.63% MLCellLinOp::defineAuxData() 6 0.006869 0.006869 0.006869 1.60% Castro::expand_state() 5 0.006329 0.006329 0.006329 1.47% FabArray::Saxpy() 597 0.006308 0.006308 0.006308 1.47% Castro::post_restart() 1 0.00618 0.00618 0.00618 1.44% MLCGSolver::ParallelAllReduce 798 0.006055 0.006055 0.006055 1.41% MLMG::addInterpCorrection() 180 0.005906 0.005906 0.005906 1.37% Gravity::multilevel_solve_for_new_phi() 1 0.005787 0.005787 0.005787 1.35% Gravity::actual_multilevel_solve() 1 0.00577 0.00577 0.00577 1.34% MLMG::mgVcycle_up::4 36 0.005758 0.005758 0.005758 1.34% MLMG::mgVcycle_up::1 36 0.005745 0.005745 0.005745 1.34% amrex::Copy() 221 0.005741 0.005741 0.005741 1.34% Castro::initialize_advance() 5 0.005641 0.005641 0.005641 1.31% MLMG::mgVcycle_up::2 36 0.005607 0.005607 0.005607 1.30% amrex::average_down 180 0.005531 0.005531 0.005531 1.29% Castro::post_timestep() 5 0.005526 0.005526 0.005526 1.29% MLMG::mgVcycle_up::3 36 0.005514 0.005514 0.005514 1.28% Gravity::fill_multipole_BCs() 6 0.005241 0.005241 0.005241 1.22% Castro::normalize_species() 30 0.005182 0.005182 0.005182 1.21% MLPoisson::Fapply() 464 0.004898 0.004898 0.004898 1.14% MLCellLinOp::solutionResidual() 42 0.003941 0.003941 0.003941 0.92% FabArray::Xpay() 325 0.003761 0.003761 0.003761 0.88% Castro::enforce_min_density() 30 0.003745 0.003745 0.003745 0.87% MLCellLinOp::defineBC() 6 0.003166 0.003166 0.003166 0.74% MLMG::prepareForSolve() 6 0.003114 0.003114 0.003114 0.72% MLMG::computeResidual() 36 0.003094 0.003094 0.003094 0.72% BndryData::define() 6 0.003009 0.003009 0.003009 0.70% Castro::estTimeStep() 10 0.00269 0.00269 0.00269 0.63% Castro::construct_new_source() 25 0.001948 0.001948 0.001948 0.45% Castro::construct_new_gravity_source() 5 0.001918 0.001918 0.001918 0.45% Castro::computeNewDt() 5 0.001707 0.001707 0.001707 0.40% amrex::Add() 36 0.001649 0.001649 0.001649 0.38% Castro::construct_old_source() 25 0.001602 0.001602 0.001602 0.37% Castro::construct_old_gravity_source() 5 0.001528 0.001528 0.001528 0.36% Castro::reset_internal_energy(Fab) 240 0.001196 0.001196 0.001196 0.28% MLMG::ResNormInf() 42 0.001074 0.001074 0.001074 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001055 0.001055 0.001055 0.25% Castro::finalize_do_advance() 5 0.001035 0.001035 0.001035 0.24% Castro::apply_source_to_state() 10 0.0009721 0.0009721 0.0009721 0.23% MLMG::getGradSolution() 6 0.0009567 0.0009567 0.0009567 0.22% MLCellLinOp::compGrad() 6 0.0009535 0.0009535 0.0009535 0.22% MLCellLinOp::setLevelBC() 6 0.0009494 0.0009494 0.0009494 0.22% MLMG::computeMLResidual() 6 0.0008672 0.0008672 0.0008672 0.20% FabArrayBase::getCPC() 632 0.0008316 0.0008316 0.0008316 0.19% MLPoisson::prepareForSolve() 6 0.0008153 0.0008153 0.0008153 0.19% MLCellLinOp::prepareForSolve() 6 0.0008058 0.0008058 0.0008058 0.19% Gravity::update_max_rhs() 6 0.0007652 0.0007652 0.0007652 0.18% FabArray::setDomainBndry() 20 0.0007424 0.0007424 0.0007424 0.17% FabArray::mult() 22 0.0007191 0.0007191 0.0007191 0.17% check_for_negative_density() 5 0.0007029 0.0007029 0.0007029 0.16% Castro::check_for_nan() 10 0.0006831 0.0006831 0.0006831 0.16% Other 2190 0.003919 0.003919 0.003919 0.91% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 12 MiB 9037 MiB Castro::initMFs() 48 48 56 MiB 68 MiB Castro::swap_state_time_levels() 32 32 45 MiB 55 MiB StateData::restartDoit() 32 32 52 MiB 55 MiB FillPatchIterator::Initialize 160 160 1102 KiB 39 MiB Castro::initialize_do_advance() 40 40 28 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1817 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6352 KiB 14 MiB MLMG::prepareForSolve() 361 361 3591 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 188 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 192 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6338 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 19 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3547 B 2048 KiB Gravity::solve_for_phi() 40 40 659 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 27 KiB 2048 KiB BndryData::define() 576 576 331 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 212 KiB 671 KiB Castro::estTimeStep() 10 10 2910 B 480 KiB VisMF::Write(FabArray) 112 112 1365 B 320 KiB Castro::normalize_species() 30 30 3909 B 320 KiB amrex::average_down 469 469 1471 B 257 KiB MLMG::addInterpCorrection() 468 468 1159 B 257 KiB amrex::Dot() 592 592 3434 B 160 KiB FabArray::norminf() 501 501 3368 B 160 KiB check_for_negative_density() 5 5 260 B 160 KiB MultiFab::max() 6 6 80 B 160 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MultiFab::contains_nan() 10 10 31 B 20 KiB MLPoisson::Fsmooth() 60 60 3480 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 46 B 10 KiB FillBoundary_nowait() 336 336 292 B 9648 B MLCellLinOp::applyBC() 3820 3820 224 B 9344 B amrex::Copy() 56 56 5843 B 8816 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B StateData::FillBoundary(geom) 960 960 39 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLCellLinOp::defineBC() 36 36 372 B 1248 B MLCGSolver::bicgstab 180 180 95 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1427 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 92 KiB 8192 KiB VisMF::Write(FabArray) 120 120 174 KiB 3584 KiB VisMF::Read() 24 24 223 KiB 3000 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MLPoisson::Fsmooth() 60 60 3480 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 46 B 10 KiB FillBoundary_nowait() 336 336 292 B 9648 B MLCellLinOp::applyBC() 1910 1910 223 B 9328 B amrex::Copy() 56 56 5843 B 8816 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B Gravity::get_old_grav_vector() 3 3 2483 B 3072 B StateData::FillBoundary(geom) 960 960 39 B 3024 B Gravity::fill_multipole_BCs() 18 18 6 B 2832 B MLMG::prepareForSolve() 7 7 814 B 1648 B amrex::average_down 37 37 456 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 25 B 400 B FabArray::norminf() 501 501 9 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2109 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.07-15-gcca5aade6929) finalized