Initializing AMReX (24.07-29-g3cb7f1555494)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.07-29-g3cb7f1555494) initialized Starting run at 08:02:14 UTC on 2024-07-24. Successfully read inputs file ... Castro git describe: 24.07-18-g56f51f6c1 AMReX git describe: 24.07-29-g3cb7f1555 Microphysics git describe: 24.07-31-g4c8331df reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.045817324 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.02561677 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.063337496 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.05048108 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.059974705 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.073586263 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.084758501 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.045620468 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.054849005 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.053553516 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.053991382 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060290391 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.079835996 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.056559986 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.025874662 seconds Ending run at 08:02:15 UTC on 2024-07-24. Run time = 0.88474229 Run time without initialization = 0.763380461 Average number of zones advanced per microsecond: 3.434 Average number of zones advanced per microsecond per rank: 3.434 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.8848 ... 0.8848 ... 0.8848 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2375 0.2375 0.2375 26.85% VisMF::Write(FabArray) 11 0.1798 0.1798 0.1798 20.32% MLCellLinOp::applyBC() 4298 0.081 0.081 0.081 9.16% MLPoisson::Fsmooth() 3240 0.03389 0.03389 0.03389 3.83% FillBoundary_nowait() 3893 0.03309 0.03309 0.03309 3.74% StateData::FillBoundary(geom) 328 0.02612 0.02612 0.02612 2.95% amrex::Dot() 1100 0.02136 0.02136 0.02136 2.41% Castro::reset_internal_energy(MultiFab) 63 0.02114 0.02114 0.02114 2.39% FabArray::norminf() 1048 0.01994 0.01994 0.01994 2.25% Castro::computeTemp() 63 0.01905 0.01905 0.01905 2.15% Amr::checkPoint() 3 0.01743 0.01743 0.01743 1.97% FabArray::ParallelCopy_nowait() 851 0.01372 0.01372 0.01372 1.55% FabArray::setVal() 1054 0.01349 0.01349 0.01349 1.52% FabArray::Saxpy() 1353 0.01313 0.01313 0.01313 1.48% StateDataPhysBCFunct::() 41 0.01299 0.01299 0.01299 1.47% amrex::average_down 405 0.01263 0.01263 0.01263 1.43% amrex::Copy() 469 0.01101 0.01101 0.01101 1.24% Castro::normalize_species() 62 0.0109 0.0109 0.0109 1.23% MLPoisson::Fapply() 1047 0.01042 0.01042 0.01042 1.18% MLCellLinOp::defineAuxData() 11 0.01009 0.01009 0.01009 1.14% Castro::enforce_min_density() 62 0.008696 0.008696 0.008696 0.98% FabArray::Xpay() 730 0.007861 0.007861 0.007861 0.89% Gravity::fill_multipole_BCs() 11 0.007424 0.007424 0.007424 0.84% MLMG::addInterpCorrection() 405 0.007009 0.007009 0.007009 0.79% Castro::estTimeStep() 21 0.005876 0.005876 0.005876 0.66% BndryData::define() 11 0.003898 0.003898 0.003898 0.44% Castro::construct_new_gravity_source() 10 0.003791 0.003791 0.003791 0.43% amrex::Add() 81 0.003725 0.003725 0.003725 0.42% Castro::construct_old_gravity_source() 10 0.003208 0.003208 0.003208 0.36% Castro::reset_internal_energy(Fab) 504 0.002825 0.002825 0.002825 0.32% Castro::initData() 1 0.001942 0.001942 0.001942 0.22% Amr::writePlotFile() 2 0.001828 0.001828 0.001828 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001793 0.001793 0.001793 0.20% MLCGSolver::bicgstab 81 0.001678 0.001678 0.001678 0.19% MLCellLinOp::setLevelBC() 11 0.001608 0.001608 0.001608 0.18% Gravity::actual_solve_with_mlmg() 11 0.001543 0.001543 0.001543 0.17% check_for_negative_density() 10 0.001424 0.001424 0.001424 0.16% FabArray::mult() 43 0.001391 0.001391 0.001391 0.16% FabArray::setDomainBndry() 41 0.001384 0.001384 0.001384 0.16% MLCellLinOp::prepareForSolve() 11 0.001317 0.001317 0.001317 0.15% MultiFab::contains_nan() 20 0.001296 0.001296 0.001296 0.15% MLCellLinOp::smooth() 1620 0.001187 0.001187 0.001187 0.13% MLCellLinOp::compGrad() 11 0.001113 0.001113 0.001113 0.13% FabArrayBase::getCPC() 1313 0.0008276 0.0008276 0.0008276 0.09% FabArray::FillBoundary() 3893 0.000752 0.000752 0.000752 0.08% MLMG::prepareForSolve() 11 0.0006918 0.0006918 0.0006918 0.08% Gravity::get_new_grav_vector() 11 0.0006268 0.0006268 0.0006268 0.07% Gravity::get_old_grav_vector() 10 0.0004927 0.0004927 0.0004927 0.06% MLCellLinOp::apply() 1047 0.0004176 0.0004176 0.0004176 0.05% AmrLevel::FillPatch() 41 0.0004124 0.0004124 0.0004124 0.05% Amr::coarseTimeStep() 10 0.0003396 0.0003396 0.0003396 0.04% MLCGSolver::ParallelAllReduce 1809 0.0003149 0.0003149 0.0003149 0.04% main() 1 0.0002784 0.0002784 0.0002784 0.03% Castro::construct_new_source() 50 0.0002718 0.0002718 0.0002718 0.03% FabArray::ParallelCopy() 851 0.0002658 0.0002658 0.0002658 0.03% MLCellLinOp::defineBC() 11 0.0002639 0.0002639 0.0002639 0.03% FillPatchIterator::Initialize 41 0.0002366 0.0002366 0.0002366 0.03% MLMG::mgVcycle() 81 0.0001839 0.0001839 0.0001839 0.02% MLCellLinOp::correctionResidual() 405 0.0001721 0.0001721 0.0001721 0.02% Castro::subcycle_advance_ctu() 10 0.0001607 0.0001607 0.0001607 0.02% Amr::timeStep() 10 0.000159 0.000159 0.000159 0.02% Gravity::solve_for_phi() 10 0.0001517 0.0001517 0.0001517 0.02% Castro::advance() 10 0.0001246 0.0001246 0.0001246 0.01% StateData::checkPoint() 12 0.0001153 0.0001153 0.0001153 0.01% Castro::do_advance_ctu() 10 0.0001103 0.0001103 0.0001103 0.01% MLMG:computeResOfCorrection() 405 0.0001085 0.0001085 0.0001085 0.01% MLMG::actualBottomSolve() 81 8.583e-05 8.583e-05 8.583e-05 0.01% Castro::do_new_sources() 10 8.511e-05 8.511e-05 8.511e-05 0.01% MLMG::mgVcycle_down::0 81 8.367e-05 8.367e-05 8.367e-05 0.01% Castro::initialize_do_advance() 10 8.155e-05 8.155e-05 8.155e-05 0.01% MLMG::mgVcycle_down::1 81 7.965e-05 7.965e-05 7.965e-05 0.01% MLMG::mgVcycle_down::2 81 7.703e-05 7.703e-05 7.703e-05 0.01% Castro::initialize_advance() 10 7.695e-05 7.695e-05 7.695e-05 0.01% MLMG::solve() 11 7.486e-05 7.486e-05 7.486e-05 0.01% MLMG::mgVcycle_down::4 81 7.21e-05 7.21e-05 7.21e-05 0.01% MLMG::mgVcycle_down::3 81 6.924e-05 6.924e-05 6.924e-05 0.01% Castro::clean_state() 62 6.369e-05 6.369e-05 6.369e-05 0.01% AmrLevel::checkPoint() 3 6.144e-05 6.144e-05 6.144e-05 0.01% MLMG::mgVcycle_up::4 81 5.468e-05 5.468e-05 5.468e-05 0.01% MLMG::oneIter() 81 5.041e-05 5.041e-05 5.041e-05 0.01% FillPatchIterator::FillFromLevel0() 41 5.036e-05 5.036e-05 5.036e-05 0.01% MLMG::mgVcycle_up::0 81 4.724e-05 4.724e-05 4.724e-05 0.01% MLMG::mgVcycle_up::1 81 4.701e-05 4.701e-05 4.701e-05 0.01% MLMG::mgVcycle_up::3 81 4.681e-05 4.681e-05 4.681e-05 0.01% Castro::construct_old_source() 50 4.636e-05 4.636e-05 4.636e-05 0.01% Castro::finalize_do_advance() 10 4.626e-05 4.626e-05 4.626e-05 0.01% MLMG::mgVcycle_up::2 81 4.514e-05 4.514e-05 4.514e-05 0.01% MLCellLinOp::solutionResidual() 92 4.349e-05 4.349e-05 4.349e-05 0.00% FillPatchSingleLevel 41 3.707e-05 3.707e-05 3.707e-05 0.00% Castro::construct_new_gravity() 10 3.415e-05 3.415e-05 3.415e-05 0.00% MLMG::ResNormInf() 92 3.272e-05 3.272e-05 3.272e-05 0.00% MLMG::mgVcycle_bottom 81 3.226e-05 3.226e-05 3.226e-05 0.00% MLMG::computeResidual() 81 3.201e-05 3.201e-05 3.201e-05 0.00% Amr::defBaseLevel() 1 2.942e-05 2.942e-05 2.942e-05 0.00% Castro::do_old_sources() 10 2.177e-05 2.177e-05 2.177e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.173e-05 2.173e-05 2.173e-05 0.00% MLPoisson::define() 11 1.92e-05 1.92e-05 1.92e-05 0.00% Amr::FinalizeInit() 1 1.83e-05 1.83e-05 1.83e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.685e-05 1.685e-05 1.685e-05 0.00% MLPoisson::prepareForSolve() 11 1.447e-05 1.447e-05 1.447e-05 0.00% Castro::apply_source_to_state() 20 1.205e-05 1.205e-05 1.205e-05 0.00% Castro::check_for_nan() 20 1.181e-05 1.181e-05 1.181e-05 0.00% Castro::construct_old_gravity() 10 1.126e-05 1.126e-05 1.126e-05 0.00% Castro::post_init() 1 1.076e-05 1.076e-05 1.076e-05 0.00% Gravity::actual_multilevel_solve() 1 8.568e-06 8.568e-06 8.568e-06 0.00% MLMG::computeMLResidual() 11 8.395e-06 8.395e-06 8.395e-06 0.00% Castro::post_timestep() 10 7.64e-06 7.64e-06 7.64e-06 0.00% Castro::computeNewDt() 9 6.757e-06 6.757e-06 6.757e-06 0.00% Castro::expand_state() 10 6.145e-06 6.145e-06 6.145e-06 0.00% MLMG::getGradSolution() 11 5.797e-06 5.797e-06 5.797e-06 0.00% Amr::InitializeInit() 1 5.236e-06 5.236e-06 5.236e-06 0.00% Amr::init() 1 2.565e-06 2.565e-06 2.565e-06 0.00% Amr::initialInit() 1 1.024e-06 1.024e-06 1.024e-06 0.00% Other 4767 0.002919 0.002919 0.002919 0.33% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8848 0.8848 0.8848 100.00% Amr::coarseTimeStep() 10 0.7373 0.7373 0.7373 83.33% Amr::timeStep() 10 0.6313 0.6313 0.6313 71.35% Castro::advance() 10 0.6206 0.6206 0.6206 70.14% Castro::subcycle_advance_ctu() 10 0.6076 0.6076 0.6076 68.68% Castro::do_advance_ctu() 10 0.6075 0.6075 0.6075 68.66% Gravity::solve_phi_with_mlmg() 11 0.3016 0.3016 0.3016 34.09% Gravity::actual_solve_with_mlmg() 11 0.2937 0.2937 0.2937 33.19% Castro::construct_new_gravity() 10 0.2786 0.2786 0.2786 31.49% MLMG::solve() 11 0.2719 0.2719 0.2719 30.73% Gravity::solve_for_phi() 10 0.2611 0.2611 0.2611 29.51% MLMG::oneIter() 81 0.2565 0.2565 0.2565 28.99% MLMG::mgVcycle() 81 0.2527 0.2527 0.2527 28.56% Castro::construct_ctu_hydro_source() 10 0.25 0.25 0.25 28.25% VisMF::Write(FabArray) 11 0.1798 0.1798 0.1798 20.32% Amr::checkPoint() 3 0.1482 0.1482 0.1482 16.75% AmrLevel::checkPoint() 3 0.1307 0.1307 0.1307 14.77% StateData::checkPoint() 12 0.1307 0.1307 0.1307 14.77% MLCellLinOp::smooth() 1620 0.1242 0.1242 0.1242 14.04% Amr::init() 1 0.1209 0.1209 0.1209 13.67% MLCellLinOp::applyBC() 4298 0.1156 0.1156 0.1156 13.06% MLMG::mgVcycle_bottom 81 0.07269 0.07269 0.07269 8.22% MLMG::actualBottomSolve() 81 0.07266 0.07266 0.07266 8.21% MLCGSolver::bicgstab 81 0.07184 0.07184 0.07184 8.12% Castro::clean_state() 62 0.06193 0.06193 0.06193 7.00% Amr::writePlotFile() 2 0.05162 0.05162 0.05162 5.83% Amr::initialInit() 1 0.04938 0.04938 0.04938 5.58% AmrLevel::FillPatch() 41 0.04922 0.04922 0.04922 5.56% FillPatchIterator::Initialize 41 0.04492 0.04492 0.04492 5.08% Amr::FinalizeInit() 1 0.04418 0.04418 0.04418 4.99% FillPatchIterator::FillFromLevel0() 41 0.0433 0.0433 0.0433 4.89% Castro::post_init() 1 0.04325 0.04325 0.04325 4.89% FillPatchSingleLevel 41 0.04325 0.04325 0.04325 4.89% Castro::computeTemp() 63 0.04301 0.04301 0.04301 4.86% MLMG::mgVcycle_down::0 81 0.04251 0.04251 0.04251 4.81% Gravity::multilevel_solve_for_new_phi() 1 0.04099 0.04099 0.04099 4.63% Gravity::actual_multilevel_solve() 1 0.04097 0.04097 0.04097 4.63% StateDataPhysBCFunct::() 41 0.0391 0.0391 0.0391 4.42% MLCellLinOp::apply() 1047 0.03676 0.03676 0.03676 4.15% FabArray::FillBoundary() 3893 0.03459 0.03459 0.03459 3.91% MLPoisson::Fsmooth() 3240 0.03389 0.03389 0.03389 3.83% FillBoundary_nowait() 3893 0.03384 0.03384 0.03384 3.82% MLMG::mgVcycle_up::0 81 0.0272 0.0272 0.0272 3.07% StateData::FillBoundary(geom) 328 0.02612 0.02612 0.02612 2.95% Castro::reset_internal_energy(MultiFab) 63 0.02396 0.02396 0.02396 2.71% Castro::initialize_do_advance() 10 0.02318 0.02318 0.02318 2.62% amrex::Dot() 1100 0.02136 0.02136 0.02136 2.41% MLMG:computeResOfCorrection() 405 0.02073 0.02073 0.02073 2.34% MLCellLinOp::correctionResidual() 405 0.02062 0.02062 0.02062 2.33% Castro::do_old_sources() 10 0.02058 0.02058 0.02058 2.33% FabArray::norminf() 1048 0.01994 0.01994 0.01994 2.25% Gravity::get_new_grav_vector() 11 0.01931 0.01931 0.01931 2.18% amrex::average_down 405 0.01808 0.01808 0.01808 2.04% MLPoisson::define() 11 0.01697 0.01697 0.01697 1.92% MLMG::mgVcycle_down::1 81 0.01679 0.01679 0.01679 1.90% Castro::do_new_sources() 10 0.01657 0.01657 0.01657 1.87% MLMG::mgVcycle_down::2 81 0.01559 0.01559 0.01559 1.76% Castro::construct_old_gravity() 10 0.01528 0.01528 0.01528 1.73% Gravity::get_old_grav_vector() 10 0.01526 0.01526 0.01526 1.73% MLMG::mgVcycle_down::3 81 0.01515 0.01515 0.01515 1.71% MLMG::mgVcycle_down::4 81 0.01506 0.01506 0.01506 1.70% FabArray::ParallelCopy() 851 0.01483 0.01483 0.01483 1.68% FabArray::ParallelCopy_nowait() 851 0.01456 0.01456 0.01456 1.65% FabArray::setVal() 1054 0.01349 0.01349 0.01349 1.52% FabArray::Saxpy() 1353 0.01313 0.01313 0.01313 1.48% MLCGSolver::ParallelAllReduce 1809 0.01278 0.01278 0.01278 1.44% MLMG::addInterpCorrection() 405 0.01228 0.01228 0.01228 1.39% Castro::initialize_advance() 10 0.01218 0.01218 0.01218 1.38% MLMG::mgVcycle_up::1 81 0.01211 0.01211 0.01211 1.37% MLMG::mgVcycle_up::4 81 0.01202 0.01202 0.01202 1.36% Castro::expand_state() 10 0.01184 0.01184 0.01184 1.34% MLMG::mgVcycle_up::2 81 0.01181 0.01181 0.01181 1.34% MLMG::mgVcycle_up::3 81 0.01155 0.01155 0.01155 1.31% MLCellLinOp::defineAuxData() 11 0.01153 0.01153 0.01153 1.30% amrex::Copy() 469 0.01101 0.01101 0.01101 1.24% Castro::normalize_species() 62 0.0109 0.0109 0.0109 1.23% Castro::post_timestep() 10 0.01054 0.01054 0.01054 1.19% MLPoisson::Fapply() 1047 0.01042 0.01042 0.01042 1.18% Castro::enforce_min_density() 62 0.008696 0.008696 0.008696 0.98% MLCellLinOp::solutionResidual() 92 0.007938 0.007938 0.007938 0.90% FabArray::Xpay() 730 0.007861 0.007861 0.007861 0.89% Gravity::fill_multipole_BCs() 11 0.0077 0.0077 0.0077 0.87% MLMG::computeResidual() 81 0.006598 0.006598 0.006598 0.75% Castro::estTimeStep() 21 0.005876 0.005876 0.005876 0.66% Amr::InitializeInit() 1 0.005197 0.005197 0.005197 0.59% Amr::defBaseLevel() 1 0.005192 0.005192 0.005192 0.59% MLCellLinOp::defineBC() 11 0.005181 0.005181 0.005181 0.59% MLMG::prepareForSolve() 11 0.004924 0.004924 0.004924 0.56% BndryData::define() 11 0.004917 0.004917 0.004917 0.56% Castro::initData() 1 0.004522 0.004522 0.004522 0.51% Castro::construct_new_source() 50 0.004063 0.004063 0.004063 0.46% Castro::construct_new_gravity_source() 10 0.003791 0.003791 0.003791 0.43% amrex::Add() 81 0.003725 0.003725 0.003725 0.42% Castro::construct_old_source() 50 0.003254 0.003254 0.003254 0.37% Castro::construct_old_gravity_source() 10 0.003208 0.003208 0.003208 0.36% Castro::reset_internal_energy(Fab) 504 0.002825 0.002825 0.002825 0.32% Castro::computeNewDt() 9 0.002736 0.002736 0.002736 0.31% Castro::finalize_do_advance() 10 0.002274 0.002274 0.002274 0.26% MLMG::ResNormInf() 92 0.002232 0.002232 0.002232 0.25% Castro::apply_source_to_state() 20 0.001882 0.001882 0.001882 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001793 0.001793 0.001793 0.20% MLMG::getGradSolution() 11 0.001642 0.001642 0.001642 0.19% MLCellLinOp::compGrad() 11 0.001636 0.001636 0.001636 0.18% MLCellLinOp::setLevelBC() 11 0.001608 0.001608 0.001608 0.18% FabArrayBase::getCPC() 1313 0.001503 0.001503 0.001503 0.17% check_for_negative_density() 10 0.001424 0.001424 0.001424 0.16% FabArray::mult() 43 0.001391 0.001391 0.001391 0.16% FabArray::setDomainBndry() 41 0.001384 0.001384 0.001384 0.16% MLMG::computeMLResidual() 11 0.00138 0.00138 0.00138 0.16% MLPoisson::prepareForSolve() 11 0.001331 0.001331 0.001331 0.15% MLCellLinOp::prepareForSolve() 11 0.001317 0.001317 0.001317 0.15% Castro::check_for_nan() 20 0.001308 0.001308 0.001308 0.15% MultiFab::contains_nan() 20 0.001296 0.001296 0.001296 0.15% Other 4767 0.008207 0.008207 0.008207 0.93% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 6076 KiB 9037 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1068 KiB 39 MiB Castro::initialize_do_advance() 80 80 26 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1637 KiB 28 MiB Castro::initialize_advance() 80 80 16 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7618 KiB 14 MiB MLMG::prepareForSolve() 649 649 3784 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 226 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 176 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7521 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 13 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2209 B 2048 KiB Gravity::solve_for_phi() 80 80 603 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 94 KiB 2048 KiB BndryData::define() 1056 1056 345 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 219 KiB 671 KiB Castro::estTimeStep() 21 21 2875 B 480 KiB VisMF::Write(FabArray) 656 656 3582 B 320 KiB Castro::normalize_species() 62 62 4002 B 320 KiB amrex::average_down 1054 1054 3558 B 257 KiB MLMG::addInterpCorrection() 1053 1053 1200 B 257 KiB amrex::Dot() 1343 1343 3566 B 160 KiB FabArray::norminf() 1129 1129 3479 B 160 KiB check_for_negative_density() 10 10 257 B 160 KiB Castro::initData() 1 1 53 B 160 KiB MultiFab::max() 11 11 59 B 160 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MultiFab::contains_nan() 20 20 29 B 20 KiB MLPoisson::Fsmooth() 132 132 3695 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 45 B 10 KiB FillBoundary_nowait() 751 751 327 B 9648 B MLCellLinOp::applyBC() 8596 8596 229 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3956 B 6144 B StateData::FillBoundary(geom) 1992 1992 47 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 389 B 1248 B MLCGSolver::bicgstab 405 405 98 B 1216 B MLPoisson::Fapply() 11 11 310 B 1024 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 692 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 41 KiB 8192 KiB VisMF::Write(FabArray) 744 744 446 KiB 3584 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MLPoisson::Fsmooth() 132 132 3695 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 45 B 10 KiB FillBoundary_nowait() 751 751 327 B 9648 B MLCellLinOp::applyBC() 4298 4298 227 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3956 B 6144 B Gravity::get_new_grav_vector() 3 3 2903 B 3072 B StateData::FillBoundary(geom) 1992 1992 48 B 3024 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B amrex::average_down 82 82 625 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLMG::addInterpCorrection() 81 81 2 B 1024 B MLPoisson::Fapply() 11 11 310 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1343 1343 26 B 400 B FabArray::norminf() 1129 1129 10 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2109 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.07-29-g3cb7f1555494) finalized Initializing AMReX (24.07-29-g3cb7f1555494)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.07-29-g3cb7f1555494) initialized Starting run at 08:02:15 UTC on 2024-07-24. Successfully read inputs file ... Castro git describe: 24.07-18-g56f51f6c1 AMReX git describe: 24.07-29-g3cb7f1555 Microphysics git describe: 24.07-31-g4c8331df reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.499175249 Restart time = 0.072902129 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.067264771 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.054322715 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.062058182 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.0741404 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.077896259 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.028432882 seconds Ending run at 08:02:16 UTC on 2024-07-24. Run time = 0.437770325 Run time without initialization = 0.364485083 Average number of zones advanced per microsecond: 3.596 Average number of zones advanced per microsecond per rank: 3.596 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.4378 ... 0.4378 ... 0.4378 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1338 0.1338 0.1338 30.56% VisMF::Read() 3 0.06142 0.06142 0.06142 14.03% MLCellLinOp::applyBC() 1910 0.0368 0.0368 0.0368 8.41% VisMF::Write(FabArray) 1 0.02596 0.02596 0.02596 5.93% MLPoisson::Fsmooth() 1440 0.01536 0.01536 0.01536 3.51% FillBoundary_nowait() 1730 0.01478 0.01478 0.01478 3.38% StateData::FillBoundary(geom) 160 0.01305 0.01305 0.01305 2.98% Castro::reset_internal_energy(MultiFab) 30 0.01078 0.01078 0.01078 2.46% amrex::Dot() 484 0.009448 0.009448 0.009448 2.16% FabArray::norminf() 465 0.008894 0.008894 0.008894 2.03% FabArray::setVal() 501 0.006802 0.006802 0.006802 1.55% Castro::normalize_species() 30 0.006743 0.006743 0.006743 1.54% FabArray::ParallelCopy_nowait() 380 0.006324 0.006324 0.006324 1.44% StateDataPhysBCFunct::() 20 0.00605 0.00605 0.00605 1.38% FabArray::Saxpy() 597 0.006008 0.006008 0.006008 1.37% Castro::computeTemp() 30 0.005979 0.005979 0.005979 1.37% amrex::Copy() 221 0.005601 0.005601 0.005601 1.28% MLCellLinOp::defineAuxData() 6 0.005538 0.005538 0.005538 1.26% Amr::restart() 1 0.004995 0.004995 0.004995 1.14% Castro::enforce_min_density() 30 0.004786 0.004786 0.004786 1.09% MLPoisson::Fapply() 464 0.004734 0.004734 0.004734 1.08% Gravity::fill_multipole_BCs() 6 0.004505 0.004505 0.004505 1.03% FabArray::Xpay() 325 0.003572 0.003572 0.003572 0.82% MLMG::addInterpCorrection() 180 0.003209 0.003209 0.003209 0.73% amrex::average_down 180 0.002991 0.002991 0.002991 0.68% Castro::estTimeStep() 10 0.002546 0.002546 0.002546 0.58% Amr::writePlotFile() 1 0.002261 0.002261 0.002261 0.52% Castro::reset_internal_energy(Fab) 240 0.002193 0.002193 0.002193 0.50% BndryData::define() 6 0.002167 0.002167 0.002167 0.49% Castro::construct_new_gravity_source() 5 0.001936 0.001936 0.001936 0.44% amrex::Add() 36 0.001653 0.001653 0.001653 0.38% Castro::construct_old_gravity_source() 5 0.001565 0.001565 0.001565 0.36% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009898 0.0009898 0.0009898 0.23% MLCGSolver::bicgstab 36 0.000895 0.000895 0.000895 0.20% MLCellLinOp::setLevelBC() 6 0.0008917 0.0008917 0.0008917 0.20% check_for_negative_density() 5 0.0008901 0.0008901 0.0008901 0.20% Gravity::actual_solve_with_mlmg() 6 0.0008432 0.0008432 0.0008432 0.19% MLCellLinOp::prepareForSolve() 6 0.0007771 0.0007771 0.0007771 0.18% MLCellLinOp::smooth() 720 0.0007149 0.0007149 0.0007149 0.16% FabArray::mult() 22 0.0007126 0.0007126 0.0007126 0.16% FabArray::setDomainBndry() 20 0.0007005 0.0007005 0.0007005 0.16% MLCellLinOp::compGrad() 6 0.0006152 0.0006152 0.0006152 0.14% MLMG::prepareForSolve() 6 0.0004003 0.0004003 0.0004003 0.09% FabArrayBase::getCPC() 632 0.0003914 0.0003914 0.0003914 0.09% FabArray::FillBoundary() 1730 0.0003526 0.0003526 0.0003526 0.08% Gravity::get_old_grav_vector() 5 0.0003143 0.0003143 0.0003143 0.07% main() 1 0.0002552 0.0002552 0.0002552 0.06% Gravity::get_new_grav_vector() 5 0.000234 0.000234 0.000234 0.05% MLCellLinOp::apply() 464 0.0002243 0.0002243 0.0002243 0.05% AmrLevel::FillPatch() 20 0.0002045 0.0002045 0.0002045 0.05% MLCGSolver::ParallelAllReduce 798 0.000179 0.000179 0.000179 0.04% Amr::coarseTimeStep() 5 0.0001545 0.0001545 0.0001545 0.04% MLCellLinOp::defineBC() 6 0.0001473 0.0001473 0.0001473 0.03% FabArray::ParallelCopy() 380 0.0001236 0.0001236 0.0001236 0.03% FillPatchIterator::Initialize 20 0.0001107 0.0001107 0.0001107 0.03% Castro::subcycle_advance_ctu() 5 9.973e-05 9.973e-05 9.973e-05 0.02% MLCellLinOp::correctionResidual() 180 9.495e-05 9.495e-05 9.495e-05 0.02% MLMG::mgVcycle() 36 9.203e-05 9.203e-05 9.203e-05 0.02% Amr::timeStep() 5 8.391e-05 8.391e-05 8.391e-05 0.02% Castro::do_advance_ctu() 5 8.287e-05 8.287e-05 8.287e-05 0.02% AmrLevel::restart() 1 7.148e-05 7.148e-05 7.148e-05 0.02% Castro::construct_new_source() 25 7.027e-05 7.027e-05 7.027e-05 0.02% Castro::advance() 5 6.782e-05 6.782e-05 6.782e-05 0.02% StateData::restartDoit() 4 6.501e-05 6.501e-05 6.501e-05 0.01% Castro::construct_old_source() 25 6.26e-05 6.26e-05 6.26e-05 0.01% Gravity::update_max_rhs() 6 6.146e-05 6.146e-05 6.146e-05 0.01% MLMG:computeResOfCorrection() 180 5.752e-05 5.752e-05 5.752e-05 0.01% Castro::finalize_do_advance() 5 5.538e-05 5.538e-05 5.538e-05 0.01% Castro::initialize_do_advance() 5 5.377e-05 5.377e-05 5.377e-05 0.01% Gravity::solve_for_phi() 5 4.923e-05 4.923e-05 4.923e-05 0.01% MLMG::mgVcycle_down::0 36 4.827e-05 4.827e-05 4.827e-05 0.01% MLMG::actualBottomSolve() 36 4.573e-05 4.573e-05 4.573e-05 0.01% MLMG::mgVcycle_down::1 36 4.552e-05 4.552e-05 4.552e-05 0.01% MLMG::mgVcycle_down::2 36 4.285e-05 4.285e-05 4.285e-05 0.01% MLMG::mgVcycle_down::4 36 4.222e-05 4.222e-05 4.222e-05 0.01% MLMG::mgVcycle_down::3 36 4.102e-05 4.102e-05 4.102e-05 0.01% MLMG::solve() 6 4.011e-05 4.011e-05 4.011e-05 0.01% Castro::initialize_advance() 5 3.883e-05 3.883e-05 3.883e-05 0.01% Castro::clean_state() 30 3.713e-05 3.713e-05 3.713e-05 0.01% MLMG::mgVcycle_up::4 36 3.092e-05 3.092e-05 3.092e-05 0.01% MLMG::mgVcycle_up::3 36 2.902e-05 2.902e-05 2.902e-05 0.01% MLMG::oneIter() 36 2.833e-05 2.833e-05 2.833e-05 0.01% Castro::post_restart() 1 2.755e-05 2.755e-05 2.755e-05 0.01% MLMG::mgVcycle_up::2 36 2.734e-05 2.734e-05 2.734e-05 0.01% MLMG::mgVcycle_up::0 36 2.671e-05 2.671e-05 2.671e-05 0.01% MLMG::mgVcycle_up::1 36 2.625e-05 2.625e-05 2.625e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.368e-05 2.368e-05 2.368e-05 0.01% MLCellLinOp::solutionResidual() 42 2.33e-05 2.33e-05 2.33e-05 0.01% MLMG::ResNormInf() 42 1.918e-05 1.918e-05 1.918e-05 0.00% FillPatchSingleLevel 20 1.801e-05 1.801e-05 1.801e-05 0.00% MLMG::computeResidual() 36 1.6e-05 1.6e-05 1.6e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.579e-05 1.579e-05 1.579e-05 0.00% MLMG::mgVcycle_bottom 36 1.455e-05 1.455e-05 1.455e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.346e-05 1.346e-05 1.346e-05 0.00% MLPoisson::define() 6 1.267e-05 1.267e-05 1.267e-05 0.00% Castro::do_old_sources() 5 1.248e-05 1.248e-05 1.248e-05 0.00% Castro::construct_new_gravity() 5 1.224e-05 1.224e-05 1.224e-05 0.00% Castro::do_new_sources() 5 1.102e-05 1.102e-05 1.102e-05 0.00% MLPoisson::prepareForSolve() 6 1.078e-05 1.078e-05 1.078e-05 0.00% Castro::expand_state() 5 9.85e-06 9.85e-06 9.85e-06 0.00% Gravity::actual_multilevel_solve() 1 8.433e-06 8.433e-06 8.433e-06 0.00% Castro::apply_source_to_state() 10 6.227e-06 6.227e-06 6.227e-06 0.00% Castro::construct_old_gravity() 5 5.767e-06 5.767e-06 5.767e-06 0.00% Castro::post_timestep() 5 4.364e-06 4.364e-06 4.364e-06 0.00% MLMG::computeMLResidual() 6 3.964e-06 3.964e-06 3.964e-06 0.00% MLMG::getGradSolution() 6 3.543e-06 3.543e-06 3.543e-06 0.00% Castro::computeNewDt() 5 3.279e-06 3.279e-06 3.279e-06 0.00% Amr::init() 1 9.05e-07 9.05e-07 9.05e-07 0.00% Other 2200 0.002314 0.002314 0.002314 0.53% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4378 0.4378 0.4378 99.99% Amr::coarseTimeStep() 5 0.3358 0.3358 0.3358 76.71% Amr::timeStep() 5 0.3341 0.3341 0.3341 76.32% Castro::advance() 5 0.3278 0.3278 0.3278 74.87% Castro::subcycle_advance_ctu() 5 0.3201 0.3201 0.3201 73.11% Castro::do_advance_ctu() 5 0.32 0.32 0.32 73.09% Castro::construct_new_gravity() 5 0.1406 0.1406 0.1406 32.12% Castro::construct_ctu_hydro_source() 5 0.1398 0.1398 0.1398 31.93% Gravity::solve_phi_with_mlmg() 6 0.1378 0.1378 0.1378 31.48% Gravity::actual_solve_with_mlmg() 6 0.133 0.133 0.133 30.39% Gravity::solve_for_phi() 5 0.1325 0.1325 0.1325 30.25% MLMG::solve() 6 0.121 0.121 0.121 27.63% MLMG::oneIter() 36 0.1131 0.1131 0.1131 25.83% MLMG::mgVcycle() 36 0.1114 0.1114 0.1114 25.45% Amr::init() 1 0.07295 0.07295 0.07295 16.66% Amr::restart() 1 0.07294 0.07294 0.07294 16.66% AmrLevel::restart() 1 0.06179 0.06179 0.06179 14.11% StateData::restartDoit() 4 0.06171 0.06171 0.06171 14.10% VisMF::Read() 3 0.06142 0.06142 0.06142 14.03% MLCellLinOp::smooth() 720 0.05614 0.05614 0.05614 12.82% MLCellLinOp::applyBC() 1910 0.05227 0.05227 0.05227 11.94% MLMG::mgVcycle_bottom 36 0.03264 0.03264 0.03264 7.46% MLMG::actualBottomSolve() 36 0.03263 0.03263 0.03263 7.45% MLCGSolver::bicgstab 36 0.03225 0.03225 0.03225 7.37% Castro::clean_state() 30 0.03057 0.03057 0.03057 6.98% Amr::writePlotFile() 1 0.0285 0.0285 0.0285 6.51% VisMF::Write(FabArray) 1 0.02596 0.02596 0.02596 5.93% AmrLevel::FillPatch() 20 0.02414 0.02414 0.02414 5.51% FillPatchIterator::Initialize 20 0.02202 0.02202 0.02202 5.03% FillPatchIterator::FillFromLevel0() 20 0.02121 0.02121 0.02121 4.84% FillPatchSingleLevel 20 0.02118 0.02118 0.02118 4.84% StateDataPhysBCFunct::() 20 0.0191 0.0191 0.0191 4.36% Castro::computeTemp() 30 0.01895 0.01895 0.01895 4.33% MLCellLinOp::apply() 464 0.01686 0.01686 0.01686 3.85% MLMG::mgVcycle_down::0 36 0.01637 0.01637 0.01637 3.74% FabArray::FillBoundary() 1730 0.01547 0.01547 0.01547 3.53% MLPoisson::Fsmooth() 1440 0.01536 0.01536 0.01536 3.51% FillBoundary_nowait() 1730 0.01512 0.01512 0.01512 3.45% StateData::FillBoundary(geom) 160 0.01305 0.01305 0.01305 2.98% Castro::reset_internal_energy(MultiFab) 30 0.01297 0.01297 0.01297 2.96% Castro::initialize_do_advance() 5 0.01244 0.01244 0.01244 2.84% MLMG::mgVcycle_up::0 36 0.01231 0.01231 0.01231 2.81% Castro::do_old_sources() 5 0.01023 0.01023 0.01023 2.34% amrex::Dot() 484 0.009448 0.009448 0.009448 2.16% MLMG:computeResOfCorrection() 180 0.009429 0.009429 0.009429 2.15% MLPoisson::define() 6 0.009413 0.009413 0.009413 2.15% MLCellLinOp::correctionResidual() 180 0.009372 0.009372 0.009372 2.14% FabArray::norminf() 465 0.008894 0.008894 0.008894 2.03% Gravity::get_new_grav_vector() 5 0.008026 0.008026 0.008026 1.83% Castro::construct_old_gravity() 5 0.007781 0.007781 0.007781 1.78% Gravity::get_old_grav_vector() 5 0.007775 0.007775 0.007775 1.78% MLMG::mgVcycle_down::1 36 0.007739 0.007739 0.007739 1.77% Castro::do_new_sources() 5 0.007516 0.007516 0.007516 1.72% Castro::initialize_advance() 5 0.007298 0.007298 0.007298 1.67% MLMG::mgVcycle_down::2 36 0.00704 0.00704 0.00704 1.61% MLMG::mgVcycle_down::3 36 0.006865 0.006865 0.006865 1.57% FabArray::ParallelCopy() 380 0.006852 0.006852 0.006852 1.57% FabArray::setVal() 501 0.006802 0.006802 0.006802 1.55% MLMG::mgVcycle_down::4 36 0.006784 0.006784 0.006784 1.55% Castro::normalize_species() 30 0.006743 0.006743 0.006743 1.54% Castro::expand_state() 5 0.006738 0.006738 0.006738 1.54% FabArray::ParallelCopy_nowait() 380 0.006728 0.006728 0.006728 1.54% MLCellLinOp::defineAuxData() 6 0.006344 0.006344 0.006344 1.45% Castro::post_timestep() 5 0.006286 0.006286 0.006286 1.44% FabArray::Saxpy() 597 0.006008 0.006008 0.006008 1.37% Castro::post_restart() 1 0.005971 0.005971 0.005971 1.36% MLCGSolver::ParallelAllReduce 798 0.005738 0.005738 0.005738 1.31% amrex::Copy() 221 0.005601 0.005601 0.005601 1.28% Gravity::multilevel_solve_for_new_phi() 1 0.005599 0.005599 0.005599 1.28% MLMG::addInterpCorrection() 180 0.005587 0.005587 0.005587 1.28% Gravity::actual_multilevel_solve() 1 0.005583 0.005583 0.005583 1.28% MLMG::mgVcycle_up::1 36 0.005487 0.005487 0.005487 1.25% MLMG::mgVcycle_up::4 36 0.005459 0.005459 0.005459 1.25% amrex::average_down 180 0.005398 0.005398 0.005398 1.23% MLMG::mgVcycle_up::2 36 0.005365 0.005365 0.005365 1.23% MLMG::mgVcycle_up::3 36 0.005244 0.005244 0.005244 1.20% Castro::enforce_min_density() 30 0.004786 0.004786 0.004786 1.09% MLPoisson::Fapply() 464 0.004734 0.004734 0.004734 1.08% Gravity::fill_multipole_BCs() 6 0.004644 0.004644 0.004644 1.06% MLCellLinOp::solutionResidual() 42 0.003846 0.003846 0.003846 0.88% FabArray::Xpay() 325 0.003572 0.003572 0.003572 0.82% MLMG::computeResidual() 36 0.002978 0.002978 0.002978 0.68% MLCellLinOp::defineBC() 6 0.002921 0.002921 0.002921 0.67% MLMG::prepareForSolve() 6 0.002801 0.002801 0.002801 0.64% BndryData::define() 6 0.002774 0.002774 0.002774 0.63% Castro::estTimeStep() 10 0.002546 0.002546 0.002546 0.58% Castro::reset_internal_energy(Fab) 240 0.002193 0.002193 0.002193 0.50% Castro::construct_new_source() 25 0.002006 0.002006 0.002006 0.46% Castro::construct_new_gravity_source() 5 0.001936 0.001936 0.001936 0.44% amrex::Add() 36 0.001653 0.001653 0.001653 0.38% Castro::construct_old_source() 25 0.001628 0.001628 0.001628 0.37% Castro::construct_old_gravity_source() 5 0.001565 0.001565 0.001565 0.36% Castro::computeNewDt() 5 0.001536 0.001536 0.001536 0.35% Castro::finalize_do_advance() 5 0.001069 0.001069 0.001069 0.24% MLMG::ResNormInf() 42 0.00105 0.00105 0.00105 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009898 0.0009898 0.0009898 0.23% Castro::apply_source_to_state() 10 0.0009757 0.0009757 0.0009757 0.22% MLMG::getGradSolution() 6 0.0009233 0.0009233 0.0009233 0.21% MLCellLinOp::compGrad() 6 0.0009198 0.0009198 0.0009198 0.21% MLCellLinOp::setLevelBC() 6 0.0008917 0.0008917 0.0008917 0.20% check_for_negative_density() 5 0.0008901 0.0008901 0.0008901 0.20% MLMG::computeMLResidual() 6 0.0008878 0.0008878 0.0008878 0.20% FabArrayBase::getCPC() 632 0.0008279 0.0008279 0.0008279 0.19% MLPoisson::prepareForSolve() 6 0.0007879 0.0007879 0.0007879 0.18% MLCellLinOp::prepareForSolve() 6 0.0007771 0.0007771 0.0007771 0.18% Gravity::update_max_rhs() 6 0.0007282 0.0007282 0.0007282 0.17% FabArray::mult() 22 0.0007126 0.0007126 0.0007126 0.16% FabArray::setDomainBndry() 20 0.0007005 0.0007005 0.0007005 0.16% Other 2200 0.004283 0.004283 0.004283 0.98% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 11 MiB 9037 MiB Castro::initMFs() 48 48 57 MiB 68 MiB Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB StateData::restartDoit() 32 32 52 MiB 55 MiB FillPatchIterator::Initialize 160 160 1127 KiB 39 MiB Castro::initialize_do_advance() 40 40 28 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1749 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6415 KiB 14 MiB MLMG::prepareForSolve() 354 354 3396 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 184 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 187 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6402 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 16 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3318 B 2048 KiB Gravity::solve_for_phi() 40 40 618 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 26 KiB 2048 KiB BndryData::define() 576 576 312 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 200 KiB 671 KiB Castro::estTimeStep() 10 10 2728 B 480 KiB VisMF::Write(FabArray) 112 112 1326 B 320 KiB Castro::normalize_species() 30 30 5008 B 320 KiB amrex::average_down 469 469 1489 B 257 KiB MLMG::addInterpCorrection() 468 468 1098 B 257 KiB amrex::Dot() 592 592 3183 B 160 KiB FabArray::norminf() 501 501 3140 B 160 KiB check_for_negative_density() 5 5 326 B 160 KiB MultiFab::max() 6 6 76 B 160 KiB FabArray::setVal() 67 67 21 KiB 28 KiB MultiFab::contains_nan() 10 10 31 B 20 KiB MLPoisson::Fsmooth() 60 60 3291 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 45 B 10 KiB FillBoundary_nowait() 336 336 294 B 9648 B MLCellLinOp::applyBC() 3820 3820 211 B 9344 B amrex::Copy() 56 56 5880 B 8816 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B StateData::FillBoundary(geom) 960 960 45 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCellLinOp::defineBC() 36 36 351 B 1248 B MLCGSolver::bicgstab 180 180 88 B 1216 B MLPoisson::Fapply() 6 6 277 B 1024 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1797 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 83 KiB 8192 KiB VisMF::Write(FabArray) 120 120 167 KiB 3584 KiB VisMF::Read() 24 24 212 KiB 3000 KiB FabArray::setVal() 67 67 21 KiB 28 KiB MLPoisson::Fsmooth() 60 60 3291 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 45 B 10 KiB FillBoundary_nowait() 336 336 294 B 9648 B MLCellLinOp::applyBC() 1910 1910 210 B 9328 B amrex::Copy() 56 56 5880 B 8816 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B Gravity::get_old_grav_vector() 3 3 2507 B 3072 B StateData::FillBoundary(geom) 960 960 46 B 3024 B Gravity::fill_multipole_BCs() 18 18 6 B 2832 B amrex::average_down 37 37 466 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLPoisson::Fapply() 6 6 277 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 23 B 400 B FabArray::norminf() 501 501 9 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2109 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.07-29-g3cb7f1555494) finalized