Initializing AMReX (24.07-31-g11d31e5f787d)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.07-31-g11d31e5f787d) initialized Starting run at 08:07:08 UTC on 2024-07-26. Successfully read inputs file ... Castro git describe: 24.07-23-g8ffd8d763 AMReX git describe: 24.07-31-g11d31e5f7 Microphysics git describe: 24.07-33-g997a4262 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.046931397 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025731091 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.064107875 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.054203571 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.073402068 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.077660749 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.056746233 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.053335086 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.055507029 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.053287126 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.06549688 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.063503054 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.064977157 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.04541998 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.025186651 seconds Ending run at 08:07:09 UTC on 2024-07-26. Run time = 0.878699128 Run time without initialization = 0.753427178 Average number of zones advanced per microsecond: 3.479 Average number of zones advanced per microsecond per rank: 3.479 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.8787 ... 0.8787 ... 0.8787 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2254 0.2254 0.2254 25.65% VisMF::Write(FabArray) 11 0.1876 0.1876 0.1876 21.35% MLCellLinOp::applyBC() 4351 0.08189 0.08189 0.08189 9.32% FillBoundary_nowait() 3941 0.03409 0.03409 0.03409 3.88% MLPoisson::Fsmooth() 3280 0.03404 0.03404 0.03404 3.87% StateDataPhysBCFunct::() 41 0.02742 0.02742 0.02742 3.12% StateData::FillBoundary(geom) 328 0.02647 0.02647 0.02647 3.01% amrex::Dot() 1114 0.0216 0.0216 0.0216 2.46% Castro::reset_internal_energy(MultiFab) 63 0.0213 0.0213 0.0213 2.42% FabArray::norminf() 1061 0.02012 0.02012 0.02012 2.29% Castro::computeTemp() 63 0.01731 0.01731 0.01731 1.97% FabArray::ParallelCopy_nowait() 861 0.01378 0.01378 0.01378 1.57% FabArray::setVal() 1062 0.01355 0.01355 0.01355 1.54% FabArray::Saxpy() 1370 0.01315 0.01315 0.01315 1.50% Castro::normalize_species() 62 0.01155 0.01155 0.01155 1.31% amrex::Copy() 472 0.01101 0.01101 0.01101 1.25% MLPoisson::Fapply() 1060 0.01045 0.01045 0.01045 1.19% MLCellLinOp::defineAuxData() 11 0.01001 0.01001 0.01001 1.14% FabArray::Xpay() 739 0.007927 0.007927 0.007927 0.90% Castro::enforce_min_density() 62 0.007694 0.007694 0.007694 0.88% Gravity::fill_multipole_BCs() 11 0.007552 0.007552 0.007552 0.86% MLMG::addInterpCorrection() 410 0.007024 0.007024 0.007024 0.80% Amr::checkPoint() 3 0.00676 0.00676 0.00676 0.77% amrex::average_down 410 0.006282 0.006282 0.006282 0.71% Castro::estTimeStep() 21 0.006003 0.006003 0.006003 0.68% BndryData::define() 11 0.003908 0.003908 0.003908 0.44% Castro::construct_new_gravity_source() 10 0.003769 0.003769 0.003769 0.43% amrex::Add() 82 0.003755 0.003755 0.003755 0.43% Castro::construct_old_gravity_source() 10 0.003212 0.003212 0.003212 0.37% Castro::reset_internal_energy(Fab) 504 0.002692 0.002692 0.002692 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001768 0.001768 0.001768 0.20% Amr::writePlotFile() 2 0.00174 0.00174 0.00174 0.20% MLCGSolver::bicgstab 82 0.001679 0.001679 0.001679 0.19% Castro::initData() 1 0.001584 0.001584 0.001584 0.18% MLCellLinOp::setLevelBC() 11 0.001579 0.001579 0.001579 0.18% Gravity::actual_solve_with_mlmg() 11 0.001539 0.001539 0.001539 0.18% FabArray::mult() 43 0.001403 0.001403 0.001403 0.16% FabArray::setDomainBndry() 41 0.001372 0.001372 0.001372 0.16% MLCellLinOp::prepareForSolve() 11 0.001315 0.001315 0.001315 0.15% MultiFab::contains_nan() 20 0.001299 0.001299 0.001299 0.15% check_for_negative_density() 10 0.001217 0.001217 0.001217 0.14% MLCellLinOp::smooth() 1640 0.001145 0.001145 0.001145 0.13% MLCellLinOp::compGrad() 11 0.0011 0.0011 0.0011 0.13% FabArrayBase::getCPC() 1323 0.000774 0.000774 0.000774 0.09% FabArray::FillBoundary() 3941 0.0007523 0.0007523 0.0007523 0.09% MLMG::prepareForSolve() 11 0.0006852 0.0006852 0.0006852 0.08% Gravity::get_new_grav_vector() 11 0.000616 0.000616 0.000616 0.07% Gravity::get_old_grav_vector() 10 0.0004996 0.0004996 0.0004996 0.06% Castro::subcycle_advance_ctu() 10 0.0004993 0.0004993 0.0004993 0.06% AmrLevel::FillPatch() 41 0.0004165 0.0004165 0.0004165 0.05% MLCellLinOp::apply() 1060 0.0003975 0.0003975 0.0003975 0.05% Amr::coarseTimeStep() 10 0.0003248 0.0003248 0.0003248 0.04% MLCGSolver::ParallelAllReduce 1832 0.000315 0.000315 0.000315 0.04% main() 1 0.0002735 0.0002735 0.0002735 0.03% MLCellLinOp::defineBC() 11 0.0002707 0.0002707 0.0002707 0.03% FabArray::ParallelCopy() 861 0.0002417 0.0002417 0.0002417 0.03% Castro::construct_new_source() 50 0.000229 0.000229 0.000229 0.03% FillPatchIterator::Initialize 41 0.0002275 0.0002275 0.0002275 0.03% MLMG::mgVcycle() 82 0.0001986 0.0001986 0.0001986 0.02% MLCellLinOp::correctionResidual() 410 0.0001843 0.0001843 0.0001843 0.02% Castro::finalize_do_advance() 10 0.0001638 0.0001638 0.0001638 0.02% Amr::timeStep() 10 0.0001609 0.0001609 0.0001609 0.02% Gravity::solve_for_phi() 10 0.0001501 0.0001501 0.0001501 0.02% Castro::do_advance_ctu() 10 0.0001495 0.0001495 0.0001495 0.02% StateData::checkPoint() 12 0.0001326 0.0001326 0.0001326 0.02% MLMG:computeResOfCorrection() 410 0.0001159 0.0001159 0.0001159 0.01% Castro::advance() 10 0.0001087 0.0001087 0.0001087 0.01% Castro::construct_old_source() 50 9.265e-05 9.265e-05 9.265e-05 0.01% MLMG::mgVcycle_down::0 82 8.884e-05 8.884e-05 8.884e-05 0.01% MLMG::mgVcycle_down::2 82 8.687e-05 8.687e-05 8.687e-05 0.01% MLMG::actualBottomSolve() 82 8.465e-05 8.465e-05 8.465e-05 0.01% Castro::do_new_sources() 10 8.201e-05 8.201e-05 8.201e-05 0.01% Castro::initialize_advance() 10 8.129e-05 8.129e-05 8.129e-05 0.01% MLMG::mgVcycle_down::1 82 8.039e-05 8.039e-05 8.039e-05 0.01% MLMG::solve() 11 7.902e-05 7.902e-05 7.902e-05 0.01% MLMG::mgVcycle_down::4 82 7.402e-05 7.402e-05 7.402e-05 0.01% MLMG::mgVcycle_down::3 82 7.153e-05 7.153e-05 7.153e-05 0.01% AmrLevel::checkPoint() 3 6.458e-05 6.458e-05 6.458e-05 0.01% Castro::clean_state() 62 6.387e-05 6.387e-05 6.387e-05 0.01% Castro::initialize_do_advance() 10 6.086e-05 6.086e-05 6.086e-05 0.01% MLMG::mgVcycle_up::4 82 5.636e-05 5.636e-05 5.636e-05 0.01% MLMG::oneIter() 82 5.323e-05 5.323e-05 5.323e-05 0.01% MLMG::mgVcycle_up::3 82 4.888e-05 4.888e-05 4.888e-05 0.01% MLMG::mgVcycle_up::0 82 4.764e-05 4.764e-05 4.764e-05 0.01% FillPatchIterator::FillFromLevel0() 41 4.707e-05 4.707e-05 4.707e-05 0.01% MLMG::mgVcycle_up::1 82 4.684e-05 4.684e-05 4.684e-05 0.01% MLMG::mgVcycle_up::2 82 4.659e-05 4.659e-05 4.659e-05 0.01% MLCellLinOp::solutionResidual() 93 4.573e-05 4.573e-05 4.573e-05 0.01% FillPatchSingleLevel 41 3.555e-05 3.555e-05 3.555e-05 0.00% MLMG::mgVcycle_bottom 82 3.313e-05 3.313e-05 3.313e-05 0.00% MLMG::ResNormInf() 93 3.167e-05 3.167e-05 3.167e-05 0.00% MLMG::computeResidual() 82 3.087e-05 3.087e-05 3.087e-05 0.00% Castro::construct_new_gravity() 10 2.422e-05 2.422e-05 2.422e-05 0.00% Amr::defBaseLevel() 1 2.24e-05 2.24e-05 2.24e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.236e-05 2.236e-05 2.236e-05 0.00% Castro::do_old_sources() 10 2.064e-05 2.064e-05 2.064e-05 0.00% Amr::FinalizeInit() 1 1.853e-05 1.853e-05 1.853e-05 0.00% MLPoisson::define() 11 1.846e-05 1.846e-05 1.846e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.689e-05 1.689e-05 1.689e-05 0.00% MLPoisson::prepareForSolve() 11 1.315e-05 1.315e-05 1.315e-05 0.00% Castro::apply_source_to_state() 20 1.145e-05 1.145e-05 1.145e-05 0.00% Castro::check_for_nan() 20 1.126e-05 1.126e-05 1.126e-05 0.00% MLMG::computeMLResidual() 11 1.084e-05 1.084e-05 1.084e-05 0.00% Castro::construct_old_gravity() 10 1.054e-05 1.054e-05 1.054e-05 0.00% Castro::computeNewDt() 9 1.035e-05 1.035e-05 1.035e-05 0.00% Castro::post_init() 1 1.006e-05 1.006e-05 1.006e-05 0.00% Gravity::actual_multilevel_solve() 1 8.538e-06 8.538e-06 8.538e-06 0.00% Castro::post_timestep() 10 7.855e-06 7.855e-06 7.855e-06 0.00% Castro::expand_state() 10 5.795e-06 5.795e-06 5.795e-06 0.00% MLMG::getGradSolution() 11 5.628e-06 5.628e-06 5.628e-06 0.00% Amr::InitializeInit() 1 5.508e-06 5.508e-06 5.508e-06 0.00% Amr::init() 1 2.101e-06 2.101e-06 2.101e-06 0.00% Amr::initialInit() 1 1.36e-06 1.36e-06 1.36e-06 0.00% Other 4815 0.00301 0.00301 0.00301 0.34% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8787 0.8787 0.8787 100.00% Amr::coarseTimeStep() 10 0.728 0.728 0.728 82.85% Amr::timeStep() 10 0.626 0.626 0.626 71.24% Castro::advance() 10 0.6151 0.6151 0.6151 70.00% Castro::subcycle_advance_ctu() 10 0.6013 0.6013 0.6013 68.43% Castro::do_advance_ctu() 10 0.6008 0.6008 0.6008 68.37% Gravity::solve_phi_with_mlmg() 11 0.2978 0.2978 0.2978 33.89% Gravity::actual_solve_with_mlmg() 11 0.2898 0.2898 0.2898 32.98% Castro::construct_new_gravity() 10 0.2767 0.2767 0.2767 31.49% MLMG::solve() 11 0.2681 0.2681 0.2681 30.52% Gravity::solve_for_phi() 10 0.2542 0.2542 0.2542 28.93% MLMG::oneIter() 82 0.2526 0.2526 0.2526 28.75% MLMG::mgVcycle() 82 0.2488 0.2488 0.2488 28.32% Castro::construct_ctu_hydro_source() 10 0.2362 0.2362 0.2362 26.88% VisMF::Write(FabArray) 11 0.1876 0.1876 0.1876 21.35% Amr::checkPoint() 3 0.1458 0.1458 0.1458 16.59% AmrLevel::checkPoint() 3 0.139 0.139 0.139 15.82% StateData::checkPoint() 12 0.139 0.139 0.139 15.82% MLCellLinOp::smooth() 1640 0.1257 0.1257 0.1257 14.30% Amr::init() 1 0.1248 0.1248 0.1248 14.21% MLCellLinOp::applyBC() 4351 0.1175 0.1175 0.1175 13.37% MLMG::mgVcycle_bottom 82 0.07336 0.07336 0.07336 8.35% MLMG::actualBottomSolve() 82 0.07333 0.07333 0.07333 8.34% MLCGSolver::bicgstab 82 0.0725 0.0725 0.0725 8.25% AmrLevel::FillPatch() 41 0.0641 0.0641 0.0641 7.29% Castro::clean_state() 62 0.06011 0.06011 0.06011 6.84% FillPatchIterator::Initialize 41 0.05979 0.05979 0.05979 6.80% FillPatchIterator::FillFromLevel0() 41 0.05819 0.05819 0.05819 6.62% FillPatchSingleLevel 41 0.05815 0.05815 0.05815 6.62% StateDataPhysBCFunct::() 41 0.05389 0.05389 0.05389 6.13% Amr::initialInit() 1 0.05206 0.05206 0.05206 5.92% Amr::writePlotFile() 2 0.05101 0.05101 0.05101 5.81% Amr::FinalizeInit() 1 0.0474 0.0474 0.0474 5.39% Castro::post_init() 1 0.04649 0.04649 0.04649 5.29% Gravity::multilevel_solve_for_new_phi() 1 0.04409 0.04409 0.04409 5.02% Gravity::actual_multilevel_solve() 1 0.04407 0.04407 0.04407 5.02% Castro::computeTemp() 63 0.0413 0.0413 0.0413 4.70% MLCellLinOp::apply() 1060 0.03728 0.03728 0.03728 4.24% MLMG::mgVcycle_down::0 82 0.03674 0.03674 0.03674 4.18% FabArray::FillBoundary() 3941 0.03557 0.03557 0.03557 4.05% FillBoundary_nowait() 3941 0.03482 0.03482 0.03482 3.96% MLPoisson::Fsmooth() 3280 0.03404 0.03404 0.03404 3.87% MLMG::mgVcycle_up::0 82 0.02779 0.02779 0.02779 3.16% StateData::FillBoundary(geom) 328 0.02647 0.02647 0.02647 3.01% Castro::initialize_do_advance() 10 0.02473 0.02473 0.02473 2.81% Gravity::get_new_grav_vector() 11 0.02451 0.02451 0.02451 2.79% Castro::reset_internal_energy(MultiFab) 63 0.02399 0.02399 0.02399 2.73% Castro::do_old_sources() 10 0.02258 0.02258 0.02258 2.57% Castro::construct_old_gravity() 10 0.02175 0.02175 0.02175 2.48% Gravity::get_old_grav_vector() 10 0.02174 0.02174 0.02174 2.47% amrex::Dot() 1114 0.0216 0.0216 0.0216 2.46% MLMG:computeResOfCorrection() 410 0.02114 0.02114 0.02114 2.41% MLCellLinOp::correctionResidual() 410 0.02102 0.02102 0.02102 2.39% FabArray::norminf() 1061 0.02012 0.02012 0.02012 2.29% MLMG::mgVcycle_down::1 82 0.01687 0.01687 0.01687 1.92% MLPoisson::define() 11 0.01687 0.01687 0.01687 1.92% MLMG::mgVcycle_down::2 82 0.01566 0.01566 0.01566 1.78% MLMG::mgVcycle_down::3 82 0.01527 0.01527 0.01527 1.74% MLMG::mgVcycle_down::4 82 0.01513 0.01513 0.01513 1.72% FabArray::ParallelCopy() 861 0.01482 0.01482 0.01482 1.69% Castro::do_new_sources() 10 0.01469 0.01469 0.01469 1.67% FabArray::ParallelCopy_nowait() 861 0.01458 0.01458 0.01458 1.66% FabArray::setVal() 1062 0.01355 0.01355 0.01355 1.54% FabArray::Saxpy() 1370 0.01315 0.01315 0.01315 1.50% Castro::expand_state() 10 0.01311 0.01311 0.01311 1.49% Castro::initialize_advance() 10 0.01302 0.01302 0.01302 1.48% MLCGSolver::ParallelAllReduce 1832 0.01291 0.01291 0.01291 1.47% MLMG::addInterpCorrection() 410 0.01229 0.01229 0.01229 1.40% MLMG::mgVcycle_up::1 82 0.0122 0.0122 0.0122 1.39% MLMG::mgVcycle_up::4 82 0.01206 0.01206 0.01206 1.37% MLMG::mgVcycle_up::2 82 0.01193 0.01193 0.01193 1.36% amrex::average_down 410 0.01162 0.01162 0.01162 1.32% MLMG::mgVcycle_up::3 82 0.01161 0.01161 0.01161 1.32% Castro::normalize_species() 62 0.01155 0.01155 0.01155 1.31% MLCellLinOp::defineAuxData() 11 0.01141 0.01141 0.01141 1.30% amrex::Copy() 472 0.01101 0.01101 0.01101 1.25% Castro::post_timestep() 10 0.01076 0.01076 0.01076 1.22% MLPoisson::Fapply() 1060 0.01045 0.01045 0.01045 1.19% MLCellLinOp::solutionResidual() 93 0.007993 0.007993 0.007993 0.91% FabArray::Xpay() 739 0.007927 0.007927 0.007927 0.90% Gravity::fill_multipole_BCs() 11 0.007791 0.007791 0.007791 0.89% Castro::enforce_min_density() 62 0.007694 0.007694 0.007694 0.88% MLMG::computeResidual() 82 0.006633 0.006633 0.006633 0.75% Castro::estTimeStep() 21 0.006003 0.006003 0.006003 0.68% MLCellLinOp::defineBC() 11 0.005197 0.005197 0.005197 0.59% BndryData::define() 11 0.004926 0.004926 0.004926 0.56% MLMG::prepareForSolve() 11 0.004913 0.004913 0.004913 0.56% Amr::InitializeInit() 1 0.004657 0.004657 0.004657 0.53% Amr::defBaseLevel() 1 0.004651 0.004651 0.004651 0.53% Castro::construct_new_source() 50 0.003998 0.003998 0.003998 0.45% Castro::initData() 1 0.003988 0.003988 0.003988 0.45% Castro::construct_new_gravity_source() 10 0.003769 0.003769 0.003769 0.43% amrex::Add() 82 0.003755 0.003755 0.003755 0.43% Castro::construct_old_source() 50 0.003305 0.003305 0.003305 0.38% Castro::construct_old_gravity_source() 10 0.003212 0.003212 0.003212 0.37% Castro::finalize_do_advance() 10 0.003028 0.003028 0.003028 0.34% Castro::reset_internal_energy(Fab) 504 0.002692 0.002692 0.002692 0.31% MLMG::ResNormInf() 93 0.002238 0.002238 0.002238 0.25% Castro::computeNewDt() 9 0.00221 0.00221 0.00221 0.25% Castro::apply_source_to_state() 20 0.001881 0.001881 0.001881 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001768 0.001768 0.001768 0.20% MLMG::getGradSolution() 11 0.001636 0.001636 0.001636 0.19% MLCellLinOp::compGrad() 11 0.00163 0.00163 0.00163 0.19% MLCellLinOp::setLevelBC() 11 0.001579 0.001579 0.001579 0.18% FabArrayBase::getCPC() 1323 0.001448 0.001448 0.001448 0.16% FabArray::mult() 43 0.001403 0.001403 0.001403 0.16% MLMG::computeMLResidual() 11 0.001401 0.001401 0.001401 0.16% FabArray::setDomainBndry() 41 0.001372 0.001372 0.001372 0.16% MLPoisson::prepareForSolve() 11 0.001328 0.001328 0.001328 0.15% MLCellLinOp::prepareForSolve() 11 0.001315 0.001315 0.001315 0.15% Castro::check_for_nan() 20 0.00131 0.00131 0.00131 0.15% MultiFab::contains_nan() 20 0.001299 0.001299 0.001299 0.15% check_for_negative_density() 10 0.001217 0.001217 0.001217 0.14% Other 4815 0.008314 0.008314 0.008314 0.95% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 6159 KiB 9037 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1323 KiB 39 MiB Castro::initialize_do_advance() 80 80 26 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1632 KiB 28 MiB Castro::initialize_advance() 80 80 16 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7614 KiB 14 MiB MLMG::prepareForSolve() 649 649 3757 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 288 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 253 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7525 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 14 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2264 B 2048 KiB Gravity::solve_for_phi() 80 80 591 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 102 KiB 2048 KiB BndryData::define() 1056 1056 343 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 218 KiB 671 KiB Castro::estTimeStep() 21 21 3286 B 480 KiB VisMF::Write(FabArray) 656 656 3570 B 320 KiB Castro::normalize_species() 62 62 4273 B 320 KiB amrex::average_down 1067 1067 1652 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1212 B 257 KiB amrex::Dot() 1360 1360 3636 B 160 KiB FabArray::norminf() 1143 1143 3545 B 160 KiB check_for_negative_density() 10 10 219 B 160 KiB Castro::initData() 1 1 53 B 160 KiB MultiFab::max() 11 11 60 B 160 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MultiFab::contains_nan() 20 20 29 B 20 KiB MLPoisson::Fsmooth() 132 132 3668 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB FillBoundary_nowait() 760 760 340 B 9648 B MLCellLinOp::applyBC() 8702 8702 232 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3928 B 6144 B StateData::FillBoundary(geom) 1992 1992 97 B 3744 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 386 B 1248 B MLCGSolver::bicgstab 410 410 99 B 1216 B MLPoisson::Fapply() 11 11 307 B 1024 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 698 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 62 KiB 8192 KiB VisMF::Write(FabArray) 744 744 470 KiB 3584 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MLPoisson::Fsmooth() 132 132 3668 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB FillBoundary_nowait() 760 760 339 B 9648 B MLCellLinOp::applyBC() 4351 4351 230 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3928 B 6144 B StateData::FillBoundary(geom) 1992 1992 98 B 3744 B Gravity::get_new_grav_vector() 3 3 2893 B 3072 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B amrex::average_down 83 83 618 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLPoisson::Fapply() 11 11 307 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 26 B 400 B FabArray::norminf() 1143 1143 10 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2109 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.07-31-g11d31e5f787d) finalized Initializing AMReX (24.07-31-g11d31e5f787d)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.07-31-g11d31e5f787d) initialized Starting run at 08:07:10 UTC on 2024-07-26. Successfully read inputs file ... Castro git describe: 24.07-23-g8ffd8d763 AMReX git describe: 24.07-31-g11d31e5f7 Microphysics git describe: 24.07-33-g997a4262 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.504754423 Restart time = 0.072267609 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.066114397 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051347696 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.059053394 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059433122 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.075262223 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.028136068 seconds Ending run at 08:07:10 UTC on 2024-07-26. Run time = 0.41241611 Run time without initialization = 0.339756408 Average number of zones advanced per microsecond: 3.858 Average number of zones advanced per microsecond per rank: 3.858 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.4125 ... 0.4125 ... 0.4125 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1169 0.1169 0.1169 28.33% VisMF::Read() 3 0.06089 0.06089 0.06089 14.76% MLCellLinOp::applyBC() 1910 0.03665 0.03665 0.03665 8.89% VisMF::Write(FabArray) 1 0.02563 0.02563 0.02563 6.21% MLPoisson::Fsmooth() 1440 0.01514 0.01514 0.01514 3.67% FillBoundary_nowait() 1730 0.01493 0.01493 0.01493 3.62% StateData::FillBoundary(geom) 160 0.01293 0.01293 0.01293 3.14% Castro::reset_internal_energy(MultiFab) 30 0.00953 0.00953 0.00953 2.31% amrex::Dot() 484 0.009412 0.009412 0.009412 2.28% FabArray::norminf() 465 0.008814 0.008814 0.008814 2.14% FabArray::setVal() 501 0.006751 0.006751 0.006751 1.64% FabArray::ParallelCopy_nowait() 380 0.006278 0.006278 0.006278 1.52% Castro::normalize_species() 30 0.00618 0.00618 0.00618 1.50% FabArray::Saxpy() 597 0.005938 0.005938 0.005938 1.44% amrex::Copy() 221 0.005556 0.005556 0.005556 1.35% MLCellLinOp::defineAuxData() 6 0.005502 0.005502 0.005502 1.33% Castro::computeTemp() 30 0.005473 0.005473 0.005473 1.33% Amr::restart() 1 0.004913 0.004913 0.004913 1.19% Gravity::fill_multipole_BCs() 6 0.004779 0.004779 0.004779 1.16% StateDataPhysBCFunct::() 20 0.004743 0.004743 0.004743 1.15% MLPoisson::Fapply() 464 0.004674 0.004674 0.004674 1.13% Castro::enforce_min_density() 30 0.003634 0.003634 0.003634 0.88% FabArray::Xpay() 325 0.003553 0.003553 0.003553 0.86% MLMG::addInterpCorrection() 180 0.003176 0.003176 0.003176 0.77% amrex::average_down 180 0.002953 0.002953 0.002953 0.72% Castro::estTimeStep() 10 0.002321 0.002321 0.002321 0.56% Amr::writePlotFile() 1 0.002313 0.002313 0.002313 0.56% BndryData::define() 6 0.002136 0.002136 0.002136 0.52% Castro::construct_new_gravity_source() 5 0.0019 0.0019 0.0019 0.46% amrex::Add() 36 0.001655 0.001655 0.001655 0.40% Castro::construct_old_gravity_source() 5 0.001562 0.001562 0.001562 0.38% Castro::reset_internal_energy(Fab) 240 0.00115 0.00115 0.00115 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001055 0.001055 0.001055 0.26% MLCellLinOp::setLevelBC() 6 0.0008917 0.0008917 0.0008917 0.22% Gravity::actual_solve_with_mlmg() 6 0.0008534 0.0008534 0.0008534 0.21% MLCellLinOp::prepareForSolve() 6 0.0007805 0.0007805 0.0007805 0.19% MLCGSolver::bicgstab 36 0.0007354 0.0007354 0.0007354 0.18% FabArray::mult() 22 0.0007064 0.0007064 0.0007064 0.17% FabArray::setDomainBndry() 20 0.000704 0.000704 0.000704 0.17% MultiFab::contains_nan() 10 0.0006823 0.0006823 0.0006823 0.17% MLCellLinOp::compGrad() 6 0.0006271 0.0006271 0.0006271 0.15% MLCellLinOp::smooth() 720 0.0005064 0.0005064 0.0005064 0.12% MLMG::prepareForSolve() 6 0.0003905 0.0003905 0.0003905 0.09% FabArrayBase::getCPC() 632 0.0003845 0.0003845 0.0003845 0.09% FabArray::FillBoundary() 1730 0.000334 0.000334 0.000334 0.08% Gravity::get_old_grav_vector() 5 0.0003033 0.0003033 0.0003033 0.07% main() 1 0.0002517 0.0002517 0.0002517 0.06% Gravity::get_new_grav_vector() 5 0.0002358 0.0002358 0.0002358 0.06% AmrLevel::FillPatch() 20 0.0002033 0.0002033 0.0002033 0.05% Amr::coarseTimeStep() 5 0.0002021 0.0002021 0.0002021 0.05% MLCellLinOp::apply() 464 0.0001744 0.0001744 0.0001744 0.04% MLCellLinOp::defineBC() 6 0.0001536 0.0001536 0.0001536 0.04% MLCGSolver::ParallelAllReduce 798 0.0001356 0.0001356 0.0001356 0.03% FabArray::ParallelCopy() 380 0.0001139 0.0001139 0.0001139 0.03% Castro::subcycle_advance_ctu() 5 0.0001107 0.0001107 0.0001107 0.03% FillPatchIterator::Initialize 20 0.0001102 0.0001102 0.0001102 0.03% Castro::do_advance_ctu() 5 0.0001025 0.0001025 0.0001025 0.02% Amr::timeStep() 5 8.703e-05 8.703e-05 8.703e-05 0.02% MLMG::mgVcycle() 36 8.188e-05 8.188e-05 8.188e-05 0.02% MLCellLinOp::correctionResidual() 180 7.725e-05 7.725e-05 7.725e-05 0.02% Castro::advance() 5 7.076e-05 7.076e-05 7.076e-05 0.02% AmrLevel::restart() 1 6.994e-05 6.994e-05 6.994e-05 0.02% StateData::restartDoit() 4 6.668e-05 6.668e-05 6.668e-05 0.02% Gravity::update_max_rhs() 6 6.363e-05 6.363e-05 6.363e-05 0.02% MLMG:computeResOfCorrection() 180 5.537e-05 5.537e-05 5.537e-05 0.01% Gravity::solve_for_phi() 5 4.794e-05 4.794e-05 4.794e-05 0.01% MLMG::mgVcycle_down::0 36 4.038e-05 4.038e-05 4.038e-05 0.01% Castro::initialize_advance() 5 3.851e-05 3.851e-05 3.851e-05 0.01% MLMG::actualBottomSolve() 36 3.824e-05 3.824e-05 3.824e-05 0.01% MLMG::mgVcycle_down::1 36 3.591e-05 3.591e-05 3.591e-05 0.01% MLMG::solve() 6 3.552e-05 3.552e-05 3.552e-05 0.01% MLMG::mgVcycle_down::2 36 3.329e-05 3.329e-05 3.329e-05 0.01% MLMG::mgVcycle_down::4 36 3.283e-05 3.283e-05 3.283e-05 0.01% Castro::initialize_do_advance() 5 3.122e-05 3.122e-05 3.122e-05 0.01% Castro::clean_state() 30 3.067e-05 3.067e-05 3.067e-05 0.01% MLMG::mgVcycle_down::3 36 3.052e-05 3.052e-05 3.052e-05 0.01% Castro::post_restart() 1 2.789e-05 2.789e-05 2.789e-05 0.01% MLMG::mgVcycle_up::4 36 2.631e-05 2.631e-05 2.631e-05 0.01% Castro::finalize_do_advance() 5 2.523e-05 2.523e-05 2.523e-05 0.01% MLMG::oneIter() 36 2.258e-05 2.258e-05 2.258e-05 0.01% MLMG::mgVcycle_up::3 36 2.238e-05 2.238e-05 2.238e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.2e-05 2.2e-05 2.2e-05 0.01% MLMG::mgVcycle_up::2 36 2.1e-05 2.1e-05 2.1e-05 0.01% MLCellLinOp::solutionResidual() 42 2.095e-05 2.095e-05 2.095e-05 0.01% MLMG::mgVcycle_up::1 36 2.029e-05 2.029e-05 2.029e-05 0.00% MLMG::mgVcycle_up::0 36 2.019e-05 2.019e-05 2.019e-05 0.00% FillPatchSingleLevel 20 1.691e-05 1.691e-05 1.691e-05 0.00% MLMG::ResNormInf() 42 1.684e-05 1.684e-05 1.684e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.637e-05 1.637e-05 1.637e-05 0.00% MLMG::mgVcycle_bottom 36 1.443e-05 1.443e-05 1.443e-05 0.00% MLMG::computeResidual() 36 1.401e-05 1.401e-05 1.401e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.305e-05 1.305e-05 1.305e-05 0.00% MLPoisson::define() 6 1.243e-05 1.243e-05 1.243e-05 0.00% Castro::expand_state() 5 1.193e-05 1.193e-05 1.193e-05 0.00% Castro::construct_new_gravity() 5 1.132e-05 1.132e-05 1.132e-05 0.00% Castro::construct_new_source() 25 1.078e-05 1.078e-05 1.078e-05 0.00% Castro::construct_old_source() 25 1.069e-05 1.069e-05 1.069e-05 0.00% Castro::do_old_sources() 5 1.067e-05 1.067e-05 1.067e-05 0.00% MLPoisson::prepareForSolve() 6 9.651e-06 9.651e-06 9.651e-06 0.00% Castro::do_new_sources() 5 9.478e-06 9.478e-06 9.478e-06 0.00% Gravity::actual_multilevel_solve() 1 8.368e-06 8.368e-06 8.368e-06 0.00% Castro::check_for_nan() 10 6.489e-06 6.489e-06 6.489e-06 0.00% Castro::apply_source_to_state() 10 6.031e-06 6.031e-06 6.031e-06 0.00% Castro::construct_old_gravity() 5 5.78e-06 5.78e-06 5.78e-06 0.00% Castro::post_timestep() 5 5.117e-06 5.117e-06 5.117e-06 0.00% MLMG::computeMLResidual() 6 3.857e-06 3.857e-06 3.857e-06 0.00% Castro::computeNewDt() 5 3.484e-06 3.484e-06 3.484e-06 0.00% MLMG::getGradSolution() 6 3.035e-06 3.035e-06 3.035e-06 0.00% Amr::init() 1 8.32e-07 8.32e-07 8.32e-07 0.00% Other 2185 0.002318 0.002318 0.002318 0.56% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4124 0.4124 0.4124 99.99% Amr::coarseTimeStep() 5 0.3114 0.3114 0.3114 75.50% Amr::timeStep() 5 0.3097 0.3097 0.3097 75.10% Castro::advance() 5 0.3033 0.3033 0.3033 73.54% Castro::subcycle_advance_ctu() 5 0.2969 0.2969 0.2969 71.99% Castro::do_advance_ctu() 5 0.2968 0.2968 0.2968 71.97% Castro::construct_new_gravity() 5 0.1395 0.1395 0.1395 33.82% Gravity::solve_phi_with_mlmg() 6 0.1369 0.1369 0.1369 33.18% Gravity::actual_solve_with_mlmg() 6 0.1318 0.1318 0.1318 31.96% Gravity::solve_for_phi() 5 0.1315 0.1315 0.1315 31.89% Castro::construct_ctu_hydro_source() 5 0.1218 0.1218 0.1218 29.52% MLMG::solve() 6 0.1197 0.1197 0.1197 29.03% MLMG::oneIter() 36 0.1119 0.1119 0.1119 27.12% MLMG::mgVcycle() 36 0.1102 0.1102 0.1102 26.71% Amr::init() 1 0.07231 0.07231 0.07231 17.53% Amr::restart() 1 0.07231 0.07231 0.07231 17.53% AmrLevel::restart() 1 0.06125 0.06125 0.06125 14.85% StateData::restartDoit() 4 0.06118 0.06118 0.06118 14.83% VisMF::Read() 3 0.06089 0.06089 0.06089 14.76% MLCellLinOp::smooth() 720 0.05575 0.05575 0.05575 13.52% MLCellLinOp::applyBC() 1910 0.05227 0.05227 0.05227 12.67% MLMG::mgVcycle_bottom 36 0.03211 0.03211 0.03211 7.79% MLMG::actualBottomSolve() 36 0.0321 0.0321 0.0321 7.78% MLCGSolver::bicgstab 36 0.03174 0.03174 0.03174 7.69% Amr::writePlotFile() 1 0.0282 0.0282 0.0282 6.84% Castro::clean_state() 30 0.02606 0.02606 0.02606 6.32% VisMF::Write(FabArray) 1 0.02563 0.02563 0.02563 6.21% AmrLevel::FillPatch() 20 0.02272 0.02272 0.02272 5.51% FillPatchIterator::Initialize 20 0.02061 0.02061 0.02061 5.00% FillPatchIterator::FillFromLevel0() 20 0.01979 0.01979 0.01979 4.80% FillPatchSingleLevel 20 0.01977 0.01977 0.01977 4.79% StateDataPhysBCFunct::() 20 0.01768 0.01768 0.01768 4.29% MLCellLinOp::apply() 464 0.01671 0.01671 0.01671 4.05% MLMG::mgVcycle_down::0 36 0.01642 0.01642 0.01642 3.98% Castro::computeTemp() 30 0.01615 0.01615 0.01615 3.92% FabArray::FillBoundary() 1730 0.01562 0.01562 0.01562 3.79% FillBoundary_nowait() 1730 0.01528 0.01528 0.01528 3.71% MLPoisson::Fsmooth() 1440 0.01514 0.01514 0.01514 3.67% StateData::FillBoundary(geom) 160 0.01293 0.01293 0.01293 3.14% MLMG::mgVcycle_up::0 36 0.01224 0.01224 0.01224 2.97% Castro::reset_internal_energy(MultiFab) 30 0.01068 0.01068 0.01068 2.59% Castro::initialize_do_advance() 5 0.01003 0.01003 0.01003 2.43% amrex::Dot() 484 0.009412 0.009412 0.009412 2.28% MLPoisson::define() 6 0.009407 0.009407 0.009407 2.28% MLMG:computeResOfCorrection() 180 0.00935 0.00935 0.00935 2.27% MLCellLinOp::correctionResidual() 180 0.009294 0.009294 0.009294 2.25% Castro::do_old_sources() 5 0.009074 0.009074 0.009074 2.20% FabArray::norminf() 465 0.008814 0.008814 0.008814 2.14% Gravity::get_new_grav_vector() 5 0.007863 0.007863 0.007863 1.91% Castro::construct_old_gravity() 5 0.007698 0.007698 0.007698 1.87% Gravity::get_old_grav_vector() 5 0.007692 0.007692 0.007692 1.86% MLMG::mgVcycle_down::1 36 0.007653 0.007653 0.007653 1.86% Castro::do_new_sources() 5 0.007295 0.007295 0.007295 1.77% MLMG::mgVcycle_down::2 36 0.006958 0.006958 0.006958 1.69% FabArray::ParallelCopy() 380 0.006793 0.006793 0.006793 1.65% MLMG::mgVcycle_down::3 36 0.006771 0.006771 0.006771 1.64% FabArray::setVal() 501 0.006751 0.006751 0.006751 1.64% MLMG::mgVcycle_down::4 36 0.006689 0.006689 0.006689 1.62% FabArray::ParallelCopy_nowait() 380 0.006679 0.006679 0.006679 1.62% MLCellLinOp::defineAuxData() 6 0.006356 0.006356 0.006356 1.54% Castro::post_timestep() 5 0.006331 0.006331 0.006331 1.53% Castro::normalize_species() 30 0.00618 0.00618 0.00618 1.50% Castro::initialize_advance() 5 0.005995 0.005995 0.005995 1.45% Castro::post_restart() 1 0.005957 0.005957 0.005957 1.44% FabArray::Saxpy() 597 0.005938 0.005938 0.005938 1.44% Castro::expand_state() 5 0.005743 0.005743 0.005743 1.39% MLCGSolver::ParallelAllReduce 798 0.005684 0.005684 0.005684 1.38% Gravity::multilevel_solve_for_new_phi() 1 0.005586 0.005586 0.005586 1.35% Gravity::actual_multilevel_solve() 1 0.005569 0.005569 0.005569 1.35% amrex::Copy() 221 0.005556 0.005556 0.005556 1.35% MLMG::addInterpCorrection() 180 0.005517 0.005517 0.005517 1.34% MLMG::mgVcycle_up::1 36 0.005427 0.005427 0.005427 1.32% MLMG::mgVcycle_up::4 36 0.005379 0.005379 0.005379 1.30% amrex::average_down 180 0.005328 0.005328 0.005328 1.29% MLMG::mgVcycle_up::2 36 0.005281 0.005281 0.005281 1.28% MLMG::mgVcycle_up::3 36 0.005158 0.005158 0.005158 1.25% Gravity::fill_multipole_BCs() 6 0.004916 0.004916 0.004916 1.19% MLPoisson::Fapply() 464 0.004674 0.004674 0.004674 1.13% MLCellLinOp::solutionResidual() 42 0.003833 0.003833 0.003833 0.93% Castro::enforce_min_density() 30 0.003634 0.003634 0.003634 0.88% FabArray::Xpay() 325 0.003553 0.003553 0.003553 0.86% MLMG::computeResidual() 36 0.002965 0.002965 0.002965 0.72% MLCellLinOp::defineBC() 6 0.002894 0.002894 0.002894 0.70% MLMG::prepareForSolve() 6 0.00281 0.00281 0.00281 0.68% BndryData::define() 6 0.002741 0.002741 0.002741 0.66% Castro::estTimeStep() 10 0.002321 0.002321 0.002321 0.56% Castro::construct_new_source() 25 0.001911 0.001911 0.001911 0.46% Castro::construct_new_gravity_source() 5 0.0019 0.0019 0.0019 0.46% amrex::Add() 36 0.001655 0.001655 0.001655 0.40% Castro::construct_old_source() 25 0.001573 0.001573 0.001573 0.38% Castro::construct_old_gravity_source() 5 0.001562 0.001562 0.001562 0.38% Castro::computeNewDt() 5 0.001463 0.001463 0.001463 0.35% Castro::reset_internal_energy(Fab) 240 0.00115 0.00115 0.00115 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001055 0.001055 0.001055 0.26% MLMG::ResNormInf() 42 0.001044 0.001044 0.001044 0.25% Castro::apply_source_to_state() 10 0.0009728 0.0009728 0.0009728 0.24% MLMG::getGradSolution() 6 0.0009389 0.0009389 0.0009389 0.23% MLCellLinOp::compGrad() 6 0.0009359 0.0009359 0.0009359 0.23% MLCellLinOp::setLevelBC() 6 0.0008917 0.0008917 0.0008917 0.22% Castro::finalize_do_advance() 5 0.000887 0.000887 0.000887 0.22% MLMG::computeMLResidual() 6 0.0008853 0.0008853 0.0008853 0.21% FabArrayBase::getCPC() 632 0.0008046 0.0008046 0.0008046 0.20% MLPoisson::prepareForSolve() 6 0.0007901 0.0007901 0.0007901 0.19% MLCellLinOp::prepareForSolve() 6 0.0007805 0.0007805 0.0007805 0.19% Gravity::update_max_rhs() 6 0.0007295 0.0007295 0.0007295 0.18% FabArray::mult() 22 0.0007064 0.0007064 0.0007064 0.17% FabArray::setDomainBndry() 20 0.000704 0.000704 0.000704 0.17% Castro::check_for_nan() 10 0.0006888 0.0006888 0.0006888 0.17% MultiFab::contains_nan() 10 0.0006823 0.0006823 0.0006823 0.17% Other 2185 0.003581 0.003581 0.003581 0.87% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 12 MiB 9037 MiB Castro::initMFs() 48 48 56 MiB 68 MiB Castro::swap_state_time_levels() 32 32 45 MiB 55 MiB StateData::restartDoit() 32 32 52 MiB 55 MiB FillPatchIterator::Initialize 160 160 1083 KiB 39 MiB Castro::initialize_do_advance() 40 40 28 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1830 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6354 KiB 14 MiB MLMG::prepareForSolve() 354 354 3568 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 193 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 195 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6341 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 18 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3517 B 2048 KiB Gravity::solve_for_phi() 40 40 651 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 27 KiB 2048 KiB BndryData::define() 576 576 328 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 210 KiB 671 KiB Castro::estTimeStep() 10 10 2627 B 480 KiB VisMF::Write(FabArray) 112 112 1381 B 320 KiB Castro::normalize_species() 30 30 4868 B 320 KiB amrex::average_down 469 469 1537 B 257 KiB MLMG::addInterpCorrection() 468 468 1156 B 257 KiB amrex::Dot() 592 592 3366 B 160 KiB FabArray::norminf() 501 501 3308 B 160 KiB check_for_negative_density() 5 5 257 B 160 KiB MultiFab::max() 6 6 81 B 160 KiB FabArray::setVal() 67 67 21 KiB 28 KiB MultiFab::contains_nan() 10 10 32 B 20 KiB MLPoisson::Fsmooth() 60 60 3456 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 48 B 10 KiB FillBoundary_nowait() 336 336 315 B 9648 B MLCellLinOp::applyBC() 3820 3820 224 B 9344 B amrex::Copy() 56 56 5838 B 8816 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B StateData::FillBoundary(geom) 960 960 41 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLCellLinOp::defineBC() 36 36 369 B 1248 B MLCGSolver::bicgstab 180 180 92 B 1216 B MLPoisson::Fapply() 6 6 291 B 1024 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1447 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 88 KiB 8192 KiB VisMF::Write(FabArray) 120 120 174 KiB 3584 KiB VisMF::Read() 24 24 223 KiB 3000 KiB FabArray::setVal() 67 67 21 KiB 28 KiB MLPoisson::Fsmooth() 60 60 3456 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 48 B 10 KiB FillBoundary_nowait() 336 336 315 B 9648 B MLCellLinOp::applyBC() 1910 1910 223 B 9328 B amrex::Copy() 56 56 5838 B 8816 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B Gravity::get_old_grav_vector() 3 3 2482 B 3072 B StateData::FillBoundary(geom) 960 960 42 B 3024 B Gravity::fill_multipole_BCs() 18 18 6 B 2832 B amrex::average_down 37 37 457 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLPoisson::Fapply() 6 6 291 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 24 B 400 B FabArray::norminf() 501 501 9 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2109 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.07-31-g11d31e5f787d) finalized