Initializing AMReX (24.07-18-gfdea28f16727)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.07-18-gfdea28f16727) initialized Starting run at 07:52:28 UTC on 2024-07-18. Successfully read inputs file ... Castro git describe: 24.07-15-gb4b35ba86 AMReX git describe: 24.07-18-gfdea28f16 Microphysics git describe: 24.07-24-g17a2fa77 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.049765032 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.027667096 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.064402762 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050699124 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.060929476 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.077855177 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.087764206 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.071298912 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.05681086 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.059324373 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.06138188 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.065983972 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.081924173 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.047360366 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.027262945 seconds Ending run at 07:52:29 UTC on 2024-07-18. Run time = 0.945367469 Run time without initialization = 0.813679092 Average number of zones advanced per microsecond: 3.222 Average number of zones advanced per microsecond per rank: 3.222 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.9454 ... 0.9454 ... 0.9454 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2611 0.2611 0.2611 27.62% VisMF::Write(FabArray) 11 0.193 0.193 0.193 20.41% MLCellLinOp::applyBC() 4351 0.08435 0.08435 0.08435 8.92% MLPoisson::Fsmooth() 3280 0.03521 0.03521 0.03521 3.72% FillBoundary_nowait() 3941 0.03453 0.03453 0.03453 3.65% Amr::checkPoint() 3 0.02812 0.02812 0.02812 2.97% StateData::FillBoundary(geom) 328 0.02755 0.02755 0.02755 2.91% amrex::Dot() 1114 0.02254 0.02254 0.02254 2.38% Castro::reset_internal_energy(MultiFab) 63 0.02161 0.02161 0.02161 2.29% FabArray::norminf() 1061 0.02081 0.02081 0.02081 2.20% Castro::computeTemp() 63 0.01882 0.01882 0.01882 1.99% FabArray::ParallelCopy_nowait() 861 0.01416 0.01416 0.01416 1.50% FabArray::setVal() 1062 0.01397 0.01397 0.01397 1.48% FabArray::Saxpy() 1370 0.01362 0.01362 0.01362 1.44% StateDataPhysBCFunct::() 41 0.01161 0.01161 0.01161 1.23% amrex::Copy() 472 0.01122 0.01122 0.01122 1.19% MLPoisson::Fapply() 1060 0.01089 0.01089 0.01089 1.15% MLCellLinOp::defineAuxData() 11 0.01081 0.01081 0.01081 1.14% Castro::normalize_species() 62 0.01077 0.01077 0.01077 1.14% Castro::construct_new_gravity_source() 10 0.01046 0.01046 0.01046 1.11% Gravity::fill_multipole_BCs() 11 0.008314 0.008314 0.008314 0.88% FabArray::Xpay() 739 0.008161 0.008161 0.008161 0.86% Castro::enforce_min_density() 62 0.007983 0.007983 0.007983 0.84% MLMG::addInterpCorrection() 410 0.007288 0.007288 0.007288 0.77% amrex::average_down 410 0.00651 0.00651 0.00651 0.69% Castro::estTimeStep() 21 0.005624 0.005624 0.005624 0.59% BndryData::define() 11 0.004137 0.004137 0.004137 0.44% amrex::Add() 82 0.003805 0.003805 0.003805 0.40% Castro::construct_old_gravity_source() 10 0.003032 0.003032 0.003032 0.32% Castro::reset_internal_energy(Fab) 504 0.002441 0.002441 0.002441 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001882 0.001882 0.001882 0.20% Amr::writePlotFile() 2 0.001821 0.001821 0.001821 0.19% MLCellLinOp::setLevelBC() 11 0.001636 0.001636 0.001636 0.17% MLCGSolver::bicgstab 82 0.001592 0.001592 0.001592 0.17% check_for_negative_density() 10 0.001566 0.001566 0.001566 0.17% Gravity::actual_solve_with_mlmg() 11 0.001525 0.001525 0.001525 0.16% Castro::initData() 1 0.001497 0.001497 0.001497 0.16% FabArray::mult() 43 0.00143 0.00143 0.00143 0.15% FabArray::setDomainBndry() 41 0.001422 0.001422 0.001422 0.15% MLCellLinOp::prepareForSolve() 11 0.001371 0.001371 0.001371 0.14% MultiFab::contains_nan() 20 0.001299 0.001299 0.001299 0.14% MLCellLinOp::compGrad() 11 0.001116 0.001116 0.001116 0.12% MLCellLinOp::smooth() 1640 0.001101 0.001101 0.001101 0.12% MLMG::prepareForSolve() 11 0.0009781 0.0009781 0.0009781 0.10% FabArrayBase::getCPC() 1323 0.0008373 0.0008373 0.0008373 0.09% FabArray::FillBoundary() 3941 0.0007963 0.0007963 0.0007963 0.08% Castro::subcycle_advance_ctu() 10 0.0007819 0.0007819 0.0007819 0.08% Gravity::get_new_grav_vector() 11 0.0006285 0.0006285 0.0006285 0.07% Gravity::get_old_grav_vector() 10 0.0004924 0.0004924 0.0004924 0.05% AmrLevel::FillPatch() 41 0.0004264 0.0004264 0.0004264 0.05% MLCellLinOp::apply() 1060 0.0004227 0.0004227 0.0004227 0.04% MLCGSolver::ParallelAllReduce 1832 0.0003447 0.0003447 0.0003447 0.04% Amr::coarseTimeStep() 10 0.0003423 0.0003423 0.0003423 0.04% main() 1 0.0003106 0.0003106 0.0003106 0.03% MLCellLinOp::defineBC() 11 0.0002666 0.0002666 0.0002666 0.03% FabArray::ParallelCopy() 861 0.0002532 0.0002532 0.0002532 0.03% FillPatchIterator::Initialize 41 0.0002209 0.0002209 0.0002209 0.02% MLMG::mgVcycle() 82 0.0002116 0.0002116 0.0002116 0.02% Castro::construct_new_source() 50 0.0002031 0.0002031 0.0002031 0.02% Castro::do_advance_ctu() 10 0.0001868 0.0001868 0.0001868 0.02% MLCellLinOp::correctionResidual() 410 0.000162 0.000162 0.000162 0.02% Amr::timeStep() 10 0.0001585 0.0001585 0.0001585 0.02% MLMG:computeResOfCorrection() 410 0.0001262 0.0001262 0.0001262 0.01% Castro::advance() 10 0.0001253 0.0001253 0.0001253 0.01% StateData::checkPoint() 12 0.000112 0.000112 0.000112 0.01% Gravity::solve_for_phi() 10 0.0001046 0.0001046 0.0001046 0.01% Castro::finalize_do_advance() 10 0.000104 0.000104 0.000104 0.01% Castro::construct_old_source() 50 9.221e-05 9.221e-05 9.221e-05 0.01% Castro::initialize_do_advance() 10 8.518e-05 8.518e-05 8.518e-05 0.01% MLMG::mgVcycle_down::0 82 8.221e-05 8.221e-05 8.221e-05 0.01% MLMG::actualBottomSolve() 82 8.149e-05 8.149e-05 8.149e-05 0.01% Castro::initialize_advance() 10 8.109e-05 8.109e-05 8.109e-05 0.01% MLMG::mgVcycle_down::2 82 7.24e-05 7.24e-05 7.24e-05 0.01% MLMG::mgVcycle_down::1 82 7.168e-05 7.168e-05 7.168e-05 0.01% MLMG::solve() 11 7.156e-05 7.156e-05 7.156e-05 0.01% MLMG::mgVcycle_down::4 82 7.09e-05 7.09e-05 7.09e-05 0.01% Castro::clean_state() 62 6.961e-05 6.961e-05 6.961e-05 0.01% MLMG::mgVcycle_down::3 82 6.771e-05 6.771e-05 6.771e-05 0.01% AmrLevel::checkPoint() 3 6.422e-05 6.422e-05 6.422e-05 0.01% MLMG::oneIter() 82 5.834e-05 5.834e-05 5.834e-05 0.01% Castro::do_old_sources() 10 5.617e-05 5.617e-05 5.617e-05 0.01% MLMG::mgVcycle_up::4 82 5.443e-05 5.443e-05 5.443e-05 0.01% MLMG::mgVcycle_up::0 82 4.976e-05 4.976e-05 4.976e-05 0.01% MLMG::mgVcycle_up::3 82 4.808e-05 4.808e-05 4.808e-05 0.01% MLMG::mgVcycle_up::1 82 4.799e-05 4.799e-05 4.799e-05 0.01% FillPatchIterator::FillFromLevel0() 41 4.768e-05 4.768e-05 4.768e-05 0.01% MLMG::mgVcycle_up::2 82 4.713e-05 4.713e-05 4.713e-05 0.00% MLCellLinOp::solutionResidual() 93 4.508e-05 4.508e-05 4.508e-05 0.00% FillPatchSingleLevel 41 3.676e-05 3.676e-05 3.676e-05 0.00% MLMG::mgVcycle_bottom 82 3.481e-05 3.481e-05 3.481e-05 0.00% MLMG::computeResidual() 82 3.478e-05 3.478e-05 3.478e-05 0.00% MLMG::ResNormInf() 93 3.36e-05 3.36e-05 3.36e-05 0.00% Amr::defBaseLevel() 1 3.302e-05 3.302e-05 3.302e-05 0.00% MLPoisson::define() 11 2.44e-05 2.44e-05 2.44e-05 0.00% Castro::apply_source_to_state() 20 2.427e-05 2.427e-05 2.427e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.396e-05 2.396e-05 2.396e-05 0.00% Castro::construct_new_gravity() 10 2.359e-05 2.359e-05 2.359e-05 0.00% Castro::do_new_sources() 10 2.085e-05 2.085e-05 2.085e-05 0.00% Amr::FinalizeInit() 1 1.933e-05 1.933e-05 1.933e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.786e-05 1.786e-05 1.786e-05 0.00% MLPoisson::prepareForSolve() 11 1.3e-05 1.3e-05 1.3e-05 0.00% MLMG::computeMLResidual() 11 1.132e-05 1.132e-05 1.132e-05 0.00% Castro::check_for_nan() 20 1.119e-05 1.119e-05 1.119e-05 0.00% Castro::construct_old_gravity() 10 1.116e-05 1.116e-05 1.116e-05 0.00% Castro::post_timestep() 10 8.977e-06 8.977e-06 8.977e-06 0.00% Gravity::actual_multilevel_solve() 1 8.733e-06 8.733e-06 8.733e-06 0.00% Castro::post_init() 1 8.525e-06 8.525e-06 8.525e-06 0.00% Castro::computeNewDt() 9 7.148e-06 7.148e-06 7.148e-06 0.00% Amr::InitializeInit() 1 6.209e-06 6.209e-06 6.209e-06 0.00% Castro::expand_state() 10 5.837e-06 5.837e-06 5.837e-06 0.00% MLMG::getGradSolution() 11 5.834e-06 5.834e-06 5.834e-06 0.00% Amr::init() 1 2.563e-06 2.563e-06 2.563e-06 0.00% Amr::initialInit() 1 1.033e-06 1.033e-06 1.033e-06 0.00% Other 4815 0.003259 0.003259 0.003259 0.34% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9454 0.9454 0.9454 100.00% Amr::coarseTimeStep() 10 0.7862 0.7862 0.7862 83.16% Amr::timeStep() 10 0.6636 0.6636 0.6636 70.20% Castro::advance() 10 0.6524 0.6524 0.6524 69.01% Castro::subcycle_advance_ctu() 10 0.6396 0.6396 0.6396 67.65% Castro::do_advance_ctu() 10 0.6388 0.6388 0.6388 67.57% Gravity::solve_phi_with_mlmg() 11 0.3083 0.3083 0.3083 32.61% Gravity::actual_solve_with_mlmg() 11 0.2995 0.2995 0.2995 31.68% Castro::construct_new_gravity() 10 0.2806 0.2806 0.2806 29.68% MLMG::solve() 11 0.2766 0.2766 0.2766 29.26% Castro::construct_ctu_hydro_source() 10 0.2733 0.2733 0.2733 28.91% Gravity::solve_for_phi() 10 0.2631 0.2631 0.2631 27.83% MLMG::oneIter() 82 0.2605 0.2605 0.2605 27.55% MLMG::mgVcycle() 82 0.2566 0.2566 0.2566 27.14% VisMF::Write(FabArray) 11 0.193 0.193 0.193 20.41% Amr::checkPoint() 3 0.1686 0.1686 0.1686 17.83% AmrLevel::checkPoint() 3 0.1405 0.1405 0.1405 14.86% StateData::checkPoint() 12 0.1404 0.1404 0.1404 14.85% Amr::init() 1 0.131 0.131 0.131 13.86% MLCellLinOp::smooth() 1640 0.1291 0.1291 0.1291 13.66% MLCellLinOp::applyBC() 4351 0.1204 0.1204 0.1204 12.74% MLMG::mgVcycle_bottom 82 0.07609 0.07609 0.07609 8.05% MLMG::actualBottomSolve() 82 0.07606 0.07606 0.07606 8.04% MLCGSolver::bicgstab 82 0.07521 0.07521 0.07521 7.96% Castro::clean_state() 62 0.06117 0.06117 0.06117 6.47% Amr::writePlotFile() 2 0.05506 0.05506 0.05506 5.82% Amr::initialInit() 1 0.05346 0.05346 0.05346 5.66% AmrLevel::FillPatch() 41 0.04941 0.04941 0.04941 5.23% Amr::FinalizeInit() 1 0.04891 0.04891 0.04891 5.17% Castro::post_init() 1 0.04797 0.04797 0.04797 5.07% Gravity::multilevel_solve_for_new_phi() 1 0.04562 0.04562 0.04562 4.83% Gravity::actual_multilevel_solve() 1 0.0456 0.0456 0.0456 4.82% FillPatchIterator::Initialize 41 0.04505 0.04505 0.04505 4.76% FillPatchIterator::FillFromLevel0() 41 0.0434 0.0434 0.0434 4.59% FillPatchSingleLevel 41 0.04336 0.04336 0.04336 4.59% Castro::computeTemp() 63 0.04287 0.04287 0.04287 4.53% StateDataPhysBCFunct::() 41 0.03916 0.03916 0.03916 4.14% MLCellLinOp::apply() 1060 0.03836 0.03836 0.03836 4.06% MLMG::mgVcycle_down::0 82 0.03737 0.03737 0.03737 3.95% FabArray::FillBoundary() 3941 0.03606 0.03606 0.03606 3.81% FillBoundary_nowait() 3941 0.03526 0.03526 0.03526 3.73% MLPoisson::Fsmooth() 3280 0.03521 0.03521 0.03521 3.72% MLMG::mgVcycle_up::0 82 0.02836 0.02836 0.02836 3.00% StateData::FillBoundary(geom) 328 0.02755 0.02755 0.02755 2.91% Castro::reset_internal_energy(MultiFab) 63 0.02405 0.02405 0.02405 2.54% Castro::initialize_do_advance() 10 0.02324 0.02324 0.02324 2.46% Castro::do_new_sources() 10 0.02269 0.02269 0.02269 2.40% amrex::Dot() 1114 0.02254 0.02254 0.02254 2.38% MLMG:computeResOfCorrection() 410 0.02161 0.02161 0.02161 2.29% MLCellLinOp::correctionResidual() 410 0.02149 0.02149 0.02149 2.27% FabArray::norminf() 1061 0.02081 0.02081 0.02081 2.20% Castro::do_old_sources() 10 0.02 0.02 0.02 2.12% Gravity::get_new_grav_vector() 11 0.01937 0.01937 0.01937 2.05% MLPoisson::define() 11 0.01803 0.01803 0.01803 1.91% MLMG::mgVcycle_down::1 82 0.01736 0.01736 0.01736 1.84% MLMG::mgVcycle_down::2 82 0.01622 0.01622 0.01622 1.72% Castro::construct_old_gravity() 10 0.01586 0.01586 0.01586 1.68% Gravity::get_old_grav_vector() 10 0.01585 0.01585 0.01585 1.68% MLMG::mgVcycle_down::3 82 0.01577 0.01577 0.01577 1.67% MLMG::mgVcycle_down::4 82 0.01572 0.01572 0.01572 1.66% FabArray::ParallelCopy() 861 0.01527 0.01527 0.01527 1.61% FabArray::ParallelCopy_nowait() 861 0.01501 0.01501 0.01501 1.59% FabArray::setVal() 1062 0.01397 0.01397 0.01397 1.48% FabArray::Saxpy() 1370 0.01362 0.01362 0.01362 1.44% MLCGSolver::ParallelAllReduce 1832 0.01349 0.01349 0.01349 1.43% MLMG::addInterpCorrection() 410 0.01281 0.01281 0.01281 1.36% MLMG::mgVcycle_up::1 82 0.01262 0.01262 0.01262 1.33% MLMG::mgVcycle_up::4 82 0.01254 0.01254 0.01254 1.33% MLMG::mgVcycle_up::2 82 0.01231 0.01231 0.01231 1.30% MLCellLinOp::defineAuxData() 11 0.0123 0.0123 0.0123 1.30% amrex::average_down 410 0.0121 0.0121 0.0121 1.28% MLMG::mgVcycle_up::3 82 0.01205 0.01205 0.01205 1.27% Castro::initialize_advance() 10 0.01204 0.01204 0.01204 1.27% Castro::expand_state() 10 0.01133 0.01133 0.01133 1.20% amrex::Copy() 472 0.01122 0.01122 0.01122 1.19% Castro::post_timestep() 10 0.01102 0.01102 0.01102 1.17% MLPoisson::Fapply() 1060 0.01089 0.01089 0.01089 1.15% Castro::normalize_species() 62 0.01077 0.01077 0.01077 1.14% Castro::construct_new_source() 50 0.01066 0.01066 0.01066 1.13% Castro::construct_new_gravity_source() 10 0.01046 0.01046 0.01046 1.11% Gravity::fill_multipole_BCs() 11 0.008567 0.008567 0.008567 0.91% FabArray::Xpay() 739 0.008161 0.008161 0.008161 0.86% MLCellLinOp::solutionResidual() 93 0.008116 0.008116 0.008116 0.86% Castro::enforce_min_density() 62 0.007983 0.007983 0.007983 0.84% MLMG::computeResidual() 82 0.006808 0.006808 0.006808 0.72% Castro::estTimeStep() 21 0.005624 0.005624 0.005624 0.59% MLCellLinOp::defineBC() 11 0.005456 0.005456 0.005456 0.58% MLMG::prepareForSolve() 11 0.005381 0.005381 0.005381 0.57% BndryData::define() 11 0.005189 0.005189 0.005189 0.55% Amr::InitializeInit() 1 0.004557 0.004557 0.004557 0.48% Amr::defBaseLevel() 1 0.00455 0.00455 0.00455 0.48% Castro::initData() 1 0.003843 0.003843 0.003843 0.41% amrex::Add() 82 0.003805 0.003805 0.003805 0.40% Castro::construct_old_source() 50 0.003124 0.003124 0.003124 0.33% Castro::construct_old_gravity_source() 10 0.003032 0.003032 0.003032 0.32% Castro::computeNewDt() 9 0.002731 0.002731 0.002731 0.29% Castro::reset_internal_energy(Fab) 504 0.002441 0.002441 0.002441 0.26% MLMG::ResNormInf() 93 0.002289 0.002289 0.002289 0.24% Castro::finalize_do_advance() 10 0.002023 0.002023 0.002023 0.21% Castro::apply_source_to_state() 20 0.001979 0.001979 0.001979 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001882 0.001882 0.001882 0.20% MLMG::getGradSolution() 11 0.001654 0.001654 0.001654 0.17% MLCellLinOp::compGrad() 11 0.001648 0.001648 0.001648 0.17% MLCellLinOp::setLevelBC() 11 0.001636 0.001636 0.001636 0.17% check_for_negative_density() 10 0.001566 0.001566 0.001566 0.17% FabArrayBase::getCPC() 1323 0.00151 0.00151 0.00151 0.16% FabArray::mult() 43 0.00143 0.00143 0.00143 0.15% FabArray::setDomainBndry() 41 0.001422 0.001422 0.001422 0.15% MLPoisson::prepareForSolve() 11 0.001384 0.001384 0.001384 0.15% MLCellLinOp::prepareForSolve() 11 0.001371 0.001371 0.001371 0.14% MLMG::computeMLResidual() 11 0.001354 0.001354 0.001354 0.14% Castro::check_for_nan() 20 0.00131 0.00131 0.00131 0.14% MultiFab::contains_nan() 20 0.001299 0.001299 0.001299 0.14% Other 4815 0.008782 0.008782 0.008782 0.93% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 5887 KiB 9037 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 987 KiB 39 MiB Castro::initialize_do_advance() 80 80 26 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1638 KiB 28 MiB Castro::initialize_advance() 80 80 16 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7609 KiB 14 MiB MLMG::prepareForSolve() 660 660 3599 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 212 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 171 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7529 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 14 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2157 B 2048 KiB Gravity::solve_for_phi() 80 80 569 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 98 KiB 2048 KiB BndryData::define() 1056 1056 329 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 209 KiB 671 KiB Castro::estTimeStep() 21 21 2468 B 480 KiB VisMF::Write(FabArray) 656 656 3412 B 320 KiB Castro::normalize_species() 62 62 3695 B 320 KiB amrex::average_down 1067 1067 1615 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1165 B 257 KiB amrex::Dot() 1360 1360 3525 B 160 KiB FabArray::norminf() 1143 1143 3406 B 160 KiB check_for_negative_density() 10 10 264 B 160 KiB Castro::initData() 1 1 52 B 160 KiB MultiFab::max() 11 11 57 B 160 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MultiFab::contains_nan() 20 20 27 B 20 KiB MLPoisson::Fsmooth() 132 132 3514 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 43 B 10 KiB FillBoundary_nowait() 760 760 319 B 9648 B MLCellLinOp::applyBC() 8702 8702 222 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3914 B 6144 B StateData::FillBoundary(geom) 1992 1992 39 B 3616 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 370 B 1248 B MLCGSolver::bicgstab 410 410 96 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 596 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 39 KiB 8192 KiB VisMF::Write(FabArray) 744 744 458 KiB 3584 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MLPoisson::Fsmooth() 132 132 3514 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 43 B 10 KiB FillBoundary_nowait() 760 760 319 B 9648 B MLCellLinOp::applyBC() 4351 4351 220 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3914 B 6144 B StateData::FillBoundary(geom) 1992 1992 40 B 3616 B Gravity::get_new_grav_vector() 3 3 2900 B 3072 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B amrex::average_down 83 83 618 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 299 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 25 B 400 B FabArray::norminf() 1143 1143 10 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2109 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.07-18-gfdea28f16727) finalized Initializing AMReX (24.07-18-gfdea28f16727)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.07-18-gfdea28f16727) initialized Starting run at 07:52:30 UTC on 2024-07-18. Successfully read inputs file ... Castro git describe: 24.07-15-gb4b35ba86 AMReX git describe: 24.07-18-gfdea28f16 Microphysics git describe: 24.07-24-g17a2fa77 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.544571825 Restart time = 0.260545223 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.06913518 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051118241 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058232973 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.078286543 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.082229823 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.030306919 seconds Ending run at 07:52:30 UTC on 2024-07-18. Run time = 0.630955261 Run time without initialization = 0.369784963 Average number of zones advanced per microsecond: 3.545 Average number of zones advanced per microsecond per rank: 3.545 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.631 ... 0.631 ... 0.631 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Read() 3 0.2482 0.2482 0.2482 39.33% Castro::construct_ctu_hydro_source() 5 0.1336 0.1336 0.1336 21.17% MLCellLinOp::applyBC() 1910 0.03761 0.03761 0.03761 5.96% VisMF::Write(FabArray) 1 0.02764 0.02764 0.02764 4.38% MLPoisson::Fsmooth() 1440 0.01596 0.01596 0.01596 2.53% FillBoundary_nowait() 1730 0.01483 0.01483 0.01483 2.35% StateData::FillBoundary(geom) 160 0.0134 0.0134 0.0134 2.12% amrex::Dot() 484 0.009837 0.009837 0.009837 1.56% FabArray::norminf() 465 0.009236 0.009236 0.009236 1.46% Castro::reset_internal_energy(MultiFab) 30 0.008515 0.008515 0.008515 1.35% Castro::computeTemp() 30 0.008027 0.008027 0.008027 1.27% FabArray::setVal() 501 0.007022 0.007022 0.007022 1.11% Castro::normalize_species() 30 0.0065 0.0065 0.0065 1.03% FabArray::ParallelCopy_nowait() 380 0.00645 0.00645 0.00645 1.02% FabArray::Saxpy() 597 0.0062 0.0062 0.0062 0.98% MLCellLinOp::defineAuxData() 6 0.005953 0.005953 0.005953 0.94% Castro::enforce_min_density() 30 0.005726 0.005726 0.005726 0.91% StateDataPhysBCFunct::() 20 0.005717 0.005717 0.005717 0.91% amrex::Copy() 221 0.005704 0.005704 0.005704 0.90% Amr::restart() 1 0.005466 0.005466 0.005466 0.87% Gravity::fill_multipole_BCs() 6 0.005342 0.005342 0.005342 0.85% MLPoisson::Fapply() 464 0.004901 0.004901 0.004901 0.78% FabArray::Xpay() 325 0.003709 0.003709 0.003709 0.59% MLMG::addInterpCorrection() 180 0.003321 0.003321 0.003321 0.53% amrex::average_down 180 0.002977 0.002977 0.002977 0.47% Castro::estTimeStep() 10 0.00263 0.00263 0.00263 0.42% Amr::writePlotFile() 1 0.002474 0.002474 0.002474 0.39% BndryData::define() 6 0.002248 0.002248 0.002248 0.36% Castro::construct_new_gravity_source() 5 0.001854 0.001854 0.001854 0.29% amrex::Add() 36 0.001652 0.001652 0.001652 0.26% Castro::construct_old_gravity_source() 5 0.001536 0.001536 0.001536 0.24% Castro::reset_internal_energy(Fab) 240 0.001208 0.001208 0.001208 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001067 0.001067 0.001067 0.17% check_for_negative_density() 5 0.0009539 0.0009539 0.0009539 0.15% MLCellLinOp::setLevelBC() 6 0.0009125 0.0009125 0.0009125 0.14% Gravity::actual_solve_with_mlmg() 6 0.0008288 0.0008288 0.0008288 0.13% MLCellLinOp::prepareForSolve() 6 0.0007798 0.0007798 0.0007798 0.12% MLCGSolver::bicgstab 36 0.0007242 0.0007242 0.0007242 0.11% MLCellLinOp::compGrad() 6 0.000629 0.000629 0.000629 0.10% MLMG::prepareForSolve() 6 0.0005705 0.0005705 0.0005705 0.09% MLCellLinOp::smooth() 720 0.0005079 0.0005079 0.0005079 0.08% FabArrayBase::getCPC() 632 0.0003773 0.0003773 0.0003773 0.06% Gravity::get_old_grav_vector() 5 0.0003544 0.0003544 0.0003544 0.06% FabArray::FillBoundary() 1730 0.0003482 0.0003482 0.0003482 0.06% main() 1 0.0002824 0.0002824 0.0002824 0.04% Gravity::get_new_grav_vector() 5 0.0002399 0.0002399 0.0002399 0.04% Amr::coarseTimeStep() 5 0.0002216 0.0002216 0.0002216 0.04% AmrLevel::FillPatch() 20 0.0002065 0.0002065 0.0002065 0.03% MLCellLinOp::apply() 464 0.0001892 0.0001892 0.0001892 0.03% Castro::subcycle_advance_ctu() 5 0.0001557 0.0001557 0.0001557 0.02% MLCellLinOp::defineBC() 6 0.000153 0.000153 0.000153 0.02% MLCGSolver::ParallelAllReduce 798 0.0001478 0.0001478 0.0001478 0.02% FabArray::ParallelCopy() 380 0.0001175 0.0001175 0.0001175 0.02% FillPatchIterator::Initialize 20 0.0001108 0.0001108 0.0001108 0.02% MLMG::mgVcycle() 36 8.912e-05 8.912e-05 8.912e-05 0.01% Amr::timeStep() 5 8.878e-05 8.878e-05 8.878e-05 0.01% AmrLevel::restart() 1 8.186e-05 8.186e-05 8.186e-05 0.01% StateData::restartDoit() 4 7.93e-05 7.93e-05 7.93e-05 0.01% MLCellLinOp::correctionResidual() 180 7.384e-05 7.384e-05 7.384e-05 0.01% Castro::construct_new_source() 25 7.16e-05 7.16e-05 7.16e-05 0.01% Gravity::update_max_rhs() 6 7.155e-05 7.155e-05 7.155e-05 0.01% Castro::initialize_do_advance() 5 6.838e-05 6.838e-05 6.838e-05 0.01% MLMG:computeResOfCorrection() 180 5.477e-05 5.477e-05 5.477e-05 0.01% Gravity::solve_for_phi() 5 5.115e-05 5.115e-05 5.115e-05 0.01% Castro::initialize_advance() 5 4.627e-05 4.627e-05 4.627e-05 0.01% Castro::finalize_do_advance() 5 4.531e-05 4.531e-05 4.531e-05 0.01% MLMG::mgVcycle_down::0 36 4.201e-05 4.201e-05 4.201e-05 0.01% MLMG::actualBottomSolve() 36 3.901e-05 3.901e-05 3.901e-05 0.01% MLMG::solve() 6 3.71e-05 3.71e-05 3.71e-05 0.01% Castro::advance() 5 3.671e-05 3.671e-05 3.671e-05 0.01% MLMG::mgVcycle_down::1 36 3.431e-05 3.431e-05 3.431e-05 0.01% Castro::clean_state() 30 3.333e-05 3.333e-05 3.333e-05 0.01% FillPatchIterator::FillFromLevel0() 20 3.312e-05 3.312e-05 3.312e-05 0.01% MLMG::mgVcycle_down::2 36 3.245e-05 3.245e-05 3.245e-05 0.01% MLMG::mgVcycle_down::4 36 3.038e-05 3.038e-05 3.038e-05 0.00% Castro::construct_old_source() 25 3.011e-05 3.011e-05 3.011e-05 0.00% MLMG::mgVcycle_down::3 36 2.973e-05 2.973e-05 2.973e-05 0.00% Castro::construct_old_gravity() 5 2.831e-05 2.831e-05 2.831e-05 0.00% Castro::do_advance_ctu() 5 2.731e-05 2.731e-05 2.731e-05 0.00% MLMG::oneIter() 36 2.677e-05 2.677e-05 2.677e-05 0.00% Castro::post_restart() 1 2.672e-05 2.672e-05 2.672e-05 0.00% Gravity::solve_phi_with_mlmg() 6 2.647e-05 2.647e-05 2.647e-05 0.00% MLMG::mgVcycle_up::4 36 2.639e-05 2.639e-05 2.639e-05 0.00% MLMG::mgVcycle_up::3 36 2.293e-05 2.293e-05 2.293e-05 0.00% MLCellLinOp::solutionResidual() 42 2.221e-05 2.221e-05 2.221e-05 0.00% MLMG::mgVcycle_up::0 36 2.147e-05 2.147e-05 2.147e-05 0.00% MLMG::mgVcycle_up::2 36 2.135e-05 2.135e-05 2.135e-05 0.00% MLMG::mgVcycle_up::1 36 2.056e-05 2.056e-05 2.056e-05 0.00% MLPoisson::define() 6 1.967e-05 1.967e-05 1.967e-05 0.00% MLMG::ResNormInf() 42 1.826e-05 1.826e-05 1.826e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.692e-05 1.692e-05 1.692e-05 0.00% FillPatchSingleLevel 20 1.685e-05 1.685e-05 1.685e-05 0.00% MLMG::mgVcycle_bottom 36 1.653e-05 1.653e-05 1.653e-05 0.00% MLMG::computeResidual() 36 1.526e-05 1.526e-05 1.526e-05 0.00% Castro::construct_new_gravity() 5 1.35e-05 1.35e-05 1.35e-05 0.00% Castro::do_old_sources() 5 1.122e-05 1.122e-05 1.122e-05 0.00% MLPoisson::prepareForSolve() 6 1.001e-05 1.001e-05 1.001e-05 0.00% Castro::do_new_sources() 5 9.85e-06 9.85e-06 9.85e-06 0.00% Castro::expand_state() 5 9.042e-06 9.042e-06 9.042e-06 0.00% Gravity::actual_multilevel_solve() 1 8.915e-06 8.915e-06 8.915e-06 0.00% Castro::apply_source_to_state() 10 6.122e-06 6.122e-06 6.122e-06 0.00% Castro::post_timestep() 5 4.571e-06 4.571e-06 4.571e-06 0.00% MLMG::computeMLResidual() 6 4.369e-06 4.369e-06 4.369e-06 0.00% Castro::computeNewDt() 5 3.816e-06 3.816e-06 3.816e-06 0.00% MLMG::getGradSolution() 6 3.713e-06 3.713e-06 3.713e-06 0.00% Amr::init() 1 9.02e-07 9.02e-07 9.02e-07 0.00% Other 2242 0.004069 0.004069 0.004069 0.64% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.631 0.631 0.631 100.00% Amr::coarseTimeStep() 5 0.3392 0.3392 0.3392 53.76% Amr::timeStep() 5 0.3375 0.3375 0.3375 53.49% Castro::advance() 5 0.3313 0.3313 0.3313 52.50% Castro::subcycle_advance_ctu() 5 0.3245 0.3245 0.3245 51.43% Castro::do_advance_ctu() 5 0.3244 0.3244 0.3244 51.41% Amr::init() 1 0.2606 0.2606 0.2606 41.30% Amr::restart() 1 0.2606 0.2606 0.2606 41.30% AmrLevel::restart() 1 0.2486 0.2486 0.2486 39.39% StateData::restartDoit() 4 0.2485 0.2485 0.2485 39.38% VisMF::Read() 3 0.2482 0.2482 0.2482 39.33% Castro::construct_new_gravity() 5 0.1445 0.1445 0.1445 22.91% Gravity::solve_phi_with_mlmg() 6 0.142 0.142 0.142 22.50% Castro::construct_ctu_hydro_source() 5 0.1403 0.1403 0.1403 22.24% Gravity::actual_solve_with_mlmg() 6 0.1364 0.1364 0.1364 21.61% Gravity::solve_for_phi() 5 0.1362 0.1362 0.1362 21.59% MLMG::solve() 6 0.1237 0.1237 0.1237 19.60% MLMG::oneIter() 36 0.1155 0.1155 0.1155 18.30% MLMG::mgVcycle() 36 0.1138 0.1138 0.1138 18.03% MLCellLinOp::smooth() 720 0.05719 0.05719 0.05719 9.06% MLCellLinOp::applyBC() 1910 0.05311 0.05311 0.05311 8.42% MLMG::mgVcycle_bottom 36 0.03362 0.03362 0.03362 5.33% MLMG::actualBottomSolve() 36 0.03361 0.03361 0.03361 5.33% MLCGSolver::bicgstab 36 0.03322 0.03322 0.03322 5.27% Amr::writePlotFile() 1 0.03039 0.03039 0.03039 4.82% Castro::clean_state() 30 0.03016 0.03016 0.03016 4.78% VisMF::Write(FabArray) 1 0.02764 0.02764 0.02764 4.38% AmrLevel::FillPatch() 20 0.02424 0.02424 0.02424 3.84% FillPatchIterator::Initialize 20 0.02211 0.02211 0.02211 3.50% FillPatchIterator::FillFromLevel0() 20 0.02125 0.02125 0.02125 3.37% FillPatchSingleLevel 20 0.02122 0.02122 0.02122 3.36% StateDataPhysBCFunct::() 20 0.01912 0.01912 0.01912 3.03% Castro::computeTemp() 30 0.01775 0.01775 0.01775 2.81% MLCellLinOp::apply() 464 0.01718 0.01718 0.01718 2.72% MLMG::mgVcycle_down::0 36 0.01658 0.01658 0.01658 2.63% MLPoisson::Fsmooth() 1440 0.01596 0.01596 0.01596 2.53% FabArray::FillBoundary() 1730 0.0155 0.0155 0.0155 2.46% FillBoundary_nowait() 1730 0.01515 0.01515 0.01515 2.40% StateData::FillBoundary(geom) 160 0.0134 0.0134 0.0134 2.12% MLMG::mgVcycle_up::0 36 0.01233 0.01233 0.01233 1.95% Castro::initialize_do_advance() 5 0.01182 0.01182 0.01182 1.87% MLPoisson::define() 6 0.009997 0.009997 0.009997 1.58% amrex::Dot() 484 0.009837 0.009837 0.009837 1.56% Castro::do_old_sources() 5 0.009736 0.009736 0.009736 1.54% Castro::reset_internal_energy(MultiFab) 30 0.009723 0.009723 0.009723 1.54% MLMG:computeResOfCorrection() 180 0.009573 0.009573 0.009573 1.52% MLCellLinOp::correctionResidual() 180 0.009518 0.009518 0.009518 1.51% FabArray::norminf() 465 0.009236 0.009236 0.009236 1.46% Gravity::get_new_grav_vector() 5 0.008162 0.008162 0.008162 1.29% Castro::do_new_sources() 5 0.00816 0.00816 0.00816 1.29% Castro::construct_old_gravity() 5 0.008072 0.008072 0.008072 1.28% Gravity::get_old_grav_vector() 5 0.008044 0.008044 0.008044 1.27% MLMG::mgVcycle_down::1 36 0.007858 0.007858 0.007858 1.25% MLMG::mgVcycle_down::2 36 0.007183 0.007183 0.007183 1.14% MLMG::mgVcycle_down::3 36 0.007099 0.007099 0.007099 1.13% FabArray::setVal() 501 0.007022 0.007022 0.007022 1.11% FabArray::ParallelCopy() 380 0.00697 0.00697 0.00697 1.10% MLMG::mgVcycle_down::4 36 0.006941 0.006941 0.006941 1.10% FabArray::ParallelCopy_nowait() 380 0.006853 0.006853 0.006853 1.09% MLCellLinOp::defineAuxData() 6 0.006795 0.006795 0.006795 1.08% Castro::expand_state() 5 0.006722 0.006722 0.006722 1.07% Castro::normalize_species() 30 0.0065 0.0065 0.0065 1.03% Castro::initialize_advance() 5 0.006378 0.006378 0.006378 1.01% Castro::post_restart() 1 0.006376 0.006376 0.006376 1.01% FabArray::Saxpy() 597 0.0062 0.0062 0.0062 0.98% Castro::post_timestep() 5 0.006175 0.006175 0.006175 0.98% Gravity::multilevel_solve_for_new_phi() 1 0.005975 0.005975 0.005975 0.95% Gravity::actual_multilevel_solve() 1 0.005959 0.005959 0.005959 0.94% MLCGSolver::ParallelAllReduce 798 0.005945 0.005945 0.005945 0.94% MLMG::addInterpCorrection() 180 0.00575 0.00575 0.00575 0.91% Castro::enforce_min_density() 30 0.005726 0.005726 0.005726 0.91% amrex::Copy() 221 0.005704 0.005704 0.005704 0.90% MLMG::mgVcycle_up::1 36 0.005616 0.005616 0.005616 0.89% MLMG::mgVcycle_up::4 36 0.00561 0.00561 0.00561 0.89% MLMG::mgVcycle_up::2 36 0.00549 0.00549 0.00549 0.87% Gravity::fill_multipole_BCs() 6 0.005479 0.005479 0.005479 0.87% amrex::average_down 180 0.005431 0.005431 0.005431 0.86% MLMG::mgVcycle_up::3 36 0.005366 0.005366 0.005366 0.85% MLPoisson::Fapply() 464 0.004901 0.004901 0.004901 0.78% MLCellLinOp::solutionResidual() 42 0.003909 0.003909 0.003909 0.62% FabArray::Xpay() 325 0.003709 0.003709 0.003709 0.59% MLMG::computeResidual() 36 0.00305 0.00305 0.00305 0.48% MLMG::prepareForSolve() 6 0.003045 0.003045 0.003045 0.48% MLCellLinOp::defineBC() 6 0.003032 0.003032 0.003032 0.48% BndryData::define() 6 0.002879 0.002879 0.002879 0.46% Castro::estTimeStep() 10 0.00263 0.00263 0.00263 0.42% Castro::construct_new_source() 25 0.001926 0.001926 0.001926 0.31% Castro::construct_new_gravity_source() 5 0.001854 0.001854 0.001854 0.29% amrex::Add() 36 0.001652 0.001652 0.001652 0.26% Castro::construct_old_source() 25 0.001566 0.001566 0.001566 0.25% Castro::construct_old_gravity_source() 5 0.001536 0.001536 0.001536 0.24% Castro::computeNewDt() 5 0.001461 0.001461 0.001461 0.23% Castro::finalize_do_advance() 5 0.001218 0.001218 0.001218 0.19% Castro::reset_internal_energy(Fab) 240 0.001208 0.001208 0.001208 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001067 0.001067 0.001067 0.17% MLMG::ResNormInf() 42 0.001064 0.001064 0.001064 0.17% Castro::apply_source_to_state() 10 0.0009772 0.0009772 0.0009772 0.15% check_for_negative_density() 5 0.0009539 0.0009539 0.0009539 0.15% MLMG::getGradSolution() 6 0.0009355 0.0009355 0.0009355 0.15% MLCellLinOp::compGrad() 6 0.0009318 0.0009318 0.0009318 0.15% MLCellLinOp::setLevelBC() 6 0.0009125 0.0009125 0.0009125 0.14% MLMG::computeMLResidual() 6 0.000878 0.000878 0.000878 0.14% FabArrayBase::getCPC() 632 0.0008087 0.0008087 0.0008087 0.13% MLPoisson::prepareForSolve() 6 0.0007898 0.0007898 0.0007898 0.13% MLCellLinOp::prepareForSolve() 6 0.0007798 0.0007798 0.0007798 0.12% Gravity::update_max_rhs() 6 0.0007718 0.0007718 0.0007718 0.12% Other 2242 0.006065 0.006065 0.006065 0.96% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 9502 KiB 9037 MiB Castro::initMFs() 48 48 40 MiB 68 MiB Castro::swap_state_time_levels() 32 32 32 MiB 55 MiB StateData::restartDoit() 32 32 43 MiB 55 MiB FillPatchIterator::Initialize 160 160 779 KiB 39 MiB Castro::initialize_do_advance() 40 40 20 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1293 KiB 28 MiB Castro::initialize_advance() 40 40 12 MiB 23 MiB Castro::buildMetrics() 32 32 9713 KiB 15 MiB Castro::post_restart() 48 48 4522 KiB 14 MiB MLMG::prepareForSolve() 361 361 2407 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 132 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 132 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 4512 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 13 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2435 B 2048 KiB Gravity::solve_for_phi() 40 40 441 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 19 KiB 2048 KiB BndryData::define() 576 576 221 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 142 KiB 671 KiB Castro::estTimeStep() 10 10 1979 B 480 KiB VisMF::Write(FabArray) 112 112 955 B 320 KiB Castro::normalize_species() 30 30 3348 B 320 KiB amrex::average_down 469 469 1006 B 257 KiB MLMG::addInterpCorrection() 468 468 774 B 257 KiB amrex::Dot() 592 592 2306 B 160 KiB FabArray::norminf() 501 501 2267 B 160 KiB check_for_negative_density() 5 5 243 B 160 KiB MultiFab::max() 6 6 56 B 160 KiB FabArray::setVal() 66 66 14 KiB 27 KiB MultiFab::contains_nan() 10 10 21 B 20 KiB MLPoisson::Fsmooth() 60 60 2332 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 31 B 10 KiB FillBoundary_nowait() 336 336 204 B 9648 B MLCellLinOp::applyBC() 3820 3820 149 B 9344 B amrex::Copy() 56 56 4145 B 8816 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B StateData::FillBoundary(geom) 960 960 30 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCellLinOp::defineBC() 36 36 249 B 1248 B MLCGSolver::bicgstab 180 180 63 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 916 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 60 KiB 8192 KiB VisMF::Write(FabArray) 120 120 125 KiB 3584 KiB VisMF::Read() 24 24 158 KiB 3000 KiB FabArray::setVal() 66 66 14 KiB 27 KiB MLPoisson::Fsmooth() 60 60 2332 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 31 B 10 KiB FillBoundary_nowait() 336 336 204 B 9648 B MLCellLinOp::applyBC() 1910 1910 148 B 9328 B amrex::Copy() 56 56 4145 B 8816 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B Gravity::get_old_grav_vector() 3 3 1769 B 3072 B StateData::FillBoundary(geom) 960 960 30 B 3024 B Gravity::fill_multipole_BCs() 18 18 4 B 2832 B MLMG::prepareForSolve() 7 7 567 B 1648 B amrex::average_down 37 37 326 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::addInterpCorrection() 36 36 1 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 16 B 400 B FabArray::norminf() 501 501 6 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2109 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.07-18-gfdea28f16727) finalized