Initializing AMReX (24.07-10-g85ae84ea8002)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.07-10-g85ae84ea8002) initialized Starting run at 12:36:19 UTC on 2024-07-08. Successfully read inputs file ... Castro git describe: 24.07-3-g79841a784 AMReX git describe: 24.07-10-g85ae84ea8 Microphysics git describe: 24.07-4-g52f71c84 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.049536319 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.0277951 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.060878819 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052879597 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.050554664 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.061560607 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.062555933 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.056254006 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052343599 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.053879596 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.068726173 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.071442098 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062751745 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.05882753 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.027420614 seconds Ending run at 12:36:20 UTC on 2024-07-08. Run time = 0.87348807 Run time without initialization = 0.740690215 Average number of zones advanced per microsecond: 3.539 Average number of zones advanced per microsecond per rank: 3.539 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.8735 ... 0.8735 ... 0.8735 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2111 0.2111 0.2111 24.16% VisMF::Write(FabArray) 11 0.2006 0.2006 0.2006 22.97% MLCellLinOp::applyBC() 4351 0.08498 0.08498 0.08498 9.73% MLPoisson::Fsmooth() 3280 0.03553 0.03553 0.03553 4.07% FillBoundary_nowait() 3941 0.03293 0.03293 0.03293 3.77% StateData::FillBoundary(geom) 328 0.02746 0.02746 0.02746 3.14% amrex::Dot() 1114 0.02289 0.02289 0.02289 2.62% FabArray::norminf() 1061 0.02114 0.02114 0.02114 2.42% Castro::reset_internal_energy(MultiFab) 63 0.01824 0.01824 0.01824 2.09% Amr::checkPoint() 3 0.01687 0.01687 0.01687 1.93% FabArray::ParallelCopy_nowait() 861 0.01424 0.01424 0.01424 1.63% Castro::normalize_species() 62 0.0142 0.0142 0.0142 1.63% FabArray::setVal() 1062 0.0141 0.0141 0.0141 1.61% FabArray::Saxpy() 1370 0.01387 0.01387 0.01387 1.59% StateDataPhysBCFunct::() 41 0.01179 0.01179 0.01179 1.35% amrex::Copy() 472 0.01126 0.01126 0.01126 1.29% MLPoisson::Fapply() 1060 0.01096 0.01096 0.01096 1.25% MLCellLinOp::defineAuxData() 11 0.01075 0.01075 0.01075 1.23% Gravity::fill_multipole_BCs() 11 0.009885 0.009885 0.009885 1.13% Castro::enforce_min_density() 62 0.009021 0.009021 0.009021 1.03% FabArray::Xpay() 739 0.008282 0.008282 0.008282 0.95% MLMG::addInterpCorrection() 410 0.007297 0.007297 0.007297 0.84% amrex::average_down 410 0.006584 0.006584 0.006584 0.75% Castro::estTimeStep() 21 0.005004 0.005004 0.005004 0.57% BndryData::define() 11 0.004176 0.004176 0.004176 0.48% amrex::Add() 82 0.0038 0.0038 0.0038 0.43% Castro::construct_new_gravity_source() 10 0.003553 0.003553 0.003553 0.41% Castro::computeTemp() 63 0.0031 0.0031 0.0031 0.35% Castro::enforce_speed_limit() 62 0.003 0.003 0.003 0.34% Castro::construct_old_gravity_source() 10 0.002904 0.002904 0.002904 0.33% Castro::reset_internal_energy(Fab) 504 0.001945 0.001945 0.001945 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001854 0.001854 0.001854 0.21% Amr::writePlotFile() 2 0.00182 0.00182 0.00182 0.21% Castro::initData() 1 0.001657 0.001657 0.001657 0.19% MLCellLinOp::setLevelBC() 11 0.001655 0.001655 0.001655 0.19% MLCGSolver::bicgstab 82 0.00164 0.00164 0.00164 0.19% Gravity::actual_solve_with_mlmg() 11 0.001594 0.001594 0.001594 0.18% FabArray::setDomainBndry() 41 0.001458 0.001458 0.001458 0.17% FabArray::mult() 43 0.001418 0.001418 0.001418 0.16% MLCellLinOp::prepareForSolve() 11 0.001378 0.001378 0.001378 0.16% MultiFab::contains_nan() 20 0.001301 0.001301 0.001301 0.15% check_for_negative_density() 10 0.001287 0.001287 0.001287 0.15% MLCellLinOp::smooth() 1640 0.001205 0.001205 0.001205 0.14% MLCellLinOp::compGrad() 11 0.001124 0.001124 0.001124 0.13% MLMG::prepareForSolve() 11 0.0009652 0.0009652 0.0009652 0.11% FabArrayBase::getCPC() 1323 0.0007907 0.0007907 0.0007907 0.09% FabArray::FillBoundary() 3941 0.0007666 0.0007666 0.0007666 0.09% Gravity::get_new_grav_vector() 11 0.0006815 0.0006815 0.0006815 0.08% Gravity::get_old_grav_vector() 10 0.0005992 0.0005992 0.0005992 0.07% MLCellLinOp::apply() 1060 0.0004342 0.0004342 0.0004342 0.05% AmrLevel::FillPatch() 41 0.0004101 0.0004101 0.0004101 0.05% MLCGSolver::ParallelAllReduce 1832 0.0003273 0.0003273 0.0003273 0.04% main() 1 0.000319 0.000319 0.000319 0.04% Amr::coarseTimeStep() 10 0.0003122 0.0003122 0.0003122 0.04% Castro::subcycle_advance_ctu() 10 0.0002619 0.0002619 0.0002619 0.03% MLCellLinOp::defineBC() 11 0.0002611 0.0002611 0.0002611 0.03% FabArray::ParallelCopy() 861 0.0002469 0.0002469 0.0002469 0.03% FillPatchIterator::Initialize 41 0.0002234 0.0002234 0.0002234 0.03% MLMG::mgVcycle() 82 0.0002137 0.0002137 0.0002137 0.02% MLCellLinOp::correctionResidual() 410 0.0001663 0.0001663 0.0001663 0.02% Amr::timeStep() 10 0.0001603 0.0001603 0.0001603 0.02% Castro::advance() 10 0.0001509 0.0001509 0.0001509 0.02% StateData::checkPoint() 12 0.000129 0.000129 0.000129 0.01% MLMG:computeResOfCorrection() 410 0.0001257 0.0001257 0.0001257 0.01% Gravity::solve_for_phi() 10 0.0001068 0.0001068 0.0001068 0.01% MLMG::mgVcycle_down::0 82 8.684e-05 8.684e-05 8.684e-05 0.01% MLMG::actualBottomSolve() 82 8.629e-05 8.629e-05 8.629e-05 0.01% Castro::initialize_advance() 10 8.307e-05 8.307e-05 8.307e-05 0.01% MLMG::mgVcycle_down::2 82 8.049e-05 8.049e-05 8.049e-05 0.01% MLMG::mgVcycle_down::1 82 7.898e-05 7.898e-05 7.898e-05 0.01% MLMG::mgVcycle_down::4 82 7.648e-05 7.648e-05 7.648e-05 0.01% MLMG::mgVcycle_down::3 82 7.349e-05 7.349e-05 7.349e-05 0.01% Castro::do_advance_ctu() 10 7.011e-05 7.011e-05 7.011e-05 0.01% Castro::clean_state() 62 6.866e-05 6.866e-05 6.866e-05 0.01% MLMG::mgVcycle_up::0 82 6.856e-05 6.856e-05 6.856e-05 0.01% MLMG::solve() 11 6.828e-05 6.828e-05 6.828e-05 0.01% AmrLevel::checkPoint() 3 6.695e-05 6.695e-05 6.695e-05 0.01% Castro::initialize_do_advance() 10 6.09e-05 6.09e-05 6.09e-05 0.01% MLMG::mgVcycle_up::4 82 5.602e-05 5.602e-05 5.602e-05 0.01% MLMG::oneIter() 82 5.237e-05 5.237e-05 5.237e-05 0.01% MLMG::mgVcycle_up::3 82 5.195e-05 5.195e-05 5.195e-05 0.01% MLMG::mgVcycle_up::1 82 5.103e-05 5.103e-05 5.103e-05 0.01% MLMG::mgVcycle_up::2 82 5.028e-05 5.028e-05 5.028e-05 0.01% FillPatchIterator::FillFromLevel0() 41 4.729e-05 4.729e-05 4.729e-05 0.01% Castro::finalize_do_advance() 10 4.583e-05 4.583e-05 4.583e-05 0.01% MLCellLinOp::solutionResidual() 93 4.511e-05 4.511e-05 4.511e-05 0.01% MLMG::ResNormInf() 93 3.992e-05 3.992e-05 3.992e-05 0.00% Castro::do_new_sources() 10 3.841e-05 3.841e-05 3.841e-05 0.00% FillPatchSingleLevel 41 3.507e-05 3.507e-05 3.507e-05 0.00% MLMG::mgVcycle_bottom 82 3.442e-05 3.442e-05 3.442e-05 0.00% MLMG::computeResidual() 82 3.346e-05 3.346e-05 3.346e-05 0.00% Amr::defBaseLevel() 1 3.134e-05 3.134e-05 3.134e-05 0.00% Castro::construct_new_gravity() 10 2.722e-05 2.722e-05 2.722e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.463e-05 2.463e-05 2.463e-05 0.00% MLPoisson::define() 11 2.413e-05 2.413e-05 2.413e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.293e-05 2.293e-05 2.293e-05 0.00% Castro::do_old_sources() 10 2.092e-05 2.092e-05 2.092e-05 0.00% Castro::construct_new_source() 50 2.043e-05 2.043e-05 2.043e-05 0.00% Amr::FinalizeInit() 1 1.978e-05 1.978e-05 1.978e-05 0.00% Castro::construct_old_source() 50 1.826e-05 1.826e-05 1.826e-05 0.00% MLPoisson::prepareForSolve() 11 1.408e-05 1.408e-05 1.408e-05 0.00% Castro::construct_old_gravity() 10 1.227e-05 1.227e-05 1.227e-05 0.00% Castro::computeNewDt() 9 1.171e-05 1.171e-05 1.171e-05 0.00% Castro::apply_source_to_state() 20 1.158e-05 1.158e-05 1.158e-05 0.00% Castro::check_for_nan() 20 1.089e-05 1.089e-05 1.089e-05 0.00% MLMG::computeMLResidual() 11 1.079e-05 1.079e-05 1.079e-05 0.00% Gravity::actual_multilevel_solve() 1 9.148e-06 9.148e-06 9.148e-06 0.00% Castro::post_timestep() 10 8.568e-06 8.568e-06 8.568e-06 0.00% MLMG::getGradSolution() 11 5.825e-06 5.825e-06 5.825e-06 0.00% Amr::InitializeInit() 1 5.655e-06 5.655e-06 5.655e-06 0.00% Castro::expand_state() 10 5.383e-06 5.383e-06 5.383e-06 0.00% Castro::post_init() 1 4.333e-06 4.333e-06 4.333e-06 0.00% Amr::init() 1 2.589e-06 2.589e-06 2.589e-06 0.00% Amr::initialInit() 1 1.148e-06 1.148e-06 1.148e-06 0.00% Other 4753 0.003107 0.003107 0.003107 0.36% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8735 0.8735 0.8735 100.00% Amr::coarseTimeStep() 10 0.713 0.713 0.713 81.63% Amr::timeStep() 10 0.5947 0.5947 0.5947 68.08% Castro::advance() 10 0.5851 0.5851 0.5851 66.98% Castro::subcycle_advance_ctu() 10 0.5738 0.5738 0.5738 65.69% Castro::do_advance_ctu() 10 0.5735 0.5735 0.5735 65.66% Gravity::solve_phi_with_mlmg() 11 0.3108 0.3108 0.3108 35.58% Gravity::actual_solve_with_mlmg() 11 0.3005 0.3005 0.3005 34.40% Castro::construct_new_gravity() 10 0.2819 0.2819 0.2819 32.27% MLMG::solve() 11 0.2776 0.2776 0.2776 31.78% Gravity::solve_for_phi() 10 0.2653 0.2653 0.2653 30.37% MLMG::oneIter() 82 0.2614 0.2614 0.2614 29.93% MLMG::mgVcycle() 82 0.2576 0.2576 0.2576 29.49% Castro::construct_ctu_hydro_source() 10 0.2212 0.2212 0.2212 25.33% VisMF::Write(FabArray) 11 0.2006 0.2006 0.2006 22.97% Amr::checkPoint() 3 0.1647 0.1647 0.1647 18.86% AmrLevel::checkPoint() 3 0.1478 0.1478 0.1478 16.93% StateData::checkPoint() 12 0.1478 0.1478 0.1478 16.92% Amr::init() 1 0.1322 0.1322 0.1322 15.13% MLCellLinOp::smooth() 1640 0.1286 0.1286 0.1286 14.73% MLCellLinOp::applyBC() 4351 0.1194 0.1194 0.1194 13.67% MLMG::mgVcycle_bottom 82 0.07737 0.07737 0.07737 8.86% MLMG::actualBottomSolve() 82 0.07733 0.07733 0.07733 8.85% MLCGSolver::bicgstab 82 0.07647 0.07647 0.07647 8.75% Amr::writePlotFile() 2 0.05535 0.05535 0.05535 6.34% Amr::initialInit() 1 0.05469 0.05469 0.05469 6.26% Amr::FinalizeInit() 1 0.04952 0.04952 0.04952 5.67% AmrLevel::FillPatch() 41 0.0495 0.0495 0.0495 5.67% Castro::clean_state() 62 0.04873 0.04873 0.04873 5.58% Castro::post_init() 1 0.04856 0.04856 0.04856 5.56% Gravity::multilevel_solve_for_new_phi() 1 0.04601 0.04601 0.04601 5.27% Gravity::actual_multilevel_solve() 1 0.04598 0.04598 0.04598 5.26% FillPatchIterator::Initialize 41 0.04517 0.04517 0.04517 5.17% FillPatchIterator::FillFromLevel0() 41 0.04349 0.04349 0.04349 4.98% FillPatchSingleLevel 41 0.04344 0.04344 0.04344 4.97% StateDataPhysBCFunct::() 41 0.03925 0.03925 0.03925 4.49% MLCellLinOp::apply() 1060 0.03834 0.03834 0.03834 4.39% MLMG::mgVcycle_down::0 82 0.03649 0.03649 0.03649 4.18% MLPoisson::Fsmooth() 3280 0.03553 0.03553 0.03553 4.07% FabArray::FillBoundary() 3941 0.0344 0.0344 0.0344 3.94% FillBoundary_nowait() 3941 0.03363 0.03363 0.03363 3.85% MLMG::mgVcycle_up::0 82 0.02769 0.02769 0.02769 3.17% StateData::FillBoundary(geom) 328 0.02746 0.02746 0.02746 3.14% Castro::computeTemp() 63 0.02329 0.02329 0.02329 2.67% amrex::Dot() 1114 0.02289 0.02289 0.02289 2.62% MLMG:computeResOfCorrection() 410 0.02152 0.02152 0.02152 2.46% MLCellLinOp::correctionResidual() 410 0.0214 0.0214 0.0214 2.45% FabArray::norminf() 1061 0.02114 0.02114 0.02114 2.42% Castro::initialize_do_advance() 10 0.02056 0.02056 0.02056 2.35% Castro::reset_internal_energy(MultiFab) 63 0.02019 0.02019 0.02019 2.31% Gravity::get_new_grav_vector() 11 0.01871 0.01871 0.01871 2.14% Castro::do_old_sources() 10 0.01859 0.01859 0.01859 2.13% MLPoisson::define() 11 0.01795 0.01795 0.01795 2.06% MLMG::mgVcycle_down::1 82 0.0176 0.0176 0.0176 2.01% MLMG::mgVcycle_down::2 82 0.0164 0.0164 0.0164 1.88% Castro::construct_old_gravity() 10 0.01599 0.01599 0.01599 1.83% Gravity::get_old_grav_vector() 10 0.01598 0.01598 0.01598 1.83% MLMG::mgVcycle_down::3 82 0.01594 0.01594 0.01594 1.82% MLMG::mgVcycle_down::4 82 0.01589 0.01589 0.01589 1.82% FabArray::ParallelCopy() 861 0.0153 0.0153 0.0153 1.75% FabArray::ParallelCopy_nowait() 861 0.01505 0.01505 0.01505 1.72% Castro::normalize_species() 62 0.0142 0.0142 0.0142 1.63% FabArray::setVal() 1062 0.0141 0.0141 0.0141 1.61% FabArray::Saxpy() 1370 0.01387 0.01387 0.01387 1.59% MLCGSolver::ParallelAllReduce 1832 0.01371 0.01371 0.01371 1.57% MLMG::addInterpCorrection() 410 0.01285 0.01285 0.01285 1.47% MLMG::mgVcycle_up::1 82 0.01274 0.01274 0.01274 1.46% MLMG::mgVcycle_up::4 82 0.01267 0.01267 0.01267 1.45% MLMG::mgVcycle_up::2 82 0.01241 0.01241 0.01241 1.42% Castro::do_new_sources() 10 0.01233 0.01233 0.01233 1.41% MLCellLinOp::defineAuxData() 11 0.0122 0.0122 0.0122 1.40% amrex::average_down 410 0.01218 0.01218 0.01218 1.39% MLMG::mgVcycle_up::3 82 0.01215 0.01215 0.01215 1.39% Castro::expand_state() 10 0.01209 0.01209 0.01209 1.38% amrex::Copy() 472 0.01126 0.01126 0.01126 1.29% MLPoisson::Fapply() 1060 0.01096 0.01096 0.01096 1.25% Castro::initialize_advance() 10 0.01047 0.01047 0.01047 1.20% Gravity::fill_multipole_BCs() 11 0.01014 0.01014 0.01014 1.16% Castro::post_timestep() 10 0.009471 0.009471 0.009471 1.08% Castro::enforce_min_density() 62 0.009021 0.009021 0.009021 1.03% FabArray::Xpay() 739 0.008282 0.008282 0.008282 0.95% MLCellLinOp::solutionResidual() 93 0.008144 0.008144 0.008144 0.93% MLMG::computeResidual() 82 0.006835 0.006835 0.006835 0.78% MLCellLinOp::defineBC() 11 0.005489 0.005489 0.005489 0.63% MLMG::prepareForSolve() 11 0.005364 0.005364 0.005364 0.61% BndryData::define() 11 0.005228 0.005228 0.005228 0.60% Amr::InitializeInit() 1 0.005167 0.005167 0.005167 0.59% Amr::defBaseLevel() 1 0.005161 0.005161 0.005161 0.59% Castro::estTimeStep() 21 0.005004 0.005004 0.005004 0.57% Castro::initData() 1 0.004422 0.004422 0.004422 0.51% amrex::Add() 82 0.0038 0.0038 0.0038 0.43% Castro::construct_new_source() 50 0.003573 0.003573 0.003573 0.41% Castro::construct_new_gravity_source() 10 0.003553 0.003553 0.003553 0.41% Castro::enforce_speed_limit() 62 0.003 0.003 0.003 0.34% Castro::construct_old_source() 50 0.002922 0.002922 0.002922 0.33% Castro::construct_old_gravity_source() 10 0.002904 0.002904 0.002904 0.33% MLMG::ResNormInf() 93 0.002298 0.002298 0.002298 0.26% Castro::computeNewDt() 9 0.002162 0.002162 0.002162 0.25% Castro::reset_internal_energy(Fab) 504 0.001945 0.001945 0.001945 0.22% Castro::finalize_do_advance() 10 0.00193 0.00193 0.00193 0.22% Castro::apply_source_to_state() 20 0.001891 0.001891 0.001891 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001854 0.001854 0.001854 0.21% MLMG::getGradSolution() 11 0.001663 0.001663 0.001663 0.19% MLCellLinOp::compGrad() 11 0.001657 0.001657 0.001657 0.19% MLCellLinOp::setLevelBC() 11 0.001655 0.001655 0.001655 0.19% FabArrayBase::getCPC() 1323 0.001464 0.001464 0.001464 0.17% FabArray::setDomainBndry() 41 0.001458 0.001458 0.001458 0.17% FabArray::mult() 43 0.001418 0.001418 0.001418 0.16% MLPoisson::prepareForSolve() 11 0.001392 0.001392 0.001392 0.16% MLCellLinOp::prepareForSolve() 11 0.001378 0.001378 0.001378 0.16% MLMG::computeMLResidual() 11 0.001353 0.001353 0.001353 0.15% Castro::check_for_nan() 20 0.001312 0.001312 0.001312 0.15% MultiFab::contains_nan() 20 0.001301 0.001301 0.001301 0.15% check_for_negative_density() 10 0.001287 0.001287 0.001287 0.15% Other 4753 0.008688 0.008688 0.008688 0.99% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 6478 KiB 9037 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1096 KiB 39 MiB Castro::initialize_do_advance() 80 80 25 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1782 KiB 28 MiB Castro::initialize_advance() 80 80 15 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7616 KiB 14 MiB MLMG::prepareForSolve() 660 660 3909 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 222 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 187 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7518 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 19 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2410 B 2048 KiB Gravity::solve_for_phi() 80 80 621 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 107 KiB 2048 KiB BndryData::define() 1056 1056 357 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 227 KiB 671 KiB Castro::estTimeStep() 21 21 2784 B 480 KiB VisMF::Write(FabArray) 656 656 3733 B 320 KiB Castro::normalize_species() 62 62 5285 B 320 KiB amrex::average_down 1067 1067 1695 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1258 B 257 KiB amrex::Dot() 1360 1360 3884 B 160 KiB FabArray::norminf() 1143 1143 3751 B 160 KiB check_for_negative_density() 10 10 234 B 160 KiB Castro::initData() 1 1 58 B 160 KiB MultiFab::max() 11 11 62 B 160 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MultiFab::contains_nan() 20 20 29 B 20 KiB MLPoisson::Fsmooth() 132 132 3817 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 47 B 10 KiB FillBoundary_nowait() 760 760 328 B 9648 B MLCellLinOp::applyBC() 8702 8702 242 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3918 B 6144 B StateData::FillBoundary(geom) 1992 1992 43 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 402 B 1248 B MLCGSolver::bicgstab 410 410 105 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 778 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 42 KiB 8192 KiB VisMF::Write(FabArray) 744 744 515 KiB 3584 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MLPoisson::Fsmooth() 132 132 3817 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 47 B 10 KiB FillBoundary_nowait() 760 760 327 B 9648 B MLCellLinOp::applyBC() 4351 4351 240 B 9328 B MLCellLinOp::prepareForSolve() 66 66 4 B 7792 B amrex::Copy() 100 100 3918 B 6144 B Gravity::get_new_grav_vector() 3 3 2882 B 3072 B StateData::FillBoundary(geom) 1992 1992 44 B 3024 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B amrex::average_down 83 83 617 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 325 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 28 B 400 B FabArray::norminf() 1143 1143 11 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2109 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.07-10-g85ae84ea8002) finalized Initializing AMReX (24.07-10-g85ae84ea8002)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.07-10-g85ae84ea8002) initialized Starting run at 12:36:21 UTC on 2024-07-08. Successfully read inputs file ... Castro git describe: 24.07-3-g79841a784 AMReX git describe: 24.07-10-g85ae84ea8 Microphysics git describe: 24.07-4-g52f71c84 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.477458275 Restart time = 0.074326877 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.068842972 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.057116476 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057277597 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060012461 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.058173988 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.030051867 seconds Ending run at 12:36:21 UTC on 2024-07-08. Run time = 0.406781939 Run time without initialization = 0.331897826 Average number of zones advanced per microsecond: 3.949 Average number of zones advanced per microsecond per rank: 3.949 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.4068 ... 0.4068 ... 0.4068 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1010 0.1010 0.1010 24.83% VisMF::Read() 3 0.06235 0.06235 0.06235 15.33% MLCellLinOp::applyBC() 1910 0.03741 0.03741 0.03741 9.20% VisMF::Write(FabArray) 1 0.02752 0.02752 0.02752 6.76% MLPoisson::Fsmooth() 1440 0.0159 0.0159 0.0159 3.91% FillBoundary_nowait() 1730 0.01504 0.01504 0.01504 3.70% StateData::FillBoundary(geom) 160 0.01321 0.01321 0.01321 3.25% amrex::Dot() 484 0.009764 0.009764 0.009764 2.40% FabArray::norminf() 465 0.009174 0.009174 0.009174 2.26% Castro::reset_internal_energy(MultiFab) 30 0.008395 0.008395 0.008395 2.06% Castro::normalize_species() 30 0.008011 0.008011 0.008011 1.97% FabArray::setVal() 501 0.006963 0.006963 0.006963 1.71% FabArray::ParallelCopy_nowait() 380 0.006399 0.006399 0.006399 1.57% FabArray::Saxpy() 597 0.006161 0.006161 0.006161 1.51% MLCellLinOp::defineAuxData() 6 0.005953 0.005953 0.005953 1.46% amrex::Copy() 221 0.005698 0.005698 0.005698 1.40% Gravity::fill_multipole_BCs() 6 0.005693 0.005693 0.005693 1.40% StateDataPhysBCFunct::() 20 0.005568 0.005568 0.005568 1.37% Amr::restart() 1 0.005184 0.005184 0.005184 1.27% MLPoisson::Fapply() 464 0.004805 0.004805 0.004805 1.18% Castro::estTimeStep() 10 0.004723 0.004723 0.004723 1.16% FabArray::Xpay() 325 0.003675 0.003675 0.003675 0.90% MLMG::addInterpCorrection() 180 0.003292 0.003292 0.003292 0.81% amrex::average_down 180 0.002926 0.002926 0.002926 0.72% Castro::enforce_min_density() 30 0.002648 0.002648 0.002648 0.65% Amr::writePlotFile() 1 0.002366 0.002366 0.002366 0.58% BndryData::define() 6 0.002333 0.002333 0.002333 0.57% Castro::construct_new_gravity_source() 5 0.001916 0.001916 0.001916 0.47% Castro::computeTemp() 30 0.001844 0.001844 0.001844 0.45% amrex::Add() 36 0.001645 0.001645 0.001645 0.40% Castro::construct_old_gravity_source() 5 0.001556 0.001556 0.001556 0.38% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001033 0.001033 0.001033 0.25% Castro::reset_internal_energy(Fab) 240 0.001016 0.001016 0.001016 0.25% MLCellLinOp::setLevelBC() 6 0.0009283 0.0009283 0.0009283 0.23% Gravity::actual_solve_with_mlmg() 6 0.0008706 0.0008706 0.0008706 0.21% Castro::finalize_do_advance() 5 0.0008633 0.0008633 0.0008633 0.21% Castro::enforce_speed_limit() 30 0.0008476 0.0008476 0.0008476 0.21% MLCellLinOp::prepareForSolve() 6 0.0007721 0.0007721 0.0007721 0.19% FabArray::setDomainBndry() 20 0.0007312 0.0007312 0.0007312 0.18% FabArray::mult() 22 0.0007134 0.0007134 0.0007134 0.18% MLCGSolver::bicgstab 36 0.0007101 0.0007101 0.0007101 0.17% MLCellLinOp::compGrad() 6 0.0006672 0.0006672 0.0006672 0.16% MLMG::prepareForSolve() 6 0.0005622 0.0005622 0.0005622 0.14% MLCellLinOp::smooth() 720 0.0005022 0.0005022 0.0005022 0.12% Gravity::get_old_grav_vector() 5 0.0004045 0.0004045 0.0004045 0.10% FabArrayBase::getCPC() 632 0.0003894 0.0003894 0.0003894 0.10% FabArray::FillBoundary() 1730 0.0003342 0.0003342 0.0003342 0.08% Gravity::get_new_grav_vector() 5 0.0002795 0.0002795 0.0002795 0.07% main() 1 0.0002684 0.0002684 0.0002684 0.07% Castro::subcycle_advance_ctu() 5 0.0002201 0.0002201 0.0002201 0.05% AmrLevel::FillPatch() 20 0.0002032 0.0002032 0.0002032 0.05% MLCellLinOp::apply() 464 0.0001901 0.0001901 0.0001901 0.05% Amr::coarseTimeStep() 5 0.0001518 0.0001518 0.0001518 0.04% Castro::advance() 5 0.0001486 0.0001486 0.0001486 0.04% MLCellLinOp::defineBC() 6 0.0001458 0.0001458 0.0001458 0.04% MLCGSolver::ParallelAllReduce 798 0.0001437 0.0001437 0.0001437 0.04% FabArray::ParallelCopy() 380 0.0001098 0.0001098 0.0001098 0.03% Castro::initialize_do_advance() 5 0.0001055 0.0001055 0.0001055 0.03% FillPatchIterator::Initialize 20 0.0001044 0.0001044 0.0001044 0.03% Amr::timeStep() 5 8.791e-05 8.791e-05 8.791e-05 0.02% MLMG::mgVcycle() 36 8.691e-05 8.691e-05 8.691e-05 0.02% AmrLevel::restart() 1 7.852e-05 7.852e-05 7.852e-05 0.02% MLCellLinOp::correctionResidual() 180 7.281e-05 7.281e-05 7.281e-05 0.02% Gravity::update_max_rhs() 6 6.818e-05 6.818e-05 6.818e-05 0.02% Castro::initialize_advance() 5 6.456e-05 6.456e-05 6.456e-05 0.02% Castro::do_new_sources() 5 6.124e-05 6.124e-05 6.124e-05 0.02% StateData::restartDoit() 4 6.12e-05 6.12e-05 6.12e-05 0.02% MLMG:computeResOfCorrection() 180 5.686e-05 5.686e-05 5.686e-05 0.01% Castro::do_advance_ctu() 5 5.237e-05 5.237e-05 5.237e-05 0.01% Gravity::solve_for_phi() 5 4.859e-05 4.859e-05 4.859e-05 0.01% MLMG::mgVcycle_down::0 36 4.165e-05 4.165e-05 4.165e-05 0.01% MLMG::actualBottomSolve() 36 3.811e-05 3.811e-05 3.811e-05 0.01% Castro::do_old_sources() 5 3.642e-05 3.642e-05 3.642e-05 0.01% MLMG::mgVcycle_down::1 36 3.5e-05 3.5e-05 3.5e-05 0.01% MLMG::solve() 6 3.369e-05 3.369e-05 3.369e-05 0.01% MLMG::mgVcycle_down::2 36 3.258e-05 3.258e-05 3.258e-05 0.01% Castro::clean_state() 30 3.255e-05 3.255e-05 3.255e-05 0.01% MLMG::mgVcycle_down::4 36 3.057e-05 3.057e-05 3.057e-05 0.01% MLMG::mgVcycle_down::3 36 3.026e-05 3.026e-05 3.026e-05 0.01% MLMG::mgVcycle_up::4 36 2.675e-05 2.675e-05 2.675e-05 0.01% Castro::post_restart() 1 2.469e-05 2.469e-05 2.469e-05 0.01% MLMG::oneIter() 36 2.419e-05 2.419e-05 2.419e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.339e-05 2.339e-05 2.339e-05 0.01% MLMG::mgVcycle_up::3 36 2.331e-05 2.331e-05 2.331e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 2.236e-05 2.236e-05 2.236e-05 0.01% MLMG::mgVcycle_up::0 36 2.139e-05 2.139e-05 2.139e-05 0.01% MLMG::mgVcycle_up::2 36 2.119e-05 2.119e-05 2.119e-05 0.01% MLCellLinOp::solutionResidual() 42 2.074e-05 2.074e-05 2.074e-05 0.01% MLMG::ResNormInf() 42 2.058e-05 2.058e-05 2.058e-05 0.01% MLMG::mgVcycle_up::1 36 1.985e-05 1.985e-05 1.985e-05 0.00% MLPoisson::define() 6 1.943e-05 1.943e-05 1.943e-05 0.00% FillPatchSingleLevel 20 1.668e-05 1.668e-05 1.668e-05 0.00% MLMG::mgVcycle_bottom 36 1.55e-05 1.55e-05 1.55e-05 0.00% MLMG::computeResidual() 36 1.5e-05 1.5e-05 1.5e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.335e-05 1.335e-05 1.335e-05 0.00% Castro::construct_new_gravity() 5 1.305e-05 1.305e-05 1.305e-05 0.00% Castro::construct_new_source() 25 1.061e-05 1.061e-05 1.061e-05 0.00% MLPoisson::prepareForSolve() 6 1.053e-05 1.053e-05 1.053e-05 0.00% Castro::construct_old_source() 25 1.018e-05 1.018e-05 1.018e-05 0.00% Castro::expand_state() 5 8.324e-06 8.324e-06 8.324e-06 0.00% Gravity::actual_multilevel_solve() 1 8.172e-06 8.172e-06 8.172e-06 0.00% Castro::apply_source_to_state() 10 6.077e-06 6.077e-06 6.077e-06 0.00% Castro::construct_old_gravity() 5 5.51e-06 5.51e-06 5.51e-06 0.00% Castro::check_for_nan() 10 5.356e-06 5.356e-06 5.356e-06 0.00% Castro::post_timestep() 5 4.799e-06 4.799e-06 4.799e-06 0.00% MLMG::computeMLResidual() 6 4.427e-06 4.427e-06 4.427e-06 0.00% Castro::computeNewDt() 5 3.981e-06 3.981e-06 3.981e-06 0.00% MLMG::getGradSolution() 6 3.325e-06 3.325e-06 3.325e-06 0.00% Amr::init() 1 8.68e-07 8.68e-07 8.68e-07 0.00% Other 2165 0.002772 0.002772 0.002772 0.68% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4068 0.4068 0.4068 99.99% Amr::coarseTimeStep() 5 0.3016 0.3016 0.3016 74.13% Amr::timeStep() 5 0.2985 0.2985 0.2985 73.38% Castro::advance() 5 0.2951 0.2951 0.2951 72.53% Castro::subcycle_advance_ctu() 5 0.2891 0.2891 0.2891 71.06% Castro::do_advance_ctu() 5 0.2888 0.2888 0.2888 71.00% Castro::construct_new_gravity() 5 0.1445 0.1445 0.1445 35.52% Gravity::solve_phi_with_mlmg() 6 0.1419 0.1419 0.1419 34.88% Gravity::solve_for_phi() 5 0.1363 0.1363 0.1363 33.50% Gravity::actual_solve_with_mlmg() 6 0.1359 0.1359 0.1359 33.42% MLMG::solve() 6 0.1231 0.1231 0.1231 30.27% MLMG::oneIter() 36 0.1149 0.1149 0.1149 28.25% MLMG::mgVcycle() 36 0.1133 0.1133 0.1133 27.84% Castro::construct_ctu_hydro_source() 5 0.1054 0.1054 0.1054 25.91% Amr::init() 1 0.07437 0.07437 0.07437 18.28% Amr::restart() 1 0.07437 0.07437 0.07437 18.28% AmrLevel::restart() 1 0.06272 0.06272 0.06272 15.42% StateData::restartDoit() 4 0.06264 0.06264 0.06264 15.40% VisMF::Read() 3 0.06235 0.06235 0.06235 15.33% MLCellLinOp::smooth() 720 0.05714 0.05714 0.05714 14.05% MLCellLinOp::applyBC() 1910 0.05311 0.05311 0.05311 13.06% MLMG::mgVcycle_bottom 36 0.03334 0.03334 0.03334 8.20% MLMG::actualBottomSolve() 36 0.03333 0.03333 0.03333 8.19% MLCGSolver::bicgstab 36 0.03295 0.03295 0.03295 8.10% Amr::writePlotFile() 1 0.03016 0.03016 0.03016 7.41% VisMF::Write(FabArray) 1 0.02752 0.02752 0.02752 6.76% AmrLevel::FillPatch() 20 0.02386 0.02386 0.02386 5.86% Castro::clean_state() 30 0.02279 0.02279 0.02279 5.60% FillPatchIterator::Initialize 20 0.02173 0.02173 0.02173 5.34% FillPatchIterator::FillFromLevel0() 20 0.02089 0.02089 0.02089 5.14% FillPatchSingleLevel 20 0.02087 0.02087 0.02087 5.13% StateDataPhysBCFunct::() 20 0.01878 0.01878 0.01878 4.62% MLCellLinOp::apply() 464 0.01706 0.01706 0.01706 4.19% MLMG::mgVcycle_down::0 36 0.01659 0.01659 0.01659 4.08% MLPoisson::Fsmooth() 1440 0.0159 0.0159 0.0159 3.91% FabArray::FillBoundary() 1730 0.0157 0.0157 0.0157 3.86% FillBoundary_nowait() 1730 0.01536 0.01536 0.01536 3.78% StateData::FillBoundary(geom) 160 0.01321 0.01321 0.01321 3.25% MLMG::mgVcycle_up::0 36 0.01244 0.01244 0.01244 3.06% Castro::computeTemp() 30 0.01126 0.01126 0.01126 2.77% Castro::do_old_sources() 5 0.01104 0.01104 0.01104 2.71% Castro::initialize_do_advance() 5 0.01081 0.01081 0.01081 2.66% MLPoisson::define() 6 0.01004 0.01004 0.01004 2.47% amrex::Dot() 484 0.009764 0.009764 0.009764 2.40% MLMG:computeResOfCorrection() 180 0.00954 0.00954 0.00954 2.34% MLCellLinOp::correctionResidual() 180 0.009483 0.009483 0.009483 2.33% Castro::reset_internal_energy(MultiFab) 30 0.009411 0.009411 0.009411 2.31% FabArray::norminf() 465 0.009174 0.009174 0.009174 2.26% Gravity::get_new_grav_vector() 5 0.008116 0.008116 0.008116 2.00% Castro::construct_old_gravity() 5 0.008061 0.008061 0.008061 1.98% Gravity::get_old_grav_vector() 5 0.008055 0.008055 0.008055 1.98% Castro::normalize_species() 30 0.008011 0.008011 0.008011 1.97% MLMG::mgVcycle_down::1 36 0.007811 0.007811 0.007811 1.92% MLMG::mgVcycle_down::2 36 0.007167 0.007167 0.007167 1.76% FabArray::setVal() 501 0.006963 0.006963 0.006963 1.71% MLMG::mgVcycle_down::3 36 0.006959 0.006959 0.006959 1.71% MLMG::mgVcycle_down::4 36 0.006923 0.006923 0.006923 1.70% FabArray::ParallelCopy() 380 0.006908 0.006908 0.006908 1.70% FabArray::ParallelCopy_nowait() 380 0.006798 0.006798 0.006798 1.67% MLCellLinOp::defineAuxData() 6 0.00678 0.00678 0.00678 1.67% Castro::expand_state() 5 0.00635 0.00635 0.00635 1.56% Castro::post_restart() 1 0.006274 0.006274 0.006274 1.54% FabArray::Saxpy() 597 0.006161 0.006161 0.006161 1.51% Gravity::multilevel_solve_for_new_phi() 1 0.005883 0.005883 0.005883 1.45% MLCGSolver::ParallelAllReduce 798 0.005882 0.005882 0.005882 1.45% Gravity::actual_multilevel_solve() 1 0.00586 0.00586 0.00586 1.44% Gravity::fill_multipole_BCs() 6 0.005826 0.005826 0.005826 1.43% Castro::do_new_sources() 5 0.00581 0.00581 0.00581 1.43% amrex::Copy() 221 0.005698 0.005698 0.005698 1.40% MLMG::addInterpCorrection() 180 0.005688 0.005688 0.005688 1.40% MLMG::mgVcycle_up::4 36 0.005592 0.005592 0.005592 1.37% MLMG::mgVcycle_up::1 36 0.005559 0.005559 0.005559 1.37% Castro::initialize_advance() 5 0.005536 0.005536 0.005536 1.36% MLMG::mgVcycle_up::2 36 0.005444 0.005444 0.005444 1.34% amrex::average_down 180 0.005366 0.005366 0.005366 1.32% MLMG::mgVcycle_up::3 36 0.005359 0.005359 0.005359 1.32% MLPoisson::Fapply() 464 0.004805 0.004805 0.004805 1.18% Castro::estTimeStep() 10 0.004723 0.004723 0.004723 1.16% MLCellLinOp::solutionResidual() 42 0.003868 0.003868 0.003868 0.95% FabArray::Xpay() 325 0.003675 0.003675 0.003675 0.90% Castro::post_timestep() 5 0.003361 0.003361 0.003361 0.83% MLCellLinOp::defineBC() 6 0.0031 0.0031 0.0031 0.76% MLMG::prepareForSolve() 6 0.003078 0.003078 0.003078 0.76% MLMG::computeResidual() 36 0.003022 0.003022 0.003022 0.74% BndryData::define() 6 0.002954 0.002954 0.002954 0.73% Castro::computeNewDt() 5 0.002896 0.002896 0.002896 0.71% Castro::finalize_do_advance() 5 0.002694 0.002694 0.002694 0.66% Castro::enforce_min_density() 30 0.002648 0.002648 0.002648 0.65% Castro::construct_new_source() 25 0.001926 0.001926 0.001926 0.47% Castro::construct_new_gravity_source() 5 0.001916 0.001916 0.001916 0.47% amrex::Add() 36 0.001645 0.001645 0.001645 0.40% Castro::construct_old_source() 25 0.001566 0.001566 0.001566 0.38% Castro::construct_old_gravity_source() 5 0.001556 0.001556 0.001556 0.38% MLMG::ResNormInf() 42 0.001059 0.001059 0.001059 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001033 0.001033 0.001033 0.25% Castro::reset_internal_energy(Fab) 240 0.001016 0.001016 0.001016 0.25% MLMG::getGradSolution() 6 0.0009741 0.0009741 0.0009741 0.24% Castro::apply_source_to_state() 10 0.0009712 0.0009712 0.0009712 0.24% MLCellLinOp::compGrad() 6 0.0009708 0.0009708 0.0009708 0.24% MLCellLinOp::setLevelBC() 6 0.0009283 0.0009283 0.0009283 0.23% MLMG::computeMLResidual() 6 0.0008657 0.0008657 0.0008657 0.21% Castro::enforce_speed_limit() 30 0.0008476 0.0008476 0.0008476 0.21% FabArrayBase::getCPC() 632 0.0008142 0.0008142 0.0008142 0.20% MLPoisson::prepareForSolve() 6 0.0007826 0.0007826 0.0007826 0.19% MLCellLinOp::prepareForSolve() 6 0.0007721 0.0007721 0.0007721 0.19% Gravity::update_max_rhs() 6 0.0007585 0.0007585 0.0007585 0.19% FabArray::setDomainBndry() 20 0.0007312 0.0007312 0.0007312 0.18% FabArray::mult() 22 0.0007134 0.0007134 0.0007134 0.18% Castro::check_for_nan() 10 0.000685 0.000685 0.000685 0.17% Other 2165 0.004065 0.004065 0.004065 1.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 13 MiB 9037 MiB Castro::initMFs() 48 48 56 MiB 68 MiB Castro::swap_state_time_levels() 32 32 45 MiB 55 MiB StateData::restartDoit() 32 32 52 MiB 55 MiB FillPatchIterator::Initialize 160 160 1174 KiB 39 MiB Castro::initialize_do_advance() 40 40 27 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1993 KiB 28 MiB Castro::initialize_advance() 40 40 16 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6302 KiB 14 MiB MLMG::prepareForSolve() 361 361 3716 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 205 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 204 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6288 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 23 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3718 B 2048 KiB Gravity::solve_for_phi() 40 40 684 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 29 KiB 2048 KiB BndryData::define() 576 576 342 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 220 KiB 671 KiB Castro::estTimeStep() 10 10 5530 B 480 KiB VisMF::Write(FabArray) 112 112 1486 B 320 KiB Castro::normalize_species() 30 30 6403 B 320 KiB amrex::average_down 469 469 1492 B 257 KiB MLMG::addInterpCorrection() 468 468 1194 B 257 KiB amrex::Dot() 592 592 3546 B 160 KiB FabArray::norminf() 501 501 3485 B 160 KiB check_for_negative_density() 5 5 164 B 160 KiB MultiFab::max() 6 6 85 B 160 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MultiFab::contains_nan() 10 10 33 B 20 KiB MLPoisson::Fsmooth() 60 60 3600 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 49 B 10 KiB FillBoundary_nowait() 336 336 322 B 9648 B MLCellLinOp::applyBC() 3820 3820 231 B 9344 B amrex::Copy() 56 56 5807 B 8816 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B StateData::FillBoundary(geom) 960 960 51 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLCellLinOp::defineBC() 36 36 385 B 1248 B MLCGSolver::bicgstab 180 180 97 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1439 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 88 KiB 8192 KiB VisMF::Write(FabArray) 120 120 190 KiB 3584 KiB VisMF::Read() 24 24 233 KiB 3000 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MLPoisson::Fsmooth() 60 60 3600 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 49 B 10 KiB FillBoundary_nowait() 336 336 322 B 9648 B MLCellLinOp::applyBC() 1910 1910 230 B 9328 B amrex::Copy() 56 56 5807 B 8816 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B Gravity::get_old_grav_vector() 3 3 2455 B 3072 B StateData::FillBoundary(geom) 960 960 51 B 3024 B Gravity::fill_multipole_BCs() 18 18 6 B 2832 B MLMG::prepareForSolve() 7 7 820 B 1648 B amrex::average_down 37 37 449 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 26 B 400 B FabArray::norminf() 501 501 10 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2109 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.07-10-g85ae84ea8002) finalized