Initializing AMReX (24.06-21-g259db7cfb99e)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.06-21-g259db7cfb99e) initialized Starting run at 16:31:47 UTC on 2024-06-21. Successfully read inputs file ... Castro git describe: 24.06-14-g5f48f9561 AMReX git describe: 24.06-21-g259db7cfb Microphysics git describe: 24.06-10-g30469dd4 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.049473385 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.02780029 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.073538119 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.048472766 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.068723717 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.064169358 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.065794492 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.04866498 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.048825344 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051677567 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.061967517 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.067497671 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.07450236 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.048294423 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.027295933 seconds Ending run at 16:31:48 UTC on 2024-06-21. Run time = 0.878549333 Run time without initialization = 0.750067219 Average number of zones advanced per microsecond: 3.495 Average number of zones advanced per microsecond per rank: 3.495 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.8786 ... 0.8786 ... 0.8786 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2462 0.2462 0.2462 28.02% VisMF::Write(FabArray) 11 0.1923 0.1923 0.1923 21.88% MLCellLinOp::applyBC() 4351 0.08863 0.08863 0.08863 10.09% MLPoisson::Fsmooth() 3280 0.03593 0.03593 0.03593 4.09% FillBoundary_nowait() 3941 0.03012 0.03012 0.03012 3.43% StateData::FillBoundary(geom) 328 0.0267 0.0267 0.0267 3.04% amrex::Dot() 1114 0.02257 0.02257 0.02257 2.57% FabArray::norminf() 1061 0.02094 0.02094 0.02094 2.38% Castro::normalize_species() 62 0.01591 0.01591 0.01591 1.81% FabArray::ParallelCopy_nowait() 861 0.01419 0.01419 0.01419 1.61% FabArray::setVal() 1062 0.0138 0.0138 0.0138 1.57% FabArray::Saxpy() 1370 0.0137 0.0137 0.0137 1.56% StateDataPhysBCFunct::() 41 0.01194 0.01194 0.01194 1.36% amrex::Copy() 472 0.01125 0.01125 0.01125 1.28% MLCellLinOp::defineAuxData() 11 0.01084 0.01084 0.01084 1.23% MLPoisson::Fapply() 1060 0.01082 0.01082 0.01082 1.23% Castro::reset_internal_energy(MultiFab) 63 0.01074 0.01074 0.01074 1.22% Castro::enforce_min_density() 62 0.009968 0.009968 0.009968 1.13% FabArray::Xpay() 739 0.008116 0.008116 0.008116 0.92% Amr::checkPoint() 3 0.007033 0.007033 0.007033 0.80% MLMG::addInterpCorrection() 410 0.007025 0.007025 0.007025 0.80% Gravity::fill_multipole_BCs() 11 0.006811 0.006811 0.006811 0.78% amrex::average_down 410 0.006399 0.006399 0.006399 0.73% Castro::estTimeStep() 21 0.004244 0.004244 0.004244 0.48% BndryData::define() 11 0.004161 0.004161 0.004161 0.47% Castro::computeTemp() 63 0.003292 0.003292 0.003292 0.37% amrex::Add() 82 0.003231 0.003231 0.003231 0.37% Castro::construct_new_gravity_source() 10 0.002946 0.002946 0.002946 0.34% Castro::construct_old_gravity_source() 10 0.002242 0.002242 0.002242 0.26% Castro::subcycle_advance_ctu() 10 0.001983 0.001983 0.001983 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001862 0.001862 0.001862 0.21% Gravity::actual_solve_with_mlmg() 11 0.001855 0.001855 0.001855 0.21% Amr::writePlotFile() 2 0.001823 0.001823 0.001823 0.21% Castro::reset_internal_energy(Fab) 504 0.001747 0.001747 0.001747 0.20% MLCGSolver::bicgstab 82 0.001646 0.001646 0.001646 0.19% MLCellLinOp::setLevelBC() 11 0.001637 0.001637 0.001637 0.19% FabArray::setDomainBndry() 41 0.001461 0.001461 0.001461 0.17% FabArray::mult() 43 0.001393 0.001393 0.001393 0.16% MLCellLinOp::prepareForSolve() 11 0.001383 0.001383 0.001383 0.16% MultiFab::contains_nan() 20 0.001275 0.001275 0.001275 0.15% Castro::initData() 1 0.001223 0.001223 0.001223 0.14% MLCellLinOp::smooth() 1640 0.001157 0.001157 0.001157 0.13% MLCellLinOp::compGrad() 11 0.0011 0.0011 0.0011 0.13% check_for_negative_density() 10 0.001022 0.001022 0.001022 0.12% MLMG::prepareForSolve() 11 0.0009651 0.0009651 0.0009651 0.11% FabArrayBase::getCPC() 1323 0.0008484 0.0008484 0.0008484 0.10% FabArray::FillBoundary() 3941 0.0007963 0.0007963 0.0007963 0.09% Castro::do_advance_ctu() 10 0.0007645 0.0007645 0.0007645 0.09% Gravity::get_new_grav_vector() 11 0.0006917 0.0006917 0.0006917 0.08% Gravity::get_old_grav_vector() 10 0.000522 0.000522 0.000522 0.06% MLCellLinOp::apply() 1060 0.0004304 0.0004304 0.0004304 0.05% AmrLevel::FillPatch() 41 0.0004168 0.0004168 0.0004168 0.05% MLCGSolver::ParallelAllReduce 1832 0.0003261 0.0003261 0.0003261 0.04% Amr::coarseTimeStep() 10 0.0003144 0.0003144 0.0003144 0.04% Castro::initialize_advance() 10 0.000314 0.000314 0.000314 0.04% main() 1 0.0003074 0.0003074 0.0003074 0.03% MLCellLinOp::defineBC() 11 0.0002766 0.0002766 0.0002766 0.03% FabArray::ParallelCopy() 861 0.0002404 0.0002404 0.0002404 0.03% FillPatchIterator::Initialize 41 0.0002295 0.0002295 0.0002295 0.03% MLMG::mgVcycle() 82 0.0002085 0.0002085 0.0002085 0.02% MLCellLinOp::correctionResidual() 410 0.0001863 0.0001863 0.0001863 0.02% Amr::timeStep() 10 0.000163 0.000163 0.000163 0.02% Castro::construct_new_source() 50 0.0001616 0.0001616 0.0001616 0.02% Gravity::update_max_rhs() 11 0.0001277 0.0001277 0.0001277 0.01% Gravity::solve_for_phi() 10 0.0001209 0.0001209 0.0001209 0.01% MLMG:computeResOfCorrection() 410 0.000119 0.000119 0.000119 0.01% StateData::checkPoint() 12 0.0001109 0.0001109 0.0001109 0.01% MLMG::actualBottomSolve() 82 8.426e-05 8.426e-05 8.426e-05 0.01% MLMG::mgVcycle_down::0 82 8.403e-05 8.403e-05 8.403e-05 0.01% MLMG::mgVcycle_down::2 82 8.331e-05 8.331e-05 8.331e-05 0.01% MLMG::mgVcycle_down::1 82 8.049e-05 8.049e-05 8.049e-05 0.01% MLMG::mgVcycle_down::4 82 7.903e-05 7.903e-05 7.903e-05 0.01% MLMG::solve() 11 7.74e-05 7.74e-05 7.74e-05 0.01% MLMG::mgVcycle_down::3 82 7.696e-05 7.696e-05 7.696e-05 0.01% Castro::clean_state() 62 6.687e-05 6.687e-05 6.687e-05 0.01% AmrLevel::checkPoint() 3 6.155e-05 6.155e-05 6.155e-05 0.01% Castro::initialize_do_advance() 10 6.067e-05 6.067e-05 6.067e-05 0.01% Castro::finalize_do_advance() 10 5.637e-05 5.637e-05 5.637e-05 0.01% MLMG::mgVcycle_up::0 82 5.604e-05 5.604e-05 5.604e-05 0.01% MLMG::mgVcycle_up::4 82 5.598e-05 5.598e-05 5.598e-05 0.01% MLMG::oneIter() 82 5.53e-05 5.53e-05 5.53e-05 0.01% FillPatchIterator::FillFromLevel0() 41 5.477e-05 5.477e-05 5.477e-05 0.01% MLMG::mgVcycle_up::3 82 5.18e-05 5.18e-05 5.18e-05 0.01% MLMG::mgVcycle_up::1 82 5.129e-05 5.129e-05 5.129e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 4.986e-05 4.986e-05 4.986e-05 0.01% MLMG::mgVcycle_up::2 82 4.844e-05 4.844e-05 4.844e-05 0.01% MLCellLinOp::solutionResidual() 93 4.714e-05 4.714e-05 4.714e-05 0.01% Amr::defBaseLevel() 1 4.427e-05 4.427e-05 4.427e-05 0.01% MLPoisson::define() 11 4.086e-05 4.086e-05 4.086e-05 0.00% FillPatchSingleLevel 41 3.618e-05 3.618e-05 3.618e-05 0.00% MLMG::computeResidual() 82 3.259e-05 3.259e-05 3.259e-05 0.00% MLMG::ResNormInf() 93 3.227e-05 3.227e-05 3.227e-05 0.00% Castro::construct_new_gravity() 10 3.111e-05 3.111e-05 3.111e-05 0.00% MLMG::mgVcycle_bottom 82 3.098e-05 3.098e-05 3.098e-05 0.00% Castro::post_timestep() 10 2.724e-05 2.724e-05 2.724e-05 0.00% Castro::advance() 10 2.511e-05 2.511e-05 2.511e-05 0.00% Castro::do_new_sources() 10 2.363e-05 2.363e-05 2.363e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.269e-05 2.269e-05 2.269e-05 0.00% Castro::do_old_sources() 10 2.072e-05 2.072e-05 2.072e-05 0.00% Amr::FinalizeInit() 1 1.926e-05 1.926e-05 1.926e-05 0.00% Castro::construct_old_source() 50 1.788e-05 1.788e-05 1.788e-05 0.00% Castro::apply_source_to_state() 20 1.162e-05 1.162e-05 1.162e-05 0.00% Castro::check_for_nan() 20 1.153e-05 1.153e-05 1.153e-05 0.00% Castro::construct_old_gravity() 10 1.098e-05 1.098e-05 1.098e-05 0.00% Castro::post_init() 1 1.074e-05 1.074e-05 1.074e-05 0.00% MLMG::computeMLResidual() 11 9.24e-06 9.24e-06 9.24e-06 0.00% Gravity::actual_multilevel_solve() 1 8.166e-06 8.166e-06 8.166e-06 0.00% MLPoisson::prepareForSolve() 11 7.98e-06 7.98e-06 7.98e-06 0.00% Amr::InitializeInit() 1 7.658e-06 7.658e-06 7.658e-06 0.00% Castro::expand_state() 10 6.471e-06 6.471e-06 6.471e-06 0.00% Castro::computeNewDt() 9 6.464e-06 6.464e-06 6.464e-06 0.00% MLMG::getGradSolution() 11 5.404e-06 5.404e-06 5.404e-06 0.00% Amr::init() 1 3.48e-06 3.48e-06 3.48e-06 0.00% Amr::initialInit() 1 1.035e-06 1.035e-06 1.035e-06 0.00% Other 4804 0.003623 0.003623 0.003623 0.41% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8786 0.8786 0.8786 100.00% Amr::coarseTimeStep() 10 0.7225 0.7225 0.7225 82.24% Amr::timeStep() 10 0.6232 0.6232 0.6232 70.93% Castro::advance() 10 0.6157 0.6157 0.6157 70.08% Castro::subcycle_advance_ctu() 10 0.6031 0.6031 0.6031 68.65% Castro::do_advance_ctu() 10 0.6011 0.6011 0.6011 68.42% Gravity::solve_phi_with_mlmg() 11 0.3072 0.3072 0.3072 34.97% Gravity::actual_solve_with_mlmg() 11 0.2999 0.2999 0.2999 34.13% Castro::construct_new_gravity() 10 0.2803 0.2803 0.2803 31.91% MLMG::solve() 11 0.2766 0.2766 0.2766 31.48% Gravity::solve_for_phi() 10 0.2637 0.2637 0.2637 30.01% MLMG::oneIter() 82 0.2606 0.2606 0.2606 29.66% MLMG::mgVcycle() 82 0.2573 0.2573 0.2573 29.29% Castro::construct_ctu_hydro_source() 10 0.2528 0.2528 0.2528 28.77% VisMF::Write(FabArray) 11 0.1923 0.1923 0.1923 21.88% Amr::checkPoint() 3 0.1466 0.1466 0.1466 16.69% AmrLevel::checkPoint() 3 0.1396 0.1396 0.1396 15.89% StateData::checkPoint() 12 0.1395 0.1395 0.1395 15.88% MLCellLinOp::smooth() 1640 0.1302 0.1302 0.1302 14.82% Amr::init() 1 0.1278 0.1278 0.1278 14.55% MLCellLinOp::applyBC() 4351 0.1203 0.1203 0.1203 13.69% MLMG::mgVcycle_bottom 82 0.07658 0.07658 0.07658 8.72% MLMG::actualBottomSolve() 82 0.07655 0.07655 0.07655 8.71% MLCGSolver::bicgstab 82 0.07568 0.07568 0.07568 8.61% Amr::writePlotFile() 2 0.05524 0.05524 0.05524 6.29% Amr::initialInit() 1 0.0504 0.0504 0.0504 5.74% AmrLevel::FillPatch() 41 0.04894 0.04894 0.04894 5.57% Amr::FinalizeInit() 1 0.04752 0.04752 0.04752 5.41% Castro::post_init() 1 0.04637 0.04637 0.04637 5.28% FillPatchIterator::Initialize 41 0.04462 0.04462 0.04462 5.08% Gravity::multilevel_solve_for_new_phi() 1 0.04401 0.04401 0.04401 5.01% Gravity::actual_multilevel_solve() 1 0.04396 0.04396 0.04396 5.00% FillPatchIterator::FillFromLevel0() 41 0.04293 0.04293 0.04293 4.89% FillPatchSingleLevel 41 0.04287 0.04287 0.04287 4.88% Castro::clean_state() 62 0.04194 0.04194 0.04194 4.77% StateDataPhysBCFunct::() 41 0.03864 0.03864 0.03864 4.40% MLCellLinOp::apply() 1060 0.03786 0.03786 0.03786 4.31% MLPoisson::Fsmooth() 3280 0.03593 0.03593 0.03593 4.09% MLMG::mgVcycle_down::0 82 0.03503 0.03503 0.03503 3.99% FabArray::FillBoundary() 3941 0.03167 0.03167 0.03167 3.61% FillBoundary_nowait() 3941 0.03088 0.03088 0.03088 3.51% StateData::FillBoundary(geom) 328 0.0267 0.0267 0.0267 3.04% MLMG::mgVcycle_up::0 82 0.02639 0.02639 0.02639 3.00% amrex::Dot() 1114 0.02257 0.02257 0.02257 2.57% MLMG:computeResOfCorrection() 410 0.02109 0.02109 0.02109 2.40% MLCellLinOp::correctionResidual() 410 0.02097 0.02097 0.02097 2.39% FabArray::norminf() 1061 0.02094 0.02094 0.02094 2.38% Castro::initialize_do_advance() 10 0.019 0.019 0.019 2.16% Gravity::get_new_grav_vector() 11 0.01853 0.01853 0.01853 2.11% MLPoisson::define() 11 0.01818 0.01818 0.01818 2.07% MLMG::mgVcycle_down::1 82 0.01736 0.01736 0.01736 1.98% MLMG::mgVcycle_up::1 82 0.01695 0.01695 0.01695 1.93% MLMG::mgVcycle_down::2 82 0.01628 0.01628 0.01628 1.85% Castro::construct_old_gravity() 10 0.01611 0.01611 0.01611 1.83% Gravity::get_old_grav_vector() 10 0.0161 0.0161 0.0161 1.83% Castro::do_old_sources() 10 0.01597 0.01597 0.01597 1.82% Castro::normalize_species() 62 0.01591 0.01591 0.01591 1.81% MLMG::mgVcycle_down::3 82 0.01583 0.01583 0.01583 1.80% Castro::computeTemp() 63 0.01578 0.01578 0.01578 1.80% MLMG::mgVcycle_down::4 82 0.01576 0.01576 0.01576 1.79% FabArray::ParallelCopy() 861 0.01527 0.01527 0.01527 1.74% FabArray::ParallelCopy_nowait() 861 0.01503 0.01503 0.01503 1.71% FabArray::setVal() 1062 0.0138 0.0138 0.0138 1.57% FabArray::Saxpy() 1370 0.0137 0.0137 0.0137 1.56% MLCGSolver::ParallelAllReduce 1832 0.01348 0.01348 0.01348 1.53% Castro::do_new_sources() 10 0.01312 0.01312 0.01312 1.49% MLMG::addInterpCorrection() 410 0.01256 0.01256 0.01256 1.43% MLMG::mgVcycle_up::4 82 0.01252 0.01252 0.01252 1.42% Castro::reset_internal_energy(MultiFab) 63 0.01249 0.01249 0.01249 1.42% MLCellLinOp::defineAuxData() 11 0.01237 0.01237 0.01237 1.41% MLMG::mgVcycle_up::2 82 0.01233 0.01233 0.01233 1.40% MLMG::mgVcycle_up::3 82 0.01208 0.01208 0.01208 1.37% amrex::average_down 410 0.01194 0.01194 0.01194 1.36% Castro::expand_state() 10 0.01189 0.01189 0.01189 1.35% Castro::initialize_advance() 10 0.01186 0.01186 0.01186 1.35% amrex::Copy() 472 0.01125 0.01125 0.01125 1.28% MLPoisson::Fapply() 1060 0.01082 0.01082 0.01082 1.23% Castro::enforce_min_density() 62 0.009968 0.009968 0.009968 1.13% FabArray::Xpay() 739 0.008116 0.008116 0.008116 0.92% MLCellLinOp::solutionResidual() 93 0.008092 0.008092 0.008092 0.92% Castro::post_timestep() 10 0.007367 0.007367 0.007367 0.84% Gravity::fill_multipole_BCs() 11 0.007064 0.007064 0.007064 0.80% MLMG::computeResidual() 82 0.006795 0.006795 0.006795 0.77% MLCellLinOp::defineBC() 11 0.005506 0.005506 0.005506 0.63% MLMG::prepareForSolve() 11 0.005352 0.005352 0.005352 0.61% BndryData::define() 11 0.005229 0.005229 0.005229 0.60% Castro::estTimeStep() 21 0.004244 0.004244 0.004244 0.48% amrex::Add() 82 0.003231 0.003231 0.003231 0.37% Castro::construct_new_source() 50 0.003108 0.003108 0.003108 0.35% Castro::construct_new_gravity_source() 10 0.002946 0.002946 0.002946 0.34% Amr::InitializeInit() 1 0.002874 0.002874 0.002874 0.33% Amr::defBaseLevel() 1 0.002867 0.002867 0.002867 0.33% Castro::construct_old_source() 50 0.002259 0.002259 0.002259 0.26% Castro::initData() 1 0.002252 0.002252 0.002252 0.26% Castro::construct_old_gravity_source() 10 0.002242 0.002242 0.002242 0.26% MLMG::ResNormInf() 93 0.002206 0.002206 0.002206 0.25% Castro::finalize_do_advance() 10 0.002109 0.002109 0.002109 0.24% Castro::apply_source_to_state() 20 0.001891 0.001891 0.001891 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001862 0.001862 0.001862 0.21% Castro::reset_internal_energy(Fab) 504 0.001747 0.001747 0.001747 0.20% MLCellLinOp::setLevelBC() 11 0.001637 0.001637 0.001637 0.19% MLMG::getGradSolution() 11 0.001636 0.001636 0.001636 0.19% MLCellLinOp::compGrad() 11 0.00163 0.00163 0.00163 0.19% FabArrayBase::getCPC() 1323 0.001584 0.001584 0.001584 0.18% Castro::computeNewDt() 9 0.001476 0.001476 0.001476 0.17% FabArray::setDomainBndry() 41 0.001461 0.001461 0.001461 0.17% FabArray::mult() 43 0.001393 0.001393 0.001393 0.16% MLPoisson::prepareForSolve() 11 0.001391 0.001391 0.001391 0.16% MLCellLinOp::prepareForSolve() 11 0.001383 0.001383 0.001383 0.16% MLMG::computeMLResidual() 11 0.001339 0.001339 0.001339 0.15% Castro::check_for_nan() 20 0.001286 0.001286 0.001286 0.15% MultiFab::contains_nan() 20 0.001275 0.001275 0.001275 0.15% check_for_negative_density() 10 0.001022 0.001022 0.001022 0.12% Gravity::update_max_rhs() 11 0.0009726 0.0009726 0.0009726 0.11% Other 4804 0.007915 0.007915 0.007915 0.90% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 6413 KiB 9037 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1070 KiB 39 MiB Castro::initialize_do_advance() 80 80 26 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1768 KiB 28 MiB Castro::initialize_advance() 80 80 16 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7599 KiB 14 MiB MLMG::prepareForSolve() 660 660 3872 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 219 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 187 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7536 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 12 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2140 B 2048 KiB Gravity::solve_for_phi() 80 80 614 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 102 KiB 2048 KiB BndryData::define() 1056 1056 354 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 225 KiB 671 KiB Castro::estTimeStep() 21 21 2315 B 480 KiB VisMF::Write(FabArray) 656 656 3790 B 320 KiB Castro::normalize_species() 62 62 5277 B 320 KiB amrex::average_down 1067 1067 1677 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1216 B 257 KiB amrex::Dot() 1360 1360 3793 B 160 KiB FabArray::norminf() 1143 1143 3682 B 160 KiB check_for_negative_density() 10 10 184 B 160 KiB Castro::initData() 1 1 34 B 160 KiB MultiFab::max() 11 11 54 B 160 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MultiFab::contains_nan() 20 20 28 B 20 KiB MLPoisson::Fsmooth() 132 132 3781 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB FillBoundary_nowait() 760 760 296 B 9648 B MLCellLinOp::applyBC() 8702 8702 246 B 9344 B MLCellLinOp::prepareForSolve() 66 66 4 B 7792 B amrex::Copy() 100 100 3923 B 6144 B StateData::FillBoundary(geom) 1992 1992 42 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 399 B 1248 B MLCGSolver::bicgstab 410 410 103 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 690 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 44 KiB 8192 KiB VisMF::Write(FabArray) 744 744 488 KiB 3584 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MLPoisson::Fsmooth() 132 132 3781 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 46 B 10 KiB FillBoundary_nowait() 760 760 295 B 9648 B MLCellLinOp::applyBC() 4351 4351 244 B 9328 B MLCellLinOp::prepareForSolve() 66 66 4 B 7792 B amrex::Copy() 100 100 3923 B 6144 B Gravity::get_new_grav_vector() 3 3 2898 B 3072 B StateData::FillBoundary(geom) 1992 1992 43 B 3024 B Gravity::fill_multipole_BCs() 33 33 3 B 2832 B amrex::average_down 83 83 619 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 322 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 27 B 400 B FabArray::norminf() 1143 1143 10 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2109 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.06-21-g259db7cfb99e) finalized Initializing AMReX (24.06-21-g259db7cfb99e)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.06-21-g259db7cfb99e) initialized Starting run at 16:31:48 UTC on 2024-06-21. Successfully read inputs file ... Castro git describe: 24.06-14-g5f48f9561 AMReX git describe: 24.06-21-g259db7cfb Microphysics git describe: 24.06-10-g30469dd4 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.497850584 Restart time = 0.071583298 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.066185103 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051161598 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.066779961 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.066425994 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.06639246 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.089127827 seconds Ending run at 16:31:49 UTC on 2024-06-21. Run time = 0.478652455 Run time without initialization = 0.40650533 Average number of zones advanced per microsecond: 3.224 Average number of zones advanced per microsecond per rank: 3.224 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.4787 ... 0.4787 ... 0.4787 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1232 0.1232 0.1232 25.74% VisMF::Read() 3 0.06129 0.06129 0.06129 12.80% Amr::writePlotFile() 1 0.06123 0.06123 0.06123 12.79% MLCellLinOp::applyBC() 1910 0.03748 0.03748 0.03748 7.83% VisMF::Write(FabArray) 1 0.02773 0.02773 0.02773 5.79% MLPoisson::Fsmooth() 1440 0.01575 0.01575 0.01575 3.29% FillBoundary_nowait() 1730 0.01341 0.01341 0.01341 2.80% StateData::FillBoundary(geom) 160 0.01323 0.01323 0.01323 2.76% amrex::Dot() 484 0.009826 0.009826 0.009826 2.05% FabArray::norminf() 465 0.009217 0.009217 0.009217 1.93% Castro::normalize_species() 30 0.008402 0.008402 0.008402 1.76% FabArray::setVal() 501 0.006757 0.006757 0.006757 1.41% FabArray::ParallelCopy_nowait() 380 0.006457 0.006457 0.006457 1.35% FabArray::Saxpy() 597 0.006265 0.006265 0.006265 1.31% Castro::enforce_min_density() 30 0.006053 0.006053 0.006053 1.26% MLCellLinOp::defineAuxData() 6 0.005969 0.005969 0.005969 1.25% StateDataPhysBCFunct::() 20 0.005802 0.005802 0.005802 1.21% Castro::reset_internal_energy(MultiFab) 30 0.005624 0.005624 0.005624 1.17% amrex::Copy() 221 0.005548 0.005548 0.005548 1.16% Amr::restart() 1 0.004946 0.004946 0.004946 1.03% MLPoisson::Fapply() 464 0.004796 0.004796 0.004796 1.00% Gravity::fill_multipole_BCs() 6 0.003885 0.003885 0.003885 0.81% FabArray::Xpay() 325 0.003618 0.003618 0.003618 0.76% MLMG::addInterpCorrection() 180 0.003127 0.003127 0.003127 0.65% amrex::average_down 180 0.002826 0.002826 0.002826 0.59% BndryData::define() 6 0.00229 0.00229 0.00229 0.48% Castro::estTimeStep() 10 0.001842 0.001842 0.001842 0.38% Castro::computeTemp() 30 0.001523 0.001523 0.001523 0.32% Castro::construct_new_gravity_source() 5 0.001444 0.001444 0.001444 0.30% amrex::Add() 36 0.001424 0.001424 0.001424 0.30% Castro::construct_old_gravity_source() 5 0.001148 0.001148 0.001148 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001016 0.001016 0.001016 0.21% Gravity::actual_solve_with_mlmg() 6 0.001001 0.001001 0.001001 0.21% MLCellLinOp::setLevelBC() 6 0.000916 0.000916 0.000916 0.19% Castro::reset_internal_energy(Fab) 240 0.0008184 0.0008184 0.0008184 0.17% MLCellLinOp::prepareForSolve() 6 0.0007752 0.0007752 0.0007752 0.16% Castro::subcycle_advance_ctu() 5 0.000744 0.000744 0.000744 0.16% MLCGSolver::bicgstab 36 0.0007372 0.0007372 0.0007372 0.15% FabArray::mult() 22 0.0007228 0.0007228 0.0007228 0.15% FabArray::setDomainBndry() 20 0.0007141 0.0007141 0.0007141 0.15% MultiFab::contains_nan() 10 0.0006511 0.0006511 0.0006511 0.14% MLCellLinOp::compGrad() 6 0.0005944 0.0005944 0.0005944 0.12% MLMG::prepareForSolve() 6 0.0005665 0.0005665 0.0005665 0.12% MLCellLinOp::smooth() 720 0.0005053 0.0005053 0.0005053 0.11% Gravity::get_old_grav_vector() 5 0.0003678 0.0003678 0.0003678 0.08% FabArrayBase::getCPC() 632 0.0003667 0.0003667 0.0003667 0.08% FabArray::FillBoundary() 1730 0.0003388 0.0003388 0.0003388 0.07% Gravity::get_new_grav_vector() 5 0.000266 0.000266 0.000266 0.06% main() 1 0.0002625 0.0002625 0.0002625 0.05% AmrLevel::FillPatch() 20 0.000204 0.000204 0.000204 0.04% MLCellLinOp::apply() 464 0.0001977 0.0001977 0.0001977 0.04% Amr::coarseTimeStep() 5 0.0001505 0.0001505 0.0001505 0.03% MLCellLinOp::defineBC() 6 0.0001496 0.0001496 0.0001496 0.03% MLCGSolver::ParallelAllReduce 798 0.0001346 0.0001346 0.0001346 0.03% Castro::construct_new_source() 25 0.000125 0.000125 0.000125 0.03% FabArray::ParallelCopy() 380 0.0001169 0.0001169 0.0001169 0.02% FillPatchIterator::Initialize 20 0.0001056 0.0001056 0.0001056 0.02% Amr::timeStep() 5 9.648e-05 9.648e-05 9.648e-05 0.02% MLMG::mgVcycle() 36 9.157e-05 9.157e-05 9.157e-05 0.02% Castro::initialize_advance() 5 8.757e-05 8.757e-05 8.757e-05 0.02% MLCellLinOp::correctionResidual() 180 7.626e-05 7.626e-05 7.626e-05 0.02% AmrLevel::restart() 1 6.976e-05 6.976e-05 6.976e-05 0.01% StateData::restartDoit() 4 6.86e-05 6.86e-05 6.86e-05 0.01% Castro::do_advance_ctu() 5 6.53e-05 6.53e-05 6.53e-05 0.01% MLMG:computeResOfCorrection() 180 5.141e-05 5.141e-05 5.141e-05 0.01% Gravity::solve_for_phi() 5 4.983e-05 4.983e-05 4.983e-05 0.01% MLMG::actualBottomSolve() 36 3.886e-05 3.886e-05 3.886e-05 0.01% MLMG::solve() 6 3.763e-05 3.763e-05 3.763e-05 0.01% MLMG::mgVcycle_down::0 36 3.574e-05 3.574e-05 3.574e-05 0.01% MLMG::mgVcycle_down::1 36 3.491e-05 3.491e-05 3.491e-05 0.01% MLMG::mgVcycle_down::2 36 3.33e-05 3.33e-05 3.33e-05 0.01% Castro::clean_state() 30 3.308e-05 3.308e-05 3.308e-05 0.01% MLMG::mgVcycle_down::4 36 3.209e-05 3.209e-05 3.209e-05 0.01% MLMG::mgVcycle_down::3 36 3.068e-05 3.068e-05 3.068e-05 0.01% Castro::initialize_do_advance() 5 3.005e-05 3.005e-05 3.005e-05 0.01% Castro::post_restart() 1 2.851e-05 2.851e-05 2.851e-05 0.01% MLPoisson::define() 6 2.687e-05 2.687e-05 2.687e-05 0.01% MLMG::mgVcycle_up::4 36 2.592e-05 2.592e-05 2.592e-05 0.01% Castro::post_timestep() 5 2.473e-05 2.473e-05 2.473e-05 0.01% MLMG::oneIter() 36 2.379e-05 2.379e-05 2.379e-05 0.00% Castro::finalize_do_advance() 5 2.357e-05 2.357e-05 2.357e-05 0.00% MLMG::mgVcycle_up::3 36 2.283e-05 2.283e-05 2.283e-05 0.00% MLCellLinOp::solutionResidual() 42 2.138e-05 2.138e-05 2.138e-05 0.00% MLMG::mgVcycle_up::2 36 2.104e-05 2.104e-05 2.104e-05 0.00% MLMG::mgVcycle_up::0 36 2.094e-05 2.094e-05 2.094e-05 0.00% FillPatchIterator::FillFromLevel0() 20 2.044e-05 2.044e-05 2.044e-05 0.00% MLMG::mgVcycle_up::1 36 1.99e-05 1.99e-05 1.99e-05 0.00% FillPatchSingleLevel 20 1.64e-05 1.64e-05 1.64e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.608e-05 1.608e-05 1.608e-05 0.00% MLMG::ResNormInf() 42 1.584e-05 1.584e-05 1.584e-05 0.00% MLMG::mgVcycle_bottom 36 1.466e-05 1.466e-05 1.466e-05 0.00% Castro::advance() 5 1.445e-05 1.445e-05 1.445e-05 0.00% MLMG::computeResidual() 36 1.444e-05 1.444e-05 1.444e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.442e-05 1.442e-05 1.442e-05 0.00% Castro::construct_new_gravity() 5 1.176e-05 1.176e-05 1.176e-05 0.00% Castro::do_old_sources() 5 1.07e-05 1.07e-05 1.07e-05 0.00% Castro::do_new_sources() 5 1.043e-05 1.043e-05 1.043e-05 0.00% Castro::construct_old_source() 25 9.464e-06 9.464e-06 9.464e-06 0.00% Gravity::actual_multilevel_solve() 1 8.48e-06 8.48e-06 8.48e-06 0.00% Castro::apply_source_to_state() 10 6.033e-06 6.033e-06 6.033e-06 0.00% Castro::check_for_nan() 10 5.863e-06 5.863e-06 5.863e-06 0.00% Castro::construct_old_gravity() 5 5.328e-06 5.328e-06 5.328e-06 0.00% MLPoisson::prepareForSolve() 6 3.93e-06 3.93e-06 3.93e-06 0.00% MLMG::computeMLResidual() 6 3.892e-06 3.892e-06 3.892e-06 0.00% Castro::expand_state() 5 3.247e-06 3.247e-06 3.247e-06 0.00% Castro::computeNewDt() 5 3.239e-06 3.239e-06 3.239e-06 0.00% MLMG::getGradSolution() 6 3.226e-06 3.226e-06 3.226e-06 0.00% Amr::init() 1 1e-06 1e-06 1e-06 0.00% Other 2191 0.002536 0.002536 0.002536 0.53% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4787 0.4787 0.4787 100.00% Amr::coarseTimeStep() 5 0.3171 0.3171 0.3171 66.24% Amr::timeStep() 5 0.3158 0.3158 0.3158 65.98% Castro::advance() 5 0.3112 0.3112 0.3112 65.01% Castro::subcycle_advance_ctu() 5 0.3042 0.3042 0.3042 63.55% Castro::do_advance_ctu() 5 0.3035 0.3035 0.3035 63.40% Castro::construct_new_gravity() 5 0.1417 0.1417 0.1417 29.60% Gravity::solve_phi_with_mlmg() 6 0.1379 0.1379 0.1379 28.81% Gravity::actual_solve_with_mlmg() 6 0.1338 0.1338 0.1338 27.95% Gravity::solve_for_phi() 5 0.1334 0.1334 0.1334 27.88% Castro::construct_ctu_hydro_source() 5 0.1266 0.1266 0.1266 26.46% MLMG::solve() 6 0.121 0.121 0.121 25.27% MLMG::oneIter() 36 0.113 0.113 0.113 23.61% MLMG::mgVcycle() 36 0.1116 0.1116 0.1116 23.31% Amr::writePlotFile() 1 0.08924 0.08924 0.08924 18.64% Amr::init() 1 0.07164 0.07164 0.07164 14.97% Amr::restart() 1 0.07164 0.07164 0.07164 14.97% AmrLevel::restart() 1 0.06154 0.06154 0.06154 12.86% StateData::restartDoit() 4 0.06147 0.06147 0.06147 12.84% VisMF::Read() 3 0.06129 0.06129 0.06129 12.80% MLCellLinOp::smooth() 720 0.05558 0.05558 0.05558 11.61% MLCellLinOp::applyBC() 1910 0.05155 0.05155 0.05155 10.77% MLMG::mgVcycle_bottom 36 0.03365 0.03365 0.03365 7.03% MLMG::actualBottomSolve() 36 0.03363 0.03363 0.03363 7.03% MLCGSolver::bicgstab 36 0.03326 0.03326 0.03326 6.95% VisMF::Write(FabArray) 1 0.02773 0.02773 0.02773 5.79% AmrLevel::FillPatch() 20 0.02414 0.02414 0.02414 5.04% Castro::clean_state() 30 0.02263 0.02263 0.02263 4.73% FillPatchIterator::Initialize 20 0.022 0.022 0.022 4.60% FillPatchIterator::FillFromLevel0() 20 0.02118 0.02118 0.02118 4.43% FillPatchSingleLevel 20 0.02116 0.02116 0.02116 4.42% StateDataPhysBCFunct::() 20 0.01903 0.01903 0.01903 3.98% MLCellLinOp::apply() 464 0.01692 0.01692 0.01692 3.53% MLPoisson::Fsmooth() 1440 0.01575 0.01575 0.01575 3.29% MLMG::mgVcycle_down::0 36 0.01553 0.01553 0.01553 3.24% FabArray::FillBoundary() 1730 0.01407 0.01407 0.01407 2.94% FillBoundary_nowait() 1730 0.01373 0.01373 0.01373 2.87% StateData::FillBoundary(geom) 160 0.01323 0.01323 0.01323 2.76% MLMG::mgVcycle_up::0 36 0.01159 0.01159 0.01159 2.42% Castro::initialize_do_advance() 5 0.01004 0.01004 0.01004 2.10% MLPoisson::define() 6 0.01001 0.01001 0.01001 2.09% amrex::Dot() 484 0.009826 0.009826 0.009826 2.05% MLMG:computeResOfCorrection() 180 0.009376 0.009376 0.009376 1.96% MLCellLinOp::correctionResidual() 180 0.009324 0.009324 0.009324 1.95% FabArray::norminf() 465 0.009217 0.009217 0.009217 1.93% Castro::do_old_sources() 5 0.008479 0.008479 0.008479 1.77% Castro::normalize_species() 30 0.008402 0.008402 0.008402 1.76% Castro::construct_old_gravity() 5 0.008354 0.008354 0.008354 1.75% Gravity::get_old_grav_vector() 5 0.008349 0.008349 0.008349 1.74% Gravity::get_new_grav_vector() 5 0.008111 0.008111 0.008111 1.69% Castro::computeTemp() 30 0.007965 0.007965 0.007965 1.66% MLMG::mgVcycle_down::1 36 0.007707 0.007707 0.007707 1.61% MLMG::mgVcycle_down::2 36 0.007218 0.007218 0.007218 1.51% MLMG::mgVcycle_down::3 36 0.007014 0.007014 0.007014 1.47% FabArray::ParallelCopy() 380 0.006965 0.006965 0.006965 1.46% Castro::do_new_sources() 5 0.006956 0.006956 0.006956 1.45% MLMG::mgVcycle_down::4 36 0.006938 0.006938 0.006938 1.45% FabArray::ParallelCopy_nowait() 380 0.006848 0.006848 0.006848 1.43% MLCellLinOp::defineAuxData() 6 0.006772 0.006772 0.006772 1.41% FabArray::setVal() 501 0.006757 0.006757 0.006757 1.41% Castro::initialize_advance() 5 0.006604 0.006604 0.006604 1.38% Castro::reset_internal_energy(MultiFab) 30 0.006442 0.006442 0.006442 1.35% Castro::expand_state() 5 0.006402 0.006402 0.006402 1.34% FabArray::Saxpy() 597 0.006265 0.006265 0.006265 1.31% Castro::enforce_min_density() 30 0.006053 0.006053 0.006053 1.26% MLCGSolver::ParallelAllReduce 798 0.005936 0.005936 0.005936 1.24% amrex::Copy() 221 0.005548 0.005548 0.005548 1.16% MLMG::mgVcycle_up::4 36 0.005542 0.005542 0.005542 1.16% MLMG::addInterpCorrection() 180 0.005532 0.005532 0.005532 1.16% MLMG::mgVcycle_up::1 36 0.005524 0.005524 0.005524 1.15% MLMG::mgVcycle_up::2 36 0.005433 0.005433 0.005433 1.14% MLMG::mgVcycle_up::3 36 0.005321 0.005321 0.005321 1.11% amrex::average_down 180 0.005271 0.005271 0.005271 1.10% Castro::post_restart() 1 0.004954 0.004954 0.004954 1.03% MLPoisson::Fapply() 464 0.004796 0.004796 0.004796 1.00% Gravity::multilevel_solve_for_new_phi() 1 0.004719 0.004719 0.004719 0.99% Gravity::actual_multilevel_solve() 1 0.004703 0.004703 0.004703 0.98% Castro::post_timestep() 5 0.004534 0.004534 0.004534 0.95% Gravity::fill_multipole_BCs() 6 0.004018 0.004018 0.004018 0.84% MLCellLinOp::solutionResidual() 42 0.003784 0.003784 0.003784 0.79% FabArray::Xpay() 325 0.003618 0.003618 0.003618 0.76% MLCellLinOp::defineBC() 6 0.003058 0.003058 0.003058 0.64% MLMG::prepareForSolve() 6 0.002996 0.002996 0.002996 0.63% MLMG::computeResidual() 36 0.002986 0.002986 0.002986 0.62% BndryData::define() 6 0.002908 0.002908 0.002908 0.61% Castro::estTimeStep() 10 0.001842 0.001842 0.001842 0.38% Castro::construct_new_source() 25 0.001569 0.001569 0.001569 0.33% Castro::construct_new_gravity_source() 5 0.001444 0.001444 0.001444 0.30% amrex::Add() 36 0.001424 0.001424 0.001424 0.30% Castro::construct_old_source() 25 0.001157 0.001157 0.001157 0.24% Castro::construct_old_gravity_source() 5 0.001148 0.001148 0.001148 0.24% Castro::computeNewDt() 5 0.001098 0.001098 0.001098 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001016 0.001016 0.001016 0.21% MLMG::ResNormInf() 42 0.0009881 0.0009881 0.0009881 0.21% Castro::apply_source_to_state() 10 0.0009689 0.0009689 0.0009689 0.20% MLCellLinOp::setLevelBC() 6 0.000916 0.000916 0.000916 0.19% MLMG::getGradSolution() 6 0.0008985 0.0008985 0.0008985 0.19% MLCellLinOp::compGrad() 6 0.0008953 0.0008953 0.0008953 0.19% Castro::reset_internal_energy(Fab) 240 0.0008184 0.0008184 0.0008184 0.17% MLMG::computeMLResidual() 6 0.0008161 0.0008161 0.0008161 0.17% FabArrayBase::getCPC() 632 0.0007965 0.0007965 0.0007965 0.17% MLPoisson::prepareForSolve() 6 0.0007792 0.0007792 0.0007792 0.16% MLCellLinOp::prepareForSolve() 6 0.0007752 0.0007752 0.0007752 0.16% Castro::finalize_do_advance() 5 0.0007716 0.0007716 0.0007716 0.16% FabArray::mult() 22 0.0007228 0.0007228 0.0007228 0.15% FabArray::setDomainBndry() 20 0.0007141 0.0007141 0.0007141 0.15% Castro::check_for_nan() 10 0.0006569 0.0006569 0.0006569 0.14% MultiFab::contains_nan() 10 0.0006511 0.0006511 0.0006511 0.14% Other 2191 0.004362 0.004362 0.004362 0.91% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 10 MiB 9037 MiB Castro::initMFs() 48 48 58 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::restartDoit() 32 32 52 MiB 55 MiB FillPatchIterator::Initialize 160 160 1005 KiB 39 MiB Castro::initialize_do_advance() 40 40 24 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1708 KiB 28 MiB Castro::initialize_advance() 40 40 15 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6515 KiB 14 MiB MLMG::prepareForSolve() 361 361 3102 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 180 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 173 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6508 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 13 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2464 B 2048 KiB Gravity::solve_for_phi() 40 40 570 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 20 KiB 2048 KiB BndryData::define() 576 576 286 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 184 KiB 671 KiB Castro::estTimeStep() 10 10 1825 B 480 KiB VisMF::Write(FabArray) 112 112 1219 B 320 KiB Castro::normalize_species() 30 30 5092 B 320 KiB amrex::average_down 469 469 1344 B 257 KiB MLMG::addInterpCorrection() 468 468 963 B 257 KiB amrex::Dot() 592 592 3037 B 160 KiB FabArray::norminf() 501 501 2980 B 160 KiB check_for_negative_density() 5 5 174 B 160 KiB MultiFab::max() 6 6 61 B 160 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MultiFab::contains_nan() 10 10 27 B 20 KiB MLPoisson::Fsmooth() 60 60 3008 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 42 B 10 KiB FillBoundary_nowait() 336 336 241 B 9648 B MLCellLinOp::applyBC() 3820 3820 198 B 9344 B amrex::Copy() 56 56 5855 B 8816 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B StateData::FillBoundary(geom) 960 960 40 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCellLinOp::defineBC() 36 36 321 B 1248 B MLCGSolver::bicgstab 180 180 83 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1255 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 84 KiB 8192 KiB VisMF::Write(FabArray) 120 120 165 KiB 3584 KiB VisMF::Read() 24 24 194 KiB 3000 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MLPoisson::Fsmooth() 60 60 3008 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 42 B 10 KiB FillBoundary_nowait() 336 336 241 B 9648 B MLCellLinOp::applyBC() 1910 1910 197 B 9328 B amrex::Copy() 56 56 5855 B 8816 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B Gravity::get_old_grav_vector() 3 3 2572 B 3072 B StateData::FillBoundary(geom) 960 960 40 B 3024 B Gravity::fill_multipole_BCs() 18 18 4 B 2832 B MLMG::prepareForSolve() 7 7 789 B 1648 B amrex::average_down 37 37 481 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 22 B 400 B FabArray::norminf() 501 501 8 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2109 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.06-21-g259db7cfb99e) finalized