Initializing AMReX (24.02-21-g99b47cb58a3c)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.02-21-g99b47cb58a3c) initialized Starting run at 09:20:37 UTC on 2024-02-20. Successfully read inputs file ... Castro git describe: 24.02-21-g56d243e7a AMReX git describe: 24.02-21-g99b47cb58 Microphysics git describe: 24.02-19-g32b2c42a reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.045419033 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025643373 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.069984116 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.063803973 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.086060692 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.076137048 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.052056385 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.046097447 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.07340677 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.077098594 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.062942099 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.064353406 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.066972758 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.043447282 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.02503439 seconds Ending run at 09:20:38 UTC on 2024-02-20. Run time = 0.93226429 Run time without initialization = 0.808116977 Average number of zones advanced per microsecond: 3.244 Average number of zones advanced per microsecond per rank: 3.244 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.9323 ... 0.9323 ... 0.9323 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.3035 0.3035 0.3035 32.55% VisMF::Write(FabArray) 11 0.1779 0.1779 0.1779 19.08% MLCellLinOp::applyBC() 4298 0.08742 0.08742 0.08742 9.38% MLPoisson::Fsmooth() 3240 0.03369 0.03369 0.03369 3.61% FillBoundary_nowait() 3893 0.03104 0.03104 0.03104 3.33% StateData::FillBoundary(geom) 328 0.0269 0.0269 0.0269 2.89% amrex::Dot() 1100 0.02143 0.02143 0.02143 2.30% Castro::normalize_species() 62 0.02121 0.02121 0.02121 2.28% FabArray::norminf() 1048 0.02004 0.02004 0.02004 2.15% Castro::computeTemp() 63 0.01569 0.01569 0.01569 1.68% FabArray::ParallelCopy_nowait() 851 0.01379 0.01379 0.01379 1.48% FabArray::setVal() 1054 0.01351 0.01351 0.01351 1.45% FabArray::Saxpy() 1353 0.01316 0.01316 0.01316 1.41% StateDataPhysBCFunct::() 41 0.0119 0.0119 0.0119 1.28% Castro::enforce_min_density() 62 0.01171 0.01171 0.01171 1.26% amrex::Copy() 469 0.01091 0.01091 0.01091 1.17% MLCellLinOp::defineAuxData() 11 0.01046 0.01046 0.01046 1.12% MLPoisson::Fapply() 1047 0.01034 0.01034 0.01034 1.11% Gravity::fill_multipole_BCs() 11 0.009082 0.009082 0.009082 0.97% FabArray::Xpay() 730 0.007935 0.007935 0.007935 0.85% MLMG::addInterpCorrection() 405 0.007057 0.007057 0.007057 0.76% amrex::average_down 405 0.006209 0.006209 0.006209 0.67% Castro::estTimeStep() 21 0.005818 0.005818 0.005818 0.62% Amr::checkPoint() 3 0.005443 0.005443 0.005443 0.58% Castro::reset_internal_energy(MultiFab) 63 0.005322 0.005322 0.005322 0.57% BndryData::define() 11 0.004142 0.004142 0.004142 0.44% amrex::Add() 81 0.003592 0.003592 0.003592 0.39% Castro::construct_new_gravity_source() 10 0.003286 0.003286 0.003286 0.35% Castro::enforce_speed_limit() 62 0.002997 0.002997 0.002997 0.32% Castro::construct_old_gravity_source() 10 0.002806 0.002806 0.002806 0.30% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001825 0.001825 0.001825 0.20% check_for_negative_density() 10 0.001807 0.001807 0.001807 0.19% Amr::writePlotFile() 2 0.001786 0.001786 0.001786 0.19% Castro::reset_internal_energy(Fab) 504 0.001668 0.001668 0.001668 0.18% MLCGSolver::bicgstab 81 0.001642 0.001642 0.001642 0.18% Gravity::actual_solve_with_mlmg() 11 0.001614 0.001614 0.001614 0.17% MLCellLinOp::setLevelBC() 11 0.001611 0.001611 0.001611 0.17% Castro::initData() 1 0.001561 0.001561 0.001561 0.17% FabArray::mult() 43 0.001383 0.001383 0.001383 0.15% FabArray::setDomainBndry() 41 0.001369 0.001369 0.001369 0.15% MLCellLinOp::prepareForSolve() 11 0.001368 0.001368 0.001368 0.15% MultiFab::contains_nan() 20 0.001269 0.001269 0.001269 0.14% MLCellLinOp::smooth() 1620 0.001178 0.001178 0.001178 0.13% MLCellLinOp::compGrad() 11 0.001105 0.001105 0.001105 0.12% MLMG::prepareForSolve() 11 0.001004 0.001004 0.001004 0.11% FabArrayBase::getCPC() 1313 0.0008166 0.0008166 0.0008166 0.09% FabArray::FillBoundary() 3893 0.0008061 0.0008061 0.0008061 0.09% FabArrayBase::CPC::define() 454 0.0006728 0.0006728 0.0006728 0.07% FabArrayBase::getFB() 3893 0.0006276 0.0006276 0.0006276 0.07% Gravity::get_new_grav_vector() 11 0.0006154 0.0006154 0.0006154 0.07% Amr::InitAmr() 1 0.0005762 0.0005762 0.0005762 0.06% Gravity::get_old_grav_vector() 10 0.0004793 0.0004793 0.0004793 0.05% MLCellLinOp::apply() 1047 0.0004593 0.0004593 0.0004593 0.05% Amr::coarseTimeStep() 10 0.0004078 0.0004078 0.0004078 0.04% AmrLevel::FillPatch() 41 0.0004027 0.0004027 0.0004027 0.04% main() 1 0.0003467 0.0003467 0.0003467 0.04% MLCGSolver::ParallelAllReduce 1809 0.000337 0.000337 0.000337 0.04% MultiFab::max() 11 0.0003316 0.0003316 0.0003316 0.04% MLCellLinOp::defineBC() 11 0.0002828 0.0002828 0.0002828 0.03% FabArray::ParallelCopy() 851 0.0002401 0.0002401 0.0002401 0.03% FillPatchIterator::Initialize 41 0.0002113 0.0002113 0.0002113 0.02% MLMG::mgVcycle() 81 0.0001968 0.0001968 0.0001968 0.02% Castro::create_source_corrector() 10 0.0001798 0.0001798 0.0001798 0.02% Castro::subcycle_advance_ctu() 10 0.0001737 0.0001737 0.0001737 0.02% MLLinOp::defineGrids() 11 0.0001703 0.0001703 0.0001703 0.02% MLCellLinOp::correctionResidual() 405 0.0001652 0.0001652 0.0001652 0.02% Amr::timeStep() 10 0.0001575 0.0001575 0.0001575 0.02% StateData::checkPoint() 12 0.0001368 0.0001368 0.0001368 0.01% Gravity::update_max_rhs() 11 0.0001296 0.0001296 0.0001296 0.01% MLMG:computeResOfCorrection() 405 0.0001153 0.0001153 0.0001153 0.01% Gravity::solve_for_phi() 10 0.000114 0.000114 0.000114 0.01% FabArrayBase::FB::FB() 56 0.0001014 0.0001014 0.0001014 0.01% Castro::post_timestep() 10 9.37e-05 9.37e-05 9.37e-05 0.01% MLMG::mgVcycle_down::0 81 9.04e-05 9.04e-05 9.04e-05 0.01% Castro::Castro() 1 8.989e-05 8.989e-05 8.989e-05 0.01% MLMG::actualBottomSolve() 81 8.854e-05 8.854e-05 8.854e-05 0.01% MLMG::mgVcycle_down::1 81 7.898e-05 7.898e-05 7.898e-05 0.01% Castro::initialize_advance() 10 7.847e-05 7.847e-05 7.847e-05 0.01% AmrLevel::checkPoint() 3 7.803e-05 7.803e-05 7.803e-05 0.01% Castro::construct_new_source() 50 7.521e-05 7.521e-05 7.521e-05 0.01% MLMG::mgVcycle_down::4 81 7.487e-05 7.487e-05 7.487e-05 0.01% MLMG::mgVcycle_down::2 81 7.48e-05 7.48e-05 7.48e-05 0.01% MLMG::solve() 11 7.124e-05 7.124e-05 7.124e-05 0.01% MLMG::mgVcycle_down::3 81 7.024e-05 7.024e-05 7.024e-05 0.01% Castro::finalize_advance() 10 6.835e-05 6.835e-05 6.835e-05 0.01% Castro::clean_state() 62 6.691e-05 6.691e-05 6.691e-05 0.01% Castro::initialize_do_advance() 10 6.654e-05 6.654e-05 6.654e-05 0.01% Castro::enforce_consistent_e() 1 6.552e-05 6.552e-05 6.552e-05 0.01% MLMG::mgVcycle_up::4 81 6.345e-05 6.345e-05 6.345e-05 0.01% MLMG::oneIter() 81 5.579e-05 5.579e-05 5.579e-05 0.01% MLMG::mgVcycle_up::0 81 5.467e-05 5.467e-05 5.467e-05 0.01% Castro::do_advance_ctu() 10 5.118e-05 5.118e-05 5.118e-05 0.01% MLCellLinOp::solutionResidual() 92 5.108e-05 5.108e-05 5.108e-05 0.01% MLMG::mgVcycle_up::3 81 4.788e-05 4.788e-05 4.788e-05 0.01% MLMG::mgVcycle_up::1 81 4.738e-05 4.738e-05 4.738e-05 0.01% Castro::finalize_do_advance() 10 4.621e-05 4.621e-05 4.621e-05 0.00% FillPatchIterator::FillFromLevel0() 41 4.574e-05 4.574e-05 4.574e-05 0.00% MLMG::mgVcycle_up::2 81 4.565e-05 4.565e-05 4.565e-05 0.00% Castro::advance() 10 4.536e-05 4.536e-05 4.536e-05 0.00% StateData::define() 4 4.002e-05 4.002e-05 4.002e-05 0.00% MLMG::ResNormInf() 92 3.748e-05 3.748e-05 3.748e-05 0.00% Castro::swap_state_time_levels() 10 3.573e-05 3.573e-05 3.573e-05 0.00% MLMG::computeResidual() 81 3.447e-05 3.447e-05 3.447e-05 0.00% MLMG::mgVcycle_bottom 81 3.363e-05 3.363e-05 3.363e-05 0.00% FillPatchSingleLevel 41 3.35e-05 3.35e-05 3.35e-05 0.00% Castro::initMFs() 1 3.252e-05 3.252e-05 3.252e-05 0.00% Amr::writeSmallPlotFile() 1 3.251e-05 3.251e-05 3.251e-05 0.00% Amr::defBaseLevel() 1 3.16e-05 3.16e-05 3.16e-05 0.00% makeSFC 55 2.723e-05 2.723e-05 2.723e-05 0.00% Castro::buildMetrics() 1 2.622e-05 2.622e-05 2.622e-05 0.00% Castro::construct_new_gravity() 10 2.608e-05 2.608e-05 2.608e-05 0.00% Amr::FinalizeInit() 1 2.573e-05 2.573e-05 2.573e-05 0.00% MLPoisson::define() 11 2.48e-05 2.48e-05 2.48e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.41e-05 2.41e-05 2.41e-05 0.00% Castro::do_old_sources() 10 2.18e-05 2.18e-05 2.18e-05 0.00% Castro::do_new_sources() 10 2.018e-05 2.018e-05 2.018e-05 0.00% Castro::construct_old_source() 50 1.98e-05 1.98e-05 1.98e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.706e-05 1.706e-05 1.706e-05 0.00% DistributionMapping::Distribute() 56 1.639e-05 1.639e-05 1.639e-05 0.00% MLPoisson::prepareForSolve() 11 1.574e-05 1.574e-05 1.574e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.41e-05 1.41e-05 1.41e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.358e-05 1.358e-05 1.358e-05 0.00% Castro::check_for_nan() 20 1.271e-05 1.271e-05 1.271e-05 0.00% MLLinOp::define() 11 1.191e-05 1.191e-05 1.191e-05 0.00% Castro::apply_source_to_state() 20 1.178e-05 1.178e-05 1.178e-05 0.00% Castro::construct_old_gravity() 10 1.162e-05 1.162e-05 1.162e-05 0.00% Amr::initSubcycle() 1 1.162e-05 1.162e-05 1.162e-05 0.00% Castro::post_init() 1 1.062e-05 1.062e-05 1.062e-05 0.00% MLMG::computeMLResidual() 11 1.008e-05 1.008e-05 1.008e-05 0.00% Gravity::swapTimeLevels() 10 8.976e-06 8.976e-06 8.976e-06 0.00% Gravity::actual_multilevel_solve() 1 7.432e-06 7.432e-06 7.432e-06 0.00% Castro::computeNewDt() 9 7.063e-06 7.063e-06 7.063e-06 0.00% MLMG::getGradSolution() 11 6.746e-06 6.746e-06 6.746e-06 0.00% Castro::expand_state() 10 5.553e-06 5.553e-06 5.553e-06 0.00% Amr::InitializeInit() 1 5.265e-06 5.265e-06 5.265e-06 0.00% Castro::retry_advance_ctu() 10 4.758e-06 4.758e-06 4.758e-06 0.00% AmrLevel::checkPointPost() 3 4.29e-06 4.29e-06 4.29e-06 0.00% Gravity::set_mass_offset() 11 4.173e-06 4.173e-06 4.173e-06 0.00% MLMG::MLRhsNormInf() 11 4.135e-06 4.135e-06 4.135e-06 0.00% MLMG::MLResNormInf() 11 4.062e-06 4.062e-06 4.062e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.101e-06 3.101e-06 3.101e-06 0.00% Castro::FluxRegCrseInit 10 2.88e-06 2.88e-06 2.88e-06 0.00% Castro::computeInitialDt() 2 2.73e-06 2.73e-06 2.73e-06 0.00% Amr::init() 1 2.662e-06 2.662e-06 2.662e-06 0.00% AmrLevel::checkPointPre() 3 2.23e-06 2.23e-06 2.23e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.067e-06 2.067e-06 2.067e-06 0.00% Castro::FluxRegFineAdd() 10 2.063e-06 2.063e-06 2.063e-06 0.00% Castro::post_regrid() 1 1.318e-06 1.318e-06 1.318e-06 0.00% Amr::initialInit() 1 1.151e-06 1.151e-06 1.151e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9323 0.9323 0.9323 100.00% Amr::coarseTimeStep() 10 0.7828 0.7828 0.7828 83.97% Amr::timeStep() 10 0.6895 0.6895 0.6895 73.95% Castro::advance() 10 0.6775 0.6775 0.6775 72.67% Castro::subcycle_advance_ctu() 10 0.6635 0.6635 0.6635 71.16% Castro::do_advance_ctu() 10 0.6633 0.6633 0.6633 71.14% Castro::construct_ctu_hydro_source() 10 0.3145 0.3145 0.3145 33.73% Gravity::solve_phi_with_mlmg() 11 0.3025 0.3025 0.3025 32.44% Gravity::actual_solve_with_mlmg() 11 0.2929 0.2929 0.2929 31.42% Castro::construct_new_gravity() 10 0.2765 0.2765 0.2765 29.66% MLMG::solve() 11 0.2704 0.2704 0.2704 29.00% Gravity::solve_for_phi() 10 0.26 0.26 0.26 27.89% MLMG::oneIter() 81 0.2547 0.2547 0.2547 27.32% MLMG::mgVcycle() 81 0.251 0.251 0.251 26.93% VisMF::Write(FabArray) 11 0.1779 0.1779 0.1779 19.08% Amr::checkPoint() 3 0.1351 0.1351 0.1351 14.49% AmrLevel::checkPoint() 3 0.1297 0.1297 0.1297 13.91% StateData::checkPoint() 12 0.1296 0.1296 0.1296 13.90% MLCellLinOp::smooth() 1620 0.1289 0.1289 0.1289 13.83% Amr::init() 1 0.1234 0.1234 0.1234 13.23% MLCellLinOp::applyBC() 4298 0.12 0.12 0.12 12.87% MLMG::mgVcycle_bottom 81 0.07299 0.07299 0.07299 7.83% MLMG::actualBottomSolve() 81 0.07296 0.07296 0.07296 7.83% MLCGSolver::bicgstab 81 0.07213 0.07213 0.07213 7.74% Castro::clean_state() 62 0.05775 0.05775 0.05775 6.19% Amr::initialInit() 1 0.0522 0.0522 0.0522 5.60% Amr::writePlotFile() 2 0.05079 0.05079 0.05079 5.45% AmrLevel::FillPatch() 41 0.04884 0.04884 0.04884 5.24% Amr::FinalizeInit() 1 0.04698 0.04698 0.04698 5.04% Castro::post_init() 1 0.04549 0.04549 0.04549 4.88% FillPatchIterator::Initialize 41 0.04456 0.04456 0.04456 4.78% FillPatchIterator::FillFromLevel0() 41 0.04298 0.04298 0.04298 4.61% FillPatchSingleLevel 41 0.04293 0.04293 0.04293 4.60% Gravity::multilevel_solve_for_new_phi() 1 0.04284 0.04284 0.04284 4.60% Gravity::actual_multilevel_solve() 1 0.04282 0.04282 0.04282 4.59% StateDataPhysBCFunct::() 41 0.0388 0.0388 0.0388 4.16% MLCellLinOp::apply() 1047 0.03624 0.03624 0.03624 3.89% MLMG::mgVcycle_down::0 81 0.03478 0.03478 0.03478 3.73% MLPoisson::Fsmooth() 3240 0.03369 0.03369 0.03369 3.61% FabArray::FillBoundary() 3893 0.03257 0.03257 0.03257 3.49% FillBoundary_nowait() 3893 0.03176 0.03176 0.03176 3.41% StateData::FillBoundary(geom) 328 0.0269 0.0269 0.0269 2.89% MLMG::mgVcycle_up::0 81 0.02646 0.02646 0.02646 2.84% Castro::computeTemp() 63 0.02268 0.02268 0.02268 2.43% MLMG::mgVcycle_down::4 81 0.02205 0.02205 0.02205 2.37% amrex::Dot() 1100 0.02143 0.02143 0.02143 2.30% Castro::initialize_do_advance() 10 0.02128 0.02128 0.02128 2.28% Castro::normalize_species() 62 0.02121 0.02121 0.02121 2.28% MLMG:computeResOfCorrection() 405 0.02041 0.02041 0.02041 2.19% MLCellLinOp::correctionResidual() 405 0.02029 0.02029 0.02029 2.18% FabArray::norminf() 1048 0.02004 0.02004 0.02004 2.15% Gravity::get_new_grav_vector() 11 0.01865 0.01865 0.01865 2.00% Castro::do_old_sources() 10 0.01838 0.01838 0.01838 1.97% MLPoisson::define() 11 0.01765 0.01765 0.01765 1.89% MLMG::mgVcycle_down::1 81 0.01663 0.01663 0.01663 1.78% Castro::construct_old_gravity() 10 0.01555 0.01555 0.01555 1.67% Gravity::get_old_grav_vector() 10 0.01554 0.01554 0.01554 1.67% MLMG::mgVcycle_down::2 81 0.01547 0.01547 0.01547 1.66% MLMG::mgVcycle_down::3 81 0.01505 0.01505 0.01505 1.61% FabArray::ParallelCopy() 851 0.01485 0.01485 0.01485 1.59% FabArray::ParallelCopy_nowait() 851 0.01461 0.01461 0.01461 1.57% Castro::do_new_sources() 10 0.01384 0.01384 0.01384 1.48% FabArray::setVal() 1054 0.01351 0.01351 0.01351 1.45% Castro::initialize_advance() 10 0.01337 0.01337 0.01337 1.43% FabArray::Saxpy() 1353 0.01316 0.01316 0.01316 1.41% MLCGSolver::ParallelAllReduce 1809 0.01291 0.01291 0.01291 1.38% MLMG::addInterpCorrection() 405 0.01242 0.01242 0.01242 1.33% MLMG::mgVcycle_up::1 81 0.01203 0.01203 0.01203 1.29% MLMG::mgVcycle_up::4 81 0.012 0.012 0.012 1.29% Castro::expand_state() 10 0.01197 0.01197 0.01197 1.28% MLCellLinOp::defineAuxData() 11 0.01192 0.01192 0.01192 1.28% Castro::post_timestep() 10 0.01184 0.01184 0.01184 1.27% MLMG::mgVcycle_up::2 81 0.01181 0.01181 0.01181 1.27% Castro::enforce_min_density() 62 0.01171 0.01171 0.01171 1.26% amrex::average_down 405 0.0116 0.0116 0.0116 1.24% MLMG::mgVcycle_up::3 81 0.01156 0.01156 0.01156 1.24% amrex::Copy() 469 0.01091 0.01091 0.01091 1.17% MLPoisson::Fapply() 1047 0.01034 0.01034 0.01034 1.11% Gravity::fill_multipole_BCs() 11 0.009334 0.009334 0.009334 1.00% FabArray::Xpay() 730 0.007935 0.007935 0.007935 0.85% MLCellLinOp::solutionResidual() 92 0.007801 0.007801 0.007801 0.84% Castro::reset_internal_energy(MultiFab) 63 0.00699 0.00699 0.00699 0.75% MLMG::computeResidual() 81 0.006457 0.006457 0.006457 0.69% Castro::estTimeStep() 21 0.005818 0.005818 0.005818 0.62% MLCellLinOp::defineBC() 11 0.005462 0.005462 0.005462 0.59% MLMG::prepareForSolve() 11 0.00533 0.00533 0.00533 0.57% Amr::InitializeInit() 1 0.005214 0.005214 0.005214 0.56% Amr::defBaseLevel() 1 0.005209 0.005209 0.005209 0.56% BndryData::define() 11 0.00518 0.00518 0.00518 0.56% Castro::initData() 1 0.004486 0.004486 0.004486 0.48% amrex::Add() 81 0.003592 0.003592 0.003592 0.39% Castro::construct_new_source() 50 0.003361 0.003361 0.003361 0.36% Castro::construct_new_gravity_source() 10 0.003286 0.003286 0.003286 0.35% Castro::enforce_speed_limit() 62 0.002997 0.002997 0.002997 0.32% Castro::construct_old_source() 50 0.002826 0.002826 0.002826 0.30% Castro::construct_old_gravity_source() 10 0.002806 0.002806 0.002806 0.30% Castro::computeNewDt() 9 0.002546 0.002546 0.002546 0.27% Castro::finalize_do_advance() 10 0.002274 0.002274 0.002274 0.24% MLMG::ResNormInf() 92 0.002228 0.002228 0.002228 0.24% Castro::apply_source_to_state() 20 0.001865 0.001865 0.001865 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001825 0.001825 0.001825 0.20% check_for_negative_density() 10 0.001807 0.001807 0.001807 0.19% Castro::reset_internal_energy(Fab) 504 0.001668 0.001668 0.001668 0.18% MLCellLinOp::setLevelBC() 11 0.001611 0.001611 0.001611 0.17% MLMG::getGradSolution() 11 0.001609 0.001609 0.001609 0.17% MLCellLinOp::compGrad() 11 0.001602 0.001602 0.001602 0.17% FabArrayBase::getCPC() 1313 0.001489 0.001489 0.001489 0.16% MLMG::computeMLResidual() 11 0.001389 0.001389 0.001389 0.15% MLPoisson::prepareForSolve() 11 0.001384 0.001384 0.001384 0.15% FabArray::mult() 43 0.001383 0.001383 0.001383 0.15% FabArray::setDomainBndry() 41 0.001369 0.001369 0.001369 0.15% MLCellLinOp::prepareForSolve() 11 0.001368 0.001368 0.001368 0.15% Castro::check_for_nan() 20 0.001281 0.001281 0.001281 0.14% MultiFab::contains_nan() 20 0.001269 0.001269 0.001269 0.14% Castro::post_regrid() 1 0.00118 0.00118 0.00118 0.13% Castro::computeInitialDt() 2 0.001053 0.001053 0.001053 0.11% Gravity::update_max_rhs() 11 0.001011 0.001011 0.001011 0.11% FabArrayBase::getFB() 3893 0.000729 0.000729 0.000729 0.08% FabArrayBase::CPC::define() 454 0.0006728 0.0006728 0.0006728 0.07% Castro::Castro() 1 0.0006325 0.0006325 0.0006325 0.07% Castro::finalize_advance() 10 0.0006142 0.0006142 0.0006142 0.07% Amr::InitAmr() 1 0.0005878 0.0005878 0.0005878 0.06% Gravity::swapTimeLevels() 10 0.0004558 0.0004558 0.0004558 0.05% MLMG::MLResNormInf() 11 0.0003379 0.0003379 0.0003379 0.04% Castro::buildMetrics() 1 0.0003332 0.0003332 0.0003332 0.04% MultiFab::max() 11 0.0003316 0.0003316 0.0003316 0.04% MLLinOp::define() 11 0.0002403 0.0002403 0.0002403 0.03% MLMG::MLRhsNormInf() 11 0.000235 0.000235 0.000235 0.03% MLLinOp::defineGrids() 11 0.0002284 0.0002284 0.0002284 0.02% Castro::create_source_corrector() 10 0.0001798 0.0001798 0.0001798 0.02% FabArrayBase::FB::FB() 56 0.0001014 0.0001014 0.0001014 0.01% Castro::enforce_consistent_e() 1 6.552e-05 6.552e-05 6.552e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.601e-05 5.601e-05 5.601e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.412e-05 5.412e-05 5.412e-05 0.01% makeSFC 55 4.243e-05 4.243e-05 4.243e-05 0.00% StateData::define() 4 4.002e-05 4.002e-05 4.002e-05 0.00% Castro::swap_state_time_levels() 10 3.573e-05 3.573e-05 3.573e-05 0.00% Castro::initMFs() 1 3.252e-05 3.252e-05 3.252e-05 0.00% Amr::writeSmallPlotFile() 1 3.251e-05 3.251e-05 3.251e-05 0.00% DistributionMapping::Distribute() 56 1.639e-05 1.639e-05 1.639e-05 0.00% Amr::initSubcycle() 1 1.162e-05 1.162e-05 1.162e-05 0.00% Castro::retry_advance_ctu() 10 4.758e-06 4.758e-06 4.758e-06 0.00% AmrLevel::checkPointPost() 3 4.29e-06 4.29e-06 4.29e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.286e-06 4.286e-06 4.286e-06 0.00% Gravity::set_mass_offset() 11 4.173e-06 4.173e-06 4.173e-06 0.00% Castro::FluxRegCrseInit 10 2.88e-06 2.88e-06 2.88e-06 0.00% AmrLevel::checkPointPre() 3 2.23e-06 2.23e-06 2.23e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.067e-06 2.067e-06 2.067e-06 0.00% Castro::FluxRegFineAdd() 10 2.063e-06 2.063e-06 2.063e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 84 MiB 9042 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1011 KiB 39 MiB Castro::initialize_do_advance() 80 80 27 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1531 KiB 28 MiB Castro::initialize_advance() 80 80 17 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7617 KiB 14 MiB MLMG::prepareForSolve() 660 660 3566 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 207 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 170 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7516 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 16 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2102 B 2048 KiB Gravity::solve_for_phi() 80 80 570 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 93 KiB 2048 KiB BndryData::define() 1056 1056 326 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 207 KiB 671 KiB Castro::estTimeStep() 21 21 3040 B 480 KiB VisMF::Write(FabArray) 656 656 3307 B 320 KiB Castro::normalize_species() 62 62 7414 B 320 KiB amrex::average_down 1054 1054 1236 B 257 KiB MLMG::addInterpCorrection() 1053 1053 1136 B 257 KiB amrex::Dot() 1343 1343 3394 B 160 KiB FabArray::norminf() 1129 1129 3319 B 160 KiB check_for_negative_density() 10 10 311 B 160 KiB Castro::initData() 1 1 49 B 160 KiB MultiFab::max() 11 11 55 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 27 B 20 KiB MLPoisson::Fsmooth() 132 132 3481 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 43 B 10 KiB FillBoundary_nowait() 751 751 291 B 9648 B MLCellLinOp::applyBC() 8596 8596 216 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3949 B 6144 B StateData::FillBoundary(geom) 1992 1992 43 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 367 B 1248 B MLCGSolver::bicgstab 405 405 93 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 7439 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 39 KiB 8192 KiB VisMF::Write(FabArray) 744 744 418 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3481 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 43 B 10 KiB FillBoundary_nowait() 751 751 292 B 9648 B MLCellLinOp::applyBC() 4298 4298 215 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3949 B 6144 B Gravity::get_new_grav_vector() 3 3 2902 B 3072 B StateData::FillBoundary(geom) 1992 1992 43 B 3024 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 82 82 270 B 1296 B MLMG::addInterpCorrection() 81 81 2 B 1024 B MLMG::prepareForSolve() 11 11 297 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1343 1343 24 B 400 B FabArray::norminf() 1129 1129 9 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2167 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.02-21-g99b47cb58a3c) finalized Initializing AMReX (24.02-21-g99b47cb58a3c)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.02-21-g99b47cb58a3c) initialized Starting run at 09:20:39 UTC on 2024-02-20. Successfully read inputs file ... Castro git describe: 24.02-21-g56d243e7a AMReX git describe: 24.02-21-g99b47cb58 Microphysics git describe: 24.02-19-g32b2c42a reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.518302887 Restart time = 0.102684979 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.071124684 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049770388 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.080092129 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.077864185 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.070679838 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.028407913 seconds Ending run at 09:20:39 UTC on 2024-02-20. Run time = 0.481790337 Run time without initialization = 0.378443991 Average number of zones advanced per microsecond: 3.463 Average number of zones advanced per microsecond per rank: 3.463 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.4818 ... 0.4818 ... 0.4818 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1467 0.1467 0.1467 30.44% VisMF::Read() 3 0.06037 0.06037 0.06037 12.53% MLCellLinOp::applyBC() 1910 0.03613 0.03613 0.03613 7.50% Amr::restart() 1 0.03542 0.03542 0.03542 7.35% VisMF::Write(FabArray) 1 0.02594 0.02594 0.02594 5.38% FillBoundary_nowait() 1730 0.01987 0.01987 0.01987 4.12% MLPoisson::Fsmooth() 1440 0.01518 0.01518 0.01518 3.15% StateData::FillBoundary(geom) 160 0.01324 0.01324 0.01324 2.75% Castro::normalize_species() 30 0.01067 0.01067 0.01067 2.22% amrex::Dot() 484 0.009362 0.009362 0.009362 1.94% FabArray::norminf() 465 0.008783 0.008783 0.008783 1.82% Castro::computeTemp() 30 0.007513 0.007513 0.007513 1.56% FabArray::setVal() 501 0.006586 0.006586 0.006586 1.37% FabArray::ParallelCopy_nowait() 380 0.006285 0.006285 0.006285 1.30% FabArray::Saxpy() 597 0.005915 0.005915 0.005915 1.23% MLCellLinOp::defineAuxData() 6 0.005704 0.005704 0.005704 1.18% Castro::enforce_min_density() 30 0.005633 0.005633 0.005633 1.17% Gravity::fill_multipole_BCs() 6 0.005478 0.005478 0.005478 1.14% amrex::Copy() 221 0.005442 0.005442 0.005442 1.13% StateDataPhysBCFunct::() 20 0.004988 0.004988 0.004988 1.04% MLPoisson::Fapply() 464 0.004572 0.004572 0.004572 0.95% Castro::estTimeStep() 10 0.003598 0.003598 0.003598 0.75% FabArray::Xpay() 325 0.003531 0.003531 0.003531 0.73% MLMG::addInterpCorrection() 180 0.003157 0.003157 0.003157 0.66% amrex::average_down 180 0.002775 0.002775 0.002775 0.58% Amr::writePlotFile() 1 0.002308 0.002308 0.002308 0.48% BndryData::define() 6 0.002232 0.002232 0.002232 0.46% Castro::reset_internal_energy(MultiFab) 30 0.002109 0.002109 0.002109 0.44% Castro::construct_new_gravity_source() 5 0.001858 0.001858 0.001858 0.39% amrex::Add() 36 0.001585 0.001585 0.001585 0.33% Castro::construct_old_gravity_source() 5 0.001426 0.001426 0.001426 0.30% Castro::enforce_speed_limit() 30 0.001202 0.001202 0.001202 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001014 0.001014 0.001014 0.21% Gravity::actual_solve_with_mlmg() 6 0.0008913 0.0008913 0.0008913 0.18% MLCellLinOp::setLevelBC() 6 0.0008822 0.0008822 0.0008822 0.18% Castro::reset_internal_energy(Fab) 240 0.0008093 0.0008093 0.0008093 0.17% MLCellLinOp::prepareForSolve() 6 0.0007681 0.0007681 0.0007681 0.16% MLCGSolver::bicgstab 36 0.0007188 0.0007188 0.0007188 0.15% FabArray::setDomainBndry() 20 0.0007123 0.0007123 0.0007123 0.15% FabArray::mult() 22 0.0006978 0.0006978 0.0006978 0.14% check_for_negative_density() 5 0.0006898 0.0006898 0.0006898 0.14% MultiFab::contains_nan() 10 0.000654 0.000654 0.000654 0.14% MLCellLinOp::compGrad() 6 0.0006258 0.0006258 0.0006258 0.13% MLMG::prepareForSolve() 6 0.0005586 0.0005586 0.0005586 0.12% MLCellLinOp::smooth() 720 0.0005261 0.0005261 0.0005261 0.11% Amr::InitAmr() 1 0.0004915 0.0004915 0.0004915 0.10% FabArrayBase::CPC::define() 244 0.0004192 0.0004192 0.0004192 0.09% FabArrayBase::getCPC() 632 0.0004168 0.0004168 0.0004168 0.09% FabArray::FillBoundary() 1730 0.0003679 0.0003679 0.0003679 0.08% Gravity::get_old_grav_vector() 5 0.0003519 0.0003519 0.0003519 0.07% Gravity::get_new_grav_vector() 5 0.0003214 0.0003214 0.0003214 0.07% main() 1 0.0003065 0.0003065 0.0003065 0.06% FabArrayBase::getFB() 1730 0.0002692 0.0002692 0.0002692 0.06% MultiFab::max() 6 0.0002083 0.0002083 0.0002083 0.04% AmrLevel::FillPatch() 20 0.0002028 0.0002028 0.0002028 0.04% MLCellLinOp::apply() 464 0.0002016 0.0002016 0.0002016 0.04% Amr::coarseTimeStep() 5 0.0001984 0.0001984 0.0001984 0.04% MLCellLinOp::defineBC() 6 0.0001518 0.0001518 0.0001518 0.03% MLCGSolver::ParallelAllReduce 798 0.000145 0.000145 0.000145 0.03% Castro::advance() 5 0.0001265 0.0001265 0.0001265 0.03% FabArray::ParallelCopy() 380 0.0001173 0.0001173 0.0001173 0.02% FillPatchIterator::Initialize 20 0.0001161 0.0001161 0.0001161 0.02% Castro::create_source_corrector() 5 0.000105 0.000105 0.000105 0.02% Castro::subcycle_advance_ctu() 5 9.908e-05 9.908e-05 9.908e-05 0.02% MLMG::mgVcycle() 36 9.474e-05 9.474e-05 9.474e-05 0.02% MLLinOp::defineGrids() 6 9.385e-05 9.385e-05 9.385e-05 0.02% Amr::timeStep() 5 9.163e-05 9.163e-05 9.163e-05 0.02% Castro::do_advance_ctu() 5 8.598e-05 8.598e-05 8.598e-05 0.02% Castro::finalize_do_advance() 5 7.963e-05 7.963e-05 7.963e-05 0.02% AmrLevel::restart() 1 7.725e-05 7.725e-05 7.725e-05 0.02% StateData::restartDoit() 4 7.439e-05 7.439e-05 7.439e-05 0.02% Castro::construct_new_source() 25 7.386e-05 7.386e-05 7.386e-05 0.02% MLCellLinOp::correctionResidual() 180 7.202e-05 7.202e-05 7.202e-05 0.01% Castro::initialize_do_advance() 5 6.846e-05 6.846e-05 6.846e-05 0.01% Gravity::update_max_rhs() 6 6.375e-05 6.375e-05 6.375e-05 0.01% FabArrayBase::FB::FB() 26 6.044e-05 6.044e-05 6.044e-05 0.01% Gravity::solve_for_phi() 5 5.49e-05 5.49e-05 5.49e-05 0.01% MLMG:computeResOfCorrection() 180 5.422e-05 5.422e-05 5.422e-05 0.01% Castro::construct_old_source() 25 4.733e-05 4.733e-05 4.733e-05 0.01% Amr::writeSmallPlotFile() 1 4.181e-05 4.181e-05 4.181e-05 0.01% MLMG::actualBottomSolve() 36 4.067e-05 4.067e-05 4.067e-05 0.01% Castro::swap_state_time_levels() 5 3.977e-05 3.977e-05 3.977e-05 0.01% MLMG::mgVcycle_down::0 36 3.845e-05 3.845e-05 3.845e-05 0.01% Castro::initialize_advance() 5 3.672e-05 3.672e-05 3.672e-05 0.01% MLMG::mgVcycle_down::1 36 3.628e-05 3.628e-05 3.628e-05 0.01% MLMG::solve() 6 3.517e-05 3.517e-05 3.517e-05 0.01% MLMG::mgVcycle_down::2 36 3.351e-05 3.351e-05 3.351e-05 0.01% MLMG::mgVcycle_down::4 36 3.3e-05 3.3e-05 3.3e-05 0.01% Castro::do_old_sources() 5 3.266e-05 3.266e-05 3.266e-05 0.01% Castro::post_restart() 1 3.196e-05 3.196e-05 3.196e-05 0.01% Castro::clean_state() 30 3.113e-05 3.113e-05 3.113e-05 0.01% MLMG::mgVcycle_down::3 36 3.086e-05 3.086e-05 3.086e-05 0.01% Castro::finalize_advance() 5 2.954e-05 2.954e-05 2.954e-05 0.01% MLMG::mgVcycle_up::4 36 2.878e-05 2.878e-05 2.878e-05 0.01% MLMG::oneIter() 36 2.84e-05 2.84e-05 2.84e-05 0.01% Castro::buildMetrics() 1 2.805e-05 2.805e-05 2.805e-05 0.01% Castro::do_new_sources() 5 2.793e-05 2.793e-05 2.793e-05 0.01% Castro::initMFs() 1 2.544e-05 2.544e-05 2.544e-05 0.01% MLCellLinOp::solutionResidual() 42 2.304e-05 2.304e-05 2.304e-05 0.00% MLMG::mgVcycle_up::0 36 2.295e-05 2.295e-05 2.295e-05 0.00% FillPatchIterator::FillFromLevel0() 20 2.246e-05 2.246e-05 2.246e-05 0.00% MLMG::mgVcycle_up::3 36 2.148e-05 2.148e-05 2.148e-05 0.00% MLMG::ResNormInf() 42 2.102e-05 2.102e-05 2.102e-05 0.00% MLMG::computeResidual() 36 2.043e-05 2.043e-05 2.043e-05 0.00% MLMG::mgVcycle_up::2 36 1.994e-05 1.994e-05 1.994e-05 0.00% MLMG::mgVcycle_up::1 36 1.959e-05 1.959e-05 1.959e-05 0.00% MLPoisson::define() 6 1.849e-05 1.849e-05 1.849e-05 0.00% Castro::construct_new_gravity() 5 1.758e-05 1.758e-05 1.758e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.667e-05 1.667e-05 1.667e-05 0.00% FillPatchSingleLevel 20 1.535e-05 1.535e-05 1.535e-05 0.00% MLMG::mgVcycle_bottom 36 1.512e-05 1.512e-05 1.512e-05 0.00% makeSFC 30 1.433e-05 1.433e-05 1.433e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.416e-05 1.416e-05 1.416e-05 0.00% MLPoisson::prepareForSolve() 6 1.147e-05 1.147e-05 1.147e-05 0.00% DistributionMapping::Distribute() 31 9.414e-06 9.414e-06 9.414e-06 0.00% Amr::initSubcycle() 1 8.702e-06 8.702e-06 8.702e-06 0.00% Gravity::actual_multilevel_solve() 1 8.532e-06 8.532e-06 8.532e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.166e-06 7.166e-06 7.166e-06 0.00% Castro::check_for_nan() 10 6.788e-06 6.788e-06 6.788e-06 0.00% MLLinOp::define() 6 6.562e-06 6.562e-06 6.562e-06 0.00% Castro::construct_old_gravity() 5 6.212e-06 6.212e-06 6.212e-06 0.00% Castro::apply_source_to_state() 10 5.823e-06 5.823e-06 5.823e-06 0.00% Castro::post_timestep() 5 5.37e-06 5.37e-06 5.37e-06 0.00% Gravity::swapTimeLevels() 5 4.236e-06 4.236e-06 4.236e-06 0.00% MLMG::computeMLResidual() 6 4.024e-06 4.024e-06 4.024e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.971e-06 3.971e-06 3.971e-06 0.00% Castro::computeNewDt() 5 3.685e-06 3.685e-06 3.685e-06 0.00% MLMG::getGradSolution() 6 3.247e-06 3.247e-06 3.247e-06 0.00% Castro::expand_state() 5 2.964e-06 2.964e-06 2.964e-06 0.00% MLMG::MLResNormInf() 6 2.905e-06 2.905e-06 2.905e-06 0.00% Castro::retry_advance_ctu() 5 2.351e-06 2.351e-06 2.351e-06 0.00% MLMG::MLRhsNormInf() 6 2.246e-06 2.246e-06 2.246e-06 0.00% Gravity::set_mass_offset() 6 1.929e-06 1.929e-06 1.929e-06 0.00% Castro::FluxRegCrseInit 5 1.544e-06 1.544e-06 1.544e-06 0.00% Castro::FluxRegFineAdd() 5 1.245e-06 1.245e-06 1.245e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.14e-06 1.14e-06 1.14e-06 0.00% Amr::init() 1 1.002e-06 1.002e-06 1.002e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4818 0.4818 0.4818 100.00% Amr::coarseTimeStep() 5 0.3497 0.3497 0.3497 72.58% Amr::timeStep() 5 0.3472 0.3472 0.3472 72.06% Castro::advance() 5 0.3414 0.3414 0.3414 70.86% Castro::subcycle_advance_ctu() 5 0.3336 0.3336 0.3336 69.23% Castro::do_advance_ctu() 5 0.3335 0.3335 0.3335 69.21% Castro::construct_ctu_hydro_source() 5 0.1513 0.1513 0.1513 31.41% Castro::construct_new_gravity() 5 0.1444 0.1444 0.1444 29.98% Gravity::solve_phi_with_mlmg() 6 0.142 0.142 0.142 29.46% Gravity::actual_solve_with_mlmg() 6 0.1362 0.1362 0.1362 28.27% Gravity::solve_for_phi() 5 0.1362 0.1362 0.1362 28.26% MLMG::solve() 6 0.1239 0.1239 0.1239 25.71% MLMG::oneIter() 36 0.1161 0.1161 0.1161 24.09% MLMG::mgVcycle() 36 0.1145 0.1145 0.1145 23.76% Amr::init() 1 0.1027 0.1027 0.1027 21.32% Amr::restart() 1 0.1027 0.1027 0.1027 21.32% AmrLevel::restart() 1 0.06074 0.06074 0.06074 12.61% MLCellLinOp::smooth() 720 0.06073 0.06073 0.06073 12.60% StateData::restartDoit() 4 0.06065 0.06065 0.06065 12.59% VisMF::Read() 3 0.06037 0.06037 0.06037 12.53% MLCellLinOp::applyBC() 1910 0.0567 0.0567 0.0567 11.77% MLMG::mgVcycle_bottom 36 0.03194 0.03194 0.03194 6.63% MLMG::actualBottomSolve() 36 0.03192 0.03192 0.03192 6.62% MLCGSolver::bicgstab 36 0.03155 0.03155 0.03155 6.55% Amr::writePlotFile() 1 0.0285 0.0285 0.0285 5.92% Castro::clean_state() 30 0.02797 0.02797 0.02797 5.81% VisMF::Write(FabArray) 1 0.02594 0.02594 0.02594 5.38% AmrLevel::FillPatch() 20 0.02326 0.02326 0.02326 4.83% FillPatchIterator::Initialize 20 0.02114 0.02114 0.02114 4.39% FabArray::FillBoundary() 1730 0.02057 0.02057 0.02057 4.27% FillPatchIterator::FillFromLevel0() 20 0.02032 0.02032 0.02032 4.22% FillPatchSingleLevel 20 0.02029 0.02029 0.02029 4.21% FillBoundary_nowait() 1730 0.0202 0.0202 0.0202 4.19% MLMG::mgVcycle_up::0 36 0.01843 0.01843 0.01843 3.82% StateDataPhysBCFunct::() 20 0.01822 0.01822 0.01822 3.78% MLCellLinOp::apply() 464 0.01616 0.01616 0.01616 3.35% MLPoisson::Fsmooth() 1440 0.01518 0.01518 0.01518 3.15% MLMG::mgVcycle_down::0 36 0.01515 0.01515 0.01515 3.14% StateData::FillBoundary(geom) 160 0.01324 0.01324 0.01324 2.75% Castro::initialize_do_advance() 5 0.01088 0.01088 0.01088 2.26% Castro::normalize_species() 30 0.01067 0.01067 0.01067 2.22% Castro::computeTemp() 30 0.01043 0.01043 0.01043 2.16% MLPoisson::define() 6 0.009669 0.009669 0.009669 2.01% Castro::do_old_sources() 5 0.009598 0.009598 0.009598 1.99% amrex::Dot() 484 0.009362 0.009362 0.009362 1.94% MLMG:computeResOfCorrection() 180 0.009015 0.009015 0.009015 1.87% MLCellLinOp::correctionResidual() 180 0.008961 0.008961 0.008961 1.86% FabArray::norminf() 465 0.008783 0.008783 0.008783 1.82% Gravity::get_new_grav_vector() 5 0.008135 0.008135 0.008135 1.69% Castro::construct_old_gravity() 5 0.007816 0.007816 0.007816 1.62% Gravity::get_old_grav_vector() 5 0.00781 0.00781 0.00781 1.62% MLMG::mgVcycle_down::1 36 0.007701 0.007701 0.007701 1.60% Castro::do_new_sources() 5 0.007489 0.007489 0.007489 1.55% Castro::initialize_advance() 5 0.0074 0.0074 0.0074 1.54% MLMG::mgVcycle_down::2 36 0.00686 0.00686 0.00686 1.42% FabArray::ParallelCopy() 380 0.006822 0.006822 0.006822 1.42% FabArray::ParallelCopy_nowait() 380 0.006704 0.006704 0.006704 1.39% MLMG::mgVcycle_down::3 36 0.006699 0.006699 0.006699 1.39% MLMG::mgVcycle_down::4 36 0.006638 0.006638 0.006638 1.38% FabArray::setVal() 501 0.006586 0.006586 0.006586 1.37% MLCellLinOp::defineAuxData() 6 0.006532 0.006532 0.006532 1.36% Castro::post_restart() 1 0.006391 0.006391 0.006391 1.33% Gravity::multilevel_solve_for_new_phi() 1 0.006012 0.006012 0.006012 1.25% Gravity::actual_multilevel_solve() 1 0.005995 0.005995 0.005995 1.24% Castro::expand_state() 5 0.005922 0.005922 0.005922 1.23% FabArray::Saxpy() 597 0.005915 0.005915 0.005915 1.23% Castro::post_timestep() 5 0.005703 0.005703 0.005703 1.18% MLCGSolver::ParallelAllReduce 798 0.005701 0.005701 0.005701 1.18% Castro::enforce_min_density() 30 0.005633 0.005633 0.005633 1.17% Gravity::fill_multipole_BCs() 6 0.005604 0.005604 0.005604 1.16% MLMG::addInterpCorrection() 180 0.00551 0.00551 0.00551 1.14% amrex::Copy() 221 0.005442 0.005442 0.005442 1.13% MLMG::mgVcycle_up::4 36 0.005325 0.005325 0.005325 1.11% MLMG::mgVcycle_up::1 36 0.00532 0.00532 0.00532 1.10% MLMG::mgVcycle_up::2 36 0.005203 0.005203 0.005203 1.08% amrex::average_down 180 0.005188 0.005188 0.005188 1.08% MLMG::mgVcycle_up::3 36 0.005104 0.005104 0.005104 1.06% MLPoisson::Fapply() 464 0.004572 0.004572 0.004572 0.95% MLCellLinOp::solutionResidual() 42 0.003665 0.003665 0.003665 0.76% Castro::estTimeStep() 10 0.003598 0.003598 0.003598 0.75% FabArray::Xpay() 325 0.003531 0.003531 0.003531 0.73% MLCellLinOp::defineBC() 6 0.002988 0.002988 0.002988 0.62% Castro::reset_internal_energy(MultiFab) 30 0.002918 0.002918 0.002918 0.61% MLMG::prepareForSolve() 6 0.002916 0.002916 0.002916 0.61% MLMG::computeResidual() 36 0.002871 0.002871 0.002871 0.60% BndryData::define() 6 0.002836 0.002836 0.002836 0.59% Castro::computeNewDt() 5 0.002323 0.002323 0.002323 0.48% Castro::construct_new_source() 25 0.001932 0.001932 0.001932 0.40% Castro::construct_new_gravity_source() 5 0.001858 0.001858 0.001858 0.39% amrex::Add() 36 0.001585 0.001585 0.001585 0.33% Castro::construct_old_source() 25 0.001473 0.001473 0.001473 0.31% Castro::construct_old_gravity_source() 5 0.001426 0.001426 0.001426 0.30% Castro::finalize_do_advance() 5 0.001359 0.001359 0.001359 0.28% Castro::enforce_speed_limit() 30 0.001202 0.001202 0.001202 0.25% MLMG::ResNormInf() 42 0.001037 0.001037 0.001037 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001014 0.001014 0.001014 0.21% Castro::apply_source_to_state() 10 0.000962 0.000962 0.000962 0.20% MLMG::getGradSolution() 6 0.0009119 0.0009119 0.0009119 0.19% MLCellLinOp::compGrad() 6 0.0009087 0.0009087 0.0009087 0.19% MLCellLinOp::setLevelBC() 6 0.0008822 0.0008822 0.0008822 0.18% FabArrayBase::getCPC() 632 0.0008359 0.0008359 0.0008359 0.17% MLMG::computeMLResidual() 6 0.0008187 0.0008187 0.0008187 0.17% Castro::reset_internal_energy(Fab) 240 0.0008093 0.0008093 0.0008093 0.17% MLPoisson::prepareForSolve() 6 0.0007795 0.0007795 0.0007795 0.16% MLCellLinOp::prepareForSolve() 6 0.0007681 0.0007681 0.0007681 0.16% Gravity::update_max_rhs() 6 0.0007181 0.0007181 0.0007181 0.15% FabArray::setDomainBndry() 20 0.0007123 0.0007123 0.0007123 0.15% FabArray::mult() 22 0.0006978 0.0006978 0.0006978 0.14% check_for_negative_density() 5 0.0006898 0.0006898 0.0006898 0.14% Castro::check_for_nan() 10 0.0006607 0.0006607 0.0006607 0.14% MultiFab::contains_nan() 10 0.000654 0.000654 0.000654 0.14% Amr::InitAmr() 1 0.0005002 0.0005002 0.0005002 0.10% FabArrayBase::CPC::define() 244 0.0004192 0.0004192 0.0004192 0.09% FabArrayBase::getFB() 1730 0.0003296 0.0003296 0.0003296 0.07% Castro::finalize_advance() 5 0.0002992 0.0002992 0.0002992 0.06% Gravity::swapTimeLevels() 5 0.0002371 0.0002371 0.0002371 0.05% MultiFab::max() 6 0.0002083 0.0002083 0.0002083 0.04% MLMG::MLResNormInf() 6 0.0001929 0.0001929 0.0001929 0.04% Castro::buildMetrics() 1 0.0001529 0.0001529 0.0001529 0.03% MLLinOp::define() 6 0.000131 0.000131 0.000131 0.03% MLMG::MLRhsNormInf() 6 0.0001257 0.0001257 0.0001257 0.03% MLLinOp::defineGrids() 6 0.0001245 0.0001245 0.0001245 0.03% Castro::create_source_corrector() 5 0.000105 0.000105 0.000105 0.02% FabArrayBase::FB::FB() 26 6.044e-05 6.044e-05 6.044e-05 0.01% Amr::writeSmallPlotFile() 1 4.181e-05 4.181e-05 4.181e-05 0.01% Castro::swap_state_time_levels() 5 3.977e-05 3.977e-05 3.977e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.947e-05 2.947e-05 2.947e-05 0.01% Castro::initMFs() 1 2.544e-05 2.544e-05 2.544e-05 0.01% makeSFC 30 2.231e-05 2.231e-05 2.231e-05 0.00% DistributionMapping::Distribute() 31 9.414e-06 9.414e-06 9.414e-06 0.00% Amr::initSubcycle() 1 8.702e-06 8.702e-06 8.702e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.413e-06 5.413e-06 5.413e-06 0.00% Castro::retry_advance_ctu() 5 2.351e-06 2.351e-06 2.351e-06 0.00% Gravity::set_mass_offset() 6 1.929e-06 1.929e-06 1.929e-06 0.00% Castro::FluxRegCrseInit 5 1.544e-06 1.544e-06 1.544e-06 0.00% Castro::FluxRegFineAdd() 5 1.245e-06 1.245e-06 1.245e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.14e-06 1.14e-06 1.14e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 166 MiB 9042 MiB Castro::initMFs() 48 48 54 MiB 68 MiB Castro::swap_state_time_levels() 32 32 43 MiB 55 MiB StateData::restartDoit() 32 32 49 MiB 55 MiB FillPatchIterator::Initialize 160 160 952 KiB 39 MiB Castro::initialize_do_advance() 40 40 26 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1589 KiB 28 MiB Castro::initialize_advance() 40 40 16 MiB 23 MiB Castro::buildMetrics() 32 32 12 MiB 15 MiB Castro::post_restart() 48 48 6054 KiB 14 MiB MLMG::prepareForSolve() 361 361 3157 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 168 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 172 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6042 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 19 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2965 B 2048 KiB Gravity::solve_for_phi() 40 40 577 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 25 KiB 2048 KiB BndryData::define() 576 576 290 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 186 KiB 671 KiB Castro::estTimeStep() 10 10 3510 B 480 KiB VisMF::Write(FabArray) 112 112 2105 B 320 KiB Castro::normalize_species() 30 30 7215 B 320 KiB amrex::average_down 469 469 1048 B 257 KiB MLMG::addInterpCorrection() 468 468 985 B 257 KiB amrex::Dot() 592 592 2868 B 160 KiB FabArray::norminf() 501 501 2811 B 160 KiB check_for_negative_density() 5 5 228 B 160 KiB MultiFab::max() 6 6 67 B 160 KiB FabArray::setVal() 66 66 16 KiB 23 KiB MultiFab::contains_nan() 10 10 27 B 20 KiB MLPoisson::Fsmooth() 60 60 3064 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 41 B 10 KiB FillBoundary_nowait() 336 336 371 B 9648 B MLCellLinOp::applyBC() 3820 3820 189 B 9344 B amrex::Copy() 56 56 5252 B 8464 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B StateData::FillBoundary(geom) 960 960 37 B 2496 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCellLinOp::defineBC() 36 36 326 B 1248 B MLCGSolver::bicgstab 180 180 78 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1005 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 76 KiB 8192 KiB VisMF::Write(FabArray) 120 120 141 KiB 3584 KiB VisMF::Read() 24 24 189 KiB 3000 KiB FabArray::setVal() 66 66 16 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3064 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 41 B 10 KiB FillBoundary_nowait() 336 336 371 B 9648 B MLCellLinOp::applyBC() 1910 1910 187 B 9328 B amrex::Copy() 56 56 5252 B 8464 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B Gravity::get_old_grav_vector() 3 3 2374 B 3072 B Gravity::fill_multipole_BCs() 18 18 5 B 2832 B StateData::FillBoundary(geom) 960 960 37 B 2496 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::prepareForSolve() 7 7 477 B 1296 B amrex::average_down 37 37 191 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 21 B 400 B FabArray::norminf() 501 501 8 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2167 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.02-21-g99b47cb58a3c) finalized