Initializing AMReX (24.04-9-g2a3955a5f5aa)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.04-9-g2a3955a5f5aa) initialized Starting run at 09:30:04 UTC on 2024-04-09. Successfully read inputs file ... Castro git describe: 24.04-11-g2407e8176 AMReX git describe: 24.04-9-g2a3955a5f Microphysics git describe: 24.04-4-ge5fecb7a reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.043553425 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.024060633 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.070142307 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.068166678 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.070003306 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.067094199 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.063403349 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.050565705 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.053120194 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.0716591 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.07106742 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.066017422 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.067614843 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.042750287 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.023996104 seconds Ending run at 09:30:05 UTC on 2024-04-09. Run time = 0.907707792 Run time without initialization = 0.786289304 Average number of zones advanced per microsecond: 3.334 Average number of zones advanced per microsecond per rank: 3.334 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.9077 ... 0.9077 ... 0.9077 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2904 0.2904 0.2904 31.99% VisMF::Write(FabArray) 11 0.1753 0.1753 0.1753 19.32% MLCellLinOp::applyBC() 4351 0.08031 0.08031 0.08031 8.85% MLPoisson::Fsmooth() 3280 0.03328 0.03328 0.03328 3.67% FillBoundary_nowait() 3941 0.03159 0.03159 0.03159 3.48% StateData::FillBoundary(geom) 328 0.02487 0.02487 0.02487 2.74% amrex::Dot() 1114 0.02111 0.02111 0.02111 2.33% FabArray::norminf() 1061 0.01962 0.01962 0.01962 2.16% Castro::normalize_species() 62 0.01785 0.01785 0.01785 1.97% StateDataPhysBCFunct::() 41 0.01707 0.01707 0.01707 1.88% Castro::computeTemp() 63 0.0148 0.0148 0.0148 1.63% FabArray::ParallelCopy_nowait() 861 0.01369 0.01369 0.01369 1.51% FabArray::setVal() 1062 0.01335 0.01335 0.01335 1.47% FabArray::Saxpy() 1370 0.01298 0.01298 0.01298 1.43% Castro::enforce_min_density() 62 0.01216 0.01216 0.01216 1.34% amrex::Copy() 472 0.01092 0.01092 0.01092 1.20% MLCellLinOp::defineAuxData() 11 0.01021 0.01021 0.01021 1.12% MLPoisson::Fapply() 1060 0.0102 0.0102 0.0102 1.12% Gravity::fill_multipole_BCs() 11 0.009262 0.009262 0.009262 1.02% FabArray::Xpay() 739 0.007776 0.007776 0.007776 0.86% Amr::checkPoint() 3 0.00701 0.00701 0.00701 0.77% MLMG::addInterpCorrection() 410 0.006967 0.006967 0.006967 0.77% amrex::average_down 410 0.006161 0.006161 0.006161 0.68% Castro::estTimeStep() 21 0.004977 0.004977 0.004977 0.55% Castro::reset_internal_energy(MultiFab) 63 0.00496 0.00496 0.00496 0.55% Castro::enforce_speed_limit() 62 0.004431 0.004431 0.004431 0.49% BndryData::define() 11 0.003837 0.003837 0.003837 0.42% amrex::Add() 82 0.003627 0.003627 0.003627 0.40% Castro::construct_new_gravity_source() 10 0.00297 0.00297 0.00297 0.33% Castro::construct_old_gravity_source() 10 0.002556 0.002556 0.002556 0.28% Amr::writePlotFile() 2 0.002144 0.002144 0.002144 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001813 0.001813 0.001813 0.20% MLCGSolver::bicgstab 82 0.001669 0.001669 0.001669 0.18% check_for_negative_density() 10 0.00166 0.00166 0.00166 0.18% Castro::reset_internal_energy(Fab) 504 0.001653 0.001653 0.001653 0.18% MLCellLinOp::setLevelBC() 11 0.001566 0.001566 0.001566 0.17% Gravity::actual_solve_with_mlmg() 11 0.001535 0.001535 0.001535 0.17% FabArray::mult() 43 0.001379 0.001379 0.001379 0.15% FabArray::setDomainBndry() 41 0.001345 0.001345 0.001345 0.15% Castro::initData() 1 0.001343 0.001343 0.001343 0.15% MLCellLinOp::prepareForSolve() 11 0.001302 0.001302 0.001302 0.14% MultiFab::contains_nan() 20 0.001263 0.001263 0.001263 0.14% MLCellLinOp::smooth() 1640 0.001201 0.001201 0.001201 0.13% MLCellLinOp::compGrad() 11 0.00105 0.00105 0.00105 0.12% MLMG::prepareForSolve() 11 0.0009711 0.0009711 0.0009711 0.11% FabArrayBase::getCPC() 1323 0.0007825 0.0007825 0.0007825 0.09% FabArray::FillBoundary() 3941 0.0007689 0.0007689 0.0007689 0.08% FabArrayBase::CPC::define() 454 0.0006523 0.0006523 0.0006523 0.07% FabArrayBase::getFB() 3941 0.0006163 0.0006163 0.0006163 0.07% Gravity::get_new_grav_vector() 11 0.0005959 0.0005959 0.0005959 0.07% Amr::InitAmr() 1 0.0005424 0.0005424 0.0005424 0.06% Gravity::get_old_grav_vector() 10 0.0004896 0.0004896 0.0004896 0.05% MLCellLinOp::apply() 1060 0.0004566 0.0004566 0.0004566 0.05% AmrLevel::FillPatch() 41 0.0004173 0.0004173 0.0004173 0.05% MultiFab::max() 11 0.0003398 0.0003398 0.0003398 0.04% Amr::coarseTimeStep() 10 0.000339 0.000339 0.000339 0.04% MLCGSolver::ParallelAllReduce 1832 0.0003104 0.0003104 0.0003104 0.03% main() 1 0.0002843 0.0002843 0.0002843 0.03% MLCellLinOp::defineBC() 11 0.0002604 0.0002604 0.0002604 0.03% FabArray::ParallelCopy() 861 0.0002562 0.0002562 0.0002562 0.03% FillPatchIterator::Initialize 41 0.0002171 0.0002171 0.0002171 0.02% MLMG::mgVcycle() 82 0.0002135 0.0002135 0.0002135 0.02% MLLinOp::defineGrids() 11 0.0001864 0.0001864 0.0001864 0.02% MLCellLinOp::correctionResidual() 410 0.0001725 0.0001725 0.0001725 0.02% Castro::subcycle_advance_ctu() 10 0.0001669 0.0001669 0.0001669 0.02% Amr::timeStep() 10 0.0001639 0.0001639 0.0001639 0.02% Castro::create_source_corrector() 10 0.0001635 0.0001635 0.0001635 0.02% Gravity::update_max_rhs() 11 0.0001425 0.0001425 0.0001425 0.02% StateData::checkPoint() 12 0.0001319 0.0001319 0.0001319 0.01% MLMG:computeResOfCorrection() 410 0.0001186 0.0001186 0.0001186 0.01% Gravity::solve_for_phi() 10 0.0001061 0.0001061 0.0001061 0.01% MLMG::actualBottomSolve() 82 9.149e-05 9.149e-05 9.149e-05 0.01% Castro::initialize_advance() 10 9.088e-05 9.088e-05 9.088e-05 0.01% Castro::Castro() 1 8.812e-05 8.812e-05 8.812e-05 0.01% Castro::post_timestep() 10 8.648e-05 8.648e-05 8.648e-05 0.01% FabArrayBase::FB::FB() 56 8.262e-05 8.262e-05 8.262e-05 0.01% MLMG::mgVcycle_down::0 82 8.116e-05 8.116e-05 8.116e-05 0.01% MLMG::mgVcycle_down::1 82 7.909e-05 7.909e-05 7.909e-05 0.01% MLMG::mgVcycle_down::2 82 7.784e-05 7.784e-05 7.784e-05 0.01% Castro::construct_new_source() 50 7.642e-05 7.642e-05 7.642e-05 0.01% MLMG::solve() 11 7.389e-05 7.389e-05 7.389e-05 0.01% MLMG::mgVcycle_down::4 82 7.378e-05 7.378e-05 7.378e-05 0.01% Castro::advance() 10 7.299e-05 7.299e-05 7.299e-05 0.01% MLMG::mgVcycle_down::3 82 7.264e-05 7.264e-05 7.264e-05 0.01% Castro::clean_state() 62 6.991e-05 6.991e-05 6.991e-05 0.01% Castro::finalize_advance() 10 6.941e-05 6.941e-05 6.941e-05 0.01% AmrLevel::checkPoint() 3 6.777e-05 6.777e-05 6.777e-05 0.01% Castro::enforce_consistent_e() 1 6.774e-05 6.774e-05 6.774e-05 0.01% Castro::initialize_do_advance() 10 6.03e-05 6.03e-05 6.03e-05 0.01% MLMG::oneIter() 82 5.552e-05 5.552e-05 5.552e-05 0.01% MLMG::mgVcycle_up::4 82 5.523e-05 5.523e-05 5.523e-05 0.01% MLMG::mgVcycle_up::3 82 4.925e-05 4.925e-05 4.925e-05 0.01% Castro::do_advance_ctu() 10 4.92e-05 4.92e-05 4.92e-05 0.01% MLMG::mgVcycle_up::1 82 4.795e-05 4.795e-05 4.795e-05 0.01% MLMG::mgVcycle_up::0 82 4.765e-05 4.765e-05 4.765e-05 0.01% MLMG::mgVcycle_up::2 82 4.715e-05 4.715e-05 4.715e-05 0.01% Castro::finalize_do_advance() 10 4.627e-05 4.627e-05 4.627e-05 0.01% MLCellLinOp::solutionResidual() 93 4.613e-05 4.613e-05 4.613e-05 0.01% FillPatchIterator::FillFromLevel0() 41 4.507e-05 4.507e-05 4.507e-05 0.00% Castro::swap_state_time_levels() 10 3.958e-05 3.958e-05 3.958e-05 0.00% FillPatchSingleLevel 41 3.726e-05 3.726e-05 3.726e-05 0.00% StateData::define() 4 3.454e-05 3.454e-05 3.454e-05 0.00% MLMG::mgVcycle_bottom 82 3.417e-05 3.417e-05 3.417e-05 0.00% Amr::defBaseLevel() 1 3.297e-05 3.297e-05 3.297e-05 0.00% MLMG::ResNormInf() 93 3.209e-05 3.209e-05 3.209e-05 0.00% MLMG::computeResidual() 82 3.018e-05 3.018e-05 3.018e-05 0.00% Amr::writeSmallPlotFile() 1 3.003e-05 3.003e-05 3.003e-05 0.00% Castro::initMFs() 1 2.898e-05 2.898e-05 2.898e-05 0.00% Castro::buildMetrics() 1 2.835e-05 2.835e-05 2.835e-05 0.00% makeSFC 55 2.824e-05 2.824e-05 2.824e-05 0.00% MLPoisson::define() 11 2.763e-05 2.763e-05 2.763e-05 0.00% Castro::construct_new_gravity() 10 2.453e-05 2.453e-05 2.453e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.43e-05 2.43e-05 2.43e-05 0.00% Castro::do_old_sources() 10 2.401e-05 2.401e-05 2.401e-05 0.00% Amr::FinalizeInit() 1 2.046e-05 2.046e-05 2.046e-05 0.00% Castro::construct_old_source() 50 2.027e-05 2.027e-05 2.027e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.831e-05 1.831e-05 1.831e-05 0.00% Castro::do_new_sources() 10 1.825e-05 1.825e-05 1.825e-05 0.00% MLLinOp::define() 11 1.813e-05 1.813e-05 1.813e-05 0.00% DistributionMapping::Distribute() 56 1.709e-05 1.709e-05 1.709e-05 0.00% Gravity::actual_multilevel_solve() 1 1.509e-05 1.509e-05 1.509e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.386e-05 1.386e-05 1.386e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.282e-05 1.282e-05 1.282e-05 0.00% Castro::check_for_nan() 20 1.191e-05 1.191e-05 1.191e-05 0.00% Castro::apply_source_to_state() 20 1.131e-05 1.131e-05 1.131e-05 0.00% Castro::construct_old_gravity() 10 1.074e-05 1.074e-05 1.074e-05 0.00% Castro::computeNewDt() 9 1.037e-05 1.037e-05 1.037e-05 0.00% Gravity::swapTimeLevels() 10 1.031e-05 1.031e-05 1.031e-05 0.00% Amr::initSubcycle() 1 1.008e-05 1.008e-05 1.008e-05 0.00% MLMG::computeMLResidual() 11 9.808e-06 9.808e-06 9.808e-06 0.00% AmrLevel::checkPointPost() 3 7.299e-06 7.299e-06 7.299e-06 0.00% MLMG::getGradSolution() 11 6.411e-06 6.411e-06 6.411e-06 0.00% MLPoisson::prepareForSolve() 11 6.264e-06 6.264e-06 6.264e-06 0.00% Amr::InitializeInit() 1 5.967e-06 5.967e-06 5.967e-06 0.00% Castro::expand_state() 10 5.948e-06 5.948e-06 5.948e-06 0.00% Castro::retry_advance_ctu() 10 4.655e-06 4.655e-06 4.655e-06 0.00% Gravity::set_mass_offset() 11 4.306e-06 4.306e-06 4.306e-06 0.00% Castro::post_init() 1 4.27e-06 4.27e-06 4.27e-06 0.00% MLMG::MLRhsNormInf() 11 3.967e-06 3.967e-06 3.967e-06 0.00% MLMG::MLResNormInf() 11 3.541e-06 3.541e-06 3.541e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.52e-06 3.52e-06 3.52e-06 0.00% Castro::FluxRegCrseInit 10 3.109e-06 3.109e-06 3.109e-06 0.00% Castro::computeInitialDt() 2 2.89e-06 2.89e-06 2.89e-06 0.00% Castro::FluxRegFineAdd() 10 2.589e-06 2.589e-06 2.589e-06 0.00% Amr::init() 1 2.542e-06 2.542e-06 2.542e-06 0.00% AmrLevel::checkPointPre() 3 2.121e-06 2.121e-06 2.121e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.033e-06 2.033e-06 2.033e-06 0.00% Castro::post_regrid() 1 1.242e-06 1.242e-06 1.242e-06 0.00% Amr::initialInit() 1 1.016e-06 1.016e-06 1.016e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9077 0.9077 0.9077 100.00% Amr::coarseTimeStep() 10 0.762 0.762 0.762 83.95% Amr::timeStep() 10 0.6654 0.6654 0.6654 73.30% Castro::advance() 10 0.6536 0.6536 0.6536 72.00% Castro::subcycle_advance_ctu() 10 0.6395 0.6395 0.6395 70.45% Castro::do_advance_ctu() 10 0.6393 0.6393 0.6393 70.43% Castro::construct_ctu_hydro_source() 10 0.2999 0.2999 0.2999 33.04% Gravity::solve_phi_with_mlmg() 11 0.293 0.293 0.293 32.28% Gravity::actual_solve_with_mlmg() 11 0.2833 0.2833 0.2833 31.21% Castro::construct_new_gravity() 10 0.2663 0.2663 0.2663 29.34% MLMG::solve() 11 0.2616 0.2616 0.2616 28.82% Gravity::solve_for_phi() 10 0.2489 0.2489 0.2489 27.42% MLMG::oneIter() 82 0.2462 0.2462 0.2462 27.12% MLMG::mgVcycle() 82 0.2425 0.2425 0.2425 26.72% VisMF::Write(FabArray) 11 0.1753 0.1753 0.1753 19.32% Amr::checkPoint() 3 0.137 0.137 0.137 15.09% AmrLevel::checkPoint() 3 0.13 0.13 0.13 14.32% StateData::checkPoint() 12 0.1299 0.1299 0.1299 14.31% MLCellLinOp::smooth() 1640 0.1218 0.1218 0.1218 13.42% Amr::init() 1 0.1208 0.1208 0.1208 13.30% MLCellLinOp::applyBC() 4351 0.1134 0.1134 0.1134 12.49% MLMG::mgVcycle_bottom 82 0.07188 0.07188 0.07188 7.92% MLMG::actualBottomSolve() 82 0.07185 0.07185 0.07185 7.92% MLCGSolver::bicgstab 82 0.07105 0.07105 0.07105 7.83% Castro::clean_state() 62 0.05504 0.05504 0.05504 6.06% Amr::initialInit() 1 0.05301 0.05301 0.05301 5.84% AmrLevel::FillPatch() 41 0.05204 0.05204 0.05204 5.73% Amr::FinalizeInit() 1 0.04831 0.04831 0.04831 5.32% Amr::writePlotFile() 2 0.04821 0.04821 0.04821 5.31% FillPatchIterator::Initialize 41 0.04773 0.04773 0.04773 5.26% Castro::post_init() 1 0.04683 0.04683 0.04683 5.16% FillPatchIterator::FillFromLevel0() 41 0.04617 0.04617 0.04617 5.09% FillPatchSingleLevel 41 0.04612 0.04612 0.04612 5.08% Gravity::multilevel_solve_for_new_phi() 1 0.04457 0.04457 0.04457 4.91% Gravity::actual_multilevel_solve() 1 0.04456 0.04456 0.04456 4.91% StateDataPhysBCFunct::() 41 0.04194 0.04194 0.04194 4.62% MLCellLinOp::apply() 1060 0.03615 0.03615 0.03615 3.98% MLMG::mgVcycle_down::0 82 0.03493 0.03493 0.03493 3.85% MLPoisson::Fsmooth() 3280 0.03328 0.03328 0.03328 3.67% FabArray::FillBoundary() 3941 0.03305 0.03305 0.03305 3.64% FillBoundary_nowait() 3941 0.03228 0.03228 0.03228 3.56% MLMG::mgVcycle_up::0 82 0.0266 0.0266 0.0266 2.93% StateData::FillBoundary(geom) 328 0.02487 0.02487 0.02487 2.74% Castro::initialize_do_advance() 10 0.02244 0.02244 0.02244 2.47% Castro::computeTemp() 63 0.02141 0.02141 0.02141 2.36% amrex::Dot() 1114 0.02111 0.02111 0.02111 2.33% Castro::do_old_sources() 10 0.02036 0.02036 0.02036 2.24% MLMG:computeResOfCorrection() 410 0.02034 0.02034 0.02034 2.24% MLCellLinOp::correctionResidual() 410 0.02022 0.02022 0.02022 2.23% FabArray::norminf() 1061 0.01962 0.01962 0.01962 2.16% Gravity::get_new_grav_vector() 11 0.01923 0.01923 0.01923 2.12% Castro::normalize_species() 62 0.01785 0.01785 0.01785 1.97% MLPoisson::define() 11 0.01705 0.01705 0.01705 1.88% MLMG::mgVcycle_down::1 82 0.01655 0.01655 0.01655 1.82% Castro::construct_old_gravity() 10 0.01636 0.01636 0.01636 1.80% Gravity::get_old_grav_vector() 10 0.01635 0.01635 0.01635 1.80% MLMG::mgVcycle_down::2 82 0.01541 0.01541 0.01541 1.70% MLMG::mgVcycle_down::3 82 0.01503 0.01503 0.01503 1.66% MLMG::mgVcycle_down::4 82 0.01488 0.01488 0.01488 1.64% FabArray::ParallelCopy() 861 0.01474 0.01474 0.01474 1.62% FabArray::ParallelCopy_nowait() 861 0.01448 0.01448 0.01448 1.60% Castro::initialize_advance() 10 0.01344 0.01344 0.01344 1.48% FabArray::setVal() 1062 0.01335 0.01335 0.01335 1.47% FabArray::Saxpy() 1370 0.01298 0.01298 0.01298 1.43% MLCGSolver::ParallelAllReduce 1832 0.01262 0.01262 0.01262 1.39% Castro::expand_state() 10 0.01259 0.01259 0.01259 1.39% MLMG::addInterpCorrection() 410 0.01223 0.01223 0.01223 1.35% Castro::enforce_min_density() 62 0.01216 0.01216 0.01216 1.34% MLMG::mgVcycle_up::1 82 0.01196 0.01196 0.01196 1.32% MLMG::mgVcycle_up::4 82 0.01189 0.01189 0.01189 1.31% MLMG::mgVcycle_up::2 82 0.01168 0.01168 0.01168 1.29% MLCellLinOp::defineAuxData() 11 0.01165 0.01165 0.01165 1.28% Castro::post_timestep() 10 0.0116 0.0116 0.0116 1.28% amrex::average_down 410 0.01149 0.01149 0.01149 1.27% MLMG::mgVcycle_up::3 82 0.01147 0.01147 0.01147 1.26% Castro::do_new_sources() 10 0.01108 0.01108 0.01108 1.22% amrex::Copy() 472 0.01092 0.01092 0.01092 1.20% MLPoisson::Fapply() 1060 0.0102 0.0102 0.0102 1.12% Gravity::fill_multipole_BCs() 11 0.0095 0.0095 0.0095 1.05% MLCellLinOp::solutionResidual() 93 0.007785 0.007785 0.007785 0.86% FabArray::Xpay() 739 0.007776 0.007776 0.007776 0.86% Castro::reset_internal_energy(MultiFab) 63 0.006613 0.006613 0.006613 0.73% MLMG::computeResidual() 82 0.006528 0.006528 0.006528 0.72% MLCellLinOp::defineBC() 11 0.005106 0.005106 0.005106 0.56% MLMG::prepareForSolve() 11 0.005105 0.005105 0.005105 0.56% Castro::estTimeStep() 21 0.004977 0.004977 0.004977 0.55% BndryData::define() 11 0.004845 0.004845 0.004845 0.53% Amr::InitializeInit() 1 0.004697 0.004697 0.004697 0.52% Amr::defBaseLevel() 1 0.004691 0.004691 0.004691 0.52% Castro::enforce_speed_limit() 62 0.004431 0.004431 0.004431 0.49% Castro::initData() 1 0.004008 0.004008 0.004008 0.44% amrex::Add() 82 0.003627 0.003627 0.003627 0.40% Castro::construct_new_source() 50 0.003047 0.003047 0.003047 0.34% Castro::construct_new_gravity_source() 10 0.00297 0.00297 0.00297 0.33% Castro::construct_old_source() 50 0.002576 0.002576 0.002576 0.28% Castro::construct_old_gravity_source() 10 0.002556 0.002556 0.002556 0.28% Castro::computeNewDt() 9 0.002262 0.002262 0.002262 0.25% MLMG::ResNormInf() 93 0.002165 0.002165 0.002165 0.24% Castro::apply_source_to_state() 20 0.001882 0.001882 0.001882 0.21% Castro::finalize_do_advance() 10 0.001866 0.001866 0.001866 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001813 0.001813 0.001813 0.20% check_for_negative_density() 10 0.00166 0.00166 0.00166 0.18% Castro::reset_internal_energy(Fab) 504 0.001653 0.001653 0.001653 0.18% MLMG::getGradSolution() 11 0.001568 0.001568 0.001568 0.17% MLCellLinOp::setLevelBC() 11 0.001566 0.001566 0.001566 0.17% MLCellLinOp::compGrad() 11 0.001562 0.001562 0.001562 0.17% FabArrayBase::getCPC() 1323 0.001435 0.001435 0.001435 0.16% FabArray::mult() 43 0.001379 0.001379 0.001379 0.15% FabArray::setDomainBndry() 41 0.001345 0.001345 0.001345 0.15% MLPoisson::prepareForSolve() 11 0.001308 0.001308 0.001308 0.14% MLCellLinOp::prepareForSolve() 11 0.001302 0.001302 0.001302 0.14% MLMG::computeMLResidual() 11 0.001297 0.001297 0.001297 0.14% Castro::check_for_nan() 20 0.001275 0.001275 0.001275 0.14% MultiFab::contains_nan() 20 0.001263 0.001263 0.001263 0.14% Castro::post_regrid() 1 0.001193 0.001193 0.001193 0.13% Gravity::update_max_rhs() 11 0.001037 0.001037 0.001037 0.11% Castro::computeInitialDt() 2 0.0009088 0.0009088 0.0009088 0.10% FabArrayBase::getFB() 3941 0.000699 0.000699 0.000699 0.08% FabArrayBase::CPC::define() 454 0.0006523 0.0006523 0.0006523 0.07% Castro::finalize_advance() 10 0.0006224 0.0006224 0.0006224 0.07% Castro::Castro() 1 0.0005979 0.0005979 0.0005979 0.07% Amr::InitAmr() 1 0.0005525 0.0005525 0.0005525 0.06% Gravity::swapTimeLevels() 10 0.0004755 0.0004755 0.0004755 0.05% MultiFab::max() 11 0.0003398 0.0003398 0.0003398 0.04% MLMG::MLResNormInf() 11 0.0003284 0.0003284 0.0003284 0.04% Castro::buildMetrics() 1 0.0003112 0.0003112 0.0003112 0.03% MLLinOp::define() 11 0.0002647 0.0002647 0.0002647 0.03% MLLinOp::defineGrids() 11 0.0002465 0.0002465 0.0002465 0.03% MLMG::MLRhsNormInf() 11 0.0002279 0.0002279 0.0002279 0.03% Castro::create_source_corrector() 10 0.0001635 0.0001635 0.0001635 0.02% FabArrayBase::FB::FB() 56 8.262e-05 8.262e-05 8.262e-05 0.01% Castro::enforce_consistent_e() 1 6.774e-05 6.774e-05 6.774e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.811e-05 5.811e-05 5.811e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.736e-05 4.736e-05 4.736e-05 0.01% makeSFC 55 4.424e-05 4.424e-05 4.424e-05 0.00% Castro::swap_state_time_levels() 10 3.958e-05 3.958e-05 3.958e-05 0.00% StateData::define() 4 3.454e-05 3.454e-05 3.454e-05 0.00% Amr::writeSmallPlotFile() 1 3.003e-05 3.003e-05 3.003e-05 0.00% Castro::initMFs() 1 2.898e-05 2.898e-05 2.898e-05 0.00% DistributionMapping::Distribute() 56 1.709e-05 1.709e-05 1.709e-05 0.00% Amr::initSubcycle() 1 1.008e-05 1.008e-05 1.008e-05 0.00% AmrLevel::checkPointPost() 3 7.299e-06 7.299e-06 7.299e-06 0.00% Castro::retry_advance_ctu() 10 4.655e-06 4.655e-06 4.655e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.61e-06 4.61e-06 4.61e-06 0.00% Gravity::set_mass_offset() 11 4.306e-06 4.306e-06 4.306e-06 0.00% Castro::FluxRegCrseInit 10 3.109e-06 3.109e-06 3.109e-06 0.00% Castro::FluxRegFineAdd() 10 2.589e-06 2.589e-06 2.589e-06 0.00% AmrLevel::checkPointPre() 3 2.121e-06 2.121e-06 2.121e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.033e-06 2.033e-06 2.033e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 5205 KiB 9037 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1113 KiB 39 MiB Castro::initialize_do_advance() 80 80 27 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1479 KiB 28 MiB Castro::initialize_advance() 80 80 16 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7615 KiB 14 MiB MLMG::prepareForSolve() 660 660 3544 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 219 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 184 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7519 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 17 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2196 B 2048 KiB Gravity::solve_for_phi() 80 80 560 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 100 KiB 2048 KiB BndryData::define() 1056 1056 324 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 206 KiB 671 KiB Castro::estTimeStep() 21 21 2663 B 480 KiB VisMF::Write(FabArray) 656 656 3363 B 320 KiB Castro::normalize_species() 62 62 6400 B 320 KiB amrex::average_down 1067 1067 1596 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1163 B 257 KiB amrex::Dot() 1360 1360 3430 B 160 KiB FabArray::norminf() 1143 1143 3335 B 160 KiB check_for_negative_density() 10 10 293 B 160 KiB Castro::initData() 1 1 51 B 160 KiB MultiFab::max() 11 11 58 B 160 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MultiFab::contains_nan() 20 20 27 B 20 KiB MLPoisson::Fsmooth() 132 132 3460 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 44 B 10 KiB FillBoundary_nowait() 760 760 303 B 9648 B MLCellLinOp::applyBC() 8702 8702 220 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3929 B 6144 B StateData::FillBoundary(geom) 1992 1992 62 B 4096 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 365 B 1248 B MLCGSolver::bicgstab 410 410 94 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 797 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 40 KiB 8192 KiB VisMF::Write(FabArray) 744 744 416 KiB 3584 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MLPoisson::Fsmooth() 132 132 3460 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 44 B 10 KiB FillBoundary_nowait() 760 760 303 B 9648 B MLCellLinOp::applyBC() 4351 4351 218 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3929 B 6144 B StateData::FillBoundary(geom) 1992 1992 62 B 4096 B Gravity::get_new_grav_vector() 3 3 2894 B 3072 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B amrex::average_down 83 83 617 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 295 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 25 B 400 B FabArray::norminf() 1143 1143 9 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2105 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.04-9-g2a3955a5f5aa) finalized Initializing AMReX (24.04-9-g2a3955a5f5aa)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.04-9-g2a3955a5f5aa) initialized Starting run at 09:30:05 UTC on 2024-04-09. Successfully read inputs file ... Castro git describe: 24.04-11-g2407e8176 AMReX git describe: 24.04-9-g2a3955a5f Microphysics git describe: 24.04-4-ge5fecb7a reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.510833495 Restart time = 0.071957912 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.068534663 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.04914459 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.072034096 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.073847292 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065559279 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.025837511 seconds Ending run at 09:30:06 UTC on 2024-04-09. Run time = 0.427980137 Run time without initialization = 0.355384971 Average number of zones advanced per microsecond: 3.688 Average number of zones advanced per microsecond per rank: 3.688 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.428 ... 0.428 ... 0.428 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1350 0.1350 0.1350 31.55% VisMF::Read() 3 0.06085 0.06085 0.06085 14.22% MLCellLinOp::applyBC() 1910 0.03535 0.03535 0.03535 8.26% VisMF::Write(FabArray) 1 0.02321 0.02321 0.02321 5.42% MLPoisson::Fsmooth() 1440 0.0146 0.0146 0.0146 3.41% StateData::FillBoundary(geom) 160 0.01314 0.01314 0.01314 3.07% FillBoundary_nowait() 1730 0.01303 0.01303 0.01303 3.04% Castro::normalize_species() 30 0.01041 0.01041 0.01041 2.43% amrex::Dot() 484 0.009135 0.009135 0.009135 2.13% FabArray::norminf() 465 0.008583 0.008583 0.008583 2.01% Castro::computeTemp() 30 0.007389 0.007389 0.007389 1.73% Castro::enforce_min_density() 30 0.006809 0.006809 0.006809 1.59% FabArray::setVal() 501 0.006588 0.006588 0.006588 1.54% FabArray::ParallelCopy_nowait() 380 0.006209 0.006209 0.006209 1.45% FabArray::Saxpy() 597 0.005778 0.005778 0.005778 1.35% MLCellLinOp::defineAuxData() 6 0.005681 0.005681 0.005681 1.33% amrex::Copy() 221 0.00543 0.00543 0.00543 1.27% Gravity::fill_multipole_BCs() 6 0.005262 0.005262 0.005262 1.23% StateDataPhysBCFunct::() 20 0.005142 0.005142 0.005142 1.20% Amr::restart() 1 0.004861 0.004861 0.004861 1.14% MLPoisson::Fapply() 464 0.004445 0.004445 0.004445 1.04% FabArray::Xpay() 325 0.003434 0.003434 0.003434 0.80% MLMG::addInterpCorrection() 180 0.003128 0.003128 0.003128 0.73% Castro::estTimeStep() 10 0.002979 0.002979 0.002979 0.70% amrex::average_down 180 0.002763 0.002763 0.002763 0.65% Amr::writePlotFile() 1 0.002476 0.002476 0.002476 0.58% BndryData::define() 6 0.002213 0.002213 0.002213 0.52% Castro::reset_internal_energy(MultiFab) 30 0.002058 0.002058 0.002058 0.48% Castro::construct_new_gravity_source() 5 0.001688 0.001688 0.001688 0.39% amrex::Add() 36 0.00154 0.00154 0.00154 0.36% Castro::construct_old_gravity_source() 5 0.001308 0.001308 0.001308 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001002 0.001002 0.001002 0.23% MLCellLinOp::setLevelBC() 6 0.0008898 0.0008898 0.0008898 0.21% check_for_negative_density() 5 0.0008637 0.0008637 0.0008637 0.20% Gravity::actual_solve_with_mlmg() 6 0.0008415 0.0008415 0.0008415 0.20% Castro::do_old_sources() 5 0.000815 0.000815 0.000815 0.19% Castro::reset_internal_energy(Fab) 240 0.0007993 0.0007993 0.0007993 0.19% MLCellLinOp::prepareForSolve() 6 0.0007726 0.0007726 0.0007726 0.18% MLCGSolver::bicgstab 36 0.0007548 0.0007548 0.0007548 0.18% FabArray::setDomainBndry() 20 0.0007306 0.0007306 0.0007306 0.17% FabArray::mult() 22 0.0006937 0.0006937 0.0006937 0.16% MultiFab::contains_nan() 10 0.0006725 0.0006725 0.0006725 0.16% MLCellLinOp::compGrad() 6 0.0005948 0.0005948 0.0005948 0.14% MLMG::prepareForSolve() 6 0.0005867 0.0005867 0.0005867 0.14% MLCellLinOp::smooth() 720 0.0005311 0.0005311 0.0005311 0.12% Amr::InitAmr() 1 0.0004623 0.0004623 0.0004623 0.11% FabArrayBase::CPC::define() 244 0.0004362 0.0004362 0.0004362 0.10% Castro::enforce_speed_limit() 30 0.0003955 0.0003955 0.0003955 0.09% FabArrayBase::getCPC() 632 0.0003865 0.0003865 0.0003865 0.09% Gravity::get_old_grav_vector() 5 0.0003717 0.0003717 0.0003717 0.09% Gravity::get_new_grav_vector() 5 0.0003409 0.0003409 0.0003409 0.08% FabArray::FillBoundary() 1730 0.0003399 0.0003399 0.0003399 0.08% FabArrayBase::getFB() 1730 0.0002631 0.0002631 0.0002631 0.06% main() 1 0.0002584 0.0002584 0.0002584 0.06% AmrLevel::FillPatch() 20 0.0002166 0.0002166 0.0002166 0.05% MLCellLinOp::apply() 464 0.000208 0.000208 0.000208 0.05% MultiFab::max() 6 0.0002029 0.0002029 0.0002029 0.05% Amr::coarseTimeStep() 5 0.0001566 0.0001566 0.0001566 0.04% MLCellLinOp::defineBC() 6 0.0001542 0.0001542 0.0001542 0.04% FillPatchIterator::Initialize 20 0.0001396 0.0001396 0.0001396 0.03% MLCGSolver::ParallelAllReduce 798 0.000135 0.000135 0.000135 0.03% FabArray::ParallelCopy() 380 0.0001167 0.0001167 0.0001167 0.03% Castro::create_source_corrector() 5 0.0001123 0.0001123 0.0001123 0.03% MLLinOp::defineGrids() 6 0.0001089 0.0001089 0.0001089 0.03% Castro::subcycle_advance_ctu() 5 9.494e-05 9.494e-05 9.494e-05 0.02% Amr::timeStep() 5 9.09e-05 9.09e-05 9.09e-05 0.02% MLMG::mgVcycle() 36 8.741e-05 8.741e-05 8.741e-05 0.02% Castro::finalize_do_advance() 5 8.253e-05 8.253e-05 8.253e-05 0.02% MLCellLinOp::correctionResidual() 180 7.424e-05 7.424e-05 7.424e-05 0.02% AmrLevel::restart() 1 7.08e-05 7.08e-05 7.08e-05 0.02% StateData::restartDoit() 4 7.025e-05 7.025e-05 7.025e-05 0.02% Castro::do_advance_ctu() 5 6.801e-05 6.801e-05 6.801e-05 0.02% Gravity::update_max_rhs() 6 6.602e-05 6.602e-05 6.602e-05 0.02% Castro::initialize_do_advance() 5 6.526e-05 6.526e-05 6.526e-05 0.02% FabArrayBase::FB::FB() 26 6.003e-05 6.003e-05 6.003e-05 0.01% Castro::post_timestep() 5 5.576e-05 5.576e-05 5.576e-05 0.01% Gravity::solve_for_phi() 5 5.323e-05 5.323e-05 5.323e-05 0.01% Castro::advance() 5 5.316e-05 5.316e-05 5.316e-05 0.01% Castro::construct_old_source() 25 5.011e-05 5.011e-05 5.011e-05 0.01% MLMG:computeResOfCorrection() 180 4.859e-05 4.859e-05 4.859e-05 0.01% Castro::construct_new_source() 25 4.539e-05 4.539e-05 4.539e-05 0.01% MLMG::actualBottomSolve() 36 3.975e-05 3.975e-05 3.975e-05 0.01% MLMG::mgVcycle_down::2 36 3.808e-05 3.808e-05 3.808e-05 0.01% Castro::initialize_advance() 5 3.715e-05 3.715e-05 3.715e-05 0.01% MLMG::solve() 6 3.672e-05 3.672e-05 3.672e-05 0.01% MLMG::mgVcycle_down::0 36 3.661e-05 3.661e-05 3.661e-05 0.01% MLMG::mgVcycle_down::1 36 3.638e-05 3.638e-05 3.638e-05 0.01% Castro::clean_state() 30 3.269e-05 3.269e-05 3.269e-05 0.01% Amr::writeSmallPlotFile() 1 3.235e-05 3.235e-05 3.235e-05 0.01% Castro::finalize_advance() 5 3.152e-05 3.152e-05 3.152e-05 0.01% MLMG::mgVcycle_down::3 36 3.099e-05 3.099e-05 3.099e-05 0.01% MLMG::mgVcycle_down::4 36 3.094e-05 3.094e-05 3.094e-05 0.01% MLMG::oneIter() 36 3.006e-05 3.006e-05 3.006e-05 0.01% Castro::do_new_sources() 5 2.909e-05 2.909e-05 2.909e-05 0.01% Castro::buildMetrics() 1 2.84e-05 2.84e-05 2.84e-05 0.01% Castro::swap_state_time_levels() 5 2.693e-05 2.693e-05 2.693e-05 0.01% MLMG::mgVcycle_up::4 36 2.634e-05 2.634e-05 2.634e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.42e-05 2.42e-05 2.42e-05 0.01% Castro::initMFs() 1 2.393e-05 2.393e-05 2.393e-05 0.01% FillPatchSingleLevel 20 2.373e-05 2.373e-05 2.373e-05 0.01% MLMG::mgVcycle_up::3 36 2.292e-05 2.292e-05 2.292e-05 0.01% Castro::post_restart() 1 2.192e-05 2.192e-05 2.192e-05 0.01% MLCellLinOp::solutionResidual() 42 2.156e-05 2.156e-05 2.156e-05 0.01% MLPoisson::define() 6 2.149e-05 2.149e-05 2.149e-05 0.01% MLMG::mgVcycle_up::0 36 2.081e-05 2.081e-05 2.081e-05 0.00% MLMG::mgVcycle_up::2 36 2.049e-05 2.049e-05 2.049e-05 0.00% MLMG::mgVcycle_up::1 36 2.017e-05 2.017e-05 2.017e-05 0.00% MLMG::computeResidual() 36 1.838e-05 1.838e-05 1.838e-05 0.00% MLMG::ResNormInf() 42 1.727e-05 1.727e-05 1.727e-05 0.00% Gravity::actual_multilevel_solve() 1 1.642e-05 1.642e-05 1.642e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.638e-05 1.638e-05 1.638e-05 0.00% Amr::initSubcycle() 1 1.577e-05 1.577e-05 1.577e-05 0.00% MLMG::mgVcycle_bottom 36 1.558e-05 1.558e-05 1.558e-05 0.00% makeSFC 30 1.551e-05 1.551e-05 1.551e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.412e-05 1.412e-05 1.412e-05 0.00% Castro::construct_new_gravity() 5 1.349e-05 1.349e-05 1.349e-05 0.00% MLLinOp::define() 6 1.195e-05 1.195e-05 1.195e-05 0.00% DistributionMapping::Distribute() 31 9.597e-06 9.597e-06 9.597e-06 0.00% Castro::expand_state() 5 8.921e-06 8.921e-06 8.921e-06 0.00% Castro::computeNewDt() 5 8.538e-06 8.538e-06 8.538e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.913e-06 7.913e-06 7.913e-06 0.00% Castro::check_for_nan() 10 6.767e-06 6.767e-06 6.767e-06 0.00% Castro::construct_old_gravity() 5 6.728e-06 6.728e-06 6.728e-06 0.00% Castro::apply_source_to_state() 10 5.852e-06 5.852e-06 5.852e-06 0.00% MLMG::computeMLResidual() 6 4.441e-06 4.441e-06 4.441e-06 0.00% Gravity::swapTimeLevels() 5 4.384e-06 4.384e-06 4.384e-06 0.00% MLPoisson::prepareForSolve() 6 3.687e-06 3.687e-06 3.687e-06 0.00% MLMG::getGradSolution() 6 3.392e-06 3.392e-06 3.392e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.051e-06 3.051e-06 3.051e-06 0.00% MLMG::MLResNormInf() 6 2.534e-06 2.534e-06 2.534e-06 0.00% Castro::retry_advance_ctu() 5 2.471e-06 2.471e-06 2.471e-06 0.00% MLMG::MLRhsNormInf() 6 2.398e-06 2.398e-06 2.398e-06 0.00% Gravity::set_mass_offset() 6 2.211e-06 2.211e-06 2.211e-06 0.00% Castro::FluxRegCrseInit 5 1.671e-06 1.671e-06 1.671e-06 0.00% Castro::FluxRegFineAdd() 5 1.276e-06 1.276e-06 1.276e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.131e-06 1.131e-06 1.131e-06 0.00% Amr::init() 1 9.03e-07 9.03e-07 9.03e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.428 0.428 0.428 99.99% Amr::coarseTimeStep() 5 0.3293 0.3293 0.3293 76.93% Amr::timeStep() 5 0.3272 0.3272 0.3272 76.44% Castro::advance() 5 0.3217 0.3217 0.3217 75.16% Castro::subcycle_advance_ctu() 5 0.3147 0.3147 0.3147 73.52% Castro::do_advance_ctu() 5 0.3146 0.3146 0.3146 73.50% Castro::construct_ctu_hydro_source() 5 0.14 0.14 0.14 32.71% Castro::construct_new_gravity() 5 0.1352 0.1352 0.1352 31.59% Gravity::solve_phi_with_mlmg() 6 0.1324 0.1324 0.1324 30.93% Gravity::solve_for_phi() 5 0.1273 0.1273 0.1273 29.73% Gravity::actual_solve_with_mlmg() 6 0.1269 0.1269 0.1269 29.65% MLMG::solve() 6 0.1146 0.1146 0.1146 26.78% MLMG::oneIter() 36 0.1069 0.1069 0.1069 24.97% MLMG::mgVcycle() 36 0.1053 0.1053 0.1053 24.60% Amr::init() 1 0.07201 0.07201 0.07201 16.82% Amr::restart() 1 0.07201 0.07201 0.07201 16.82% AmrLevel::restart() 1 0.06121 0.06121 0.06121 14.30% StateData::restartDoit() 4 0.06114 0.06114 0.06114 14.28% VisMF::Read() 3 0.06085 0.06085 0.06085 14.22% MLCellLinOp::smooth() 720 0.05259 0.05259 0.05259 12.29% MLCellLinOp::applyBC() 1910 0.04904 0.04904 0.04904 11.46% MLMG::mgVcycle_bottom 36 0.03128 0.03128 0.03128 7.31% MLMG::actualBottomSolve() 36 0.03126 0.03126 0.03126 7.30% MLCGSolver::bicgstab 36 0.0309 0.0309 0.0309 7.22% Castro::clean_state() 30 0.02789 0.02789 0.02789 6.52% Amr::writePlotFile() 1 0.02594 0.02594 0.02594 6.06% AmrLevel::FillPatch() 20 0.02342 0.02342 0.02342 5.47% VisMF::Write(FabArray) 1 0.02321 0.02321 0.02321 5.42% FillPatchIterator::Initialize 20 0.0213 0.0213 0.0213 4.98% FillPatchIterator::FillFromLevel0() 20 0.02043 0.02043 0.02043 4.77% FillPatchSingleLevel 20 0.0204 0.0204 0.0204 4.77% StateDataPhysBCFunct::() 20 0.01829 0.01829 0.01829 4.27% MLCellLinOp::apply() 464 0.01595 0.01595 0.01595 3.73% MLMG::mgVcycle_down::0 36 0.01502 0.01502 0.01502 3.51% MLPoisson::Fsmooth() 1440 0.0146 0.0146 0.0146 3.41% FabArray::FillBoundary() 1730 0.01369 0.01369 0.01369 3.20% FillBoundary_nowait() 1730 0.01335 0.01335 0.01335 3.12% StateData::FillBoundary(geom) 160 0.01314 0.01314 0.01314 3.07% Castro::initialize_do_advance() 5 0.01159 0.01159 0.01159 2.71% MLMG::mgVcycle_up::0 36 0.01129 0.01129 0.01129 2.64% Castro::do_old_sources() 5 0.01077 0.01077 0.01077 2.52% Castro::normalize_species() 30 0.01041 0.01041 0.01041 2.43% Castro::computeTemp() 30 0.01025 0.01025 0.01025 2.39% MLPoisson::define() 6 0.009639 0.009639 0.009639 2.25% amrex::Dot() 484 0.009135 0.009135 0.009135 2.13% MLMG:computeResOfCorrection() 180 0.008821 0.008821 0.008821 2.06% MLCellLinOp::correctionResidual() 180 0.008772 0.008772 0.008772 2.05% FabArray::norminf() 465 0.008583 0.008583 0.008583 2.01% Castro::construct_old_gravity() 5 0.007977 0.007977 0.007977 1.86% Gravity::get_old_grav_vector() 5 0.007971 0.007971 0.007971 1.86% Gravity::get_new_grav_vector() 5 0.007828 0.007828 0.007828 1.83% Castro::do_new_sources() 5 0.007363 0.007363 0.007363 1.72% MLMG::mgVcycle_down::1 36 0.007327 0.007327 0.007327 1.71% Castro::enforce_min_density() 30 0.006809 0.006809 0.006809 1.59% FabArray::ParallelCopy() 380 0.006736 0.006736 0.006736 1.57% MLMG::mgVcycle_down::2 36 0.006719 0.006719 0.006719 1.57% Castro::initialize_advance() 5 0.006683 0.006683 0.006683 1.56% FabArray::ParallelCopy_nowait() 380 0.006619 0.006619 0.006619 1.55% MLMG::mgVcycle_down::3 36 0.006594 0.006594 0.006594 1.54% FabArray::setVal() 501 0.006588 0.006588 0.006588 1.54% MLCellLinOp::defineAuxData() 6 0.006501 0.006501 0.006501 1.52% MLMG::mgVcycle_down::4 36 0.006476 0.006476 0.006476 1.51% Castro::expand_state() 5 0.006054 0.006054 0.006054 1.41% FabArray::Saxpy() 597 0.005778 0.005778 0.005778 1.35% Castro::post_restart() 1 0.005754 0.005754 0.005754 1.34% MLCGSolver::ParallelAllReduce 798 0.005514 0.005514 0.005514 1.29% amrex::Copy() 221 0.00543 0.00543 0.00543 1.27% MLMG::addInterpCorrection() 180 0.005421 0.005421 0.005421 1.27% Gravity::multilevel_solve_for_new_phi() 1 0.005392 0.005392 0.005392 1.26% Gravity::fill_multipole_BCs() 6 0.005386 0.005386 0.005386 1.26% Gravity::actual_multilevel_solve() 1 0.005376 0.005376 0.005376 1.26% Castro::post_timestep() 5 0.005366 0.005366 0.005366 1.25% MLMG::mgVcycle_up::4 36 0.005225 0.005225 0.005225 1.22% MLMG::mgVcycle_up::1 36 0.005189 0.005189 0.005189 1.21% amrex::average_down 180 0.005115 0.005115 0.005115 1.19% MLMG::mgVcycle_up::2 36 0.005087 0.005087 0.005087 1.19% MLMG::mgVcycle_up::3 36 0.004999 0.004999 0.004999 1.17% MLPoisson::Fapply() 464 0.004445 0.004445 0.004445 1.04% MLCellLinOp::solutionResidual() 42 0.00366 0.00366 0.00366 0.86% FabArray::Xpay() 325 0.003434 0.003434 0.003434 0.80% Castro::estTimeStep() 10 0.002979 0.002979 0.002979 0.70% MLCellLinOp::defineBC() 6 0.002962 0.002962 0.002962 0.69% MLMG::prepareForSolve() 6 0.002948 0.002948 0.002948 0.69% Castro::reset_internal_energy(MultiFab) 30 0.002857 0.002857 0.002857 0.67% MLMG::computeResidual() 36 0.002852 0.002852 0.002852 0.67% BndryData::define() 6 0.002808 0.002808 0.002808 0.66% Castro::computeNewDt() 5 0.001944 0.001944 0.001944 0.45% Castro::construct_new_source() 25 0.001734 0.001734 0.001734 0.41% Castro::construct_new_gravity_source() 5 0.001688 0.001688 0.001688 0.39% amrex::Add() 36 0.00154 0.00154 0.00154 0.36% Castro::construct_old_source() 25 0.001358 0.001358 0.001358 0.32% Castro::construct_old_gravity_source() 5 0.001308 0.001308 0.001308 0.31% Castro::finalize_do_advance() 5 0.001126 0.001126 0.001126 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001002 0.001002 0.001002 0.23% MLMG::ResNormInf() 42 0.0009839 0.0009839 0.0009839 0.23% Castro::apply_source_to_state() 10 0.0009607 0.0009607 0.0009607 0.22% MLCellLinOp::setLevelBC() 6 0.0008898 0.0008898 0.0008898 0.21% MLMG::getGradSolution() 6 0.0008889 0.0008889 0.0008889 0.21% MLCellLinOp::compGrad() 6 0.0008855 0.0008855 0.0008855 0.21% check_for_negative_density() 5 0.0008637 0.0008637 0.0008637 0.20% MLMG::computeMLResidual() 6 0.0008299 0.0008299 0.0008299 0.19% FabArrayBase::getCPC() 632 0.0008227 0.0008227 0.0008227 0.19% Castro::reset_internal_energy(Fab) 240 0.0007993 0.0007993 0.0007993 0.19% MLPoisson::prepareForSolve() 6 0.0007763 0.0007763 0.0007763 0.18% MLCellLinOp::prepareForSolve() 6 0.0007726 0.0007726 0.0007726 0.18% FabArray::setDomainBndry() 20 0.0007306 0.0007306 0.0007306 0.17% Gravity::update_max_rhs() 6 0.0007182 0.0007182 0.0007182 0.17% FabArray::mult() 22 0.0006937 0.0006937 0.0006937 0.16% Castro::check_for_nan() 10 0.0006793 0.0006793 0.0006793 0.16% MultiFab::contains_nan() 10 0.0006725 0.0006725 0.0006725 0.16% Amr::InitAmr() 1 0.0004781 0.0004781 0.0004781 0.11% FabArrayBase::CPC::define() 244 0.0004362 0.0004362 0.0004362 0.10% Castro::enforce_speed_limit() 30 0.0003955 0.0003955 0.0003955 0.09% FabArrayBase::getFB() 1730 0.0003232 0.0003232 0.0003232 0.08% Castro::finalize_advance() 5 0.0003076 0.0003076 0.0003076 0.07% Gravity::swapTimeLevels() 5 0.0002486 0.0002486 0.0002486 0.06% MultiFab::max() 6 0.0002029 0.0002029 0.0002029 0.05% MLMG::MLResNormInf() 6 0.000193 0.000193 0.000193 0.05% Castro::buildMetrics() 1 0.0001624 0.0001624 0.0001624 0.04% MLLinOp::define() 6 0.0001538 0.0001538 0.0001538 0.04% MLLinOp::defineGrids() 6 0.0001419 0.0001419 0.0001419 0.03% MLMG::MLRhsNormInf() 6 0.000123 0.000123 0.000123 0.03% Castro::create_source_corrector() 5 0.0001123 0.0001123 0.0001123 0.03% FabArrayBase::FB::FB() 26 6.003e-05 6.003e-05 6.003e-05 0.01% Amr::writeSmallPlotFile() 1 3.235e-05 3.235e-05 3.235e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.181e-05 3.181e-05 3.181e-05 0.01% Castro::swap_state_time_levels() 5 2.693e-05 2.693e-05 2.693e-05 0.01% Castro::initMFs() 1 2.393e-05 2.393e-05 2.393e-05 0.01% makeSFC 30 2.389e-05 2.389e-05 2.389e-05 0.01% Amr::initSubcycle() 1 1.577e-05 1.577e-05 1.577e-05 0.00% DistributionMapping::Distribute() 31 9.597e-06 9.597e-06 9.597e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.267e-06 4.267e-06 4.267e-06 0.00% Castro::retry_advance_ctu() 5 2.471e-06 2.471e-06 2.471e-06 0.00% Gravity::set_mass_offset() 6 2.211e-06 2.211e-06 2.211e-06 0.00% Castro::FluxRegCrseInit 5 1.671e-06 1.671e-06 1.671e-06 0.00% Castro::FluxRegFineAdd() 5 1.276e-06 1.276e-06 1.276e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.131e-06 1.131e-06 1.131e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 10 MiB 9037 MiB Castro::initMFs() 48 48 57 MiB 68 MiB Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB StateData::restartDoit() 32 32 52 MiB 55 MiB FillPatchIterator::Initialize 160 160 1089 KiB 39 MiB Castro::initialize_do_advance() 40 40 28 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1606 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6395 KiB 14 MiB MLMG::prepareForSolve() 361 361 3286 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 193 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 187 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6382 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 20 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3333 B 2048 KiB Gravity::solve_for_phi() 40 40 607 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 25 KiB 2048 KiB BndryData::define() 576 576 303 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 195 KiB 671 KiB Castro::estTimeStep() 10 10 3240 B 480 KiB VisMF::Write(FabArray) 112 112 1275 B 320 KiB Castro::normalize_species() 30 30 7917 B 320 KiB amrex::average_down 469 469 1397 B 257 KiB MLMG::addInterpCorrection() 468 468 1072 B 257 KiB amrex::Dot() 592 592 3141 B 160 KiB FabArray::norminf() 501 501 3092 B 160 KiB check_for_negative_density() 5 5 323 B 160 KiB MultiFab::max() 6 6 74 B 160 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MultiFab::contains_nan() 10 10 31 B 20 KiB MLPoisson::Fsmooth() 60 60 3181 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 46 B 10 KiB FillBoundary_nowait() 336 336 261 B 9648 B MLCellLinOp::applyBC() 3820 3820 205 B 9344 B amrex::Copy() 56 56 5836 B 8816 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B StateData::FillBoundary(geom) 960 960 43 B 2976 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLCellLinOp::defineBC() 36 36 340 B 1248 B MLCGSolver::bicgstab 180 180 87 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1697 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 84 KiB 8192 KiB VisMF::Write(FabArray) 120 120 150 KiB 3584 KiB VisMF::Read() 24 24 216 KiB 3000 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MLPoisson::Fsmooth() 60 60 3181 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 46 B 10 KiB FillBoundary_nowait() 336 336 261 B 9648 B MLCellLinOp::applyBC() 1910 1910 203 B 9328 B amrex::Copy() 56 56 5836 B 8816 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B Gravity::get_old_grav_vector() 3 3 2510 B 3072 B StateData::FillBoundary(geom) 960 960 43 B 2976 B Gravity::fill_multipole_BCs() 18 18 5 B 2832 B MLMG::prepareForSolve() 7 7 793 B 1648 B amrex::average_down 37 37 459 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 23 B 400 B FabArray::norminf() 501 501 9 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2105 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.04-9-g2a3955a5f5aa) finalized