Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-16-g826cd378f8ba) initialized Starting run at 08:30:33 UTC on 2022-09-16. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09-16-g826cd378f Microphysics git describe: 22.08-13-g1adf1bdb reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.051269615 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.029418408 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.047931691 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052147141 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.050416956 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.064099982 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.076909463 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.047105236 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.063117248 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.064251957 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.05804624 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.05985968 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.06179128 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.047231642 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029204217 seconds Ending run at 08:30:34 UTC on 2022-09-16. Run time = 0.855357644 Run time without initialization = 0.722761576 Average number of zones advanced per microsecond: 3.627 Average number of zones advanced per microsecond per rank: 3.627 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8554 ... 0.8554 ... 0.8554 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2071 0.2071 0.2071 24.21% VisMF::Write(FabArray) 11 0.197 0.197 0.197 23.03% MLCellLinOp::applyBC() 4433 0.0789 0.0789 0.0789 9.22% MLPoisson::Fsmooth() 3280 0.06367 0.06367 0.06367 7.44% StateDataPhysBCFunct::() 41 0.02374 0.02374 0.02374 2.78% MLCGSolver::bicgstab 82 0.02346 0.02346 0.02346 2.74% StateData::FillBoundary(geom) 328 0.02297 0.02297 0.02297 2.69% MultiFab::Dot() 1114 0.02195 0.02195 0.02195 2.57% Castro::normalize_species() 62 0.01516 0.01516 0.01516 1.77% FillBoundary_nowait() 4023 0.01413 0.01413 0.01413 1.65% MultiFab::LinComb() 1586 0.01413 0.01413 0.01413 1.65% FabArray::setVal() 1144 0.01399 0.01399 0.01399 1.63% Castro::computeTemp() 63 0.01377 0.01377 0.01377 1.61% FabArray::ParallelCopy_nowait() 861 0.01289 0.01289 0.01289 1.51% MLPoisson::Fapply() 1142 0.01153 0.01153 0.01153 1.35% MLCellLinOp::defineAuxData() 11 0.01141 0.01141 0.01141 1.33% Gravity::fill_multipole_BCs() 11 0.008431 0.008431 0.008431 0.99% Castro::enforce_min_density() 62 0.008052 0.008052 0.008052 0.94% MLMG::addInterpCorrection() 410 0.007735 0.007735 0.007735 0.90% amrex::average_down 410 0.006791 0.006791 0.006791 0.79% MultiFab::Xpay() 585 0.006489 0.006489 0.006489 0.76% Castro::estTimeStep() 21 0.005904 0.005904 0.005904 0.69% Amr::checkPoint() 3 0.004752 0.004752 0.004752 0.56% Castro::reset_internal_energy(MultiFab) 63 0.004727 0.004727 0.004727 0.55% Castro::do_advance_ctu() 10 0.004662 0.004662 0.004662 0.54% BndryData::define() 11 0.003747 0.003747 0.003747 0.44% Castro::construct_new_gravity_source() 10 0.003309 0.003309 0.003309 0.39% Castro::construct_old_gravity_source() 10 0.002726 0.002726 0.002726 0.32% Amr::writePlotFile() 2 0.002526 0.002526 0.002526 0.30% Castro::reset_internal_energy(Fab) 504 0.002305 0.002305 0.002305 0.27% MLMG::ResNormInf() 93 0.002073 0.002073 0.002073 0.24% Gravity::get_new_grav_vector() 11 0.001911 0.001911 0.001911 0.22% MultiFab::Saxpy() 20 0.001803 0.001803 0.001803 0.21% Castro::expand_state() 10 0.001731 0.001731 0.001731 0.20% Gravity::get_old_grav_vector() 10 0.001724 0.001724 0.001724 0.20% MultiFab::Add() 82 0.001659 0.001659 0.001659 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001617 0.001617 0.001617 0.19% MLCellLinOp::setLevelBC() 11 0.001519 0.001519 0.001519 0.18% Gravity::actual_solve_with_mlmg() 11 0.001411 0.001411 0.001411 0.16% FabArray::mult() 43 0.001324 0.001324 0.001324 0.15% FabArray::setDomainBndry() 41 0.001303 0.001303 0.001303 0.15% Castro::initData() 1 0.001279 0.001279 0.001279 0.15% Castro::enforce_speed_limit() 62 0.001242 0.001242 0.001242 0.15% MLMG::prepareForSolve() 11 0.001215 0.001215 0.001215 0.14% MultiFab::contains_nan() 20 0.001176 0.001176 0.001176 0.14% MLCellLinOp::prepareForSolve() 11 0.001145 0.001145 0.001145 0.13% MLCellLinOp::smooth() 1640 0.001017 0.001017 0.001017 0.12% MLCellLinOp::compGrad() 11 0.0009228 0.0009228 0.0009228 0.11% FabArray::FillBoundary() 4023 0.0008511 0.0008511 0.0008511 0.10% FabArrayBase::getCPC() 1323 0.0007516 0.0007516 0.0007516 0.09% FabArrayBase::CPC::define() 454 0.0006538 0.0006538 0.0006538 0.08% FabArrayBase::getFB() 4023 0.0005995 0.0005995 0.0005995 0.07% Amr::InitAmr() 1 0.0004916 0.0004916 0.0004916 0.06% MLCellLinOp::apply() 1142 0.000451 0.000451 0.000451 0.05% Gravity::solve_for_phi() 10 0.0004499 0.0004499 0.0004499 0.05% Gravity::update_max_rhs() 11 0.0004098 0.0004098 0.0004098 0.05% Amr::coarseTimeStep() 10 0.0003682 0.0003682 0.0003682 0.04% CGSolver::sxay() 1586 0.0003445 0.0003445 0.0003445 0.04% MultiFab::Copy() 11 0.0003224 0.0003224 0.0003224 0.04% FillPatchIterator::Initialize 41 0.0002988 0.0002988 0.0002988 0.03% MLCGSolver::ParallelAllReduce 1514 0.0002957 0.0002957 0.0002957 0.03% MLCellLinOp::defineBC() 11 0.0002826 0.0002826 0.0002826 0.03% FabArray::ParallelCopy() 861 0.0002608 0.0002608 0.0002608 0.03% main() 1 0.0002606 0.0002606 0.0002606 0.03% MultiFab::max() 11 0.000256 0.000256 0.000256 0.03% MLCellLinOp::correctionResidual() 492 0.0002157 0.0002157 0.0002157 0.03% MLMG::MLRhsNormInf() 11 0.0002146 0.0002146 0.0002146 0.03% Castro::construct_new_gravity() 10 0.0002065 0.0002065 0.0002065 0.02% MLMG::mgVcycle() 82 0.0001989 0.0001989 0.0001989 0.02% Castro::subcycle_advance_ctu() 10 0.0001741 0.0001741 0.0001741 0.02% Amr::timeStep() 10 0.0001542 0.0001542 0.0001542 0.02% MLLinOp::defineGrids() 11 0.0001481 0.0001481 0.0001481 0.02% MLMG:computeResOfCorrection() 410 0.0001457 0.0001457 0.0001457 0.02% StateData::checkPoint() 12 0.0001351 0.0001351 0.0001351 0.02% MLMG::mgVcycle_down::0 82 0.00011 0.00011 0.00011 0.01% Castro::advance() 10 9.939e-05 9.939e-05 9.939e-05 0.01% Castro::initialize_advance() 10 9.285e-05 9.285e-05 9.285e-05 0.01% MLMG::mgVcycle_down::1 82 9.156e-05 9.156e-05 9.156e-05 0.01% Castro::Castro() 1 9.136e-05 9.136e-05 9.136e-05 0.01% MLMG::mgVcycle_down::2 82 8.626e-05 8.626e-05 8.626e-05 0.01% FabArrayBase::FB::FB() 56 8.357e-05 8.357e-05 8.357e-05 0.01% Castro::finalize_advance() 10 8.276e-05 8.276e-05 8.276e-05 0.01% MLMG::mgVcycle_down::3 82 8.015e-05 8.015e-05 8.015e-05 0.01% MLMG::mgVcycle_down::4 82 7.718e-05 7.718e-05 7.718e-05 0.01% Castro::clean_state() 62 7.691e-05 7.691e-05 7.691e-05 0.01% MLMG::actualBottomSolve() 82 7.613e-05 7.613e-05 7.613e-05 0.01% MLMG::solve() 11 7.344e-05 7.344e-05 7.344e-05 0.01% AmrLevel::checkPoint() 3 7.285e-05 7.285e-05 7.285e-05 0.01% MLMG::mgVcycle_up::4 82 6.872e-05 6.872e-05 6.872e-05 0.01% Castro::initialize_do_advance() 10 6.162e-05 6.162e-05 6.162e-05 0.01% MLMG::mgVcycle_up::0 82 5.831e-05 5.831e-05 5.831e-05 0.01% MLMG::oneIter() 82 5.548e-05 5.548e-05 5.548e-05 0.01% MLMG::mgVcycle_up::3 82 5.43e-05 5.43e-05 5.43e-05 0.01% MLMG::mgVcycle_up::1 82 5.377e-05 5.377e-05 5.377e-05 0.01% MLMG::mgVcycle_up::2 82 5.202e-05 5.202e-05 5.202e-05 0.01% MLCellLinOp::solutionResidual() 93 5.028e-05 5.028e-05 5.028e-05 0.01% StateData::define() 4 4.496e-05 4.496e-05 4.496e-05 0.01% Castro::swap_state_time_levels() 10 4.075e-05 4.075e-05 4.075e-05 0.00% MLMG::computeResidual() 82 3.827e-05 3.827e-05 3.827e-05 0.00% Castro::post_timestep() 10 3.377e-05 3.377e-05 3.377e-05 0.00% Castro::enforce_consistent_e() 1 3.375e-05 3.375e-05 3.375e-05 0.00% Castro::finalize_do_advance() 10 3.235e-05 3.235e-05 3.235e-05 0.00% MLMG::mgVcycle_bottom 82 3.11e-05 3.11e-05 3.11e-05 0.00% Gravity::actual_multilevel_solve() 1 3.041e-05 3.041e-05 3.041e-05 0.00% MLPoisson::define() 11 3.035e-05 3.035e-05 3.035e-05 0.00% Castro::initMFs() 1 2.816e-05 2.816e-05 2.816e-05 0.00% Amr::writeSmallPlotFile() 1 2.672e-05 2.672e-05 2.672e-05 0.00% FillPatchSingleLevel 41 2.671e-05 2.671e-05 2.671e-05 0.00% makeSFC 55 2.646e-05 2.646e-05 2.646e-05 0.00% Castro::create_source_corrector() 10 2.413e-05 2.413e-05 2.413e-05 0.00% Castro::buildMetrics() 1 2.393e-05 2.393e-05 2.393e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.388e-05 2.388e-05 2.388e-05 0.00% Amr::defBaseLevel() 1 2.311e-05 2.311e-05 2.311e-05 0.00% MLLinOp::define() 11 2.196e-05 2.196e-05 2.196e-05 0.00% Amr::FinalizeInit() 1 1.937e-05 1.937e-05 1.937e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.817e-05 1.817e-05 1.817e-05 0.00% Castro::construct_new_source() 50 1.696e-05 1.696e-05 1.696e-05 0.00% Castro::construct_old_source() 50 1.662e-05 1.662e-05 1.662e-05 0.00% Castro::do_new_sources() 10 1.649e-05 1.649e-05 1.649e-05 0.00% Castro::do_old_sources() 10 1.599e-05 1.599e-05 1.599e-05 0.00% DistributionMapping::Distribute() 56 1.468e-05 1.468e-05 1.468e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.448e-05 1.448e-05 1.448e-05 0.00% Castro::check_for_nan() 20 1.305e-05 1.305e-05 1.305e-05 0.00% Castro::apply_source_to_state() 20 1.094e-05 1.094e-05 1.094e-05 0.00% MLMG::computeMLResidual() 11 9.972e-06 9.972e-06 9.972e-06 0.00% Castro::construct_old_gravity() 10 9.913e-06 9.913e-06 9.913e-06 0.00% Amr::initSubcycle() 1 9.016e-06 9.016e-06 9.016e-06 0.00% MLPoisson::prepareForSolve() 11 8.662e-06 8.662e-06 8.662e-06 0.00% Gravity::swapTimeLevels() 10 8.581e-06 8.581e-06 8.581e-06 0.00% MLMG::getGradSolution() 11 6.507e-06 6.507e-06 6.507e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.423e-06 6.423e-06 6.423e-06 0.00% Castro::computeNewDt() 9 5.865e-06 5.865e-06 5.865e-06 0.00% AmrLevel::checkPointPost() 3 5.616e-06 5.616e-06 5.616e-06 0.00% Amr::InitializeInit() 1 5.028e-06 5.028e-06 5.028e-06 0.00% Gravity::set_mass_offset() 11 4.556e-06 4.556e-06 4.556e-06 0.00% Castro::post_init() 1 3.766e-06 3.766e-06 3.766e-06 0.00% Castro::retry_advance_ctu() 10 3.639e-06 3.639e-06 3.639e-06 0.00% MLMG::MLResNormInf() 11 3.422e-06 3.422e-06 3.422e-06 0.00% Castro::computeInitialDt() 2 3.198e-06 3.198e-06 3.198e-06 0.00% Castro::FluxRegCrseInit 10 2.9e-06 2.9e-06 2.9e-06 0.00% Amr::init() 1 2.782e-06 2.782e-06 2.782e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.717e-06 2.717e-06 2.717e-06 0.00% Castro::FluxRegFineAdd() 10 1.909e-06 1.909e-06 1.909e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.884e-06 1.884e-06 1.884e-06 0.00% AmrLevel::checkPointPre() 3 1.884e-06 1.884e-06 1.884e-06 0.00% Castro::post_regrid() 1 1.341e-06 1.341e-06 1.341e-06 0.00% Amr::initialInit() 1 1.053e-06 1.053e-06 1.053e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8554 0.8554 0.8554 100.00% Amr::coarseTimeStep() 10 0.6933 0.6933 0.6933 81.06% Amr::timeStep() 10 0.5955 0.5955 0.5955 69.62% Castro::advance() 10 0.5878 0.5878 0.5878 68.71% Castro::subcycle_advance_ctu() 10 0.5765 0.5765 0.5765 67.40% Castro::do_advance_ctu() 10 0.5763 0.5763 0.5763 67.38% Gravity::solve_phi_with_mlmg() 11 0.3104 0.3104 0.3104 36.29% Gravity::actual_solve_with_mlmg() 11 0.3017 0.3017 0.3017 35.28% Castro::construct_new_gravity() 10 0.2822 0.2822 0.2822 32.99% MLMG::solve() 11 0.2795 0.2795 0.2795 32.67% Gravity::solve_for_phi() 10 0.2671 0.2671 0.2671 31.23% MLMG::oneIter() 82 0.2647 0.2647 0.2647 30.94% MLMG::mgVcycle() 82 0.263 0.263 0.263 30.74% Castro::construct_ctu_hydro_source() 10 0.2071 0.2071 0.2071 24.21% VisMF::Write(FabArray) 11 0.197 0.197 0.197 23.03% Amr::checkPoint() 3 0.1457 0.1457 0.1457 17.04% AmrLevel::checkPoint() 3 0.141 0.141 0.141 16.48% StateData::checkPoint() 12 0.1409 0.1409 0.1409 16.47% MLCellLinOp::smooth() 1640 0.135 0.135 0.135 15.78% Amr::init() 1 0.132 0.132 0.132 15.43% MLCellLinOp::applyBC() 4433 0.09457 0.09457 0.09457 11.06% MLMG::mgVcycle_bottom 82 0.08043 0.08043 0.08043 9.40% MLMG::actualBottomSolve() 82 0.0804 0.0804 0.0804 9.40% MLCGSolver::bicgstab 82 0.07962 0.07962 0.07962 9.31% MLPoisson::Fsmooth() 3280 0.06367 0.06367 0.06367 7.44% Amr::writePlotFile() 2 0.05874 0.05874 0.05874 6.87% FillPatchIterator::Initialize 41 0.05231 0.05231 0.05231 6.11% Amr::initialInit() 1 0.05118 0.05118 0.05118 5.98% FillPatchSingleLevel 41 0.05071 0.05071 0.05071 5.93% Amr::FinalizeInit() 1 0.04698 0.04698 0.04698 5.49% StateDataPhysBCFunct::() 41 0.04671 0.04671 0.04671 5.46% Castro::post_init() 1 0.04563 0.04563 0.04563 5.33% Castro::clean_state() 62 0.04454 0.04454 0.04454 5.21% Gravity::multilevel_solve_for_new_phi() 1 0.04376 0.04376 0.04376 5.12% Gravity::actual_multilevel_solve() 1 0.04374 0.04374 0.04374 5.11% MLCellLinOp::apply() 1142 0.03572 0.03572 0.03572 4.18% MLMG::mgVcycle_down::0 82 0.03516 0.03516 0.03516 4.11% MLMG::mgVcycle_up::0 82 0.03027 0.03027 0.03027 3.54% Castro::initialize_do_advance() 10 0.02937 0.02937 0.02937 3.43% StateData::FillBoundary(geom) 328 0.02297 0.02297 0.02297 2.69% Castro::expand_state() 10 0.02245 0.02245 0.02245 2.62% MultiFab::Dot() 1114 0.02195 0.02195 0.02195 2.57% MLCellLinOp::correctionResidual() 492 0.02096 0.02096 0.02096 2.45% Castro::computeTemp() 63 0.0208 0.0208 0.0208 2.43% MLMG:computeResOfCorrection() 410 0.0181 0.0181 0.0181 2.12% MLPoisson::define() 11 0.01792 0.01792 0.01792 2.09% MLMG::mgVcycle_down::1 82 0.01749 0.01749 0.01749 2.04% MLMG::mgVcycle_down::2 82 0.01701 0.01701 0.01701 1.99% Gravity::get_new_grav_vector() 11 0.01664 0.01664 0.01664 1.95% MLMG::mgVcycle_down::3 82 0.01614 0.01614 0.01614 1.89% FabArray::FillBoundary() 4023 0.01567 0.01567 0.01567 1.83% MLMG::mgVcycle_down::4 82 0.01536 0.01536 0.01536 1.80% Castro::normalize_species() 62 0.01516 0.01516 0.01516 1.77% FillBoundary_nowait() 4023 0.01482 0.01482 0.01482 1.73% Castro::construct_old_gravity() 10 0.01455 0.01455 0.01455 1.70% Gravity::get_old_grav_vector() 10 0.01454 0.01454 0.01454 1.70% CGSolver::sxay() 1586 0.01447 0.01447 0.01447 1.69% MultiFab::LinComb() 1586 0.01413 0.01413 0.01413 1.65% FabArray::setVal() 1144 0.01399 0.01399 0.01399 1.63% FabArray::ParallelCopy() 861 0.01394 0.01394 0.01394 1.63% FabArray::ParallelCopy_nowait() 861 0.01368 0.01368 0.01368 1.60% MLMG::mgVcycle_up::2 82 0.01316 0.01316 0.01316 1.54% MLCGSolver::ParallelAllReduce 1514 0.01312 0.01312 0.01312 1.53% MLMG::mgVcycle_up::1 82 0.0129 0.0129 0.0129 1.51% MLMG::addInterpCorrection() 410 0.01275 0.01275 0.01275 1.49% MLCellLinOp::defineAuxData() 11 0.01271 0.01271 0.01271 1.49% MLMG::mgVcycle_up::3 82 0.01244 0.01244 0.01244 1.45% MLMG::mgVcycle_up::4 82 0.01242 0.01242 0.01242 1.45% amrex::average_down 410 0.01176 0.01176 0.01176 1.37% Castro::do_old_sources() 10 0.01173 0.01173 0.01173 1.37% MLPoisson::Fapply() 1142 0.01153 0.01153 0.01153 1.35% Castro::do_new_sources() 10 0.01146 0.01146 0.01146 1.34% Castro::initialize_advance() 10 0.01107 0.01107 0.01107 1.29% Gravity::fill_multipole_BCs() 11 0.008431 0.008431 0.008431 0.99% Castro::enforce_min_density() 62 0.008052 0.008052 0.008052 0.94% Castro::post_timestep() 10 0.007587 0.007587 0.007587 0.89% MLCellLinOp::solutionResidual() 93 0.007061 0.007061 0.007061 0.83% Castro::reset_internal_energy(MultiFab) 63 0.007031 0.007031 0.007031 0.82% MultiFab::Xpay() 585 0.006489 0.006489 0.006489 0.76% MLMG::computeResidual() 82 0.006099 0.006099 0.006099 0.71% Castro::estTimeStep() 21 0.005904 0.005904 0.005904 0.69% MLMG::prepareForSolve() 11 0.005306 0.005306 0.005306 0.62% MLCellLinOp::defineBC() 11 0.00495 0.00495 0.00495 0.58% BndryData::define() 11 0.004667 0.004667 0.004667 0.55% Amr::InitializeInit() 1 0.004204 0.004204 0.004204 0.49% Amr::defBaseLevel() 1 0.004199 0.004199 0.004199 0.49% Castro::initData() 1 0.003675 0.003675 0.003675 0.43% Castro::construct_new_source() 50 0.003326 0.003326 0.003326 0.39% Castro::construct_new_gravity_source() 10 0.003309 0.003309 0.003309 0.39% Castro::construct_old_source() 50 0.002743 0.002743 0.002743 0.32% Castro::construct_old_gravity_source() 10 0.002726 0.002726 0.002726 0.32% Castro::computeNewDt() 9 0.002539 0.002539 0.002539 0.30% Castro::reset_internal_energy(Fab) 504 0.002305 0.002305 0.002305 0.27% MLMG::ResNormInf() 93 0.002073 0.002073 0.002073 0.24% Castro::apply_source_to_state() 20 0.001814 0.001814 0.001814 0.21% MultiFab::Saxpy() 20 0.001803 0.001803 0.001803 0.21% MultiFab::Add() 82 0.001659 0.001659 0.001659 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001617 0.001617 0.001617 0.19% MLCellLinOp::setLevelBC() 11 0.001519 0.001519 0.001519 0.18% MLMG::getGradSolution() 11 0.001425 0.001425 0.001425 0.17% MLCellLinOp::compGrad() 11 0.001418 0.001418 0.001418 0.17% FabArrayBase::getCPC() 1323 0.001405 0.001405 0.001405 0.16% FabArray::mult() 43 0.001324 0.001324 0.001324 0.15% FabArray::setDomainBndry() 41 0.001303 0.001303 0.001303 0.15% Castro::enforce_speed_limit() 62 0.001242 0.001242 0.001242 0.15% Castro::check_for_nan() 20 0.001189 0.001189 0.001189 0.14% MultiFab::contains_nan() 20 0.001176 0.001176 0.001176 0.14% MLPoisson::prepareForSolve() 11 0.001154 0.001154 0.001154 0.13% MLCellLinOp::prepareForSolve() 11 0.001145 0.001145 0.001145 0.13% Castro::post_regrid() 1 0.001087 0.001087 0.001087 0.13% MLMG::computeMLResidual() 11 0.00101 0.00101 0.00101 0.12% Gravity::update_max_rhs() 11 0.0008133 0.0008133 0.0008133 0.10% Castro::computeInitialDt() 2 0.0007445 0.0007445 0.0007445 0.09% FabArrayBase::getFB() 4023 0.000683 0.000683 0.000683 0.08% FabArrayBase::CPC::define() 454 0.0006538 0.0006538 0.0006538 0.08% Amr::InitAmr() 1 0.0005006 0.0005006 0.0005006 0.06% Castro::Castro() 1 0.0004464 0.0004464 0.0004464 0.05% Gravity::swapTimeLevels() 10 0.0004354 0.0004354 0.0004354 0.05% MultiFab::Copy() 11 0.0003224 0.0003224 0.0003224 0.04% MLMG::MLResNormInf() 11 0.0002797 0.0002797 0.0002797 0.03% MultiFab::max() 11 0.000256 0.000256 0.000256 0.03% MLLinOp::define() 11 0.0002264 0.0002264 0.0002264 0.03% MLMG::MLRhsNormInf() 11 0.0002146 0.0002146 0.0002146 0.03% MLLinOp::defineGrids() 11 0.0002045 0.0002045 0.0002045 0.02% Castro::buildMetrics() 1 0.0001642 0.0001642 0.0001642 0.02% Castro::finalize_advance() 10 8.757e-05 8.757e-05 8.757e-05 0.01% FabArrayBase::FB::FB() 56 8.357e-05 8.357e-05 8.357e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.45e-05 5.45e-05 5.45e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.139e-05 5.139e-05 5.139e-05 0.01% StateData::define() 4 4.496e-05 4.496e-05 4.496e-05 0.01% Castro::swap_state_time_levels() 10 4.075e-05 4.075e-05 4.075e-05 0.00% makeSFC 55 4.002e-05 4.002e-05 4.002e-05 0.00% Castro::enforce_consistent_e() 1 3.375e-05 3.375e-05 3.375e-05 0.00% Castro::finalize_do_advance() 10 3.235e-05 3.235e-05 3.235e-05 0.00% Castro::initMFs() 1 2.816e-05 2.816e-05 2.816e-05 0.00% Amr::writeSmallPlotFile() 1 2.672e-05 2.672e-05 2.672e-05 0.00% Castro::create_source_corrector() 10 2.413e-05 2.413e-05 2.413e-05 0.00% DistributionMapping::Distribute() 56 1.468e-05 1.468e-05 1.468e-05 0.00% Amr::initSubcycle() 1 9.016e-06 9.016e-06 9.016e-06 0.00% AmrLevel::checkPointPost() 3 5.616e-06 5.616e-06 5.616e-06 0.00% Gravity::set_mass_offset() 11 4.556e-06 4.556e-06 4.556e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.843e-06 3.843e-06 3.843e-06 0.00% Castro::retry_advance_ctu() 10 3.639e-06 3.639e-06 3.639e-06 0.00% Castro::FluxRegCrseInit 10 2.9e-06 2.9e-06 2.9e-06 0.00% Castro::FluxRegFineAdd() 10 1.909e-06 1.909e-06 1.909e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.884e-06 1.884e-06 1.884e-06 0.00% AmrLevel::checkPointPre() 3 1.884e-06 1.884e-06 1.884e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-16-g826cd378f8ba) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-16-g826cd378f8ba) initialized Starting run at 08:30:35 UTC on 2022-09-16. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09-16-g826cd378f Microphysics git describe: 22.08-13-g1adf1bdb reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.471275715 Restart time = 0.048075531 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052128066 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.05091711 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.05647592 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059990865 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.076889075 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031059005 seconds Ending run at 08:30:35 UTC on 2022-09-16. Run time = 0.376566051 Run time without initialization = 0.327929246 Average number of zones advanced per microsecond: 3.997 Average number of zones advanced per microsecond per rank: 3.997 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3766 ... 0.3766 ... 0.3766 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1010 0.1010 0.1010 26.82% VisMF::Read() 3 0.04027 0.04027 0.04027 10.69% MLCellLinOp::applyBC() 1946 0.0336 0.0336 0.0336 8.92% VisMF::Write(FabArray) 1 0.02971 0.02971 0.02971 7.89% MLPoisson::Fsmooth() 1440 0.02649 0.02649 0.02649 7.03% StateData::FillBoundary(geom) 160 0.01137 0.01137 0.01137 3.02% Castro::normalize_species() 30 0.009944 0.009944 0.009944 2.64% MLCGSolver::bicgstab 36 0.009885 0.009885 0.009885 2.62% MultiFab::Dot() 484 0.009238 0.009238 0.009238 2.45% Castro::computeTemp() 30 0.008595 0.008595 0.008595 2.28% FabArray::setVal() 537 0.006533 0.006533 0.006533 1.73% FillBoundary_nowait() 1766 0.00615 0.00615 0.00615 1.63% MLCellLinOp::defineAuxData() 6 0.005988 0.005988 0.005988 1.59% MultiFab::LinComb() 690 0.005911 0.005911 0.005911 1.57% FabArray::ParallelCopy_nowait() 380 0.005821 0.005821 0.005821 1.55% StateDataPhysBCFunct::() 20 0.005695 0.005695 0.005695 1.51% Castro::enforce_min_density() 30 0.00532 0.00532 0.00532 1.41% MLPoisson::Fapply() 500 0.004897 0.004897 0.004897 1.30% Gravity::fill_multipole_BCs() 6 0.004631 0.004631 0.004631 1.23% Amr::restart() 1 0.003579 0.003579 0.003579 0.95% MLMG::addInterpCorrection() 180 0.003289 0.003289 0.003289 0.87% amrex::average_down 180 0.002898 0.002898 0.002898 0.77% MultiFab::Xpay() 258 0.002758 0.002758 0.002758 0.73% Castro::estTimeStep() 10 0.002716 0.002716 0.002716 0.72% Castro::do_advance_ctu() 5 0.002459 0.002459 0.002459 0.65% BndryData::define() 6 0.002025 0.002025 0.002025 0.54% Castro::reset_internal_energy(MultiFab) 30 0.001741 0.001741 0.001741 0.46% Castro::construct_new_gravity_source() 5 0.001733 0.001733 0.001733 0.46% Amr::writePlotFile() 1 0.00143 0.00143 0.00143 0.38% Castro::construct_old_gravity_source() 5 0.00141 0.00141 0.00141 0.37% Gravity::get_old_grav_vector() 5 0.0009557 0.0009557 0.0009557 0.25% MultiFab::Saxpy() 10 0.0009198 0.0009198 0.0009198 0.24% Castro::reset_internal_energy(Fab) 240 0.0009174 0.0009174 0.0009174 0.24% MLMG::ResNormInf() 42 0.0009043 0.0009043 0.0009043 0.24% Gravity::get_new_grav_vector() 5 0.0008906 0.0008906 0.0008906 0.24% Castro::expand_state() 5 0.0008708 0.0008708 0.0008708 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000863 0.000863 0.000863 0.23% MLCellLinOp::setLevelBC() 6 0.0007958 0.0007958 0.0007958 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007294 0.0007294 0.0007294 0.19% MultiFab::Add() 36 0.0007102 0.0007102 0.0007102 0.19% FabArray::mult() 22 0.0006477 0.0006477 0.0006477 0.17% MLMG::prepareForSolve() 6 0.0006395 0.0006395 0.0006395 0.17% FabArray::setDomainBndry() 20 0.000634 0.000634 0.000634 0.17% MLCellLinOp::prepareForSolve() 6 0.0006125 0.0006125 0.0006125 0.16% MultiFab::contains_nan() 10 0.0005833 0.0005833 0.0005833 0.15% MLCellLinOp::compGrad() 6 0.0004773 0.0004773 0.0004773 0.13% Castro::enforce_speed_limit() 30 0.000452 0.000452 0.000452 0.12% MLCellLinOp::smooth() 720 0.0004513 0.0004513 0.0004513 0.12% Amr::InitAmr() 1 0.0004068 0.0004068 0.0004068 0.11% FabArrayBase::CPC::define() 244 0.0003843 0.0003843 0.0003843 0.10% FabArray::FillBoundary() 1766 0.0003812 0.0003812 0.0003812 0.10% FabArrayBase::getCPC() 632 0.0003683 0.0003683 0.0003683 0.10% main() 1 0.0002503 0.0002503 0.0002503 0.07% FabArrayBase::getFB() 1766 0.0002499 0.0002499 0.0002499 0.07% Amr::coarseTimeStep() 5 0.000233 0.000233 0.000233 0.06% Gravity::update_max_rhs() 6 0.0002227 0.0002227 0.0002227 0.06% Gravity::solve_for_phi() 5 0.0002026 0.0002026 0.0002026 0.05% MLCellLinOp::apply() 500 0.0002001 0.0002001 0.0002001 0.05% Castro::construct_new_gravity() 5 0.0001715 0.0001715 0.0001715 0.05% MultiFab::Copy() 6 0.0001703 0.0001703 0.0001703 0.05% CGSolver::sxay() 690 0.0001627 0.0001627 0.0001627 0.04% Castro::subcycle_advance_ctu() 5 0.0001509 0.0001509 0.0001509 0.04% MLCellLinOp::defineBC() 6 0.0001471 0.0001471 0.0001471 0.04% FillPatchIterator::Initialize 20 0.0001341 0.0001341 0.0001341 0.04% MultiFab::max() 6 0.0001332 0.0001332 0.0001332 0.04% Castro::advance() 5 0.0001324 0.0001324 0.0001324 0.04% MLCGSolver::ParallelAllReduce 659 0.0001287 0.0001287 0.0001287 0.03% Castro::construct_new_source() 25 0.0001256 0.0001256 0.0001256 0.03% FabArray::ParallelCopy() 380 0.0001208 0.0001208 0.0001208 0.03% MLMG::MLRhsNormInf() 6 0.0001114 0.0001114 0.0001114 0.03% MLCellLinOp::correctionResidual() 216 9.63e-05 9.63e-05 9.63e-05 0.03% MLMG::mgVcycle() 36 9.137e-05 9.137e-05 9.137e-05 0.02% StateData::restartDoit() 4 8.11e-05 8.11e-05 8.11e-05 0.02% Amr::timeStep() 5 7.931e-05 7.931e-05 7.931e-05 0.02% MLLinOp::defineGrids() 6 7.72e-05 7.72e-05 7.72e-05 0.02% AmrLevel::restart() 1 7.572e-05 7.572e-05 7.572e-05 0.02% Castro::create_source_corrector() 5 7.162e-05 7.162e-05 7.162e-05 0.02% Castro::finalize_advance() 5 7.033e-05 7.033e-05 7.033e-05 0.02% MLMG:computeResOfCorrection() 180 6.684e-05 6.684e-05 6.684e-05 0.02% FabArrayBase::FB::FB() 26 5.636e-05 5.636e-05 5.636e-05 0.01% MLMG::mgVcycle_down::0 36 4.472e-05 4.472e-05 4.472e-05 0.01% Castro::initialize_do_advance() 5 4.429e-05 4.429e-05 4.429e-05 0.01% MLMG::mgVcycle_down::1 36 4.059e-05 4.059e-05 4.059e-05 0.01% Castro::initialize_advance() 5 3.925e-05 3.925e-05 3.925e-05 0.01% Castro::buildMetrics() 1 3.915e-05 3.915e-05 3.915e-05 0.01% Castro::clean_state() 30 3.849e-05 3.849e-05 3.849e-05 0.01% MLMG::mgVcycle_down::2 36 3.695e-05 3.695e-05 3.695e-05 0.01% Castro::construct_old_source() 25 3.672e-05 3.672e-05 3.672e-05 0.01% MLMG::mgVcycle_down::4 36 3.477e-05 3.477e-05 3.477e-05 0.01% MLMG::mgVcycle_down::3 36 3.433e-05 3.433e-05 3.433e-05 0.01% MLMG::actualBottomSolve() 36 3.425e-05 3.425e-05 3.425e-05 0.01% MLMG::mgVcycle_up::4 36 3.305e-05 3.305e-05 3.305e-05 0.01% MLMG::solve() 6 3.229e-05 3.229e-05 3.229e-05 0.01% MLMG::mgVcycle_up::3 36 3e-05 3e-05 3e-05 0.01% Castro::post_restart() 1 2.953e-05 2.953e-05 2.953e-05 0.01% Gravity::actual_multilevel_solve() 1 2.933e-05 2.933e-05 2.933e-05 0.01% Castro::initMFs() 1 2.879e-05 2.879e-05 2.879e-05 0.01% Castro::swap_state_time_levels() 5 2.851e-05 2.851e-05 2.851e-05 0.01% MLMG::mgVcycle_up::2 36 2.831e-05 2.831e-05 2.831e-05 0.01% Amr::writeSmallPlotFile() 1 2.615e-05 2.615e-05 2.615e-05 0.01% MLMG::mgVcycle_up::0 36 2.614e-05 2.614e-05 2.614e-05 0.01% MLMG::oneIter() 36 2.54e-05 2.54e-05 2.54e-05 0.01% MLCellLinOp::solutionResidual() 42 2.347e-05 2.347e-05 2.347e-05 0.01% Castro::post_timestep() 5 2.341e-05 2.341e-05 2.341e-05 0.01% MLMG::mgVcycle_up::1 36 2.232e-05 2.232e-05 2.232e-05 0.01% MLPoisson::define() 6 2.18e-05 2.18e-05 2.18e-05 0.01% Castro::computeNewDt() 5 2.149e-05 2.149e-05 2.149e-05 0.01% MLLinOp::define() 6 2.046e-05 2.046e-05 2.046e-05 0.01% Castro::construct_old_gravity() 5 1.981e-05 1.981e-05 1.981e-05 0.01% MLMG::computeResidual() 36 1.822e-05 1.822e-05 1.822e-05 0.00% Castro::finalize_do_advance() 5 1.803e-05 1.803e-05 1.803e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.746e-05 1.746e-05 1.746e-05 0.00% makeSFC 30 1.436e-05 1.436e-05 1.436e-05 0.00% MLMG::mgVcycle_bottom 36 1.375e-05 1.375e-05 1.375e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.361e-05 1.361e-05 1.361e-05 0.00% FillPatchSingleLevel 20 1.321e-05 1.321e-05 1.321e-05 0.00% Castro::do_new_sources() 5 8.937e-06 8.937e-06 8.937e-06 0.00% DistributionMapping::Distribute() 31 8.605e-06 8.605e-06 8.605e-06 0.00% Amr::initSubcycle() 1 8.435e-06 8.435e-06 8.435e-06 0.00% Castro::do_old_sources() 5 8.276e-06 8.276e-06 8.276e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.957e-06 6.957e-06 6.957e-06 0.00% Castro::check_for_nan() 10 5.611e-06 5.611e-06 5.611e-06 0.00% Castro::apply_source_to_state() 10 5.272e-06 5.272e-06 5.272e-06 0.00% MLPoisson::prepareForSolve() 6 4.801e-06 4.801e-06 4.801e-06 0.00% MLMG::computeMLResidual() 6 4.458e-06 4.458e-06 4.458e-06 0.00% Gravity::swapTimeLevels() 5 4.326e-06 4.326e-06 4.326e-06 0.00% MLMG::getGradSolution() 6 3.157e-06 3.157e-06 3.157e-06 0.00% Gravity::set_mass_offset() 6 3.074e-06 3.074e-06 3.074e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.793e-06 2.793e-06 2.793e-06 0.00% MLMG::MLResNormInf() 6 2.276e-06 2.276e-06 2.276e-06 0.00% Castro::retry_advance_ctu() 5 1.988e-06 1.988e-06 1.988e-06 0.00% Castro::FluxRegCrseInit 5 1.648e-06 1.648e-06 1.648e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.107e-06 1.107e-06 1.107e-06 0.00% Amr::init() 1 1.057e-06 1.057e-06 1.057e-06 0.00% Castro::FluxRegFineAdd() 5 9.98e-07 9.98e-07 9.98e-07 0.00% AmrLevel::AmrLevel() 1 8.42e-07 8.42e-07 8.42e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3766 0.3766 0.3766 100.00% Amr::coarseTimeStep() 5 0.2966 0.2966 0.2966 78.77% Amr::timeStep() 5 0.295 0.295 0.295 78.34% Castro::advance() 5 0.2889 0.2889 0.2889 76.71% Castro::subcycle_advance_ctu() 5 0.2831 0.2831 0.2831 75.18% Castro::do_advance_ctu() 5 0.283 0.283 0.283 75.14% Castro::construct_new_gravity() 5 0.1396 0.1396 0.1396 37.08% Gravity::solve_phi_with_mlmg() 6 0.1355 0.1355 0.1355 35.98% Gravity::solve_for_phi() 5 0.132 0.132 0.132 35.05% Gravity::actual_solve_with_mlmg() 6 0.1307 0.1307 0.1307 34.72% MLMG::solve() 6 0.1189 0.1189 0.1189 31.58% MLMG::oneIter() 36 0.1119 0.1119 0.1119 29.72% MLMG::mgVcycle() 36 0.1112 0.1112 0.1112 29.52% Castro::construct_ctu_hydro_source() 5 0.101 0.101 0.101 26.82% MLCellLinOp::smooth() 720 0.05689 0.05689 0.05689 15.11% Amr::init() 1 0.04812 0.04812 0.04812 12.78% Amr::restart() 1 0.04812 0.04812 0.04812 12.78% AmrLevel::restart() 1 0.04048 0.04048 0.04048 10.75% MLCellLinOp::applyBC() 1946 0.04044 0.04044 0.04044 10.74% StateData::restartDoit() 4 0.0404 0.0404 0.0404 10.73% VisMF::Read() 3 0.04027 0.04027 0.04027 10.69% MLMG::mgVcycle_bottom 36 0.03395 0.03395 0.03395 9.02% MLMG::actualBottomSolve() 36 0.03394 0.03394 0.03394 9.01% MLCGSolver::bicgstab 36 0.03361 0.03361 0.03361 8.92% Amr::writePlotFile() 1 0.03114 0.03114 0.03114 8.27% VisMF::Write(FabArray) 1 0.02971 0.02971 0.02971 7.89% Castro::clean_state() 30 0.02701 0.02701 0.02701 7.17% MLPoisson::Fsmooth() 1440 0.02649 0.02649 0.02649 7.03% FillPatchIterator::Initialize 20 0.01983 0.01983 0.01983 5.27% FillPatchSingleLevel 20 0.01906 0.01906 0.01906 5.06% StateDataPhysBCFunct::() 20 0.01707 0.01707 0.01707 4.53% MLCellLinOp::apply() 500 0.01532 0.01532 0.01532 4.07% MLMG::mgVcycle_down::0 36 0.01494 0.01494 0.01494 3.97% MLMG::mgVcycle_up::0 36 0.0128 0.0128 0.0128 3.40% StateData::FillBoundary(geom) 160 0.01137 0.01137 0.01137 3.02% Castro::computeTemp() 30 0.01125 0.01125 0.01125 2.99% Castro::initialize_do_advance() 5 0.01065 0.01065 0.01065 2.83% Castro::normalize_species() 30 0.009944 0.009944 0.009944 2.64% MLPoisson::define() 6 0.009552 0.009552 0.009552 2.54% MultiFab::Dot() 484 0.009238 0.009238 0.009238 2.45% MLCellLinOp::correctionResidual() 216 0.008906 0.008906 0.008906 2.36% MLMG:computeResOfCorrection() 180 0.007683 0.007683 0.007683 2.04% Castro::do_new_sources() 5 0.00766 0.00766 0.00766 2.03% Gravity::get_new_grav_vector() 5 0.007461 0.007461 0.007461 1.98% MLMG::mgVcycle_down::1 36 0.007407 0.007407 0.007407 1.97% Castro::construct_old_gravity() 5 0.007269 0.007269 0.007269 1.93% Gravity::get_old_grav_vector() 5 0.007249 0.007249 0.007249 1.92% MLMG::mgVcycle_down::2 36 0.007205 0.007205 0.007205 1.91% FabArray::FillBoundary() 1766 0.006837 0.006837 0.006837 1.82% MLMG::mgVcycle_down::3 36 0.00682 0.00682 0.00682 1.81% MLCellLinOp::defineAuxData() 6 0.006707 0.006707 0.006707 1.78% FabArray::setVal() 537 0.006533 0.006533 0.006533 1.73% MLMG::mgVcycle_down::4 36 0.00652 0.00652 0.00652 1.73% FillBoundary_nowait() 1766 0.006456 0.006456 0.006456 1.71% FabArray::ParallelCopy() 380 0.006314 0.006314 0.006314 1.68% FabArray::ParallelCopy_nowait() 380 0.006193 0.006193 0.006193 1.64% Castro::do_old_sources() 5 0.006166 0.006166 0.006166 1.64% CGSolver::sxay() 690 0.006073 0.006073 0.006073 1.61% Castro::post_timestep() 5 0.00606 0.00606 0.00606 1.61% MultiFab::LinComb() 690 0.005911 0.005911 0.005911 1.57% Castro::expand_state() 5 0.005806 0.005806 0.005806 1.54% MLCGSolver::ParallelAllReduce 659 0.005568 0.005568 0.005568 1.48% Castro::initialize_advance() 5 0.005554 0.005554 0.005554 1.47% MLMG::mgVcycle_up::2 36 0.005539 0.005539 0.005539 1.47% MLMG::addInterpCorrection() 180 0.005445 0.005445 0.005445 1.45% MLMG::mgVcycle_up::1 36 0.005424 0.005424 0.005424 1.44% Castro::enforce_min_density() 30 0.00532 0.00532 0.00532 1.41% MLMG::mgVcycle_up::4 36 0.005263 0.005263 0.005263 1.40% MLMG::mgVcycle_up::3 36 0.005218 0.005218 0.005218 1.39% amrex::average_down 180 0.005071 0.005071 0.005071 1.35% MLPoisson::Fapply() 500 0.004897 0.004897 0.004897 1.30% Gravity::fill_multipole_BCs() 6 0.004631 0.004631 0.004631 1.23% Castro::post_restart() 1 0.003866 0.003866 0.003866 1.03% Gravity::multilevel_solve_for_new_phi() 1 0.003744 0.003744 0.003744 0.99% Gravity::actual_multilevel_solve() 1 0.003726 0.003726 0.003726 0.99% MLCellLinOp::solutionResidual() 42 0.003138 0.003138 0.003138 0.83% MLMG::prepareForSolve() 6 0.002789 0.002789 0.002789 0.74% MultiFab::Xpay() 258 0.002758 0.002758 0.002758 0.73% Castro::estTimeStep() 10 0.002716 0.002716 0.002716 0.72% MLCellLinOp::defineBC() 6 0.002696 0.002696 0.002696 0.72% Castro::reset_internal_energy(MultiFab) 30 0.002658 0.002658 0.002658 0.71% MLMG::computeResidual() 36 0.002595 0.002595 0.002595 0.69% BndryData::define() 6 0.002549 0.002549 0.002549 0.68% Castro::construct_new_source() 25 0.001859 0.001859 0.001859 0.49% Castro::construct_new_gravity_source() 5 0.001733 0.001733 0.001733 0.46% Castro::construct_old_source() 25 0.001447 0.001447 0.001447 0.38% Castro::construct_old_gravity_source() 5 0.00141 0.00141 0.00141 0.37% Castro::computeNewDt() 5 0.001377 0.001377 0.001377 0.37% Castro::apply_source_to_state() 10 0.000925 0.000925 0.000925 0.25% MultiFab::Saxpy() 10 0.0009198 0.0009198 0.0009198 0.24% Castro::reset_internal_energy(Fab) 240 0.0009174 0.0009174 0.0009174 0.24% MLMG::ResNormInf() 42 0.0009043 0.0009043 0.0009043 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000863 0.000863 0.000863 0.23% MLCellLinOp::setLevelBC() 6 0.0007958 0.0007958 0.0007958 0.21% FabArrayBase::getCPC() 632 0.0007526 0.0007526 0.0007526 0.20% MLMG::getGradSolution() 6 0.0007475 0.0007475 0.0007475 0.20% MLCellLinOp::compGrad() 6 0.0007444 0.0007444 0.0007444 0.20% MultiFab::Add() 36 0.0007102 0.0007102 0.0007102 0.19% FabArray::mult() 22 0.0006477 0.0006477 0.0006477 0.17% FabArray::setDomainBndry() 20 0.000634 0.000634 0.000634 0.17% MLPoisson::prepareForSolve() 6 0.0006173 0.0006173 0.0006173 0.16% MLCellLinOp::prepareForSolve() 6 0.0006125 0.0006125 0.0006125 0.16% Castro::check_for_nan() 10 0.0005889 0.0005889 0.0005889 0.16% MultiFab::contains_nan() 10 0.0005833 0.0005833 0.0005833 0.15% MLMG::computeMLResidual() 6 0.0005651 0.0005651 0.0005651 0.15% Castro::enforce_speed_limit() 30 0.000452 0.000452 0.000452 0.12% Gravity::update_max_rhs() 6 0.0004317 0.0004317 0.0004317 0.11% Amr::InitAmr() 1 0.0004152 0.0004152 0.0004152 0.11% FabArrayBase::CPC::define() 244 0.0003843 0.0003843 0.0003843 0.10% FabArrayBase::getFB() 1766 0.0003062 0.0003062 0.0003062 0.08% Gravity::swapTimeLevels() 5 0.0002287 0.0002287 0.0002287 0.06% MultiFab::Copy() 6 0.0001703 0.0001703 0.0001703 0.05% Castro::buildMetrics() 1 0.0001586 0.0001586 0.0001586 0.04% MLMG::MLResNormInf() 6 0.000146 0.000146 0.000146 0.04% MultiFab::max() 6 0.0001332 0.0001332 0.0001332 0.04% MLLinOp::define() 6 0.0001274 0.0001274 0.0001274 0.03% MLMG::MLRhsNormInf() 6 0.0001114 0.0001114 0.0001114 0.03% MLLinOp::defineGrids() 6 0.000107 0.000107 0.000107 0.03% Castro::finalize_advance() 5 7.298e-05 7.298e-05 7.298e-05 0.02% Castro::create_source_corrector() 5 7.162e-05 7.162e-05 7.162e-05 0.02% FabArrayBase::FB::FB() 26 5.636e-05 5.636e-05 5.636e-05 0.01% Castro::initMFs() 1 2.879e-05 2.879e-05 2.879e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.868e-05 2.868e-05 2.868e-05 0.01% Castro::swap_state_time_levels() 5 2.851e-05 2.851e-05 2.851e-05 0.01% Amr::writeSmallPlotFile() 1 2.615e-05 2.615e-05 2.615e-05 0.01% makeSFC 30 2.172e-05 2.172e-05 2.172e-05 0.01% Castro::finalize_do_advance() 5 1.803e-05 1.803e-05 1.803e-05 0.00% DistributionMapping::Distribute() 31 8.605e-06 8.605e-06 8.605e-06 0.00% Amr::initSubcycle() 1 8.435e-06 8.435e-06 8.435e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.038e-06 4.038e-06 4.038e-06 0.00% Gravity::set_mass_offset() 6 3.074e-06 3.074e-06 3.074e-06 0.00% Castro::retry_advance_ctu() 5 1.988e-06 1.988e-06 1.988e-06 0.00% Castro::FluxRegCrseInit 5 1.648e-06 1.648e-06 1.648e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.107e-06 1.107e-06 1.107e-06 0.00% Castro::FluxRegFineAdd() 5 9.98e-07 9.98e-07 9.98e-07 0.00% AmrLevel::AmrLevel() 1 8.42e-07 8.42e-07 8.42e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-16-g826cd378f8ba) finalized