Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-24-gea7e8a507e77) initialized Starting run at 10:08:21 UTC on 2022-12-28. Successfully read inputs file ... Castro git describe: 22.12-11-g45d01ffe1 AMReX git describe: 22.12-24-gea7e8a507 Microphysics git describe: 22.12-28-gdc4a7e21 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.056654184 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.032662646 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.044735734 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.046801091 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.047658044 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.063355675 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.072985434 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.055937783 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.05231135 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.044874991 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.045746041 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.045744195 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.058784628 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.056045301 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032521007 seconds Ending run at 10:08:22 UTC on 2022-12-28. Run time = 0.804364531 Run time without initialization = 0.668117875 Average number of zones advanced per microsecond: 3.924 Average number of zones advanced per microsecond per rank: 3.924 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8044 ... 0.8044 ... 0.8044 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2271 0.2271 0.2271 28.23% Castro::construct_ctu_hydro_source() 10 0.1748 0.1748 0.1748 21.73% MLCellLinOp::applyBC() 4433 0.07379 0.07379 0.07379 9.17% MLPoisson::Fsmooth() 3280 0.03152 0.03152 0.03152 3.92% FillBoundary_nowait() 4023 0.03109 0.03109 0.03109 3.86% StateData::FillBoundary(geom) 328 0.02253 0.02253 0.02253 2.80% amrex::Dot() 1114 0.01991 0.01991 0.01991 2.47% StateDataPhysBCFunct::() 41 0.01828 0.01828 0.01828 2.27% amrex::Copy() 1029 0.01453 0.01453 0.01453 1.81% FabArray::norminf() 743 0.01389 0.01389 0.01389 1.73% Castro::normalize_species() 62 0.01326 0.01326 0.01326 1.65% Castro::computeTemp() 63 0.01307 0.01307 0.01307 1.63% FabArray::setVal() 1144 0.01285 0.01285 0.01285 1.60% FabArray::ParallelCopy_nowait() 861 0.01277 0.01277 0.01277 1.59% MLPoisson::Fapply() 1142 0.01014 0.01014 0.01014 1.26% MLCellLinOp::defineAuxData() 11 0.009541 0.009541 0.009541 1.19% Castro::enforce_min_density() 62 0.008895 0.008895 0.008895 1.11% FabArray::Saxpy() 813 0.007959 0.007959 0.007959 0.99% FabArray::Xpay() 821 0.007912 0.007912 0.007912 0.98% MLMG::addInterpCorrection() 410 0.006459 0.006459 0.006459 0.80% Gravity::fill_multipole_BCs() 11 0.00591 0.00591 0.00591 0.73% amrex::average_down 410 0.005721 0.005721 0.005721 0.71% FabArray::LinComb() 557 0.004383 0.004383 0.004383 0.54% amrex::Add() 164 0.004296 0.004296 0.004296 0.53% Castro::estTimeStep() 21 0.004268 0.004268 0.004268 0.53% Castro::reset_internal_energy(MultiFab) 63 0.004002 0.004002 0.004002 0.50% Amr::checkPoint() 3 0.003943 0.003943 0.003943 0.49% BndryData::define() 11 0.003485 0.003485 0.003485 0.43% Castro::construct_new_gravity_source() 10 0.002727 0.002727 0.002727 0.34% Amr::writePlotFile() 2 0.002329 0.002329 0.002329 0.29% MLCGSolver::bicgstab 82 0.002148 0.002148 0.002148 0.27% Castro::do_advance_ctu() 10 0.002142 0.002142 0.002142 0.27% Castro::construct_old_gravity_source() 10 0.002014 0.002014 0.002014 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001638 0.001638 0.001638 0.20% Castro::reset_internal_energy(Fab) 504 0.001433 0.001433 0.001433 0.18% Gravity::actual_solve_with_mlmg() 11 0.001382 0.001382 0.001382 0.17% MLCellLinOp::setLevelBC() 11 0.00135 0.00135 0.00135 0.17% FabArray::setDomainBndry() 41 0.001325 0.001325 0.001325 0.16% FabArray::mult() 43 0.001322 0.001322 0.001322 0.16% Castro::enforce_speed_limit() 62 0.001294 0.001294 0.001294 0.16% MultiFab::contains_nan() 20 0.001172 0.001172 0.001172 0.15% MLCellLinOp::smooth() 1640 0.001167 0.001167 0.001167 0.15% Castro::initData() 1 0.001096 0.001096 0.001096 0.14% MLCellLinOp::prepareForSolve() 11 0.001077 0.001077 0.001077 0.13% MLCellLinOp::compGrad() 11 0.0009013 0.0009013 0.0009013 0.11% MLMG::prepareForSolve() 11 0.0008482 0.0008482 0.0008482 0.11% FabArray::FillBoundary() 4023 0.0007737 0.0007737 0.0007737 0.10% FabArrayBase::getCPC() 1323 0.0007369 0.0007369 0.0007369 0.09% FabArrayBase::CPC::define() 454 0.0006877 0.0006877 0.0006877 0.09% Gravity::get_new_grav_vector() 11 0.0006152 0.0006152 0.0006152 0.08% FabArrayBase::getFB() 4023 0.0005904 0.0005904 0.0005904 0.07% Gravity::get_old_grav_vector() 10 0.0005277 0.0005277 0.0005277 0.07% Amr::InitAmr() 1 0.0004768 0.0004768 0.0004768 0.06% MLCellLinOp::apply() 1142 0.0004765 0.0004765 0.0004765 0.06% MLMG::mgVcycle() 82 0.0003769 0.0003769 0.0003769 0.05% Amr::coarseTimeStep() 10 0.0003453 0.0003453 0.0003453 0.04% main() 1 0.0002948 0.0002948 0.0002948 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002748 0.0002748 0.0002748 0.03% MultiFab::max() 11 0.0002566 0.0002566 0.0002566 0.03% FabArray::ParallelCopy() 861 0.0002396 0.0002396 0.0002396 0.03% MLCellLinOp::correctionResidual() 492 0.0002266 0.0002266 0.0002266 0.03% Castro::subcycle_advance_ctu() 10 0.0002202 0.0002202 0.0002202 0.03% FillPatchIterator::Initialize 41 0.0002101 0.0002101 0.0002101 0.03% MLCellLinOp::defineBC() 11 0.000203 0.000203 0.000203 0.03% MLLinOp::defineGrids() 11 0.0001772 0.0001772 0.0001772 0.02% Amr::timeStep() 10 0.0001576 0.0001576 0.0001576 0.02% Gravity::solve_for_phi() 10 0.0001463 0.0001463 0.0001463 0.02% StateData::checkPoint() 12 0.0001373 0.0001373 0.0001373 0.02% Gravity::update_max_rhs() 11 0.0001169 0.0001169 0.0001169 0.01% MLMG:computeResOfCorrection() 410 0.0001103 0.0001103 0.0001103 0.01% MLMG::mgVcycle_down::0 82 9.481e-05 9.481e-05 9.481e-05 0.01% MLMG::actualBottomSolve() 82 9.042e-05 9.042e-05 9.042e-05 0.01% MLMG::mgVcycle_down::1 82 8.913e-05 8.913e-05 8.913e-05 0.01% AmrLevel::checkPoint() 3 8.439e-05 8.439e-05 8.439e-05 0.01% FabArrayBase::FB::FB() 56 8.402e-05 8.402e-05 8.402e-05 0.01% MLMG::mgVcycle_down::2 82 8.324e-05 8.324e-05 8.324e-05 0.01% MLMG::mgVcycle_down::3 82 7.933e-05 7.933e-05 7.933e-05 0.01% Castro::clean_state() 62 7.932e-05 7.932e-05 7.932e-05 0.01% MLMG::mgVcycle_down::4 82 7.81e-05 7.81e-05 7.81e-05 0.01% MLMG::solve() 11 7.567e-05 7.567e-05 7.567e-05 0.01% Castro::Castro() 1 7.544e-05 7.544e-05 7.544e-05 0.01% Castro::expand_state() 10 7.427e-05 7.427e-05 7.427e-05 0.01% Castro::initialize_advance() 10 6.546e-05 6.546e-05 6.546e-05 0.01% Castro::advance() 10 6.426e-05 6.426e-05 6.426e-05 0.01% Castro::finalize_advance() 10 6.397e-05 6.397e-05 6.397e-05 0.01% MLMG::mgVcycle_up::4 82 6.324e-05 6.324e-05 6.324e-05 0.01% MLMG::mgVcycle_up::0 82 5.391e-05 5.391e-05 5.391e-05 0.01% MLMG::oneIter() 82 5.316e-05 5.316e-05 5.316e-05 0.01% MLMG::mgVcycle_up::1 82 5.228e-05 5.228e-05 5.228e-05 0.01% MLMG::mgVcycle_up::3 82 5.152e-05 5.152e-05 5.152e-05 0.01% MLCellLinOp::solutionResidual() 93 5.035e-05 5.035e-05 5.035e-05 0.01% MLMG::mgVcycle_up::2 82 5.021e-05 5.021e-05 5.021e-05 0.01% Castro::initialize_do_advance() 10 4.432e-05 4.432e-05 4.432e-05 0.01% Castro::construct_new_source() 50 4.195e-05 4.195e-05 4.195e-05 0.01% Castro::finalize_do_advance() 10 3.724e-05 3.724e-05 3.724e-05 0.00% Castro::swap_state_time_levels() 10 3.604e-05 3.604e-05 3.604e-05 0.00% StateData::define() 4 3.436e-05 3.436e-05 3.436e-05 0.00% Castro::enforce_consistent_e() 1 3.346e-05 3.346e-05 3.346e-05 0.00% MLMG::ResNormInf() 93 3.281e-05 3.281e-05 3.281e-05 0.00% MLMG::mgVcycle_bottom 82 3.195e-05 3.195e-05 3.195e-05 0.00% MLMG::computeResidual() 82 3.028e-05 3.028e-05 3.028e-05 0.00% Castro::construct_new_gravity() 10 2.937e-05 2.937e-05 2.937e-05 0.00% FillPatchSingleLevel 41 2.926e-05 2.926e-05 2.926e-05 0.00% makeSFC 55 2.716e-05 2.716e-05 2.716e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.28e-05 2.28e-05 2.28e-05 0.00% Amr::writeSmallPlotFile() 1 2.269e-05 2.269e-05 2.269e-05 0.00% MLPoisson::define() 11 2.165e-05 2.165e-05 2.165e-05 0.00% Amr::FinalizeInit() 1 2.032e-05 2.032e-05 2.032e-05 0.00% Castro::initMFs() 1 1.858e-05 1.858e-05 1.858e-05 0.00% Castro::do_new_sources() 10 1.747e-05 1.747e-05 1.747e-05 0.00% Castro::construct_old_source() 50 1.728e-05 1.728e-05 1.728e-05 0.00% DistributionMapping::Distribute() 56 1.662e-05 1.662e-05 1.662e-05 0.00% Castro::buildMetrics() 1 1.603e-05 1.603e-05 1.603e-05 0.00% Castro::do_old_sources() 10 1.555e-05 1.555e-05 1.555e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.446e-05 1.446e-05 1.446e-05 0.00% Amr::defBaseLevel() 1 1.415e-05 1.415e-05 1.415e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.385e-05 1.385e-05 1.385e-05 0.00% Castro::check_for_nan() 20 1.161e-05 1.161e-05 1.161e-05 0.00% Castro::post_timestep() 10 1.07e-05 1.07e-05 1.07e-05 0.00% Castro::post_init() 1 1.068e-05 1.068e-05 1.068e-05 0.00% MLLinOp::define() 11 1.062e-05 1.062e-05 1.062e-05 0.00% Castro::apply_source_to_state() 20 1.016e-05 1.016e-05 1.016e-05 0.00% Gravity::swapTimeLevels() 10 9.42e-06 9.42e-06 9.42e-06 0.00% Castro::construct_old_gravity() 10 9.378e-06 9.378e-06 9.378e-06 0.00% AmrLevel::AmrLevel(dm) 1 9.305e-06 9.305e-06 9.305e-06 0.00% MLMG::MLRhsNormInf() 11 8.983e-06 8.983e-06 8.983e-06 0.00% MLPoisson::prepareForSolve() 11 8.477e-06 8.477e-06 8.477e-06 0.00% Amr::initSubcycle() 1 8.468e-06 8.468e-06 8.468e-06 0.00% Gravity::actual_multilevel_solve() 1 8.05e-06 8.05e-06 8.05e-06 0.00% MLMG::computeMLResidual() 11 7.055e-06 7.055e-06 7.055e-06 0.00% Castro::computeNewDt() 9 6.64e-06 6.64e-06 6.64e-06 0.00% MLMG::getGradSolution() 11 5.753e-06 5.753e-06 5.753e-06 0.00% Amr::InitializeInit() 1 5.724e-06 5.724e-06 5.724e-06 0.00% AmrLevel::checkPointPost() 3 5.246e-06 5.246e-06 5.246e-06 0.00% Castro::retry_advance_ctu() 10 4.435e-06 4.435e-06 4.435e-06 0.00% Gravity::set_mass_offset() 11 4.082e-06 4.082e-06 4.082e-06 0.00% MLMG::MLResNormInf() 11 3.856e-06 3.856e-06 3.856e-06 0.00% Castro::create_source_corrector() 10 3.102e-06 3.102e-06 3.102e-06 0.00% Castro::FluxRegCrseInit 10 2.907e-06 2.907e-06 2.907e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.748e-06 2.748e-06 2.748e-06 0.00% Castro::computeInitialDt() 2 2.744e-06 2.744e-06 2.744e-06 0.00% Amr::init() 1 2.627e-06 2.627e-06 2.627e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.224e-06 2.224e-06 2.224e-06 0.00% Castro::FluxRegFineAdd() 10 2.128e-06 2.128e-06 2.128e-06 0.00% AmrLevel::checkPointPre() 3 1.889e-06 1.889e-06 1.889e-06 0.00% Castro::post_regrid() 1 1.349e-06 1.349e-06 1.349e-06 0.00% Amr::initialInit() 1 1.057e-06 1.057e-06 1.057e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8044 0.8044 0.8044 100.00% Amr::coarseTimeStep() 10 0.6354 0.6354 0.6354 78.99% Amr::timeStep() 10 0.5208 0.5208 0.5208 64.74% Castro::advance() 10 0.5138 0.5138 0.5138 63.88% Castro::subcycle_advance_ctu() 10 0.5014 0.5014 0.5014 62.34% Castro::do_advance_ctu() 10 0.5012 0.5012 0.5012 62.31% Gravity::solve_phi_with_mlmg() 11 0.2747 0.2747 0.2747 34.16% Gravity::actual_solve_with_mlmg() 11 0.2684 0.2684 0.2684 33.36% Castro::construct_new_gravity() 10 0.2545 0.2545 0.2545 31.64% MLMG::solve() 11 0.2485 0.2485 0.2485 30.89% Gravity::solve_for_phi() 10 0.2358 0.2358 0.2358 29.32% MLMG::oneIter() 82 0.2345 0.2345 0.2345 29.16% MLMG::mgVcycle() 82 0.2309 0.2309 0.2309 28.71% VisMF::Write(FabArray) 11 0.2271 0.2271 0.2271 28.23% Castro::construct_ctu_hydro_source() 10 0.1748 0.1748 0.1748 21.73% Amr::checkPoint() 3 0.1688 0.1688 0.1688 20.98% AmrLevel::checkPoint() 3 0.1648 0.1648 0.1648 20.49% StateData::checkPoint() 12 0.1647 0.1647 0.1647 20.48% Amr::init() 1 0.1356 0.1356 0.1356 16.86% MLCellLinOp::smooth() 1640 0.1141 0.1141 0.1141 14.18% MLCellLinOp::applyBC() 4433 0.1063 0.1063 0.1063 13.22% MLMG::mgVcycle_bottom 82 0.07123 0.07123 0.07123 8.86% MLMG::actualBottomSolve() 82 0.0712 0.0712 0.0712 8.85% MLCGSolver::bicgstab 82 0.07052 0.07052 0.07052 8.77% Amr::writePlotFile() 2 0.06531 0.06531 0.06531 8.12% FillPatchIterator::Initialize 41 0.04634 0.04634 0.04634 5.76% Amr::initialInit() 1 0.04617 0.04617 0.04617 5.74% FillPatchSingleLevel 41 0.04481 0.04481 0.04481 5.57% Amr::FinalizeInit() 1 0.04242 0.04242 0.04242 5.27% Castro::clean_state() 62 0.04136 0.04136 0.04136 5.14% Castro::post_init() 1 0.04113 0.04113 0.04113 5.11% StateDataPhysBCFunct::() 41 0.04081 0.04081 0.04081 5.07% Gravity::multilevel_solve_for_new_phi() 1 0.03937 0.03937 0.03937 4.89% Gravity::actual_multilevel_solve() 1 0.03935 0.03935 0.03935 4.89% MLCellLinOp::apply() 1142 0.0351 0.0351 0.0351 4.36% MLMG::mgVcycle_down::0 82 0.03312 0.03312 0.03312 4.12% FabArray::FillBoundary() 4023 0.03253 0.03253 0.03253 4.04% FillBoundary_nowait() 4023 0.03176 0.03176 0.03176 3.95% MLPoisson::Fsmooth() 3280 0.03152 0.03152 0.03152 3.92% MLMG::mgVcycle_up::0 82 0.02529 0.02529 0.02529 3.14% StateData::FillBoundary(geom) 328 0.02253 0.02253 0.02253 2.80% MLCellLinOp::correctionResidual() 492 0.02156 0.02156 0.02156 2.68% Castro::initialize_do_advance() 10 0.02108 0.02108 0.02108 2.62% Gravity::get_new_grav_vector() 11 0.02015 0.02015 0.02015 2.51% amrex::Dot() 1114 0.01991 0.01991 0.01991 2.47% MLMG:computeResOfCorrection() 410 0.01903 0.01903 0.01903 2.37% Castro::computeTemp() 63 0.01851 0.01851 0.01851 2.30% MLPoisson::define() 11 0.01575 0.01575 0.01575 1.96% MLMG::mgVcycle_down::1 82 0.01526 0.01526 0.01526 1.90% amrex::Copy() 1029 0.01453 0.01453 0.01453 1.81% MLMG::mgVcycle_down::2 82 0.01428 0.01428 0.01428 1.78% Castro::construct_old_gravity() 10 0.01422 0.01422 0.01422 1.77% Gravity::get_old_grav_vector() 10 0.01421 0.01421 0.01421 1.77% MLMG::mgVcycle_down::3 82 0.01389 0.01389 0.01389 1.73% FabArray::norminf() 743 0.01389 0.01389 0.01389 1.73% FabArray::ParallelCopy() 861 0.01382 0.01382 0.01382 1.72% MLMG::mgVcycle_down::4 82 0.01374 0.01374 0.01374 1.71% FabArray::ParallelCopy_nowait() 861 0.01358 0.01358 0.01358 1.69% Castro::expand_state() 10 0.01352 0.01352 0.01352 1.68% Castro::normalize_species() 62 0.01326 0.01326 0.01326 1.65% FabArray::setVal() 1144 0.01285 0.01285 0.01285 1.60% MLCGSolver::ParallelAllReduce 1514 0.01195 0.01195 0.01195 1.49% Castro::initialize_advance() 10 0.01175 0.01175 0.01175 1.46% MLMG::addInterpCorrection() 410 0.01143 0.01143 0.01143 1.42% MLMG::mgVcycle_up::4 82 0.01119 0.01119 0.01119 1.39% MLMG::mgVcycle_up::1 82 0.01111 0.01111 0.01111 1.38% MLCellLinOp::defineAuxData() 11 0.01087 0.01087 0.01087 1.35% MLMG::mgVcycle_up::2 82 0.01081 0.01081 0.01081 1.34% MLMG::mgVcycle_up::3 82 0.01062 0.01062 0.01062 1.32% amrex::average_down 410 0.01061 0.01061 0.01061 1.32% Castro::do_new_sources() 10 0.01043 0.01043 0.01043 1.30% MLPoisson::Fapply() 1142 0.01014 0.01014 0.01014 1.26% Castro::enforce_min_density() 62 0.008895 0.008895 0.008895 1.11% Castro::do_old_sources() 10 0.008562 0.008562 0.008562 1.06% FabArray::Saxpy() 813 0.007959 0.007959 0.007959 0.99% FabArray::Xpay() 821 0.007912 0.007912 0.007912 0.98% MLCellLinOp::solutionResidual() 93 0.007012 0.007012 0.007012 0.87% Castro::post_timestep() 10 0.006763 0.006763 0.006763 0.84% Gravity::fill_multipole_BCs() 11 0.006168 0.006168 0.006168 0.77% MLMG::computeResidual() 82 0.006062 0.006062 0.006062 0.75% Castro::reset_internal_energy(MultiFab) 63 0.005435 0.005435 0.005435 0.68% MLCellLinOp::defineBC() 11 0.004609 0.004609 0.004609 0.57% MLMG::prepareForSolve() 11 0.004526 0.004526 0.004526 0.56% BndryData::define() 11 0.004406 0.004406 0.004406 0.55% FabArray::LinComb() 557 0.004383 0.004383 0.004383 0.54% amrex::Add() 164 0.004296 0.004296 0.004296 0.53% Castro::estTimeStep() 21 0.004268 0.004268 0.004268 0.53% Amr::InitializeInit() 1 0.003751 0.003751 0.003751 0.47% Amr::defBaseLevel() 1 0.003745 0.003745 0.003745 0.47% Castro::initData() 1 0.003281 0.003281 0.003281 0.41% Castro::construct_new_source() 50 0.002769 0.002769 0.002769 0.34% Castro::construct_new_gravity_source() 10 0.002727 0.002727 0.002727 0.34% MLMG::ResNormInf() 93 0.002094 0.002094 0.002094 0.26% Castro::construct_old_source() 50 0.002032 0.002032 0.002032 0.25% Castro::construct_old_gravity_source() 10 0.002014 0.002014 0.002014 0.25% Castro::apply_source_to_state() 20 0.001819 0.001819 0.001819 0.23% Castro::computeNewDt() 9 0.00167 0.00167 0.00167 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001638 0.001638 0.001638 0.20% Castro::reset_internal_energy(Fab) 504 0.001433 0.001433 0.001433 0.18% FabArrayBase::getCPC() 1323 0.001425 0.001425 0.001425 0.18% MLMG::getGradSolution() 11 0.001365 0.001365 0.001365 0.17% MLCellLinOp::compGrad() 11 0.001359 0.001359 0.001359 0.17% MLCellLinOp::setLevelBC() 11 0.00135 0.00135 0.00135 0.17% FabArray::setDomainBndry() 41 0.001325 0.001325 0.001325 0.16% FabArray::mult() 43 0.001322 0.001322 0.001322 0.16% Castro::enforce_speed_limit() 62 0.001294 0.001294 0.001294 0.16% Castro::check_for_nan() 20 0.001184 0.001184 0.001184 0.15% MultiFab::contains_nan() 20 0.001172 0.001172 0.001172 0.15% Castro::post_regrid() 1 0.001117 0.001117 0.001117 0.14% MLPoisson::prepareForSolve() 11 0.001085 0.001085 0.001085 0.13% MLCellLinOp::prepareForSolve() 11 0.001077 0.001077 0.001077 0.13% MLMG::computeMLResidual() 11 0.0009868 0.0009868 0.0009868 0.12% Gravity::update_max_rhs() 11 0.0008088 0.0008088 0.0008088 0.10% Castro::computeInitialDt() 2 0.0007125 0.0007125 0.0007125 0.09% FabArrayBase::CPC::define() 454 0.0006877 0.0006877 0.0006877 0.09% FabArrayBase::getFB() 4023 0.0006744 0.0006744 0.0006744 0.08% Castro::finalize_advance() 10 0.0005791 0.0005791 0.0005791 0.07% Amr::InitAmr() 1 0.0004853 0.0004853 0.0004853 0.06% Gravity::swapTimeLevels() 10 0.000432 0.000432 0.000432 0.05% Castro::Castro() 1 0.0004021 0.0004021 0.0004021 0.05% MLMG::MLResNormInf() 11 0.0002805 0.0002805 0.0002805 0.03% MultiFab::max() 11 0.0002566 0.0002566 0.0002566 0.03% MLLinOp::define() 11 0.0002465 0.0002465 0.0002465 0.03% MLLinOp::defineGrids() 11 0.0002359 0.0002359 0.0002359 0.03% MLMG::MLRhsNormInf() 11 0.0002217 0.0002217 0.0002217 0.03% Castro::buildMetrics() 1 0.000149 0.000149 0.000149 0.02% FabArrayBase::FB::FB() 56 8.402e-05 8.402e-05 8.402e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.645e-05 5.645e-05 5.645e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.366e-05 4.366e-05 4.366e-05 0.01% makeSFC 55 4.26e-05 4.26e-05 4.26e-05 0.01% Castro::finalize_do_advance() 10 3.724e-05 3.724e-05 3.724e-05 0.00% Castro::swap_state_time_levels() 10 3.604e-05 3.604e-05 3.604e-05 0.00% StateData::define() 4 3.436e-05 3.436e-05 3.436e-05 0.00% Castro::enforce_consistent_e() 1 3.346e-05 3.346e-05 3.346e-05 0.00% Amr::writeSmallPlotFile() 1 2.269e-05 2.269e-05 2.269e-05 0.00% Castro::initMFs() 1 1.858e-05 1.858e-05 1.858e-05 0.00% DistributionMapping::Distribute() 56 1.662e-05 1.662e-05 1.662e-05 0.00% Amr::initSubcycle() 1 8.468e-06 8.468e-06 8.468e-06 0.00% AmrLevel::checkPointPost() 3 5.246e-06 5.246e-06 5.246e-06 0.00% Castro::retry_advance_ctu() 10 4.435e-06 4.435e-06 4.435e-06 0.00% Gravity::set_mass_offset() 11 4.082e-06 4.082e-06 4.082e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.932e-06 3.932e-06 3.932e-06 0.00% Castro::create_source_corrector() 10 3.102e-06 3.102e-06 3.102e-06 0.00% Castro::FluxRegCrseInit 10 2.907e-06 2.907e-06 2.907e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.224e-06 2.224e-06 2.224e-06 0.00% Castro::FluxRegFineAdd() 10 2.128e-06 2.128e-06 2.128e-06 0.00% AmrLevel::checkPointPre() 3 1.889e-06 1.889e-06 1.889e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-24-gea7e8a507e77) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-24-gea7e8a507e77) initialized Starting run at 10:08:22 UTC on 2022-12-28. Successfully read inputs file ... Castro git describe: 22.12-11-g45d01ffe1 AMReX git describe: 22.12-24-gea7e8a507 Microphysics git describe: 22.12-28-gdc4a7e21 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.46771892 Restart time = 0.047213609 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.045996422 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.044234683 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.044920736 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.044960751 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.049851632 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032410554 seconds Ending run at 10:08:23 UTC on 2022-12-28. Run time = 0.310538045 Run time without initialization = 0.262757831 Average number of zones advanced per microsecond: 4.988 Average number of zones advanced per microsecond per rank: 4.988 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3106 ... 0.3106 ... 0.3106 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0618 0.0618 0.0618 19.90% VisMF::Read() 3 0.04121 0.04121 0.04121 13.27% MLCellLinOp::applyBC() 1946 0.03154 0.03154 0.03154 10.15% VisMF::Write(FabArray) 1 0.03093 0.03093 0.03093 9.96% MLPoisson::Fsmooth() 1440 0.01353 0.01353 0.01353 4.35% FillBoundary_nowait() 1766 0.0132 0.0132 0.0132 4.25% StateData::FillBoundary(geom) 160 0.01105 0.01105 0.01105 3.56% amrex::Dot() 484 0.0083 0.0083 0.0083 2.67% amrex::Copy() 463 0.006786 0.006786 0.006786 2.19% Castro::normalize_species() 30 0.006082 0.006082 0.006082 1.96% FabArray::setVal() 537 0.006028 0.006028 0.006028 1.94% FabArray::norminf() 326 0.005938 0.005938 0.005938 1.91% FabArray::ParallelCopy_nowait() 380 0.00572 0.00572 0.00572 1.84% StateDataPhysBCFunct::() 20 0.005609 0.005609 0.005609 1.81% Castro::computeTemp() 30 0.005433 0.005433 0.005433 1.75% MLCellLinOp::defineAuxData() 6 0.005129 0.005129 0.005129 1.65% MLPoisson::Fapply() 500 0.004334 0.004334 0.004334 1.40% FabArray::Saxpy() 355 0.003518 0.003518 0.003518 1.13% FabArray::Xpay() 361 0.003398 0.003398 0.003398 1.09% Gravity::fill_multipole_BCs() 6 0.002857 0.002857 0.002857 0.92% MLMG::addInterpCorrection() 180 0.00276 0.00276 0.00276 0.89% Castro::enforce_min_density() 30 0.002529 0.002529 0.002529 0.81% Amr::restart() 1 0.002478 0.002478 0.002478 0.80% amrex::average_down 180 0.002466 0.002466 0.002466 0.79% Castro::estTimeStep() 10 0.002089 0.002089 0.002089 0.67% BndryData::define() 6 0.001902 0.001902 0.001902 0.61% FabArray::LinComb() 242 0.001857 0.001857 0.001857 0.60% amrex::Add() 72 0.001821 0.001821 0.001821 0.59% Castro::reset_internal_energy(MultiFab) 30 0.001753 0.001753 0.001753 0.56% Castro::construct_new_gravity_source() 5 0.00134 0.00134 0.00134 0.43% Amr::writePlotFile() 1 0.001308 0.001308 0.001308 0.42% Castro::construct_old_gravity_source() 5 0.001045 0.001045 0.001045 0.34% MLCGSolver::bicgstab 36 0.0009433 0.0009433 0.0009433 0.30% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008736 0.0008736 0.0008736 0.28% Castro::do_advance_ctu() 5 0.0008514 0.0008514 0.0008514 0.27% Gravity::actual_solve_with_mlmg() 6 0.000742 0.000742 0.000742 0.24% MLCellLinOp::setLevelBC() 6 0.0007142 0.0007142 0.0007142 0.23% Castro::reset_internal_energy(Fab) 240 0.0006644 0.0006644 0.0006644 0.21% FabArray::mult() 22 0.0006518 0.0006518 0.0006518 0.21% FabArray::setDomainBndry() 20 0.0006367 0.0006367 0.0006367 0.20% MultiFab::contains_nan() 10 0.000586 0.000586 0.000586 0.19% MLCellLinOp::prepareForSolve() 6 0.0005811 0.0005811 0.0005811 0.19% Castro::enforce_speed_limit() 30 0.0005581 0.0005581 0.0005581 0.18% MLCellLinOp::smooth() 720 0.0005297 0.0005297 0.0005297 0.17% MLCellLinOp::compGrad() 6 0.0004817 0.0004817 0.0004817 0.16% MLMG::prepareForSolve() 6 0.0004604 0.0004604 0.0004604 0.15% FabArrayBase::CPC::define() 244 0.00041 0.00041 0.00041 0.13% Amr::InitAmr() 1 0.0003898 0.0003898 0.0003898 0.13% FabArrayBase::getCPC() 632 0.0003426 0.0003426 0.0003426 0.11% FabArray::FillBoundary() 1766 0.0003308 0.0003308 0.0003308 0.11% Gravity::get_old_grav_vector() 5 0.0002918 0.0002918 0.0002918 0.09% Gravity::get_new_grav_vector() 5 0.000266 0.000266 0.000266 0.09% main() 1 0.0002603 0.0002603 0.0002603 0.08% FabArrayBase::getFB() 1766 0.0002466 0.0002466 0.0002466 0.08% MLCellLinOp::apply() 500 0.0002284 0.0002284 0.0002284 0.07% MLMG::mgVcycle() 36 0.0001776 0.0001776 0.0001776 0.06% Amr::coarseTimeStep() 5 0.0001624 0.0001624 0.0001624 0.05% MLLinOp::defineGrids() 6 0.0001476 0.0001476 0.0001476 0.05% MultiFab::max() 6 0.0001349 0.0001349 0.0001349 0.04% MLCGSolver::ParallelAllReduce 659 0.0001267 0.0001267 0.0001267 0.04% Castro::subcycle_advance_ctu() 5 0.0001199 0.0001199 0.0001199 0.04% FillPatchIterator::Initialize 20 0.0001178 0.0001178 0.0001178 0.04% MLCellLinOp::defineBC() 6 0.0001081 0.0001081 0.0001081 0.03% FabArray::ParallelCopy() 380 0.0001054 0.0001054 0.0001054 0.03% MLCellLinOp::correctionResidual() 216 9.375e-05 9.375e-05 9.375e-05 0.03% Amr::timeStep() 5 9.134e-05 9.134e-05 9.134e-05 0.03% AmrLevel::restart() 1 7.482e-05 7.482e-05 7.482e-05 0.02% Gravity::solve_for_phi() 5 6.784e-05 6.784e-05 6.784e-05 0.02% Gravity::update_max_rhs() 6 6.132e-05 6.132e-05 6.132e-05 0.02% StateData::restartDoit() 4 5.977e-05 5.977e-05 5.977e-05 0.02% FabArrayBase::FB::FB() 26 5.677e-05 5.677e-05 5.677e-05 0.02% MLMG:computeResOfCorrection() 180 4.733e-05 4.733e-05 4.733e-05 0.02% MLMG::mgVcycle_down::0 36 4.438e-05 4.438e-05 4.438e-05 0.01% Castro::finalize_advance() 5 4.147e-05 4.147e-05 4.147e-05 0.01% MLMG::mgVcycle_down::1 36 4.095e-05 4.095e-05 4.095e-05 0.01% Castro::clean_state() 30 3.955e-05 3.955e-05 3.955e-05 0.01% MLMG::actualBottomSolve() 36 3.925e-05 3.925e-05 3.925e-05 0.01% Castro::expand_state() 5 3.849e-05 3.849e-05 3.849e-05 0.01% MLMG::mgVcycle_down::2 36 3.785e-05 3.785e-05 3.785e-05 0.01% MLMG::mgVcycle_down::4 36 3.767e-05 3.767e-05 3.767e-05 0.01% MLMG::mgVcycle_down::3 36 3.636e-05 3.636e-05 3.636e-05 0.01% MLMG::solve() 6 3.607e-05 3.607e-05 3.607e-05 0.01% Castro::initialize_advance() 5 3.197e-05 3.197e-05 3.197e-05 0.01% MLMG::mgVcycle_up::4 36 2.974e-05 2.974e-05 2.974e-05 0.01% Castro::buildMetrics() 1 2.725e-05 2.725e-05 2.725e-05 0.01% MLMG::mgVcycle_up::3 36 2.4e-05 2.4e-05 2.4e-05 0.01% MLMG::mgVcycle_up::0 36 2.39e-05 2.39e-05 2.39e-05 0.01% MLMG::oneIter() 36 2.383e-05 2.383e-05 2.383e-05 0.01% Amr::writeSmallPlotFile() 1 2.363e-05 2.363e-05 2.363e-05 0.01% Castro::initialize_do_advance() 5 2.252e-05 2.252e-05 2.252e-05 0.01% Castro::finalize_do_advance() 5 2.233e-05 2.233e-05 2.233e-05 0.01% MLMG::mgVcycle_up::2 36 2.208e-05 2.208e-05 2.208e-05 0.01% Castro::post_restart() 1 2.204e-05 2.204e-05 2.204e-05 0.01% MLMG::mgVcycle_up::1 36 2.175e-05 2.175e-05 2.175e-05 0.01% MLCellLinOp::solutionResidual() 42 2.122e-05 2.122e-05 2.122e-05 0.01% Castro::swap_state_time_levels() 5 2.093e-05 2.093e-05 2.093e-05 0.01% Castro::initMFs() 1 1.975e-05 1.975e-05 1.975e-05 0.01% Gravity::solve_phi_with_mlmg() 6 1.792e-05 1.792e-05 1.792e-05 0.01% MLMG::ResNormInf() 42 1.625e-05 1.625e-05 1.625e-05 0.01% makeSFC 30 1.552e-05 1.552e-05 1.552e-05 0.00% MLPoisson::define() 6 1.492e-05 1.492e-05 1.492e-05 0.00% MLMG::computeResidual() 36 1.462e-05 1.462e-05 1.462e-05 0.00% MLMG::mgVcycle_bottom 36 1.442e-05 1.442e-05 1.442e-05 0.00% FillPatchSingleLevel 20 1.398e-05 1.398e-05 1.398e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.351e-05 1.351e-05 1.351e-05 0.00% Castro::advance() 5 1.262e-05 1.262e-05 1.262e-05 0.00% Castro::construct_new_gravity() 5 1.169e-05 1.169e-05 1.169e-05 0.00% DistributionMapping::Distribute() 31 9.397e-06 9.397e-06 9.397e-06 0.00% Castro::do_new_sources() 5 8.822e-06 8.822e-06 8.822e-06 0.00% Castro::construct_new_source() 25 8.655e-06 8.655e-06 8.655e-06 0.00% Castro::do_old_sources() 5 8.655e-06 8.655e-06 8.655e-06 0.00% Castro::construct_old_source() 25 8.65e-06 8.65e-06 8.65e-06 0.00% Amr::initSubcycle() 1 8.084e-06 8.084e-06 8.084e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.409e-06 7.409e-06 7.409e-06 0.00% Gravity::actual_multilevel_solve() 1 7.13e-06 7.13e-06 7.13e-06 0.00% Castro::check_for_nan() 10 6.77e-06 6.77e-06 6.77e-06 0.00% Castro::apply_source_to_state() 10 5.93e-06 5.93e-06 5.93e-06 0.00% MLLinOp::define() 6 5.891e-06 5.891e-06 5.891e-06 0.00% Castro::post_timestep() 5 5.005e-06 5.005e-06 5.005e-06 0.00% Castro::construct_old_gravity() 5 4.932e-06 4.932e-06 4.932e-06 0.00% Gravity::swapTimeLevels() 5 4.562e-06 4.562e-06 4.562e-06 0.00% MLPoisson::prepareForSolve() 6 4.309e-06 4.309e-06 4.309e-06 0.00% MLMG::computeMLResidual() 6 3.657e-06 3.657e-06 3.657e-06 0.00% Castro::computeNewDt() 5 3.552e-06 3.552e-06 3.552e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.081e-06 3.081e-06 3.081e-06 0.00% MLMG::getGradSolution() 6 2.949e-06 2.949e-06 2.949e-06 0.00% MLMG::MLResNormInf() 6 2.293e-06 2.293e-06 2.293e-06 0.00% Castro::retry_advance_ctu() 5 2.117e-06 2.117e-06 2.117e-06 0.00% MLMG::MLRhsNormInf() 6 2.106e-06 2.106e-06 2.106e-06 0.00% Gravity::set_mass_offset() 6 2.021e-06 2.021e-06 2.021e-06 0.00% Castro::create_source_corrector() 5 1.897e-06 1.897e-06 1.897e-06 0.00% Castro::FluxRegCrseInit 5 1.464e-06 1.464e-06 1.464e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.211e-06 1.211e-06 1.211e-06 0.00% Castro::FluxRegFineAdd() 5 1.107e-06 1.107e-06 1.107e-06 0.00% AmrLevel::AmrLevel() 1 1.054e-06 1.054e-06 1.054e-06 0.00% Amr::init() 1 8.62e-07 8.62e-07 8.62e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3105 0.3105 0.3105 99.99% Amr::coarseTimeStep() 5 0.2301 0.2301 0.2301 74.10% Amr::timeStep() 5 0.2286 0.2286 0.2286 73.62% Castro::advance() 5 0.2263 0.2263 0.2263 72.86% Castro::subcycle_advance_ctu() 5 0.2203 0.2203 0.2203 70.94% Castro::do_advance_ctu() 5 0.2202 0.2202 0.2202 70.90% Castro::construct_new_gravity() 5 0.125 0.125 0.125 40.25% Gravity::solve_phi_with_mlmg() 6 0.1204 0.1204 0.1204 38.77% Gravity::solve_for_phi() 5 0.1176 0.1176 0.1176 37.86% Gravity::actual_solve_with_mlmg() 6 0.1173 0.1173 0.1173 37.77% MLMG::solve() 6 0.1065 0.1065 0.1065 34.30% MLMG::oneIter() 36 0.09989 0.09989 0.09989 32.16% MLMG::mgVcycle() 36 0.09836 0.09836 0.09836 31.67% Castro::construct_ctu_hydro_source() 5 0.0618 0.0618 0.0618 19.90% MLCellLinOp::smooth() 720 0.04866 0.04866 0.04866 15.67% Amr::init() 1 0.04725 0.04725 0.04725 15.22% Amr::restart() 1 0.04725 0.04725 0.04725 15.22% MLCellLinOp::applyBC() 1946 0.04537 0.04537 0.04537 14.61% AmrLevel::restart() 1 0.0414 0.0414 0.0414 13.33% StateData::restartDoit() 4 0.04132 0.04132 0.04132 13.30% VisMF::Read() 3 0.04121 0.04121 0.04121 13.27% Amr::writePlotFile() 1 0.03249 0.03249 0.03249 10.46% VisMF::Write(FabArray) 1 0.03093 0.03093 0.03093 9.96% MLMG::mgVcycle_bottom 36 0.03015 0.03015 0.03015 9.71% MLMG::actualBottomSolve() 36 0.03014 0.03014 0.03014 9.70% MLCGSolver::bicgstab 36 0.02985 0.02985 0.02985 9.61% FillPatchIterator::Initialize 20 0.01939 0.01939 0.01939 6.24% FillPatchSingleLevel 20 0.01864 0.01864 0.01864 6.00% Castro::clean_state() 30 0.01706 0.01706 0.01706 5.49% StateDataPhysBCFunct::() 20 0.01665 0.01665 0.01665 5.36% MLCellLinOp::apply() 500 0.01508 0.01508 0.01508 4.85% MLMG::mgVcycle_down::0 36 0.01414 0.01414 0.01414 4.55% FabArray::FillBoundary() 1766 0.01383 0.01383 0.01383 4.45% MLPoisson::Fsmooth() 1440 0.01353 0.01353 0.01353 4.35% FillBoundary_nowait() 1766 0.0135 0.0135 0.0135 4.35% StateData::FillBoundary(geom) 160 0.01105 0.01105 0.01105 3.56% MLMG::mgVcycle_up::0 36 0.0108 0.0108 0.0108 3.48% MLCellLinOp::correctionResidual() 216 0.009202 0.009202 0.009202 2.96% Castro::initialize_do_advance() 5 0.009048 0.009048 0.009048 2.91% MLPoisson::define() 6 0.008591 0.008591 0.008591 2.77% amrex::Dot() 484 0.0083 0.0083 0.0083 2.67% MLMG:computeResOfCorrection() 180 0.0081 0.0081 0.0081 2.61% Castro::computeTemp() 30 0.007851 0.007851 0.007851 2.53% Castro::construct_old_gravity() 5 0.007352 0.007352 0.007352 2.37% Gravity::get_old_grav_vector() 5 0.007347 0.007347 0.007347 2.37% Gravity::get_new_grav_vector() 5 0.007333 0.007333 0.007333 2.36% amrex::Copy() 463 0.006786 0.006786 0.006786 2.19% MLMG::mgVcycle_down::1 36 0.006544 0.006544 0.006544 2.11% FabArray::ParallelCopy() 380 0.006201 0.006201 0.006201 2.00% FabArray::ParallelCopy_nowait() 380 0.006096 0.006096 0.006096 1.96% Castro::normalize_species() 30 0.006082 0.006082 0.006082 1.96% MLMG::mgVcycle_down::2 36 0.006076 0.006076 0.006076 1.96% FabArray::setVal() 537 0.006028 0.006028 0.006028 1.94% FabArray::norminf() 326 0.005938 0.005938 0.005938 1.91% MLMG::mgVcycle_down::3 36 0.005934 0.005934 0.005934 1.91% MLCellLinOp::defineAuxData() 6 0.005868 0.005868 0.005868 1.89% MLMG::mgVcycle_down::4 36 0.005857 0.005857 0.005857 1.89% Castro::initialize_advance() 5 0.005666 0.005666 0.005666 1.82% Castro::do_new_sources() 5 0.005469 0.005469 0.005469 1.76% Castro::expand_state() 5 0.005356 0.005356 0.005356 1.72% MLCGSolver::ParallelAllReduce 659 0.005011 0.005011 0.005011 1.61% MLMG::addInterpCorrection() 180 0.004881 0.004881 0.004881 1.57% MLMG::mgVcycle_up::4 36 0.004763 0.004763 0.004763 1.53% MLMG::mgVcycle_up::1 36 0.004754 0.004754 0.004754 1.53% MLMG::mgVcycle_up::2 36 0.004642 0.004642 0.004642 1.49% amrex::average_down 180 0.004579 0.004579 0.004579 1.47% MLMG::mgVcycle_up::3 36 0.004529 0.004529 0.004529 1.46% MLPoisson::Fapply() 500 0.004334 0.004334 0.004334 1.40% Castro::do_old_sources() 5 0.003847 0.003847 0.003847 1.24% FabArray::Saxpy() 355 0.003518 0.003518 0.003518 1.13% FabArray::Xpay() 361 0.003398 0.003398 0.003398 1.09% Castro::post_restart() 1 0.003215 0.003215 0.003215 1.04% MLCellLinOp::solutionResidual() 42 0.003127 0.003127 0.003127 1.01% Gravity::multilevel_solve_for_new_phi() 1 0.003096 0.003096 0.003096 1.00% Gravity::actual_multilevel_solve() 1 0.003083 0.003083 0.003083 0.99% Gravity::fill_multipole_BCs() 6 0.002993 0.002993 0.002993 0.96% MLMG::computeResidual() 36 0.002589 0.002589 0.002589 0.83% Castro::enforce_min_density() 30 0.002529 0.002529 0.002529 0.81% MLCellLinOp::defineBC() 6 0.002523 0.002523 0.002523 0.81% Castro::reset_internal_energy(MultiFab) 30 0.002418 0.002418 0.002418 0.78% BndryData::define() 6 0.002415 0.002415 0.002415 0.78% MLMG::prepareForSolve() 6 0.002404 0.002404 0.002404 0.77% Castro::post_timestep() 5 0.002271 0.002271 0.002271 0.73% Castro::estTimeStep() 10 0.002089 0.002089 0.002089 0.67% FabArray::LinComb() 242 0.001857 0.001857 0.001857 0.60% amrex::Add() 72 0.001821 0.001821 0.001821 0.59% Castro::construct_new_source() 25 0.001348 0.001348 0.001348 0.43% Castro::construct_new_gravity_source() 5 0.00134 0.00134 0.00134 0.43% Castro::computeNewDt() 5 0.001307 0.001307 0.001307 0.42% Castro::construct_old_source() 25 0.001054 0.001054 0.001054 0.34% Castro::construct_old_gravity_source() 5 0.001045 0.001045 0.001045 0.34% MLMG::ResNormInf() 42 0.0009231 0.0009231 0.0009231 0.30% Castro::apply_source_to_state() 10 0.000916 0.000916 0.000916 0.29% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008736 0.0008736 0.0008736 0.28% FabArrayBase::getCPC() 632 0.0007526 0.0007526 0.0007526 0.24% MLMG::getGradSolution() 6 0.0007374 0.0007374 0.0007374 0.24% MLCellLinOp::compGrad() 6 0.0007344 0.0007344 0.0007344 0.24% MLCellLinOp::setLevelBC() 6 0.0007142 0.0007142 0.0007142 0.23% Castro::reset_internal_energy(Fab) 240 0.0006644 0.0006644 0.0006644 0.21% FabArray::mult() 22 0.0006518 0.0006518 0.0006518 0.21% FabArray::setDomainBndry() 20 0.0006367 0.0006367 0.0006367 0.20% Castro::check_for_nan() 10 0.0005927 0.0005927 0.0005927 0.19% MultiFab::contains_nan() 10 0.000586 0.000586 0.000586 0.19% MLPoisson::prepareForSolve() 6 0.0005855 0.0005855 0.0005855 0.19% MLCellLinOp::prepareForSolve() 6 0.0005811 0.0005811 0.0005811 0.19% Castro::enforce_speed_limit() 30 0.0005581 0.0005581 0.0005581 0.18% MLMG::computeMLResidual() 6 0.000557 0.000557 0.000557 0.18% Gravity::update_max_rhs() 6 0.0004421 0.0004421 0.0004421 0.14% FabArrayBase::CPC::define() 244 0.00041 0.00041 0.00041 0.13% Amr::InitAmr() 1 0.0003979 0.0003979 0.0003979 0.13% FabArrayBase::getFB() 1766 0.0003033 0.0003033 0.0003033 0.10% Castro::finalize_advance() 5 0.0002974 0.0002974 0.0002974 0.10% Gravity::swapTimeLevels() 5 0.0002243 0.0002243 0.0002243 0.07% MLLinOp::define() 6 0.0001856 0.0001856 0.0001856 0.06% MLLinOp::defineGrids() 6 0.0001797 0.0001797 0.0001797 0.06% MLMG::MLResNormInf() 6 0.0001499 0.0001499 0.0001499 0.05% Castro::buildMetrics() 1 0.0001429 0.0001429 0.0001429 0.05% MultiFab::max() 6 0.0001349 0.0001349 0.0001349 0.04% MLMG::MLRhsNormInf() 6 0.000113 0.000113 0.000113 0.04% FabArrayBase::FB::FB() 26 5.677e-05 5.677e-05 5.677e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 3.087e-05 3.087e-05 3.087e-05 0.01% Amr::writeSmallPlotFile() 1 2.363e-05 2.363e-05 2.363e-05 0.01% makeSFC 30 2.346e-05 2.346e-05 2.346e-05 0.01% Castro::finalize_do_advance() 5 2.233e-05 2.233e-05 2.233e-05 0.01% Castro::swap_state_time_levels() 5 2.093e-05 2.093e-05 2.093e-05 0.01% Castro::initMFs() 1 1.975e-05 1.975e-05 1.975e-05 0.01% DistributionMapping::Distribute() 31 9.397e-06 9.397e-06 9.397e-06 0.00% Amr::initSubcycle() 1 8.084e-06 8.084e-06 8.084e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.535e-06 4.535e-06 4.535e-06 0.00% Castro::retry_advance_ctu() 5 2.117e-06 2.117e-06 2.117e-06 0.00% Gravity::set_mass_offset() 6 2.021e-06 2.021e-06 2.021e-06 0.00% Castro::create_source_corrector() 5 1.897e-06 1.897e-06 1.897e-06 0.00% Castro::FluxRegCrseInit 5 1.464e-06 1.464e-06 1.464e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.211e-06 1.211e-06 1.211e-06 0.00% Castro::FluxRegFineAdd() 5 1.107e-06 1.107e-06 1.107e-06 0.00% AmrLevel::AmrLevel() 1 1.054e-06 1.054e-06 1.054e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-24-gea7e8a507e77) finalized