Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.03-31-g619630d18031) initialized Starting run at 08:17:37 UTC on 2022-03-25. Successfully read inputs file ... Castro git describe: 22.02-15-g933259a8d AMReX git describe: 22.03-31-g619630d18 Microphysics git describe: 22.03-11-g198df193 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.038179797 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022692996 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.11177644 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.048091658 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.048852611 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.054629717 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.058771253 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.074275862 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.060345098 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047568214 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.049035428 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.054652263 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062246184 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.037211646 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.022088115 seconds Ending run at 08:17:38 UTC on 2022-03-25. Run time = 0.849070789 Run time without initialization = 0.730049511 Average number of zones advanced per microsecond: 3.591 Average number of zones advanced per microsecond per rank: 3.591 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8491 ... 0.8491 ... 0.8491 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2033 0.2033 0.2033 23.94% VisMF::Write(FabArray) 11 0.1868 0.1868 0.1868 22.00% MLCellLinOp::applyBC() 4379 0.09146 0.09146 0.09146 10.77% MLPoisson::Fsmooth() 3240 0.05801 0.05801 0.05801 6.83% FabArray::setVal() 1135 0.0263 0.0263 0.0263 3.10% StateData::FillBoundary(geom) 328 0.02289 0.02289 0.02289 2.70% MLCGSolver::bicgstab 81 0.021 0.021 0.021 2.47% MultiFab::Dot() 1100 0.02081 0.02081 0.02081 2.45% FillBoundary_nowait() 3974 0.0166 0.0166 0.0166 1.95% StateDataPhysBCFunct::() 41 0.01626 0.01626 0.01626 1.91% FabArray::ParallelCopy_nowait() 851 0.01567 0.01567 0.01567 1.85% Castro::computeTemp() 63 0.01453 0.01453 0.01453 1.71% Gravity::fill_multipole_BCs() 11 0.0123 0.0123 0.0123 1.45% MultiFab::LinComb() 1566 0.01198 0.01198 0.01198 1.41% MLPoisson::Fapply() 1128 0.01026 0.01026 0.01026 1.21% MLCellLinOp::defineAuxData() 11 0.009869 0.009869 0.009869 1.16% Castro::enforce_min_density() 62 0.009764 0.009764 0.009764 1.15% MLMG::addInterpCorrection() 405 0.006856 0.006856 0.006856 0.81% amrex::average_down 405 0.00644 0.00644 0.00644 0.76% Castro::estTimeStep() 21 0.006062 0.006062 0.006062 0.71% MultiFab::Xpay() 578 0.005902 0.005902 0.005902 0.70% FabArray::setDomainBndry() 41 0.005675 0.005675 0.005675 0.67% Castro::reset_internal_energy() 63 0.005612 0.005612 0.005612 0.66% Castro::expand_state() 10 0.004859 0.004859 0.004859 0.57% Amr::checkPoint() 3 0.004584 0.004584 0.004584 0.54% BndryData::define() 11 0.003529 0.003529 0.003529 0.42% Castro::do_advance_ctu() 10 0.003105 0.003105 0.003105 0.37% Amr::writePlotFile() 2 0.003049 0.003049 0.003049 0.36% Castro::construct_new_gravity_source() 10 0.002942 0.002942 0.002942 0.35% Gravity::get_new_grav_vector() 11 0.002733 0.002733 0.002733 0.32% Castro::normalize_species() 62 0.002559 0.002559 0.002559 0.30% Castro::enforce_speed_limit() 62 0.002558 0.002558 0.002558 0.30% Castro::construct_old_gravity_source() 10 0.002298 0.002298 0.002298 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002235 0.002235 0.002235 0.26% MLCellLinOp::compGrad() 11 0.002002 0.002002 0.002002 0.24% MLMG::ResNormInf() 92 0.001859 0.001859 0.001859 0.22% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.21% Gravity::get_old_grav_vector() 10 0.001747 0.001747 0.001747 0.21% MLMG::oneIter() 81 0.001618 0.001618 0.001618 0.19% MLCellLinOp::setLevelBC() 11 0.001524 0.001524 0.001524 0.18% Gravity::actual_solve_with_mlmg() 11 0.001421 0.001421 0.001421 0.17% MLCellLinOp::prepareForSolve() 11 0.001339 0.001339 0.001339 0.16% FabArray::mult() 43 0.00132 0.00132 0.00132 0.16% Castro::initData() 1 0.001244 0.001244 0.001244 0.15% MLCellLinOp::smooth() 1620 0.00101 0.00101 0.00101 0.12% FabArrayBase::getCPC() 1313 0.0008456 0.0008456 0.0008456 0.10% FabArray::FillBoundary() 3974 0.0008403 0.0008403 0.0008403 0.10% MLMG::prepareForSolve() 11 0.0007603 0.0007603 0.0007603 0.09% FabArrayBase::CPC::define() 454 0.0007193 0.0007193 0.0007193 0.08% FabArrayBase::getFB() 3974 0.0006643 0.0006643 0.0006643 0.08% Gravity::update_max_rhs() 11 0.0005391 0.0005391 0.0005391 0.06% MultiFab::Copy() 11 0.0005253 0.0005253 0.0005253 0.06% Gravity::solve_for_phi() 10 0.0004757 0.0004757 0.0004757 0.06% MLCellLinOp::apply() 1128 0.000465 0.000465 0.000465 0.05% CGSolver::sxay() 1566 0.0004565 0.0004565 0.0004565 0.05% Amr::InitAmr() 1 0.0004263 0.0004263 0.0004263 0.05% MLMG::mgVcycle() 81 0.0004231 0.0004231 0.0004231 0.05% MultiFab::min() 10 0.0003747 0.0003747 0.0003747 0.04% MLCGSolver::ParallelAllReduce 1495 0.000313 0.000313 0.000313 0.04% main() 1 0.000295 0.000295 0.000295 0.03% FabArray::ParallelCopy() 851 0.000281 0.000281 0.000281 0.03% Castro::construct_new_gravity() 10 0.0002675 0.0002675 0.0002675 0.03% MultiFab::max() 11 0.0002567 0.0002567 0.0002567 0.03% FillPatchIterator::Initialize 41 0.0002505 0.0002505 0.0002505 0.03% MLCellLinOp::correctionResidual() 486 0.000226 0.000226 0.000226 0.03% Gravity::actual_multilevel_solve() 1 0.0002182 0.0002182 0.0002182 0.03% Amr::coarseTimeStep() 10 0.0002111 0.0002111 0.0002111 0.02% Amr::timeStep() 10 0.0002073 0.0002073 0.0002073 0.02% Castro::subcycle_advance_ctu() 10 0.0001998 0.0001998 0.0001998 0.02% MLMG::MLRhsNormInf() 11 0.000195 0.000195 0.000195 0.02% MLCellLinOp::defineBC() 11 0.0001924 0.0001924 0.0001924 0.02% MLLinOp::defineGrids() 11 0.0001737 0.0001737 0.0001737 0.02% Amr::defBaseLevel() 1 0.0001467 0.0001467 0.0001467 0.02% MLMG:computeResOfCorrection() 405 0.000143 0.000143 0.000143 0.02% StateData::checkPoint() 12 0.0001414 0.0001414 0.0001414 0.02% MLMG::actualBottomSolve() 81 0.0001079 0.0001079 0.0001079 0.01% MLMG::mgVcycle_down::0 81 9.404e-05 9.404e-05 9.404e-05 0.01% FabArrayBase::FB::FB() 56 8.734e-05 8.734e-05 8.734e-05 0.01% MLMG::mgVcycle_down::1 81 8.34e-05 8.34e-05 8.34e-05 0.01% Castro::advance() 10 8.157e-05 8.157e-05 8.157e-05 0.01% MLMG::mgVcycle_down::2 81 8.136e-05 8.136e-05 8.136e-05 0.01% MLMG::mgVcycle_down::4 81 7.778e-05 7.778e-05 7.778e-05 0.01% Castro::initialize_advance() 10 7.718e-05 7.718e-05 7.718e-05 0.01% MLMG::mgVcycle_down::3 81 7.693e-05 7.693e-05 7.693e-05 0.01% MLMG::solve() 11 7.571e-05 7.571e-05 7.571e-05 0.01% Castro::clean_state() 62 7.328e-05 7.328e-05 7.328e-05 0.01% AmrLevel::checkPoint() 3 7.284e-05 7.284e-05 7.284e-05 0.01% MLMG::mgVcycle_up::4 81 5.787e-05 5.787e-05 5.787e-05 0.01% Castro::construct_new_source() 50 5.734e-05 5.734e-05 5.734e-05 0.01% Castro::initialize_do_advance() 10 5.681e-05 5.681e-05 5.681e-05 0.01% Castro::finalize_advance() 10 5.596e-05 5.596e-05 5.596e-05 0.01% Castro::post_timestep() 10 5.216e-05 5.216e-05 5.216e-05 0.01% MLMG::mgVcycle_up::0 81 5.088e-05 5.088e-05 5.088e-05 0.01% MLMG::mgVcycle_up::1 81 4.996e-05 4.996e-05 4.996e-05 0.01% MLMG::mgVcycle_up::3 81 4.86e-05 4.86e-05 4.86e-05 0.01% MLMG::mgVcycle_up::2 81 4.798e-05 4.798e-05 4.798e-05 0.01% MLCellLinOp::solutionResidual() 92 4.787e-05 4.787e-05 4.787e-05 0.01% StateData::define() 4 4.171e-05 4.171e-05 4.171e-05 0.00% Castro::swap_state_time_levels() 10 3.882e-05 3.882e-05 3.882e-05 0.00% Castro::enforce_consistent_e() 1 3.872e-05 3.872e-05 3.872e-05 0.00% Castro::finalize_do_advance() 10 3.823e-05 3.823e-05 3.823e-05 0.00% MLMG::mgVcycle_bottom 81 3.608e-05 3.608e-05 3.608e-05 0.00% MLMG::computeResidual() 81 3.596e-05 3.596e-05 3.596e-05 0.00% FillPatchSingleLevel 41 2.805e-05 2.805e-05 2.805e-05 0.00% MLLinOp::define() 11 2.773e-05 2.773e-05 2.773e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.761e-05 2.761e-05 2.761e-05 0.00% makeSFC 55 2.65e-05 2.65e-05 2.65e-05 0.00% Amr::writeSmallPlotFile() 1 2.525e-05 2.525e-05 2.525e-05 0.00% MLPoisson::define() 11 2.493e-05 2.493e-05 2.493e-05 0.00% Amr::FinalizeInit() 1 2.055e-05 2.055e-05 2.055e-05 0.00% Castro::construct_old_source() 50 1.947e-05 1.947e-05 1.947e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.732e-05 1.732e-05 1.732e-05 0.00% Castro::do_new_sources() 10 1.613e-05 1.613e-05 1.613e-05 0.00% DistributionMapping::Distribute() 56 1.592e-05 1.592e-05 1.592e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.585e-05 1.585e-05 1.585e-05 0.00% Castro::do_old_sources() 10 1.446e-05 1.446e-05 1.446e-05 0.00% Castro::apply_source_to_state() 20 1.2e-05 1.2e-05 1.2e-05 0.00% Castro::construct_old_gravity() 10 1.07e-05 1.07e-05 1.07e-05 0.00% Gravity::swapTimeLevels() 10 9.442e-06 9.442e-06 9.442e-06 0.00% Amr::initSubcycle() 1 9.378e-06 9.378e-06 9.378e-06 0.00% AmrLevel::AmrLevel(dm) 1 8.954e-06 8.954e-06 8.954e-06 0.00% MLPoisson::prepareForSolve() 11 8.281e-06 8.281e-06 8.281e-06 0.00% Castro::computeNewDt() 9 6.676e-06 6.676e-06 6.676e-06 0.00% MLMG::computeMLResidual() 11 6.505e-06 6.505e-06 6.505e-06 0.00% Amr::InitializeInit() 1 5.882e-06 5.882e-06 5.882e-06 0.00% MLMG::buildFineMask() 11 5.718e-06 5.718e-06 5.718e-06 0.00% AmrLevel::checkPointPost() 3 5.574e-06 5.574e-06 5.574e-06 0.00% MLMG::getGradSolution() 11 5.437e-06 5.437e-06 5.437e-06 0.00% Castro::create_source_corrector() 10 5.396e-06 5.396e-06 5.396e-06 0.00% MLMG::MLResNormInf() 11 4.972e-06 4.972e-06 4.972e-06 0.00% Gravity::set_mass_offset() 11 4.36e-06 4.36e-06 4.36e-06 0.00% Castro::retry_advance_ctu() 10 4.115e-06 4.115e-06 4.115e-06 0.00% Castro::post_init() 1 3.819e-06 3.819e-06 3.819e-06 0.00% Castro::FluxRegFineAdd() 10 3.208e-06 3.208e-06 3.208e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.937e-06 2.937e-06 2.937e-06 0.00% Castro::computeInitialDt() 2 2.574e-06 2.574e-06 2.574e-06 0.00% Amr::init() 1 2.219e-06 2.219e-06 2.219e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.121e-06 2.121e-06 2.121e-06 0.00% AmrLevel::checkPointPre() 3 1.935e-06 1.935e-06 1.935e-06 0.00% Castro::post_regrid() 1 1.256e-06 1.256e-06 1.256e-06 0.00% Amr::initialInit() 1 9.69e-07 9.69e-07 9.69e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8491 0.8491 0.8491 100.00% Amr::coarseTimeStep() 10 0.7077 0.7077 0.7077 83.35% Amr::timeStep() 10 0.5929 0.5929 0.5929 69.82% Castro::advance() 10 0.5849 0.5849 0.5849 68.88% Castro::subcycle_advance_ctu() 10 0.5715 0.5715 0.5715 67.30% Castro::do_advance_ctu() 10 0.5713 0.5713 0.5713 67.28% Gravity::solve_phi_with_mlmg() 11 0.3171 0.3171 0.3171 37.34% Gravity::actual_solve_with_mlmg() 11 0.3045 0.3045 0.3045 35.87% Castro::construct_new_gravity() 10 0.2911 0.2911 0.2911 34.29% MLMG::solve() 11 0.2823 0.2823 0.2823 33.24% Gravity::solve_for_phi() 10 0.2731 0.2731 0.2731 32.16% MLMG::oneIter() 81 0.2668 0.2668 0.2668 31.43% MLMG::mgVcycle() 81 0.2652 0.2652 0.2652 31.24% Castro::construct_ctu_hydro_source() 10 0.2033 0.2033 0.2033 23.94% VisMF::Write(FabArray) 11 0.1868 0.1868 0.1868 22.00% Amr::checkPoint() 3 0.1498 0.1498 0.1498 17.64% AmrLevel::checkPoint() 3 0.1452 0.1452 0.1452 17.10% StateData::checkPoint() 12 0.1451 0.1451 0.1451 17.09% MLCellLinOp::smooth() 1620 0.1403 0.1403 0.1403 16.52% Amr::init() 1 0.1185 0.1185 0.1185 13.95% MLCellLinOp::applyBC() 4379 0.1096 0.1096 0.1096 12.91% MLMG::mgVcycle_bottom 81 0.07563 0.07563 0.07563 8.91% MLMG::actualBottomSolve() 81 0.0756 0.0756 0.0756 8.90% MLCGSolver::bicgstab 81 0.07488 0.07488 0.07488 8.82% MLPoisson::Fsmooth() 3240 0.05801 0.05801 0.05801 6.83% Amr::initialInit() 1 0.05747 0.05747 0.05747 6.77% FillPatchIterator::Initialize 41 0.04928 0.04928 0.04928 5.80% Amr::FinalizeInit() 1 0.04804 0.04804 0.04804 5.66% Castro::post_init() 1 0.04732 0.04732 0.04732 5.57% Amr::writePlotFile() 2 0.0449 0.0449 0.0449 5.29% Gravity::multilevel_solve_for_new_phi() 1 0.04471 0.04471 0.04471 5.27% Gravity::actual_multilevel_solve() 1 0.04469 0.04469 0.04469 5.26% FillPatchSingleLevel 41 0.04335 0.04335 0.04335 5.11% StateDataPhysBCFunct::() 41 0.03915 0.03915 0.03915 4.61% MLCellLinOp::apply() 1128 0.03859 0.03859 0.03859 4.54% MLMG::mgVcycle_down::0 81 0.03682 0.03682 0.03682 4.34% Castro::clean_state() 62 0.03437 0.03437 0.03437 4.05% MLMG::mgVcycle_up::0 81 0.03156 0.03156 0.03156 3.72% FabArray::setVal() 1135 0.0263 0.0263 0.0263 3.10% Castro::initialize_do_advance() 10 0.02383 0.02383 0.02383 2.81% StateData::FillBoundary(geom) 328 0.02289 0.02289 0.02289 2.70% MLCellLinOp::correctionResidual() 486 0.02164 0.02164 0.02164 2.55% MultiFab::Dot() 1100 0.02081 0.02081 0.02081 2.45% Gravity::get_new_grav_vector() 11 0.02021 0.02021 0.02021 2.38% Castro::computeTemp() 63 0.02014 0.02014 0.02014 2.37% MLMG:computeResOfCorrection() 405 0.01873 0.01873 0.01873 2.21% Castro::expand_state() 10 0.01843 0.01843 0.01843 2.17% FabArray::FillBoundary() 3974 0.01819 0.01819 0.01819 2.14% MLMG::mgVcycle_down::1 81 0.01804 0.01804 0.01804 2.12% MLMG::mgVcycle_down::2 81 0.01746 0.01746 0.01746 2.06% FillBoundary_nowait() 3974 0.01735 0.01735 0.01735 2.04% FabArray::ParallelCopy() 851 0.01684 0.01684 0.01684 1.98% MLPoisson::define() 11 0.01679 0.01679 0.01679 1.98% MLMG::mgVcycle_down::3 81 0.0167 0.0167 0.0167 1.97% FabArray::ParallelCopy_nowait() 851 0.01656 0.01656 0.01656 1.95% MLMG::mgVcycle_down::4 81 0.01592 0.01592 0.01592 1.88% Castro::construct_old_gravity() 10 0.01582 0.01582 0.01582 1.86% Gravity::get_old_grav_vector() 10 0.01581 0.01581 0.01581 1.86% MLMG::mgVcycle_up::2 81 0.01361 0.01361 0.01361 1.60% MLMG::mgVcycle_up::1 81 0.01336 0.01336 0.01336 1.57% Castro::initialize_advance() 10 0.01325 0.01325 0.01325 1.56% MLMG::addInterpCorrection() 405 0.0132 0.0132 0.0132 1.56% MLMG::mgVcycle_up::3 81 0.01291 0.01291 0.01291 1.52% MLMG::mgVcycle_up::4 81 0.01277 0.01277 0.01277 1.50% amrex::average_down 405 0.01275 0.01275 0.01275 1.50% MLCGSolver::ParallelAllReduce 1495 0.01254 0.01254 0.01254 1.48% CGSolver::sxay() 1566 0.01244 0.01244 0.01244 1.46% Gravity::fill_multipole_BCs() 11 0.0123 0.0123 0.0123 1.45% MultiFab::LinComb() 1566 0.01198 0.01198 0.01198 1.41% Castro::do_new_sources() 10 0.01153 0.01153 0.01153 1.36% MLCellLinOp::defineAuxData() 11 0.01152 0.01152 0.01152 1.36% MLPoisson::Fapply() 1128 0.01026 0.01026 0.01026 1.21% Castro::enforce_min_density() 62 0.009764 0.009764 0.009764 1.15% Amr::InitializeInit() 1 0.009425 0.009425 0.009425 1.11% Amr::defBaseLevel() 1 0.009419 0.009419 0.009419 1.11% Castro::post_timestep() 10 0.00777 0.00777 0.00777 0.92% Castro::do_old_sources() 10 0.007595 0.007595 0.007595 0.89% MLCellLinOp::solutionResidual() 92 0.007519 0.007519 0.007519 0.89% MLMG::computeResidual() 81 0.00648 0.00648 0.00648 0.76% Castro::estTimeStep() 21 0.006062 0.006062 0.006062 0.71% MultiFab::Xpay() 578 0.005902 0.005902 0.005902 0.70% MLMG::prepareForSolve() 11 0.005736 0.005736 0.005736 0.68% FabArray::setDomainBndry() 41 0.005675 0.005675 0.005675 0.67% Castro::reset_internal_energy() 63 0.005612 0.005612 0.005612 0.66% MLCellLinOp::defineBC() 11 0.004978 0.004978 0.004978 0.59% BndryData::define() 11 0.004786 0.004786 0.004786 0.56% Castro::initData() 1 0.003363 0.003363 0.003363 0.40% Castro::construct_new_source() 50 0.002999 0.002999 0.002999 0.35% Castro::construct_new_gravity_source() 10 0.002942 0.002942 0.002942 0.35% Castro::computeNewDt() 9 0.002639 0.002639 0.002639 0.31% Castro::normalize_species() 62 0.002559 0.002559 0.002559 0.30% Castro::enforce_speed_limit() 62 0.002558 0.002558 0.002558 0.30% MLMG::getGradSolution() 11 0.002539 0.002539 0.002539 0.30% MLCellLinOp::compGrad() 11 0.002534 0.002534 0.002534 0.30% Castro::construct_old_source() 50 0.002317 0.002317 0.002317 0.27% Castro::construct_old_gravity_source() 10 0.002298 0.002298 0.002298 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002235 0.002235 0.002235 0.26% MLMG::ResNormInf() 92 0.001859 0.001859 0.001859 0.22% Castro::apply_source_to_state() 20 0.001823 0.001823 0.001823 0.21% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.21% FabArrayBase::getCPC() 1313 0.001565 0.001565 0.001565 0.18% MLCellLinOp::setLevelBC() 11 0.001524 0.001524 0.001524 0.18% MLPoisson::prepareForSolve() 11 0.001347 0.001347 0.001347 0.16% MLCellLinOp::prepareForSolve() 11 0.001339 0.001339 0.001339 0.16% FabArray::mult() 43 0.00132 0.00132 0.00132 0.16% Gravity::swapTimeLevels() 10 0.001087 0.001087 0.001087 0.13% MLMG::computeMLResidual() 11 0.001082 0.001082 0.001082 0.13% Gravity::update_max_rhs() 11 0.0009363 0.0009363 0.0009363 0.11% FabArrayBase::getFB() 3974 0.0007517 0.0007517 0.0007517 0.09% FabArrayBase::CPC::define() 454 0.0007193 0.0007193 0.0007193 0.08% Castro::computeInitialDt() 2 0.0006741 0.0006741 0.0006741 0.08% MultiFab::Copy() 11 0.0005253 0.0005253 0.0005253 0.06% Castro::post_regrid() 1 0.0004949 0.0004949 0.0004949 0.06% Amr::InitAmr() 1 0.0004357 0.0004357 0.0004357 0.05% MultiFab::min() 10 0.0003747 0.0003747 0.0003747 0.04% MLLinOp::define() 11 0.0002606 0.0002606 0.0002606 0.03% MultiFab::max() 11 0.0002567 0.0002567 0.0002567 0.03% MLMG::MLResNormInf() 11 0.0002536 0.0002536 0.0002536 0.03% MLLinOp::defineGrids() 11 0.0002328 0.0002328 0.0002328 0.03% MLMG::MLRhsNormInf() 11 0.000195 0.000195 0.000195 0.02% FabArrayBase::FB::FB() 56 8.734e-05 8.734e-05 8.734e-05 0.01% Castro::finalize_advance() 10 5.917e-05 5.917e-05 5.917e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.696e-05 5.696e-05 5.696e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.067e-05 5.067e-05 5.067e-05 0.01% StateData::define() 4 4.171e-05 4.171e-05 4.171e-05 0.00% makeSFC 55 4.111e-05 4.111e-05 4.111e-05 0.00% Castro::swap_state_time_levels() 10 3.882e-05 3.882e-05 3.882e-05 0.00% Castro::enforce_consistent_e() 1 3.872e-05 3.872e-05 3.872e-05 0.00% Castro::finalize_do_advance() 10 3.823e-05 3.823e-05 3.823e-05 0.00% Amr::writeSmallPlotFile() 1 2.525e-05 2.525e-05 2.525e-05 0.00% DistributionMapping::Distribute() 56 1.592e-05 1.592e-05 1.592e-05 0.00% Amr::initSubcycle() 1 9.378e-06 9.378e-06 9.378e-06 0.00% MLMG::buildFineMask() 11 5.718e-06 5.718e-06 5.718e-06 0.00% AmrLevel::checkPointPost() 3 5.574e-06 5.574e-06 5.574e-06 0.00% Castro::create_source_corrector() 10 5.396e-06 5.396e-06 5.396e-06 0.00% Gravity::set_mass_offset() 11 4.36e-06 4.36e-06 4.36e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.244e-06 4.244e-06 4.244e-06 0.00% Castro::retry_advance_ctu() 10 4.115e-06 4.115e-06 4.115e-06 0.00% Castro::FluxRegFineAdd() 10 3.208e-06 3.208e-06 3.208e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.121e-06 2.121e-06 2.121e-06 0.00% AmrLevel::checkPointPre() 3 1.935e-06 1.935e-06 1.935e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.03-31-g619630d18031) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.03-31-g619630d18031) initialized Starting run at 08:17:39 UTC on 2022-03-25. Successfully read inputs file ... Castro git describe: 22.02-15-g933259a8d AMReX git describe: 22.03-31-g619630d18 Microphysics git describe: 22.03-11-g198df193 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.515355301 Restart time = 0.055072423 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.114459146 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048304294 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.048108138 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.048745664 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063035889 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.023606776 seconds Ending run at 08:17:39 UTC on 2022-03-25. Run time = 0.402269309 Run time without initialization = 0.346612611 Average number of zones advanced per microsecond: 3.782 Average number of zones advanced per microsecond per rank: 3.782 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4023 ... 0.4023 ... 0.4023 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1185 0.1185 0.1185 29.46% VisMF::Read() 3 0.04333 0.04333 0.04333 10.77% MLCellLinOp::applyBC() 1946 0.0375 0.0375 0.0375 9.32% MLPoisson::Fsmooth() 1440 0.0254 0.0254 0.0254 6.31% VisMF::Write(FabArray) 1 0.02235 0.02235 0.02235 5.56% FabArray::setVal() 537 0.01645 0.01645 0.01645 4.09% StateData::FillBoundary(geom) 160 0.01141 0.01141 0.01141 2.83% MLCGSolver::bicgstab 36 0.009193 0.009193 0.009193 2.28% MultiFab::Dot() 484 0.009064 0.009064 0.009064 2.25% FillBoundary_nowait() 1766 0.00709 0.00709 0.00709 1.76% FabArray::ParallelCopy_nowait() 380 0.007021 0.007021 0.007021 1.75% Castro::computeTemp() 30 0.006856 0.006856 0.006856 1.70% StateDataPhysBCFunct::() 20 0.006846 0.006846 0.006846 1.70% Gravity::fill_multipole_BCs() 6 0.006394 0.006394 0.006394 1.59% MLCellLinOp::defineAuxData() 6 0.005404 0.005404 0.005404 1.34% MultiFab::LinComb() 690 0.005225 0.005225 0.005225 1.30% MLPoisson::Fapply() 500 0.004536 0.004536 0.004536 1.13% Castro::enforce_min_density() 30 0.004329 0.004329 0.004329 1.08% Castro::expand_state() 5 0.004188 0.004188 0.004188 1.04% FabArray::setDomainBndry() 20 0.004118 0.004118 0.004118 1.02% Amr::restart() 1 0.003008 0.003008 0.003008 0.75% MLMG::addInterpCorrection() 180 0.002985 0.002985 0.002985 0.74% Castro::estTimeStep() 10 0.002965 0.002965 0.002965 0.74% Castro::do_advance_ctu() 5 0.002889 0.002889 0.002889 0.72% amrex::average_down 180 0.00277 0.00277 0.00277 0.69% MultiFab::Xpay() 258 0.00262 0.00262 0.00262 0.65% Castro::reset_internal_energy() 30 0.002449 0.002449 0.002449 0.61% BndryData::define() 6 0.001942 0.001942 0.001942 0.48% Gravity::get_new_grav_vector() 5 0.001804 0.001804 0.001804 0.45% Castro::construct_new_gravity_source() 5 0.001533 0.001533 0.001533 0.38% MLCellLinOp::compGrad() 6 0.001379 0.001379 0.001379 0.34% Amr::writePlotFile() 1 0.001344 0.001344 0.001344 0.33% Castro::normalize_species() 30 0.001287 0.001287 0.001287 0.32% Castro::construct_old_gravity_source() 5 0.001238 0.001238 0.001238 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001185 0.001185 0.001185 0.29% Castro::enforce_speed_limit() 30 0.001079 0.001079 0.001079 0.27% MultiFab::Saxpy() 10 0.0009149 0.0009149 0.0009149 0.23% Gravity::get_old_grav_vector() 5 0.0009087 0.0009087 0.0009087 0.23% MLMG::ResNormInf() 42 0.000829 0.000829 0.000829 0.21% MLCellLinOp::setLevelBC() 6 0.0008129 0.0008129 0.0008129 0.20% MLMG::oneIter() 36 0.0007285 0.0007285 0.0007285 0.18% Gravity::actual_solve_with_mlmg() 6 0.0007281 0.0007281 0.0007281 0.18% MLCellLinOp::prepareForSolve() 6 0.0006883 0.0006883 0.0006883 0.17% FabArray::mult() 22 0.0006545 0.0006545 0.0006545 0.16% MLCellLinOp::smooth() 720 0.0005475 0.0005475 0.0005475 0.14% Gravity::update_max_rhs() 6 0.0005364 0.0005364 0.0005364 0.13% MLMG::prepareForSolve() 6 0.0004324 0.0004324 0.0004324 0.11% FabArrayBase::CPC::define() 244 0.0004215 0.0004215 0.0004215 0.10% FabArrayBase::getCPC() 632 0.0004094 0.0004094 0.0004094 0.10% FabArray::FillBoundary() 1766 0.000395 0.000395 0.000395 0.10% MultiFab::Copy() 6 0.0003821 0.0003821 0.0003821 0.09% Amr::InitAmr() 1 0.0003792 0.0003792 0.0003792 0.09% main() 1 0.0003055 0.0003055 0.0003055 0.08% FabArrayBase::getFB() 1766 0.0002973 0.0002973 0.0002973 0.07% Gravity::actual_multilevel_solve() 1 0.0002298 0.0002298 0.0002298 0.06% MLCellLinOp::apply() 500 0.0002294 0.0002294 0.0002294 0.06% Gravity::solve_for_phi() 5 0.0002184 0.0002184 0.0002184 0.05% CGSolver::sxay() 690 0.0002165 0.0002165 0.0002165 0.05% Castro::construct_new_gravity() 5 0.0002051 0.0002051 0.0002051 0.05% MLMG::mgVcycle() 36 0.0001704 0.0001704 0.0001704 0.04% MultiFab::min() 5 0.0001607 0.0001607 0.0001607 0.04% MLCGSolver::ParallelAllReduce 659 0.0001381 0.0001381 0.0001381 0.03% FabArray::ParallelCopy() 380 0.0001378 0.0001378 0.0001378 0.03% MultiFab::max() 6 0.0001323 0.0001323 0.0001323 0.03% Castro::construct_new_source() 25 0.0001284 0.0001284 0.0001284 0.03% FillPatchIterator::Initialize 20 0.0001227 0.0001227 0.0001227 0.03% Amr::coarseTimeStep() 5 0.0001109 0.0001109 0.0001109 0.03% MLCellLinOp::correctionResidual() 216 0.0001085 0.0001085 0.0001085 0.03% MLMG::MLRhsNormInf() 6 0.0001041 0.0001041 0.0001041 0.03% Amr::timeStep() 5 0.0001034 0.0001034 0.0001034 0.03% MLCellLinOp::defineBC() 6 9.915e-05 9.915e-05 9.915e-05 0.02% MLLinOp::defineGrids() 6 9.297e-05 9.297e-05 9.297e-05 0.02% Castro::subcycle_advance_ctu() 5 8.44e-05 8.44e-05 8.44e-05 0.02% StateData::restartDoit() 4 7.403e-05 7.403e-05 7.403e-05 0.02% AmrLevel::restart() 1 7.179e-05 7.179e-05 7.179e-05 0.02% Castro::advance() 5 7.142e-05 7.142e-05 7.142e-05 0.02% MLMG:computeResOfCorrection() 180 6.826e-05 6.826e-05 6.826e-05 0.02% FabArrayBase::FB::FB() 26 6.396e-05 6.396e-05 6.396e-05 0.02% Castro::create_source_corrector() 5 6.065e-05 6.065e-05 6.065e-05 0.02% Castro::initialize_advance() 5 5.812e-05 5.812e-05 5.812e-05 0.01% MLMG::actualBottomSolve() 36 4.993e-05 4.993e-05 4.993e-05 0.01% MLMG::mgVcycle_down::1 36 4.431e-05 4.431e-05 4.431e-05 0.01% MLMG::mgVcycle_down::2 36 4.223e-05 4.223e-05 4.223e-05 0.01% MLMG::mgVcycle_down::0 36 4.216e-05 4.216e-05 4.216e-05 0.01% MLMG::mgVcycle_down::4 36 4.134e-05 4.134e-05 4.134e-05 0.01% MLMG::mgVcycle_down::3 36 4.132e-05 4.132e-05 4.132e-05 0.01% Castro::post_timestep() 5 3.961e-05 3.961e-05 3.961e-05 0.01% Castro::do_new_sources() 5 3.796e-05 3.796e-05 3.796e-05 0.01% Castro::clean_state() 30 3.565e-05 3.565e-05 3.565e-05 0.01% MLMG::solve() 6 3.544e-05 3.544e-05 3.544e-05 0.01% Amr::writeSmallPlotFile() 1 3.091e-05 3.091e-05 3.091e-05 0.01% Castro::initialize_do_advance() 5 3.035e-05 3.035e-05 3.035e-05 0.01% MLMG::mgVcycle_up::4 36 3.014e-05 3.014e-05 3.014e-05 0.01% MLLinOp::define() 6 2.76e-05 2.76e-05 2.76e-05 0.01% Castro::swap_state_time_levels() 5 2.717e-05 2.717e-05 2.717e-05 0.01% MLMG::mgVcycle_up::0 36 2.65e-05 2.65e-05 2.65e-05 0.01% MLMG::mgVcycle_up::3 36 2.641e-05 2.641e-05 2.641e-05 0.01% Castro::finalize_advance() 5 2.604e-05 2.604e-05 2.604e-05 0.01% MLMG::mgVcycle_up::2 36 2.543e-05 2.543e-05 2.543e-05 0.01% MLMG::mgVcycle_up::1 36 2.5e-05 2.5e-05 2.5e-05 0.01% MLCellLinOp::solutionResidual() 42 2.365e-05 2.365e-05 2.365e-05 0.01% Castro::post_restart() 1 2.353e-05 2.353e-05 2.353e-05 0.01% MLPoisson::define() 6 2.232e-05 2.232e-05 2.232e-05 0.01% Castro::finalize_do_advance() 5 1.942e-05 1.942e-05 1.942e-05 0.00% MLMG::computeResidual() 36 1.721e-05 1.721e-05 1.721e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.672e-05 1.672e-05 1.672e-05 0.00% MLMG::mgVcycle_bottom 36 1.648e-05 1.648e-05 1.648e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.632e-05 1.632e-05 1.632e-05 0.00% FillPatchSingleLevel 20 1.472e-05 1.472e-05 1.472e-05 0.00% makeSFC 30 1.401e-05 1.401e-05 1.401e-05 0.00% Castro::construct_old_source() 25 1.218e-05 1.218e-05 1.218e-05 0.00% DistributionMapping::Distribute() 31 9.446e-06 9.446e-06 9.446e-06 0.00% Amr::initSubcycle() 1 8.711e-06 8.711e-06 8.711e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.308e-06 8.308e-06 8.308e-06 0.00% Castro::do_old_sources() 5 8.151e-06 8.151e-06 8.151e-06 0.00% Castro::apply_source_to_state() 10 6.629e-06 6.629e-06 6.629e-06 0.00% Castro::construct_old_gravity() 5 6.323e-06 6.323e-06 6.323e-06 0.00% Gravity::swapTimeLevels() 5 4.907e-06 4.907e-06 4.907e-06 0.00% MLPoisson::prepareForSolve() 6 4.414e-06 4.414e-06 4.414e-06 0.00% Castro::computeNewDt() 5 3.564e-06 3.564e-06 3.564e-06 0.00% MLMG::buildFineMask() 6 3.394e-06 3.394e-06 3.394e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.068e-06 3.068e-06 3.068e-06 0.00% MLMG::computeMLResidual() 6 3.021e-06 3.021e-06 3.021e-06 0.00% MLMG::getGradSolution() 6 2.985e-06 2.985e-06 2.985e-06 0.00% MLMG::MLResNormInf() 6 2.512e-06 2.512e-06 2.512e-06 0.00% Gravity::set_mass_offset() 6 2.375e-06 2.375e-06 2.375e-06 0.00% Castro::FluxRegFineAdd() 5 1.885e-06 1.885e-06 1.885e-06 0.00% Castro::retry_advance_ctu() 5 1.545e-06 1.545e-06 1.545e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.116e-06 1.116e-06 1.116e-06 0.00% Amr::init() 1 1.083e-06 1.083e-06 1.083e-06 0.00% AmrLevel::AmrLevel() 1 8.86e-07 8.86e-07 8.86e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4023 0.4023 0.4023 100.00% Amr::coarseTimeStep() 5 0.3228 0.3228 0.3228 80.22% Amr::timeStep() 5 0.3209 0.3209 0.3209 79.77% Castro::advance() 5 0.3176 0.3176 0.3176 78.95% Castro::subcycle_advance_ctu() 5 0.3073 0.3073 0.3073 76.39% Castro::do_advance_ctu() 5 0.3072 0.3072 0.3072 76.37% Castro::construct_new_gravity() 5 0.1448 0.1448 0.1448 35.98% Gravity::solve_phi_with_mlmg() 6 0.141 0.141 0.141 35.04% Gravity::solve_for_phi() 5 0.1352 0.1352 0.1352 33.61% Gravity::actual_solve_with_mlmg() 6 0.1345 0.1345 0.1345 33.42% MLMG::solve() 6 0.122 0.122 0.122 30.33% Castro::construct_ctu_hydro_source() 5 0.1185 0.1185 0.1185 29.46% MLMG::oneIter() 36 0.1142 0.1142 0.1142 28.39% MLMG::mgVcycle() 36 0.1135 0.1135 0.1135 28.21% MLCellLinOp::smooth() 720 0.05942 0.05942 0.05942 14.77% Amr::init() 1 0.05513 0.05513 0.05513 13.70% Amr::restart() 1 0.05512 0.05512 0.05512 13.70% MLCellLinOp::applyBC() 1946 0.04535 0.04535 0.04535 11.27% AmrLevel::restart() 1 0.04426 0.04426 0.04426 11.00% StateData::restartDoit() 4 0.04418 0.04418 0.04418 10.98% VisMF::Read() 3 0.04333 0.04333 0.04333 10.77% MLMG::mgVcycle_bottom 36 0.0327 0.0327 0.0327 8.13% MLMG::actualBottomSolve() 36 0.03269 0.03269 0.03269 8.12% MLCGSolver::bicgstab 36 0.03237 0.03237 0.03237 8.05% MLPoisson::Fsmooth() 1440 0.0254 0.0254 0.0254 6.31% FillPatchIterator::Initialize 20 0.02457 0.02457 0.02457 6.11% Amr::writePlotFile() 1 0.02369 0.02369 0.02369 5.89% VisMF::Write(FabArray) 1 0.02235 0.02235 0.02235 5.56% FillPatchSingleLevel 20 0.02033 0.02033 0.02033 5.05% StateDataPhysBCFunct::() 20 0.01825 0.01825 0.01825 4.54% FabArray::setVal() 537 0.01645 0.01645 0.01645 4.09% MLCellLinOp::apply() 500 0.01635 0.01635 0.01635 4.06% Castro::clean_state() 30 0.01604 0.01604 0.01604 3.99% MLMG::mgVcycle_down::0 36 0.0158 0.0158 0.0158 3.93% Castro::initialize_do_advance() 5 0.01501 0.01501 0.01501 3.73% MLMG::mgVcycle_up::0 36 0.01349 0.01349 0.01349 3.35% Castro::expand_state() 5 0.01226 0.01226 0.01226 3.05% StateData::FillBoundary(geom) 160 0.01141 0.01141 0.01141 2.83% Castro::initialize_advance() 5 0.0102 0.0102 0.0102 2.54% Gravity::get_new_grav_vector() 5 0.009308 0.009308 0.009308 2.31% Castro::computeTemp() 30 0.009305 0.009305 0.009305 2.31% MLCellLinOp::correctionResidual() 216 0.009229 0.009229 0.009229 2.29% MLPoisson::define() 6 0.009223 0.009223 0.009223 2.29% MultiFab::Dot() 484 0.009064 0.009064 0.009064 2.25% MLMG:computeResOfCorrection() 180 0.008003 0.008003 0.008003 1.99% Castro::construct_old_gravity() 5 0.007882 0.007882 0.007882 1.96% Gravity::get_old_grav_vector() 5 0.007876 0.007876 0.007876 1.96% FabArray::FillBoundary() 1766 0.007846 0.007846 0.007846 1.95% MLMG::mgVcycle_down::1 36 0.007694 0.007694 0.007694 1.91% FabArray::ParallelCopy() 380 0.00757 0.00757 0.00757 1.88% FillBoundary_nowait() 1766 0.007451 0.007451 0.007451 1.85% MLMG::mgVcycle_down::2 36 0.007438 0.007438 0.007438 1.85% FabArray::ParallelCopy_nowait() 380 0.007433 0.007433 0.007433 1.85% MLMG::mgVcycle_down::3 36 0.007075 0.007075 0.007075 1.76% Castro::do_new_sources() 5 0.006889 0.006889 0.006889 1.71% MLMG::mgVcycle_down::4 36 0.00678 0.00678 0.00678 1.69% Castro::post_restart() 1 0.006612 0.006612 0.006612 1.64% Gravity::fill_multipole_BCs() 6 0.006394 0.006394 0.006394 1.59% MLCellLinOp::defineAuxData() 6 0.006302 0.006302 0.006302 1.57% Gravity::multilevel_solve_for_new_phi() 1 0.006219 0.006219 0.006219 1.55% Gravity::actual_multilevel_solve() 1 0.006202 0.006202 0.006202 1.54% MLMG::mgVcycle_up::2 36 0.005767 0.005767 0.005767 1.43% MLMG::addInterpCorrection() 180 0.005735 0.005735 0.005735 1.43% MLMG::mgVcycle_up::1 36 0.005687 0.005687 0.005687 1.41% amrex::average_down 180 0.005526 0.005526 0.005526 1.37% MLCGSolver::ParallelAllReduce 659 0.005504 0.005504 0.005504 1.37% MLMG::mgVcycle_up::3 36 0.005476 0.005476 0.005476 1.36% CGSolver::sxay() 690 0.005442 0.005442 0.005442 1.35% MLMG::mgVcycle_up::4 36 0.005405 0.005405 0.005405 1.34% MultiFab::LinComb() 690 0.005225 0.005225 0.005225 1.30% MLPoisson::Fapply() 500 0.004536 0.004536 0.004536 1.13% Castro::do_old_sources() 5 0.00446 0.00446 0.00446 1.11% Castro::enforce_min_density() 30 0.004329 0.004329 0.004329 1.08% FabArray::setDomainBndry() 20 0.004118 0.004118 0.004118 1.02% MLMG::prepareForSolve() 6 0.003457 0.003457 0.003457 0.86% MLCellLinOp::solutionResidual() 42 0.003366 0.003366 0.003366 0.84% Castro::post_timestep() 5 0.003223 0.003223 0.003223 0.80% Castro::estTimeStep() 10 0.002965 0.002965 0.002965 0.74% MLMG::computeResidual() 36 0.002794 0.002794 0.002794 0.69% MLCellLinOp::defineBC() 6 0.002748 0.002748 0.002748 0.68% BndryData::define() 6 0.002648 0.002648 0.002648 0.66% MultiFab::Xpay() 258 0.00262 0.00262 0.00262 0.65% Castro::reset_internal_energy() 30 0.002449 0.002449 0.002449 0.61% Castro::computeNewDt() 5 0.001697 0.001697 0.001697 0.42% MLMG::getGradSolution() 6 0.001669 0.001669 0.001669 0.41% MLCellLinOp::compGrad() 6 0.001666 0.001666 0.001666 0.41% Castro::construct_new_source() 25 0.001661 0.001661 0.001661 0.41% Castro::construct_new_gravity_source() 5 0.001533 0.001533 0.001533 0.38% Castro::normalize_species() 30 0.001287 0.001287 0.001287 0.32% Castro::construct_old_source() 25 0.00125 0.00125 0.00125 0.31% Castro::construct_old_gravity_source() 5 0.001238 0.001238 0.001238 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001185 0.001185 0.001185 0.29% Castro::enforce_speed_limit() 30 0.001079 0.001079 0.001079 0.27% Castro::apply_source_to_state() 10 0.0009215 0.0009215 0.0009215 0.23% MultiFab::Saxpy() 10 0.0009149 0.0009149 0.0009149 0.23% Gravity::swapTimeLevels() 5 0.0008905 0.0008905 0.0008905 0.22% FabArrayBase::getCPC() 632 0.0008309 0.0008309 0.0008309 0.21% MLMG::ResNormInf() 42 0.000829 0.000829 0.000829 0.21% MLCellLinOp::setLevelBC() 6 0.0008129 0.0008129 0.0008129 0.20% Gravity::update_max_rhs() 6 0.0007454 0.0007454 0.0007454 0.19% MLPoisson::prepareForSolve() 6 0.0006927 0.0006927 0.0006927 0.17% MLCellLinOp::prepareForSolve() 6 0.0006883 0.0006883 0.0006883 0.17% FabArray::mult() 22 0.0006545 0.0006545 0.0006545 0.16% MLMG::computeMLResidual() 6 0.0005923 0.0005923 0.0005923 0.15% FabArrayBase::CPC::define() 244 0.0004215 0.0004215 0.0004215 0.10% Amr::InitAmr() 1 0.0003879 0.0003879 0.0003879 0.10% MultiFab::Copy() 6 0.0003821 0.0003821 0.0003821 0.09% FabArrayBase::getFB() 1766 0.0003612 0.0003612 0.0003612 0.09% MultiFab::min() 5 0.0001607 0.0001607 0.0001607 0.04% MLLinOp::define() 6 0.0001519 0.0001519 0.0001519 0.04% MLMG::MLResNormInf() 6 0.0001326 0.0001326 0.0001326 0.03% MultiFab::max() 6 0.0001323 0.0001323 0.0001323 0.03% MLLinOp::defineGrids() 6 0.0001243 0.0001243 0.0001243 0.03% MLMG::MLRhsNormInf() 6 0.0001041 0.0001041 0.0001041 0.03% FabArrayBase::FB::FB() 26 6.396e-05 6.396e-05 6.396e-05 0.02% Castro::create_source_corrector() 5 6.065e-05 6.065e-05 6.065e-05 0.02% Amr::writeSmallPlotFile() 1 3.091e-05 3.091e-05 3.091e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.019e-05 3.019e-05 3.019e-05 0.01% Castro::finalize_advance() 5 2.792e-05 2.792e-05 2.792e-05 0.01% Castro::swap_state_time_levels() 5 2.717e-05 2.717e-05 2.717e-05 0.01% makeSFC 30 2.188e-05 2.188e-05 2.188e-05 0.01% Castro::finalize_do_advance() 5 1.942e-05 1.942e-05 1.942e-05 0.00% DistributionMapping::Distribute() 31 9.446e-06 9.446e-06 9.446e-06 0.00% Amr::initSubcycle() 1 8.711e-06 8.711e-06 8.711e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.642e-06 4.642e-06 4.642e-06 0.00% MLMG::buildFineMask() 6 3.394e-06 3.394e-06 3.394e-06 0.00% Gravity::set_mass_offset() 6 2.375e-06 2.375e-06 2.375e-06 0.00% Castro::FluxRegFineAdd() 5 1.885e-06 1.885e-06 1.885e-06 0.00% Castro::retry_advance_ctu() 5 1.545e-06 1.545e-06 1.545e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.116e-06 1.116e-06 1.116e-06 0.00% AmrLevel::AmrLevel() 1 8.86e-07 8.86e-07 8.86e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.03-31-g619630d18031) finalized