Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-39-g50466d11e7b2) initialized Starting run at 08:30:25 UTC on 2022-05-02. Successfully read inputs file ... Castro git describe: 22.04-59-gdd7643ce9 AMReX git describe: 22.04-39-g50466d11e Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.038741481 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022209074 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.04598017 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.047964147 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.058537763 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.058682262 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.057729374 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.035152169 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.057116078 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.057378225 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.060773265 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.058098606 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.052229006 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.035525753 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.02223892 seconds Ending run at 08:30:26 UTC on 2022-05-02. Run time = 0.759094618 Run time without initialization = 0.647889183 Average number of zones advanced per microsecond: 4.046 Average number of zones advanced per microsecond per rank: 4.046 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.7591 ... 0.7591 ... 0.7591 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1866 0.1866 0.1866 24.58% VisMF::Write(FabArray) 11 0.1474 0.1474 0.1474 19.42% MLCellLinOp::applyBC() 4433 0.07277 0.07277 0.07277 9.59% MLPoisson::Fsmooth() 3280 0.0592 0.0592 0.0592 7.80% StateData::FillBoundary(geom) 328 0.02355 0.02355 0.02355 3.10% MLCGSolver::bicgstab 82 0.02099 0.02099 0.02099 2.76% MultiFab::Dot() 1114 0.02055 0.02055 0.02055 2.71% Castro::normalize_species() 62 0.0173 0.0173 0.0173 2.28% Castro::computeTemp() 63 0.0154 0.0154 0.0154 2.03% FillBoundary_nowait() 4023 0.01424 0.01424 0.01424 1.88% FabArray::setVal() 1144 0.01309 0.01309 0.01309 1.72% FabArray::ParallelCopy_nowait() 861 0.01304 0.01304 0.01304 1.72% Castro::enforce_min_density() 62 0.01264 0.01264 0.01264 1.67% MultiFab::LinComb() 1586 0.01223 0.01223 0.01223 1.61% StateDataPhysBCFunct::() 41 0.01146 0.01146 0.01146 1.51% MLPoisson::Fapply() 1142 0.01047 0.01047 0.01047 1.38% MLCellLinOp::defineAuxData() 11 0.009646 0.009646 0.009646 1.27% Gravity::fill_multipole_BCs() 11 0.008553 0.008553 0.008553 1.13% Castro::estTimeStep() 21 0.007023 0.007023 0.007023 0.93% MLMG::addInterpCorrection() 410 0.006655 0.006655 0.006655 0.88% amrex::average_down 410 0.006128 0.006128 0.006128 0.81% MultiFab::Xpay() 585 0.00597 0.00597 0.00597 0.79% Castro::reset_internal_energy(MultiFab) 63 0.004945 0.004945 0.004945 0.65% Castro::do_advance_ctu() 10 0.004836 0.004836 0.004836 0.64% Amr::checkPoint() 3 0.004035 0.004035 0.004035 0.53% BndryData::define() 11 0.003584 0.003584 0.003584 0.47% Castro::construct_new_gravity_source() 10 0.003152 0.003152 0.003152 0.42% Amr::writePlotFile() 2 0.002436 0.002436 0.002436 0.32% Castro::construct_old_gravity_source() 10 0.002103 0.002103 0.002103 0.28% Castro::enforce_speed_limit() 62 0.001953 0.001953 0.001953 0.26% Gravity::get_new_grav_vector() 11 0.001918 0.001918 0.001918 0.25% MLMG::ResNormInf() 93 0.001847 0.001847 0.001847 0.24% MultiFab::Saxpy() 20 0.001809 0.001809 0.001809 0.24% Castro::expand_state() 10 0.001731 0.001731 0.001731 0.23% Gravity::get_old_grav_vector() 10 0.001719 0.001719 0.001719 0.23% MLMG::oneIter() 82 0.001677 0.001677 0.001677 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001631 0.001631 0.001631 0.21% Castro::reset_internal_energy(Fab) 504 0.001541 0.001541 0.001541 0.20% Gravity::actual_solve_with_mlmg() 11 0.001356 0.001356 0.001356 0.18% FabArray::mult() 43 0.001314 0.001314 0.001314 0.17% MLCellLinOp::setLevelBC() 11 0.001293 0.001293 0.001293 0.17% FabArray::setDomainBndry() 41 0.001266 0.001266 0.001266 0.17% Castro::initData() 1 0.001205 0.001205 0.001205 0.16% MultiFab::contains_nan() 20 0.001176 0.001176 0.001176 0.15% MLCellLinOp::smooth() 1640 0.001141 0.001141 0.001141 0.15% MLCellLinOp::prepareForSolve() 11 0.001067 0.001067 0.001067 0.14% MLCellLinOp::compGrad() 11 0.0008837 0.0008837 0.0008837 0.12% FabArrayBase::getCPC() 1323 0.0008647 0.0008647 0.0008647 0.11% FabArray::FillBoundary() 4023 0.0008416 0.0008416 0.0008416 0.11% MLMG::prepareForSolve() 11 0.0007821 0.0007821 0.0007821 0.10% FabArrayBase::CPC::define() 454 0.0007165 0.0007165 0.0007165 0.09% FabArrayBase::getFB() 4023 0.0007158 0.0007158 0.0007158 0.09% MLCellLinOp::apply() 1142 0.0004931 0.0004931 0.0004931 0.06% Amr::InitAmr() 1 0.0004542 0.0004542 0.0004542 0.06% CGSolver::sxay() 1586 0.0004195 0.0004195 0.0004195 0.06% Gravity::update_max_rhs() 11 0.0004014 0.0004014 0.0004014 0.05% MLMG::mgVcycle() 82 0.0003949 0.0003949 0.0003949 0.05% MLLinOp::defineGrids() 11 0.0003857 0.0003857 0.0003857 0.05% Gravity::solve_for_phi() 10 0.0003731 0.0003731 0.0003731 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003397 0.0003397 0.0003397 0.04% main() 1 0.0003143 0.0003143 0.0003143 0.04% FabArray::ParallelCopy() 861 0.0002919 0.0002919 0.0002919 0.04% MultiFab::Copy() 11 0.0002614 0.0002614 0.0002614 0.03% MultiFab::max() 11 0.0002536 0.0002536 0.0002536 0.03% FillPatchIterator::Initialize 41 0.0002473 0.0002473 0.0002473 0.03% Castro::construct_new_gravity() 10 0.0002339 0.0002339 0.0002339 0.03% MLCellLinOp::correctionResidual() 492 0.0002335 0.0002335 0.0002335 0.03% Amr::coarseTimeStep() 10 0.0002075 0.0002075 0.0002075 0.03% Castro::subcycle_advance_ctu() 10 0.0002043 0.0002043 0.0002043 0.03% MLMG::MLRhsNormInf() 11 0.0001973 0.0001973 0.0001973 0.03% Amr::timeStep() 10 0.0001922 0.0001922 0.0001922 0.03% MLCellLinOp::defineBC() 11 0.0001861 0.0001861 0.0001861 0.02% MLMG:computeResOfCorrection() 410 0.0001484 0.0001484 0.0001484 0.02% StateData::checkPoint() 12 0.0001328 0.0001328 0.0001328 0.02% Castro::advance() 10 0.0001191 0.0001191 0.0001191 0.02% MLMG::actualBottomSolve() 82 0.000115 0.000115 0.000115 0.02% Castro::Castro() 1 9.545e-05 9.545e-05 9.545e-05 0.01% MLMG::mgVcycle_down::0 82 8.941e-05 8.941e-05 8.941e-05 0.01% FabArrayBase::FB::FB() 56 8.814e-05 8.814e-05 8.814e-05 0.01% MLMG::solve() 11 8.172e-05 8.172e-05 8.172e-05 0.01% MLMG::mgVcycle_down::1 82 7.872e-05 7.872e-05 7.872e-05 0.01% MLMG::mgVcycle_down::2 82 7.681e-05 7.681e-05 7.681e-05 0.01% Castro::initialize_advance() 10 7.53e-05 7.53e-05 7.53e-05 0.01% Castro::clean_state() 62 7.448e-05 7.448e-05 7.448e-05 0.01% MLMG::mgVcycle_down::4 82 7.245e-05 7.245e-05 7.245e-05 0.01% MLMG::mgVcycle_down::3 82 7.099e-05 7.099e-05 7.099e-05 0.01% AmrLevel::checkPoint() 3 7.084e-05 7.084e-05 7.084e-05 0.01% MLMG::mgVcycle_up::4 82 6.015e-05 6.015e-05 6.015e-05 0.01% Castro::initialize_do_advance() 10 5.535e-05 5.535e-05 5.535e-05 0.01% Castro::finalize_advance() 10 5.45e-05 5.45e-05 5.45e-05 0.01% MLCellLinOp::solutionResidual() 93 5.082e-05 5.082e-05 5.082e-05 0.01% MLMG::mgVcycle_up::0 82 4.859e-05 4.859e-05 4.859e-05 0.01% MLMG::mgVcycle_up::3 82 4.785e-05 4.785e-05 4.785e-05 0.01% MLMG::mgVcycle_up::1 82 4.743e-05 4.743e-05 4.743e-05 0.01% MLMG::mgVcycle_up::2 82 4.723e-05 4.723e-05 4.723e-05 0.01% StateData::define() 4 4.377e-05 4.377e-05 4.377e-05 0.01% Gravity::actual_multilevel_solve() 1 4.195e-05 4.195e-05 4.195e-05 0.01% Castro::swap_state_time_levels() 10 3.902e-05 3.902e-05 3.902e-05 0.01% Castro::finalize_do_advance() 10 3.672e-05 3.672e-05 3.672e-05 0.00% FillPatchSingleLevel 41 3.434e-05 3.434e-05 3.434e-05 0.00% Castro::enforce_consistent_e() 1 3.387e-05 3.387e-05 3.387e-05 0.00% MLMG::mgVcycle_bottom 82 3.328e-05 3.328e-05 3.328e-05 0.00% MLMG::computeResidual() 82 3.328e-05 3.328e-05 3.328e-05 0.00% makeSFC 55 2.643e-05 2.643e-05 2.643e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.526e-05 2.526e-05 2.526e-05 0.00% Amr::writeSmallPlotFile() 1 2.443e-05 2.443e-05 2.443e-05 0.00% Castro::initMFs() 1 2.421e-05 2.421e-05 2.421e-05 0.00% MLLinOp::define() 11 2.392e-05 2.392e-05 2.392e-05 0.00% MLPoisson::define() 11 2.235e-05 2.235e-05 2.235e-05 0.00% Castro::buildMetrics() 1 2.112e-05 2.112e-05 2.112e-05 0.00% Amr::FinalizeInit() 1 1.917e-05 1.917e-05 1.917e-05 0.00% Castro::construct_old_source() 50 1.831e-05 1.831e-05 1.831e-05 0.00% Castro::construct_new_source() 50 1.778e-05 1.778e-05 1.778e-05 0.00% Castro::do_new_sources() 10 1.711e-05 1.711e-05 1.711e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.666e-05 1.666e-05 1.666e-05 0.00% Amr::defBaseLevel() 1 1.628e-05 1.628e-05 1.628e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.577e-05 1.577e-05 1.577e-05 0.00% Castro::do_old_sources() 10 1.564e-05 1.564e-05 1.564e-05 0.00% DistributionMapping::Distribute() 56 1.42e-05 1.42e-05 1.42e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.389e-05 1.389e-05 1.389e-05 0.00% Castro::apply_source_to_state() 20 1.127e-05 1.127e-05 1.127e-05 0.00% Castro::check_for_nan() 20 1.124e-05 1.124e-05 1.124e-05 0.00% Castro::construct_old_gravity() 10 1.056e-05 1.056e-05 1.056e-05 0.00% Gravity::swapTimeLevels() 10 9.97e-06 9.97e-06 9.97e-06 0.00% Amr::InitializeInit() 1 9.921e-06 9.921e-06 9.921e-06 0.00% Amr::initSubcycle() 1 9.905e-06 9.905e-06 9.905e-06 0.00% MLPoisson::prepareForSolve() 11 8.512e-06 8.512e-06 8.512e-06 0.00% Castro::post_timestep() 10 7.161e-06 7.161e-06 7.161e-06 0.00% Castro::computeNewDt() 9 6.643e-06 6.643e-06 6.643e-06 0.00% MLMG::computeMLResidual() 11 6.588e-06 6.588e-06 6.588e-06 0.00% MLMG::getGradSolution() 11 5.811e-06 5.811e-06 5.811e-06 0.00% MLMG::buildFineMask() 11 5.477e-06 5.477e-06 5.477e-06 0.00% AmrLevel::checkPointPost() 3 5.353e-06 5.353e-06 5.353e-06 0.00% MLMG::MLResNormInf() 11 5.193e-06 5.193e-06 5.193e-06 0.00% Castro::create_source_corrector() 10 4.276e-06 4.276e-06 4.276e-06 0.00% Gravity::set_mass_offset() 11 4.226e-06 4.226e-06 4.226e-06 0.00% Castro::FluxRegCrseInit 10 3.875e-06 3.875e-06 3.875e-06 0.00% Castro::post_init() 1 3.789e-06 3.789e-06 3.789e-06 0.00% Castro::retry_advance_ctu() 10 3.708e-06 3.708e-06 3.708e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.706e-06 2.706e-06 2.706e-06 0.00% Castro::computeInitialDt() 2 2.589e-06 2.589e-06 2.589e-06 0.00% Castro::FluxRegFineAdd() 10 2.518e-06 2.518e-06 2.518e-06 0.00% Amr::init() 1 2.406e-06 2.406e-06 2.406e-06 0.00% AmrLevel::checkPointPre() 3 2.099e-06 2.099e-06 2.099e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.757e-06 1.757e-06 1.757e-06 0.00% Castro::post_regrid() 1 1.177e-06 1.177e-06 1.177e-06 0.00% Amr::initialInit() 1 1.009e-06 1.009e-06 1.009e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.7591 0.7591 0.7591 100.00% Amr::coarseTimeStep() 10 0.6254 0.6254 0.6254 82.39% Amr::timeStep() 10 0.5505 0.5505 0.5505 72.52% Castro::advance() 10 0.5416 0.5416 0.5416 71.35% Castro::subcycle_advance_ctu() 10 0.5287 0.5287 0.5287 69.65% Castro::do_advance_ctu() 10 0.5285 0.5285 0.5285 69.62% Gravity::solve_phi_with_mlmg() 11 0.288 0.288 0.288 37.94% Gravity::actual_solve_with_mlmg() 11 0.2792 0.2792 0.2792 36.78% Castro::construct_new_gravity() 10 0.2613 0.2613 0.2613 34.42% MLMG::solve() 11 0.259 0.259 0.259 34.12% Gravity::solve_for_phi() 10 0.2461 0.2461 0.2461 32.41% MLMG::oneIter() 82 0.2454 0.2454 0.2454 32.33% MLMG::mgVcycle() 82 0.2437 0.2437 0.2437 32.11% Castro::construct_ctu_hydro_source() 10 0.1866 0.1866 0.1866 24.58% VisMF::Write(FabArray) 11 0.1474 0.1474 0.1474 19.42% MLCellLinOp::smooth() 1640 0.1261 0.1261 0.1261 16.62% Amr::init() 1 0.1106 0.1106 0.1106 14.57% Amr::checkPoint() 3 0.1096 0.1096 0.1096 14.43% AmrLevel::checkPoint() 3 0.1055 0.1055 0.1055 13.90% StateData::checkPoint() 12 0.1054 0.1054 0.1054 13.89% MLCellLinOp::applyBC() 4433 0.08866 0.08866 0.08866 11.68% MLMG::mgVcycle_bottom 82 0.07289 0.07289 0.07289 9.60% MLMG::actualBottomSolve() 82 0.07286 0.07286 0.07286 9.60% MLCGSolver::bicgstab 82 0.07211 0.07211 0.07211 9.50% MLPoisson::Fsmooth() 3280 0.0592 0.0592 0.0592 7.80% Castro::clean_state() 62 0.05314 0.05314 0.05314 7.00% Amr::initialInit() 1 0.0495 0.0495 0.0495 6.52% Amr::FinalizeInit() 1 0.04547 0.04547 0.04547 5.99% Amr::writePlotFile() 2 0.04456 0.04456 0.04456 5.87% Castro::post_init() 1 0.04413 0.04413 0.04413 5.81% Gravity::multilevel_solve_for_new_phi() 1 0.04236 0.04236 0.04236 5.58% Gravity::actual_multilevel_solve() 1 0.04235 0.04235 0.04235 5.58% FillPatchIterator::Initialize 41 0.04052 0.04052 0.04052 5.34% FillPatchSingleLevel 41 0.03901 0.03901 0.03901 5.14% StateDataPhysBCFunct::() 41 0.03501 0.03501 0.03501 4.61% MLMG::mgVcycle_down::0 82 0.03415 0.03415 0.03415 4.50% MLCellLinOp::apply() 1142 0.03338 0.03338 0.03338 4.40% MLMG::mgVcycle_up::0 82 0.02925 0.02925 0.02925 3.85% StateData::FillBoundary(geom) 328 0.02355 0.02355 0.02355 3.10% Castro::computeTemp() 63 0.02189 0.02189 0.02189 2.88% MultiFab::Dot() 1114 0.02055 0.02055 0.02055 2.71% Castro::initialize_do_advance() 10 0.01987 0.01987 0.01987 2.62% MLCellLinOp::correctionResidual() 492 0.01947 0.01947 0.01947 2.57% Castro::normalize_species() 62 0.0173 0.0173 0.0173 2.28% MLMG:computeResOfCorrection() 410 0.0169 0.0169 0.0169 2.23% Gravity::get_new_grav_vector() 11 0.0167 0.0167 0.0167 2.20% MLMG::mgVcycle_down::1 82 0.01629 0.01629 0.01629 2.15% MLPoisson::define() 11 0.0162 0.0162 0.0162 2.13% FabArray::FillBoundary() 4023 0.01588 0.01588 0.01588 2.09% MLMG::mgVcycle_down::2 82 0.0156 0.0156 0.0156 2.06% FillBoundary_nowait() 4023 0.01504 0.01504 0.01504 1.98% MLMG::mgVcycle_down::3 82 0.01478 0.01478 0.01478 1.95% FabArray::ParallelCopy() 861 0.01425 0.01425 0.01425 1.88% Castro::construct_old_gravity() 10 0.0142 0.0142 0.0142 1.87% Gravity::get_old_grav_vector() 10 0.01419 0.01419 0.01419 1.87% MLMG::mgVcycle_down::4 82 0.01402 0.01402 0.01402 1.85% FabArray::ParallelCopy_nowait() 861 0.01395 0.01395 0.01395 1.84% Castro::do_new_sources() 10 0.01314 0.01314 0.01314 1.73% FabArray::setVal() 1144 0.01309 0.01309 0.01309 1.72% Castro::initialize_advance() 10 0.0127 0.0127 0.0127 1.67% CGSolver::sxay() 1586 0.01265 0.01265 0.01265 1.67% Castro::enforce_min_density() 62 0.01264 0.01264 0.01264 1.67% MLCGSolver::ParallelAllReduce 1514 0.01237 0.01237 0.01237 1.63% MultiFab::LinComb() 1586 0.01223 0.01223 0.01223 1.61% MLMG::mgVcycle_up::2 82 0.01203 0.01203 0.01203 1.58% MLMG::mgVcycle_up::1 82 0.01182 0.01182 0.01182 1.56% MLMG::addInterpCorrection() 410 0.01177 0.01177 0.01177 1.55% MLMG::mgVcycle_up::3 82 0.0113 0.0113 0.0113 1.49% amrex::average_down 410 0.01129 0.01129 0.01129 1.49% MLMG::mgVcycle_up::4 82 0.01121 0.01121 0.01121 1.48% Castro::expand_state() 10 0.0111 0.0111 0.0111 1.46% MLCellLinOp::defineAuxData() 11 0.011 0.011 0.011 1.45% Castro::do_old_sources() 10 0.01067 0.01067 0.01067 1.41% MLPoisson::Fapply() 1142 0.01047 0.01047 0.01047 1.38% Castro::post_timestep() 10 0.008719 0.008719 0.008719 1.15% Gravity::fill_multipole_BCs() 11 0.008553 0.008553 0.008553 1.13% Castro::estTimeStep() 21 0.007023 0.007023 0.007023 0.93% MLCellLinOp::solutionResidual() 93 0.006968 0.006968 0.006968 0.92% Castro::reset_internal_energy(MultiFab) 63 0.006486 0.006486 0.006486 0.85% MLMG::computeResidual() 82 0.006012 0.006012 0.006012 0.79% MultiFab::Xpay() 585 0.00597 0.00597 0.00597 0.79% MLCellLinOp::defineBC() 11 0.004715 0.004715 0.004715 0.62% BndryData::define() 11 0.004529 0.004529 0.004529 0.60% MLMG::prepareForSolve() 11 0.004466 0.004466 0.004466 0.59% Amr::InitializeInit() 1 0.004037 0.004037 0.004037 0.53% Amr::defBaseLevel() 1 0.004027 0.004027 0.004027 0.53% Castro::initData() 1 0.003506 0.003506 0.003506 0.46% Castro::computeNewDt() 9 0.003481 0.003481 0.003481 0.46% Castro::construct_new_source() 50 0.003169 0.003169 0.003169 0.42% Castro::construct_new_gravity_source() 10 0.003152 0.003152 0.003152 0.42% Castro::construct_old_source() 50 0.002121 0.002121 0.002121 0.28% Castro::construct_old_gravity_source() 10 0.002103 0.002103 0.002103 0.28% Castro::enforce_speed_limit() 62 0.001953 0.001953 0.001953 0.26% MLMG::ResNormInf() 93 0.001847 0.001847 0.001847 0.24% Castro::apply_source_to_state() 20 0.00182 0.00182 0.00182 0.24% MultiFab::Saxpy() 20 0.001809 0.001809 0.001809 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001631 0.001631 0.001631 0.21% FabArrayBase::getCPC() 1323 0.001581 0.001581 0.001581 0.21% Castro::reset_internal_energy(Fab) 504 0.001541 0.001541 0.001541 0.20% MLMG::getGradSolution() 11 0.001341 0.001341 0.001341 0.18% MLCellLinOp::compGrad() 11 0.001335 0.001335 0.001335 0.18% FabArray::mult() 43 0.001314 0.001314 0.001314 0.17% MLCellLinOp::setLevelBC() 11 0.001293 0.001293 0.001293 0.17% FabArray::setDomainBndry() 41 0.001266 0.001266 0.001266 0.17% Castro::check_for_nan() 20 0.001187 0.001187 0.001187 0.16% MultiFab::contains_nan() 20 0.001176 0.001176 0.001176 0.15% Castro::post_regrid() 1 0.001104 0.001104 0.001104 0.15% MLPoisson::prepareForSolve() 11 0.001076 0.001076 0.001076 0.14% MLCellLinOp::prepareForSolve() 11 0.001067 0.001067 0.001067 0.14% MLMG::computeMLResidual() 11 0.0009953 0.0009953 0.0009953 0.13% FabArrayBase::getFB() 4023 0.0008039 0.0008039 0.0008039 0.11% Gravity::update_max_rhs() 11 0.0007976 0.0007976 0.0007976 0.11% FabArrayBase::CPC::define() 454 0.0007165 0.0007165 0.0007165 0.09% Castro::computeInitialDt() 2 0.0006874 0.0006874 0.0006874 0.09% MLLinOp::define() 11 0.0004647 0.0004647 0.0004647 0.06% Amr::InitAmr() 1 0.0004641 0.0004641 0.0004641 0.06% Castro::Castro() 1 0.0004409 0.0004409 0.0004409 0.06% MLLinOp::defineGrids() 11 0.0004407 0.0004407 0.0004407 0.06% Gravity::swapTimeLevels() 10 0.0004344 0.0004344 0.0004344 0.06% MultiFab::Copy() 11 0.0002614 0.0002614 0.0002614 0.03% MultiFab::max() 11 0.0002536 0.0002536 0.0002536 0.03% MLMG::MLResNormInf() 11 0.0002518 0.0002518 0.0002518 0.03% MLMG::MLRhsNormInf() 11 0.0001973 0.0001973 0.0001973 0.03% Castro::buildMetrics() 1 0.0001577 0.0001577 0.0001577 0.02% FabArrayBase::FB::FB() 56 8.814e-05 8.814e-05 8.814e-05 0.01% Castro::finalize_advance() 10 6.089e-05 6.089e-05 6.089e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.954e-05 5.954e-05 5.954e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.33e-05 5.33e-05 5.33e-05 0.01% StateData::define() 4 4.377e-05 4.377e-05 4.377e-05 0.01% makeSFC 55 3.941e-05 3.941e-05 3.941e-05 0.01% Castro::swap_state_time_levels() 10 3.902e-05 3.902e-05 3.902e-05 0.01% Castro::finalize_do_advance() 10 3.672e-05 3.672e-05 3.672e-05 0.00% Castro::enforce_consistent_e() 1 3.387e-05 3.387e-05 3.387e-05 0.00% Amr::writeSmallPlotFile() 1 2.443e-05 2.443e-05 2.443e-05 0.00% Castro::initMFs() 1 2.421e-05 2.421e-05 2.421e-05 0.00% DistributionMapping::Distribute() 56 1.42e-05 1.42e-05 1.42e-05 0.00% Amr::initSubcycle() 1 9.905e-06 9.905e-06 9.905e-06 0.00% MLMG::buildFineMask() 11 5.477e-06 5.477e-06 5.477e-06 0.00% AmrLevel::checkPointPost() 3 5.353e-06 5.353e-06 5.353e-06 0.00% Castro::create_source_corrector() 10 4.276e-06 4.276e-06 4.276e-06 0.00% Gravity::set_mass_offset() 11 4.226e-06 4.226e-06 4.226e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.931e-06 3.931e-06 3.931e-06 0.00% Castro::FluxRegCrseInit 10 3.875e-06 3.875e-06 3.875e-06 0.00% Castro::retry_advance_ctu() 10 3.708e-06 3.708e-06 3.708e-06 0.00% Castro::FluxRegFineAdd() 10 2.518e-06 2.518e-06 2.518e-06 0.00% AmrLevel::checkPointPre() 3 2.099e-06 2.099e-06 2.099e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.757e-06 1.757e-06 1.757e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-39-g50466d11e7b2) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-39-g50466d11e7b2) initialized Starting run at 08:30:27 UTC on 2022-05-02. Successfully read inputs file ... Castro git describe: 22.04-59-gdd7643ce9 AMReX git describe: 22.04-39-g50466d11e Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.415164628 Restart time = 0.045747608 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.050198918 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.046832488 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.047005262 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.058969986 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063391312 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.058475852 seconds Ending run at 08:30:27 UTC on 2022-05-02. Run time = 0.371504836 Run time without initialization = 0.32523139 Average number of zones advanced per microsecond: 4.030 Average number of zones advanced per microsecond per rank: 4.030 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3715 ... 0.3715 ... 0.3715 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0780 0.0780 0.0780 20.98% VisMF::Read() 3 0.03873 0.03873 0.03873 10.42% Amr::writePlotFile() 1 0.03604 0.03604 0.03604 9.70% MLCellLinOp::applyBC() 1946 0.03137 0.03137 0.03137 8.44% MLPoisson::Fsmooth() 1440 0.02545 0.02545 0.02545 6.85% VisMF::Write(FabArray) 1 0.02252 0.02252 0.02252 6.06% StateData::FillBoundary(geom) 160 0.01148 0.01148 0.01148 3.09% Castro::computeTemp() 30 0.01019 0.01019 0.01019 2.74% MLCGSolver::bicgstab 36 0.008977 0.008977 0.008977 2.42% MultiFab::Dot() 484 0.008762 0.008762 0.008762 2.36% Castro::normalize_species() 30 0.007644 0.007644 0.007644 2.06% FabArray::setVal() 537 0.006242 0.006242 0.006242 1.68% FillBoundary_nowait() 1766 0.006195 0.006195 0.006195 1.67% FabArray::ParallelCopy_nowait() 380 0.005911 0.005911 0.005911 1.59% Castro::enforce_min_density() 30 0.005439 0.005439 0.005439 1.46% MLCellLinOp::defineAuxData() 6 0.005237 0.005237 0.005237 1.41% MultiFab::LinComb() 690 0.005203 0.005203 0.005203 1.40% StateDataPhysBCFunct::() 20 0.00501 0.00501 0.00501 1.35% MLPoisson::Fapply() 500 0.004517 0.004517 0.004517 1.22% Gravity::fill_multipole_BCs() 6 0.004374 0.004374 0.004374 1.18% Castro::estTimeStep() 10 0.003305 0.003305 0.003305 0.89% Amr::restart() 1 0.002981 0.002981 0.002981 0.80% MLMG::addInterpCorrection() 180 0.002863 0.002863 0.002863 0.77% amrex::average_down 180 0.002682 0.002682 0.002682 0.72% MultiFab::Xpay() 258 0.002629 0.002629 0.002629 0.71% Castro::do_advance_ctu() 5 0.002201 0.002201 0.002201 0.59% BndryData::define() 6 0.001977 0.001977 0.001977 0.53% Castro::construct_new_gravity_source() 5 0.001695 0.001695 0.001695 0.46% Castro::reset_internal_energy(MultiFab) 30 0.00164 0.00164 0.00164 0.44% Castro::construct_old_gravity_source() 5 0.001255 0.001255 0.001255 0.34% Castro::enforce_speed_limit() 30 0.00108 0.00108 0.00108 0.29% Castro::reset_internal_energy(Fab) 240 0.0009741 0.0009741 0.0009741 0.26% Gravity::get_old_grav_vector() 5 0.0009739 0.0009739 0.0009739 0.26% Gravity::get_new_grav_vector() 5 0.000941 0.000941 0.000941 0.25% MultiFab::Saxpy() 10 0.0009255 0.0009255 0.0009255 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008747 0.0008747 0.0008747 0.24% Castro::expand_state() 5 0.0008698 0.0008698 0.0008698 0.23% MLMG::ResNormInf() 42 0.0008183 0.0008183 0.0008183 0.22% Castro::subcycle_advance_ctu() 5 0.000817 0.000817 0.000817 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007857 0.0007857 0.0007857 0.21% MLMG::oneIter() 36 0.0007299 0.0007299 0.0007299 0.20% MLCellLinOp::setLevelBC() 6 0.0007041 0.0007041 0.0007041 0.19% FabArray::setDomainBndry() 20 0.0006624 0.0006624 0.0006624 0.18% FabArray::mult() 22 0.0006425 0.0006425 0.0006425 0.17% MultiFab::contains_nan() 10 0.0005853 0.0005853 0.0005853 0.16% MLCellLinOp::prepareForSolve() 6 0.0005691 0.0005691 0.0005691 0.15% MLCellLinOp::compGrad() 6 0.0004824 0.0004824 0.0004824 0.13% MLCellLinOp::smooth() 720 0.000464 0.000464 0.000464 0.12% MLMG::prepareForSolve() 6 0.0004251 0.0004251 0.0004251 0.11% FabArrayBase::CPC::define() 244 0.0004161 0.0004161 0.0004161 0.11% FabArrayBase::getCPC() 632 0.0003925 0.0003925 0.0003925 0.11% FabArray::FillBoundary() 1766 0.0003775 0.0003775 0.0003775 0.10% Amr::InitAmr() 1 0.0003552 0.0003552 0.0003552 0.10% FabArrayBase::getFB() 1766 0.0002806 0.0002806 0.0002806 0.08% main() 1 0.0002705 0.0002705 0.0002705 0.07% Gravity::update_max_rhs() 6 0.0002328 0.0002328 0.0002328 0.06% MLCellLinOp::apply() 500 0.0002145 0.0002145 0.0002145 0.06% Gravity::solve_for_phi() 5 0.000199 0.000199 0.000199 0.05% CGSolver::sxay() 690 0.0001908 0.0001908 0.0001908 0.05% MLLinOp::defineGrids() 6 0.0001818 0.0001818 0.0001818 0.05% MLMG::mgVcycle() 36 0.0001706 0.0001706 0.0001706 0.05% Castro::advance() 5 0.000162 0.000162 0.000162 0.04% Castro::construct_new_source() 25 0.0001555 0.0001555 0.0001555 0.04% MLCGSolver::ParallelAllReduce 659 0.0001462 0.0001462 0.0001462 0.04% Castro::create_source_corrector() 5 0.0001397 0.0001397 0.0001397 0.04% MultiFab::Copy() 6 0.0001395 0.0001395 0.0001395 0.04% Castro::post_timestep() 5 0.0001364 0.0001364 0.0001364 0.04% Castro::construct_new_gravity() 5 0.0001347 0.0001347 0.0001347 0.04% MultiFab::max() 6 0.0001329 0.0001329 0.0001329 0.04% FillPatchIterator::Initialize 20 0.0001322 0.0001322 0.0001322 0.04% Amr::coarseTimeStep() 5 0.0001303 0.0001303 0.0001303 0.04% FabArray::ParallelCopy() 380 0.00013 0.00013 0.00013 0.03% Castro::initialize_advance() 5 0.0001166 0.0001166 0.0001166 0.03% MLCellLinOp::correctionResidual() 216 0.0001056 0.0001056 0.0001056 0.03% MLMG::MLRhsNormInf() 6 0.0001051 0.0001051 0.0001051 0.03% Amr::timeStep() 5 0.0001036 0.0001036 0.0001036 0.03% MLCellLinOp::defineBC() 6 0.0001011 0.0001011 0.0001011 0.03% AmrLevel::restart() 1 7.56e-05 7.56e-05 7.56e-05 0.02% StateData::restartDoit() 4 6.982e-05 6.982e-05 6.982e-05 0.02% Gravity::solve_phi_with_mlmg() 6 6.925e-05 6.925e-05 6.925e-05 0.02% Castro::finalize_advance() 5 6.368e-05 6.368e-05 6.368e-05 0.02% FabArrayBase::FB::FB() 26 6.095e-05 6.095e-05 6.095e-05 0.02% MLMG:computeResOfCorrection() 180 5.682e-05 5.682e-05 5.682e-05 0.02% Castro::initialize_do_advance() 5 5.443e-05 5.443e-05 5.443e-05 0.01% MLMG::actualBottomSolve() 36 5.092e-05 5.092e-05 5.092e-05 0.01% Castro::construct_old_source() 25 4.782e-05 4.782e-05 4.782e-05 0.01% MLMG::mgVcycle_down::0 36 3.846e-05 3.846e-05 3.846e-05 0.01% MLMG::mgVcycle_down::1 36 3.8e-05 3.8e-05 3.8e-05 0.01% Castro::clean_state() 30 3.748e-05 3.748e-05 3.748e-05 0.01% MLMG::solve() 6 3.545e-05 3.545e-05 3.545e-05 0.01% MLMG::mgVcycle_down::2 36 3.495e-05 3.495e-05 3.495e-05 0.01% MLMG::mgVcycle_down::4 36 3.414e-05 3.414e-05 3.414e-05 0.01% Castro::construct_old_gravity() 5 3.41e-05 3.41e-05 3.41e-05 0.01% MLMG::mgVcycle_down::3 36 3.301e-05 3.301e-05 3.301e-05 0.01% Castro::buildMetrics() 1 3.284e-05 3.284e-05 3.284e-05 0.01% Castro::post_restart() 1 3.218e-05 3.218e-05 3.218e-05 0.01% MLLinOp::define() 6 2.918e-05 2.918e-05 2.918e-05 0.01% Gravity::actual_multilevel_solve() 1 2.868e-05 2.868e-05 2.868e-05 0.01% Castro::initMFs() 1 2.675e-05 2.675e-05 2.675e-05 0.01% MLMG::mgVcycle_up::4 36 2.66e-05 2.66e-05 2.66e-05 0.01% Castro::swap_state_time_levels() 5 2.644e-05 2.644e-05 2.644e-05 0.01% Amr::writeSmallPlotFile() 1 2.533e-05 2.533e-05 2.533e-05 0.01% MLCellLinOp::solutionResidual() 42 2.301e-05 2.301e-05 2.301e-05 0.01% MLMG::mgVcycle_up::3 36 2.214e-05 2.214e-05 2.214e-05 0.01% MLMG::mgVcycle_up::0 36 2.193e-05 2.193e-05 2.193e-05 0.01% MLMG::mgVcycle_up::2 36 2.159e-05 2.159e-05 2.159e-05 0.01% MLMG::mgVcycle_up::1 36 2.035e-05 2.035e-05 2.035e-05 0.01% Castro::finalize_do_advance() 5 1.961e-05 1.961e-05 1.961e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.596e-05 1.596e-05 1.596e-05 0.00% MLMG::mgVcycle_bottom 36 1.592e-05 1.592e-05 1.592e-05 0.00% MLMG::computeResidual() 36 1.568e-05 1.568e-05 1.568e-05 0.00% FillPatchSingleLevel 20 1.522e-05 1.522e-05 1.522e-05 0.00% MLPoisson::define() 6 1.475e-05 1.475e-05 1.475e-05 0.00% makeSFC 30 1.428e-05 1.428e-05 1.428e-05 0.00% Castro::do_new_sources() 5 9.666e-06 9.666e-06 9.666e-06 0.00% Amr::initSubcycle() 1 9.227e-06 9.227e-06 9.227e-06 0.00% DistributionMapping::Distribute() 31 9.092e-06 9.092e-06 9.092e-06 0.00% Castro::do_old_sources() 5 8.902e-06 8.902e-06 8.902e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.347e-06 7.347e-06 7.347e-06 0.00% Castro::apply_source_to_state() 10 5.834e-06 5.834e-06 5.834e-06 0.00% Castro::check_for_nan() 10 5.793e-06 5.793e-06 5.793e-06 0.00% Gravity::swapTimeLevels() 5 4.632e-06 4.632e-06 4.632e-06 0.00% MLPoisson::prepareForSolve() 6 4.142e-06 4.142e-06 4.142e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.661e-06 3.661e-06 3.661e-06 0.00% MLMG::computeMLResidual() 6 3.358e-06 3.358e-06 3.358e-06 0.00% Castro::retry_advance_ctu() 5 3.21e-06 3.21e-06 3.21e-06 0.00% MLMG::buildFineMask() 6 3.152e-06 3.152e-06 3.152e-06 0.00% MLMG::getGradSolution() 6 2.743e-06 2.743e-06 2.743e-06 0.00% Castro::computeNewDt() 5 2.738e-06 2.738e-06 2.738e-06 0.00% MLMG::MLResNormInf() 6 2.417e-06 2.417e-06 2.417e-06 0.00% Castro::FluxRegCrseInit 5 2.06e-06 2.06e-06 2.06e-06 0.00% Gravity::set_mass_offset() 6 1.932e-06 1.932e-06 1.932e-06 0.00% Castro::FluxRegFineAdd() 5 1.319e-06 1.319e-06 1.319e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.301e-06 1.301e-06 1.301e-06 0.00% Amr::init() 1 1.007e-06 1.007e-06 1.007e-06 0.00% AmrLevel::AmrLevel() 1 9.82e-07 9.82e-07 9.82e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3715 0.3715 0.3715 100.00% Amr::coarseTimeStep() 5 0.2665 0.2665 0.2665 71.74% Amr::timeStep() 5 0.2646 0.2646 0.2646 71.23% Castro::advance() 5 0.2599 0.2599 0.2599 69.96% Castro::subcycle_advance_ctu() 5 0.2531 0.2531 0.2531 68.13% Castro::do_advance_ctu() 5 0.2523 0.2523 0.2523 67.91% Castro::construct_new_gravity() 5 0.1319 0.1319 0.1319 35.50% Gravity::solve_phi_with_mlmg() 6 0.1276 0.1276 0.1276 34.35% Gravity::solve_for_phi() 5 0.1243 0.1243 0.1243 33.46% Gravity::actual_solve_with_mlmg() 6 0.1231 0.1231 0.1231 33.13% MLMG::solve() 6 0.112 0.112 0.112 30.15% MLMG::oneIter() 36 0.1055 0.1055 0.1055 28.39% MLMG::mgVcycle() 36 0.1048 0.1048 0.1048 28.20% Castro::construct_ctu_hydro_source() 5 0.07796 0.07796 0.07796 20.98% Amr::writePlotFile() 1 0.05856 0.05856 0.05856 15.76% MLCellLinOp::smooth() 720 0.05418 0.05418 0.05418 14.58% Amr::init() 1 0.04578 0.04578 0.04578 12.32% Amr::restart() 1 0.04578 0.04578 0.04578 12.32% AmrLevel::restart() 1 0.03894 0.03894 0.03894 10.48% StateData::restartDoit() 4 0.03886 0.03886 0.03886 10.46% VisMF::Read() 3 0.03873 0.03873 0.03873 10.42% MLCellLinOp::applyBC() 1946 0.03828 0.03828 0.03828 10.30% MLMG::mgVcycle_bottom 36 0.03119 0.03119 0.03119 8.40% MLMG::actualBottomSolve() 36 0.03118 0.03118 0.03118 8.39% MLCGSolver::bicgstab 36 0.03085 0.03085 0.03085 8.30% Castro::clean_state() 30 0.027 0.027 0.027 7.27% MLPoisson::Fsmooth() 1440 0.02545 0.02545 0.02545 6.85% VisMF::Write(FabArray) 1 0.02252 0.02252 0.02252 6.06% FillPatchIterator::Initialize 20 0.01931 0.01931 0.01931 5.20% FillPatchSingleLevel 20 0.01852 0.01852 0.01852 4.98% StateDataPhysBCFunct::() 20 0.01649 0.01649 0.01649 4.44% MLMG::mgVcycle_down::0 36 0.0148 0.0148 0.0148 3.98% MLCellLinOp::apply() 500 0.01449 0.01449 0.01449 3.90% Castro::computeTemp() 30 0.0128 0.0128 0.0128 3.45% MLMG::mgVcycle_up::0 36 0.01254 0.01254 0.01254 3.38% StateData::FillBoundary(geom) 160 0.01148 0.01148 0.01148 3.09% Castro::initialize_do_advance() 5 0.01088 0.01088 0.01088 2.93% MLPoisson::define() 6 0.008846 0.008846 0.008846 2.38% MultiFab::Dot() 484 0.008762 0.008762 0.008762 2.36% MLCellLinOp::correctionResidual() 216 0.008479 0.008479 0.008479 2.28% Castro::normalize_species() 30 0.007644 0.007644 0.007644 2.06% Gravity::get_new_grav_vector() 5 0.007447 0.007447 0.007447 2.00% MLMG:computeResOfCorrection() 180 0.007326 0.007326 0.007326 1.97% Castro::construct_old_gravity() 5 0.00711 0.00711 0.00711 1.91% Gravity::get_old_grav_vector() 5 0.007076 0.007076 0.007076 1.90% MLMG::mgVcycle_down::1 36 0.006983 0.006983 0.006983 1.88% Castro::do_new_sources() 5 0.006932 0.006932 0.006932 1.87% FabArray::FillBoundary() 1766 0.006914 0.006914 0.006914 1.86% Castro::do_old_sources() 5 0.00676 0.00676 0.00676 1.82% MLMG::mgVcycle_down::2 36 0.006709 0.006709 0.006709 1.81% FillBoundary_nowait() 1766 0.006537 0.006537 0.006537 1.76% Castro::initialize_advance() 5 0.006536 0.006536 0.006536 1.76% FabArray::ParallelCopy() 380 0.006449 0.006449 0.006449 1.74% MLMG::mgVcycle_down::3 36 0.006339 0.006339 0.006339 1.71% FabArray::ParallelCopy_nowait() 380 0.006319 0.006319 0.006319 1.70% FabArray::setVal() 537 0.006242 0.006242 0.006242 1.68% MLMG::mgVcycle_down::4 36 0.006066 0.006066 0.006066 1.63% MLCellLinOp::defineAuxData() 6 0.005978 0.005978 0.005978 1.61% Castro::expand_state() 5 0.005643 0.005643 0.005643 1.52% Castro::enforce_min_density() 30 0.005439 0.005439 0.005439 1.46% CGSolver::sxay() 690 0.005394 0.005394 0.005394 1.45% MLCGSolver::ParallelAllReduce 659 0.005277 0.005277 0.005277 1.42% MultiFab::LinComb() 690 0.005203 0.005203 0.005203 1.40% MLMG::mgVcycle_up::2 36 0.005185 0.005185 0.005185 1.40% MLMG::mgVcycle_up::1 36 0.005089 0.005089 0.005089 1.37% MLMG::addInterpCorrection() 180 0.005067 0.005067 0.005067 1.36% amrex::average_down 180 0.004911 0.004911 0.004911 1.32% MLMG::mgVcycle_up::3 36 0.00487 0.00487 0.00487 1.31% MLMG::mgVcycle_up::4 36 0.004809 0.004809 0.004809 1.29% Castro::post_timestep() 5 0.004632 0.004632 0.004632 1.25% MLPoisson::Fapply() 500 0.004517 0.004517 0.004517 1.22% Gravity::fill_multipole_BCs() 6 0.004374 0.004374 0.004374 1.18% Castro::post_restart() 1 0.003675 0.003675 0.003675 0.99% Gravity::multilevel_solve_for_new_phi() 1 0.003548 0.003548 0.003548 0.95% Gravity::actual_multilevel_solve() 1 0.003532 0.003532 0.003532 0.95% Castro::estTimeStep() 10 0.003305 0.003305 0.003305 0.89% MLCellLinOp::solutionResidual() 42 0.00314 0.00314 0.00314 0.85% MultiFab::Xpay() 258 0.002629 0.002629 0.002629 0.71% Castro::reset_internal_energy(MultiFab) 30 0.002614 0.002614 0.002614 0.70% MLCellLinOp::defineBC() 6 0.002612 0.002612 0.002612 0.70% MLMG::computeResidual() 36 0.002598 0.002598 0.002598 0.70% BndryData::define() 6 0.002511 0.002511 0.002511 0.68% MLMG::prepareForSolve() 6 0.002393 0.002393 0.002393 0.64% Castro::construct_new_source() 25 0.001851 0.001851 0.001851 0.50% Castro::computeNewDt() 5 0.001746 0.001746 0.001746 0.47% Castro::construct_new_gravity_source() 5 0.001695 0.001695 0.001695 0.46% Castro::construct_old_source() 25 0.001303 0.001303 0.001303 0.35% Castro::construct_old_gravity_source() 5 0.001255 0.001255 0.001255 0.34% Castro::enforce_speed_limit() 30 0.00108 0.00108 0.00108 0.29% Castro::reset_internal_energy(Fab) 240 0.0009741 0.0009741 0.0009741 0.26% Castro::apply_source_to_state() 10 0.0009313 0.0009313 0.0009313 0.25% MultiFab::Saxpy() 10 0.0009255 0.0009255 0.0009255 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008747 0.0008747 0.0008747 0.24% MLMG::ResNormInf() 42 0.0008183 0.0008183 0.0008183 0.22% FabArrayBase::getCPC() 632 0.0008086 0.0008086 0.0008086 0.22% MLMG::getGradSolution() 6 0.0007377 0.0007377 0.0007377 0.20% MLCellLinOp::compGrad() 6 0.0007349 0.0007349 0.0007349 0.20% MLCellLinOp::setLevelBC() 6 0.0007041 0.0007041 0.0007041 0.19% FabArray::setDomainBndry() 20 0.0006624 0.0006624 0.0006624 0.18% FabArray::mult() 22 0.0006425 0.0006425 0.0006425 0.17% Castro::check_for_nan() 10 0.0005911 0.0005911 0.0005911 0.16% MultiFab::contains_nan() 10 0.0005853 0.0005853 0.0005853 0.16% MLPoisson::prepareForSolve() 6 0.0005732 0.0005732 0.0005732 0.15% MLCellLinOp::prepareForSolve() 6 0.0005691 0.0005691 0.0005691 0.15% MLMG::computeMLResidual() 6 0.0005614 0.0005614 0.0005614 0.15% Gravity::update_max_rhs() 6 0.0004431 0.0004431 0.0004431 0.12% FabArrayBase::CPC::define() 244 0.0004161 0.0004161 0.0004161 0.11% Amr::InitAmr() 1 0.0003645 0.0003645 0.0003645 0.10% FabArrayBase::getFB() 1766 0.0003415 0.0003415 0.0003415 0.09% MLLinOp::define() 6 0.0002417 0.0002417 0.0002417 0.07% Gravity::swapTimeLevels() 5 0.000226 0.000226 0.000226 0.06% MLLinOp::defineGrids() 6 0.0002125 0.0002125 0.0002125 0.06% Castro::buildMetrics() 1 0.0001571 0.0001571 0.0001571 0.04% Castro::create_source_corrector() 5 0.0001397 0.0001397 0.0001397 0.04% MultiFab::Copy() 6 0.0001395 0.0001395 0.0001395 0.04% MLMG::MLResNormInf() 6 0.0001348 0.0001348 0.0001348 0.04% MultiFab::max() 6 0.0001329 0.0001329 0.0001329 0.04% MLMG::MLRhsNormInf() 6 0.0001051 0.0001051 0.0001051 0.03% Castro::finalize_advance() 5 6.706e-05 6.706e-05 6.706e-05 0.02% FabArrayBase::FB::FB() 26 6.095e-05 6.095e-05 6.095e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.942e-05 2.942e-05 2.942e-05 0.01% Castro::initMFs() 1 2.675e-05 2.675e-05 2.675e-05 0.01% Castro::swap_state_time_levels() 5 2.644e-05 2.644e-05 2.644e-05 0.01% Amr::writeSmallPlotFile() 1 2.533e-05 2.533e-05 2.533e-05 0.01% makeSFC 30 2.207e-05 2.207e-05 2.207e-05 0.01% Castro::finalize_do_advance() 5 1.961e-05 1.961e-05 1.961e-05 0.01% Amr::initSubcycle() 1 9.227e-06 9.227e-06 9.227e-06 0.00% DistributionMapping::Distribute() 31 9.092e-06 9.092e-06 9.092e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.96e-06 4.96e-06 4.96e-06 0.00% Castro::retry_advance_ctu() 5 3.21e-06 3.21e-06 3.21e-06 0.00% MLMG::buildFineMask() 6 3.152e-06 3.152e-06 3.152e-06 0.00% Castro::FluxRegCrseInit 5 2.06e-06 2.06e-06 2.06e-06 0.00% Gravity::set_mass_offset() 6 1.932e-06 1.932e-06 1.932e-06 0.00% Castro::FluxRegFineAdd() 5 1.319e-06 1.319e-06 1.319e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.301e-06 1.301e-06 1.301e-06 0.00% AmrLevel::AmrLevel() 1 9.82e-07 9.82e-07 9.82e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-39-g50466d11e7b2) finalized