Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-25-g735c3513153f) initialized Starting run at 08:22:13 UTC on 2022-10-31. Successfully read inputs file ... Castro git describe: 22.09-8-g4d18c91ff AMReX git describe: 22.10-25-g735c35131 Microphysics git describe: 22.10-7-gced9a15b reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.051704643 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.029782259 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.052958676 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.056259966 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.057554643 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.059293395 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.075419252 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.05061227 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.061800134 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.058978174 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058259326 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060099477 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063244707 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.075274145 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029727733 seconds Ending run at 08:22:14 UTC on 2022-10-31. Run time = 0.89397008 Run time without initialization = 0.760092984 Average number of zones advanced per microsecond: 3.449 Average number of zones advanced per microsecond per rank: 3.449 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.894 ... 0.894 ... 0.894 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2203 0.2203 0.2203 24.64% VisMF::Write(FabArray) 11 0.201 0.201 0.201 22.49% MLCellLinOp::applyBC() 4433 0.07873 0.07873 0.07873 8.81% MLPoisson::Fsmooth() 3280 0.06263 0.06263 0.06263 7.01% Amr::checkPoint() 3 0.03323 0.03323 0.03323 3.72% MLCGSolver::bicgstab 82 0.0233 0.0233 0.0233 2.61% StateData::FillBoundary(geom) 328 0.02294 0.02294 0.02294 2.57% MultiFab::Dot() 1114 0.0218 0.0218 0.0218 2.44% StateDataPhysBCFunct::() 41 0.01618 0.01618 0.01618 1.81% Castro::normalize_species() 62 0.01585 0.01585 0.01585 1.77% Castro::computeTemp() 63 0.01427 0.01427 0.01427 1.60% FillBoundary_nowait() 4023 0.01404 0.01404 0.01404 1.57% MultiFab::LinComb() 1586 0.01397 0.01397 0.01397 1.56% FabArray::setVal() 1144 0.01392 0.01392 0.01392 1.56% FabArray::ParallelCopy_nowait() 861 0.01274 0.01274 0.01274 1.42% MLPoisson::Fapply() 1142 0.01155 0.01155 0.01155 1.29% MLCellLinOp::defineAuxData() 11 0.01117 0.01117 0.01117 1.25% Castro::enforce_min_density() 62 0.01088 0.01088 0.01088 1.22% Gravity::fill_multipole_BCs() 11 0.008315 0.008315 0.008315 0.93% MLMG::addInterpCorrection() 410 0.007583 0.007583 0.007583 0.85% amrex::average_down 410 0.006664 0.006664 0.006664 0.75% MultiFab::Xpay() 585 0.006401 0.006401 0.006401 0.72% Castro::do_advance_ctu() 10 0.004688 0.004688 0.004688 0.52% Castro::estTimeStep() 21 0.004385 0.004385 0.004385 0.49% Castro::reset_internal_energy(MultiFab) 63 0.004345 0.004345 0.004345 0.49% BndryData::define() 11 0.003759 0.003759 0.003759 0.42% Castro::construct_new_gravity_source() 10 0.00334 0.00334 0.00334 0.37% Amr::writePlotFile() 2 0.002832 0.002832 0.002832 0.32% Castro::construct_old_gravity_source() 10 0.002667 0.002667 0.002667 0.30% MLMG::ResNormInf() 93 0.002024 0.002024 0.002024 0.23% Gravity::get_new_grav_vector() 11 0.001931 0.001931 0.001931 0.22% MultiFab::Saxpy() 20 0.001804 0.001804 0.001804 0.20% Castro::expand_state() 10 0.001726 0.001726 0.001726 0.19% Gravity::get_old_grav_vector() 10 0.001716 0.001716 0.001716 0.19% MultiFab::Add() 82 0.001612 0.001612 0.001612 0.18% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001608 0.001608 0.001608 0.18% Castro::reset_internal_energy(Fab) 504 0.001556 0.001556 0.001556 0.17% MLCellLinOp::setLevelBC() 11 0.0015 0.0015 0.0015 0.17% Castro::enforce_speed_limit() 62 0.001484 0.001484 0.001484 0.17% Gravity::actual_solve_with_mlmg() 11 0.001464 0.001464 0.001464 0.16% FabArray::mult() 43 0.001323 0.001323 0.001323 0.15% FabArray::setDomainBndry() 41 0.001287 0.001287 0.001287 0.14% Castro::initData() 1 0.001268 0.001268 0.001268 0.14% MLMG::prepareForSolve() 11 0.001222 0.001222 0.001222 0.14% MultiFab::contains_nan() 20 0.001181 0.001181 0.001181 0.13% MLCellLinOp::prepareForSolve() 11 0.001163 0.001163 0.001163 0.13% MLCellLinOp::smooth() 1640 0.001078 0.001078 0.001078 0.12% MLCellLinOp::compGrad() 11 0.0009079 0.0009079 0.0009079 0.10% Castro::subcycle_advance_ctu() 10 0.0009004 0.0009004 0.0009004 0.10% FabArray::FillBoundary() 4023 0.0007914 0.0007914 0.0007914 0.09% FabArrayBase::getCPC() 1323 0.0007499 0.0007499 0.0007499 0.08% FabArrayBase::CPC::define() 454 0.0006547 0.0006547 0.0006547 0.07% FabArrayBase::getFB() 4023 0.000598 0.000598 0.000598 0.07% Castro::finalize_advance() 10 0.0005701 0.0005701 0.0005701 0.06% Gravity::solve_for_phi() 10 0.0004597 0.0004597 0.0004597 0.05% MLCellLinOp::apply() 1142 0.0004582 0.0004582 0.0004582 0.05% Amr::InitAmr() 1 0.0004556 0.0004556 0.0004556 0.05% Gravity::update_max_rhs() 11 0.0004218 0.0004218 0.0004218 0.05% CGSolver::sxay() 1586 0.0003776 0.0003776 0.0003776 0.04% Amr::coarseTimeStep() 10 0.0003211 0.0003211 0.0003211 0.04% MultiFab::Copy() 11 0.0003119 0.0003119 0.0003119 0.03% MLCGSolver::ParallelAllReduce 1514 0.0002964 0.0002964 0.0002964 0.03% FillPatchIterator::Initialize 41 0.0002828 0.0002828 0.0002828 0.03% MLCellLinOp::defineBC() 11 0.0002813 0.0002813 0.0002813 0.03% main() 1 0.000258 0.000258 0.000258 0.03% MultiFab::max() 11 0.0002554 0.0002554 0.0002554 0.03% FabArray::ParallelCopy() 861 0.00025 0.00025 0.00025 0.03% MLCellLinOp::correctionResidual() 492 0.00023 0.00023 0.00023 0.03% Castro::construct_new_gravity() 10 0.0002175 0.0002175 0.0002175 0.02% MLMG::mgVcycle() 82 0.0002162 0.0002162 0.0002162 0.02% MLMG::MLRhsNormInf() 11 0.0002123 0.0002123 0.0002123 0.02% MLLinOp::defineGrids() 11 0.0001739 0.0001739 0.0001739 0.02% Amr::timeStep() 10 0.0001566 0.0001566 0.0001566 0.02% MLMG:computeResOfCorrection() 410 0.0001527 0.0001527 0.0001527 0.02% StateData::checkPoint() 12 0.0001501 0.0001501 0.0001501 0.02% MLMG::mgVcycle_down::0 82 0.0001224 0.0001224 0.0001224 0.01% AmrLevel::checkPoint() 3 0.0001092 0.0001092 0.0001092 0.01% MLMG::mgVcycle_down::1 82 0.0001055 0.0001055 0.0001055 0.01% MLMG::mgVcycle_down::2 82 0.0001018 0.0001018 0.0001018 0.01% MLMG::mgVcycle_down::3 82 9.587e-05 9.587e-05 9.587e-05 0.01% MLMG::mgVcycle_down::4 82 9.454e-05 9.454e-05 9.454e-05 0.01% Castro::initialize_advance() 10 8.58e-05 8.58e-05 8.58e-05 0.01% MLMG::actualBottomSolve() 82 8.556e-05 8.556e-05 8.556e-05 0.01% Castro::Castro() 1 8.525e-05 8.525e-05 8.525e-05 0.01% FabArrayBase::FB::FB() 56 8.245e-05 8.245e-05 8.245e-05 0.01% Castro::clean_state() 62 8.24e-05 8.24e-05 8.24e-05 0.01% MLMG::mgVcycle_up::4 82 7.685e-05 7.685e-05 7.685e-05 0.01% MLMG::solve() 11 6.992e-05 6.992e-05 6.992e-05 0.01% Castro::advance() 10 6.778e-05 6.778e-05 6.778e-05 0.01% MLMG::oneIter() 82 6.481e-05 6.481e-05 6.481e-05 0.01% Castro::initialize_do_advance() 10 6.366e-05 6.366e-05 6.366e-05 0.01% MLMG::mgVcycle_up::0 82 6.063e-05 6.063e-05 6.063e-05 0.01% MLMG::mgVcycle_up::3 82 5.751e-05 5.751e-05 5.751e-05 0.01% MLMG::mgVcycle_up::1 82 5.651e-05 5.651e-05 5.651e-05 0.01% MLMG::mgVcycle_up::2 82 5.431e-05 5.431e-05 5.431e-05 0.01% MLCellLinOp::solutionResidual() 93 5.139e-05 5.139e-05 5.139e-05 0.01% StateData::define() 4 4.037e-05 4.037e-05 4.037e-05 0.00% Castro::swap_state_time_levels() 10 3.882e-05 3.882e-05 3.882e-05 0.00% MLMG::computeResidual() 82 3.711e-05 3.711e-05 3.711e-05 0.00% Castro::finalize_do_advance() 10 3.253e-05 3.253e-05 3.253e-05 0.00% Castro::enforce_consistent_e() 1 3.249e-05 3.249e-05 3.249e-05 0.00% MLMG::mgVcycle_bottom 82 3.226e-05 3.226e-05 3.226e-05 0.00% MLPoisson::define() 11 3.113e-05 3.113e-05 3.113e-05 0.00% Gravity::actual_multilevel_solve() 1 3.11e-05 3.11e-05 3.11e-05 0.00% FillPatchSingleLevel 41 2.739e-05 2.739e-05 2.739e-05 0.00% Castro::initMFs() 1 2.576e-05 2.576e-05 2.576e-05 0.00% makeSFC 55 2.517e-05 2.517e-05 2.517e-05 0.00% Amr::writeSmallPlotFile() 1 2.506e-05 2.506e-05 2.506e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.465e-05 2.465e-05 2.465e-05 0.00% Castro::buildMetrics() 1 2.228e-05 2.228e-05 2.228e-05 0.00% MLLinOp::define() 11 2.204e-05 2.204e-05 2.204e-05 0.00% Amr::defBaseLevel() 1 2.085e-05 2.085e-05 2.085e-05 0.00% Amr::FinalizeInit() 1 1.981e-05 1.981e-05 1.981e-05 0.00% Castro::construct_old_source() 50 1.974e-05 1.974e-05 1.974e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.849e-05 1.849e-05 1.849e-05 0.00% DistributionMapping::Distribute() 56 1.585e-05 1.585e-05 1.585e-05 0.00% Castro::do_new_sources() 10 1.582e-05 1.582e-05 1.582e-05 0.00% Castro::do_old_sources() 10 1.57e-05 1.57e-05 1.57e-05 0.00% Castro::construct_new_source() 50 1.561e-05 1.561e-05 1.561e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.54e-05 1.54e-05 1.54e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.298e-05 1.298e-05 1.298e-05 0.00% Castro::check_for_nan() 20 1.25e-05 1.25e-05 1.25e-05 0.00% Castro::apply_source_to_state() 20 1.128e-05 1.128e-05 1.128e-05 0.00% Gravity::swapTimeLevels() 10 1.035e-05 1.035e-05 1.035e-05 0.00% Castro::construct_old_gravity() 10 1.028e-05 1.028e-05 1.028e-05 0.00% MLMG::computeMLResidual() 11 1.013e-05 1.013e-05 1.013e-05 0.00% Castro::post_timestep() 10 9.776e-06 9.776e-06 9.776e-06 0.00% MLPoisson::prepareForSolve() 11 8.7e-06 8.7e-06 8.7e-06 0.00% Amr::initSubcycle() 1 8.058e-06 8.058e-06 8.058e-06 0.00% MLMG::getGradSolution() 11 6.778e-06 6.778e-06 6.778e-06 0.00% Castro::computeNewDt() 9 6.474e-06 6.474e-06 6.474e-06 0.00% AmrLevel::checkPointPost() 3 5.792e-06 5.792e-06 5.792e-06 0.00% Amr::InitializeInit() 1 4.919e-06 4.919e-06 4.919e-06 0.00% AmrLevel::checkPointPre() 3 4.362e-06 4.362e-06 4.362e-06 0.00% Gravity::set_mass_offset() 11 3.923e-06 3.923e-06 3.923e-06 0.00% Castro::retry_advance_ctu() 10 3.896e-06 3.896e-06 3.896e-06 0.00% MLMG::MLResNormInf() 11 3.699e-06 3.699e-06 3.699e-06 0.00% Castro::create_source_corrector() 10 3.561e-06 3.561e-06 3.561e-06 0.00% Castro::post_init() 1 3.429e-06 3.429e-06 3.429e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.736e-06 2.736e-06 2.736e-06 0.00% Castro::FluxRegCrseInit 10 2.694e-06 2.694e-06 2.694e-06 0.00% Castro::computeInitialDt() 2 2.454e-06 2.454e-06 2.454e-06 0.00% Amr::init() 1 2.379e-06 2.379e-06 2.379e-06 0.00% Castro::FluxRegFineAdd() 10 2.025e-06 2.025e-06 2.025e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.87e-06 1.87e-06 1.87e-06 0.00% Amr::initialInit() 1 1.224e-06 1.224e-06 1.224e-06 0.00% Castro::post_regrid() 1 1.209e-06 1.209e-06 1.209e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.894 0.894 0.894 100.00% Amr::coarseTimeStep() 10 0.7301 0.7301 0.7301 81.67% Amr::timeStep() 10 0.6017 0.6017 0.6017 67.30% Castro::advance() 10 0.5934 0.5934 0.5934 66.38% Castro::subcycle_advance_ctu() 10 0.5809 0.5809 0.5809 64.98% Castro::do_advance_ctu() 10 0.58 0.58 0.58 64.88% Gravity::solve_phi_with_mlmg() 11 0.3079 0.3079 0.3079 34.44% Gravity::actual_solve_with_mlmg() 11 0.2994 0.2994 0.2994 33.49% Castro::construct_new_gravity() 10 0.2806 0.2806 0.2806 31.39% MLMG::solve() 11 0.2773 0.2773 0.2773 31.02% Gravity::solve_for_phi() 10 0.2645 0.2645 0.2645 29.59% MLMG::oneIter() 82 0.2626 0.2626 0.2626 29.37% MLMG::mgVcycle() 82 0.2609 0.2609 0.2609 29.19% Castro::construct_ctu_hydro_source() 10 0.2203 0.2203 0.2203 24.64% VisMF::Write(FabArray) 11 0.201 0.201 0.201 22.49% Amr::checkPoint() 3 0.1777 0.1777 0.1777 19.88% AmrLevel::checkPoint() 3 0.1445 0.1445 0.1445 16.16% StateData::checkPoint() 12 0.1444 0.1444 0.1444 16.15% MLCellLinOp::smooth() 1640 0.1338 0.1338 0.1338 14.97% Amr::init() 1 0.1333 0.1333 0.1333 14.91% MLCellLinOp::applyBC() 4433 0.09424 0.09424 0.09424 10.54% MLMG::mgVcycle_bottom 82 0.07993 0.07993 0.07993 8.94% MLMG::actualBottomSolve() 82 0.0799 0.0799 0.0799 8.94% MLCGSolver::bicgstab 82 0.07911 0.07911 0.07911 8.85% MLPoisson::Fsmooth() 3280 0.06263 0.06263 0.06263 7.01% Amr::writePlotFile() 2 0.05963 0.05963 0.05963 6.67% Amr::initialInit() 1 0.05171 0.05171 0.05171 5.78% Castro::clean_state() 62 0.04757 0.04757 0.04757 5.32% Amr::FinalizeInit() 1 0.04739 0.04739 0.04739 5.30% Castro::post_init() 1 0.04606 0.04606 0.04606 5.15% FillPatchIterator::Initialize 41 0.04463 0.04463 0.04463 4.99% Gravity::multilevel_solve_for_new_phi() 1 0.04389 0.04389 0.04389 4.91% Gravity::actual_multilevel_solve() 1 0.04387 0.04387 0.04387 4.91% FillPatchSingleLevel 41 0.04306 0.04306 0.04306 4.82% StateDataPhysBCFunct::() 41 0.03913 0.03913 0.03913 4.38% MLCellLinOp::apply() 1142 0.03565 0.03565 0.03565 3.99% MLMG::mgVcycle_down::0 82 0.03482 0.03482 0.03482 3.89% MLMG::mgVcycle_up::0 82 0.02987 0.02987 0.02987 3.34% StateData::FillBoundary(geom) 328 0.02294 0.02294 0.02294 2.57% MultiFab::Dot() 1114 0.0218 0.0218 0.0218 2.44% MLCellLinOp::correctionResidual() 492 0.02088 0.02088 0.02088 2.34% Castro::initialize_do_advance() 10 0.02022 0.02022 0.02022 2.26% Castro::computeTemp() 63 0.02017 0.02017 0.02017 2.26% MLMG:computeResOfCorrection() 410 0.01804 0.01804 0.01804 2.02% Gravity::get_new_grav_vector() 11 0.01796 0.01796 0.01796 2.01% MLPoisson::define() 11 0.01771 0.01771 0.01771 1.98% MLMG::mgVcycle_down::1 82 0.01741 0.01741 0.01741 1.95% MLMG::mgVcycle_down::2 82 0.01691 0.01691 0.01691 1.89% MLMG::mgVcycle_down::3 82 0.0161 0.0161 0.0161 1.80% Castro::normalize_species() 62 0.01585 0.01585 0.01585 1.77% FabArray::FillBoundary() 4023 0.01551 0.01551 0.01551 1.73% Castro::construct_old_gravity() 10 0.01545 0.01545 0.01545 1.73% Gravity::get_old_grav_vector() 10 0.01544 0.01544 0.01544 1.73% MLMG::mgVcycle_down::4 82 0.01529 0.01529 0.01529 1.71% FillBoundary_nowait() 4023 0.01472 0.01472 0.01472 1.65% CGSolver::sxay() 1586 0.01435 0.01435 0.01435 1.61% MultiFab::LinComb() 1586 0.01397 0.01397 0.01397 1.56% FabArray::setVal() 1144 0.01392 0.01392 0.01392 1.56% FabArray::ParallelCopy() 861 0.01379 0.01379 0.01379 1.54% FabArray::ParallelCopy_nowait() 861 0.01354 0.01354 0.01354 1.51% Castro::do_new_sources() 10 0.01311 0.01311 0.01311 1.47% MLMG::mgVcycle_up::2 82 0.01302 0.01302 0.01302 1.46% MLCGSolver::ParallelAllReduce 1514 0.013 0.013 0.013 1.45% MLMG::mgVcycle_up::1 82 0.01287 0.01287 0.01287 1.44% MLMG::addInterpCorrection() 410 0.01249 0.01249 0.01249 1.40% MLCellLinOp::defineAuxData() 11 0.01247 0.01247 0.01247 1.39% MLMG::mgVcycle_up::3 82 0.01233 0.01233 0.01233 1.38% MLMG::mgVcycle_up::4 82 0.01217 0.01217 0.01217 1.36% Castro::expand_state() 10 0.01216 0.01216 0.01216 1.36% Castro::initialize_advance() 10 0.01187 0.01187 0.01187 1.33% amrex::average_down 410 0.01163 0.01163 0.01163 1.30% MLPoisson::Fapply() 1142 0.01155 0.01155 0.01155 1.29% Castro::enforce_min_density() 62 0.01088 0.01088 0.01088 1.22% Castro::do_old_sources() 10 0.01057 0.01057 0.01057 1.18% Gravity::fill_multipole_BCs() 11 0.008315 0.008315 0.008315 0.93% Castro::post_timestep() 10 0.008104 0.008104 0.008104 0.91% MLCellLinOp::solutionResidual() 93 0.007045 0.007045 0.007045 0.79% MultiFab::Xpay() 585 0.006401 0.006401 0.006401 0.72% MLMG::computeResidual() 82 0.006064 0.006064 0.006064 0.68% Castro::reset_internal_energy(MultiFab) 63 0.005901 0.005901 0.005901 0.66% MLMG::prepareForSolve() 11 0.00529 0.00529 0.00529 0.59% MLCellLinOp::defineBC() 11 0.004957 0.004957 0.004957 0.55% BndryData::define() 11 0.004676 0.004676 0.004676 0.52% Castro::estTimeStep() 21 0.004385 0.004385 0.004385 0.49% Amr::InitializeInit() 1 0.004322 0.004322 0.004322 0.48% Amr::defBaseLevel() 1 0.004317 0.004317 0.004317 0.48% Castro::initData() 1 0.003795 0.003795 0.003795 0.42% Castro::construct_new_source() 50 0.003355 0.003355 0.003355 0.38% Castro::construct_new_gravity_source() 10 0.00334 0.00334 0.00334 0.37% Castro::construct_old_source() 50 0.002686 0.002686 0.002686 0.30% Castro::construct_old_gravity_source() 10 0.002667 0.002667 0.002667 0.30% MLMG::ResNormInf() 93 0.002024 0.002024 0.002024 0.23% Castro::apply_source_to_state() 20 0.001815 0.001815 0.001815 0.20% MultiFab::Saxpy() 20 0.001804 0.001804 0.001804 0.20% MultiFab::Add() 82 0.001612 0.001612 0.001612 0.18% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001608 0.001608 0.001608 0.18% Castro::reset_internal_energy(Fab) 504 0.001556 0.001556 0.001556 0.17% MLCellLinOp::setLevelBC() 11 0.0015 0.0015 0.0015 0.17% Castro::enforce_speed_limit() 62 0.001484 0.001484 0.001484 0.17% Castro::computeNewDt() 9 0.001481 0.001481 0.001481 0.17% MLMG::getGradSolution() 11 0.001405 0.001405 0.001405 0.16% FabArrayBase::getCPC() 1323 0.001405 0.001405 0.001405 0.16% MLCellLinOp::compGrad() 11 0.001398 0.001398 0.001398 0.16% FabArray::mult() 43 0.001323 0.001323 0.001323 0.15% FabArray::setDomainBndry() 41 0.001287 0.001287 0.001287 0.14% Castro::check_for_nan() 20 0.001193 0.001193 0.001193 0.13% MultiFab::contains_nan() 20 0.001181 0.001181 0.001181 0.13% MLPoisson::prepareForSolve() 11 0.001171 0.001171 0.001171 0.13% MLCellLinOp::prepareForSolve() 11 0.001163 0.001163 0.001163 0.13% Castro::post_regrid() 1 0.001141 0.001141 0.001141 0.13% MLMG::computeMLResidual() 11 0.001029 0.001029 0.001029 0.12% Castro::computeInitialDt() 2 0.0008695 0.0008695 0.0008695 0.10% Gravity::update_max_rhs() 11 0.0008191 0.0008191 0.0008191 0.09% FabArrayBase::getFB() 4023 0.0006804 0.0006804 0.0006804 0.08% FabArrayBase::CPC::define() 454 0.0006547 0.0006547 0.0006547 0.07% Castro::finalize_advance() 10 0.0005748 0.0005748 0.0005748 0.06% Amr::InitAmr() 1 0.0004637 0.0004637 0.0004637 0.05% Castro::Castro() 1 0.0004407 0.0004407 0.0004407 0.05% Gravity::swapTimeLevels() 10 0.0004352 0.0004352 0.0004352 0.05% MultiFab::Copy() 11 0.0003119 0.0003119 0.0003119 0.03% MLMG::MLResNormInf() 11 0.0002819 0.0002819 0.0002819 0.03% MultiFab::max() 11 0.0002554 0.0002554 0.0002554 0.03% MLLinOp::define() 11 0.00025 0.00025 0.00025 0.03% MLLinOp::defineGrids() 11 0.0002279 0.0002279 0.0002279 0.03% MLMG::MLRhsNormInf() 11 0.0002123 0.0002123 0.0002123 0.02% Castro::buildMetrics() 1 0.0001664 0.0001664 0.0001664 0.02% FabArrayBase::FB::FB() 56 8.245e-05 8.245e-05 8.245e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.576e-05 5.576e-05 5.576e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.214e-05 5.214e-05 5.214e-05 0.01% StateData::define() 4 4.037e-05 4.037e-05 4.037e-05 0.00% makeSFC 55 3.916e-05 3.916e-05 3.916e-05 0.00% Castro::swap_state_time_levels() 10 3.882e-05 3.882e-05 3.882e-05 0.00% Castro::finalize_do_advance() 10 3.253e-05 3.253e-05 3.253e-05 0.00% Castro::enforce_consistent_e() 1 3.249e-05 3.249e-05 3.249e-05 0.00% Castro::initMFs() 1 2.576e-05 2.576e-05 2.576e-05 0.00% Amr::writeSmallPlotFile() 1 2.506e-05 2.506e-05 2.506e-05 0.00% DistributionMapping::Distribute() 56 1.585e-05 1.585e-05 1.585e-05 0.00% Amr::initSubcycle() 1 8.058e-06 8.058e-06 8.058e-06 0.00% AmrLevel::checkPointPost() 3 5.792e-06 5.792e-06 5.792e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.599e-06 4.599e-06 4.599e-06 0.00% AmrLevel::checkPointPre() 3 4.362e-06 4.362e-06 4.362e-06 0.00% Gravity::set_mass_offset() 11 3.923e-06 3.923e-06 3.923e-06 0.00% Castro::retry_advance_ctu() 10 3.896e-06 3.896e-06 3.896e-06 0.00% Castro::create_source_corrector() 10 3.561e-06 3.561e-06 3.561e-06 0.00% Castro::FluxRegCrseInit 10 2.694e-06 2.694e-06 2.694e-06 0.00% Castro::FluxRegFineAdd() 10 2.025e-06 2.025e-06 2.025e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.87e-06 1.87e-06 1.87e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-25-g735c3513153f) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-25-g735c3513153f) initialized Starting run at 08:22:14 UTC on 2022-10-31. Successfully read inputs file ... Castro git describe: 22.09-8-g4d18c91ff AMReX git describe: 22.10-25-g735c35131 Microphysics git describe: 22.10-7-gced9a15b reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.486006835 Restart time = 0.048883978 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.053228702 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050205302 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057001022 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059228129 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.077647679 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031974641 seconds Ending run at 08:22:15 UTC on 2022-10-31. Run time = 0.379141308 Run time without initialization = 0.329696029 Average number of zones advanced per microsecond: 3.976 Average number of zones advanced per microsecond per rank: 3.976 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3792 ... 0.3792 ... 0.3792 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1008 0.1008 0.1008 26.59% VisMF::Read() 3 0.04098 0.04098 0.04098 10.81% MLCellLinOp::applyBC() 1946 0.03498 0.03498 0.03498 9.23% VisMF::Write(FabArray) 1 0.03038 0.03038 0.03038 8.01% MLPoisson::Fsmooth() 1440 0.02705 0.02705 0.02705 7.13% StateData::FillBoundary(geom) 160 0.0108 0.0108 0.0108 2.85% Castro::normalize_species() 30 0.01046 0.01046 0.01046 2.76% MLCGSolver::bicgstab 36 0.01019 0.01019 0.01019 2.69% MultiFab::Dot() 484 0.009447 0.009447 0.009447 2.49% Castro::computeTemp() 30 0.008743 0.008743 0.008743 2.31% FabArray::setVal() 537 0.006644 0.006644 0.006644 1.75% FillBoundary_nowait() 1766 0.006215 0.006215 0.006215 1.64% MLCellLinOp::defineAuxData() 6 0.006101 0.006101 0.006101 1.61% MultiFab::LinComb() 690 0.006048 0.006048 0.006048 1.59% FabArray::ParallelCopy_nowait() 380 0.005957 0.005957 0.005957 1.57% StateDataPhysBCFunct::() 20 0.005372 0.005372 0.005372 1.42% MLPoisson::Fapply() 500 0.005033 0.005033 0.005033 1.33% Gravity::fill_multipole_BCs() 6 0.004693 0.004693 0.004693 1.24% Amr::restart() 1 0.003618 0.003618 0.003618 0.95% MLMG::addInterpCorrection() 180 0.003338 0.003338 0.003338 0.88% Castro::enforce_min_density() 30 0.003087 0.003087 0.003087 0.81% amrex::average_down 180 0.002921 0.002921 0.002921 0.77% MultiFab::Xpay() 258 0.002829 0.002829 0.002829 0.75% Castro::estTimeStep() 10 0.002433 0.002433 0.002433 0.64% Castro::do_advance_ctu() 5 0.002357 0.002357 0.002357 0.62% BndryData::define() 6 0.002062 0.002062 0.002062 0.54% Castro::construct_new_gravity_source() 5 0.001838 0.001838 0.001838 0.48% Castro::reset_internal_energy(MultiFab) 30 0.001696 0.001696 0.001696 0.45% Amr::writePlotFile() 1 0.001681 0.001681 0.001681 0.44% Castro::construct_old_gravity_source() 5 0.001381 0.001381 0.001381 0.36% Gravity::get_old_grav_vector() 5 0.0009847 0.0009847 0.0009847 0.26% Castro::reset_internal_energy(Fab) 240 0.0009801 0.0009801 0.0009801 0.26% MultiFab::Saxpy() 10 0.0009167 0.0009167 0.0009167 0.24% MLMG::ResNormInf() 42 0.0009058 0.0009058 0.0009058 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008937 0.0008937 0.0008937 0.24% Castro::expand_state() 5 0.0008667 0.0008667 0.0008667 0.23% Gravity::get_new_grav_vector() 5 0.0008618 0.0008618 0.0008618 0.23% MLCellLinOp::setLevelBC() 6 0.000802 0.000802 0.000802 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007467 0.0007467 0.0007467 0.20% MultiFab::Add() 36 0.0007159 0.0007159 0.0007159 0.19% MLMG::prepareForSolve() 6 0.0006569 0.0006569 0.0006569 0.17% FabArray::mult() 22 0.0006504 0.0006504 0.0006504 0.17% FabArray::setDomainBndry() 20 0.000636 0.000636 0.000636 0.17% MLCellLinOp::prepareForSolve() 6 0.000629 0.000629 0.000629 0.17% MultiFab::contains_nan() 10 0.000586 0.000586 0.000586 0.15% MLCellLinOp::compGrad() 6 0.0004831 0.0004831 0.0004831 0.13% MLCellLinOp::smooth() 720 0.0004797 0.0004797 0.0004797 0.13% Castro::enforce_speed_limit() 30 0.0004792 0.0004792 0.0004792 0.13% Amr::InitAmr() 1 0.0003922 0.0003922 0.0003922 0.10% FabArrayBase::CPC::define() 244 0.0003833 0.0003833 0.0003833 0.10% FabArray::FillBoundary() 1766 0.0003585 0.0003585 0.0003585 0.09% Castro::finalize_advance() 5 0.0003583 0.0003583 0.0003583 0.09% FabArrayBase::getCPC() 632 0.0003498 0.0003498 0.0003498 0.09% FabArrayBase::getFB() 1766 0.0002569 0.0002569 0.0002569 0.07% main() 1 0.0002391 0.0002391 0.0002391 0.06% Gravity::update_max_rhs() 6 0.000235 0.000235 0.000235 0.06% Gravity::solve_for_phi() 5 0.0002211 0.0002211 0.0002211 0.06% MLCellLinOp::apply() 500 0.000206 0.000206 0.000206 0.05% CGSolver::sxay() 690 0.0001856 0.0001856 0.0001856 0.05% Amr::coarseTimeStep() 5 0.0001801 0.0001801 0.0001801 0.05% MultiFab::Copy() 6 0.0001727 0.0001727 0.0001727 0.05% Castro::construct_new_gravity() 5 0.0001694 0.0001694 0.0001694 0.04% Castro::subcycle_advance_ctu() 5 0.0001527 0.0001527 0.0001527 0.04% MLCellLinOp::defineBC() 6 0.0001515 0.0001515 0.0001515 0.04% MultiFab::max() 6 0.0001374 0.0001374 0.0001374 0.04% Castro::construct_new_source() 25 0.0001364 0.0001364 0.0001364 0.04% Castro::advance() 5 0.0001332 0.0001332 0.0001332 0.04% FillPatchIterator::Initialize 20 0.0001329 0.0001329 0.0001329 0.04% MLCGSolver::ParallelAllReduce 659 0.0001254 0.0001254 0.0001254 0.03% FabArray::ParallelCopy() 380 0.0001199 0.0001199 0.0001199 0.03% MLMG::MLRhsNormInf() 6 0.0001135 0.0001135 0.0001135 0.03% MLCellLinOp::correctionResidual() 216 9.927e-05 9.927e-05 9.927e-05 0.03% MLMG::mgVcycle() 36 9.567e-05 9.567e-05 9.567e-05 0.03% Amr::timeStep() 5 8.215e-05 8.215e-05 8.215e-05 0.02% AmrLevel::restart() 1 8.161e-05 8.161e-05 8.161e-05 0.02% MLMG:computeResOfCorrection() 180 7.579e-05 7.579e-05 7.579e-05 0.02% StateData::restartDoit() 4 7.481e-05 7.481e-05 7.481e-05 0.02% MLLinOp::defineGrids() 6 7.446e-05 7.446e-05 7.446e-05 0.02% FabArrayBase::FB::FB() 26 5.596e-05 5.596e-05 5.596e-05 0.01% Castro::create_source_corrector() 5 5.363e-05 5.363e-05 5.363e-05 0.01% MLMG::mgVcycle_down::0 36 5.178e-05 5.178e-05 5.178e-05 0.01% MLMG::mgVcycle_down::1 36 4.689e-05 4.689e-05 4.689e-05 0.01% Castro::initialize_do_advance() 5 4.583e-05 4.583e-05 4.583e-05 0.01% MLMG::mgVcycle_down::2 36 4.433e-05 4.433e-05 4.433e-05 0.01% MLMG::mgVcycle_down::4 36 4.155e-05 4.155e-05 4.155e-05 0.01% MLMG::mgVcycle_down::3 36 4.131e-05 4.131e-05 4.131e-05 0.01% Castro::clean_state() 30 4.068e-05 4.068e-05 4.068e-05 0.01% MLMG::actualBottomSolve() 36 3.888e-05 3.888e-05 3.888e-05 0.01% Castro::initialize_advance() 5 3.876e-05 3.876e-05 3.876e-05 0.01% Castro::construct_old_source() 25 3.688e-05 3.688e-05 3.688e-05 0.01% Castro::initMFs() 1 3.592e-05 3.592e-05 3.592e-05 0.01% MLMG::mgVcycle_up::4 36 3.405e-05 3.405e-05 3.405e-05 0.01% MLMG::solve() 6 3.26e-05 3.26e-05 3.26e-05 0.01% Castro::buildMetrics() 1 3.158e-05 3.158e-05 3.158e-05 0.01% Gravity::actual_multilevel_solve() 1 2.988e-05 2.988e-05 2.988e-05 0.01% Castro::post_restart() 1 2.901e-05 2.901e-05 2.901e-05 0.01% MLMG::mgVcycle_up::0 36 2.753e-05 2.753e-05 2.753e-05 0.01% Castro::swap_state_time_levels() 5 2.751e-05 2.751e-05 2.751e-05 0.01% MLMG::oneIter() 36 2.73e-05 2.73e-05 2.73e-05 0.01% MLMG::mgVcycle_up::3 36 2.665e-05 2.665e-05 2.665e-05 0.01% Amr::writeSmallPlotFile() 1 2.571e-05 2.571e-05 2.571e-05 0.01% MLMG::mgVcycle_up::2 36 2.543e-05 2.543e-05 2.543e-05 0.01% MLCellLinOp::solutionResidual() 42 2.339e-05 2.339e-05 2.339e-05 0.01% MLMG::mgVcycle_up::1 36 2.308e-05 2.308e-05 2.308e-05 0.01% Castro::post_timestep() 5 2.285e-05 2.285e-05 2.285e-05 0.01% MLPoisson::define() 6 2.284e-05 2.284e-05 2.284e-05 0.01% Castro::computeNewDt() 5 2.252e-05 2.252e-05 2.252e-05 0.01% MLLinOp::define() 6 2.161e-05 2.161e-05 2.161e-05 0.01% Castro::construct_old_gravity() 5 2.084e-05 2.084e-05 2.084e-05 0.01% Castro::finalize_do_advance() 5 1.757e-05 1.757e-05 1.757e-05 0.00% MLMG::computeResidual() 36 1.751e-05 1.751e-05 1.751e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.744e-05 1.744e-05 1.744e-05 0.00% MLMG::mgVcycle_bottom 36 1.496e-05 1.496e-05 1.496e-05 0.00% makeSFC 30 1.443e-05 1.443e-05 1.443e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.411e-05 1.411e-05 1.411e-05 0.00% FillPatchSingleLevel 20 1.224e-05 1.224e-05 1.224e-05 0.00% Castro::do_new_sources() 5 9.158e-06 9.158e-06 9.158e-06 0.00% DistributionMapping::Distribute() 31 8.882e-06 8.882e-06 8.882e-06 0.00% Amr::initSubcycle() 1 8.539e-06 8.539e-06 8.539e-06 0.00% Castro::do_old_sources() 5 7.809e-06 7.809e-06 7.809e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.186e-06 7.186e-06 7.186e-06 0.00% Castro::check_for_nan() 10 6.717e-06 6.717e-06 6.717e-06 0.00% Castro::apply_source_to_state() 10 5.231e-06 5.231e-06 5.231e-06 0.00% MLPoisson::prepareForSolve() 6 4.797e-06 4.797e-06 4.797e-06 0.00% MLMG::computeMLResidual() 6 4.697e-06 4.697e-06 4.697e-06 0.00% Gravity::swapTimeLevels() 5 4.212e-06 4.212e-06 4.212e-06 0.00% MLMG::getGradSolution() 6 3.549e-06 3.549e-06 3.549e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.038e-06 3.038e-06 3.038e-06 0.00% Gravity::set_mass_offset() 6 2.171e-06 2.171e-06 2.171e-06 0.00% MLMG::MLResNormInf() 6 2.063e-06 2.063e-06 2.063e-06 0.00% Castro::FluxRegCrseInit 5 1.775e-06 1.775e-06 1.775e-06 0.00% Castro::retry_advance_ctu() 5 1.661e-06 1.661e-06 1.661e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.118e-06 1.118e-06 1.118e-06 0.00% Amr::init() 1 9.5e-07 9.5e-07 9.5e-07 0.00% Castro::FluxRegFineAdd() 5 9.38e-07 9.38e-07 9.38e-07 0.00% AmrLevel::AmrLevel() 1 9.11e-07 9.11e-07 9.11e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3792 0.3792 0.3792 100.00% Amr::coarseTimeStep() 5 0.2975 0.2975 0.2975 78.46% Amr::timeStep() 5 0.296 0.296 0.296 78.06% Castro::advance() 5 0.2904 0.2904 0.2904 76.59% Castro::subcycle_advance_ctu() 5 0.2845 0.2845 0.2845 75.04% Castro::do_advance_ctu() 5 0.2844 0.2844 0.2844 75.00% Castro::construct_new_gravity() 5 0.1428 0.1428 0.1428 37.67% Gravity::solve_phi_with_mlmg() 6 0.1391 0.1391 0.1391 36.67% Gravity::solve_for_phi() 5 0.1355 0.1355 0.1355 35.74% Gravity::actual_solve_with_mlmg() 6 0.1342 0.1342 0.1342 35.40% MLMG::solve() 6 0.1222 0.1222 0.1222 32.23% MLMG::oneIter() 36 0.1151 0.1151 0.1151 30.34% MLMG::mgVcycle() 36 0.1143 0.1143 0.1143 30.15% Castro::construct_ctu_hydro_source() 5 0.1008 0.1008 0.1008 26.59% MLCellLinOp::smooth() 720 0.05859 0.05859 0.05859 15.45% Amr::init() 1 0.04894 0.04894 0.04894 12.91% Amr::restart() 1 0.04894 0.04894 0.04894 12.91% MLCellLinOp::applyBC() 1946 0.04187 0.04187 0.04187 11.04% AmrLevel::restart() 1 0.04119 0.04119 0.04119 10.86% StateData::restartDoit() 4 0.0411 0.0411 0.0411 10.84% VisMF::Read() 3 0.04098 0.04098 0.04098 10.81% MLMG::mgVcycle_bottom 36 0.03485 0.03485 0.03485 9.19% MLMG::actualBottomSolve() 36 0.03483 0.03483 0.03483 9.19% MLCGSolver::bicgstab 36 0.03448 0.03448 0.03448 9.09% Amr::writePlotFile() 1 0.03206 0.03206 0.03206 8.46% VisMF::Write(FabArray) 1 0.03038 0.03038 0.03038 8.01% MLPoisson::Fsmooth() 1440 0.02705 0.02705 0.02705 7.13% Castro::clean_state() 30 0.02549 0.02549 0.02549 6.72% FillPatchIterator::Initialize 20 0.01893 0.01893 0.01893 4.99% FillPatchSingleLevel 20 0.01816 0.01816 0.01816 4.79% StateDataPhysBCFunct::() 20 0.01617 0.01617 0.01617 4.26% MLCellLinOp::apply() 500 0.01577 0.01577 0.01577 4.16% MLMG::mgVcycle_down::0 36 0.01531 0.01531 0.01531 4.04% MLMG::mgVcycle_up::0 36 0.01316 0.01316 0.01316 3.47% Castro::computeTemp() 30 0.01142 0.01142 0.01142 3.01% StateData::FillBoundary(geom) 160 0.0108 0.0108 0.0108 2.85% Castro::initialize_do_advance() 5 0.0105 0.0105 0.0105 2.77% Castro::normalize_species() 30 0.01046 0.01046 0.01046 2.76% MLPoisson::define() 6 0.009729 0.009729 0.009729 2.57% MultiFab::Dot() 484 0.009447 0.009447 0.009447 2.49% MLCellLinOp::correctionResidual() 216 0.009194 0.009194 0.009194 2.42% MLMG:computeResOfCorrection() 180 0.007954 0.007954 0.007954 2.10% Castro::do_new_sources() 5 0.007841 0.007841 0.007841 2.07% MLMG::mgVcycle_down::1 36 0.00762 0.00762 0.00762 2.01% MLMG::mgVcycle_down::2 36 0.007397 0.007397 0.007397 1.95% Gravity::get_new_grav_vector() 5 0.007166 0.007166 0.007166 1.89% MLMG::mgVcycle_down::3 36 0.007029 0.007029 0.007029 1.85% Castro::construct_old_gravity() 5 0.006991 0.006991 0.006991 1.84% Gravity::get_old_grav_vector() 5 0.00697 0.00697 0.00697 1.84% FabArray::FillBoundary() 1766 0.006887 0.006887 0.006887 1.82% MLCellLinOp::defineAuxData() 6 0.006822 0.006822 0.006822 1.80% MLMG::mgVcycle_down::4 36 0.006699 0.006699 0.006699 1.77% FabArray::setVal() 537 0.006644 0.006644 0.006644 1.75% FillBoundary_nowait() 1766 0.006528 0.006528 0.006528 1.72% FabArray::ParallelCopy() 380 0.006439 0.006439 0.006439 1.70% FabArray::ParallelCopy_nowait() 380 0.006319 0.006319 0.006319 1.67% CGSolver::sxay() 690 0.006233 0.006233 0.006233 1.64% MultiFab::LinComb() 690 0.006048 0.006048 0.006048 1.59% Castro::do_old_sources() 5 0.005987 0.005987 0.005987 1.58% MLMG::mgVcycle_up::2 36 0.005703 0.005703 0.005703 1.50% MLCGSolver::ParallelAllReduce 659 0.005652 0.005652 0.005652 1.49% MLMG::mgVcycle_up::1 36 0.005633 0.005633 0.005633 1.49% Castro::expand_state() 5 0.005576 0.005576 0.005576 1.47% MLMG::addInterpCorrection() 180 0.005564 0.005564 0.005564 1.47% Castro::post_timestep() 5 0.005469 0.005469 0.005469 1.44% MLMG::mgVcycle_up::3 36 0.005418 0.005418 0.005418 1.43% MLMG::mgVcycle_up::4 36 0.005399 0.005399 0.005399 1.42% Castro::initialize_advance() 5 0.005397 0.005397 0.005397 1.42% amrex::average_down 180 0.005151 0.005151 0.005151 1.36% MLPoisson::Fapply() 500 0.005033 0.005033 0.005033 1.33% Gravity::fill_multipole_BCs() 6 0.004693 0.004693 0.004693 1.24% Castro::post_restart() 1 0.003946 0.003946 0.003946 1.04% Gravity::multilevel_solve_for_new_phi() 1 0.003823 0.003823 0.003823 1.01% Gravity::actual_multilevel_solve() 1 0.003806 0.003806 0.003806 1.00% MLCellLinOp::solutionResidual() 42 0.003219 0.003219 0.003219 0.85% Castro::enforce_min_density() 30 0.003087 0.003087 0.003087 0.81% MLMG::prepareForSolve() 6 0.002854 0.002854 0.002854 0.75% MultiFab::Xpay() 258 0.002829 0.002829 0.002829 0.75% MLCellLinOp::defineBC() 6 0.002758 0.002758 0.002758 0.73% Castro::reset_internal_energy(MultiFab) 30 0.002676 0.002676 0.002676 0.71% MLMG::computeResidual() 36 0.002675 0.002675 0.002675 0.71% BndryData::define() 6 0.002607 0.002607 0.002607 0.69% Castro::estTimeStep() 10 0.002433 0.002433 0.002433 0.64% Castro::construct_new_source() 25 0.001975 0.001975 0.001975 0.52% Castro::construct_new_gravity_source() 5 0.001838 0.001838 0.001838 0.48% Castro::construct_old_source() 25 0.001418 0.001418 0.001418 0.37% Castro::construct_old_gravity_source() 5 0.001381 0.001381 0.001381 0.36% Castro::computeNewDt() 5 0.001338 0.001338 0.001338 0.35% Castro::reset_internal_energy(Fab) 240 0.0009801 0.0009801 0.0009801 0.26% Castro::apply_source_to_state() 10 0.000922 0.000922 0.000922 0.24% MultiFab::Saxpy() 10 0.0009167 0.0009167 0.0009167 0.24% MLMG::ResNormInf() 42 0.0009058 0.0009058 0.0009058 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008937 0.0008937 0.0008937 0.24% MLCellLinOp::setLevelBC() 6 0.000802 0.000802 0.000802 0.21% MLMG::getGradSolution() 6 0.0007585 0.0007585 0.0007585 0.20% MLCellLinOp::compGrad() 6 0.000755 0.000755 0.000755 0.20% FabArrayBase::getCPC() 632 0.0007331 0.0007331 0.0007331 0.19% MultiFab::Add() 36 0.0007159 0.0007159 0.0007159 0.19% FabArray::mult() 22 0.0006504 0.0006504 0.0006504 0.17% FabArray::setDomainBndry() 20 0.000636 0.000636 0.000636 0.17% MLPoisson::prepareForSolve() 6 0.0006338 0.0006338 0.0006338 0.17% MLCellLinOp::prepareForSolve() 6 0.000629 0.000629 0.000629 0.17% Castro::check_for_nan() 10 0.0005927 0.0005927 0.0005927 0.16% MultiFab::contains_nan() 10 0.000586 0.000586 0.000586 0.15% MLMG::computeMLResidual() 6 0.0005667 0.0005667 0.0005667 0.15% Castro::enforce_speed_limit() 30 0.0004792 0.0004792 0.0004792 0.13% Gravity::update_max_rhs() 6 0.0004509 0.0004509 0.0004509 0.12% Amr::InitAmr() 1 0.0004007 0.0004007 0.0004007 0.11% FabArrayBase::CPC::define() 244 0.0003833 0.0003833 0.0003833 0.10% Castro::finalize_advance() 5 0.000361 0.000361 0.000361 0.10% FabArrayBase::getFB() 1766 0.0003128 0.0003128 0.0003128 0.08% Gravity::swapTimeLevels() 5 0.000226 0.000226 0.000226 0.06% MultiFab::Copy() 6 0.0001727 0.0001727 0.0001727 0.05% Castro::buildMetrics() 1 0.0001507 0.0001507 0.0001507 0.04% MLMG::MLResNormInf() 6 0.0001491 0.0001491 0.0001491 0.04% MultiFab::max() 6 0.0001374 0.0001374 0.0001374 0.04% MLLinOp::define() 6 0.0001265 0.0001265 0.0001265 0.03% MLMG::MLRhsNormInf() 6 0.0001135 0.0001135 0.0001135 0.03% MLLinOp::defineGrids() 6 0.0001049 0.0001049 0.0001049 0.03% FabArrayBase::FB::FB() 26 5.596e-05 5.596e-05 5.596e-05 0.01% Castro::create_source_corrector() 5 5.363e-05 5.363e-05 5.363e-05 0.01% Castro::initMFs() 1 3.592e-05 3.592e-05 3.592e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.933e-05 2.933e-05 2.933e-05 0.01% Castro::swap_state_time_levels() 5 2.751e-05 2.751e-05 2.751e-05 0.01% Amr::writeSmallPlotFile() 1 2.571e-05 2.571e-05 2.571e-05 0.01% makeSFC 30 2.215e-05 2.215e-05 2.215e-05 0.01% Castro::finalize_do_advance() 5 1.757e-05 1.757e-05 1.757e-05 0.00% DistributionMapping::Distribute() 31 8.882e-06 8.882e-06 8.882e-06 0.00% Amr::initSubcycle() 1 8.539e-06 8.539e-06 8.539e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.205e-06 4.205e-06 4.205e-06 0.00% Gravity::set_mass_offset() 6 2.171e-06 2.171e-06 2.171e-06 0.00% Castro::FluxRegCrseInit 5 1.775e-06 1.775e-06 1.775e-06 0.00% Castro::retry_advance_ctu() 5 1.661e-06 1.661e-06 1.661e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.118e-06 1.118e-06 1.118e-06 0.00% Castro::FluxRegFineAdd() 5 9.38e-07 9.38e-07 9.38e-07 0.00% AmrLevel::AmrLevel() 1 9.11e-07 9.11e-07 9.11e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-25-g735c3513153f) finalized