Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-17-g8294c3afbcbb) initialized Starting run at 08:35:38 UTC on 2022-08-24. Successfully read inputs file ... Castro git describe: 22.08-11-ga978fcf88 AMReX git describe: 22.08-17-g8294c3afb Microphysics git describe: 22.08-4-gb9a0f9aa reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.053531928 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030781947 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.049654965 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051630835 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.05279167 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.063978607 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.062723471 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.049880898 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.072248195 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.076909168 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.066543352 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.058328186 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.053574627 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.073980998 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.030694155 seconds Ending run at 08:35:39 UTC on 2022-08-24. Run time = 0.897634644 Run time without initialization = 0.76358131 Average number of zones advanced per microsecond: 3.433 Average number of zones advanced per microsecond per rank: 3.433 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8977 ... 0.8977 ... 0.8977 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2153 0.2153 0.2153 23.98% VisMF::Write(FabArray) 11 0.2068 0.2068 0.2068 23.04% MLCellLinOp::applyBC() 4379 0.07998 0.07998 0.07998 8.91% MLPoisson::Fsmooth() 3240 0.06384 0.06384 0.06384 7.11% Amr::checkPoint() 3 0.02926 0.02926 0.02926 3.26% StateData::FillBoundary(geom) 328 0.02493 0.02493 0.02493 2.78% MLCGSolver::bicgstab 81 0.02408 0.02408 0.02408 2.68% MultiFab::Dot() 1100 0.02234 0.02234 0.02234 2.49% Castro::normalize_species() 62 0.01473 0.01473 0.01473 1.64% MultiFab::LinComb() 1566 0.01451 0.01451 0.01451 1.62% Castro::computeTemp() 63 0.01439 0.01439 0.01439 1.60% FabArray::setVal() 1135 0.01434 0.01434 0.01434 1.60% FillBoundary_nowait() 3974 0.01431 0.01431 0.01431 1.59% FabArray::ParallelCopy_nowait() 851 0.01314 0.01314 0.01314 1.46% StateDataPhysBCFunct::() 41 0.01256 0.01256 0.01256 1.40% MLCellLinOp::defineAuxData() 11 0.01187 0.01187 0.01187 1.32% MLPoisson::Fapply() 1128 0.01179 0.01179 0.01179 1.31% Gravity::fill_multipole_BCs() 11 0.01038 0.01038 0.01038 1.16% Castro::enforce_min_density() 62 0.009689 0.009689 0.009689 1.08% MLMG::addInterpCorrection() 405 0.007631 0.007631 0.007631 0.85% amrex::average_down 405 0.006864 0.006864 0.006864 0.76% MultiFab::Xpay() 578 0.006625 0.006625 0.006625 0.74% Castro::estTimeStep() 21 0.006371 0.006371 0.006371 0.71% Castro::do_advance_ctu() 10 0.005217 0.005217 0.005217 0.58% Castro::reset_internal_energy(MultiFab) 63 0.004054 0.004054 0.004054 0.45% BndryData::define() 11 0.003923 0.003923 0.003923 0.44% Castro::construct_new_gravity_source() 10 0.003372 0.003372 0.003372 0.38% Castro::construct_old_gravity_source() 10 0.002892 0.002892 0.002892 0.32% Amr::writePlotFile() 2 0.002841 0.002841 0.002841 0.32% MLMG::ResNormInf() 92 0.002151 0.002151 0.002151 0.24% Gravity::get_new_grav_vector() 11 0.001936 0.001936 0.001936 0.22% MultiFab::Saxpy() 20 0.001819 0.001819 0.001819 0.20% Gravity::get_old_grav_vector() 10 0.001746 0.001746 0.001746 0.19% Castro::expand_state() 10 0.001741 0.001741 0.001741 0.19% MultiFab::Add() 81 0.001675 0.001675 0.001675 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001668 0.001668 0.001668 0.19% MLCellLinOp::setLevelBC() 11 0.001561 0.001561 0.001561 0.17% Castro::reset_internal_energy(Fab) 504 0.001488 0.001488 0.001488 0.17% Gravity::actual_solve_with_mlmg() 11 0.001427 0.001427 0.001427 0.16% Castro::enforce_speed_limit() 62 0.001393 0.001393 0.001393 0.16% FabArray::mult() 43 0.001325 0.001325 0.001325 0.15% FabArray::setDomainBndry() 41 0.001321 0.001321 0.001321 0.15% MLMG::prepareForSolve() 11 0.00121 0.00121 0.00121 0.13% MultiFab::contains_nan() 20 0.001193 0.001193 0.001193 0.13% Castro::initData() 1 0.001186 0.001186 0.001186 0.13% MLCellLinOp::prepareForSolve() 11 0.001181 0.001181 0.001181 0.13% MLCellLinOp::smooth() 1620 0.001044 0.001044 0.001044 0.12% MLCellLinOp::compGrad() 11 0.0009285 0.0009285 0.0009285 0.10% FabArray::FillBoundary() 3974 0.000847 0.000847 0.000847 0.09% FabArrayBase::getCPC() 1313 0.000745 0.000745 0.000745 0.08% FabArrayBase::CPC::define() 454 0.0006674 0.0006674 0.0006674 0.07% FabArrayBase::getFB() 3974 0.0005999 0.0005999 0.0005999 0.07% Amr::InitAmr() 1 0.0004969 0.0004969 0.0004969 0.06% Gravity::solve_for_phi() 10 0.0004621 0.0004621 0.0004621 0.05% MLCellLinOp::apply() 1128 0.0004565 0.0004565 0.0004565 0.05% Gravity::update_max_rhs() 11 0.0004205 0.0004205 0.0004205 0.05% CGSolver::sxay() 1566 0.000403 0.000403 0.000403 0.04% Amr::coarseTimeStep() 10 0.000344 0.000344 0.000344 0.04% MultiFab::Copy() 11 0.0003304 0.0003304 0.0003304 0.04% MLCGSolver::ParallelAllReduce 1495 0.0002978 0.0002978 0.0002978 0.03% FillPatchIterator::Initialize 41 0.000292 0.000292 0.000292 0.03% main() 1 0.0002832 0.0002832 0.0002832 0.03% MLCellLinOp::defineBC() 11 0.0002779 0.0002779 0.0002779 0.03% MultiFab::max() 11 0.0002609 0.0002609 0.0002609 0.03% FabArray::ParallelCopy() 851 0.0002567 0.0002567 0.0002567 0.03% MLCellLinOp::correctionResidual() 486 0.0002227 0.0002227 0.0002227 0.02% MLMG::MLRhsNormInf() 11 0.0002201 0.0002201 0.0002201 0.02% Castro::construct_new_gravity() 10 0.0002079 0.0002079 0.0002079 0.02% MLMG::mgVcycle() 81 0.0002058 0.0002058 0.0002058 0.02% Amr::timeStep() 10 0.0002002 0.0002002 0.0002002 0.02% Castro::create_source_corrector() 10 0.0001948 0.0001948 0.0001948 0.02% Castro::subcycle_advance_ctu() 10 0.0001769 0.0001769 0.0001769 0.02% MLMG:computeResOfCorrection() 405 0.0001409 0.0001409 0.0001409 0.02% MLLinOp::defineGrids() 11 0.0001338 0.0001338 0.0001338 0.01% StateData::checkPoint() 12 0.0001292 0.0001292 0.0001292 0.01% MLMG::mgVcycle_down::0 81 0.0001048 0.0001048 0.0001048 0.01% Castro::Castro() 1 9.351e-05 9.351e-05 9.351e-05 0.01% MLMG::mgVcycle_down::1 81 9.02e-05 9.02e-05 9.02e-05 0.01% MLMG::mgVcycle_down::2 81 8.726e-05 8.726e-05 8.726e-05 0.01% FabArrayBase::FB::FB() 56 8.563e-05 8.563e-05 8.563e-05 0.01% Castro::initialize_advance() 10 8.268e-05 8.268e-05 8.268e-05 0.01% MLMG::mgVcycle_down::3 81 8.257e-05 8.257e-05 8.257e-05 0.01% MLMG::mgVcycle_down::4 81 8.124e-05 8.124e-05 8.124e-05 0.01% MLMG::actualBottomSolve() 81 7.99e-05 7.99e-05 7.99e-05 0.01% Castro::clean_state() 62 7.815e-05 7.815e-05 7.815e-05 0.01% AmrLevel::checkPoint() 3 7.394e-05 7.394e-05 7.394e-05 0.01% MLMG::solve() 11 6.747e-05 6.747e-05 6.747e-05 0.01% MLMG::mgVcycle_up::4 81 6.499e-05 6.499e-05 6.499e-05 0.01% Castro::initialize_do_advance() 10 6.461e-05 6.461e-05 6.461e-05 0.01% Castro::advance() 10 5.672e-05 5.672e-05 5.672e-05 0.01% MLMG::mgVcycle_up::1 81 5.489e-05 5.489e-05 5.489e-05 0.01% MLMG::mgVcycle_up::2 81 5.421e-05 5.421e-05 5.421e-05 0.01% MLMG::mgVcycle_up::0 81 5.419e-05 5.419e-05 5.419e-05 0.01% MLMG::mgVcycle_up::3 81 5.36e-05 5.36e-05 5.36e-05 0.01% MLMG::oneIter() 81 5.167e-05 5.167e-05 5.167e-05 0.01% Castro::finalize_advance() 10 5.145e-05 5.145e-05 5.145e-05 0.01% MLCellLinOp::solutionResidual() 92 4.81e-05 4.81e-05 4.81e-05 0.01% StateData::define() 4 4.317e-05 4.317e-05 4.317e-05 0.00% Castro::swap_state_time_levels() 10 3.982e-05 3.982e-05 3.982e-05 0.00% MLMG::computeResidual() 81 3.718e-05 3.718e-05 3.718e-05 0.00% Castro::enforce_consistent_e() 1 3.546e-05 3.546e-05 3.546e-05 0.00% Castro::finalize_do_advance() 10 3.473e-05 3.473e-05 3.473e-05 0.00% Castro::construct_new_source() 50 3.419e-05 3.419e-05 3.419e-05 0.00% Gravity::actual_multilevel_solve() 1 3.322e-05 3.322e-05 3.322e-05 0.00% MLMG::mgVcycle_bottom 81 3.182e-05 3.182e-05 3.182e-05 0.00% MLPoisson::define() 11 2.8e-05 2.8e-05 2.8e-05 0.00% FillPatchSingleLevel 41 2.789e-05 2.789e-05 2.789e-05 0.00% makeSFC 55 2.739e-05 2.739e-05 2.739e-05 0.00% Castro::initMFs() 1 2.702e-05 2.702e-05 2.702e-05 0.00% Amr::writeSmallPlotFile() 1 2.66e-05 2.66e-05 2.66e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.544e-05 2.544e-05 2.544e-05 0.00% Amr::defBaseLevel() 1 2.503e-05 2.503e-05 2.503e-05 0.00% MLLinOp::define() 11 2.404e-05 2.404e-05 2.404e-05 0.00% Castro::buildMetrics() 1 2.36e-05 2.36e-05 2.36e-05 0.00% Amr::FinalizeInit() 1 2.159e-05 2.159e-05 2.159e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.07e-05 2.07e-05 2.07e-05 0.00% Castro::construct_old_source() 50 2.029e-05 2.029e-05 2.029e-05 0.00% Castro::do_new_sources() 10 1.683e-05 1.683e-05 1.683e-05 0.00% Castro::do_old_sources() 10 1.535e-05 1.535e-05 1.535e-05 0.00% DistributionMapping::Distribute() 56 1.518e-05 1.518e-05 1.518e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.339e-05 1.339e-05 1.339e-05 0.00% Castro::check_for_nan() 20 1.255e-05 1.255e-05 1.255e-05 0.00% Castro::apply_source_to_state() 20 1.121e-05 1.121e-05 1.121e-05 0.00% Castro::construct_old_gravity() 10 1.005e-05 1.005e-05 1.005e-05 0.00% Amr::initSubcycle() 1 9.316e-06 9.316e-06 9.316e-06 0.00% MLMG::computeMLResidual() 11 9.285e-06 9.285e-06 9.285e-06 0.00% Gravity::swapTimeLevels() 10 9.096e-06 9.096e-06 9.096e-06 0.00% MLPoisson::prepareForSolve() 11 8.876e-06 8.876e-06 8.876e-06 0.00% Castro::post_timestep() 10 8.781e-06 8.781e-06 8.781e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.126e-06 7.126e-06 7.126e-06 0.00% Amr::InitializeInit() 1 6.876e-06 6.876e-06 6.876e-06 0.00% Castro::computeNewDt() 9 6.207e-06 6.207e-06 6.207e-06 0.00% MLMG::getGradSolution() 11 6.036e-06 6.036e-06 6.036e-06 0.00% AmrLevel::checkPointPost() 3 4.184e-06 4.184e-06 4.184e-06 0.00% Gravity::set_mass_offset() 11 4.111e-06 4.111e-06 4.111e-06 0.00% Castro::retry_advance_ctu() 10 4.023e-06 4.023e-06 4.023e-06 0.00% Castro::post_init() 1 3.843e-06 3.843e-06 3.843e-06 0.00% MLMG::MLResNormInf() 11 3.304e-06 3.304e-06 3.304e-06 0.00% Amr::init() 1 3.102e-06 3.102e-06 3.102e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.995e-06 2.995e-06 2.995e-06 0.00% Castro::FluxRegCrseInit 10 2.742e-06 2.742e-06 2.742e-06 0.00% AmrLevel::checkPointPre() 3 2.73e-06 2.73e-06 2.73e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.472e-06 2.472e-06 2.472e-06 0.00% Castro::computeInitialDt() 2 2.403e-06 2.403e-06 2.403e-06 0.00% Castro::FluxRegFineAdd() 10 2.359e-06 2.359e-06 2.359e-06 0.00% Castro::post_regrid() 1 2.064e-06 2.064e-06 2.064e-06 0.00% Amr::initialInit() 1 9.64e-07 9.64e-07 9.64e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8976 0.8976 0.8976 100.00% Amr::coarseTimeStep() 10 0.7327 0.7327 0.7327 81.62% Amr::timeStep() 10 0.6045 0.6045 0.6045 67.34% Castro::advance() 10 0.5971 0.5971 0.5971 66.52% Castro::subcycle_advance_ctu() 10 0.5858 0.5858 0.5858 65.26% Castro::do_advance_ctu() 10 0.5856 0.5856 0.5856 65.24% Gravity::solve_phi_with_mlmg() 11 0.317 0.317 0.317 35.31% Gravity::actual_solve_with_mlmg() 11 0.3064 0.3064 0.3064 34.13% Castro::construct_new_gravity() 10 0.2915 0.2915 0.2915 32.48% MLMG::solve() 11 0.2834 0.2834 0.2834 31.57% Gravity::solve_for_phi() 10 0.2758 0.2758 0.2758 30.73% MLMG::oneIter() 81 0.2683 0.2683 0.2683 29.89% MLMG::mgVcycle() 81 0.2666 0.2666 0.2666 29.70% Castro::construct_ctu_hydro_source() 10 0.2153 0.2153 0.2153 23.98% VisMF::Write(FabArray) 11 0.2068 0.2068 0.2068 23.04% Amr::checkPoint() 3 0.1775 0.1775 0.1775 19.78% AmrLevel::checkPoint() 3 0.1483 0.1483 0.1483 16.52% StateData::checkPoint() 12 0.1482 0.1482 0.1482 16.51% MLCellLinOp::smooth() 1620 0.1362 0.1362 0.1362 15.17% Amr::init() 1 0.1334 0.1334 0.1334 14.86% MLCellLinOp::applyBC() 4379 0.09582 0.09582 0.09582 10.67% MLMG::mgVcycle_bottom 81 0.08227 0.08227 0.08227 9.16% MLMG::actualBottomSolve() 81 0.08224 0.08224 0.08224 9.16% MLCGSolver::bicgstab 81 0.08142 0.08142 0.08142 9.07% MLPoisson::Fsmooth() 3240 0.06384 0.06384 0.06384 7.11% Amr::writePlotFile() 2 0.06161 0.06161 0.06161 6.86% Amr::initialInit() 1 0.04897 0.04897 0.04897 5.46% Castro::clean_state() 62 0.04512 0.04512 0.04512 5.03% Amr::FinalizeInit() 1 0.04506 0.04506 0.04506 5.02% Castro::post_init() 1 0.0436 0.0436 0.0436 4.86% FillPatchIterator::Initialize 41 0.04312 0.04312 0.04312 4.80% Gravity::multilevel_solve_for_new_phi() 1 0.04165 0.04165 0.04165 4.64% Gravity::actual_multilevel_solve() 1 0.04163 0.04163 0.04163 4.64% FillPatchSingleLevel 41 0.0415 0.0415 0.0415 4.62% StateDataPhysBCFunct::() 41 0.03749 0.03749 0.03749 4.18% MLCellLinOp::apply() 1128 0.03628 0.03628 0.03628 4.04% MLMG::mgVcycle_down::0 81 0.03536 0.03536 0.03536 3.94% MLMG::mgVcycle_up::0 81 0.03014 0.03014 0.03014 3.36% StateData::FillBoundary(geom) 328 0.02493 0.02493 0.02493 2.78% MultiFab::Dot() 1100 0.02234 0.02234 0.02234 2.49% MLCellLinOp::correctionResidual() 486 0.0213 0.0213 0.0213 2.37% Castro::initialize_do_advance() 10 0.02023 0.02023 0.02023 2.25% Castro::computeTemp() 63 0.01993 0.01993 0.01993 2.22% MLPoisson::define() 11 0.01859 0.01859 0.01859 2.07% MLMG:computeResOfCorrection() 405 0.01838 0.01838 0.01838 2.05% MLMG::mgVcycle_down::1 81 0.01776 0.01776 0.01776 1.98% Gravity::get_new_grav_vector() 11 0.01732 0.01732 0.01732 1.93% MLMG::mgVcycle_down::2 81 0.01729 0.01729 0.01729 1.93% MLMG::mgVcycle_down::3 81 0.01647 0.01647 0.01647 1.83% FabArray::FillBoundary() 3974 0.01584 0.01584 0.01584 1.76% MLMG::mgVcycle_down::4 81 0.01565 0.01565 0.01565 1.74% Castro::construct_old_gravity() 10 0.01527 0.01527 0.01527 1.70% Gravity::get_old_grav_vector() 10 0.01526 0.01526 0.01526 1.70% FillBoundary_nowait() 3974 0.01499 0.01499 0.01499 1.67% CGSolver::sxay() 1566 0.01491 0.01491 0.01491 1.66% Castro::normalize_species() 62 0.01473 0.01473 0.01473 1.64% MultiFab::LinComb() 1566 0.01451 0.01451 0.01451 1.62% FabArray::setVal() 1135 0.01434 0.01434 0.01434 1.60% FabArray::ParallelCopy() 851 0.0142 0.0142 0.0142 1.58% FabArray::ParallelCopy_nowait() 851 0.01395 0.01395 0.01395 1.55% MLMG::mgVcycle_up::2 81 0.01333 0.01333 0.01333 1.49% MLCGSolver::ParallelAllReduce 1495 0.01331 0.01331 0.01331 1.48% MLCellLinOp::defineAuxData() 11 0.0132 0.0132 0.0132 1.47% MLMG::mgVcycle_up::1 81 0.01305 0.01305 0.01305 1.45% MLMG::addInterpCorrection() 405 0.0127 0.0127 0.0127 1.41% MLMG::mgVcycle_up::3 81 0.01258 0.01258 0.01258 1.40% MLMG::mgVcycle_up::4 81 0.01248 0.01248 0.01248 1.39% Castro::do_new_sources() 10 0.01206 0.01206 0.01206 1.34% amrex::average_down 405 0.01201 0.01201 0.01201 1.34% MLPoisson::Fapply() 1128 0.01179 0.01179 0.01179 1.31% Castro::expand_state() 10 0.01163 0.01163 0.01163 1.30% Castro::initialize_advance() 10 0.01126 0.01126 0.01126 1.25% Castro::do_old_sources() 10 0.0109 0.0109 0.0109 1.21% Gravity::fill_multipole_BCs() 11 0.01038 0.01038 0.01038 1.16% Castro::enforce_min_density() 62 0.009689 0.009689 0.009689 1.08% MLCellLinOp::solutionResidual() 92 0.007165 0.007165 0.007165 0.80% Castro::post_timestep() 10 0.007146 0.007146 0.007146 0.80% MultiFab::Xpay() 578 0.006625 0.006625 0.006625 0.74% Castro::estTimeStep() 21 0.006371 0.006371 0.006371 0.71% MLMG::computeResidual() 81 0.006175 0.006175 0.006175 0.69% Castro::reset_internal_energy(MultiFab) 63 0.005541 0.005541 0.005541 0.62% MLMG::prepareForSolve() 11 0.005395 0.005395 0.005395 0.60% MLCellLinOp::defineBC() 11 0.005149 0.005149 0.005149 0.57% BndryData::define() 11 0.004872 0.004872 0.004872 0.54% Amr::InitializeInit() 1 0.003913 0.003913 0.003913 0.44% Amr::defBaseLevel() 1 0.003906 0.003906 0.003906 0.44% Castro::construct_new_source() 50 0.003406 0.003406 0.003406 0.38% Castro::initData() 1 0.003383 0.003383 0.003383 0.38% Castro::computeNewDt() 9 0.003374 0.003374 0.003374 0.38% Castro::construct_new_gravity_source() 10 0.003372 0.003372 0.003372 0.38% Castro::construct_old_source() 50 0.002912 0.002912 0.002912 0.32% Castro::construct_old_gravity_source() 10 0.002892 0.002892 0.002892 0.32% MLMG::ResNormInf() 92 0.002151 0.002151 0.002151 0.24% Castro::apply_source_to_state() 20 0.00183 0.00183 0.00183 0.20% MultiFab::Saxpy() 20 0.001819 0.001819 0.001819 0.20% MultiFab::Add() 81 0.001675 0.001675 0.001675 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001668 0.001668 0.001668 0.19% MLCellLinOp::setLevelBC() 11 0.001561 0.001561 0.001561 0.17% Castro::reset_internal_energy(Fab) 504 0.001488 0.001488 0.001488 0.17% MLMG::getGradSolution() 11 0.001441 0.001441 0.001441 0.16% MLCellLinOp::compGrad() 11 0.001435 0.001435 0.001435 0.16% FabArrayBase::getCPC() 1313 0.001412 0.001412 0.001412 0.16% Castro::enforce_speed_limit() 62 0.001393 0.001393 0.001393 0.16% FabArray::mult() 43 0.001325 0.001325 0.001325 0.15% FabArray::setDomainBndry() 41 0.001321 0.001321 0.001321 0.15% Castro::post_regrid() 1 0.001256 0.001256 0.001256 0.14% Castro::check_for_nan() 20 0.001205 0.001205 0.001205 0.13% MultiFab::contains_nan() 20 0.001193 0.001193 0.001193 0.13% MLPoisson::prepareForSolve() 11 0.00119 0.00119 0.00119 0.13% MLCellLinOp::prepareForSolve() 11 0.001181 0.001181 0.001181 0.13% MLMG::computeMLResidual() 11 0.001036 0.001036 0.001036 0.12% Gravity::update_max_rhs() 11 0.0008284 0.0008284 0.0008284 0.09% FabArrayBase::getFB() 3974 0.0006855 0.0006855 0.0006855 0.08% Castro::computeInitialDt() 2 0.000683 0.000683 0.000683 0.08% FabArrayBase::CPC::define() 454 0.0006674 0.0006674 0.0006674 0.07% Amr::InitAmr() 1 0.0005062 0.0005062 0.0005062 0.06% Gravity::swapTimeLevels() 10 0.0004527 0.0004527 0.0004527 0.05% Castro::Castro() 1 0.0004432 0.0004432 0.0004432 0.05% MultiFab::Copy() 11 0.0003304 0.0003304 0.0003304 0.04% MLMG::MLResNormInf() 11 0.0002887 0.0002887 0.0002887 0.03% MultiFab::max() 11 0.0002609 0.0002609 0.0002609 0.03% MLMG::MLRhsNormInf() 11 0.0002201 0.0002201 0.0002201 0.02% MLLinOp::define() 11 0.0002151 0.0002151 0.0002151 0.02% Castro::create_source_corrector() 10 0.0001948 0.0001948 0.0001948 0.02% MLLinOp::defineGrids() 11 0.0001911 0.0001911 0.0001911 0.02% Castro::buildMetrics() 1 0.0001569 0.0001569 0.0001569 0.02% FabArrayBase::FB::FB() 56 8.563e-05 8.563e-05 8.563e-05 0.01% Castro::finalize_advance() 10 5.655e-05 5.655e-05 5.655e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.487e-05 5.487e-05 5.487e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.029e-05 5.029e-05 5.029e-05 0.01% StateData::define() 4 4.317e-05 4.317e-05 4.317e-05 0.00% makeSFC 55 4.148e-05 4.148e-05 4.148e-05 0.00% Castro::swap_state_time_levels() 10 3.982e-05 3.982e-05 3.982e-05 0.00% Castro::enforce_consistent_e() 1 3.546e-05 3.546e-05 3.546e-05 0.00% Castro::finalize_do_advance() 10 3.473e-05 3.473e-05 3.473e-05 0.00% Castro::initMFs() 1 2.702e-05 2.702e-05 2.702e-05 0.00% Amr::writeSmallPlotFile() 1 2.66e-05 2.66e-05 2.66e-05 0.00% DistributionMapping::Distribute() 56 1.518e-05 1.518e-05 1.518e-05 0.00% Amr::initSubcycle() 1 9.316e-06 9.316e-06 9.316e-06 0.00% AmrLevel::checkPointPost() 3 4.184e-06 4.184e-06 4.184e-06 0.00% Gravity::set_mass_offset() 11 4.111e-06 4.111e-06 4.111e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.085e-06 4.085e-06 4.085e-06 0.00% Castro::retry_advance_ctu() 10 4.023e-06 4.023e-06 4.023e-06 0.00% Castro::FluxRegCrseInit 10 2.742e-06 2.742e-06 2.742e-06 0.00% AmrLevel::checkPointPre() 3 2.73e-06 2.73e-06 2.73e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.472e-06 2.472e-06 2.472e-06 0.00% Castro::FluxRegFineAdd() 10 2.359e-06 2.359e-06 2.359e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-17-g8294c3afbcbb) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-17-g8294c3afbcbb) initialized Starting run at 08:35:40 UTC on 2022-08-24. Successfully read inputs file ... Castro git describe: 22.08-11-ga978fcf88 AMReX git describe: 22.08-17-g8294c3afb Microphysics git describe: 22.08-4-gb9a0f9aa reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.464724482 Restart time = 0.048315874 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.053353294 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050227992 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.062492737 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.073150887 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.07436493 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032535901 seconds Ending run at 08:35:40 UTC on 2022-08-24. Run time = 0.395450223 Run time without initialization = 0.346543226 Average number of zones advanced per microsecond: 3.782 Average number of zones advanced per microsecond per rank: 3.782 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3955 ... 0.3955 ... 0.3955 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1175 0.1175 0.1175 29.72% VisMF::Read() 3 0.0404 0.0404 0.0404 10.21% MLCellLinOp::applyBC() 1946 0.03467 0.03467 0.03467 8.77% VisMF::Write(FabArray) 1 0.03098 0.03098 0.03098 7.83% MLPoisson::Fsmooth() 1440 0.02718 0.02718 0.02718 6.87% StateData::FillBoundary(geom) 160 0.01269 0.01269 0.01269 3.21% MLCGSolver::bicgstab 36 0.01028 0.01028 0.01028 2.60% MultiFab::Dot() 484 0.009564 0.009564 0.009564 2.42% FabArray::setVal() 537 0.006782 0.006782 0.006782 1.71% MLCellLinOp::defineAuxData() 6 0.006369 0.006369 0.006369 1.61% Castro::enforce_min_density() 30 0.006336 0.006336 0.006336 1.60% FillBoundary_nowait() 1766 0.006283 0.006283 0.006283 1.59% Castro::computeTemp() 30 0.006197 0.006197 0.006197 1.57% MultiFab::LinComb() 690 0.006156 0.006156 0.006156 1.56% FabArray::ParallelCopy_nowait() 380 0.005982 0.005982 0.005982 1.51% Castro::normalize_species() 30 0.005841 0.005841 0.005841 1.48% Gravity::fill_multipole_BCs() 6 0.005746 0.005746 0.005746 1.45% MLPoisson::Fapply() 500 0.00507 0.00507 0.00507 1.28% StateDataPhysBCFunct::() 20 0.005045 0.005045 0.005045 1.28% Amr::restart() 1 0.003631 0.003631 0.003631 0.92% MLMG::addInterpCorrection() 180 0.003288 0.003288 0.003288 0.83% amrex::average_down 180 0.002971 0.002971 0.002971 0.75% Castro::estTimeStep() 10 0.002937 0.002937 0.002937 0.74% MultiFab::Xpay() 258 0.00289 0.00289 0.00289 0.73% Castro::do_advance_ctu() 5 0.002273 0.002273 0.002273 0.57% Castro::reset_internal_energy(MultiFab) 30 0.002198 0.002198 0.002198 0.56% BndryData::define() 6 0.002154 0.002154 0.002154 0.54% Amr::writePlotFile() 1 0.001651 0.001651 0.001651 0.42% Castro::construct_new_gravity_source() 5 0.001643 0.001643 0.001643 0.42% Castro::construct_old_gravity_source() 5 0.001304 0.001304 0.001304 0.33% MLMG::ResNormInf() 42 0.0009459 0.0009459 0.0009459 0.24% MultiFab::Saxpy() 10 0.0009137 0.0009137 0.0009137 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009116 0.0009116 0.0009116 0.23% Gravity::get_old_grav_vector() 5 0.0009072 0.0009072 0.0009072 0.23% Castro::expand_state() 5 0.0008719 0.0008719 0.0008719 0.22% Gravity::get_new_grav_vector() 5 0.0008678 0.0008678 0.0008678 0.22% MLCellLinOp::setLevelBC() 6 0.0008334 0.0008334 0.0008334 0.21% Castro::reset_internal_energy(Fab) 240 0.00075 0.00075 0.00075 0.19% Gravity::actual_solve_with_mlmg() 6 0.0007429 0.0007429 0.0007429 0.19% MultiFab::Add() 36 0.0007224 0.0007224 0.0007224 0.18% Castro::enforce_speed_limit() 30 0.0007053 0.0007053 0.0007053 0.18% MLMG::prepareForSolve() 6 0.0006537 0.0006537 0.0006537 0.17% FabArray::mult() 22 0.0006534 0.0006534 0.0006534 0.17% FabArray::setDomainBndry() 20 0.0006418 0.0006418 0.0006418 0.16% MLCellLinOp::prepareForSolve() 6 0.0006396 0.0006396 0.0006396 0.16% MultiFab::contains_nan() 10 0.0005955 0.0005955 0.0005955 0.15% MLCellLinOp::compGrad() 6 0.0004946 0.0004946 0.0004946 0.13% MLCellLinOp::smooth() 720 0.0004762 0.0004762 0.0004762 0.12% Amr::InitAmr() 1 0.0004137 0.0004137 0.0004137 0.10% FabArray::FillBoundary() 1766 0.0003935 0.0003935 0.0003935 0.10% FabArrayBase::CPC::define() 244 0.00039 0.00039 0.00039 0.10% FabArrayBase::getCPC() 632 0.0003604 0.0003604 0.0003604 0.09% FabArrayBase::getFB() 1766 0.0002646 0.0002646 0.0002646 0.07% main() 1 0.0002526 0.0002526 0.0002526 0.06% Gravity::solve_for_phi() 5 0.0002294 0.0002294 0.0002294 0.06% Gravity::update_max_rhs() 6 0.000227 0.000227 0.000227 0.06% MLCellLinOp::apply() 500 0.0002004 0.0002004 0.0002004 0.05% CGSolver::sxay() 690 0.0001789 0.0001789 0.0001789 0.05% MultiFab::Copy() 6 0.0001762 0.0001762 0.0001762 0.04% Amr::coarseTimeStep() 5 0.0001726 0.0001726 0.0001726 0.04% MLCellLinOp::defineBC() 6 0.0001494 0.0001494 0.0001494 0.04% FillPatchIterator::Initialize 20 0.0001401 0.0001401 0.0001401 0.04% MultiFab::max() 6 0.0001368 0.0001368 0.0001368 0.03% MLCGSolver::ParallelAllReduce 659 0.0001252 0.0001252 0.0001252 0.03% FabArray::ParallelCopy() 380 0.0001195 0.0001195 0.0001195 0.03% MLMG::MLRhsNormInf() 6 0.0001174 0.0001174 0.0001174 0.03% Castro::construct_new_gravity() 5 0.000116 0.000116 0.000116 0.03% Amr::timeStep() 5 0.0001011 0.0001011 0.0001011 0.03% Castro::subcycle_advance_ctu() 5 0.0001003 0.0001003 0.0001003 0.03% Castro::create_source_corrector() 5 9.775e-05 9.775e-05 9.775e-05 0.02% MLCellLinOp::correctionResidual() 216 9.743e-05 9.743e-05 9.743e-05 0.02% MLMG::mgVcycle() 36 8.798e-05 8.798e-05 8.798e-05 0.02% AmrLevel::restart() 1 7.494e-05 7.494e-05 7.494e-05 0.02% StateData::restartDoit() 4 7.369e-05 7.369e-05 7.369e-05 0.02% MLLinOp::defineGrids() 6 7e-05 7e-05 7e-05 0.02% MLMG:computeResOfCorrection() 180 6.636e-05 6.636e-05 6.636e-05 0.02% Castro::finalize_advance() 5 6.221e-05 6.221e-05 6.221e-05 0.02% FabArrayBase::FB::FB() 26 5.636e-05 5.636e-05 5.636e-05 0.01% MLMG::mgVcycle_down::0 36 4.714e-05 4.714e-05 4.714e-05 0.01% Castro::clean_state() 30 4.412e-05 4.412e-05 4.412e-05 0.01% MLMG::mgVcycle_down::1 36 4.155e-05 4.155e-05 4.155e-05 0.01% Castro::initialize_advance() 5 4.059e-05 4.059e-05 4.059e-05 0.01% MLMG::mgVcycle_down::2 36 4.035e-05 4.035e-05 4.035e-05 0.01% MLMG::mgVcycle_down::4 36 3.853e-05 3.853e-05 3.853e-05 0.01% MLMG::mgVcycle_down::3 36 3.731e-05 3.731e-05 3.731e-05 0.01% MLMG::actualBottomSolve() 36 3.657e-05 3.657e-05 3.657e-05 0.01% MLMG::solve() 6 3.286e-05 3.286e-05 3.286e-05 0.01% Castro::initialize_do_advance() 5 3.211e-05 3.211e-05 3.211e-05 0.01% Castro::buildMetrics() 1 3.136e-05 3.136e-05 3.136e-05 0.01% MLMG::mgVcycle_up::4 36 3.123e-05 3.123e-05 3.123e-05 0.01% Gravity::actual_multilevel_solve() 1 3.008e-05 3.008e-05 3.008e-05 0.01% MLMG::mgVcycle_up::0 36 2.862e-05 2.862e-05 2.862e-05 0.01% Castro::post_restart() 1 2.828e-05 2.828e-05 2.828e-05 0.01% Castro::advance() 5 2.798e-05 2.798e-05 2.798e-05 0.01% Castro::swap_state_time_levels() 5 2.745e-05 2.745e-05 2.745e-05 0.01% Amr::writeSmallPlotFile() 1 2.66e-05 2.66e-05 2.66e-05 0.01% Castro::initMFs() 1 2.636e-05 2.636e-05 2.636e-05 0.01% Castro::construct_old_source() 25 2.564e-05 2.564e-05 2.564e-05 0.01% MLMG::mgVcycle_up::3 36 2.528e-05 2.528e-05 2.528e-05 0.01% MLMG::mgVcycle_up::2 36 2.41e-05 2.41e-05 2.41e-05 0.01% MLMG::oneIter() 36 2.395e-05 2.395e-05 2.395e-05 0.01% MLMG::mgVcycle_up::1 36 2.32e-05 2.32e-05 2.32e-05 0.01% MLCellLinOp::solutionResidual() 42 2.214e-05 2.214e-05 2.214e-05 0.01% MLLinOp::define() 6 2.109e-05 2.109e-05 2.109e-05 0.01% MLPoisson::define() 6 1.99e-05 1.99e-05 1.99e-05 0.01% Castro::finalize_do_advance() 5 1.803e-05 1.803e-05 1.803e-05 0.00% MLMG::computeResidual() 36 1.743e-05 1.743e-05 1.743e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.705e-05 1.705e-05 1.705e-05 0.00% MLMG::mgVcycle_bottom 36 1.521e-05 1.521e-05 1.521e-05 0.00% makeSFC 30 1.414e-05 1.414e-05 1.414e-05 0.00% FillPatchSingleLevel 20 1.371e-05 1.371e-05 1.371e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.35e-05 1.35e-05 1.35e-05 0.00% Castro::construct_new_source() 25 9.808e-06 9.808e-06 9.808e-06 0.00% Castro::do_new_sources() 5 8.904e-06 8.904e-06 8.904e-06 0.00% Amr::initSubcycle() 1 8.787e-06 8.787e-06 8.787e-06 0.00% DistributionMapping::Distribute() 31 8.349e-06 8.349e-06 8.349e-06 0.00% Castro::do_old_sources() 5 8.224e-06 8.224e-06 8.224e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.561e-06 7.561e-06 7.561e-06 0.00% Castro::check_for_nan() 10 7.044e-06 7.044e-06 7.044e-06 0.00% Castro::construct_old_gravity() 5 5.871e-06 5.871e-06 5.871e-06 0.00% Castro::apply_source_to_state() 10 5.382e-06 5.382e-06 5.382e-06 0.00% Castro::post_timestep() 5 5.025e-06 5.025e-06 5.025e-06 0.00% MLPoisson::prepareForSolve() 6 4.951e-06 4.951e-06 4.951e-06 0.00% MLMG::computeMLResidual() 6 4.836e-06 4.836e-06 4.836e-06 0.00% Gravity::swapTimeLevels() 5 4.532e-06 4.532e-06 4.532e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.669e-06 3.669e-06 3.669e-06 0.00% Castro::computeNewDt() 5 3.38e-06 3.38e-06 3.38e-06 0.00% MLMG::getGradSolution() 6 2.986e-06 2.986e-06 2.986e-06 0.00% MLMG::MLResNormInf() 6 2.173e-06 2.173e-06 2.173e-06 0.00% Gravity::set_mass_offset() 6 2.167e-06 2.167e-06 2.167e-06 0.00% Castro::retry_advance_ctu() 5 1.92e-06 1.92e-06 1.92e-06 0.00% Castro::FluxRegCrseInit 5 1.8e-06 1.8e-06 1.8e-06 0.00% Castro::FluxRegFineAdd() 5 1.122e-06 1.122e-06 1.122e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.117e-06 1.117e-06 1.117e-06 0.00% Amr::init() 1 1.102e-06 1.102e-06 1.102e-06 0.00% AmrLevel::AmrLevel() 1 9.95e-07 9.95e-07 9.95e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3955 0.3955 0.3955 100.00% Amr::coarseTimeStep() 5 0.3138 0.3138 0.3138 79.34% Amr::timeStep() 5 0.3117 0.3117 0.3117 78.82% Castro::advance() 5 0.3085 0.3085 0.3085 78.01% Castro::subcycle_advance_ctu() 5 0.3026 0.3026 0.3026 76.52% Castro::do_advance_ctu() 5 0.3025 0.3025 0.3025 76.49% Castro::construct_new_gravity() 5 0.1454 0.1454 0.1454 36.77% Gravity::solve_phi_with_mlmg() 6 0.141 0.141 0.141 35.64% Gravity::solve_for_phi() 5 0.1374 0.1374 0.1374 34.74% Gravity::actual_solve_with_mlmg() 6 0.1351 0.1351 0.1351 34.16% MLMG::solve() 6 0.1226 0.1226 0.1226 31.01% Castro::construct_ctu_hydro_source() 5 0.1175 0.1175 0.1175 29.72% MLMG::oneIter() 36 0.1154 0.1154 0.1154 29.18% MLMG::mgVcycle() 36 0.1146 0.1146 0.1146 28.99% MLCellLinOp::smooth() 720 0.05849 0.05849 0.05849 14.79% Amr::init() 1 0.04838 0.04838 0.04838 12.23% Amr::restart() 1 0.04838 0.04838 0.04838 12.23% MLCellLinOp::applyBC() 1946 0.04167 0.04167 0.04167 10.54% AmrLevel::restart() 1 0.04061 0.04061 0.04061 10.27% StateData::restartDoit() 4 0.04053 0.04053 0.04053 10.25% VisMF::Read() 3 0.0404 0.0404 0.0404 10.21% MLMG::mgVcycle_bottom 36 0.0352 0.0352 0.0352 8.90% MLMG::actualBottomSolve() 36 0.03518 0.03518 0.03518 8.90% MLCGSolver::bicgstab 36 0.03483 0.03483 0.03483 8.81% Amr::writePlotFile() 1 0.03263 0.03263 0.03263 8.25% VisMF::Write(FabArray) 1 0.03098 0.03098 0.03098 7.83% MLPoisson::Fsmooth() 1440 0.02718 0.02718 0.02718 6.87% Castro::clean_state() 30 0.02207 0.02207 0.02207 5.58% FillPatchIterator::Initialize 20 0.02052 0.02052 0.02052 5.19% FillPatchSingleLevel 20 0.01974 0.01974 0.01974 4.99% StateDataPhysBCFunct::() 20 0.01774 0.01774 0.01774 4.48% MLCellLinOp::apply() 500 0.01582 0.01582 0.01582 4.00% MLMG::mgVcycle_down::0 36 0.01539 0.01539 0.01539 3.89% MLMG::mgVcycle_up::0 36 0.01311 0.01311 0.01311 3.32% StateData::FillBoundary(geom) 160 0.01269 0.01269 0.01269 3.21% MLPoisson::define() 6 0.0101 0.0101 0.0101 2.56% Castro::initialize_do_advance() 5 0.009652 0.009652 0.009652 2.44% MultiFab::Dot() 484 0.009564 0.009564 0.009564 2.42% MLCellLinOp::correctionResidual() 216 0.009262 0.009262 0.009262 2.34% Castro::computeTemp() 30 0.009145 0.009145 0.009145 2.31% MLMG:computeResOfCorrection() 180 0.008 0.008 0.008 2.02% Gravity::get_new_grav_vector() 5 0.007888 0.007888 0.007888 1.99% MLMG::mgVcycle_down::1 36 0.007671 0.007671 0.007671 1.94% Castro::construct_old_gravity() 5 0.007647 0.007647 0.007647 1.93% Gravity::get_old_grav_vector() 5 0.007641 0.007641 0.007641 1.93% MLMG::mgVcycle_down::2 36 0.00739 0.00739 0.00739 1.87% MLCellLinOp::defineAuxData() 6 0.007117 0.007117 0.007117 1.80% MLMG::mgVcycle_down::3 36 0.007049 0.007049 0.007049 1.78% FabArray::FillBoundary() 1766 0.006997 0.006997 0.006997 1.77% Castro::do_new_sources() 5 0.006949 0.006949 0.006949 1.76% FabArray::setVal() 537 0.006782 0.006782 0.006782 1.71% MLMG::mgVcycle_down::4 36 0.006744 0.006744 0.006744 1.71% FillBoundary_nowait() 1766 0.006604 0.006604 0.006604 1.67% FabArray::ParallelCopy() 380 0.006472 0.006472 0.006472 1.64% FabArray::ParallelCopy_nowait() 380 0.006352 0.006352 0.006352 1.61% Castro::enforce_min_density() 30 0.006336 0.006336 0.006336 1.60% CGSolver::sxay() 690 0.006335 0.006335 0.006335 1.60% MultiFab::LinComb() 690 0.006156 0.006156 0.006156 1.56% Castro::normalize_species() 30 0.005841 0.005841 0.005841 1.48% Castro::initialize_advance() 5 0.005817 0.005817 0.005817 1.47% Gravity::fill_multipole_BCs() 6 0.005746 0.005746 0.005746 1.45% MLCGSolver::ParallelAllReduce 659 0.005735 0.005735 0.005735 1.45% MLMG::mgVcycle_up::2 36 0.005684 0.005684 0.005684 1.44% MLMG::mgVcycle_up::1 36 0.005582 0.005582 0.005582 1.41% MLMG::addInterpCorrection() 180 0.005502 0.005502 0.005502 1.39% Castro::expand_state() 5 0.005413 0.005413 0.005413 1.37% MLMG::mgVcycle_up::3 36 0.005384 0.005384 0.005384 1.36% MLMG::mgVcycle_up::4 36 0.005354 0.005354 0.005354 1.35% amrex::average_down 180 0.00524 0.00524 0.00524 1.33% Castro::do_old_sources() 5 0.005084 0.005084 0.005084 1.29% MLPoisson::Fapply() 500 0.00507 0.00507 0.00507 1.28% Castro::post_restart() 1 0.003951 0.003951 0.003951 1.00% Gravity::multilevel_solve_for_new_phi() 1 0.003829 0.003829 0.003829 0.97% Gravity::actual_multilevel_solve() 1 0.003812 0.003812 0.003812 0.96% MLCellLinOp::solutionResidual() 42 0.00323 0.00323 0.00323 0.82% Castro::post_timestep() 5 0.003091 0.003091 0.003091 0.78% Castro::reset_internal_energy(MultiFab) 30 0.002948 0.002948 0.002948 0.75% Castro::estTimeStep() 10 0.002937 0.002937 0.002937 0.74% MLMG::prepareForSolve() 6 0.002892 0.002892 0.002892 0.73% MultiFab::Xpay() 258 0.00289 0.00289 0.00289 0.73% MLCellLinOp::defineBC() 6 0.002847 0.002847 0.002847 0.72% BndryData::define() 6 0.002698 0.002698 0.002698 0.68% MLMG::computeResidual() 36 0.002684 0.002684 0.002684 0.68% Castro::computeNewDt() 5 0.00187 0.00187 0.00187 0.47% Castro::construct_new_source() 25 0.001652 0.001652 0.001652 0.42% Castro::construct_new_gravity_source() 5 0.001643 0.001643 0.001643 0.42% Castro::construct_old_source() 25 0.001329 0.001329 0.001329 0.34% Castro::construct_old_gravity_source() 5 0.001304 0.001304 0.001304 0.33% MLMG::ResNormInf() 42 0.0009459 0.0009459 0.0009459 0.24% Castro::apply_source_to_state() 10 0.0009191 0.0009191 0.0009191 0.23% MultiFab::Saxpy() 10 0.0009137 0.0009137 0.0009137 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009116 0.0009116 0.0009116 0.23% MLCellLinOp::setLevelBC() 6 0.0008334 0.0008334 0.0008334 0.21% MLMG::getGradSolution() 6 0.0007758 0.0007758 0.0007758 0.20% MLCellLinOp::compGrad() 6 0.0007729 0.0007729 0.0007729 0.20% FabArrayBase::getCPC() 632 0.0007504 0.0007504 0.0007504 0.19% Castro::reset_internal_energy(Fab) 240 0.00075 0.00075 0.00075 0.19% MultiFab::Add() 36 0.0007224 0.0007224 0.0007224 0.18% Castro::enforce_speed_limit() 30 0.0007053 0.0007053 0.0007053 0.18% FabArray::mult() 22 0.0006534 0.0006534 0.0006534 0.17% MLPoisson::prepareForSolve() 6 0.0006445 0.0006445 0.0006445 0.16% FabArray::setDomainBndry() 20 0.0006418 0.0006418 0.0006418 0.16% MLCellLinOp::prepareForSolve() 6 0.0006396 0.0006396 0.0006396 0.16% Castro::check_for_nan() 10 0.0006025 0.0006025 0.0006025 0.15% MultiFab::contains_nan() 10 0.0005955 0.0005955 0.0005955 0.15% MLMG::computeMLResidual() 6 0.0005684 0.0005684 0.0005684 0.14% Gravity::update_max_rhs() 6 0.0004415 0.0004415 0.0004415 0.11% Amr::InitAmr() 1 0.0004225 0.0004225 0.0004225 0.11% FabArrayBase::CPC::define() 244 0.00039 0.00039 0.00039 0.10% FabArrayBase::getFB() 1766 0.000321 0.000321 0.000321 0.08% Gravity::swapTimeLevels() 5 0.0002307 0.0002307 0.0002307 0.06% MultiFab::Copy() 6 0.0001762 0.0001762 0.0001762 0.04% Castro::buildMetrics() 1 0.0001615 0.0001615 0.0001615 0.04% MLMG::MLResNormInf() 6 0.0001518 0.0001518 0.0001518 0.04% MultiFab::max() 6 0.0001368 0.0001368 0.0001368 0.03% MLLinOp::define() 6 0.0001209 0.0001209 0.0001209 0.03% MLMG::MLRhsNormInf() 6 0.0001174 0.0001174 0.0001174 0.03% MLLinOp::defineGrids() 6 9.982e-05 9.982e-05 9.982e-05 0.03% Castro::create_source_corrector() 5 9.775e-05 9.775e-05 9.775e-05 0.02% Castro::finalize_advance() 5 6.514e-05 6.514e-05 6.514e-05 0.02% FabArrayBase::FB::FB() 26 5.636e-05 5.636e-05 5.636e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.87e-05 2.87e-05 2.87e-05 0.01% Castro::swap_state_time_levels() 5 2.745e-05 2.745e-05 2.745e-05 0.01% Amr::writeSmallPlotFile() 1 2.66e-05 2.66e-05 2.66e-05 0.01% Castro::initMFs() 1 2.636e-05 2.636e-05 2.636e-05 0.01% makeSFC 30 2.114e-05 2.114e-05 2.114e-05 0.01% Castro::finalize_do_advance() 5 1.803e-05 1.803e-05 1.803e-05 0.00% Amr::initSubcycle() 1 8.787e-06 8.787e-06 8.787e-06 0.00% DistributionMapping::Distribute() 31 8.349e-06 8.349e-06 8.349e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.023e-06 5.023e-06 5.023e-06 0.00% Gravity::set_mass_offset() 6 2.167e-06 2.167e-06 2.167e-06 0.00% Castro::retry_advance_ctu() 5 1.92e-06 1.92e-06 1.92e-06 0.00% Castro::FluxRegCrseInit 5 1.8e-06 1.8e-06 1.8e-06 0.00% Castro::FluxRegFineAdd() 5 1.122e-06 1.122e-06 1.122e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.117e-06 1.117e-06 1.117e-06 0.00% AmrLevel::AmrLevel() 1 9.95e-07 9.95e-07 9.95e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-17-g8294c3afbcbb) finalized