Initializing CUDA... CUDA initialized with 1 GPU AMReX (23.01-1-g30f2290cf7c3) initialized Starting run at 10:09:07 UTC on 2023-01-02. Successfully read inputs file ... Castro git describe: 22.12-11-g45d01ffe1 AMReX git describe: 23.01-1-g30f2290cf Microphysics git describe: 22.12-28-gdc4a7e21 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.056617842 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.032584212 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.045825597 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.04807282 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.049338065 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.05685962 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.062368876 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.0566395 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.074256869 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.05303309 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.045657065 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.054828106 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.058172341 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.055647711 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032499362 seconds Ending run at 10:09:08 UTC on 2023-01-02. Run time = 0.830262002 Run time without initialization = 0.693836023 Average number of zones advanced per microsecond: 3.778 Average number of zones advanced per microsecond per rank: 3.778 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8303 ... 0.8303 ... 0.8303 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2263 0.2263 0.2263 27.25% Castro::construct_ctu_hydro_source() 10 0.1965 0.1965 0.1965 23.66% MLCellLinOp::applyBC() 4433 0.07284 0.07284 0.07284 8.77% FillBoundary_nowait() 4023 0.03211 0.03211 0.03211 3.87% MLPoisson::Fsmooth() 3280 0.03122 0.03122 0.03122 3.76% StateData::FillBoundary(geom) 328 0.02238 0.02238 0.02238 2.70% Castro::normalize_species() 62 0.02092 0.02092 0.02092 2.52% amrex::Dot() 1114 0.01948 0.01948 0.01948 2.35% amrex::Copy() 1029 0.01466 0.01466 0.01466 1.77% Castro::computeTemp() 63 0.01418 0.01418 0.01418 1.71% StateDataPhysBCFunct::() 41 0.01394 0.01394 0.01394 1.68% FabArray::norminf() 743 0.01376 0.01376 0.01376 1.66% FabArray::setVal() 1144 0.01276 0.01276 0.01276 1.54% FabArray::ParallelCopy_nowait() 861 0.01267 0.01267 0.01267 1.53% MLPoisson::Fapply() 1142 0.01007 0.01007 0.01007 1.21% MLCellLinOp::defineAuxData() 11 0.00933 0.00933 0.00933 1.12% FabArray::Saxpy() 813 0.007885 0.007885 0.007885 0.95% FabArray::Xpay() 821 0.00783 0.00783 0.00783 0.94% Castro::enforce_min_density() 62 0.00642 0.00642 0.00642 0.77% MLMG::addInterpCorrection() 410 0.006371 0.006371 0.006371 0.77% Gravity::fill_multipole_BCs() 11 0.006173 0.006173 0.006173 0.74% amrex::average_down 410 0.005639 0.005639 0.005639 0.68% Amr::checkPoint() 3 0.004885 0.004885 0.004885 0.59% Castro::estTimeStep() 21 0.004881 0.004881 0.004881 0.59% Castro::reset_internal_energy(MultiFab) 63 0.004334 0.004334 0.004334 0.52% FabArray::LinComb() 557 0.004326 0.004326 0.004326 0.52% amrex::Add() 164 0.004279 0.004279 0.004279 0.52% BndryData::define() 11 0.003454 0.003454 0.003454 0.42% Castro::construct_new_gravity_source() 10 0.003268 0.003268 0.003268 0.39% Castro::construct_old_gravity_source() 10 0.002656 0.002656 0.002656 0.32% Castro::do_advance_ctu() 10 0.002602 0.002602 0.002602 0.31% Amr::writePlotFile() 2 0.002342 0.002342 0.002342 0.28% MLCGSolver::bicgstab 82 0.002134 0.002134 0.002134 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001608 0.001608 0.001608 0.19% Castro::reset_internal_energy(Fab) 504 0.001509 0.001509 0.001509 0.18% MLCellLinOp::setLevelBC() 11 0.001368 0.001368 0.001368 0.16% Gravity::actual_solve_with_mlmg() 11 0.001365 0.001365 0.001365 0.16% FabArray::mult() 43 0.001324 0.001324 0.001324 0.16% Castro::enforce_speed_limit() 62 0.00131 0.00131 0.00131 0.16% FabArray::setDomainBndry() 41 0.001271 0.001271 0.001271 0.15% Castro::initData() 1 0.001243 0.001243 0.001243 0.15% MultiFab::contains_nan() 20 0.001205 0.001205 0.001205 0.15% MLCellLinOp::smooth() 1640 0.001196 0.001196 0.001196 0.14% MLCellLinOp::prepareForSolve() 11 0.001079 0.001079 0.001079 0.13% MLCellLinOp::compGrad() 11 0.0009075 0.0009075 0.0009075 0.11% MLMG::prepareForSolve() 11 0.0008312 0.0008312 0.0008312 0.10% FabArray::FillBoundary() 4023 0.000789 0.000789 0.000789 0.10% FabArrayBase::getCPC() 1323 0.0007167 0.0007167 0.0007167 0.09% FabArrayBase::CPC::define() 454 0.0006701 0.0006701 0.0006701 0.08% Gravity::get_new_grav_vector() 11 0.0006062 0.0006062 0.0006062 0.07% FabArrayBase::getFB() 4023 0.0005892 0.0005892 0.0005892 0.07% Castro::create_source_corrector() 10 0.0005776 0.0005776 0.0005776 0.07% Gravity::get_old_grav_vector() 10 0.0005459 0.0005459 0.0005459 0.07% Amr::InitAmr() 1 0.000518 0.000518 0.000518 0.06% MLCellLinOp::apply() 1142 0.0004995 0.0004995 0.0004995 0.06% MLMG::mgVcycle() 82 0.0004381 0.0004381 0.0004381 0.05% Amr::coarseTimeStep() 10 0.000346 0.000346 0.000346 0.04% main() 1 0.0003056 0.0003056 0.0003056 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002839 0.0002839 0.0002839 0.03% MultiFab::max() 11 0.0002631 0.0002631 0.0002631 0.03% FabArray::ParallelCopy() 861 0.0002482 0.0002482 0.0002482 0.03% MLCellLinOp::correctionResidual() 492 0.0002394 0.0002394 0.0002394 0.03% FillPatchIterator::Initialize 41 0.0002089 0.0002089 0.0002089 0.03% MLCellLinOp::defineBC() 11 0.0002037 0.0002037 0.0002037 0.02% MLLinOp::defineGrids() 11 0.000176 0.000176 0.000176 0.02% Gravity::solve_for_phi() 10 0.0001676 0.0001676 0.0001676 0.02% Castro::subcycle_advance_ctu() 10 0.0001547 0.0001547 0.0001547 0.02% Amr::timeStep() 10 0.0001539 0.0001539 0.0001539 0.02% StateData::checkPoint() 12 0.0001352 0.0001352 0.0001352 0.02% Castro::advance() 10 0.0001242 0.0001242 0.0001242 0.01% Gravity::update_max_rhs() 11 0.0001132 0.0001132 0.0001132 0.01% MLMG:computeResOfCorrection() 410 0.0001083 0.0001083 0.0001083 0.01% MLMG::mgVcycle_down::0 82 9.597e-05 9.597e-05 9.597e-05 0.01% MLMG::actualBottomSolve() 82 9.158e-05 9.158e-05 9.158e-05 0.01% Castro::clean_state() 62 9.053e-05 9.053e-05 9.053e-05 0.01% MLMG::mgVcycle_down::1 82 8.934e-05 8.934e-05 8.934e-05 0.01% MLMG::mgVcycle_down::2 82 8.445e-05 8.445e-05 8.445e-05 0.01% FabArrayBase::FB::FB() 56 8.275e-05 8.275e-05 8.275e-05 0.01% MLMG::mgVcycle_down::3 82 8.069e-05 8.069e-05 8.069e-05 0.01% MLMG::mgVcycle_down::4 82 8.059e-05 8.059e-05 8.059e-05 0.01% Castro::expand_state() 10 8.032e-05 8.032e-05 8.032e-05 0.01% Castro::Castro() 1 7.904e-05 7.904e-05 7.904e-05 0.01% AmrLevel::checkPoint() 3 7.532e-05 7.532e-05 7.532e-05 0.01% MLMG::solve() 11 7.249e-05 7.249e-05 7.249e-05 0.01% Castro::finalize_advance() 10 6.909e-05 6.909e-05 6.909e-05 0.01% Castro::initialize_advance() 10 6.539e-05 6.539e-05 6.539e-05 0.01% MLMG::mgVcycle_up::4 82 6.524e-05 6.524e-05 6.524e-05 0.01% MLMG::mgVcycle_up::0 82 5.677e-05 5.677e-05 5.677e-05 0.01% MLMG::mgVcycle_up::1 82 5.384e-05 5.384e-05 5.384e-05 0.01% MLMG::oneIter() 82 5.186e-05 5.186e-05 5.186e-05 0.01% MLMG::mgVcycle_up::3 82 5.181e-05 5.181e-05 5.181e-05 0.01% MLMG::mgVcycle_up::2 82 5.132e-05 5.132e-05 5.132e-05 0.01% MLCellLinOp::solutionResidual() 93 5.106e-05 5.106e-05 5.106e-05 0.01% Castro::initialize_do_advance() 10 4.359e-05 4.359e-05 4.359e-05 0.01% Castro::swap_state_time_levels() 10 4.003e-05 4.003e-05 4.003e-05 0.00% Castro::enforce_consistent_e() 1 3.594e-05 3.594e-05 3.594e-05 0.00% StateData::define() 4 3.49e-05 3.49e-05 3.49e-05 0.00% MLMG::ResNormInf() 93 3.26e-05 3.26e-05 3.26e-05 0.00% MLMG::mgVcycle_bottom 82 3.208e-05 3.208e-05 3.208e-05 0.00% MLMG::computeResidual() 82 3.17e-05 3.17e-05 3.17e-05 0.00% Castro::finalize_do_advance() 10 3.131e-05 3.131e-05 3.131e-05 0.00% FillPatchSingleLevel 41 2.976e-05 2.976e-05 2.976e-05 0.00% makeSFC 55 2.599e-05 2.599e-05 2.599e-05 0.00% Castro::construct_new_gravity() 10 2.4e-05 2.4e-05 2.4e-05 0.00% Amr::writeSmallPlotFile() 1 2.294e-05 2.294e-05 2.294e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.161e-05 2.161e-05 2.161e-05 0.00% MLPoisson::define() 11 2.078e-05 2.078e-05 2.078e-05 0.00% Amr::FinalizeInit() 1 1.979e-05 1.979e-05 1.979e-05 0.00% Castro::construct_old_source() 50 1.93e-05 1.93e-05 1.93e-05 0.00% Castro::initMFs() 1 1.777e-05 1.777e-05 1.777e-05 0.00% Castro::do_new_sources() 10 1.775e-05 1.775e-05 1.775e-05 0.00% Castro::buildMetrics() 1 1.591e-05 1.591e-05 1.591e-05 0.00% DistributionMapping::Distribute() 56 1.579e-05 1.579e-05 1.579e-05 0.00% Castro::construct_old_gravity() 10 1.576e-05 1.576e-05 1.576e-05 0.00% Castro::do_old_sources() 10 1.557e-05 1.557e-05 1.557e-05 0.00% Castro::construct_new_source() 50 1.517e-05 1.517e-05 1.517e-05 0.00% Castro::check_for_nan() 20 1.487e-05 1.487e-05 1.487e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.346e-05 1.346e-05 1.346e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.343e-05 1.343e-05 1.343e-05 0.00% Amr::defBaseLevel() 1 1.336e-05 1.336e-05 1.336e-05 0.00% Castro::apply_source_to_state() 20 1.033e-05 1.033e-05 1.033e-05 0.00% Gravity::swapTimeLevels() 10 9.429e-06 9.429e-06 9.429e-06 0.00% MLLinOp::define() 11 9.402e-06 9.402e-06 9.402e-06 0.00% AmrLevel::AmrLevel(dm) 1 9.355e-06 9.355e-06 9.355e-06 0.00% Amr::initSubcycle() 1 9.174e-06 9.174e-06 9.174e-06 0.00% Castro::post_timestep() 10 9.146e-06 9.146e-06 9.146e-06 0.00% Castro::post_init() 1 9.132e-06 9.132e-06 9.132e-06 0.00% MLPoisson::prepareForSolve() 11 7.874e-06 7.874e-06 7.874e-06 0.00% Gravity::actual_multilevel_solve() 1 7.217e-06 7.217e-06 7.217e-06 0.00% Castro::computeNewDt() 9 7.053e-06 7.053e-06 7.053e-06 0.00% MLMG::computeMLResidual() 11 6.984e-06 6.984e-06 6.984e-06 0.00% MLMG::getGradSolution() 11 5.53e-06 5.53e-06 5.53e-06 0.00% Amr::InitializeInit() 1 5.037e-06 5.037e-06 5.037e-06 0.00% AmrLevel::checkPointPost() 3 5.005e-06 5.005e-06 5.005e-06 0.00% Gravity::set_mass_offset() 11 3.97e-06 3.97e-06 3.97e-06 0.00% MLMG::MLRhsNormInf() 11 3.88e-06 3.88e-06 3.88e-06 0.00% Castro::retry_advance_ctu() 10 3.542e-06 3.542e-06 3.542e-06 0.00% MLMG::MLResNormInf() 11 3.428e-06 3.428e-06 3.428e-06 0.00% Amr::init() 1 3.136e-06 3.136e-06 3.136e-06 0.00% Castro::FluxRegCrseInit 10 2.818e-06 2.818e-06 2.818e-06 0.00% Castro::computeInitialDt() 2 2.773e-06 2.773e-06 2.773e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.727e-06 2.727e-06 2.727e-06 0.00% AmrLevel::checkPointPre() 3 2.092e-06 2.092e-06 2.092e-06 0.00% Castro::FluxRegFineAdd() 10 2.045e-06 2.045e-06 2.045e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.978e-06 1.978e-06 1.978e-06 0.00% Castro::post_regrid() 1 1.071e-06 1.071e-06 1.071e-06 0.00% Amr::initialInit() 1 1.005e-06 1.005e-06 1.005e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8303 0.8303 0.8303 100.00% Amr::coarseTimeStep() 10 0.6611 0.6611 0.6611 79.62% Amr::timeStep() 10 0.5453 0.5453 0.5453 65.68% Castro::advance() 10 0.5384 0.5384 0.5384 64.84% Castro::subcycle_advance_ctu() 10 0.5262 0.5262 0.5262 63.38% Castro::do_advance_ctu() 10 0.526 0.526 0.526 63.36% Gravity::solve_phi_with_mlmg() 11 0.2733 0.2733 0.2733 32.91% Gravity::actual_solve_with_mlmg() 11 0.2666 0.2666 0.2666 32.11% Castro::construct_new_gravity() 10 0.2501 0.2501 0.2501 30.12% MLMG::solve() 11 0.2471 0.2471 0.2471 29.76% Gravity::solve_for_phi() 10 0.2345 0.2345 0.2345 28.25% MLMG::oneIter() 82 0.2332 0.2332 0.2332 28.09% MLMG::mgVcycle() 82 0.2296 0.2296 0.2296 27.65% VisMF::Write(FabArray) 11 0.2263 0.2263 0.2263 27.25% Castro::construct_ctu_hydro_source() 10 0.1965 0.1965 0.1965 23.66% Amr::checkPoint() 3 0.169 0.169 0.169 20.36% AmrLevel::checkPoint() 3 0.1641 0.1641 0.1641 19.77% StateData::checkPoint() 12 0.1641 0.1641 0.1641 19.76% Amr::init() 1 0.1358 0.1358 0.1358 16.35% MLCellLinOp::smooth() 1640 0.1139 0.1139 0.1139 13.72% MLCellLinOp::applyBC() 4433 0.1064 0.1064 0.1064 12.82% MLMG::mgVcycle_bottom 82 0.07031 0.07031 0.07031 8.47% MLMG::actualBottomSolve() 82 0.07028 0.07028 0.07028 8.46% MLCGSolver::bicgstab 82 0.06961 0.06961 0.06961 8.38% Amr::writePlotFile() 2 0.06522 0.06522 0.06522 7.85% Castro::clean_state() 62 0.04792 0.04792 0.04792 5.77% Amr::initialInit() 1 0.04642 0.04642 0.04642 5.59% Amr::FinalizeInit() 1 0.04227 0.04227 0.04227 5.09% FillPatchIterator::Initialize 41 0.04183 0.04183 0.04183 5.04% Castro::post_init() 1 0.04099 0.04099 0.04099 4.94% FillPatchSingleLevel 41 0.04035 0.04035 0.04035 4.86% Gravity::multilevel_solve_for_new_phi() 1 0.03921 0.03921 0.03921 4.72% Gravity::actual_multilevel_solve() 1 0.03919 0.03919 0.03919 4.72% StateDataPhysBCFunct::() 41 0.03632 0.03632 0.03632 4.37% MLCellLinOp::apply() 1142 0.03506 0.03506 0.03506 4.22% MLMG::mgVcycle_down::0 82 0.03368 0.03368 0.03368 4.06% FabArray::FillBoundary() 4023 0.03357 0.03357 0.03357 4.04% FillBoundary_nowait() 4023 0.03278 0.03278 0.03278 3.95% MLPoisson::Fsmooth() 3280 0.03122 0.03122 0.03122 3.76% MLMG::mgVcycle_up::0 82 0.02559 0.02559 0.02559 3.08% Castro::initialize_do_advance() 10 0.02443 0.02443 0.02443 2.94% StateData::FillBoundary(geom) 328 0.02238 0.02238 0.02238 2.70% MLCellLinOp::correctionResidual() 492 0.02162 0.02162 0.02162 2.60% Castro::normalize_species() 62 0.02092 0.02092 0.02092 2.52% Castro::computeTemp() 63 0.02003 0.02003 0.02003 2.41% amrex::Dot() 1114 0.01948 0.01948 0.01948 2.35% MLMG:computeResOfCorrection() 410 0.01909 0.01909 0.01909 2.30% Gravity::get_new_grav_vector() 11 0.01683 0.01683 0.01683 2.03% MLPoisson::define() 11 0.01545 0.01545 0.01545 1.86% MLMG::mgVcycle_down::1 82 0.01508 0.01508 0.01508 1.82% amrex::Copy() 1029 0.01466 0.01466 0.01466 1.77% Castro::construct_old_gravity() 10 0.01453 0.01453 0.01453 1.75% Gravity::get_old_grav_vector() 10 0.01451 0.01451 0.01451 1.75% MLMG::mgVcycle_down::2 82 0.01404 0.01404 0.01404 1.69% FabArray::norminf() 743 0.01376 0.01376 0.01376 1.66% MLMG::mgVcycle_down::3 82 0.01375 0.01375 0.01375 1.66% FabArray::ParallelCopy() 861 0.01372 0.01372 0.01372 1.65% MLMG::mgVcycle_down::4 82 0.01353 0.01353 0.01353 1.63% FabArray::ParallelCopy_nowait() 861 0.01347 0.01347 0.01347 1.62% FabArray::setVal() 1144 0.01276 0.01276 0.01276 1.54% Castro::expand_state() 10 0.01195 0.01195 0.01195 1.44% MLCGSolver::ParallelAllReduce 1514 0.01171 0.01171 0.01171 1.41% Castro::do_new_sources() 10 0.0115 0.0115 0.0115 1.38% Castro::initialize_advance() 10 0.01148 0.01148 0.01148 1.38% MLMG::addInterpCorrection() 410 0.01126 0.01126 0.01126 1.36% MLMG::mgVcycle_up::4 82 0.01104 0.01104 0.01104 1.33% MLMG::mgVcycle_up::1 82 0.01095 0.01095 0.01095 1.32% MLMG::mgVcycle_up::2 82 0.01068 0.01068 0.01068 1.29% MLCellLinOp::defineAuxData() 11 0.01062 0.01062 0.01062 1.28% MLMG::mgVcycle_up::3 82 0.01053 0.01053 0.01053 1.27% amrex::average_down 410 0.01047 0.01047 0.01047 1.26% Castro::do_old_sources() 10 0.01041 0.01041 0.01041 1.25% MLPoisson::Fapply() 1142 0.01007 0.01007 0.01007 1.21% FabArray::Saxpy() 813 0.007885 0.007885 0.007885 0.95% FabArray::Xpay() 821 0.00783 0.00783 0.00783 0.94% MLCellLinOp::solutionResidual() 93 0.006992 0.006992 0.006992 0.84% Castro::post_timestep() 10 0.006798 0.006798 0.006798 0.82% Gravity::fill_multipole_BCs() 11 0.006426 0.006426 0.006426 0.77% Castro::enforce_min_density() 62 0.00642 0.00642 0.00642 0.77% MLMG::computeResidual() 82 0.006023 0.006023 0.006023 0.73% Castro::reset_internal_energy(MultiFab) 63 0.005843 0.005843 0.005843 0.70% Castro::estTimeStep() 21 0.004881 0.004881 0.004881 0.59% MLCellLinOp::defineBC() 11 0.004564 0.004564 0.004564 0.55% MLMG::prepareForSolve() 11 0.004492 0.004492 0.004492 0.54% BndryData::define() 11 0.00436 0.00436 0.00436 0.53% FabArray::LinComb() 557 0.004326 0.004326 0.004326 0.52% amrex::Add() 164 0.004279 0.004279 0.004279 0.52% Amr::InitializeInit() 1 0.004152 0.004152 0.004152 0.50% Amr::defBaseLevel() 1 0.004147 0.004147 0.004147 0.50% Castro::initData() 1 0.003677 0.003677 0.003677 0.44% Castro::construct_new_source() 50 0.003283 0.003283 0.003283 0.40% Castro::construct_new_gravity_source() 10 0.003268 0.003268 0.003268 0.39% Castro::construct_old_source() 50 0.002676 0.002676 0.002676 0.32% Castro::construct_old_gravity_source() 10 0.002656 0.002656 0.002656 0.32% Castro::computeNewDt() 9 0.002352 0.002352 0.002352 0.28% MLMG::ResNormInf() 93 0.002078 0.002078 0.002078 0.25% Castro::apply_source_to_state() 20 0.001823 0.001823 0.001823 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001608 0.001608 0.001608 0.19% Castro::reset_internal_energy(Fab) 504 0.001509 0.001509 0.001509 0.18% FabArrayBase::getCPC() 1323 0.001387 0.001387 0.001387 0.17% MLMG::getGradSolution() 11 0.001368 0.001368 0.001368 0.16% MLCellLinOp::setLevelBC() 11 0.001368 0.001368 0.001368 0.16% MLCellLinOp::compGrad() 11 0.001363 0.001363 0.001363 0.16% FabArray::mult() 43 0.001324 0.001324 0.001324 0.16% Castro::enforce_speed_limit() 62 0.00131 0.00131 0.00131 0.16% FabArray::setDomainBndry() 41 0.001271 0.001271 0.001271 0.15% Castro::check_for_nan() 20 0.00122 0.00122 0.00122 0.15% MultiFab::contains_nan() 20 0.001205 0.001205 0.001205 0.15% Castro::post_regrid() 1 0.001102 0.001102 0.001102 0.13% MLPoisson::prepareForSolve() 11 0.001086 0.001086 0.001086 0.13% MLCellLinOp::prepareForSolve() 11 0.001079 0.001079 0.001079 0.13% MLMG::computeMLResidual() 11 0.001008 0.001008 0.001008 0.12% Castro::computeInitialDt() 2 0.0008654 0.0008654 0.0008654 0.10% Gravity::update_max_rhs() 11 0.00081 0.00081 0.00081 0.10% FabArrayBase::getFB() 4023 0.0006719 0.0006719 0.0006719 0.08% FabArrayBase::CPC::define() 454 0.0006701 0.0006701 0.0006701 0.08% Castro::create_source_corrector() 10 0.0005776 0.0005776 0.0005776 0.07% Castro::finalize_advance() 10 0.000577 0.000577 0.000577 0.07% Amr::InitAmr() 1 0.0005271 0.0005271 0.0005271 0.06% Gravity::swapTimeLevels() 10 0.0004272 0.0004272 0.0004272 0.05% Castro::Castro() 1 0.0004085 0.0004085 0.0004085 0.05% MLMG::MLResNormInf() 11 0.0002861 0.0002861 0.0002861 0.03% MultiFab::max() 11 0.0002631 0.0002631 0.0002631 0.03% MLLinOp::define() 11 0.0002414 0.0002414 0.0002414 0.03% MLLinOp::defineGrids() 11 0.0002319 0.0002319 0.0002319 0.03% MLMG::MLRhsNormInf() 11 0.0002142 0.0002142 0.0002142 0.03% Castro::buildMetrics() 1 0.0001526 0.0001526 0.0001526 0.02% FabArrayBase::FB::FB() 56 8.275e-05 8.275e-05 8.275e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.393e-05 5.393e-05 5.393e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.425e-05 4.425e-05 4.425e-05 0.01% makeSFC 55 4.051e-05 4.051e-05 4.051e-05 0.00% Castro::swap_state_time_levels() 10 4.003e-05 4.003e-05 4.003e-05 0.00% Castro::enforce_consistent_e() 1 3.594e-05 3.594e-05 3.594e-05 0.00% StateData::define() 4 3.49e-05 3.49e-05 3.49e-05 0.00% Castro::finalize_do_advance() 10 3.131e-05 3.131e-05 3.131e-05 0.00% Amr::writeSmallPlotFile() 1 2.294e-05 2.294e-05 2.294e-05 0.00% Castro::initMFs() 1 1.777e-05 1.777e-05 1.777e-05 0.00% DistributionMapping::Distribute() 56 1.579e-05 1.579e-05 1.579e-05 0.00% Amr::initSubcycle() 1 9.174e-06 9.174e-06 9.174e-06 0.00% AmrLevel::checkPointPost() 3 5.005e-06 5.005e-06 5.005e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.01e-06 4.01e-06 4.01e-06 0.00% Gravity::set_mass_offset() 11 3.97e-06 3.97e-06 3.97e-06 0.00% Castro::retry_advance_ctu() 10 3.542e-06 3.542e-06 3.542e-06 0.00% Castro::FluxRegCrseInit 10 2.818e-06 2.818e-06 2.818e-06 0.00% AmrLevel::checkPointPre() 3 2.092e-06 2.092e-06 2.092e-06 0.00% Castro::FluxRegFineAdd() 10 2.045e-06 2.045e-06 2.045e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.978e-06 1.978e-06 1.978e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-1-g30f2290cf7c3) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (23.01-1-g30f2290cf7c3) initialized Starting run at 10:09:09 UTC on 2023-01-02. Successfully read inputs file ... Castro git describe: 22.12-11-g45d01ffe1 AMReX git describe: 23.01-1-g30f2290cf Microphysics git describe: 22.12-28-gdc4a7e21 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.455531265 Restart time = 0.059768918 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.045441137 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047491518 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.054738244 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057025277 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.060584891 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032269333 seconds Ending run at 10:09:09 UTC on 2023-01-02. Run time = 0.358279422 Run time without initialization = 0.297932258 Average number of zones advanced per microsecond: 4.399 Average number of zones advanced per microsecond per rank: 4.399 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3583 ... 0.3583 ... 0.3583 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0957 0.0957 0.0957 26.70% VisMF::Read() 3 0.04048 0.04048 0.04048 11.30% MLCellLinOp::applyBC() 1946 0.0316 0.0316 0.0316 8.82% VisMF::Write(FabArray) 1 0.03075 0.03075 0.03075 8.58% Amr::restart() 1 0.0157 0.0157 0.0157 4.38% MLPoisson::Fsmooth() 1440 0.01362 0.01362 0.01362 3.80% FillBoundary_nowait() 1766 0.01257 0.01257 0.01257 3.51% StateData::FillBoundary(geom) 160 0.01079 0.01079 0.01079 3.01% amrex::Dot() 484 0.008383 0.008383 0.008383 2.34% Castro::normalize_species() 30 0.007381 0.007381 0.007381 2.06% amrex::Copy() 463 0.006852 0.006852 0.006852 1.91% FabArray::setVal() 537 0.006089 0.006089 0.006089 1.70% FabArray::norminf() 326 0.005978 0.005978 0.005978 1.67% FabArray::ParallelCopy_nowait() 380 0.005794 0.005794 0.005794 1.62% StateDataPhysBCFunct::() 20 0.005553 0.005553 0.005553 1.55% MLCellLinOp::defineAuxData() 6 0.005106 0.005106 0.005106 1.42% Castro::computeTemp() 30 0.005034 0.005034 0.005034 1.40% MLPoisson::Fapply() 500 0.004335 0.004335 0.004335 1.21% Castro::enforce_min_density() 30 0.003804 0.003804 0.003804 1.06% FabArray::Saxpy() 355 0.003532 0.003532 0.003532 0.99% FabArray::Xpay() 361 0.003422 0.003422 0.003422 0.96% Gravity::fill_multipole_BCs() 6 0.00279 0.00279 0.00279 0.78% MLMG::addInterpCorrection() 180 0.002757 0.002757 0.002757 0.77% amrex::average_down 180 0.002463 0.002463 0.002463 0.69% Castro::estTimeStep() 10 0.002073 0.002073 0.002073 0.58% FabArray::LinComb() 242 0.001887 0.001887 0.001887 0.53% BndryData::define() 6 0.001877 0.001877 0.001877 0.52% amrex::Add() 72 0.001822 0.001822 0.001822 0.51% Castro::reset_internal_energy(MultiFab) 30 0.001642 0.001642 0.001642 0.46% Castro::construct_new_gravity_source() 5 0.00138 0.00138 0.00138 0.39% Amr::writePlotFile() 1 0.001334 0.001334 0.001334 0.37% MLCGSolver::bicgstab 36 0.0009552 0.0009552 0.0009552 0.27% Castro::do_advance_ctu() 5 0.0009401 0.0009401 0.0009401 0.26% Castro::construct_old_gravity_source() 5 0.0009095 0.0009095 0.0009095 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008755 0.0008755 0.0008755 0.24% MLCellLinOp::setLevelBC() 6 0.0007187 0.0007187 0.0007187 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007104 0.0007104 0.0007104 0.20% Castro::enforce_speed_limit() 30 0.0006757 0.0006757 0.0006757 0.19% Castro::reset_internal_energy(Fab) 240 0.0006619 0.0006619 0.0006619 0.18% FabArray::mult() 22 0.0006488 0.0006488 0.0006488 0.18% FabArray::setDomainBndry() 20 0.0006298 0.0006298 0.0006298 0.18% MultiFab::contains_nan() 10 0.0005827 0.0005827 0.0005827 0.16% MLCellLinOp::prepareForSolve() 6 0.0005721 0.0005721 0.0005721 0.16% MLCellLinOp::smooth() 720 0.000519 0.000519 0.000519 0.14% MLCellLinOp::compGrad() 6 0.000477 0.000477 0.000477 0.13% MLMG::prepareForSolve() 6 0.0004637 0.0004637 0.0004637 0.13% FabArrayBase::CPC::define() 244 0.0004082 0.0004082 0.0004082 0.11% Amr::InitAmr() 1 0.0003945 0.0003945 0.0003945 0.11% FabArrayBase::getCPC() 632 0.0003456 0.0003456 0.0003456 0.10% FabArray::FillBoundary() 1766 0.0003358 0.0003358 0.0003358 0.09% Gravity::get_old_grav_vector() 5 0.000284 0.000284 0.000284 0.08% main() 1 0.0002692 0.0002692 0.0002692 0.08% Gravity::get_new_grav_vector() 5 0.0002686 0.0002686 0.0002686 0.07% FabArrayBase::getFB() 1766 0.0002552 0.0002552 0.0002552 0.07% MLCellLinOp::apply() 500 0.0002111 0.0002111 0.0002111 0.06% MLMG::mgVcycle() 36 0.0001831 0.0001831 0.0001831 0.05% Amr::coarseTimeStep() 5 0.0001552 0.0001552 0.0001552 0.04% MultiFab::max() 6 0.0001339 0.0001339 0.0001339 0.04% MLCGSolver::ParallelAllReduce 659 0.000122 0.000122 0.000122 0.03% MLLinOp::defineGrids() 6 0.0001118 0.0001118 0.0001118 0.03% MLCellLinOp::correctionResidual() 216 0.0001102 0.0001102 0.0001102 0.03% FabArray::ParallelCopy() 380 0.0001098 0.0001098 0.0001098 0.03% MLCellLinOp::defineBC() 6 0.000105 0.000105 0.000105 0.03% FillPatchIterator::Initialize 20 0.0001034 0.0001034 0.0001034 0.03% Amr::timeStep() 5 7.601e-05 7.601e-05 7.601e-05 0.02% AmrLevel::restart() 1 7.554e-05 7.554e-05 7.554e-05 0.02% Gravity::solve_for_phi() 5 6.757e-05 6.757e-05 6.757e-05 0.02% Castro::subcycle_advance_ctu() 5 6.74e-05 6.74e-05 6.74e-05 0.02% StateData::restartDoit() 4 6.227e-05 6.227e-05 6.227e-05 0.02% Gravity::update_max_rhs() 6 6.068e-05 6.068e-05 6.068e-05 0.02% FabArrayBase::FB::FB() 26 5.978e-05 5.978e-05 5.978e-05 0.02% MLMG:computeResOfCorrection() 180 4.935e-05 4.935e-05 4.935e-05 0.01% MLMG::mgVcycle_down::0 36 4.493e-05 4.493e-05 4.493e-05 0.01% MLMG::mgVcycle_down::1 36 4.218e-05 4.218e-05 4.218e-05 0.01% Castro::clean_state() 30 4.078e-05 4.078e-05 4.078e-05 0.01% MLMG::actualBottomSolve() 36 4.056e-05 4.056e-05 4.056e-05 0.01% Castro::finalize_advance() 5 4.038e-05 4.038e-05 4.038e-05 0.01% MLMG::mgVcycle_down::4 36 3.954e-05 3.954e-05 3.954e-05 0.01% MLMG::mgVcycle_down::2 36 3.87e-05 3.87e-05 3.87e-05 0.01% Castro::expand_state() 5 3.804e-05 3.804e-05 3.804e-05 0.01% MLMG::mgVcycle_down::3 36 3.775e-05 3.775e-05 3.775e-05 0.01% MLMG::solve() 6 3.639e-05 3.639e-05 3.639e-05 0.01% MLMG::mgVcycle_up::4 36 3.539e-05 3.539e-05 3.539e-05 0.01% Castro::initialize_advance() 5 3.374e-05 3.374e-05 3.374e-05 0.01% MLMG::mgVcycle_up::0 36 2.578e-05 2.578e-05 2.578e-05 0.01% Castro::buildMetrics() 1 2.541e-05 2.541e-05 2.541e-05 0.01% Amr::writeSmallPlotFile() 1 2.455e-05 2.455e-05 2.455e-05 0.01% MLMG::mgVcycle_up::3 36 2.38e-05 2.38e-05 2.38e-05 0.01% Castro::initialize_do_advance() 5 2.35e-05 2.35e-05 2.35e-05 0.01% MLMG::mgVcycle_up::2 36 2.344e-05 2.344e-05 2.344e-05 0.01% MLCellLinOp::solutionResidual() 42 2.32e-05 2.32e-05 2.32e-05 0.01% MLMG::oneIter() 36 2.268e-05 2.268e-05 2.268e-05 0.01% MLMG::mgVcycle_up::1 36 2.26e-05 2.26e-05 2.26e-05 0.01% Castro::post_restart() 1 2.215e-05 2.215e-05 2.215e-05 0.01% Castro::swap_state_time_levels() 5 2.084e-05 2.084e-05 2.084e-05 0.01% Castro::finalize_do_advance() 5 1.921e-05 1.921e-05 1.921e-05 0.01% Castro::initMFs() 1 1.899e-05 1.899e-05 1.899e-05 0.01% Castro::construct_new_gravity() 5 1.648e-05 1.648e-05 1.648e-05 0.00% MLMG::ResNormInf() 42 1.632e-05 1.632e-05 1.632e-05 0.00% FillPatchSingleLevel 20 1.456e-05 1.456e-05 1.456e-05 0.00% MLMG::mgVcycle_bottom 36 1.455e-05 1.455e-05 1.455e-05 0.00% makeSFC 30 1.414e-05 1.414e-05 1.414e-05 0.00% MLPoisson::define() 6 1.411e-05 1.411e-05 1.411e-05 0.00% MLMG::computeResidual() 36 1.344e-05 1.344e-05 1.344e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.334e-05 1.334e-05 1.334e-05 0.00% Castro::advance() 5 1.252e-05 1.252e-05 1.252e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.24e-05 1.24e-05 1.24e-05 0.00% Castro::construct_new_source() 25 1.086e-05 1.086e-05 1.086e-05 0.00% Castro::construct_old_source() 25 9.567e-06 9.567e-06 9.567e-06 0.00% DistributionMapping::Distribute() 31 9.013e-06 9.013e-06 9.013e-06 0.00% Castro::do_new_sources() 5 9.012e-06 9.012e-06 9.012e-06 0.00% Castro::do_old_sources() 5 8.812e-06 8.812e-06 8.812e-06 0.00% Amr::initSubcycle() 1 8.203e-06 8.203e-06 8.203e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.139e-06 7.139e-06 7.139e-06 0.00% Gravity::actual_multilevel_solve() 1 6.931e-06 6.931e-06 6.931e-06 0.00% Castro::retry_advance_ctu() 5 6.812e-06 6.812e-06 6.812e-06 0.00% Castro::check_for_nan() 10 6.681e-06 6.681e-06 6.681e-06 0.00% Castro::apply_source_to_state() 10 5.515e-06 5.515e-06 5.515e-06 0.00% Castro::post_timestep() 5 5.255e-06 5.255e-06 5.255e-06 0.00% MLLinOp::define() 6 5.006e-06 5.006e-06 5.006e-06 0.00% Castro::construct_old_gravity() 5 5.001e-06 5.001e-06 5.001e-06 0.00% Gravity::swapTimeLevels() 5 4.495e-06 4.495e-06 4.495e-06 0.00% MLPoisson::prepareForSolve() 6 4.12e-06 4.12e-06 4.12e-06 0.00% Castro::computeNewDt() 5 3.374e-06 3.374e-06 3.374e-06 0.00% MLMG::computeMLResidual() 6 3.176e-06 3.176e-06 3.176e-06 0.00% MLMG::getGradSolution() 6 2.986e-06 2.986e-06 2.986e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.945e-06 2.945e-06 2.945e-06 0.00% AmrLevel::AmrLevel() 1 2.921e-06 2.921e-06 2.921e-06 0.00% MLMG::MLResNormInf() 6 2.197e-06 2.197e-06 2.197e-06 0.00% MLMG::MLRhsNormInf() 6 2.062e-06 2.062e-06 2.062e-06 0.00% Castro::create_source_corrector() 5 1.969e-06 1.969e-06 1.969e-06 0.00% Gravity::set_mass_offset() 6 1.845e-06 1.845e-06 1.845e-06 0.00% Castro::FluxRegCrseInit 5 1.407e-06 1.407e-06 1.407e-06 0.00% Castro::FluxRegFineAdd() 5 1.082e-06 1.082e-06 1.082e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.079e-06 1.079e-06 1.079e-06 0.00% Amr::init() 1 8.87e-07 8.87e-07 8.87e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3583 0.3583 0.3583 99.99% Amr::coarseTimeStep() 5 0.2654 0.2654 0.2654 74.08% Amr::timeStep() 5 0.2639 0.2639 0.2639 73.66% Castro::advance() 5 0.2603 0.2603 0.2603 72.65% Castro::subcycle_advance_ctu() 5 0.2532 0.2532 0.2532 70.67% Castro::do_advance_ctu() 5 0.2531 0.2531 0.2531 70.65% Castro::construct_new_gravity() 5 0.1245 0.1245 0.1245 34.75% Gravity::solve_phi_with_mlmg() 6 0.1201 0.1201 0.1201 33.52% Gravity::solve_for_phi() 5 0.1172 0.1172 0.1172 32.71% Gravity::actual_solve_with_mlmg() 6 0.1171 0.1171 0.1171 32.67% MLMG::solve() 6 0.1064 0.1064 0.1064 29.70% MLMG::oneIter() 36 0.09974 0.09974 0.09974 27.84% MLMG::mgVcycle() 36 0.09822 0.09822 0.09822 27.41% Castro::construct_ctu_hydro_source() 5 0.09568 0.09568 0.09568 26.70% Amr::init() 1 0.05981 0.05981 0.05981 16.69% Amr::restart() 1 0.05981 0.05981 0.05981 16.69% MLCellLinOp::smooth() 720 0.0482 0.0482 0.0482 13.45% MLCellLinOp::applyBC() 1946 0.04482 0.04482 0.04482 12.51% AmrLevel::restart() 1 0.04068 0.04068 0.04068 11.35% StateData::restartDoit() 4 0.0406 0.0406 0.0406 11.33% VisMF::Read() 3 0.04048 0.04048 0.04048 11.30% Amr::writePlotFile() 1 0.03235 0.03235 0.03235 9.03% VisMF::Write(FabArray) 1 0.03075 0.03075 0.03075 8.58% MLMG::mgVcycle_bottom 36 0.0304 0.0304 0.0304 8.49% MLMG::actualBottomSolve() 36 0.03039 0.03039 0.03039 8.48% MLCGSolver::bicgstab 36 0.03009 0.03009 0.03009 8.40% Castro::clean_state() 30 0.01924 0.01924 0.01924 5.37% FillPatchIterator::Initialize 20 0.01908 0.01908 0.01908 5.32% FillPatchSingleLevel 20 0.01835 0.01835 0.01835 5.12% StateDataPhysBCFunct::() 20 0.01634 0.01634 0.01634 4.56% MLCellLinOp::apply() 500 0.01506 0.01506 0.01506 4.20% MLMG::mgVcycle_down::0 36 0.01387 0.01387 0.01387 3.87% MLPoisson::Fsmooth() 1440 0.01362 0.01362 0.01362 3.80% FabArray::FillBoundary() 1766 0.01322 0.01322 0.01322 3.69% FillBoundary_nowait() 1766 0.01289 0.01289 0.01289 3.60% StateData::FillBoundary(geom) 160 0.01079 0.01079 0.01079 3.01% MLMG::mgVcycle_up::0 36 0.01047 0.01047 0.01047 2.92% MLCellLinOp::correctionResidual() 216 0.0092 0.0092 0.0092 2.57% Castro::initialize_do_advance() 5 0.008777 0.008777 0.008777 2.45% MLPoisson::define() 6 0.008505 0.008505 0.008505 2.37% amrex::Dot() 484 0.008383 0.008383 0.008383 2.34% MLMG:computeResOfCorrection() 180 0.00809 0.00809 0.00809 2.26% Castro::normalize_species() 30 0.007381 0.007381 0.007381 2.06% Castro::computeTemp() 30 0.007338 0.007338 0.007338 2.05% Gravity::get_new_grav_vector() 5 0.007181 0.007181 0.007181 2.00% Castro::construct_old_gravity() 5 0.006902 0.006902 0.006902 1.93% Gravity::get_old_grav_vector() 5 0.006897 0.006897 0.006897 1.92% amrex::Copy() 463 0.006852 0.006852 0.006852 1.91% Castro::initialize_advance() 5 0.006772 0.006772 0.006772 1.89% MLMG::mgVcycle_down::1 36 0.006553 0.006553 0.006553 1.83% FabArray::ParallelCopy() 380 0.006276 0.006276 0.006276 1.75% FabArray::ParallelCopy_nowait() 380 0.006166 0.006166 0.006166 1.72% MLMG::mgVcycle_down::2 36 0.00612 0.00612 0.00612 1.71% FabArray::setVal() 537 0.006089 0.006089 0.006089 1.70% FabArray::norminf() 326 0.005978 0.005978 0.005978 1.67% MLMG::mgVcycle_down::3 36 0.005944 0.005944 0.005944 1.66% MLMG::mgVcycle_down::4 36 0.005883 0.005883 0.005883 1.64% MLCellLinOp::defineAuxData() 6 0.005836 0.005836 0.005836 1.63% Castro::expand_state() 5 0.005603 0.005603 0.005603 1.56% Castro::do_new_sources() 5 0.005478 0.005478 0.005478 1.53% MLCGSolver::ParallelAllReduce 659 0.005057 0.005057 0.005057 1.41% MLMG::addInterpCorrection() 180 0.004905 0.004905 0.004905 1.37% MLMG::mgVcycle_up::4 36 0.004797 0.004797 0.004797 1.34% MLMG::mgVcycle_up::1 36 0.004771 0.004771 0.004771 1.33% MLMG::mgVcycle_up::2 36 0.004657 0.004657 0.004657 1.30% amrex::average_down 180 0.0046 0.0046 0.0046 1.28% MLMG::mgVcycle_up::3 36 0.00456 0.00456 0.00456 1.27% MLPoisson::Fapply() 500 0.004335 0.004335 0.004335 1.21% Castro::enforce_min_density() 30 0.003804 0.003804 0.003804 1.06% Castro::do_old_sources() 5 0.003626 0.003626 0.003626 1.01% Castro::post_timestep() 5 0.00357 0.00357 0.00357 1.00% FabArray::Saxpy() 355 0.003532 0.003532 0.003532 0.99% FabArray::Xpay() 361 0.003422 0.003422 0.003422 0.96% Castro::post_restart() 1 0.003263 0.003263 0.003263 0.91% MLCellLinOp::solutionResidual() 42 0.003154 0.003154 0.003154 0.88% Gravity::multilevel_solve_for_new_phi() 1 0.003149 0.003149 0.003149 0.88% Gravity::actual_multilevel_solve() 1 0.003136 0.003136 0.003136 0.88% Gravity::fill_multipole_BCs() 6 0.002928 0.002928 0.002928 0.82% MLMG::computeResidual() 36 0.002617 0.002617 0.002617 0.73% MLCellLinOp::defineBC() 6 0.002508 0.002508 0.002508 0.70% MLMG::prepareForSolve() 6 0.002415 0.002415 0.002415 0.67% BndryData::define() 6 0.002403 0.002403 0.002403 0.67% Castro::reset_internal_energy(MultiFab) 30 0.002304 0.002304 0.002304 0.64% Castro::estTimeStep() 10 0.002073 0.002073 0.002073 0.58% FabArray::LinComb() 242 0.001887 0.001887 0.001887 0.53% amrex::Add() 72 0.001822 0.001822 0.001822 0.51% Castro::construct_new_source() 25 0.001391 0.001391 0.001391 0.39% Castro::construct_new_gravity_source() 5 0.00138 0.00138 0.00138 0.39% Castro::computeNewDt() 5 0.001334 0.001334 0.001334 0.37% MLMG::ResNormInf() 42 0.0009218 0.0009218 0.0009218 0.26% Castro::construct_old_source() 25 0.000919 0.000919 0.000919 0.26% Castro::apply_source_to_state() 10 0.0009157 0.0009157 0.0009157 0.26% Castro::construct_old_gravity_source() 5 0.0009095 0.0009095 0.0009095 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008755 0.0008755 0.0008755 0.24% FabArrayBase::getCPC() 632 0.0007538 0.0007538 0.0007538 0.21% MLMG::getGradSolution() 6 0.0007332 0.0007332 0.0007332 0.20% MLCellLinOp::compGrad() 6 0.0007302 0.0007302 0.0007302 0.20% MLCellLinOp::setLevelBC() 6 0.0007187 0.0007187 0.0007187 0.20% Castro::enforce_speed_limit() 30 0.0006757 0.0006757 0.0006757 0.19% Castro::reset_internal_energy(Fab) 240 0.0006619 0.0006619 0.0006619 0.18% FabArray::mult() 22 0.0006488 0.0006488 0.0006488 0.18% FabArray::setDomainBndry() 20 0.0006298 0.0006298 0.0006298 0.18% Castro::check_for_nan() 10 0.0005893 0.0005893 0.0005893 0.16% MultiFab::contains_nan() 10 0.0005827 0.0005827 0.0005827 0.16% MLPoisson::prepareForSolve() 6 0.0005762 0.0005762 0.0005762 0.16% MLCellLinOp::prepareForSolve() 6 0.0005721 0.0005721 0.0005721 0.16% MLMG::computeMLResidual() 6 0.0005535 0.0005535 0.0005535 0.15% Gravity::update_max_rhs() 6 0.0004481 0.0004481 0.0004481 0.13% FabArrayBase::CPC::define() 244 0.0004082 0.0004082 0.0004082 0.11% Amr::InitAmr() 1 0.0004027 0.0004027 0.0004027 0.11% FabArrayBase::getFB() 1766 0.000315 0.000315 0.000315 0.09% Castro::finalize_advance() 5 0.000295 0.000295 0.000295 0.08% Gravity::swapTimeLevels() 5 0.0002215 0.0002215 0.0002215 0.06% MLMG::MLResNormInf() 6 0.0001477 0.0001477 0.0001477 0.04% MLLinOp::define() 6 0.0001469 0.0001469 0.0001469 0.04% MLLinOp::defineGrids() 6 0.0001418 0.0001418 0.0001418 0.04% Castro::buildMetrics() 1 0.0001413 0.0001413 0.0001413 0.04% MultiFab::max() 6 0.0001339 0.0001339 0.0001339 0.04% MLMG::MLRhsNormInf() 6 0.0001137 0.0001137 0.0001137 0.03% FabArrayBase::FB::FB() 26 5.978e-05 5.978e-05 5.978e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.898e-05 2.898e-05 2.898e-05 0.01% Amr::writeSmallPlotFile() 1 2.455e-05 2.455e-05 2.455e-05 0.01% makeSFC 30 2.185e-05 2.185e-05 2.185e-05 0.01% Castro::swap_state_time_levels() 5 2.084e-05 2.084e-05 2.084e-05 0.01% Castro::finalize_do_advance() 5 1.921e-05 1.921e-05 1.921e-05 0.01% Castro::initMFs() 1 1.899e-05 1.899e-05 1.899e-05 0.01% DistributionMapping::Distribute() 31 9.013e-06 9.013e-06 9.013e-06 0.00% Amr::initSubcycle() 1 8.203e-06 8.203e-06 8.203e-06 0.00% Castro::retry_advance_ctu() 5 6.812e-06 6.812e-06 6.812e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.255e-06 4.255e-06 4.255e-06 0.00% AmrLevel::AmrLevel() 1 2.921e-06 2.921e-06 2.921e-06 0.00% Castro::create_source_corrector() 5 1.969e-06 1.969e-06 1.969e-06 0.00% Gravity::set_mass_offset() 6 1.845e-06 1.845e-06 1.845e-06 0.00% Castro::FluxRegCrseInit 5 1.407e-06 1.407e-06 1.407e-06 0.00% Castro::FluxRegFineAdd() 5 1.082e-06 1.082e-06 1.082e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.079e-06 1.079e-06 1.079e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-1-g30f2290cf7c3) finalized