Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-10-gf84c7a8f77d6) initialized Starting run at 08:36:13 UTC on 2022-10-14. Successfully read inputs file ... Castro git describe: 22.09-1-g65b273ad0 AMReX git describe: 22.10-10-gf84c7a8f7 Microphysics git describe: 22.10-4-g1dbcf8c2 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.053467094 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030592372 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.050051172 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052401889 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.052225234 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.051499843 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.078401172 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.048922452 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.081394627 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.0587002 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.049493545 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.05179732 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.053361283 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.049041292 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.046414548 seconds Ending run at 08:36:14 UTC on 2022-10-14. Run time = 0.861704222 Run time without initialization = 0.724356186 Average number of zones advanced per microsecond: 3.619 Average number of zones advanced per microsecond per rank: 3.619 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8617 ... 0.8617 ... 0.8617 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2204 0.2204 0.2204 25.58% Castro::construct_ctu_hydro_source() 10 0.1766 0.1766 0.1766 20.49% MLCellLinOp::applyBC() 4433 0.08179 0.08179 0.08179 9.49% MLPoisson::Fsmooth() 3280 0.06524 0.06524 0.06524 7.57% MLCGSolver::bicgstab 82 0.02441 0.02441 0.02441 2.83% StateData::FillBoundary(geom) 328 0.02382 0.02382 0.02382 2.76% MultiFab::Dot() 1114 0.02294 0.02294 0.02294 2.66% StateDataPhysBCFunct::() 41 0.02147 0.02147 0.02147 2.49% MultiFab::LinComb() 1586 0.01475 0.01475 0.01475 1.71% FabArray::setVal() 1144 0.01454 0.01454 0.01454 1.69% Castro::computeTemp() 63 0.01452 0.01452 0.01452 1.69% Castro::normalize_species() 62 0.0145 0.0145 0.0145 1.68% FillBoundary_nowait() 4023 0.01424 0.01424 0.01424 1.65% FabArray::ParallelCopy_nowait() 861 0.01346 0.01346 0.01346 1.56% MLPoisson::Fapply() 1142 0.01206 0.01206 0.01206 1.40% MLCellLinOp::defineAuxData() 11 0.01193 0.01193 0.01193 1.38% Gravity::fill_multipole_BCs() 11 0.01032 0.01032 0.01032 1.20% Castro::enforce_min_density() 62 0.008731 0.008731 0.008731 1.01% MLMG::addInterpCorrection() 410 0.00782 0.00782 0.00782 0.91% amrex::average_down 410 0.006956 0.006956 0.006956 0.81% Castro::estTimeStep() 21 0.006913 0.006913 0.006913 0.80% MultiFab::Xpay() 585 0.006744 0.006744 0.006744 0.78% Amr::checkPoint() 3 0.005159 0.005159 0.005159 0.60% Castro::do_advance_ctu() 10 0.005095 0.005095 0.005095 0.59% Castro::reset_internal_energy(MultiFab) 63 0.004061 0.004061 0.004061 0.47% BndryData::define() 11 0.003923 0.003923 0.003923 0.46% Castro::construct_new_gravity_source() 10 0.003432 0.003432 0.003432 0.40% Amr::writePlotFile() 2 0.002914 0.002914 0.002914 0.34% Castro::construct_old_gravity_source() 10 0.002872 0.002872 0.002872 0.33% MLMG::ResNormInf() 93 0.002084 0.002084 0.002084 0.24% Gravity::get_new_grav_vector() 11 0.001937 0.001937 0.001937 0.22% MultiFab::Saxpy() 20 0.001823 0.001823 0.001823 0.21% Castro::expand_state() 10 0.001738 0.001738 0.001738 0.20% Gravity::get_old_grav_vector() 10 0.001733 0.001733 0.001733 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001676 0.001676 0.001676 0.19% MultiFab::Add() 82 0.001663 0.001663 0.001663 0.19% MLCellLinOp::setLevelBC() 11 0.00155 0.00155 0.00155 0.18% Castro::reset_internal_energy(Fab) 504 0.001531 0.001531 0.001531 0.18% Gravity::actual_solve_with_mlmg() 11 0.0014 0.0014 0.0014 0.16% FabArray::mult() 43 0.001332 0.001332 0.001332 0.15% FabArray::setDomainBndry() 41 0.001321 0.001321 0.001321 0.15% MLMG::prepareForSolve() 11 0.001227 0.001227 0.001227 0.14% Castro::initData() 1 0.00122 0.00122 0.00122 0.14% MultiFab::contains_nan() 20 0.001188 0.001188 0.001188 0.14% MLCellLinOp::prepareForSolve() 11 0.001176 0.001176 0.001176 0.14% Castro::enforce_speed_limit() 62 0.001174 0.001174 0.001174 0.14% MLCellLinOp::smooth() 1640 0.001049 0.001049 0.001049 0.12% Castro::subcycle_advance_ctu() 10 0.0009673 0.0009673 0.0009673 0.11% MLCellLinOp::compGrad() 11 0.0009172 0.0009172 0.0009172 0.11% FabArray::FillBoundary() 4023 0.0008303 0.0008303 0.0008303 0.10% FabArrayBase::getCPC() 1323 0.0007572 0.0007572 0.0007572 0.09% FabArrayBase::CPC::define() 454 0.0006899 0.0006899 0.0006899 0.08% FabArrayBase::getFB() 4023 0.000587 0.000587 0.000587 0.07% Amr::InitAmr() 1 0.0004758 0.0004758 0.0004758 0.06% MLCellLinOp::apply() 1142 0.0004583 0.0004583 0.0004583 0.05% Gravity::solve_for_phi() 10 0.0004549 0.0004549 0.0004549 0.05% Gravity::update_max_rhs() 11 0.0004271 0.0004271 0.0004271 0.05% CGSolver::sxay() 1586 0.0003495 0.0003495 0.0003495 0.04% Amr::coarseTimeStep() 10 0.0003375 0.0003375 0.0003375 0.04% MultiFab::Copy() 11 0.000337 0.000337 0.000337 0.04% main() 1 0.0002998 0.0002998 0.0002998 0.03% MLCGSolver::ParallelAllReduce 1514 0.0002962 0.0002962 0.0002962 0.03% FillPatchIterator::Initialize 41 0.0002836 0.0002836 0.0002836 0.03% MLCellLinOp::defineBC() 11 0.0002751 0.0002751 0.0002751 0.03% MultiFab::max() 11 0.0002614 0.0002614 0.0002614 0.03% FabArray::ParallelCopy() 861 0.0002577 0.0002577 0.0002577 0.03% MLMG::mgVcycle() 82 0.0002298 0.0002298 0.0002298 0.03% MLCellLinOp::correctionResidual() 492 0.0002221 0.0002221 0.0002221 0.03% MLMG::MLRhsNormInf() 11 0.0002187 0.0002187 0.0002187 0.03% Castro::construct_new_gravity() 10 0.0002145 0.0002145 0.0002145 0.02% Amr::timeStep() 10 0.000173 0.000173 0.000173 0.02% MLLinOp::defineGrids() 11 0.0001647 0.0001647 0.0001647 0.02% MLMG:computeResOfCorrection() 410 0.0001563 0.0001563 0.0001563 0.02% Castro::create_source_corrector() 10 0.0001322 0.0001322 0.0001322 0.02% StateData::checkPoint() 12 0.0001264 0.0001264 0.0001264 0.01% MLMG::mgVcycle_down::0 82 0.0001108 0.0001108 0.0001108 0.01% MLMG::mgVcycle_down::1 82 9.919e-05 9.919e-05 9.919e-05 0.01% FabArrayBase::FB::FB() 56 9.105e-05 9.105e-05 9.105e-05 0.01% MLMG::mgVcycle_down::2 82 8.926e-05 8.926e-05 8.926e-05 0.01% Castro::Castro() 1 8.835e-05 8.835e-05 8.835e-05 0.01% MLMG::mgVcycle_down::3 82 8.359e-05 8.359e-05 8.359e-05 0.01% MLMG::mgVcycle_down::4 82 8.3e-05 8.3e-05 8.3e-05 0.01% Castro::initialize_advance() 10 8.223e-05 8.223e-05 8.223e-05 0.01% Castro::clean_state() 62 8.162e-05 8.162e-05 8.162e-05 0.01% MLMG::actualBottomSolve() 82 7.965e-05 7.965e-05 7.965e-05 0.01% AmrLevel::checkPoint() 3 7.16e-05 7.16e-05 7.16e-05 0.01% MLMG::mgVcycle_up::4 82 6.73e-05 6.73e-05 6.73e-05 0.01% MLMG::solve() 11 6.64e-05 6.64e-05 6.64e-05 0.01% MLMG::oneIter() 82 6.165e-05 6.165e-05 6.165e-05 0.01% Castro::initialize_do_advance() 10 6.086e-05 6.086e-05 6.086e-05 0.01% Castro::finalize_advance() 10 6.081e-05 6.081e-05 6.081e-05 0.01% MLMG::mgVcycle_up::3 82 5.468e-05 5.468e-05 5.468e-05 0.01% MLMG::mgVcycle_up::1 82 5.459e-05 5.459e-05 5.459e-05 0.01% MLMG::mgVcycle_up::0 82 5.441e-05 5.441e-05 5.441e-05 0.01% MLMG::mgVcycle_up::2 82 5.438e-05 5.438e-05 5.438e-05 0.01% MLCellLinOp::solutionResidual() 93 4.955e-05 4.955e-05 4.955e-05 0.01% StateData::define() 4 4.099e-05 4.099e-05 4.099e-05 0.00% MLMG::computeResidual() 82 4.07e-05 4.07e-05 4.07e-05 0.00% Castro::swap_state_time_levels() 10 3.973e-05 3.973e-05 3.973e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.918e-05 3.918e-05 3.918e-05 0.00% Castro::advance() 10 3.587e-05 3.587e-05 3.587e-05 0.00% Castro::enforce_consistent_e() 1 3.536e-05 3.536e-05 3.536e-05 0.00% MLMG::mgVcycle_bottom 82 3.472e-05 3.472e-05 3.472e-05 0.00% Castro::construct_new_source() 50 3.438e-05 3.438e-05 3.438e-05 0.00% Castro::finalize_do_advance() 10 3.411e-05 3.411e-05 3.411e-05 0.00% Gravity::actual_multilevel_solve() 1 3.193e-05 3.193e-05 3.193e-05 0.00% Amr::writeSmallPlotFile() 1 2.898e-05 2.898e-05 2.898e-05 0.00% FillPatchSingleLevel 41 2.761e-05 2.761e-05 2.761e-05 0.00% Castro::initMFs() 1 2.668e-05 2.668e-05 2.668e-05 0.00% makeSFC 55 2.649e-05 2.649e-05 2.649e-05 0.00% MLLinOp::define() 11 2.4e-05 2.4e-05 2.4e-05 0.00% Castro::buildMetrics() 1 2.389e-05 2.389e-05 2.389e-05 0.00% MLPoisson::define() 11 2.349e-05 2.349e-05 2.349e-05 0.00% Amr::defBaseLevel() 1 2.08e-05 2.08e-05 2.08e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.899e-05 1.899e-05 1.899e-05 0.00% Amr::FinalizeInit() 1 1.891e-05 1.891e-05 1.891e-05 0.00% Castro::do_new_sources() 10 1.738e-05 1.738e-05 1.738e-05 0.00% Castro::construct_old_source() 50 1.58e-05 1.58e-05 1.58e-05 0.00% Castro::do_old_sources() 10 1.56e-05 1.56e-05 1.56e-05 0.00% DistributionMapping::Distribute() 56 1.44e-05 1.44e-05 1.44e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.405e-05 1.405e-05 1.405e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.226e-05 1.226e-05 1.226e-05 0.00% Castro::apply_source_to_state() 20 1.114e-05 1.114e-05 1.114e-05 0.00% Castro::check_for_nan() 20 1.06e-05 1.06e-05 1.06e-05 0.00% Castro::construct_old_gravity() 10 1.035e-05 1.035e-05 1.035e-05 0.00% Castro::post_timestep() 10 1.022e-05 1.022e-05 1.022e-05 0.00% MLMG::computeMLResidual() 11 9.835e-06 9.835e-06 9.835e-06 0.00% Amr::initSubcycle() 1 8.535e-06 8.535e-06 8.535e-06 0.00% Gravity::swapTimeLevels() 10 8.289e-06 8.289e-06 8.289e-06 0.00% MLPoisson::prepareForSolve() 11 8.189e-06 8.189e-06 8.189e-06 0.00% MLMG::getGradSolution() 11 6.937e-06 6.937e-06 6.937e-06 0.00% Castro::computeNewDt() 9 6.207e-06 6.207e-06 6.207e-06 0.00% Castro::retry_advance_ctu() 10 5.178e-06 5.178e-06 5.178e-06 0.00% Amr::InitializeInit() 1 4.719e-06 4.719e-06 4.719e-06 0.00% AmrLevel::checkPointPost() 3 4.305e-06 4.305e-06 4.305e-06 0.00% Castro::post_init() 1 3.875e-06 3.875e-06 3.875e-06 0.00% Gravity::set_mass_offset() 11 3.787e-06 3.787e-06 3.787e-06 0.00% Castro::FluxRegCrseInit 10 3.631e-06 3.631e-06 3.631e-06 0.00% MLMG::MLResNormInf() 11 3.497e-06 3.497e-06 3.497e-06 0.00% Amr::init() 1 2.723e-06 2.723e-06 2.723e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.6e-06 2.6e-06 2.6e-06 0.00% Castro::FluxRegFineAdd() 10 2.556e-06 2.556e-06 2.556e-06 0.00% Castro::computeInitialDt() 2 2.529e-06 2.529e-06 2.529e-06 0.00% AmrLevel::checkPointPre() 3 2.204e-06 2.204e-06 2.204e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.867e-06 1.867e-06 1.867e-06 0.00% Castro::post_regrid() 1 1.324e-06 1.324e-06 1.324e-06 0.00% Amr::initialInit() 1 1.029e-06 1.029e-06 1.029e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8617 0.8617 0.8617 100.00% Amr::coarseTimeStep() 10 0.6777 0.6777 0.6777 78.64% Amr::timeStep() 10 0.5747 0.5747 0.5747 66.69% Castro::advance() 10 0.5675 0.5675 0.5675 65.86% Castro::subcycle_advance_ctu() 10 0.5563 0.5563 0.5563 64.56% Castro::do_advance_ctu() 10 0.5553 0.5553 0.5553 64.44% Gravity::solve_phi_with_mlmg() 11 0.3225 0.3225 0.3225 37.42% Gravity::actual_solve_with_mlmg() 11 0.3119 0.3119 0.3119 36.19% Castro::construct_new_gravity() 10 0.2933 0.2933 0.2933 34.04% MLMG::solve() 11 0.2888 0.2888 0.2888 33.52% Gravity::solve_for_phi() 10 0.2778 0.2778 0.2778 32.24% MLMG::oneIter() 82 0.2737 0.2737 0.2737 31.76% MLMG::mgVcycle() 82 0.272 0.272 0.272 31.56% VisMF::Write(FabArray) 11 0.2204 0.2204 0.2204 25.58% Castro::construct_ctu_hydro_source() 10 0.1766 0.1766 0.1766 20.49% Amr::checkPoint() 3 0.1515 0.1515 0.1515 17.59% AmrLevel::checkPoint() 3 0.1464 0.1464 0.1464 16.99% StateData::checkPoint() 12 0.1463 0.1463 0.1463 16.98% MLCellLinOp::smooth() 1640 0.1389 0.1389 0.1389 16.11% Amr::init() 1 0.1367 0.1367 0.1367 15.87% MLCellLinOp::applyBC() 4433 0.09754 0.09754 0.09754 11.32% MLMG::mgVcycle_bottom 82 0.0839 0.0839 0.0839 9.74% MLMG::actualBottomSolve() 82 0.08387 0.08387 0.08387 9.73% MLCGSolver::bicgstab 82 0.08304 0.08304 0.08304 9.64% Amr::writePlotFile() 2 0.07714 0.07714 0.07714 8.95% MLPoisson::Fsmooth() 3280 0.06524 0.06524 0.06524 7.57% Amr::initialInit() 1 0.05256 0.05256 0.05256 6.10% FillPatchIterator::Initialize 41 0.05093 0.05093 0.05093 5.91% FillPatchSingleLevel 41 0.04933 0.04933 0.04933 5.72% Amr::FinalizeInit() 1 0.04862 0.04862 0.04862 5.64% Castro::post_init() 1 0.04724 0.04724 0.04724 5.48% StateDataPhysBCFunct::() 41 0.04529 0.04529 0.04529 5.26% Gravity::multilevel_solve_for_new_phi() 1 0.04519 0.04519 0.04519 5.24% Gravity::actual_multilevel_solve() 1 0.04517 0.04517 0.04517 5.24% Castro::clean_state() 62 0.04386 0.04386 0.04386 5.09% MLCellLinOp::apply() 1142 0.03698 0.03698 0.03698 4.29% MLMG::mgVcycle_down::0 82 0.03586 0.03586 0.03586 4.16% MLMG::mgVcycle_up::0 82 0.03076 0.03076 0.03076 3.57% Castro::initialize_do_advance() 10 0.02839 0.02839 0.02839 3.29% StateData::FillBoundary(geom) 328 0.02382 0.02382 0.02382 2.76% MultiFab::Dot() 1114 0.02294 0.02294 0.02294 2.66% MLCellLinOp::correctionResidual() 492 0.02168 0.02168 0.02168 2.52% Castro::computeTemp() 63 0.02012 0.02012 0.02012 2.33% Castro::expand_state() 10 0.01975 0.01975 0.01975 2.29% MLMG:computeResOfCorrection() 410 0.01871 0.01871 0.01871 2.17% MLPoisson::define() 11 0.0187 0.0187 0.0187 2.17% MLMG::mgVcycle_down::1 82 0.01836 0.01836 0.01836 2.13% MLMG::mgVcycle_down::2 82 0.01765 0.01765 0.01765 2.05% Gravity::get_new_grav_vector() 11 0.0173 0.0173 0.0173 2.01% MLMG::mgVcycle_down::3 82 0.01676 0.01676 0.01676 1.94% MLMG::mgVcycle_down::4 82 0.01597 0.01597 0.01597 1.85% FabArray::FillBoundary() 4023 0.01575 0.01575 0.01575 1.83% CGSolver::sxay() 1586 0.0151 0.0151 0.0151 1.75% FillBoundary_nowait() 4023 0.01491 0.01491 0.01491 1.73% Castro::construct_old_gravity() 10 0.01486 0.01486 0.01486 1.72% Gravity::get_old_grav_vector() 10 0.01485 0.01485 0.01485 1.72% MultiFab::LinComb() 1586 0.01475 0.01475 0.01475 1.71% FabArray::setVal() 1144 0.01454 0.01454 0.01454 1.69% FabArray::ParallelCopy() 861 0.01453 0.01453 0.01453 1.69% Castro::normalize_species() 62 0.0145 0.0145 0.0145 1.68% FabArray::ParallelCopy_nowait() 861 0.01428 0.01428 0.01428 1.66% MLCGSolver::ParallelAllReduce 1514 0.01367 0.01367 0.01367 1.59% MLMG::mgVcycle_up::2 82 0.01357 0.01357 0.01357 1.57% MLMG::mgVcycle_up::1 82 0.01336 0.01336 0.01336 1.55% MLCellLinOp::defineAuxData() 11 0.01329 0.01329 0.01329 1.54% MLMG::addInterpCorrection() 410 0.01308 0.01308 0.01308 1.52% MLMG::mgVcycle_up::3 82 0.01287 0.01287 0.01287 1.49% MLMG::mgVcycle_up::4 82 0.0127 0.0127 0.0127 1.47% amrex::average_down 410 0.01222 0.01222 0.01222 1.42% MLPoisson::Fapply() 1142 0.01206 0.01206 0.01206 1.40% Castro::do_new_sources() 10 0.01194 0.01194 0.01194 1.39% Castro::initialize_advance() 10 0.01111 0.01111 0.01111 1.29% Castro::do_old_sources() 10 0.01077 0.01077 0.01077 1.25% Gravity::fill_multipole_BCs() 11 0.01032 0.01032 0.01032 1.20% Castro::enforce_min_density() 62 0.008731 0.008731 0.008731 1.01% MLCellLinOp::solutionResidual() 93 0.007228 0.007228 0.007228 0.84% Castro::post_timestep() 10 0.00696 0.00696 0.00696 0.81% Castro::estTimeStep() 21 0.006913 0.006913 0.006913 0.80% MultiFab::Xpay() 585 0.006744 0.006744 0.006744 0.78% MLMG::computeResidual() 82 0.006251 0.006251 0.006251 0.73% Castro::reset_internal_energy(MultiFab) 63 0.005592 0.005592 0.005592 0.65% MLMG::prepareForSolve() 11 0.005445 0.005445 0.005445 0.63% MLCellLinOp::defineBC() 11 0.005148 0.005148 0.005148 0.60% BndryData::define() 11 0.004873 0.004873 0.004873 0.57% Castro::computeNewDt() 9 0.004153 0.004153 0.004153 0.48% Amr::InitializeInit() 1 0.003936 0.003936 0.003936 0.46% Amr::defBaseLevel() 1 0.003931 0.003931 0.003931 0.46% Castro::construct_new_source() 50 0.003466 0.003466 0.003466 0.40% Castro::construct_new_gravity_source() 10 0.003432 0.003432 0.003432 0.40% Castro::initData() 1 0.00341 0.00341 0.00341 0.40% Castro::construct_old_source() 50 0.002888 0.002888 0.002888 0.34% Castro::construct_old_gravity_source() 10 0.002872 0.002872 0.002872 0.33% MLMG::ResNormInf() 93 0.002084 0.002084 0.002084 0.24% Castro::apply_source_to_state() 20 0.001835 0.001835 0.001835 0.21% MultiFab::Saxpy() 20 0.001823 0.001823 0.001823 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001676 0.001676 0.001676 0.19% MultiFab::Add() 82 0.001663 0.001663 0.001663 0.19% MLCellLinOp::setLevelBC() 11 0.00155 0.00155 0.00155 0.18% Castro::reset_internal_energy(Fab) 504 0.001531 0.001531 0.001531 0.18% FabArrayBase::getCPC() 1323 0.001447 0.001447 0.001447 0.17% MLMG::getGradSolution() 11 0.001434 0.001434 0.001434 0.17% MLCellLinOp::compGrad() 11 0.001427 0.001427 0.001427 0.17% FabArray::mult() 43 0.001332 0.001332 0.001332 0.15% FabArray::setDomainBndry() 41 0.001321 0.001321 0.001321 0.15% Castro::check_for_nan() 20 0.001198 0.001198 0.001198 0.14% MultiFab::contains_nan() 20 0.001188 0.001188 0.001188 0.14% MLPoisson::prepareForSolve() 11 0.001185 0.001185 0.001185 0.14% MLCellLinOp::prepareForSolve() 11 0.001176 0.001176 0.001176 0.14% Castro::post_regrid() 1 0.001176 0.001176 0.001176 0.14% Castro::enforce_speed_limit() 62 0.001174 0.001174 0.001174 0.14% MLMG::computeMLResidual() 11 0.001027 0.001027 0.001027 0.12% Gravity::update_max_rhs() 11 0.0008423 0.0008423 0.0008423 0.10% FabArrayBase::CPC::define() 454 0.0006899 0.0006899 0.0006899 0.08% Castro::computeInitialDt() 2 0.0006884 0.0006884 0.0006884 0.08% FabArrayBase::getFB() 4023 0.000678 0.000678 0.000678 0.08% Amr::InitAmr() 1 0.0004844 0.0004844 0.0004844 0.06% Gravity::swapTimeLevels() 10 0.0004475 0.0004475 0.0004475 0.05% Castro::Castro() 1 0.000444 0.000444 0.000444 0.05% MultiFab::Copy() 11 0.000337 0.000337 0.000337 0.04% MLMG::MLResNormInf() 11 0.0002803 0.0002803 0.0002803 0.03% MultiFab::max() 11 0.0002614 0.0002614 0.0002614 0.03% MLLinOp::define() 11 0.0002443 0.0002443 0.0002443 0.03% MLLinOp::defineGrids() 11 0.0002203 0.0002203 0.0002203 0.03% MLMG::MLRhsNormInf() 11 0.0002187 0.0002187 0.0002187 0.03% Castro::buildMetrics() 1 0.000163 0.000163 0.000163 0.02% Castro::create_source_corrector() 10 0.0001322 0.0001322 0.0001322 0.02% FabArrayBase::FB::FB() 56 9.105e-05 9.105e-05 9.105e-05 0.01% Castro::finalize_advance() 10 6.7e-05 6.7e-05 6.7e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.367e-05 5.367e-05 5.367e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.325e-05 5.325e-05 5.325e-05 0.01% StateData::define() 4 4.099e-05 4.099e-05 4.099e-05 0.00% Castro::swap_state_time_levels() 10 3.973e-05 3.973e-05 3.973e-05 0.00% makeSFC 55 3.963e-05 3.963e-05 3.963e-05 0.00% Castro::enforce_consistent_e() 1 3.536e-05 3.536e-05 3.536e-05 0.00% Castro::finalize_do_advance() 10 3.411e-05 3.411e-05 3.411e-05 0.00% Amr::writeSmallPlotFile() 1 2.898e-05 2.898e-05 2.898e-05 0.00% Castro::initMFs() 1 2.668e-05 2.668e-05 2.668e-05 0.00% DistributionMapping::Distribute() 56 1.44e-05 1.44e-05 1.44e-05 0.00% Amr::initSubcycle() 1 8.535e-06 8.535e-06 8.535e-06 0.00% Castro::retry_advance_ctu() 10 5.178e-06 5.178e-06 5.178e-06 0.00% AmrLevel::checkPointPost() 3 4.305e-06 4.305e-06 4.305e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.87e-06 3.87e-06 3.87e-06 0.00% Gravity::set_mass_offset() 11 3.787e-06 3.787e-06 3.787e-06 0.00% Castro::FluxRegCrseInit 10 3.631e-06 3.631e-06 3.631e-06 0.00% Castro::FluxRegFineAdd() 10 2.556e-06 2.556e-06 2.556e-06 0.00% AmrLevel::checkPointPre() 3 2.204e-06 2.204e-06 2.204e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.867e-06 1.867e-06 1.867e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-10-gf84c7a8f77d6) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-10-gf84c7a8f77d6) initialized Starting run at 08:36:14 UTC on 2022-10-14. Successfully read inputs file ... Castro git describe: 22.09-1-g65b273ad0 AMReX git describe: 22.10-10-gf84c7a8f7 Microphysics git describe: 22.10-4-g1dbcf8c2 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.470862531 Restart time = 0.048964812 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051608915 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050029565 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.050573863 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.050168131 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065116676 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032165089 seconds Ending run at 08:36:15 UTC on 2022-10-14. Run time = 0.349583994 Run time without initialization = 0.30006339 Average number of zones advanced per microsecond: 4.368 Average number of zones advanced per microsecond per rank: 4.368 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3496 ... 0.3496 ... 0.3496 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0713 0.0713 0.0713 20.39% VisMF::Read() 3 0.04101 0.04101 0.04101 11.73% MLCellLinOp::applyBC() 1946 0.03516 0.03516 0.03516 10.06% VisMF::Write(FabArray) 1 0.03059 0.03059 0.03059 8.75% MLPoisson::Fsmooth() 1440 0.02739 0.02739 0.02739 7.83% StateData::FillBoundary(geom) 160 0.01211 0.01211 0.01211 3.46% MLCGSolver::bicgstab 36 0.01027 0.01027 0.01027 2.94% MultiFab::Dot() 484 0.009576 0.009576 0.009576 2.74% FabArray::setVal() 537 0.006744 0.006744 0.006744 1.93% Castro::computeTemp() 30 0.006675 0.006675 0.006675 1.91% MLCellLinOp::defineAuxData() 6 0.00634 0.00634 0.00634 1.81% FillBoundary_nowait() 1766 0.006279 0.006279 0.006279 1.80% MultiFab::LinComb() 690 0.006172 0.006172 0.006172 1.77% FabArray::ParallelCopy_nowait() 380 0.006047 0.006047 0.006047 1.73% Castro::normalize_species() 30 0.005893 0.005893 0.005893 1.69% Gravity::fill_multipole_BCs() 6 0.005821 0.005821 0.005821 1.67% StateDataPhysBCFunct::() 20 0.005383 0.005383 0.005383 1.54% Castro::enforce_min_density() 30 0.005329 0.005329 0.005329 1.52% MLPoisson::Fapply() 500 0.00513 0.00513 0.00513 1.47% Amr::restart() 1 0.003605 0.003605 0.003605 1.03% MLMG::addInterpCorrection() 180 0.003353 0.003353 0.003353 0.96% Castro::estTimeStep() 10 0.003318 0.003318 0.003318 0.95% amrex::average_down 180 0.002983 0.002983 0.002983 0.85% MultiFab::Xpay() 258 0.002887 0.002887 0.002887 0.83% Castro::reset_internal_energy(MultiFab) 30 0.002181 0.002181 0.002181 0.62% BndryData::define() 6 0.00216 0.00216 0.00216 0.62% Castro::do_advance_ctu() 5 0.001911 0.001911 0.001911 0.55% Castro::construct_new_gravity_source() 5 0.001666 0.001666 0.001666 0.48% Amr::writePlotFile() 1 0.001655 0.001655 0.001655 0.47% Castro::construct_old_gravity_source() 5 0.001485 0.001485 0.001485 0.42% MLMG::ResNormInf() 42 0.0009193 0.0009193 0.0009193 0.26% MultiFab::Saxpy() 10 0.0009166 0.0009166 0.0009166 0.26% Gravity::get_old_grav_vector() 5 0.0009148 0.0009148 0.0009148 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009008 0.0009008 0.0009008 0.26% Castro::expand_state() 5 0.0008802 0.0008802 0.0008802 0.25% Gravity::get_new_grav_vector() 5 0.0008748 0.0008748 0.0008748 0.25% MLCellLinOp::setLevelBC() 6 0.0008206 0.0008206 0.0008206 0.23% Castro::reset_internal_energy(Fab) 240 0.000772 0.000772 0.000772 0.22% Gravity::actual_solve_with_mlmg() 6 0.000741 0.000741 0.000741 0.21% MultiFab::Add() 36 0.0007116 0.0007116 0.0007116 0.20% MLMG::prepareForSolve() 6 0.000663 0.000663 0.000663 0.19% FabArray::mult() 22 0.0006553 0.0006553 0.0006553 0.19% FabArray::setDomainBndry() 20 0.0006447 0.0006447 0.0006447 0.18% MLCellLinOp::prepareForSolve() 6 0.0006373 0.0006373 0.0006373 0.18% MultiFab::contains_nan() 10 0.0005875 0.0005875 0.0005875 0.17% MLCellLinOp::compGrad() 6 0.0004943 0.0004943 0.0004943 0.14% MLCellLinOp::smooth() 720 0.0004806 0.0004806 0.0004806 0.14% Castro::enforce_speed_limit() 30 0.0004444 0.0004444 0.0004444 0.13% FabArrayBase::CPC::define() 244 0.0004048 0.0004048 0.0004048 0.12% Amr::InitAmr() 1 0.0003968 0.0003968 0.0003968 0.11% FabArray::FillBoundary() 1766 0.0003844 0.0003844 0.0003844 0.11% FabArrayBase::getCPC() 632 0.0003706 0.0003706 0.0003706 0.11% FabArrayBase::getFB() 1766 0.0002641 0.0002641 0.0002641 0.08% main() 1 0.0002529 0.0002529 0.0002529 0.07% Gravity::solve_for_phi() 5 0.000235 0.000235 0.000235 0.07% Gravity::update_max_rhs() 6 0.0002302 0.0002302 0.0002302 0.07% MLCellLinOp::apply() 500 0.0002074 0.0002074 0.0002074 0.06% MultiFab::Copy() 6 0.000179 0.000179 0.000179 0.05% CGSolver::sxay() 690 0.0001696 0.0001696 0.0001696 0.05% Amr::coarseTimeStep() 5 0.0001641 0.0001641 0.0001641 0.05% MLCellLinOp::defineBC() 6 0.000148 0.000148 0.000148 0.04% MultiFab::max() 6 0.0001389 0.0001389 0.0001389 0.04% FillPatchIterator::Initialize 20 0.0001384 0.0001384 0.0001384 0.04% MLCGSolver::ParallelAllReduce 659 0.000126 0.000126 0.000126 0.04% Castro::subcycle_advance_ctu() 5 0.0001208 0.0001208 0.0001208 0.03% FabArray::ParallelCopy() 380 0.0001201 0.0001201 0.0001201 0.03% Castro::construct_new_gravity() 5 0.0001156 0.0001156 0.0001156 0.03% MLMG::MLRhsNormInf() 6 0.0001137 0.0001137 0.0001137 0.03% MLCellLinOp::correctionResidual() 216 0.0001025 0.0001025 0.0001025 0.03% MLMG::mgVcycle() 36 0.0001022 0.0001022 0.0001022 0.03% AmrLevel::restart() 1 8.272e-05 8.272e-05 8.272e-05 0.02% Amr::timeStep() 5 8.225e-05 8.225e-05 8.225e-05 0.02% MLLinOp::defineGrids() 6 7.642e-05 7.642e-05 7.642e-05 0.02% StateData::restartDoit() 4 7.369e-05 7.369e-05 7.369e-05 0.02% MLMG:computeResOfCorrection() 180 7.367e-05 7.367e-05 7.367e-05 0.02% FabArrayBase::FB::FB() 26 5.897e-05 5.897e-05 5.897e-05 0.02% Castro::create_source_corrector() 5 5.383e-05 5.383e-05 5.383e-05 0.02% Castro::advance() 5 4.699e-05 4.699e-05 4.699e-05 0.01% MLMG::mgVcycle_down::0 36 4.638e-05 4.638e-05 4.638e-05 0.01% Castro::clean_state() 30 4.632e-05 4.632e-05 4.632e-05 0.01% MLMG::mgVcycle_down::1 36 4.321e-05 4.321e-05 4.321e-05 0.01% Castro::construct_old_source() 25 4.318e-05 4.318e-05 4.318e-05 0.01% MLMG::mgVcycle_down::2 36 3.979e-05 3.979e-05 3.979e-05 0.01% MLMG::mgVcycle_down::4 36 3.929e-05 3.929e-05 3.929e-05 0.01% Castro::initialize_advance() 5 3.929e-05 3.929e-05 3.929e-05 0.01% Castro::finalize_advance() 5 3.906e-05 3.906e-05 3.906e-05 0.01% MLMG::mgVcycle_down::3 36 3.845e-05 3.845e-05 3.845e-05 0.01% MLMG::actualBottomSolve() 36 3.614e-05 3.614e-05 3.614e-05 0.01% Castro::buildMetrics() 1 3.26e-05 3.26e-05 3.26e-05 0.01% MLMG::solve() 6 3.137e-05 3.137e-05 3.137e-05 0.01% Gravity::actual_multilevel_solve() 1 3.126e-05 3.126e-05 3.126e-05 0.01% Castro::post_restart() 1 3.11e-05 3.11e-05 3.11e-05 0.01% MLMG::mgVcycle_up::4 36 3.095e-05 3.095e-05 3.095e-05 0.01% Castro::initialize_do_advance() 5 3.091e-05 3.091e-05 3.091e-05 0.01% Castro::initMFs() 1 2.905e-05 2.905e-05 2.905e-05 0.01% MLMG::mgVcycle_up::3 36 2.784e-05 2.784e-05 2.784e-05 0.01% MLMG::oneIter() 36 2.713e-05 2.713e-05 2.713e-05 0.01% Castro::swap_state_time_levels() 5 2.697e-05 2.697e-05 2.697e-05 0.01% Amr::writeSmallPlotFile() 1 2.609e-05 2.609e-05 2.609e-05 0.01% MLMG::mgVcycle_up::1 36 2.584e-05 2.584e-05 2.584e-05 0.01% MLMG::mgVcycle_up::0 36 2.476e-05 2.476e-05 2.476e-05 0.01% MLMG::mgVcycle_up::2 36 2.347e-05 2.347e-05 2.347e-05 0.01% MLLinOp::define() 6 2.3e-05 2.3e-05 2.3e-05 0.01% MLCellLinOp::solutionResidual() 42 2.289e-05 2.289e-05 2.289e-05 0.01% MLMG::computeResidual() 36 1.89e-05 1.89e-05 1.89e-05 0.01% Castro::finalize_do_advance() 5 1.841e-05 1.841e-05 1.841e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.789e-05 1.789e-05 1.789e-05 0.01% MLPoisson::define() 6 1.645e-05 1.645e-05 1.645e-05 0.00% MLMG::mgVcycle_bottom 36 1.478e-05 1.478e-05 1.478e-05 0.00% makeSFC 30 1.435e-05 1.435e-05 1.435e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.382e-05 1.382e-05 1.382e-05 0.00% FillPatchSingleLevel 20 1.365e-05 1.365e-05 1.365e-05 0.00% Castro::construct_new_source() 25 1.017e-05 1.017e-05 1.017e-05 0.00% Amr::initSubcycle() 1 8.877e-06 8.877e-06 8.877e-06 0.00% Castro::do_new_sources() 5 8.529e-06 8.529e-06 8.529e-06 0.00% DistributionMapping::Distribute() 31 8.458e-06 8.458e-06 8.458e-06 0.00% Castro::do_old_sources() 5 8.235e-06 8.235e-06 8.235e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.232e-06 7.232e-06 7.232e-06 0.00% Castro::check_for_nan() 10 6.802e-06 6.802e-06 6.802e-06 0.00% Castro::construct_old_gravity() 5 6.644e-06 6.644e-06 6.644e-06 0.00% Castro::apply_source_to_state() 10 5.41e-06 5.41e-06 5.41e-06 0.00% Castro::post_timestep() 5 5.171e-06 5.171e-06 5.171e-06 0.00% MLPoisson::prepareForSolve() 6 4.963e-06 4.963e-06 4.963e-06 0.00% MLMG::computeMLResidual() 6 4.537e-06 4.537e-06 4.537e-06 0.00% Gravity::swapTimeLevels() 5 4.19e-06 4.19e-06 4.19e-06 0.00% Castro::computeNewDt() 5 3.42e-06 3.42e-06 3.42e-06 0.00% MLMG::getGradSolution() 6 3.213e-06 3.213e-06 3.213e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.039e-06 3.039e-06 3.039e-06 0.00% MLMG::MLResNormInf() 6 2.233e-06 2.233e-06 2.233e-06 0.00% Gravity::set_mass_offset() 6 2.09e-06 2.09e-06 2.09e-06 0.00% Castro::retry_advance_ctu() 5 2.019e-06 2.019e-06 2.019e-06 0.00% Castro::FluxRegCrseInit 5 1.604e-06 1.604e-06 1.604e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.172e-06 1.172e-06 1.172e-06 0.00% Castro::FluxRegFineAdd() 5 1.058e-06 1.058e-06 1.058e-06 0.00% Amr::init() 1 9.13e-07 9.13e-07 9.13e-07 0.00% AmrLevel::AmrLevel() 1 8.17e-07 8.17e-07 8.17e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3496 0.3496 0.3496 100.00% Amr::coarseTimeStep() 5 0.2677 0.2677 0.2677 76.56% Amr::timeStep() 5 0.2652 0.2652 0.2652 75.86% Castro::advance() 5 0.2621 0.2621 0.2621 74.97% Castro::subcycle_advance_ctu() 5 0.2567 0.2567 0.2567 73.43% Castro::do_advance_ctu() 5 0.2566 0.2566 0.2566 73.40% Castro::construct_new_gravity() 5 0.1462 0.1462 0.1462 41.83% Gravity::solve_phi_with_mlmg() 6 0.1419 0.1419 0.1419 40.60% Gravity::solve_for_phi() 5 0.1383 0.1383 0.1383 39.57% Gravity::actual_solve_with_mlmg() 6 0.136 0.136 0.136 38.90% MLMG::solve() 6 0.1236 0.1236 0.1236 35.34% MLMG::oneIter() 36 0.1163 0.1163 0.1163 33.28% MLMG::mgVcycle() 36 0.1156 0.1156 0.1156 33.07% Castro::construct_ctu_hydro_source() 5 0.07128 0.07128 0.07128 20.39% MLCellLinOp::smooth() 720 0.05913 0.05913 0.05913 16.91% Amr::init() 1 0.04901 0.04901 0.04901 14.02% Amr::restart() 1 0.04901 0.04901 0.04901 14.02% MLCellLinOp::applyBC() 1946 0.04214 0.04214 0.04214 12.05% AmrLevel::restart() 1 0.04123 0.04123 0.04123 11.79% StateData::restartDoit() 4 0.04114 0.04114 0.04114 11.77% VisMF::Read() 3 0.04101 0.04101 0.04101 11.73% MLMG::mgVcycle_bottom 36 0.03529 0.03529 0.03529 10.09% MLMG::actualBottomSolve() 36 0.03527 0.03527 0.03527 10.09% MLCGSolver::bicgstab 36 0.03492 0.03492 0.03492 9.99% Amr::writePlotFile() 1 0.03225 0.03225 0.03225 9.22% VisMF::Write(FabArray) 1 0.03059 0.03059 0.03059 8.75% MLPoisson::Fsmooth() 1440 0.02739 0.02739 0.02739 7.83% Castro::clean_state() 30 0.02134 0.02134 0.02134 6.10% FillPatchIterator::Initialize 20 0.02029 0.02029 0.02029 5.80% FillPatchSingleLevel 20 0.01951 0.01951 0.01951 5.58% StateDataPhysBCFunct::() 20 0.01749 0.01749 0.01749 5.00% MLCellLinOp::apply() 500 0.01595 0.01595 0.01595 4.56% MLMG::mgVcycle_down::0 36 0.01547 0.01547 0.01547 4.42% MLMG::mgVcycle_up::0 36 0.01319 0.01319 0.01319 3.77% StateData::FillBoundary(geom) 160 0.01211 0.01211 0.01211 3.46% MLPoisson::define() 6 0.01008 0.01008 0.01008 2.88% Castro::initialize_do_advance() 5 0.009795 0.009795 0.009795 2.80% Castro::computeTemp() 30 0.009628 0.009628 0.009628 2.75% MultiFab::Dot() 484 0.009576 0.009576 0.009576 2.74% MLCellLinOp::correctionResidual() 216 0.009321 0.009321 0.009321 2.67% MLMG:computeResOfCorrection() 180 0.008056 0.008056 0.008056 2.30% Gravity::get_new_grav_vector() 5 0.007776 0.007776 0.007776 2.22% MLMG::mgVcycle_down::1 36 0.007774 0.007774 0.007774 2.22% Castro::construct_old_gravity() 5 0.007598 0.007598 0.007598 2.17% Gravity::get_old_grav_vector() 5 0.007592 0.007592 0.007592 2.17% MLMG::mgVcycle_down::2 36 0.007485 0.007485 0.007485 2.14% Castro::do_new_sources() 5 0.007278 0.007278 0.007278 2.08% MLMG::mgVcycle_down::3 36 0.007115 0.007115 0.007115 2.04% MLCellLinOp::defineAuxData() 6 0.007091 0.007091 0.007091 2.03% FabArray::FillBoundary() 1766 0.006987 0.006987 0.006987 2.00% MLMG::mgVcycle_down::4 36 0.006847 0.006847 0.006847 1.96% FabArray::setVal() 537 0.006744 0.006744 0.006744 1.93% FillBoundary_nowait() 1766 0.006602 0.006602 0.006602 1.89% FabArray::ParallelCopy() 380 0.006559 0.006559 0.006559 1.88% FabArray::ParallelCopy_nowait() 380 0.006439 0.006439 0.006439 1.84% CGSolver::sxay() 690 0.006341 0.006341 0.006341 1.81% MultiFab::LinComb() 690 0.006172 0.006172 0.006172 1.77% Castro::normalize_species() 30 0.005893 0.005893 0.005893 1.69% Gravity::fill_multipole_BCs() 6 0.005821 0.005821 0.005821 1.67% MLCGSolver::ParallelAllReduce 659 0.005745 0.005745 0.005745 1.64% MLMG::mgVcycle_up::2 36 0.005736 0.005736 0.005736 1.64% MLMG::mgVcycle_up::1 36 0.005684 0.005684 0.005684 1.63% MLMG::addInterpCorrection() 180 0.00561 0.00561 0.00561 1.60% Castro::expand_state() 5 0.005491 0.005491 0.005491 1.57% MLMG::mgVcycle_up::3 36 0.005474 0.005474 0.005474 1.57% MLMG::mgVcycle_up::4 36 0.005442 0.005442 0.005442 1.56% Castro::enforce_min_density() 30 0.005329 0.005329 0.005329 1.52% Castro::initialize_advance() 5 0.005304 0.005304 0.005304 1.52% amrex::average_down 180 0.005285 0.005285 0.005285 1.51% MLPoisson::Fapply() 500 0.00513 0.00513 0.00513 1.47% Castro::do_old_sources() 5 0.005069 0.005069 0.005069 1.45% Castro::post_restart() 1 0.003993 0.003993 0.003993 1.14% Gravity::multilevel_solve_for_new_phi() 1 0.003867 0.003867 0.003867 1.11% Gravity::actual_multilevel_solve() 1 0.003849 0.003849 0.003849 1.10% Castro::estTimeStep() 10 0.003318 0.003318 0.003318 0.95% MLCellLinOp::solutionResidual() 42 0.003236 0.003236 0.003236 0.93% Castro::post_timestep() 5 0.003015 0.003015 0.003015 0.86% Castro::reset_internal_energy(MultiFab) 30 0.002953 0.002953 0.002953 0.84% MLMG::prepareForSolve() 6 0.002895 0.002895 0.002895 0.83% MultiFab::Xpay() 258 0.002887 0.002887 0.002887 0.83% MLCellLinOp::defineBC() 6 0.002841 0.002841 0.002841 0.81% BndryData::define() 6 0.002693 0.002693 0.002693 0.77% MLMG::computeResidual() 36 0.002692 0.002692 0.002692 0.77% Castro::computeNewDt() 5 0.002271 0.002271 0.002271 0.65% Castro::construct_new_source() 25 0.001676 0.001676 0.001676 0.48% Castro::construct_new_gravity_source() 5 0.001666 0.001666 0.001666 0.48% Castro::construct_old_source() 25 0.001528 0.001528 0.001528 0.44% Castro::construct_old_gravity_source() 5 0.001485 0.001485 0.001485 0.42% Castro::apply_source_to_state() 10 0.000922 0.000922 0.000922 0.26% MLMG::ResNormInf() 42 0.0009193 0.0009193 0.0009193 0.26% MultiFab::Saxpy() 10 0.0009166 0.0009166 0.0009166 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009008 0.0009008 0.0009008 0.26% MLCellLinOp::setLevelBC() 6 0.0008206 0.0008206 0.0008206 0.23% FabArrayBase::getCPC() 632 0.0007754 0.0007754 0.0007754 0.22% MLMG::getGradSolution() 6 0.0007746 0.0007746 0.0007746 0.22% Castro::reset_internal_energy(Fab) 240 0.000772 0.000772 0.000772 0.22% MLCellLinOp::compGrad() 6 0.0007714 0.0007714 0.0007714 0.22% MultiFab::Add() 36 0.0007116 0.0007116 0.0007116 0.20% FabArray::mult() 22 0.0006553 0.0006553 0.0006553 0.19% FabArray::setDomainBndry() 20 0.0006447 0.0006447 0.0006447 0.18% MLPoisson::prepareForSolve() 6 0.0006422 0.0006422 0.0006422 0.18% MLCellLinOp::prepareForSolve() 6 0.0006373 0.0006373 0.0006373 0.18% Castro::check_for_nan() 10 0.0005943 0.0005943 0.0005943 0.17% MultiFab::contains_nan() 10 0.0005875 0.0005875 0.0005875 0.17% MLMG::computeMLResidual() 6 0.0005679 0.0005679 0.0005679 0.16% Gravity::update_max_rhs() 6 0.0004499 0.0004499 0.0004499 0.13% Castro::enforce_speed_limit() 30 0.0004444 0.0004444 0.0004444 0.13% Amr::InitAmr() 1 0.0004057 0.0004057 0.0004057 0.12% FabArrayBase::CPC::define() 244 0.0004048 0.0004048 0.0004048 0.12% FabArrayBase::getFB() 1766 0.0003231 0.0003231 0.0003231 0.09% Gravity::swapTimeLevels() 5 0.0002303 0.0002303 0.0002303 0.07% MultiFab::Copy() 6 0.000179 0.000179 0.000179 0.05% Castro::buildMetrics() 1 0.0001533 0.0001533 0.0001533 0.04% MLMG::MLResNormInf() 6 0.0001495 0.0001495 0.0001495 0.04% MultiFab::max() 6 0.0001389 0.0001389 0.0001389 0.04% MLLinOp::define() 6 0.0001295 0.0001295 0.0001295 0.04% MLMG::MLRhsNormInf() 6 0.0001137 0.0001137 0.0001137 0.03% MLLinOp::defineGrids() 6 0.0001065 0.0001065 0.0001065 0.03% FabArrayBase::FB::FB() 26 5.897e-05 5.897e-05 5.897e-05 0.02% Castro::create_source_corrector() 5 5.383e-05 5.383e-05 5.383e-05 0.02% Castro::finalize_advance() 5 4.172e-05 4.172e-05 4.172e-05 0.01% Castro::initMFs() 1 2.905e-05 2.905e-05 2.905e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.889e-05 2.889e-05 2.889e-05 0.01% Castro::swap_state_time_levels() 5 2.697e-05 2.697e-05 2.697e-05 0.01% Amr::writeSmallPlotFile() 1 2.609e-05 2.609e-05 2.609e-05 0.01% makeSFC 30 2.166e-05 2.166e-05 2.166e-05 0.01% Castro::finalize_do_advance() 5 1.841e-05 1.841e-05 1.841e-05 0.01% Amr::initSubcycle() 1 8.877e-06 8.877e-06 8.877e-06 0.00% DistributionMapping::Distribute() 31 8.458e-06 8.458e-06 8.458e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.188e-06 4.188e-06 4.188e-06 0.00% Gravity::set_mass_offset() 6 2.09e-06 2.09e-06 2.09e-06 0.00% Castro::retry_advance_ctu() 5 2.019e-06 2.019e-06 2.019e-06 0.00% Castro::FluxRegCrseInit 5 1.604e-06 1.604e-06 1.604e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.172e-06 1.172e-06 1.172e-06 0.00% Castro::FluxRegFineAdd() 5 1.058e-06 1.058e-06 1.058e-06 0.00% AmrLevel::AmrLevel() 1 8.17e-07 8.17e-07 8.17e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-10-gf84c7a8f77d6) finalized