Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-11-g9aa23c202a13) initialized Starting run at 08:32:12 UTC on 2022-09-13. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09-11-g9aa23c202 Microphysics git describe: 22.08-12-g91211ba4 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.052968045 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030518574 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048617149 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052389897 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.050120725 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.048520711 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.065437361 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.049192565 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.073139539 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.061584888 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.063471732 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.054827797 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.057247425 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.049225887 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031571995 seconds Ending run at 08:32:13 UTC on 2022-09-13. Run time = 0.842697543 Run time without initialization = 0.705976756 Average number of zones advanced per microsecond: 3.713 Average number of zones advanced per microsecond per rank: 3.713 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8427 ... 0.8427 ... 0.8427 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2061 0.2061 0.2061 24.46% Castro::construct_ctu_hydro_source() 10 0.1875 0.1875 0.1875 22.25% MLCellLinOp::applyBC() 4433 0.08109 0.08109 0.08109 9.62% MLPoisson::Fsmooth() 3280 0.06423 0.06423 0.06423 7.62% StateData::FillBoundary(geom) 328 0.02446 0.02446 0.02446 2.90% MLCGSolver::bicgstab 82 0.02413 0.02413 0.02413 2.86% MultiFab::Dot() 1114 0.02272 0.02272 0.02272 2.70% Castro::computeTemp() 63 0.01519 0.01519 0.01519 1.80% MultiFab::LinComb() 1586 0.01456 0.01456 0.01456 1.73% FillBoundary_nowait() 4023 0.01437 0.01437 0.01437 1.70% FabArray::setVal() 1144 0.01435 0.01435 0.01435 1.70% FabArray::ParallelCopy_nowait() 861 0.01331 0.01331 0.01331 1.58% Castro::normalize_species() 62 0.01258 0.01258 0.01258 1.49% StateDataPhysBCFunct::() 41 0.01253 0.01253 0.01253 1.49% MLCellLinOp::defineAuxData() 11 0.01196 0.01196 0.01196 1.42% MLPoisson::Fapply() 1142 0.01187 0.01187 0.01187 1.41% Castro::enforce_min_density() 62 0.01118 0.01118 0.01118 1.33% Gravity::fill_multipole_BCs() 11 0.008345 0.008345 0.008345 0.99% MLMG::addInterpCorrection() 410 0.007809 0.007809 0.007809 0.93% amrex::average_down 410 0.006877 0.006877 0.006877 0.82% MultiFab::Xpay() 585 0.006688 0.006688 0.006688 0.79% Castro::do_advance_ctu() 10 0.005089 0.005089 0.005089 0.60% Castro::estTimeStep() 21 0.004933 0.004933 0.004933 0.59% Amr::checkPoint() 3 0.004788 0.004788 0.004788 0.57% Castro::reset_internal_energy(MultiFab) 63 0.004135 0.004135 0.004135 0.49% BndryData::define() 11 0.003989 0.003989 0.003989 0.47% Castro::construct_new_gravity_source() 10 0.003299 0.003299 0.003299 0.39% Amr::writePlotFile() 2 0.002641 0.002641 0.002641 0.31% Castro::construct_old_gravity_source() 10 0.002597 0.002597 0.002597 0.31% MLMG::ResNormInf() 93 0.002099 0.002099 0.002099 0.25% Gravity::get_new_grav_vector() 11 0.001928 0.001928 0.001928 0.23% MultiFab::Saxpy() 20 0.001813 0.001813 0.001813 0.22% Castro::expand_state() 10 0.00174 0.00174 0.00174 0.21% Gravity::get_old_grav_vector() 10 0.001737 0.001737 0.001737 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001685 0.001685 0.001685 0.20% MultiFab::Add() 82 0.001679 0.001679 0.001679 0.20% MLCellLinOp::setLevelBC() 11 0.001551 0.001551 0.001551 0.18% Castro::reset_internal_energy(Fab) 504 0.001528 0.001528 0.001528 0.18% Gravity::actual_solve_with_mlmg() 11 0.001457 0.001457 0.001457 0.17% FabArray::mult() 43 0.001337 0.001337 0.001337 0.16% FabArray::setDomainBndry() 41 0.001308 0.001308 0.001308 0.16% MLMG::prepareForSolve() 11 0.001232 0.001232 0.001232 0.15% MLCellLinOp::prepareForSolve() 11 0.001205 0.001205 0.001205 0.14% Castro::initData() 1 0.001193 0.001193 0.001193 0.14% MultiFab::contains_nan() 20 0.001179 0.001179 0.001179 0.14% MLCellLinOp::smooth() 1640 0.001166 0.001166 0.001166 0.14% Castro::enforce_speed_limit() 62 0.001115 0.001115 0.001115 0.13% MLCellLinOp::compGrad() 11 0.000914 0.000914 0.000914 0.11% FabArray::FillBoundary() 4023 0.0008771 0.0008771 0.0008771 0.10% FabArrayBase::getCPC() 1323 0.0007448 0.0007448 0.0007448 0.09% FabArrayBase::CPC::define() 454 0.0006755 0.0006755 0.0006755 0.08% FabArrayBase::getFB() 4023 0.0005909 0.0005909 0.0005909 0.07% Amr::InitAmr() 1 0.0004936 0.0004936 0.0004936 0.06% Gravity::solve_for_phi() 10 0.0004621 0.0004621 0.0004621 0.05% MLCellLinOp::apply() 1142 0.0004375 0.0004375 0.0004375 0.05% Gravity::update_max_rhs() 11 0.000413 0.000413 0.000413 0.05% CGSolver::sxay() 1586 0.0003558 0.0003558 0.0003558 0.04% Amr::coarseTimeStep() 10 0.0003288 0.0003288 0.0003288 0.04% MultiFab::Copy() 11 0.0003285 0.0003285 0.0003285 0.04% main() 1 0.0003058 0.0003058 0.0003058 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002972 0.0002972 0.0002972 0.04% FillPatchIterator::Initialize 41 0.0002904 0.0002904 0.0002904 0.03% MLCellLinOp::defineBC() 11 0.0002815 0.0002815 0.0002815 0.03% FabArray::ParallelCopy() 861 0.0002641 0.0002641 0.0002641 0.03% MultiFab::max() 11 0.0002551 0.0002551 0.0002551 0.03% MLMG::mgVcycle() 82 0.0002347 0.0002347 0.0002347 0.03% Castro::subcycle_advance_ctu() 10 0.0002271 0.0002271 0.0002271 0.03% MLCellLinOp::correctionResidual() 492 0.0002188 0.0002188 0.0002188 0.03% MLMG::MLRhsNormInf() 11 0.0002162 0.0002162 0.0002162 0.03% Castro::construct_new_gravity() 10 0.0002066 0.0002066 0.0002066 0.02% Amr::timeStep() 10 0.0001644 0.0001644 0.0001644 0.02% MLMG:computeResOfCorrection() 410 0.0001497 0.0001497 0.0001497 0.02% MLLinOp::defineGrids() 11 0.0001383 0.0001383 0.0001383 0.02% StateData::checkPoint() 12 0.0001269 0.0001269 0.0001269 0.02% MLMG::mgVcycle_down::0 82 0.0001046 0.0001046 0.0001046 0.01% MLMG::mgVcycle_down::1 82 9.146e-05 9.146e-05 9.146e-05 0.01% Castro::Castro() 1 9.039e-05 9.039e-05 9.039e-05 0.01% FabArrayBase::FB::FB() 56 8.883e-05 8.883e-05 8.883e-05 0.01% MLMG::mgVcycle_down::2 82 8.665e-05 8.665e-05 8.665e-05 0.01% Castro::initialize_advance() 10 8.563e-05 8.563e-05 8.563e-05 0.01% Castro::clean_state() 62 8.218e-05 8.218e-05 8.218e-05 0.01% MLMG::mgVcycle_down::3 82 8.111e-05 8.111e-05 8.111e-05 0.01% MLMG::mgVcycle_down::4 82 8.033e-05 8.033e-05 8.033e-05 0.01% MLMG::actualBottomSolve() 82 7.657e-05 7.657e-05 7.657e-05 0.01% MLMG::mgVcycle_up::4 82 7.643e-05 7.643e-05 7.643e-05 0.01% AmrLevel::checkPoint() 3 7.1e-05 7.1e-05 7.1e-05 0.01% MLMG::solve() 11 7.094e-05 7.094e-05 7.094e-05 0.01% Castro::finalize_advance() 10 6.587e-05 6.587e-05 6.587e-05 0.01% Castro::initialize_do_advance() 10 6.285e-05 6.285e-05 6.285e-05 0.01% MLMG::mgVcycle_up::0 82 6.021e-05 6.021e-05 6.021e-05 0.01% MLMG::mgVcycle_up::1 82 5.798e-05 5.798e-05 5.798e-05 0.01% MLMG::oneIter() 82 5.785e-05 5.785e-05 5.785e-05 0.01% MLMG::mgVcycle_up::3 82 5.758e-05 5.758e-05 5.758e-05 0.01% MLMG::mgVcycle_up::2 82 5.551e-05 5.551e-05 5.551e-05 0.01% MLCellLinOp::solutionResidual() 93 5.145e-05 5.145e-05 5.145e-05 0.01% StateData::define() 4 4.386e-05 4.386e-05 4.386e-05 0.01% Castro::advance() 10 4.172e-05 4.172e-05 4.172e-05 0.00% Castro::swap_state_time_levels() 10 4.134e-05 4.134e-05 4.134e-05 0.00% MLMG::computeResidual() 82 3.926e-05 3.926e-05 3.926e-05 0.00% Castro::enforce_consistent_e() 1 3.53e-05 3.53e-05 3.53e-05 0.00% Gravity::actual_multilevel_solve() 1 3.41e-05 3.41e-05 3.41e-05 0.00% Castro::finalize_do_advance() 10 3.388e-05 3.388e-05 3.388e-05 0.00% MLMG::mgVcycle_bottom 82 3.31e-05 3.31e-05 3.31e-05 0.00% MLPoisson::define() 11 2.972e-05 2.972e-05 2.972e-05 0.00% FillPatchSingleLevel 41 2.881e-05 2.881e-05 2.881e-05 0.00% Castro::initMFs() 1 2.849e-05 2.849e-05 2.849e-05 0.00% makeSFC 55 2.759e-05 2.759e-05 2.759e-05 0.00% Amr::defBaseLevel() 1 2.755e-05 2.755e-05 2.755e-05 0.00% Amr::writeSmallPlotFile() 1 2.718e-05 2.718e-05 2.718e-05 0.00% MLLinOp::define() 11 2.539e-05 2.539e-05 2.539e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.479e-05 2.479e-05 2.479e-05 0.00% Castro::buildMetrics() 1 2.362e-05 2.362e-05 2.362e-05 0.00% Amr::FinalizeInit() 1 2.042e-05 2.042e-05 2.042e-05 0.00% Castro::construct_old_source() 50 1.954e-05 1.954e-05 1.954e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.929e-05 1.929e-05 1.929e-05 0.00% Castro::construct_new_source() 50 1.862e-05 1.862e-05 1.862e-05 0.00% Castro::do_new_sources() 10 1.76e-05 1.76e-05 1.76e-05 0.00% Castro::do_old_sources() 10 1.626e-05 1.626e-05 1.626e-05 0.00% DistributionMapping::Distribute() 56 1.516e-05 1.516e-05 1.516e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.361e-05 1.361e-05 1.361e-05 0.00% Castro::check_for_nan() 20 1.229e-05 1.229e-05 1.229e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 1.169e-05 1.169e-05 1.169e-05 0.00% Castro::apply_source_to_state() 20 1.164e-05 1.164e-05 1.164e-05 0.00% Castro::construct_old_gravity() 10 1.036e-05 1.036e-05 1.036e-05 0.00% MLMG::computeMLResidual() 11 9.376e-06 9.376e-06 9.376e-06 0.00% Amr::initSubcycle() 1 9.088e-06 9.088e-06 9.088e-06 0.00% Gravity::swapTimeLevels() 10 8.844e-06 8.844e-06 8.844e-06 0.00% MLPoisson::prepareForSolve() 11 8.835e-06 8.835e-06 8.835e-06 0.00% Castro::post_timestep() 10 8.078e-06 8.078e-06 8.078e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.967e-06 6.967e-06 6.967e-06 0.00% MLMG::getGradSolution() 11 6.761e-06 6.761e-06 6.761e-06 0.00% Castro::computeNewDt() 9 5.655e-06 5.655e-06 5.655e-06 0.00% AmrLevel::checkPointPost() 3 5.171e-06 5.171e-06 5.171e-06 0.00% Amr::InitializeInit() 1 4.925e-06 4.925e-06 4.925e-06 0.00% Castro::create_source_corrector() 10 3.946e-06 3.946e-06 3.946e-06 0.00% Gravity::set_mass_offset() 11 3.898e-06 3.898e-06 3.898e-06 0.00% Castro::retry_advance_ctu() 10 3.866e-06 3.866e-06 3.866e-06 0.00% MLMG::MLResNormInf() 11 3.791e-06 3.791e-06 3.791e-06 0.00% Castro::post_init() 1 3.471e-06 3.471e-06 3.471e-06 0.00% Castro::FluxRegCrseInit 10 2.983e-06 2.983e-06 2.983e-06 0.00% Castro::FluxRegFineAdd() 10 2.429e-06 2.429e-06 2.429e-06 0.00% Amr::init() 1 2.206e-06 2.206e-06 2.206e-06 0.00% Castro::computeInitialDt() 2 2.202e-06 2.202e-06 2.202e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.883e-06 1.883e-06 1.883e-06 0.00% AmrLevel::checkPointPre() 3 1.632e-06 1.632e-06 1.632e-06 0.00% Castro::post_regrid() 1 1.223e-06 1.223e-06 1.223e-06 0.00% Amr::initialInit() 1 1.12e-06 1.12e-06 1.12e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8427 0.8427 0.8427 100.00% Amr::coarseTimeStep() 10 0.6742 0.6742 0.6742 80.00% Amr::timeStep() 10 0.5728 0.5728 0.5728 67.97% Castro::advance() 10 0.5658 0.5658 0.5658 67.14% Castro::subcycle_advance_ctu() 10 0.5549 0.5549 0.5549 65.85% Castro::do_advance_ctu() 10 0.5547 0.5547 0.5547 65.82% Gravity::solve_phi_with_mlmg() 11 0.3179 0.3179 0.3179 37.72% Gravity::actual_solve_with_mlmg() 11 0.3093 0.3093 0.3093 36.71% Castro::construct_new_gravity() 10 0.2891 0.2891 0.2891 34.30% MLMG::solve() 11 0.2861 0.2861 0.2861 33.95% Gravity::solve_for_phi() 10 0.2735 0.2735 0.2735 32.45% MLMG::oneIter() 82 0.2711 0.2711 0.2711 32.16% MLMG::mgVcycle() 82 0.2693 0.2693 0.2693 31.96% VisMF::Write(FabArray) 11 0.2061 0.2061 0.2061 24.46% Castro::construct_ctu_hydro_source() 10 0.1875 0.1875 0.1875 22.25% Amr::checkPoint() 3 0.1515 0.1515 0.1515 17.98% AmrLevel::checkPoint() 3 0.1467 0.1467 0.1467 17.41% StateData::checkPoint() 12 0.1467 0.1467 0.1467 17.40% MLCellLinOp::smooth() 1640 0.1375 0.1375 0.1375 16.32% Amr::init() 1 0.1361 0.1361 0.1361 16.15% MLCellLinOp::applyBC() 4433 0.09702 0.09702 0.09702 11.51% MLMG::mgVcycle_bottom 82 0.08302 0.08302 0.08302 9.85% MLMG::actualBottomSolve() 82 0.08299 0.08299 0.08299 9.85% MLCGSolver::bicgstab 82 0.08217 0.08217 0.08217 9.75% MLPoisson::Fsmooth() 3280 0.06423 0.06423 0.06423 7.62% Amr::writePlotFile() 2 0.06222 0.06222 0.06222 7.38% Amr::initialInit() 1 0.05246 0.05246 0.05246 6.22% Amr::FinalizeInit() 1 0.04826 0.04826 0.04826 5.73% Castro::post_init() 1 0.04687 0.04687 0.04687 5.56% Castro::clean_state() 62 0.04494 0.04494 0.04494 5.33% Gravity::multilevel_solve_for_new_phi() 1 0.04493 0.04493 0.04493 5.33% Gravity::actual_multilevel_solve() 1 0.04491 0.04491 0.04491 5.33% FillPatchIterator::Initialize 41 0.04258 0.04258 0.04258 5.05% FillPatchSingleLevel 41 0.04098 0.04098 0.04098 4.86% StateDataPhysBCFunct::() 41 0.03699 0.03699 0.03699 4.39% MLCellLinOp::apply() 1142 0.03668 0.03668 0.03668 4.35% MLMG::mgVcycle_down::0 82 0.03555 0.03555 0.03555 4.22% MLMG::mgVcycle_up::0 82 0.03061 0.03061 0.03061 3.63% StateData::FillBoundary(geom) 328 0.02446 0.02446 0.02446 2.90% MultiFab::Dot() 1114 0.02272 0.02272 0.02272 2.70% MLCellLinOp::correctionResidual() 492 0.02151 0.02151 0.02151 2.55% Castro::computeTemp() 63 0.02085 0.02085 0.02085 2.47% Castro::initialize_do_advance() 10 0.02008 0.02008 0.02008 2.38% MLPoisson::define() 11 0.01879 0.01879 0.01879 2.23% MLMG:computeResOfCorrection() 410 0.01857 0.01857 0.01857 2.20% MLMG::mgVcycle_down::1 82 0.01791 0.01791 0.01791 2.13% MLMG::mgVcycle_down::2 82 0.01746 0.01746 0.01746 2.07% Gravity::get_new_grav_vector() 11 0.01725 0.01725 0.01725 2.05% MLMG::mgVcycle_down::3 82 0.01658 0.01658 0.01658 1.97% FabArray::FillBoundary() 4023 0.01592 0.01592 0.01592 1.89% MLMG::mgVcycle_down::4 82 0.0158 0.0158 0.0158 1.88% FillBoundary_nowait() 4023 0.01505 0.01505 0.01505 1.79% Castro::construct_old_gravity() 10 0.01503 0.01503 0.01503 1.78% Gravity::get_old_grav_vector() 10 0.01501 0.01501 0.01501 1.78% CGSolver::sxay() 1586 0.01492 0.01492 0.01492 1.77% MultiFab::LinComb() 1586 0.01456 0.01456 0.01456 1.73% FabArray::ParallelCopy() 861 0.01436 0.01436 0.01436 1.70% FabArray::setVal() 1144 0.01435 0.01435 0.01435 1.70% FabArray::ParallelCopy_nowait() 861 0.0141 0.0141 0.0141 1.67% MLCGSolver::ParallelAllReduce 1514 0.01355 0.01355 0.01355 1.61% MLMG::mgVcycle_up::2 82 0.0134 0.0134 0.0134 1.59% MLCellLinOp::defineAuxData() 11 0.01333 0.01333 0.01333 1.58% MLMG::mgVcycle_up::1 82 0.01326 0.01326 0.01326 1.57% MLMG::addInterpCorrection() 410 0.01301 0.01301 0.01301 1.54% MLMG::mgVcycle_up::4 82 0.01276 0.01276 0.01276 1.51% MLMG::mgVcycle_up::3 82 0.01275 0.01275 0.01275 1.51% Castro::normalize_species() 62 0.01258 0.01258 0.01258 1.49% amrex::average_down 410 0.01209 0.01209 0.01209 1.43% Castro::do_new_sources() 10 0.01207 0.01207 0.01207 1.43% MLPoisson::Fapply() 1142 0.01187 0.01187 0.01187 1.41% Castro::expand_state() 10 0.0114 0.0114 0.0114 1.35% Castro::enforce_min_density() 62 0.01118 0.01118 0.01118 1.33% Castro::do_old_sources() 10 0.01106 0.01106 0.01106 1.31% Castro::initialize_advance() 10 0.0108 0.0108 0.0108 1.28% Gravity::fill_multipole_BCs() 11 0.008345 0.008345 0.008345 0.99% MLCellLinOp::solutionResidual() 93 0.007157 0.007157 0.007157 0.85% Castro::post_timestep() 10 0.006784 0.006784 0.006784 0.81% MultiFab::Xpay() 585 0.006688 0.006688 0.006688 0.79% MLMG::computeResidual() 82 0.006164 0.006164 0.006164 0.73% Castro::reset_internal_energy(MultiFab) 63 0.005663 0.005663 0.005663 0.67% MLMG::prepareForSolve() 11 0.005445 0.005445 0.005445 0.65% MLCellLinOp::defineBC() 11 0.005218 0.005218 0.005218 0.62% BndryData::define() 11 0.004936 0.004936 0.004936 0.59% Castro::estTimeStep() 21 0.004933 0.004933 0.004933 0.59% Amr::InitializeInit() 1 0.004195 0.004195 0.004195 0.50% Amr::defBaseLevel() 1 0.00419 0.00419 0.00419 0.50% Castro::initData() 1 0.003651 0.003651 0.003651 0.43% Castro::construct_new_source() 50 0.003318 0.003318 0.003318 0.39% Castro::construct_new_gravity_source() 10 0.003299 0.003299 0.003299 0.39% Castro::construct_old_source() 50 0.002616 0.002616 0.002616 0.31% Castro::construct_old_gravity_source() 10 0.002597 0.002597 0.002597 0.31% MLMG::ResNormInf() 93 0.002099 0.002099 0.002099 0.25% Castro::computeNewDt() 9 0.00209 0.00209 0.00209 0.25% Castro::apply_source_to_state() 20 0.001824 0.001824 0.001824 0.22% MultiFab::Saxpy() 20 0.001813 0.001813 0.001813 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001685 0.001685 0.001685 0.20% MultiFab::Add() 82 0.001679 0.001679 0.001679 0.20% MLCellLinOp::setLevelBC() 11 0.001551 0.001551 0.001551 0.18% Castro::reset_internal_energy(Fab) 504 0.001528 0.001528 0.001528 0.18% MLMG::getGradSolution() 11 0.001423 0.001423 0.001423 0.17% FabArrayBase::getCPC() 1323 0.00142 0.00142 0.00142 0.17% MLCellLinOp::compGrad() 11 0.001416 0.001416 0.001416 0.17% FabArray::mult() 43 0.001337 0.001337 0.001337 0.16% FabArray::setDomainBndry() 41 0.001308 0.001308 0.001308 0.16% MLPoisson::prepareForSolve() 11 0.001213 0.001213 0.001213 0.14% MLCellLinOp::prepareForSolve() 11 0.001205 0.001205 0.001205 0.14% Castro::check_for_nan() 20 0.001191 0.001191 0.001191 0.14% MultiFab::contains_nan() 20 0.001179 0.001179 0.001179 0.14% Castro::post_regrid() 1 0.00113 0.00113 0.00113 0.13% Castro::enforce_speed_limit() 62 0.001115 0.001115 0.001115 0.13% MLMG::computeMLResidual() 11 0.001042 0.001042 0.001042 0.12% Gravity::update_max_rhs() 11 0.000819 0.000819 0.000819 0.10% Castro::computeInitialDt() 2 0.0006949 0.0006949 0.0006949 0.08% FabArrayBase::getFB() 4023 0.0006797 0.0006797 0.0006797 0.08% FabArrayBase::CPC::define() 454 0.0006755 0.0006755 0.0006755 0.08% Amr::InitAmr() 1 0.0005027 0.0005027 0.0005027 0.06% Gravity::swapTimeLevels() 10 0.0004492 0.0004492 0.0004492 0.05% Castro::Castro() 1 0.000448 0.000448 0.000448 0.05% MultiFab::Copy() 11 0.0003285 0.0003285 0.0003285 0.04% MLMG::MLResNormInf() 11 0.00028 0.00028 0.00028 0.03% MultiFab::max() 11 0.0002551 0.0002551 0.0002551 0.03% MLLinOp::define() 11 0.0002208 0.0002208 0.0002208 0.03% MLMG::MLRhsNormInf() 11 0.0002162 0.0002162 0.0002162 0.03% MLLinOp::defineGrids() 11 0.0001955 0.0001955 0.0001955 0.02% Castro::buildMetrics() 1 0.0001625 0.0001625 0.0001625 0.02% FabArrayBase::FB::FB() 56 8.883e-05 8.883e-05 8.883e-05 0.01% Castro::finalize_advance() 10 7.129e-05 7.129e-05 7.129e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.529e-05 5.529e-05 5.529e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.082e-05 5.082e-05 5.082e-05 0.01% StateData::define() 4 4.386e-05 4.386e-05 4.386e-05 0.01% makeSFC 55 4.167e-05 4.167e-05 4.167e-05 0.00% Castro::swap_state_time_levels() 10 4.134e-05 4.134e-05 4.134e-05 0.00% Castro::enforce_consistent_e() 1 3.53e-05 3.53e-05 3.53e-05 0.00% Castro::finalize_do_advance() 10 3.388e-05 3.388e-05 3.388e-05 0.00% Castro::initMFs() 1 2.849e-05 2.849e-05 2.849e-05 0.00% Amr::writeSmallPlotFile() 1 2.718e-05 2.718e-05 2.718e-05 0.00% DistributionMapping::Distribute() 56 1.516e-05 1.516e-05 1.516e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 1.277e-05 1.277e-05 1.277e-05 0.00% Amr::initSubcycle() 1 9.088e-06 9.088e-06 9.088e-06 0.00% AmrLevel::checkPointPost() 3 5.171e-06 5.171e-06 5.171e-06 0.00% Castro::create_source_corrector() 10 3.946e-06 3.946e-06 3.946e-06 0.00% Gravity::set_mass_offset() 11 3.898e-06 3.898e-06 3.898e-06 0.00% Castro::retry_advance_ctu() 10 3.866e-06 3.866e-06 3.866e-06 0.00% Castro::FluxRegCrseInit 10 2.983e-06 2.983e-06 2.983e-06 0.00% Castro::FluxRegFineAdd() 10 2.429e-06 2.429e-06 2.429e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.883e-06 1.883e-06 1.883e-06 0.00% AmrLevel::checkPointPre() 3 1.632e-06 1.632e-06 1.632e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-11-g9aa23c202a13) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-11-g9aa23c202a13) initialized Starting run at 08:32:14 UTC on 2022-09-13. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09-11-g9aa23c202 Microphysics git describe: 22.08-12-g91211ba4 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.450993109 Restart time = 0.04827824 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051795913 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048922815 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.051343601 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.058439159 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065983397 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031879751 seconds Ending run at 08:32:14 UTC on 2022-09-13. Run time = 0.35762653 Run time without initialization = 0.308767827 Average number of zones advanced per microsecond: 4.245 Average number of zones advanced per microsecond per rank: 4.245 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3577 ... 0.3577 ... 0.3577 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0768 0.0768 0.0768 21.49% VisMF::Read() 3 0.0403 0.0403 0.0403 11.27% MLCellLinOp::applyBC() 1946 0.03433 0.03433 0.03433 9.60% VisMF::Write(FabArray) 1 0.03043 0.03043 0.03043 8.51% MLPoisson::Fsmooth() 1440 0.02715 0.02715 0.02715 7.59% StateData::FillBoundary(geom) 160 0.0117 0.0117 0.0117 3.27% MLCGSolver::bicgstab 36 0.01021 0.01021 0.01021 2.85% MultiFab::Dot() 484 0.009573 0.009573 0.009573 2.68% Castro::normalize_species() 30 0.009046 0.009046 0.009046 2.53% Castro::computeTemp() 30 0.007231 0.007231 0.007231 2.02% FabArray::setVal() 537 0.006785 0.006785 0.006785 1.90% MLCellLinOp::defineAuxData() 6 0.006279 0.006279 0.006279 1.76% FillBoundary_nowait() 1766 0.006206 0.006206 0.006206 1.74% MultiFab::LinComb() 690 0.006155 0.006155 0.006155 1.72% FabArray::ParallelCopy_nowait() 380 0.005958 0.005958 0.005958 1.67% StateDataPhysBCFunct::() 20 0.005181 0.005181 0.005181 1.45% MLPoisson::Fapply() 500 0.005058 0.005058 0.005058 1.41% Castro::enforce_min_density() 30 0.004924 0.004924 0.004924 1.38% Gravity::fill_multipole_BCs() 6 0.004761 0.004761 0.004761 1.33% Amr::restart() 1 0.003646 0.003646 0.003646 1.02% MLMG::addInterpCorrection() 180 0.00335 0.00335 0.00335 0.94% amrex::average_down 180 0.002964 0.002964 0.002964 0.83% MultiFab::Xpay() 258 0.002855 0.002855 0.002855 0.80% Castro::do_advance_ctu() 5 0.002464 0.002464 0.002464 0.69% Castro::subcycle_advance_ctu() 5 0.002411 0.002411 0.002411 0.67% Castro::estTimeStep() 10 0.002385 0.002385 0.002385 0.67% BndryData::define() 6 0.002117 0.002117 0.002117 0.59% Castro::construct_new_gravity_source() 5 0.001775 0.001775 0.001775 0.50% Castro::reset_internal_energy(MultiFab) 30 0.001709 0.001709 0.001709 0.48% Amr::writePlotFile() 1 0.001537 0.001537 0.001537 0.43% Castro::construct_old_gravity_source() 5 0.001451 0.001451 0.001451 0.41% Castro::enforce_speed_limit() 30 0.001292 0.001292 0.001292 0.36% Gravity::get_old_grav_vector() 5 0.000969 0.000969 0.000969 0.27% Castro::reset_internal_energy(Fab) 240 0.0009579 0.0009579 0.0009579 0.27% MLMG::ResNormInf() 42 0.0009195 0.0009195 0.0009195 0.26% MultiFab::Saxpy() 10 0.0009183 0.0009183 0.0009183 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008798 0.0008798 0.0008798 0.25% Castro::expand_state() 5 0.0008711 0.0008711 0.0008711 0.24% Gravity::get_new_grav_vector() 5 0.0008614 0.0008614 0.0008614 0.24% MLCellLinOp::setLevelBC() 6 0.0008173 0.0008173 0.0008173 0.23% Gravity::actual_solve_with_mlmg() 6 0.0007351 0.0007351 0.0007351 0.21% MultiFab::Add() 36 0.0007253 0.0007253 0.0007253 0.20% MLMG::prepareForSolve() 6 0.0006647 0.0006647 0.0006647 0.19% FabArray::mult() 22 0.0006509 0.0006509 0.0006509 0.18% FabArray::setDomainBndry() 20 0.0006385 0.0006385 0.0006385 0.18% MLCellLinOp::prepareForSolve() 6 0.0006305 0.0006305 0.0006305 0.18% MultiFab::contains_nan() 10 0.0005996 0.0005996 0.0005996 0.17% MLCellLinOp::smooth() 720 0.0004982 0.0004982 0.0004982 0.14% MLCellLinOp::compGrad() 6 0.0004977 0.0004977 0.0004977 0.14% Amr::InitAmr() 1 0.000407 0.000407 0.000407 0.11% FabArray::FillBoundary() 1766 0.0004007 0.0004007 0.0004007 0.11% FabArrayBase::CPC::define() 244 0.0003882 0.0003882 0.0003882 0.11% FabArrayBase::getCPC() 632 0.0003655 0.0003655 0.0003655 0.10% main() 1 0.0002476 0.0002476 0.0002476 0.07% FabArrayBase::getFB() 1766 0.0002453 0.0002453 0.0002453 0.07% Castro::create_source_corrector() 5 0.0002335 0.0002335 0.0002335 0.07% Gravity::solve_for_phi() 5 0.0002303 0.0002303 0.0002303 0.06% Gravity::update_max_rhs() 6 0.0002294 0.0002294 0.0002294 0.06% MLCellLinOp::apply() 500 0.0001862 0.0001862 0.0001862 0.05% Castro::construct_new_gravity() 5 0.0001855 0.0001855 0.0001855 0.05% CGSolver::sxay() 690 0.0001801 0.0001801 0.0001801 0.05% MultiFab::Copy() 6 0.0001756 0.0001756 0.0001756 0.05% Amr::coarseTimeStep() 5 0.0001697 0.0001697 0.0001697 0.05% Castro::construct_new_source() 25 0.0001599 0.0001599 0.0001599 0.04% MLCellLinOp::defineBC() 6 0.0001494 0.0001494 0.0001494 0.04% FillPatchIterator::Initialize 20 0.0001393 0.0001393 0.0001393 0.04% MultiFab::max() 6 0.0001354 0.0001354 0.0001354 0.04% MLCGSolver::ParallelAllReduce 659 0.0001228 0.0001228 0.0001228 0.03% FabArray::ParallelCopy() 380 0.0001187 0.0001187 0.0001187 0.03% MLMG::MLRhsNormInf() 6 0.0001123 0.0001123 0.0001123 0.03% AmrLevel::restart() 1 9.855e-05 9.855e-05 9.855e-05 0.03% Castro::advance() 5 9.565e-05 9.565e-05 9.565e-05 0.03% Amr::timeStep() 5 9.429e-05 9.429e-05 9.429e-05 0.03% MLCellLinOp::correctionResidual() 216 9.341e-05 9.341e-05 9.341e-05 0.03% Castro::initialize_do_advance() 5 9.295e-05 9.295e-05 9.295e-05 0.03% MLMG::mgVcycle() 36 8.961e-05 8.961e-05 8.961e-05 0.03% MLLinOp::defineGrids() 6 8.817e-05 8.817e-05 8.817e-05 0.02% Castro::finalize_advance() 5 7.608e-05 7.608e-05 7.608e-05 0.02% StateData::restartDoit() 4 7.473e-05 7.473e-05 7.473e-05 0.02% MLMG:computeResOfCorrection() 180 7.077e-05 7.077e-05 7.077e-05 0.02% Castro::initialize_advance() 5 7.051e-05 7.051e-05 7.051e-05 0.02% Castro::computeNewDt() 5 7.033e-05 7.033e-05 7.033e-05 0.02% Castro::post_timestep() 5 6.916e-05 6.916e-05 6.916e-05 0.02% FabArrayBase::FB::FB() 26 5.741e-05 5.741e-05 5.741e-05 0.02% Castro::construct_old_source() 25 5.338e-05 5.338e-05 5.338e-05 0.01% MLMG::mgVcycle_down::1 36 4.664e-05 4.664e-05 4.664e-05 0.01% MLMG::mgVcycle_down::0 36 4.332e-05 4.332e-05 4.332e-05 0.01% Castro::clean_state() 30 4.189e-05 4.189e-05 4.189e-05 0.01% MLMG::mgVcycle_down::2 36 4.058e-05 4.058e-05 4.058e-05 0.01% MLMG::mgVcycle_down::4 36 3.586e-05 3.586e-05 3.586e-05 0.01% MLMG::mgVcycle_down::3 36 3.545e-05 3.545e-05 3.545e-05 0.01% MLMG::actualBottomSolve() 36 3.46e-05 3.46e-05 3.46e-05 0.01% Castro::buildMetrics() 1 3.326e-05 3.326e-05 3.326e-05 0.01% MLMG::mgVcycle_up::4 36 3.299e-05 3.299e-05 3.299e-05 0.01% MLMG::solve() 6 3.238e-05 3.238e-05 3.238e-05 0.01% Castro::post_restart() 1 3.182e-05 3.182e-05 3.182e-05 0.01% Gravity::actual_multilevel_solve() 1 3.048e-05 3.048e-05 3.048e-05 0.01% Castro::swap_state_time_levels() 5 2.739e-05 2.739e-05 2.739e-05 0.01% Castro::initMFs() 1 2.713e-05 2.713e-05 2.713e-05 0.01% MLMG::mgVcycle_up::0 36 2.708e-05 2.708e-05 2.708e-05 0.01% Amr::writeSmallPlotFile() 1 2.634e-05 2.634e-05 2.634e-05 0.01% MLMG::mgVcycle_up::3 36 2.597e-05 2.597e-05 2.597e-05 0.01% MLMG::mgVcycle_up::2 36 2.519e-05 2.519e-05 2.519e-05 0.01% MLMG::oneIter() 36 2.518e-05 2.518e-05 2.518e-05 0.01% MLMG::mgVcycle_up::1 36 2.415e-05 2.415e-05 2.415e-05 0.01% Castro::construct_old_gravity() 5 2.362e-05 2.362e-05 2.362e-05 0.01% MLCellLinOp::solutionResidual() 42 2.297e-05 2.297e-05 2.297e-05 0.01% MLPoisson::define() 6 2.073e-05 2.073e-05 2.073e-05 0.01% MLLinOp::define() 6 1.974e-05 1.974e-05 1.974e-05 0.01% Castro::finalize_do_advance() 5 1.886e-05 1.886e-05 1.886e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.852e-05 1.852e-05 1.852e-05 0.01% MLMG::computeResidual() 36 1.733e-05 1.733e-05 1.733e-05 0.00% MLMG::mgVcycle_bottom 36 1.537e-05 1.537e-05 1.537e-05 0.00% DistributionMapping::Distribute() 31 1.505e-05 1.505e-05 1.505e-05 0.00% FillPatchSingleLevel 20 1.418e-05 1.418e-05 1.418e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.396e-05 1.396e-05 1.396e-05 0.00% makeSFC 30 1.393e-05 1.393e-05 1.393e-05 0.00% Castro::do_new_sources() 5 1.015e-05 1.015e-05 1.015e-05 0.00% Castro::check_for_nan() 10 8.648e-06 8.648e-06 8.648e-06 0.00% Amr::initSubcycle() 1 8.627e-06 8.627e-06 8.627e-06 0.00% Castro::do_old_sources() 5 8.456e-06 8.456e-06 8.456e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.302e-06 7.302e-06 7.302e-06 0.00% Castro::apply_source_to_state() 10 6.352e-06 6.352e-06 6.352e-06 0.00% MLPoisson::prepareForSolve() 6 4.758e-06 4.758e-06 4.758e-06 0.00% MLMG::computeMLResidual() 6 4.475e-06 4.475e-06 4.475e-06 0.00% Gravity::swapTimeLevels() 5 4.192e-06 4.192e-06 4.192e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.136e-06 3.136e-06 3.136e-06 0.00% MLMG::getGradSolution() 6 2.985e-06 2.985e-06 2.985e-06 0.00% Castro::retry_advance_ctu() 5 2.893e-06 2.893e-06 2.893e-06 0.00% Gravity::set_mass_offset() 6 2.6e-06 2.6e-06 2.6e-06 0.00% Castro::FluxRegCrseInit 5 2.315e-06 2.315e-06 2.315e-06 0.00% MLMG::MLResNormInf() 6 2.02e-06 2.02e-06 2.02e-06 0.00% Castro::FluxRegFineAdd() 5 1.605e-06 1.605e-06 1.605e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.215e-06 1.215e-06 1.215e-06 0.00% Amr::init() 1 1.19e-06 1.19e-06 1.19e-06 0.00% AmrLevel::AmrLevel() 1 9.69e-07 9.69e-07 9.69e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3576 0.3576 0.3576 100.00% Amr::coarseTimeStep() 5 0.2766 0.2766 0.2766 77.35% Amr::timeStep() 5 0.2752 0.2752 0.2752 76.94% Castro::advance() 5 0.2713 0.2713 0.2713 75.85% Castro::subcycle_advance_ctu() 5 0.2649 0.2649 0.2649 74.05% Castro::do_advance_ctu() 5 0.2624 0.2624 0.2624 73.38% Castro::construct_new_gravity() 5 0.1434 0.1434 0.1434 40.09% Gravity::solve_phi_with_mlmg() 6 0.1393 0.1393 0.1393 38.94% Gravity::solve_for_phi() 5 0.1357 0.1357 0.1357 37.94% Gravity::actual_solve_with_mlmg() 6 0.1344 0.1344 0.1344 37.57% MLMG::solve() 6 0.1221 0.1221 0.1221 34.13% MLMG::oneIter() 36 0.1149 0.1149 0.1149 32.12% MLMG::mgVcycle() 36 0.1141 0.1141 0.1141 31.91% Castro::construct_ctu_hydro_source() 5 0.07685 0.07685 0.07685 21.49% MLCellLinOp::smooth() 720 0.05821 0.05821 0.05821 16.28% Amr::init() 1 0.04834 0.04834 0.04834 13.51% Amr::restart() 1 0.04834 0.04834 0.04834 13.51% MLCellLinOp::applyBC() 1946 0.04124 0.04124 0.04124 11.53% AmrLevel::restart() 1 0.04053 0.04053 0.04053 11.33% StateData::restartDoit() 4 0.04043 0.04043 0.04043 11.30% VisMF::Read() 3 0.0403 0.0403 0.0403 11.27% MLMG::mgVcycle_bottom 36 0.0351 0.0351 0.0351 9.81% MLMG::actualBottomSolve() 36 0.03508 0.03508 0.03508 9.81% MLCGSolver::bicgstab 36 0.03473 0.03473 0.03473 9.71% Amr::writePlotFile() 1 0.03196 0.03196 0.03196 8.94% VisMF::Write(FabArray) 1 0.03043 0.03043 0.03043 8.51% MLPoisson::Fsmooth() 1440 0.02715 0.02715 0.02715 7.59% Castro::clean_state() 30 0.0252 0.0252 0.0252 7.05% FillPatchIterator::Initialize 20 0.01968 0.01968 0.01968 5.50% FillPatchSingleLevel 20 0.0189 0.0189 0.0189 5.28% StateDataPhysBCFunct::() 20 0.01688 0.01688 0.01688 4.72% MLCellLinOp::apply() 500 0.01565 0.01565 0.01565 4.38% MLMG::mgVcycle_down::0 36 0.0152 0.0152 0.0152 4.25% MLMG::mgVcycle_up::0 36 0.01306 0.01306 0.01306 3.65% StateData::FillBoundary(geom) 160 0.0117 0.0117 0.0117 3.27% Castro::initialize_do_advance() 5 0.01131 0.01131 0.01131 3.16% MLPoisson::define() 6 0.009982 0.009982 0.009982 2.79% Castro::computeTemp() 30 0.009898 0.009898 0.009898 2.77% MultiFab::Dot() 484 0.009573 0.009573 0.009573 2.68% MLCellLinOp::correctionResidual() 216 0.009147 0.009147 0.009147 2.56% Castro::normalize_species() 30 0.009046 0.009046 0.009046 2.53% MLMG:computeResOfCorrection() 180 0.007887 0.007887 0.007887 2.21% Castro::do_new_sources() 5 0.007851 0.007851 0.007851 2.20% MLMG::mgVcycle_down::1 36 0.007592 0.007592 0.007592 2.12% Castro::construct_old_gravity() 5 0.00756 0.00756 0.00756 2.11% Gravity::get_old_grav_vector() 5 0.007536 0.007536 0.007536 2.11% Gravity::get_new_grav_vector() 5 0.007511 0.007511 0.007511 2.10% MLMG::mgVcycle_down::2 36 0.00735 0.00735 0.00735 2.06% MLCellLinOp::defineAuxData() 6 0.007019 0.007019 0.007019 1.96% MLMG::mgVcycle_down::3 36 0.007 0.007 0.007 1.96% FabArray::FillBoundary() 1766 0.00691 0.00691 0.00691 1.93% FabArray::setVal() 537 0.006785 0.006785 0.006785 1.90% MLMG::mgVcycle_down::4 36 0.006725 0.006725 0.006725 1.88% FillBoundary_nowait() 1766 0.006509 0.006509 0.006509 1.82% FabArray::ParallelCopy() 380 0.006438 0.006438 0.006438 1.80% CGSolver::sxay() 690 0.006335 0.006335 0.006335 1.77% FabArray::ParallelCopy_nowait() 380 0.006319 0.006319 0.006319 1.77% Castro::initialize_advance() 5 0.006243 0.006243 0.006243 1.75% MultiFab::LinComb() 690 0.006155 0.006155 0.006155 1.72% MLCGSolver::ParallelAllReduce 659 0.005755 0.005755 0.005755 1.61% MLMG::mgVcycle_up::2 36 0.005669 0.005669 0.005669 1.59% MLMG::mgVcycle_up::1 36 0.005577 0.005577 0.005577 1.56% MLMG::addInterpCorrection() 180 0.00557 0.00557 0.00557 1.56% MLMG::mgVcycle_up::4 36 0.005402 0.005402 0.005402 1.51% MLMG::mgVcycle_up::3 36 0.005385 0.005385 0.005385 1.51% Castro::expand_state() 5 0.005352 0.005352 0.005352 1.50% amrex::average_down 180 0.005177 0.005177 0.005177 1.45% Castro::do_old_sources() 5 0.005097 0.005097 0.005097 1.43% MLPoisson::Fapply() 500 0.005058 0.005058 0.005058 1.41% Castro::enforce_min_density() 30 0.004924 0.004924 0.004924 1.38% Gravity::fill_multipole_BCs() 6 0.004761 0.004761 0.004761 1.33% Castro::post_restart() 1 0.003977 0.003977 0.003977 1.11% Gravity::multilevel_solve_for_new_phi() 1 0.003852 0.003852 0.003852 1.08% Gravity::actual_multilevel_solve() 1 0.003833 0.003833 0.003833 1.07% Castro::post_timestep() 5 0.00381 0.00381 0.00381 1.07% MLCellLinOp::solutionResidual() 42 0.003187 0.003187 0.003187 0.89% MLMG::prepareForSolve() 6 0.002898 0.002898 0.002898 0.81% MultiFab::Xpay() 258 0.002855 0.002855 0.002855 0.80% MLCellLinOp::defineBC() 6 0.002798 0.002798 0.002798 0.78% Castro::reset_internal_energy(MultiFab) 30 0.002667 0.002667 0.002667 0.75% BndryData::define() 6 0.002648 0.002648 0.002648 0.74% MLMG::computeResidual() 36 0.002647 0.002647 0.002647 0.74% Castro::estTimeStep() 10 0.002385 0.002385 0.002385 0.67% Castro::construct_new_source() 25 0.001935 0.001935 0.001935 0.54% Castro::construct_new_gravity_source() 5 0.001775 0.001775 0.001775 0.50% Castro::construct_old_source() 25 0.001504 0.001504 0.001504 0.42% Castro::construct_old_gravity_source() 5 0.001451 0.001451 0.001451 0.41% Castro::computeNewDt() 5 0.001301 0.001301 0.001301 0.36% Castro::enforce_speed_limit() 30 0.001292 0.001292 0.001292 0.36% Castro::reset_internal_energy(Fab) 240 0.0009579 0.0009579 0.0009579 0.27% Castro::apply_source_to_state() 10 0.0009247 0.0009247 0.0009247 0.26% MLMG::ResNormInf() 42 0.0009195 0.0009195 0.0009195 0.26% MultiFab::Saxpy() 10 0.0009183 0.0009183 0.0009183 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008798 0.0008798 0.0008798 0.25% MLCellLinOp::setLevelBC() 6 0.0008173 0.0008173 0.0008173 0.23% MLMG::getGradSolution() 6 0.0007709 0.0007709 0.0007709 0.22% MLCellLinOp::compGrad() 6 0.0007679 0.0007679 0.0007679 0.21% FabArrayBase::getCPC() 632 0.0007537 0.0007537 0.0007537 0.21% MultiFab::Add() 36 0.0007253 0.0007253 0.0007253 0.20% FabArray::mult() 22 0.0006509 0.0006509 0.0006509 0.18% FabArray::setDomainBndry() 20 0.0006385 0.0006385 0.0006385 0.18% MLPoisson::prepareForSolve() 6 0.0006353 0.0006353 0.0006353 0.18% MLCellLinOp::prepareForSolve() 6 0.0006305 0.0006305 0.0006305 0.18% Castro::check_for_nan() 10 0.0006083 0.0006083 0.0006083 0.17% MultiFab::contains_nan() 10 0.0005996 0.0005996 0.0005996 0.17% MLMG::computeMLResidual() 6 0.0005619 0.0005619 0.0005619 0.16% Gravity::update_max_rhs() 6 0.0004437 0.0004437 0.0004437 0.12% Amr::InitAmr() 1 0.0004157 0.0004157 0.0004157 0.12% FabArrayBase::CPC::define() 244 0.0003882 0.0003882 0.0003882 0.11% FabArrayBase::getFB() 1766 0.0003027 0.0003027 0.0003027 0.08% Gravity::swapTimeLevels() 5 0.0002395 0.0002395 0.0002395 0.07% Castro::create_source_corrector() 5 0.0002335 0.0002335 0.0002335 0.07% MultiFab::Copy() 6 0.0001756 0.0001756 0.0001756 0.05% Castro::buildMetrics() 1 0.0001521 0.0001521 0.0001521 0.04% MLMG::MLResNormInf() 6 0.0001472 0.0001472 0.0001472 0.04% MLLinOp::define() 6 0.0001441 0.0001441 0.0001441 0.04% MultiFab::max() 6 0.0001354 0.0001354 0.0001354 0.04% MLLinOp::defineGrids() 6 0.0001244 0.0001244 0.0001244 0.03% MLMG::MLRhsNormInf() 6 0.0001123 0.0001123 0.0001123 0.03% Castro::finalize_advance() 5 8e-05 8e-05 8e-05 0.02% FabArrayBase::FB::FB() 26 5.741e-05 5.741e-05 5.741e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 3.502e-05 3.502e-05 3.502e-05 0.01% makeSFC 30 2.772e-05 2.772e-05 2.772e-05 0.01% Castro::swap_state_time_levels() 5 2.739e-05 2.739e-05 2.739e-05 0.01% Castro::initMFs() 1 2.713e-05 2.713e-05 2.713e-05 0.01% Amr::writeSmallPlotFile() 1 2.634e-05 2.634e-05 2.634e-05 0.01% Castro::finalize_do_advance() 5 1.886e-05 1.886e-05 1.886e-05 0.01% DistributionMapping::Distribute() 31 1.505e-05 1.505e-05 1.505e-05 0.00% Amr::initSubcycle() 1 8.627e-06 8.627e-06 8.627e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.395e-06 4.395e-06 4.395e-06 0.00% Castro::retry_advance_ctu() 5 2.893e-06 2.893e-06 2.893e-06 0.00% Gravity::set_mass_offset() 6 2.6e-06 2.6e-06 2.6e-06 0.00% Castro::FluxRegCrseInit 5 2.315e-06 2.315e-06 2.315e-06 0.00% Castro::FluxRegFineAdd() 5 1.605e-06 1.605e-06 1.605e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.215e-06 1.215e-06 1.215e-06 0.00% AmrLevel::AmrLevel() 1 9.69e-07 9.69e-07 9.69e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-11-g9aa23c202a13) finalized