Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-4-ge4ab0485621d) initialized Starting run at 08:32:00 UTC on 2022-10-06. Successfully read inputs file ... Castro git describe: 22.09-1-g65b273ad0 AMReX git describe: 22.10-4-ge4ab04856 Microphysics git describe: 22.10-2-gbaa4f033 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.068438522 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.029772446 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048603918 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051733405 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.052026316 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.052355486 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.073572411 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.050896772 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.062057269 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.060006864 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.060908988 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059563783 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.055061027 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.047527784 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029447697 seconds Ending run at 08:32:01 UTC on 2022-10-06. Run time = 0.854553919 Run time without initialization = 0.704414713 Average number of zones advanced per microsecond: 3.721 Average number of zones advanced per microsecond per rank: 3.721 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8546 ... 0.8546 ... 0.8546 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2171 0.2171 0.2171 25.41% Castro::construct_ctu_hydro_source() 10 0.1962 0.1962 0.1962 22.96% MLCellLinOp::applyBC() 4433 0.07822 0.07822 0.07822 9.15% MLPoisson::Fsmooth() 3280 0.06316 0.06316 0.06316 7.39% MLCGSolver::bicgstab 82 0.02781 0.02781 0.02781 3.25% StateData::FillBoundary(geom) 328 0.02303 0.02303 0.02303 2.69% MultiFab::Dot() 1114 0.02177 0.02177 0.02177 2.55% MultiFab::LinComb() 1586 0.01416 0.01416 0.01416 1.66% FabArray::setVal() 1144 0.01403 0.01403 0.01403 1.64% FillBoundary_nowait() 4023 0.014 0.014 0.014 1.64% Castro::computeTemp() 63 0.01301 0.01301 0.01301 1.52% StateDataPhysBCFunct::() 41 0.01301 0.01301 0.01301 1.52% Castro::normalize_species() 62 0.01295 0.01295 0.01295 1.52% FabArray::ParallelCopy_nowait() 861 0.01288 0.01288 0.01288 1.51% MLPoisson::Fapply() 1142 0.01156 0.01156 0.01156 1.35% MLCellLinOp::defineAuxData() 11 0.01153 0.01153 0.01153 1.35% Castro::enforce_min_density() 62 0.00917 0.00917 0.00917 1.07% Gravity::fill_multipole_BCs() 11 0.008235 0.008235 0.008235 0.96% MLMG::addInterpCorrection() 410 0.007614 0.007614 0.007614 0.89% amrex::average_down 410 0.006788 0.006788 0.006788 0.79% MultiFab::Xpay() 585 0.006483 0.006483 0.006483 0.76% Amr::checkPoint() 3 0.006164 0.006164 0.006164 0.72% Castro::estTimeStep() 21 0.004736 0.004736 0.004736 0.55% Castro::do_advance_ctu() 10 0.00446 0.00446 0.00446 0.52% Castro::reset_internal_energy(MultiFab) 63 0.004154 0.004154 0.004154 0.49% BndryData::define() 11 0.003728 0.003728 0.003728 0.44% Castro::construct_new_gravity_source() 10 0.003153 0.003153 0.003153 0.37% Amr::writePlotFile() 2 0.002868 0.002868 0.002868 0.34% Castro::construct_old_gravity_source() 10 0.00263 0.00263 0.00263 0.31% MLMG::ResNormInf() 93 0.002024 0.002024 0.002024 0.24% Gravity::get_new_grav_vector() 11 0.001929 0.001929 0.001929 0.23% MultiFab::Saxpy() 20 0.001809 0.001809 0.001809 0.21% Castro::enforce_speed_limit() 62 0.001764 0.001764 0.001764 0.21% Castro::expand_state() 10 0.001734 0.001734 0.001734 0.20% Gravity::get_old_grav_vector() 10 0.001715 0.001715 0.001715 0.20% MultiFab::Add() 82 0.001638 0.001638 0.001638 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001612 0.001612 0.001612 0.19% MLCellLinOp::setLevelBC() 11 0.001524 0.001524 0.001524 0.18% Gravity::actual_solve_with_mlmg() 11 0.001483 0.001483 0.001483 0.17% Castro::reset_internal_energy(Fab) 504 0.001479 0.001479 0.001479 0.17% FabArray::mult() 43 0.001332 0.001332 0.001332 0.16% FabArray::setDomainBndry() 41 0.001281 0.001281 0.001281 0.15% MLMG::prepareForSolve() 11 0.001249 0.001249 0.001249 0.15% Castro::initData() 1 0.001193 0.001193 0.001193 0.14% MultiFab::contains_nan() 20 0.001185 0.001185 0.001185 0.14% MLCellLinOp::prepareForSolve() 11 0.001164 0.001164 0.001164 0.14% MLCellLinOp::smooth() 1640 0.001127 0.001127 0.001127 0.13% MLCellLinOp::compGrad() 11 0.0009231 0.0009231 0.0009231 0.11% FabArray::FillBoundary() 4023 0.0009038 0.0009038 0.0009038 0.11% FabArrayBase::getCPC() 1323 0.0007866 0.0007866 0.0007866 0.09% FabArrayBase::CPC::define() 454 0.0006805 0.0006805 0.0006805 0.08% FabArrayBase::getFB() 4023 0.0006077 0.0006077 0.0006077 0.07% MLCellLinOp::apply() 1142 0.0004964 0.0004964 0.0004964 0.06% Amr::InitAmr() 1 0.0004715 0.0004715 0.0004715 0.06% Gravity::solve_for_phi() 10 0.0004494 0.0004494 0.0004494 0.05% Gravity::update_max_rhs() 11 0.0004115 0.0004115 0.0004115 0.05% CGSolver::sxay() 1586 0.0003571 0.0003571 0.0003571 0.04% Amr::coarseTimeStep() 10 0.0003462 0.0003462 0.0003462 0.04% MultiFab::Copy() 11 0.0003381 0.0003381 0.0003381 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003068 0.0003068 0.0003068 0.04% FillPatchIterator::Initialize 41 0.0002948 0.0002948 0.0002948 0.03% MLCellLinOp::defineBC() 11 0.0002828 0.0002828 0.0002828 0.03% FabArray::ParallelCopy() 861 0.0002793 0.0002793 0.0002793 0.03% main() 1 0.0002719 0.0002719 0.0002719 0.03% MultiFab::max() 11 0.0002633 0.0002633 0.0002633 0.03% MLMG::mgVcycle() 82 0.000237 0.000237 0.000237 0.03% MLCellLinOp::correctionResidual() 492 0.0002303 0.0002303 0.0002303 0.03% Castro::subcycle_advance_ctu() 10 0.0002266 0.0002266 0.0002266 0.03% MLMG::MLRhsNormInf() 11 0.000213 0.000213 0.000213 0.02% Castro::construct_new_gravity() 10 0.0002094 0.0002094 0.0002094 0.02% MLLinOp::defineGrids() 11 0.0002 0.0002 0.0002 0.02% Amr::timeStep() 10 0.0001787 0.0001787 0.0001787 0.02% MLMG:computeResOfCorrection() 410 0.0001569 0.0001569 0.0001569 0.02% StateData::checkPoint() 12 0.0001317 0.0001317 0.0001317 0.02% MLMG::mgVcycle_down::0 82 0.0001099 0.0001099 0.0001099 0.01% MLMG::mgVcycle_down::1 82 9.141e-05 9.141e-05 9.141e-05 0.01% Castro::initialize_advance() 10 9.061e-05 9.061e-05 9.061e-05 0.01% MLMG::mgVcycle_down::2 82 8.853e-05 8.853e-05 8.853e-05 0.01% Castro::Castro() 1 8.758e-05 8.758e-05 8.758e-05 0.01% Castro::clean_state() 62 8.64e-05 8.64e-05 8.64e-05 0.01% MLMG::actualBottomSolve() 82 8.626e-05 8.626e-05 8.626e-05 0.01% FabArrayBase::FB::FB() 56 8.534e-05 8.534e-05 8.534e-05 0.01% MLMG::mgVcycle_down::3 82 8.244e-05 8.244e-05 8.244e-05 0.01% MLMG::mgVcycle_down::4 82 8.078e-05 8.078e-05 8.078e-05 0.01% MLMG::solve() 11 7.467e-05 7.467e-05 7.467e-05 0.01% AmrLevel::checkPoint() 3 7.331e-05 7.331e-05 7.331e-05 0.01% Castro::finalize_advance() 10 6.938e-05 6.938e-05 6.938e-05 0.01% MLMG::mgVcycle_up::4 82 6.583e-05 6.583e-05 6.583e-05 0.01% Castro::initialize_do_advance() 10 6.451e-05 6.451e-05 6.451e-05 0.01% MLMG::oneIter() 82 6.272e-05 6.272e-05 6.272e-05 0.01% MLMG::mgVcycle_up::0 82 5.766e-05 5.766e-05 5.766e-05 0.01% MLMG::mgVcycle_up::3 82 5.409e-05 5.409e-05 5.409e-05 0.01% MLMG::mgVcycle_up::2 82 5.296e-05 5.296e-05 5.296e-05 0.01% MLMG::mgVcycle_up::1 82 5.254e-05 5.254e-05 5.254e-05 0.01% MLCellLinOp::solutionResidual() 93 5.146e-05 5.146e-05 5.146e-05 0.01% Castro::advance() 10 4.241e-05 4.241e-05 4.241e-05 0.00% StateData::define() 4 4.067e-05 4.067e-05 4.067e-05 0.00% MLMG::computeResidual() 82 4.015e-05 4.015e-05 4.015e-05 0.00% Castro::swap_state_time_levels() 10 4.011e-05 4.011e-05 4.011e-05 0.00% Castro::finalize_do_advance() 10 3.627e-05 3.627e-05 3.627e-05 0.00% MLMG::mgVcycle_bottom 82 3.521e-05 3.521e-05 3.521e-05 0.00% Castro::enforce_consistent_e() 1 3.333e-05 3.333e-05 3.333e-05 0.00% Gravity::actual_multilevel_solve() 1 3.19e-05 3.19e-05 3.19e-05 0.00% Castro::do_new_sources() 10 3.092e-05 3.092e-05 3.092e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.025e-05 3.025e-05 3.025e-05 0.00% makeSFC 55 2.944e-05 2.944e-05 2.944e-05 0.00% FillPatchSingleLevel 41 2.709e-05 2.709e-05 2.709e-05 0.00% MLLinOp::define() 11 2.608e-05 2.608e-05 2.608e-05 0.00% Castro::initMFs() 1 2.592e-05 2.592e-05 2.592e-05 0.00% MLPoisson::define() 11 2.579e-05 2.579e-05 2.579e-05 0.00% Amr::writeSmallPlotFile() 1 2.53e-05 2.53e-05 2.53e-05 0.00% Castro::buildMetrics() 1 2.347e-05 2.347e-05 2.347e-05 0.00% Amr::defBaseLevel() 1 2.232e-05 2.232e-05 2.232e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.963e-05 1.963e-05 1.963e-05 0.00% Amr::FinalizeInit() 1 1.924e-05 1.924e-05 1.924e-05 0.00% Castro::construct_new_source() 50 1.781e-05 1.781e-05 1.781e-05 0.00% Castro::construct_old_source() 50 1.756e-05 1.756e-05 1.756e-05 0.00% Castro::do_old_sources() 10 1.618e-05 1.618e-05 1.618e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.523e-05 1.523e-05 1.523e-05 0.00% DistributionMapping::Distribute() 56 1.464e-05 1.464e-05 1.464e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.342e-05 1.342e-05 1.342e-05 0.00% Castro::check_for_nan() 20 1.265e-05 1.265e-05 1.265e-05 0.00% Castro::post_timestep() 10 1.166e-05 1.166e-05 1.166e-05 0.00% Castro::apply_source_to_state() 20 1.151e-05 1.151e-05 1.151e-05 0.00% Castro::construct_old_gravity() 10 1.076e-05 1.076e-05 1.076e-05 0.00% MLMG::computeMLResidual() 11 1.031e-05 1.031e-05 1.031e-05 0.00% Gravity::swapTimeLevels() 10 1.007e-05 1.007e-05 1.007e-05 0.00% MLPoisson::prepareForSolve() 11 8.887e-06 8.887e-06 8.887e-06 0.00% Amr::initSubcycle() 1 8.812e-06 8.812e-06 8.812e-06 0.00% MLMG::getGradSolution() 11 8.219e-06 8.219e-06 8.219e-06 0.00% Castro::computeNewDt() 9 6.396e-06 6.396e-06 6.396e-06 0.00% Gravity::set_mass_offset() 11 5.137e-06 5.137e-06 5.137e-06 0.00% AmrLevel::checkPointPost() 3 5e-06 5e-06 5e-06 0.00% Amr::InitializeInit() 1 4.665e-06 4.665e-06 4.665e-06 0.00% Castro::retry_advance_ctu() 10 4.44e-06 4.44e-06 4.44e-06 0.00% Castro::post_init() 1 4.046e-06 4.046e-06 4.046e-06 0.00% MLMG::MLResNormInf() 11 3.691e-06 3.691e-06 3.691e-06 0.00% Castro::create_source_corrector() 10 3.599e-06 3.599e-06 3.599e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.101e-06 3.101e-06 3.101e-06 0.00% Castro::FluxRegCrseInit 10 2.74e-06 2.74e-06 2.74e-06 0.00% Amr::init() 1 2.403e-06 2.403e-06 2.403e-06 0.00% Castro::computeInitialDt() 2 2.309e-06 2.309e-06 2.309e-06 0.00% Castro::FluxRegFineAdd() 10 2.306e-06 2.306e-06 2.306e-06 0.00% AmrLevel::checkPointPre() 3 2.149e-06 2.149e-06 2.149e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.903e-06 1.903e-06 1.903e-06 0.00% Castro::post_regrid() 1 1.7e-06 1.7e-06 1.7e-06 0.00% Amr::initialInit() 1 1e-06 1e-06 1e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8546 0.8546 0.8546 100.00% Amr::coarseTimeStep() 10 0.6747 0.6747 0.6747 78.95% Amr::timeStep() 10 0.5734 0.5734 0.5734 67.09% Castro::advance() 10 0.5665 0.5665 0.5665 66.29% Castro::subcycle_advance_ctu() 10 0.5542 0.5542 0.5542 64.85% Castro::do_advance_ctu() 10 0.5539 0.5539 0.5539 64.82% Gravity::solve_phi_with_mlmg() 11 0.3136 0.3136 0.3136 36.70% Gravity::actual_solve_with_mlmg() 11 0.3052 0.3052 0.3052 35.71% Castro::construct_new_gravity() 10 0.2852 0.2852 0.2852 33.37% MLMG::solve() 11 0.2826 0.2826 0.2826 33.07% Gravity::solve_for_phi() 10 0.2701 0.2701 0.2701 31.60% MLMG::oneIter() 82 0.2679 0.2679 0.2679 31.35% MLMG::mgVcycle() 82 0.2662 0.2662 0.2662 31.15% VisMF::Write(FabArray) 11 0.2171 0.2171 0.2171 25.41% Castro::construct_ctu_hydro_source() 10 0.1962 0.1962 0.1962 22.96% Amr::checkPoint() 3 0.167 0.167 0.167 19.54% AmrLevel::checkPoint() 3 0.1608 0.1608 0.1608 18.82% StateData::checkPoint() 12 0.1608 0.1608 0.1608 18.81% Amr::init() 1 0.1496 0.1496 0.1496 17.50% MLCellLinOp::smooth() 1640 0.1339 0.1339 0.1339 15.67% MLCellLinOp::applyBC() 4433 0.09381 0.09381 0.09381 10.98% MLMG::mgVcycle_bottom 82 0.08483 0.08483 0.08483 9.93% MLMG::actualBottomSolve() 82 0.0848 0.0848 0.0848 9.92% MLCGSolver::bicgstab 82 0.08399 0.08399 0.08399 9.83% MLPoisson::Fsmooth() 3280 0.06316 0.06316 0.06316 7.39% Amr::writePlotFile() 2 0.05937 0.05937 0.05937 6.95% Amr::initialInit() 1 0.05119 0.05119 0.05119 5.99% Amr::FinalizeInit() 1 0.04729 0.04729 0.04729 5.53% Castro::post_init() 1 0.04592 0.04592 0.04592 5.37% Gravity::multilevel_solve_for_new_phi() 1 0.04406 0.04406 0.04406 5.16% Gravity::actual_multilevel_solve() 1 0.04404 0.04404 0.04404 5.15% Castro::clean_state() 62 0.04194 0.04194 0.04194 4.91% FillPatchIterator::Initialize 41 0.04164 0.04164 0.04164 4.87% FillPatchSingleLevel 41 0.04007 0.04007 0.04007 4.69% StateDataPhysBCFunct::() 41 0.03604 0.03604 0.03604 4.22% MLCellLinOp::apply() 1142 0.03573 0.03573 0.03573 4.18% MLMG::mgVcycle_down::0 82 0.0349 0.0349 0.0349 4.08% MLMG::mgVcycle_up::0 82 0.02986 0.02986 0.02986 3.49% StateData::FillBoundary(geom) 328 0.02303 0.02303 0.02303 2.69% MultiFab::Dot() 1114 0.02177 0.02177 0.02177 2.55% MLCellLinOp::correctionResidual() 492 0.02083 0.02083 0.02083 2.44% Castro::initialize_do_advance() 10 0.01872 0.01872 0.01872 2.19% Castro::computeTemp() 63 0.01864 0.01864 0.01864 2.18% MLPoisson::define() 11 0.01808 0.01808 0.01808 2.12% MLMG:computeResOfCorrection() 410 0.01798 0.01798 0.01798 2.10% MLMG::mgVcycle_down::1 82 0.01748 0.01748 0.01748 2.05% MLMG::mgVcycle_down::2 82 0.01695 0.01695 0.01695 1.98% Gravity::get_new_grav_vector() 11 0.01668 0.01668 0.01668 1.95% MLMG::mgVcycle_down::3 82 0.01609 0.01609 0.01609 1.88% FabArray::FillBoundary() 4023 0.0156 0.0156 0.0156 1.82% MLMG::mgVcycle_down::4 82 0.0153 0.0153 0.0153 1.79% FillBoundary_nowait() 4023 0.01469 0.01469 0.01469 1.72% Castro::construct_old_gravity() 10 0.01458 0.01458 0.01458 1.71% Gravity::get_old_grav_vector() 10 0.01457 0.01457 0.01457 1.70% CGSolver::sxay() 1586 0.01452 0.01452 0.01452 1.70% MultiFab::LinComb() 1586 0.01416 0.01416 0.01416 1.66% FabArray::setVal() 1144 0.01403 0.01403 0.01403 1.64% FabArray::ParallelCopy() 861 0.014 0.014 0.014 1.64% FabArray::ParallelCopy_nowait() 861 0.01372 0.01372 0.01372 1.61% MLMG::mgVcycle_up::2 82 0.01305 0.01305 0.01305 1.53% MLCGSolver::ParallelAllReduce 1514 0.01301 0.01301 0.01301 1.52% Castro::normalize_species() 62 0.01295 0.01295 0.01295 1.52% MLCellLinOp::defineAuxData() 11 0.01285 0.01285 0.01285 1.50% MLMG::mgVcycle_up::1 82 0.01279 0.01279 0.01279 1.50% MLMG::addInterpCorrection() 410 0.01261 0.01261 0.01261 1.48% MLMG::mgVcycle_up::3 82 0.01237 0.01237 0.01237 1.45% MLMG::mgVcycle_up::4 82 0.01232 0.01232 0.01232 1.44% Castro::initialize_advance() 10 0.01227 0.01227 0.01227 1.44% amrex::average_down 410 0.01179 0.01179 0.01179 1.38% MLPoisson::Fapply() 1142 0.01156 0.01156 0.01156 1.35% Castro::expand_state() 10 0.01156 0.01156 0.01156 1.35% Castro::do_new_sources() 10 0.0104 0.0104 0.0104 1.22% Castro::do_old_sources() 10 0.009937 0.009937 0.009937 1.16% Castro::enforce_min_density() 62 0.00917 0.00917 0.00917 1.07% Gravity::fill_multipole_BCs() 11 0.008235 0.008235 0.008235 0.96% MLCellLinOp::solutionResidual() 93 0.007057 0.007057 0.007057 0.83% Castro::post_timestep() 10 0.006656 0.006656 0.006656 0.78% MultiFab::Xpay() 585 0.006483 0.006483 0.006483 0.76% MLMG::computeResidual() 82 0.006091 0.006091 0.006091 0.71% Castro::reset_internal_energy(MultiFab) 63 0.005633 0.005633 0.005633 0.66% MLMG::prepareForSolve() 11 0.005347 0.005347 0.005347 0.63% MLCellLinOp::defineBC() 11 0.004915 0.004915 0.004915 0.58% Castro::estTimeStep() 21 0.004736 0.004736 0.004736 0.55% BndryData::define() 11 0.004632 0.004632 0.004632 0.54% Amr::InitializeInit() 1 0.003903 0.003903 0.003903 0.46% Amr::defBaseLevel() 1 0.003899 0.003899 0.003899 0.46% Castro::initData() 1 0.003381 0.003381 0.003381 0.40% Castro::construct_new_source() 50 0.003171 0.003171 0.003171 0.37% Castro::construct_new_gravity_source() 10 0.003153 0.003153 0.003153 0.37% Castro::construct_old_source() 50 0.002648 0.002648 0.002648 0.31% Castro::construct_old_gravity_source() 10 0.00263 0.00263 0.00263 0.31% MLMG::ResNormInf() 93 0.002024 0.002024 0.002024 0.24% Castro::computeNewDt() 9 0.001967 0.001967 0.001967 0.23% Castro::apply_source_to_state() 20 0.001821 0.001821 0.001821 0.21% MultiFab::Saxpy() 20 0.001809 0.001809 0.001809 0.21% Castro::enforce_speed_limit() 62 0.001764 0.001764 0.001764 0.21% MultiFab::Add() 82 0.001638 0.001638 0.001638 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001612 0.001612 0.001612 0.19% MLCellLinOp::setLevelBC() 11 0.001524 0.001524 0.001524 0.18% Castro::reset_internal_energy(Fab) 504 0.001479 0.001479 0.001479 0.17% FabArrayBase::getCPC() 1323 0.001467 0.001467 0.001467 0.17% MLMG::getGradSolution() 11 0.001423 0.001423 0.001423 0.17% MLCellLinOp::compGrad() 11 0.001415 0.001415 0.001415 0.17% FabArray::mult() 43 0.001332 0.001332 0.001332 0.16% FabArray::setDomainBndry() 41 0.001281 0.001281 0.001281 0.15% Castro::check_for_nan() 20 0.001197 0.001197 0.001197 0.14% MultiFab::contains_nan() 20 0.001185 0.001185 0.001185 0.14% MLPoisson::prepareForSolve() 11 0.001173 0.001173 0.001173 0.14% MLCellLinOp::prepareForSolve() 11 0.001164 0.001164 0.001164 0.14% Castro::post_regrid() 1 0.001114 0.001114 0.001114 0.13% MLMG::computeMLResidual() 11 0.001015 0.001015 0.001015 0.12% Gravity::update_max_rhs() 11 0.0008234 0.0008234 0.0008234 0.10% Castro::computeInitialDt() 2 0.0007645 0.0007645 0.0007645 0.09% FabArrayBase::getFB() 4023 0.0006931 0.0006931 0.0006931 0.08% FabArrayBase::CPC::define() 454 0.0006805 0.0006805 0.0006805 0.08% Amr::InitAmr() 1 0.0004803 0.0004803 0.0004803 0.06% Castro::Castro() 1 0.0004371 0.0004371 0.0004371 0.05% Gravity::swapTimeLevels() 10 0.000436 0.000436 0.000436 0.05% MultiFab::Copy() 11 0.0003381 0.0003381 0.0003381 0.04% MLLinOp::define() 11 0.0002862 0.0002862 0.0002862 0.03% MLMG::MLResNormInf() 11 0.0002756 0.0002756 0.0002756 0.03% MultiFab::max() 11 0.0002633 0.0002633 0.0002633 0.03% MLLinOp::defineGrids() 11 0.0002602 0.0002602 0.0002602 0.03% MLMG::MLRhsNormInf() 11 0.000213 0.000213 0.000213 0.02% Castro::buildMetrics() 1 0.0001572 0.0001572 0.0001572 0.02% FabArrayBase::FB::FB() 56 8.534e-05 8.534e-05 8.534e-05 0.01% Castro::finalize_advance() 10 7.443e-05 7.443e-05 7.443e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.823e-05 5.823e-05 5.823e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.409e-05 5.409e-05 5.409e-05 0.01% makeSFC 55 4.301e-05 4.301e-05 4.301e-05 0.01% StateData::define() 4 4.067e-05 4.067e-05 4.067e-05 0.00% Castro::swap_state_time_levels() 10 4.011e-05 4.011e-05 4.011e-05 0.00% Castro::finalize_do_advance() 10 3.627e-05 3.627e-05 3.627e-05 0.00% Castro::enforce_consistent_e() 1 3.333e-05 3.333e-05 3.333e-05 0.00% Castro::initMFs() 1 2.592e-05 2.592e-05 2.592e-05 0.00% Amr::writeSmallPlotFile() 1 2.53e-05 2.53e-05 2.53e-05 0.00% DistributionMapping::Distribute() 56 1.464e-05 1.464e-05 1.464e-05 0.00% Amr::initSubcycle() 1 8.812e-06 8.812e-06 8.812e-06 0.00% Gravity::set_mass_offset() 11 5.137e-06 5.137e-06 5.137e-06 0.00% AmrLevel::checkPointPost() 3 5e-06 5e-06 5e-06 0.00% Castro::retry_advance_ctu() 10 4.44e-06 4.44e-06 4.44e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.172e-06 4.172e-06 4.172e-06 0.00% Castro::create_source_corrector() 10 3.599e-06 3.599e-06 3.599e-06 0.00% Castro::FluxRegCrseInit 10 2.74e-06 2.74e-06 2.74e-06 0.00% Castro::FluxRegFineAdd() 10 2.306e-06 2.306e-06 2.306e-06 0.00% AmrLevel::checkPointPre() 3 2.149e-06 2.149e-06 2.149e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.903e-06 1.903e-06 1.903e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-4-ge4ab0485621d) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-4-ge4ab0485621d) initialized Starting run at 08:32:02 UTC on 2022-10-06. Successfully read inputs file ... Castro git describe: 22.09-1-g65b273ad0 AMReX git describe: 22.10-4-ge4ab04856 Microphysics git describe: 22.10-2-gbaa4f033 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.479379957 Restart time = 0.049084184 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.050950737 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.0505992 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.05966791 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.062430824 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.076707928 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.03162489 seconds Ending run at 08:32:02 UTC on 2022-10-06. Run time = 0.382026222 Run time without initialization = 0.332398109 Average number of zones advanced per microsecond: 3.943 Average number of zones advanced per microsecond per rank: 3.943 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3821 ... 0.3821 ... 0.3821 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1006 0.1006 0.1006 26.32% VisMF::Read() 3 0.0412 0.0412 0.0412 10.78% MLCellLinOp::applyBC() 1946 0.03464 0.03464 0.03464 9.07% VisMF::Write(FabArray) 1 0.02998 0.02998 0.02998 7.85% MLPoisson::Fsmooth() 1440 0.02714 0.02714 0.02714 7.10% StateData::FillBoundary(geom) 160 0.01109 0.01109 0.01109 2.90% MLCGSolver::bicgstab 36 0.01007 0.01007 0.01007 2.64% MultiFab::Dot() 484 0.009412 0.009412 0.009412 2.46% Castro::computeTemp() 30 0.008331 0.008331 0.008331 2.18% Castro::normalize_species() 30 0.007819 0.007819 0.007819 2.05% FabArray::setVal() 537 0.006686 0.006686 0.006686 1.75% MLCellLinOp::defineAuxData() 6 0.006186 0.006186 0.006186 1.62% FillBoundary_nowait() 1766 0.006161 0.006161 0.006161 1.61% MultiFab::LinComb() 690 0.006089 0.006089 0.006089 1.59% Castro::enforce_min_density() 30 0.006058 0.006058 0.006058 1.59% FabArray::ParallelCopy_nowait() 380 0.005923 0.005923 0.005923 1.55% StateDataPhysBCFunct::() 20 0.005404 0.005404 0.005404 1.41% Gravity::fill_multipole_BCs() 6 0.005046 0.005046 0.005046 1.32% MLPoisson::Fapply() 500 0.005 0.005 0.005 1.31% Amr::restart() 1 0.003638 0.003638 0.003638 0.95% MLMG::addInterpCorrection() 180 0.003313 0.003313 0.003313 0.87% Castro::do_advance_ctu() 5 0.003191 0.003191 0.003191 0.84% amrex::average_down 180 0.002962 0.002962 0.002962 0.78% MultiFab::Xpay() 258 0.002829 0.002829 0.002829 0.74% Castro::estTimeStep() 10 0.002166 0.002166 0.002166 0.57% BndryData::define() 6 0.002054 0.002054 0.002054 0.54% Castro::subcycle_advance_ctu() 5 0.001732 0.001732 0.001732 0.45% Castro::construct_new_gravity_source() 5 0.00173 0.00173 0.00173 0.45% Amr::writePlotFile() 1 0.001726 0.001726 0.001726 0.45% Castro::reset_internal_energy(MultiFab) 30 0.001702 0.001702 0.001702 0.45% Castro::construct_old_gravity_source() 5 0.001471 0.001471 0.001471 0.39% Castro::enforce_speed_limit() 30 0.001239 0.001239 0.001239 0.32% Gravity::get_old_grav_vector() 5 0.0009692 0.0009692 0.0009692 0.25% MultiFab::Saxpy() 10 0.0009234 0.0009234 0.0009234 0.24% MLMG::ResNormInf() 42 0.000901 0.000901 0.000901 0.24% Castro::reset_internal_energy(Fab) 240 0.0008926 0.0008926 0.0008926 0.23% Castro::expand_state() 5 0.0008764 0.0008764 0.0008764 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008742 0.0008742 0.0008742 0.23% Gravity::get_new_grav_vector() 5 0.0008635 0.0008635 0.0008635 0.23% Gravity::actual_solve_with_mlmg() 6 0.0008483 0.0008483 0.0008483 0.22% MLCellLinOp::setLevelBC() 6 0.000806 0.000806 0.000806 0.21% MultiFab::Add() 36 0.0007015 0.0007015 0.0007015 0.18% MLMG::prepareForSolve() 6 0.0006662 0.0006662 0.0006662 0.17% FabArray::mult() 22 0.0006464 0.0006464 0.0006464 0.17% MLCellLinOp::prepareForSolve() 6 0.0006341 0.0006341 0.0006341 0.17% FabArray::setDomainBndry() 20 0.0006287 0.0006287 0.0006287 0.16% MultiFab::contains_nan() 10 0.0005883 0.0005883 0.0005883 0.15% MLCellLinOp::compGrad() 6 0.0004946 0.0004946 0.0004946 0.13% MLCellLinOp::smooth() 720 0.0004635 0.0004635 0.0004635 0.12% FabArray::FillBoundary() 1766 0.0004083 0.0004083 0.0004083 0.11% Amr::InitAmr() 1 0.0003893 0.0003893 0.0003893 0.10% FabArrayBase::CPC::define() 244 0.0003867 0.0003867 0.0003867 0.10% FabArrayBase::getCPC() 632 0.000355 0.000355 0.000355 0.09% FabArrayBase::getFB() 1766 0.0002554 0.0002554 0.0002554 0.07% main() 1 0.0002536 0.0002536 0.0002536 0.07% Gravity::update_max_rhs() 6 0.0002289 0.0002289 0.0002289 0.06% MLCellLinOp::apply() 500 0.0001994 0.0001994 0.0001994 0.05% Gravity::solve_for_phi() 5 0.0001932 0.0001932 0.0001932 0.05% Castro::create_source_corrector() 5 0.0001904 0.0001904 0.0001904 0.05% Castro::construct_new_gravity() 5 0.0001811 0.0001811 0.0001811 0.05% MultiFab::Copy() 6 0.0001755 0.0001755 0.0001755 0.05% Amr::coarseTimeStep() 5 0.0001753 0.0001753 0.0001753 0.05% CGSolver::sxay() 690 0.0001689 0.0001689 0.0001689 0.04% Castro::construct_new_source() 25 0.0001554 0.0001554 0.0001554 0.04% MLCellLinOp::defineBC() 6 0.0001486 0.0001486 0.0001486 0.04% FillPatchIterator::Initialize 20 0.0001391 0.0001391 0.0001391 0.04% MultiFab::max() 6 0.000137 0.000137 0.000137 0.04% FabArray::ParallelCopy() 380 0.0001201 0.0001201 0.0001201 0.03% MLCGSolver::ParallelAllReduce 659 0.0001194 0.0001194 0.0001194 0.03% MLMG::MLRhsNormInf() 6 0.0001122 0.0001122 0.0001122 0.03% MLLinOp::defineGrids() 6 0.0001083 0.0001083 0.0001083 0.03% Castro::initialize_do_advance() 5 0.0001067 0.0001067 0.0001067 0.03% Castro::post_timestep() 5 0.0001012 0.0001012 0.0001012 0.03% MLCellLinOp::correctionResidual() 216 9.903e-05 9.903e-05 9.903e-05 0.03% Castro::advance() 5 9.389e-05 9.389e-05 9.389e-05 0.02% MLMG::mgVcycle() 36 9.329e-05 9.329e-05 9.329e-05 0.02% Castro::initialize_advance() 5 8.558e-05 8.558e-05 8.558e-05 0.02% Amr::timeStep() 5 8.265e-05 8.265e-05 8.265e-05 0.02% AmrLevel::restart() 1 8.075e-05 8.075e-05 8.075e-05 0.02% StateData::restartDoit() 4 7.436e-05 7.436e-05 7.436e-05 0.02% Castro::construct_old_source() 25 7.277e-05 7.277e-05 7.277e-05 0.02% MLMG:computeResOfCorrection() 180 7.158e-05 7.158e-05 7.158e-05 0.02% Castro::finalize_advance() 5 6.94e-05 6.94e-05 6.94e-05 0.02% Castro::computeNewDt() 5 6.891e-05 6.891e-05 6.891e-05 0.02% FabArrayBase::FB::FB() 26 5.94e-05 5.94e-05 5.94e-05 0.02% MLMG::mgVcycle_down::0 36 4.438e-05 4.438e-05 4.438e-05 0.01% MLMG::mgVcycle_down::1 36 4.071e-05 4.071e-05 4.071e-05 0.01% Castro::construct_old_gravity() 5 4.052e-05 4.052e-05 4.052e-05 0.01% Castro::clean_state() 30 3.987e-05 3.987e-05 3.987e-05 0.01% MLMG::mgVcycle_down::2 36 3.893e-05 3.893e-05 3.893e-05 0.01% MLMG::mgVcycle_down::4 36 3.65e-05 3.65e-05 3.65e-05 0.01% MLMG::actualBottomSolve() 36 3.523e-05 3.523e-05 3.523e-05 0.01% MLMG::mgVcycle_down::3 36 3.493e-05 3.493e-05 3.493e-05 0.01% MLMG::solve() 6 3.256e-05 3.256e-05 3.256e-05 0.01% Castro::buildMetrics() 1 3.17e-05 3.17e-05 3.17e-05 0.01% MLMG::mgVcycle_up::4 36 3.128e-05 3.128e-05 3.128e-05 0.01% Castro::post_restart() 1 3.125e-05 3.125e-05 3.125e-05 0.01% Gravity::actual_multilevel_solve() 1 2.965e-05 2.965e-05 2.965e-05 0.01% Castro::initMFs() 1 2.793e-05 2.793e-05 2.793e-05 0.01% Castro::swap_state_time_levels() 5 2.779e-05 2.779e-05 2.779e-05 0.01% MLMG::mgVcycle_up::1 36 2.753e-05 2.753e-05 2.753e-05 0.01% Gravity::solve_phi_with_mlmg() 6 2.697e-05 2.697e-05 2.697e-05 0.01% Amr::writeSmallPlotFile() 1 2.639e-05 2.639e-05 2.639e-05 0.01% MLMG::mgVcycle_up::0 36 2.6e-05 2.6e-05 2.6e-05 0.01% MLMG::oneIter() 36 2.568e-05 2.568e-05 2.568e-05 0.01% MLMG::mgVcycle_up::3 36 2.538e-05 2.538e-05 2.538e-05 0.01% MLMG::mgVcycle_up::2 36 2.348e-05 2.348e-05 2.348e-05 0.01% MLCellLinOp::solutionResidual() 42 2.28e-05 2.28e-05 2.28e-05 0.01% MLLinOp::define() 6 2.231e-05 2.231e-05 2.231e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.786e-05 1.786e-05 1.786e-05 0.00% MLMG::computeResidual() 36 1.777e-05 1.777e-05 1.777e-05 0.00% Castro::finalize_do_advance() 5 1.776e-05 1.776e-05 1.776e-05 0.00% MLPoisson::define() 6 1.658e-05 1.658e-05 1.658e-05 0.00% MLMG::mgVcycle_bottom 36 1.486e-05 1.486e-05 1.486e-05 0.00% makeSFC 30 1.402e-05 1.402e-05 1.402e-05 0.00% FillPatchSingleLevel 20 1.282e-05 1.282e-05 1.282e-05 0.00% Castro::do_new_sources() 5 1.006e-05 1.006e-05 1.006e-05 0.00% DistributionMapping::Distribute() 31 8.853e-06 8.853e-06 8.853e-06 0.00% Amr::initSubcycle() 1 8.8e-06 8.8e-06 8.8e-06 0.00% Castro::do_old_sources() 5 8.335e-06 8.335e-06 8.335e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.183e-06 7.183e-06 7.183e-06 0.00% Castro::check_for_nan() 10 6.525e-06 6.525e-06 6.525e-06 0.00% Castro::apply_source_to_state() 10 5.879e-06 5.879e-06 5.879e-06 0.00% AmrLevel::AmrLevel() 1 5.2e-06 5.2e-06 5.2e-06 0.00% MLPoisson::prepareForSolve() 6 4.765e-06 4.765e-06 4.765e-06 0.00% MLMG::computeMLResidual() 6 4.715e-06 4.715e-06 4.715e-06 0.00% Gravity::swapTimeLevels() 5 4.536e-06 4.536e-06 4.536e-06 0.00% MLMG::getGradSolution() 6 3.677e-06 3.677e-06 3.677e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.449e-06 3.449e-06 3.449e-06 0.00% Gravity::set_mass_offset() 6 2.393e-06 2.393e-06 2.393e-06 0.00% MLMG::MLResNormInf() 6 2.069e-06 2.069e-06 2.069e-06 0.00% Castro::retry_advance_ctu() 5 2.058e-06 2.058e-06 2.058e-06 0.00% Castro::FluxRegCrseInit 5 1.98e-06 1.98e-06 1.98e-06 0.00% Castro::FluxRegFineAdd() 5 1.264e-06 1.264e-06 1.264e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.056e-06 1.056e-06 1.056e-06 0.00% Amr::init() 1 9.87e-07 9.87e-07 9.87e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.382 0.382 0.382 100.00% Amr::coarseTimeStep() 5 0.3005 0.3005 0.3005 78.66% Amr::timeStep() 5 0.2991 0.2991 0.2991 78.29% Castro::advance() 5 0.2953 0.2953 0.2953 77.30% Castro::subcycle_advance_ctu() 5 0.2888 0.2888 0.2888 75.60% Castro::do_advance_ctu() 5 0.2871 0.2871 0.2871 75.15% Castro::construct_new_gravity() 5 0.1432 0.1432 0.1432 37.48% Gravity::solve_phi_with_mlmg() 6 0.1392 0.1392 0.1392 36.43% Gravity::solve_for_phi() 5 0.1356 0.1356 0.1356 35.50% Gravity::actual_solve_with_mlmg() 6 0.134 0.134 0.134 35.07% MLMG::solve() 6 0.1218 0.1218 0.1218 31.87% MLMG::oneIter() 36 0.1146 0.1146 0.1146 30.00% MLMG::mgVcycle() 36 0.1139 0.1139 0.1139 29.81% Castro::construct_ctu_hydro_source() 5 0.1006 0.1006 0.1006 26.32% MLCellLinOp::smooth() 720 0.05838 0.05838 0.05838 15.28% Amr::init() 1 0.04913 0.04913 0.04913 12.86% Amr::restart() 1 0.04913 0.04913 0.04913 12.86% MLCellLinOp::applyBC() 1946 0.04153 0.04153 0.04153 10.87% AmrLevel::restart() 1 0.04141 0.04141 0.04141 10.84% StateData::restartDoit() 4 0.04132 0.04132 0.04132 10.82% VisMF::Read() 3 0.0412 0.0412 0.0412 10.78% MLMG::mgVcycle_bottom 36 0.03471 0.03471 0.03471 9.08% MLMG::actualBottomSolve() 36 0.03469 0.03469 0.03469 9.08% MLCGSolver::bicgstab 36 0.03435 0.03435 0.03435 8.99% Amr::writePlotFile() 1 0.03171 0.03171 0.03171 8.30% VisMF::Write(FabArray) 1 0.02998 0.02998 0.02998 7.85% MLPoisson::Fsmooth() 1440 0.02714 0.02714 0.02714 7.10% Castro::clean_state() 30 0.02608 0.02608 0.02608 6.83% FillPatchIterator::Initialize 20 0.01924 0.01924 0.01924 5.04% FillPatchSingleLevel 20 0.01847 0.01847 0.01847 4.84% StateDataPhysBCFunct::() 20 0.0165 0.0165 0.0165 4.32% MLCellLinOp::apply() 500 0.01568 0.01568 0.01568 4.10% MLMG::mgVcycle_down::0 36 0.01526 0.01526 0.01526 3.99% MLMG::mgVcycle_up::0 36 0.01307 0.01307 0.01307 3.42% StateData::FillBoundary(geom) 160 0.01109 0.01109 0.01109 2.90% Castro::initialize_do_advance() 5 0.01099 0.01099 0.01099 2.88% Castro::computeTemp() 30 0.01093 0.01093 0.01093 2.86% MLPoisson::define() 6 0.009813 0.009813 0.009813 2.57% MultiFab::Dot() 484 0.009412 0.009412 0.009412 2.46% MLCellLinOp::correctionResidual() 216 0.00913 0.00913 0.00913 2.39% MLMG:computeResOfCorrection() 180 0.007881 0.007881 0.007881 2.06% Castro::normalize_species() 30 0.007819 0.007819 0.007819 2.05% MLMG::mgVcycle_down::1 36 0.007613 0.007613 0.007613 1.99% Castro::do_new_sources() 5 0.007433 0.007433 0.007433 1.95% MLMG::mgVcycle_down::2 36 0.007383 0.007383 0.007383 1.93% Gravity::get_new_grav_vector() 5 0.007357 0.007357 0.007357 1.93% Castro::construct_old_gravity() 5 0.007342 0.007342 0.007342 1.92% Gravity::get_old_grav_vector() 5 0.007301 0.007301 0.007301 1.91% MLMG::mgVcycle_down::3 36 0.006994 0.006994 0.006994 1.83% MLCellLinOp::defineAuxData() 6 0.006914 0.006914 0.006914 1.81% FabArray::FillBoundary() 1766 0.006884 0.006884 0.006884 1.80% MLMG::mgVcycle_down::4 36 0.006711 0.006711 0.006711 1.76% FabArray::setVal() 537 0.006686 0.006686 0.006686 1.75% FillBoundary_nowait() 1766 0.006476 0.006476 0.006476 1.70% FabArray::ParallelCopy() 380 0.006412 0.006412 0.006412 1.68% Castro::initialize_advance() 5 0.006325 0.006325 0.006325 1.66% FabArray::ParallelCopy_nowait() 380 0.006291 0.006291 0.006291 1.65% CGSolver::sxay() 690 0.006258 0.006258 0.006258 1.64% Castro::do_old_sources() 5 0.006229 0.006229 0.006229 1.63% MultiFab::LinComb() 690 0.006089 0.006089 0.006089 1.59% Castro::enforce_min_density() 30 0.006058 0.006058 0.006058 1.59% MLMG::mgVcycle_up::2 36 0.005684 0.005684 0.005684 1.49% MLCGSolver::ParallelAllReduce 659 0.005655 0.005655 0.005655 1.48% MLMG::mgVcycle_up::1 36 0.005604 0.005604 0.005604 1.47% MLMG::addInterpCorrection() 180 0.00553 0.00553 0.00553 1.45% MLMG::mgVcycle_up::3 36 0.005394 0.005394 0.005394 1.41% MLMG::mgVcycle_up::4 36 0.005381 0.005381 0.005381 1.41% Castro::expand_state() 5 0.005278 0.005278 0.005278 1.38% amrex::average_down 180 0.00519 0.00519 0.00519 1.36% Gravity::fill_multipole_BCs() 6 0.005046 0.005046 0.005046 1.32% MLPoisson::Fapply() 500 0.005 0.005 0.005 1.31% Castro::post_restart() 1 0.003898 0.003898 0.003898 1.02% Gravity::multilevel_solve_for_new_phi() 1 0.003772 0.003772 0.003772 0.99% Gravity::actual_multilevel_solve() 1 0.003754 0.003754 0.003754 0.98% Castro::post_timestep() 5 0.003689 0.003689 0.003689 0.97% MLCellLinOp::solutionResidual() 42 0.003187 0.003187 0.003187 0.83% MLMG::prepareForSolve() 6 0.002891 0.002891 0.002891 0.76% MultiFab::Xpay() 258 0.002829 0.002829 0.002829 0.74% MLCellLinOp::defineBC() 6 0.002723 0.002723 0.002723 0.71% MLMG::computeResidual() 36 0.002644 0.002644 0.002644 0.69% Castro::reset_internal_energy(MultiFab) 30 0.002594 0.002594 0.002594 0.68% BndryData::define() 6 0.002574 0.002574 0.002574 0.67% Castro::estTimeStep() 10 0.002166 0.002166 0.002166 0.57% Castro::construct_new_source() 25 0.001886 0.001886 0.001886 0.49% Castro::construct_new_gravity_source() 5 0.00173 0.00173 0.00173 0.45% Castro::construct_old_source() 25 0.001544 0.001544 0.001544 0.40% Castro::construct_old_gravity_source() 5 0.001471 0.001471 0.001471 0.39% Castro::computeNewDt() 5 0.00124 0.00124 0.00124 0.32% Castro::enforce_speed_limit() 30 0.001239 0.001239 0.001239 0.32% Castro::apply_source_to_state() 10 0.0009293 0.0009293 0.0009293 0.24% MultiFab::Saxpy() 10 0.0009234 0.0009234 0.0009234 0.24% MLMG::ResNormInf() 42 0.000901 0.000901 0.000901 0.24% Castro::reset_internal_energy(Fab) 240 0.0008926 0.0008926 0.0008926 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008742 0.0008742 0.0008742 0.23% MLCellLinOp::setLevelBC() 6 0.000806 0.000806 0.000806 0.21% MLMG::getGradSolution() 6 0.0007693 0.0007693 0.0007693 0.20% MLCellLinOp::compGrad() 6 0.0007656 0.0007656 0.0007656 0.20% FabArrayBase::getCPC() 632 0.0007417 0.0007417 0.0007417 0.19% MultiFab::Add() 36 0.0007015 0.0007015 0.0007015 0.18% FabArray::mult() 22 0.0006464 0.0006464 0.0006464 0.17% MLPoisson::prepareForSolve() 6 0.0006389 0.0006389 0.0006389 0.17% MLCellLinOp::prepareForSolve() 6 0.0006341 0.0006341 0.0006341 0.17% FabArray::setDomainBndry() 20 0.0006287 0.0006287 0.0006287 0.16% Castro::check_for_nan() 10 0.0005948 0.0005948 0.0005948 0.16% MultiFab::contains_nan() 10 0.0005883 0.0005883 0.0005883 0.15% MLMG::computeMLResidual() 6 0.000566 0.000566 0.000566 0.15% Gravity::update_max_rhs() 6 0.0004441 0.0004441 0.0004441 0.12% Amr::InitAmr() 1 0.0003981 0.0003981 0.0003981 0.10% FabArrayBase::CPC::define() 244 0.0003867 0.0003867 0.0003867 0.10% FabArrayBase::getFB() 1766 0.0003148 0.0003148 0.0003148 0.08% Gravity::swapTimeLevels() 5 0.000227 0.000227 0.000227 0.06% Castro::create_source_corrector() 5 0.0001904 0.0001904 0.0001904 0.05% MultiFab::Copy() 6 0.0001755 0.0001755 0.0001755 0.05% MLLinOp::define() 6 0.0001602 0.0001602 0.0001602 0.04% Castro::buildMetrics() 1 0.0001484 0.0001484 0.0001484 0.04% MLMG::MLResNormInf() 6 0.0001459 0.0001459 0.0001459 0.04% MLLinOp::defineGrids() 6 0.0001379 0.0001379 0.0001379 0.04% MultiFab::max() 6 0.000137 0.000137 0.000137 0.04% MLMG::MLRhsNormInf() 6 0.0001122 0.0001122 0.0001122 0.03% Castro::finalize_advance() 5 7.264e-05 7.264e-05 7.264e-05 0.02% FabArrayBase::FB::FB() 26 5.94e-05 5.94e-05 5.94e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.856e-05 2.856e-05 2.856e-05 0.01% Castro::initMFs() 1 2.793e-05 2.793e-05 2.793e-05 0.01% Castro::swap_state_time_levels() 5 2.779e-05 2.779e-05 2.779e-05 0.01% Amr::writeSmallPlotFile() 1 2.639e-05 2.639e-05 2.639e-05 0.01% makeSFC 30 2.137e-05 2.137e-05 2.137e-05 0.01% Castro::finalize_do_advance() 5 1.776e-05 1.776e-05 1.776e-05 0.00% DistributionMapping::Distribute() 31 8.853e-06 8.853e-06 8.853e-06 0.00% Amr::initSubcycle() 1 8.8e-06 8.8e-06 8.8e-06 0.00% AmrLevel::AmrLevel() 1 5.2e-06 5.2e-06 5.2e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.946e-06 4.946e-06 4.946e-06 0.00% Gravity::set_mass_offset() 6 2.393e-06 2.393e-06 2.393e-06 0.00% Castro::retry_advance_ctu() 5 2.058e-06 2.058e-06 2.058e-06 0.00% Castro::FluxRegCrseInit 5 1.98e-06 1.98e-06 1.98e-06 0.00% Castro::FluxRegFineAdd() 5 1.264e-06 1.264e-06 1.264e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.056e-06 1.056e-06 1.056e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-4-ge4ab0485621d) finalized