Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-18-g9502b99cd98c) initialized Starting run at 08:40:59 UTC on 2022-10-19. Successfully read inputs file ... Castro git describe: 22.09-1-g65b273ad0 AMReX git describe: 22.10-18-g9502b99cd Microphysics git describe: 22.10-4-g1dbcf8c2 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.05096225 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.029399348 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.050294379 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.054501381 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.062594321 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.060572953 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.073937394 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.047283995 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.062608341 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.057177643 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.059318067 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.061233914 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062347673 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.047299221 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029038779 seconds Ending run at 08:41:00 UTC on 2022-10-19. Run time = 0.861681171 Run time without initialization = 0.728817991 Average number of zones advanced per microsecond: 3.597 Average number of zones advanced per microsecond per rank: 3.597 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8617 ... 0.8617 ... 0.8617 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2136 0.2136 0.2136 24.78% VisMF::Write(FabArray) 11 0.1961 0.1961 0.1961 22.76% MLCellLinOp::applyBC() 4433 0.07909 0.07909 0.07909 9.18% MLPoisson::Fsmooth() 3280 0.06351 0.06351 0.06351 7.37% StateDataPhysBCFunct::() 41 0.02463 0.02463 0.02463 2.86% MLCGSolver::bicgstab 82 0.02335 0.02335 0.02335 2.71% MultiFab::Dot() 1114 0.02185 0.02185 0.02185 2.54% StateData::FillBoundary(geom) 328 0.02167 0.02167 0.02167 2.52% Castro::normalize_species() 62 0.01589 0.01589 0.01589 1.84% Castro::computeTemp() 63 0.01424 0.01424 0.01424 1.65% MultiFab::LinComb() 1586 0.01406 0.01406 0.01406 1.63% FabArray::setVal() 1144 0.01401 0.01401 0.01401 1.63% FillBoundary_nowait() 4023 0.01396 0.01396 0.01396 1.62% FabArray::ParallelCopy_nowait() 861 0.0131 0.0131 0.0131 1.52% MLPoisson::Fapply() 1142 0.01151 0.01151 0.01151 1.34% MLCellLinOp::defineAuxData() 11 0.01111 0.01111 0.01111 1.29% Castro::enforce_min_density() 62 0.009218 0.009218 0.009218 1.07% Gravity::fill_multipole_BCs() 11 0.008358 0.008358 0.008358 0.97% MLMG::addInterpCorrection() 410 0.007684 0.007684 0.007684 0.89% amrex::average_down 410 0.006788 0.006788 0.006788 0.79% MultiFab::Xpay() 585 0.006467 0.006467 0.006467 0.75% Castro::estTimeStep() 21 0.005523 0.005523 0.005523 0.64% Amr::checkPoint() 3 0.005111 0.005111 0.005111 0.59% Castro::do_advance_ctu() 10 0.004651 0.004651 0.004651 0.54% Castro::reset_internal_energy(MultiFab) 63 0.004008 0.004008 0.004008 0.47% BndryData::define() 11 0.003702 0.003702 0.003702 0.43% Castro::construct_new_gravity_source() 10 0.003264 0.003264 0.003264 0.38% Amr::writePlotFile() 2 0.002803 0.002803 0.002803 0.33% Castro::construct_old_gravity_source() 10 0.002666 0.002666 0.002666 0.31% MLMG::ResNormInf() 93 0.002003 0.002003 0.002003 0.23% Gravity::get_new_grav_vector() 11 0.001913 0.001913 0.001913 0.22% MultiFab::Saxpy() 20 0.001803 0.001803 0.001803 0.21% Gravity::get_old_grav_vector() 10 0.001745 0.001745 0.001745 0.20% Castro::expand_state() 10 0.001727 0.001727 0.001727 0.20% MultiFab::Add() 82 0.001659 0.001659 0.001659 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001624 0.001624 0.001624 0.19% Castro::reset_internal_energy(Fab) 504 0.001566 0.001566 0.001566 0.18% MLCellLinOp::setLevelBC() 11 0.001513 0.001513 0.001513 0.18% Gravity::actual_solve_with_mlmg() 11 0.001398 0.001398 0.001398 0.16% Castro::enforce_speed_limit() 62 0.001356 0.001356 0.001356 0.16% FabArray::mult() 43 0.00131 0.00131 0.00131 0.15% FabArray::setDomainBndry() 41 0.00128 0.00128 0.00128 0.15% Castro::initData() 1 0.001257 0.001257 0.001257 0.15% MLMG::prepareForSolve() 11 0.001212 0.001212 0.001212 0.14% MultiFab::contains_nan() 20 0.001176 0.001176 0.001176 0.14% MLCellLinOp::prepareForSolve() 11 0.001148 0.001148 0.001148 0.13% MLCellLinOp::smooth() 1640 0.001009 0.001009 0.001009 0.12% MLCellLinOp::compGrad() 11 0.0009051 0.0009051 0.0009051 0.11% FabArray::FillBoundary() 4023 0.0008348 0.0008348 0.0008348 0.10% Castro::create_source_corrector() 10 0.0008002 0.0008002 0.0008002 0.09% FabArrayBase::getCPC() 1323 0.0007806 0.0007806 0.0007806 0.09% FabArrayBase::CPC::define() 454 0.0006509 0.0006509 0.0006509 0.08% FabArrayBase::getFB() 4023 0.0006029 0.0006029 0.0006029 0.07% Amr::InitAmr() 1 0.0004794 0.0004794 0.0004794 0.06% Gravity::solve_for_phi() 10 0.0004154 0.0004154 0.0004154 0.05% MLCellLinOp::apply() 1142 0.0004073 0.0004073 0.0004073 0.05% Gravity::update_max_rhs() 11 0.0004034 0.0004034 0.0004034 0.05% CGSolver::sxay() 1586 0.0003388 0.0003388 0.0003388 0.04% Amr::coarseTimeStep() 10 0.0003262 0.0003262 0.0003262 0.04% MultiFab::Copy() 11 0.0003193 0.0003193 0.0003193 0.04% FillPatchIterator::Initialize 41 0.000285 0.000285 0.000285 0.03% MLCGSolver::ParallelAllReduce 1514 0.0002797 0.0002797 0.0002797 0.03% MLCellLinOp::defineBC() 11 0.0002707 0.0002707 0.0002707 0.03% FabArray::ParallelCopy() 861 0.0002597 0.0002597 0.0002597 0.03% MultiFab::max() 11 0.000254 0.000254 0.000254 0.03% main() 1 0.0002494 0.0002494 0.0002494 0.03% MLCellLinOp::correctionResidual() 492 0.0002286 0.0002286 0.0002286 0.03% MLMG::MLRhsNormInf() 11 0.0002153 0.0002153 0.0002153 0.02% MLMG::mgVcycle() 82 0.0002113 0.0002113 0.0002113 0.02% Castro::construct_new_gravity() 10 0.0002064 0.0002064 0.0002064 0.02% MLLinOp::defineGrids() 11 0.000174 0.000174 0.000174 0.02% MLMG:computeResOfCorrection() 410 0.0001582 0.0001582 0.0001582 0.02% Castro::subcycle_advance_ctu() 10 0.0001573 0.0001573 0.0001573 0.02% Amr::timeStep() 10 0.0001558 0.0001558 0.0001558 0.02% Castro::finalize_advance() 10 0.000145 0.000145 0.000145 0.02% StateData::checkPoint() 12 0.0001307 0.0001307 0.0001307 0.02% MLMG::mgVcycle_down::0 82 0.0001217 0.0001217 0.0001217 0.01% MLMG::mgVcycle_down::1 82 9.986e-05 9.986e-05 9.986e-05 0.01% MLMG::mgVcycle_down::2 82 9.589e-05 9.589e-05 9.589e-05 0.01% MLMG::mgVcycle_down::3 82 9.338e-05 9.338e-05 9.338e-05 0.01% MLMG::actualBottomSolve() 82 8.973e-05 8.973e-05 8.973e-05 0.01% Castro::Castro() 1 8.859e-05 8.859e-05 8.859e-05 0.01% MLMG::mgVcycle_down::4 82 8.636e-05 8.636e-05 8.636e-05 0.01% Castro::advance() 10 8.555e-05 8.555e-05 8.555e-05 0.01% FabArrayBase::FB::FB() 56 8.22e-05 8.22e-05 8.22e-05 0.01% Castro::initialize_advance() 10 8.169e-05 8.169e-05 8.169e-05 0.01% Castro::clean_state() 62 8.047e-05 8.047e-05 8.047e-05 0.01% AmrLevel::checkPoint() 3 7.426e-05 7.426e-05 7.426e-05 0.01% MLMG::mgVcycle_up::4 82 7.212e-05 7.212e-05 7.212e-05 0.01% MLMG::mgVcycle_up::0 82 6.892e-05 6.892e-05 6.892e-05 0.01% MLMG::solve() 11 6.869e-05 6.869e-05 6.869e-05 0.01% MLMG::oneIter() 82 6.583e-05 6.583e-05 6.583e-05 0.01% Castro::initialize_do_advance() 10 6.235e-05 6.235e-05 6.235e-05 0.01% MLMG::mgVcycle_up::3 82 6.046e-05 6.046e-05 6.046e-05 0.01% MLMG::mgVcycle_up::1 82 5.89e-05 5.89e-05 5.89e-05 0.01% MLMG::mgVcycle_up::2 82 5.866e-05 5.866e-05 5.866e-05 0.01% Gravity::solve_phi_with_mlmg() 11 5.692e-05 5.692e-05 5.692e-05 0.01% MLCellLinOp::solutionResidual() 93 4.994e-05 4.994e-05 4.994e-05 0.01% StateData::define() 4 4.282e-05 4.282e-05 4.282e-05 0.00% MLMG::computeResidual() 82 4.2e-05 4.2e-05 4.2e-05 0.00% Castro::swap_state_time_levels() 10 3.908e-05 3.908e-05 3.908e-05 0.00% Castro::finalize_do_advance() 10 3.448e-05 3.448e-05 3.448e-05 0.00% Castro::enforce_consistent_e() 1 3.299e-05 3.299e-05 3.299e-05 0.00% MLMG::mgVcycle_bottom 82 3.23e-05 3.23e-05 3.23e-05 0.00% Gravity::actual_multilevel_solve() 1 3.13e-05 3.13e-05 3.13e-05 0.00% MLPoisson::define() 11 2.853e-05 2.853e-05 2.853e-05 0.00% Castro::initMFs() 1 2.817e-05 2.817e-05 2.817e-05 0.00% FillPatchSingleLevel 41 2.765e-05 2.765e-05 2.765e-05 0.00% makeSFC 55 2.722e-05 2.722e-05 2.722e-05 0.00% Amr::writeSmallPlotFile() 1 2.595e-05 2.595e-05 2.595e-05 0.00% Castro::buildMetrics() 1 2.407e-05 2.407e-05 2.407e-05 0.00% MLLinOp::define() 11 2.195e-05 2.195e-05 2.195e-05 0.00% Amr::FinalizeInit() 1 2.093e-05 2.093e-05 2.093e-05 0.00% Amr::defBaseLevel() 1 1.93e-05 1.93e-05 1.93e-05 0.00% Castro::construct_old_source() 50 1.913e-05 1.913e-05 1.913e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.805e-05 1.805e-05 1.805e-05 0.00% Castro::do_new_sources() 10 1.736e-05 1.736e-05 1.736e-05 0.00% Castro::construct_new_source() 50 1.68e-05 1.68e-05 1.68e-05 0.00% Castro::do_old_sources() 10 1.678e-05 1.678e-05 1.678e-05 0.00% DistributionMapping::Distribute() 56 1.419e-05 1.419e-05 1.419e-05 0.00% Castro::check_for_nan() 20 1.4e-05 1.4e-05 1.4e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.289e-05 1.289e-05 1.289e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.192e-05 1.192e-05 1.192e-05 0.00% Castro::apply_source_to_state() 20 1.111e-05 1.111e-05 1.111e-05 0.00% Castro::construct_old_gravity() 10 1.09e-05 1.09e-05 1.09e-05 0.00% MLMG::computeMLResidual() 11 1.052e-05 1.052e-05 1.052e-05 0.00% Amr::initSubcycle() 1 9.055e-06 9.055e-06 9.055e-06 0.00% Castro::post_timestep() 10 8.981e-06 8.981e-06 8.981e-06 0.00% Gravity::swapTimeLevels() 10 8.788e-06 8.788e-06 8.788e-06 0.00% MLPoisson::prepareForSolve() 11 8.293e-06 8.293e-06 8.293e-06 0.00% Castro::computeNewDt() 9 7.291e-06 7.291e-06 7.291e-06 0.00% MLMG::getGradSolution() 11 7.076e-06 7.076e-06 7.076e-06 0.00% AmrLevel::checkPointPost() 3 5.628e-06 5.628e-06 5.628e-06 0.00% Amr::InitializeInit() 1 4.905e-06 4.905e-06 4.905e-06 0.00% Gravity::set_mass_offset() 11 4.622e-06 4.622e-06 4.622e-06 0.00% Castro::retry_advance_ctu() 10 3.93e-06 3.93e-06 3.93e-06 0.00% Castro::post_init() 1 3.431e-06 3.431e-06 3.431e-06 0.00% MLMG::MLResNormInf() 11 3.217e-06 3.217e-06 3.217e-06 0.00% Castro::FluxRegCrseInit 10 3.126e-06 3.126e-06 3.126e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.744e-06 2.744e-06 2.744e-06 0.00% Amr::init() 1 2.58e-06 2.58e-06 2.58e-06 0.00% Castro::FluxRegFineAdd() 10 2.423e-06 2.423e-06 2.423e-06 0.00% Castro::computeInitialDt() 2 2.368e-06 2.368e-06 2.368e-06 0.00% AmrLevel::checkPointPre() 3 2.017e-06 2.017e-06 2.017e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.703e-06 1.703e-06 1.703e-06 0.00% Castro::post_regrid() 1 1.514e-06 1.514e-06 1.514e-06 0.00% Amr::initialInit() 1 1.279e-06 1.279e-06 1.279e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8617 0.8617 0.8617 100.00% Amr::coarseTimeStep() 10 0.6996 0.6996 0.6996 81.18% Amr::timeStep() 10 0.6017 0.6017 0.6017 69.83% Castro::advance() 10 0.5946 0.5946 0.5946 69.00% Castro::subcycle_advance_ctu() 10 0.5833 0.5833 0.5833 67.69% Castro::do_advance_ctu() 10 0.5832 0.5832 0.5832 67.68% Gravity::solve_phi_with_mlmg() 11 0.3097 0.3097 0.3097 35.94% Gravity::actual_solve_with_mlmg() 11 0.3011 0.3011 0.3011 34.94% Castro::construct_new_gravity() 10 0.2846 0.2846 0.2846 33.03% MLMG::solve() 11 0.2791 0.2791 0.2791 32.39% Gravity::solve_for_phi() 10 0.2665 0.2665 0.2665 30.93% MLMG::oneIter() 82 0.2645 0.2645 0.2645 30.69% MLMG::mgVcycle() 82 0.2627 0.2627 0.2627 30.49% Castro::construct_ctu_hydro_source() 10 0.2136 0.2136 0.2136 24.78% VisMF::Write(FabArray) 11 0.1961 0.1961 0.1961 22.76% Amr::checkPoint() 3 0.1457 0.1457 0.1457 16.91% AmrLevel::checkPoint() 3 0.1406 0.1406 0.1406 16.31% StateData::checkPoint() 12 0.1405 0.1405 0.1405 16.30% MLCellLinOp::smooth() 1640 0.1348 0.1348 0.1348 15.65% Amr::init() 1 0.1323 0.1323 0.1323 15.35% MLCellLinOp::applyBC() 4433 0.09457 0.09457 0.09457 10.98% MLMG::mgVcycle_bottom 82 0.08012 0.08012 0.08012 9.30% MLMG::actualBottomSolve() 82 0.08009 0.08009 0.08009 9.29% MLCGSolver::bicgstab 82 0.07928 0.07928 0.07928 9.20% MLPoisson::Fsmooth() 3280 0.06351 0.06351 0.06351 7.37% Amr::writePlotFile() 2 0.05856 0.05856 0.05856 6.80% FillPatchIterator::Initialize 41 0.05192 0.05192 0.05192 6.03% Amr::initialInit() 1 0.05179 0.05179 0.05179 6.01% FillPatchSingleLevel 41 0.05036 0.05036 0.05036 5.84% Amr::FinalizeInit() 1 0.04759 0.04759 0.04759 5.52% StateDataPhysBCFunct::() 41 0.04631 0.04631 0.04631 5.37% Castro::post_init() 1 0.04618 0.04618 0.04618 5.36% Castro::clean_state() 62 0.04558 0.04558 0.04558 5.29% Gravity::multilevel_solve_for_new_phi() 1 0.04364 0.04364 0.04364 5.06% Gravity::actual_multilevel_solve() 1 0.04362 0.04362 0.04362 5.06% MLCellLinOp::apply() 1142 0.03568 0.03568 0.03568 4.14% MLMG::mgVcycle_down::0 82 0.03502 0.03502 0.03502 4.06% MLMG::mgVcycle_up::0 82 0.0301 0.0301 0.0301 3.49% MultiFab::Dot() 1114 0.02185 0.02185 0.02185 2.54% StateData::FillBoundary(geom) 328 0.02167 0.02167 0.02167 2.52% MLCellLinOp::correctionResidual() 492 0.02095 0.02095 0.02095 2.43% Castro::initialize_do_advance() 10 0.02068 0.02068 0.02068 2.40% Gravity::get_new_grav_vector() 11 0.02039 0.02039 0.02039 2.37% Castro::computeTemp() 63 0.01982 0.01982 0.01982 2.30% Castro::construct_old_gravity() 10 0.01844 0.01844 0.01844 2.14% Gravity::get_old_grav_vector() 10 0.01843 0.01843 0.01843 2.14% MLMG:computeResOfCorrection() 410 0.01812 0.01812 0.01812 2.10% MLPoisson::define() 11 0.0176 0.0176 0.0176 2.04% MLMG::mgVcycle_down::1 82 0.01758 0.01758 0.01758 2.04% MLMG::mgVcycle_down::2 82 0.01707 0.01707 0.01707 1.98% MLMG::mgVcycle_down::3 82 0.01622 0.01622 0.01622 1.88% Castro::normalize_species() 62 0.01589 0.01589 0.01589 1.84% FabArray::FillBoundary() 4023 0.01548 0.01548 0.01548 1.80% MLMG::mgVcycle_down::4 82 0.01541 0.01541 0.01541 1.79% FillBoundary_nowait() 4023 0.01464 0.01464 0.01464 1.70% CGSolver::sxay() 1586 0.01439 0.01439 0.01439 1.67% FabArray::ParallelCopy() 861 0.01417 0.01417 0.01417 1.64% MultiFab::LinComb() 1586 0.01406 0.01406 0.01406 1.63% FabArray::setVal() 1144 0.01401 0.01401 0.01401 1.63% FabArray::ParallelCopy_nowait() 861 0.01391 0.01391 0.01391 1.61% MLMG::mgVcycle_up::2 82 0.01324 0.01324 0.01324 1.54% MLCGSolver::ParallelAllReduce 1514 0.01301 0.01301 0.01301 1.51% MLMG::mgVcycle_up::1 82 0.01297 0.01297 0.01297 1.50% Castro::expand_state() 10 0.01292 0.01292 0.01292 1.50% MLMG::addInterpCorrection() 410 0.01276 0.01276 0.01276 1.48% MLMG::mgVcycle_up::3 82 0.0125 0.0125 0.0125 1.45% MLCellLinOp::defineAuxData() 11 0.01243 0.01243 0.01243 1.44% MLMG::mgVcycle_up::4 82 0.01231 0.01231 0.01231 1.43% Castro::do_old_sources() 10 0.01226 0.01226 0.01226 1.42% amrex::average_down 410 0.01187 0.01187 0.01187 1.38% Castro::do_new_sources() 10 0.01182 0.01182 0.01182 1.37% MLPoisson::Fapply() 1142 0.01151 0.01151 0.01151 1.34% Castro::initialize_advance() 10 0.011 0.011 0.011 1.28% Castro::enforce_min_density() 62 0.009218 0.009218 0.009218 1.07% Gravity::fill_multipole_BCs() 11 0.008358 0.008358 0.008358 0.97% MLCellLinOp::solutionResidual() 93 0.007044 0.007044 0.007044 0.82% Castro::post_timestep() 10 0.007033 0.007033 0.007033 0.82% MultiFab::Xpay() 585 0.006467 0.006467 0.006467 0.75% MLMG::computeResidual() 82 0.006085 0.006085 0.006085 0.71% Castro::reset_internal_energy(MultiFab) 63 0.005575 0.005575 0.005575 0.65% Castro::estTimeStep() 21 0.005523 0.005523 0.005523 0.64% MLMG::prepareForSolve() 11 0.005279 0.005279 0.005279 0.61% MLCellLinOp::defineBC() 11 0.004898 0.004898 0.004898 0.57% BndryData::define() 11 0.004627 0.004627 0.004627 0.54% Amr::InitializeInit() 1 0.004202 0.004202 0.004202 0.49% Amr::defBaseLevel() 1 0.004197 0.004197 0.004197 0.49% Castro::initData() 1 0.003679 0.003679 0.003679 0.43% Castro::construct_new_source() 50 0.003281 0.003281 0.003281 0.38% Castro::construct_new_gravity_source() 10 0.003264 0.003264 0.003264 0.38% Castro::construct_old_source() 50 0.002685 0.002685 0.002685 0.31% Castro::construct_old_gravity_source() 10 0.002666 0.002666 0.002666 0.31% Castro::computeNewDt() 9 0.002309 0.002309 0.002309 0.27% MLMG::ResNormInf() 93 0.002003 0.002003 0.002003 0.23% Castro::apply_source_to_state() 20 0.001815 0.001815 0.001815 0.21% MultiFab::Saxpy() 20 0.001803 0.001803 0.001803 0.21% MultiFab::Add() 82 0.001659 0.001659 0.001659 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001624 0.001624 0.001624 0.19% Castro::reset_internal_energy(Fab) 504 0.001566 0.001566 0.001566 0.18% MLCellLinOp::setLevelBC() 11 0.001513 0.001513 0.001513 0.18% FabArrayBase::getCPC() 1323 0.001431 0.001431 0.001431 0.17% MLMG::getGradSolution() 11 0.001404 0.001404 0.001404 0.16% MLCellLinOp::compGrad() 11 0.001397 0.001397 0.001397 0.16% Castro::enforce_speed_limit() 62 0.001356 0.001356 0.001356 0.16% FabArray::mult() 43 0.00131 0.00131 0.00131 0.15% FabArray::setDomainBndry() 41 0.00128 0.00128 0.00128 0.15% Castro::check_for_nan() 20 0.00119 0.00119 0.00119 0.14% MultiFab::contains_nan() 20 0.001176 0.001176 0.001176 0.14% MLPoisson::prepareForSolve() 11 0.001157 0.001157 0.001157 0.13% MLCellLinOp::prepareForSolve() 11 0.001148 0.001148 0.001148 0.13% Castro::post_regrid() 1 0.001148 0.001148 0.001148 0.13% MLMG::computeMLResidual() 11 0.001011 0.001011 0.001011 0.12% Gravity::update_max_rhs() 11 0.000802 0.000802 0.000802 0.09% Castro::create_source_corrector() 10 0.0008002 0.0008002 0.0008002 0.09% Castro::computeInitialDt() 2 0.0007511 0.0007511 0.0007511 0.09% FabArrayBase::getFB() 4023 0.0006851 0.0006851 0.0006851 0.08% FabArrayBase::CPC::define() 454 0.0006509 0.0006509 0.0006509 0.08% Amr::InitAmr() 1 0.0004885 0.0004885 0.0004885 0.06% Castro::Castro() 1 0.0004395 0.0004395 0.0004395 0.05% Gravity::swapTimeLevels() 10 0.0004343 0.0004343 0.0004343 0.05% MultiFab::Copy() 11 0.0003193 0.0003193 0.0003193 0.04% MLMG::MLResNormInf() 11 0.0002831 0.0002831 0.0002831 0.03% MultiFab::max() 11 0.000254 0.000254 0.000254 0.03% MLLinOp::define() 11 0.0002506 0.0002506 0.0002506 0.03% MLLinOp::defineGrids() 11 0.0002286 0.0002286 0.0002286 0.03% MLMG::MLRhsNormInf() 11 0.0002153 0.0002153 0.0002153 0.02% Castro::buildMetrics() 1 0.0001603 0.0001603 0.0001603 0.02% Castro::finalize_advance() 10 0.0001505 0.0001505 0.0001505 0.02% FabArrayBase::FB::FB() 56 8.22e-05 8.22e-05 8.22e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.474e-05 5.474e-05 5.474e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.288e-05 5.288e-05 5.288e-05 0.01% StateData::define() 4 4.282e-05 4.282e-05 4.282e-05 0.00% makeSFC 55 3.999e-05 3.999e-05 3.999e-05 0.00% Castro::swap_state_time_levels() 10 3.908e-05 3.908e-05 3.908e-05 0.00% Castro::finalize_do_advance() 10 3.448e-05 3.448e-05 3.448e-05 0.00% Castro::enforce_consistent_e() 1 3.299e-05 3.299e-05 3.299e-05 0.00% Castro::initMFs() 1 2.817e-05 2.817e-05 2.817e-05 0.00% Amr::writeSmallPlotFile() 1 2.595e-05 2.595e-05 2.595e-05 0.00% DistributionMapping::Distribute() 56 1.419e-05 1.419e-05 1.419e-05 0.00% Amr::initSubcycle() 1 9.055e-06 9.055e-06 9.055e-06 0.00% AmrLevel::checkPointPost() 3 5.628e-06 5.628e-06 5.628e-06 0.00% Gravity::set_mass_offset() 11 4.622e-06 4.622e-06 4.622e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.161e-06 4.161e-06 4.161e-06 0.00% Castro::retry_advance_ctu() 10 3.93e-06 3.93e-06 3.93e-06 0.00% Castro::FluxRegCrseInit 10 3.126e-06 3.126e-06 3.126e-06 0.00% Castro::FluxRegFineAdd() 10 2.423e-06 2.423e-06 2.423e-06 0.00% AmrLevel::checkPointPre() 3 2.017e-06 2.017e-06 2.017e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.703e-06 1.703e-06 1.703e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-18-g9502b99cd98c) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-18-g9502b99cd98c) initialized Starting run at 08:41:01 UTC on 2022-10-19. Successfully read inputs file ... Castro git describe: 22.09-1-g65b273ad0 AMReX git describe: 22.10-18-g9502b99cd Microphysics git describe: 22.10-4-g1dbcf8c2 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.482080098 Restart time = 0.047369172 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.054918342 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.061303288 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.06051784 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.062625588 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.07783283 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031264863 seconds Ending run at 08:41:01 UTC on 2022-10-19. Run time = 0.396737607 Run time without initialization = 0.348845775 Average number of zones advanced per microsecond: 3.757 Average number of zones advanced per microsecond per rank: 3.757 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3968 ... 0.3968 ... 0.3968 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1181 0.1181 0.1181 29.76% VisMF::Read() 3 0.03958 0.03958 0.03958 9.98% MLCellLinOp::applyBC() 1946 0.03437 0.03437 0.03437 8.66% VisMF::Write(FabArray) 1 0.02973 0.02973 0.02973 7.49% MLPoisson::Fsmooth() 1440 0.02703 0.02703 0.02703 6.81% StateData::FillBoundary(geom) 160 0.01123 0.01123 0.01123 2.83% MLCGSolver::bicgstab 36 0.01003 0.01003 0.01003 2.53% MultiFab::Dot() 484 0.009341 0.009341 0.009341 2.35% Castro::normalize_species() 30 0.009001 0.009001 0.009001 2.27% Castro::computeTemp() 30 0.008685 0.008685 0.008685 2.19% FabArray::setVal() 537 0.006648 0.006648 0.006648 1.68% FillBoundary_nowait() 1766 0.006141 0.006141 0.006141 1.55% StateDataPhysBCFunct::() 20 0.006112 0.006112 0.006112 1.54% MLCellLinOp::defineAuxData() 6 0.00607 0.00607 0.00607 1.53% MultiFab::LinComb() 690 0.006043 0.006043 0.006043 1.52% FabArray::ParallelCopy_nowait() 380 0.00591 0.00591 0.00591 1.49% MLPoisson::Fapply() 500 0.005 0.005 0.005 1.26% Gravity::fill_multipole_BCs() 6 0.004762 0.004762 0.004762 1.20% Castro::enforce_min_density() 30 0.004739 0.004739 0.004739 1.19% Amr::restart() 1 0.003533 0.003533 0.003533 0.89% MLMG::addInterpCorrection() 180 0.003334 0.003334 0.003334 0.84% Castro::estTimeStep() 10 0.002994 0.002994 0.002994 0.75% amrex::average_down 180 0.00295 0.00295 0.00295 0.74% MultiFab::Xpay() 258 0.002866 0.002866 0.002866 0.72% Castro::do_advance_ctu() 5 0.002271 0.002271 0.002271 0.57% BndryData::define() 6 0.002058 0.002058 0.002058 0.52% Castro::construct_new_gravity_source() 5 0.001777 0.001777 0.001777 0.45% Castro::reset_internal_energy(MultiFab) 30 0.001667 0.001667 0.001667 0.42% Amr::writePlotFile() 1 0.00161 0.00161 0.00161 0.41% Castro::subcycle_advance_ctu() 5 0.001519 0.001519 0.001519 0.38% Castro::construct_old_gravity_source() 5 0.001445 0.001445 0.001445 0.36% Gravity::get_old_grav_vector() 5 0.0009563 0.0009563 0.0009563 0.24% Castro::construct_new_source() 25 0.0009382 0.0009382 0.0009382 0.24% MultiFab::Saxpy() 10 0.0009331 0.0009331 0.0009331 0.24% Castro::reset_internal_energy(Fab) 240 0.0009116 0.0009116 0.0009116 0.23% MLMG::ResNormInf() 42 0.0008894 0.0008894 0.0008894 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008699 0.0008699 0.0008699 0.22% Castro::expand_state() 5 0.0008681 0.0008681 0.0008681 0.22% Gravity::get_new_grav_vector() 5 0.000862 0.000862 0.000862 0.22% MLCellLinOp::setLevelBC() 6 0.0007958 0.0007958 0.0007958 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007462 0.0007462 0.0007462 0.19% MultiFab::Add() 36 0.0007247 0.0007247 0.0007247 0.18% MLMG::prepareForSolve() 6 0.0006544 0.0006544 0.0006544 0.16% FabArray::mult() 22 0.0006468 0.0006468 0.0006468 0.16% FabArray::setDomainBndry() 20 0.0006395 0.0006395 0.0006395 0.16% MLCellLinOp::prepareForSolve() 6 0.0006187 0.0006187 0.0006187 0.16% MultiFab::contains_nan() 10 0.0006054 0.0006054 0.0006054 0.15% MLCellLinOp::compGrad() 6 0.0004821 0.0004821 0.0004821 0.12% Castro::enforce_speed_limit() 30 0.0004782 0.0004782 0.0004782 0.12% MLCellLinOp::smooth() 720 0.0004511 0.0004511 0.0004511 0.11% FabArrayBase::CPC::define() 244 0.0003859 0.0003859 0.0003859 0.10% FabArray::FillBoundary() 1766 0.0003766 0.0003766 0.0003766 0.09% Amr::InitAmr() 1 0.0003759 0.0003759 0.0003759 0.09% FabArrayBase::getCPC() 632 0.0003594 0.0003594 0.0003594 0.09% FabArrayBase::getFB() 1766 0.0002567 0.0002567 0.0002567 0.06% main() 1 0.0002273 0.0002273 0.0002273 0.06% Gravity::update_max_rhs() 6 0.0002239 0.0002239 0.0002239 0.06% Gravity::solve_for_phi() 5 0.0002079 0.0002079 0.0002079 0.05% MLCellLinOp::apply() 500 0.0001872 0.0001872 0.0001872 0.05% MultiFab::Copy() 6 0.0001745 0.0001745 0.0001745 0.04% Castro::construct_new_gravity() 5 0.0001714 0.0001714 0.0001714 0.04% Amr::coarseTimeStep() 5 0.000166 0.000166 0.000166 0.04% CGSolver::sxay() 690 0.0001657 0.0001657 0.0001657 0.04% MLCellLinOp::defineBC() 6 0.0001439 0.0001439 0.0001439 0.04% MultiFab::max() 6 0.0001357 0.0001357 0.0001357 0.03% FillPatchIterator::Initialize 20 0.0001341 0.0001341 0.0001341 0.03% FabArray::ParallelCopy() 380 0.0001202 0.0001202 0.0001202 0.03% MLCGSolver::ParallelAllReduce 659 0.0001183 0.0001183 0.0001183 0.03% MLMG::MLRhsNormInf() 6 0.0001132 0.0001132 0.0001132 0.03% MLMG::mgVcycle() 36 0.000102 0.000102 0.000102 0.03% MLCellLinOp::correctionResidual() 216 0.0001003 0.0001003 0.0001003 0.03% Castro::finalize_advance() 5 9.079e-05 9.079e-05 9.079e-05 0.02% Castro::create_source_corrector() 5 8.661e-05 8.661e-05 8.661e-05 0.02% AmrLevel::restart() 1 8.158e-05 8.158e-05 8.158e-05 0.02% Castro::construct_old_source() 25 7.878e-05 7.878e-05 7.878e-05 0.02% Amr::timeStep() 5 7.8e-05 7.8e-05 7.8e-05 0.02% MLLinOp::defineGrids() 6 7.265e-05 7.265e-05 7.265e-05 0.02% MLMG:computeResOfCorrection() 180 7.039e-05 7.039e-05 7.039e-05 0.02% StateData::restartDoit() 4 7.018e-05 7.018e-05 7.018e-05 0.02% Castro::advance() 5 6.745e-05 6.745e-05 6.745e-05 0.02% Castro::initialize_do_advance() 5 5.86e-05 5.86e-05 5.86e-05 0.01% Castro::post_timestep() 5 5.807e-05 5.807e-05 5.807e-05 0.01% MLMG::mgVcycle_down::0 36 5.759e-05 5.759e-05 5.759e-05 0.01% FabArrayBase::FB::FB() 26 5.603e-05 5.603e-05 5.603e-05 0.01% MLMG::mgVcycle_down::1 36 4.629e-05 4.629e-05 4.629e-05 0.01% MLMG::mgVcycle_down::2 36 4.259e-05 4.259e-05 4.259e-05 0.01% Castro::clean_state() 30 4.122e-05 4.122e-05 4.122e-05 0.01% MLMG::actualBottomSolve() 36 4.006e-05 4.006e-05 4.006e-05 0.01% MLMG::mgVcycle_down::4 36 3.954e-05 3.954e-05 3.954e-05 0.01% Castro::initialize_advance() 5 3.932e-05 3.932e-05 3.932e-05 0.01% MLMG::mgVcycle_down::3 36 3.861e-05 3.861e-05 3.861e-05 0.01% MLMG::mgVcycle_up::4 36 3.594e-05 3.594e-05 3.594e-05 0.01% MLMG::solve() 6 3.355e-05 3.355e-05 3.355e-05 0.01% MLMG::mgVcycle_up::0 36 3.09e-05 3.09e-05 3.09e-05 0.01% Castro::buildMetrics() 1 3.083e-05 3.083e-05 3.083e-05 0.01% MLMG::mgVcycle_up::3 36 3.002e-05 3.002e-05 3.002e-05 0.01% Gravity::actual_multilevel_solve() 1 2.974e-05 2.974e-05 2.974e-05 0.01% MLMG::oneIter() 36 2.906e-05 2.906e-05 2.906e-05 0.01% Castro::post_restart() 1 2.834e-05 2.834e-05 2.834e-05 0.01% MLMG::mgVcycle_up::2 36 2.814e-05 2.814e-05 2.814e-05 0.01% MLMG::mgVcycle_up::1 36 2.732e-05 2.732e-05 2.732e-05 0.01% Amr::writeSmallPlotFile() 1 2.713e-05 2.713e-05 2.713e-05 0.01% Castro::swap_state_time_levels() 5 2.64e-05 2.64e-05 2.64e-05 0.01% Castro::initMFs() 1 2.496e-05 2.496e-05 2.496e-05 0.01% MLCellLinOp::solutionResidual() 42 2.292e-05 2.292e-05 2.292e-05 0.01% Castro::computeNewDt() 5 2.099e-05 2.099e-05 2.099e-05 0.01% Castro::construct_old_gravity() 5 2.086e-05 2.086e-05 2.086e-05 0.01% MLPoisson::define() 6 2.06e-05 2.06e-05 2.06e-05 0.01% MLLinOp::define() 6 1.991e-05 1.991e-05 1.991e-05 0.01% MLMG::computeResidual() 36 1.893e-05 1.893e-05 1.893e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.832e-05 1.832e-05 1.832e-05 0.00% Castro::finalize_do_advance() 5 1.695e-05 1.695e-05 1.695e-05 0.00% MLMG::mgVcycle_bottom 36 1.619e-05 1.619e-05 1.619e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.446e-05 1.446e-05 1.446e-05 0.00% makeSFC 30 1.355e-05 1.355e-05 1.355e-05 0.00% FillPatchSingleLevel 20 1.353e-05 1.353e-05 1.353e-05 0.00% Castro::do_new_sources() 5 1.09e-05 1.09e-05 1.09e-05 0.00% Castro::check_for_nan() 10 1.05e-05 1.05e-05 1.05e-05 0.00% DistributionMapping::Distribute() 31 8.574e-06 8.574e-06 8.574e-06 0.00% Amr::initSubcycle() 1 8.455e-06 8.455e-06 8.455e-06 0.00% Castro::do_old_sources() 5 8.106e-06 8.106e-06 8.106e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.9e-06 6.9e-06 6.9e-06 0.00% Castro::apply_source_to_state() 10 6.686e-06 6.686e-06 6.686e-06 0.00% MLMG::computeMLResidual() 6 4.939e-06 4.939e-06 4.939e-06 0.00% MLPoisson::prepareForSolve() 6 4.369e-06 4.369e-06 4.369e-06 0.00% Gravity::swapTimeLevels() 5 4.233e-06 4.233e-06 4.233e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.484e-06 3.484e-06 3.484e-06 0.00% MLMG::getGradSolution() 6 3.25e-06 3.25e-06 3.25e-06 0.00% Castro::retry_advance_ctu() 5 2.839e-06 2.839e-06 2.839e-06 0.00% Gravity::set_mass_offset() 6 2.25e-06 2.25e-06 2.25e-06 0.00% MLMG::MLResNormInf() 6 2.155e-06 2.155e-06 2.155e-06 0.00% Castro::FluxRegCrseInit 5 2.094e-06 2.094e-06 2.094e-06 0.00% Castro::FluxRegFineAdd() 5 1.427e-06 1.427e-06 1.427e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.153e-06 1.153e-06 1.153e-06 0.00% Amr::init() 1 1.058e-06 1.058e-06 1.058e-06 0.00% AmrLevel::AmrLevel() 1 9.4e-07 9.4e-07 9.4e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3967 0.3967 0.3967 100.00% Amr::coarseTimeStep() 5 0.3174 0.3174 0.3174 79.99% Amr::timeStep() 5 0.3155 0.3155 0.3155 79.52% Castro::advance() 5 0.3106 0.3106 0.3106 78.29% Castro::subcycle_advance_ctu() 5 0.3055 0.3055 0.3055 76.99% Castro::do_advance_ctu() 5 0.304 0.304 0.304 76.61% Castro::construct_new_gravity() 5 0.142 0.142 0.142 35.80% Gravity::solve_phi_with_mlmg() 6 0.138 0.138 0.138 34.79% Gravity::solve_for_phi() 5 0.1345 0.1345 0.1345 33.90% Gravity::actual_solve_with_mlmg() 6 0.1331 0.1331 0.1331 33.56% MLMG::solve() 6 0.1212 0.1212 0.1212 30.54% Castro::construct_ctu_hydro_source() 5 0.1181 0.1181 0.1181 29.76% MLMG::oneIter() 36 0.1141 0.1141 0.1141 28.75% MLMG::mgVcycle() 36 0.1133 0.1133 0.1133 28.56% MLCellLinOp::smooth() 720 0.058 0.058 0.058 14.62% Amr::init() 1 0.04741 0.04741 0.04741 11.95% Amr::restart() 1 0.04741 0.04741 0.04741 11.95% MLCellLinOp::applyBC() 1946 0.0412 0.0412 0.0412 10.38% AmrLevel::restart() 1 0.03979 0.03979 0.03979 10.03% StateData::restartDoit() 4 0.03971 0.03971 0.03971 10.01% VisMF::Read() 3 0.03958 0.03958 0.03958 9.98% MLMG::mgVcycle_bottom 36 0.03449 0.03449 0.03449 8.69% MLMG::actualBottomSolve() 36 0.03447 0.03447 0.03447 8.69% MLCGSolver::bicgstab 36 0.03412 0.03412 0.03412 8.60% Amr::writePlotFile() 1 0.03134 0.03134 0.03134 7.90% VisMF::Write(FabArray) 1 0.02973 0.02973 0.02973 7.49% MLPoisson::Fsmooth() 1440 0.02703 0.02703 0.02703 6.81% Castro::clean_state() 30 0.02552 0.02552 0.02552 6.43% FillPatchIterator::Initialize 20 0.02011 0.02011 0.02011 5.07% FillPatchSingleLevel 20 0.01933 0.01933 0.01933 4.87% StateDataPhysBCFunct::() 20 0.01734 0.01734 0.01734 4.37% MLCellLinOp::apply() 500 0.01559 0.01559 0.01559 3.93% MLMG::mgVcycle_down::0 36 0.01522 0.01522 0.01522 3.84% MLMG::mgVcycle_up::0 36 0.01303 0.01303 0.01303 3.28% Castro::computeTemp() 30 0.01126 0.01126 0.01126 2.84% StateData::FillBoundary(geom) 160 0.01123 0.01123 0.01123 2.83% Castro::initialize_do_advance() 5 0.01058 0.01058 0.01058 2.67% MLPoisson::define() 6 0.009659 0.009659 0.009659 2.43% MultiFab::Dot() 484 0.009341 0.009341 0.009341 2.35% MLCellLinOp::correctionResidual() 216 0.009148 0.009148 0.009148 2.31% Castro::normalize_species() 30 0.009001 0.009001 0.009001 2.27% Castro::do_new_sources() 5 0.008054 0.008054 0.008054 2.03% MLMG:computeResOfCorrection() 180 0.007889 0.007889 0.007889 1.99% MLMG::mgVcycle_down::1 36 0.007579 0.007579 0.007579 1.91% Castro::construct_old_gravity() 5 0.00739 0.00739 0.00739 1.86% Gravity::get_new_grav_vector() 5 0.00739 0.00739 0.00739 1.86% Gravity::get_old_grav_vector() 5 0.007369 0.007369 0.007369 1.86% MLMG::mgVcycle_down::2 36 0.007358 0.007358 0.007358 1.85% MLMG::mgVcycle_down::3 36 0.006981 0.006981 0.006981 1.76% FabArray::FillBoundary() 1766 0.006831 0.006831 0.006831 1.72% MLCellLinOp::defineAuxData() 6 0.006791 0.006791 0.006791 1.71% MLMG::mgVcycle_down::4 36 0.006668 0.006668 0.006668 1.68% FabArray::setVal() 537 0.006648 0.006648 0.006648 1.68% Castro::do_old_sources() 5 0.006634 0.006634 0.006634 1.67% FillBoundary_nowait() 1766 0.006454 0.006454 0.006454 1.63% FabArray::ParallelCopy() 380 0.006401 0.006401 0.006401 1.61% FabArray::ParallelCopy_nowait() 380 0.00628 0.00628 0.00628 1.58% CGSolver::sxay() 690 0.006208 0.006208 0.006208 1.56% MultiFab::LinComb() 690 0.006043 0.006043 0.006043 1.52% Castro::expand_state() 5 0.005879 0.005879 0.005879 1.48% MLMG::mgVcycle_up::2 36 0.005664 0.005664 0.005664 1.43% MLCGSolver::ParallelAllReduce 659 0.005597 0.005597 0.005597 1.41% MLMG::mgVcycle_up::1 36 0.005569 0.005569 0.005569 1.40% MLMG::addInterpCorrection() 180 0.00552 0.00552 0.00552 1.39% MLMG::mgVcycle_up::3 36 0.005361 0.005361 0.005361 1.35% MLMG::mgVcycle_up::4 36 0.005304 0.005304 0.005304 1.34% amrex::average_down 180 0.005189 0.005189 0.005189 1.31% MLPoisson::Fapply() 500 0.005 0.005 0.005 1.26% Castro::initialize_advance() 5 0.004994 0.004994 0.004994 1.26% Castro::post_timestep() 5 0.004794 0.004794 0.004794 1.21% Gravity::fill_multipole_BCs() 6 0.004762 0.004762 0.004762 1.20% Castro::enforce_min_density() 30 0.004739 0.004739 0.004739 1.19% Castro::post_restart() 1 0.003909 0.003909 0.003909 0.99% Gravity::multilevel_solve_for_new_phi() 1 0.003789 0.003789 0.003789 0.95% Gravity::actual_multilevel_solve() 1 0.003771 0.003771 0.003771 0.95% MLCellLinOp::solutionResidual() 42 0.003195 0.003195 0.003195 0.81% Castro::estTimeStep() 10 0.002994 0.002994 0.002994 0.75% MultiFab::Xpay() 258 0.002866 0.002866 0.002866 0.72% MLMG::prepareForSolve() 6 0.002856 0.002856 0.002856 0.72% MLCellLinOp::defineBC() 6 0.002726 0.002726 0.002726 0.69% Castro::construct_new_source() 25 0.002715 0.002715 0.002715 0.68% MLMG::computeResidual() 36 0.002655 0.002655 0.002655 0.67% BndryData::define() 6 0.002582 0.002582 0.002582 0.65% Castro::reset_internal_energy(MultiFab) 30 0.002579 0.002579 0.002579 0.65% Castro::construct_new_gravity_source() 5 0.001777 0.001777 0.001777 0.45% Castro::computeNewDt() 5 0.001679 0.001679 0.001679 0.42% Castro::construct_old_source() 25 0.001523 0.001523 0.001523 0.38% Castro::construct_old_gravity_source() 5 0.001445 0.001445 0.001445 0.36% Castro::apply_source_to_state() 10 0.0009398 0.0009398 0.0009398 0.24% MultiFab::Saxpy() 10 0.0009331 0.0009331 0.0009331 0.24% Castro::reset_internal_energy(Fab) 240 0.0009116 0.0009116 0.0009116 0.23% MLMG::ResNormInf() 42 0.0008894 0.0008894 0.0008894 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008699 0.0008699 0.0008699 0.22% MLCellLinOp::setLevelBC() 6 0.0007958 0.0007958 0.0007958 0.20% MLMG::getGradSolution() 6 0.0007539 0.0007539 0.0007539 0.19% MLCellLinOp::compGrad() 6 0.0007507 0.0007507 0.0007507 0.19% FabArrayBase::getCPC() 632 0.0007453 0.0007453 0.0007453 0.19% MultiFab::Add() 36 0.0007247 0.0007247 0.0007247 0.18% FabArray::mult() 22 0.0006468 0.0006468 0.0006468 0.16% FabArray::setDomainBndry() 20 0.0006395 0.0006395 0.0006395 0.16% MLPoisson::prepareForSolve() 6 0.0006231 0.0006231 0.0006231 0.16% MLCellLinOp::prepareForSolve() 6 0.0006187 0.0006187 0.0006187 0.16% Castro::check_for_nan() 10 0.0006159 0.0006159 0.0006159 0.16% MultiFab::contains_nan() 10 0.0006054 0.0006054 0.0006054 0.15% MLMG::computeMLResidual() 6 0.0005638 0.0005638 0.0005638 0.14% Castro::enforce_speed_limit() 30 0.0004782 0.0004782 0.0004782 0.12% Gravity::update_max_rhs() 6 0.0004375 0.0004375 0.0004375 0.11% FabArrayBase::CPC::define() 244 0.0003859 0.0003859 0.0003859 0.10% Amr::InitAmr() 1 0.0003843 0.0003843 0.0003843 0.10% FabArrayBase::getFB() 1766 0.0003127 0.0003127 0.0003127 0.08% Gravity::swapTimeLevels() 5 0.0002245 0.0002245 0.0002245 0.06% MultiFab::Copy() 6 0.0001745 0.0001745 0.0001745 0.04% Castro::buildMetrics() 1 0.0001481 0.0001481 0.0001481 0.04% MLMG::MLResNormInf() 6 0.0001477 0.0001477 0.0001477 0.04% MultiFab::max() 6 0.0001357 0.0001357 0.0001357 0.03% MLLinOp::define() 6 0.0001216 0.0001216 0.0001216 0.03% MLMG::MLRhsNormInf() 6 0.0001132 0.0001132 0.0001132 0.03% MLLinOp::defineGrids() 6 0.0001016 0.0001016 0.0001016 0.03% Castro::finalize_advance() 5 9.432e-05 9.432e-05 9.432e-05 0.02% Castro::create_source_corrector() 5 8.661e-05 8.661e-05 8.661e-05 0.02% FabArrayBase::FB::FB() 26 5.603e-05 5.603e-05 5.603e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.784e-05 2.784e-05 2.784e-05 0.01% Amr::writeSmallPlotFile() 1 2.713e-05 2.713e-05 2.713e-05 0.01% Castro::swap_state_time_levels() 5 2.64e-05 2.64e-05 2.64e-05 0.01% Castro::initMFs() 1 2.496e-05 2.496e-05 2.496e-05 0.01% makeSFC 30 2.094e-05 2.094e-05 2.094e-05 0.01% Castro::finalize_do_advance() 5 1.695e-05 1.695e-05 1.695e-05 0.00% DistributionMapping::Distribute() 31 8.574e-06 8.574e-06 8.574e-06 0.00% Amr::initSubcycle() 1 8.455e-06 8.455e-06 8.455e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.665e-06 4.665e-06 4.665e-06 0.00% Castro::retry_advance_ctu() 5 2.839e-06 2.839e-06 2.839e-06 0.00% Gravity::set_mass_offset() 6 2.25e-06 2.25e-06 2.25e-06 0.00% Castro::FluxRegCrseInit 5 2.094e-06 2.094e-06 2.094e-06 0.00% Castro::FluxRegFineAdd() 5 1.427e-06 1.427e-06 1.427e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.153e-06 1.153e-06 1.153e-06 0.00% AmrLevel::AmrLevel() 1 9.4e-07 9.4e-07 9.4e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-18-g9502b99cd98c) finalized