Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.06-33-g6f72de283c38) initialized Starting run at 08:29:11 UTC on 2022-06-16. Successfully read inputs file ... Castro git describe: 22.06-12-g556652b03 AMReX git describe: 22.06-33-g6f72de283 Microphysics git describe: 22.06-2-g35a553f4 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.043885718 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025235908 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048121476 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050727917 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.050682607 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.050867171 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.06535928 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.045405964 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.080882722 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.058356381 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.053920111 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.064851392 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.068532418 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.042389941 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024639582 seconds Ending run at 08:29:12 UTC on 2022-06-16. Run time = 0.827715909 Run time without initialization = 0.705407973 Average number of zones advanced per microsecond: 3.716 Average number of zones advanced per microsecond per rank: 3.716 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8278 ... 0.8278 ... 0.8278 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2093 0.2093 0.2093 25.29% VisMF::Write(FabArray) 11 0.1742 0.1742 0.1742 21.05% MLCellLinOp::applyBC() 4433 0.08073 0.08073 0.08073 9.75% MLPoisson::Fsmooth() 3280 0.06379 0.06379 0.06379 7.71% StateData::FillBoundary(geom) 328 0.02477 0.02477 0.02477 2.99% MLCGSolver::bicgstab 82 0.02425 0.02425 0.02425 2.93% MultiFab::Dot() 1114 0.02249 0.02249 0.02249 2.72% Castro::computeTemp() 63 0.01451 0.01451 0.01451 1.75% MultiFab::LinComb() 1586 0.01445 0.01445 0.01445 1.75% FabArray::setVal() 1144 0.01432 0.01432 0.01432 1.73% FillBoundary_nowait() 4023 0.0143 0.0143 0.0143 1.73% Castro::normalize_species() 62 0.01359 0.01359 0.01359 1.64% FabArray::ParallelCopy_nowait() 861 0.01307 0.01307 0.01307 1.58% MLPoisson::Fapply() 1142 0.01182 0.01182 0.01182 1.43% MLCellLinOp::defineAuxData() 11 0.01166 0.01166 0.01166 1.41% StateDataPhysBCFunct::() 41 0.01156 0.01156 0.01156 1.40% Castro::enforce_min_density() 62 0.01049 0.01049 0.01049 1.27% Gravity::fill_multipole_BCs() 11 0.007973 0.007973 0.007973 0.96% MLMG::addInterpCorrection() 410 0.00753 0.00753 0.00753 0.91% amrex::average_down 410 0.006891 0.006891 0.006891 0.83% MultiFab::Xpay() 585 0.006683 0.006683 0.006683 0.81% Castro::estTimeStep() 21 0.006192 0.006192 0.006192 0.75% Amr::checkPoint() 3 0.005071 0.005071 0.005071 0.61% Castro::do_advance_ctu() 10 0.004572 0.004572 0.004572 0.55% BndryData::define() 11 0.003892 0.003892 0.003892 0.47% Castro::reset_internal_energy(MultiFab) 63 0.003864 0.003864 0.003864 0.47% Castro::construct_new_gravity_source() 10 0.002745 0.002745 0.002745 0.33% Amr::writePlotFile() 2 0.002339 0.002339 0.002339 0.28% MLMG::ResNormInf() 93 0.001937 0.001937 0.001937 0.23% Gravity::get_new_grav_vector() 11 0.001932 0.001932 0.001932 0.23% Castro::construct_old_gravity_source() 10 0.001839 0.001839 0.001839 0.22% MultiFab::Saxpy() 20 0.001816 0.001816 0.001816 0.22% Gravity::get_old_grav_vector() 10 0.001753 0.001753 0.001753 0.21% Castro::expand_state() 10 0.001731 0.001731 0.001731 0.21% MLMG::oneIter() 82 0.001669 0.001669 0.001669 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001656 0.001656 0.001656 0.20% MLCellLinOp::setLevelBC() 11 0.00154 0.00154 0.00154 0.19% Castro::reset_internal_energy(Fab) 504 0.001491 0.001491 0.001491 0.18% Gravity::actual_solve_with_mlmg() 11 0.001401 0.001401 0.001401 0.17% FabArray::mult() 43 0.001323 0.001323 0.001323 0.16% FabArray::setDomainBndry() 41 0.001303 0.001303 0.001303 0.16% MLCellLinOp::smooth() 1640 0.00122 0.00122 0.00122 0.15% MultiFab::contains_nan() 20 0.001188 0.001188 0.001188 0.14% MLCellLinOp::prepareForSolve() 11 0.001163 0.001163 0.001163 0.14% Castro::initData() 1 0.001158 0.001158 0.001158 0.14% MLMG::prepareForSolve() 11 0.001039 0.001039 0.001039 0.13% MLCellLinOp::compGrad() 11 0.0009458 0.0009458 0.0009458 0.11% Castro::enforce_speed_limit() 62 0.0008338 0.0008338 0.0008338 0.10% FabArray::FillBoundary() 4023 0.0008142 0.0008142 0.0008142 0.10% FabArrayBase::getCPC() 1323 0.0007799 0.0007799 0.0007799 0.09% FabArrayBase::CPC::define() 454 0.0007132 0.0007132 0.0007132 0.09% FabArrayBase::getFB() 4023 0.0006122 0.0006122 0.0006122 0.07% Amr::InitAmr() 1 0.0004991 0.0004991 0.0004991 0.06% MLCellLinOp::apply() 1142 0.0004781 0.0004781 0.0004781 0.06% Gravity::solve_for_phi() 10 0.0004486 0.0004486 0.0004486 0.05% Gravity::update_max_rhs() 11 0.0004358 0.0004358 0.0004358 0.05% CGSolver::sxay() 1586 0.0003509 0.0003509 0.0003509 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003142 0.0003142 0.0003142 0.04% Amr::coarseTimeStep() 10 0.0003127 0.0003127 0.0003127 0.04% main() 1 0.000305 0.000305 0.000305 0.04% MLCellLinOp::defineBC() 11 0.0002979 0.0002979 0.0002979 0.04% FabArray::ParallelCopy() 861 0.0002929 0.0002929 0.0002929 0.04% FillPatchIterator::Initialize 41 0.0002928 0.0002928 0.0002928 0.04% MultiFab::max() 11 0.000261 0.000261 0.000261 0.03% MultiFab::Copy() 11 0.0002561 0.0002561 0.0002561 0.03% MLCellLinOp::correctionResidual() 492 0.0002209 0.0002209 0.0002209 0.03% Castro::construct_new_gravity() 10 0.0002081 0.0002081 0.0002081 0.03% MLMG::MLRhsNormInf() 11 0.0002003 0.0002003 0.0002003 0.02% MLMG::mgVcycle() 82 0.0001987 0.0001987 0.0001987 0.02% Amr::timeStep() 10 0.0001979 0.0001979 0.0001979 0.02% MLLinOp::defineGrids() 11 0.0001926 0.0001926 0.0001926 0.02% Castro::subcycle_advance_ctu() 10 0.0001783 0.0001783 0.0001783 0.02% MLMG:computeResOfCorrection() 410 0.0001425 0.0001425 0.0001425 0.02% StateData::checkPoint() 12 0.0001384 0.0001384 0.0001384 0.02% Castro::advance() 10 0.0001152 0.0001152 0.0001152 0.01% MLMG::actualBottomSolve() 82 0.0001029 0.0001029 0.0001029 0.01% Castro::construct_new_source() 50 9.453e-05 9.453e-05 9.453e-05 0.01% Castro::Castro() 1 9.352e-05 9.352e-05 9.352e-05 0.01% MLMG::mgVcycle_down::0 82 9.087e-05 9.087e-05 9.087e-05 0.01% Castro::initialize_advance() 10 9.043e-05 9.043e-05 9.043e-05 0.01% FabArrayBase::FB::FB() 56 8.546e-05 8.546e-05 8.546e-05 0.01% Castro::finalize_advance() 10 8.545e-05 8.545e-05 8.545e-05 0.01% MLMG::mgVcycle_down::1 82 8.222e-05 8.222e-05 8.222e-05 0.01% MLMG::mgVcycle_down::2 82 8.01e-05 8.01e-05 8.01e-05 0.01% Castro::clean_state() 62 7.898e-05 7.898e-05 7.898e-05 0.01% MLMG::solve() 11 7.641e-05 7.641e-05 7.641e-05 0.01% MLMG::mgVcycle_down::3 82 7.557e-05 7.557e-05 7.557e-05 0.01% MLMG::mgVcycle_down::4 82 7.479e-05 7.479e-05 7.479e-05 0.01% AmrLevel::checkPoint() 3 7.452e-05 7.452e-05 7.452e-05 0.01% Castro::initialize_do_advance() 10 6.64e-05 6.64e-05 6.64e-05 0.01% MLMG::mgVcycle_up::4 82 6.306e-05 6.306e-05 6.306e-05 0.01% MLMG::mgVcycle_up::0 82 5.219e-05 5.219e-05 5.219e-05 0.01% MLMG::mgVcycle_up::1 82 5.033e-05 5.033e-05 5.033e-05 0.01% MLMG::mgVcycle_up::3 82 5.02e-05 5.02e-05 5.02e-05 0.01% MLCellLinOp::solutionResidual() 93 5.015e-05 5.015e-05 5.015e-05 0.01% MLMG::mgVcycle_up::2 82 4.858e-05 4.858e-05 4.858e-05 0.01% Castro::swap_state_time_levels() 10 4.26e-05 4.26e-05 4.26e-05 0.01% StateData::define() 4 4.204e-05 4.204e-05 4.204e-05 0.01% Castro::create_source_corrector() 10 4.067e-05 4.067e-05 4.067e-05 0.00% Castro::finalize_do_advance() 10 3.766e-05 3.766e-05 3.766e-05 0.00% Castro::enforce_consistent_e() 1 3.575e-05 3.575e-05 3.575e-05 0.00% MLMG::mgVcycle_bottom 82 3.449e-05 3.449e-05 3.449e-05 0.00% makeSFC 55 3.393e-05 3.393e-05 3.393e-05 0.00% MLMG::computeResidual() 82 3.239e-05 3.239e-05 3.239e-05 0.00% Gravity::actual_multilevel_solve() 1 3.2e-05 3.2e-05 3.2e-05 0.00% Amr::defBaseLevel() 1 3.148e-05 3.148e-05 3.148e-05 0.00% Amr::writeSmallPlotFile() 1 3.026e-05 3.026e-05 3.026e-05 0.00% FillPatchSingleLevel 41 2.994e-05 2.994e-05 2.994e-05 0.00% Castro::initMFs() 1 2.954e-05 2.954e-05 2.954e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.63e-05 2.63e-05 2.63e-05 0.00% MLPoisson::define() 11 2.537e-05 2.537e-05 2.537e-05 0.00% Castro::buildMetrics() 1 2.517e-05 2.517e-05 2.517e-05 0.00% Amr::FinalizeInit() 1 2.203e-05 2.203e-05 2.203e-05 0.00% MLLinOp::define() 11 2.183e-05 2.183e-05 2.183e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.938e-05 1.938e-05 1.938e-05 0.00% Castro::construct_old_source() 50 1.851e-05 1.851e-05 1.851e-05 0.00% Castro::do_new_sources() 10 1.651e-05 1.651e-05 1.651e-05 0.00% Castro::do_old_sources() 10 1.578e-05 1.578e-05 1.578e-05 0.00% DistributionMapping::Distribute() 56 1.486e-05 1.486e-05 1.486e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.331e-05 1.331e-05 1.331e-05 0.00% Castro::check_for_nan() 20 1.266e-05 1.266e-05 1.266e-05 0.00% Gravity::swapTimeLevels() 10 1.127e-05 1.127e-05 1.127e-05 0.00% Castro::construct_old_gravity() 10 1.108e-05 1.108e-05 1.108e-05 0.00% Castro::apply_source_to_state() 20 1.063e-05 1.063e-05 1.063e-05 0.00% Amr::initSubcycle() 1 9.546e-06 9.546e-06 9.546e-06 0.00% MLPoisson::prepareForSolve() 11 9.137e-06 9.137e-06 9.137e-06 0.00% Castro::post_timestep() 10 9.07e-06 9.07e-06 9.07e-06 0.00% AmrLevel::checkPointPost() 3 8.718e-06 8.718e-06 8.718e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.675e-06 7.675e-06 7.675e-06 0.00% Amr::InitializeInit() 1 7.247e-06 7.247e-06 7.247e-06 0.00% MLMG::computeMLResidual() 11 7.135e-06 7.135e-06 7.135e-06 0.00% Castro::computeNewDt() 9 6.887e-06 6.887e-06 6.887e-06 0.00% MLMG::getGradSolution() 11 6.361e-06 6.361e-06 6.361e-06 0.00% MLMG::buildFineMask() 11 4.983e-06 4.983e-06 4.983e-06 0.00% MLMG::MLResNormInf() 11 4.629e-06 4.629e-06 4.629e-06 0.00% Castro::post_init() 1 4.471e-06 4.471e-06 4.471e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.247e-06 4.247e-06 4.247e-06 0.00% Castro::retry_advance_ctu() 10 3.997e-06 3.997e-06 3.997e-06 0.00% Gravity::set_mass_offset() 11 3.714e-06 3.714e-06 3.714e-06 0.00% Castro::computeInitialDt() 2 3.198e-06 3.198e-06 3.198e-06 0.00% Castro::FluxRegCrseInit 10 2.909e-06 2.909e-06 2.909e-06 0.00% Amr::init() 1 2.886e-06 2.886e-06 2.886e-06 0.00% Castro::FluxRegFineAdd() 10 2.585e-06 2.585e-06 2.585e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.947e-06 1.947e-06 1.947e-06 0.00% AmrLevel::checkPointPre() 3 1.901e-06 1.901e-06 1.901e-06 0.00% Amr::initialInit() 1 1.403e-06 1.403e-06 1.403e-06 0.00% Castro::post_regrid() 1 1.202e-06 1.202e-06 1.202e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8277 0.8277 0.8277 100.00% Amr::coarseTimeStep() 10 0.6805 0.6805 0.6805 82.21% Amr::timeStep() 10 0.5884 0.5884 0.5884 71.08% Castro::advance() 10 0.5815 0.5815 0.5815 70.26% Castro::subcycle_advance_ctu() 10 0.5708 0.5708 0.5708 68.96% Castro::do_advance_ctu() 10 0.5706 0.5706 0.5706 68.93% Gravity::solve_phi_with_mlmg() 11 0.315 0.315 0.315 38.05% Gravity::actual_solve_with_mlmg() 11 0.3068 0.3068 0.3068 37.06% Castro::construct_new_gravity() 10 0.2855 0.2855 0.2855 34.50% MLMG::solve() 11 0.284 0.284 0.284 34.31% Gravity::solve_for_phi() 10 0.2704 0.2704 0.2704 32.67% MLMG::oneIter() 82 0.2694 0.2694 0.2694 32.55% MLMG::mgVcycle() 82 0.2677 0.2677 0.2677 32.34% Castro::construct_ctu_hydro_source() 10 0.2093 0.2093 0.2093 25.29% VisMF::Write(FabArray) 11 0.1742 0.1742 0.1742 21.05% MLCellLinOp::smooth() 1640 0.1368 0.1368 0.1368 16.53% Amr::checkPoint() 3 0.1319 0.1319 0.1319 15.93% AmrLevel::checkPoint() 3 0.1268 0.1268 0.1268 15.32% StateData::checkPoint() 12 0.1267 0.1267 0.1267 15.31% Amr::init() 1 0.1217 0.1217 0.1217 14.70% MLCellLinOp::applyBC() 4433 0.09654 0.09654 0.09654 11.66% MLMG::mgVcycle_bottom 82 0.08269 0.08269 0.08269 9.99% MLMG::actualBottomSolve() 82 0.08266 0.08266 0.08266 9.99% MLCGSolver::bicgstab 82 0.08183 0.08183 0.08183 9.89% MLPoisson::Fsmooth() 3280 0.06379 0.06379 0.06379 7.71% Amr::initialInit() 1 0.05242 0.05242 0.05242 6.33% Amr::writePlotFile() 2 0.04999 0.04999 0.04999 6.04% Amr::FinalizeInit() 1 0.04838 0.04838 0.04838 5.85% Castro::post_init() 1 0.04709 0.04709 0.04709 5.69% Gravity::multilevel_solve_for_new_phi() 1 0.04509 0.04509 0.04509 5.45% Gravity::actual_multilevel_solve() 1 0.04507 0.04507 0.04507 5.44% Castro::clean_state() 62 0.04408 0.04408 0.04408 5.33% FillPatchIterator::Initialize 41 0.04189 0.04189 0.04189 5.06% FillPatchSingleLevel 41 0.0403 0.0403 0.0403 4.87% MLCellLinOp::apply() 1142 0.03655 0.03655 0.03655 4.42% StateDataPhysBCFunct::() 41 0.03633 0.03633 0.03633 4.39% MLMG::mgVcycle_down::0 82 0.0356 0.0356 0.0356 4.30% MLMG::mgVcycle_up::0 82 0.03044 0.03044 0.03044 3.68% StateData::FillBoundary(geom) 328 0.02477 0.02477 0.02477 2.99% MultiFab::Dot() 1114 0.02249 0.02249 0.02249 2.72% MLCellLinOp::correctionResidual() 492 0.02147 0.02147 0.02147 2.59% Castro::computeTemp() 63 0.01987 0.01987 0.01987 2.40% Castro::initialize_do_advance() 10 0.01895 0.01895 0.01895 2.29% MLMG:computeResOfCorrection() 410 0.01856 0.01856 0.01856 2.24% MLPoisson::define() 11 0.01846 0.01846 0.01846 2.23% MLMG::mgVcycle_down::1 82 0.01779 0.01779 0.01779 2.15% MLMG::mgVcycle_down::2 82 0.01736 0.01736 0.01736 2.10% Gravity::get_new_grav_vector() 11 0.01684 0.01684 0.01684 2.03% MLMG::mgVcycle_down::3 82 0.01647 0.01647 0.01647 1.99% FabArray::FillBoundary() 4023 0.01581 0.01581 0.01581 1.91% MLMG::mgVcycle_down::4 82 0.0157 0.0157 0.0157 1.90% Castro::construct_old_gravity() 10 0.01516 0.01516 0.01516 1.83% Gravity::get_old_grav_vector() 10 0.01515 0.01515 0.01515 1.83% FillBoundary_nowait() 4023 0.015 0.015 0.015 1.81% CGSolver::sxay() 1586 0.0148 0.0148 0.0148 1.79% MultiFab::LinComb() 1586 0.01445 0.01445 0.01445 1.75% FabArray::setVal() 1144 0.01432 0.01432 0.01432 1.73% FabArray::ParallelCopy() 861 0.01421 0.01421 0.01421 1.72% FabArray::ParallelCopy_nowait() 861 0.01392 0.01392 0.01392 1.68% Castro::normalize_species() 62 0.01359 0.01359 0.01359 1.64% MLCGSolver::ParallelAllReduce 1514 0.01345 0.01345 0.01345 1.62% MLMG::mgVcycle_up::2 82 0.01336 0.01336 0.01336 1.61% MLMG::mgVcycle_up::1 82 0.01308 0.01308 0.01308 1.58% MLCellLinOp::defineAuxData() 11 0.01302 0.01302 0.01302 1.57% MLMG::addInterpCorrection() 410 0.01264 0.01264 0.01264 1.53% MLMG::mgVcycle_up::3 82 0.01258 0.01258 0.01258 1.52% MLMG::mgVcycle_up::4 82 0.01246 0.01246 0.01246 1.51% amrex::average_down 410 0.01204 0.01204 0.01204 1.45% Castro::do_new_sources() 10 0.01204 0.01204 0.01204 1.45% MLPoisson::Fapply() 1142 0.01182 0.01182 0.01182 1.43% Castro::expand_state() 10 0.01123 0.01123 0.01123 1.36% Castro::initialize_advance() 10 0.01054 0.01054 0.01054 1.27% Castro::enforce_min_density() 62 0.01049 0.01049 0.01049 1.27% Castro::do_old_sources() 10 0.009527 0.009527 0.009527 1.15% Gravity::fill_multipole_BCs() 11 0.007973 0.007973 0.007973 0.96% MLCellLinOp::solutionResidual() 93 0.007194 0.007194 0.007194 0.87% MultiFab::Xpay() 585 0.006683 0.006683 0.006683 0.81% Castro::post_timestep() 10 0.006659 0.006659 0.006659 0.80% Castro::estTimeStep() 21 0.006192 0.006192 0.006192 0.75% MLMG::computeResidual() 82 0.006183 0.006183 0.006183 0.75% Castro::reset_internal_energy(MultiFab) 63 0.005355 0.005355 0.005355 0.65% MLCellLinOp::defineBC() 11 0.005133 0.005133 0.005133 0.62% MLMG::prepareForSolve() 11 0.005119 0.005119 0.005119 0.62% BndryData::define() 11 0.004835 0.004835 0.004835 0.58% Amr::InitializeInit() 1 0.004031 0.004031 0.004031 0.49% Amr::defBaseLevel() 1 0.004024 0.004024 0.004024 0.49% Castro::initData() 1 0.003476 0.003476 0.003476 0.42% Castro::computeNewDt() 9 0.003386 0.003386 0.003386 0.41% Castro::construct_new_source() 50 0.00284 0.00284 0.00284 0.34% Castro::construct_new_gravity_source() 10 0.002745 0.002745 0.002745 0.33% MLMG::ResNormInf() 93 0.001937 0.001937 0.001937 0.23% Castro::construct_old_source() 50 0.001857 0.001857 0.001857 0.22% Castro::construct_old_gravity_source() 10 0.001839 0.001839 0.001839 0.22% Castro::apply_source_to_state() 20 0.001826 0.001826 0.001826 0.22% MultiFab::Saxpy() 20 0.001816 0.001816 0.001816 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001656 0.001656 0.001656 0.20% MLCellLinOp::setLevelBC() 11 0.00154 0.00154 0.00154 0.19% FabArrayBase::getCPC() 1323 0.001493 0.001493 0.001493 0.18% Castro::reset_internal_energy(Fab) 504 0.001491 0.001491 0.001491 0.18% MLMG::getGradSolution() 11 0.001437 0.001437 0.001437 0.17% MLCellLinOp::compGrad() 11 0.001431 0.001431 0.001431 0.17% FabArray::mult() 43 0.001323 0.001323 0.001323 0.16% FabArray::setDomainBndry() 41 0.001303 0.001303 0.001303 0.16% Castro::check_for_nan() 20 0.0012 0.0012 0.0012 0.15% MultiFab::contains_nan() 20 0.001188 0.001188 0.001188 0.14% MLPoisson::prepareForSolve() 11 0.001172 0.001172 0.001172 0.14% MLCellLinOp::prepareForSolve() 11 0.001163 0.001163 0.001163 0.14% MLMG::computeMLResidual() 11 0.00105 0.00105 0.00105 0.13% Castro::post_regrid() 1 0.001047 0.001047 0.001047 0.13% Gravity::update_max_rhs() 11 0.0008486 0.0008486 0.0008486 0.10% Castro::enforce_speed_limit() 62 0.0008338 0.0008338 0.0008338 0.10% Castro::computeInitialDt() 2 0.000715 0.000715 0.000715 0.09% FabArrayBase::CPC::define() 454 0.0007132 0.0007132 0.0007132 0.09% FabArrayBase::getFB() 4023 0.0006977 0.0006977 0.0006977 0.08% Amr::InitAmr() 1 0.0005086 0.0005086 0.0005086 0.06% Castro::Castro() 1 0.0004614 0.0004614 0.0004614 0.06% Gravity::swapTimeLevels() 10 0.0004397 0.0004397 0.0004397 0.05% MLLinOp::define() 11 0.0002772 0.0002772 0.0002772 0.03% MultiFab::max() 11 0.000261 0.000261 0.000261 0.03% MLMG::MLResNormInf() 11 0.00026 0.00026 0.00026 0.03% MultiFab::Copy() 11 0.0002561 0.0002561 0.0002561 0.03% MLLinOp::defineGrids() 11 0.0002553 0.0002553 0.0002553 0.03% MLMG::MLRhsNormInf() 11 0.0002003 0.0002003 0.0002003 0.02% Castro::buildMetrics() 1 0.000171 0.000171 0.000171 0.02% Castro::finalize_advance() 10 9.095e-05 9.095e-05 9.095e-05 0.01% FabArrayBase::FB::FB() 56 8.546e-05 8.546e-05 8.546e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 6.079e-05 6.079e-05 6.079e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.971e-05 4.971e-05 4.971e-05 0.01% makeSFC 55 4.748e-05 4.748e-05 4.748e-05 0.01% Castro::swap_state_time_levels() 10 4.26e-05 4.26e-05 4.26e-05 0.01% StateData::define() 4 4.204e-05 4.204e-05 4.204e-05 0.01% Castro::create_source_corrector() 10 4.067e-05 4.067e-05 4.067e-05 0.00% Castro::finalize_do_advance() 10 3.766e-05 3.766e-05 3.766e-05 0.00% Castro::enforce_consistent_e() 1 3.575e-05 3.575e-05 3.575e-05 0.00% Amr::writeSmallPlotFile() 1 3.026e-05 3.026e-05 3.026e-05 0.00% Castro::initMFs() 1 2.954e-05 2.954e-05 2.954e-05 0.00% DistributionMapping::Distribute() 56 1.486e-05 1.486e-05 1.486e-05 0.00% Amr::initSubcycle() 1 9.546e-06 9.546e-06 9.546e-06 0.00% AmrLevel::checkPointPost() 3 8.718e-06 8.718e-06 8.718e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.562e-06 5.562e-06 5.562e-06 0.00% MLMG::buildFineMask() 11 4.983e-06 4.983e-06 4.983e-06 0.00% Castro::retry_advance_ctu() 10 3.997e-06 3.997e-06 3.997e-06 0.00% Gravity::set_mass_offset() 11 3.714e-06 3.714e-06 3.714e-06 0.00% Castro::FluxRegCrseInit 10 2.909e-06 2.909e-06 2.909e-06 0.00% Castro::FluxRegFineAdd() 10 2.585e-06 2.585e-06 2.585e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.947e-06 1.947e-06 1.947e-06 0.00% AmrLevel::checkPointPre() 3 1.901e-06 1.901e-06 1.901e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.06-33-g6f72de283c38) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.06-33-g6f72de283c38) initialized Starting run at 08:29:13 UTC on 2022-06-16. Successfully read inputs file ... Castro git describe: 22.06-12-g556652b03 AMReX git describe: 22.06-33-g6f72de283 Microphysics git describe: 22.06-2-g35a553f4 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.433486182 Restart time = 0.047391984 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.050759365 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051744024 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.064168361 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.064322384 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.066410871 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026502923 seconds Ending run at 08:29:13 UTC on 2022-06-16. Run time = 0.372340262 Run time without initialization = 0.324347972 Average number of zones advanced per microsecond: 4.041 Average number of zones advanced per microsecond per rank: 4.041 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3724 ... 0.3724 ... 0.3724 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0991 0.0991 0.0991 26.62% VisMF::Read() 3 0.03991 0.03991 0.03991 10.72% MLCellLinOp::applyBC() 1946 0.03518 0.03518 0.03518 9.45% MLPoisson::Fsmooth() 1440 0.0276 0.0276 0.0276 7.41% VisMF::Write(FabArray) 1 0.02505 0.02505 0.02505 6.73% StateData::FillBoundary(geom) 160 0.01172 0.01172 0.01172 3.15% MLCGSolver::bicgstab 36 0.01042 0.01042 0.01042 2.80% MultiFab::Dot() 484 0.00971 0.00971 0.00971 2.61% Castro::computeTemp() 30 0.00796 0.00796 0.00796 2.14% Castro::normalize_species() 30 0.006985 0.006985 0.006985 1.88% FabArray::setVal() 537 0.006874 0.006874 0.006874 1.85% MLCellLinOp::defineAuxData() 6 0.00635 0.00635 0.00635 1.71% FillBoundary_nowait() 1766 0.00626 0.00626 0.00626 1.68% MultiFab::LinComb() 690 0.006196 0.006196 0.006196 1.66% FabArray::ParallelCopy_nowait() 380 0.006029 0.006029 0.006029 1.62% Castro::enforce_min_density() 30 0.005961 0.005961 0.005961 1.60% StateDataPhysBCFunct::() 20 0.005248 0.005248 0.005248 1.41% MLPoisson::Fapply() 500 0.005102 0.005102 0.005102 1.37% Gravity::fill_multipole_BCs() 6 0.00422 0.00422 0.00422 1.13% Castro::estTimeStep() 10 0.003343 0.003343 0.003343 0.90% MLMG::addInterpCorrection() 180 0.003241 0.003241 0.003241 0.87% Amr::restart() 1 0.00315 0.00315 0.00315 0.85% amrex::average_down 180 0.002978 0.002978 0.002978 0.80% MultiFab::Xpay() 258 0.0029 0.0029 0.0029 0.78% Castro::do_advance_ctu() 5 0.002252 0.002252 0.002252 0.60% BndryData::define() 6 0.00218 0.00218 0.00218 0.59% Castro::reset_internal_energy(MultiFab) 30 0.001568 0.001568 0.001568 0.42% Amr::writePlotFile() 1 0.001546 0.001546 0.001546 0.42% Castro::construct_new_gravity_source() 5 0.001507 0.001507 0.001507 0.40% Castro::construct_old_gravity_source() 5 0.001314 0.001314 0.001314 0.35% Gravity::get_old_grav_vector() 5 0.001056 0.001056 0.001056 0.28% Castro::reset_internal_energy(Fab) 240 0.001001 0.001001 0.001001 0.27% Gravity::get_new_grav_vector() 5 0.0009552 0.0009552 0.0009552 0.26% MultiFab::Saxpy() 10 0.0009194 0.0009194 0.0009194 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008936 0.0008936 0.0008936 0.24% Castro::expand_state() 5 0.0008702 0.0008702 0.0008702 0.23% MLMG::ResNormInf() 42 0.0008528 0.0008528 0.0008528 0.23% MLCellLinOp::setLevelBC() 6 0.0008232 0.0008232 0.0008232 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007634 0.0007634 0.0007634 0.21% MLMG::oneIter() 36 0.0007312 0.0007312 0.0007312 0.20% FabArray::mult() 22 0.0006628 0.0006628 0.0006628 0.18% MLCellLinOp::prepareForSolve() 6 0.0006582 0.0006582 0.0006582 0.18% FabArray::setDomainBndry() 20 0.0006502 0.0006502 0.0006502 0.17% MultiFab::contains_nan() 10 0.000605 0.000605 0.000605 0.16% MLMG::prepareForSolve() 6 0.0005721 0.0005721 0.0005721 0.15% MLCellLinOp::smooth() 720 0.0005238 0.0005238 0.0005238 0.14% MLCellLinOp::compGrad() 6 0.0005024 0.0005024 0.0005024 0.13% Castro::enforce_speed_limit() 30 0.0004882 0.0004882 0.0004882 0.13% Amr::InitAmr() 1 0.0004376 0.0004376 0.0004376 0.12% FabArrayBase::CPC::define() 244 0.0004138 0.0004138 0.0004138 0.11% FabArray::FillBoundary() 1766 0.0003739 0.0003739 0.0003739 0.10% FabArrayBase::getCPC() 632 0.0003709 0.0003709 0.0003709 0.10% main() 1 0.0002936 0.0002936 0.0002936 0.08% Gravity::update_max_rhs() 6 0.000272 0.000272 0.000272 0.07% FabArrayBase::getFB() 1766 0.000261 0.000261 0.000261 0.07% Castro::subcycle_advance_ctu() 5 0.0002262 0.0002262 0.0002262 0.06% Gravity::solve_for_phi() 5 0.0002197 0.0002197 0.0002197 0.06% MLCellLinOp::apply() 500 0.0001978 0.0001978 0.0001978 0.05% Castro::create_source_corrector() 5 0.0001947 0.0001947 0.0001947 0.05% CGSolver::sxay() 690 0.0001761 0.0001761 0.0001761 0.05% MLCellLinOp::defineBC() 6 0.0001569 0.0001569 0.0001569 0.04% Amr::coarseTimeStep() 5 0.0001558 0.0001558 0.0001558 0.04% FillPatchIterator::Initialize 20 0.0001522 0.0001522 0.0001522 0.04% MultiFab::max() 6 0.0001477 0.0001477 0.0001477 0.04% Castro::advance() 5 0.0001453 0.0001453 0.0001453 0.04% MultiFab::Copy() 6 0.0001384 0.0001384 0.0001384 0.04% Castro::construct_new_gravity() 5 0.0001375 0.0001375 0.0001375 0.04% Castro::construct_new_source() 25 0.0001331 0.0001331 0.0001331 0.04% FabArray::ParallelCopy() 380 0.0001276 0.0001276 0.0001276 0.03% MLCGSolver::ParallelAllReduce 659 0.0001276 0.0001276 0.0001276 0.03% MLLinOp::defineGrids() 6 0.0001259 0.0001259 0.0001259 0.03% Amr::timeStep() 5 0.0001163 0.0001163 0.0001163 0.03% MLMG::MLRhsNormInf() 6 0.000106 0.000106 0.000106 0.03% Castro::construct_old_source() 25 9.543e-05 9.543e-05 9.543e-05 0.03% MLCellLinOp::correctionResidual() 216 9.352e-05 9.352e-05 9.352e-05 0.03% MLMG::mgVcycle() 36 8.47e-05 8.47e-05 8.47e-05 0.02% AmrLevel::restart() 1 8.414e-05 8.414e-05 8.414e-05 0.02% Castro::computeNewDt() 5 8.066e-05 8.066e-05 8.066e-05 0.02% StateData::restartDoit() 4 7.721e-05 7.721e-05 7.721e-05 0.02% Castro::finalize_advance() 5 6.974e-05 6.974e-05 6.974e-05 0.02% FabArrayBase::FB::FB() 26 5.792e-05 5.792e-05 5.792e-05 0.02% MLMG:computeResOfCorrection() 180 5.456e-05 5.456e-05 5.456e-05 0.01% Castro::initialize_advance() 5 5.185e-05 5.185e-05 5.185e-05 0.01% Castro::initialize_do_advance() 5 4.701e-05 4.701e-05 4.701e-05 0.01% Castro::post_restart() 1 4.39e-05 4.39e-05 4.39e-05 0.01% Castro::clean_state() 30 4.282e-05 4.282e-05 4.282e-05 0.01% MLMG::actualBottomSolve() 36 4.262e-05 4.262e-05 4.262e-05 0.01% MLMG::mgVcycle_down::0 36 3.997e-05 3.997e-05 3.997e-05 0.01% MLMG::mgVcycle_down::1 36 3.874e-05 3.874e-05 3.874e-05 0.01% Castro::buildMetrics() 1 3.794e-05 3.794e-05 3.794e-05 0.01% MLMG::mgVcycle_down::2 36 3.761e-05 3.761e-05 3.761e-05 0.01% MLMG::solve() 6 3.687e-05 3.687e-05 3.687e-05 0.01% MLMG::mgVcycle_down::4 36 3.521e-05 3.521e-05 3.521e-05 0.01% MLMG::mgVcycle_down::3 36 3.42e-05 3.42e-05 3.42e-05 0.01% Castro::swap_state_time_levels() 5 3.12e-05 3.12e-05 3.12e-05 0.01% Gravity::actual_multilevel_solve() 1 3.109e-05 3.109e-05 3.109e-05 0.01% Amr::writeSmallPlotFile() 1 2.832e-05 2.832e-05 2.832e-05 0.01% MLMG::mgVcycle_up::4 36 2.82e-05 2.82e-05 2.82e-05 0.01% Castro::initMFs() 1 2.71e-05 2.71e-05 2.71e-05 0.01% MLCellLinOp::solutionResidual() 42 2.351e-05 2.351e-05 2.351e-05 0.01% MLLinOp::define() 6 2.333e-05 2.333e-05 2.333e-05 0.01% MLMG::mgVcycle_up::0 36 2.333e-05 2.333e-05 2.333e-05 0.01% MLMG::mgVcycle_up::3 36 2.275e-05 2.275e-05 2.275e-05 0.01% MLMG::mgVcycle_up::2 36 2.217e-05 2.217e-05 2.217e-05 0.01% MLMG::mgVcycle_up::1 36 2.129e-05 2.129e-05 2.129e-05 0.01% Castro::post_timestep() 5 2.085e-05 2.085e-05 2.085e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.836e-05 1.836e-05 1.836e-05 0.00% MLPoisson::define() 6 1.78e-05 1.78e-05 1.78e-05 0.00% Castro::finalize_do_advance() 5 1.76e-05 1.76e-05 1.76e-05 0.00% makeSFC 30 1.555e-05 1.555e-05 1.555e-05 0.00% MLMG::mgVcycle_bottom 36 1.489e-05 1.489e-05 1.489e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.469e-05 1.469e-05 1.469e-05 0.00% FillPatchSingleLevel 20 1.445e-05 1.445e-05 1.445e-05 0.00% MLMG::computeResidual() 36 1.433e-05 1.433e-05 1.433e-05 0.00% Castro::do_new_sources() 5 1.055e-05 1.055e-05 1.055e-05 0.00% Amr::initSubcycle() 1 1.003e-05 1.003e-05 1.003e-05 0.00% DistributionMapping::Distribute() 31 9.77e-06 9.77e-06 9.77e-06 0.00% Castro::check_for_nan() 10 9.733e-06 9.733e-06 9.733e-06 0.00% Castro::do_old_sources() 5 9.441e-06 9.441e-06 9.441e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.365e-06 8.365e-06 8.365e-06 0.00% Castro::construct_old_gravity() 5 8.238e-06 8.238e-06 8.238e-06 0.00% Castro::apply_source_to_state() 10 6.432e-06 6.432e-06 6.432e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.434e-06 5.434e-06 5.434e-06 0.00% Gravity::swapTimeLevels() 5 5.192e-06 5.192e-06 5.192e-06 0.00% MLPoisson::prepareForSolve() 6 4.926e-06 4.926e-06 4.926e-06 0.00% MLMG::computeMLResidual() 6 3.52e-06 3.52e-06 3.52e-06 0.00% MLMG::getGradSolution() 6 3.177e-06 3.177e-06 3.177e-06 0.00% MLMG::buildFineMask() 6 2.86e-06 2.86e-06 2.86e-06 0.00% MLMG::MLResNormInf() 6 2.682e-06 2.682e-06 2.682e-06 0.00% Castro::retry_advance_ctu() 5 2.584e-06 2.584e-06 2.584e-06 0.00% Gravity::set_mass_offset() 6 2.58e-06 2.58e-06 2.58e-06 0.00% Castro::FluxRegCrseInit 5 1.859e-06 1.859e-06 1.859e-06 0.00% Castro::FluxRegFineAdd() 5 1.25e-06 1.25e-06 1.25e-06 0.00% AmrLevel::AmrLevel() 1 1.25e-06 1.25e-06 1.25e-06 0.00% Amr::init() 1 1.223e-06 1.223e-06 1.223e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.096e-06 1.096e-06 1.096e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3724 0.3724 0.3724 100.00% Amr::coarseTimeStep() 5 0.2976 0.2976 0.2976 79.91% Amr::timeStep() 5 0.295 0.295 0.295 79.22% Castro::advance() 5 0.2913 0.2913 0.2913 78.24% Castro::subcycle_advance_ctu() 5 0.286 0.286 0.286 76.80% Castro::do_advance_ctu() 5 0.2858 0.2858 0.2858 76.74% Castro::construct_new_gravity() 5 0.1449 0.1449 0.1449 38.92% Gravity::solve_phi_with_mlmg() 6 0.1407 0.1407 0.1407 37.77% Gravity::solve_for_phi() 5 0.1371 0.1371 0.1371 36.82% Gravity::actual_solve_with_mlmg() 6 0.1363 0.1363 0.1363 36.61% MLMG::solve() 6 0.1238 0.1238 0.1238 33.24% MLMG::oneIter() 36 0.1167 0.1167 0.1167 31.34% MLMG::mgVcycle() 36 0.116 0.116 0.116 31.14% Castro::construct_ctu_hydro_source() 5 0.09914 0.09914 0.09914 26.62% MLCellLinOp::smooth() 720 0.05937 0.05937 0.05937 15.94% Amr::init() 1 0.04744 0.04744 0.04744 12.74% Amr::restart() 1 0.04744 0.04744 0.04744 12.74% MLCellLinOp::applyBC() 1946 0.04213 0.04213 0.04213 11.31% AmrLevel::restart() 1 0.04013 0.04013 0.04013 10.78% StateData::restartDoit() 4 0.04004 0.04004 0.04004 10.75% VisMF::Read() 3 0.03991 0.03991 0.03991 10.72% MLMG::mgVcycle_bottom 36 0.03562 0.03562 0.03562 9.57% MLMG::actualBottomSolve() 36 0.03561 0.03561 0.03561 9.56% MLCGSolver::bicgstab 36 0.03524 0.03524 0.03524 9.46% MLPoisson::Fsmooth() 1440 0.0276 0.0276 0.0276 7.41% Amr::writePlotFile() 1 0.02659 0.02659 0.02659 7.14% VisMF::Write(FabArray) 1 0.02505 0.02505 0.02505 6.73% Castro::clean_state() 30 0.02401 0.02401 0.02401 6.45% FillPatchIterator::Initialize 20 0.01978 0.01978 0.01978 5.31% FillPatchSingleLevel 20 0.01897 0.01897 0.01897 5.10% StateDataPhysBCFunct::() 20 0.01696 0.01696 0.01696 4.56% MLCellLinOp::apply() 500 0.01591 0.01591 0.01591 4.27% MLMG::mgVcycle_down::0 36 0.01547 0.01547 0.01547 4.16% MLMG::mgVcycle_up::0 36 0.0133 0.0133 0.0133 3.57% StateData::FillBoundary(geom) 160 0.01172 0.01172 0.01172 3.15% Castro::initialize_do_advance() 5 0.01156 0.01156 0.01156 3.10% Castro::computeTemp() 30 0.01053 0.01053 0.01053 2.83% MLPoisson::define() 6 0.01019 0.01019 0.01019 2.74% MultiFab::Dot() 484 0.00971 0.00971 0.00971 2.61% MLCellLinOp::correctionResidual() 216 0.009294 0.009294 0.009294 2.50% MLMG:computeResOfCorrection() 180 0.008018 0.008018 0.008018 2.15% Castro::construct_old_gravity() 5 0.007768 0.007768 0.007768 2.09% Gravity::get_old_grav_vector() 5 0.00776 0.00776 0.00776 2.08% MLMG::mgVcycle_down::1 36 0.007721 0.007721 0.007721 2.07% Gravity::get_new_grav_vector() 5 0.007689 0.007689 0.007689 2.06% MLMG::mgVcycle_down::2 36 0.007504 0.007504 0.007504 2.02% MLMG::mgVcycle_down::3 36 0.007124 0.007124 0.007124 1.91% MLCellLinOp::defineAuxData() 6 0.007114 0.007114 0.007114 1.91% Castro::normalize_species() 30 0.006985 0.006985 0.006985 1.88% FabArray::FillBoundary() 1766 0.006952 0.006952 0.006952 1.87% Castro::do_new_sources() 5 0.006898 0.006898 0.006898 1.85% FabArray::setVal() 537 0.006874 0.006874 0.006874 1.85% MLMG::mgVcycle_down::4 36 0.006822 0.006822 0.006822 1.83% FillBoundary_nowait() 1766 0.006579 0.006579 0.006579 1.77% FabArray::ParallelCopy() 380 0.006532 0.006532 0.006532 1.75% FabArray::ParallelCopy_nowait() 380 0.006405 0.006405 0.006405 1.72% CGSolver::sxay() 690 0.006372 0.006372 0.006372 1.71% MultiFab::LinComb() 690 0.006196 0.006196 0.006196 1.66% Castro::enforce_min_density() 30 0.005961 0.005961 0.005961 1.60% MLCGSolver::ParallelAllReduce 659 0.005836 0.005836 0.005836 1.57% MLMG::mgVcycle_up::2 36 0.005734 0.005734 0.005734 1.54% MLMG::mgVcycle_up::1 36 0.005714 0.005714 0.005714 1.53% Castro::do_old_sources() 5 0.005655 0.005655 0.005655 1.52% MLMG::addInterpCorrection() 180 0.005507 0.005507 0.005507 1.48% MLMG::mgVcycle_up::3 36 0.005453 0.005453 0.005453 1.46% MLMG::mgVcycle_up::4 36 0.005403 0.005403 0.005403 1.45% amrex::average_down 180 0.00525 0.00525 0.00525 1.41% Castro::expand_state() 5 0.005137 0.005137 0.005137 1.38% Castro::initialize_advance() 5 0.005107 0.005107 0.005107 1.37% MLPoisson::Fapply() 500 0.005102 0.005102 0.005102 1.37% Gravity::fill_multipole_BCs() 6 0.00422 0.00422 0.00422 1.13% Castro::post_restart() 1 0.003967 0.003967 0.003967 1.07% Gravity::multilevel_solve_for_new_phi() 1 0.003821 0.003821 0.003821 1.03% Gravity::actual_multilevel_solve() 1 0.003802 0.003802 0.003802 1.02% Castro::post_timestep() 5 0.003558 0.003558 0.003558 0.96% Castro::estTimeStep() 10 0.003343 0.003343 0.003343 0.90% MLCellLinOp::solutionResidual() 42 0.003242 0.003242 0.003242 0.87% MultiFab::Xpay() 258 0.0029 0.0029 0.0029 0.78% MLCellLinOp::defineBC() 6 0.002875 0.002875 0.002875 0.77% MLMG::prepareForSolve() 6 0.002824 0.002824 0.002824 0.76% BndryData::define() 6 0.002718 0.002718 0.002718 0.73% MLMG::computeResidual() 36 0.002678 0.002678 0.002678 0.72% Castro::reset_internal_energy(MultiFab) 30 0.002569 0.002569 0.002569 0.69% Castro::computeNewDt() 5 0.002396 0.002396 0.002396 0.64% Castro::construct_new_source() 25 0.00164 0.00164 0.00164 0.44% Castro::construct_new_gravity_source() 5 0.001507 0.001507 0.001507 0.40% Castro::construct_old_source() 25 0.00141 0.00141 0.00141 0.38% Castro::construct_old_gravity_source() 5 0.001314 0.001314 0.001314 0.35% Castro::reset_internal_energy(Fab) 240 0.001001 0.001001 0.001001 0.27% Castro::apply_source_to_state() 10 0.0009259 0.0009259 0.0009259 0.25% MultiFab::Saxpy() 10 0.0009194 0.0009194 0.0009194 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008936 0.0008936 0.0008936 0.24% MLMG::ResNormInf() 42 0.0008528 0.0008528 0.0008528 0.23% MLCellLinOp::setLevelBC() 6 0.0008232 0.0008232 0.0008232 0.22% FabArrayBase::getCPC() 632 0.0007847 0.0007847 0.0007847 0.21% MLMG::getGradSolution() 6 0.0007794 0.0007794 0.0007794 0.21% MLCellLinOp::compGrad() 6 0.0007762 0.0007762 0.0007762 0.21% MLPoisson::prepareForSolve() 6 0.0006631 0.0006631 0.0006631 0.18% FabArray::mult() 22 0.0006628 0.0006628 0.0006628 0.18% MLCellLinOp::prepareForSolve() 6 0.0006582 0.0006582 0.0006582 0.18% FabArray::setDomainBndry() 20 0.0006502 0.0006502 0.0006502 0.17% Castro::check_for_nan() 10 0.0006147 0.0006147 0.0006147 0.17% MultiFab::contains_nan() 10 0.000605 0.000605 0.000605 0.16% MLMG::computeMLResidual() 6 0.0005811 0.0005811 0.0005811 0.16% Gravity::update_max_rhs() 6 0.0005042 0.0005042 0.0005042 0.14% Castro::enforce_speed_limit() 30 0.0004882 0.0004882 0.0004882 0.13% Amr::InitAmr() 1 0.0004476 0.0004476 0.0004476 0.12% FabArrayBase::CPC::define() 244 0.0004138 0.0004138 0.0004138 0.11% FabArrayBase::getFB() 1766 0.0003189 0.0003189 0.0003189 0.09% Gravity::swapTimeLevels() 5 0.0002524 0.0002524 0.0002524 0.07% Castro::create_source_corrector() 5 0.0001947 0.0001947 0.0001947 0.05% MLLinOp::define() 6 0.000182 0.000182 0.000182 0.05% Castro::buildMetrics() 1 0.0001591 0.0001591 0.0001591 0.04% MLLinOp::defineGrids() 6 0.0001587 0.0001587 0.0001587 0.04% MultiFab::max() 6 0.0001477 0.0001477 0.0001477 0.04% MultiFab::Copy() 6 0.0001384 0.0001384 0.0001384 0.04% MLMG::MLResNormInf() 6 0.0001367 0.0001367 0.0001367 0.04% MLMG::MLRhsNormInf() 6 0.000106 0.000106 0.000106 0.03% Castro::finalize_advance() 5 7.285e-05 7.285e-05 7.285e-05 0.02% FabArrayBase::FB::FB() 26 5.792e-05 5.792e-05 5.792e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 3.167e-05 3.167e-05 3.167e-05 0.01% Castro::swap_state_time_levels() 5 3.12e-05 3.12e-05 3.12e-05 0.01% Amr::writeSmallPlotFile() 1 2.832e-05 2.832e-05 2.832e-05 0.01% Castro::initMFs() 1 2.71e-05 2.71e-05 2.71e-05 0.01% makeSFC 30 2.33e-05 2.33e-05 2.33e-05 0.01% Castro::finalize_do_advance() 5 1.76e-05 1.76e-05 1.76e-05 0.00% Amr::initSubcycle() 1 1.003e-05 1.003e-05 1.003e-05 0.00% DistributionMapping::Distribute() 31 9.77e-06 9.77e-06 9.77e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 7.447e-06 7.447e-06 7.447e-06 0.00% MLMG::buildFineMask() 6 2.86e-06 2.86e-06 2.86e-06 0.00% Castro::retry_advance_ctu() 5 2.584e-06 2.584e-06 2.584e-06 0.00% Gravity::set_mass_offset() 6 2.58e-06 2.58e-06 2.58e-06 0.00% Castro::FluxRegCrseInit 5 1.859e-06 1.859e-06 1.859e-06 0.00% Castro::FluxRegFineAdd() 5 1.25e-06 1.25e-06 1.25e-06 0.00% AmrLevel::AmrLevel() 1 1.25e-06 1.25e-06 1.25e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.096e-06 1.096e-06 1.096e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.06-33-g6f72de283c38) finalized