Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-10-gf84c7a8f77d6) initialized Starting run at 08:36:22 UTC on 2022-10-13. Successfully read inputs file ... Castro git describe: 22.09-1-g65b273ad0 AMReX git describe: 22.10-10-gf84c7a8f7 Microphysics git describe: 22.10-4-g1dbcf8c2 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.05360108 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.03085779 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.049537475 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050687953 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.051788713 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.056226916 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.070846868 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.049391941 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.071387208 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051357486 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.051171419 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.05107432 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.064302462 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.049519133 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.030656004 seconds Ending run at 08:36:23 UTC on 2022-10-13. Run time = 0.836297512 Run time without initialization = 0.698603282 Average number of zones advanced per microsecond: 3.752 Average number of zones advanced per microsecond per rank: 3.752 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8363 ... 0.8363 ... 0.8363 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2060 0.2060 0.2060 24.63% Castro::construct_ctu_hydro_source() 10 0.1751 0.1751 0.1751 20.93% MLCellLinOp::applyBC() 4433 0.08106 0.08106 0.08106 9.69% MLPoisson::Fsmooth() 3280 0.0649 0.0649 0.0649 7.76% MLCGSolver::bicgstab 82 0.02428 0.02428 0.02428 2.90% StateData::FillBoundary(geom) 328 0.02375 0.02375 0.02375 2.84% MultiFab::Dot() 1114 0.0228 0.0228 0.0228 2.73% Castro::normalize_species() 62 0.0157 0.0157 0.0157 1.88% MultiFab::LinComb() 1586 0.01466 0.01466 0.01466 1.75% FabArray::setVal() 1144 0.01445 0.01445 0.01445 1.73% Castro::computeTemp() 63 0.01426 0.01426 0.01426 1.70% FillBoundary_nowait() 4023 0.01423 0.01423 0.01423 1.70% StateDataPhysBCFunct::() 41 0.01363 0.01363 0.01363 1.63% FabArray::ParallelCopy_nowait() 861 0.01341 0.01341 0.01341 1.60% MLPoisson::Fapply() 1142 0.01199 0.01199 0.01199 1.43% MLCellLinOp::defineAuxData() 11 0.01184 0.01184 0.01184 1.42% Gravity::fill_multipole_BCs() 11 0.01024 0.01024 0.01024 1.22% Castro::enforce_min_density() 62 0.008976 0.008976 0.008976 1.07% MLMG::addInterpCorrection() 410 0.007793 0.007793 0.007793 0.93% amrex::average_down 410 0.006925 0.006925 0.006925 0.83% MultiFab::Xpay() 585 0.006671 0.006671 0.006671 0.80% Castro::estTimeStep() 21 0.006285 0.006285 0.006285 0.75% Amr::checkPoint() 3 0.005183 0.005183 0.005183 0.62% Castro::do_advance_ctu() 10 0.005094 0.005094 0.005094 0.61% BndryData::define() 11 0.004002 0.004002 0.004002 0.48% Castro::reset_internal_energy(MultiFab) 63 0.003936 0.003936 0.003936 0.47% Castro::construct_new_gravity_source() 10 0.003307 0.003307 0.003307 0.40% Castro::construct_old_gravity_source() 10 0.002946 0.002946 0.002946 0.35% Amr::writePlotFile() 2 0.002891 0.002891 0.002891 0.35% MLMG::ResNormInf() 93 0.002066 0.002066 0.002066 0.25% Gravity::get_new_grav_vector() 11 0.00193 0.00193 0.00193 0.23% MultiFab::Saxpy() 20 0.001818 0.001818 0.001818 0.22% Gravity::get_old_grav_vector() 10 0.001749 0.001749 0.001749 0.21% Castro::expand_state() 10 0.001734 0.001734 0.001734 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001693 0.001693 0.001693 0.20% MultiFab::Add() 82 0.001656 0.001656 0.001656 0.20% Castro::reset_internal_energy(Fab) 504 0.001602 0.001602 0.001602 0.19% MLCellLinOp::setLevelBC() 11 0.001588 0.001588 0.001588 0.19% Gravity::actual_solve_with_mlmg() 11 0.001435 0.001435 0.001435 0.17% Castro::enforce_speed_limit() 62 0.001412 0.001412 0.001412 0.17% FabArray::mult() 43 0.001342 0.001342 0.001342 0.16% FabArray::setDomainBndry() 41 0.00131 0.00131 0.00131 0.16% Castro::initData() 1 0.001276 0.001276 0.001276 0.15% MLMG::prepareForSolve() 11 0.001265 0.001265 0.001265 0.15% MLCellLinOp::prepareForSolve() 11 0.001215 0.001215 0.001215 0.15% MultiFab::contains_nan() 20 0.001173 0.001173 0.001173 0.14% MLCellLinOp::smooth() 1640 0.00104 0.00104 0.00104 0.12% MLCellLinOp::compGrad() 11 0.0009221 0.0009221 0.0009221 0.11% FabArray::FillBoundary() 4023 0.0008376 0.0008376 0.0008376 0.10% FabArrayBase::getCPC() 1323 0.0007795 0.0007795 0.0007795 0.09% FabArrayBase::CPC::define() 454 0.0006906 0.0006906 0.0006906 0.08% FabArrayBase::getFB() 4023 0.0006067 0.0006067 0.0006067 0.07% Amr::InitAmr() 1 0.0005115 0.0005115 0.0005115 0.06% Gravity::solve_for_phi() 10 0.0004682 0.0004682 0.0004682 0.06% MLCellLinOp::apply() 1142 0.0004596 0.0004596 0.0004596 0.05% Gravity::update_max_rhs() 11 0.0004197 0.0004197 0.0004197 0.05% Amr::coarseTimeStep() 10 0.0003745 0.0003745 0.0003745 0.04% CGSolver::sxay() 1586 0.0003471 0.0003471 0.0003471 0.04% MultiFab::Copy() 11 0.0003349 0.0003349 0.0003349 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002968 0.0002968 0.0002968 0.04% FillPatchIterator::Initialize 41 0.0002955 0.0002955 0.0002955 0.04% MLCellLinOp::defineBC() 11 0.0002849 0.0002849 0.0002849 0.03% main() 1 0.0002786 0.0002786 0.0002786 0.03% MultiFab::max() 11 0.0002635 0.0002635 0.0002635 0.03% FabArray::ParallelCopy() 861 0.0002632 0.0002632 0.0002632 0.03% Castro::construct_new_source() 50 0.000235 0.000235 0.000235 0.03% MLCellLinOp::correctionResidual() 492 0.0002246 0.0002246 0.0002246 0.03% MLMG::MLRhsNormInf() 11 0.0002184 0.0002184 0.0002184 0.03% MLMG::mgVcycle() 82 0.0002088 0.0002088 0.0002088 0.02% Castro::construct_new_gravity() 10 0.0002043 0.0002043 0.0002043 0.02% Castro::subcycle_advance_ctu() 10 0.0002002 0.0002002 0.0002002 0.02% MLLinOp::defineGrids() 11 0.0001726 0.0001726 0.0001726 0.02% Amr::timeStep() 10 0.0001578 0.0001578 0.0001578 0.02% MLMG:computeResOfCorrection() 410 0.0001535 0.0001535 0.0001535 0.02% StateData::checkPoint() 12 0.0001255 0.0001255 0.0001255 0.02% MLMG::mgVcycle_down::0 82 0.0001111 0.0001111 0.0001111 0.01% Castro::Castro() 1 9.98e-05 9.98e-05 9.98e-05 0.01% MLMG::mgVcycle_down::1 82 9.705e-05 9.705e-05 9.705e-05 0.01% MLMG::mgVcycle_down::2 82 9.259e-05 9.259e-05 9.259e-05 0.01% Castro::initialize_advance() 10 8.97e-05 8.97e-05 8.97e-05 0.01% MLMG::mgVcycle_down::3 82 8.697e-05 8.697e-05 8.697e-05 0.01% MLMG::mgVcycle_down::4 82 8.641e-05 8.641e-05 8.641e-05 0.01% FabArrayBase::FB::FB() 56 8.518e-05 8.518e-05 8.518e-05 0.01% MLMG::actualBottomSolve() 82 8.212e-05 8.212e-05 8.212e-05 0.01% Castro::clean_state() 62 7.988e-05 7.988e-05 7.988e-05 0.01% AmrLevel::checkPoint() 3 7.005e-05 7.005e-05 7.005e-05 0.01% MLMG::solve() 11 6.869e-05 6.869e-05 6.869e-05 0.01% MLMG::mgVcycle_up::4 82 6.848e-05 6.848e-05 6.848e-05 0.01% MLMG::oneIter() 82 5.998e-05 5.998e-05 5.998e-05 0.01% Castro::initialize_do_advance() 10 5.907e-05 5.907e-05 5.907e-05 0.01% MLMG::mgVcycle_up::0 82 5.658e-05 5.658e-05 5.658e-05 0.01% MLMG::mgVcycle_up::1 82 5.578e-05 5.578e-05 5.578e-05 0.01% MLMG::mgVcycle_up::2 82 5.574e-05 5.574e-05 5.574e-05 0.01% MLMG::mgVcycle_up::3 82 5.525e-05 5.525e-05 5.525e-05 0.01% Castro::finalize_advance() 10 5.161e-05 5.161e-05 5.161e-05 0.01% MLCellLinOp::solutionResidual() 93 4.944e-05 4.944e-05 4.944e-05 0.01% StateData::define() 4 4.258e-05 4.258e-05 4.258e-05 0.01% MLMG::computeResidual() 82 4.256e-05 4.256e-05 4.256e-05 0.01% Castro::swap_state_time_levels() 10 3.859e-05 3.859e-05 3.859e-05 0.00% Castro::advance() 10 3.755e-05 3.755e-05 3.755e-05 0.00% Castro::enforce_consistent_e() 1 3.418e-05 3.418e-05 3.418e-05 0.00% Castro::finalize_do_advance() 10 3.366e-05 3.366e-05 3.366e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.288e-05 3.288e-05 3.288e-05 0.00% MLMG::mgVcycle_bottom 82 3.255e-05 3.255e-05 3.255e-05 0.00% Gravity::actual_multilevel_solve() 1 3.182e-05 3.182e-05 3.182e-05 0.00% Castro::post_timestep() 10 3.058e-05 3.058e-05 3.058e-05 0.00% Castro::initMFs() 1 2.855e-05 2.855e-05 2.855e-05 0.00% FillPatchSingleLevel 41 2.74e-05 2.74e-05 2.74e-05 0.00% Amr::writeSmallPlotFile() 1 2.582e-05 2.582e-05 2.582e-05 0.00% makeSFC 55 2.541e-05 2.541e-05 2.541e-05 0.00% Castro::buildMetrics() 1 2.403e-05 2.403e-05 2.403e-05 0.00% MLPoisson::define() 11 2.355e-05 2.355e-05 2.355e-05 0.00% MLLinOp::define() 11 2.256e-05 2.256e-05 2.256e-05 0.00% Amr::defBaseLevel() 1 2.242e-05 2.242e-05 2.242e-05 0.00% Amr::FinalizeInit() 1 1.905e-05 1.905e-05 1.905e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.77e-05 1.77e-05 1.77e-05 0.00% Castro::do_new_sources() 10 1.69e-05 1.69e-05 1.69e-05 0.00% Castro::construct_old_source() 50 1.647e-05 1.647e-05 1.647e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.52e-05 1.52e-05 1.52e-05 0.00% Castro::do_old_sources() 10 1.506e-05 1.506e-05 1.506e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.438e-05 1.438e-05 1.438e-05 0.00% DistributionMapping::Distribute() 56 1.354e-05 1.354e-05 1.354e-05 0.00% Castro::check_for_nan() 20 1.184e-05 1.184e-05 1.184e-05 0.00% Castro::apply_source_to_state() 20 1.094e-05 1.094e-05 1.094e-05 0.00% Castro::construct_old_gravity() 10 9.752e-06 9.752e-06 9.752e-06 0.00% MLMG::computeMLResidual() 11 9.661e-06 9.661e-06 9.661e-06 0.00% Amr::initSubcycle() 1 9.306e-06 9.306e-06 9.306e-06 0.00% MLPoisson::prepareForSolve() 11 8.624e-06 8.624e-06 8.624e-06 0.00% Gravity::swapTimeLevels() 10 7.837e-06 7.837e-06 7.837e-06 0.00% MLMG::getGradSolution() 11 6.582e-06 6.582e-06 6.582e-06 0.00% Castro::computeNewDt() 9 6.018e-06 6.018e-06 6.018e-06 0.00% AmrLevel::checkPointPost() 3 5.248e-06 5.248e-06 5.248e-06 0.00% Castro::create_source_corrector() 10 4.816e-06 4.816e-06 4.816e-06 0.00% Amr::InitializeInit() 1 4.747e-06 4.747e-06 4.747e-06 0.00% Gravity::set_mass_offset() 11 4.244e-06 4.244e-06 4.244e-06 0.00% Castro::retry_advance_ctu() 10 3.716e-06 3.716e-06 3.716e-06 0.00% MLMG::MLResNormInf() 11 3.691e-06 3.691e-06 3.691e-06 0.00% Castro::post_init() 1 3.505e-06 3.505e-06 3.505e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.853e-06 2.853e-06 2.853e-06 0.00% Castro::computeInitialDt() 2 2.821e-06 2.821e-06 2.821e-06 0.00% Castro::FluxRegCrseInit 10 2.678e-06 2.678e-06 2.678e-06 0.00% Amr::init() 1 2.656e-06 2.656e-06 2.656e-06 0.00% Castro::FluxRegFineAdd() 10 2.445e-06 2.445e-06 2.445e-06 0.00% AmrLevel::checkPointPre() 3 2.074e-06 2.074e-06 2.074e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.926e-06 1.926e-06 1.926e-06 0.00% Castro::post_regrid() 1 1.269e-06 1.269e-06 1.269e-06 0.00% Amr::initialInit() 1 1.128e-06 1.128e-06 1.128e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8363 0.8363 0.8363 100.00% Amr::coarseTimeStep() 10 0.6677 0.6677 0.6677 79.84% Amr::timeStep() 10 0.5645 0.5645 0.5645 67.50% Castro::advance() 10 0.5564 0.5564 0.5564 66.53% Castro::subcycle_advance_ctu() 10 0.5454 0.5454 0.5454 65.22% Castro::do_advance_ctu() 10 0.5452 0.5452 0.5452 65.19% Gravity::solve_phi_with_mlmg() 11 0.3208 0.3208 0.3208 38.36% Gravity::actual_solve_with_mlmg() 11 0.3103 0.3103 0.3103 37.11% Castro::construct_new_gravity() 10 0.292 0.292 0.292 34.92% MLMG::solve() 11 0.2871 0.2871 0.2871 34.33% Gravity::solve_for_phi() 10 0.2765 0.2765 0.2765 33.06% MLMG::oneIter() 82 0.272 0.272 0.272 32.53% MLMG::mgVcycle() 82 0.2703 0.2703 0.2703 32.32% VisMF::Write(FabArray) 11 0.206 0.206 0.206 24.63% Castro::construct_ctu_hydro_source() 10 0.1751 0.1751 0.1751 20.93% Amr::checkPoint() 3 0.1526 0.1526 0.1526 18.25% AmrLevel::checkPoint() 3 0.1474 0.1474 0.1474 17.63% StateData::checkPoint() 12 0.1474 0.1474 0.1474 17.62% MLCellLinOp::smooth() 1640 0.138 0.138 0.138 16.50% Amr::init() 1 0.137 0.137 0.137 16.39% MLCellLinOp::applyBC() 4433 0.09682 0.09682 0.09682 11.58% MLMG::mgVcycle_bottom 82 0.08335 0.08335 0.08335 9.97% MLMG::actualBottomSolve() 82 0.08332 0.08332 0.08332 9.96% MLCGSolver::bicgstab 82 0.08249 0.08249 0.08249 9.86% MLPoisson::Fsmooth() 3280 0.0649 0.0649 0.0649 7.76% Amr::writePlotFile() 2 0.06165 0.06165 0.06165 7.37% Amr::initialInit() 1 0.05245 0.05245 0.05245 6.27% Amr::FinalizeInit() 1 0.04809 0.04809 0.04809 5.75% Castro::post_init() 1 0.04664 0.04664 0.04664 5.58% Castro::clean_state() 62 0.04501 0.04501 0.04501 5.38% Gravity::multilevel_solve_for_new_phi() 1 0.04485 0.04485 0.04485 5.36% Gravity::actual_multilevel_solve() 1 0.04483 0.04483 0.04483 5.36% FillPatchIterator::Initialize 41 0.04302 0.04302 0.04302 5.14% FillPatchSingleLevel 41 0.04141 0.04141 0.04141 4.95% StateDataPhysBCFunct::() 41 0.03738 0.03738 0.03738 4.47% MLCellLinOp::apply() 1142 0.03674 0.03674 0.03674 4.39% MLMG::mgVcycle_down::0 82 0.03574 0.03574 0.03574 4.27% MLMG::mgVcycle_up::0 82 0.03072 0.03072 0.03072 3.67% StateData::FillBoundary(geom) 328 0.02375 0.02375 0.02375 2.84% MultiFab::Dot() 1114 0.0228 0.0228 0.0228 2.73% MLCellLinOp::correctionResidual() 492 0.02154 0.02154 0.02154 2.58% Castro::initialize_do_advance() 10 0.02045 0.02045 0.02045 2.44% Castro::computeTemp() 63 0.0198 0.0198 0.0198 2.37% MLPoisson::define() 11 0.01874 0.01874 0.01874 2.24% MLMG:computeResOfCorrection() 410 0.0186 0.0186 0.0186 2.22% MLMG::mgVcycle_down::1 82 0.01806 0.01806 0.01806 2.16% MLMG::mgVcycle_down::2 82 0.01755 0.01755 0.01755 2.10% Gravity::get_new_grav_vector() 11 0.01704 0.01704 0.01704 2.04% MLMG::mgVcycle_down::3 82 0.01662 0.01662 0.01662 1.99% MLMG::mgVcycle_down::4 82 0.0159 0.0159 0.0159 1.90% FabArray::FillBoundary() 4023 0.01576 0.01576 0.01576 1.88% Castro::normalize_species() 62 0.0157 0.0157 0.0157 1.88% CGSolver::sxay() 1586 0.01501 0.01501 0.01501 1.79% Castro::construct_old_gravity() 10 0.01498 0.01498 0.01498 1.79% Gravity::get_old_grav_vector() 10 0.01497 0.01497 0.01497 1.79% FillBoundary_nowait() 4023 0.01492 0.01492 0.01492 1.78% MultiFab::LinComb() 1586 0.01466 0.01466 0.01466 1.75% FabArray::ParallelCopy() 861 0.0145 0.0145 0.0145 1.73% FabArray::setVal() 1144 0.01445 0.01445 0.01445 1.73% FabArray::ParallelCopy_nowait() 861 0.01423 0.01423 0.01423 1.70% MLCGSolver::ParallelAllReduce 1514 0.01358 0.01358 0.01358 1.62% MLMG::mgVcycle_up::2 82 0.01348 0.01348 0.01348 1.61% MLMG::mgVcycle_up::1 82 0.01327 0.01327 0.01327 1.59% MLCellLinOp::defineAuxData() 11 0.01322 0.01322 0.01322 1.58% MLMG::addInterpCorrection() 410 0.013 0.013 0.013 1.55% MLMG::mgVcycle_up::3 82 0.0128 0.0128 0.0128 1.53% MLMG::mgVcycle_up::4 82 0.01262 0.01262 0.01262 1.51% Castro::expand_state() 10 0.01234 0.01234 0.01234 1.48% amrex::average_down 410 0.0122 0.0122 0.0122 1.46% Castro::do_new_sources() 10 0.01209 0.01209 0.01209 1.45% MLPoisson::Fapply() 1142 0.01199 0.01199 0.01199 1.43% Castro::initialize_advance() 10 0.0109 0.0109 0.0109 1.30% Castro::do_old_sources() 10 0.01088 0.01088 0.01088 1.30% Gravity::fill_multipole_BCs() 11 0.01024 0.01024 0.01024 1.22% Castro::enforce_min_density() 62 0.008976 0.008976 0.008976 1.07% Castro::post_timestep() 10 0.007975 0.007975 0.007975 0.95% MLCellLinOp::solutionResidual() 93 0.007185 0.007185 0.007185 0.86% MultiFab::Xpay() 585 0.006671 0.006671 0.006671 0.80% Castro::estTimeStep() 21 0.006285 0.006285 0.006285 0.75% MLMG::computeResidual() 82 0.006206 0.006206 0.006206 0.74% Castro::reset_internal_energy(MultiFab) 63 0.005539 0.005539 0.005539 0.66% MLMG::prepareForSolve() 11 0.005509 0.005509 0.005509 0.66% MLCellLinOp::defineBC() 11 0.005251 0.005251 0.005251 0.63% BndryData::define() 11 0.004966 0.004966 0.004966 0.59% Amr::InitializeInit() 1 0.004365 0.004365 0.004365 0.52% Amr::defBaseLevel() 1 0.00436 0.00436 0.00436 0.52% Castro::initData() 1 0.003819 0.003819 0.003819 0.46% Castro::construct_new_source() 50 0.003542 0.003542 0.003542 0.42% Castro::computeNewDt() 9 0.003353 0.003353 0.003353 0.40% Castro::construct_new_gravity_source() 10 0.003307 0.003307 0.003307 0.40% Castro::construct_old_source() 50 0.002962 0.002962 0.002962 0.35% Castro::construct_old_gravity_source() 10 0.002946 0.002946 0.002946 0.35% MLMG::ResNormInf() 93 0.002066 0.002066 0.002066 0.25% Castro::apply_source_to_state() 20 0.001829 0.001829 0.001829 0.22% MultiFab::Saxpy() 20 0.001818 0.001818 0.001818 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001693 0.001693 0.001693 0.20% MultiFab::Add() 82 0.001656 0.001656 0.001656 0.20% Castro::reset_internal_energy(Fab) 504 0.001602 0.001602 0.001602 0.19% MLCellLinOp::setLevelBC() 11 0.001588 0.001588 0.001588 0.19% FabArrayBase::getCPC() 1323 0.00147 0.00147 0.00147 0.18% MLMG::getGradSolution() 11 0.001427 0.001427 0.001427 0.17% MLCellLinOp::compGrad() 11 0.001421 0.001421 0.001421 0.17% Castro::enforce_speed_limit() 62 0.001412 0.001412 0.001412 0.17% FabArray::mult() 43 0.001342 0.001342 0.001342 0.16% FabArray::setDomainBndry() 41 0.00131 0.00131 0.00131 0.16% MLPoisson::prepareForSolve() 11 0.001224 0.001224 0.001224 0.15% MLCellLinOp::prepareForSolve() 11 0.001215 0.001215 0.001215 0.15% Castro::post_regrid() 1 0.001205 0.001205 0.001205 0.14% Castro::check_for_nan() 20 0.001185 0.001185 0.001185 0.14% MultiFab::contains_nan() 20 0.001173 0.001173 0.001173 0.14% MLMG::computeMLResidual() 11 0.001032 0.001032 0.001032 0.12% Gravity::update_max_rhs() 11 0.0008426 0.0008426 0.0008426 0.10% Castro::computeInitialDt() 2 0.0006924 0.0006924 0.0006924 0.08% FabArrayBase::getFB() 4023 0.0006919 0.0006919 0.0006919 0.08% FabArrayBase::CPC::define() 454 0.0006906 0.0006906 0.0006906 0.08% Amr::InitAmr() 1 0.0005208 0.0005208 0.0005208 0.06% Castro::Castro() 1 0.0004575 0.0004575 0.0004575 0.05% Gravity::swapTimeLevels() 10 0.0004477 0.0004477 0.0004477 0.05% MultiFab::Copy() 11 0.0003349 0.0003349 0.0003349 0.04% MLMG::MLResNormInf() 11 0.0002806 0.0002806 0.0002806 0.03% MultiFab::max() 11 0.0002635 0.0002635 0.0002635 0.03% MLLinOp::define() 11 0.0002499 0.0002499 0.0002499 0.03% MLLinOp::defineGrids() 11 0.0002273 0.0002273 0.0002273 0.03% MLMG::MLRhsNormInf() 11 0.0002184 0.0002184 0.0002184 0.03% Castro::buildMetrics() 1 0.0001631 0.0001631 0.0001631 0.02% FabArrayBase::FB::FB() 56 8.518e-05 8.518e-05 8.518e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.696e-05 5.696e-05 5.696e-05 0.01% Castro::finalize_advance() 10 5.673e-05 5.673e-05 5.673e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.281e-05 5.281e-05 5.281e-05 0.01% StateData::define() 4 4.258e-05 4.258e-05 4.258e-05 0.01% Castro::swap_state_time_levels() 10 3.859e-05 3.859e-05 3.859e-05 0.00% makeSFC 55 3.761e-05 3.761e-05 3.761e-05 0.00% Castro::enforce_consistent_e() 1 3.418e-05 3.418e-05 3.418e-05 0.00% Castro::finalize_do_advance() 10 3.366e-05 3.366e-05 3.366e-05 0.00% Castro::initMFs() 1 2.855e-05 2.855e-05 2.855e-05 0.00% Amr::writeSmallPlotFile() 1 2.582e-05 2.582e-05 2.582e-05 0.00% DistributionMapping::Distribute() 56 1.354e-05 1.354e-05 1.354e-05 0.00% Amr::initSubcycle() 1 9.306e-06 9.306e-06 9.306e-06 0.00% AmrLevel::checkPointPost() 3 5.248e-06 5.248e-06 5.248e-06 0.00% Castro::create_source_corrector() 10 4.816e-06 4.816e-06 4.816e-06 0.00% Gravity::set_mass_offset() 11 4.244e-06 4.244e-06 4.244e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.196e-06 4.196e-06 4.196e-06 0.00% Castro::retry_advance_ctu() 10 3.716e-06 3.716e-06 3.716e-06 0.00% Castro::FluxRegCrseInit 10 2.678e-06 2.678e-06 2.678e-06 0.00% Castro::FluxRegFineAdd() 10 2.445e-06 2.445e-06 2.445e-06 0.00% AmrLevel::checkPointPre() 3 2.074e-06 2.074e-06 2.074e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.926e-06 1.926e-06 1.926e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-10-gf84c7a8f77d6) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-10-gf84c7a8f77d6) initialized Starting run at 08:36:23 UTC on 2022-10-13. Successfully read inputs file ... Castro git describe: 22.09-1-g65b273ad0 AMReX git describe: 22.10-10-gf84c7a8f7 Microphysics git describe: 22.10-4-g1dbcf8c2 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.466208984 Restart time = 0.049006219 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052349251 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050275547 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057054011 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.06333326 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065098813 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.045618774 seconds Ending run at 08:36:24 UTC on 2022-10-13. Run time = 0.383772405 Run time without initialization = 0.334174765 Average number of zones advanced per microsecond: 3.922 Average number of zones advanced per microsecond per rank: 3.922 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3838 ... 0.3838 ... 0.3838 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0892 0.0892 0.0892 23.25% VisMF::Write(FabArray) 1 0.044 0.044 0.044 11.46% VisMF::Read() 3 0.0411 0.0411 0.0411 10.71% MLCellLinOp::applyBC() 1946 0.03485 0.03485 0.03485 9.08% MLPoisson::Fsmooth() 1440 0.02735 0.02735 0.02735 7.13% StateData::FillBoundary(geom) 160 0.01159 0.01159 0.01159 3.02% MLCGSolver::bicgstab 36 0.01027 0.01027 0.01027 2.68% MultiFab::Dot() 484 0.009632 0.009632 0.009632 2.51% Castro::computeTemp() 30 0.007255 0.007255 0.007255 1.89% Castro::normalize_species() 30 0.006979 0.006979 0.006979 1.82% Castro::enforce_min_density() 30 0.00692 0.00692 0.00692 1.80% FabArray::setVal() 537 0.006759 0.006759 0.006759 1.76% FillBoundary_nowait() 1766 0.006254 0.006254 0.006254 1.63% MLCellLinOp::defineAuxData() 6 0.006221 0.006221 0.006221 1.62% StateDataPhysBCFunct::() 20 0.00616 0.00616 0.00616 1.61% MultiFab::LinComb() 690 0.00615 0.00615 0.00615 1.60% Gravity::fill_multipole_BCs() 6 0.006016 0.006016 0.006016 1.57% FabArray::ParallelCopy_nowait() 380 0.006006 0.006006 0.006006 1.56% MLPoisson::Fapply() 500 0.005103 0.005103 0.005103 1.33% Amr::restart() 1 0.003636 0.003636 0.003636 0.95% MLMG::addInterpCorrection() 180 0.003332 0.003332 0.003332 0.87% amrex::average_down 180 0.002972 0.002972 0.002972 0.77% MultiFab::Xpay() 258 0.002875 0.002875 0.002875 0.75% Castro::estTimeStep() 10 0.002685 0.002685 0.002685 0.70% Castro::do_advance_ctu() 5 0.002277 0.002277 0.002277 0.59% BndryData::define() 6 0.002151 0.002151 0.002151 0.56% Amr::writePlotFile() 1 0.001736 0.001736 0.001736 0.45% Castro::reset_internal_energy(MultiFab) 30 0.001682 0.001682 0.001682 0.44% Castro::construct_new_gravity_source() 5 0.001643 0.001643 0.001643 0.43% Castro::construct_old_gravity_source() 5 0.001344 0.001344 0.001344 0.35% MultiFab::Saxpy() 10 0.0009182 0.0009182 0.0009182 0.24% MLMG::ResNormInf() 42 0.0009127 0.0009127 0.0009127 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009068 0.0009068 0.0009068 0.24% Gravity::get_old_grav_vector() 5 0.0008951 0.0008951 0.0008951 0.23% Castro::expand_state() 5 0.0008744 0.0008744 0.0008744 0.23% Gravity::get_new_grav_vector() 5 0.0008689 0.0008689 0.0008689 0.23% Castro::enforce_speed_limit() 30 0.0008462 0.0008462 0.0008462 0.22% MLCellLinOp::setLevelBC() 6 0.000822 0.000822 0.000822 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007422 0.0007422 0.0007422 0.19% Castro::reset_internal_energy(Fab) 240 0.0007421 0.0007421 0.0007421 0.19% MultiFab::Add() 36 0.0007161 0.0007161 0.0007161 0.19% FabArray::mult() 22 0.0006561 0.0006561 0.0006561 0.17% MLMG::prepareForSolve() 6 0.000651 0.000651 0.000651 0.17% MLCellLinOp::prepareForSolve() 6 0.0006383 0.0006383 0.0006383 0.17% FabArray::setDomainBndry() 20 0.0006334 0.0006334 0.0006334 0.17% MultiFab::contains_nan() 10 0.0005888 0.0005888 0.0005888 0.15% MLCellLinOp::compGrad() 6 0.0004905 0.0004905 0.0004905 0.13% MLCellLinOp::smooth() 720 0.0004748 0.0004748 0.0004748 0.12% Amr::InitAmr() 1 0.0004061 0.0004061 0.0004061 0.11% FabArrayBase::CPC::define() 244 0.0003944 0.0003944 0.0003944 0.10% FabArray::FillBoundary() 1766 0.0003861 0.0003861 0.0003861 0.10% FabArrayBase::getCPC() 632 0.0003674 0.0003674 0.0003674 0.10% main() 1 0.0002683 0.0002683 0.0002683 0.07% FabArrayBase::getFB() 1766 0.0002602 0.0002602 0.0002602 0.07% Gravity::update_max_rhs() 6 0.0002298 0.0002298 0.0002298 0.06% MLCellLinOp::apply() 500 0.0002113 0.0002113 0.0002113 0.06% Gravity::solve_for_phi() 5 0.000206 0.000206 0.000206 0.05% MultiFab::Copy() 6 0.0001795 0.0001795 0.0001795 0.05% CGSolver::sxay() 690 0.0001774 0.0001774 0.0001774 0.05% Amr::coarseTimeStep() 5 0.0001605 0.0001605 0.0001605 0.04% Castro::subcycle_advance_ctu() 5 0.0001525 0.0001525 0.0001525 0.04% Castro::construct_new_source() 25 0.0001474 0.0001474 0.0001474 0.04% MLCellLinOp::defineBC() 6 0.0001443 0.0001443 0.0001443 0.04% MultiFab::max() 6 0.0001371 0.0001371 0.0001371 0.04% FillPatchIterator::Initialize 20 0.0001362 0.0001362 0.0001362 0.04% MLCGSolver::ParallelAllReduce 659 0.0001352 0.0001352 0.0001352 0.04% FabArray::ParallelCopy() 380 0.0001238 0.0001238 0.0001238 0.03% Castro::construct_new_gravity() 5 0.0001147 0.0001147 0.0001147 0.03% MLMG::MLRhsNormInf() 6 0.0001125 0.0001125 0.0001125 0.03% MLCellLinOp::correctionResidual() 216 9.989e-05 9.989e-05 9.989e-05 0.03% MLMG::mgVcycle() 36 9.353e-05 9.353e-05 9.353e-05 0.02% MLLinOp::defineGrids() 6 9.078e-05 9.078e-05 9.078e-05 0.02% AmrLevel::restart() 1 8.289e-05 8.289e-05 8.289e-05 0.02% Amr::timeStep() 5 8.13e-05 8.13e-05 8.13e-05 0.02% StateData::restartDoit() 4 7.724e-05 7.724e-05 7.724e-05 0.02% MLMG:computeResOfCorrection() 180 7.242e-05 7.242e-05 7.242e-05 0.02% FabArrayBase::FB::FB() 26 5.644e-05 5.644e-05 5.644e-05 0.01% Castro::finalize_advance() 5 5.405e-05 5.405e-05 5.405e-05 0.01% MLMG::mgVcycle_down::0 36 4.693e-05 4.693e-05 4.693e-05 0.01% MLMG::mgVcycle_down::1 36 4.368e-05 4.368e-05 4.368e-05 0.01% MLMG::mgVcycle_down::2 36 4.066e-05 4.066e-05 4.066e-05 0.01% Castro::initialize_advance() 5 3.996e-05 3.996e-05 3.996e-05 0.01% Castro::clean_state() 30 3.847e-05 3.847e-05 3.847e-05 0.01% MLMG::mgVcycle_down::4 36 3.821e-05 3.821e-05 3.821e-05 0.01% MLMG::mgVcycle_down::3 36 3.741e-05 3.741e-05 3.741e-05 0.01% MLMG::actualBottomSolve() 36 3.594e-05 3.594e-05 3.594e-05 0.01% MLMG::mgVcycle_up::4 36 3.378e-05 3.378e-05 3.378e-05 0.01% MLMG::solve() 6 3.324e-05 3.324e-05 3.324e-05 0.01% Castro::buildMetrics() 1 3.298e-05 3.298e-05 3.298e-05 0.01% Castro::post_restart() 1 3.106e-05 3.106e-05 3.106e-05 0.01% Castro::advance() 5 3.094e-05 3.094e-05 3.094e-05 0.01% Gravity::actual_multilevel_solve() 1 3.037e-05 3.037e-05 3.037e-05 0.01% Castro::initMFs() 1 3.035e-05 3.035e-05 3.035e-05 0.01% Castro::initialize_do_advance() 5 3.012e-05 3.012e-05 3.012e-05 0.01% Amr::writeSmallPlotFile() 1 2.986e-05 2.986e-05 2.986e-05 0.01% Castro::swap_state_time_levels() 5 2.797e-05 2.797e-05 2.797e-05 0.01% MLMG::oneIter() 36 2.792e-05 2.792e-05 2.792e-05 0.01% MLMG::mgVcycle_up::0 36 2.654e-05 2.654e-05 2.654e-05 0.01% MLMG::mgVcycle_up::3 36 2.648e-05 2.648e-05 2.648e-05 0.01% Castro::construct_old_source() 25 2.598e-05 2.598e-05 2.598e-05 0.01% MLMG::mgVcycle_up::2 36 2.516e-05 2.516e-05 2.516e-05 0.01% MLCellLinOp::solutionResidual() 42 2.415e-05 2.415e-05 2.415e-05 0.01% MLMG::mgVcycle_up::1 36 2.333e-05 2.333e-05 2.333e-05 0.01% Castro::finalize_do_advance() 5 2.047e-05 2.047e-05 2.047e-05 0.01% MLLinOp::define() 6 2.031e-05 2.031e-05 2.031e-05 0.01% MLMG::computeResidual() 36 1.873e-05 1.873e-05 1.873e-05 0.00% Castro::create_source_corrector() 5 1.839e-05 1.839e-05 1.839e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.769e-05 1.769e-05 1.769e-05 0.00% MLPoisson::define() 6 1.622e-05 1.622e-05 1.622e-05 0.00% MLMG::mgVcycle_bottom 36 1.59e-05 1.59e-05 1.59e-05 0.00% makeSFC 30 1.44e-05 1.44e-05 1.44e-05 0.00% FillPatchSingleLevel 20 1.42e-05 1.42e-05 1.42e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.419e-05 1.419e-05 1.419e-05 0.00% Castro::construct_old_gravity() 5 1.163e-05 1.163e-05 1.163e-05 0.00% Amr::initSubcycle() 1 8.712e-06 8.712e-06 8.712e-06 0.00% Castro::do_new_sources() 5 8.59e-06 8.59e-06 8.59e-06 0.00% Castro::do_old_sources() 5 8.441e-06 8.441e-06 8.441e-06 0.00% DistributionMapping::Distribute() 31 8.325e-06 8.325e-06 8.325e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.99e-06 6.99e-06 6.99e-06 0.00% Castro::check_for_nan() 10 6.364e-06 6.364e-06 6.364e-06 0.00% Castro::apply_source_to_state() 10 5.837e-06 5.837e-06 5.837e-06 0.00% Castro::post_timestep() 5 5.148e-06 5.148e-06 5.148e-06 0.00% MLMG::computeMLResidual() 6 4.909e-06 4.909e-06 4.909e-06 0.00% MLPoisson::prepareForSolve() 6 4.741e-06 4.741e-06 4.741e-06 0.00% Gravity::swapTimeLevels() 5 4.21e-06 4.21e-06 4.21e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.404e-06 3.404e-06 3.404e-06 0.00% MLMG::getGradSolution() 6 3.232e-06 3.232e-06 3.232e-06 0.00% Castro::computeNewDt() 5 2.867e-06 2.867e-06 2.867e-06 0.00% MLMG::MLResNormInf() 6 2.304e-06 2.304e-06 2.304e-06 0.00% Gravity::set_mass_offset() 6 1.926e-06 1.926e-06 1.926e-06 0.00% Castro::retry_advance_ctu() 5 1.797e-06 1.797e-06 1.797e-06 0.00% Castro::FluxRegCrseInit 5 1.784e-06 1.784e-06 1.784e-06 0.00% Castro::FluxRegFineAdd() 5 1.291e-06 1.291e-06 1.291e-06 0.00% Amr::init() 1 1.11e-06 1.11e-06 1.11e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.056e-06 1.056e-06 1.056e-06 0.00% AmrLevel::AmrLevel() 1 8.22e-07 8.22e-07 8.22e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3838 0.3838 0.3838 100.00% Amr::coarseTimeStep() 5 0.2883 0.2883 0.2883 75.11% Amr::timeStep() 5 0.2865 0.2865 0.2865 74.65% Castro::advance() 5 0.2831 0.2831 0.2831 73.76% Castro::subcycle_advance_ctu() 5 0.2772 0.2772 0.2772 72.22% Castro::do_advance_ctu() 5 0.277 0.277 0.277 72.18% Castro::construct_new_gravity() 5 0.1457 0.1457 0.1457 37.96% Gravity::solve_phi_with_mlmg() 6 0.1416 0.1416 0.1416 36.88% Gravity::solve_for_phi() 5 0.138 0.138 0.138 35.96% Gravity::actual_solve_with_mlmg() 6 0.1354 0.1354 0.1354 35.28% MLMG::solve() 6 0.1231 0.1231 0.1231 32.08% MLMG::oneIter() 36 0.1159 0.1159 0.1159 30.20% MLMG::mgVcycle() 36 0.1152 0.1152 0.1152 30.00% Castro::construct_ctu_hydro_source() 5 0.08922 0.08922 0.08922 23.25% MLCellLinOp::smooth() 720 0.05879 0.05879 0.05879 15.32% Amr::init() 1 0.04907 0.04907 0.04907 12.79% Amr::restart() 1 0.04907 0.04907 0.04907 12.79% Amr::writePlotFile() 1 0.04573 0.04573 0.04573 11.92% VisMF::Write(FabArray) 1 0.044 0.044 0.044 11.46% MLCellLinOp::applyBC() 1946 0.0418 0.0418 0.0418 10.89% AmrLevel::restart() 1 0.04131 0.04131 0.04131 10.76% StateData::restartDoit() 4 0.04123 0.04123 0.04123 10.74% VisMF::Read() 3 0.0411 0.0411 0.0411 10.71% MLMG::mgVcycle_bottom 36 0.03534 0.03534 0.03534 9.21% MLMG::actualBottomSolve() 36 0.03532 0.03532 0.03532 9.20% MLCGSolver::bicgstab 36 0.03497 0.03497 0.03497 9.11% MLPoisson::Fsmooth() 1440 0.02735 0.02735 0.02735 7.13% Castro::clean_state() 30 0.02446 0.02446 0.02446 6.37% FillPatchIterator::Initialize 20 0.02053 0.02053 0.02053 5.35% FillPatchSingleLevel 20 0.01976 0.01976 0.01976 5.15% StateDataPhysBCFunct::() 20 0.01775 0.01775 0.01775 4.63% MLCellLinOp::apply() 500 0.01588 0.01588 0.01588 4.14% MLMG::mgVcycle_down::0 36 0.01538 0.01538 0.01538 4.01% MLMG::mgVcycle_up::0 36 0.01317 0.01317 0.01317 3.43% Castro::initialize_do_advance() 5 0.01173 0.01173 0.01173 3.06% StateData::FillBoundary(geom) 160 0.01159 0.01159 0.01159 3.02% MLPoisson::define() 6 0.009966 0.009966 0.009966 2.60% Castro::computeTemp() 30 0.009679 0.009679 0.009679 2.52% MultiFab::Dot() 484 0.009632 0.009632 0.009632 2.51% MLCellLinOp::correctionResidual() 216 0.009244 0.009244 0.009244 2.41% MLMG:computeResOfCorrection() 180 0.007976 0.007976 0.007976 2.08% MLMG::mgVcycle_down::1 36 0.007691 0.007691 0.007691 2.00% Gravity::get_new_grav_vector() 5 0.007589 0.007589 0.007589 1.98% MLMG::mgVcycle_down::2 36 0.007416 0.007416 0.007416 1.93% Castro::construct_old_gravity() 5 0.007394 0.007394 0.007394 1.93% Gravity::get_old_grav_vector() 5 0.007382 0.007382 0.007382 1.92% Castro::do_new_sources() 5 0.007159 0.007159 0.007159 1.87% MLMG::mgVcycle_down::3 36 0.007081 0.007081 0.007081 1.84% Castro::normalize_species() 30 0.006979 0.006979 0.006979 1.82% MLCellLinOp::defineAuxData() 6 0.006973 0.006973 0.006973 1.82% FabArray::FillBoundary() 1766 0.006956 0.006956 0.006956 1.81% Castro::enforce_min_density() 30 0.00692 0.00692 0.00692 1.80% MLMG::mgVcycle_down::4 36 0.006762 0.006762 0.006762 1.76% FabArray::setVal() 537 0.006759 0.006759 0.006759 1.76% FillBoundary_nowait() 1766 0.00657 0.00657 0.00657 1.71% FabArray::ParallelCopy() 380 0.006505 0.006505 0.006505 1.69% FabArray::ParallelCopy_nowait() 380 0.006381 0.006381 0.006381 1.66% CGSolver::sxay() 690 0.006328 0.006328 0.006328 1.65% MultiFab::LinComb() 690 0.00615 0.00615 0.00615 1.60% Castro::expand_state() 5 0.006117 0.006117 0.006117 1.59% Gravity::fill_multipole_BCs() 6 0.006016 0.006016 0.006016 1.57% Castro::initialize_advance() 5 0.005829 0.005829 0.005829 1.52% MLCGSolver::ParallelAllReduce 659 0.005781 0.005781 0.005781 1.51% MLMG::mgVcycle_up::2 36 0.005724 0.005724 0.005724 1.49% MLMG::mgVcycle_up::1 36 0.005647 0.005647 0.005647 1.47% MLMG::addInterpCorrection() 180 0.005573 0.005573 0.005573 1.45% MLMG::mgVcycle_up::3 36 0.005449 0.005449 0.005449 1.42% MLMG::mgVcycle_up::4 36 0.005409 0.005409 0.005409 1.41% Castro::do_old_sources() 5 0.005345 0.005345 0.005345 1.39% amrex::average_down 180 0.005242 0.005242 0.005242 1.37% MLPoisson::Fapply() 500 0.005103 0.005103 0.005103 1.33% Castro::post_restart() 1 0.003941 0.003941 0.003941 1.03% Gravity::multilevel_solve_for_new_phi() 1 0.003814 0.003814 0.003814 0.99% Gravity::actual_multilevel_solve() 1 0.003797 0.003797 0.003797 0.99% Castro::post_timestep() 5 0.003344 0.003344 0.003344 0.87% MLCellLinOp::solutionResidual() 42 0.003242 0.003242 0.003242 0.84% MLMG::prepareForSolve() 6 0.002894 0.002894 0.002894 0.75% MultiFab::Xpay() 258 0.002875 0.002875 0.002875 0.75% MLCellLinOp::defineBC() 6 0.002836 0.002836 0.002836 0.74% BndryData::define() 6 0.002692 0.002692 0.002692 0.70% MLMG::computeResidual() 36 0.002692 0.002692 0.002692 0.70% Castro::estTimeStep() 10 0.002685 0.002685 0.002685 0.70% Castro::reset_internal_energy(MultiFab) 30 0.002424 0.002424 0.002424 0.63% Castro::construct_new_source() 25 0.00179 0.00179 0.00179 0.47% Castro::construct_new_gravity_source() 5 0.001643 0.001643 0.001643 0.43% Castro::computeNewDt() 5 0.001577 0.001577 0.001577 0.41% Castro::construct_old_source() 25 0.00137 0.00137 0.00137 0.36% Castro::construct_old_gravity_source() 5 0.001344 0.001344 0.001344 0.35% Castro::apply_source_to_state() 10 0.000924 0.000924 0.000924 0.24% MultiFab::Saxpy() 10 0.0009182 0.0009182 0.0009182 0.24% MLMG::ResNormInf() 42 0.0009127 0.0009127 0.0009127 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009068 0.0009068 0.0009068 0.24% Castro::enforce_speed_limit() 30 0.0008462 0.0008462 0.0008462 0.22% MLCellLinOp::setLevelBC() 6 0.000822 0.000822 0.000822 0.21% MLMG::getGradSolution() 6 0.0007654 0.0007654 0.0007654 0.20% MLCellLinOp::compGrad() 6 0.0007622 0.0007622 0.0007622 0.20% FabArrayBase::getCPC() 632 0.0007618 0.0007618 0.0007618 0.20% Castro::reset_internal_energy(Fab) 240 0.0007421 0.0007421 0.0007421 0.19% MultiFab::Add() 36 0.0007161 0.0007161 0.0007161 0.19% FabArray::mult() 22 0.0006561 0.0006561 0.0006561 0.17% MLPoisson::prepareForSolve() 6 0.0006431 0.0006431 0.0006431 0.17% MLCellLinOp::prepareForSolve() 6 0.0006383 0.0006383 0.0006383 0.17% FabArray::setDomainBndry() 20 0.0006334 0.0006334 0.0006334 0.17% Castro::check_for_nan() 10 0.0005951 0.0005951 0.0005951 0.16% MultiFab::contains_nan() 10 0.0005888 0.0005888 0.0005888 0.15% MLMG::computeMLResidual() 6 0.0005738 0.0005738 0.0005738 0.15% Gravity::update_max_rhs() 6 0.0004471 0.0004471 0.0004471 0.12% Amr::InitAmr() 1 0.0004148 0.0004148 0.0004148 0.11% FabArrayBase::CPC::define() 244 0.0003944 0.0003944 0.0003944 0.10% FabArrayBase::getFB() 1766 0.0003166 0.0003166 0.0003166 0.08% Gravity::swapTimeLevels() 5 0.000228 0.000228 0.000228 0.06% MultiFab::Copy() 6 0.0001795 0.0001795 0.0001795 0.05% Castro::buildMetrics() 1 0.0001516 0.0001516 0.0001516 0.04% MLMG::MLResNormInf() 6 0.0001489 0.0001489 0.0001489 0.04% MLLinOp::define() 6 0.0001406 0.0001406 0.0001406 0.04% MultiFab::max() 6 0.0001371 0.0001371 0.0001371 0.04% MLLinOp::defineGrids() 6 0.0001203 0.0001203 0.0001203 0.03% MLMG::MLRhsNormInf() 6 0.0001125 0.0001125 0.0001125 0.03% Castro::finalize_advance() 5 5.713e-05 5.713e-05 5.713e-05 0.01% FabArrayBase::FB::FB() 26 5.644e-05 5.644e-05 5.644e-05 0.01% Castro::initMFs() 1 3.035e-05 3.035e-05 3.035e-05 0.01% Amr::writeSmallPlotFile() 1 2.986e-05 2.986e-05 2.986e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.842e-05 2.842e-05 2.842e-05 0.01% Castro::swap_state_time_levels() 5 2.797e-05 2.797e-05 2.797e-05 0.01% makeSFC 30 2.143e-05 2.143e-05 2.143e-05 0.01% Castro::finalize_do_advance() 5 2.047e-05 2.047e-05 2.047e-05 0.01% Castro::create_source_corrector() 5 1.839e-05 1.839e-05 1.839e-05 0.00% Amr::initSubcycle() 1 8.712e-06 8.712e-06 8.712e-06 0.00% DistributionMapping::Distribute() 31 8.325e-06 8.325e-06 8.325e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.699e-06 4.699e-06 4.699e-06 0.00% Gravity::set_mass_offset() 6 1.926e-06 1.926e-06 1.926e-06 0.00% Castro::retry_advance_ctu() 5 1.797e-06 1.797e-06 1.797e-06 0.00% Castro::FluxRegCrseInit 5 1.784e-06 1.784e-06 1.784e-06 0.00% Castro::FluxRegFineAdd() 5 1.291e-06 1.291e-06 1.291e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.056e-06 1.056e-06 1.056e-06 0.00% AmrLevel::AmrLevel() 1 8.22e-07 8.22e-07 8.22e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-10-gf84c7a8f77d6) finalized