Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-3-g1bc4e4eb5a25) initialized Starting run at 08:34:46 UTC on 2022-10-04. Successfully read inputs file ... Castro git describe: 22.09-1-g65b273ad0 AMReX git describe: 22.10-3-g1bc4e4eb5 Microphysics git describe: 22.10-2-gbaa4f033 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.053119834 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030629529 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.049589986 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051702274 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.050811228 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.057515443 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.062949145 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.062622769 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.078788403 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.064699778 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.061994687 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.062696404 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.066265965 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.317464103 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.035595672 seconds Ending run at 08:34:48 UTC on 2022-10-04. Run time = 1.160558791 Run time without initialization = 1.023424715 Average number of zones advanced per microsecond: 2.561 Average number of zones advanced per microsecond per rank: 2.561 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 1.161 ... 1.161 ... 1.161 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.4777 0.4777 0.4777 41.16% Castro::construct_ctu_hydro_source() 10 0.2152 0.2152 0.2152 18.54% MLCellLinOp::applyBC() 4433 0.082 0.082 0.082 7.07% MLPoisson::Fsmooth() 3280 0.06526 0.06526 0.06526 5.62% MLCGSolver::bicgstab 82 0.02472 0.02472 0.02472 2.13% StateData::FillBoundary(geom) 328 0.02407 0.02407 0.02407 2.07% MultiFab::Dot() 1114 0.02305 0.02305 0.02305 1.99% Amr::checkPoint() 3 0.01877 0.01877 0.01877 1.62% Castro::normalize_species() 62 0.01496 0.01496 0.01496 1.29% MultiFab::LinComb() 1586 0.01486 0.01486 0.01486 1.28% FabArray::setVal() 1144 0.01454 0.01454 0.01454 1.25% FillBoundary_nowait() 4023 0.01444 0.01444 0.01444 1.24% FabArray::ParallelCopy_nowait() 861 0.01351 0.01351 0.01351 1.16% Castro::computeTemp() 63 0.01312 0.01312 0.01312 1.13% StateDataPhysBCFunct::() 41 0.01293 0.01293 0.01293 1.11% MLCellLinOp::defineAuxData() 11 0.01242 0.01242 0.01242 1.07% MLPoisson::Fapply() 1142 0.01201 0.01201 0.01201 1.04% Castro::enforce_min_density() 62 0.01049 0.01049 0.01049 0.90% Gravity::fill_multipole_BCs() 11 0.008378 0.008378 0.008378 0.72% MLMG::addInterpCorrection() 410 0.007771 0.007771 0.007771 0.67% amrex::average_down 410 0.006987 0.006987 0.006987 0.60% MultiFab::Xpay() 585 0.00674 0.00674 0.00674 0.58% Castro::estTimeStep() 21 0.005991 0.005991 0.005991 0.52% Castro::do_advance_ctu() 10 0.004606 0.004606 0.004606 0.40% BndryData::define() 11 0.00414 0.00414 0.00414 0.36% Castro::reset_internal_energy(MultiFab) 63 0.003959 0.003959 0.003959 0.34% Amr::writePlotFile() 2 0.00309 0.00309 0.00309 0.27% Castro::construct_new_gravity_source() 10 0.003027 0.003027 0.003027 0.26% Castro::construct_old_gravity_source() 10 0.002205 0.002205 0.002205 0.19% MLMG::ResNormInf() 93 0.002104 0.002104 0.002104 0.18% Gravity::get_new_grav_vector() 11 0.001943 0.001943 0.001943 0.17% MultiFab::Saxpy() 20 0.001814 0.001814 0.001814 0.16% Gravity::get_old_grav_vector() 10 0.001751 0.001751 0.001751 0.15% Castro::expand_state() 10 0.00174 0.00174 0.00174 0.15% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001729 0.001729 0.001729 0.15% MultiFab::Add() 82 0.001671 0.001671 0.001671 0.14% MLCellLinOp::setLevelBC() 11 0.001604 0.001604 0.001604 0.14% Castro::reset_internal_energy(Fab) 504 0.001554 0.001554 0.001554 0.13% Gravity::actual_solve_with_mlmg() 11 0.001442 0.001442 0.001442 0.12% FabArray::mult() 43 0.001334 0.001334 0.001334 0.11% FabArray::setDomainBndry() 41 0.001334 0.001334 0.001334 0.11% Castro::enforce_speed_limit() 62 0.001247 0.001247 0.001247 0.11% MLMG::prepareForSolve() 11 0.001236 0.001236 0.001236 0.11% MLCellLinOp::prepareForSolve() 11 0.0012 0.0012 0.0012 0.10% MultiFab::contains_nan() 20 0.001192 0.001192 0.001192 0.10% Castro::initData() 1 0.001181 0.001181 0.001181 0.10% MLCellLinOp::smooth() 1640 0.001112 0.001112 0.001112 0.10% MLCellLinOp::compGrad() 11 0.0009331 0.0009331 0.0009331 0.08% FabArray::FillBoundary() 4023 0.0008565 0.0008565 0.0008565 0.07% FabArrayBase::getCPC() 1323 0.0007534 0.0007534 0.0007534 0.06% FabArrayBase::CPC::define() 454 0.0006775 0.0006775 0.0006775 0.06% FabArrayBase::getFB() 4023 0.0005908 0.0005908 0.0005908 0.05% Amr::InitAmr() 1 0.0004815 0.0004815 0.0004815 0.04% Gravity::solve_for_phi() 10 0.0004748 0.0004748 0.0004748 0.04% MLCellLinOp::apply() 1142 0.0004718 0.0004718 0.0004718 0.04% Gravity::update_max_rhs() 11 0.0004157 0.0004157 0.0004157 0.04% Amr::coarseTimeStep() 10 0.0003534 0.0003534 0.0003534 0.03% CGSolver::sxay() 1586 0.0003481 0.0003481 0.0003481 0.03% MultiFab::Copy() 11 0.00033 0.00033 0.00033 0.03% MLCGSolver::ParallelAllReduce 1514 0.0003005 0.0003005 0.0003005 0.03% FillPatchIterator::Initialize 41 0.0002977 0.0002977 0.0002977 0.03% main() 1 0.0002967 0.0002967 0.0002967 0.03% MLCellLinOp::defineBC() 11 0.0002811 0.0002811 0.0002811 0.02% MultiFab::max() 11 0.0002598 0.0002598 0.0002598 0.02% FabArray::ParallelCopy() 861 0.0002593 0.0002593 0.0002593 0.02% MLCellLinOp::correctionResidual() 492 0.000246 0.000246 0.000246 0.02% MLMG::MLRhsNormInf() 11 0.0002196 0.0002196 0.0002196 0.02% Castro::construct_new_gravity() 10 0.00021 0.00021 0.00021 0.02% MLMG::mgVcycle() 82 0.0002068 0.0002068 0.0002068 0.02% Amr::timeStep() 10 0.0001554 0.0001554 0.0001554 0.01% MLMG:computeResOfCorrection() 410 0.0001526 0.0001526 0.0001526 0.01% Castro::subcycle_advance_ctu() 10 0.0001467 0.0001467 0.0001467 0.01% MLLinOp::defineGrids() 11 0.0001403 0.0001403 0.0001403 0.01% StateData::checkPoint() 12 0.0001305 0.0001305 0.0001305 0.01% MLMG::mgVcycle_down::0 82 0.0001198 0.0001198 0.0001198 0.01% MLMG::mgVcycle_down::1 82 9.773e-05 9.773e-05 9.773e-05 0.01% MLMG::mgVcycle_down::2 82 9.639e-05 9.639e-05 9.639e-05 0.01% MLMG::mgVcycle_down::3 82 8.938e-05 8.938e-05 8.938e-05 0.01% Castro::Castro() 1 8.861e-05 8.861e-05 8.861e-05 0.01% MLMG::mgVcycle_down::4 82 8.777e-05 8.777e-05 8.777e-05 0.01% MLMG::mgVcycle_up::4 82 8.724e-05 8.724e-05 8.724e-05 0.01% FabArrayBase::FB::FB() 56 8.591e-05 8.591e-05 8.591e-05 0.01% Castro::initialize_advance() 10 8.554e-05 8.554e-05 8.554e-05 0.01% MLMG::actualBottomSolve() 82 8.214e-05 8.214e-05 8.214e-05 0.01% Castro::clean_state() 62 8.181e-05 8.181e-05 8.181e-05 0.01% Castro::advance() 10 8.179e-05 8.179e-05 8.179e-05 0.01% AmrLevel::checkPoint() 3 7.416e-05 7.416e-05 7.416e-05 0.01% MLMG::solve() 11 7.261e-05 7.261e-05 7.261e-05 0.01% Castro::finalize_advance() 10 6.635e-05 6.635e-05 6.635e-05 0.01% MLMG::oneIter() 82 6.363e-05 6.363e-05 6.363e-05 0.01% Castro::initialize_do_advance() 10 6.313e-05 6.313e-05 6.313e-05 0.01% MLMG::mgVcycle_up::0 82 6.229e-05 6.229e-05 6.229e-05 0.01% MLMG::mgVcycle_up::3 82 5.775e-05 5.775e-05 5.775e-05 0.00% MLMG::mgVcycle_up::1 82 5.728e-05 5.728e-05 5.728e-05 0.00% MLMG::mgVcycle_up::2 82 5.623e-05 5.623e-05 5.623e-05 0.00% MLCellLinOp::solutionResidual() 93 5.371e-05 5.371e-05 5.371e-05 0.00% StateData::define() 4 4.089e-05 4.089e-05 4.089e-05 0.00% MLMG::computeResidual() 82 4.017e-05 4.017e-05 4.017e-05 0.00% Castro::swap_state_time_levels() 10 3.911e-05 3.911e-05 3.911e-05 0.00% Amr::writeSmallPlotFile() 1 3.768e-05 3.768e-05 3.768e-05 0.00% Castro::enforce_consistent_e() 1 3.569e-05 3.569e-05 3.569e-05 0.00% Gravity::actual_multilevel_solve() 1 3.372e-05 3.372e-05 3.372e-05 0.00% MLMG::mgVcycle_bottom 82 3.327e-05 3.327e-05 3.327e-05 0.00% Castro::finalize_do_advance() 10 3.313e-05 3.313e-05 3.313e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.854e-05 2.854e-05 2.854e-05 0.00% makeSFC 55 2.81e-05 2.81e-05 2.81e-05 0.00% FillPatchSingleLevel 41 2.778e-05 2.778e-05 2.778e-05 0.00% Castro::initMFs() 1 2.502e-05 2.502e-05 2.502e-05 0.00% MLLinOp::define() 11 2.421e-05 2.421e-05 2.421e-05 0.00% Castro::buildMetrics() 1 2.293e-05 2.293e-05 2.293e-05 0.00% Amr::defBaseLevel() 1 2.109e-05 2.109e-05 2.109e-05 0.00% Amr::FinalizeInit() 1 1.996e-05 1.996e-05 1.996e-05 0.00% Castro::construct_new_source() 50 1.968e-05 1.968e-05 1.968e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.933e-05 1.933e-05 1.933e-05 0.00% MLPoisson::define() 11 1.849e-05 1.849e-05 1.849e-05 0.00% Castro::construct_old_source() 50 1.845e-05 1.845e-05 1.845e-05 0.00% Castro::do_new_sources() 10 1.705e-05 1.705e-05 1.705e-05 0.00% DistributionMapping::Distribute() 56 1.625e-05 1.625e-05 1.625e-05 0.00% Castro::do_old_sources() 10 1.608e-05 1.608e-05 1.608e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.379e-05 1.379e-05 1.379e-05 0.00% Castro::check_for_nan() 20 1.25e-05 1.25e-05 1.25e-05 0.00% Castro::apply_source_to_state() 20 1.156e-05 1.156e-05 1.156e-05 0.00% MLMG::computeMLResidual() 11 1.016e-05 1.016e-05 1.016e-05 0.00% Castro::construct_old_gravity() 10 9.33e-06 9.33e-06 9.33e-06 0.00% Amr::initSubcycle() 1 9.035e-06 9.035e-06 9.035e-06 0.00% Gravity::swapTimeLevels() 10 8.987e-06 8.987e-06 8.987e-06 0.00% MLPoisson::prepareForSolve() 11 8.841e-06 8.841e-06 8.841e-06 0.00% Castro::post_timestep() 10 8.686e-06 8.686e-06 8.686e-06 0.00% Castro::computeNewDt() 9 7.054e-06 7.054e-06 7.054e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.989e-06 6.989e-06 6.989e-06 0.00% MLMG::getGradSolution() 11 6.802e-06 6.802e-06 6.802e-06 0.00% AmrLevel::checkPointPost() 3 5.804e-06 5.804e-06 5.804e-06 0.00% Amr::InitializeInit() 1 4.778e-06 4.778e-06 4.778e-06 0.00% Castro::retry_advance_ctu() 10 4.065e-06 4.065e-06 4.065e-06 0.00% Gravity::set_mass_offset() 11 3.957e-06 3.957e-06 3.957e-06 0.00% MLMG::MLResNormInf() 11 3.676e-06 3.676e-06 3.676e-06 0.00% Castro::create_source_corrector() 10 3.608e-06 3.608e-06 3.608e-06 0.00% Castro::post_init() 1 3.488e-06 3.488e-06 3.488e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.175e-06 3.175e-06 3.175e-06 0.00% Castro::computeInitialDt() 2 2.861e-06 2.861e-06 2.861e-06 0.00% Castro::FluxRegCrseInit 10 2.754e-06 2.754e-06 2.754e-06 0.00% Castro::FluxRegFineAdd() 10 2.37e-06 2.37e-06 2.37e-06 0.00% Amr::init() 1 2.331e-06 2.331e-06 2.331e-06 0.00% AmrLevel::checkPointPre() 3 2.081e-06 2.081e-06 2.081e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.944e-06 1.944e-06 1.944e-06 0.00% Castro::post_regrid() 1 1.148e-06 1.148e-06 1.148e-06 0.00% Amr::initialInit() 1 9.57e-07 9.57e-07 9.57e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 1.161 1.161 1.161 100.00% Amr::coarseTimeStep() 10 0.9876 0.9876 0.9876 85.09% Amr::timeStep() 10 0.6038 0.6038 0.6038 52.03% Castro::advance() 10 0.5963 0.5963 0.5963 51.38% Castro::subcycle_advance_ctu() 10 0.5831 0.5831 0.5831 50.24% Castro::do_advance_ctu() 10 0.5829 0.5829 0.5829 50.23% VisMF::Write(FabArray) 11 0.4777 0.4777 0.4777 41.16% Amr::checkPoint() 3 0.4334 0.4334 0.4334 37.34% AmrLevel::checkPoint() 3 0.4146 0.4146 0.4146 35.73% StateData::checkPoint() 12 0.4145 0.4145 0.4145 35.72% Gravity::solve_phi_with_mlmg() 11 0.3225 0.3225 0.3225 27.79% Gravity::actual_solve_with_mlmg() 11 0.3139 0.3139 0.3139 27.05% Castro::construct_new_gravity() 10 0.2935 0.2935 0.2935 25.29% MLMG::solve() 11 0.29 0.29 0.29 24.99% Gravity::solve_for_phi() 10 0.2778 0.2778 0.2778 23.94% MLMG::oneIter() 82 0.2748 0.2748 0.2748 23.68% MLMG::mgVcycle() 82 0.2731 0.2731 0.2731 23.53% Castro::construct_ctu_hydro_source() 10 0.2152 0.2152 0.2152 18.54% MLCellLinOp::smooth() 1640 0.1392 0.1392 0.1392 11.99% Amr::init() 1 0.1365 0.1365 0.1365 11.76% MLCellLinOp::applyBC() 4433 0.09797 0.09797 0.09797 8.44% MLMG::mgVcycle_bottom 82 0.08454 0.08454 0.08454 7.28% MLMG::actualBottomSolve() 82 0.08451 0.08451 0.08451 7.28% MLCGSolver::bicgstab 82 0.08368 0.08368 0.08368 7.21% Amr::writePlotFile() 2 0.06635 0.06635 0.06635 5.72% MLPoisson::Fsmooth() 3280 0.06526 0.06526 0.06526 5.62% Amr::initialInit() 1 0.05264 0.05264 0.05264 4.54% Amr::FinalizeInit() 1 0.04851 0.04851 0.04851 4.18% Castro::post_init() 1 0.04713 0.04713 0.04713 4.06% Gravity::multilevel_solve_for_new_phi() 1 0.04523 0.04523 0.04523 3.90% Gravity::actual_multilevel_solve() 1 0.04522 0.04522 0.04522 3.90% Castro::clean_state() 62 0.04455 0.04455 0.04455 3.84% FillPatchIterator::Initialize 41 0.04267 0.04267 0.04267 3.68% FillPatchSingleLevel 41 0.04104 0.04104 0.04104 3.54% MLCellLinOp::apply() 1142 0.03716 0.03716 0.03716 3.20% StateDataPhysBCFunct::() 41 0.03701 0.03701 0.03701 3.19% MLMG::mgVcycle_down::0 82 0.03614 0.03614 0.03614 3.11% MLMG::mgVcycle_up::0 82 0.03103 0.03103 0.03103 2.67% StateData::FillBoundary(geom) 328 0.02407 0.02407 0.02407 2.07% MultiFab::Dot() 1114 0.02305 0.02305 0.02305 1.99% MLCellLinOp::correctionResidual() 492 0.02182 0.02182 0.02182 1.88% Castro::initialize_do_advance() 10 0.01945 0.01945 0.01945 1.68% MLPoisson::define() 11 0.01944 0.01944 0.01944 1.67% MLMG:computeResOfCorrection() 410 0.01882 0.01882 0.01882 1.62% Castro::computeTemp() 63 0.01864 0.01864 0.01864 1.61% MLMG::mgVcycle_down::1 82 0.01812 0.01812 0.01812 1.56% MLMG::mgVcycle_down::2 82 0.01774 0.01774 0.01774 1.53% Gravity::get_new_grav_vector() 11 0.01724 0.01724 0.01724 1.49% MLMG::mgVcycle_down::3 82 0.01678 0.01678 0.01678 1.45% MLMG::mgVcycle_down::4 82 0.01604 0.01604 0.01604 1.38% FabArray::FillBoundary() 4023 0.01598 0.01598 0.01598 1.38% CGSolver::sxay() 1586 0.01521 0.01521 0.01521 1.31% FillBoundary_nowait() 4023 0.01512 0.01512 0.01512 1.30% Castro::normalize_species() 62 0.01496 0.01496 0.01496 1.29% Castro::construct_old_gravity() 10 0.01494 0.01494 0.01494 1.29% Gravity::get_old_grav_vector() 10 0.01493 0.01493 0.01493 1.29% MultiFab::LinComb() 1586 0.01486 0.01486 0.01486 1.28% FabArray::ParallelCopy() 861 0.01457 0.01457 0.01457 1.26% FabArray::setVal() 1144 0.01454 0.01454 0.01454 1.25% FabArray::ParallelCopy_nowait() 861 0.01431 0.01431 0.01431 1.23% MLCellLinOp::defineAuxData() 11 0.0138 0.0138 0.0138 1.19% MLCGSolver::ParallelAllReduce 1514 0.01375 0.01375 0.01375 1.19% MLMG::mgVcycle_up::2 82 0.01354 0.01354 0.01354 1.17% MLMG::mgVcycle_up::1 82 0.01334 0.01334 0.01334 1.15% Castro::initialize_advance() 10 0.01308 0.01308 0.01308 1.13% MLMG::addInterpCorrection() 410 0.01303 0.01303 0.01303 1.12% MLMG::mgVcycle_up::3 82 0.01288 0.01288 0.01288 1.11% MLMG::mgVcycle_up::4 82 0.01274 0.01274 0.01274 1.10% amrex::average_down 410 0.0123 0.0123 0.0123 1.06% MLPoisson::Fapply() 1142 0.01201 0.01201 0.01201 1.04% Castro::expand_state() 10 0.01171 0.01171 0.01171 1.01% Castro::enforce_min_density() 62 0.01049 0.01049 0.01049 0.90% Castro::do_new_sources() 10 0.01045 0.01045 0.01045 0.90% Castro::do_old_sources() 10 0.009931 0.009931 0.009931 0.86% Gravity::fill_multipole_BCs() 11 0.008378 0.008378 0.008378 0.72% Castro::post_timestep() 10 0.007372 0.007372 0.007372 0.64% MLCellLinOp::solutionResidual() 93 0.007215 0.007215 0.007215 0.62% MultiFab::Xpay() 585 0.00674 0.00674 0.00674 0.58% MLMG::computeResidual() 82 0.006229 0.006229 0.006229 0.54% Castro::estTimeStep() 21 0.005991 0.005991 0.005991 0.52% Castro::reset_internal_energy(MultiFab) 63 0.005513 0.005513 0.005513 0.48% MLMG::prepareForSolve() 11 0.005493 0.005493 0.005493 0.47% MLCellLinOp::defineBC() 11 0.005398 0.005398 0.005398 0.47% BndryData::define() 11 0.005117 0.005117 0.005117 0.44% Amr::InitializeInit() 1 0.004128 0.004128 0.004128 0.36% Amr::defBaseLevel() 1 0.004123 0.004123 0.004123 0.36% Castro::initData() 1 0.003609 0.003609 0.003609 0.31% Castro::construct_new_source() 50 0.003047 0.003047 0.003047 0.26% Castro::construct_new_gravity_source() 10 0.003027 0.003027 0.003027 0.26% Castro::computeNewDt() 9 0.002796 0.002796 0.002796 0.24% Castro::construct_old_source() 50 0.002223 0.002223 0.002223 0.19% Castro::construct_old_gravity_source() 10 0.002205 0.002205 0.002205 0.19% MLMG::ResNormInf() 93 0.002104 0.002104 0.002104 0.18% Castro::apply_source_to_state() 20 0.001825 0.001825 0.001825 0.16% MultiFab::Saxpy() 20 0.001814 0.001814 0.001814 0.16% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001729 0.001729 0.001729 0.15% MultiFab::Add() 82 0.001671 0.001671 0.001671 0.14% MLCellLinOp::setLevelBC() 11 0.001604 0.001604 0.001604 0.14% Castro::reset_internal_energy(Fab) 504 0.001554 0.001554 0.001554 0.13% MLMG::getGradSolution() 11 0.001451 0.001451 0.001451 0.12% MLCellLinOp::compGrad() 11 0.001444 0.001444 0.001444 0.12% FabArrayBase::getCPC() 1323 0.001431 0.001431 0.001431 0.12% FabArray::mult() 43 0.001334 0.001334 0.001334 0.11% FabArray::setDomainBndry() 41 0.001334 0.001334 0.001334 0.11% Castro::enforce_speed_limit() 62 0.001247 0.001247 0.001247 0.11% MLPoisson::prepareForSolve() 11 0.001209 0.001209 0.001209 0.10% Castro::check_for_nan() 20 0.001205 0.001205 0.001205 0.10% MLCellLinOp::prepareForSolve() 11 0.0012 0.0012 0.0012 0.10% MultiFab::contains_nan() 20 0.001192 0.001192 0.001192 0.10% Castro::post_regrid() 1 0.001123 0.001123 0.001123 0.10% MLMG::computeMLResidual() 11 0.001036 0.001036 0.001036 0.09% Gravity::update_max_rhs() 11 0.0008302 0.0008302 0.0008302 0.07% FabArrayBase::CPC::define() 454 0.0006775 0.0006775 0.0006775 0.06% FabArrayBase::getFB() 4023 0.0006768 0.0006768 0.0006768 0.06% Castro::computeInitialDt() 2 0.000594 0.000594 0.000594 0.05% Amr::InitAmr() 1 0.0004906 0.0004906 0.0004906 0.04% Gravity::swapTimeLevels() 10 0.0004582 0.0004582 0.0004582 0.04% Castro::Castro() 1 0.000441 0.000441 0.000441 0.04% MultiFab::Copy() 11 0.00033 0.00033 0.00033 0.03% MLMG::MLResNormInf() 11 0.0002825 0.0002825 0.0002825 0.02% MultiFab::max() 11 0.0002598 0.0002598 0.0002598 0.02% MLLinOp::define() 11 0.0002234 0.0002234 0.0002234 0.02% MLMG::MLRhsNormInf() 11 0.0002196 0.0002196 0.0002196 0.02% MLLinOp::defineGrids() 11 0.0001992 0.0001992 0.0001992 0.02% Castro::buildMetrics() 1 0.0001605 0.0001605 0.0001605 0.01% FabArrayBase::FB::FB() 56 8.591e-05 8.591e-05 8.591e-05 0.01% Castro::finalize_advance() 10 7.148e-05 7.148e-05 7.148e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.695e-05 5.695e-05 5.695e-05 0.00% AmrLevel::AmrLevel(dm) 1 4.788e-05 4.788e-05 4.788e-05 0.00% makeSFC 55 4.317e-05 4.317e-05 4.317e-05 0.00% StateData::define() 4 4.089e-05 4.089e-05 4.089e-05 0.00% Castro::swap_state_time_levels() 10 3.911e-05 3.911e-05 3.911e-05 0.00% Amr::writeSmallPlotFile() 1 3.768e-05 3.768e-05 3.768e-05 0.00% Castro::enforce_consistent_e() 1 3.569e-05 3.569e-05 3.569e-05 0.00% Castro::finalize_do_advance() 10 3.313e-05 3.313e-05 3.313e-05 0.00% Castro::initMFs() 1 2.502e-05 2.502e-05 2.502e-05 0.00% DistributionMapping::Distribute() 56 1.625e-05 1.625e-05 1.625e-05 0.00% Amr::initSubcycle() 1 9.035e-06 9.035e-06 9.035e-06 0.00% AmrLevel::checkPointPost() 3 5.804e-06 5.804e-06 5.804e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.363e-06 4.363e-06 4.363e-06 0.00% Castro::retry_advance_ctu() 10 4.065e-06 4.065e-06 4.065e-06 0.00% Gravity::set_mass_offset() 11 3.957e-06 3.957e-06 3.957e-06 0.00% Castro::create_source_corrector() 10 3.608e-06 3.608e-06 3.608e-06 0.00% Castro::FluxRegCrseInit 10 2.754e-06 2.754e-06 2.754e-06 0.00% Castro::FluxRegFineAdd() 10 2.37e-06 2.37e-06 2.37e-06 0.00% AmrLevel::checkPointPre() 3 2.081e-06 2.081e-06 2.081e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.944e-06 1.944e-06 1.944e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-3-g1bc4e4eb5a25) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-3-g1bc4e4eb5a25) initialized Starting run at 08:34:48 UTC on 2022-10-04. Successfully read inputs file ... Castro git describe: 22.09-1-g65b273ad0 AMReX git describe: 22.10-3-g1bc4e4eb5 Microphysics git describe: 22.10-2-gbaa4f033 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.459081091 Restart time = 0.047936955 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.054983818 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.052182489 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057603286 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.07150644 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.078599644 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.036913901 seconds Ending run at 08:34:49 UTC on 2022-10-04. Run time = 0.400839891 Run time without initialization = 0.352316812 Average number of zones advanced per microsecond: 3.720 Average number of zones advanced per microsecond per rank: 3.720 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4009 ... 0.4009 ... 0.4009 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1111 0.1111 0.1111 27.70% VisMF::Read() 3 0.04006 0.04006 0.04006 9.99% MLCellLinOp::applyBC() 1946 0.03603 0.03603 0.03603 8.99% VisMF::Write(FabArray) 1 0.03525 0.03525 0.03525 8.79% MLPoisson::Fsmooth() 1440 0.02779 0.02779 0.02779 6.93% StateData::FillBoundary(geom) 160 0.01159 0.01159 0.01159 2.89% MLCGSolver::bicgstab 36 0.01049 0.01049 0.01049 2.62% Castro::normalize_species() 30 0.01045 0.01045 0.01045 2.61% Castro::computeTemp() 30 0.0101 0.0101 0.0101 2.52% MultiFab::Dot() 484 0.009855 0.009855 0.009855 2.46% FabArray::setVal() 537 0.006862 0.006862 0.006862 1.71% MLCellLinOp::defineAuxData() 6 0.006344 0.006344 0.006344 1.58% MultiFab::LinComb() 690 0.0063 0.0063 0.0063 1.57% FillBoundary_nowait() 1766 0.006288 0.006288 0.006288 1.57% FabArray::ParallelCopy_nowait() 380 0.006189 0.006189 0.006189 1.54% StateDataPhysBCFunct::() 20 0.005761 0.005761 0.005761 1.44% MLPoisson::Fapply() 500 0.005134 0.005134 0.005134 1.28% Gravity::fill_multipole_BCs() 6 0.004522 0.004522 0.004522 1.13% Amr::restart() 1 0.003594 0.003594 0.003594 0.90% MLMG::addInterpCorrection() 180 0.00336 0.00336 0.00336 0.84% Castro::enforce_min_density() 30 0.003142 0.003142 0.003142 0.78% amrex::average_down 180 0.003008 0.003008 0.003008 0.75% Castro::estTimeStep() 10 0.002934 0.002934 0.002934 0.73% MultiFab::Xpay() 258 0.00292 0.00292 0.00292 0.73% Castro::do_advance_ctu() 5 0.00245 0.00245 0.00245 0.61% BndryData::define() 6 0.00221 0.00221 0.00221 0.55% Amr::writePlotFile() 1 0.001784 0.001784 0.001784 0.44% Castro::construct_new_gravity_source() 5 0.001727 0.001727 0.001727 0.43% Castro::reset_internal_energy(MultiFab) 30 0.00172 0.00172 0.00172 0.43% Castro::construct_old_gravity_source() 5 0.001556 0.001556 0.001556 0.39% Castro::reset_internal_energy(Fab) 240 0.001058 0.001058 0.001058 0.26% Gravity::get_old_grav_vector() 5 0.001043 0.001043 0.001043 0.26% Gravity::get_new_grav_vector() 5 0.0009638 0.0009638 0.0009638 0.24% MLMG::ResNormInf() 42 0.0009279 0.0009279 0.0009279 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009278 0.0009278 0.0009278 0.23% MultiFab::Saxpy() 10 0.000923 0.000923 0.000923 0.23% Castro::expand_state() 5 0.0008811 0.0008811 0.0008811 0.22% MLCellLinOp::setLevelBC() 6 0.0008536 0.0008536 0.0008536 0.21% Gravity::actual_solve_with_mlmg() 6 0.0008079 0.0008079 0.0008079 0.20% MultiFab::Add() 36 0.0007124 0.0007124 0.0007124 0.18% MLMG::prepareForSolve() 6 0.0006657 0.0006657 0.0006657 0.17% MLCellLinOp::prepareForSolve() 6 0.0006545 0.0006545 0.0006545 0.16% FabArray::setDomainBndry() 20 0.0006506 0.0006506 0.0006506 0.16% FabArray::mult() 22 0.0006492 0.0006492 0.0006492 0.16% MultiFab::contains_nan() 10 0.0005876 0.0005876 0.0005876 0.15% MLCellLinOp::compGrad() 6 0.0004957 0.0004957 0.0004957 0.12% MLCellLinOp::smooth() 720 0.0004883 0.0004883 0.0004883 0.12% Amr::InitAmr() 1 0.0004262 0.0004262 0.0004262 0.11% Castro::enforce_speed_limit() 30 0.0003955 0.0003955 0.0003955 0.10% FabArrayBase::CPC::define() 244 0.0003949 0.0003949 0.0003949 0.10% FabArray::FillBoundary() 1766 0.0003846 0.0003846 0.0003846 0.10% FabArrayBase::getCPC() 632 0.0003523 0.0003523 0.0003523 0.09% main() 1 0.0003317 0.0003317 0.0003317 0.08% FabArrayBase::getFB() 1766 0.0002525 0.0002525 0.0002525 0.06% Gravity::update_max_rhs() 6 0.0002295 0.0002295 0.0002295 0.06% Gravity::solve_for_phi() 5 0.000216 0.000216 0.000216 0.05% Castro::subcycle_advance_ctu() 5 0.0002071 0.0002071 0.0002071 0.05% MLCellLinOp::apply() 500 0.0002041 0.0002041 0.0002041 0.05% MultiFab::Copy() 6 0.0001804 0.0001804 0.0001804 0.05% Castro::construct_new_gravity() 5 0.0001789 0.0001789 0.0001789 0.04% Amr::coarseTimeStep() 5 0.0001663 0.0001663 0.0001663 0.04% CGSolver::sxay() 690 0.0001587 0.0001587 0.0001587 0.04% MLCellLinOp::defineBC() 6 0.0001509 0.0001509 0.0001509 0.04% FillPatchIterator::Initialize 20 0.0001385 0.0001385 0.0001385 0.03% MultiFab::max() 6 0.0001356 0.0001356 0.0001356 0.03% MLCGSolver::ParallelAllReduce 659 0.0001294 0.0001294 0.0001294 0.03% FabArray::ParallelCopy() 380 0.000123 0.000123 0.000123 0.03% MLMG::MLRhsNormInf() 6 0.0001205 0.0001205 0.0001205 0.03% Castro::construct_new_source() 25 0.0001157 0.0001157 0.0001157 0.03% Castro::post_timestep() 5 0.0001118 0.0001118 0.0001118 0.03% MLMG::mgVcycle() 36 0.0001038 0.0001038 0.0001038 0.03% MLCellLinOp::correctionResidual() 216 9.633e-05 9.633e-05 9.633e-05 0.02% Castro::advance() 5 9.093e-05 9.093e-05 9.093e-05 0.02% Amr::timeStep() 5 7.688e-05 7.688e-05 7.688e-05 0.02% AmrLevel::restart() 1 7.604e-05 7.604e-05 7.604e-05 0.02% StateData::restartDoit() 4 7.507e-05 7.507e-05 7.507e-05 0.02% MLLinOp::defineGrids() 6 7.405e-05 7.405e-05 7.405e-05 0.02% Castro::finalize_advance() 5 7.289e-05 7.289e-05 7.289e-05 0.02% MLMG:computeResOfCorrection() 180 6.66e-05 6.66e-05 6.66e-05 0.02% Castro::create_source_corrector() 5 6.506e-05 6.506e-05 6.506e-05 0.02% FabArrayBase::FB::FB() 26 5.615e-05 5.615e-05 5.615e-05 0.01% Castro::construct_old_source() 25 5.466e-05 5.466e-05 5.466e-05 0.01% MLMG::mgVcycle_down::0 36 4.964e-05 4.964e-05 4.964e-05 0.01% Castro::initialize_do_advance() 5 4.708e-05 4.708e-05 4.708e-05 0.01% MLMG::mgVcycle_down::1 36 4.462e-05 4.462e-05 4.462e-05 0.01% MLMG::mgVcycle_down::2 36 4.211e-05 4.211e-05 4.211e-05 0.01% Castro::clean_state() 30 4.089e-05 4.089e-05 4.089e-05 0.01% Castro::initialize_advance() 5 4.062e-05 4.062e-05 4.062e-05 0.01% MLMG::mgVcycle_down::4 36 3.97e-05 3.97e-05 3.97e-05 0.01% MLMG::mgVcycle_down::3 36 3.935e-05 3.935e-05 3.935e-05 0.01% MLMG::actualBottomSolve() 36 3.542e-05 3.542e-05 3.542e-05 0.01% MLMG::mgVcycle_up::4 36 3.426e-05 3.426e-05 3.426e-05 0.01% Amr::writeSmallPlotFile() 1 3.384e-05 3.384e-05 3.384e-05 0.01% Castro::buildMetrics() 1 3.372e-05 3.372e-05 3.372e-05 0.01% MLMG::solve() 6 3.317e-05 3.317e-05 3.317e-05 0.01% Gravity::actual_multilevel_solve() 1 3.141e-05 3.141e-05 3.141e-05 0.01% Castro::post_restart() 1 3.132e-05 3.132e-05 3.132e-05 0.01% Castro::swap_state_time_levels() 5 2.818e-05 2.818e-05 2.818e-05 0.01% MLMG::oneIter() 36 2.803e-05 2.803e-05 2.803e-05 0.01% Castro::initMFs() 1 2.795e-05 2.795e-05 2.795e-05 0.01% MLMG::mgVcycle_up::0 36 2.777e-05 2.777e-05 2.777e-05 0.01% MLMG::mgVcycle_up::3 36 2.657e-05 2.657e-05 2.657e-05 0.01% Castro::do_old_sources() 5 2.609e-05 2.609e-05 2.609e-05 0.01% MLMG::mgVcycle_up::2 36 2.45e-05 2.45e-05 2.45e-05 0.01% MLCellLinOp::solutionResidual() 42 2.361e-05 2.361e-05 2.361e-05 0.01% MLMG::mgVcycle_up::1 36 2.345e-05 2.345e-05 2.345e-05 0.01% Castro::finalize_do_advance() 5 2.339e-05 2.339e-05 2.339e-05 0.01% Castro::construct_old_gravity() 5 2.192e-05 2.192e-05 2.192e-05 0.01% MLLinOp::define() 6 2.014e-05 2.014e-05 2.014e-05 0.01% Castro::computeNewDt() 5 2.009e-05 2.009e-05 2.009e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.88e-05 1.88e-05 1.88e-05 0.00% MLMG::computeResidual() 36 1.75e-05 1.75e-05 1.75e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.525e-05 1.525e-05 1.525e-05 0.00% MLMG::mgVcycle_bottom 36 1.511e-05 1.511e-05 1.511e-05 0.00% makeSFC 30 1.439e-05 1.439e-05 1.439e-05 0.00% FillPatchSingleLevel 20 1.363e-05 1.363e-05 1.363e-05 0.00% MLPoisson::define() 6 1.034e-05 1.034e-05 1.034e-05 0.00% Castro::do_new_sources() 5 9.613e-06 9.613e-06 9.613e-06 0.00% DistributionMapping::Distribute() 31 8.93e-06 8.93e-06 8.93e-06 0.00% Amr::initSubcycle() 1 8.765e-06 8.765e-06 8.765e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.182e-06 7.182e-06 7.182e-06 0.00% Castro::check_for_nan() 10 6.692e-06 6.692e-06 6.692e-06 0.00% Castro::apply_source_to_state() 10 5.796e-06 5.796e-06 5.796e-06 0.00% MLMG::computeMLResidual() 6 4.903e-06 4.903e-06 4.903e-06 0.00% MLPoisson::prepareForSolve() 6 4.88e-06 4.88e-06 4.88e-06 0.00% Gravity::swapTimeLevels() 5 3.966e-06 3.966e-06 3.966e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.793e-06 3.793e-06 3.793e-06 0.00% MLMG::getGradSolution() 6 3.16e-06 3.16e-06 3.16e-06 0.00% Gravity::set_mass_offset() 6 2.324e-06 2.324e-06 2.324e-06 0.00% MLMG::MLResNormInf() 6 2.222e-06 2.222e-06 2.222e-06 0.00% Castro::retry_advance_ctu() 5 1.875e-06 1.875e-06 1.875e-06 0.00% Castro::FluxRegCrseInit 5 1.639e-06 1.639e-06 1.639e-06 0.00% Castro::FluxRegFineAdd() 5 1.242e-06 1.242e-06 1.242e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.134e-06 1.134e-06 1.134e-06 0.00% Amr::init() 1 1.106e-06 1.106e-06 1.106e-06 0.00% AmrLevel::AmrLevel() 1 9.98e-07 9.98e-07 9.98e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4009 0.4009 0.4009 100.00% Amr::coarseTimeStep() 5 0.315 0.315 0.315 78.59% Amr::timeStep() 5 0.3134 0.3134 0.3134 78.19% Castro::advance() 5 0.3073 0.3073 0.3073 76.67% Castro::subcycle_advance_ctu() 5 0.3015 0.3015 0.3015 75.21% Castro::do_advance_ctu() 5 0.3013 0.3013 0.3013 75.16% Castro::construct_new_gravity() 5 0.1471 0.1471 0.1471 36.70% Gravity::solve_phi_with_mlmg() 6 0.143 0.143 0.143 35.68% Gravity::solve_for_phi() 5 0.1395 0.1395 0.1395 34.79% Gravity::actual_solve_with_mlmg() 6 0.1384 0.1384 0.1384 34.52% MLMG::solve() 6 0.1258 0.1258 0.1258 31.38% MLMG::oneIter() 36 0.1184 0.1184 0.1184 29.55% MLMG::mgVcycle() 36 0.1177 0.1177 0.1177 29.36% Castro::construct_ctu_hydro_source() 5 0.1111 0.1111 0.1111 27.70% MLCellLinOp::smooth() 720 0.06013 0.06013 0.06013 15.00% Amr::init() 1 0.04798 0.04798 0.04798 11.97% Amr::restart() 1 0.04798 0.04798 0.04798 11.97% MLCellLinOp::applyBC() 1946 0.04301 0.04301 0.04301 10.73% AmrLevel::restart() 1 0.04027 0.04027 0.04027 10.05% StateData::restartDoit() 4 0.04019 0.04019 0.04019 10.03% VisMF::Read() 3 0.04006 0.04006 0.04006 9.99% Amr::writePlotFile() 1 0.03704 0.03704 0.03704 9.24% MLMG::mgVcycle_bottom 36 0.03613 0.03613 0.03613 9.01% MLMG::actualBottomSolve() 36 0.03612 0.03612 0.03612 9.01% MLCGSolver::bicgstab 36 0.03576 0.03576 0.03576 8.92% VisMF::Write(FabArray) 1 0.03525 0.03525 0.03525 8.79% MLPoisson::Fsmooth() 1440 0.02779 0.02779 0.02779 6.93% Castro::clean_state() 30 0.02691 0.02691 0.02691 6.71% FillPatchIterator::Initialize 20 0.02015 0.02015 0.02015 5.03% FillPatchSingleLevel 20 0.01936 0.01936 0.01936 4.83% StateDataPhysBCFunct::() 20 0.01735 0.01735 0.01735 4.33% MLCellLinOp::apply() 500 0.01622 0.01622 0.01622 4.05% MLMG::mgVcycle_down::0 36 0.01566 0.01566 0.01566 3.91% MLMG::mgVcycle_up::0 36 0.01342 0.01342 0.01342 3.35% Castro::computeTemp() 30 0.01288 0.01288 0.01288 3.21% StateData::FillBoundary(geom) 160 0.01159 0.01159 0.01159 2.89% Castro::normalize_species() 30 0.01045 0.01045 0.01045 2.61% MLPoisson::define() 6 0.01016 0.01016 0.01016 2.53% Castro::initialize_do_advance() 5 0.009899 0.009899 0.009899 2.47% MultiFab::Dot() 484 0.009855 0.009855 0.009855 2.46% MLCellLinOp::correctionResidual() 216 0.009438 0.009438 0.009438 2.35% MLMG:computeResOfCorrection() 180 0.008138 0.008138 0.008138 2.03% MLMG::mgVcycle_down::1 36 0.007868 0.007868 0.007868 1.96% Castro::construct_old_gravity() 5 0.007811 0.007811 0.007811 1.95% Gravity::get_old_grav_vector() 5 0.007789 0.007789 0.007789 1.94% Castro::do_new_sources() 5 0.007672 0.007672 0.007672 1.91% MLMG::mgVcycle_down::2 36 0.007603 0.007603 0.007603 1.90% Gravity::get_new_grav_vector() 5 0.007478 0.007478 0.007478 1.87% MLMG::mgVcycle_down::3 36 0.007241 0.007241 0.007241 1.81% MLCellLinOp::defineAuxData() 6 0.007109 0.007109 0.007109 1.77% FabArray::FillBoundary() 1766 0.006981 0.006981 0.006981 1.74% MLMG::mgVcycle_down::4 36 0.00695 0.00695 0.00695 1.73% FabArray::setVal() 537 0.006862 0.006862 0.006862 1.71% FabArray::ParallelCopy() 380 0.006671 0.006671 0.006671 1.66% FillBoundary_nowait() 1766 0.006596 0.006596 0.006596 1.65% FabArray::ParallelCopy_nowait() 380 0.006548 0.006548 0.006548 1.63% Castro::do_old_sources() 5 0.006502 0.006502 0.006502 1.62% CGSolver::sxay() 690 0.006459 0.006459 0.006459 1.61% MultiFab::LinComb() 690 0.0063 0.0063 0.0063 1.57% Castro::post_timestep() 5 0.006012 0.006012 0.006012 1.50% MLCGSolver::ParallelAllReduce 659 0.005899 0.005899 0.005899 1.47% MLMG::mgVcycle_up::2 36 0.005861 0.005861 0.005861 1.46% MLMG::mgVcycle_up::1 36 0.005781 0.005781 0.005781 1.44% Castro::initialize_advance() 5 0.005691 0.005691 0.005691 1.42% MLMG::addInterpCorrection() 180 0.005681 0.005681 0.005681 1.42% Castro::expand_state() 5 0.005668 0.005668 0.005668 1.41% MLMG::mgVcycle_up::3 36 0.005573 0.005573 0.005573 1.39% MLMG::mgVcycle_up::4 36 0.005516 0.005516 0.005516 1.38% amrex::average_down 180 0.005362 0.005362 0.005362 1.34% MLPoisson::Fapply() 500 0.005134 0.005134 0.005134 1.28% Gravity::fill_multipole_BCs() 6 0.004522 0.004522 0.004522 1.13% Castro::post_restart() 1 0.00393 0.00393 0.00393 0.98% Gravity::multilevel_solve_for_new_phi() 1 0.003805 0.003805 0.003805 0.95% Gravity::actual_multilevel_solve() 1 0.003786 0.003786 0.003786 0.94% MLCellLinOp::solutionResidual() 42 0.00325 0.00325 0.00325 0.81% Castro::enforce_min_density() 30 0.003142 0.003142 0.003142 0.78% MLMG::prepareForSolve() 6 0.002976 0.002976 0.002976 0.74% Castro::estTimeStep() 10 0.002934 0.002934 0.002934 0.73% MultiFab::Xpay() 258 0.00292 0.00292 0.00292 0.73% MLCellLinOp::defineBC() 6 0.002912 0.002912 0.002912 0.73% Castro::reset_internal_energy(MultiFab) 30 0.002778 0.002778 0.002778 0.69% BndryData::define() 6 0.002761 0.002761 0.002761 0.69% MLMG::computeResidual() 36 0.002699 0.002699 0.002699 0.67% Castro::construct_new_source() 25 0.001843 0.001843 0.001843 0.46% Castro::construct_new_gravity_source() 5 0.001727 0.001727 0.001727 0.43% Castro::construct_old_source() 25 0.001611 0.001611 0.001611 0.40% Castro::construct_old_gravity_source() 5 0.001556 0.001556 0.001556 0.39% Castro::computeNewDt() 5 0.00143 0.00143 0.00143 0.36% Castro::reset_internal_energy(Fab) 240 0.001058 0.001058 0.001058 0.26% Castro::apply_source_to_state() 10 0.0009288 0.0009288 0.0009288 0.23% MLMG::ResNormInf() 42 0.0009279 0.0009279 0.0009279 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009278 0.0009278 0.0009278 0.23% MultiFab::Saxpy() 10 0.000923 0.000923 0.000923 0.23% MLCellLinOp::setLevelBC() 6 0.0008536 0.0008536 0.0008536 0.21% MLMG::getGradSolution() 6 0.0007741 0.0007741 0.0007741 0.19% MLCellLinOp::compGrad() 6 0.000771 0.000771 0.000771 0.19% FabArrayBase::getCPC() 632 0.0007472 0.0007472 0.0007472 0.19% MultiFab::Add() 36 0.0007124 0.0007124 0.0007124 0.18% MLPoisson::prepareForSolve() 6 0.0006594 0.0006594 0.0006594 0.16% MLCellLinOp::prepareForSolve() 6 0.0006545 0.0006545 0.0006545 0.16% FabArray::setDomainBndry() 20 0.0006506 0.0006506 0.0006506 0.16% FabArray::mult() 22 0.0006492 0.0006492 0.0006492 0.16% Castro::check_for_nan() 10 0.0005943 0.0005943 0.0005943 0.15% MultiFab::contains_nan() 10 0.0005876 0.0005876 0.0005876 0.15% MLMG::computeMLResidual() 6 0.0005735 0.0005735 0.0005735 0.14% Gravity::update_max_rhs() 6 0.0004442 0.0004442 0.0004442 0.11% Amr::InitAmr() 1 0.000435 0.000435 0.000435 0.11% Castro::enforce_speed_limit() 30 0.0003955 0.0003955 0.0003955 0.10% FabArrayBase::CPC::define() 244 0.0003949 0.0003949 0.0003949 0.10% FabArrayBase::getFB() 1766 0.0003087 0.0003087 0.0003087 0.08% Gravity::swapTimeLevels() 5 0.0002306 0.0002306 0.0002306 0.06% MultiFab::Copy() 6 0.0001804 0.0001804 0.0001804 0.05% Castro::buildMetrics() 1 0.0001546 0.0001546 0.0001546 0.04% MLMG::MLResNormInf() 6 0.0001508 0.0001508 0.0001508 0.04% MultiFab::max() 6 0.0001356 0.0001356 0.0001356 0.03% MLLinOp::define() 6 0.0001243 0.0001243 0.0001243 0.03% MLMG::MLRhsNormInf() 6 0.0001205 0.0001205 0.0001205 0.03% MLLinOp::defineGrids() 6 0.0001042 0.0001042 0.0001042 0.03% Castro::finalize_advance() 5 7.577e-05 7.577e-05 7.577e-05 0.02% Castro::create_source_corrector() 5 6.506e-05 6.506e-05 6.506e-05 0.02% FabArrayBase::FB::FB() 26 5.615e-05 5.615e-05 5.615e-05 0.01% Amr::writeSmallPlotFile() 1 3.384e-05 3.384e-05 3.384e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.903e-05 2.903e-05 2.903e-05 0.01% Castro::swap_state_time_levels() 5 2.818e-05 2.818e-05 2.818e-05 0.01% Castro::initMFs() 1 2.795e-05 2.795e-05 2.795e-05 0.01% Castro::finalize_do_advance() 5 2.339e-05 2.339e-05 2.339e-05 0.01% makeSFC 30 2.184e-05 2.184e-05 2.184e-05 0.01% DistributionMapping::Distribute() 31 8.93e-06 8.93e-06 8.93e-06 0.00% Amr::initSubcycle() 1 8.765e-06 8.765e-06 8.765e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.27e-06 5.27e-06 5.27e-06 0.00% Gravity::set_mass_offset() 6 2.324e-06 2.324e-06 2.324e-06 0.00% Castro::retry_advance_ctu() 5 1.875e-06 1.875e-06 1.875e-06 0.00% Castro::FluxRegCrseInit 5 1.639e-06 1.639e-06 1.639e-06 0.00% Castro::FluxRegFineAdd() 5 1.242e-06 1.242e-06 1.242e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.134e-06 1.134e-06 1.134e-06 0.00% AmrLevel::AmrLevel() 1 9.98e-07 9.98e-07 9.98e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-3-g1bc4e4eb5a25) finalized