Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-15-gff1cce552060) initialized Starting run at 10:12:12 UTC on 2022-12-16. Successfully read inputs file ... Castro git describe: 22.12-5-g3ad2ef0c0 AMReX git describe: 22.12-15-gff1cce552 Microphysics git describe: 22.12-19-g0d4e0eca reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.058268395 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.033771552 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048049912 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.048616849 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.048260953 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.057443134 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.075304873 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.057649742 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.058311243 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047396593 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.04642499 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.046082325 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.060398045 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.057888071 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033406093 seconds Ending run at 10:12:13 UTC on 2022-12-16. Run time = 0.827473818 Run time without initialization = 0.685871566 Average number of zones advanced per microsecond: 3.822 Average number of zones advanced per microsecond per rank: 3.822 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8275 ... 0.8275 ... 0.8275 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2342 0.2342 0.2342 28.31% Castro::construct_ctu_hydro_source() 10 0.1751 0.1751 0.1751 21.16% MLCellLinOp::applyBC() 4433 0.07514 0.07514 0.07514 9.08% MLPoisson::Fsmooth() 3280 0.06036 0.06036 0.06036 7.29% StateData::FillBoundary(geom) 328 0.02356 0.02356 0.02356 2.85% amrex::Dot() 1114 0.02042 0.02042 0.02042 2.47% amrex::Copy() 1029 0.01483 0.01483 0.01483 1.79% Castro::computeTemp() 63 0.01469 0.01469 0.01469 1.78% FabArray::norminf() 743 0.0144 0.0144 0.0144 1.74% Castro::normalize_species() 62 0.01409 0.01409 0.01409 1.70% FillBoundary_nowait() 4023 0.01404 0.01404 0.01404 1.70% FabArray::setVal() 1144 0.01327 0.01327 0.01327 1.60% FabArray::ParallelCopy_nowait() 861 0.01313 0.01313 0.01313 1.59% StateDataPhysBCFunct::() 41 0.01241 0.01241 0.01241 1.50% MLPoisson::Fapply() 1142 0.01048 0.01048 0.01048 1.27% MLCellLinOp::defineAuxData() 11 0.009993 0.009993 0.009993 1.21% Castro::enforce_min_density() 62 0.009985 0.009985 0.009985 1.21% FabArray::Saxpy() 813 0.008169 0.008169 0.008169 0.99% FabArray::Xpay() 821 0.008156 0.008156 0.008156 0.99% MLMG::addInterpCorrection() 410 0.006526 0.006526 0.006526 0.79% Gravity::fill_multipole_BCs() 11 0.006301 0.006301 0.006301 0.76% amrex::average_down 410 0.005943 0.005943 0.005943 0.72% Castro::estTimeStep() 21 0.004546 0.004546 0.004546 0.55% FabArray::LinComb() 557 0.004499 0.004499 0.004499 0.54% Castro::reset_internal_energy(MultiFab) 63 0.004017 0.004017 0.004017 0.49% Amr::checkPoint() 3 0.003931 0.003931 0.003931 0.48% BndryData::define() 11 0.003732 0.003732 0.003732 0.45% Castro::construct_new_gravity_source() 10 0.002858 0.002858 0.002858 0.35% Castro::do_advance_ctu() 10 0.00258 0.00258 0.00258 0.31% amrex::Add() 164 0.002391 0.002391 0.002391 0.29% Amr::writePlotFile() 2 0.002368 0.002368 0.002368 0.29% MLCGSolver::bicgstab 82 0.002161 0.002161 0.002161 0.26% Castro::construct_old_gravity_source() 10 0.001989 0.001989 0.001989 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001644 0.001644 0.001644 0.20% Castro::reset_internal_energy(Fab) 504 0.00148 0.00148 0.00148 0.18% MLCellLinOp::setLevelBC() 11 0.001386 0.001386 0.001386 0.17% FabArray::mult() 43 0.001337 0.001337 0.001337 0.16% FabArray::setDomainBndry() 41 0.001326 0.001326 0.001326 0.16% Gravity::actual_solve_with_mlmg() 11 0.001312 0.001312 0.001312 0.16% Castro::enforce_speed_limit() 62 0.001233 0.001233 0.001233 0.15% MLCellLinOp::smooth() 1640 0.001221 0.001221 0.001221 0.15% Castro::initData() 1 0.0012 0.0012 0.0012 0.14% MultiFab::contains_nan() 20 0.001177 0.001177 0.001177 0.14% MLCellLinOp::prepareForSolve() 11 0.001116 0.001116 0.001116 0.13% MLCellLinOp::compGrad() 11 0.0009151 0.0009151 0.0009151 0.11% MLMG::prepareForSolve() 11 0.0008626 0.0008626 0.0008626 0.10% FabArray::FillBoundary() 4023 0.000783 0.000783 0.000783 0.09% FabArrayBase::getCPC() 1323 0.0007419 0.0007419 0.0007419 0.09% FabArrayBase::CPC::define() 454 0.0006891 0.0006891 0.0006891 0.08% Gravity::get_new_grav_vector() 11 0.0006276 0.0006276 0.0006276 0.08% FabArrayBase::getFB() 4023 0.0006072 0.0006072 0.0006072 0.07% Gravity::get_old_grav_vector() 10 0.0005354 0.0005354 0.0005354 0.06% Amr::InitAmr() 1 0.0005154 0.0005154 0.0005154 0.06% MLCellLinOp::apply() 1142 0.0004839 0.0004839 0.0004839 0.06% MLMG::mgVcycle() 82 0.0003889 0.0003889 0.0003889 0.05% Amr::coarseTimeStep() 10 0.0003656 0.0003656 0.0003656 0.04% main() 1 0.0002906 0.0002906 0.0002906 0.04% MLCGSolver::ParallelAllReduce 1514 0.000278 0.000278 0.000278 0.03% MLLinOp::defineGrids() 11 0.0002655 0.0002655 0.0002655 0.03% MultiFab::max() 11 0.0002592 0.0002592 0.0002592 0.03% FabArray::ParallelCopy() 861 0.0002485 0.0002485 0.0002485 0.03% Castro::subcycle_advance_ctu() 10 0.0002322 0.0002322 0.0002322 0.03% MLCellLinOp::correctionResidual() 492 0.0002295 0.0002295 0.0002295 0.03% FillPatchIterator::Initialize 41 0.0002056 0.0002056 0.0002056 0.02% MLCellLinOp::defineBC() 11 0.0002028 0.0002028 0.0002028 0.02% Amr::timeStep() 10 0.0001586 0.0001586 0.0001586 0.02% Gravity::solve_for_phi() 10 0.0001377 0.0001377 0.0001377 0.02% StateData::checkPoint() 12 0.0001319 0.0001319 0.0001319 0.02% Gravity::update_max_rhs() 11 0.0001151 0.0001151 0.0001151 0.01% MLMG:computeResOfCorrection() 410 0.0001133 0.0001133 0.0001133 0.01% MLMG::actualBottomSolve() 82 9.524e-05 9.524e-05 9.524e-05 0.01% MLMG::mgVcycle_down::0 82 9.442e-05 9.442e-05 9.442e-05 0.01% FabArrayBase::FB::FB() 56 8.821e-05 8.821e-05 8.821e-05 0.01% Castro::Castro() 1 8.401e-05 8.401e-05 8.401e-05 0.01% Castro::expand_state() 10 8.388e-05 8.388e-05 8.388e-05 0.01% MLMG::mgVcycle_down::1 82 8.24e-05 8.24e-05 8.24e-05 0.01% MLMG::mgVcycle_down::2 82 7.871e-05 7.871e-05 7.871e-05 0.01% MLMG::solve() 11 7.845e-05 7.845e-05 7.845e-05 0.01% Castro::clean_state() 62 7.755e-05 7.755e-05 7.755e-05 0.01% AmrLevel::checkPoint() 3 7.404e-05 7.404e-05 7.404e-05 0.01% Castro::initialize_advance() 10 7.273e-05 7.273e-05 7.273e-05 0.01% MLMG::mgVcycle_down::3 82 7.247e-05 7.247e-05 7.247e-05 0.01% MLMG::mgVcycle_down::4 82 7.014e-05 7.014e-05 7.014e-05 0.01% MLMG::mgVcycle_up::4 82 6.182e-05 6.182e-05 6.182e-05 0.01% Castro::advance() 10 6.035e-05 6.035e-05 6.035e-05 0.01% Castro::finalize_advance() 10 5.953e-05 5.953e-05 5.953e-05 0.01% MLMG::oneIter() 82 5.362e-05 5.362e-05 5.362e-05 0.01% MLMG::mgVcycle_up::0 82 5.33e-05 5.33e-05 5.33e-05 0.01% MLCellLinOp::solutionResidual() 93 5.308e-05 5.308e-05 5.308e-05 0.01% MLMG::mgVcycle_up::1 82 5.2e-05 5.2e-05 5.2e-05 0.01% MLMG::mgVcycle_up::3 82 5.098e-05 5.098e-05 5.098e-05 0.01% MLMG::mgVcycle_up::2 82 5.081e-05 5.081e-05 5.081e-05 0.01% Castro::initialize_do_advance() 10 4.476e-05 4.476e-05 4.476e-05 0.01% Castro::construct_new_source() 50 4.082e-05 4.082e-05 4.082e-05 0.00% Castro::swap_state_time_levels() 10 3.596e-05 3.596e-05 3.596e-05 0.00% StateData::define() 4 3.572e-05 3.572e-05 3.572e-05 0.00% Castro::finalize_do_advance() 10 3.514e-05 3.514e-05 3.514e-05 0.00% Castro::enforce_consistent_e() 1 3.398e-05 3.398e-05 3.398e-05 0.00% MLMG::ResNormInf() 93 3.374e-05 3.374e-05 3.374e-05 0.00% MLMG::computeResidual() 82 3.245e-05 3.245e-05 3.245e-05 0.00% MLMG::mgVcycle_bottom 82 3.102e-05 3.102e-05 3.102e-05 0.00% FillPatchSingleLevel 41 3.05e-05 3.05e-05 3.05e-05 0.00% makeSFC 55 2.756e-05 2.756e-05 2.756e-05 0.00% Castro::construct_new_gravity() 10 2.396e-05 2.396e-05 2.396e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.333e-05 2.333e-05 2.333e-05 0.00% Amr::FinalizeInit() 1 2.3e-05 2.3e-05 2.3e-05 0.00% Amr::writeSmallPlotFile() 1 2.273e-05 2.273e-05 2.273e-05 0.00% MLPoisson::define() 11 2.26e-05 2.26e-05 2.26e-05 0.00% Castro::initMFs() 1 2.112e-05 2.112e-05 2.112e-05 0.00% Castro::construct_old_source() 50 1.961e-05 1.961e-05 1.961e-05 0.00% Castro::buildMetrics() 1 1.841e-05 1.841e-05 1.841e-05 0.00% Castro::do_new_sources() 10 1.742e-05 1.742e-05 1.742e-05 0.00% Amr::defBaseLevel() 1 1.706e-05 1.706e-05 1.706e-05 0.00% DistributionMapping::Distribute() 56 1.69e-05 1.69e-05 1.69e-05 0.00% Castro::do_old_sources() 10 1.661e-05 1.661e-05 1.661e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.515e-05 1.515e-05 1.515e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.477e-05 1.477e-05 1.477e-05 0.00% Castro::post_init() 1 1.238e-05 1.238e-05 1.238e-05 0.00% MLLinOp::define() 11 1.162e-05 1.162e-05 1.162e-05 0.00% Castro::check_for_nan() 20 1.155e-05 1.155e-05 1.155e-05 0.00% Castro::post_timestep() 10 1.041e-05 1.041e-05 1.041e-05 0.00% Gravity::swapTimeLevels() 10 1.04e-05 1.04e-05 1.04e-05 0.00% Castro::construct_old_gravity() 10 9.826e-06 9.826e-06 9.826e-06 0.00% AmrLevel::AmrLevel(dm) 1 9.793e-06 9.793e-06 9.793e-06 0.00% Castro::apply_source_to_state() 20 9.305e-06 9.305e-06 9.305e-06 0.00% MLPoisson::prepareForSolve() 11 9.123e-06 9.123e-06 9.123e-06 0.00% Gravity::actual_multilevel_solve() 1 8.133e-06 8.133e-06 8.133e-06 0.00% Amr::initSubcycle() 1 8.058e-06 8.058e-06 8.058e-06 0.00% MLMG::computeMLResidual() 11 7.844e-06 7.844e-06 7.844e-06 0.00% Castro::computeNewDt() 9 7.841e-06 7.841e-06 7.841e-06 0.00% MLMG::getGradSolution() 11 5.733e-06 5.733e-06 5.733e-06 0.00% Amr::InitializeInit() 1 5.368e-06 5.368e-06 5.368e-06 0.00% Gravity::set_mass_offset() 11 5.142e-06 5.142e-06 5.142e-06 0.00% Castro::retry_advance_ctu() 10 4.624e-06 4.624e-06 4.624e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.185e-06 4.185e-06 4.185e-06 0.00% AmrLevel::checkPointPost() 3 4.17e-06 4.17e-06 4.17e-06 0.00% MLMG::MLRhsNormInf() 11 4.035e-06 4.035e-06 4.035e-06 0.00% Castro::create_source_corrector() 10 3.685e-06 3.685e-06 3.685e-06 0.00% MLMG::MLResNormInf() 11 3.578e-06 3.578e-06 3.578e-06 0.00% Castro::computeInitialDt() 2 3.114e-06 3.114e-06 3.114e-06 0.00% Castro::FluxRegCrseInit 10 2.934e-06 2.934e-06 2.934e-06 0.00% Amr::init() 1 2.638e-06 2.638e-06 2.638e-06 0.00% Castro::FluxRegFineAdd() 10 2.218e-06 2.218e-06 2.218e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.132e-06 2.132e-06 2.132e-06 0.00% AmrLevel::checkPointPre() 3 1.671e-06 1.671e-06 1.671e-06 0.00% Amr::initialInit() 1 1.423e-06 1.423e-06 1.423e-06 0.00% Castro::post_regrid() 1 1.337e-06 1.337e-06 1.337e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8275 0.8275 0.8275 100.00% Amr::coarseTimeStep() 10 0.6523 0.6523 0.6523 78.82% Amr::timeStep() 10 0.5338 0.5338 0.5338 64.51% Castro::advance() 10 0.5268 0.5268 0.5268 63.66% Castro::subcycle_advance_ctu() 10 0.515 0.515 0.515 62.24% Castro::do_advance_ctu() 10 0.5148 0.5148 0.5148 62.21% Gravity::solve_phi_with_mlmg() 11 0.2904 0.2904 0.2904 35.09% Gravity::actual_solve_with_mlmg() 11 0.2836 0.2836 0.2836 34.27% Castro::construct_new_gravity() 10 0.2646 0.2646 0.2646 31.97% MLMG::solve() 11 0.263 0.263 0.263 31.78% Gravity::solve_for_phi() 10 0.2493 0.2493 0.2493 30.12% MLMG::oneIter() 82 0.2486 0.2486 0.2486 30.04% MLMG::mgVcycle() 82 0.2469 0.2469 0.2469 29.84% VisMF::Write(FabArray) 11 0.2342 0.2342 0.2342 28.31% Castro::construct_ctu_hydro_source() 10 0.1751 0.1751 0.1751 21.16% Amr::checkPoint() 3 0.1739 0.1739 0.1739 21.02% AmrLevel::checkPoint() 3 0.17 0.17 0.17 20.54% StateData::checkPoint() 12 0.1699 0.1699 0.1699 20.53% Amr::init() 1 0.1409 0.1409 0.1409 17.03% MLCellLinOp::smooth() 1640 0.129 0.129 0.129 15.59% MLCellLinOp::applyBC() 4433 0.09066 0.09066 0.09066 10.96% MLMG::mgVcycle_bottom 82 0.07344 0.07344 0.07344 8.87% MLMG::actualBottomSolve() 82 0.07341 0.07341 0.07341 8.87% MLCGSolver::bicgstab 82 0.0727 0.0727 0.0727 8.79% Amr::writePlotFile() 2 0.06732 0.06732 0.06732 8.14% MLPoisson::Fsmooth() 3280 0.06036 0.06036 0.06036 7.29% Amr::initialInit() 1 0.04875 0.04875 0.04875 5.89% Castro::clean_state() 62 0.04477 0.04477 0.04477 5.41% Amr::FinalizeInit() 1 0.04473 0.04473 0.04473 5.41% Castro::post_init() 1 0.04335 0.04335 0.04335 5.24% Gravity::multilevel_solve_for_new_phi() 1 0.04157 0.04157 0.04157 5.02% Gravity::actual_multilevel_solve() 1 0.04156 0.04156 0.04156 5.02% FillPatchIterator::Initialize 41 0.04155 0.04155 0.04155 5.02% FillPatchSingleLevel 41 0.04002 0.04002 0.04002 4.84% StateDataPhysBCFunct::() 41 0.03597 0.03597 0.03597 4.35% MLMG::mgVcycle_down::0 82 0.03501 0.03501 0.03501 4.23% MLCellLinOp::apply() 1142 0.03371 0.03371 0.03371 4.07% MLMG::mgVcycle_up::0 82 0.02982 0.02982 0.02982 3.60% StateData::FillBoundary(geom) 328 0.02356 0.02356 0.02356 2.85% amrex::Dot() 1114 0.02042 0.02042 0.02042 2.47% Castro::computeTemp() 63 0.02019 0.02019 0.02019 2.44% MLCellLinOp::correctionResidual() 492 0.01977 0.01977 0.01977 2.39% Castro::initialize_do_advance() 10 0.01967 0.01967 0.01967 2.38% MLMG:computeResOfCorrection() 410 0.01717 0.01717 0.01717 2.08% Gravity::get_new_grav_vector() 11 0.01679 0.01679 0.01679 2.03% MLPoisson::define() 11 0.01656 0.01656 0.01656 2.00% MLMG::mgVcycle_down::1 82 0.01631 0.01631 0.01631 1.97% MLMG::mgVcycle_down::2 82 0.01576 0.01576 0.01576 1.90% FabArray::FillBoundary() 4023 0.01552 0.01552 0.01552 1.88% MLMG::mgVcycle_down::3 82 0.01492 0.01492 0.01492 1.80% amrex::Copy() 1029 0.01483 0.01483 0.01483 1.79% Castro::construct_old_gravity() 10 0.01482 0.01482 0.01482 1.79% Gravity::get_old_grav_vector() 10 0.01481 0.01481 0.01481 1.79% FillBoundary_nowait() 4023 0.01474 0.01474 0.01474 1.78% FabArray::norminf() 743 0.0144 0.0144 0.0144 1.74% FabArray::ParallelCopy() 861 0.01418 0.01418 0.01418 1.71% Castro::normalize_species() 62 0.01409 0.01409 0.01409 1.70% MLMG::mgVcycle_down::4 82 0.01409 0.01409 0.01409 1.70% FabArray::ParallelCopy_nowait() 861 0.01394 0.01394 0.01394 1.68% FabArray::setVal() 1144 0.01327 0.01327 0.01327 1.60% MLCGSolver::ParallelAllReduce 1514 0.01224 0.01224 0.01224 1.48% MLMG::mgVcycle_up::2 82 0.01221 0.01221 0.01221 1.48% MLMG::mgVcycle_up::1 82 0.01201 0.01201 0.01201 1.45% Castro::do_new_sources() 10 0.01186 0.01186 0.01186 1.43% MLMG::addInterpCorrection() 410 0.01163 0.01163 0.01163 1.41% MLMG::mgVcycle_up::3 82 0.0115 0.0115 0.0115 1.39% MLMG::mgVcycle_up::4 82 0.01148 0.01148 0.01148 1.39% Castro::expand_state() 10 0.01132 0.01132 0.01132 1.37% MLCellLinOp::defineAuxData() 11 0.01132 0.01132 0.01132 1.37% Castro::initialize_advance() 10 0.01112 0.01112 0.01112 1.34% amrex::average_down 410 0.01101 0.01101 0.01101 1.33% MLPoisson::Fapply() 1142 0.01048 0.01048 0.01048 1.27% Castro::enforce_min_density() 62 0.009985 0.009985 0.009985 1.21% Castro::do_old_sources() 10 0.009803 0.009803 0.009803 1.18% FabArray::Saxpy() 813 0.008169 0.008169 0.008169 0.99% FabArray::Xpay() 821 0.008156 0.008156 0.008156 0.99% MLCellLinOp::solutionResidual() 93 0.007165 0.007165 0.007165 0.87% Castro::post_timestep() 10 0.00686 0.00686 0.00686 0.83% Gravity::fill_multipole_BCs() 11 0.006553 0.006553 0.006553 0.79% MLMG::computeResidual() 82 0.006169 0.006169 0.006169 0.75% Castro::reset_internal_energy(MultiFab) 63 0.005497 0.005497 0.005497 0.66% MLCellLinOp::defineBC() 11 0.004878 0.004878 0.004878 0.59% MLMG::prepareForSolve() 11 0.004714 0.004714 0.004714 0.57% BndryData::define() 11 0.004676 0.004676 0.004676 0.57% Castro::estTimeStep() 21 0.004546 0.004546 0.004546 0.55% FabArray::LinComb() 557 0.004499 0.004499 0.004499 0.54% Amr::InitializeInit() 1 0.004022 0.004022 0.004022 0.49% Amr::defBaseLevel() 1 0.004016 0.004016 0.004016 0.49% Castro::initData() 1 0.003493 0.003493 0.003493 0.42% Castro::construct_new_source() 50 0.002899 0.002899 0.002899 0.35% Castro::construct_new_gravity_source() 10 0.002858 0.002858 0.002858 0.35% amrex::Add() 164 0.002391 0.002391 0.002391 0.29% MLMG::ResNormInf() 93 0.002142 0.002142 0.002142 0.26% Castro::construct_old_source() 50 0.002009 0.002009 0.002009 0.24% Castro::construct_old_gravity_source() 10 0.001989 0.001989 0.001989 0.24% Castro::apply_source_to_state() 20 0.001817 0.001817 0.001817 0.22% Castro::computeNewDt() 9 0.001812 0.001812 0.001812 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001644 0.001644 0.001644 0.20% Castro::reset_internal_energy(Fab) 504 0.00148 0.00148 0.00148 0.18% FabArrayBase::getCPC() 1323 0.001431 0.001431 0.001431 0.17% MLCellLinOp::setLevelBC() 11 0.001386 0.001386 0.001386 0.17% MLMG::getGradSolution() 11 0.001384 0.001384 0.001384 0.17% MLCellLinOp::compGrad() 11 0.001378 0.001378 0.001378 0.17% FabArray::mult() 43 0.001337 0.001337 0.001337 0.16% FabArray::setDomainBndry() 41 0.001326 0.001326 0.001326 0.16% Castro::enforce_speed_limit() 62 0.001233 0.001233 0.001233 0.15% Castro::check_for_nan() 20 0.001188 0.001188 0.001188 0.14% MultiFab::contains_nan() 20 0.001177 0.001177 0.001177 0.14% Castro::post_regrid() 1 0.001159 0.001159 0.001159 0.14% MLPoisson::prepareForSolve() 11 0.001125 0.001125 0.001125 0.14% MLCellLinOp::prepareForSolve() 11 0.001116 0.001116 0.001116 0.13% MLMG::computeMLResidual() 11 0.001036 0.001036 0.001036 0.13% Gravity::update_max_rhs() 11 0.0008272 0.0008272 0.0008272 0.10% Castro::computeInitialDt() 2 0.0008262 0.0008262 0.0008262 0.10% FabArrayBase::getFB() 4023 0.0006954 0.0006954 0.0006954 0.08% FabArrayBase::CPC::define() 454 0.0006891 0.0006891 0.0006891 0.08% Castro::finalize_advance() 10 0.0005858 0.0005858 0.0005858 0.07% Amr::InitAmr() 1 0.0005234 0.0005234 0.0005234 0.06% Castro::Castro() 1 0.0004545 0.0004545 0.0004545 0.05% Gravity::swapTimeLevels() 10 0.0004425 0.0004425 0.0004425 0.05% MLLinOp::define() 11 0.0003369 0.0003369 0.0003369 0.04% MLLinOp::defineGrids() 11 0.0003253 0.0003253 0.0003253 0.04% MLMG::MLResNormInf() 11 0.0002892 0.0002892 0.0002892 0.03% MultiFab::max() 11 0.0002592 0.0002592 0.0002592 0.03% MLMG::MLRhsNormInf() 11 0.0002216 0.0002216 0.0002216 0.03% Castro::buildMetrics() 1 0.0001868 0.0001868 0.0001868 0.02% FabArrayBase::FB::FB() 56 8.821e-05 8.821e-05 8.821e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.77e-05 5.77e-05 5.77e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.551e-05 4.551e-05 4.551e-05 0.01% makeSFC 55 4.293e-05 4.293e-05 4.293e-05 0.01% Castro::swap_state_time_levels() 10 3.596e-05 3.596e-05 3.596e-05 0.00% StateData::define() 4 3.572e-05 3.572e-05 3.572e-05 0.00% Castro::finalize_do_advance() 10 3.514e-05 3.514e-05 3.514e-05 0.00% Castro::enforce_consistent_e() 1 3.398e-05 3.398e-05 3.398e-05 0.00% Amr::writeSmallPlotFile() 1 2.273e-05 2.273e-05 2.273e-05 0.00% Castro::initMFs() 1 2.112e-05 2.112e-05 2.112e-05 0.00% DistributionMapping::Distribute() 56 1.69e-05 1.69e-05 1.69e-05 0.00% Amr::initSubcycle() 1 8.058e-06 8.058e-06 8.058e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.72e-06 5.72e-06 5.72e-06 0.00% Gravity::set_mass_offset() 11 5.142e-06 5.142e-06 5.142e-06 0.00% Castro::retry_advance_ctu() 10 4.624e-06 4.624e-06 4.624e-06 0.00% AmrLevel::checkPointPost() 3 4.17e-06 4.17e-06 4.17e-06 0.00% Castro::create_source_corrector() 10 3.685e-06 3.685e-06 3.685e-06 0.00% Castro::FluxRegCrseInit 10 2.934e-06 2.934e-06 2.934e-06 0.00% Castro::FluxRegFineAdd() 10 2.218e-06 2.218e-06 2.218e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.132e-06 2.132e-06 2.132e-06 0.00% AmrLevel::checkPointPre() 3 1.671e-06 1.671e-06 1.671e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-15-gff1cce552060) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-15-gff1cce552060) initialized Starting run at 10:12:13 UTC on 2022-12-16. Successfully read inputs file ... Castro git describe: 22.12-5-g3ad2ef0c0 AMReX git describe: 22.12-15-gff1cce552 Microphysics git describe: 22.12-19-g0d4e0eca reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.476909484 Restart time = 0.047153329 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.049315692 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.054612996 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.060511435 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059176887 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.054707462 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033297027 seconds Ending run at 10:12:14 UTC on 2022-12-16. Run time = 0.359851341 Run time without initialization = 0.312094849 Average number of zones advanced per microsecond: 4.200 Average number of zones advanced per microsecond per rank: 4.200 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3599 ... 0.3599 ... 0.3599 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0998 0.0998 0.0998 27.72% VisMF::Read() 3 0.04078 0.04078 0.04078 11.33% MLCellLinOp::applyBC() 1946 0.03266 0.03266 0.03266 9.08% VisMF::Write(FabArray) 1 0.03175 0.03175 0.03175 8.82% MLPoisson::Fsmooth() 1440 0.02592 0.02592 0.02592 7.20% StateData::FillBoundary(geom) 160 0.01149 0.01149 0.01149 3.19% amrex::Dot() 484 0.008783 0.008783 0.008783 2.44% amrex::Copy() 463 0.006989 0.006989 0.006989 1.94% Castro::enforce_min_density() 30 0.006552 0.006552 0.006552 1.82% FabArray::setVal() 537 0.006291 0.006291 0.006291 1.75% FabArray::norminf() 326 0.006237 0.006237 0.006237 1.73% FillBoundary_nowait() 1766 0.006139 0.006139 0.006139 1.71% Castro::normalize_species() 30 0.005981 0.005981 0.005981 1.66% FabArray::ParallelCopy_nowait() 380 0.005973 0.005973 0.005973 1.66% MLCellLinOp::defineAuxData() 6 0.005246 0.005246 0.005246 1.46% StateDataPhysBCFunct::() 20 0.005193 0.005193 0.005193 1.44% MLPoisson::Fapply() 500 0.004534 0.004534 0.004534 1.26% Castro::computeTemp() 30 0.004273 0.004273 0.004273 1.19% FabArray::Saxpy() 355 0.003671 0.003671 0.003671 1.02% FabArray::Xpay() 361 0.003557 0.003557 0.003557 0.99% MLMG::addInterpCorrection() 180 0.002821 0.002821 0.002821 0.78% amrex::average_down 180 0.00259 0.00259 0.00259 0.72% Amr::restart() 1 0.002569 0.002569 0.002569 0.71% Gravity::fill_multipole_BCs() 6 0.002445 0.002445 0.002445 0.68% BndryData::define() 6 0.002032 0.002032 0.002032 0.56% Castro::estTimeStep() 10 0.001968 0.001968 0.001968 0.55% FabArray::LinComb() 242 0.001941 0.001941 0.001941 0.54% Castro::reset_internal_energy(MultiFab) 30 0.001736 0.001736 0.001736 0.48% Amr::writePlotFile() 1 0.001439 0.001439 0.001439 0.40% Castro::do_advance_ctu() 5 0.001212 0.001212 0.001212 0.34% Castro::construct_new_gravity_source() 5 0.001186 0.001186 0.001186 0.33% amrex::Add() 72 0.001041 0.001041 0.001041 0.29% MLCGSolver::bicgstab 36 0.0009513 0.0009513 0.0009513 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008872 0.0008872 0.0008872 0.25% Castro::construct_old_gravity_source() 5 0.0007964 0.0007964 0.0007964 0.22% Castro::reset_internal_energy(Fab) 240 0.0007596 0.0007596 0.0007596 0.21% MLCellLinOp::setLevelBC() 6 0.0007403 0.0007403 0.0007403 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007136 0.0007136 0.0007136 0.20% Castro::enforce_speed_limit() 30 0.0006728 0.0006728 0.0006728 0.19% FabArray::mult() 22 0.0006691 0.0006691 0.0006691 0.19% FabArray::setDomainBndry() 20 0.0006428 0.0006428 0.0006428 0.18% MultiFab::contains_nan() 10 0.0006014 0.0006014 0.0006014 0.17% MLCellLinOp::prepareForSolve() 6 0.0005956 0.0005956 0.0005956 0.17% MLCellLinOp::smooth() 720 0.0004971 0.0004971 0.0004971 0.14% MLCellLinOp::compGrad() 6 0.0004959 0.0004959 0.0004959 0.14% MLMG::prepareForSolve() 6 0.0004582 0.0004582 0.0004582 0.13% Amr::InitAmr() 1 0.0004062 0.0004062 0.0004062 0.11% FabArrayBase::CPC::define() 244 0.0003985 0.0003985 0.0003985 0.11% FabArrayBase::getCPC() 632 0.000347 0.000347 0.000347 0.10% FabArray::FillBoundary() 1766 0.0003416 0.0003416 0.0003416 0.09% Gravity::get_old_grav_vector() 5 0.0003025 0.0003025 0.0003025 0.08% main() 1 0.0002852 0.0002852 0.0002852 0.08% Gravity::get_new_grav_vector() 5 0.0002687 0.0002687 0.0002687 0.07% FabArrayBase::getFB() 1766 0.0002464 0.0002464 0.0002464 0.07% MLCellLinOp::apply() 500 0.0002022 0.0002022 0.0002022 0.06% Amr::coarseTimeStep() 5 0.0001795 0.0001795 0.0001795 0.05% MLMG::mgVcycle() 36 0.000167 0.000167 0.000167 0.05% MultiFab::max() 6 0.0001433 0.0001433 0.0001433 0.04% MLCGSolver::ParallelAllReduce 659 0.0001216 0.0001216 0.0001216 0.03% FabArray::ParallelCopy() 380 0.0001209 0.0001209 0.0001209 0.03% MLLinOp::defineGrids() 6 0.0001174 0.0001174 0.0001174 0.03% FillPatchIterator::Initialize 20 0.0001148 0.0001148 0.0001148 0.03% Castro::subcycle_advance_ctu() 5 0.0001116 0.0001116 0.0001116 0.03% MLCellLinOp::defineBC() 6 0.0001066 0.0001066 0.0001066 0.03% MLCellLinOp::correctionResidual() 216 9.719e-05 9.719e-05 9.719e-05 0.03% Amr::timeStep() 5 9.392e-05 9.392e-05 9.392e-05 0.03% AmrLevel::restart() 1 7.641e-05 7.641e-05 7.641e-05 0.02% Gravity::solve_for_phi() 5 6.924e-05 6.924e-05 6.924e-05 0.02% StateData::restartDoit() 4 6.35e-05 6.35e-05 6.35e-05 0.02% Gravity::update_max_rhs() 6 6.297e-05 6.297e-05 6.297e-05 0.02% FabArrayBase::FB::FB() 26 5.682e-05 5.682e-05 5.682e-05 0.02% Castro::initialize_advance() 5 4.998e-05 4.998e-05 4.998e-05 0.01% MLMG:computeResOfCorrection() 180 4.975e-05 4.975e-05 4.975e-05 0.01% Castro::clean_state() 30 4.596e-05 4.596e-05 4.596e-05 0.01% Castro::finalize_advance() 5 4.494e-05 4.494e-05 4.494e-05 0.01% MLMG::actualBottomSolve() 36 4.303e-05 4.303e-05 4.303e-05 0.01% MLMG::mgVcycle_down::0 36 4.223e-05 4.223e-05 4.223e-05 0.01% Castro::expand_state() 5 4.034e-05 4.034e-05 4.034e-05 0.01% MLMG::mgVcycle_down::1 36 3.814e-05 3.814e-05 3.814e-05 0.01% MLMG::mgVcycle_down::2 36 3.469e-05 3.469e-05 3.469e-05 0.01% MLMG::mgVcycle_down::4 36 3.415e-05 3.415e-05 3.415e-05 0.01% MLMG::solve() 6 3.414e-05 3.414e-05 3.414e-05 0.01% MLMG::mgVcycle_down::3 36 3.308e-05 3.308e-05 3.308e-05 0.01% MLMG::mgVcycle_up::4 36 2.908e-05 2.908e-05 2.908e-05 0.01% Castro::initialize_do_advance() 5 2.729e-05 2.729e-05 2.729e-05 0.01% Amr::writeSmallPlotFile() 1 2.614e-05 2.614e-05 2.614e-05 0.01% MLMG::oneIter() 36 2.496e-05 2.496e-05 2.496e-05 0.01% Castro::buildMetrics() 1 2.418e-05 2.418e-05 2.418e-05 0.01% Castro::post_restart() 1 2.393e-05 2.393e-05 2.393e-05 0.01% MLMG::mgVcycle_up::0 36 2.365e-05 2.365e-05 2.365e-05 0.01% MLMG::mgVcycle_up::3 36 2.317e-05 2.317e-05 2.317e-05 0.01% Castro::swap_state_time_levels() 5 2.315e-05 2.315e-05 2.315e-05 0.01% Castro::finalize_do_advance() 5 2.283e-05 2.283e-05 2.283e-05 0.01% MLCellLinOp::solutionResidual() 42 2.263e-05 2.263e-05 2.263e-05 0.01% MLMG::mgVcycle_up::2 36 2.195e-05 2.195e-05 2.195e-05 0.01% MLMG::mgVcycle_up::1 36 2.125e-05 2.125e-05 2.125e-05 0.01% Castro::initMFs() 1 2.062e-05 2.062e-05 2.062e-05 0.01% MLMG::ResNormInf() 42 1.767e-05 1.767e-05 1.767e-05 0.00% Castro::advance() 5 1.584e-05 1.584e-05 1.584e-05 0.00% FillPatchSingleLevel 20 1.571e-05 1.571e-05 1.571e-05 0.00% MLMG::computeResidual() 36 1.436e-05 1.436e-05 1.436e-05 0.00% MLMG::mgVcycle_bottom 36 1.416e-05 1.416e-05 1.416e-05 0.00% MLPoisson::define() 6 1.409e-05 1.409e-05 1.409e-05 0.00% makeSFC 30 1.348e-05 1.348e-05 1.348e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.342e-05 1.342e-05 1.342e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.306e-05 1.306e-05 1.306e-05 0.00% Castro::construct_new_gravity() 5 1.267e-05 1.267e-05 1.267e-05 0.00% Castro::construct_old_source() 25 1.011e-05 1.011e-05 1.011e-05 0.00% Castro::construct_new_source() 25 9.821e-06 9.821e-06 9.821e-06 0.00% Castro::do_new_sources() 5 9.786e-06 9.786e-06 9.786e-06 0.00% DistributionMapping::Distribute() 31 9.361e-06 9.361e-06 9.361e-06 0.00% Castro::do_old_sources() 5 8.869e-06 8.869e-06 8.869e-06 0.00% Castro::check_for_nan() 10 8.281e-06 8.281e-06 8.281e-06 0.00% Amr::initSubcycle() 1 7.81e-06 7.81e-06 7.81e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.386e-06 7.386e-06 7.386e-06 0.00% Castro::construct_old_gravity() 5 6.999e-06 6.999e-06 6.999e-06 0.00% Gravity::actual_multilevel_solve() 1 6.755e-06 6.755e-06 6.755e-06 0.00% Castro::apply_source_to_state() 10 6.117e-06 6.117e-06 6.117e-06 0.00% Castro::post_timestep() 5 5.856e-06 5.856e-06 5.856e-06 0.00% MLLinOp::define() 6 5.469e-06 5.469e-06 5.469e-06 0.00% Gravity::swapTimeLevels() 5 4.573e-06 4.573e-06 4.573e-06 0.00% MLPoisson::prepareForSolve() 6 4.468e-06 4.468e-06 4.468e-06 0.00% Castro::computeNewDt() 5 3.968e-06 3.968e-06 3.968e-06 0.00% MLMG::computeMLResidual() 6 3.545e-06 3.545e-06 3.545e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.208e-06 3.208e-06 3.208e-06 0.00% Gravity::set_mass_offset() 6 3.108e-06 3.108e-06 3.108e-06 0.00% MLMG::getGradSolution() 6 2.891e-06 2.891e-06 2.891e-06 0.00% Castro::create_source_corrector() 5 2.803e-06 2.803e-06 2.803e-06 0.00% Castro::retry_advance_ctu() 5 2.418e-06 2.418e-06 2.418e-06 0.00% MLMG::MLResNormInf() 6 2.183e-06 2.183e-06 2.183e-06 0.00% MLMG::MLRhsNormInf() 6 2.05e-06 2.05e-06 2.05e-06 0.00% Castro::FluxRegCrseInit 5 1.896e-06 1.896e-06 1.896e-06 0.00% Castro::FluxRegFineAdd() 5 1.545e-06 1.545e-06 1.545e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.101e-06 1.101e-06 1.101e-06 0.00% AmrLevel::AmrLevel() 1 1.056e-06 1.056e-06 1.056e-06 0.00% Amr::init() 1 1.016e-06 1.016e-06 1.016e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3599 0.3599 0.3599 100.00% Amr::coarseTimeStep() 5 0.2785 0.2785 0.2785 77.39% Amr::timeStep() 5 0.277 0.277 0.277 76.98% Castro::advance() 5 0.2738 0.2738 0.2738 76.09% Castro::subcycle_advance_ctu() 5 0.2677 0.2677 0.2677 74.38% Castro::do_advance_ctu() 5 0.2676 0.2676 0.2676 74.35% Castro::construct_new_gravity() 5 0.1323 0.1323 0.1323 36.77% Gravity::solve_phi_with_mlmg() 6 0.128 0.128 0.128 35.56% Gravity::actual_solve_with_mlmg() 6 0.1253 0.1253 0.1253 34.82% Gravity::solve_for_phi() 5 0.1249 0.1249 0.1249 34.70% MLMG::solve() 6 0.1143 0.1143 0.1143 31.75% MLMG::oneIter() 36 0.1074 0.1074 0.1074 29.86% MLMG::mgVcycle() 36 0.1067 0.1067 0.1067 29.65% Castro::construct_ctu_hydro_source() 5 0.09976 0.09976 0.09976 27.72% MLCellLinOp::smooth() 720 0.05563 0.05563 0.05563 15.46% Amr::init() 1 0.0472 0.0472 0.0472 13.11% Amr::restart() 1 0.0472 0.0472 0.0472 13.11% AmrLevel::restart() 1 0.04098 0.04098 0.04098 11.39% StateData::restartDoit() 4 0.0409 0.0409 0.0409 11.36% VisMF::Read() 3 0.04078 0.04078 0.04078 11.33% MLCellLinOp::applyBC() 1946 0.03945 0.03945 0.03945 10.96% Amr::writePlotFile() 1 0.03344 0.03344 0.03344 9.29% VisMF::Write(FabArray) 1 0.03175 0.03175 0.03175 8.82% MLMG::mgVcycle_bottom 36 0.03172 0.03172 0.03172 8.81% MLMG::actualBottomSolve() 36 0.0317 0.0317 0.0317 8.81% MLCGSolver::bicgstab 36 0.03139 0.03139 0.03139 8.72% MLPoisson::Fsmooth() 1440 0.02592 0.02592 0.02592 7.20% Castro::clean_state() 30 0.02002 0.02002 0.02002 5.56% FillPatchIterator::Initialize 20 0.01947 0.01947 0.01947 5.41% FillPatchSingleLevel 20 0.01871 0.01871 0.01871 5.20% StateDataPhysBCFunct::() 20 0.01669 0.01669 0.01669 4.64% MLMG::mgVcycle_down::0 36 0.01521 0.01521 0.01521 4.23% MLCellLinOp::apply() 500 0.01471 0.01471 0.01471 4.09% MLMG::mgVcycle_up::0 36 0.01294 0.01294 0.01294 3.60% StateData::FillBoundary(geom) 160 0.01149 0.01149 0.01149 3.19% Castro::initialize_do_advance() 5 0.009498 0.009498 0.009498 2.64% MLPoisson::define() 6 0.008806 0.008806 0.008806 2.45% amrex::Dot() 484 0.008783 0.008783 0.008783 2.44% MLCellLinOp::correctionResidual() 216 0.008604 0.008604 0.008604 2.39% MLMG:computeResOfCorrection() 180 0.007446 0.007446 0.007446 2.07% Gravity::get_new_grav_vector() 5 0.007347 0.007347 0.007347 2.04% Castro::construct_old_gravity() 5 0.007151 0.007151 0.007151 1.99% Gravity::get_old_grav_vector() 5 0.007144 0.007144 0.007144 1.99% MLMG::mgVcycle_down::1 36 0.007036 0.007036 0.007036 1.96% amrex::Copy() 463 0.006989 0.006989 0.006989 1.94% FabArray::FillBoundary() 1766 0.006784 0.006784 0.006784 1.88% MLMG::mgVcycle_down::2 36 0.006779 0.006779 0.006779 1.88% Castro::computeTemp() 30 0.006769 0.006769 0.006769 1.88% Castro::enforce_min_density() 30 0.006552 0.006552 0.006552 1.82% FabArray::ParallelCopy() 380 0.006472 0.006472 0.006472 1.80% FillBoundary_nowait() 1766 0.006442 0.006442 0.006442 1.79% MLMG::mgVcycle_down::3 36 0.006437 0.006437 0.006437 1.79% FabArray::ParallelCopy_nowait() 380 0.006351 0.006351 0.006351 1.76% FabArray::setVal() 537 0.006291 0.006291 0.006291 1.75% FabArray::norminf() 326 0.006237 0.006237 0.006237 1.73% MLMG::mgVcycle_down::4 36 0.006113 0.006113 0.006113 1.70% Castro::normalize_species() 30 0.005981 0.005981 0.005981 1.66% MLCellLinOp::defineAuxData() 6 0.005964 0.005964 0.005964 1.66% Castro::initialize_advance() 5 0.005808 0.005808 0.005808 1.61% Castro::expand_state() 5 0.005591 0.005591 0.005591 1.55% MLCGSolver::ParallelAllReduce 659 0.005286 0.005286 0.005286 1.47% MLMG::mgVcycle_up::2 36 0.005251 0.005251 0.005251 1.46% MLMG::mgVcycle_up::1 36 0.005152 0.005152 0.005152 1.43% MLMG::addInterpCorrection() 180 0.00506 0.00506 0.00506 1.41% MLMG::mgVcycle_up::3 36 0.004978 0.004978 0.004978 1.38% MLMG::mgVcycle_up::4 36 0.004935 0.004935 0.004935 1.37% Castro::do_new_sources() 5 0.004876 0.004876 0.004876 1.35% amrex::average_down 180 0.004814 0.004814 0.004814 1.34% Castro::do_old_sources() 5 0.004772 0.004772 0.004772 1.33% MLPoisson::Fapply() 500 0.004534 0.004534 0.004534 1.26% FabArray::Saxpy() 355 0.003671 0.003671 0.003671 1.02% FabArray::Xpay() 361 0.003557 0.003557 0.003557 0.99% Castro::post_restart() 1 0.003478 0.003478 0.003478 0.97% Gravity::multilevel_solve_for_new_phi() 1 0.003362 0.003362 0.003362 0.93% Gravity::actual_multilevel_solve() 1 0.003349 0.003349 0.003349 0.93% MLCellLinOp::solutionResidual() 42 0.00322 0.00322 0.00322 0.89% Castro::post_timestep() 5 0.003107 0.003107 0.003107 0.86% MLCellLinOp::defineBC() 6 0.002676 0.002676 0.002676 0.74% MLMG::computeResidual() 36 0.002665 0.002665 0.002665 0.74% Gravity::fill_multipole_BCs() 6 0.002573 0.002573 0.002573 0.71% BndryData::define() 6 0.002569 0.002569 0.002569 0.71% Castro::reset_internal_energy(MultiFab) 30 0.002496 0.002496 0.002496 0.69% MLMG::prepareForSolve() 6 0.00249 0.00249 0.00249 0.69% Castro::estTimeStep() 10 0.001968 0.001968 0.001968 0.55% FabArray::LinComb() 242 0.001941 0.001941 0.001941 0.54% Castro::computeNewDt() 5 0.001284 0.001284 0.001284 0.36% Castro::construct_new_source() 25 0.001196 0.001196 0.001196 0.33% Castro::construct_new_gravity_source() 5 0.001186 0.001186 0.001186 0.33% amrex::Add() 72 0.001041 0.001041 0.001041 0.29% MLMG::ResNormInf() 42 0.0009462 0.0009462 0.0009462 0.26% Castro::apply_source_to_state() 10 0.0009288 0.0009288 0.0009288 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008872 0.0008872 0.0008872 0.25% Castro::construct_old_source() 25 0.0008065 0.0008065 0.0008065 0.22% Castro::construct_old_gravity_source() 5 0.0007964 0.0007964 0.0007964 0.22% MLMG::getGradSolution() 6 0.0007611 0.0007611 0.0007611 0.21% Castro::reset_internal_energy(Fab) 240 0.0007596 0.0007596 0.0007596 0.21% MLCellLinOp::compGrad() 6 0.0007582 0.0007582 0.0007582 0.21% FabArrayBase::getCPC() 632 0.0007455 0.0007455 0.0007455 0.21% MLCellLinOp::setLevelBC() 6 0.0007403 0.0007403 0.0007403 0.21% Castro::enforce_speed_limit() 30 0.0006728 0.0006728 0.0006728 0.19% FabArray::mult() 22 0.0006691 0.0006691 0.0006691 0.19% FabArray::setDomainBndry() 20 0.0006428 0.0006428 0.0006428 0.18% Castro::check_for_nan() 10 0.0006097 0.0006097 0.0006097 0.17% MultiFab::contains_nan() 10 0.0006014 0.0006014 0.0006014 0.17% MLPoisson::prepareForSolve() 6 0.0006 0.0006 0.0006 0.17% MLCellLinOp::prepareForSolve() 6 0.0005956 0.0005956 0.0005956 0.17% MLMG::computeMLResidual() 6 0.0005736 0.0005736 0.0005736 0.16% Gravity::update_max_rhs() 6 0.0004581 0.0004581 0.0004581 0.13% Amr::InitAmr() 1 0.000414 0.000414 0.000414 0.12% FabArrayBase::CPC::define() 244 0.0003985 0.0003985 0.0003985 0.11% Castro::finalize_advance() 5 0.000319 0.000319 0.000319 0.09% FabArrayBase::getFB() 1766 0.0003032 0.0003032 0.0003032 0.08% Gravity::swapTimeLevels() 5 0.0002257 0.0002257 0.0002257 0.06% MLMG::MLResNormInf() 6 0.0001528 0.0001528 0.0001528 0.04% MLLinOp::define() 6 0.0001527 0.0001527 0.0001527 0.04% Castro::buildMetrics() 1 0.0001477 0.0001477 0.0001477 0.04% MLLinOp::defineGrids() 6 0.0001473 0.0001473 0.0001473 0.04% MultiFab::max() 6 0.0001433 0.0001433 0.0001433 0.04% MLMG::MLRhsNormInf() 6 0.0001157 0.0001157 0.0001157 0.03% FabArrayBase::FB::FB() 26 5.682e-05 5.682e-05 5.682e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.88e-05 2.88e-05 2.88e-05 0.01% Amr::writeSmallPlotFile() 1 2.614e-05 2.614e-05 2.614e-05 0.01% Castro::swap_state_time_levels() 5 2.315e-05 2.315e-05 2.315e-05 0.01% Castro::finalize_do_advance() 5 2.283e-05 2.283e-05 2.283e-05 0.01% makeSFC 30 2.141e-05 2.141e-05 2.141e-05 0.01% Castro::initMFs() 1 2.062e-05 2.062e-05 2.062e-05 0.01% DistributionMapping::Distribute() 31 9.361e-06 9.361e-06 9.361e-06 0.00% Amr::initSubcycle() 1 7.81e-06 7.81e-06 7.81e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.637e-06 4.637e-06 4.637e-06 0.00% Gravity::set_mass_offset() 6 3.108e-06 3.108e-06 3.108e-06 0.00% Castro::create_source_corrector() 5 2.803e-06 2.803e-06 2.803e-06 0.00% Castro::retry_advance_ctu() 5 2.418e-06 2.418e-06 2.418e-06 0.00% Castro::FluxRegCrseInit 5 1.896e-06 1.896e-06 1.896e-06 0.00% Castro::FluxRegFineAdd() 5 1.545e-06 1.545e-06 1.545e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.101e-06 1.101e-06 1.101e-06 0.00% AmrLevel::AmrLevel() 1 1.056e-06 1.056e-06 1.056e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-15-gff1cce552060) finalized