Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.06-20-g74183fe377df) initialized Starting run at 08:28:48 UTC on 2022-06-13. Successfully read inputs file ... Castro git describe: 22.06-12-g556652b03 AMReX git describe: 22.06-20-g74183fe37 Microphysics git describe: 22.06-1-gb8e427c1 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.044364468 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025897838 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.047894651 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051195532 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.050917911 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.062074761 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.092205924 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.042976929 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.057742228 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.05373697 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.067780255 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.065625226 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.06850223 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.041175992 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.025859398 seconds Ending run at 08:28:49 UTC on 2022-06-13. Run time = 0.851477662 Run time without initialization = 0.728290556 Average number of zones advanced per microsecond: 3.599 Average number of zones advanced per microsecond per rank: 3.599 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8515 ... 0.8515 ... 0.8515 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2202 0.2202 0.2202 25.86% VisMF::Write(FabArray) 11 0.1732 0.1732 0.1732 20.34% MLCellLinOp::applyBC() 4433 0.08143 0.08143 0.08143 9.56% MLPoisson::Fsmooth() 3280 0.06409 0.06409 0.06409 7.53% StateData::FillBoundary(geom) 328 0.02441 0.02441 0.02441 2.87% MLCGSolver::bicgstab 82 0.02434 0.02434 0.02434 2.86% MultiFab::Dot() 1114 0.02258 0.02258 0.02258 2.65% StateDataPhysBCFunct::() 41 0.01774 0.01774 0.01774 2.08% MultiFab::LinComb() 1586 0.01462 0.01462 0.01462 1.72% FabArray::setVal() 1144 0.01439 0.01439 0.01439 1.69% FillBoundary_nowait() 4023 0.01431 0.01431 0.01431 1.68% Castro::computeTemp() 63 0.0143 0.0143 0.0143 1.68% Castro::normalize_species() 62 0.01365 0.01365 0.01365 1.60% FabArray::ParallelCopy_nowait() 861 0.01321 0.01321 0.01321 1.55% MLPoisson::Fapply() 1142 0.01193 0.01193 0.01193 1.40% MLCellLinOp::defineAuxData() 11 0.01172 0.01172 0.01172 1.38% Castro::enforce_min_density() 62 0.01167 0.01167 0.01167 1.37% Gravity::fill_multipole_BCs() 11 0.008164 0.008164 0.008164 0.96% MLMG::addInterpCorrection() 410 0.007637 0.007637 0.007637 0.90% BndryData::define() 11 0.007547 0.007547 0.007547 0.89% amrex::average_down 410 0.00692 0.00692 0.00692 0.81% MultiFab::Xpay() 585 0.006625 0.006625 0.006625 0.78% Castro::estTimeStep() 21 0.005757 0.005757 0.005757 0.68% Castro::do_advance_ctu() 10 0.00515 0.00515 0.00515 0.60% Amr::checkPoint() 3 0.004801 0.004801 0.004801 0.56% Castro::reset_internal_energy(MultiFab) 63 0.003933 0.003933 0.003933 0.46% Castro::construct_new_gravity_source() 10 0.00322 0.00322 0.00322 0.38% Castro::construct_old_gravity_source() 10 0.002503 0.002503 0.002503 0.29% Amr::writePlotFile() 2 0.002333 0.002333 0.002333 0.27% Gravity::get_new_grav_vector() 11 0.001955 0.001955 0.001955 0.23% MLMG::ResNormInf() 93 0.001954 0.001954 0.001954 0.23% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00178 0.00178 0.00178 0.21% Gravity::get_old_grav_vector() 10 0.001774 0.001774 0.001774 0.21% Castro::expand_state() 10 0.00173 0.00173 0.00173 0.20% MLMG::oneIter() 82 0.001699 0.001699 0.001699 0.20% MLCellLinOp::setLevelBC() 11 0.001558 0.001558 0.001558 0.18% Castro::reset_internal_energy(Fab) 504 0.001543 0.001543 0.001543 0.18% Gravity::actual_solve_with_mlmg() 11 0.001386 0.001386 0.001386 0.16% FabArray::mult() 43 0.001334 0.001334 0.001334 0.16% FabArray::setDomainBndry() 41 0.001329 0.001329 0.001329 0.16% MLCellLinOp::smooth() 1640 0.001253 0.001253 0.001253 0.15% Castro::initData() 1 0.00121 0.00121 0.00121 0.14% MLCellLinOp::prepareForSolve() 11 0.001205 0.001205 0.001205 0.14% MultiFab::contains_nan() 20 0.001172 0.001172 0.001172 0.14% MLMG::prepareForSolve() 11 0.001047 0.001047 0.001047 0.12% Castro::enforce_speed_limit() 62 0.0009898 0.0009898 0.0009898 0.12% MLCellLinOp::compGrad() 11 0.0009413 0.0009413 0.0009413 0.11% FabArray::FillBoundary() 4023 0.000797 0.000797 0.000797 0.09% FabArrayBase::getCPC() 1323 0.0007961 0.0007961 0.0007961 0.09% FabArrayBase::CPC::define() 454 0.0006649 0.0006649 0.0006649 0.08% FabArrayBase::getFB() 4023 0.0006217 0.0006217 0.0006217 0.07% MLCellLinOp::apply() 1142 0.0005062 0.0005062 0.0005062 0.06% Amr::InitAmr() 1 0.0004936 0.0004936 0.0004936 0.06% Gravity::solve_for_phi() 10 0.000424 0.000424 0.000424 0.05% Gravity::update_max_rhs() 11 0.0004062 0.0004062 0.0004062 0.05% CGSolver::sxay() 1586 0.0003761 0.0003761 0.0003761 0.04% Amr::coarseTimeStep() 10 0.0003171 0.0003171 0.0003171 0.04% MLCellLinOp::defineBC() 11 0.000312 0.000312 0.000312 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003012 0.0003012 0.0003012 0.04% FillPatchIterator::Initialize 41 0.0002968 0.0002968 0.0002968 0.03% main() 1 0.0002818 0.0002818 0.0002818 0.03% FabArray::ParallelCopy() 861 0.0002774 0.0002774 0.0002774 0.03% MultiFab::Copy() 11 0.0002631 0.0002631 0.0002631 0.03% MultiFab::max() 11 0.0002574 0.0002574 0.0002574 0.03% MLCellLinOp::correctionResidual() 492 0.0002322 0.0002322 0.0002322 0.03% MLMG::mgVcycle() 82 0.0002257 0.0002257 0.0002257 0.03% MLLinOp::defineGrids() 11 0.0002157 0.0002157 0.0002157 0.03% Castro::construct_new_gravity() 10 0.0002049 0.0002049 0.0002049 0.02% MLMG::MLRhsNormInf() 11 0.0002032 0.0002032 0.0002032 0.02% Amr::timeStep() 10 0.000198 0.000198 0.000198 0.02% Castro::subcycle_advance_ctu() 10 0.0001895 0.0001895 0.0001895 0.02% StateData::checkPoint() 12 0.0001257 0.0001257 0.0001257 0.01% MLMG:computeResOfCorrection() 410 0.0001228 0.0001228 0.0001228 0.01% Castro::advance() 10 0.0001169 0.0001169 0.0001169 0.01% MLMG::actualBottomSolve() 82 0.0001055 0.0001055 0.0001055 0.01% MLMG::mgVcycle_down::0 82 9.003e-05 9.003e-05 9.003e-05 0.01% FabArrayBase::FB::FB() 56 8.648e-05 8.648e-05 8.648e-05 0.01% Castro::Castro() 1 8.406e-05 8.406e-05 8.406e-05 0.01% MLMG::mgVcycle_down::1 82 8.231e-05 8.231e-05 8.231e-05 0.01% MLMG::mgVcycle_down::2 82 8.195e-05 8.195e-05 8.195e-05 0.01% Castro::initialize_advance() 10 8.177e-05 8.177e-05 8.177e-05 0.01% MLMG::solve() 11 7.81e-05 7.81e-05 7.81e-05 0.01% Castro::clean_state() 62 7.583e-05 7.583e-05 7.583e-05 0.01% MLMG::mgVcycle_down::3 82 7.271e-05 7.271e-05 7.271e-05 0.01% MLMG::mgVcycle_down::4 82 7.165e-05 7.165e-05 7.165e-05 0.01% AmrLevel::checkPoint() 3 7.134e-05 7.134e-05 7.134e-05 0.01% Castro::initialize_do_advance() 10 6.913e-05 6.913e-05 6.913e-05 0.01% MLMG::mgVcycle_up::4 82 5.802e-05 5.802e-05 5.802e-05 0.01% MLMG::mgVcycle_up::0 82 5.328e-05 5.328e-05 5.328e-05 0.01% MLCellLinOp::solutionResidual() 93 5.18e-05 5.18e-05 5.18e-05 0.01% MLMG::mgVcycle_up::1 82 5.09e-05 5.09e-05 5.09e-05 0.01% MLMG::mgVcycle_up::3 82 5e-05 5e-05 5e-05 0.01% MLMG::mgVcycle_up::2 82 4.996e-05 4.996e-05 4.996e-05 0.01% Castro::finalize_advance() 10 4.98e-05 4.98e-05 4.98e-05 0.01% Castro::construct_new_source() 50 4.022e-05 4.022e-05 4.022e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.751e-05 3.751e-05 3.751e-05 0.00% StateData::define() 4 3.707e-05 3.707e-05 3.707e-05 0.00% Castro::swap_state_time_levels() 10 3.703e-05 3.703e-05 3.703e-05 0.00% Castro::finalize_do_advance() 10 3.395e-05 3.395e-05 3.395e-05 0.00% MLMG::mgVcycle_bottom 82 3.393e-05 3.393e-05 3.393e-05 0.00% Castro::enforce_consistent_e() 1 3.351e-05 3.351e-05 3.351e-05 0.00% MLMG::computeResidual() 82 3.259e-05 3.259e-05 3.259e-05 0.00% Gravity::actual_multilevel_solve() 1 3.138e-05 3.138e-05 3.138e-05 0.00% FillPatchSingleLevel 41 2.891e-05 2.891e-05 2.891e-05 0.00% Castro::initMFs() 1 2.772e-05 2.772e-05 2.772e-05 0.00% makeSFC 55 2.636e-05 2.636e-05 2.636e-05 0.00% Amr::writeSmallPlotFile() 1 2.502e-05 2.502e-05 2.502e-05 0.00% Amr::defBaseLevel() 1 2.323e-05 2.323e-05 2.323e-05 0.00% Castro::buildMetrics() 1 2.285e-05 2.285e-05 2.285e-05 0.00% MLLinOp::define() 11 2.221e-05 2.221e-05 2.221e-05 0.00% MLPoisson::define() 11 2.106e-05 2.106e-05 2.106e-05 0.00% Amr::FinalizeInit() 1 1.957e-05 1.957e-05 1.957e-05 0.00% Castro::construct_old_source() 50 1.849e-05 1.849e-05 1.849e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.761e-05 1.761e-05 1.761e-05 0.00% Castro::do_new_sources() 10 1.671e-05 1.671e-05 1.671e-05 0.00% Castro::do_old_sources() 10 1.636e-05 1.636e-05 1.636e-05 0.00% DistributionMapping::Distribute() 56 1.473e-05 1.473e-05 1.473e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.421e-05 1.421e-05 1.421e-05 0.00% Castro::check_for_nan() 20 1.197e-05 1.197e-05 1.197e-05 0.00% Castro::apply_source_to_state() 20 1.062e-05 1.062e-05 1.062e-05 0.00% Castro::construct_old_gravity() 10 9.739e-06 9.739e-06 9.739e-06 0.00% Castro::post_timestep() 10 9.578e-06 9.578e-06 9.578e-06 0.00% Amr::initSubcycle() 1 8.86e-06 8.86e-06 8.86e-06 0.00% Gravity::swapTimeLevels() 10 8.647e-06 8.647e-06 8.647e-06 0.00% MLPoisson::prepareForSolve() 11 8.185e-06 8.185e-06 8.185e-06 0.00% MLMG::computeMLResidual() 11 8.166e-06 8.166e-06 8.166e-06 0.00% Castro::computeNewDt() 9 6.692e-06 6.692e-06 6.692e-06 0.00% Amr::InitializeInit() 1 6.613e-06 6.613e-06 6.613e-06 0.00% AmrLevel::AmrLevel(dm) 1 5.938e-06 5.938e-06 5.938e-06 0.00% MLMG::getGradSolution() 11 5.882e-06 5.882e-06 5.882e-06 0.00% MLMG::buildFineMask() 11 4.822e-06 4.822e-06 4.822e-06 0.00% Castro::create_source_corrector() 10 4.624e-06 4.624e-06 4.624e-06 0.00% MLMG::MLResNormInf() 11 4.466e-06 4.466e-06 4.466e-06 0.00% Gravity::set_mass_offset() 11 4.436e-06 4.436e-06 4.436e-06 0.00% AmrLevel::checkPointPost() 3 4.284e-06 4.284e-06 4.284e-06 0.00% Castro::post_init() 1 3.96e-06 3.96e-06 3.96e-06 0.00% Castro::retry_advance_ctu() 10 3.69e-06 3.69e-06 3.69e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.32e-06 3.32e-06 3.32e-06 0.00% Castro::FluxRegCrseInit 10 3.3e-06 3.3e-06 3.3e-06 0.00% Amr::init() 1 2.934e-06 2.934e-06 2.934e-06 0.00% Castro::computeInitialDt() 2 2.692e-06 2.692e-06 2.692e-06 0.00% Castro::FluxRegFineAdd() 10 2.39e-06 2.39e-06 2.39e-06 0.00% AmrLevel::checkPointPre() 3 2.089e-06 2.089e-06 2.089e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.971e-06 1.971e-06 1.971e-06 0.00% Amr::initialInit() 1 1.417e-06 1.417e-06 1.417e-06 0.00% Castro::post_regrid() 1 1.296e-06 1.296e-06 1.296e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8515 0.8515 0.8515 100.00% Amr::coarseTimeStep() 10 0.7022 0.7022 0.7022 82.47% Amr::timeStep() 10 0.6144 0.6144 0.6144 72.16% Castro::advance() 10 0.6077 0.6077 0.6077 71.37% Castro::subcycle_advance_ctu() 10 0.5959 0.5959 0.5959 69.98% Castro::do_advance_ctu() 10 0.5957 0.5957 0.5957 69.96% Gravity::solve_phi_with_mlmg() 11 0.321 0.321 0.321 37.70% Gravity::actual_solve_with_mlmg() 11 0.3126 0.3126 0.3126 36.71% Castro::construct_new_gravity() 10 0.2983 0.2983 0.2983 35.03% MLMG::solve() 11 0.2859 0.2859 0.2859 33.57% Gravity::solve_for_phi() 10 0.2764 0.2764 0.2764 32.46% MLMG::oneIter() 82 0.2712 0.2712 0.2712 31.85% MLMG::mgVcycle() 82 0.2695 0.2695 0.2695 31.65% Castro::construct_ctu_hydro_source() 10 0.2202 0.2202 0.2202 25.86% VisMF::Write(FabArray) 11 0.1732 0.1732 0.1732 20.34% MLCellLinOp::smooth() 1640 0.1377 0.1377 0.1377 16.17% Amr::checkPoint() 3 0.1286 0.1286 0.1286 15.11% AmrLevel::checkPoint() 3 0.1238 0.1238 0.1238 14.54% StateData::checkPoint() 12 0.1238 0.1238 0.1238 14.53% Amr::init() 1 0.1226 0.1226 0.1226 14.39% MLCellLinOp::applyBC() 4433 0.09725 0.09725 0.09725 11.42% MLMG::mgVcycle_bottom 82 0.08315 0.08315 0.08315 9.77% MLMG::actualBottomSolve() 82 0.08312 0.08312 0.08312 9.76% MLCGSolver::bicgstab 82 0.08228 0.08228 0.08228 9.66% MLPoisson::Fsmooth() 3280 0.06409 0.06409 0.06409 7.53% Amr::initialInit() 1 0.05218 0.05218 0.05218 6.13% Amr::writePlotFile() 2 0.05188 0.05188 0.05188 6.09% Amr::FinalizeInit() 1 0.04817 0.04817 0.04817 5.66% FillPatchIterator::Initialize 41 0.04774 0.04774 0.04774 5.61% Castro::post_init() 1 0.04685 0.04685 0.04685 5.50% FillPatchSingleLevel 41 0.04612 0.04612 0.04612 5.42% Castro::clean_state() 62 0.04533 0.04533 0.04533 5.32% Gravity::multilevel_solve_for_new_phi() 1 0.04509 0.04509 0.04509 5.29% Gravity::actual_multilevel_solve() 1 0.04507 0.04507 0.04507 5.29% StateDataPhysBCFunct::() 41 0.04216 0.04216 0.04216 4.95% MLCellLinOp::apply() 1142 0.0368 0.0368 0.0368 4.32% MLMG::mgVcycle_down::0 82 0.03584 0.03584 0.03584 4.21% MLMG::mgVcycle_up::0 82 0.03072 0.03072 0.03072 3.61% StateData::FillBoundary(geom) 328 0.02441 0.02441 0.02441 2.87% Gravity::get_new_grav_vector() 11 0.02336 0.02336 0.02336 2.74% MultiFab::Dot() 1114 0.02258 0.02258 0.02258 2.65% MLPoisson::define() 11 0.02232 0.02232 0.02232 2.62% MLCellLinOp::correctionResidual() 492 0.02158 0.02158 0.02158 2.53% Castro::computeTemp() 63 0.01978 0.01978 0.01978 2.32% MLMG:computeResOfCorrection() 410 0.01862 0.01862 0.01862 2.19% Castro::initialize_do_advance() 10 0.0181 0.0181 0.0181 2.13% MLMG::mgVcycle_down::1 82 0.01793 0.01793 0.01793 2.11% MLMG::mgVcycle_down::2 82 0.01746 0.01746 0.01746 2.05% MLMG::mgVcycle_down::3 82 0.01654 0.01654 0.01654 1.94% FabArray::FillBoundary() 4023 0.01581 0.01581 0.01581 1.86% MLMG::mgVcycle_down::4 82 0.01574 0.01574 0.01574 1.85% FillBoundary_nowait() 4023 0.01501 0.01501 0.01501 1.76% CGSolver::sxay() 1586 0.01499 0.01499 0.01499 1.76% Castro::construct_old_gravity() 10 0.01488 0.01488 0.01488 1.75% Gravity::get_old_grav_vector() 10 0.01487 0.01487 0.01487 1.75% MultiFab::LinComb() 1586 0.01462 0.01462 0.01462 1.72% FabArray::setVal() 1144 0.01439 0.01439 0.01439 1.69% FabArray::ParallelCopy() 861 0.01431 0.01431 0.01431 1.68% FabArray::ParallelCopy_nowait() 861 0.01403 0.01403 0.01403 1.65% Castro::do_new_sources() 10 0.0137 0.0137 0.0137 1.61% Castro::normalize_species() 62 0.01365 0.01365 0.01365 1.60% MLCGSolver::ParallelAllReduce 1514 0.01346 0.01346 0.01346 1.58% MLMG::mgVcycle_up::2 82 0.0134 0.0134 0.0134 1.57% MLMG::mgVcycle_up::1 82 0.01319 0.01319 0.01319 1.55% MLCellLinOp::defineAuxData() 11 0.01306 0.01306 0.01306 1.53% MLMG::addInterpCorrection() 410 0.01281 0.01281 0.01281 1.50% MLMG::mgVcycle_up::3 82 0.01271 0.01271 0.01271 1.49% MLMG::mgVcycle_up::4 82 0.01256 0.01256 0.01256 1.47% amrex::average_down 410 0.01212 0.01212 0.01212 1.42% MLPoisson::Fapply() 1142 0.01193 0.01193 0.01193 1.40% Castro::initialize_advance() 10 0.01168 0.01168 0.01168 1.37% Castro::enforce_min_density() 62 0.01167 0.01167 0.01167 1.37% Castro::expand_state() 10 0.01109 0.01109 0.01109 1.30% Castro::do_old_sources() 10 0.01057 0.01057 0.01057 1.24% MLCellLinOp::defineBC() 11 0.008939 0.008939 0.008939 1.05% BndryData::define() 11 0.008627 0.008627 0.008627 1.01% Gravity::fill_multipole_BCs() 11 0.008164 0.008164 0.008164 0.96% MLCellLinOp::solutionResidual() 93 0.007217 0.007217 0.007217 0.85% MultiFab::Xpay() 585 0.006625 0.006625 0.006625 0.78% Castro::post_timestep() 10 0.006485 0.006485 0.006485 0.76% MLMG::computeResidual() 82 0.006197 0.006197 0.006197 0.73% Castro::estTimeStep() 21 0.005757 0.005757 0.005757 0.68% Castro::reset_internal_energy(MultiFab) 63 0.005477 0.005477 0.005477 0.64% MLMG::prepareForSolve() 11 0.005219 0.005219 0.005219 0.61% Amr::InitializeInit() 1 0.004006 0.004006 0.004006 0.47% Amr::defBaseLevel() 1 0.004 0.004 0.004 0.47% Castro::initData() 1 0.003491 0.003491 0.003491 0.41% Castro::construct_new_source() 50 0.00326 0.00326 0.00326 0.38% Castro::construct_new_gravity_source() 10 0.00322 0.00322 0.00322 0.38% Castro::computeNewDt() 9 0.002771 0.002771 0.002771 0.33% Castro::construct_old_source() 50 0.002521 0.002521 0.002521 0.30% Castro::construct_old_gravity_source() 10 0.002503 0.002503 0.002503 0.29% MLMG::ResNormInf() 93 0.001954 0.001954 0.001954 0.23% Castro::apply_source_to_state() 20 0.001822 0.001822 0.001822 0.21% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00178 0.00178 0.00178 0.21% MLCellLinOp::setLevelBC() 11 0.001558 0.001558 0.001558 0.18% Castro::reset_internal_energy(Fab) 504 0.001543 0.001543 0.001543 0.18% FabArrayBase::getCPC() 1323 0.001461 0.001461 0.001461 0.17% MLMG::getGradSolution() 11 0.00145 0.00145 0.00145 0.17% MLCellLinOp::compGrad() 11 0.001444 0.001444 0.001444 0.17% FabArray::mult() 43 0.001334 0.001334 0.001334 0.16% FabArray::setDomainBndry() 41 0.001329 0.001329 0.001329 0.16% MLPoisson::prepareForSolve() 11 0.001213 0.001213 0.001213 0.14% MLCellLinOp::prepareForSolve() 11 0.001205 0.001205 0.001205 0.14% Castro::check_for_nan() 20 0.001184 0.001184 0.001184 0.14% MultiFab::contains_nan() 20 0.001172 0.001172 0.001172 0.14% Castro::post_regrid() 1 0.00109 0.00109 0.00109 0.13% MLMG::computeMLResidual() 11 0.001061 0.001061 0.001061 0.12% Castro::enforce_speed_limit() 62 0.0009898 0.0009898 0.0009898 0.12% Gravity::update_max_rhs() 11 0.0008166 0.0008166 0.0008166 0.10% FabArrayBase::getFB() 4023 0.0007082 0.0007082 0.0007082 0.08% Castro::computeInitialDt() 2 0.0006965 0.0006965 0.0006965 0.08% FabArrayBase::CPC::define() 454 0.0006649 0.0006649 0.0006649 0.08% Amr::InitAmr() 1 0.0005025 0.0005025 0.0005025 0.06% Castro::Castro() 1 0.0004383 0.0004383 0.0004383 0.05% Gravity::swapTimeLevels() 10 0.0004355 0.0004355 0.0004355 0.05% MLLinOp::define() 11 0.0002941 0.0002941 0.0002941 0.03% MLLinOp::defineGrids() 11 0.0002719 0.0002719 0.0002719 0.03% MultiFab::Copy() 11 0.0002631 0.0002631 0.0002631 0.03% MLMG::MLResNormInf() 11 0.0002617 0.0002617 0.0002617 0.03% MultiFab::max() 11 0.0002574 0.0002574 0.0002574 0.03% MLMG::MLRhsNormInf() 11 0.0002032 0.0002032 0.0002032 0.02% Castro::buildMetrics() 1 0.000162 0.000162 0.000162 0.02% FabArrayBase::FB::FB() 56 8.648e-05 8.648e-05 8.648e-05 0.01% Castro::finalize_advance() 10 5.549e-05 5.549e-05 5.549e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.43e-05 5.43e-05 5.43e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.301e-05 4.301e-05 4.301e-05 0.01% makeSFC 55 4.009e-05 4.009e-05 4.009e-05 0.00% StateData::define() 4 3.707e-05 3.707e-05 3.707e-05 0.00% Castro::swap_state_time_levels() 10 3.703e-05 3.703e-05 3.703e-05 0.00% Castro::finalize_do_advance() 10 3.395e-05 3.395e-05 3.395e-05 0.00% Castro::enforce_consistent_e() 1 3.351e-05 3.351e-05 3.351e-05 0.00% Castro::initMFs() 1 2.772e-05 2.772e-05 2.772e-05 0.00% Amr::writeSmallPlotFile() 1 2.502e-05 2.502e-05 2.502e-05 0.00% DistributionMapping::Distribute() 56 1.473e-05 1.473e-05 1.473e-05 0.00% Amr::initSubcycle() 1 8.86e-06 8.86e-06 8.86e-06 0.00% MLMG::buildFineMask() 11 4.822e-06 4.822e-06 4.822e-06 0.00% Castro::create_source_corrector() 10 4.624e-06 4.624e-06 4.624e-06 0.00% Gravity::set_mass_offset() 11 4.436e-06 4.436e-06 4.436e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.318e-06 4.318e-06 4.318e-06 0.00% AmrLevel::checkPointPost() 3 4.284e-06 4.284e-06 4.284e-06 0.00% Castro::retry_advance_ctu() 10 3.69e-06 3.69e-06 3.69e-06 0.00% Castro::FluxRegCrseInit 10 3.3e-06 3.3e-06 3.3e-06 0.00% Castro::FluxRegFineAdd() 10 2.39e-06 2.39e-06 2.39e-06 0.00% AmrLevel::checkPointPre() 3 2.089e-06 2.089e-06 2.089e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.971e-06 1.971e-06 1.971e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.06-20-g74183fe377df) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.06-20-g74183fe377df) initialized Starting run at 08:28:49 UTC on 2022-06-13. Successfully read inputs file ... Castro git describe: 22.06-12-g556652b03 AMReX git describe: 22.06-20-g74183fe37 Microphysics git describe: 22.06-1-gb8e427c1 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.470470669 Restart time = 0.046407015 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052275655 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049243078 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.062338689 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.065751587 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.069718532 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.027022931 seconds Ending run at 08:28:50 UTC on 2022-06-13. Run time = 0.373778576 Run time without initialization = 0.326735809 Average number of zones advanced per microsecond: 4.012 Average number of zones advanced per microsecond per rank: 4.012 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3738 ... 0.3738 ... 0.3738 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1015 0.1015 0.1015 27.16% VisMF::Read() 3 0.03888 0.03888 0.03888 10.40% MLCellLinOp::applyBC() 1946 0.03484 0.03484 0.03484 9.32% MLPoisson::Fsmooth() 1440 0.02711 0.02711 0.02711 7.25% VisMF::Write(FabArray) 1 0.0256 0.0256 0.0256 6.85% StateData::FillBoundary(geom) 160 0.01198 0.01198 0.01198 3.21% MLCGSolver::bicgstab 36 0.01032 0.01032 0.01032 2.76% MultiFab::Dot() 484 0.00953 0.00953 0.00953 2.55% Castro::computeTemp() 30 0.007749 0.007749 0.007749 2.07% Castro::normalize_species() 30 0.00727 0.00727 0.00727 1.94% FabArray::setVal() 537 0.006785 0.006785 0.006785 1.82% Castro::enforce_min_density() 30 0.006696 0.006696 0.006696 1.79% MLCellLinOp::defineAuxData() 6 0.00631 0.00631 0.00631 1.69% FillBoundary_nowait() 1766 0.006231 0.006231 0.006231 1.67% MultiFab::LinComb() 690 0.006161 0.006161 0.006161 1.65% FabArray::ParallelCopy_nowait() 380 0.005943 0.005943 0.005943 1.59% StateDataPhysBCFunct::() 20 0.005097 0.005097 0.005097 1.36% MLPoisson::Fapply() 500 0.005059 0.005059 0.005059 1.35% Gravity::fill_multipole_BCs() 6 0.004404 0.004404 0.004404 1.18% MLMG::addInterpCorrection() 180 0.00326 0.00326 0.00326 0.87% Amr::restart() 1 0.003094 0.003094 0.003094 0.83% amrex::average_down 180 0.002985 0.002985 0.002985 0.80% MultiFab::Xpay() 258 0.002833 0.002833 0.002833 0.76% Castro::do_advance_ctu() 5 0.002433 0.002433 0.002433 0.65% Castro::estTimeStep() 10 0.002194 0.002194 0.002194 0.59% BndryData::define() 6 0.00218 0.00218 0.00218 0.58% Castro::construct_new_gravity_source() 5 0.001743 0.001743 0.001743 0.47% Castro::reset_internal_energy(MultiFab) 30 0.001665 0.001665 0.001665 0.45% Amr::writePlotFile() 1 0.001511 0.001511 0.001511 0.40% Castro::construct_old_gravity_source() 5 0.00151 0.00151 0.00151 0.40% Castro::enforce_speed_limit() 30 0.001271 0.001271 0.001271 0.34% Gravity::get_old_grav_vector() 5 0.0009846 0.0009846 0.0009846 0.26% Castro::reset_internal_energy(Fab) 240 0.0009671 0.0009671 0.0009671 0.26% Gravity::get_new_grav_vector() 5 0.0009534 0.0009534 0.0009534 0.26% MultiFab::Saxpy() 10 0.0009239 0.0009239 0.0009239 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008861 0.0008861 0.0008861 0.24% Castro::expand_state() 5 0.0008667 0.0008667 0.0008667 0.23% MLMG::ResNormInf() 42 0.0008479 0.0008479 0.0008479 0.23% MLCellLinOp::setLevelBC() 6 0.0008229 0.0008229 0.0008229 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007577 0.0007577 0.0007577 0.20% MLMG::oneIter() 36 0.0007398 0.0007398 0.0007398 0.20% MLCellLinOp::prepareForSolve() 6 0.0006504 0.0006504 0.0006504 0.17% FabArray::mult() 22 0.0006445 0.0006445 0.0006445 0.17% FabArray::setDomainBndry() 20 0.0006379 0.0006379 0.0006379 0.17% MultiFab::contains_nan() 10 0.0005935 0.0005935 0.0005935 0.16% MLMG::prepareForSolve() 6 0.000593 0.000593 0.000593 0.16% MLCellLinOp::smooth() 720 0.0005053 0.0005053 0.0005053 0.14% MLCellLinOp::compGrad() 6 0.0004862 0.0004862 0.0004862 0.13% Amr::InitAmr() 1 0.0004547 0.0004547 0.0004547 0.12% FabArrayBase::CPC::define() 244 0.0004119 0.0004119 0.0004119 0.11% FabArray::FillBoundary() 1766 0.000379 0.000379 0.000379 0.10% FabArrayBase::getCPC() 632 0.0003663 0.0003663 0.0003663 0.10% FabArrayBase::getFB() 1766 0.0002581 0.0002581 0.0002581 0.07% Castro::subcycle_advance_ctu() 5 0.0002499 0.0002499 0.0002499 0.07% main() 1 0.0002494 0.0002494 0.0002494 0.07% Gravity::update_max_rhs() 6 0.0002274 0.0002274 0.0002274 0.06% Gravity::solve_for_phi() 5 0.0002213 0.0002213 0.0002213 0.06% MLCellLinOp::apply() 500 0.0002054 0.0002054 0.0002054 0.05% Castro::create_source_corrector() 5 0.0002028 0.0002028 0.0002028 0.05% CGSolver::sxay() 690 0.0001749 0.0001749 0.0001749 0.05% MLCellLinOp::defineBC() 6 0.0001585 0.0001585 0.0001585 0.04% Castro::advance() 5 0.0001567 0.0001567 0.0001567 0.04% Castro::construct_new_source() 25 0.0001528 0.0001528 0.0001528 0.04% Amr::coarseTimeStep() 5 0.0001524 0.0001524 0.0001524 0.04% MultiFab::Copy() 6 0.0001384 0.0001384 0.0001384 0.04% Castro::construct_new_gravity() 5 0.0001377 0.0001377 0.0001377 0.04% FillPatchIterator::Initialize 20 0.0001366 0.0001366 0.0001366 0.04% MultiFab::max() 6 0.0001336 0.0001336 0.0001336 0.04% FabArray::ParallelCopy() 380 0.0001286 0.0001286 0.0001286 0.03% MLCGSolver::ParallelAllReduce 659 0.0001196 0.0001196 0.0001196 0.03% MLMG::MLRhsNormInf() 6 0.0001067 0.0001067 0.0001067 0.03% Amr::timeStep() 5 9.888e-05 9.888e-05 9.888e-05 0.03% MLCellLinOp::correctionResidual() 216 9.848e-05 9.848e-05 9.848e-05 0.03% MLMG::mgVcycle() 36 9.447e-05 9.447e-05 9.447e-05 0.03% MLLinOp::defineGrids() 6 9.234e-05 9.234e-05 9.234e-05 0.02% Castro::construct_old_source() 25 8.813e-05 8.813e-05 8.813e-05 0.02% Castro::computeNewDt() 5 8.029e-05 8.029e-05 8.029e-05 0.02% AmrLevel::restart() 1 7.646e-05 7.646e-05 7.646e-05 0.02% StateData::restartDoit() 4 7.556e-05 7.556e-05 7.556e-05 0.02% Castro::finalize_advance() 5 6.682e-05 6.682e-05 6.682e-05 0.02% FabArrayBase::FB::FB() 26 5.575e-05 5.575e-05 5.575e-05 0.01% MLMG:computeResOfCorrection() 180 5.459e-05 5.459e-05 5.459e-05 0.01% MLMG::actualBottomSolve() 36 4.626e-05 4.626e-05 4.626e-05 0.01% Castro::initialize_do_advance() 5 4.503e-05 4.503e-05 4.503e-05 0.01% Castro::initialize_advance() 5 3.941e-05 3.941e-05 3.941e-05 0.01% Castro::clean_state() 30 3.91e-05 3.91e-05 3.91e-05 0.01% MLMG::mgVcycle_down::0 36 3.87e-05 3.87e-05 3.87e-05 0.01% Castro::post_restart() 1 3.71e-05 3.71e-05 3.71e-05 0.01% MLMG::mgVcycle_down::1 36 3.612e-05 3.612e-05 3.612e-05 0.01% MLMG::solve() 6 3.6e-05 3.6e-05 3.6e-05 0.01% MLMG::mgVcycle_down::2 36 3.552e-05 3.552e-05 3.552e-05 0.01% MLMG::mgVcycle_down::4 36 3.319e-05 3.319e-05 3.319e-05 0.01% Castro::buildMetrics() 1 3.239e-05 3.239e-05 3.239e-05 0.01% MLMG::mgVcycle_down::3 36 3.158e-05 3.158e-05 3.158e-05 0.01% Gravity::actual_multilevel_solve() 1 2.985e-05 2.985e-05 2.985e-05 0.01% MLMG::mgVcycle_up::4 36 2.808e-05 2.808e-05 2.808e-05 0.01% Castro::swap_state_time_levels() 5 2.77e-05 2.77e-05 2.77e-05 0.01% Castro::initMFs() 1 2.653e-05 2.653e-05 2.653e-05 0.01% Amr::writeSmallPlotFile() 1 2.621e-05 2.621e-05 2.621e-05 0.01% MLMG::mgVcycle_up::0 36 2.398e-05 2.398e-05 2.398e-05 0.01% MLCellLinOp::solutionResidual() 42 2.386e-05 2.386e-05 2.386e-05 0.01% MLMG::mgVcycle_up::3 36 2.34e-05 2.34e-05 2.34e-05 0.01% MLMG::mgVcycle_up::2 36 2.303e-05 2.303e-05 2.303e-05 0.01% Castro::post_timestep() 5 2.23e-05 2.23e-05 2.23e-05 0.01% MLLinOp::define() 6 2.138e-05 2.138e-05 2.138e-05 0.01% MLMG::mgVcycle_up::1 36 2.088e-05 2.088e-05 2.088e-05 0.01% Castro::finalize_do_advance() 5 1.84e-05 1.84e-05 1.84e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.771e-05 1.771e-05 1.771e-05 0.00% MLMG::computeResidual() 36 1.562e-05 1.562e-05 1.562e-05 0.00% MLMG::mgVcycle_bottom 36 1.42e-05 1.42e-05 1.42e-05 0.00% MLPoisson::define() 6 1.384e-05 1.384e-05 1.384e-05 0.00% makeSFC 30 1.339e-05 1.339e-05 1.339e-05 0.00% FillPatchSingleLevel 20 1.328e-05 1.328e-05 1.328e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.309e-05 1.309e-05 1.309e-05 0.00% Amr::initSubcycle() 1 9.8e-06 9.8e-06 9.8e-06 0.00% Castro::do_new_sources() 5 9.404e-06 9.404e-06 9.404e-06 0.00% DistributionMapping::Distribute() 31 8.663e-06 8.663e-06 8.663e-06 0.00% Castro::check_for_nan() 10 8.625e-06 8.625e-06 8.625e-06 0.00% Castro::do_old_sources() 5 8.507e-06 8.507e-06 8.507e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.133e-06 7.133e-06 7.133e-06 0.00% Castro::apply_source_to_state() 10 5.778e-06 5.778e-06 5.778e-06 0.00% Castro::construct_old_gravity() 5 5.173e-06 5.173e-06 5.173e-06 0.00% MLPoisson::prepareForSolve() 6 4.72e-06 4.72e-06 4.72e-06 0.00% Gravity::swapTimeLevels() 5 4.691e-06 4.691e-06 4.691e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.989e-06 3.989e-06 3.989e-06 0.00% MLMG::computeMLResidual() 6 3.123e-06 3.123e-06 3.123e-06 0.00% MLMG::getGradSolution() 6 3.047e-06 3.047e-06 3.047e-06 0.00% MLMG::buildFineMask() 6 3.027e-06 3.027e-06 3.027e-06 0.00% MLMG::MLResNormInf() 6 2.318e-06 2.318e-06 2.318e-06 0.00% Gravity::set_mass_offset() 6 2.307e-06 2.307e-06 2.307e-06 0.00% Castro::retry_advance_ctu() 5 2.094e-06 2.094e-06 2.094e-06 0.00% Castro::FluxRegCrseInit 5 1.69e-06 1.69e-06 1.69e-06 0.00% Amr::init() 1 1.616e-06 1.616e-06 1.616e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.165e-06 1.165e-06 1.165e-06 0.00% Castro::FluxRegFineAdd() 5 1.143e-06 1.143e-06 1.143e-06 0.00% AmrLevel::AmrLevel() 1 1.053e-06 1.053e-06 1.053e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3738 0.3738 0.3738 100.00% Amr::coarseTimeStep() 5 0.2995 0.2995 0.2995 80.11% Amr::timeStep() 5 0.2982 0.2982 0.2982 79.76% Castro::advance() 5 0.2945 0.2945 0.2945 78.78% Castro::subcycle_advance_ctu() 5 0.2878 0.2878 0.2878 76.99% Castro::do_advance_ctu() 5 0.2876 0.2876 0.2876 76.93% Castro::construct_new_gravity() 5 0.1434 0.1434 0.1434 38.36% Gravity::solve_phi_with_mlmg() 6 0.1393 0.1393 0.1393 37.28% Gravity::solve_for_phi() 5 0.1356 0.1356 0.1356 36.29% Gravity::actual_solve_with_mlmg() 6 0.1348 0.1348 0.1348 36.07% MLMG::solve() 6 0.1224 0.1224 0.1224 32.74% MLMG::oneIter() 36 0.1153 0.1153 0.1153 30.84% MLMG::mgVcycle() 36 0.1146 0.1146 0.1146 30.64% Castro::construct_ctu_hydro_source() 5 0.1015 0.1015 0.1015 27.16% MLCellLinOp::smooth() 720 0.05857 0.05857 0.05857 15.67% Amr::init() 1 0.04647 0.04647 0.04647 12.43% Amr::restart() 1 0.04647 0.04647 0.04647 12.43% MLCellLinOp::applyBC() 1946 0.04176 0.04176 0.04176 11.17% AmrLevel::restart() 1 0.03909 0.03909 0.03909 10.46% StateData::restartDoit() 4 0.03901 0.03901 0.03901 10.44% VisMF::Read() 3 0.03888 0.03888 0.03888 10.40% MLMG::mgVcycle_bottom 36 0.0352 0.0352 0.0352 9.42% MLMG::actualBottomSolve() 36 0.03519 0.03519 0.03519 9.41% MLCGSolver::bicgstab 36 0.03483 0.03483 0.03483 9.32% MLPoisson::Fsmooth() 1440 0.02711 0.02711 0.02711 7.25% Amr::writePlotFile() 1 0.02711 0.02711 0.02711 7.25% Castro::clean_state() 30 0.02566 0.02566 0.02566 6.86% VisMF::Write(FabArray) 1 0.0256 0.0256 0.0256 6.85% FillPatchIterator::Initialize 20 0.01987 0.01987 0.01987 5.31% FillPatchSingleLevel 20 0.01909 0.01909 0.01909 5.11% StateDataPhysBCFunct::() 20 0.01708 0.01708 0.01708 4.57% MLCellLinOp::apply() 500 0.0158 0.0158 0.0158 4.23% MLMG::mgVcycle_down::0 36 0.01535 0.01535 0.01535 4.11% MLMG::mgVcycle_up::0 36 0.01312 0.01312 0.01312 3.51% StateData::FillBoundary(geom) 160 0.01198 0.01198 0.01198 3.21% Castro::initialize_do_advance() 5 0.01099 0.01099 0.01099 2.94% Castro::computeTemp() 30 0.01038 0.01038 0.01038 2.78% MLPoisson::define() 6 0.01009 0.01009 0.01009 2.70% MultiFab::Dot() 484 0.00953 0.00953 0.00953 2.55% MLCellLinOp::correctionResidual() 216 0.009196 0.009196 0.009196 2.46% MLMG:computeResOfCorrection() 180 0.007922 0.007922 0.007922 2.12% Castro::construct_old_gravity() 5 0.007774 0.007774 0.007774 2.08% Gravity::get_old_grav_vector() 5 0.007769 0.007769 0.007769 2.08% MLMG::mgVcycle_down::1 36 0.007648 0.007648 0.007648 2.05% Gravity::get_new_grav_vector() 5 0.007635 0.007635 0.007635 2.04% MLMG::mgVcycle_down::2 36 0.007439 0.007439 0.007439 1.99% Castro::do_new_sources() 5 0.007385 0.007385 0.007385 1.98% Castro::normalize_species() 30 0.00727 0.00727 0.00727 1.94% MLCellLinOp::defineAuxData() 6 0.007054 0.007054 0.007054 1.89% MLMG::mgVcycle_down::3 36 0.007003 0.007003 0.007003 1.87% FabArray::FillBoundary() 1766 0.006924 0.006924 0.006924 1.85% FabArray::setVal() 537 0.006785 0.006785 0.006785 1.82% Castro::enforce_min_density() 30 0.006696 0.006696 0.006696 1.79% MLMG::mgVcycle_down::4 36 0.006692 0.006692 0.006692 1.79% FillBoundary_nowait() 1766 0.006545 0.006545 0.006545 1.75% Castro::initialize_advance() 5 0.00647 0.00647 0.00647 1.73% FabArray::ParallelCopy() 380 0.006452 0.006452 0.006452 1.73% CGSolver::sxay() 690 0.006336 0.006336 0.006336 1.69% FabArray::ParallelCopy_nowait() 380 0.006323 0.006323 0.006323 1.69% MultiFab::LinComb() 690 0.006161 0.006161 0.006161 1.65% MLCGSolver::ParallelAllReduce 659 0.005698 0.005698 0.005698 1.52% MLMG::mgVcycle_up::2 36 0.005692 0.005692 0.005692 1.52% MLMG::mgVcycle_up::1 36 0.005568 0.005568 0.005568 1.49% MLMG::addInterpCorrection() 180 0.005465 0.005465 0.005465 1.46% Castro::do_old_sources() 5 0.005395 0.005395 0.005395 1.44% MLMG::mgVcycle_up::3 36 0.005392 0.005392 0.005392 1.44% MLMG::mgVcycle_up::4 36 0.005355 0.005355 0.005355 1.43% amrex::average_down 180 0.005231 0.005231 0.005231 1.40% Castro::expand_state() 5 0.005199 0.005199 0.005199 1.39% MLPoisson::Fapply() 500 0.005059 0.005059 0.005059 1.35% Gravity::fill_multipole_BCs() 6 0.004404 0.004404 0.004404 1.18% Castro::post_restart() 1 0.0041 0.0041 0.0041 1.10% Gravity::multilevel_solve_for_new_phi() 1 0.003969 0.003969 0.003969 1.06% Gravity::actual_multilevel_solve() 1 0.003951 0.003951 0.003951 1.06% Castro::post_timestep() 5 0.003567 0.003567 0.003567 0.95% MLCellLinOp::solutionResidual() 42 0.003232 0.003232 0.003232 0.86% MLCellLinOp::defineBC() 6 0.002878 0.002878 0.002878 0.77% MLMG::prepareForSolve() 6 0.002857 0.002857 0.002857 0.76% MultiFab::Xpay() 258 0.002833 0.002833 0.002833 0.76% BndryData::define() 6 0.00272 0.00272 0.00272 0.73% MLMG::computeResidual() 36 0.002665 0.002665 0.002665 0.71% Castro::reset_internal_energy(MultiFab) 30 0.002632 0.002632 0.002632 0.70% Castro::estTimeStep() 10 0.002194 0.002194 0.002194 0.59% Castro::construct_new_source() 25 0.001896 0.001896 0.001896 0.51% Castro::construct_new_gravity_source() 5 0.001743 0.001743 0.001743 0.47% Castro::construct_old_source() 25 0.001598 0.001598 0.001598 0.43% Castro::construct_old_gravity_source() 5 0.00151 0.00151 0.00151 0.40% Castro::enforce_speed_limit() 30 0.001271 0.001271 0.001271 0.34% Castro::computeNewDt() 5 0.001155 0.001155 0.001155 0.31% Castro::reset_internal_energy(Fab) 240 0.0009671 0.0009671 0.0009671 0.26% Castro::apply_source_to_state() 10 0.0009296 0.0009296 0.0009296 0.25% MultiFab::Saxpy() 10 0.0009239 0.0009239 0.0009239 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008861 0.0008861 0.0008861 0.24% MLMG::ResNormInf() 42 0.0008479 0.0008479 0.0008479 0.23% MLCellLinOp::setLevelBC() 6 0.0008229 0.0008229 0.0008229 0.22% FabArrayBase::getCPC() 632 0.0007783 0.0007783 0.0007783 0.21% MLMG::getGradSolution() 6 0.0007577 0.0007577 0.0007577 0.20% MLCellLinOp::compGrad() 6 0.0007547 0.0007547 0.0007547 0.20% MLPoisson::prepareForSolve() 6 0.0006551 0.0006551 0.0006551 0.18% MLCellLinOp::prepareForSolve() 6 0.0006504 0.0006504 0.0006504 0.17% FabArray::mult() 22 0.0006445 0.0006445 0.0006445 0.17% FabArray::setDomainBndry() 20 0.0006379 0.0006379 0.0006379 0.17% Castro::check_for_nan() 10 0.0006021 0.0006021 0.0006021 0.16% MultiFab::contains_nan() 10 0.0005935 0.0005935 0.0005935 0.16% MLMG::computeMLResidual() 6 0.0005854 0.0005854 0.0005854 0.16% Amr::InitAmr() 1 0.0004645 0.0004645 0.0004645 0.12% Gravity::update_max_rhs() 6 0.0004387 0.0004387 0.0004387 0.12% FabArrayBase::CPC::define() 244 0.0004119 0.0004119 0.0004119 0.11% FabArrayBase::getFB() 1766 0.0003139 0.0003139 0.0003139 0.08% Gravity::swapTimeLevels() 5 0.0002265 0.0002265 0.0002265 0.06% Castro::create_source_corrector() 5 0.0002028 0.0002028 0.0002028 0.05% Castro::buildMetrics() 1 0.000153 0.000153 0.000153 0.04% MLLinOp::define() 6 0.0001425 0.0001425 0.0001425 0.04% MultiFab::Copy() 6 0.0001384 0.0001384 0.0001384 0.04% MLMG::MLResNormInf() 6 0.0001341 0.0001341 0.0001341 0.04% MultiFab::max() 6 0.0001336 0.0001336 0.0001336 0.04% MLLinOp::defineGrids() 6 0.0001212 0.0001212 0.0001212 0.03% MLMG::MLRhsNormInf() 6 0.0001067 0.0001067 0.0001067 0.03% Castro::finalize_advance() 5 6.965e-05 6.965e-05 6.965e-05 0.02% FabArrayBase::FB::FB() 26 5.575e-05 5.575e-05 5.575e-05 0.01% Castro::swap_state_time_levels() 5 2.77e-05 2.77e-05 2.77e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.766e-05 2.766e-05 2.766e-05 0.01% Castro::initMFs() 1 2.653e-05 2.653e-05 2.653e-05 0.01% Amr::writeSmallPlotFile() 1 2.621e-05 2.621e-05 2.621e-05 0.01% makeSFC 30 2.052e-05 2.052e-05 2.052e-05 0.01% Castro::finalize_do_advance() 5 1.84e-05 1.84e-05 1.84e-05 0.00% Amr::initSubcycle() 1 9.8e-06 9.8e-06 9.8e-06 0.00% DistributionMapping::Distribute() 31 8.663e-06 8.663e-06 8.663e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.52e-06 5.52e-06 5.52e-06 0.00% MLMG::buildFineMask() 6 3.027e-06 3.027e-06 3.027e-06 0.00% Gravity::set_mass_offset() 6 2.307e-06 2.307e-06 2.307e-06 0.00% Castro::retry_advance_ctu() 5 2.094e-06 2.094e-06 2.094e-06 0.00% Castro::FluxRegCrseInit 5 1.69e-06 1.69e-06 1.69e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.165e-06 1.165e-06 1.165e-06 0.00% Castro::FluxRegFineAdd() 5 1.143e-06 1.143e-06 1.143e-06 0.00% AmrLevel::AmrLevel() 1 1.053e-06 1.053e-06 1.053e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.06-20-g74183fe377df) finalized