Initializing CUDA... CUDA initialized with 1 device. AMReX (23.01-4-gb3241d7b368a) initialized Starting run at 10:11:39 UTC on 2023-01-04. Successfully read inputs file ... Castro git describe: 22.12-11-g45d01ffe1 AMReX git describe: 23.01-4-gb3241d7b3 Microphysics git describe: 22.12-28-gdc4a7e21 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.058285937 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.033646609 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.046481254 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049731026 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.04803519 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.059006827 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.070245181 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.057648609 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.079981242 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048877428 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.056725781 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057902006 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.06040661 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.057908176 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033934383 seconds Ending run at 10:11:40 UTC on 2023-01-04. Run time = 0.868133153 Run time without initialization = 0.727588997 Average number of zones advanced per microsecond: 3.603 Average number of zones advanced per microsecond per rank: 3.603 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8682 ... 0.8682 ... 0.8682 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2344 0.2344 0.2344 27.00% Castro::construct_ctu_hydro_source() 10 0.2128 0.2128 0.2128 24.51% MLCellLinOp::applyBC() 4433 0.07508 0.07508 0.07508 8.65% MLPoisson::Fsmooth() 3280 0.03234 0.03234 0.03234 3.72% FillBoundary_nowait() 4023 0.03214 0.03214 0.03214 3.70% StateDataPhysBCFunct::() 41 0.02417 0.02417 0.02417 2.78% StateData::FillBoundary(geom) 328 0.02343 0.02343 0.02343 2.70% amrex::Dot() 1114 0.02051 0.02051 0.02051 2.36% Castro::normalize_species() 62 0.01591 0.01591 0.01591 1.83% amrex::Copy() 1029 0.01483 0.01483 0.01483 1.71% FabArray::norminf() 743 0.01435 0.01435 0.01435 1.65% Castro::computeTemp() 63 0.01385 0.01385 0.01385 1.60% FabArray::setVal() 1144 0.01313 0.01313 0.01313 1.51% FabArray::ParallelCopy_nowait() 861 0.01311 0.01311 0.01311 1.51% MLPoisson::Fapply() 1142 0.01044 0.01044 0.01044 1.20% MLCellLinOp::defineAuxData() 11 0.009647 0.009647 0.009647 1.11% FabArray::Saxpy() 813 0.008149 0.008149 0.008149 0.94% FabArray::Xpay() 821 0.008077 0.008077 0.008077 0.93% Castro::enforce_min_density() 62 0.006696 0.006696 0.006696 0.77% MLMG::addInterpCorrection() 410 0.00658 0.00658 0.00658 0.76% Gravity::fill_multipole_BCs() 11 0.00612 0.00612 0.00612 0.70% amrex::average_down 410 0.005811 0.005811 0.005811 0.67% Castro::estTimeStep() 21 0.005154 0.005154 0.005154 0.59% FabArray::LinComb() 557 0.004559 0.004559 0.004559 0.53% amrex::Add() 164 0.004343 0.004343 0.004343 0.50% Castro::reset_internal_energy(MultiFab) 63 0.00427 0.00427 0.00427 0.49% Amr::checkPoint() 3 0.004093 0.004093 0.004093 0.47% BndryData::define() 11 0.003675 0.003675 0.003675 0.42% Castro::construct_new_gravity_source() 10 0.002833 0.002833 0.002833 0.33% Castro::do_advance_ctu() 10 0.002608 0.002608 0.002608 0.30% Amr::writePlotFile() 2 0.002444 0.002444 0.002444 0.28% Castro::reset_internal_energy(Fab) 504 0.002315 0.002315 0.002315 0.27% MLCGSolver::bicgstab 82 0.002028 0.002028 0.002028 0.23% Castro::construct_old_gravity_source() 10 0.001955 0.001955 0.001955 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001659 0.001659 0.001659 0.19% MLCellLinOp::setLevelBC() 11 0.001389 0.001389 0.001389 0.16% Gravity::actual_solve_with_mlmg() 11 0.001379 0.001379 0.001379 0.16% FabArray::mult() 43 0.001345 0.001345 0.001345 0.15% FabArray::setDomainBndry() 41 0.001316 0.001316 0.001316 0.15% Castro::enforce_speed_limit() 62 0.00128 0.00128 0.00128 0.15% Castro::initData() 1 0.001218 0.001218 0.001218 0.14% MultiFab::contains_nan() 20 0.001183 0.001183 0.001183 0.14% MLCellLinOp::smooth() 1640 0.001135 0.001135 0.001135 0.13% MLCellLinOp::prepareForSolve() 11 0.001101 0.001101 0.001101 0.13% MLCellLinOp::compGrad() 11 0.000904 0.000904 0.000904 0.10% MLMG::prepareForSolve() 11 0.0008311 0.0008311 0.0008311 0.10% FabArray::FillBoundary() 4023 0.0008034 0.0008034 0.0008034 0.09% Castro::subcycle_advance_ctu() 10 0.0007379 0.0007379 0.0007379 0.08% FabArrayBase::getCPC() 1323 0.0007205 0.0007205 0.0007205 0.08% FabArrayBase::CPC::define() 454 0.000688 0.000688 0.000688 0.08% Gravity::get_new_grav_vector() 11 0.00061 0.00061 0.00061 0.07% FabArrayBase::getFB() 4023 0.0006066 0.0006066 0.0006066 0.07% Amr::InitAmr() 1 0.0005303 0.0005303 0.0005303 0.06% Gravity::get_old_grav_vector() 10 0.0005263 0.0005263 0.0005263 0.06% MLCellLinOp::apply() 1142 0.0005011 0.0005011 0.0005011 0.06% Amr::coarseTimeStep() 10 0.0003813 0.0003813 0.0003813 0.04% MLMG::mgVcycle() 82 0.0003753 0.0003753 0.0003753 0.04% main() 1 0.0003148 0.0003148 0.0003148 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002932 0.0002932 0.0002932 0.03% MultiFab::max() 11 0.00026 0.00026 0.00026 0.03% FabArray::ParallelCopy() 861 0.0002349 0.0002349 0.0002349 0.03% MLCellLinOp::correctionResidual() 492 0.0002276 0.0002276 0.0002276 0.03% FillPatchIterator::Initialize 41 0.0002071 0.0002071 0.0002071 0.02% MLCellLinOp::defineBC() 11 0.0002018 0.0002018 0.0002018 0.02% Gravity::solve_for_phi() 10 0.0001818 0.0001818 0.0001818 0.02% MLLinOp::defineGrids() 11 0.0001561 0.0001561 0.0001561 0.02% Amr::timeStep() 10 0.000155 0.000155 0.000155 0.02% StateData::checkPoint() 12 0.0001325 0.0001325 0.0001325 0.02% Gravity::update_max_rhs() 11 0.0001252 0.0001252 0.0001252 0.01% MLMG:computeResOfCorrection() 410 0.0001209 0.0001209 0.0001209 0.01% MLMG::mgVcycle_down::0 82 9.32e-05 9.32e-05 9.32e-05 0.01% MLMG::actualBottomSolve() 82 9.169e-05 9.169e-05 9.169e-05 0.01% Castro::finalize_advance() 10 8.916e-05 8.916e-05 8.916e-05 0.01% Castro::Castro() 1 8.548e-05 8.548e-05 8.548e-05 0.01% Castro::advance() 10 8.512e-05 8.512e-05 8.512e-05 0.01% FabArrayBase::FB::FB() 56 8.404e-05 8.404e-05 8.404e-05 0.01% MLMG::mgVcycle_down::1 82 8.308e-05 8.308e-05 8.308e-05 0.01% MLMG::mgVcycle_down::2 82 7.993e-05 7.993e-05 7.993e-05 0.01% Castro::clean_state() 62 7.779e-05 7.779e-05 7.779e-05 0.01% MLMG::mgVcycle_down::4 82 7.494e-05 7.494e-05 7.494e-05 0.01% MLMG::mgVcycle_down::3 82 7.478e-05 7.478e-05 7.478e-05 0.01% Castro::expand_state() 10 7.424e-05 7.424e-05 7.424e-05 0.01% AmrLevel::checkPoint() 3 7.347e-05 7.347e-05 7.347e-05 0.01% MLMG::solve() 11 7.256e-05 7.256e-05 7.256e-05 0.01% MLMG::mgVcycle_up::4 82 7.205e-05 7.205e-05 7.205e-05 0.01% Castro::initialize_advance() 10 7.067e-05 7.067e-05 7.067e-05 0.01% MLMG::oneIter() 82 5.515e-05 5.515e-05 5.515e-05 0.01% MLMG::mgVcycle_up::0 82 5.344e-05 5.344e-05 5.344e-05 0.01% MLMG::mgVcycle_up::1 82 5.077e-05 5.077e-05 5.077e-05 0.01% MLCellLinOp::solutionResidual() 93 5.069e-05 5.069e-05 5.069e-05 0.01% MLMG::mgVcycle_up::3 82 5.036e-05 5.036e-05 5.036e-05 0.01% MLMG::mgVcycle_up::2 82 4.947e-05 4.947e-05 4.947e-05 0.01% Castro::initialize_do_advance() 10 4.355e-05 4.355e-05 4.355e-05 0.01% Castro::enforce_consistent_e() 1 3.62e-05 3.62e-05 3.62e-05 0.00% Castro::construct_new_source() 50 3.544e-05 3.544e-05 3.544e-05 0.00% Castro::swap_state_time_levels() 10 3.51e-05 3.51e-05 3.51e-05 0.00% StateData::define() 4 3.48e-05 3.48e-05 3.48e-05 0.00% MLMG::computeResidual() 82 3.366e-05 3.366e-05 3.366e-05 0.00% Castro::finalize_do_advance() 10 3.242e-05 3.242e-05 3.242e-05 0.00% MLMG::mgVcycle_bottom 82 3.191e-05 3.191e-05 3.191e-05 0.00% MLMG::ResNormInf() 93 3.104e-05 3.104e-05 3.104e-05 0.00% FillPatchSingleLevel 41 2.878e-05 2.878e-05 2.878e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.843e-05 2.843e-05 2.843e-05 0.00% MLPoisson::define() 11 2.815e-05 2.815e-05 2.815e-05 0.00% Amr::writeSmallPlotFile() 1 2.616e-05 2.616e-05 2.616e-05 0.00% makeSFC 55 2.539e-05 2.539e-05 2.539e-05 0.00% Castro::construct_new_gravity() 10 2.521e-05 2.521e-05 2.521e-05 0.00% Castro::initMFs() 1 2.038e-05 2.038e-05 2.038e-05 0.00% Amr::FinalizeInit() 1 1.986e-05 1.986e-05 1.986e-05 0.00% Amr::defBaseLevel() 1 1.777e-05 1.777e-05 1.777e-05 0.00% Castro::buildMetrics() 1 1.758e-05 1.758e-05 1.758e-05 0.00% Castro::do_new_sources() 10 1.732e-05 1.732e-05 1.732e-05 0.00% Castro::construct_old_source() 50 1.697e-05 1.697e-05 1.697e-05 0.00% DistributionMapping::Distribute() 56 1.665e-05 1.665e-05 1.665e-05 0.00% Castro::do_old_sources() 10 1.522e-05 1.522e-05 1.522e-05 0.00% Castro::computeNewDt() 9 1.475e-05 1.475e-05 1.475e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.465e-05 1.465e-05 1.465e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.321e-05 1.321e-05 1.321e-05 0.00% Castro::post_init() 1 1.169e-05 1.169e-05 1.169e-05 0.00% Castro::check_for_nan() 20 1.084e-05 1.084e-05 1.084e-05 0.00% Castro::apply_source_to_state() 20 1.025e-05 1.025e-05 1.025e-05 0.00% MLLinOp::define() 11 9.731e-06 9.731e-06 9.731e-06 0.00% Castro::construct_old_gravity() 10 9.659e-06 9.659e-06 9.659e-06 0.00% AmrLevel::AmrLevel(dm) 1 9.652e-06 9.652e-06 9.652e-06 0.00% Amr::InitializeInit() 1 9.53e-06 9.53e-06 9.53e-06 0.00% Gravity::swapTimeLevels() 10 9.351e-06 9.351e-06 9.351e-06 0.00% Amr::initSubcycle() 1 8.79e-06 8.79e-06 8.79e-06 0.00% Castro::post_timestep() 10 8.501e-06 8.501e-06 8.501e-06 0.00% MLPoisson::prepareForSolve() 11 7.906e-06 7.906e-06 7.906e-06 0.00% Gravity::actual_multilevel_solve() 1 7.716e-06 7.716e-06 7.716e-06 0.00% MLMG::computeMLResidual() 11 7.578e-06 7.578e-06 7.578e-06 0.00% MLMG::getGradSolution() 11 6.119e-06 6.119e-06 6.119e-06 0.00% Gravity::set_mass_offset() 11 4.668e-06 4.668e-06 4.668e-06 0.00% AmrLevel::checkPointPost() 3 4.624e-06 4.624e-06 4.624e-06 0.00% Castro::FluxRegCrseInit 10 4.143e-06 4.143e-06 4.143e-06 0.00% Castro::retry_advance_ctu() 10 3.761e-06 3.761e-06 3.761e-06 0.00% MLMG::MLRhsNormInf() 11 3.668e-06 3.668e-06 3.668e-06 0.00% Castro::create_source_corrector() 10 3.659e-06 3.659e-06 3.659e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.47e-06 3.47e-06 3.47e-06 0.00% MLMG::MLResNormInf() 11 3.319e-06 3.319e-06 3.319e-06 0.00% Castro::computeInitialDt() 2 2.782e-06 2.782e-06 2.782e-06 0.00% Amr::init() 1 2.419e-06 2.419e-06 2.419e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.018e-06 2.018e-06 2.018e-06 0.00% Castro::FluxRegFineAdd() 10 1.988e-06 1.988e-06 1.988e-06 0.00% AmrLevel::checkPointPre() 3 1.859e-06 1.859e-06 1.859e-06 0.00% Castro::post_regrid() 1 1.198e-06 1.198e-06 1.198e-06 0.00% Amr::initialInit() 1 1.034e-06 1.034e-06 1.034e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8681 0.8681 0.8681 100.00% Amr::coarseTimeStep() 10 0.6934 0.6934 0.6934 79.87% Amr::timeStep() 10 0.574 0.574 0.574 66.12% Castro::advance() 10 0.5651 0.5651 0.5651 65.09% Castro::subcycle_advance_ctu() 10 0.5529 0.5529 0.5529 63.69% Castro::do_advance_ctu() 10 0.5522 0.5522 0.5522 63.60% Gravity::solve_phi_with_mlmg() 11 0.2813 0.2813 0.2813 32.40% Gravity::actual_solve_with_mlmg() 11 0.2747 0.2747 0.2747 31.64% Castro::construct_new_gravity() 10 0.2569 0.2569 0.2569 29.59% MLMG::solve() 11 0.2545 0.2545 0.2545 29.32% Gravity::solve_for_phi() 10 0.2414 0.2414 0.2414 27.81% MLMG::oneIter() 82 0.2404 0.2404 0.2404 27.69% MLMG::mgVcycle() 82 0.2368 0.2368 0.2368 27.27% VisMF::Write(FabArray) 11 0.2344 0.2344 0.2344 27.00% Castro::construct_ctu_hydro_source() 10 0.2128 0.2128 0.2128 24.51% Amr::checkPoint() 3 0.174 0.174 0.174 20.04% AmrLevel::checkPoint() 3 0.1699 0.1699 0.1699 19.57% StateData::checkPoint() 12 0.1698 0.1698 0.1698 19.56% Amr::init() 1 0.1399 0.1399 0.1399 16.11% MLCellLinOp::smooth() 1640 0.1166 0.1166 0.1166 13.43% MLCellLinOp::applyBC() 4433 0.1087 0.1087 0.1087 12.52% MLMG::mgVcycle_bottom 82 0.07344 0.07344 0.07344 8.46% MLMG::actualBottomSolve() 82 0.07341 0.07341 0.07341 8.46% MLCGSolver::bicgstab 82 0.07271 0.07271 0.07271 8.37% Amr::writePlotFile() 2 0.06772 0.06772 0.06772 7.80% FillPatchIterator::Initialize 41 0.05313 0.05313 0.05313 6.12% FillPatchSingleLevel 41 0.0516 0.0516 0.0516 5.94% Amr::initialInit() 1 0.0478 0.0478 0.0478 5.51% StateDataPhysBCFunct::() 41 0.04761 0.04761 0.04761 5.48% Amr::FinalizeInit() 1 0.04374 0.04374 0.04374 5.04% Castro::clean_state() 62 0.04359 0.04359 0.04359 5.02% Castro::post_init() 1 0.04237 0.04237 0.04237 4.88% Gravity::multilevel_solve_for_new_phi() 1 0.0404 0.0404 0.0404 4.65% Gravity::actual_multilevel_solve() 1 0.04038 0.04038 0.04038 4.65% MLCellLinOp::apply() 1142 0.03604 0.03604 0.03604 4.15% MLMG::mgVcycle_down::0 82 0.03408 0.03408 0.03408 3.93% FabArray::FillBoundary() 4023 0.03363 0.03363 0.03363 3.87% FillBoundary_nowait() 4023 0.03283 0.03283 0.03283 3.78% MLPoisson::Fsmooth() 3280 0.03234 0.03234 0.03234 3.72% MLMG::mgVcycle_up::0 82 0.02593 0.02593 0.02593 2.99% Castro::initialize_do_advance() 10 0.02576 0.02576 0.02576 2.97% StateData::FillBoundary(geom) 328 0.02343 0.02343 0.02343 2.70% MLCellLinOp::correctionResidual() 492 0.02211 0.02211 0.02211 2.55% amrex::Dot() 1114 0.02051 0.02051 0.02051 2.36% Castro::computeTemp() 63 0.02044 0.02044 0.02044 2.35% MLMG:computeResOfCorrection() 410 0.0195 0.0195 0.0195 2.25% Castro::expand_state() 10 0.01874 0.01874 0.01874 2.16% Castro::construct_old_gravity() 10 0.01854 0.01854 0.01854 2.14% Gravity::get_old_grav_vector() 10 0.01853 0.01853 0.01853 2.13% Gravity::get_new_grav_vector() 11 0.01713 0.01713 0.01713 1.97% MLPoisson::define() 11 0.01603 0.01603 0.01603 1.85% Castro::normalize_species() 62 0.01591 0.01591 0.01591 1.83% MLMG::mgVcycle_down::1 82 0.01554 0.01554 0.01554 1.79% amrex::Copy() 1029 0.01483 0.01483 0.01483 1.71% MLMG::mgVcycle_down::2 82 0.01456 0.01456 0.01456 1.68% FabArray::norminf() 743 0.01435 0.01435 0.01435 1.65% MLMG::mgVcycle_down::3 82 0.01422 0.01422 0.01422 1.64% FabArray::ParallelCopy() 861 0.01416 0.01416 0.01416 1.63% MLMG::mgVcycle_down::4 82 0.01403 0.01403 0.01403 1.62% FabArray::ParallelCopy_nowait() 861 0.01392 0.01392 0.01392 1.60% FabArray::setVal() 1144 0.01313 0.01313 0.01313 1.51% MLCGSolver::ParallelAllReduce 1514 0.01232 0.01232 0.01232 1.42% MLMG::addInterpCorrection() 410 0.01168 0.01168 0.01168 1.35% Castro::initialize_advance() 10 0.01148 0.01148 0.01148 1.32% MLMG::mgVcycle_up::4 82 0.01142 0.01142 0.01142 1.32% MLMG::mgVcycle_up::1 82 0.01128 0.01128 0.01128 1.30% MLMG::mgVcycle_up::2 82 0.01108 0.01108 0.01108 1.28% MLCellLinOp::defineAuxData() 11 0.01095 0.01095 0.01095 1.26% amrex::average_down 410 0.0109 0.0109 0.0109 1.26% MLMG::mgVcycle_up::3 82 0.01083 0.01083 0.01083 1.25% Castro::do_new_sources() 10 0.01055 0.01055 0.01055 1.22% MLPoisson::Fapply() 1142 0.01044 0.01044 0.01044 1.20% Castro::do_old_sources() 10 0.009726 0.009726 0.009726 1.12% Castro::post_timestep() 10 0.008737 0.008737 0.008737 1.01% FabArray::Saxpy() 813 0.008149 0.008149 0.008149 0.94% FabArray::Xpay() 821 0.008077 0.008077 0.008077 0.93% MLCellLinOp::solutionResidual() 93 0.007103 0.007103 0.007103 0.82% Castro::enforce_min_density() 62 0.006696 0.006696 0.006696 0.77% Castro::reset_internal_energy(MultiFab) 63 0.006585 0.006585 0.006585 0.76% Gravity::fill_multipole_BCs() 11 0.006368 0.006368 0.006368 0.73% MLMG::computeResidual() 82 0.006135 0.006135 0.006135 0.71% Castro::estTimeStep() 21 0.005154 0.005154 0.005154 0.59% MLCellLinOp::defineBC() 11 0.00483 0.00483 0.00483 0.56% BndryData::define() 11 0.004628 0.004628 0.004628 0.53% MLMG::prepareForSolve() 11 0.004581 0.004581 0.004581 0.53% FabArray::LinComb() 557 0.004559 0.004559 0.004559 0.53% amrex::Add() 164 0.004343 0.004343 0.004343 0.50% Amr::InitializeInit() 1 0.004066 0.004066 0.004066 0.47% Amr::defBaseLevel() 1 0.004057 0.004057 0.004057 0.47% Castro::initData() 1 0.003555 0.003555 0.003555 0.41% Castro::construct_new_source() 50 0.002868 0.002868 0.002868 0.33% Castro::construct_new_gravity_source() 10 0.002833 0.002833 0.002833 0.33% Castro::computeNewDt() 9 0.002615 0.002615 0.002615 0.30% Castro::reset_internal_energy(Fab) 504 0.002315 0.002315 0.002315 0.27% MLMG::ResNormInf() 93 0.002111 0.002111 0.002111 0.24% Castro::construct_old_source() 50 0.001972 0.001972 0.001972 0.23% Castro::construct_old_gravity_source() 10 0.001955 0.001955 0.001955 0.23% Castro::apply_source_to_state() 20 0.001818 0.001818 0.001818 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001659 0.001659 0.001659 0.19% FabArrayBase::getCPC() 1323 0.001409 0.001409 0.001409 0.16% MLCellLinOp::setLevelBC() 11 0.001389 0.001389 0.001389 0.16% MLMG::getGradSolution() 11 0.001379 0.001379 0.001379 0.16% MLCellLinOp::compGrad() 11 0.001373 0.001373 0.001373 0.16% FabArray::mult() 43 0.001345 0.001345 0.001345 0.15% FabArray::setDomainBndry() 41 0.001316 0.001316 0.001316 0.15% Castro::enforce_speed_limit() 62 0.00128 0.00128 0.00128 0.15% Castro::check_for_nan() 20 0.001194 0.001194 0.001194 0.14% Castro::post_regrid() 1 0.001183 0.001183 0.001183 0.14% MultiFab::contains_nan() 20 0.001183 0.001183 0.001183 0.14% MLPoisson::prepareForSolve() 11 0.001109 0.001109 0.001109 0.13% MLCellLinOp::prepareForSolve() 11 0.001101 0.001101 0.001101 0.13% MLMG::computeMLResidual() 11 0.001009 0.001009 0.001009 0.12% Castro::computeInitialDt() 2 0.0008996 0.0008996 0.0008996 0.10% Gravity::update_max_rhs() 11 0.0008284 0.0008284 0.0008284 0.10% FabArrayBase::getFB() 4023 0.0006906 0.0006906 0.0006906 0.08% FabArrayBase::CPC::define() 454 0.000688 0.000688 0.000688 0.08% Castro::finalize_advance() 10 0.0006212 0.0006212 0.0006212 0.07% Amr::InitAmr() 1 0.0005391 0.0005391 0.0005391 0.06% Gravity::swapTimeLevels() 10 0.0004359 0.0004359 0.0004359 0.05% Castro::Castro() 1 0.0004349 0.0004349 0.0004349 0.05% MLMG::MLResNormInf() 11 0.0002834 0.0002834 0.0002834 0.03% MultiFab::max() 11 0.00026 0.00026 0.00026 0.03% MLLinOp::define() 11 0.0002217 0.0002217 0.0002217 0.03% MLMG::MLRhsNormInf() 11 0.0002182 0.0002182 0.0002182 0.03% MLLinOp::defineGrids() 11 0.0002119 0.0002119 0.0002119 0.02% Castro::buildMetrics() 1 0.0001666 0.0001666 0.0001666 0.02% FabArrayBase::FB::FB() 56 8.404e-05 8.404e-05 8.404e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.377e-05 5.377e-05 5.377e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.445e-05 4.445e-05 4.445e-05 0.01% makeSFC 55 4.057e-05 4.057e-05 4.057e-05 0.00% Castro::enforce_consistent_e() 1 3.62e-05 3.62e-05 3.62e-05 0.00% Castro::swap_state_time_levels() 10 3.51e-05 3.51e-05 3.51e-05 0.00% StateData::define() 4 3.48e-05 3.48e-05 3.48e-05 0.00% Castro::finalize_do_advance() 10 3.242e-05 3.242e-05 3.242e-05 0.00% Amr::writeSmallPlotFile() 1 2.616e-05 2.616e-05 2.616e-05 0.00% Castro::initMFs() 1 2.038e-05 2.038e-05 2.038e-05 0.00% DistributionMapping::Distribute() 56 1.665e-05 1.665e-05 1.665e-05 0.00% Amr::initSubcycle() 1 8.79e-06 8.79e-06 8.79e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.938e-06 4.938e-06 4.938e-06 0.00% Gravity::set_mass_offset() 11 4.668e-06 4.668e-06 4.668e-06 0.00% AmrLevel::checkPointPost() 3 4.624e-06 4.624e-06 4.624e-06 0.00% Castro::FluxRegCrseInit 10 4.143e-06 4.143e-06 4.143e-06 0.00% Castro::retry_advance_ctu() 10 3.761e-06 3.761e-06 3.761e-06 0.00% Castro::create_source_corrector() 10 3.659e-06 3.659e-06 3.659e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.018e-06 2.018e-06 2.018e-06 0.00% Castro::FluxRegFineAdd() 10 1.988e-06 1.988e-06 1.988e-06 0.00% AmrLevel::checkPointPre() 3 1.859e-06 1.859e-06 1.859e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-4-gb3241d7b368a) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.01-4-gb3241d7b368a) initialized Starting run at 10:11:41 UTC on 2023-01-04. Successfully read inputs file ... Castro git describe: 22.12-11-g45d01ffe1 AMReX git describe: 23.01-4-gb3241d7b3 Microphysics git describe: 22.12-28-gdc4a7e21 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.47169575 Restart time = 0.047119805 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.047774132 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047667884 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.046674645 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.056193014 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.060563414 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033034703 seconds Ending run at 10:11:41 UTC on 2023-01-04. Run time = 0.34001502 Run time without initialization = 0.292287943 Average number of zones advanced per microsecond: 4.484 Average number of zones advanced per microsecond per rank: 4.484 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.34 ... 0.34 ... 0.34 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0829 0.0829 0.0829 24.37% VisMF::Read() 3 0.04087 0.04087 0.04087 12.02% MLCellLinOp::applyBC() 1946 0.03239 0.03239 0.03239 9.52% VisMF::Write(FabArray) 1 0.03149 0.03149 0.03149 9.26% MLPoisson::Fsmooth() 1440 0.01404 0.01404 0.01404 4.13% FillBoundary_nowait() 1766 0.01288 0.01288 0.01288 3.79% StateData::FillBoundary(geom) 160 0.01148 0.01148 0.01148 3.38% amrex::Dot() 484 0.008771 0.008771 0.008771 2.58% Castro::normalize_species() 30 0.007094 0.007094 0.007094 2.09% amrex::Copy() 463 0.007047 0.007047 0.007047 2.07% FabArray::setVal() 537 0.006247 0.006247 0.006247 1.84% FabArray::norminf() 326 0.006226 0.006226 0.006226 1.83% Castro::computeTemp() 30 0.005984 0.005984 0.005984 1.76% FabArray::ParallelCopy_nowait() 380 0.005937 0.005937 0.005937 1.75% StateDataPhysBCFunct::() 20 0.005735 0.005735 0.005735 1.69% MLCellLinOp::defineAuxData() 6 0.005182 0.005182 0.005182 1.52% MLPoisson::Fapply() 500 0.004492 0.004492 0.004492 1.32% FabArray::Saxpy() 355 0.003666 0.003666 0.003666 1.08% FabArray::Xpay() 361 0.003511 0.003511 0.003511 1.03% Castro::enforce_min_density() 30 0.003155 0.003155 0.003155 0.93% Gravity::fill_multipole_BCs() 6 0.002947 0.002947 0.002947 0.87% MLMG::addInterpCorrection() 180 0.002871 0.002871 0.002871 0.84% Amr::restart() 1 0.002599 0.002599 0.002599 0.76% amrex::average_down 180 0.002544 0.002544 0.002544 0.75% BndryData::define() 6 0.00201 0.00201 0.00201 0.59% Castro::reset_internal_energy(MultiFab) 30 0.001953 0.001953 0.001953 0.57% Castro::estTimeStep() 10 0.001943 0.001943 0.001943 0.57% FabArray::LinComb() 242 0.00193 0.00193 0.00193 0.57% amrex::Add() 72 0.001846 0.001846 0.001846 0.54% Castro::construct_new_gravity_source() 5 0.001659 0.001659 0.001659 0.49% Amr::writePlotFile() 1 0.001376 0.001376 0.001376 0.40% Castro::construct_old_gravity_source() 5 0.001284 0.001284 0.001284 0.38% Castro::do_advance_ctu() 5 0.001268 0.001268 0.001268 0.37% MLCGSolver::bicgstab 36 0.0009076 0.0009076 0.0009076 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008822 0.0008822 0.0008822 0.26% Castro::subcycle_advance_ctu() 5 0.0008282 0.0008282 0.0008282 0.24% MLCellLinOp::setLevelBC() 6 0.0007496 0.0007496 0.0007496 0.22% Castro::reset_internal_energy(Fab) 240 0.0007133 0.0007133 0.0007133 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007117 0.0007117 0.0007117 0.21% FabArray::mult() 22 0.0006511 0.0006511 0.0006511 0.19% FabArray::setDomainBndry() 20 0.0006504 0.0006504 0.0006504 0.19% MLCellLinOp::prepareForSolve() 6 0.0005917 0.0005917 0.0005917 0.17% MultiFab::contains_nan() 10 0.0005903 0.0005903 0.0005903 0.17% MLCellLinOp::smooth() 720 0.0005656 0.0005656 0.0005656 0.17% Castro::enforce_speed_limit() 30 0.0005324 0.0005324 0.0005324 0.16% MLCellLinOp::compGrad() 6 0.0004902 0.0004902 0.0004902 0.14% MLMG::prepareForSolve() 6 0.0004551 0.0004551 0.0004551 0.13% Amr::InitAmr() 1 0.0004129 0.0004129 0.0004129 0.12% FabArrayBase::CPC::define() 244 0.0003977 0.0003977 0.0003977 0.12% FabArray::FillBoundary() 1766 0.0003466 0.0003466 0.0003466 0.10% FabArrayBase::getCPC() 632 0.0003397 0.0003397 0.0003397 0.10% Gravity::get_old_grav_vector() 5 0.000286 0.000286 0.000286 0.08% main() 1 0.000276 0.000276 0.000276 0.08% Gravity::get_new_grav_vector() 5 0.0002711 0.0002711 0.0002711 0.08% FabArrayBase::getFB() 1766 0.0002529 0.0002529 0.0002529 0.07% MLCellLinOp::apply() 500 0.0002354 0.0002354 0.0002354 0.07% MLMG::mgVcycle() 36 0.0001591 0.0001591 0.0001591 0.05% Amr::coarseTimeStep() 5 0.0001555 0.0001555 0.0001555 0.05% MultiFab::max() 6 0.0001353 0.0001353 0.0001353 0.04% MLCGSolver::ParallelAllReduce 659 0.0001284 0.0001284 0.0001284 0.04% FabArray::ParallelCopy() 380 0.0001104 0.0001104 0.0001104 0.03% MLCellLinOp::correctionResidual() 216 0.0001055 0.0001055 0.0001055 0.03% MLCellLinOp::defineBC() 6 0.0001047 0.0001047 0.0001047 0.03% FillPatchIterator::Initialize 20 9.803e-05 9.803e-05 9.803e-05 0.03% Gravity::solve_for_phi() 5 9.359e-05 9.359e-05 9.359e-05 0.03% MLLinOp::defineGrids() 6 8.278e-05 8.278e-05 8.278e-05 0.02% Amr::timeStep() 5 7.63e-05 7.63e-05 7.63e-05 0.02% AmrLevel::restart() 1 6.934e-05 6.934e-05 6.934e-05 0.02% StateData::restartDoit() 4 6.122e-05 6.122e-05 6.122e-05 0.02% Gravity::update_max_rhs() 6 5.923e-05 5.923e-05 5.923e-05 0.02% FabArrayBase::FB::FB() 26 5.745e-05 5.745e-05 5.745e-05 0.02% MLMG:computeResOfCorrection() 180 5.452e-05 5.452e-05 5.452e-05 0.02% MLMG::actualBottomSolve() 36 4.352e-05 4.352e-05 4.352e-05 0.01% Castro::finalize_advance() 5 4.242e-05 4.242e-05 4.242e-05 0.01% MLMG::mgVcycle_down::0 36 4.228e-05 4.228e-05 4.228e-05 0.01% Castro::clean_state() 30 3.989e-05 3.989e-05 3.989e-05 0.01% MLMG::mgVcycle_down::1 36 3.856e-05 3.856e-05 3.856e-05 0.01% Castro::expand_state() 5 3.75e-05 3.75e-05 3.75e-05 0.01% MLMG::mgVcycle_down::2 36 3.606e-05 3.606e-05 3.606e-05 0.01% MLMG::mgVcycle_down::4 36 3.462e-05 3.462e-05 3.462e-05 0.01% MLMG::mgVcycle_down::3 36 3.41e-05 3.41e-05 3.41e-05 0.01% Castro::advance() 5 3.353e-05 3.353e-05 3.353e-05 0.01% MLMG::solve() 6 3.335e-05 3.335e-05 3.335e-05 0.01% Castro::initialize_advance() 5 3.049e-05 3.049e-05 3.049e-05 0.01% MLMG::mgVcycle_up::4 36 2.952e-05 2.952e-05 2.952e-05 0.01% Amr::writeSmallPlotFile() 1 2.585e-05 2.585e-05 2.585e-05 0.01% MLMG::oneIter() 36 2.489e-05 2.489e-05 2.489e-05 0.01% Castro::buildMetrics() 1 2.454e-05 2.454e-05 2.454e-05 0.01% MLMG::mgVcycle_up::0 36 2.425e-05 2.425e-05 2.425e-05 0.01% Castro::initialize_do_advance() 5 2.291e-05 2.291e-05 2.291e-05 0.01% MLMG::mgVcycle_up::3 36 2.245e-05 2.245e-05 2.245e-05 0.01% MLCellLinOp::solutionResidual() 42 2.243e-05 2.243e-05 2.243e-05 0.01% Castro::post_restart() 1 2.24e-05 2.24e-05 2.24e-05 0.01% MLMG::mgVcycle_up::2 36 2.207e-05 2.207e-05 2.207e-05 0.01% Castro::initMFs() 1 2.135e-05 2.135e-05 2.135e-05 0.01% Castro::swap_state_time_levels() 5 2.118e-05 2.118e-05 2.118e-05 0.01% MLMG::mgVcycle_up::1 36 2.112e-05 2.112e-05 2.112e-05 0.01% MLPoisson::define() 6 1.969e-05 1.969e-05 1.969e-05 0.01% Castro::finalize_do_advance() 5 1.89e-05 1.89e-05 1.89e-05 0.01% MLMG::ResNormInf() 42 1.685e-05 1.685e-05 1.685e-05 0.00% MLMG::computeResidual() 36 1.412e-05 1.412e-05 1.412e-05 0.00% MLMG::mgVcycle_bottom 36 1.388e-05 1.388e-05 1.388e-05 0.00% FillPatchSingleLevel 20 1.35e-05 1.35e-05 1.35e-05 0.00% Castro::construct_new_gravity() 5 1.343e-05 1.343e-05 1.343e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.332e-05 1.332e-05 1.332e-05 0.00% makeSFC 30 1.326e-05 1.326e-05 1.326e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.304e-05 1.304e-05 1.304e-05 0.00% Castro::construct_old_source() 25 1.035e-05 1.035e-05 1.035e-05 0.00% DistributionMapping::Distribute() 31 9.983e-06 9.983e-06 9.983e-06 0.00% Castro::construct_new_source() 25 9.599e-06 9.599e-06 9.599e-06 0.00% Castro::do_new_sources() 5 9.276e-06 9.276e-06 9.276e-06 0.00% Amr::initSubcycle() 1 8.391e-06 8.391e-06 8.391e-06 0.00% Castro::do_old_sources() 5 8.387e-06 8.387e-06 8.387e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.773e-06 6.773e-06 6.773e-06 0.00% Gravity::actual_multilevel_solve() 1 6.471e-06 6.471e-06 6.471e-06 0.00% Castro::check_for_nan() 10 6.177e-06 6.177e-06 6.177e-06 0.00% Castro::construct_old_gravity() 5 5.849e-06 5.849e-06 5.849e-06 0.00% Castro::apply_source_to_state() 10 5.492e-06 5.492e-06 5.492e-06 0.00% MLLinOp::define() 6 5.42e-06 5.42e-06 5.42e-06 0.00% Castro::post_timestep() 5 4.692e-06 4.692e-06 4.692e-06 0.00% Gravity::swapTimeLevels() 5 4.656e-06 4.656e-06 4.656e-06 0.00% MLPoisson::prepareForSolve() 6 4.039e-06 4.039e-06 4.039e-06 0.00% MLMG::computeMLResidual() 6 3.575e-06 3.575e-06 3.575e-06 0.00% Castro::computeNewDt() 5 3.471e-06 3.471e-06 3.471e-06 0.00% MLMG::getGradSolution() 6 2.943e-06 2.943e-06 2.943e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.814e-06 2.814e-06 2.814e-06 0.00% MLMG::MLResNormInf() 6 2.372e-06 2.372e-06 2.372e-06 0.00% Castro::create_source_corrector() 5 2.37e-06 2.37e-06 2.37e-06 0.00% Gravity::set_mass_offset() 6 2.149e-06 2.149e-06 2.149e-06 0.00% MLMG::MLRhsNormInf() 6 2.087e-06 2.087e-06 2.087e-06 0.00% Castro::retry_advance_ctu() 5 1.898e-06 1.898e-06 1.898e-06 0.00% Castro::FluxRegCrseInit 5 1.505e-06 1.505e-06 1.505e-06 0.00% Castro::FluxRegFineAdd() 5 1.206e-06 1.206e-06 1.206e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.106e-06 1.106e-06 1.106e-06 0.00% AmrLevel::AmrLevel() 1 9.88e-07 9.88e-07 9.88e-07 0.00% Amr::init() 1 9.81e-07 9.81e-07 9.81e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.34 0.34 0.34 100.00% Amr::coarseTimeStep() 5 0.259 0.259 0.259 76.17% Amr::timeStep() 5 0.2576 0.2576 0.2576 75.76% Castro::advance() 5 0.2547 0.2547 0.2547 74.90% Castro::subcycle_advance_ctu() 5 0.2476 0.2476 0.2476 72.82% Castro::do_advance_ctu() 5 0.2468 0.2468 0.2468 72.58% Castro::construct_new_gravity() 5 0.1285 0.1285 0.1285 37.78% Gravity::solve_phi_with_mlmg() 6 0.1237 0.1237 0.1237 36.37% Gravity::solve_for_phi() 5 0.1208 0.1208 0.1208 35.51% Gravity::actual_solve_with_mlmg() 6 0.1205 0.1205 0.1205 35.44% MLMG::solve() 6 0.1096 0.1096 0.1096 32.23% MLMG::oneIter() 36 0.1028 0.1028 0.1028 30.24% MLMG::mgVcycle() 36 0.1013 0.1013 0.1013 29.78% Castro::construct_ctu_hydro_source() 5 0.08288 0.08288 0.08288 24.37% MLCellLinOp::smooth() 720 0.04948 0.04948 0.04948 14.55% Amr::init() 1 0.04716 0.04716 0.04716 13.87% Amr::restart() 1 0.04716 0.04716 0.04716 13.87% MLCellLinOp::applyBC() 1946 0.04592 0.04592 0.04592 13.51% AmrLevel::restart() 1 0.04107 0.04107 0.04107 12.08% StateData::restartDoit() 4 0.04099 0.04099 0.04099 12.05% VisMF::Read() 3 0.04087 0.04087 0.04087 12.02% Amr::writePlotFile() 1 0.03312 0.03312 0.03312 9.74% MLMG::mgVcycle_bottom 36 0.03164 0.03164 0.03164 9.30% MLMG::actualBottomSolve() 36 0.03162 0.03162 0.03162 9.30% VisMF::Write(FabArray) 1 0.03149 0.03149 0.03149 9.26% MLCGSolver::bicgstab 36 0.03132 0.03132 0.03132 9.21% FillPatchIterator::Initialize 20 0.01996 0.01996 0.01996 5.87% Castro::clean_state() 30 0.01947 0.01947 0.01947 5.73% FillPatchSingleLevel 20 0.01921 0.01921 0.01921 5.65% StateDataPhysBCFunct::() 20 0.01721 0.01721 0.01721 5.06% MLCellLinOp::apply() 500 0.01551 0.01551 0.01551 4.56% MLMG::mgVcycle_down::0 36 0.01415 0.01415 0.01415 4.16% MLPoisson::Fsmooth() 1440 0.01404 0.01404 0.01404 4.13% FabArray::FillBoundary() 1766 0.01354 0.01354 0.01354 3.98% FillBoundary_nowait() 1766 0.01319 0.01319 0.01319 3.88% StateData::FillBoundary(geom) 160 0.01148 0.01148 0.01148 3.38% MLMG::mgVcycle_up::0 36 0.01079 0.01079 0.01079 3.17% MLCellLinOp::correctionResidual() 216 0.009418 0.009418 0.009418 2.77% Castro::initialize_do_advance() 5 0.009364 0.009364 0.009364 2.75% amrex::Dot() 484 0.008771 0.008771 0.008771 2.58% MLPoisson::define() 6 0.008684 0.008684 0.008684 2.55% Castro::computeTemp() 30 0.00865 0.00865 0.00865 2.54% MLMG:computeResOfCorrection() 180 0.008274 0.008274 0.008274 2.43% Gravity::get_new_grav_vector() 5 0.007601 0.007601 0.007601 2.24% Castro::construct_old_gravity() 5 0.0072 0.0072 0.0072 2.12% Gravity::get_old_grav_vector() 5 0.007194 0.007194 0.007194 2.12% Castro::normalize_species() 30 0.007094 0.007094 0.007094 2.09% amrex::Copy() 463 0.007047 0.007047 0.007047 2.07% MLMG::mgVcycle_down::1 36 0.006743 0.006743 0.006743 1.98% Castro::initialize_advance() 5 0.006732 0.006732 0.006732 1.98% FabArray::ParallelCopy() 380 0.006416 0.006416 0.006416 1.89% MLMG::mgVcycle_down::2 36 0.006327 0.006327 0.006327 1.86% FabArray::ParallelCopy_nowait() 380 0.006306 0.006306 0.006306 1.85% Castro::do_new_sources() 5 0.006279 0.006279 0.006279 1.85% FabArray::setVal() 537 0.006247 0.006247 0.006247 1.84% FabArray::norminf() 326 0.006226 0.006226 0.006226 1.83% MLMG::mgVcycle_down::3 36 0.006116 0.006116 0.006116 1.80% MLMG::mgVcycle_down::4 36 0.006075 0.006075 0.006075 1.79% MLCellLinOp::defineAuxData() 6 0.005895 0.005895 0.005895 1.73% Castro::expand_state() 5 0.005792 0.005792 0.005792 1.70% MLCGSolver::ParallelAllReduce 659 0.00529 0.00529 0.00529 1.56% MLMG::addInterpCorrection() 180 0.005083 0.005083 0.005083 1.49% MLMG::mgVcycle_up::4 36 0.00493 0.00493 0.00493 1.45% MLMG::mgVcycle_up::1 36 0.004882 0.004882 0.004882 1.44% MLMG::mgVcycle_up::2 36 0.004784 0.004784 0.004784 1.41% amrex::average_down 180 0.004765 0.004765 0.004765 1.40% MLMG::mgVcycle_up::3 36 0.004689 0.004689 0.004689 1.38% MLPoisson::Fapply() 500 0.004492 0.004492 0.004492 1.32% Castro::do_old_sources() 5 0.004194 0.004194 0.004194 1.23% FabArray::Saxpy() 355 0.003666 0.003666 0.003666 1.08% FabArray::Xpay() 361 0.003511 0.003511 0.003511 1.03% Castro::post_restart() 1 0.003325 0.003325 0.003325 0.98% MLCellLinOp::solutionResidual() 42 0.003213 0.003213 0.003213 0.94% Gravity::multilevel_solve_for_new_phi() 1 0.003203 0.003203 0.003203 0.94% Gravity::actual_multilevel_solve() 1 0.003189 0.003189 0.003189 0.94% Castro::enforce_min_density() 30 0.003155 0.003155 0.003155 0.93% Gravity::fill_multipole_BCs() 6 0.003076 0.003076 0.003076 0.90% Castro::post_timestep() 5 0.002848 0.002848 0.002848 0.84% MLMG::computeResidual() 36 0.002669 0.002669 0.002669 0.78% Castro::reset_internal_energy(MultiFab) 30 0.002667 0.002667 0.002667 0.78% MLCellLinOp::defineBC() 6 0.002652 0.002652 0.002652 0.78% BndryData::define() 6 0.002547 0.002547 0.002547 0.75% MLMG::prepareForSolve() 6 0.00247 0.00247 0.00247 0.73% Castro::estTimeStep() 10 0.001943 0.001943 0.001943 0.57% FabArray::LinComb() 242 0.00193 0.00193 0.00193 0.57% amrex::Add() 72 0.001846 0.001846 0.001846 0.54% Castro::construct_new_source() 25 0.001669 0.001669 0.001669 0.49% Castro::construct_new_gravity_source() 5 0.001659 0.001659 0.001659 0.49% Castro::construct_old_source() 25 0.001294 0.001294 0.001294 0.38% Castro::construct_old_gravity_source() 5 0.001284 0.001284 0.001284 0.38% Castro::computeNewDt() 5 0.001245 0.001245 0.001245 0.37% MLMG::ResNormInf() 42 0.0009325 0.0009325 0.0009325 0.27% Castro::apply_source_to_state() 10 0.0009165 0.0009165 0.0009165 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008822 0.0008822 0.0008822 0.26% MLMG::getGradSolution() 6 0.000753 0.000753 0.000753 0.22% MLCellLinOp::compGrad() 6 0.0007501 0.0007501 0.0007501 0.22% MLCellLinOp::setLevelBC() 6 0.0007496 0.0007496 0.0007496 0.22% FabArrayBase::getCPC() 632 0.0007374 0.0007374 0.0007374 0.22% Castro::reset_internal_energy(Fab) 240 0.0007133 0.0007133 0.0007133 0.21% FabArray::mult() 22 0.0006511 0.0006511 0.0006511 0.19% FabArray::setDomainBndry() 20 0.0006504 0.0006504 0.0006504 0.19% Castro::check_for_nan() 10 0.0005965 0.0005965 0.0005965 0.18% MLPoisson::prepareForSolve() 6 0.0005957 0.0005957 0.0005957 0.18% MLCellLinOp::prepareForSolve() 6 0.0005917 0.0005917 0.0005917 0.17% MultiFab::contains_nan() 10 0.0005903 0.0005903 0.0005903 0.17% MLMG::computeMLResidual() 6 0.0005612 0.0005612 0.0005612 0.17% Castro::enforce_speed_limit() 30 0.0005324 0.0005324 0.0005324 0.16% Gravity::update_max_rhs() 6 0.0004473 0.0004473 0.0004473 0.13% Amr::InitAmr() 1 0.0004213 0.0004213 0.0004213 0.12% FabArrayBase::CPC::define() 244 0.0003977 0.0003977 0.0003977 0.12% FabArrayBase::getFB() 1766 0.0003104 0.0003104 0.0003104 0.09% Castro::finalize_advance() 5 0.0003087 0.0003087 0.0003087 0.09% Gravity::swapTimeLevels() 5 0.0002239 0.0002239 0.0002239 0.07% Castro::buildMetrics() 1 0.000149 0.000149 0.000149 0.04% MLMG::MLResNormInf() 6 0.0001488 0.0001488 0.0001488 0.04% MultiFab::max() 6 0.0001353 0.0001353 0.0001353 0.04% MLLinOp::define() 6 0.0001177 0.0001177 0.0001177 0.03% MLMG::MLRhsNormInf() 6 0.0001152 0.0001152 0.0001152 0.03% MLLinOp::defineGrids() 6 0.0001123 0.0001123 0.0001123 0.03% FabArrayBase::FB::FB() 26 5.745e-05 5.745e-05 5.745e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.839e-05 2.839e-05 2.839e-05 0.01% Amr::writeSmallPlotFile() 1 2.585e-05 2.585e-05 2.585e-05 0.01% makeSFC 30 2.162e-05 2.162e-05 2.162e-05 0.01% Castro::initMFs() 1 2.135e-05 2.135e-05 2.135e-05 0.01% Castro::swap_state_time_levels() 5 2.118e-05 2.118e-05 2.118e-05 0.01% Castro::finalize_do_advance() 5 1.89e-05 1.89e-05 1.89e-05 0.01% DistributionMapping::Distribute() 31 9.983e-06 9.983e-06 9.983e-06 0.00% Amr::initSubcycle() 1 8.391e-06 8.391e-06 8.391e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.435e-06 4.435e-06 4.435e-06 0.00% Castro::create_source_corrector() 5 2.37e-06 2.37e-06 2.37e-06 0.00% Gravity::set_mass_offset() 6 2.149e-06 2.149e-06 2.149e-06 0.00% Castro::retry_advance_ctu() 5 1.898e-06 1.898e-06 1.898e-06 0.00% Castro::FluxRegCrseInit 5 1.505e-06 1.505e-06 1.505e-06 0.00% Castro::FluxRegFineAdd() 5 1.206e-06 1.206e-06 1.206e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.106e-06 1.106e-06 1.106e-06 0.00% AmrLevel::AmrLevel() 1 9.88e-07 9.88e-07 9.88e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-4-gb3241d7b368a) finalized