Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-8-g3f715d29c94b) initialized Starting run at 09:03:28 UTC on 2022-08-09. Successfully read inputs file ... Castro git describe: 22.08-6-g2aa2c77ae AMReX git describe: 22.08-8-g3f715d29c Microphysics git describe: 22.08-1-gffefdd62 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.053005113 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030733001 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.050613548 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.062584615 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.072431864 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.070912159 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.0605596 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.049675054 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.069092821 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.06938242 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.05503238 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.064455261 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.07370393 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.049474899 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.03059873 seconds Ending run at 09:03:29 UTC on 2022-08-09. Run time = 0.916670473 Run time without initialization = 0.779138278 Average number of zones advanced per microsecond: 3.365 Average number of zones advanced per microsecond per rank: 3.365 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.9167 ... 0.9167 ... 0.9167 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2514 0.2514 0.2514 27.42% VisMF::Write(FabArray) 11 0.2057 0.2057 0.2057 22.44% MLCellLinOp::applyBC() 4433 0.08168 0.08168 0.08168 8.91% MLPoisson::Fsmooth() 3280 0.0639 0.0639 0.0639 6.97% StateData::FillBoundary(geom) 328 0.02489 0.02489 0.02489 2.72% MLCGSolver::bicgstab 82 0.02428 0.02428 0.02428 2.65% MultiFab::Dot() 1114 0.0228 0.0228 0.0228 2.49% Castro::normalize_species() 62 0.01631 0.01631 0.01631 1.78% Castro::computeTemp() 63 0.015 0.015 0.015 1.64% FillBoundary_nowait() 4023 0.0146 0.0146 0.0146 1.59% MultiFab::LinComb() 1586 0.01446 0.01446 0.01446 1.58% FabArray::setVal() 1144 0.0143 0.0143 0.0143 1.56% FabArray::ParallelCopy_nowait() 861 0.01351 0.01351 0.01351 1.47% Castro::enforce_min_density() 62 0.01328 0.01328 0.01328 1.45% StateDataPhysBCFunct::() 41 0.01222 0.01222 0.01222 1.33% MLPoisson::Fapply() 1142 0.0119 0.0119 0.0119 1.30% MLCellLinOp::defineAuxData() 11 0.01151 0.01151 0.01151 1.26% Gravity::fill_multipole_BCs() 11 0.01029 0.01029 0.01029 1.12% MLMG::addInterpCorrection() 410 0.007806 0.007806 0.007806 0.85% amrex::average_down 410 0.006959 0.006959 0.006959 0.76% MultiFab::Xpay() 585 0.006681 0.006681 0.006681 0.73% Castro::estTimeStep() 21 0.005863 0.005863 0.005863 0.64% Castro::do_advance_ctu() 10 0.005601 0.005601 0.005601 0.61% Amr::checkPoint() 3 0.00502 0.00502 0.00502 0.55% Castro::reset_internal_energy(MultiFab) 63 0.004306 0.004306 0.004306 0.47% BndryData::define() 11 0.003897 0.003897 0.003897 0.43% Castro::construct_new_gravity_source() 10 0.003317 0.003317 0.003317 0.36% Castro::construct_old_gravity_source() 10 0.002829 0.002829 0.002829 0.31% Amr::writePlotFile() 2 0.002824 0.002824 0.002824 0.31% MLMG::ResNormInf() 93 0.002088 0.002088 0.002088 0.23% Gravity::get_new_grav_vector() 11 0.001932 0.001932 0.001932 0.21% MultiFab::Saxpy() 20 0.001814 0.001814 0.001814 0.20% Gravity::get_old_grav_vector() 10 0.001754 0.001754 0.001754 0.19% Castro::expand_state() 10 0.001738 0.001738 0.001738 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001728 0.001728 0.001728 0.19% MultiFab::Add() 82 0.001676 0.001676 0.001676 0.18% MLCellLinOp::setLevelBC() 11 0.00159 0.00159 0.00159 0.17% Castro::reset_internal_energy(Fab) 504 0.001553 0.001553 0.001553 0.17% Gravity::actual_solve_with_mlmg() 11 0.001451 0.001451 0.001451 0.16% FabArray::mult() 43 0.001315 0.001315 0.001315 0.14% FabArray::setDomainBndry() 41 0.001303 0.001303 0.001303 0.14% Castro::initData() 1 0.001227 0.001227 0.001227 0.13% MLMG::prepareForSolve() 11 0.001216 0.001216 0.001216 0.13% MLCellLinOp::prepareForSolve() 11 0.001194 0.001194 0.001194 0.13% MultiFab::contains_nan() 20 0.001185 0.001185 0.001185 0.13% MLCellLinOp::smooth() 1640 0.001151 0.001151 0.001151 0.13% Castro::enforce_speed_limit() 62 0.001051 0.001051 0.001051 0.11% MLCellLinOp::compGrad() 11 0.0009303 0.0009303 0.0009303 0.10% FabArray::FillBoundary() 4023 0.0008319 0.0008319 0.0008319 0.09% FabArrayBase::getCPC() 1323 0.0007787 0.0007787 0.0007787 0.08% FabArrayBase::CPC::define() 454 0.0006597 0.0006597 0.0006597 0.07% FabArrayBase::getFB() 4023 0.0006243 0.0006243 0.0006243 0.07% Amr::InitAmr() 1 0.0004945 0.0004945 0.0004945 0.05% MLCellLinOp::apply() 1142 0.0004567 0.0004567 0.0004567 0.05% Gravity::solve_for_phi() 10 0.0004184 0.0004184 0.0004184 0.05% Gravity::update_max_rhs() 11 0.0004112 0.0004112 0.0004112 0.04% CGSolver::sxay() 1586 0.0003855 0.0003855 0.0003855 0.04% Amr::coarseTimeStep() 10 0.0003354 0.0003354 0.0003354 0.04% MultiFab::Copy() 11 0.0003328 0.0003328 0.0003328 0.04% FillPatchIterator::Initialize 41 0.0002972 0.0002972 0.0002972 0.03% MLCGSolver::ParallelAllReduce 1514 0.0002951 0.0002951 0.0002951 0.03% MLCellLinOp::defineBC() 11 0.0002811 0.0002811 0.0002811 0.03% main() 1 0.0002658 0.0002658 0.0002658 0.03% FabArray::ParallelCopy() 861 0.0002631 0.0002631 0.0002631 0.03% MultiFab::max() 11 0.0002601 0.0002601 0.0002601 0.03% Castro::create_source_corrector() 10 0.0002369 0.0002369 0.0002369 0.03% MLCellLinOp::correctionResidual() 492 0.0002365 0.0002365 0.0002365 0.03% MLMG::MLRhsNormInf() 11 0.0002193 0.0002193 0.0002193 0.02% Castro::construct_new_gravity() 10 0.0002073 0.0002073 0.0002073 0.02% Amr::timeStep() 10 0.0002017 0.0002017 0.0002017 0.02% MLMG::mgVcycle() 82 0.0001933 0.0001933 0.0001933 0.02% MLMG:computeResOfCorrection() 410 0.0001583 0.0001583 0.0001583 0.02% Castro::subcycle_advance_ctu() 10 0.0001452 0.0001452 0.0001452 0.02% MLLinOp::defineGrids() 11 0.000129 0.000129 0.000129 0.01% StateData::checkPoint() 12 0.0001259 0.0001259 0.0001259 0.01% MLMG::mgVcycle_down::0 82 0.0001167 0.0001167 0.0001167 0.01% MLMG::mgVcycle_down::1 82 9.738e-05 9.738e-05 9.738e-05 0.01% MLMG::mgVcycle_down::2 82 9.178e-05 9.178e-05 9.178e-05 0.01% Castro::Castro() 1 9.065e-05 9.065e-05 9.065e-05 0.01% MLMG::actualBottomSolve() 82 8.737e-05 8.737e-05 8.737e-05 0.01% MLMG::mgVcycle_down::3 82 8.722e-05 8.722e-05 8.722e-05 0.01% MLMG::mgVcycle_down::4 82 8.574e-05 8.574e-05 8.574e-05 0.01% FabArrayBase::FB::FB() 56 8.514e-05 8.514e-05 8.514e-05 0.01% Castro::initialize_advance() 10 7.976e-05 7.976e-05 7.976e-05 0.01% Castro::clean_state() 62 7.697e-05 7.697e-05 7.697e-05 0.01% MLMG::mgVcycle_up::4 82 7.159e-05 7.159e-05 7.159e-05 0.01% Castro::construct_new_source() 50 7.114e-05 7.114e-05 7.114e-05 0.01% AmrLevel::checkPoint() 3 7.009e-05 7.009e-05 7.009e-05 0.01% MLMG::solve() 11 6.819e-05 6.819e-05 6.819e-05 0.01% MLMG::mgVcycle_up::1 82 6.706e-05 6.706e-05 6.706e-05 0.01% Castro::initialize_do_advance() 10 6.232e-05 6.232e-05 6.232e-05 0.01% MLMG::oneIter() 82 6.167e-05 6.167e-05 6.167e-05 0.01% MLMG::mgVcycle_up::0 82 6.162e-05 6.162e-05 6.162e-05 0.01% MLMG::mgVcycle_up::3 82 5.926e-05 5.926e-05 5.926e-05 0.01% MLMG::mgVcycle_up::2 82 5.926e-05 5.926e-05 5.926e-05 0.01% Castro::advance() 10 5.413e-05 5.413e-05 5.413e-05 0.01% MLCellLinOp::solutionResidual() 93 5.271e-05 5.271e-05 5.271e-05 0.01% Castro::finalize_advance() 10 4.954e-05 4.954e-05 4.954e-05 0.01% MLMG::computeResidual() 82 4.157e-05 4.157e-05 4.157e-05 0.00% StateData::define() 4 3.886e-05 3.886e-05 3.886e-05 0.00% Castro::swap_state_time_levels() 10 3.841e-05 3.841e-05 3.841e-05 0.00% Castro::enforce_consistent_e() 1 3.483e-05 3.483e-05 3.483e-05 0.00% MLMG::mgVcycle_bottom 82 3.429e-05 3.429e-05 3.429e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.388e-05 3.388e-05 3.388e-05 0.00% Castro::finalize_do_advance() 10 3.271e-05 3.271e-05 3.271e-05 0.00% Gravity::actual_multilevel_solve() 1 3.211e-05 3.211e-05 3.211e-05 0.00% FillPatchSingleLevel 41 2.943e-05 2.943e-05 2.943e-05 0.00% Castro::initMFs() 1 2.699e-05 2.699e-05 2.699e-05 0.00% Amr::defBaseLevel() 1 2.596e-05 2.596e-05 2.596e-05 0.00% Amr::writeSmallPlotFile() 1 2.568e-05 2.568e-05 2.568e-05 0.00% makeSFC 55 2.492e-05 2.492e-05 2.492e-05 0.00% MLLinOp::define() 11 2.446e-05 2.446e-05 2.446e-05 0.00% MLPoisson::define() 11 2.415e-05 2.415e-05 2.415e-05 0.00% Amr::FinalizeInit() 1 2.224e-05 2.224e-05 2.224e-05 0.00% Castro::buildMetrics() 1 2.179e-05 2.179e-05 2.179e-05 0.00% Castro::construct_old_source() 50 1.817e-05 1.817e-05 1.817e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.748e-05 1.748e-05 1.748e-05 0.00% Castro::do_old_sources() 10 1.635e-05 1.635e-05 1.635e-05 0.00% Castro::do_new_sources() 10 1.575e-05 1.575e-05 1.575e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.477e-05 1.477e-05 1.477e-05 0.00% DistributionMapping::Distribute() 56 1.417e-05 1.417e-05 1.417e-05 0.00% MLPoisson::prepareForSolve() 11 1.286e-05 1.286e-05 1.286e-05 0.00% Castro::check_for_nan() 20 1.235e-05 1.235e-05 1.235e-05 0.00% Castro::apply_source_to_state() 20 1.117e-05 1.117e-05 1.117e-05 0.00% MLMG::computeMLResidual() 11 1.028e-05 1.028e-05 1.028e-05 0.00% Castro::construct_old_gravity() 10 9.929e-06 9.929e-06 9.929e-06 0.00% Amr::initSubcycle() 1 8.619e-06 8.619e-06 8.619e-06 0.00% Gravity::swapTimeLevels() 10 8.255e-06 8.255e-06 8.255e-06 0.00% Castro::post_timestep() 10 7.937e-06 7.937e-06 7.937e-06 0.00% Amr::InitializeInit() 1 6.976e-06 6.976e-06 6.976e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.601e-06 6.601e-06 6.601e-06 0.00% MLMG::getGradSolution() 11 6.417e-06 6.417e-06 6.417e-06 0.00% Castro::computeNewDt() 9 6.216e-06 6.216e-06 6.216e-06 0.00% AmrLevel::checkPointPost() 3 4.556e-06 4.556e-06 4.556e-06 0.00% Gravity::set_mass_offset() 11 4.12e-06 4.12e-06 4.12e-06 0.00% Castro::post_init() 1 3.546e-06 3.546e-06 3.546e-06 0.00% MLMG::MLResNormInf() 11 3.449e-06 3.449e-06 3.449e-06 0.00% Castro::retry_advance_ctu() 10 3.299e-06 3.299e-06 3.299e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.834e-06 2.834e-06 2.834e-06 0.00% Castro::FluxRegCrseInit 10 2.749e-06 2.749e-06 2.749e-06 0.00% Castro::computeInitialDt() 2 2.617e-06 2.617e-06 2.617e-06 0.00% Amr::init() 1 2.285e-06 2.285e-06 2.285e-06 0.00% Castro::FluxRegFineAdd() 10 2.119e-06 2.119e-06 2.119e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.788e-06 1.788e-06 1.788e-06 0.00% AmrLevel::checkPointPre() 3 1.713e-06 1.713e-06 1.713e-06 0.00% Castro::post_regrid() 1 1.224e-06 1.224e-06 1.224e-06 0.00% Amr::initialInit() 1 1.149e-06 1.149e-06 1.149e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9167 0.9167 0.9167 100.00% Amr::coarseTimeStep() 10 0.7483 0.7483 0.7483 81.63% Amr::timeStep() 10 0.6454 0.6454 0.6454 70.40% Castro::advance() 10 0.6378 0.6378 0.6378 69.58% Castro::subcycle_advance_ctu() 10 0.6257 0.6257 0.6257 68.25% Castro::do_advance_ctu() 10 0.6255 0.6255 0.6255 68.24% Gravity::solve_phi_with_mlmg() 11 0.3203 0.3203 0.3203 34.94% Gravity::actual_solve_with_mlmg() 11 0.3098 0.3098 0.3098 33.79% Castro::construct_new_gravity() 10 0.2907 0.2907 0.2907 31.72% MLMG::solve() 11 0.287 0.287 0.287 31.31% Gravity::solve_for_phi() 10 0.2754 0.2754 0.2754 30.04% MLMG::oneIter() 82 0.2719 0.2719 0.2719 29.66% MLMG::mgVcycle() 82 0.2702 0.2702 0.2702 29.47% Castro::construct_ctu_hydro_source() 10 0.2514 0.2514 0.2514 27.42% VisMF::Write(FabArray) 11 0.2057 0.2057 0.2057 22.44% Amr::checkPoint() 3 0.1523 0.1523 0.1523 16.61% AmrLevel::checkPoint() 3 0.1473 0.1473 0.1473 16.06% StateData::checkPoint() 12 0.1472 0.1472 0.1472 16.06% MLCellLinOp::smooth() 1640 0.1378 0.1378 0.1378 15.03% Amr::init() 1 0.1369 0.1369 0.1369 14.94% MLCellLinOp::applyBC() 4433 0.09782 0.09782 0.09782 10.67% MLMG::mgVcycle_bottom 82 0.08316 0.08316 0.08316 9.07% MLMG::actualBottomSolve() 82 0.08312 0.08312 0.08312 9.07% MLCGSolver::bicgstab 82 0.0823 0.0823 0.0823 8.98% MLPoisson::Fsmooth() 3280 0.0639 0.0639 0.0639 6.97% Amr::writePlotFile() 2 0.06146 0.06146 0.06146 6.70% Amr::initialInit() 1 0.05305 0.05305 0.05305 5.79% Castro::clean_state() 62 0.05064 0.05064 0.05064 5.52% Amr::FinalizeInit() 1 0.04884 0.04884 0.04884 5.33% Castro::post_init() 1 0.04748 0.04748 0.04748 5.18% Gravity::multilevel_solve_for_new_phi() 1 0.04539 0.04539 0.04539 4.95% Gravity::actual_multilevel_solve() 1 0.04537 0.04537 0.04537 4.95% FillPatchIterator::Initialize 41 0.04273 0.04273 0.04273 4.66% FillPatchSingleLevel 41 0.04113 0.04113 0.04113 4.49% StateDataPhysBCFunct::() 41 0.03711 0.03711 0.03711 4.05% MLCellLinOp::apply() 1142 0.03689 0.03689 0.03689 4.02% MLMG::mgVcycle_down::0 82 0.0358 0.0358 0.0358 3.91% MLMG::mgVcycle_up::0 82 0.03059 0.03059 0.03059 3.34% StateData::FillBoundary(geom) 328 0.02489 0.02489 0.02489 2.72% MultiFab::Dot() 1114 0.0228 0.0228 0.0228 2.49% MLCellLinOp::correctionResidual() 492 0.02164 0.02164 0.02164 2.36% Castro::computeTemp() 63 0.02086 0.02086 0.02086 2.28% Castro::initialize_do_advance() 10 0.02037 0.02037 0.02037 2.22% MLMG:computeResOfCorrection() 410 0.01871 0.01871 0.01871 2.04% MLPoisson::define() 11 0.01827 0.01827 0.01827 1.99% MLMG::mgVcycle_down::1 82 0.01797 0.01797 0.01797 1.96% MLMG::mgVcycle_down::2 82 0.01754 0.01754 0.01754 1.91% Gravity::get_new_grav_vector() 11 0.01714 0.01714 0.01714 1.87% MLMG::mgVcycle_down::3 82 0.01668 0.01668 0.01668 1.82% Castro::normalize_species() 62 0.01631 0.01631 0.01631 1.78% FabArray::FillBoundary() 4023 0.01615 0.01615 0.01615 1.76% MLMG::mgVcycle_down::4 82 0.01592 0.01592 0.01592 1.74% Castro::construct_old_gravity() 10 0.01535 0.01535 0.01535 1.67% Gravity::get_old_grav_vector() 10 0.01534 0.01534 0.01534 1.67% FillBoundary_nowait() 4023 0.01531 0.01531 0.01531 1.67% CGSolver::sxay() 1586 0.01485 0.01485 0.01485 1.62% FabArray::ParallelCopy() 861 0.0146 0.0146 0.0146 1.59% MultiFab::LinComb() 1586 0.01446 0.01446 0.01446 1.58% FabArray::ParallelCopy_nowait() 861 0.01433 0.01433 0.01433 1.56% FabArray::setVal() 1144 0.0143 0.0143 0.0143 1.56% MLCGSolver::ParallelAllReduce 1514 0.01361 0.01361 0.01361 1.48% MLMG::mgVcycle_up::2 82 0.01344 0.01344 0.01344 1.47% MLMG::mgVcycle_up::1 82 0.0133 0.0133 0.0133 1.45% Castro::enforce_min_density() 62 0.01328 0.01328 0.01328 1.45% MLMG::addInterpCorrection() 410 0.01306 0.01306 0.01306 1.43% MLCellLinOp::defineAuxData() 11 0.01289 0.01289 0.01289 1.41% MLMG::mgVcycle_up::3 82 0.0128 0.0128 0.0128 1.40% MLMG::mgVcycle_up::4 82 0.01277 0.01277 0.01277 1.39% Castro::do_old_sources() 10 0.01245 0.01245 0.01245 1.36% Castro::do_new_sources() 10 0.01244 0.01244 0.01244 1.36% amrex::average_down 410 0.01231 0.01231 0.01231 1.34% Castro::initialize_advance() 10 0.01204 0.01204 0.01204 1.31% MLPoisson::Fapply() 1142 0.0119 0.0119 0.0119 1.30% Castro::expand_state() 10 0.01156 0.01156 0.01156 1.26% Gravity::fill_multipole_BCs() 11 0.01029 0.01029 0.01029 1.12% Castro::post_timestep() 10 0.007334 0.007334 0.007334 0.80% MLCellLinOp::solutionResidual() 93 0.007264 0.007264 0.007264 0.79% MultiFab::Xpay() 585 0.006681 0.006681 0.006681 0.73% MLMG::computeResidual() 82 0.006284 0.006284 0.006284 0.69% Castro::estTimeStep() 21 0.005863 0.005863 0.005863 0.64% Castro::reset_internal_energy(MultiFab) 63 0.005858 0.005858 0.005858 0.64% MLMG::prepareForSolve() 11 0.005427 0.005427 0.005427 0.59% MLCellLinOp::defineBC() 11 0.005151 0.005151 0.005151 0.56% BndryData::define() 11 0.00487 0.00487 0.00487 0.53% Amr::InitializeInit() 1 0.004209 0.004209 0.004209 0.46% Amr::defBaseLevel() 1 0.004202 0.004202 0.004202 0.46% Castro::initData() 1 0.003688 0.003688 0.003688 0.40% Castro::construct_new_source() 50 0.003388 0.003388 0.003388 0.37% Castro::construct_new_gravity_source() 10 0.003317 0.003317 0.003317 0.36% Castro::computeNewDt() 9 0.002914 0.002914 0.002914 0.32% Castro::construct_old_source() 50 0.002847 0.002847 0.002847 0.31% Castro::construct_old_gravity_source() 10 0.002829 0.002829 0.002829 0.31% MLMG::ResNormInf() 93 0.002088 0.002088 0.002088 0.23% Castro::apply_source_to_state() 20 0.001825 0.001825 0.001825 0.20% MultiFab::Saxpy() 20 0.001814 0.001814 0.001814 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001728 0.001728 0.001728 0.19% MultiFab::Add() 82 0.001676 0.001676 0.001676 0.18% MLCellLinOp::setLevelBC() 11 0.00159 0.00159 0.00159 0.17% Castro::reset_internal_energy(Fab) 504 0.001553 0.001553 0.001553 0.17% MLMG::getGradSolution() 11 0.001459 0.001459 0.001459 0.16% MLCellLinOp::compGrad() 11 0.001453 0.001453 0.001453 0.16% FabArrayBase::getCPC() 1323 0.001438 0.001438 0.001438 0.16% FabArray::mult() 43 0.001315 0.001315 0.001315 0.14% FabArray::setDomainBndry() 41 0.001303 0.001303 0.001303 0.14% MLPoisson::prepareForSolve() 11 0.001207 0.001207 0.001207 0.13% Castro::check_for_nan() 20 0.001197 0.001197 0.001197 0.13% MLCellLinOp::prepareForSolve() 11 0.001194 0.001194 0.001194 0.13% MultiFab::contains_nan() 20 0.001185 0.001185 0.001185 0.13% Castro::post_regrid() 1 0.001159 0.001159 0.001159 0.13% Castro::enforce_speed_limit() 62 0.001051 0.001051 0.001051 0.11% MLMG::computeMLResidual() 11 0.001032 0.001032 0.001032 0.11% Gravity::update_max_rhs() 11 0.0008176 0.0008176 0.0008176 0.09% FabArrayBase::getFB() 4023 0.0007094 0.0007094 0.0007094 0.08% FabArrayBase::CPC::define() 454 0.0006597 0.0006597 0.0006597 0.07% Castro::computeInitialDt() 2 0.0006579 0.0006579 0.0006579 0.07% Amr::InitAmr() 1 0.0005031 0.0005031 0.0005031 0.05% Castro::Castro() 1 0.0004389 0.0004389 0.0004389 0.05% Gravity::swapTimeLevels() 10 0.0004348 0.0004348 0.0004348 0.05% MultiFab::Copy() 11 0.0003328 0.0003328 0.0003328 0.04% MLMG::MLResNormInf() 11 0.00028 0.00028 0.00028 0.03% MultiFab::max() 11 0.0002601 0.0002601 0.0002601 0.03% Castro::create_source_corrector() 10 0.0002369 0.0002369 0.0002369 0.03% MLMG::MLRhsNormInf() 11 0.0002193 0.0002193 0.0002193 0.02% MLLinOp::define() 11 0.000208 0.000208 0.000208 0.02% MLLinOp::defineGrids() 11 0.0001836 0.0001836 0.0001836 0.02% Castro::buildMetrics() 1 0.0001557 0.0001557 0.0001557 0.02% FabArrayBase::FB::FB() 56 8.514e-05 8.514e-05 8.514e-05 0.01% Castro::finalize_advance() 10 5.441e-05 5.441e-05 5.441e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.279e-05 5.279e-05 5.279e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.546e-05 4.546e-05 4.546e-05 0.00% StateData::define() 4 3.886e-05 3.886e-05 3.886e-05 0.00% Castro::swap_state_time_levels() 10 3.841e-05 3.841e-05 3.841e-05 0.00% makeSFC 55 3.802e-05 3.802e-05 3.802e-05 0.00% Castro::enforce_consistent_e() 1 3.483e-05 3.483e-05 3.483e-05 0.00% Castro::finalize_do_advance() 10 3.271e-05 3.271e-05 3.271e-05 0.00% Castro::initMFs() 1 2.699e-05 2.699e-05 2.699e-05 0.00% Amr::writeSmallPlotFile() 1 2.568e-05 2.568e-05 2.568e-05 0.00% DistributionMapping::Distribute() 56 1.417e-05 1.417e-05 1.417e-05 0.00% Amr::initSubcycle() 1 8.619e-06 8.619e-06 8.619e-06 0.00% AmrLevel::checkPointPost() 3 4.556e-06 4.556e-06 4.556e-06 0.00% Gravity::set_mass_offset() 11 4.12e-06 4.12e-06 4.12e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.898e-06 3.898e-06 3.898e-06 0.00% Castro::retry_advance_ctu() 10 3.299e-06 3.299e-06 3.299e-06 0.00% Castro::FluxRegCrseInit 10 2.749e-06 2.749e-06 2.749e-06 0.00% Castro::FluxRegFineAdd() 10 2.119e-06 2.119e-06 2.119e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.788e-06 1.788e-06 1.788e-06 0.00% AmrLevel::checkPointPre() 3 1.713e-06 1.713e-06 1.713e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-8-g3f715d29c94b) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-8-g3f715d29c94b) initialized Starting run at 09:03:30 UTC on 2022-08-09. Successfully read inputs file ... Castro git describe: 22.08-6-g2aa2c77ae AMReX git describe: 22.08-8-g3f715d29c Microphysics git describe: 22.08-1-gffefdd62 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.504333111 Restart time = 0.048173868 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052133586 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050102254 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.051099666 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.05926604 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065060745 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032585947 seconds Ending run at 09:03:30 UTC on 2022-08-09. Run time = 0.359397846 Run time without initialization = 0.310651802 Average number of zones advanced per microsecond: 4.219 Average number of zones advanced per microsecond per rank: 4.219 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3594 ... 0.3594 ... 0.3594 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0813 0.0813 0.0813 22.63% VisMF::Read() 3 0.04029 0.04029 0.04029 11.21% MLCellLinOp::applyBC() 1946 0.03466 0.03466 0.03466 9.64% VisMF::Write(FabArray) 1 0.03106 0.03106 0.03106 8.64% MLPoisson::Fsmooth() 1440 0.02706 0.02706 0.02706 7.53% StateData::FillBoundary(geom) 160 0.01197 0.01197 0.01197 3.33% MLCGSolver::bicgstab 36 0.01022 0.01022 0.01022 2.84% MultiFab::Dot() 484 0.009557 0.009557 0.009557 2.66% Castro::normalize_species() 30 0.006884 0.006884 0.006884 1.92% FabArray::setVal() 537 0.006744 0.006744 0.006744 1.88% FillBoundary_nowait() 1766 0.006255 0.006255 0.006255 1.74% MLCellLinOp::defineAuxData() 6 0.006189 0.006189 0.006189 1.72% MultiFab::LinComb() 690 0.006127 0.006127 0.006127 1.70% Gravity::fill_multipole_BCs() 6 0.005993 0.005993 0.005993 1.67% FabArray::ParallelCopy_nowait() 380 0.005932 0.005932 0.005932 1.65% Castro::enforce_min_density() 30 0.005801 0.005801 0.005801 1.61% StateDataPhysBCFunct::() 20 0.0054 0.0054 0.0054 1.50% Castro::computeTemp() 30 0.005317 0.005317 0.005317 1.48% MLPoisson::Fapply() 500 0.005071 0.005071 0.005071 1.41% Amr::restart() 1 0.003625 0.003625 0.003625 1.01% MLMG::addInterpCorrection() 180 0.003285 0.003285 0.003285 0.91% Castro::estTimeStep() 10 0.002945 0.002945 0.002945 0.82% amrex::average_down 180 0.002937 0.002937 0.002937 0.82% MultiFab::Xpay() 258 0.002884 0.002884 0.002884 0.80% Castro::do_advance_ctu() 5 0.002247 0.002247 0.002247 0.63% Castro::reset_internal_energy(MultiFab) 30 0.002141 0.002141 0.002141 0.60% BndryData::define() 6 0.002141 0.002141 0.002141 0.60% Castro::construct_new_gravity_source() 5 0.001639 0.001639 0.001639 0.46% Amr::writePlotFile() 1 0.001613 0.001613 0.001613 0.45% Castro::construct_old_gravity_source() 5 0.001425 0.001425 0.001425 0.40% Castro::subcycle_advance_ctu() 5 0.001409 0.001409 0.001409 0.39% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009418 0.0009418 0.0009418 0.26% Gravity::get_old_grav_vector() 5 0.0009151 0.0009151 0.0009151 0.25% MultiFab::Saxpy() 10 0.0009148 0.0009148 0.0009148 0.25% MLMG::ResNormInf() 42 0.0009088 0.0009088 0.0009088 0.25% Gravity::get_new_grav_vector() 5 0.0008718 0.0008718 0.0008718 0.24% Castro::expand_state() 5 0.0008707 0.0008707 0.0008707 0.24% MLCellLinOp::setLevelBC() 6 0.0008278 0.0008278 0.0008278 0.23% Gravity::actual_solve_with_mlmg() 6 0.0007659 0.0007659 0.0007659 0.21% Castro::reset_internal_energy(Fab) 240 0.000758 0.000758 0.000758 0.21% MultiFab::Add() 36 0.0007099 0.0007099 0.0007099 0.20% FabArray::mult() 22 0.0006574 0.0006574 0.0006574 0.18% MLMG::prepareForSolve() 6 0.0006525 0.0006525 0.0006525 0.18% FabArray::setDomainBndry() 20 0.0006336 0.0006336 0.0006336 0.18% MLCellLinOp::prepareForSolve() 6 0.0006295 0.0006295 0.0006295 0.18% MultiFab::contains_nan() 10 0.0005819 0.0005819 0.0005819 0.16% Castro::enforce_speed_limit() 30 0.0005465 0.0005465 0.0005465 0.15% MLCellLinOp::compGrad() 6 0.0004962 0.0004962 0.0004962 0.14% MLCellLinOp::smooth() 720 0.0004882 0.0004882 0.0004882 0.14% Amr::InitAmr() 1 0.000409 0.000409 0.000409 0.11% FabArrayBase::CPC::define() 244 0.0003914 0.0003914 0.0003914 0.11% FabArrayBase::getCPC() 632 0.0003707 0.0003707 0.0003707 0.10% FabArray::FillBoundary() 1766 0.0003685 0.0003685 0.0003685 0.10% FabArrayBase::getFB() 1766 0.0002633 0.0002633 0.0002633 0.07% main() 1 0.000251 0.000251 0.000251 0.07% Gravity::update_max_rhs() 6 0.0002261 0.0002261 0.0002261 0.06% Gravity::solve_for_phi() 5 0.0002148 0.0002148 0.0002148 0.06% MLCellLinOp::apply() 500 0.0001922 0.0001922 0.0001922 0.05% MultiFab::Copy() 6 0.0001786 0.0001786 0.0001786 0.05% CGSolver::sxay() 690 0.0001731 0.0001731 0.0001731 0.05% Amr::coarseTimeStep() 5 0.0001652 0.0001652 0.0001652 0.05% MLCellLinOp::defineBC() 6 0.0001494 0.0001494 0.0001494 0.04% FillPatchIterator::Initialize 20 0.0001367 0.0001367 0.0001367 0.04% MultiFab::max() 6 0.0001347 0.0001347 0.0001347 0.04% MLCGSolver::ParallelAllReduce 659 0.0001231 0.0001231 0.0001231 0.03% FabArray::ParallelCopy() 380 0.0001191 0.0001191 0.0001191 0.03% Amr::timeStep() 5 0.0001155 0.0001155 0.0001155 0.03% Castro::construct_new_gravity() 5 0.0001147 0.0001147 0.0001147 0.03% MLMG::MLRhsNormInf() 6 0.0001136 0.0001136 0.0001136 0.03% Castro::create_source_corrector() 5 9.701e-05 9.701e-05 9.701e-05 0.03% MLCellLinOp::correctionResidual() 216 9.33e-05 9.33e-05 9.33e-05 0.03% MLLinOp::defineGrids() 6 9.301e-05 9.301e-05 9.301e-05 0.03% MLMG::mgVcycle() 36 8.345e-05 8.345e-05 8.345e-05 0.02% AmrLevel::restart() 1 7.519e-05 7.519e-05 7.519e-05 0.02% MLMG:computeResOfCorrection() 180 7.033e-05 7.033e-05 7.033e-05 0.02% StateData::restartDoit() 4 6.944e-05 6.944e-05 6.944e-05 0.02% Castro::finalize_advance() 5 6.377e-05 6.377e-05 6.377e-05 0.02% FabArrayBase::FB::FB() 26 5.719e-05 5.719e-05 5.719e-05 0.02% Castro::construct_new_source() 25 5.323e-05 5.323e-05 5.323e-05 0.01% Castro::advance() 5 5.054e-05 5.054e-05 5.054e-05 0.01% Castro::construct_old_source() 25 4.752e-05 4.752e-05 4.752e-05 0.01% MLMG::mgVcycle_down::0 36 4.578e-05 4.578e-05 4.578e-05 0.01% MLMG::mgVcycle_down::1 36 4.177e-05 4.177e-05 4.177e-05 0.01% Castro::clean_state() 30 4.071e-05 4.071e-05 4.071e-05 0.01% MLMG::mgVcycle_down::2 36 3.987e-05 3.987e-05 3.987e-05 0.01% MLMG::mgVcycle_down::4 36 3.843e-05 3.843e-05 3.843e-05 0.01% MLMG::mgVcycle_down::3 36 3.737e-05 3.737e-05 3.737e-05 0.01% Castro::initialize_advance() 5 3.708e-05 3.708e-05 3.708e-05 0.01% MLMG::actualBottomSolve() 36 3.63e-05 3.63e-05 3.63e-05 0.01% Castro::buildMetrics() 1 3.33e-05 3.33e-05 3.33e-05 0.01% MLMG::mgVcycle_up::4 36 3.244e-05 3.244e-05 3.244e-05 0.01% MLMG::solve() 6 3.181e-05 3.181e-05 3.181e-05 0.01% Gravity::actual_multilevel_solve() 1 3.01e-05 3.01e-05 3.01e-05 0.01% Castro::initialize_do_advance() 5 3.005e-05 3.005e-05 3.005e-05 0.01% Castro::post_restart() 1 2.993e-05 2.993e-05 2.993e-05 0.01% MLMG::oneIter() 36 2.79e-05 2.79e-05 2.79e-05 0.01% MLMG::mgVcycle_up::0 36 2.753e-05 2.753e-05 2.753e-05 0.01% MLMG::mgVcycle_up::3 36 2.716e-05 2.716e-05 2.716e-05 0.01% Castro::initMFs() 1 2.685e-05 2.685e-05 2.685e-05 0.01% MLMG::mgVcycle_up::2 36 2.623e-05 2.623e-05 2.623e-05 0.01% Amr::writeSmallPlotFile() 1 2.579e-05 2.579e-05 2.579e-05 0.01% Castro::swap_state_time_levels() 5 2.527e-05 2.527e-05 2.527e-05 0.01% MLMG::mgVcycle_up::1 36 2.47e-05 2.47e-05 2.47e-05 0.01% Castro::computeNewDt() 5 2.32e-05 2.32e-05 2.32e-05 0.01% MLCellLinOp::solutionResidual() 42 2.199e-05 2.199e-05 2.199e-05 0.01% MLLinOp::define() 6 2.134e-05 2.134e-05 2.134e-05 0.01% Castro::finalize_do_advance() 5 1.907e-05 1.907e-05 1.907e-05 0.01% MLMG::computeResidual() 36 1.83e-05 1.83e-05 1.83e-05 0.01% Gravity::solve_phi_with_mlmg() 6 1.691e-05 1.691e-05 1.691e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.685e-05 1.685e-05 1.685e-05 0.00% MLMG::mgVcycle_bottom 36 1.63e-05 1.63e-05 1.63e-05 0.00% MLPoisson::define() 6 1.613e-05 1.613e-05 1.613e-05 0.00% makeSFC 30 1.357e-05 1.357e-05 1.357e-05 0.00% FillPatchSingleLevel 20 1.332e-05 1.332e-05 1.332e-05 0.00% Amr::initSubcycle() 1 8.865e-06 8.865e-06 8.865e-06 0.00% Castro::do_new_sources() 5 8.605e-06 8.605e-06 8.605e-06 0.00% DistributionMapping::Distribute() 31 8.029e-06 8.029e-06 8.029e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.894e-06 7.894e-06 7.894e-06 0.00% Castro::do_old_sources() 5 7.821e-06 7.821e-06 7.821e-06 0.00% Castro::construct_old_gravity() 5 6.196e-06 6.196e-06 6.196e-06 0.00% Castro::post_timestep() 5 6.088e-06 6.088e-06 6.088e-06 0.00% Castro::check_for_nan() 10 5.676e-06 5.676e-06 5.676e-06 0.00% Castro::apply_source_to_state() 10 5.44e-06 5.44e-06 5.44e-06 0.00% MLMG::computeMLResidual() 6 5.038e-06 5.038e-06 5.038e-06 0.00% MLPoisson::prepareForSolve() 6 4.667e-06 4.667e-06 4.667e-06 0.00% Gravity::swapTimeLevels() 5 4.013e-06 4.013e-06 4.013e-06 0.00% Castro::retry_advance_ctu() 5 3.397e-06 3.397e-06 3.397e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.164e-06 3.164e-06 3.164e-06 0.00% MLMG::getGradSolution() 6 3.093e-06 3.093e-06 3.093e-06 0.00% Castro::FluxRegCrseInit 5 2.679e-06 2.679e-06 2.679e-06 0.00% MLMG::MLResNormInf() 6 2.097e-06 2.097e-06 2.097e-06 0.00% Gravity::set_mass_offset() 6 2.075e-06 2.075e-06 2.075e-06 0.00% Castro::FluxRegFineAdd() 5 1.679e-06 1.679e-06 1.679e-06 0.00% Amr::init() 1 1.465e-06 1.465e-06 1.465e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.009e-06 1.009e-06 1.009e-06 0.00% AmrLevel::AmrLevel() 1 9.23e-07 9.23e-07 9.23e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3594 0.3594 0.3594 100.00% Amr::coarseTimeStep() 5 0.2778 0.2778 0.2778 77.30% Amr::timeStep() 5 0.2758 0.2758 0.2758 76.74% Castro::advance() 5 0.2727 0.2727 0.2727 75.86% Castro::subcycle_advance_ctu() 5 0.2668 0.2668 0.2668 74.24% Castro::do_advance_ctu() 5 0.2654 0.2654 0.2654 73.85% Castro::construct_new_gravity() 5 0.1451 0.1451 0.1451 40.37% Gravity::solve_phi_with_mlmg() 6 0.1407 0.1407 0.1407 39.13% Gravity::solve_for_phi() 5 0.1371 0.1371 0.1371 38.15% Gravity::actual_solve_with_mlmg() 6 0.1345 0.1345 0.1345 37.43% MLMG::solve() 6 0.1222 0.1222 0.1222 34.00% MLMG::oneIter() 36 0.115 0.115 0.115 32.00% MLMG::mgVcycle() 36 0.1143 0.1143 0.1143 31.79% Castro::construct_ctu_hydro_source() 5 0.08135 0.08135 0.08135 22.63% MLCellLinOp::smooth() 720 0.05836 0.05836 0.05836 16.24% Amr::init() 1 0.04822 0.04822 0.04822 13.42% Amr::restart() 1 0.04822 0.04822 0.04822 13.42% MLCellLinOp::applyBC() 1946 0.04161 0.04161 0.04161 11.58% AmrLevel::restart() 1 0.04049 0.04049 0.04049 11.26% StateData::restartDoit() 4 0.04041 0.04041 0.04041 11.24% VisMF::Read() 3 0.04029 0.04029 0.04029 11.21% MLMG::mgVcycle_bottom 36 0.03508 0.03508 0.03508 9.76% MLMG::actualBottomSolve() 36 0.03506 0.03506 0.03506 9.76% MLCGSolver::bicgstab 36 0.03471 0.03471 0.03471 9.66% Amr::writePlotFile() 1 0.03267 0.03267 0.03267 9.09% VisMF::Write(FabArray) 1 0.03106 0.03106 0.03106 8.64% MLPoisson::Fsmooth() 1440 0.02706 0.02706 0.02706 7.53% Castro::clean_state() 30 0.02149 0.02149 0.02149 5.98% FillPatchIterator::Initialize 20 0.02013 0.02013 0.02013 5.60% FillPatchSingleLevel 20 0.01936 0.01936 0.01936 5.39% StateDataPhysBCFunct::() 20 0.01737 0.01737 0.01737 4.83% MLCellLinOp::apply() 500 0.01578 0.01578 0.01578 4.39% MLMG::mgVcycle_down::0 36 0.01533 0.01533 0.01533 4.26% MLMG::mgVcycle_up::0 36 0.0131 0.0131 0.0131 3.64% StateData::FillBoundary(geom) 160 0.01197 0.01197 0.01197 3.33% MLPoisson::define() 6 0.009963 0.009963 0.009963 2.77% MultiFab::Dot() 484 0.009557 0.009557 0.009557 2.66% Castro::initialize_do_advance() 5 0.009468 0.009468 0.009468 2.63% MLCellLinOp::correctionResidual() 216 0.009232 0.009232 0.009232 2.57% Castro::computeTemp() 30 0.008216 0.008216 0.008216 2.29% MLMG:computeResOfCorrection() 180 0.007955 0.007955 0.007955 2.21% Gravity::get_new_grav_vector() 5 0.007875 0.007875 0.007875 2.19% MLMG::mgVcycle_down::1 36 0.007625 0.007625 0.007625 2.12% Castro::construct_old_gravity() 5 0.007561 0.007561 0.007561 2.10% Gravity::get_old_grav_vector() 5 0.007555 0.007555 0.007555 2.10% MLMG::mgVcycle_down::2 36 0.007368 0.007368 0.007368 2.05% MLMG::mgVcycle_down::3 36 0.007011 0.007011 0.007011 1.95% Castro::do_new_sources() 5 0.006971 0.006971 0.006971 1.94% MLCellLinOp::defineAuxData() 6 0.006945 0.006945 0.006945 1.93% FabArray::FillBoundary() 1766 0.006944 0.006944 0.006944 1.93% Castro::normalize_species() 30 0.006884 0.006884 0.006884 1.92% FabArray::setVal() 537 0.006744 0.006744 0.006744 1.88% MLMG::mgVcycle_down::4 36 0.006707 0.006707 0.006707 1.87% FillBoundary_nowait() 1766 0.006575 0.006575 0.006575 1.83% FabArray::ParallelCopy() 380 0.00643 0.00643 0.00643 1.79% FabArray::ParallelCopy_nowait() 380 0.006311 0.006311 0.006311 1.76% CGSolver::sxay() 690 0.0063 0.0063 0.0063 1.75% MultiFab::LinComb() 690 0.006127 0.006127 0.006127 1.70% Gravity::fill_multipole_BCs() 6 0.005993 0.005993 0.005993 1.67% Castro::enforce_min_density() 30 0.005801 0.005801 0.005801 1.61% MLCGSolver::ParallelAllReduce 659 0.00573 0.00573 0.00573 1.59% Castro::initialize_advance() 5 0.005727 0.005727 0.005727 1.59% MLMG::mgVcycle_up::2 36 0.005638 0.005638 0.005638 1.57% MLMG::mgVcycle_up::1 36 0.005571 0.005571 0.005571 1.55% MLMG::addInterpCorrection() 180 0.00549 0.00549 0.00549 1.53% MLMG::mgVcycle_up::3 36 0.005388 0.005388 0.005388 1.50% MLMG::mgVcycle_up::4 36 0.005369 0.005369 0.005369 1.49% Castro::expand_state() 5 0.005342 0.005342 0.005342 1.49% amrex::average_down 180 0.00518 0.00518 0.00518 1.44% MLPoisson::Fapply() 500 0.005071 0.005071 0.005071 1.41% Castro::do_old_sources() 5 0.004914 0.004914 0.004914 1.37% Castro::post_restart() 1 0.003924 0.003924 0.003924 1.09% Gravity::multilevel_solve_for_new_phi() 1 0.003797 0.003797 0.003797 1.06% Gravity::actual_multilevel_solve() 1 0.00378 0.00378 0.00378 1.05% MLCellLinOp::solutionResidual() 42 0.00324 0.00324 0.00324 0.90% Castro::post_timestep() 5 0.003027 0.003027 0.003027 0.84% Castro::estTimeStep() 10 0.002945 0.002945 0.002945 0.82% Castro::reset_internal_energy(MultiFab) 30 0.002899 0.002899 0.002899 0.81% MLMG::prepareForSolve() 6 0.002886 0.002886 0.002886 0.80% MultiFab::Xpay() 258 0.002884 0.002884 0.002884 0.80% MLCellLinOp::defineBC() 6 0.002859 0.002859 0.002859 0.80% BndryData::define() 6 0.002709 0.002709 0.002709 0.75% MLMG::computeResidual() 36 0.002695 0.002695 0.002695 0.75% Castro::computeNewDt() 5 0.001833 0.001833 0.001833 0.51% Castro::construct_new_source() 25 0.001692 0.001692 0.001692 0.47% Castro::construct_new_gravity_source() 5 0.001639 0.001639 0.001639 0.46% Castro::construct_old_source() 25 0.001473 0.001473 0.001473 0.41% Castro::construct_old_gravity_source() 5 0.001425 0.001425 0.001425 0.40% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009418 0.0009418 0.0009418 0.26% Castro::apply_source_to_state() 10 0.0009203 0.0009203 0.0009203 0.26% MultiFab::Saxpy() 10 0.0009148 0.0009148 0.0009148 0.25% MLMG::ResNormInf() 42 0.0009088 0.0009088 0.0009088 0.25% MLCellLinOp::setLevelBC() 6 0.0008278 0.0008278 0.0008278 0.23% MLMG::getGradSolution() 6 0.0007754 0.0007754 0.0007754 0.22% MLCellLinOp::compGrad() 6 0.0007723 0.0007723 0.0007723 0.21% FabArrayBase::getCPC() 632 0.0007621 0.0007621 0.0007621 0.21% Castro::reset_internal_energy(Fab) 240 0.000758 0.000758 0.000758 0.21% MultiFab::Add() 36 0.0007099 0.0007099 0.0007099 0.20% FabArray::mult() 22 0.0006574 0.0006574 0.0006574 0.18% MLPoisson::prepareForSolve() 6 0.0006342 0.0006342 0.0006342 0.18% FabArray::setDomainBndry() 20 0.0006336 0.0006336 0.0006336 0.18% MLCellLinOp::prepareForSolve() 6 0.0006295 0.0006295 0.0006295 0.18% Castro::check_for_nan() 10 0.0005876 0.0005876 0.0005876 0.16% MultiFab::contains_nan() 10 0.0005819 0.0005819 0.0005819 0.16% MLMG::computeMLResidual() 6 0.0005677 0.0005677 0.0005677 0.16% Castro::enforce_speed_limit() 30 0.0005465 0.0005465 0.0005465 0.15% Gravity::update_max_rhs() 6 0.0004462 0.0004462 0.0004462 0.12% Amr::InitAmr() 1 0.0004179 0.0004179 0.0004179 0.12% FabArrayBase::CPC::define() 244 0.0003914 0.0003914 0.0003914 0.11% FabArrayBase::getFB() 1766 0.0003205 0.0003205 0.0003205 0.09% Gravity::swapTimeLevels() 5 0.0002255 0.0002255 0.0002255 0.06% MultiFab::Copy() 6 0.0001786 0.0001786 0.0001786 0.05% Castro::buildMetrics() 1 0.0001543 0.0001543 0.0001543 0.04% MLMG::MLResNormInf() 6 0.0001474 0.0001474 0.0001474 0.04% MLLinOp::define() 6 0.0001436 0.0001436 0.0001436 0.04% MultiFab::max() 6 0.0001347 0.0001347 0.0001347 0.04% MLLinOp::defineGrids() 6 0.0001222 0.0001222 0.0001222 0.03% MLMG::MLRhsNormInf() 6 0.0001136 0.0001136 0.0001136 0.03% Castro::create_source_corrector() 5 9.701e-05 9.701e-05 9.701e-05 0.03% Castro::finalize_advance() 5 6.813e-05 6.813e-05 6.813e-05 0.02% FabArrayBase::FB::FB() 26 5.719e-05 5.719e-05 5.719e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.821e-05 2.821e-05 2.821e-05 0.01% Castro::initMFs() 1 2.685e-05 2.685e-05 2.685e-05 0.01% Amr::writeSmallPlotFile() 1 2.579e-05 2.579e-05 2.579e-05 0.01% Castro::swap_state_time_levels() 5 2.527e-05 2.527e-05 2.527e-05 0.01% makeSFC 30 2.032e-05 2.032e-05 2.032e-05 0.01% Castro::finalize_do_advance() 5 1.907e-05 1.907e-05 1.907e-05 0.01% Amr::initSubcycle() 1 8.865e-06 8.865e-06 8.865e-06 0.00% DistributionMapping::Distribute() 31 8.029e-06 8.029e-06 8.029e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.441e-06 4.441e-06 4.441e-06 0.00% Castro::retry_advance_ctu() 5 3.397e-06 3.397e-06 3.397e-06 0.00% Castro::FluxRegCrseInit 5 2.679e-06 2.679e-06 2.679e-06 0.00% Gravity::set_mass_offset() 6 2.075e-06 2.075e-06 2.075e-06 0.00% Castro::FluxRegFineAdd() 5 1.679e-06 1.679e-06 1.679e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.009e-06 1.009e-06 1.009e-06 0.00% AmrLevel::AmrLevel() 1 9.23e-07 9.23e-07 9.23e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-8-g3f715d29c94b) finalized