Initializing AMReX (v2024-35-gb892132f6144)... Initializing CUDA... CUDA initialized with 1 device. AMReX (v2024-35-gb892132f6144) initialized Starting run at 08:24:47 UTC on 2024-03-27. Successfully read inputs file ... Castro git describe: 24.03-31-g760ecc416 AMReX git describe: v2024-35-gb892132f6 Microphysics git describe: 24.03-22-gf47a7fcc reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.043791128 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.024277528 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.071261583 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049149448 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.068429501 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.074127869 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.065620195 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.06559891 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.063834779 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.064364083 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.071414732 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.069125816 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.06638394 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.043005137 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024194109 seconds Ending run at 08:24:48 UTC on 2024-03-27. Run time = 0.919034466 Run time without initialization = 0.797233084 Average number of zones advanced per microsecond: 3.288 Average number of zones advanced per microsecond per rank: 3.288 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.9191 ... 0.9191 ... 0.9191 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2894 0.2894 0.2894 31.49% VisMF::Write(FabArray) 11 0.1914 0.1914 0.1914 20.82% MLCellLinOp::applyBC() 4351 0.07942 0.07942 0.07942 8.64% MLPoisson::Fsmooth() 3280 0.03296 0.03296 0.03296 3.59% FillBoundary_nowait() 3941 0.03058 0.03058 0.03058 3.33% StateData::FillBoundary(geom) 328 0.0261 0.0261 0.0261 2.84% amrex::Dot() 1114 0.02083 0.02083 0.02083 2.27% Castro::normalize_species() 62 0.01994 0.01994 0.01994 2.17% FabArray::norminf() 1061 0.01936 0.01936 0.01936 2.11% Castro::computeTemp() 63 0.01571 0.01571 0.01571 1.71% FabArray::ParallelCopy_nowait() 861 0.0134 0.0134 0.0134 1.46% FabArray::setVal() 1062 0.01318 0.01318 0.01318 1.43% FabArray::Saxpy() 1370 0.01282 0.01282 0.01282 1.39% StateDataPhysBCFunct::() 41 0.01281 0.01281 0.01281 1.39% Castro::enforce_min_density() 62 0.01187 0.01187 0.01187 1.29% amrex::Copy() 472 0.01081 0.01081 0.01081 1.18% MLPoisson::Fapply() 1060 0.01005 0.01005 0.01005 1.09% MLCellLinOp::defineAuxData() 11 0.009972 0.009972 0.009972 1.08% Gravity::fill_multipole_BCs() 11 0.009146 0.009146 0.009146 1.00% FabArray::Xpay() 739 0.007707 0.007707 0.007707 0.84% Amr::checkPoint() 3 0.006935 0.006935 0.006935 0.75% MLMG::addInterpCorrection() 410 0.006933 0.006933 0.006933 0.75% amrex::average_down 410 0.006104 0.006104 0.006104 0.66% Castro::estTimeStep() 21 0.006094 0.006094 0.006094 0.66% Castro::reset_internal_energy(MultiFab) 63 0.005283 0.005283 0.005283 0.57% BndryData::define() 11 0.003839 0.003839 0.003839 0.42% amrex::Add() 82 0.003633 0.003633 0.003633 0.40% Castro::construct_new_gravity_source() 10 0.003437 0.003437 0.003437 0.37% Castro::enforce_speed_limit() 62 0.003231 0.003231 0.003231 0.35% Castro::construct_old_gravity_source() 10 0.002835 0.002835 0.002835 0.31% Amr::writePlotFile() 2 0.002132 0.002132 0.002132 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001757 0.001757 0.001757 0.19% check_for_negative_density() 10 0.001669 0.001669 0.001669 0.18% Castro::reset_internal_energy(Fab) 504 0.001664 0.001664 0.001664 0.18% MLCGSolver::bicgstab 82 0.00161 0.00161 0.00161 0.18% MLCellLinOp::setLevelBC() 11 0.001545 0.001545 0.001545 0.17% Gravity::actual_solve_with_mlmg() 11 0.001537 0.001537 0.001537 0.17% Castro::initData() 1 0.001421 0.001421 0.001421 0.15% FabArray::mult() 43 0.001375 0.001375 0.001375 0.15% FabArray::setDomainBndry() 41 0.001373 0.001373 0.001373 0.15% MLCellLinOp::prepareForSolve() 11 0.00129 0.00129 0.00129 0.14% MultiFab::contains_nan() 20 0.00127 0.00127 0.00127 0.14% MLCellLinOp::compGrad() 11 0.001042 0.001042 0.001042 0.11% MLCellLinOp::smooth() 1640 0.001029 0.001029 0.001029 0.11% MLMG::prepareForSolve() 11 0.0009297 0.0009297 0.0009297 0.10% FabArray::FillBoundary() 3941 0.0008502 0.0008502 0.0008502 0.09% FabArrayBase::getCPC() 1323 0.0007933 0.0007933 0.0007933 0.09% FabArrayBase::getFB() 3941 0.0006628 0.0006628 0.0006628 0.07% FabArrayBase::CPC::define() 454 0.0006478 0.0006478 0.0006478 0.07% Gravity::get_new_grav_vector() 11 0.0006052 0.0006052 0.0006052 0.07% Amr::InitAmr() 1 0.0005245 0.0005245 0.0005245 0.06% Gravity::get_old_grav_vector() 10 0.0004832 0.0004832 0.0004832 0.05% AmrLevel::FillPatch() 41 0.0004177 0.0004177 0.0004177 0.05% MLCellLinOp::apply() 1060 0.0004094 0.0004094 0.0004094 0.04% Amr::coarseTimeStep() 10 0.0003694 0.0003694 0.0003694 0.04% MultiFab::max() 11 0.000324 0.000324 0.000324 0.04% MLCGSolver::ParallelAllReduce 1832 0.0003222 0.0003222 0.0003222 0.04% main() 1 0.0003043 0.0003043 0.0003043 0.03% MLCellLinOp::defineBC() 11 0.0002705 0.0002705 0.0002705 0.03% FabArray::ParallelCopy() 861 0.0002512 0.0002512 0.0002512 0.03% FillPatchIterator::Initialize 41 0.0002077 0.0002077 0.0002077 0.02% MLMG::mgVcycle() 82 0.0001981 0.0001981 0.0001981 0.02% MLLinOp::defineGrids() 11 0.0001844 0.0001844 0.0001844 0.02% Castro::subcycle_advance_ctu() 10 0.00018 0.00018 0.00018 0.02% MLCellLinOp::correctionResidual() 410 0.0001791 0.0001791 0.0001791 0.02% Castro::create_source_corrector() 10 0.0001634 0.0001634 0.0001634 0.02% Amr::timeStep() 10 0.0001569 0.0001569 0.0001569 0.02% Castro::construct_new_source() 50 0.000135 0.000135 0.000135 0.01% StateData::checkPoint() 12 0.0001284 0.0001284 0.0001284 0.01% Gravity::update_max_rhs() 11 0.0001267 0.0001267 0.0001267 0.01% Gravity::solve_for_phi() 10 0.00012 0.00012 0.00012 0.01% MLMG:computeResOfCorrection() 410 0.0001194 0.0001194 0.0001194 0.01% Castro::advance() 10 9.286e-05 9.286e-05 9.286e-05 0.01% Castro::Castro() 1 9.001e-05 9.001e-05 9.001e-05 0.01% FabArrayBase::FB::FB() 56 8.67e-05 8.67e-05 8.67e-05 0.01% MLMG::actualBottomSolve() 82 8.569e-05 8.569e-05 8.569e-05 0.01% MLMG::mgVcycle_down::0 82 8.517e-05 8.517e-05 8.517e-05 0.01% Castro::initialize_advance() 10 7.822e-05 7.822e-05 7.822e-05 0.01% Castro::clean_state() 62 7.755e-05 7.755e-05 7.755e-05 0.01% MLMG::solve() 11 7.201e-05 7.201e-05 7.201e-05 0.01% MLMG::mgVcycle_down::1 82 7.143e-05 7.143e-05 7.143e-05 0.01% MLMG::mgVcycle_down::2 82 7.014e-05 7.014e-05 7.014e-05 0.01% MLMG::mgVcycle_down::4 82 6.794e-05 6.794e-05 6.794e-05 0.01% Castro::enforce_consistent_e() 1 6.479e-05 6.479e-05 6.479e-05 0.01% MLMG::mgVcycle_down::3 82 6.459e-05 6.459e-05 6.459e-05 0.01% AmrLevel::checkPoint() 3 6.31e-05 6.31e-05 6.31e-05 0.01% MLMG::mgVcycle_up::4 82 5.967e-05 5.967e-05 5.967e-05 0.01% Castro::post_timestep() 10 5.883e-05 5.883e-05 5.883e-05 0.01% Castro::finalize_advance() 10 5.877e-05 5.877e-05 5.877e-05 0.01% Castro::initialize_do_advance() 10 5.748e-05 5.748e-05 5.748e-05 0.01% MLMG::mgVcycle_up::0 82 5.394e-05 5.394e-05 5.394e-05 0.01% MLCellLinOp::solutionResidual() 93 5.19e-05 5.19e-05 5.19e-05 0.01% MLMG::mgVcycle_up::3 82 5.169e-05 5.169e-05 5.169e-05 0.01% MLMG::mgVcycle_up::1 82 5.11e-05 5.11e-05 5.11e-05 0.01% MLMG::oneIter() 82 5.067e-05 5.067e-05 5.067e-05 0.01% MLMG::mgVcycle_up::2 82 4.919e-05 4.919e-05 4.919e-05 0.01% FillPatchIterator::FillFromLevel0() 41 4.816e-05 4.816e-05 4.816e-05 0.01% Castro::do_advance_ctu() 10 4.599e-05 4.599e-05 4.599e-05 0.01% StateData::define() 4 4.276e-05 4.276e-05 4.276e-05 0.00% Castro::finalize_do_advance() 10 4.254e-05 4.254e-05 4.254e-05 0.00% Castro::swap_state_time_levels() 10 3.769e-05 3.769e-05 3.769e-05 0.00% MLMG::mgVcycle_bottom 82 3.404e-05 3.404e-05 3.404e-05 0.00% FillPatchSingleLevel 41 3.383e-05 3.383e-05 3.383e-05 0.00% MLMG::computeResidual() 82 3.304e-05 3.304e-05 3.304e-05 0.00% Amr::writeSmallPlotFile() 1 3.248e-05 3.248e-05 3.248e-05 0.00% MLMG::ResNormInf() 93 3.179e-05 3.179e-05 3.179e-05 0.00% Castro::initMFs() 1 3.138e-05 3.138e-05 3.138e-05 0.00% makeSFC 55 2.878e-05 2.878e-05 2.878e-05 0.00% Castro::construct_new_gravity() 10 2.813e-05 2.813e-05 2.813e-05 0.00% Amr::defBaseLevel() 1 2.772e-05 2.772e-05 2.772e-05 0.00% MLPoisson::define() 11 2.685e-05 2.685e-05 2.685e-05 0.00% Castro::buildMetrics() 1 2.669e-05 2.669e-05 2.669e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.624e-05 2.624e-05 2.624e-05 0.00% Castro::do_old_sources() 10 2.074e-05 2.074e-05 2.074e-05 0.00% Amr::FinalizeInit() 1 2.01e-05 2.01e-05 2.01e-05 0.00% Castro::do_new_sources() 10 1.958e-05 1.958e-05 1.958e-05 0.00% Castro::construct_old_source() 50 1.876e-05 1.876e-05 1.876e-05 0.00% DistributionMapping::Distribute() 56 1.818e-05 1.818e-05 1.818e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.747e-05 1.747e-05 1.747e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.697e-05 1.697e-05 1.697e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.354e-05 1.354e-05 1.354e-05 0.00% Castro::check_for_nan() 20 1.274e-05 1.274e-05 1.274e-05 0.00% MLPoisson::prepareForSolve() 11 1.238e-05 1.238e-05 1.238e-05 0.00% Castro::apply_source_to_state() 20 1.154e-05 1.154e-05 1.154e-05 0.00% MLLinOp::define() 11 1.138e-05 1.138e-05 1.138e-05 0.00% Amr::initSubcycle() 1 1.003e-05 1.003e-05 1.003e-05 0.00% Castro::construct_old_gravity() 10 9.817e-06 9.817e-06 9.817e-06 0.00% Castro::post_init() 1 9.571e-06 9.571e-06 9.571e-06 0.00% MLMG::computeMLResidual() 11 9.119e-06 9.119e-06 9.119e-06 0.00% Gravity::swapTimeLevels() 10 8.905e-06 8.905e-06 8.905e-06 0.00% Gravity::actual_multilevel_solve() 1 7.633e-06 7.633e-06 7.633e-06 0.00% Castro::computeNewDt() 9 6.95e-06 6.95e-06 6.95e-06 0.00% Castro::expand_state() 10 6.044e-06 6.044e-06 6.044e-06 0.00% MLMG::getGradSolution() 11 5.789e-06 5.789e-06 5.789e-06 0.00% Amr::InitializeInit() 1 5.676e-06 5.676e-06 5.676e-06 0.00% AmrLevel::checkPointPost() 3 5.259e-06 5.259e-06 5.259e-06 0.00% Gravity::set_mass_offset() 11 5.007e-06 5.007e-06 5.007e-06 0.00% Castro::retry_advance_ctu() 10 4.836e-06 4.836e-06 4.836e-06 0.00% MLMG::MLRhsNormInf() 11 4.271e-06 4.271e-06 4.271e-06 0.00% Amr::initialInit() 1 3.643e-06 3.643e-06 3.643e-06 0.00% MLMG::MLResNormInf() 11 3.633e-06 3.633e-06 3.633e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.392e-06 3.392e-06 3.392e-06 0.00% Castro::FluxRegCrseInit 10 2.863e-06 2.863e-06 2.863e-06 0.00% Castro::computeInitialDt() 2 2.549e-06 2.549e-06 2.549e-06 0.00% Castro::FluxRegFineAdd() 10 2.491e-06 2.491e-06 2.491e-06 0.00% Amr::init() 1 2.437e-06 2.437e-06 2.437e-06 0.00% AmrLevel::checkPointPre() 3 2.342e-06 2.342e-06 2.342e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.206e-06 2.206e-06 2.206e-06 0.00% Castro::post_regrid() 1 1.322e-06 1.322e-06 1.322e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.919 0.919 0.919 100.00% Amr::coarseTimeStep() 10 0.7728 0.7728 0.7728 84.08% Amr::timeStep() 10 0.6602 0.6602 0.6602 71.83% Castro::advance() 10 0.6491 0.6491 0.6491 70.62% Castro::subcycle_advance_ctu() 10 0.6355 0.6355 0.6355 69.14% Castro::do_advance_ctu() 10 0.6353 0.6353 0.6353 69.12% Castro::construct_ctu_hydro_source() 10 0.2996 0.2996 0.2996 32.60% Gravity::solve_phi_with_mlmg() 11 0.2889 0.2889 0.2889 31.44% Gravity::actual_solve_with_mlmg() 11 0.2793 0.2793 0.2793 30.39% Castro::construct_new_gravity() 10 0.2614 0.2614 0.2614 28.44% MLMG::solve() 11 0.2579 0.2579 0.2579 28.06% Gravity::solve_for_phi() 10 0.2452 0.2452 0.2452 26.68% MLMG::oneIter() 82 0.2427 0.2427 0.2427 26.41% MLMG::mgVcycle() 82 0.239 0.239 0.239 26.01% VisMF::Write(FabArray) 11 0.1914 0.1914 0.1914 20.82% Amr::checkPoint() 3 0.1526 0.1526 0.1526 16.60% AmrLevel::checkPoint() 3 0.1456 0.1456 0.1456 15.84% StateData::checkPoint() 12 0.1456 0.1456 0.1456 15.84% Amr::init() 1 0.1212 0.1212 0.1212 13.18% MLCellLinOp::smooth() 1640 0.1199 0.1199 0.1199 13.05% MLCellLinOp::applyBC() 4351 0.1116 0.1116 0.1116 12.14% MLMG::mgVcycle_bottom 82 0.07093 0.07093 0.07093 7.72% MLMG::actualBottomSolve() 82 0.07089 0.07089 0.07089 7.71% MLCGSolver::bicgstab 82 0.07009 0.07009 0.07009 7.63% Castro::clean_state() 62 0.0569 0.0569 0.0569 6.19% Amr::initialInit() 1 0.05295 0.05295 0.05295 5.76% AmrLevel::FillPatch() 41 0.0489 0.0489 0.0489 5.32% Amr::writePlotFile() 2 0.04859 0.04859 0.04859 5.29% Amr::FinalizeInit() 1 0.04808 0.04808 0.04808 5.23% Castro::post_init() 1 0.04659 0.04659 0.04659 5.07% FillPatchIterator::Initialize 41 0.0446 0.0446 0.0446 4.85% Gravity::multilevel_solve_for_new_phi() 1 0.04418 0.04418 0.04418 4.81% Gravity::actual_multilevel_solve() 1 0.04416 0.04416 0.04416 4.80% FillPatchIterator::FillFromLevel0() 41 0.04302 0.04302 0.04302 4.68% FillPatchSingleLevel 41 0.04297 0.04297 0.04297 4.68% StateDataPhysBCFunct::() 41 0.03891 0.03891 0.03891 4.23% MLCellLinOp::apply() 1060 0.03564 0.03564 0.03564 3.88% MLMG::mgVcycle_down::0 82 0.03429 0.03429 0.03429 3.73% MLPoisson::Fsmooth() 3280 0.03296 0.03296 0.03296 3.59% FabArray::FillBoundary() 3941 0.03218 0.03218 0.03218 3.50% FillBoundary_nowait() 3941 0.03133 0.03133 0.03133 3.41% StateData::FillBoundary(geom) 328 0.0261 0.0261 0.0261 2.84% MLMG::mgVcycle_up::0 82 0.02598 0.02598 0.02598 2.83% Castro::computeTemp() 63 0.02265 0.02265 0.02265 2.46% Castro::initialize_do_advance() 10 0.02254 0.02254 0.02254 2.45% amrex::Dot() 1114 0.02083 0.02083 0.02083 2.27% MLMG:computeResOfCorrection() 410 0.02011 0.02011 0.02011 2.19% Castro::do_old_sources() 10 0.02005 0.02005 0.02005 2.18% MLCellLinOp::correctionResidual() 410 0.01999 0.01999 0.01999 2.18% Castro::normalize_species() 62 0.01994 0.01994 0.01994 2.17% FabArray::norminf() 1061 0.01936 0.01936 0.01936 2.11% Gravity::get_new_grav_vector() 11 0.01815 0.01815 0.01815 1.98% MLPoisson::define() 11 0.01676 0.01676 0.01676 1.82% MLMG::mgVcycle_down::1 82 0.01635 0.01635 0.01635 1.78% MLMG::mgVcycle_down::2 82 0.01516 0.01516 0.01516 1.65% Castro::construct_old_gravity() 10 0.01513 0.01513 0.01513 1.65% Gravity::get_old_grav_vector() 10 0.01513 0.01513 0.01513 1.65% MLMG::mgVcycle_down::3 82 0.01479 0.01479 0.01479 1.61% MLMG::mgVcycle_down::4 82 0.01471 0.01471 0.01471 1.60% FabArray::ParallelCopy() 861 0.01445 0.01445 0.01445 1.57% FabArray::ParallelCopy_nowait() 861 0.0142 0.0142 0.0142 1.55% Castro::do_new_sources() 10 0.01324 0.01324 0.01324 1.44% FabArray::setVal() 1062 0.01318 0.01318 0.01318 1.43% Castro::initialize_advance() 10 0.01292 0.01292 0.01292 1.41% FabArray::Saxpy() 1370 0.01282 0.01282 0.01282 1.39% MLCGSolver::ParallelAllReduce 1832 0.01253 0.01253 0.01253 1.36% Castro::expand_state() 10 0.01251 0.01251 0.01251 1.36% MLMG::addInterpCorrection() 410 0.01213 0.01213 0.01213 1.32% Castro::enforce_min_density() 62 0.01187 0.01187 0.01187 1.29% MLMG::mgVcycle_up::1 82 0.01186 0.01186 0.01186 1.29% MLMG::mgVcycle_up::4 82 0.01179 0.01179 0.01179 1.28% MLMG::mgVcycle_up::2 82 0.01159 0.01159 0.01159 1.26% MLMG::mgVcycle_up::3 82 0.01137 0.01137 0.01137 1.24% MLCellLinOp::defineAuxData() 11 0.01137 0.01137 0.01137 1.24% amrex::average_down 410 0.01133 0.01133 0.01133 1.23% Castro::post_timestep() 10 0.01093 0.01093 0.01093 1.19% amrex::Copy() 472 0.01081 0.01081 0.01081 1.18% MLPoisson::Fapply() 1060 0.01005 0.01005 0.01005 1.09% Gravity::fill_multipole_BCs() 11 0.009398 0.009398 0.009398 1.02% FabArray::Xpay() 739 0.007707 0.007707 0.007707 0.84% MLCellLinOp::solutionResidual() 93 0.007663 0.007663 0.007663 0.83% Castro::reset_internal_energy(MultiFab) 63 0.006947 0.006947 0.006947 0.76% MLMG::computeResidual() 82 0.006426 0.006426 0.006426 0.70% Castro::estTimeStep() 21 0.006094 0.006094 0.006094 0.66% MLCellLinOp::defineBC() 11 0.005103 0.005103 0.005103 0.56% MLMG::prepareForSolve() 11 0.005029 0.005029 0.005029 0.55% Amr::InitializeInit() 1 0.004863 0.004863 0.004863 0.53% Amr::defBaseLevel() 1 0.004858 0.004858 0.004858 0.53% BndryData::define() 11 0.004832 0.004832 0.004832 0.53% Castro::initData() 1 0.004226 0.004226 0.004226 0.46% amrex::Add() 82 0.003633 0.003633 0.003633 0.40% Castro::construct_new_source() 50 0.003572 0.003572 0.003572 0.39% Castro::construct_new_gravity_source() 10 0.003437 0.003437 0.003437 0.37% Castro::enforce_speed_limit() 62 0.003231 0.003231 0.003231 0.35% Castro::construct_old_source() 50 0.002854 0.002854 0.002854 0.31% Castro::construct_old_gravity_source() 10 0.002835 0.002835 0.002835 0.31% Castro::computeNewDt() 9 0.002775 0.002775 0.002775 0.30% Castro::finalize_do_advance() 10 0.002337 0.002337 0.002337 0.25% MLMG::ResNormInf() 93 0.002147 0.002147 0.002147 0.23% Castro::apply_source_to_state() 20 0.001863 0.001863 0.001863 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001757 0.001757 0.001757 0.19% check_for_negative_density() 10 0.001669 0.001669 0.001669 0.18% Castro::reset_internal_energy(Fab) 504 0.001664 0.001664 0.001664 0.18% MLMG::getGradSolution() 11 0.001557 0.001557 0.001557 0.17% MLCellLinOp::compGrad() 11 0.001552 0.001552 0.001552 0.17% MLCellLinOp::setLevelBC() 11 0.001545 0.001545 0.001545 0.17% FabArrayBase::getCPC() 1323 0.001441 0.001441 0.001441 0.16% FabArray::mult() 43 0.001375 0.001375 0.001375 0.15% FabArray::setDomainBndry() 41 0.001373 0.001373 0.001373 0.15% MLPoisson::prepareForSolve() 11 0.001303 0.001303 0.001303 0.14% MLCellLinOp::prepareForSolve() 11 0.00129 0.00129 0.00129 0.14% Castro::check_for_nan() 20 0.001283 0.001283 0.001283 0.14% MLMG::computeMLResidual() 11 0.001279 0.001279 0.001279 0.14% MultiFab::contains_nan() 20 0.00127 0.00127 0.00127 0.14% Castro::post_regrid() 1 0.001199 0.001199 0.001199 0.13% Castro::computeInitialDt() 2 0.001034 0.001034 0.001034 0.11% Gravity::update_max_rhs() 11 0.0009783 0.0009783 0.0009783 0.11% FabArrayBase::getFB() 3941 0.0007495 0.0007495 0.0007495 0.08% FabArrayBase::CPC::define() 454 0.0006478 0.0006478 0.0006478 0.07% Castro::finalize_advance() 10 0.0005973 0.0005973 0.0005973 0.06% Castro::Castro() 1 0.0005393 0.0005393 0.0005393 0.06% Amr::InitAmr() 1 0.0005345 0.0005345 0.0005345 0.06% Gravity::swapTimeLevels() 10 0.0004693 0.0004693 0.0004693 0.05% MLMG::MLResNormInf() 11 0.0003262 0.0003262 0.0003262 0.04% MultiFab::max() 11 0.000324 0.000324 0.000324 0.04% MLLinOp::define() 11 0.0002574 0.0002574 0.0002574 0.03% MLLinOp::defineGrids() 11 0.000246 0.000246 0.000246 0.03% Castro::buildMetrics() 1 0.0002446 0.0002446 0.0002446 0.03% MLMG::MLRhsNormInf() 11 0.0002262 0.0002262 0.0002262 0.02% Castro::create_source_corrector() 10 0.0001634 0.0001634 0.0001634 0.02% FabArrayBase::FB::FB() 56 8.67e-05 8.67e-05 8.67e-05 0.01% Castro::enforce_consistent_e() 1 6.479e-05 6.479e-05 6.479e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.973e-05 5.973e-05 5.973e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.94e-05 5.94e-05 5.94e-05 0.01% makeSFC 55 4.586e-05 4.586e-05 4.586e-05 0.00% StateData::define() 4 4.276e-05 4.276e-05 4.276e-05 0.00% Castro::swap_state_time_levels() 10 3.769e-05 3.769e-05 3.769e-05 0.00% Amr::writeSmallPlotFile() 1 3.248e-05 3.248e-05 3.248e-05 0.00% Castro::initMFs() 1 3.138e-05 3.138e-05 3.138e-05 0.00% DistributionMapping::Distribute() 56 1.818e-05 1.818e-05 1.818e-05 0.00% Amr::initSubcycle() 1 1.003e-05 1.003e-05 1.003e-05 0.00% AmrLevel::checkPointPost() 3 5.259e-06 5.259e-06 5.259e-06 0.00% Gravity::set_mass_offset() 11 5.007e-06 5.007e-06 5.007e-06 0.00% Castro::retry_advance_ctu() 10 4.836e-06 4.836e-06 4.836e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.479e-06 4.479e-06 4.479e-06 0.00% Castro::FluxRegCrseInit 10 2.863e-06 2.863e-06 2.863e-06 0.00% Castro::FluxRegFineAdd() 10 2.491e-06 2.491e-06 2.491e-06 0.00% AmrLevel::checkPointPre() 3 2.342e-06 2.342e-06 2.342e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.206e-06 2.206e-06 2.206e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 5302 KiB 9037 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1050 KiB 39 MiB Castro::initialize_do_advance() 80 80 26 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1473 KiB 28 MiB Castro::initialize_advance() 80 80 16 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7617 KiB 14 MiB MLMG::prepareForSolve() 660 660 3451 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 205 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 168 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7519 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 17 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2062 B 2048 KiB Gravity::solve_for_phi() 80 80 545 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 98 KiB 2048 KiB BndryData::define() 1056 1056 315 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 200 KiB 671 KiB Castro::estTimeStep() 21 21 3233 B 480 KiB VisMF::Write(FabArray) 656 656 3285 B 320 KiB Castro::normalize_species() 62 62 7065 B 320 KiB amrex::average_down 1067 1067 1578 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1134 B 257 KiB amrex::Dot() 1360 1360 3348 B 160 KiB FabArray::norminf() 1143 1143 3253 B 160 KiB check_for_negative_density() 10 10 286 B 160 KiB Castro::initData() 1 1 48 B 160 KiB MultiFab::max() 11 11 54 B 160 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MultiFab::contains_nan() 20 20 27 B 20 KiB MLPoisson::Fsmooth() 132 132 3369 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 43 B 10 KiB FillBoundary_nowait() 760 760 290 B 9648 B MLCellLinOp::applyBC() 8702 8702 215 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3903 B 6144 B StateData::FillBoundary(geom) 1992 1992 48 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 355 B 1248 B MLCGSolver::bicgstab 410 410 92 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 533 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 39 KiB 8192 KiB VisMF::Write(FabArray) 744 744 397 KiB 3584 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MLPoisson::Fsmooth() 132 132 3369 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 43 B 10 KiB FillBoundary_nowait() 760 760 290 B 9648 B MLCellLinOp::applyBC() 4351 4351 214 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3903 B 6144 B Gravity::get_new_grav_vector() 3 3 2897 B 3072 B StateData::FillBoundary(geom) 1992 1992 48 B 3024 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B amrex::average_down 83 83 617 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 287 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 24 B 400 B FabArray::norminf() 1143 1143 9 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2180 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (v2024-35-gb892132f6144) finalized Initializing AMReX (v2024-35-gb892132f6144)... Initializing CUDA... CUDA initialized with 1 device. AMReX (v2024-35-gb892132f6144) initialized Starting run at 08:24:49 UTC on 2024-03-27. Successfully read inputs file ... Castro git describe: 24.03-31-g760ecc416 AMReX git describe: v2024-35-gb892132f6 Microphysics git describe: 24.03-22-gf47a7fcc reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.516013815 Restart time = 0.071128288 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.071637653 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.054908569 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.072251161 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.075276307 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.053731153 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026592899 seconds Ending run at 08:24:49 UTC on 2024-03-27. Run time = 0.426594373 Run time without initialization = 0.354837021 Average number of zones advanced per microsecond: 3.694 Average number of zones advanced per microsecond per rank: 3.694 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.4266 ... 0.4266 ... 0.4266 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1400 0.1400 0.1400 32.82% VisMF::Read() 3 0.05998 0.05998 0.05998 14.06% MLCellLinOp::applyBC() 1910 0.03487 0.03487 0.03487 8.17% VisMF::Write(FabArray) 1 0.02386 0.02386 0.02386 5.59% MLPoisson::Fsmooth() 1440 0.01443 0.01443 0.01443 3.38% FillBoundary_nowait() 1730 0.01307 0.01307 0.01307 3.06% StateData::FillBoundary(geom) 160 0.01256 0.01256 0.01256 2.94% amrex::Dot() 484 0.008947 0.008947 0.008947 2.10% Castro::normalize_species() 30 0.008664 0.008664 0.008664 2.03% FabArray::norminf() 465 0.008448 0.008448 0.008448 1.98% Castro::computeTemp() 30 0.007438 0.007438 0.007438 1.74% FabArray::setVal() 501 0.006487 0.006487 0.006487 1.52% FabArray::ParallelCopy_nowait() 380 0.006068 0.006068 0.006068 1.42% FabArray::Saxpy() 597 0.005728 0.005728 0.005728 1.34% StateDataPhysBCFunct::() 20 0.005685 0.005685 0.005685 1.33% MLCellLinOp::defineAuxData() 6 0.005478 0.005478 0.005478 1.28% Castro::enforce_min_density() 30 0.005432 0.005432 0.005432 1.27% amrex::Copy() 221 0.005387 0.005387 0.005387 1.26% Amr::restart() 1 0.004867 0.004867 0.004867 1.14% MLPoisson::Fapply() 464 0.00438 0.00438 0.00438 1.03% Gravity::fill_multipole_BCs() 6 0.004292 0.004292 0.004292 1.01% FabArray::Xpay() 325 0.00341 0.00341 0.00341 0.80% MLMG::addInterpCorrection() 180 0.003079 0.003079 0.003079 0.72% Castro::estTimeStep() 10 0.00288 0.00288 0.00288 0.68% amrex::average_down 180 0.002698 0.002698 0.002698 0.63% Amr::writePlotFile() 1 0.002594 0.002594 0.002594 0.61% BndryData::define() 6 0.002127 0.002127 0.002127 0.50% Castro::construct_new_gravity_source() 5 0.001969 0.001969 0.001969 0.46% Castro::reset_internal_energy(MultiFab) 30 0.001789 0.001789 0.001789 0.42% Castro::construct_old_gravity_source() 5 0.00158 0.00158 0.00158 0.37% amrex::Add() 36 0.001545 0.001545 0.001545 0.36% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009573 0.0009573 0.0009573 0.22% Castro::reset_internal_energy(Fab) 240 0.0009148 0.0009148 0.0009148 0.21% MLCellLinOp::setLevelBC() 6 0.000864 0.000864 0.000864 0.20% check_for_negative_density() 5 0.0008426 0.0008426 0.0008426 0.20% Gravity::actual_solve_with_mlmg() 6 0.0008324 0.0008324 0.0008324 0.20% MLCellLinOp::prepareForSolve() 6 0.0007542 0.0007542 0.0007542 0.18% FabArray::setDomainBndry() 20 0.0007149 0.0007149 0.0007149 0.17% MLCGSolver::bicgstab 36 0.0007034 0.0007034 0.0007034 0.16% FabArray::mult() 22 0.0006933 0.0006933 0.0006933 0.16% MultiFab::contains_nan() 10 0.0006593 0.0006593 0.0006593 0.15% MLCellLinOp::compGrad() 6 0.0005927 0.0005927 0.0005927 0.14% MLMG::prepareForSolve() 6 0.0005458 0.0005458 0.0005458 0.13% Amr::InitAmr() 1 0.0004627 0.0004627 0.0004627 0.11% MLCellLinOp::smooth() 720 0.0004542 0.0004542 0.0004542 0.11% FabArrayBase::CPC::define() 244 0.0004267 0.0004267 0.0004267 0.10% Castro::enforce_speed_limit() 30 0.0004188 0.0004188 0.0004188 0.10% Gravity::get_old_grav_vector() 5 0.0003712 0.0003712 0.0003712 0.09% FabArrayBase::getCPC() 632 0.0003708 0.0003708 0.0003708 0.09% FabArray::FillBoundary() 1730 0.0003607 0.0003607 0.0003607 0.08% Gravity::get_new_grav_vector() 5 0.0003328 0.0003328 0.0003328 0.08% main() 1 0.0002668 0.0002668 0.0002668 0.06% FabArrayBase::getFB() 1730 0.0002578 0.0002578 0.0002578 0.06% MultiFab::max() 6 0.0002036 0.0002036 0.0002036 0.05% AmrLevel::FillPatch() 20 0.0001999 0.0001999 0.0001999 0.05% MLCellLinOp::apply() 464 0.0001708 0.0001708 0.0001708 0.04% Castro::construct_new_source() 25 0.0001701 0.0001701 0.0001701 0.04% Amr::coarseTimeStep() 5 0.0001562 0.0001562 0.0001562 0.04% MLCellLinOp::defineBC() 6 0.0001484 0.0001484 0.0001484 0.03% MLCGSolver::ParallelAllReduce 798 0.0001387 0.0001387 0.0001387 0.03% Castro::initialize_do_advance() 5 0.0001158 0.0001158 0.0001158 0.03% Castro::advance() 5 0.0001153 0.0001153 0.0001153 0.03% FabArray::ParallelCopy() 380 0.0001095 0.0001095 0.0001095 0.03% FillPatchIterator::Initialize 20 0.000109 0.000109 0.000109 0.03% Castro::subcycle_advance_ctu() 5 0.0001074 0.0001074 0.0001074 0.03% Castro::construct_old_source() 25 0.0001034 0.0001034 0.0001034 0.02% Castro::create_source_corrector() 5 0.0001005 0.0001005 0.0001005 0.02% Castro::post_timestep() 5 0.0001002 0.0001002 0.0001002 0.02% MLLinOp::defineGrids() 6 0.0001001 0.0001001 0.0001001 0.02% Castro::do_new_sources() 5 0.0001001 0.0001001 0.0001001 0.02% Amr::timeStep() 5 9.374e-05 9.374e-05 9.374e-05 0.02% MLMG::mgVcycle() 36 9.337e-05 9.337e-05 9.337e-05 0.02% Castro::finalize_do_advance() 5 8.122e-05 8.122e-05 8.122e-05 0.02% AmrLevel::restart() 1 7.563e-05 7.563e-05 7.563e-05 0.02% MLCellLinOp::correctionResidual() 180 7.138e-05 7.138e-05 7.138e-05 0.02% StateData::restartDoit() 4 7.135e-05 7.135e-05 7.135e-05 0.02% Gravity::update_max_rhs() 6 6.481e-05 6.481e-05 6.481e-05 0.02% Castro::finalize_advance() 5 6.175e-05 6.175e-05 6.175e-05 0.01% Castro::do_advance_ctu() 5 5.956e-05 5.956e-05 5.956e-05 0.01% FabArrayBase::FB::FB() 26 5.91e-05 5.91e-05 5.91e-05 0.01% Castro::do_old_sources() 5 5.815e-05 5.815e-05 5.815e-05 0.01% MLMG:computeResOfCorrection() 180 5.508e-05 5.508e-05 5.508e-05 0.01% Gravity::solve_for_phi() 5 5.245e-05 5.245e-05 5.245e-05 0.01% MLMG::mgVcycle_down::0 36 4.226e-05 4.226e-05 4.226e-05 0.01% Castro::initialize_advance() 5 4.04e-05 4.04e-05 4.04e-05 0.01% MLMG::actualBottomSolve() 36 3.783e-05 3.783e-05 3.783e-05 0.01% MLMG::solve() 6 3.663e-05 3.663e-05 3.663e-05 0.01% MLMG::mgVcycle_down::1 36 3.405e-05 3.405e-05 3.405e-05 0.01% MLMG::mgVcycle_down::2 36 3.292e-05 3.292e-05 3.292e-05 0.01% Amr::writeSmallPlotFile() 1 3.229e-05 3.229e-05 3.229e-05 0.01% Castro::clean_state() 30 3.227e-05 3.227e-05 3.227e-05 0.01% MLMG::mgVcycle_down::4 36 3.225e-05 3.225e-05 3.225e-05 0.01% MLMG::mgVcycle_down::3 36 3.092e-05 3.092e-05 3.092e-05 0.01% Castro::post_restart() 1 3.054e-05 3.054e-05 3.054e-05 0.01% Castro::construct_old_gravity() 5 2.873e-05 2.873e-05 2.873e-05 0.01% MLMG::mgVcycle_up::4 36 2.834e-05 2.834e-05 2.834e-05 0.01% Castro::buildMetrics() 1 2.795e-05 2.795e-05 2.795e-05 0.01% Castro::swap_state_time_levels() 5 2.526e-05 2.526e-05 2.526e-05 0.01% MLMG::mgVcycle_up::3 36 2.374e-05 2.374e-05 2.374e-05 0.01% MLCellLinOp::solutionResidual() 42 2.373e-05 2.373e-05 2.373e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.341e-05 2.341e-05 2.341e-05 0.01% Castro::initMFs() 1 2.292e-05 2.292e-05 2.292e-05 0.01% MLMG::mgVcycle_up::0 36 2.232e-05 2.232e-05 2.232e-05 0.01% MLMG::mgVcycle_up::2 36 2.216e-05 2.216e-05 2.216e-05 0.01% MLMG::oneIter() 36 2.194e-05 2.194e-05 2.194e-05 0.01% MLMG::mgVcycle_up::1 36 2.183e-05 2.183e-05 2.183e-05 0.01% MLPoisson::define() 6 1.995e-05 1.995e-05 1.995e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.71e-05 1.71e-05 1.71e-05 0.00% MLMG::ResNormInf() 42 1.662e-05 1.662e-05 1.662e-05 0.00% MLMG::mgVcycle_bottom 36 1.531e-05 1.531e-05 1.531e-05 0.00% FillPatchSingleLevel 20 1.509e-05 1.509e-05 1.509e-05 0.00% makeSFC 30 1.41e-05 1.41e-05 1.41e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.376e-05 1.376e-05 1.376e-05 0.00% MLMG::computeResidual() 36 1.374e-05 1.374e-05 1.374e-05 0.00% Castro::construct_new_gravity() 5 1.346e-05 1.346e-05 1.346e-05 0.00% Amr::initSubcycle() 1 1.037e-05 1.037e-05 1.037e-05 0.00% DistributionMapping::Distribute() 31 9.861e-06 9.861e-06 9.861e-06 0.00% MLPoisson::prepareForSolve() 6 9.132e-06 9.132e-06 9.132e-06 0.00% Gravity::actual_multilevel_solve() 1 8.192e-06 8.192e-06 8.192e-06 0.00% Castro::check_for_nan() 10 7.135e-06 7.135e-06 7.135e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.088e-06 7.088e-06 7.088e-06 0.00% MLLinOp::define() 6 6.685e-06 6.685e-06 6.685e-06 0.00% Castro::apply_source_to_state() 10 6.091e-06 6.091e-06 6.091e-06 0.00% Gravity::swapTimeLevels() 5 4.303e-06 4.303e-06 4.303e-06 0.00% MLMG::computeMLResidual() 6 4.243e-06 4.243e-06 4.243e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.451e-06 3.451e-06 3.451e-06 0.00% Castro::computeNewDt() 5 3.195e-06 3.195e-06 3.195e-06 0.00% MLMG::getGradSolution() 6 3.117e-06 3.117e-06 3.117e-06 0.00% Castro::expand_state() 5 2.978e-06 2.978e-06 2.978e-06 0.00% MLMG::MLRhsNormInf() 6 2.404e-06 2.404e-06 2.404e-06 0.00% Gravity::set_mass_offset() 6 2.385e-06 2.385e-06 2.385e-06 0.00% MLMG::MLResNormInf() 6 2.323e-06 2.323e-06 2.323e-06 0.00% Castro::retry_advance_ctu() 5 2.144e-06 2.144e-06 2.144e-06 0.00% Castro::FluxRegCrseInit 5 1.82e-06 1.82e-06 1.82e-06 0.00% Castro::FluxRegFineAdd() 5 1.315e-06 1.315e-06 1.315e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.014e-06 1.014e-06 1.014e-06 0.00% Amr::init() 1 8.66e-07 8.66e-07 8.66e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4266 0.4266 0.4266 100.00% Amr::coarseTimeStep() 5 0.328 0.328 0.328 76.87% Amr::timeStep() 5 0.3262 0.3262 0.3262 76.46% Castro::advance() 5 0.3221 0.3221 0.3221 75.50% Castro::subcycle_advance_ctu() 5 0.3149 0.3149 0.3149 73.81% Castro::do_advance_ctu() 5 0.3148 0.3148 0.3148 73.78% Castro::construct_ctu_hydro_source() 5 0.1454 0.1454 0.1454 34.07% Castro::construct_new_gravity() 5 0.1322 0.1322 0.1322 30.99% Gravity::solve_phi_with_mlmg() 6 0.1294 0.1294 0.1294 30.34% Gravity::actual_solve_with_mlmg() 6 0.1249 0.1249 0.1249 29.27% Gravity::solve_for_phi() 5 0.1242 0.1242 0.1242 29.12% MLMG::solve() 6 0.113 0.113 0.113 26.50% MLMG::oneIter() 36 0.1055 0.1055 0.1055 24.72% MLMG::mgVcycle() 36 0.1039 0.1039 0.1039 24.36% Amr::init() 1 0.07117 0.07117 0.07117 16.68% Amr::restart() 1 0.07117 0.07117 0.07117 16.68% AmrLevel::restart() 1 0.06034 0.06034 0.06034 14.14% StateData::restartDoit() 4 0.06026 0.06026 0.06026 14.13% VisMF::Read() 3 0.05998 0.05998 0.05998 14.06% MLCellLinOp::smooth() 720 0.0521 0.0521 0.0521 12.21% MLCellLinOp::applyBC() 1910 0.04862 0.04862 0.04862 11.40% MLMG::mgVcycle_bottom 36 0.03072 0.03072 0.03072 7.20% MLMG::actualBottomSolve() 36 0.03071 0.03071 0.03071 7.20% MLCGSolver::bicgstab 36 0.03035 0.03035 0.03035 7.11% Amr::writePlotFile() 1 0.02671 0.02671 0.02671 6.26% Castro::clean_state() 30 0.02469 0.02469 0.02469 5.79% VisMF::Write(FabArray) 1 0.02386 0.02386 0.02386 5.59% AmrLevel::FillPatch() 20 0.02326 0.02326 0.02326 5.45% FillPatchIterator::Initialize 20 0.02115 0.02115 0.02115 4.96% FillPatchIterator::FillFromLevel0() 20 0.02033 0.02033 0.02033 4.77% FillPatchSingleLevel 20 0.02031 0.02031 0.02031 4.76% StateDataPhysBCFunct::() 20 0.01825 0.01825 0.01825 4.28% MLCellLinOp::apply() 464 0.01568 0.01568 0.01568 3.67% MLMG::mgVcycle_down::0 36 0.01498 0.01498 0.01498 3.51% MLPoisson::Fsmooth() 1440 0.01443 0.01443 0.01443 3.38% FabArray::FillBoundary() 1730 0.01375 0.01375 0.01375 3.22% FillBoundary_nowait() 1730 0.01339 0.01339 0.01339 3.14% StateData::FillBoundary(geom) 160 0.01256 0.01256 0.01256 2.94% MLMG::mgVcycle_up::0 36 0.01124 0.01124 0.01124 2.64% Castro::initialize_do_advance() 5 0.01123 0.01123 0.01123 2.63% Castro::computeTemp() 30 0.01014 0.01014 0.01014 2.38% Castro::do_old_sources() 5 0.009547 0.009547 0.009547 2.24% MLPoisson::define() 6 0.00927 0.00927 0.00927 2.17% amrex::Dot() 484 0.008947 0.008947 0.008947 2.10% MLMG:computeResOfCorrection() 180 0.00875 0.00875 0.00875 2.05% MLCellLinOp::correctionResidual() 180 0.008695 0.008695 0.008695 2.04% Castro::normalize_species() 30 0.008664 0.008664 0.008664 2.03% FabArray::norminf() 465 0.008448 0.008448 0.008448 1.98% Gravity::get_new_grav_vector() 5 0.007823 0.007823 0.007823 1.83% Castro::construct_old_gravity() 5 0.007774 0.007774 0.007774 1.82% Gravity::get_old_grav_vector() 5 0.007746 0.007746 0.007746 1.82% MLMG::mgVcycle_down::1 36 0.007205 0.007205 0.007205 1.69% Castro::initialize_advance() 5 0.00677 0.00677 0.00677 1.59% Castro::do_new_sources() 5 0.006758 0.006758 0.006758 1.58% MLMG::mgVcycle_down::2 36 0.006607 0.006607 0.006607 1.55% FabArray::ParallelCopy() 380 0.006574 0.006574 0.006574 1.54% FabArray::setVal() 501 0.006487 0.006487 0.006487 1.52% FabArray::ParallelCopy_nowait() 380 0.006465 0.006465 0.006465 1.52% MLMG::mgVcycle_down::3 36 0.006432 0.006432 0.006432 1.51% MLMG::mgVcycle_down::4 36 0.006369 0.006369 0.006369 1.49% Castro::expand_state() 5 0.006292 0.006292 0.006292 1.47% MLCellLinOp::defineAuxData() 6 0.006262 0.006262 0.006262 1.47% Castro::post_restart() 1 0.005784 0.005784 0.005784 1.36% FabArray::Saxpy() 597 0.005728 0.005728 0.005728 1.34% Castro::enforce_min_density() 30 0.005432 0.005432 0.005432 1.27% MLCGSolver::ParallelAllReduce 798 0.005425 0.005425 0.005425 1.27% Gravity::multilevel_solve_for_new_phi() 1 0.005413 0.005413 0.005413 1.27% Gravity::actual_multilevel_solve() 1 0.005396 0.005396 0.005396 1.26% amrex::Copy() 221 0.005387 0.005387 0.005387 1.26% MLMG::addInterpCorrection() 180 0.005329 0.005329 0.005329 1.25% MLMG::mgVcycle_up::1 36 0.005151 0.005151 0.005151 1.21% MLMG::mgVcycle_up::4 36 0.005134 0.005134 0.005134 1.20% MLMG::mgVcycle_up::2 36 0.005041 0.005041 0.005041 1.18% amrex::average_down 180 0.004978 0.004978 0.004978 1.17% MLMG::mgVcycle_up::3 36 0.004933 0.004933 0.004933 1.16% Gravity::fill_multipole_BCs() 6 0.004416 0.004416 0.004416 1.04% MLPoisson::Fapply() 464 0.00438 0.00438 0.00438 1.03% Castro::post_timestep() 5 0.004016 0.004016 0.004016 0.94% MLCellLinOp::solutionResidual() 42 0.00356 0.00356 0.00356 0.83% FabArray::Xpay() 325 0.00341 0.00341 0.00341 0.80% Castro::estTimeStep() 10 0.00288 0.00288 0.00288 0.68% MLCellLinOp::defineBC() 6 0.00285 0.00285 0.00285 0.67% MLMG::prepareForSolve() 6 0.002845 0.002845 0.002845 0.67% MLMG::computeResidual() 36 0.002776 0.002776 0.002776 0.65% Castro::reset_internal_energy(MultiFab) 30 0.002704 0.002704 0.002704 0.63% BndryData::define() 6 0.002701 0.002701 0.002701 0.63% Castro::construct_new_source() 25 0.002139 0.002139 0.002139 0.50% Castro::construct_new_gravity_source() 5 0.001969 0.001969 0.001969 0.46% Castro::construct_old_source() 25 0.001684 0.001684 0.001684 0.39% Castro::computeNewDt() 5 0.001603 0.001603 0.001603 0.38% Castro::construct_old_gravity_source() 5 0.00158 0.00158 0.00158 0.37% amrex::Add() 36 0.001545 0.001545 0.001545 0.36% Castro::finalize_do_advance() 5 0.001362 0.001362 0.001362 0.32% MLMG::ResNormInf() 42 0.0009754 0.0009754 0.0009754 0.23% Castro::apply_source_to_state() 10 0.00097 0.00097 0.00097 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009573 0.0009573 0.0009573 0.22% Castro::reset_internal_energy(Fab) 240 0.0009148 0.0009148 0.0009148 0.21% MLMG::getGradSolution() 6 0.0008756 0.0008756 0.0008756 0.21% MLCellLinOp::compGrad() 6 0.0008725 0.0008725 0.0008725 0.20% MLCellLinOp::setLevelBC() 6 0.000864 0.000864 0.000864 0.20% check_for_negative_density() 5 0.0008426 0.0008426 0.0008426 0.20% MLMG::computeMLResidual() 6 0.0008025 0.0008025 0.0008025 0.19% FabArrayBase::getCPC() 632 0.0007974 0.0007974 0.0007974 0.19% MLPoisson::prepareForSolve() 6 0.0007634 0.0007634 0.0007634 0.18% MLCellLinOp::prepareForSolve() 6 0.0007542 0.0007542 0.0007542 0.18% FabArray::setDomainBndry() 20 0.0007149 0.0007149 0.0007149 0.17% Gravity::update_max_rhs() 6 0.0007064 0.0007064 0.0007064 0.17% FabArray::mult() 22 0.0006933 0.0006933 0.0006933 0.16% Castro::check_for_nan() 10 0.0006664 0.0006664 0.0006664 0.16% MultiFab::contains_nan() 10 0.0006593 0.0006593 0.0006593 0.15% Amr::InitAmr() 1 0.000473 0.000473 0.000473 0.11% FabArrayBase::CPC::define() 244 0.0004267 0.0004267 0.0004267 0.10% Castro::enforce_speed_limit() 30 0.0004188 0.0004188 0.0004188 0.10% Castro::finalize_advance() 5 0.0003266 0.0003266 0.0003266 0.08% FabArrayBase::getFB() 1730 0.0003169 0.0003169 0.0003169 0.07% Gravity::swapTimeLevels() 5 0.0002439 0.0002439 0.0002439 0.06% MultiFab::max() 6 0.0002036 0.0002036 0.0002036 0.05% MLMG::MLResNormInf() 6 0.0001914 0.0001914 0.0001914 0.04% Castro::buildMetrics() 1 0.0001534 0.0001534 0.0001534 0.04% MLLinOp::define() 6 0.0001376 0.0001376 0.0001376 0.03% MLLinOp::defineGrids() 6 0.0001309 0.0001309 0.0001309 0.03% MLMG::MLRhsNormInf() 6 0.0001256 0.0001256 0.0001256 0.03% Castro::create_source_corrector() 5 0.0001005 0.0001005 0.0001005 0.02% FabArrayBase::FB::FB() 26 5.91e-05 5.91e-05 5.91e-05 0.01% Amr::writeSmallPlotFile() 1 3.229e-05 3.229e-05 3.229e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.976e-05 2.976e-05 2.976e-05 0.01% Castro::swap_state_time_levels() 5 2.526e-05 2.526e-05 2.526e-05 0.01% Castro::initMFs() 1 2.292e-05 2.292e-05 2.292e-05 0.01% makeSFC 30 2.267e-05 2.267e-05 2.267e-05 0.01% Amr::initSubcycle() 1 1.037e-05 1.037e-05 1.037e-05 0.00% DistributionMapping::Distribute() 31 9.861e-06 9.861e-06 9.861e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.738e-06 4.738e-06 4.738e-06 0.00% Gravity::set_mass_offset() 6 2.385e-06 2.385e-06 2.385e-06 0.00% Castro::retry_advance_ctu() 5 2.144e-06 2.144e-06 2.144e-06 0.00% Castro::FluxRegCrseInit 5 1.82e-06 1.82e-06 1.82e-06 0.00% Castro::FluxRegFineAdd() 5 1.315e-06 1.315e-06 1.315e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.014e-06 1.014e-06 1.014e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 11 MiB 9037 MiB Castro::initMFs() 48 48 57 MiB 68 MiB Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB StateData::restartDoit() 32 32 52 MiB 55 MiB FillPatchIterator::Initialize 160 160 1102 KiB 39 MiB Castro::initialize_do_advance() 40 40 28 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1658 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6407 KiB 14 MiB MLMG::prepareForSolve() 361 361 3251 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 188 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 187 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6394 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 16 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3289 B 2048 KiB Gravity::solve_for_phi() 40 40 595 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 25 KiB 2048 KiB BndryData::define() 576 576 300 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 192 KiB 671 KiB Castro::estTimeStep() 10 10 3171 B 480 KiB VisMF::Write(FabArray) 112 112 1268 B 320 KiB Castro::normalize_species() 30 30 6608 B 320 KiB amrex::average_down 469 469 1371 B 257 KiB MLMG::addInterpCorrection() 468 468 1070 B 257 KiB amrex::Dot() 592 592 3096 B 160 KiB FabArray::norminf() 501 501 3055 B 160 KiB check_for_negative_density() 5 5 316 B 160 KiB MultiFab::max() 6 6 74 B 160 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MultiFab::contains_nan() 10 10 30 B 20 KiB MLPoisson::Fsmooth() 60 60 3150 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 45 B 10 KiB FillBoundary_nowait() 336 336 266 B 9648 B MLCellLinOp::applyBC() 3820 3820 204 B 9344 B amrex::Copy() 56 56 5800 B 8816 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B StateData::FillBoundary(geom) 960 960 48 B 3360 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCellLinOp::defineBC() 36 36 337 B 1248 B MLCGSolver::bicgstab 180 180 85 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1230 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 86 KiB 8192 KiB VisMF::Write(FabArray) 120 120 155 KiB 3584 KiB VisMF::Read() 24 24 213 KiB 3000 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MLPoisson::Fsmooth() 60 60 3150 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 45 B 10 KiB FillBoundary_nowait() 336 336 266 B 9648 B MLCellLinOp::applyBC() 1910 1910 203 B 9328 B amrex::Copy() 56 56 5800 B 8816 B MLCellLinOp::prepareForSolve() 36 36 5 B 7792 B StateData::FillBoundary(geom) 960 960 49 B 3360 B Gravity::get_old_grav_vector() 3 3 2504 B 3072 B Gravity::fill_multipole_BCs() 18 18 5 B 2832 B MLMG::prepareForSolve() 7 7 791 B 1648 B amrex::average_down 37 37 453 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 22 B 400 B FabArray::norminf() 501 501 8 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2180 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (v2024-35-gb892132f6144) finalized