Initializing AMReX (24.05-2-gee11254ffc7c)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.05-2-gee11254ffc7c) initialized Starting run at 08:09:05 UTC on 2024-05-03. Successfully read inputs file ... Castro git describe: 24.05 AMReX git describe: 24.05-2-gee11254ff Microphysics git describe: 24.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.04578236 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025130662 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.068291764 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.069063374 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.079105219 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.065789297 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.071374512 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.044734555 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.053116577 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.071858707 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.072311011 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.068524277 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.067973074 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.042725127 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.02466254 seconds Ending run at 08:09:06 UTC on 2024-05-03. Run time = 0.926879082 Run time without initialization = 0.800294537 Average number of zones advanced per microsecond: 3.276 Average number of zones advanced per microsecond per rank: 3.276 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.9269 ... 0.9269 ... 0.9269 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2949 0.2949 0.2949 31.81% VisMF::Write(FabArray) 11 0.1744 0.1744 0.1744 18.82% MLCellLinOp::applyBC() 4351 0.0888 0.0888 0.0888 9.58% MLPoisson::Fsmooth() 3280 0.03398 0.03398 0.03398 3.67% FillBoundary_nowait() 3941 0.0309 0.0309 0.0309 3.33% StateData::FillBoundary(geom) 328 0.02683 0.02683 0.02683 2.89% amrex::Dot() 1114 0.02199 0.02199 0.02199 2.37% Castro::normalize_species() 62 0.0215 0.0215 0.0215 2.32% FabArray::norminf() 1061 0.02028 0.02028 0.02028 2.19% Castro::computeTemp() 63 0.01796 0.01796 0.01796 1.94% FabArray::ParallelCopy_nowait() 861 0.01391 0.01391 0.01391 1.50% FabArray::setVal() 1062 0.01365 0.01365 0.01365 1.47% FabArray::Saxpy() 1370 0.01338 0.01338 0.01338 1.44% Castro::enforce_min_density() 62 0.01197 0.01197 0.01197 1.29% StateDataPhysBCFunct::() 41 0.01177 0.01177 0.01177 1.27% amrex::Copy() 472 0.01103 0.01103 0.01103 1.19% MLCellLinOp::defineAuxData() 11 0.01068 0.01068 0.01068 1.15% MLPoisson::Fapply() 1060 0.01054 0.01054 0.01054 1.14% Gravity::fill_multipole_BCs() 11 0.009646 0.009646 0.009646 1.04% FabArray::Xpay() 739 0.007983 0.007983 0.007983 0.86% MLMG::addInterpCorrection() 410 0.007124 0.007124 0.007124 0.77% Castro::estTimeStep() 21 0.006563 0.006563 0.006563 0.71% amrex::average_down 410 0.006204 0.006204 0.006204 0.67% Amr::checkPoint() 3 0.006039 0.006039 0.006039 0.65% Castro::reset_internal_energy(MultiFab) 63 0.004961 0.004961 0.004961 0.54% BndryData::define() 11 0.004104 0.004104 0.004104 0.44% amrex::Add() 82 0.003621 0.003621 0.003621 0.39% Castro::construct_new_gravity_source() 10 0.003166 0.003166 0.003166 0.34% Castro::enforce_speed_limit() 62 0.0025 0.0025 0.0025 0.27% Castro::construct_old_gravity_source() 10 0.00241 0.00241 0.00241 0.26% Amr::writePlotFile() 2 0.002148 0.002148 0.002148 0.23% Castro::reset_internal_energy(Fab) 504 0.001915 0.001915 0.001915 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001858 0.001858 0.001858 0.20% check_for_negative_density() 10 0.001779 0.001779 0.001779 0.19% MLCellLinOp::setLevelBC() 11 0.001611 0.001611 0.001611 0.17% Castro::initData() 1 0.00161 0.00161 0.00161 0.17% MLCGSolver::bicgstab 82 0.001573 0.001573 0.001573 0.17% Gravity::actual_solve_with_mlmg() 11 0.001536 0.001536 0.001536 0.17% FabArray::mult() 43 0.00139 0.00139 0.00139 0.15% FabArray::setDomainBndry() 41 0.001385 0.001385 0.001385 0.15% MLCellLinOp::prepareForSolve() 11 0.001318 0.001318 0.001318 0.14% MultiFab::contains_nan() 20 0.001278 0.001278 0.001278 0.14% MLCellLinOp::smooth() 1640 0.001119 0.001119 0.001119 0.12% MLCellLinOp::compGrad() 11 0.001076 0.001076 0.001076 0.12% MLMG::prepareForSolve() 11 0.0009585 0.0009585 0.0009585 0.10% FabArrayBase::getCPC() 1323 0.0007852 0.0007852 0.0007852 0.08% FabArray::FillBoundary() 3941 0.0007429 0.0007429 0.0007429 0.08% Gravity::get_new_grav_vector() 11 0.0006165 0.0006165 0.0006165 0.07% Gravity::get_old_grav_vector() 10 0.0004802 0.0004802 0.0004802 0.05% MLCellLinOp::apply() 1060 0.0004449 0.0004449 0.0004449 0.05% Amr::coarseTimeStep() 10 0.0004319 0.0004319 0.0004319 0.05% AmrLevel::FillPatch() 41 0.0003987 0.0003987 0.0003987 0.04% MLCGSolver::ParallelAllReduce 1832 0.0003228 0.0003228 0.0003228 0.03% main() 1 0.0003117 0.0003117 0.0003117 0.03% FabArray::ParallelCopy() 861 0.0002702 0.0002702 0.0002702 0.03% MLCellLinOp::defineBC() 11 0.0002611 0.0002611 0.0002611 0.03% FillPatchIterator::Initialize 41 0.0002428 0.0002428 0.0002428 0.03% Castro::subcycle_advance_ctu() 10 0.0001865 0.0001865 0.0001865 0.02% MLMG::mgVcycle() 82 0.0001801 0.0001801 0.0001801 0.02% MLCellLinOp::correctionResidual() 410 0.0001691 0.0001691 0.0001691 0.02% Amr::timeStep() 10 0.0001592 0.0001592 0.0001592 0.02% Castro::construct_new_source() 50 0.0001298 0.0001298 0.0001298 0.01% Castro::advance() 10 0.0001172 0.0001172 0.0001172 0.01% MLMG:computeResOfCorrection() 410 0.0001161 0.0001161 0.0001161 0.01% StateData::checkPoint() 12 0.0001119 0.0001119 0.0001119 0.01% Gravity::solve_for_phi() 10 0.0001063 0.0001063 0.0001063 0.01% MLMG::actualBottomSolve() 82 7.921e-05 7.921e-05 7.921e-05 0.01% Castro::initialize_advance() 10 7.631e-05 7.631e-05 7.631e-05 0.01% MLMG::mgVcycle_down::0 82 7.596e-05 7.596e-05 7.596e-05 0.01% MLMG::solve() 11 7.505e-05 7.505e-05 7.505e-05 0.01% Castro::clean_state() 62 7.198e-05 7.198e-05 7.198e-05 0.01% MLMG::mgVcycle_down::1 82 6.858e-05 6.858e-05 6.858e-05 0.01% MLMG::mgVcycle_down::2 82 6.571e-05 6.571e-05 6.571e-05 0.01% MLMG::mgVcycle_down::4 82 6.243e-05 6.243e-05 6.243e-05 0.01% AmrLevel::checkPoint() 3 6.189e-05 6.189e-05 6.189e-05 0.01% MLMG::mgVcycle_down::3 82 5.976e-05 5.976e-05 5.976e-05 0.01% Castro::initialize_do_advance() 10 5.906e-05 5.906e-05 5.906e-05 0.01% MLMG::oneIter() 82 5.54e-05 5.54e-05 5.54e-05 0.01% MLMG::mgVcycle_up::4 82 5.085e-05 5.085e-05 5.085e-05 0.01% MLMG::mgVcycle_up::1 82 5.058e-05 5.058e-05 5.058e-05 0.01% FillPatchIterator::FillFromLevel0() 41 4.965e-05 4.965e-05 4.965e-05 0.01% MLMG::mgVcycle_up::0 82 4.903e-05 4.903e-05 4.903e-05 0.01% Castro::do_advance_ctu() 10 4.868e-05 4.868e-05 4.868e-05 0.01% Castro::finalize_do_advance() 10 4.705e-05 4.705e-05 4.705e-05 0.01% MLMG::mgVcycle_up::3 82 4.558e-05 4.558e-05 4.558e-05 0.00% MLCellLinOp::solutionResidual() 93 4.496e-05 4.496e-05 4.496e-05 0.00% MLMG::mgVcycle_up::2 82 4.325e-05 4.325e-05 4.325e-05 0.00% Castro::post_timestep() 10 3.989e-05 3.989e-05 3.989e-05 0.00% FillPatchSingleLevel 41 3.658e-05 3.658e-05 3.658e-05 0.00% MLMG::computeResidual() 82 3.358e-05 3.358e-05 3.358e-05 0.00% MLMG::ResNormInf() 93 3.26e-05 3.26e-05 3.26e-05 0.00% MLMG::mgVcycle_bottom 82 3.18e-05 3.18e-05 3.18e-05 0.00% Amr::defBaseLevel() 1 3.144e-05 3.144e-05 3.144e-05 0.00% Castro::construct_new_gravity() 10 2.673e-05 2.673e-05 2.673e-05 0.00% MLPoisson::define() 11 2.467e-05 2.467e-05 2.467e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.209e-05 2.209e-05 2.209e-05 0.00% Castro::do_new_sources() 10 1.989e-05 1.989e-05 1.989e-05 0.00% Castro::do_old_sources() 10 1.941e-05 1.941e-05 1.941e-05 0.00% Castro::construct_old_source() 50 1.874e-05 1.874e-05 1.874e-05 0.00% Amr::FinalizeInit() 1 1.751e-05 1.751e-05 1.751e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.741e-05 1.741e-05 1.741e-05 0.00% Gravity::actual_multilevel_solve() 1 1.462e-05 1.462e-05 1.462e-05 0.00% Castro::apply_source_to_state() 20 1.364e-05 1.364e-05 1.364e-05 0.00% Castro::check_for_nan() 20 1.101e-05 1.101e-05 1.101e-05 0.00% Castro::construct_old_gravity() 10 9.962e-06 9.962e-06 9.962e-06 0.00% MLMG::computeMLResidual() 11 9.808e-06 9.808e-06 9.808e-06 0.00% Castro::computeNewDt() 9 7.656e-06 7.656e-06 7.656e-06 0.00% MLMG::getGradSolution() 11 6.604e-06 6.604e-06 6.604e-06 0.00% MLPoisson::prepareForSolve() 11 6.351e-06 6.351e-06 6.351e-06 0.00% Amr::InitializeInit() 1 5.736e-06 5.736e-06 5.736e-06 0.00% Castro::expand_state() 10 5.602e-06 5.602e-06 5.602e-06 0.00% Castro::post_init() 1 4.271e-06 4.271e-06 4.271e-06 0.00% Amr::init() 1 2.775e-06 2.775e-06 2.775e-06 0.00% Amr::initialInit() 1 1.362e-06 1.362e-06 1.362e-06 0.00% Castro::post_regrid() 1 1.242e-06 1.242e-06 1.242e-06 0.00% Other 4752 0.003254 0.003254 0.003254 0.35% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9269 0.9269 0.9269 100.00% Amr::coarseTimeStep() 10 0.7754 0.7754 0.7754 83.65% Amr::timeStep() 10 0.6837 0.6837 0.6837 73.76% Castro::advance() 10 0.6719 0.6719 0.6719 72.49% Castro::subcycle_advance_ctu() 10 0.6569 0.6569 0.6569 70.87% Castro::do_advance_ctu() 10 0.6567 0.6567 0.6567 70.85% Castro::construct_ctu_hydro_source() 10 0.3058 0.3058 0.3058 32.99% Gravity::solve_phi_with_mlmg() 11 0.3057 0.3057 0.3057 32.98% Gravity::actual_solve_with_mlmg() 11 0.2956 0.2956 0.2956 31.89% Castro::construct_new_gravity() 10 0.2767 0.2767 0.2767 29.86% MLMG::solve() 11 0.273 0.273 0.273 29.45% Gravity::solve_for_phi() 10 0.2603 0.2603 0.2603 28.08% MLMG::oneIter() 82 0.2574 0.2574 0.2574 27.77% MLMG::mgVcycle() 82 0.2537 0.2537 0.2537 27.37% VisMF::Write(FabArray) 11 0.1744 0.1744 0.1744 18.82% Amr::checkPoint() 3 0.1334 0.1334 0.1334 14.39% MLCellLinOp::smooth() 1640 0.1298 0.1298 0.1298 14.01% AmrLevel::checkPoint() 3 0.1273 0.1273 0.1273 13.74% StateData::checkPoint() 12 0.1273 0.1273 0.1273 13.73% Amr::init() 1 0.1259 0.1259 0.1259 13.58% MLCellLinOp::applyBC() 4351 0.1211 0.1211 0.1211 13.07% MLMG::mgVcycle_bottom 82 0.07426 0.07426 0.07426 8.01% MLMG::actualBottomSolve() 82 0.07423 0.07423 0.07423 8.01% MLCGSolver::bicgstab 82 0.07341 0.07341 0.07341 7.92% Castro::clean_state() 62 0.05999 0.05999 0.05999 6.47% Amr::initialInit() 1 0.05483 0.05483 0.05483 5.92% Amr::writePlotFile() 2 0.04993 0.04993 0.04993 5.39% Amr::FinalizeInit() 1 0.04976 0.04976 0.04976 5.37% AmrLevel::FillPatch() 41 0.04874 0.04874 0.04874 5.26% Castro::post_init() 1 0.04821 0.04821 0.04821 5.20% Gravity::multilevel_solve_for_new_phi() 1 0.04586 0.04586 0.04586 4.95% Gravity::actual_multilevel_solve() 1 0.04584 0.04584 0.04584 4.95% FillPatchIterator::Initialize 41 0.04443 0.04443 0.04443 4.79% FillPatchIterator::FillFromLevel0() 41 0.0428 0.0428 0.0428 4.62% FillPatchSingleLevel 41 0.04275 0.04275 0.04275 4.61% StateDataPhysBCFunct::() 41 0.0386 0.0386 0.0386 4.16% MLCellLinOp::apply() 1060 0.03684 0.03684 0.03684 3.97% MLMG::mgVcycle_down::0 82 0.03497 0.03497 0.03497 3.77% MLPoisson::Fsmooth() 3280 0.03398 0.03398 0.03398 3.67% FabArray::FillBoundary() 3941 0.03232 0.03232 0.03232 3.49% FillBoundary_nowait() 3941 0.03158 0.03158 0.03158 3.41% StateData::FillBoundary(geom) 328 0.02683 0.02683 0.02683 2.89% MLMG::mgVcycle_up::0 82 0.02642 0.02642 0.02642 2.85% Castro::computeTemp() 63 0.02484 0.02484 0.02484 2.68% Castro::initialize_do_advance() 10 0.02225 0.02225 0.02225 2.40% amrex::Dot() 1114 0.02199 0.02199 0.02199 2.37% Castro::normalize_species() 62 0.0215 0.0215 0.0215 2.32% MLMG:computeResOfCorrection() 410 0.02071 0.02071 0.02071 2.23% MLCellLinOp::correctionResidual() 410 0.02059 0.02059 0.02059 2.22% FabArray::norminf() 1061 0.02028 0.02028 0.02028 2.19% MLMG::mgVcycle_up::2 82 0.01897 0.01897 0.01897 2.05% Castro::do_old_sources() 10 0.01866 0.01866 0.01866 2.01% Gravity::get_new_grav_vector() 11 0.01838 0.01838 0.01838 1.98% MLPoisson::define() 11 0.01781 0.01781 0.01781 1.92% MLMG::mgVcycle_down::1 82 0.01684 0.01684 0.01684 1.82% Castro::construct_old_gravity() 10 0.01593 0.01593 0.01593 1.72% Gravity::get_old_grav_vector() 10 0.01592 0.01592 0.01592 1.72% MLMG::mgVcycle_down::2 82 0.01564 0.01564 0.01564 1.69% MLMG::mgVcycle_down::3 82 0.01526 0.01526 0.01526 1.65% MLMG::mgVcycle_down::4 82 0.01516 0.01516 0.01516 1.64% FabArray::ParallelCopy() 861 0.01499 0.01499 0.01499 1.62% FabArray::ParallelCopy_nowait() 861 0.01472 0.01472 0.01472 1.59% Castro::initialize_advance() 10 0.01425 0.01425 0.01425 1.54% Castro::do_new_sources() 10 0.01374 0.01374 0.01374 1.48% FabArray::setVal() 1062 0.01365 0.01365 0.01365 1.47% FabArray::Saxpy() 1370 0.01338 0.01338 0.01338 1.44% MLCGSolver::ParallelAllReduce 1832 0.01314 0.01314 0.01314 1.42% MLMG::addInterpCorrection() 410 0.01255 0.01255 0.01255 1.35% MLMG::mgVcycle_up::1 82 0.01224 0.01224 0.01224 1.32% MLCellLinOp::defineAuxData() 11 0.01214 0.01214 0.01214 1.31% MLMG::mgVcycle_up::4 82 0.01211 0.01211 0.01211 1.31% Castro::enforce_min_density() 62 0.01197 0.01197 0.01197 1.29% Castro::expand_state() 10 0.01179 0.01179 0.01179 1.27% MLMG::mgVcycle_up::3 82 0.01168 0.01168 0.01168 1.26% amrex::average_down 410 0.01165 0.01165 0.01165 1.26% Castro::post_timestep() 10 0.01161 0.01161 0.01161 1.25% amrex::Copy() 472 0.01103 0.01103 0.01103 1.19% MLPoisson::Fapply() 1060 0.01054 0.01054 0.01054 1.14% Gravity::fill_multipole_BCs() 11 0.009885 0.009885 0.009885 1.07% FabArray::Xpay() 739 0.007983 0.007983 0.007983 0.86% MLCellLinOp::solutionResidual() 93 0.007844 0.007844 0.007844 0.85% Castro::reset_internal_energy(MultiFab) 63 0.006876 0.006876 0.006876 0.74% MLMG::computeResidual() 82 0.006597 0.006597 0.006597 0.71% Castro::estTimeStep() 21 0.006563 0.006563 0.006563 0.71% MLCellLinOp::defineBC() 11 0.005413 0.005413 0.005413 0.58% MLMG::prepareForSolve() 11 0.005205 0.005205 0.005205 0.56% BndryData::define() 11 0.005152 0.005152 0.005152 0.56% Amr::InitializeInit() 1 0.005067 0.005067 0.005067 0.55% Amr::defBaseLevel() 1 0.005061 0.005061 0.005061 0.55% Castro::initData() 1 0.004367 0.004367 0.004367 0.47% amrex::Add() 82 0.003621 0.003621 0.003621 0.39% Castro::construct_new_source() 50 0.003296 0.003296 0.003296 0.36% Castro::construct_new_gravity_source() 10 0.003166 0.003166 0.003166 0.34% Castro::computeNewDt() 9 0.002936 0.002936 0.002936 0.32% Castro::finalize_do_advance() 10 0.002636 0.002636 0.002636 0.28% Castro::enforce_speed_limit() 62 0.0025 0.0025 0.0025 0.27% Castro::construct_old_source() 50 0.002429 0.002429 0.002429 0.26% Castro::construct_old_gravity_source() 10 0.00241 0.00241 0.00241 0.26% MLMG::ResNormInf() 93 0.002199 0.002199 0.002199 0.24% Castro::reset_internal_energy(Fab) 504 0.001915 0.001915 0.001915 0.21% Castro::apply_source_to_state() 20 0.001884 0.001884 0.001884 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001858 0.001858 0.001858 0.20% check_for_negative_density() 10 0.001779 0.001779 0.001779 0.19% MLCellLinOp::setLevelBC() 11 0.001611 0.001611 0.001611 0.17% MLMG::getGradSolution() 11 0.001606 0.001606 0.001606 0.17% MLCellLinOp::compGrad() 11 0.0016 0.0016 0.0016 0.17% FabArrayBase::getCPC() 1323 0.001454 0.001454 0.001454 0.16% FabArray::mult() 43 0.00139 0.00139 0.00139 0.15% FabArray::setDomainBndry() 41 0.001385 0.001385 0.001385 0.15% MLPoisson::prepareForSolve() 11 0.001325 0.001325 0.001325 0.14% MLCellLinOp::prepareForSolve() 11 0.001318 0.001318 0.001318 0.14% MLMG::computeMLResidual() 11 0.00129 0.00129 0.00129 0.14% Castro::check_for_nan() 20 0.001289 0.001289 0.001289 0.14% MultiFab::contains_nan() 20 0.001278 0.001278 0.001278 0.14% Castro::post_regrid() 1 0.001237 0.001237 0.001237 0.13% Other 4752 0.008051 0.008051 0.008051 0.87% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 5680 KiB 9037 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1011 KiB 39 MiB Castro::initialize_do_advance() 80 80 27 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1502 KiB 28 MiB Castro::initialize_advance() 80 80 16 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7618 KiB 14 MiB MLMG::prepareForSolve() 660 660 3623 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 206 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 175 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7518 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 18 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2124 B 2048 KiB Gravity::solve_for_phi() 80 80 574 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 101 KiB 2048 KiB BndryData::define() 1056 1056 331 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 210 KiB 671 KiB Castro::estTimeStep() 21 21 3455 B 480 KiB VisMF::Write(FabArray) 656 656 3355 B 320 KiB Castro::normalize_species() 62 62 7556 B 320 KiB amrex::average_down 1067 1067 1596 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1165 B 257 KiB amrex::Dot() 1360 1360 3499 B 160 KiB FabArray::norminf() 1143 1143 3380 B 160 KiB check_for_negative_density() 10 10 308 B 160 KiB Castro::initData() 1 1 51 B 160 KiB MultiFab::max() 11 11 56 B 160 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MultiFab::contains_nan() 20 20 27 B 20 KiB MLPoisson::Fsmooth() 132 132 3539 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 43 B 10 KiB FillBoundary_nowait() 760 760 290 B 9648 B MLCellLinOp::applyBC() 8702 8702 220 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3949 B 6144 B StateData::FillBoundary(geom) 1992 1992 41 B 2640 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 373 B 1248 B MLCGSolver::bicgstab 410 410 95 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 578 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 39 KiB 8192 KiB VisMF::Write(FabArray) 744 744 410 KiB 3584 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MLPoisson::Fsmooth() 132 132 3539 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 43 B 10 KiB FillBoundary_nowait() 760 760 289 B 9648 B MLCellLinOp::applyBC() 4351 4351 218 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3949 B 6144 B Gravity::get_new_grav_vector() 3 3 2892 B 3072 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B StateData::FillBoundary(geom) 1992 1992 42 B 2640 B amrex::average_down 83 83 617 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 301 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 25 B 400 B FabArray::norminf() 1143 1143 9 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2105 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.05-2-gee11254ffc7c) finalized Initializing AMReX (24.05-2-gee11254ffc7c)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.05-2-gee11254ffc7c) initialized Starting run at 08:09:06 UTC on 2024-05-03. Successfully read inputs file ... Castro git describe: 24.05 AMReX git describe: 24.05-2-gee11254ff Microphysics git describe: 24.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.525031718 Restart time = 0.071520713 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.071576355 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.057443937 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.07840059 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.078703585 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.055622848 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.070230623 seconds Ending run at 08:09:07 UTC on 2024-05-03. Run time = 0.484599279 Run time without initialization = 0.412429666 Average number of zones advanced per microsecond: 3.178 Average number of zones advanced per microsecond per rank: 3.178 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.4846 ... 0.4846 ... 0.4846 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1476 0.1476 0.1476 30.46% VisMF::Read() 3 0.06026 0.06026 0.06026 12.43% Amr::writePlotFile() 1 0.04521 0.04521 0.04521 9.33% MLCellLinOp::applyBC() 1910 0.03601 0.03601 0.03601 7.43% VisMF::Write(FabArray) 1 0.02485 0.02485 0.02485 5.13% MLPoisson::Fsmooth() 1440 0.01505 0.01505 0.01505 3.11% StateData::FillBoundary(geom) 160 0.01325 0.01325 0.01325 2.73% FillBoundary_nowait() 1730 0.01291 0.01291 0.01291 2.66% Castro::normalize_species() 30 0.009517 0.009517 0.009517 1.96% amrex::Dot() 484 0.009362 0.009362 0.009362 1.93% FabArray::norminf() 465 0.00877 0.00877 0.00877 1.81% Castro::computeTemp() 30 0.007573 0.007573 0.007573 1.56% FabArray::setVal() 501 0.006696 0.006696 0.006696 1.38% FabArray::ParallelCopy_nowait() 380 0.006277 0.006277 0.006277 1.30% FabArray::Saxpy() 597 0.006014 0.006014 0.006014 1.24% MLCellLinOp::defineAuxData() 6 0.005845 0.005845 0.005845 1.21% Castro::enforce_min_density() 30 0.005733 0.005733 0.005733 1.18% StateDataPhysBCFunct::() 20 0.005587 0.005587 0.005587 1.15% amrex::Copy() 221 0.005519 0.005519 0.005519 1.14% Gravity::fill_multipole_BCs() 6 0.005029 0.005029 0.005029 1.04% Amr::restart() 1 0.004895 0.004895 0.004895 1.01% MLPoisson::Fapply() 464 0.00455 0.00455 0.00455 0.94% FabArray::Xpay() 325 0.00356 0.00356 0.00356 0.73% MLMG::addInterpCorrection() 180 0.003133 0.003133 0.003133 0.65% amrex::average_down 180 0.002744 0.002744 0.002744 0.57% Castro::estTimeStep() 10 0.002652 0.002652 0.002652 0.55% BndryData::define() 6 0.002269 0.002269 0.002269 0.47% Castro::enforce_speed_limit() 30 0.001974 0.001974 0.001974 0.41% amrex::Add() 36 0.001563 0.001563 0.001563 0.32% Castro::construct_new_gravity_source() 5 0.001483 0.001483 0.001483 0.31% Castro::reset_internal_energy(MultiFab) 30 0.001475 0.001475 0.001475 0.30% Castro::construct_old_gravity_source() 5 0.001206 0.001206 0.001206 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009991 0.0009991 0.0009991 0.21% check_for_negative_density() 5 0.0009745 0.0009745 0.0009745 0.20% MLCellLinOp::setLevelBC() 6 0.0008937 0.0008937 0.0008937 0.18% Gravity::actual_solve_with_mlmg() 6 0.000845 0.000845 0.000845 0.17% Castro::reset_internal_energy(Fab) 240 0.0008238 0.0008238 0.0008238 0.17% MLCellLinOp::prepareForSolve() 6 0.00076 0.00076 0.00076 0.16% FabArray::setDomainBndry() 20 0.0007046 0.0007046 0.0007046 0.15% MLCGSolver::bicgstab 36 0.0007004 0.0007004 0.0007004 0.14% FabArray::mult() 22 0.000696 0.000696 0.000696 0.14% MLCellLinOp::compGrad() 6 0.0006132 0.0006132 0.0006132 0.13% MLMG::prepareForSolve() 6 0.0005591 0.0005591 0.0005591 0.12% MLCellLinOp::smooth() 720 0.0004905 0.0004905 0.0004905 0.10% FabArrayBase::getCPC() 632 0.0003896 0.0003896 0.0003896 0.08% Gravity::get_old_grav_vector() 5 0.0003449 0.0003449 0.0003449 0.07% FabArray::FillBoundary() 1730 0.0003371 0.0003371 0.0003371 0.07% main() 1 0.0002783 0.0002783 0.0002783 0.06% Gravity::get_new_grav_vector() 5 0.0002632 0.0002632 0.0002632 0.05% AmrLevel::FillPatch() 20 0.0002099 0.0002099 0.0002099 0.04% MLCellLinOp::apply() 464 0.0001954 0.0001954 0.0001954 0.04% Amr::coarseTimeStep() 5 0.0001805 0.0001805 0.0001805 0.04% MLCellLinOp::defineBC() 6 0.0001426 0.0001426 0.0001426 0.03% MLCGSolver::ParallelAllReduce 798 0.0001392 0.0001392 0.0001392 0.03% FabArray::ParallelCopy() 380 0.000121 0.000121 0.000121 0.02% FillPatchIterator::Initialize 20 0.0001052 0.0001052 0.0001052 0.02% Castro::subcycle_advance_ctu() 5 0.0001045 0.0001045 0.0001045 0.02% Castro::do_advance_ctu() 5 9.981e-05 9.981e-05 9.981e-05 0.02% Amr::timeStep() 5 8.421e-05 8.421e-05 8.421e-05 0.02% MLMG::mgVcycle() 36 7.825e-05 7.825e-05 7.825e-05 0.02% Castro::advance() 5 7.628e-05 7.628e-05 7.628e-05 0.02% AmrLevel::restart() 1 7.597e-05 7.597e-05 7.597e-05 0.02% Gravity::update_max_rhs() 6 7.484e-05 7.484e-05 7.484e-05 0.02% MLCellLinOp::correctionResidual() 180 7.27e-05 7.27e-05 7.27e-05 0.02% StateData::restartDoit() 4 6.572e-05 6.572e-05 6.572e-05 0.01% Castro::finalize_do_advance() 5 6.268e-05 6.268e-05 6.268e-05 0.01% Castro::construct_new_source() 25 5.719e-05 5.719e-05 5.719e-05 0.01% Castro::construct_old_source() 25 5.447e-05 5.447e-05 5.447e-05 0.01% MLMG:computeResOfCorrection() 180 5.2e-05 5.2e-05 5.2e-05 0.01% Castro::initialize_do_advance() 5 5.051e-05 5.051e-05 5.051e-05 0.01% Gravity::solve_for_phi() 5 4.994e-05 4.994e-05 4.994e-05 0.01% Castro::initialize_advance() 5 3.597e-05 3.597e-05 3.597e-05 0.01% MLMG::solve() 6 3.483e-05 3.483e-05 3.483e-05 0.01% MLMG::actualBottomSolve() 36 3.475e-05 3.475e-05 3.475e-05 0.01% Castro::clean_state() 30 3.369e-05 3.369e-05 3.369e-05 0.01% MLMG::mgVcycle_down::0 36 3.329e-05 3.329e-05 3.329e-05 0.01% MLMG::mgVcycle_down::1 36 3.075e-05 3.075e-05 3.075e-05 0.01% Castro::do_new_sources() 5 2.938e-05 2.938e-05 2.938e-05 0.01% MLMG::mgVcycle_down::2 36 2.807e-05 2.807e-05 2.807e-05 0.01% Castro::post_timestep() 5 2.795e-05 2.795e-05 2.795e-05 0.01% MLMG::mgVcycle_down::4 36 2.711e-05 2.711e-05 2.711e-05 0.01% MLMG::mgVcycle_down::3 36 2.608e-05 2.608e-05 2.608e-05 0.01% Castro::post_restart() 1 2.44e-05 2.44e-05 2.44e-05 0.01% MLMG::oneIter() 36 2.336e-05 2.336e-05 2.336e-05 0.00% MLMG::mgVcycle_up::4 36 2.317e-05 2.317e-05 2.317e-05 0.00% FillPatchIterator::FillFromLevel0() 20 2.172e-05 2.172e-05 2.172e-05 0.00% MLMG::mgVcycle_up::0 36 2.015e-05 2.015e-05 2.015e-05 0.00% MLCellLinOp::solutionResidual() 42 2.002e-05 2.002e-05 2.002e-05 0.00% MLMG::mgVcycle_up::3 36 1.986e-05 1.986e-05 1.986e-05 0.00% MLMG::mgVcycle_up::2 36 1.864e-05 1.864e-05 1.864e-05 0.00% MLMG::mgVcycle_up::1 36 1.828e-05 1.828e-05 1.828e-05 0.00% Gravity::actual_multilevel_solve() 1 1.674e-05 1.674e-05 1.674e-05 0.00% MLPoisson::define() 6 1.667e-05 1.667e-05 1.667e-05 0.00% FillPatchSingleLevel 20 1.65e-05 1.65e-05 1.65e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.587e-05 1.587e-05 1.587e-05 0.00% MLMG::ResNormInf() 42 1.574e-05 1.574e-05 1.574e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.407e-05 1.407e-05 1.407e-05 0.00% MLMG::computeResidual() 36 1.386e-05 1.386e-05 1.386e-05 0.00% MLMG::mgVcycle_bottom 36 1.361e-05 1.361e-05 1.361e-05 0.00% Castro::construct_new_gravity() 5 1.337e-05 1.337e-05 1.337e-05 0.00% Castro::do_old_sources() 5 1.065e-05 1.065e-05 1.065e-05 0.00% Castro::expand_state() 5 8.82e-06 8.82e-06 8.82e-06 0.00% Castro::apply_source_to_state() 10 6.425e-06 6.425e-06 6.425e-06 0.00% Castro::construct_old_gravity() 5 6.068e-06 6.068e-06 6.068e-06 0.00% MLMG::computeMLResidual() 6 3.737e-06 3.737e-06 3.737e-06 0.00% Castro::computeNewDt() 5 3.505e-06 3.505e-06 3.505e-06 0.00% MLMG::getGradSolution() 6 3.504e-06 3.504e-06 3.504e-06 0.00% MLPoisson::prepareForSolve() 6 3.335e-06 3.335e-06 3.335e-06 0.00% Amr::init() 1 8.98e-07 8.98e-07 8.98e-07 0.00% Other 2170 0.002469 0.002469 0.002469 0.51% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4846 0.4846 0.4846 100.00% Amr::coarseTimeStep() 5 0.3419 0.3419 0.3419 70.55% Amr::timeStep() 5 0.3401 0.3401 0.3401 70.18% Castro::advance() 5 0.3344 0.3344 0.3344 69.00% Castro::subcycle_advance_ctu() 5 0.3266 0.3266 0.3266 67.38% Castro::do_advance_ctu() 5 0.3264 0.3264 0.3264 67.36% Castro::construct_ctu_hydro_source() 5 0.1529 0.1529 0.1529 31.55% Castro::construct_new_gravity() 5 0.1372 0.1372 0.1372 28.32% Gravity::solve_phi_with_mlmg() 6 0.1342 0.1342 0.1342 27.70% Gravity::solve_for_phi() 5 0.129 0.129 0.129 26.61% Gravity::actual_solve_with_mlmg() 6 0.129 0.129 0.129 26.61% MLMG::solve() 6 0.1165 0.1165 0.1165 24.04% MLMG::oneIter() 36 0.1087 0.1087 0.1087 22.43% MLMG::mgVcycle() 36 0.1071 0.1071 0.1071 22.10% Amr::init() 1 0.07157 0.07157 0.07157 14.77% Amr::restart() 1 0.07157 0.07157 0.07157 14.77% Amr::writePlotFile() 1 0.07033 0.07033 0.07033 14.51% AmrLevel::restart() 1 0.06063 0.06063 0.06063 12.51% StateData::restartDoit() 4 0.06054 0.06054 0.06054 12.49% VisMF::Read() 3 0.06026 0.06026 0.06026 12.43% MLCellLinOp::smooth() 720 0.05338 0.05338 0.05338 11.01% MLCellLinOp::applyBC() 1910 0.04957 0.04957 0.04957 10.23% MLMG::mgVcycle_bottom 36 0.03208 0.03208 0.03208 6.62% MLMG::actualBottomSolve() 36 0.03207 0.03207 0.03207 6.62% MLCGSolver::bicgstab 36 0.0317 0.0317 0.0317 6.54% Castro::clean_state() 30 0.02713 0.02713 0.02713 5.60% VisMF::Write(FabArray) 1 0.02485 0.02485 0.02485 5.13% AmrLevel::FillPatch() 20 0.02387 0.02387 0.02387 4.93% FillPatchIterator::Initialize 20 0.02175 0.02175 0.02175 4.49% FillPatchIterator::FillFromLevel0() 20 0.02094 0.02094 0.02094 4.32% FillPatchSingleLevel 20 0.02091 0.02091 0.02091 4.32% StateDataPhysBCFunct::() 20 0.01884 0.01884 0.01884 3.89% MLCellLinOp::apply() 464 0.01619 0.01619 0.01619 3.34% MLMG::mgVcycle_down::0 36 0.01514 0.01514 0.01514 3.12% MLPoisson::Fsmooth() 1440 0.01505 0.01505 0.01505 3.11% FabArray::FillBoundary() 1730 0.01355 0.01355 0.01355 2.80% StateData::FillBoundary(geom) 160 0.01325 0.01325 0.01325 2.73% FillBoundary_nowait() 1730 0.01322 0.01322 0.01322 2.73% MLMG::mgVcycle_up::0 36 0.01129 0.01129 0.01129 2.33% Castro::initialize_do_advance() 5 0.0112 0.0112 0.0112 2.31% Castro::computeTemp() 30 0.009871 0.009871 0.009871 2.04% MLPoisson::define() 6 0.009801 0.009801 0.009801 2.02% Castro::normalize_species() 30 0.009517 0.009517 0.009517 1.96% amrex::Dot() 484 0.009362 0.009362 0.009362 1.93% MLMG:computeResOfCorrection() 180 0.008986 0.008986 0.008986 1.85% MLCellLinOp::correctionResidual() 180 0.008934 0.008934 0.008934 1.84% FabArray::norminf() 465 0.00877 0.00877 0.00877 1.81% Castro::do_old_sources() 5 0.008699 0.008699 0.008699 1.80% Castro::construct_old_gravity() 5 0.008186 0.008186 0.008186 1.69% Gravity::get_old_grav_vector() 5 0.00818 0.00818 0.00818 1.69% Gravity::get_new_grav_vector() 5 0.008129 0.008129 0.008129 1.68% Castro::initialize_advance() 5 0.00747 0.00747 0.00747 1.54% MLMG::mgVcycle_down::1 36 0.007458 0.007458 0.007458 1.54% MLMG::mgVcycle_down::2 36 0.006824 0.006824 0.006824 1.41% FabArray::ParallelCopy() 380 0.006798 0.006798 0.006798 1.40% FabArray::setVal() 501 0.006696 0.006696 0.006696 1.38% FabArray::ParallelCopy_nowait() 380 0.006677 0.006677 0.006677 1.38% MLMG::mgVcycle_down::3 36 0.006668 0.006668 0.006668 1.38% MLCellLinOp::defineAuxData() 6 0.006649 0.006649 0.006649 1.37% MLMG::mgVcycle_down::4 36 0.006639 0.006639 0.006639 1.37% Castro::do_new_sources() 5 0.006559 0.006559 0.006559 1.35% FabArray::Saxpy() 597 0.006014 0.006014 0.006014 1.24% Castro::expand_state() 5 0.005993 0.005993 0.005993 1.24% Castro::post_restart() 1 0.005866 0.005866 0.005866 1.21% Castro::enforce_min_density() 30 0.005733 0.005733 0.005733 1.18% MLCGSolver::ParallelAllReduce 798 0.00565 0.00565 0.00565 1.17% Castro::post_timestep() 5 0.005616 0.005616 0.005616 1.16% amrex::Copy() 221 0.005519 0.005519 0.005519 1.14% Gravity::multilevel_solve_for_new_phi() 1 0.005495 0.005495 0.005495 1.13% MLMG::addInterpCorrection() 180 0.005491 0.005491 0.005491 1.13% Gravity::actual_multilevel_solve() 1 0.005479 0.005479 0.005479 1.13% MLMG::mgVcycle_up::4 36 0.00534 0.00534 0.00534 1.10% MLMG::mgVcycle_up::1 36 0.005302 0.005302 0.005302 1.09% MLMG::mgVcycle_up::2 36 0.005182 0.005182 0.005182 1.07% Gravity::fill_multipole_BCs() 6 0.005155 0.005155 0.005155 1.06% MLMG::mgVcycle_up::3 36 0.005128 0.005128 0.005128 1.06% amrex::average_down 180 0.005121 0.005121 0.005121 1.06% MLPoisson::Fapply() 464 0.00455 0.00455 0.00455 0.94% MLCellLinOp::solutionResidual() 42 0.003703 0.003703 0.003703 0.76% FabArray::Xpay() 325 0.00356 0.00356 0.00356 0.73% MLCellLinOp::defineBC() 6 0.003006 0.003006 0.003006 0.62% MLMG::prepareForSolve() 6 0.002914 0.002914 0.002914 0.60% MLMG::computeResidual() 36 0.002894 0.002894 0.002894 0.60% BndryData::define() 6 0.002864 0.002864 0.002864 0.59% Castro::estTimeStep() 10 0.002652 0.002652 0.002652 0.55% Castro::reset_internal_energy(MultiFab) 30 0.002298 0.002298 0.002298 0.47% Castro::enforce_speed_limit() 30 0.001974 0.001974 0.001974 0.41% Castro::computeNewDt() 5 0.00163 0.00163 0.00163 0.34% amrex::Add() 36 0.001563 0.001563 0.001563 0.32% Castro::construct_new_source() 25 0.001541 0.001541 0.001541 0.32% Castro::construct_new_gravity_source() 5 0.001483 0.001483 0.001483 0.31% Castro::construct_old_source() 25 0.001261 0.001261 0.001261 0.26% Castro::construct_old_gravity_source() 5 0.001206 0.001206 0.001206 0.25% Castro::finalize_do_advance() 5 0.001089 0.001089 0.001089 0.22% MLMG::ResNormInf() 42 0.001 0.001 0.001 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009991 0.0009991 0.0009991 0.21% check_for_negative_density() 5 0.0009745 0.0009745 0.0009745 0.20% Castro::apply_source_to_state() 10 0.0009676 0.0009676 0.0009676 0.20% MLMG::getGradSolution() 6 0.0009071 0.0009071 0.0009071 0.19% MLCellLinOp::compGrad() 6 0.0009036 0.0009036 0.0009036 0.19% MLCellLinOp::setLevelBC() 6 0.0008937 0.0008937 0.0008937 0.18% MLMG::computeMLResidual() 6 0.0008264 0.0008264 0.0008264 0.17% Castro::reset_internal_energy(Fab) 240 0.0008238 0.0008238 0.0008238 0.17% FabArrayBase::getCPC() 632 0.0007982 0.0007982 0.0007982 0.16% MLPoisson::prepareForSolve() 6 0.0007633 0.0007633 0.0007633 0.16% MLCellLinOp::prepareForSolve() 6 0.00076 0.00076 0.00076 0.16% Gravity::update_max_rhs() 6 0.0007346 0.0007346 0.0007346 0.15% FabArray::setDomainBndry() 20 0.0007046 0.0007046 0.0007046 0.15% FabArray::mult() 22 0.000696 0.000696 0.000696 0.14% Other 2170 0.004351 0.004351 0.004351 0.90% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 10 MiB 9037 MiB Castro::initMFs() 48 48 58 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::restartDoit() 32 32 52 MiB 55 MiB FillPatchIterator::Initialize 160 160 967 KiB 39 MiB Castro::initialize_do_advance() 40 40 26 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1518 KiB 28 MiB Castro::initialize_advance() 40 40 16 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6544 KiB 14 MiB MLMG::prepareForSolve() 361 361 2951 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 175 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 171 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6532 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 17 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2998 B 2048 KiB Gravity::solve_for_phi() 40 40 544 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 23 KiB 2048 KiB BndryData::define() 576 576 272 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 175 KiB 671 KiB Castro::estTimeStep() 10 10 2559 B 480 KiB VisMF::Write(FabArray) 112 112 1145 B 320 KiB Castro::normalize_species() 30 30 6393 B 320 KiB amrex::average_down 469 469 1308 B 257 KiB MLMG::addInterpCorrection() 468 468 974 B 257 KiB amrex::Dot() 592 592 2847 B 160 KiB FabArray::norminf() 501 501 2797 B 160 KiB check_for_negative_density() 5 5 323 B 160 KiB MultiFab::max() 6 6 66 B 160 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MultiFab::contains_nan() 10 10 27 B 20 KiB MLPoisson::Fsmooth() 60 60 2860 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 40 B 10 KiB FillBoundary_nowait() 336 336 229 B 9648 B MLCellLinOp::applyBC() 3820 3820 185 B 9344 B amrex::Copy() 56 56 5848 B 8816 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B StateData::FillBoundary(geom) 960 960 39 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCellLinOp::defineBC() 36 36 305 B 1248 B MLCGSolver::bicgstab 180 180 78 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1085 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 76 KiB 8192 KiB VisMF::Write(FabArray) 120 120 143 KiB 3584 KiB VisMF::Read() 24 24 189 KiB 3000 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MLPoisson::Fsmooth() 60 60 2860 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 40 B 10 KiB FillBoundary_nowait() 336 336 229 B 9648 B MLCellLinOp::applyBC() 1910 1910 184 B 9328 B amrex::Copy() 56 56 5848 B 8816 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B Gravity::get_old_grav_vector() 3 3 2577 B 3072 B Gravity::fill_multipole_BCs() 18 18 5 B 2832 B StateData::FillBoundary(geom) 960 960 39 B 2832 B MLMG::prepareForSolve() 7 7 778 B 1648 B amrex::average_down 37 37 475 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 21 B 400 B FabArray::norminf() 501 501 8 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2105 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.05-2-gee11254ffc7c) finalized