Initializing AMReX (24.02-25-g2230caa24c7d)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.02-25-g2230caa24c7d) initialized Starting run at 10:33:56 UTC on 2024-02-21. Successfully read inputs file ... Castro git describe: 24.02-21-g56d243e7a AMReX git describe: 24.02-25-g2230caa24 Microphysics git describe: 24.02-22-g4f2ee5fd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.045165623 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025072882 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.067243882 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.073924599 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.076571207 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.067292519 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.067610642 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.044954822 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.081443492 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.079300929 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.05976223 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.077098131 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.084294227 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.043636631 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024976075 seconds Ending run at 10:33:57 UTC on 2024-02-21. Run time = 0.975022858 Run time without initialization = 0.848800309 Average number of zones advanced per microsecond: 3.088 Average number of zones advanced per microsecond per rank: 3.088 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.9751 ... 0.9751 ... 0.9751 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.3396 0.3396 0.3396 34.83% VisMF::Write(FabArray) 11 0.1761 0.1761 0.1761 18.06% MLCellLinOp::applyBC() 4351 0.08254 0.08254 0.08254 8.46% MLPoisson::Fsmooth() 3280 0.03389 0.03389 0.03389 3.48% FillBoundary_nowait() 3941 0.03047 0.03047 0.03047 3.13% StateData::FillBoundary(geom) 328 0.02676 0.02676 0.02676 2.74% Castro::normalize_species() 62 0.02222 0.02222 0.02222 2.28% amrex::Dot() 1114 0.02173 0.02173 0.02173 2.23% FabArray::norminf() 1061 0.02025 0.02025 0.02025 2.08% StateDataPhysBCFunct::() 41 0.01846 0.01846 0.01846 1.89% Castro::computeTemp() 63 0.01724 0.01724 0.01724 1.77% FabArray::ParallelCopy_nowait() 861 0.01395 0.01395 0.01395 1.43% Castro::enforce_min_density() 62 0.01378 0.01378 0.01378 1.41% FabArray::setVal() 1062 0.01366 0.01366 0.01366 1.40% FabArray::Saxpy() 1370 0.01333 0.01333 0.01333 1.37% amrex::Copy() 472 0.01107 0.01107 0.01107 1.14% MLCellLinOp::defineAuxData() 11 0.01069 0.01069 0.01069 1.10% MLPoisson::Fapply() 1060 0.0105 0.0105 0.0105 1.08% Gravity::fill_multipole_BCs() 11 0.009244 0.009244 0.009244 0.95% FabArray::Xpay() 739 0.008027 0.008027 0.008027 0.82% MLMG::addInterpCorrection() 410 0.007154 0.007154 0.007154 0.73% amrex::average_down 410 0.006385 0.006385 0.006385 0.65% Castro::estTimeStep() 21 0.006367 0.006367 0.006367 0.65% Amr::checkPoint() 3 0.00537 0.00537 0.00537 0.55% Castro::reset_internal_energy(MultiFab) 63 0.004766 0.004766 0.004766 0.49% BndryData::define() 11 0.004165 0.004165 0.004165 0.43% amrex::Add() 82 0.003588 0.003588 0.003588 0.37% Castro::construct_new_gravity_source() 10 0.003543 0.003543 0.003543 0.36% Castro::construct_old_gravity_source() 10 0.003034 0.003034 0.003034 0.31% check_for_negative_density() 10 0.002293 0.002293 0.002293 0.24% Castro::enforce_speed_limit() 62 0.002211 0.002211 0.002211 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001858 0.001858 0.001858 0.19% Amr::writePlotFile() 2 0.001781 0.001781 0.001781 0.18% Castro::reset_internal_energy(Fab) 504 0.001775 0.001775 0.001775 0.18% MLCGSolver::bicgstab 82 0.001645 0.001645 0.001645 0.17% MLCellLinOp::setLevelBC() 11 0.001634 0.001634 0.001634 0.17% Gravity::actual_solve_with_mlmg() 11 0.001569 0.001569 0.001569 0.16% Castro::initData() 1 0.001551 0.001551 0.001551 0.16% FabArray::mult() 43 0.001394 0.001394 0.001394 0.14% FabArray::setDomainBndry() 41 0.001384 0.001384 0.001384 0.14% MLCellLinOp::prepareForSolve() 11 0.001372 0.001372 0.001372 0.14% MultiFab::contains_nan() 20 0.001273 0.001273 0.001273 0.13% MLCellLinOp::smooth() 1640 0.001183 0.001183 0.001183 0.12% MLCellLinOp::compGrad() 11 0.001094 0.001094 0.001094 0.11% MLMG::prepareForSolve() 11 0.0009772 0.0009772 0.0009772 0.10% FabArray::FillBoundary() 3941 0.0008219 0.0008219 0.0008219 0.08% FabArrayBase::getCPC() 1323 0.0008059 0.0008059 0.0008059 0.08% FabArrayBase::CPC::define() 454 0.0006874 0.0006874 0.0006874 0.07% FabArrayBase::getFB() 3941 0.0006288 0.0006288 0.0006288 0.06% Gravity::get_new_grav_vector() 11 0.0006193 0.0006193 0.0006193 0.06% Amr::InitAmr() 1 0.0005601 0.0005601 0.0005601 0.06% Gravity::get_old_grav_vector() 10 0.0004956 0.0004956 0.0004956 0.05% AmrLevel::FillPatch() 41 0.0004411 0.0004411 0.0004411 0.05% MLCellLinOp::apply() 1060 0.0004213 0.0004213 0.0004213 0.04% MLCGSolver::ParallelAllReduce 1832 0.0004126 0.0004126 0.0004126 0.04% Amr::coarseTimeStep() 10 0.0003967 0.0003967 0.0003967 0.04% MultiFab::max() 11 0.0003291 0.0003291 0.0003291 0.03% main() 1 0.0002964 0.0002964 0.0002964 0.03% MLCellLinOp::defineBC() 11 0.0002797 0.0002797 0.0002797 0.03% FabArray::ParallelCopy() 861 0.0002582 0.0002582 0.0002582 0.03% FillPatchIterator::Initialize 41 0.0002321 0.0002321 0.0002321 0.02% MLMG::mgVcycle() 82 0.0002107 0.0002107 0.0002107 0.02% MLCellLinOp::correctionResidual() 410 0.0001876 0.0001876 0.0001876 0.02% Castro::subcycle_advance_ctu() 10 0.0001848 0.0001848 0.0001848 0.02% Castro::create_source_corrector() 10 0.0001808 0.0001808 0.0001808 0.02% MLLinOp::defineGrids() 11 0.0001739 0.0001739 0.0001739 0.02% Amr::timeStep() 10 0.0001589 0.0001589 0.0001589 0.02% Castro::construct_new_source() 50 0.0001517 0.0001517 0.0001517 0.02% MLMG:computeResOfCorrection() 410 0.0001407 0.0001407 0.0001407 0.01% StateData::checkPoint() 12 0.0001346 0.0001346 0.0001346 0.01% FabArrayBase::FB::FB() 56 0.0001272 0.0001272 0.0001272 0.01% Gravity::update_max_rhs() 11 0.0001262 0.0001262 0.0001262 0.01% Gravity::solve_for_phi() 10 0.0001063 0.0001063 0.0001063 0.01% MLMG::mgVcycle_down::0 82 9.497e-05 9.497e-05 9.497e-05 0.01% Castro::advance() 10 9.346e-05 9.346e-05 9.346e-05 0.01% MLMG::mgVcycle_down::1 82 9.219e-05 9.219e-05 9.219e-05 0.01% MLMG::actualBottomSolve() 82 9.01e-05 9.01e-05 9.01e-05 0.01% MLMG::mgVcycle_down::2 82 8.905e-05 8.905e-05 8.905e-05 0.01% Castro::Castro() 1 8.632e-05 8.632e-05 8.632e-05 0.01% MLMG::mgVcycle_down::4 82 8.581e-05 8.581e-05 8.581e-05 0.01% MLMG::mgVcycle_down::3 82 8.362e-05 8.362e-05 8.362e-05 0.01% AmrLevel::checkPoint() 3 7.813e-05 7.813e-05 7.813e-05 0.01% Castro::initialize_advance() 10 7.574e-05 7.574e-05 7.574e-05 0.01% MLMG::solve() 11 7.458e-05 7.458e-05 7.458e-05 0.01% Castro::clean_state() 62 7.151e-05 7.151e-05 7.151e-05 0.01% MLMG::mgVcycle_up::4 82 7.055e-05 7.055e-05 7.055e-05 0.01% MLMG::mgVcycle_up::0 82 6.447e-05 6.447e-05 6.447e-05 0.01% Castro::finalize_advance() 10 6.433e-05 6.433e-05 6.433e-05 0.01% Castro::enforce_consistent_e() 1 6.254e-05 6.254e-05 6.254e-05 0.01% Castro::initialize_do_advance() 10 6.157e-05 6.157e-05 6.157e-05 0.01% MLMG::mgVcycle_up::3 82 6.013e-05 6.013e-05 6.013e-05 0.01% MLMG::oneIter() 82 5.888e-05 5.888e-05 5.888e-05 0.01% MLMG::mgVcycle_up::1 82 5.854e-05 5.854e-05 5.854e-05 0.01% MLMG::mgVcycle_up::2 82 5.743e-05 5.743e-05 5.743e-05 0.01% MLCellLinOp::solutionResidual() 93 5.562e-05 5.562e-05 5.562e-05 0.01% Castro::do_advance_ctu() 10 5.511e-05 5.511e-05 5.511e-05 0.01% FillPatchIterator::FillFromLevel0() 41 5.272e-05 5.272e-05 5.272e-05 0.01% Castro::finalize_do_advance() 10 4.551e-05 4.551e-05 4.551e-05 0.00% Castro::swap_state_time_levels() 10 4.483e-05 4.483e-05 4.483e-05 0.00% FillPatchSingleLevel 41 4.169e-05 4.169e-05 4.169e-05 0.00% MLMG::ResNormInf() 93 3.698e-05 3.698e-05 3.698e-05 0.00% MLMG::mgVcycle_bottom 82 3.689e-05 3.689e-05 3.689e-05 0.00% MLMG::computeResidual() 82 3.578e-05 3.578e-05 3.578e-05 0.00% StateData::define() 4 3.545e-05 3.545e-05 3.545e-05 0.00% Castro::post_timestep() 10 3.445e-05 3.445e-05 3.445e-05 0.00% Castro::initMFs() 1 3.434e-05 3.434e-05 3.434e-05 0.00% Amr::writeSmallPlotFile() 1 3.143e-05 3.143e-05 3.143e-05 0.00% Castro::construct_new_gravity() 10 2.863e-05 2.863e-05 2.863e-05 0.00% Amr::defBaseLevel() 1 2.709e-05 2.709e-05 2.709e-05 0.00% makeSFC 55 2.659e-05 2.659e-05 2.659e-05 0.00% Castro::buildMetrics() 1 2.62e-05 2.62e-05 2.62e-05 0.00% Castro::do_old_sources() 10 2.582e-05 2.582e-05 2.582e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.377e-05 2.377e-05 2.377e-05 0.00% MLPoisson::define() 11 2.079e-05 2.079e-05 2.079e-05 0.00% Castro::do_new_sources() 10 2.055e-05 2.055e-05 2.055e-05 0.00% Castro::construct_old_source() 50 2.051e-05 2.051e-05 2.051e-05 0.00% AmrLevel::AmrLevel(dm) 1 2e-05 2e-05 2e-05 0.00% Amr::FinalizeInit() 1 1.896e-05 1.896e-05 1.896e-05 0.00% Castro::apply_source_to_state() 20 1.785e-05 1.785e-05 1.785e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.695e-05 1.695e-05 1.695e-05 0.00% DistributionMapping::Distribute() 56 1.635e-05 1.635e-05 1.635e-05 0.00% MLPoisson::prepareForSolve() 11 1.405e-05 1.405e-05 1.405e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.343e-05 1.343e-05 1.343e-05 0.00% Castro::check_for_nan() 20 1.262e-05 1.262e-05 1.262e-05 0.00% Castro::construct_old_gravity() 10 1.252e-05 1.252e-05 1.252e-05 0.00% MLLinOp::define() 11 1.221e-05 1.221e-05 1.221e-05 0.00% Castro::post_init() 1 1.002e-05 1.002e-05 1.002e-05 0.00% MLMG::computeMLResidual() 11 9.701e-06 9.701e-06 9.701e-06 0.00% Gravity::swapTimeLevels() 10 8.918e-06 8.918e-06 8.918e-06 0.00% Amr::initSubcycle() 1 8.483e-06 8.483e-06 8.483e-06 0.00% Gravity::actual_multilevel_solve() 1 7.705e-06 7.705e-06 7.705e-06 0.00% Castro::computeNewDt() 9 6.813e-06 6.813e-06 6.813e-06 0.00% MLMG::getGradSolution() 11 6.7e-06 6.7e-06 6.7e-06 0.00% Castro::expand_state() 10 5.954e-06 5.954e-06 5.954e-06 0.00% AmrLevel::checkPointPost() 3 5.488e-06 5.488e-06 5.488e-06 0.00% Gravity::set_mass_offset() 11 5.17e-06 5.17e-06 5.17e-06 0.00% Castro::retry_advance_ctu() 10 4.719e-06 4.719e-06 4.719e-06 0.00% MLMG::MLRhsNormInf() 11 4.652e-06 4.652e-06 4.652e-06 0.00% Amr::InitializeInit() 1 4.614e-06 4.614e-06 4.614e-06 0.00% MLMG::MLResNormInf() 11 4.213e-06 4.213e-06 4.213e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.016e-06 3.016e-06 3.016e-06 0.00% Castro::computeInitialDt() 2 2.889e-06 2.889e-06 2.889e-06 0.00% Castro::FluxRegCrseInit 10 2.786e-06 2.786e-06 2.786e-06 0.00% Amr::init() 1 2.721e-06 2.721e-06 2.721e-06 0.00% Castro::FluxRegFineAdd() 10 2.284e-06 2.284e-06 2.284e-06 0.00% AmrLevel::checkPointPre() 3 2.097e-06 2.097e-06 2.097e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.892e-06 1.892e-06 1.892e-06 0.00% Castro::post_regrid() 1 1.37e-06 1.37e-06 1.37e-06 0.00% Amr::initialInit() 1 9.04e-07 9.04e-07 9.04e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.975 0.975 0.975 100.00% Amr::coarseTimeStep() 10 0.8236 0.8236 0.8236 84.47% Amr::timeStep() 10 0.7311 0.7311 0.7311 74.98% Castro::advance() 10 0.7198 0.7198 0.7198 73.82% Castro::subcycle_advance_ctu() 10 0.7062 0.7062 0.7062 72.43% Castro::do_advance_ctu() 10 0.706 0.706 0.706 72.41% Castro::construct_ctu_hydro_source() 10 0.3521 0.3521 0.3521 36.11% Gravity::solve_phi_with_mlmg() 11 0.2994 0.2994 0.2994 30.71% Gravity::actual_solve_with_mlmg() 11 0.2897 0.2897 0.2897 29.71% Castro::construct_new_gravity() 10 0.27 0.27 0.27 27.69% MLMG::solve() 11 0.267 0.267 0.267 27.38% Gravity::solve_for_phi() 10 0.2537 0.2537 0.2537 26.02% MLMG::oneIter() 82 0.2512 0.2512 0.2512 25.76% MLMG::mgVcycle() 82 0.2475 0.2475 0.2475 25.39% VisMF::Write(FabArray) 11 0.1761 0.1761 0.1761 18.06% Amr::checkPoint() 3 0.1339 0.1339 0.1339 13.73% AmrLevel::checkPoint() 3 0.1285 0.1285 0.1285 13.18% StateData::checkPoint() 12 0.1284 0.1284 0.1284 13.17% Amr::init() 1 0.1255 0.1255 0.1255 12.87% MLCellLinOp::smooth() 1640 0.1231 0.1231 0.1231 12.63% MLCellLinOp::applyBC() 4351 0.1146 0.1146 0.1146 11.75% MLMG::mgVcycle_bottom 82 0.07426 0.07426 0.07426 7.62% MLMG::actualBottomSolve() 82 0.07422 0.07422 0.07422 7.61% MLCGSolver::bicgstab 82 0.07336 0.07336 0.07336 7.52% Castro::clean_state() 62 0.06108 0.06108 0.06108 6.26% AmrLevel::FillPatch() 41 0.05547 0.05547 0.05547 5.69% Amr::initialInit() 1 0.05519 0.05519 0.05519 5.66% FillPatchIterator::Initialize 41 0.05101 0.05101 0.05101 5.23% Amr::writePlotFile() 2 0.05016 0.05016 0.05016 5.14% Amr::FinalizeInit() 1 0.05013 0.05013 0.05013 5.14% FillPatchIterator::FillFromLevel0() 41 0.04939 0.04939 0.04939 5.07% FillPatchSingleLevel 41 0.04934 0.04934 0.04934 5.06% Castro::post_init() 1 0.04863 0.04863 0.04863 4.99% Gravity::multilevel_solve_for_new_phi() 1 0.04616 0.04616 0.04616 4.73% Gravity::actual_multilevel_solve() 1 0.04614 0.04614 0.04614 4.73% StateDataPhysBCFunct::() 41 0.04521 0.04521 0.04521 4.64% MLCellLinOp::apply() 1060 0.03694 0.03694 0.03694 3.79% MLMG::mgVcycle_down::0 82 0.03481 0.03481 0.03481 3.57% MLPoisson::Fsmooth() 3280 0.03389 0.03389 0.03389 3.48% FabArray::FillBoundary() 3941 0.03205 0.03205 0.03205 3.29% FillBoundary_nowait() 3941 0.03123 0.03123 0.03123 3.20% StateData::FillBoundary(geom) 328 0.02676 0.02676 0.02676 2.74% MLMG::mgVcycle_up::0 82 0.02624 0.02624 0.02624 2.69% Castro::computeTemp() 63 0.02378 0.02378 0.02378 2.44% Castro::normalize_species() 62 0.02222 0.02222 0.02222 2.28% Castro::construct_old_gravity() 10 0.02206 0.02206 0.02206 2.26% Gravity::get_old_grav_vector() 10 0.02204 0.02204 0.02204 2.26% Castro::initialize_do_advance() 10 0.02202 0.02202 0.02202 2.26% amrex::Dot() 1114 0.02173 0.02173 0.02173 2.23% Castro::do_old_sources() 10 0.02111 0.02111 0.02111 2.17% MLMG:computeResOfCorrection() 410 0.02077 0.02077 0.02077 2.13% MLCellLinOp::correctionResidual() 410 0.02063 0.02063 0.02063 2.12% FabArray::norminf() 1061 0.02025 0.02025 0.02025 2.08% Gravity::get_new_grav_vector() 11 0.01831 0.01831 0.01831 1.88% MLPoisson::define() 11 0.01795 0.01795 0.01795 1.84% MLMG::mgVcycle_down::1 82 0.0172 0.0172 0.0172 1.76% MLMG::mgVcycle_down::2 82 0.01579 0.01579 0.01579 1.62% MLMG::mgVcycle_down::3 82 0.01545 0.01545 0.01545 1.58% MLMG::mgVcycle_down::4 82 0.01534 0.01534 0.01534 1.57% Castro::do_new_sources() 10 0.01514 0.01514 0.01514 1.55% FabArray::ParallelCopy() 861 0.015 0.015 0.015 1.54% FabArray::ParallelCopy_nowait() 861 0.01474 0.01474 0.01474 1.51% Castro::enforce_min_density() 62 0.01378 0.01378 0.01378 1.41% FabArray::setVal() 1062 0.01366 0.01366 0.01366 1.40% FabArray::Saxpy() 1370 0.01333 0.01333 0.01333 1.37% MLCGSolver::ParallelAllReduce 1832 0.01314 0.01314 0.01314 1.35% Castro::initialize_advance() 10 0.0129 0.0129 0.0129 1.32% MLMG::addInterpCorrection() 410 0.01257 0.01257 0.01257 1.29% MLMG::mgVcycle_up::1 82 0.0123 0.0123 0.0123 1.26% MLCellLinOp::defineAuxData() 11 0.01219 0.01219 0.01219 1.25% MLMG::mgVcycle_up::4 82 0.01217 0.01217 0.01217 1.25% Castro::expand_state() 10 0.01198 0.01198 0.01198 1.23% MLMG::mgVcycle_up::2 82 0.01198 0.01198 0.01198 1.23% amrex::average_down 410 0.01188 0.01188 0.01188 1.22% MLMG::mgVcycle_up::3 82 0.01178 0.01178 0.01178 1.21% amrex::Copy() 472 0.01107 0.01107 0.01107 1.14% Castro::post_timestep() 10 0.01106 0.01106 0.01106 1.13% MLPoisson::Fapply() 1060 0.0105 0.0105 0.0105 1.08% Gravity::fill_multipole_BCs() 11 0.009478 0.009478 0.009478 0.97% FabArray::Xpay() 739 0.008027 0.008027 0.008027 0.82% MLCellLinOp::solutionResidual() 93 0.007879 0.007879 0.007879 0.81% Castro::reset_internal_energy(MultiFab) 63 0.006541 0.006541 0.006541 0.67% MLMG::computeResidual() 82 0.00653 0.00653 0.00653 0.67% Castro::estTimeStep() 21 0.006367 0.006367 0.006367 0.65% MLCellLinOp::defineBC() 11 0.005499 0.005499 0.005499 0.56% MLMG::prepareForSolve() 11 0.005323 0.005323 0.005323 0.55% BndryData::define() 11 0.005219 0.005219 0.005219 0.54% Amr::InitializeInit() 1 0.005061 0.005061 0.005061 0.52% Amr::defBaseLevel() 1 0.005056 0.005056 0.005056 0.52% Castro::initData() 1 0.004396 0.004396 0.004396 0.45% Castro::construct_new_source() 50 0.003695 0.003695 0.003695 0.38% amrex::Add() 82 0.003588 0.003588 0.003588 0.37% Castro::construct_new_gravity_source() 10 0.003543 0.003543 0.003543 0.36% Castro::construct_old_source() 50 0.003055 0.003055 0.003055 0.31% Castro::construct_old_gravity_source() 10 0.003034 0.003034 0.003034 0.31% Castro::computeNewDt() 9 0.002716 0.002716 0.002716 0.28% Castro::finalize_do_advance() 10 0.002666 0.002666 0.002666 0.27% check_for_negative_density() 10 0.002293 0.002293 0.002293 0.24% MLMG::ResNormInf() 93 0.002216 0.002216 0.002216 0.23% Castro::enforce_speed_limit() 62 0.002211 0.002211 0.002211 0.23% Castro::apply_source_to_state() 20 0.001883 0.001883 0.001883 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001858 0.001858 0.001858 0.19% Castro::reset_internal_energy(Fab) 504 0.001775 0.001775 0.001775 0.18% MLCellLinOp::setLevelBC() 11 0.001634 0.001634 0.001634 0.17% MLMG::getGradSolution() 11 0.001605 0.001605 0.001605 0.16% MLCellLinOp::compGrad() 11 0.001599 0.001599 0.001599 0.16% FabArrayBase::getCPC() 1323 0.001493 0.001493 0.001493 0.15% MLMG::computeMLResidual() 11 0.001395 0.001395 0.001395 0.14% FabArray::mult() 43 0.001394 0.001394 0.001394 0.14% MLPoisson::prepareForSolve() 11 0.001386 0.001386 0.001386 0.14% FabArray::setDomainBndry() 41 0.001384 0.001384 0.001384 0.14% MLCellLinOp::prepareForSolve() 11 0.001372 0.001372 0.001372 0.14% Castro::check_for_nan() 20 0.001285 0.001285 0.001285 0.13% MultiFab::contains_nan() 20 0.001273 0.001273 0.001273 0.13% Castro::post_regrid() 1 0.00119 0.00119 0.00119 0.12% Castro::computeInitialDt() 2 0.00104 0.00104 0.00104 0.11% Gravity::update_max_rhs() 11 0.0009909 0.0009909 0.0009909 0.10% FabArrayBase::getFB() 3941 0.000756 0.000756 0.000756 0.08% FabArrayBase::CPC::define() 454 0.0006874 0.0006874 0.0006874 0.07% Castro::finalize_advance() 10 0.0006134 0.0006134 0.0006134 0.06% Castro::Castro() 1 0.0005734 0.0005734 0.0005734 0.06% Amr::InitAmr() 1 0.0005686 0.0005686 0.0005686 0.06% Gravity::swapTimeLevels() 10 0.0004626 0.0004626 0.0004626 0.05% MLMG::MLResNormInf() 11 0.0003382 0.0003382 0.0003382 0.03% MultiFab::max() 11 0.0003291 0.0003291 0.0003291 0.03% Castro::buildMetrics() 1 0.0002776 0.0002776 0.0002776 0.03% MLLinOp::define() 11 0.0002431 0.0002431 0.0002431 0.02% MLMG::MLRhsNormInf() 11 0.0002327 0.0002327 0.0002327 0.02% MLLinOp::defineGrids() 11 0.0002309 0.0002309 0.0002309 0.02% Castro::create_source_corrector() 10 0.0001808 0.0001808 0.0001808 0.02% FabArrayBase::FB::FB() 56 0.0001272 0.0001272 0.0001272 0.01% Castro::enforce_consistent_e() 1 6.254e-05 6.254e-05 6.254e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.544e-05 5.544e-05 5.544e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.507e-05 5.507e-05 5.507e-05 0.01% Castro::swap_state_time_levels() 10 4.483e-05 4.483e-05 4.483e-05 0.00% makeSFC 55 4.165e-05 4.165e-05 4.165e-05 0.00% StateData::define() 4 3.545e-05 3.545e-05 3.545e-05 0.00% Castro::initMFs() 1 3.434e-05 3.434e-05 3.434e-05 0.00% Amr::writeSmallPlotFile() 1 3.143e-05 3.143e-05 3.143e-05 0.00% DistributionMapping::Distribute() 56 1.635e-05 1.635e-05 1.635e-05 0.00% Amr::initSubcycle() 1 8.483e-06 8.483e-06 8.483e-06 0.00% AmrLevel::checkPointPost() 3 5.488e-06 5.488e-06 5.488e-06 0.00% Gravity::set_mass_offset() 11 5.17e-06 5.17e-06 5.17e-06 0.00% Castro::retry_advance_ctu() 10 4.719e-06 4.719e-06 4.719e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.307e-06 4.307e-06 4.307e-06 0.00% Castro::FluxRegCrseInit 10 2.786e-06 2.786e-06 2.786e-06 0.00% Castro::FluxRegFineAdd() 10 2.284e-06 2.284e-06 2.284e-06 0.00% AmrLevel::checkPointPre() 3 2.097e-06 2.097e-06 2.097e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.892e-06 1.892e-06 1.892e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 82 MiB 9042 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1043 KiB 39 MiB Castro::initialize_do_advance() 80 80 28 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1445 KiB 28 MiB Castro::initialize_advance() 80 80 17 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7616 KiB 14 MiB MLMG::prepareForSolve() 660 660 3366 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 195 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 231 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7521 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 16 KiB 2053 KiB Gravity::update_max_rhs() 88 88 1970 B 2048 KiB Gravity::solve_for_phi() 80 80 532 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 96 KiB 2048 KiB BndryData::define() 1056 1056 308 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 196 KiB 671 KiB Castro::estTimeStep() 21 21 3185 B 480 KiB VisMF::Write(FabArray) 656 656 3160 B 320 KiB Castro::normalize_species() 62 62 7425 B 320 KiB amrex::average_down 1067 1067 1561 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1102 B 257 KiB amrex::Dot() 1360 1360 3280 B 160 KiB FabArray::norminf() 1143 1143 3202 B 160 KiB check_for_negative_density() 10 10 379 B 160 KiB Castro::initData() 1 1 46 B 160 KiB MultiFab::max() 11 11 52 B 160 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MultiFab::contains_nan() 20 20 25 B 20 KiB MLPoisson::Fsmooth() 132 132 3285 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 41 B 10 KiB FillBoundary_nowait() 760 760 273 B 9648 B MLCellLinOp::applyBC() 8702 8702 209 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3947 B 6144 B StateData::FillBoundary(geom) 1992 1992 42 B 2928 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 347 B 1248 B MLCGSolver::bicgstab 410 410 90 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 472 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 37 KiB 8192 KiB VisMF::Write(FabArray) 744 744 395 KiB 3584 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MLPoisson::Fsmooth() 132 132 3285 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 41 B 10 KiB FillBoundary_nowait() 760 760 273 B 9648 B MLCellLinOp::applyBC() 4351 4351 208 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3947 B 6144 B Gravity::get_new_grav_vector() 3 3 2900 B 3072 B StateData::FillBoundary(geom) 1992 1992 42 B 2928 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B amrex::average_down 83 83 617 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 280 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 24 B 400 B FabArray::norminf() 1143 1143 9 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2167 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.02-25-g2230caa24c7d) finalized Initializing AMReX (24.02-25-g2230caa24c7d)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.02-25-g2230caa24c7d) initialized Starting run at 10:33:58 UTC on 2024-02-21. Successfully read inputs file ... Castro git describe: 24.02-21-g56d243e7a AMReX git describe: 24.02-25-g2230caa24 Microphysics git describe: 24.02-22-g4f2ee5fd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.523856999 Restart time = 0.124730556 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.074522156 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.057239505 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.080406033 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.076971902 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.060786461 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.046623434 seconds Ending run at 10:33:59 UTC on 2024-02-21. Run time = 0.522362299 Run time without initialization = 0.396972182 Average number of zones advanced per microsecond: 3.302 Average number of zones advanced per microsecond per rank: 3.302 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.5224 ... 0.5224 ... 0.5224 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1544 0.1544 0.1544 29.56% VisMF::Read() 3 0.06067 0.06067 0.06067 11.61% Amr::restart() 1 0.05721 0.05721 0.05721 10.95% MLCellLinOp::applyBC() 1910 0.03589 0.03589 0.03589 6.87% VisMF::Write(FabArray) 1 0.02607 0.02607 0.02607 4.99% Amr::writePlotFile() 1 0.02032 0.02032 0.02032 3.89% MLPoisson::Fsmooth() 1440 0.01476 0.01476 0.01476 2.83% StateData::FillBoundary(geom) 160 0.01329 0.01329 0.01329 2.54% FillBoundary_nowait() 1730 0.0129 0.0129 0.0129 2.47% Castro::normalize_species() 30 0.01229 0.01229 0.01229 2.35% amrex::Dot() 484 0.009287 0.009287 0.009287 1.78% FabArray::norminf() 465 0.008745 0.008745 0.008745 1.67% Castro::computeTemp() 30 0.007221 0.007221 0.007221 1.38% FabArray::setVal() 501 0.006625 0.006625 0.006625 1.27% FabArray::ParallelCopy_nowait() 380 0.006218 0.006218 0.006218 1.19% FabArray::Saxpy() 597 0.005918 0.005918 0.005918 1.13% MLCellLinOp::defineAuxData() 6 0.005734 0.005734 0.005734 1.10% Gravity::fill_multipole_BCs() 6 0.005699 0.005699 0.005699 1.09% amrex::Copy() 221 0.005527 0.005527 0.005527 1.06% Castro::enforce_min_density() 30 0.005354 0.005354 0.005354 1.02% StateDataPhysBCFunct::() 20 0.004897 0.004897 0.004897 0.94% MLPoisson::Fapply() 464 0.004574 0.004574 0.004574 0.88% FabArray::Xpay() 325 0.003528 0.003528 0.003528 0.68% MLMG::addInterpCorrection() 180 0.003189 0.003189 0.003189 0.61% amrex::average_down 180 0.003023 0.003023 0.003023 0.58% Castro::estTimeStep() 10 0.002792 0.002792 0.002792 0.53% BndryData::define() 6 0.002221 0.002221 0.002221 0.43% Castro::reset_internal_energy(MultiFab) 30 0.002092 0.002092 0.002092 0.40% Castro::construct_new_gravity_source() 5 0.00173 0.00173 0.00173 0.33% amrex::Add() 36 0.001548 0.001548 0.001548 0.30% Castro::construct_old_gravity_source() 5 0.001496 0.001496 0.001496 0.29% check_for_negative_density() 5 0.001053 0.001053 0.001053 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009953 0.0009953 0.0009953 0.19% MLCellLinOp::setLevelBC() 6 0.0008771 0.0008771 0.0008771 0.17% Gravity::actual_solve_with_mlmg() 6 0.0008755 0.0008755 0.0008755 0.17% Castro::reset_internal_energy(Fab) 240 0.0008198 0.0008198 0.0008198 0.16% MLCellLinOp::prepareForSolve() 6 0.0007741 0.0007741 0.0007741 0.15% Castro::enforce_speed_limit() 30 0.0007687 0.0007687 0.0007687 0.15% MLCGSolver::bicgstab 36 0.0007103 0.0007103 0.0007103 0.14% FabArray::mult() 22 0.0007071 0.0007071 0.0007071 0.14% FabArray::setDomainBndry() 20 0.0007003 0.0007003 0.0007003 0.13% MultiFab::contains_nan() 10 0.0006755 0.0006755 0.0006755 0.13% MLCellLinOp::compGrad() 6 0.00062 0.00062 0.00062 0.12% MLMG::prepareForSolve() 6 0.0005674 0.0005674 0.0005674 0.11% MLCellLinOp::smooth() 720 0.0005105 0.0005105 0.0005105 0.10% Amr::InitAmr() 1 0.0004866 0.0004866 0.0004866 0.09% FabArrayBase::CPC::define() 244 0.0004264 0.0004264 0.0004264 0.08% FabArrayBase::getCPC() 632 0.0003794 0.0003794 0.0003794 0.07% FabArray::FillBoundary() 1730 0.0003702 0.0003702 0.0003702 0.07% Gravity::get_old_grav_vector() 5 0.0003685 0.0003685 0.0003685 0.07% Gravity::get_new_grav_vector() 5 0.0002912 0.0002912 0.0002912 0.06% main() 1 0.0002771 0.0002771 0.0002771 0.05% FabArrayBase::getFB() 1730 0.0002709 0.0002709 0.0002709 0.05% MultiFab::max() 6 0.0002075 0.0002075 0.0002075 0.04% AmrLevel::FillPatch() 20 0.0002007 0.0002007 0.0002007 0.04% MLCellLinOp::apply() 464 0.000179 0.000179 0.000179 0.03% MLCGSolver::ParallelAllReduce 798 0.0001662 0.0001662 0.0001662 0.03% Amr::coarseTimeStep() 5 0.0001573 0.0001573 0.0001573 0.03% MLCellLinOp::defineBC() 6 0.000151 0.000151 0.000151 0.03% Castro::subcycle_advance_ctu() 5 0.000117 0.000117 0.000117 0.02% FabArray::ParallelCopy() 380 0.0001119 0.0001119 0.0001119 0.02% FillPatchIterator::Initialize 20 0.000107 0.000107 0.000107 0.02% Castro::create_source_corrector() 5 0.000107 0.000107 0.000107 0.02% MLLinOp::defineGrids() 6 0.0001029 0.0001029 0.0001029 0.02% Castro::construct_new_source() 25 9.321e-05 9.321e-05 9.321e-05 0.02% MLMG::mgVcycle() 36 9.029e-05 9.029e-05 9.029e-05 0.02% Amr::timeStep() 5 8.637e-05 8.637e-05 8.637e-05 0.02% MLCellLinOp::correctionResidual() 180 8.047e-05 8.047e-05 8.047e-05 0.02% AmrLevel::restart() 1 7.989e-05 7.989e-05 7.989e-05 0.02% Castro::do_advance_ctu() 5 6.712e-05 6.712e-05 6.712e-05 0.01% Castro::initialize_do_advance() 5 6.607e-05 6.607e-05 6.607e-05 0.01% StateData::restartDoit() 4 6.59e-05 6.59e-05 6.59e-05 0.01% Gravity::update_max_rhs() 6 6.374e-05 6.374e-05 6.374e-05 0.01% FabArrayBase::FB::FB() 26 6.137e-05 6.137e-05 6.137e-05 0.01% Gravity::solve_for_phi() 5 6.104e-05 6.104e-05 6.104e-05 0.01% MLMG:computeResOfCorrection() 180 5.657e-05 5.657e-05 5.657e-05 0.01% Castro::advance() 5 5.617e-05 5.617e-05 5.617e-05 0.01% Castro::finalize_do_advance() 5 4.192e-05 4.192e-05 4.192e-05 0.01% Castro::clean_state() 30 4.054e-05 4.054e-05 4.054e-05 0.01% MLMG::actualBottomSolve() 36 4.049e-05 4.049e-05 4.049e-05 0.01% MLMG::mgVcycle_down::0 36 4.011e-05 4.011e-05 4.011e-05 0.01% MLMG::mgVcycle_down::1 36 3.809e-05 3.809e-05 3.809e-05 0.01% Castro::post_restart() 1 3.73e-05 3.73e-05 3.73e-05 0.01% Castro::initialize_advance() 5 3.555e-05 3.555e-05 3.555e-05 0.01% MLMG::mgVcycle_down::2 36 3.471e-05 3.471e-05 3.471e-05 0.01% MLMG::mgVcycle_down::4 36 3.349e-05 3.349e-05 3.349e-05 0.01% MLMG::solve() 6 3.348e-05 3.348e-05 3.348e-05 0.01% Castro::post_timestep() 5 3.277e-05 3.277e-05 3.277e-05 0.01% MLMG::mgVcycle_down::3 36 3.167e-05 3.167e-05 3.167e-05 0.01% Amr::writeSmallPlotFile() 1 3.149e-05 3.149e-05 3.149e-05 0.01% MLMG::mgVcycle_up::4 36 2.992e-05 2.992e-05 2.992e-05 0.01% Castro::finalize_advance() 5 2.978e-05 2.978e-05 2.978e-05 0.01% Castro::construct_old_source() 25 2.794e-05 2.794e-05 2.794e-05 0.01% Castro::buildMetrics() 1 2.697e-05 2.697e-05 2.697e-05 0.01% MLMG::oneIter() 36 2.605e-05 2.605e-05 2.605e-05 0.00% MLMG::mgVcycle_up::3 36 2.542e-05 2.542e-05 2.542e-05 0.00% MLMG::mgVcycle_up::0 36 2.528e-05 2.528e-05 2.528e-05 0.00% Castro::initMFs() 1 2.501e-05 2.501e-05 2.501e-05 0.00% Castro::swap_state_time_levels() 5 2.482e-05 2.482e-05 2.482e-05 0.00% MLMG::mgVcycle_up::2 36 2.421e-05 2.421e-05 2.421e-05 0.00% MLCellLinOp::solutionResidual() 42 2.299e-05 2.299e-05 2.299e-05 0.00% MLMG::mgVcycle_up::1 36 2.264e-05 2.264e-05 2.264e-05 0.00% FillPatchIterator::FillFromLevel0() 20 2.233e-05 2.233e-05 2.233e-05 0.00% Castro::construct_new_gravity() 5 2.069e-05 2.069e-05 2.069e-05 0.00% MLMG::ResNormInf() 42 1.8e-05 1.8e-05 1.8e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.729e-05 1.729e-05 1.729e-05 0.00% FillPatchSingleLevel 20 1.622e-05 1.622e-05 1.622e-05 0.00% MLPoisson::define() 6 1.505e-05 1.505e-05 1.505e-05 0.00% MLMG::mgVcycle_bottom 36 1.454e-05 1.454e-05 1.454e-05 0.00% makeSFC 30 1.44e-05 1.44e-05 1.44e-05 0.00% MLMG::computeResidual() 36 1.398e-05 1.398e-05 1.398e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.343e-05 1.343e-05 1.343e-05 0.00% MLPoisson::prepareForSolve() 6 1.202e-05 1.202e-05 1.202e-05 0.00% Castro::do_old_sources() 5 1.181e-05 1.181e-05 1.181e-05 0.00% Castro::do_new_sources() 5 1.054e-05 1.054e-05 1.054e-05 0.00% DistributionMapping::Distribute() 31 1.021e-05 1.021e-05 1.021e-05 0.00% Gravity::actual_multilevel_solve() 1 8.645e-06 8.645e-06 8.645e-06 0.00% Amr::initSubcycle() 1 8.315e-06 8.315e-06 8.315e-06 0.00% MLLinOp::define() 6 8.185e-06 8.185e-06 8.185e-06 0.00% Castro::check_for_nan() 10 7.69e-06 7.69e-06 7.69e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.662e-06 7.662e-06 7.662e-06 0.00% Castro::construct_old_gravity() 5 6.534e-06 6.534e-06 6.534e-06 0.00% Castro::apply_source_to_state() 10 6.197e-06 6.197e-06 6.197e-06 0.00% MLMG::computeMLResidual() 6 4.308e-06 4.308e-06 4.308e-06 0.00% Gravity::swapTimeLevels() 5 3.804e-06 3.804e-06 3.804e-06 0.00% Castro::computeNewDt() 5 3.473e-06 3.473e-06 3.473e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.421e-06 3.421e-06 3.421e-06 0.00% MLMG::getGradSolution() 6 3.398e-06 3.398e-06 3.398e-06 0.00% Castro::expand_state() 5 2.935e-06 2.935e-06 2.935e-06 0.00% MLMG::MLResNormInf() 6 2.53e-06 2.53e-06 2.53e-06 0.00% Gravity::set_mass_offset() 6 2.525e-06 2.525e-06 2.525e-06 0.00% MLMG::MLRhsNormInf() 6 2.288e-06 2.288e-06 2.288e-06 0.00% Castro::retry_advance_ctu() 5 2.014e-06 2.014e-06 2.014e-06 0.00% Castro::FluxRegCrseInit 5 1.436e-06 1.436e-06 1.436e-06 0.00% Amr::init() 1 1.335e-06 1.335e-06 1.335e-06 0.00% Castro::FluxRegFineAdd() 5 1.24e-06 1.24e-06 1.24e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.03e-06 1.03e-06 1.03e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.5224 0.5224 0.5224 100.00% Amr::coarseTimeStep() 5 0.3501 0.3501 0.3501 67.01% Amr::timeStep() 5 0.348 0.348 0.348 66.62% Castro::advance() 5 0.342 0.342 0.342 65.48% Castro::subcycle_advance_ctu() 5 0.3346 0.3346 0.3346 64.05% Castro::do_advance_ctu() 5 0.3344 0.3344 0.3344 64.02% Castro::construct_ctu_hydro_source() 5 0.16 0.16 0.16 30.63% Castro::construct_new_gravity() 5 0.1369 0.1369 0.1369 26.21% Gravity::solve_phi_with_mlmg() 6 0.1346 0.1346 0.1346 25.77% Gravity::solve_for_phi() 5 0.1289 0.1289 0.1289 24.68% Gravity::actual_solve_with_mlmg() 6 0.1287 0.1287 0.1287 24.63% Amr::init() 1 0.1248 0.1248 0.1248 23.89% Amr::restart() 1 0.1248 0.1248 0.1248 23.89% MLMG::solve() 6 0.1163 0.1163 0.1163 22.27% MLMG::oneIter() 36 0.1086 0.1086 0.1086 20.79% MLMG::mgVcycle() 36 0.107 0.107 0.107 20.49% AmrLevel::restart() 1 0.06104 0.06104 0.06104 11.68% StateData::restartDoit() 4 0.06095 0.06095 0.06095 11.67% VisMF::Read() 3 0.06067 0.06067 0.06067 11.61% MLCellLinOp::smooth() 720 0.05307 0.05307 0.05307 10.16% MLCellLinOp::applyBC() 1910 0.0495 0.0495 0.0495 9.48% Amr::writePlotFile() 1 0.04672 0.04672 0.04672 8.94% MLMG::mgVcycle_bottom 36 0.03196 0.03196 0.03196 6.12% MLMG::actualBottomSolve() 36 0.03194 0.03194 0.03194 6.11% MLCGSolver::bicgstab 36 0.03157 0.03157 0.03157 6.04% Castro::clean_state() 30 0.02859 0.02859 0.02859 5.47% VisMF::Write(FabArray) 1 0.02607 0.02607 0.02607 4.99% AmrLevel::FillPatch() 20 0.02318 0.02318 0.02318 4.44% FillPatchIterator::Initialize 20 0.02107 0.02107 0.02107 4.03% FillPatchIterator::FillFromLevel0() 20 0.02026 0.02026 0.02026 3.88% FillPatchSingleLevel 20 0.02024 0.02024 0.02024 3.87% StateDataPhysBCFunct::() 20 0.01819 0.01819 0.01819 3.48% MLCellLinOp::apply() 464 0.01617 0.01617 0.01617 3.09% MLMG::mgVcycle_down::0 36 0.0152 0.0152 0.0152 2.91% MLPoisson::Fsmooth() 1440 0.01476 0.01476 0.01476 2.83% FabArray::FillBoundary() 1730 0.0136 0.0136 0.0136 2.60% StateData::FillBoundary(geom) 160 0.01329 0.01329 0.01329 2.54% FillBoundary_nowait() 1730 0.01323 0.01323 0.01323 2.53% Castro::normalize_species() 30 0.01229 0.01229 0.01229 2.35% MLMG::mgVcycle_up::0 36 0.01127 0.01127 0.01127 2.16% Castro::initialize_do_advance() 5 0.01104 0.01104 0.01104 2.11% Castro::computeTemp() 30 0.01013 0.01013 0.01013 1.94% MLPoisson::define() 6 0.009668 0.009668 0.009668 1.85% Castro::do_old_sources() 5 0.009559 0.009559 0.009559 1.83% amrex::Dot() 484 0.009287 0.009287 0.009287 1.78% MLMG:computeResOfCorrection() 180 0.009003 0.009003 0.009003 1.72% MLCellLinOp::correctionResidual() 180 0.008946 0.008946 0.008946 1.71% FabArray::norminf() 465 0.008745 0.008745 0.008745 1.67% Gravity::get_new_grav_vector() 5 0.007891 0.007891 0.007891 1.51% Castro::construct_old_gravity() 5 0.007774 0.007774 0.007774 1.49% Gravity::get_old_grav_vector() 5 0.007767 0.007767 0.007767 1.49% Castro::do_new_sources() 5 0.007632 0.007632 0.007632 1.46% MLMG::mgVcycle_down::1 36 0.007506 0.007506 0.007506 1.44% Castro::initialize_advance() 5 0.007121 0.007121 0.007121 1.36% MLMG::mgVcycle_down::2 36 0.006871 0.006871 0.006871 1.32% FabArray::ParallelCopy() 380 0.006727 0.006727 0.006727 1.29% MLMG::mgVcycle_down::3 36 0.006674 0.006674 0.006674 1.28% FabArray::setVal() 501 0.006625 0.006625 0.006625 1.27% FabArray::ParallelCopy_nowait() 380 0.006615 0.006615 0.006615 1.27% MLMG::mgVcycle_down::4 36 0.006598 0.006598 0.006598 1.26% MLCellLinOp::defineAuxData() 6 0.006543 0.006543 0.006543 1.25% Castro::post_restart() 1 0.006348 0.006348 0.006348 1.22% Castro::expand_state() 5 0.006101 0.006101 0.006101 1.17% Gravity::multilevel_solve_for_new_phi() 1 0.005968 0.005968 0.005968 1.14% Gravity::actual_multilevel_solve() 1 0.005951 0.005951 0.005951 1.14% FabArray::Saxpy() 597 0.005918 0.005918 0.005918 1.13% Castro::post_timestep() 5 0.005896 0.005896 0.005896 1.13% Gravity::fill_multipole_BCs() 6 0.005825 0.005825 0.005825 1.12% MLCGSolver::ParallelAllReduce 798 0.00567 0.00567 0.00567 1.09% amrex::Copy() 221 0.005527 0.005527 0.005527 1.06% MLMG::addInterpCorrection() 180 0.005515 0.005515 0.005515 1.06% amrex::average_down 180 0.00539 0.00539 0.00539 1.03% Castro::enforce_min_density() 30 0.005354 0.005354 0.005354 1.02% MLMG::mgVcycle_up::4 36 0.005313 0.005313 0.005313 1.02% MLMG::mgVcycle_up::1 36 0.005309 0.005309 0.005309 1.02% MLMG::mgVcycle_up::2 36 0.005196 0.005196 0.005196 0.99% MLMG::mgVcycle_up::3 36 0.005071 0.005071 0.005071 0.97% MLPoisson::Fapply() 464 0.004574 0.004574 0.004574 0.88% MLCellLinOp::solutionResidual() 42 0.00362 0.00362 0.00362 0.69% FabArray::Xpay() 325 0.003528 0.003528 0.003528 0.68% MLCellLinOp::defineBC() 6 0.002967 0.002967 0.002967 0.57% MLMG::prepareForSolve() 6 0.002928 0.002928 0.002928 0.56% Castro::reset_internal_energy(MultiFab) 30 0.002912 0.002912 0.002912 0.56% MLMG::computeResidual() 36 0.002823 0.002823 0.002823 0.54% BndryData::define() 6 0.002816 0.002816 0.002816 0.54% Castro::estTimeStep() 10 0.002792 0.002792 0.002792 0.53% Castro::computeNewDt() 5 0.001891 0.001891 0.001891 0.36% Castro::construct_new_source() 25 0.001823 0.001823 0.001823 0.35% Castro::construct_new_gravity_source() 5 0.00173 0.00173 0.00173 0.33% amrex::Add() 36 0.001548 0.001548 0.001548 0.30% Castro::construct_old_source() 25 0.001524 0.001524 0.001524 0.29% Castro::construct_old_gravity_source() 5 0.001496 0.001496 0.001496 0.29% check_for_negative_density() 5 0.001053 0.001053 0.001053 0.20% MLMG::ResNormInf() 42 0.0009964 0.0009964 0.0009964 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009953 0.0009953 0.0009953 0.19% Castro::apply_source_to_state() 10 0.0009653 0.0009653 0.0009653 0.18% Castro::finalize_do_advance() 5 0.0009457 0.0009457 0.0009457 0.18% MLMG::getGradSolution() 6 0.000904 0.000904 0.000904 0.17% MLCellLinOp::compGrad() 6 0.0009006 0.0009006 0.0009006 0.17% MLCellLinOp::setLevelBC() 6 0.0008771 0.0008771 0.0008771 0.17% Castro::reset_internal_energy(Fab) 240 0.0008198 0.0008198 0.0008198 0.16% MLMG::computeMLResidual() 6 0.0008155 0.0008155 0.0008155 0.16% FabArrayBase::getCPC() 632 0.0008059 0.0008059 0.0008059 0.15% MLPoisson::prepareForSolve() 6 0.0007861 0.0007861 0.0007861 0.15% MLCellLinOp::prepareForSolve() 6 0.0007741 0.0007741 0.0007741 0.15% Castro::enforce_speed_limit() 30 0.0007687 0.0007687 0.0007687 0.15% Gravity::update_max_rhs() 6 0.0007168 0.0007168 0.0007168 0.14% FabArray::mult() 22 0.0007071 0.0007071 0.0007071 0.14% FabArray::setDomainBndry() 20 0.0007003 0.0007003 0.0007003 0.13% Castro::check_for_nan() 10 0.0006832 0.0006832 0.0006832 0.13% MultiFab::contains_nan() 10 0.0006755 0.0006755 0.0006755 0.13% Amr::InitAmr() 1 0.0004949 0.0004949 0.0004949 0.09% FabArrayBase::CPC::define() 244 0.0004264 0.0004264 0.0004264 0.08% FabArrayBase::getFB() 1730 0.0003323 0.0003323 0.0003323 0.06% Castro::finalize_advance() 5 0.0002995 0.0002995 0.0002995 0.06% Gravity::swapTimeLevels() 5 0.0002434 0.0002434 0.0002434 0.05% MultiFab::max() 6 0.0002075 0.0002075 0.0002075 0.04% MLMG::MLResNormInf() 6 0.0001967 0.0001967 0.0001967 0.04% Castro::buildMetrics() 1 0.0001535 0.0001535 0.0001535 0.03% MLLinOp::define() 6 0.0001429 0.0001429 0.0001429 0.03% MLLinOp::defineGrids() 6 0.0001348 0.0001348 0.0001348 0.03% MLMG::MLRhsNormInf() 6 0.0001225 0.0001225 0.0001225 0.02% Castro::create_source_corrector() 5 0.000107 0.000107 0.000107 0.02% FabArrayBase::FB::FB() 26 6.137e-05 6.137e-05 6.137e-05 0.01% Amr::writeSmallPlotFile() 1 3.149e-05 3.149e-05 3.149e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.08e-05 3.08e-05 3.08e-05 0.01% Castro::initMFs() 1 2.501e-05 2.501e-05 2.501e-05 0.00% Castro::swap_state_time_levels() 5 2.482e-05 2.482e-05 2.482e-05 0.00% makeSFC 30 2.313e-05 2.313e-05 2.313e-05 0.00% DistributionMapping::Distribute() 31 1.021e-05 1.021e-05 1.021e-05 0.00% Amr::initSubcycle() 1 8.315e-06 8.315e-06 8.315e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.892e-06 4.892e-06 4.892e-06 0.00% Gravity::set_mass_offset() 6 2.525e-06 2.525e-06 2.525e-06 0.00% Castro::retry_advance_ctu() 5 2.014e-06 2.014e-06 2.014e-06 0.00% Castro::FluxRegCrseInit 5 1.436e-06 1.436e-06 1.436e-06 0.00% Castro::FluxRegFineAdd() 5 1.24e-06 1.24e-06 1.24e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.03e-06 1.03e-06 1.03e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 150 MiB 9042 MiB Castro::initMFs() 48 48 52 MiB 68 MiB Castro::swap_state_time_levels() 32 32 42 MiB 55 MiB StateData::restartDoit() 32 32 47 MiB 55 MiB FillPatchIterator::Initialize 160 160 887 KiB 39 MiB Castro::initialize_do_advance() 40 40 24 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1477 KiB 28 MiB Castro::initialize_advance() 40 40 15 MiB 23 MiB Castro::buildMetrics() 32 32 12 MiB 15 MiB Castro::post_restart() 48 48 5853 KiB 14 MiB MLMG::prepareForSolve() 361 361 2734 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 154 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 154 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 5842 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 18 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2729 B 2048 KiB Gravity::solve_for_phi() 40 40 504 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 23 KiB 2048 KiB BndryData::define() 576 576 252 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 162 KiB 671 KiB Castro::estTimeStep() 10 10 2556 B 480 KiB VisMF::Write(FabArray) 112 112 1924 B 320 KiB Castro::normalize_species() 30 30 7657 B 320 KiB amrex::average_down 469 469 1311 B 257 KiB MLMG::addInterpCorrection() 468 468 901 B 257 KiB amrex::Dot() 592 592 2625 B 160 KiB FabArray::norminf() 501 501 2583 B 160 KiB check_for_negative_density() 5 5 315 B 160 KiB MultiFab::max() 6 6 62 B 160 KiB FabArray::setVal() 66 66 18 KiB 27 KiB MultiFab::contains_nan() 10 10 25 B 20 KiB MLPoisson::Fsmooth() 60 60 2649 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 37 B 10 KiB FillBoundary_nowait() 336 336 214 B 9648 B MLCellLinOp::applyBC() 3820 3820 171 B 9344 B amrex::Copy() 56 56 5261 B 8816 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B StateData::FillBoundary(geom) 960 960 34 B 2448 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCellLinOp::defineBC() 36 36 283 B 1248 B MLCGSolver::bicgstab 180 180 72 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 906 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 71 KiB 8192 KiB VisMF::Write(FabArray) 120 120 132 KiB 3584 KiB VisMF::Read() 24 24 175 KiB 3000 KiB FabArray::setVal() 66 66 18 KiB 27 KiB MLPoisson::Fsmooth() 60 60 2649 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 37 B 10 KiB FillBoundary_nowait() 336 336 214 B 9648 B MLCellLinOp::applyBC() 1910 1910 170 B 9328 B amrex::Copy() 56 56 5261 B 8816 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B Gravity::get_old_grav_vector() 3 3 2299 B 3072 B Gravity::fill_multipole_BCs() 18 18 4 B 2832 B StateData::FillBoundary(geom) 960 960 34 B 2448 B MLMG::prepareForSolve() 7 7 703 B 1648 B amrex::average_down 37 37 420 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 19 B 400 B FabArray::norminf() 501 501 7 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2167 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.02-25-g2230caa24c7d) finalized