Initializing AMReX (24.03-8-gae3af4339425)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.03-8-gae3af4339425) initialized Starting run at 08:25:56 UTC on 2024-03-14. Successfully read inputs file ... Castro git describe: 24.03-6-g9ab585a31 AMReX git describe: 24.03-8-gae3af4339 Microphysics git describe: 24.03-7-g124b71be reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.045462422 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025566428 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.066883844 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.075647086 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.078198177 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.067444539 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.070490062 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.045541839 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.074130738 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.075741418 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.062759345 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.064924439 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.081990544 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.044064216 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.025332473 seconds Ending run at 08:25:57 UTC on 2024-03-14. Run time = 0.957345068 Run time without initialization = 0.83386595 Average number of zones advanced per microsecond: 3.144 Average number of zones advanced per microsecond per rank: 3.144 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.9574 ... 0.9574 ... 0.9574 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.3366 0.3366 0.3366 35.16% VisMF::Write(FabArray) 11 0.1775 0.1775 0.1775 18.54% MLCellLinOp::applyBC() 4298 0.08034 0.08034 0.08034 8.39% MLPoisson::Fsmooth() 3240 0.03345 0.03345 0.03345 3.49% FillBoundary_nowait() 3893 0.03016 0.03016 0.03016 3.15% StateData::FillBoundary(geom) 328 0.0267 0.0267 0.0267 2.79% amrex::Dot() 1100 0.02131 0.02131 0.02131 2.23% Castro::normalize_species() 62 0.02101 0.02101 0.02101 2.19% FabArray::norminf() 1048 0.01982 0.01982 0.01982 2.07% Castro::computeTemp() 63 0.01704 0.01704 0.01704 1.78% FabArray::ParallelCopy_nowait() 851 0.01366 0.01366 0.01366 1.43% FabArray::setVal() 1054 0.01348 0.01348 0.01348 1.41% FabArray::Saxpy() 1353 0.01308 0.01308 0.01308 1.37% Castro::enforce_min_density() 62 0.01214 0.01214 0.01214 1.27% StateDataPhysBCFunct::() 41 0.0121 0.0121 0.0121 1.26% amrex::Copy() 469 0.01094 0.01094 0.01094 1.14% MLCellLinOp::defineAuxData() 11 0.01028 0.01028 0.01028 1.07% MLPoisson::Fapply() 1047 0.01025 0.01025 0.01025 1.07% Gravity::fill_multipole_BCs() 11 0.009087 0.009087 0.009087 0.95% FabArray::Xpay() 730 0.007812 0.007812 0.007812 0.82% MLMG::addInterpCorrection() 405 0.007043 0.007043 0.007043 0.74% amrex::average_down 405 0.006179 0.006179 0.006179 0.65% Amr::checkPoint() 3 0.005915 0.005915 0.005915 0.62% Castro::estTimeStep() 21 0.005624 0.005624 0.005624 0.59% Castro::reset_internal_energy(MultiFab) 63 0.00517 0.00517 0.00517 0.54% BndryData::define() 11 0.004026 0.004026 0.004026 0.42% amrex::Add() 81 0.003587 0.003587 0.003587 0.37% Castro::construct_new_gravity_source() 10 0.003359 0.003359 0.003359 0.35% Castro::construct_old_gravity_source() 10 0.002898 0.002898 0.002898 0.30% Castro::enforce_speed_limit() 62 0.002221 0.002221 0.002221 0.23% check_for_negative_density() 10 0.002203 0.002203 0.002203 0.23% Amr::writePlotFile() 2 0.002124 0.002124 0.002124 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001826 0.001826 0.001826 0.19% Castro::reset_internal_energy(Fab) 504 0.001732 0.001732 0.001732 0.18% MLCGSolver::bicgstab 81 0.001642 0.001642 0.001642 0.17% MLCellLinOp::setLevelBC() 11 0.001603 0.001603 0.001603 0.17% Castro::initData() 1 0.001542 0.001542 0.001542 0.16% Gravity::actual_solve_with_mlmg() 11 0.001536 0.001536 0.001536 0.16% FabArray::setDomainBndry() 41 0.001387 0.001387 0.001387 0.14% FabArray::mult() 43 0.001384 0.001384 0.001384 0.14% MLCellLinOp::prepareForSolve() 11 0.001335 0.001335 0.001335 0.14% MultiFab::contains_nan() 20 0.001293 0.001293 0.001293 0.14% MLCellLinOp::smooth() 1620 0.001129 0.001129 0.001129 0.12% MLCellLinOp::compGrad() 11 0.001086 0.001086 0.001086 0.11% MLMG::prepareForSolve() 11 0.0009753 0.0009753 0.0009753 0.10% FabArrayBase::getCPC() 1313 0.0007748 0.0007748 0.0007748 0.08% FabArray::FillBoundary() 3893 0.0007589 0.0007589 0.0007589 0.08% FabArrayBase::CPC::define() 454 0.0006579 0.0006579 0.0006579 0.07% Gravity::get_new_grav_vector() 11 0.0006221 0.0006221 0.0006221 0.06% FabArrayBase::getFB() 3893 0.0006183 0.0006183 0.0006183 0.06% Amr::InitAmr() 1 0.0005773 0.0005773 0.0005773 0.06% Gravity::get_old_grav_vector() 10 0.0004857 0.0004857 0.0004857 0.05% AmrLevel::FillPatch() 41 0.0004116 0.0004116 0.0004116 0.04% MLCellLinOp::apply() 1047 0.0004068 0.0004068 0.0004068 0.04% main() 1 0.0003923 0.0003923 0.0003923 0.04% Amr::coarseTimeStep() 10 0.0003716 0.0003716 0.0003716 0.04% MultiFab::max() 11 0.0003317 0.0003317 0.0003317 0.03% MLCGSolver::ParallelAllReduce 1809 0.0003197 0.0003197 0.0003197 0.03% MLCellLinOp::defineBC() 11 0.0002745 0.0002745 0.0002745 0.03% FabArray::ParallelCopy() 851 0.0002406 0.0002406 0.0002406 0.03% FillPatchIterator::Initialize 41 0.0002267 0.0002267 0.0002267 0.02% MLMG::mgVcycle() 81 0.0002052 0.0002052 0.0002052 0.02% Castro::subcycle_advance_ctu() 10 0.0001973 0.0001973 0.0001973 0.02% MLCellLinOp::correctionResidual() 405 0.0001925 0.0001925 0.0001925 0.02% Castro::create_source_corrector() 10 0.0001879 0.0001879 0.0001879 0.02% MLLinOp::defineGrids() 11 0.0001819 0.0001819 0.0001819 0.02% Amr::timeStep() 10 0.0001515 0.0001515 0.0001515 0.02% Gravity::update_max_rhs() 11 0.0001273 0.0001273 0.0001273 0.01% Gravity::solve_for_phi() 10 0.0001253 0.0001253 0.0001253 0.01% MLMG:computeResOfCorrection() 405 0.000119 0.000119 0.000119 0.01% StateData::checkPoint() 12 0.0001104 0.0001104 0.0001104 0.01% Castro::Castro() 1 0.000104 0.000104 0.000104 0.01% Castro::advance() 10 9.56e-05 9.56e-05 9.56e-05 0.01% MLMG::mgVcycle_down::0 81 9.126e-05 9.126e-05 9.126e-05 0.01% Castro::initialize_advance() 10 9.081e-05 9.081e-05 9.081e-05 0.01% FabArrayBase::FB::FB() 56 8.572e-05 8.572e-05 8.572e-05 0.01% MLMG::mgVcycle_down::1 81 8.437e-05 8.437e-05 8.437e-05 0.01% MLMG::actualBottomSolve() 81 8.312e-05 8.312e-05 8.312e-05 0.01% MLMG::mgVcycle_down::2 81 8.057e-05 8.057e-05 8.057e-05 0.01% MLMG::mgVcycle_down::4 81 8.035e-05 8.035e-05 8.035e-05 0.01% Castro::do_new_sources() 10 7.632e-05 7.632e-05 7.632e-05 0.01% Castro::construct_new_source() 50 7.568e-05 7.568e-05 7.568e-05 0.01% MLMG::mgVcycle_down::3 81 7.421e-05 7.421e-05 7.421e-05 0.01% Castro::clean_state() 62 7.392e-05 7.392e-05 7.392e-05 0.01% MLMG::solve() 11 7.133e-05 7.133e-05 7.133e-05 0.01% Castro::finalize_advance() 10 6.725e-05 6.725e-05 6.725e-05 0.01% Castro::enforce_consistent_e() 1 6.628e-05 6.628e-05 6.628e-05 0.01% AmrLevel::checkPoint() 3 6.139e-05 6.139e-05 6.139e-05 0.01% Castro::initialize_do_advance() 10 5.933e-05 5.933e-05 5.933e-05 0.01% MLMG::mgVcycle_up::4 81 5.78e-05 5.78e-05 5.78e-05 0.01% MLMG::oneIter() 81 5.743e-05 5.743e-05 5.743e-05 0.01% MLMG::mgVcycle_up::0 81 5.247e-05 5.247e-05 5.247e-05 0.01% MLCellLinOp::solutionResidual() 92 5.067e-05 5.067e-05 5.067e-05 0.01% FillPatchIterator::FillFromLevel0() 41 5.06e-05 5.06e-05 5.06e-05 0.01% MLMG::mgVcycle_up::1 81 5.059e-05 5.059e-05 5.059e-05 0.01% MLMG::mgVcycle_up::3 81 4.995e-05 4.995e-05 4.995e-05 0.01% MLMG::mgVcycle_up::2 81 4.871e-05 4.871e-05 4.871e-05 0.01% Castro::do_advance_ctu() 10 4.691e-05 4.691e-05 4.691e-05 0.00% StateData::define() 4 4.528e-05 4.528e-05 4.528e-05 0.00% Castro::post_timestep() 10 4.11e-05 4.11e-05 4.11e-05 0.00% Castro::finalize_do_advance() 10 4.076e-05 4.076e-05 4.076e-05 0.00% Castro::swap_state_time_levels() 10 3.771e-05 3.771e-05 3.771e-05 0.00% Amr::writeSmallPlotFile() 1 3.585e-05 3.585e-05 3.585e-05 0.00% FillPatchSingleLevel 41 3.33e-05 3.33e-05 3.33e-05 0.00% MLMG::ResNormInf() 92 3.305e-05 3.305e-05 3.305e-05 0.00% Castro::initMFs() 1 3.3e-05 3.3e-05 3.3e-05 0.00% MLMG::mgVcycle_bottom 81 3.22e-05 3.22e-05 3.22e-05 0.00% Amr::defBaseLevel() 1 3.193e-05 3.193e-05 3.193e-05 0.00% MLMG::computeResidual() 81 3.118e-05 3.118e-05 3.118e-05 0.00% Castro::construct_new_gravity() 10 3.033e-05 3.033e-05 3.033e-05 0.00% Castro::buildMetrics() 1 2.766e-05 2.766e-05 2.766e-05 0.00% makeSFC 55 2.537e-05 2.537e-05 2.537e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.35e-05 2.35e-05 2.35e-05 0.00% Castro::do_old_sources() 10 2.215e-05 2.215e-05 2.215e-05 0.00% Castro::construct_old_source() 50 1.971e-05 1.971e-05 1.971e-05 0.00% Amr::FinalizeInit() 1 1.943e-05 1.943e-05 1.943e-05 0.00% MLPoisson::define() 11 1.923e-05 1.923e-05 1.923e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.813e-05 1.813e-05 1.813e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.81e-05 1.81e-05 1.81e-05 0.00% DistributionMapping::Distribute() 56 1.744e-05 1.744e-05 1.744e-05 0.00% MLMG::computeMLResidual() 11 1.453e-05 1.453e-05 1.453e-05 0.00% MLPoisson::prepareForSolve() 11 1.398e-05 1.398e-05 1.398e-05 0.00% Castro::check_for_nan() 20 1.355e-05 1.355e-05 1.355e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.335e-05 1.335e-05 1.335e-05 0.00% Castro::apply_source_to_state() 20 1.275e-05 1.275e-05 1.275e-05 0.00% MLLinOp::define() 11 1.252e-05 1.252e-05 1.252e-05 0.00% Gravity::swapTimeLevels() 10 1.122e-05 1.122e-05 1.122e-05 0.00% Amr::initSubcycle() 1 1.073e-05 1.073e-05 1.073e-05 0.00% Castro::construct_old_gravity() 10 1.07e-05 1.07e-05 1.07e-05 0.00% Castro::post_init() 1 9.853e-06 9.853e-06 9.853e-06 0.00% Gravity::actual_multilevel_solve() 1 8.708e-06 8.708e-06 8.708e-06 0.00% Amr::InitializeInit() 1 7.604e-06 7.604e-06 7.604e-06 0.00% Castro::computeNewDt() 9 7.258e-06 7.258e-06 7.258e-06 0.00% MLMG::getGradSolution() 11 6.761e-06 6.761e-06 6.761e-06 0.00% Castro::expand_state() 10 6.614e-06 6.614e-06 6.614e-06 0.00% AmrLevel::checkPointPost() 3 4.834e-06 4.834e-06 4.834e-06 0.00% Castro::retry_advance_ctu() 10 4.491e-06 4.491e-06 4.491e-06 0.00% MLMG::MLRhsNormInf() 11 3.997e-06 3.997e-06 3.997e-06 0.00% Gravity::set_mass_offset() 11 3.907e-06 3.907e-06 3.907e-06 0.00% MLMG::MLResNormInf() 11 3.401e-06 3.401e-06 3.401e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.35e-06 3.35e-06 3.35e-06 0.00% Castro::FluxRegCrseInit 10 2.87e-06 2.87e-06 2.87e-06 0.00% Amr::init() 1 2.782e-06 2.782e-06 2.782e-06 0.00% Castro::computeInitialDt() 2 2.773e-06 2.773e-06 2.773e-06 0.00% Castro::FluxRegFineAdd() 10 2.259e-06 2.259e-06 2.259e-06 0.00% AmrLevel::checkPointPre() 3 2.215e-06 2.215e-06 2.215e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.914e-06 1.914e-06 1.914e-06 0.00% Castro::post_regrid() 1 1.194e-06 1.194e-06 1.194e-06 0.00% Amr::initialInit() 1 1.132e-06 1.132e-06 1.132e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9574 0.9574 0.9574 100.00% Amr::coarseTimeStep() 10 0.8083 0.8083 0.8083 84.43% Amr::timeStep() 10 0.7151 0.7151 0.7151 74.69% Castro::advance() 10 0.7046 0.7046 0.7046 73.59% Castro::subcycle_advance_ctu() 10 0.69 0.69 0.69 72.07% Castro::do_advance_ctu() 10 0.6898 0.6898 0.6898 72.05% Castro::construct_ctu_hydro_source() 10 0.3493 0.3493 0.3493 36.48% Gravity::solve_phi_with_mlmg() 11 0.2927 0.2927 0.2927 30.58% Gravity::actual_solve_with_mlmg() 11 0.2832 0.2832 0.2832 29.58% Castro::construct_new_gravity() 10 0.2671 0.2671 0.2671 27.90% MLMG::solve() 11 0.2611 0.2611 0.2611 27.28% Gravity::solve_for_phi() 10 0.2507 0.2507 0.2507 26.18% MLMG::oneIter() 81 0.2457 0.2457 0.2457 25.67% MLMG::mgVcycle() 81 0.2421 0.2421 0.2421 25.29% VisMF::Write(FabArray) 11 0.1775 0.1775 0.1775 18.54% Amr::checkPoint() 3 0.1352 0.1352 0.1352 14.13% AmrLevel::checkPoint() 3 0.1293 0.1293 0.1293 13.51% StateData::checkPoint() 12 0.1293 0.1293 0.1293 13.50% Amr::init() 1 0.1227 0.1227 0.1227 12.81% MLCellLinOp::smooth() 1620 0.1207 0.1207 0.1207 12.61% MLCellLinOp::applyBC() 4298 0.112 0.112 0.112 11.69% MLMG::mgVcycle_bottom 81 0.07252 0.07252 0.07252 7.57% MLMG::actualBottomSolve() 81 0.07248 0.07248 0.07248 7.57% MLCGSolver::bicgstab 81 0.07168 0.07168 0.07168 7.49% Castro::clean_state() 62 0.05853 0.05853 0.05853 6.11% Amr::initialInit() 1 0.05149 0.05149 0.05149 5.38% Amr::writePlotFile() 2 0.05103 0.05103 0.05103 5.33% AmrLevel::FillPatch() 41 0.04888 0.04888 0.04888 5.11% Amr::FinalizeInit() 1 0.04633 0.04633 0.04633 4.84% Castro::post_init() 1 0.0449 0.0449 0.0449 4.69% FillPatchIterator::Initialize 41 0.04455 0.04455 0.04455 4.65% FillPatchIterator::FillFromLevel0() 41 0.04294 0.04294 0.04294 4.49% FillPatchSingleLevel 41 0.04289 0.04289 0.04289 4.48% Gravity::multilevel_solve_for_new_phi() 1 0.04251 0.04251 0.04251 4.44% Gravity::actual_multilevel_solve() 1 0.0425 0.0425 0.0425 4.44% StateDataPhysBCFunct::() 41 0.0388 0.0388 0.0388 4.05% MLCellLinOp::apply() 1047 0.03597 0.03597 0.03597 3.76% MLMG::mgVcycle_down::0 81 0.03435 0.03435 0.03435 3.59% MLPoisson::Fsmooth() 3240 0.03345 0.03345 0.03345 3.49% FabArray::FillBoundary() 3893 0.03162 0.03162 0.03162 3.30% FillBoundary_nowait() 3893 0.03086 0.03086 0.03086 3.22% StateData::FillBoundary(geom) 328 0.0267 0.0267 0.0267 2.79% MLMG::mgVcycle_up::0 81 0.02586 0.02586 0.02586 2.70% Castro::computeTemp() 63 0.02395 0.02395 0.02395 2.50% amrex::Dot() 1100 0.02131 0.02131 0.02131 2.23% Castro::initialize_do_advance() 10 0.02121 0.02121 0.02121 2.22% Castro::normalize_species() 62 0.02101 0.02101 0.02101 2.19% MLMG:computeResOfCorrection() 405 0.02026 0.02026 0.02026 2.12% MLCellLinOp::correctionResidual() 405 0.02014 0.02014 0.02014 2.10% FabArray::norminf() 1048 0.01982 0.01982 0.01982 2.07% Castro::do_old_sources() 10 0.01966 0.01966 0.01966 2.05% Gravity::get_new_grav_vector() 11 0.01833 0.01833 0.01833 1.91% MLPoisson::define() 11 0.01732 0.01732 0.01732 1.81% MLMG::mgVcycle_down::1 81 0.01656 0.01656 0.01656 1.73% MLMG::mgVcycle_down::2 81 0.01539 0.01539 0.01539 1.61% Castro::construct_old_gravity() 10 0.01528 0.01528 0.01528 1.60% Gravity::get_old_grav_vector() 10 0.01527 0.01527 0.01527 1.59% MLMG::mgVcycle_down::3 81 0.01503 0.01503 0.01503 1.57% MLMG::mgVcycle_down::4 81 0.01493 0.01493 0.01493 1.56% FabArray::ParallelCopy() 851 0.01469 0.01469 0.01469 1.53% FabArray::ParallelCopy_nowait() 851 0.01445 0.01445 0.01445 1.51% Castro::do_new_sources() 10 0.01406 0.01406 0.01406 1.47% Castro::initialize_advance() 10 0.01384 0.01384 0.01384 1.45% FabArray::setVal() 1054 0.01348 0.01348 0.01348 1.41% FabArray::Saxpy() 1353 0.01308 0.01308 0.01308 1.37% MLCGSolver::ParallelAllReduce 1809 0.01279 0.01279 0.01279 1.34% MLMG::addInterpCorrection() 405 0.01235 0.01235 0.01235 1.29% Castro::enforce_min_density() 62 0.01214 0.01214 0.01214 1.27% Castro::expand_state() 10 0.01205 0.01205 0.01205 1.26% MLMG::mgVcycle_up::1 81 0.01201 0.01201 0.01201 1.25% MLMG::mgVcycle_up::4 81 0.01194 0.01194 0.01194 1.25% MLMG::mgVcycle_up::2 81 0.01177 0.01177 0.01177 1.23% MLCellLinOp::defineAuxData() 11 0.01172 0.01172 0.01172 1.22% MLMG::mgVcycle_up::3 81 0.01154 0.01154 0.01154 1.21% amrex::average_down 405 0.01151 0.01151 0.01151 1.20% amrex::Copy() 469 0.01094 0.01094 0.01094 1.14% Castro::post_timestep() 10 0.01037 0.01037 0.01037 1.08% MLPoisson::Fapply() 1047 0.01025 0.01025 0.01025 1.07% Gravity::fill_multipole_BCs() 11 0.00932 0.00932 0.00932 0.97% FabArray::Xpay() 730 0.007812 0.007812 0.007812 0.82% MLCellLinOp::solutionResidual() 92 0.007659 0.007659 0.007659 0.80% Castro::reset_internal_energy(MultiFab) 63 0.006902 0.006902 0.006902 0.72% MLMG::computeResidual() 81 0.006398 0.006398 0.006398 0.67% Castro::estTimeStep() 21 0.005624 0.005624 0.005624 0.59% MLCellLinOp::defineBC() 11 0.005324 0.005324 0.005324 0.56% MLMG::prepareForSolve() 11 0.005215 0.005215 0.005215 0.54% Amr::InitializeInit() 1 0.005162 0.005162 0.005162 0.54% Amr::defBaseLevel() 1 0.005154 0.005154 0.005154 0.54% BndryData::define() 11 0.00505 0.00505 0.00505 0.53% Castro::initData() 1 0.004445 0.004445 0.004445 0.46% amrex::Add() 81 0.003587 0.003587 0.003587 0.37% Castro::construct_new_source() 50 0.003435 0.003435 0.003435 0.36% Castro::construct_new_gravity_source() 10 0.003359 0.003359 0.003359 0.35% Castro::construct_old_source() 50 0.002918 0.002918 0.002918 0.30% Castro::construct_old_gravity_source() 10 0.002898 0.002898 0.002898 0.30% Castro::computeNewDt() 9 0.002403 0.002403 0.002403 0.25% Castro::finalize_do_advance() 10 0.002297 0.002297 0.002297 0.24% Castro::enforce_speed_limit() 62 0.002221 0.002221 0.002221 0.23% check_for_negative_density() 10 0.002203 0.002203 0.002203 0.23% MLMG::ResNormInf() 92 0.002158 0.002158 0.002158 0.23% Castro::apply_source_to_state() 20 0.001874 0.001874 0.001874 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001826 0.001826 0.001826 0.19% Castro::reset_internal_energy(Fab) 504 0.001732 0.001732 0.001732 0.18% MLCellLinOp::setLevelBC() 11 0.001603 0.001603 0.001603 0.17% MLMG::getGradSolution() 11 0.001599 0.001599 0.001599 0.17% MLCellLinOp::compGrad() 11 0.001593 0.001593 0.001593 0.17% FabArrayBase::getCPC() 1313 0.001433 0.001433 0.001433 0.15% FabArray::setDomainBndry() 41 0.001387 0.001387 0.001387 0.14% FabArray::mult() 43 0.001384 0.001384 0.001384 0.14% MLPoisson::prepareForSolve() 11 0.001349 0.001349 0.001349 0.14% MLCellLinOp::prepareForSolve() 11 0.001335 0.001335 0.001335 0.14% Castro::check_for_nan() 20 0.001306 0.001306 0.001306 0.14% MLMG::computeMLResidual() 11 0.001306 0.001306 0.001306 0.14% MultiFab::contains_nan() 20 0.001293 0.001293 0.001293 0.14% Castro::post_regrid() 1 0.001137 0.001137 0.001137 0.12% Gravity::update_max_rhs() 11 0.00099 0.00099 0.00099 0.10% Castro::computeInitialDt() 2 0.0009745 0.0009745 0.0009745 0.10% FabArrayBase::getFB() 3893 0.000704 0.000704 0.000704 0.07% FabArrayBase::CPC::define() 454 0.0006579 0.0006579 0.0006579 0.07% Castro::finalize_advance() 10 0.0006176 0.0006176 0.0006176 0.06% Castro::Castro() 1 0.00061 0.00061 0.00061 0.06% Amr::InitAmr() 1 0.0005881 0.0005881 0.0005881 0.06% Gravity::swapTimeLevels() 10 0.0004837 0.0004837 0.0004837 0.05% MultiFab::max() 11 0.0003317 0.0003317 0.0003317 0.03% MLMG::MLResNormInf() 11 0.0003273 0.0003273 0.0003273 0.03% Castro::buildMetrics() 1 0.0002976 0.0002976 0.0002976 0.03% MLLinOp::define() 11 0.0002515 0.0002515 0.0002515 0.03% MLLinOp::defineGrids() 11 0.000239 0.000239 0.000239 0.02% MLMG::MLRhsNormInf() 11 0.0002303 0.0002303 0.0002303 0.02% Castro::create_source_corrector() 10 0.0001879 0.0001879 0.0001879 0.02% FabArrayBase::FB::FB() 56 8.572e-05 8.572e-05 8.572e-05 0.01% Castro::enforce_consistent_e() 1 6.628e-05 6.628e-05 6.628e-05 0.01% AmrLevel::AmrLevel(dm) 1 6.337e-05 6.337e-05 6.337e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.511e-05 5.511e-05 5.511e-05 0.01% StateData::define() 4 4.528e-05 4.528e-05 4.528e-05 0.00% makeSFC 55 4.176e-05 4.176e-05 4.176e-05 0.00% Castro::swap_state_time_levels() 10 3.771e-05 3.771e-05 3.771e-05 0.00% Amr::writeSmallPlotFile() 1 3.585e-05 3.585e-05 3.585e-05 0.00% Castro::initMFs() 1 3.3e-05 3.3e-05 3.3e-05 0.00% DistributionMapping::Distribute() 56 1.744e-05 1.744e-05 1.744e-05 0.00% Amr::initSubcycle() 1 1.073e-05 1.073e-05 1.073e-05 0.00% AmrLevel::checkPointPost() 3 4.834e-06 4.834e-06 4.834e-06 0.00% Castro::retry_advance_ctu() 10 4.491e-06 4.491e-06 4.491e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.397e-06 4.397e-06 4.397e-06 0.00% Gravity::set_mass_offset() 11 3.907e-06 3.907e-06 3.907e-06 0.00% Castro::FluxRegCrseInit 10 2.87e-06 2.87e-06 2.87e-06 0.00% Castro::FluxRegFineAdd() 10 2.259e-06 2.259e-06 2.259e-06 0.00% AmrLevel::checkPointPre() 3 2.215e-06 2.215e-06 2.215e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.914e-06 1.914e-06 1.914e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 5513 KiB 9037 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 48 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 996 KiB 39 MiB Castro::initialize_do_advance() 80 80 28 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1488 KiB 28 MiB Castro::initialize_advance() 80 80 17 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7615 KiB 14 MiB MLMG::prepareForSolve() 660 660 3354 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 199 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 163 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7518 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 16 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2002 B 2048 KiB Gravity::solve_for_phi() 80 80 535 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 90 KiB 2048 KiB BndryData::define() 1056 1056 307 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 195 KiB 671 KiB Castro::estTimeStep() 21 21 2862 B 480 KiB VisMF::Write(FabArray) 656 656 3228 B 320 KiB Castro::normalize_species() 62 62 7145 B 320 KiB amrex::average_down 1054 1054 1550 B 257 KiB MLMG::addInterpCorrection() 1053 1053 1100 B 257 KiB amrex::Dot() 1343 1343 3285 B 160 KiB FabArray::norminf() 1129 1129 3194 B 160 KiB check_for_negative_density() 10 10 364 B 160 KiB Castro::initData() 1 1 49 B 160 KiB MultiFab::max() 11 11 54 B 160 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MultiFab::contains_nan() 20 20 26 B 20 KiB MLPoisson::Fsmooth() 132 132 3273 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 41 B 10 KiB FillBoundary_nowait() 751 751 275 B 9648 B MLCellLinOp::applyBC() 8596 8596 209 B 9344 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3941 B 6144 B StateData::FillBoundary(geom) 1992 1992 44 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 345 B 1248 B MLCGSolver::bicgstab 405 405 90 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 548 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 38 KiB 8192 KiB VisMF::Write(FabArray) 744 744 406 KiB 3584 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MLPoisson::Fsmooth() 132 132 3273 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 41 B 10 KiB FillBoundary_nowait() 751 751 275 B 9648 B MLCellLinOp::applyBC() 4298 4298 207 B 9328 B MLCellLinOp::prepareForSolve() 66 66 3 B 7792 B amrex::Copy() 100 100 3941 B 6144 B Gravity::get_new_grav_vector() 3 3 2908 B 3072 B StateData::FillBoundary(geom) 1992 1992 44 B 3024 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B amrex::average_down 82 82 617 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLMG::addInterpCorrection() 81 81 2 B 1024 B MLMG::prepareForSolve() 11 11 279 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1343 1343 24 B 400 B FabArray::norminf() 1129 1129 9 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2180 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.03-8-gae3af4339425) finalized Initializing AMReX (24.03-8-gae3af4339425)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.03-8-gae3af4339425) initialized Starting run at 08:25:57 UTC on 2024-03-14. Successfully read inputs file ... Castro git describe: 24.03-6-g9ab585a31 AMReX git describe: 24.03-8-gae3af4339 Microphysics git describe: 24.03-7-g124b71be reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.527633014 Restart time = 0.071500294 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.068627926 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048461215 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.071702042 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.07515284 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.071756641 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.027386541 seconds Ending run at 08:25:58 UTC on 2024-03-14. Run time = 0.435818366 Run time without initialization = 0.36365476 Average number of zones advanced per microsecond: 3.604 Average number of zones advanced per microsecond per rank: 3.604 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.4358 ... 0.4358 ... 0.4358 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1406 0.1406 0.1406 32.26% VisMF::Read() 3 0.06028 0.06028 0.06028 13.83% MLCellLinOp::applyBC() 1910 0.03574 0.03574 0.03574 8.20% VisMF::Write(FabArray) 1 0.02478 0.02478 0.02478 5.69% MLPoisson::Fsmooth() 1440 0.01478 0.01478 0.01478 3.39% StateData::FillBoundary(geom) 160 0.01316 0.01316 0.01316 3.02% FillBoundary_nowait() 1730 0.01297 0.01297 0.01297 2.98% Castro::normalize_species() 30 0.01072 0.01072 0.01072 2.46% amrex::Dot() 484 0.00928 0.00928 0.00928 2.13% FabArray::norminf() 465 0.008762 0.008762 0.008762 2.01% Castro::computeTemp() 30 0.007324 0.007324 0.007324 1.68% FabArray::setVal() 501 0.006633 0.006633 0.006633 1.52% Castro::enforce_min_density() 30 0.006394 0.006394 0.006394 1.47% FabArray::ParallelCopy_nowait() 380 0.006241 0.006241 0.006241 1.43% FabArray::Saxpy() 597 0.005923 0.005923 0.005923 1.36% MLCellLinOp::defineAuxData() 6 0.005788 0.005788 0.005788 1.33% amrex::Copy() 221 0.005497 0.005497 0.005497 1.26% Gravity::fill_multipole_BCs() 6 0.00498 0.00498 0.00498 1.14% Amr::restart() 1 0.004742 0.004742 0.004742 1.09% StateDataPhysBCFunct::() 20 0.004654 0.004654 0.004654 1.07% MLPoisson::Fapply() 464 0.004536 0.004536 0.004536 1.04% FabArray::Xpay() 325 0.003538 0.003538 0.003538 0.81% Castro::estTimeStep() 10 0.003335 0.003335 0.003335 0.77% MLMG::addInterpCorrection() 180 0.00316 0.00316 0.00316 0.72% amrex::average_down 180 0.002769 0.002769 0.002769 0.64% Amr::writePlotFile() 1 0.00251 0.00251 0.00251 0.58% BndryData::define() 6 0.002222 0.002222 0.002222 0.51% Castro::reset_internal_energy(MultiFab) 30 0.001973 0.001973 0.001973 0.45% Castro::enforce_speed_limit() 30 0.001934 0.001934 0.001934 0.44% Castro::construct_new_gravity_source() 5 0.001614 0.001614 0.001614 0.37% amrex::Add() 36 0.001555 0.001555 0.001555 0.36% Castro::construct_old_gravity_source() 5 0.001206 0.001206 0.001206 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009953 0.0009953 0.0009953 0.23% check_for_negative_density() 5 0.0009725 0.0009725 0.0009725 0.22% MLCellLinOp::setLevelBC() 6 0.0008891 0.0008891 0.0008891 0.20% Castro::reset_internal_energy(Fab) 240 0.0008644 0.0008644 0.0008644 0.20% Gravity::actual_solve_with_mlmg() 6 0.0008348 0.0008348 0.0008348 0.19% MLCellLinOp::prepareForSolve() 6 0.0007524 0.0007524 0.0007524 0.17% FabArray::mult() 22 0.0007066 0.0007066 0.0007066 0.16% FabArray::setDomainBndry() 20 0.0007011 0.0007011 0.0007011 0.16% MLCGSolver::bicgstab 36 0.0006996 0.0006996 0.0006996 0.16% MultiFab::contains_nan() 10 0.0006618 0.0006618 0.0006618 0.15% MLCellLinOp::compGrad() 6 0.0006071 0.0006071 0.0006071 0.14% MLMG::prepareForSolve() 6 0.0005492 0.0005492 0.0005492 0.13% MLCellLinOp::smooth() 720 0.0005003 0.0005003 0.0005003 0.11% Amr::InitAmr() 1 0.0004724 0.0004724 0.0004724 0.11% FabArrayBase::CPC::define() 244 0.0004186 0.0004186 0.0004186 0.10% FabArrayBase::getCPC() 632 0.0003663 0.0003663 0.0003663 0.08% Gravity::get_old_grav_vector() 5 0.000355 0.000355 0.000355 0.08% FabArray::FillBoundary() 1730 0.0003416 0.0003416 0.0003416 0.08% main() 1 0.0002677 0.0002677 0.0002677 0.06% FabArrayBase::getFB() 1730 0.0002649 0.0002649 0.0002649 0.06% Gravity::get_new_grav_vector() 5 0.0002624 0.0002624 0.0002624 0.06% Amr::coarseTimeStep() 5 0.0002437 0.0002437 0.0002437 0.06% MultiFab::max() 6 0.00022 0.00022 0.00022 0.05% AmrLevel::FillPatch() 20 0.0001985 0.0001985 0.0001985 0.05% MLCellLinOp::apply() 464 0.000181 0.000181 0.000181 0.04% MLCellLinOp::defineBC() 6 0.0001501 0.0001501 0.0001501 0.03% MLCGSolver::ParallelAllReduce 798 0.0001389 0.0001389 0.0001389 0.03% FillPatchIterator::Initialize 20 0.0001143 0.0001143 0.0001143 0.03% MLLinOp::defineGrids() 6 0.0001121 0.0001121 0.0001121 0.03% FabArray::ParallelCopy() 380 0.0001109 0.0001109 0.0001109 0.03% Castro::create_source_corrector() 5 0.000102 0.000102 0.000102 0.02% Castro::do_new_sources() 5 9.883e-05 9.883e-05 9.883e-05 0.02% Castro::subcycle_advance_ctu() 5 9.323e-05 9.323e-05 9.323e-05 0.02% Castro::initialize_do_advance() 5 8.708e-05 8.708e-05 8.708e-05 0.02% MLMG::mgVcycle() 36 8.58e-05 8.58e-05 8.58e-05 0.02% Amr::timeStep() 5 8.415e-05 8.415e-05 8.415e-05 0.02% MLCellLinOp::correctionResidual() 180 8.137e-05 8.137e-05 8.137e-05 0.02% Castro::construct_new_source() 25 7.607e-05 7.607e-05 7.607e-05 0.02% AmrLevel::restart() 1 7.595e-05 7.595e-05 7.595e-05 0.02% Castro::advance() 5 7.565e-05 7.565e-05 7.565e-05 0.02% StateData::restartDoit() 4 7.255e-05 7.255e-05 7.255e-05 0.02% Gravity::update_max_rhs() 6 6.739e-05 6.739e-05 6.739e-05 0.02% FabArrayBase::FB::FB() 26 6.515e-05 6.515e-05 6.515e-05 0.01% Gravity::solve_for_phi() 5 6.243e-05 6.243e-05 6.243e-05 0.01% Castro::do_advance_ctu() 5 6.214e-05 6.214e-05 6.214e-05 0.01% MLMG:computeResOfCorrection() 180 5.206e-05 5.206e-05 5.206e-05 0.01% Castro::construct_old_source() 25 4.879e-05 4.879e-05 4.879e-05 0.01% MLMG::mgVcycle_down::0 36 4.534e-05 4.534e-05 4.534e-05 0.01% Castro::finalize_do_advance() 5 4.386e-05 4.386e-05 4.386e-05 0.01% Castro::initialize_advance() 5 3.795e-05 3.795e-05 3.795e-05 0.01% MLMG::actualBottomSolve() 36 3.71e-05 3.71e-05 3.71e-05 0.01% MLMG::mgVcycle_down::1 36 3.66e-05 3.66e-05 3.66e-05 0.01% MLMG::solve() 6 3.507e-05 3.507e-05 3.507e-05 0.01% MLMG::mgVcycle_down::2 36 3.392e-05 3.392e-05 3.392e-05 0.01% Castro::clean_state() 30 3.368e-05 3.368e-05 3.368e-05 0.01% MLMG::mgVcycle_down::4 36 3.129e-05 3.129e-05 3.129e-05 0.01% Amr::writeSmallPlotFile() 1 3.109e-05 3.109e-05 3.109e-05 0.01% Castro::buildMetrics() 1 3.068e-05 3.068e-05 3.068e-05 0.01% MLMG::mgVcycle_down::3 36 3.068e-05 3.068e-05 3.068e-05 0.01% Castro::finalize_advance() 5 3.055e-05 3.055e-05 3.055e-05 0.01% Castro::post_timestep() 5 2.896e-05 2.896e-05 2.896e-05 0.01% Castro::post_restart() 1 2.87e-05 2.87e-05 2.87e-05 0.01% MLMG::mgVcycle_up::4 36 2.763e-05 2.763e-05 2.763e-05 0.01% Castro::initMFs() 1 2.549e-05 2.549e-05 2.549e-05 0.01% MLMG::oneIter() 36 2.471e-05 2.471e-05 2.471e-05 0.01% Castro::swap_state_time_levels() 5 2.419e-05 2.419e-05 2.419e-05 0.01% MLCellLinOp::solutionResidual() 42 2.414e-05 2.414e-05 2.414e-05 0.01% MLMG::mgVcycle_up::0 36 2.294e-05 2.294e-05 2.294e-05 0.01% MLMG::mgVcycle_up::3 36 2.245e-05 2.245e-05 2.245e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.185e-05 2.185e-05 2.185e-05 0.01% MLMG::mgVcycle_up::2 36 2.138e-05 2.138e-05 2.138e-05 0.00% MLMG::mgVcycle_up::1 36 2.055e-05 2.055e-05 2.055e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.685e-05 1.685e-05 1.685e-05 0.00% MLMG::ResNormInf() 42 1.666e-05 1.666e-05 1.666e-05 0.00% FillPatchSingleLevel 20 1.585e-05 1.585e-05 1.585e-05 0.00% MLMG::mgVcycle_bottom 36 1.508e-05 1.508e-05 1.508e-05 0.00% MLPoisson::define() 6 1.433e-05 1.433e-05 1.433e-05 0.00% makeSFC 30 1.387e-05 1.387e-05 1.387e-05 0.00% MLMG::computeResidual() 36 1.366e-05 1.366e-05 1.366e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.322e-05 1.322e-05 1.322e-05 0.00% Castro::construct_new_gravity() 5 1.286e-05 1.286e-05 1.286e-05 0.00% Castro::do_old_sources() 5 1.202e-05 1.202e-05 1.202e-05 0.00% MLPoisson::prepareForSolve() 6 1.024e-05 1.024e-05 1.024e-05 0.00% Amr::initSubcycle() 1 9.932e-06 9.932e-06 9.932e-06 0.00% MLMG::computeMLResidual() 6 9.662e-06 9.662e-06 9.662e-06 0.00% DistributionMapping::Distribute() 31 9.013e-06 9.013e-06 9.013e-06 0.00% Gravity::actual_multilevel_solve() 1 8.61e-06 8.61e-06 8.61e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.004e-06 8.004e-06 8.004e-06 0.00% Castro::check_for_nan() 10 6.955e-06 6.955e-06 6.955e-06 0.00% MLLinOp::define() 6 6.756e-06 6.756e-06 6.756e-06 0.00% Castro::apply_source_to_state() 10 6.375e-06 6.375e-06 6.375e-06 0.00% Castro::construct_old_gravity() 5 5.642e-06 5.642e-06 5.642e-06 0.00% Gravity::swapTimeLevels() 5 4.609e-06 4.609e-06 4.609e-06 0.00% Castro::computeNewDt() 5 3.512e-06 3.512e-06 3.512e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.408e-06 3.408e-06 3.408e-06 0.00% MLMG::getGradSolution() 6 3.181e-06 3.181e-06 3.181e-06 0.00% Castro::expand_state() 5 3.021e-06 3.021e-06 3.021e-06 0.00% Gravity::set_mass_offset() 6 2.539e-06 2.539e-06 2.539e-06 0.00% MLMG::MLResNormInf() 6 2.323e-06 2.323e-06 2.323e-06 0.00% MLMG::MLRhsNormInf() 6 2.213e-06 2.213e-06 2.213e-06 0.00% Castro::retry_advance_ctu() 5 1.903e-06 1.903e-06 1.903e-06 0.00% Castro::FluxRegCrseInit 5 1.625e-06 1.625e-06 1.625e-06 0.00% Castro::FluxRegFineAdd() 5 1.205e-06 1.205e-06 1.205e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.084e-06 1.084e-06 1.084e-06 0.00% Amr::init() 1 7.16e-07 7.16e-07 7.16e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4358 0.4358 0.4358 100.00% Amr::coarseTimeStep() 5 0.3359 0.3359 0.3359 77.08% Amr::timeStep() 5 0.3335 0.3335 0.3335 76.52% Castro::advance() 5 0.3276 0.3276 0.3276 75.17% Castro::subcycle_advance_ctu() 5 0.3194 0.3194 0.3194 73.29% Castro::do_advance_ctu() 5 0.3193 0.3193 0.3193 73.27% Castro::construct_ctu_hydro_source() 5 0.1462 0.1462 0.1462 33.55% Castro::construct_new_gravity() 5 0.1361 0.1361 0.1361 31.24% Gravity::solve_phi_with_mlmg() 6 0.1334 0.1334 0.1334 30.60% Gravity::actual_solve_with_mlmg() 6 0.1281 0.1281 0.1281 29.40% Gravity::solve_for_phi() 5 0.1281 0.1281 0.1281 29.38% MLMG::solve() 6 0.1158 0.1158 0.1158 26.57% MLMG::oneIter() 36 0.1081 0.1081 0.1081 24.81% MLMG::mgVcycle() 36 0.1065 0.1065 0.1065 24.44% Amr::init() 1 0.07157 0.07157 0.07157 16.42% Amr::restart() 1 0.07156 0.07156 0.07156 16.42% AmrLevel::restart() 1 0.06066 0.06066 0.06066 13.92% StateData::restartDoit() 4 0.06058 0.06058 0.06058 13.90% VisMF::Read() 3 0.06028 0.06028 0.06028 13.83% MLCellLinOp::smooth() 720 0.053 0.053 0.053 12.16% MLCellLinOp::applyBC() 1910 0.04938 0.04938 0.04938 11.33% MLMG::mgVcycle_bottom 36 0.03186 0.03186 0.03186 7.31% MLMG::actualBottomSolve() 36 0.03184 0.03184 0.03184 7.31% MLCGSolver::bicgstab 36 0.03147 0.03147 0.03147 7.22% Castro::clean_state() 30 0.02924 0.02924 0.02924 6.71% Amr::writePlotFile() 1 0.02755 0.02755 0.02755 6.32% VisMF::Write(FabArray) 1 0.02478 0.02478 0.02478 5.69% AmrLevel::FillPatch() 20 0.02283 0.02283 0.02283 5.24% FillPatchIterator::Initialize 20 0.02073 0.02073 0.02073 4.76% FillPatchIterator::FillFromLevel0() 20 0.01991 0.01991 0.01991 4.57% FillPatchSingleLevel 20 0.01989 0.01989 0.01989 4.56% StateDataPhysBCFunct::() 20 0.01781 0.01781 0.01781 4.09% MLCellLinOp::apply() 464 0.01609 0.01609 0.01609 3.69% MLMG::mgVcycle_down::0 36 0.01508 0.01508 0.01508 3.46% MLPoisson::Fsmooth() 1440 0.01478 0.01478 0.01478 3.39% FabArray::FillBoundary() 1730 0.01364 0.01364 0.01364 3.13% FillBoundary_nowait() 1730 0.0133 0.0133 0.0133 3.05% StateData::FillBoundary(geom) 160 0.01316 0.01316 0.01316 3.02% MLMG::mgVcycle_up::0 36 0.01131 0.01131 0.01131 2.59% Castro::initialize_do_advance() 5 0.01102 0.01102 0.01102 2.53% Castro::normalize_species() 30 0.01072 0.01072 0.01072 2.46% Castro::computeTemp() 30 0.01016 0.01016 0.01016 2.33% MLPoisson::define() 6 0.009712 0.009712 0.009712 2.23% Castro::do_old_sources() 5 0.009446 0.009446 0.009446 2.17% amrex::Dot() 484 0.00928 0.00928 0.00928 2.13% MLMG:computeResOfCorrection() 180 0.008973 0.008973 0.008973 2.06% MLCellLinOp::correctionResidual() 180 0.008921 0.008921 0.008921 2.05% FabArray::norminf() 465 0.008762 0.008762 0.008762 2.01% Gravity::get_new_grav_vector() 5 0.007961 0.007961 0.007961 1.83% Castro::initialize_advance() 5 0.007799 0.007799 0.007799 1.79% Castro::construct_old_gravity() 5 0.007765 0.007765 0.007765 1.78% Gravity::get_old_grav_vector() 5 0.007759 0.007759 0.007759 1.78% MLMG::mgVcycle_down::1 36 0.007413 0.007413 0.007413 1.70% Castro::do_new_sources() 5 0.006971 0.006971 0.006971 1.60% MLMG::mgVcycle_down::2 36 0.006832 0.006832 0.006832 1.57% FabArray::ParallelCopy() 380 0.006743 0.006743 0.006743 1.55% FabArray::setVal() 501 0.006633 0.006633 0.006633 1.52% MLMG::mgVcycle_down::3 36 0.006633 0.006633 0.006633 1.52% FabArray::ParallelCopy_nowait() 380 0.006632 0.006632 0.006632 1.52% MLCellLinOp::defineAuxData() 6 0.006592 0.006592 0.006592 1.51% MLMG::mgVcycle_down::4 36 0.006546 0.006546 0.006546 1.50% Castro::enforce_min_density() 30 0.006394 0.006394 0.006394 1.47% Castro::post_restart() 1 0.005977 0.005977 0.005977 1.37% FabArray::Saxpy() 597 0.005923 0.005923 0.005923 1.36% Castro::expand_state() 5 0.005839 0.005839 0.005839 1.34% Castro::post_timestep() 5 0.005832 0.005832 0.005832 1.34% MLCGSolver::ParallelAllReduce 798 0.005622 0.005622 0.005622 1.29% Gravity::multilevel_solve_for_new_phi() 1 0.005575 0.005575 0.005575 1.28% Gravity::actual_multilevel_solve() 1 0.005558 0.005558 0.005558 1.28% amrex::Copy() 221 0.005497 0.005497 0.005497 1.26% MLMG::addInterpCorrection() 180 0.005488 0.005488 0.005488 1.26% MLMG::mgVcycle_up::1 36 0.005297 0.005297 0.005297 1.22% MLMG::mgVcycle_up::4 36 0.005257 0.005257 0.005257 1.21% MLMG::mgVcycle_up::2 36 0.005174 0.005174 0.005174 1.19% amrex::average_down 180 0.00512 0.00512 0.00512 1.17% Gravity::fill_multipole_BCs() 6 0.005111 0.005111 0.005111 1.17% MLMG::mgVcycle_up::3 36 0.005052 0.005052 0.005052 1.16% MLPoisson::Fapply() 464 0.004536 0.004536 0.004536 1.04% MLCellLinOp::solutionResidual() 42 0.003636 0.003636 0.003636 0.83% FabArray::Xpay() 325 0.003538 0.003538 0.003538 0.81% Castro::estTimeStep() 10 0.003335 0.003335 0.003335 0.77% MLCellLinOp::defineBC() 6 0.002957 0.002957 0.002957 0.68% MLMG::prepareForSolve() 6 0.00289 0.00289 0.00289 0.66% Castro::reset_internal_energy(MultiFab) 30 0.002838 0.002838 0.002838 0.65% MLMG::computeResidual() 36 0.002828 0.002828 0.002828 0.65% BndryData::define() 6 0.002807 0.002807 0.002807 0.64% Castro::computeNewDt() 5 0.002162 0.002162 0.002162 0.50% Castro::enforce_speed_limit() 30 0.001934 0.001934 0.001934 0.44% Castro::construct_new_source() 25 0.00169 0.00169 0.00169 0.39% Castro::construct_new_gravity_source() 5 0.001614 0.001614 0.001614 0.37% amrex::Add() 36 0.001555 0.001555 0.001555 0.36% Castro::construct_old_source() 25 0.001255 0.001255 0.001255 0.29% Castro::finalize_do_advance() 5 0.001221 0.001221 0.001221 0.28% Castro::construct_old_gravity_source() 5 0.001206 0.001206 0.001206 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009953 0.0009953 0.0009953 0.23% MLMG::ResNormInf() 42 0.0009943 0.0009943 0.0009943 0.23% check_for_negative_density() 5 0.0009725 0.0009725 0.0009725 0.22% Castro::apply_source_to_state() 10 0.0009707 0.0009707 0.0009707 0.22% MLMG::getGradSolution() 6 0.0008958 0.0008958 0.0008958 0.21% MLCellLinOp::compGrad() 6 0.0008926 0.0008926 0.0008926 0.20% MLCellLinOp::setLevelBC() 6 0.0008891 0.0008891 0.0008891 0.20% Castro::reset_internal_energy(Fab) 240 0.0008644 0.0008644 0.0008644 0.20% MLMG::computeMLResidual() 6 0.0008311 0.0008311 0.0008311 0.19% FabArrayBase::getCPC() 632 0.0007849 0.0007849 0.0007849 0.18% MLPoisson::prepareForSolve() 6 0.0007626 0.0007626 0.0007626 0.17% MLCellLinOp::prepareForSolve() 6 0.0007524 0.0007524 0.0007524 0.17% Gravity::update_max_rhs() 6 0.00075 0.00075 0.00075 0.17% FabArray::mult() 22 0.0007066 0.0007066 0.0007066 0.16% FabArray::setDomainBndry() 20 0.0007011 0.0007011 0.0007011 0.16% Castro::check_for_nan() 10 0.0006688 0.0006688 0.0006688 0.15% MultiFab::contains_nan() 10 0.0006618 0.0006618 0.0006618 0.15% Amr::InitAmr() 1 0.0004823 0.0004823 0.0004823 0.11% FabArrayBase::CPC::define() 244 0.0004186 0.0004186 0.0004186 0.10% FabArrayBase::getFB() 1730 0.00033 0.00033 0.00033 0.08% Castro::finalize_advance() 5 0.0003063 0.0003063 0.0003063 0.07% Gravity::swapTimeLevels() 5 0.0002564 0.0002564 0.0002564 0.06% MultiFab::max() 6 0.00022 0.00022 0.00022 0.05% MLMG::MLResNormInf() 6 0.0001908 0.0001908 0.0001908 0.04% Castro::buildMetrics() 1 0.0001636 0.0001636 0.0001636 0.04% MLLinOp::define() 6 0.0001493 0.0001493 0.0001493 0.03% MLLinOp::defineGrids() 6 0.0001425 0.0001425 0.0001425 0.03% MLMG::MLRhsNormInf() 6 0.0001228 0.0001228 0.0001228 0.03% Castro::create_source_corrector() 5 0.000102 0.000102 0.000102 0.02% FabArrayBase::FB::FB() 26 6.515e-05 6.515e-05 6.515e-05 0.01% Amr::writeSmallPlotFile() 1 3.109e-05 3.109e-05 3.109e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.935e-05 2.935e-05 2.935e-05 0.01% Castro::initMFs() 1 2.549e-05 2.549e-05 2.549e-05 0.01% Castro::swap_state_time_levels() 5 2.419e-05 2.419e-05 2.419e-05 0.01% makeSFC 30 2.135e-05 2.135e-05 2.135e-05 0.00% Amr::initSubcycle() 1 9.932e-06 9.932e-06 9.932e-06 0.00% DistributionMapping::Distribute() 31 9.013e-06 9.013e-06 9.013e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.941e-06 4.941e-06 4.941e-06 0.00% Gravity::set_mass_offset() 6 2.539e-06 2.539e-06 2.539e-06 0.00% Castro::retry_advance_ctu() 5 1.903e-06 1.903e-06 1.903e-06 0.00% Castro::FluxRegCrseInit 5 1.625e-06 1.625e-06 1.625e-06 0.00% Castro::FluxRegFineAdd() 5 1.205e-06 1.205e-06 1.205e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.084e-06 1.084e-06 1.084e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 11 MiB 9037 MiB Castro::initMFs() 48 48 57 MiB 68 MiB Castro::swap_state_time_levels() 32 32 46 MiB 55 MiB StateData::restartDoit() 32 32 52 MiB 55 MiB FillPatchIterator::Initialize 160 160 1031 KiB 39 MiB Castro::initialize_do_advance() 40 40 28 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1680 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6429 KiB 14 MiB MLMG::prepareForSolve() 361 361 3261 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 184 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 187 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6415 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 18 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3423 B 2048 KiB Gravity::solve_for_phi() 40 40 600 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 26 KiB 2048 KiB BndryData::define() 576 576 300 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 193 KiB 671 KiB Castro::estTimeStep() 10 10 3586 B 480 KiB VisMF::Write(FabArray) 112 112 1297 B 320 KiB Castro::normalize_species() 30 30 8009 B 320 KiB amrex::average_down 469 469 1381 B 257 KiB MLMG::addInterpCorrection() 468 468 1078 B 257 KiB amrex::Dot() 592 592 3137 B 160 KiB FabArray::norminf() 501 501 3098 B 160 KiB check_for_negative_density() 5 5 358 B 160 KiB MultiFab::max() 6 6 79 B 160 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MultiFab::contains_nan() 10 10 30 B 20 KiB MLPoisson::Fsmooth() 60 60 3159 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 45 B 10 KiB FillBoundary_nowait() 336 336 258 B 9648 B MLCellLinOp::applyBC() 3820 3820 204 B 9344 B amrex::Copy() 56 56 5861 B 8816 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B StateData::FillBoundary(geom) 960 960 36 B 2448 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 1 B 1616 B MLCellLinOp::defineBC() 36 36 338 B 1248 B MLCGSolver::bicgstab 180 180 87 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1215 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 82 KiB 8192 KiB VisMF::Write(FabArray) 120 120 159 KiB 3584 KiB VisMF::Read() 24 24 210 KiB 3000 KiB FabArray::setVal() 66 66 20 KiB 27 KiB MLPoisson::Fsmooth() 60 60 3159 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 45 B 10 KiB FillBoundary_nowait() 336 336 258 B 9648 B MLCellLinOp::applyBC() 1910 1910 202 B 9328 B amrex::Copy() 56 56 5861 B 8816 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B Gravity::get_old_grav_vector() 3 3 2522 B 3072 B Gravity::fill_multipole_BCs() 18 18 5 B 2832 B StateData::FillBoundary(geom) 960 960 36 B 2448 B MLMG::prepareForSolve() 7 7 793 B 1648 B amrex::average_down 37 37 463 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 23 B 400 B FabArray::norminf() 501 501 9 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2180 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.03-8-gae3af4339425) finalized