Initializing CUDA... CUDA initialized with 1 device. AMReX (23.06-3-g572f226f4c98) initialized Starting run at 08:23:43 UTC on 2023-06-05. Successfully read inputs file ... Castro git describe: 23.06-8-g36dd1a76e AMReX git describe: 23.06-3-g572f226f4 Microphysics git describe: 23.06-2-g726fe1e2 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.059545461 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.03428987 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048825972 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.069934921 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.079692343 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.069319499 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.060972947 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.059299468 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051750423 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.076605598 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.075571193 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.056892447 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065164882 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.058897391 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.03421111 seconds Ending run at 08:23:44 UTC on 2023-06-05. Run time = 0.947348371 Run time without initialization = 0.807876941 Average number of zones advanced per microsecond: 3.245 Average number of zones advanced per microsecond per rank: 3.245 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.9474 ... 0.9474 ... 0.9474 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2849 0.2849 0.2849 30.08% VisMF::Write(FabArray) 11 0.2394 0.2394 0.2394 25.27% MLCellLinOp::applyBC() 4379 0.076 0.076 0.076 8.02% MLPoisson::Fsmooth() 3240 0.03212 0.03212 0.03212 3.39% FillBoundary_nowait() 3974 0.03011 0.03011 0.03011 3.18% StateData::FillBoundary(geom) 328 0.02568 0.02568 0.02568 2.71% amrex::Dot() 1100 0.02052 0.02052 0.02052 2.17% Castro::normalize_species() 62 0.01916 0.01916 0.01916 2.02% Castro::computeTemp() 63 0.01513 0.01513 0.01513 1.60% amrex::Copy() 1019 0.01481 0.01481 0.01481 1.56% FabArray::norminf() 734 0.01418 0.01418 0.01418 1.50% FabArray::ParallelCopy_nowait() 851 0.0133 0.0133 0.0133 1.40% FabArray::setVal() 1135 0.01313 0.01313 0.01313 1.39% StateDataPhysBCFunct::() 41 0.0114 0.0114 0.0114 1.20% Castro::enforce_min_density() 62 0.0112 0.0112 0.0112 1.18% MLPoisson::Fapply() 1128 0.01029 0.01029 0.01029 1.09% MLCellLinOp::defineAuxData() 11 0.009852 0.009852 0.009852 1.04% FabArray::Saxpy() 803 0.008101 0.008101 0.008101 0.86% FabArray::Xpay() 811 0.008065 0.008065 0.008065 0.85% MLMG::addInterpCorrection() 405 0.006642 0.006642 0.006642 0.70% Gravity::fill_multipole_BCs() 11 0.006632 0.006632 0.006632 0.70% amrex::average_down 405 0.005927 0.005927 0.005927 0.63% Castro::estTimeStep() 21 0.005735 0.005735 0.005735 0.61% Castro::reset_internal_energy(MultiFab) 63 0.00506 0.00506 0.00506 0.53% Castro::enforce_speed_limit() 62 0.004549 0.004549 0.004549 0.48% FabArray::LinComb() 550 0.00452 0.00452 0.00452 0.48% amrex::Add() 162 0.004318 0.004318 0.004318 0.46% Amr::checkPoint() 3 0.003968 0.003968 0.003968 0.42% BndryData::define() 11 0.003781 0.003781 0.003781 0.40% Castro::construct_new_gravity_source() 10 0.003199 0.003199 0.003199 0.34% Castro::construct_old_gravity_source() 10 0.002698 0.002698 0.002698 0.28% Amr::writePlotFile() 2 0.002426 0.002426 0.002426 0.26% MLCGSolver::bicgstab 81 0.002152 0.002152 0.002152 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001703 0.001703 0.001703 0.18% check_for_negative_density() 10 0.00165 0.00165 0.00165 0.17% Castro::reset_internal_energy(Fab) 504 0.00159 0.00159 0.00159 0.17% Gravity::actual_solve_with_mlmg() 11 0.001468 0.001468 0.001468 0.15% MLCellLinOp::setLevelBC() 11 0.001432 0.001432 0.001432 0.15% FabArray::mult() 43 0.001309 0.001309 0.001309 0.14% FabArray::setDomainBndry() 41 0.001289 0.001289 0.001289 0.14% Castro::initData() 1 0.001234 0.001234 0.001234 0.13% MultiFab::contains_nan() 20 0.001192 0.001192 0.001192 0.13% MLCellLinOp::prepareForSolve() 11 0.00113 0.00113 0.00113 0.12% MLCellLinOp::smooth() 1620 0.001095 0.001095 0.001095 0.12% MLCellLinOp::compGrad() 11 0.0009168 0.0009168 0.0009168 0.10% MLMG::prepareForSolve() 11 0.0008667 0.0008667 0.0008667 0.09% FabArray::FillBoundary() 3974 0.0007998 0.0007998 0.0007998 0.08% FabArrayBase::getCPC() 1313 0.0007698 0.0007698 0.0007698 0.08% FabArrayBase::CPC::define() 454 0.0006661 0.0006661 0.0006661 0.07% Gravity::get_new_grav_vector() 11 0.0006266 0.0006266 0.0006266 0.07% FabArrayBase::getFB() 3974 0.0005819 0.0005819 0.0005819 0.06% Gravity::get_old_grav_vector() 10 0.0005637 0.0005637 0.0005637 0.06% Amr::InitAmr() 1 0.000517 0.000517 0.000517 0.05% MLLinOp::defineGrids() 11 0.0004681 0.0004681 0.0004681 0.05% MLCellLinOp::apply() 1128 0.0004532 0.0004532 0.0004532 0.05% Amr::coarseTimeStep() 10 0.0004147 0.0004147 0.0004147 0.04% MLMG::mgVcycle() 81 0.0003467 0.0003467 0.0003467 0.04% main() 1 0.0003118 0.0003118 0.0003118 0.03% MLCGSolver::ParallelAllReduce 1495 0.000285 0.000285 0.000285 0.03% MultiFab::max() 11 0.0002584 0.0002584 0.0002584 0.03% FabArray::ParallelCopy() 851 0.0002395 0.0002395 0.0002395 0.03% FillPatchIterator::Initialize 41 0.0002336 0.0002336 0.0002336 0.02% MLCellLinOp::defineBC() 11 0.0002324 0.0002324 0.0002324 0.02% MLCellLinOp::correctionResidual() 486 0.0002091 0.0002091 0.0002091 0.02% Castro::subcycle_advance_ctu() 10 0.0001767 0.0001767 0.0001767 0.02% Castro::create_source_corrector() 10 0.0001602 0.0001602 0.0001602 0.02% Amr::timeStep() 10 0.0001469 0.0001469 0.0001469 0.02% StateData::checkPoint() 12 0.0001307 0.0001307 0.0001307 0.01% Gravity::update_max_rhs() 11 0.0001232 0.0001232 0.0001232 0.01% Castro::do_advance_ctu() 10 0.0001229 0.0001229 0.0001229 0.01% MLMG:computeResOfCorrection() 405 0.0001183 0.0001183 0.0001183 0.01% Gravity::solve_for_phi() 10 0.0001131 0.0001131 0.0001131 0.01% Castro::advance() 10 9.196e-05 9.196e-05 9.196e-05 0.01% Castro::Castro() 1 9.106e-05 9.106e-05 9.106e-05 0.01% MLMG::mgVcycle_down::0 81 8.977e-05 8.977e-05 8.977e-05 0.01% FabArrayBase::FB::FB() 56 8.872e-05 8.872e-05 8.872e-05 0.01% MLMG::actualBottomSolve() 81 8.563e-05 8.563e-05 8.563e-05 0.01% Castro::expand_state() 10 7.892e-05 7.892e-05 7.892e-05 0.01% MLMG::mgVcycle_down::1 81 7.756e-05 7.756e-05 7.756e-05 0.01% MLMG::mgVcycle_down::2 81 7.406e-05 7.406e-05 7.406e-05 0.01% MLMG::solve() 11 7.396e-05 7.396e-05 7.396e-05 0.01% AmrLevel::checkPoint() 3 7.356e-05 7.356e-05 7.356e-05 0.01% Castro::clean_state() 62 7.353e-05 7.353e-05 7.353e-05 0.01% MLMG::mgVcycle_down::3 81 7.022e-05 7.022e-05 7.022e-05 0.01% MLMG::mgVcycle_down::4 81 6.908e-05 6.908e-05 6.908e-05 0.01% Castro::initialize_advance() 10 6.774e-05 6.774e-05 6.774e-05 0.01% Castro::finalize_advance() 10 6.123e-05 6.123e-05 6.123e-05 0.01% MLMG::mgVcycle_up::4 81 5.892e-05 5.892e-05 5.892e-05 0.01% MLCellLinOp::solutionResidual() 92 5.542e-05 5.542e-05 5.542e-05 0.01% MLMG::oneIter() 81 5.501e-05 5.501e-05 5.501e-05 0.01% Castro::initialize_do_advance() 10 5.156e-05 5.156e-05 5.156e-05 0.01% MLMG::mgVcycle_up::1 81 5.018e-05 5.018e-05 5.018e-05 0.01% MLMG::mgVcycle_up::0 81 4.905e-05 4.905e-05 4.905e-05 0.01% MLMG::mgVcycle_up::3 81 4.891e-05 4.891e-05 4.891e-05 0.01% MLMG::mgVcycle_up::2 81 4.849e-05 4.849e-05 4.849e-05 0.01% Castro::post_timestep() 10 4.727e-05 4.727e-05 4.727e-05 0.00% Castro::construct_new_source() 50 4.475e-05 4.475e-05 4.475e-05 0.00% FillPatchSingleLevel 41 4.413e-05 4.413e-05 4.413e-05 0.00% StateData::define() 4 4.062e-05 4.062e-05 4.062e-05 0.00% Castro::swap_state_time_levels() 10 3.819e-05 3.819e-05 3.819e-05 0.00% Castro::finalize_do_advance() 10 3.804e-05 3.804e-05 3.804e-05 0.00% Castro::enforce_consistent_e() 1 3.305e-05 3.305e-05 3.305e-05 0.00% MLMG::ResNormInf() 92 3.26e-05 3.26e-05 3.26e-05 0.00% MLMG::computeResidual() 81 3.04e-05 3.04e-05 3.04e-05 0.00% MLMG::mgVcycle_bottom 81 2.938e-05 2.938e-05 2.938e-05 0.00% MLPoisson::define() 11 2.616e-05 2.616e-05 2.616e-05 0.00% Castro::initMFs() 1 2.613e-05 2.613e-05 2.613e-05 0.00% Amr::writeSmallPlotFile() 1 2.523e-05 2.523e-05 2.523e-05 0.00% Amr::defBaseLevel() 1 2.521e-05 2.521e-05 2.521e-05 0.00% makeSFC 55 2.477e-05 2.477e-05 2.477e-05 0.00% Castro::construct_new_gravity() 10 2.356e-05 2.356e-05 2.356e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.098e-05 2.098e-05 2.098e-05 0.00% Castro::construct_old_source() 50 1.868e-05 1.868e-05 1.868e-05 0.00% Amr::FinalizeInit() 1 1.854e-05 1.854e-05 1.854e-05 0.00% Castro::buildMetrics() 1 1.834e-05 1.834e-05 1.834e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.639e-05 1.639e-05 1.639e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.627e-05 1.627e-05 1.627e-05 0.00% Castro::do_new_sources() 10 1.616e-05 1.616e-05 1.616e-05 0.00% DistributionMapping::Distribute() 56 1.534e-05 1.534e-05 1.534e-05 0.00% Castro::do_old_sources() 10 1.472e-05 1.472e-05 1.472e-05 0.00% MLPoisson::prepareForSolve() 11 1.452e-05 1.452e-05 1.452e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.42e-05 1.42e-05 1.42e-05 0.00% MLLinOp::define() 11 1.129e-05 1.129e-05 1.129e-05 0.00% Castro::apply_source_to_state() 20 1.047e-05 1.047e-05 1.047e-05 0.00% Castro::check_for_nan() 20 1.044e-05 1.044e-05 1.044e-05 0.00% Castro::construct_old_gravity() 10 9.545e-06 9.545e-06 9.545e-06 0.00% Amr::initSubcycle() 1 8.926e-06 8.926e-06 8.926e-06 0.00% Gravity::swapTimeLevels() 10 8.424e-06 8.424e-06 8.424e-06 0.00% Gravity::actual_multilevel_solve() 1 8.39e-06 8.39e-06 8.39e-06 0.00% MLMG::computeMLResidual() 11 7.755e-06 7.755e-06 7.755e-06 0.00% Castro::computeNewDt() 9 7.55e-06 7.55e-06 7.55e-06 0.00% MLMG::getGradSolution() 11 5.432e-06 5.432e-06 5.432e-06 0.00% Amr::InitializeInit() 1 4.729e-06 4.729e-06 4.729e-06 0.00% AmrLevel::checkPointPost() 3 4.446e-06 4.446e-06 4.446e-06 0.00% Castro::retry_advance_ctu() 10 4.162e-06 4.162e-06 4.162e-06 0.00% Gravity::set_mass_offset() 11 3.861e-06 3.861e-06 3.861e-06 0.00% MLMG::MLRhsNormInf() 11 3.81e-06 3.81e-06 3.81e-06 0.00% MLMG::MLResNormInf() 11 3.549e-06 3.549e-06 3.549e-06 0.00% Castro::post_init() 1 3.522e-06 3.522e-06 3.522e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.303e-06 3.303e-06 3.303e-06 0.00% Castro::computeInitialDt() 2 2.783e-06 2.783e-06 2.783e-06 0.00% Castro::FluxRegCrseInit 10 2.721e-06 2.721e-06 2.721e-06 0.00% Amr::init() 1 2.515e-06 2.515e-06 2.515e-06 0.00% Castro::FluxRegFineAdd() 10 2.085e-06 2.085e-06 2.085e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.852e-06 1.852e-06 1.852e-06 0.00% AmrLevel::checkPointPre() 3 1.577e-06 1.577e-06 1.577e-06 0.00% Amr::initialInit() 1 1.405e-06 1.405e-06 1.405e-06 0.00% Castro::post_regrid() 1 9e-07 9e-07 9e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9474 0.9474 0.9474 100.00% Amr::coarseTimeStep() 10 0.7734 0.7734 0.7734 81.64% Amr::timeStep() 10 0.6514 0.6514 0.6514 68.76% Castro::advance() 10 0.6398 0.6398 0.6398 67.53% Castro::subcycle_advance_ctu() 10 0.6255 0.6255 0.6255 66.02% Castro::do_advance_ctu() 10 0.6253 0.6253 0.6253 66.00% Castro::construct_ctu_hydro_source() 10 0.2849 0.2849 0.2849 30.08% Gravity::solve_phi_with_mlmg() 11 0.2811 0.2811 0.2811 29.67% Gravity::actual_solve_with_mlmg() 11 0.274 0.274 0.274 28.92% Castro::construct_new_gravity() 10 0.26 0.26 0.26 27.44% MLMG::solve() 11 0.253 0.253 0.253 26.70% Gravity::solve_for_phi() 10 0.244 0.244 0.244 25.76% VisMF::Write(FabArray) 11 0.2394 0.2394 0.2394 25.27% MLMG::oneIter() 81 0.2388 0.2388 0.2388 25.20% MLMG::mgVcycle() 81 0.2351 0.2351 0.2351 24.82% Amr::checkPoint() 3 0.1779 0.1779 0.1779 18.78% AmrLevel::checkPoint() 3 0.1739 0.1739 0.1739 18.36% StateData::checkPoint() 12 0.1739 0.1739 0.1739 18.35% Amr::init() 1 0.1388 0.1388 0.1388 14.65% MLCellLinOp::smooth() 1620 0.1152 0.1152 0.1152 12.16% MLCellLinOp::applyBC() 4379 0.1076 0.1076 0.1076 11.36% MLMG::mgVcycle_bottom 81 0.07313 0.07313 0.07313 7.72% MLMG::actualBottomSolve() 81 0.0731 0.0731 0.0731 7.72% MLCGSolver::bicgstab 81 0.0724 0.0724 0.0724 7.64% Amr::writePlotFile() 2 0.06864 0.06864 0.06864 7.24% Castro::clean_state() 62 0.05594 0.05594 0.05594 5.90% Amr::initialInit() 1 0.04483 0.04483 0.04483 4.73% FillPatchIterator::Initialize 41 0.0427 0.0427 0.0427 4.51% FillPatchSingleLevel 41 0.04118 0.04118 0.04118 4.35% Amr::FinalizeInit() 1 0.04069 0.04069 0.04069 4.30% Castro::post_init() 1 0.03944 0.03944 0.03944 4.16% Gravity::multilevel_solve_for_new_phi() 1 0.03754 0.03754 0.03754 3.96% Gravity::actual_multilevel_solve() 1 0.03752 0.03752 0.03752 3.96% StateDataPhysBCFunct::() 41 0.03709 0.03709 0.03709 3.91% MLCellLinOp::apply() 1128 0.03591 0.03591 0.03591 3.79% MLMG::mgVcycle_down::0 81 0.03318 0.03318 0.03318 3.50% MLPoisson::Fsmooth() 3240 0.03212 0.03212 0.03212 3.39% FabArray::FillBoundary() 3974 0.03158 0.03158 0.03158 3.33% FillBoundary_nowait() 3974 0.03078 0.03078 0.03078 3.25% StateData::FillBoundary(geom) 328 0.02568 0.02568 0.02568 2.71% MLMG::mgVcycle_up::0 81 0.02503 0.02503 0.02503 2.64% MLCellLinOp::correctionResidual() 486 0.02201 0.02201 0.02201 2.32% Castro::computeTemp() 63 0.02178 0.02178 0.02178 2.30% Castro::initialize_do_advance() 10 0.02054 0.02054 0.02054 2.17% amrex::Dot() 1100 0.02052 0.02052 0.02052 2.17% MLMG:computeResOfCorrection() 405 0.01938 0.01938 0.01938 2.05% Castro::normalize_species() 62 0.01916 0.01916 0.01916 2.02% Gravity::get_new_grav_vector() 11 0.01759 0.01759 0.01759 1.86% MLPoisson::define() 11 0.01674 0.01674 0.01674 1.77% MLMG::mgVcycle_down::1 81 0.01564 0.01564 0.01564 1.65% Castro::construct_old_gravity() 10 0.01524 0.01524 0.01524 1.61% Gravity::get_old_grav_vector() 10 0.01523 0.01523 0.01523 1.61% amrex::Copy() 1019 0.01481 0.01481 0.01481 1.56% MLMG::mgVcycle_down::2 81 0.0146 0.0146 0.0146 1.54% FabArray::ParallelCopy() 851 0.01436 0.01436 0.01436 1.52% MLMG::mgVcycle_down::3 81 0.01426 0.01426 0.01426 1.50% FabArray::norminf() 734 0.01418 0.01418 0.01418 1.50% FabArray::ParallelCopy_nowait() 851 0.01412 0.01412 0.01412 1.49% MLMG::mgVcycle_down::4 81 0.01412 0.01412 0.01412 1.49% Castro::initialize_advance() 10 0.01364 0.01364 0.01364 1.44% FabArray::setVal() 1135 0.01313 0.01313 0.01313 1.39% MLCGSolver::ParallelAllReduce 1495 0.01232 0.01232 0.01232 1.30% Castro::do_new_sources() 10 0.01205 0.01205 0.01205 1.27% MLMG::addInterpCorrection() 405 0.01179 0.01179 0.01179 1.24% Castro::do_old_sources() 10 0.01149 0.01149 0.01149 1.21% Castro::post_timestep() 10 0.01146 0.01146 0.01146 1.21% MLMG::mgVcycle_up::4 81 0.01143 0.01143 0.01143 1.21% Castro::expand_state() 10 0.01142 0.01142 0.01142 1.20% MLMG::mgVcycle_up::1 81 0.01135 0.01135 0.01135 1.20% Castro::enforce_min_density() 62 0.0112 0.0112 0.0112 1.18% MLCellLinOp::defineAuxData() 11 0.01119 0.01119 0.01119 1.18% MLMG::mgVcycle_up::2 81 0.01119 0.01119 0.01119 1.18% amrex::average_down 405 0.01109 0.01109 0.01109 1.17% MLMG::mgVcycle_up::3 81 0.01086 0.01086 0.01086 1.15% MLPoisson::Fapply() 1128 0.01029 0.01029 0.01029 1.09% FabArray::Saxpy() 803 0.008101 0.008101 0.008101 0.86% FabArray::Xpay() 811 0.008065 0.008065 0.008065 0.85% MLCellLinOp::solutionResidual() 92 0.007143 0.007143 0.007143 0.75% Gravity::fill_multipole_BCs() 11 0.006897 0.006897 0.006897 0.73% Castro::reset_internal_energy(MultiFab) 63 0.00665 0.00665 0.00665 0.70% MLMG::computeResidual() 81 0.006151 0.006151 0.006151 0.65% Castro::estTimeStep() 21 0.005735 0.005735 0.005735 0.61% MLCellLinOp::defineBC() 11 0.004989 0.004989 0.004989 0.53% BndryData::define() 11 0.004757 0.004757 0.004757 0.50% MLMG::prepareForSolve() 11 0.004681 0.004681 0.004681 0.49% Castro::enforce_speed_limit() 62 0.004549 0.004549 0.004549 0.48% FabArray::LinComb() 550 0.00452 0.00452 0.00452 0.48% amrex::Add() 162 0.004318 0.004318 0.004318 0.46% Amr::InitializeInit() 1 0.004134 0.004134 0.004134 0.44% Amr::defBaseLevel() 1 0.00413 0.00413 0.00413 0.44% Castro::initData() 1 0.003598 0.003598 0.003598 0.38% Castro::construct_new_source() 50 0.003244 0.003244 0.003244 0.34% Castro::construct_new_gravity_source() 10 0.003199 0.003199 0.003199 0.34% Castro::construct_old_source() 50 0.002716 0.002716 0.002716 0.29% Castro::construct_old_gravity_source() 10 0.002698 0.002698 0.002698 0.28% Castro::computeNewDt() 9 0.002629 0.002629 0.002629 0.28% MLMG::ResNormInf() 92 0.002068 0.002068 0.002068 0.22% Castro::apply_source_to_state() 20 0.001823 0.001823 0.001823 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001703 0.001703 0.001703 0.18% check_for_negative_density() 10 0.00165 0.00165 0.00165 0.17% Castro::reset_internal_energy(Fab) 504 0.00159 0.00159 0.00159 0.17% FabArrayBase::getCPC() 1313 0.001436 0.001436 0.001436 0.15% MLCellLinOp::setLevelBC() 11 0.001432 0.001432 0.001432 0.15% MLMG::getGradSolution() 11 0.001388 0.001388 0.001388 0.15% MLCellLinOp::compGrad() 11 0.001383 0.001383 0.001383 0.15% FabArray::mult() 43 0.001309 0.001309 0.001309 0.14% FabArray::setDomainBndry() 41 0.001289 0.001289 0.001289 0.14% Castro::check_for_nan() 20 0.001202 0.001202 0.001202 0.13% MultiFab::contains_nan() 20 0.001192 0.001192 0.001192 0.13% MLPoisson::prepareForSolve() 11 0.001145 0.001145 0.001145 0.12% MLCellLinOp::prepareForSolve() 11 0.00113 0.00113 0.00113 0.12% Castro::post_regrid() 1 0.001081 0.001081 0.001081 0.11% MLMG::computeMLResidual() 11 0.001029 0.001029 0.001029 0.11% Castro::computeInitialDt() 2 0.0008401 0.0008401 0.0008401 0.09% Gravity::update_max_rhs() 11 0.0008274 0.0008274 0.0008274 0.09% FabArrayBase::getFB() 3974 0.0006706 0.0006706 0.0006706 0.07% FabArrayBase::CPC::define() 454 0.0006661 0.0006661 0.0006661 0.07% Castro::finalize_advance() 10 0.0005895 0.0005895 0.0005895 0.06% MLLinOp::define() 11 0.0005338 0.0005338 0.0005338 0.06% Amr::InitAmr() 1 0.0005259 0.0005259 0.0005259 0.06% MLLinOp::defineGrids() 11 0.0005225 0.0005225 0.0005225 0.06% Castro::Castro() 1 0.0004446 0.0004446 0.0004446 0.05% Gravity::swapTimeLevels() 10 0.0004402 0.0004402 0.0004402 0.05% MLMG::MLResNormInf() 11 0.0002861 0.0002861 0.0002861 0.03% MultiFab::max() 11 0.0002584 0.0002584 0.0002584 0.03% MLMG::MLRhsNormInf() 11 0.0002203 0.0002203 0.0002203 0.02% Castro::buildMetrics() 1 0.0001625 0.0001625 0.0001625 0.02% Castro::create_source_corrector() 10 0.0001602 0.0001602 0.0001602 0.02% FabArrayBase::FB::FB() 56 8.872e-05 8.872e-05 8.872e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.69e-05 5.69e-05 5.69e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.257e-05 5.257e-05 5.257e-05 0.01% StateData::define() 4 4.062e-05 4.062e-05 4.062e-05 0.00% makeSFC 55 3.836e-05 3.836e-05 3.836e-05 0.00% Castro::swap_state_time_levels() 10 3.819e-05 3.819e-05 3.819e-05 0.00% Castro::finalize_do_advance() 10 3.804e-05 3.804e-05 3.804e-05 0.00% Castro::enforce_consistent_e() 1 3.305e-05 3.305e-05 3.305e-05 0.00% Castro::initMFs() 1 2.613e-05 2.613e-05 2.613e-05 0.00% Amr::writeSmallPlotFile() 1 2.523e-05 2.523e-05 2.523e-05 0.00% DistributionMapping::Distribute() 56 1.534e-05 1.534e-05 1.534e-05 0.00% Amr::initSubcycle() 1 8.926e-06 8.926e-06 8.926e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.051e-06 5.051e-06 5.051e-06 0.00% AmrLevel::checkPointPost() 3 4.446e-06 4.446e-06 4.446e-06 0.00% Castro::retry_advance_ctu() 10 4.162e-06 4.162e-06 4.162e-06 0.00% Gravity::set_mass_offset() 11 3.861e-06 3.861e-06 3.861e-06 0.00% Castro::FluxRegCrseInit 10 2.721e-06 2.721e-06 2.721e-06 0.00% Castro::FluxRegFineAdd() 10 2.085e-06 2.085e-06 2.085e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.852e-06 1.852e-06 1.852e-06 0.00% AmrLevel::checkPointPre() 3 1.577e-06 1.577e-06 1.577e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 83 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 141 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 955 KiB 39 MiB Castro::initialize_do_advance() 80 80 25 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 2035 KiB 28 MiB Castro::initialize_advance() 80 80 15 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7608 KiB 14 MiB MLMG::prepareForSolve() 660 660 3285 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 193 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 164 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7529 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 12 KiB 2053 KiB Gravity::update_max_rhs() 88 88 1670 B 2048 KiB Gravity::solve_for_phi() 80 80 526 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 81 KiB 2048 KiB BndryData::define() 1056 1056 300 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 191 KiB 671 KiB Castro::estTimeStep() 21 21 2955 B 480 KiB VisMF::Write(FabArray) 656 656 3173 B 320 KiB Castro::normalize_species() 62 62 6589 B 320 KiB amrex::average_down 1054 1054 1188 B 257 KiB MLMG::addInterpCorrection() 1053 1053 1060 B 257 KiB amrex::Dot() 1343 1343 3190 B 160 KiB FabArray::norminf() 896 896 2211 B 160 KiB check_for_negative_density() 10 10 277 B 160 KiB Castro::initData() 1 1 26 B 160 KiB MultiFab::max() 11 11 42 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 25 B 20 KiB MLPoisson::Fsmooth() 132 132 3211 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 41 B 10 KiB FillBoundary_nowait() 751 751 277 B 9648 B MLCellLinOp::applyBC() 8758 8758 206 B 9344 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3865 B 6144 B StateData::FillBoundary(geom) 1992 1992 38 B 2880 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 729 729 111 B 1472 B MLCellLinOp::defineBC() 66 66 338 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 612 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 47 KiB 8192 KiB VisMF::Write(FabArray) 744 744 587 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3211 B 12 KiB FabArray::ParallelCopy_nowait() 851 851 41 B 10 KiB FillBoundary_nowait() 751 751 277 B 9648 B MLCellLinOp::applyBC() 4379 4379 205 B 9328 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3865 B 6144 B Gravity::get_new_grav_vector() 3 3 2927 B 3072 B StateData::FillBoundary(geom) 1992 1992 39 B 2880 B Gravity::fill_multipole_BCs() 33 33 3 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 82 82 272 B 1296 B MLMG::addInterpCorrection() 81 81 2 B 1024 B MLMG::prepareForSolve() 11 11 273 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1343 1343 23 B 400 B FabArray::norminf() 896 896 16 B 272 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2424 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.06-3-g572f226f4c98) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.06-3-g572f226f4c98) initialized Starting run at 08:23:45 UTC on 2023-06-05. Successfully read inputs file ... Castro git describe: 23.06-8-g36dd1a76e AMReX git describe: 23.06-3-g572f226f4 Microphysics git describe: 23.06-2-g726fe1e2 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.527581428 Restart time = 0.045981985 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.049784283 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048613742 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.056536173 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.080518255 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.082280945 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033716216 seconds Ending run at 08:23:45 UTC on 2023-06-05. Run time = 0.398435449 Run time without initialization = 0.351883359 Average number of zones advanced per microsecond: 3.725 Average number of zones advanced per microsecond per rank: 3.725 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3985 ... 0.3985 ... 0.3985 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1334 0.1334 0.1334 33.48% VisMF::Read() 3 0.03969 0.03969 0.03969 9.96% MLCellLinOp::applyBC() 1946 0.03304 0.03304 0.03304 8.29% VisMF::Write(FabArray) 1 0.03207 0.03207 0.03207 8.05% MLPoisson::Fsmooth() 1440 0.0139 0.0139 0.0139 3.49% FillBoundary_nowait() 1766 0.01288 0.01288 0.01288 3.23% StateData::FillBoundary(geom) 160 0.01219 0.01219 0.01219 3.06% Castro::normalize_species() 30 0.00939 0.00939 0.00939 2.36% amrex::Dot() 484 0.008818 0.008818 0.008818 2.21% amrex::Copy() 463 0.006953 0.006953 0.006953 1.74% FabArray::setVal() 537 0.006273 0.006273 0.006273 1.57% FabArray::norminf() 326 0.006162 0.006162 0.006162 1.55% Castro::computeTemp() 30 0.006133 0.006133 0.006133 1.54% FabArray::ParallelCopy_nowait() 380 0.00609 0.00609 0.00609 1.53% MLCellLinOp::defineAuxData() 6 0.005264 0.005264 0.005264 1.32% StateDataPhysBCFunct::() 20 0.004894 0.004894 0.004894 1.23% Castro::enforce_min_density() 30 0.004634 0.004634 0.004634 1.16% MLPoisson::Fapply() 500 0.004459 0.004459 0.004459 1.12% FabArray::Saxpy() 355 0.003634 0.003634 0.003634 0.91% Castro::estTimeStep() 10 0.003621 0.003621 0.003621 0.91% FabArray::Xpay() 361 0.003495 0.003495 0.003495 0.88% Gravity::fill_multipole_BCs() 6 0.00341 0.00341 0.00341 0.86% MLMG::addInterpCorrection() 180 0.002911 0.002911 0.002911 0.73% amrex::average_down 180 0.002586 0.002586 0.002586 0.65% Amr::restart() 1 0.002567 0.002567 0.002567 0.64% Castro::reset_internal_energy(MultiFab) 30 0.002267 0.002267 0.002267 0.57% BndryData::define() 6 0.002041 0.002041 0.002041 0.51% FabArray::LinComb() 242 0.001932 0.001932 0.001932 0.48% amrex::Add() 72 0.001837 0.001837 0.001837 0.46% Castro::subcycle_advance_ctu() 5 0.001744 0.001744 0.001744 0.44% Castro::construct_new_gravity_source() 5 0.001579 0.001579 0.001579 0.40% Amr::writePlotFile() 1 0.001496 0.001496 0.001496 0.38% Castro::construct_old_gravity_source() 5 0.001413 0.001413 0.001413 0.35% MLCGSolver::bicgstab 36 0.0009532 0.0009532 0.0009532 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008964 0.0008964 0.0008964 0.22% Castro::reset_internal_energy(Fab) 240 0.0008384 0.0008384 0.0008384 0.21% check_for_negative_density() 5 0.0008192 0.0008192 0.0008192 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007667 0.0007667 0.0007667 0.19% MLCellLinOp::setLevelBC() 6 0.0007443 0.0007443 0.0007443 0.19% FabArray::setDomainBndry() 20 0.000657 0.000657 0.000657 0.16% Castro::enforce_speed_limit() 30 0.0006546 0.0006546 0.0006546 0.16% FabArray::mult() 22 0.0006382 0.0006382 0.0006382 0.16% MLCellLinOp::prepareForSolve() 6 0.0006096 0.0006096 0.0006096 0.15% MultiFab::contains_nan() 10 0.0005901 0.0005901 0.0005901 0.15% MLCellLinOp::compGrad() 6 0.0004932 0.0004932 0.0004932 0.12% MLCellLinOp::smooth() 720 0.0004898 0.0004898 0.0004898 0.12% MLMG::prepareForSolve() 6 0.0004648 0.0004648 0.0004648 0.12% FabArrayBase::CPC::define() 244 0.0004096 0.0004096 0.0004096 0.10% Amr::InitAmr() 1 0.0003812 0.0003812 0.0003812 0.10% FabArrayBase::getCPC() 632 0.0003668 0.0003668 0.0003668 0.09% FabArray::FillBoundary() 1766 0.0003637 0.0003637 0.0003637 0.09% Gravity::get_old_grav_vector() 5 0.0003096 0.0003096 0.0003096 0.08% Gravity::get_new_grav_vector() 5 0.0002744 0.0002744 0.0002744 0.07% main() 1 0.0002737 0.0002737 0.0002737 0.07% FabArrayBase::getFB() 1766 0.0002517 0.0002517 0.0002517 0.06% MLCellLinOp::apply() 500 0.0002072 0.0002072 0.0002072 0.05% Amr::coarseTimeStep() 5 0.0001736 0.0001736 0.0001736 0.04% MLMG::mgVcycle() 36 0.0001644 0.0001644 0.0001644 0.04% MLLinOp::defineGrids() 6 0.0001641 0.0001641 0.0001641 0.04% Castro::do_advance_ctu() 5 0.0001514 0.0001514 0.0001514 0.04% MLCGSolver::ParallelAllReduce 659 0.0001406 0.0001406 0.0001406 0.04% Castro::construct_new_source() 25 0.0001357 0.0001357 0.0001357 0.03% MultiFab::max() 6 0.0001336 0.0001336 0.0001336 0.03% MLCellLinOp::defineBC() 6 0.0001199 0.0001199 0.0001199 0.03% FillPatchIterator::Initialize 20 0.0001155 0.0001155 0.0001155 0.03% FabArray::ParallelCopy() 380 0.0001119 0.0001119 0.0001119 0.03% MLCellLinOp::correctionResidual() 216 9.352e-05 9.352e-05 9.352e-05 0.02% StateData::restartDoit() 4 8.945e-05 8.945e-05 8.945e-05 0.02% Castro::create_source_corrector() 5 8.359e-05 8.359e-05 8.359e-05 0.02% AmrLevel::restart() 1 8.284e-05 8.284e-05 8.284e-05 0.02% Amr::timeStep() 5 7.646e-05 7.646e-05 7.646e-05 0.02% Castro::construct_old_source() 25 6.334e-05 6.334e-05 6.334e-05 0.02% Gravity::update_max_rhs() 6 5.959e-05 5.959e-05 5.959e-05 0.01% FabArrayBase::FB::FB() 26 5.863e-05 5.863e-05 5.863e-05 0.01% Castro::advance() 5 5.821e-05 5.821e-05 5.821e-05 0.01% Gravity::solve_for_phi() 5 5.532e-05 5.532e-05 5.532e-05 0.01% MLMG:computeResOfCorrection() 180 5.014e-05 5.014e-05 5.014e-05 0.01% Castro::expand_state() 5 4.229e-05 4.229e-05 4.229e-05 0.01% MLMG::mgVcycle_down::1 36 4.115e-05 4.115e-05 4.115e-05 0.01% MLMG::mgVcycle_down::0 36 4.068e-05 4.068e-05 4.068e-05 0.01% MLMG::actualBottomSolve() 36 3.977e-05 3.977e-05 3.977e-05 0.01% Castro::clean_state() 30 3.542e-05 3.542e-05 3.542e-05 0.01% MLMG::mgVcycle_down::2 36 3.475e-05 3.475e-05 3.475e-05 0.01% MLMG::mgVcycle_down::4 36 3.469e-05 3.469e-05 3.469e-05 0.01% Castro::post_timestep() 5 3.409e-05 3.409e-05 3.409e-05 0.01% MLMG::solve() 6 3.374e-05 3.374e-05 3.374e-05 0.01% MLMG::mgVcycle_down::3 36 3.298e-05 3.298e-05 3.298e-05 0.01% Castro::initialize_advance() 5 3.296e-05 3.296e-05 3.296e-05 0.01% MLMG::oneIter() 36 2.97e-05 2.97e-05 2.97e-05 0.01% Castro::finalize_advance() 5 2.962e-05 2.962e-05 2.962e-05 0.01% Castro::initMFs() 1 2.853e-05 2.853e-05 2.853e-05 0.01% Amr::writeSmallPlotFile() 1 2.77e-05 2.77e-05 2.77e-05 0.01% Castro::buildMetrics() 1 2.713e-05 2.713e-05 2.713e-05 0.01% MLMG::mgVcycle_up::4 36 2.696e-05 2.696e-05 2.696e-05 0.01% Castro::initialize_do_advance() 5 2.652e-05 2.652e-05 2.652e-05 0.01% Castro::post_restart() 1 2.491e-05 2.491e-05 2.491e-05 0.01% Castro::swap_state_time_levels() 5 2.215e-05 2.215e-05 2.215e-05 0.01% MLMG::mgVcycle_up::3 36 2.156e-05 2.156e-05 2.156e-05 0.01% MLCellLinOp::solutionResidual() 42 2.138e-05 2.138e-05 2.138e-05 0.01% MLMG::mgVcycle_up::2 36 2.126e-05 2.126e-05 2.126e-05 0.01% MLMG::mgVcycle_up::0 36 2.101e-05 2.101e-05 2.101e-05 0.01% MLMG::mgVcycle_up::1 36 2.08e-05 2.08e-05 2.08e-05 0.01% makeSFC 30 2.05e-05 2.05e-05 2.05e-05 0.01% Castro::finalize_do_advance() 5 1.996e-05 1.996e-05 1.996e-05 0.01% MLPoisson::define() 6 1.914e-05 1.914e-05 1.914e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.737e-05 1.737e-05 1.737e-05 0.00% MLMG::ResNormInf() 42 1.723e-05 1.723e-05 1.723e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.571e-05 1.571e-05 1.571e-05 0.00% MLMG::mgVcycle_bottom 36 1.4e-05 1.4e-05 1.4e-05 0.00% FillPatchSingleLevel 20 1.4e-05 1.4e-05 1.4e-05 0.00% MLMG::computeResidual() 36 1.369e-05 1.369e-05 1.369e-05 0.00% Castro::construct_new_gravity() 5 1.162e-05 1.162e-05 1.162e-05 0.00% MLPoisson::prepareForSolve() 6 9.625e-06 9.625e-06 9.625e-06 0.00% DistributionMapping::Distribute() 31 9.174e-06 9.174e-06 9.174e-06 0.00% Amr::initSubcycle() 1 8.977e-06 8.977e-06 8.977e-06 0.00% Castro::do_new_sources() 5 8.894e-06 8.894e-06 8.894e-06 0.00% Castro::do_old_sources() 5 7.996e-06 7.996e-06 7.996e-06 0.00% Gravity::actual_multilevel_solve() 1 7.379e-06 7.379e-06 7.379e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.091e-06 7.091e-06 7.091e-06 0.00% Castro::apply_source_to_state() 10 5.71e-06 5.71e-06 5.71e-06 0.00% Castro::check_for_nan() 10 5.633e-06 5.633e-06 5.633e-06 0.00% Castro::construct_old_gravity() 5 4.919e-06 4.919e-06 4.919e-06 0.00% MLLinOp::define() 6 4.863e-06 4.863e-06 4.863e-06 0.00% Castro::FluxRegCrseInit 5 4.134e-06 4.134e-06 4.134e-06 0.00% Gravity::swapTimeLevels() 5 4.009e-06 4.009e-06 4.009e-06 0.00% MLMG::computeMLResidual() 6 3.754e-06 3.754e-06 3.754e-06 0.00% Castro::computeNewDt() 5 3.499e-06 3.499e-06 3.499e-06 0.00% MLMG::getGradSolution() 6 3.128e-06 3.128e-06 3.128e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.885e-06 2.885e-06 2.885e-06 0.00% Castro::retry_advance_ctu() 5 2.851e-06 2.851e-06 2.851e-06 0.00% MLMG::MLResNormInf() 6 2.248e-06 2.248e-06 2.248e-06 0.00% Gravity::set_mass_offset() 6 2.153e-06 2.153e-06 2.153e-06 0.00% MLMG::MLRhsNormInf() 6 2.014e-06 2.014e-06 2.014e-06 0.00% Castro::FluxRegFineAdd() 5 1.866e-06 1.866e-06 1.866e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.007e-06 1.007e-06 1.007e-06 0.00% Amr::init() 1 8.22e-07 8.22e-07 8.22e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3984 0.3984 0.3984 100.00% Amr::coarseTimeStep() 5 0.3179 0.3179 0.3179 79.78% Amr::timeStep() 5 0.3151 0.3151 0.3151 79.07% Castro::advance() 5 0.3122 0.3122 0.3122 78.34% Castro::subcycle_advance_ctu() 5 0.3059 0.3059 0.3059 76.76% Castro::do_advance_ctu() 5 0.3041 0.3041 0.3041 76.32% Castro::construct_ctu_hydro_source() 5 0.1334 0.1334 0.1334 33.48% Castro::construct_new_gravity() 5 0.1298 0.1298 0.1298 32.58% Gravity::solve_phi_with_mlmg() 6 0.125 0.125 0.125 31.38% Gravity::solve_for_phi() 5 0.122 0.122 0.122 30.62% Gravity::actual_solve_with_mlmg() 6 0.1214 0.1214 0.1214 30.46% MLMG::solve() 6 0.1102 0.1102 0.1102 27.65% MLMG::oneIter() 36 0.1034 0.1034 0.1034 25.94% MLMG::mgVcycle() 36 0.1018 0.1018 0.1018 25.55% MLCellLinOp::smooth() 720 0.04976 0.04976 0.04976 12.49% MLCellLinOp::applyBC() 1946 0.0466 0.0466 0.0466 11.69% Amr::init() 1 0.04604 0.04604 0.04604 11.55% Amr::restart() 1 0.04604 0.04604 0.04604 11.55% AmrLevel::restart() 1 0.03992 0.03992 0.03992 10.02% StateData::restartDoit() 4 0.03984 0.03984 0.03984 10.00% VisMF::Read() 3 0.03969 0.03969 0.03969 9.96% Amr::writePlotFile() 1 0.03382 0.03382 0.03382 8.49% VisMF::Write(FabArray) 1 0.03207 0.03207 0.03207 8.05% MLMG::mgVcycle_bottom 36 0.03161 0.03161 0.03161 7.93% MLMG::actualBottomSolve() 36 0.0316 0.0316 0.0316 7.93% MLCGSolver::bicgstab 36 0.03129 0.03129 0.03129 7.85% Castro::clean_state() 30 0.02395 0.02395 0.02395 6.01% FillPatchIterator::Initialize 20 0.0199 0.0199 0.0199 4.99% FillPatchSingleLevel 20 0.01912 0.01912 0.01912 4.80% StateDataPhysBCFunct::() 20 0.01708 0.01708 0.01708 4.29% MLCellLinOp::apply() 500 0.01563 0.01563 0.01563 3.92% MLMG::mgVcycle_down::0 36 0.01433 0.01433 0.01433 3.60% MLPoisson::Fsmooth() 1440 0.0139 0.0139 0.0139 3.49% FabArray::FillBoundary() 1766 0.01356 0.01356 0.01356 3.40% FillBoundary_nowait() 1766 0.01319 0.01319 0.01319 3.31% StateData::FillBoundary(geom) 160 0.01219 0.01219 0.01219 3.06% MLMG::mgVcycle_up::0 36 0.01082 0.01082 0.01082 2.71% MLCellLinOp::correctionResidual() 216 0.009484 0.009484 0.009484 2.38% Castro::normalize_species() 30 0.00939 0.00939 0.00939 2.36% Castro::initialize_do_advance() 5 0.009376 0.009376 0.009376 2.35% Castro::computeTemp() 30 0.009238 0.009238 0.009238 2.32% MLPoisson::define() 6 0.008925 0.008925 0.008925 2.24% amrex::Dot() 484 0.008818 0.008818 0.008818 2.21% MLMG:computeResOfCorrection() 180 0.008332 0.008332 0.008332 2.09% Gravity::get_new_grav_vector() 5 0.007732 0.007732 0.007732 1.94% Castro::construct_old_gravity() 5 0.007477 0.007477 0.007477 1.88% Gravity::get_old_grav_vector() 5 0.007472 0.007472 0.007472 1.88% amrex::Copy() 463 0.006953 0.006953 0.006953 1.74% Castro::do_new_sources() 5 0.006882 0.006882 0.006882 1.73% MLMG::mgVcycle_down::1 36 0.006823 0.006823 0.006823 1.71% FabArray::ParallelCopy() 380 0.006599 0.006599 0.006599 1.66% Castro::do_old_sources() 5 0.00653 0.00653 0.00653 1.64% FabArray::ParallelCopy_nowait() 380 0.006487 0.006487 0.006487 1.63% MLMG::mgVcycle_down::2 36 0.006318 0.006318 0.006318 1.59% FabArray::setVal() 537 0.006273 0.006273 0.006273 1.57% MLMG::mgVcycle_down::3 36 0.006176 0.006176 0.006176 1.55% FabArray::norminf() 326 0.006162 0.006162 0.006162 1.55% MLMG::mgVcycle_down::4 36 0.006111 0.006111 0.006111 1.53% MLCellLinOp::defineAuxData() 6 0.005982 0.005982 0.005982 1.50% Castro::initialize_advance() 5 0.005917 0.005917 0.005917 1.48% MLCGSolver::ParallelAllReduce 659 0.005348 0.005348 0.005348 1.34% Castro::expand_state() 5 0.005283 0.005283 0.005283 1.33% MLMG::addInterpCorrection() 180 0.005168 0.005168 0.005168 1.30% MLMG::mgVcycle_up::4 36 0.004969 0.004969 0.004969 1.25% MLMG::mgVcycle_up::1 36 0.004942 0.004942 0.004942 1.24% amrex::average_down 180 0.004902 0.004902 0.004902 1.23% MLMG::mgVcycle_up::2 36 0.00482 0.00482 0.00482 1.21% MLMG::mgVcycle_up::3 36 0.004727 0.004727 0.004727 1.19% Castro::enforce_min_density() 30 0.004634 0.004634 0.004634 1.16% MLPoisson::Fapply() 500 0.004459 0.004459 0.004459 1.12% FabArray::Saxpy() 355 0.003634 0.003634 0.003634 0.91% Castro::estTimeStep() 10 0.003621 0.003621 0.003621 0.91% Gravity::fill_multipole_BCs() 6 0.003538 0.003538 0.003538 0.89% FabArray::Xpay() 361 0.003495 0.003495 0.003495 0.88% Castro::post_restart() 1 0.003371 0.003371 0.003371 0.85% Gravity::multilevel_solve_for_new_phi() 1 0.003259 0.003259 0.003259 0.82% Gravity::actual_multilevel_solve() 1 0.003243 0.003243 0.003243 0.81% MLCellLinOp::solutionResidual() 42 0.003192 0.003192 0.003192 0.80% Castro::reset_internal_energy(MultiFab) 30 0.003105 0.003105 0.003105 0.78% Castro::post_timestep() 5 0.002852 0.002852 0.002852 0.72% MLCellLinOp::defineBC() 6 0.002719 0.002719 0.002719 0.68% MLMG::computeResidual() 36 0.002645 0.002645 0.002645 0.66% Castro::computeNewDt() 5 0.002644 0.002644 0.002644 0.66% BndryData::define() 6 0.002599 0.002599 0.002599 0.65% MLMG::prepareForSolve() 6 0.002515 0.002515 0.002515 0.63% FabArray::LinComb() 242 0.001932 0.001932 0.001932 0.48% amrex::Add() 72 0.001837 0.001837 0.001837 0.46% Castro::construct_new_source() 25 0.001715 0.001715 0.001715 0.43% Castro::construct_new_gravity_source() 5 0.001579 0.001579 0.001579 0.40% Castro::construct_old_source() 25 0.001476 0.001476 0.001476 0.37% Castro::construct_old_gravity_source() 5 0.001413 0.001413 0.001413 0.35% MLMG::ResNormInf() 42 0.0009262 0.0009262 0.0009262 0.23% Castro::apply_source_to_state() 10 0.000921 0.000921 0.000921 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008964 0.0008964 0.0008964 0.22% Castro::reset_internal_energy(Fab) 240 0.0008384 0.0008384 0.0008384 0.21% check_for_negative_density() 5 0.0008192 0.0008192 0.0008192 0.21% FabArrayBase::getCPC() 632 0.0007765 0.0007765 0.0007765 0.19% MLMG::getGradSolution() 6 0.0007599 0.0007599 0.0007599 0.19% MLCellLinOp::compGrad() 6 0.0007568 0.0007568 0.0007568 0.19% MLCellLinOp::setLevelBC() 6 0.0007443 0.0007443 0.0007443 0.19% FabArray::setDomainBndry() 20 0.000657 0.000657 0.000657 0.16% Castro::enforce_speed_limit() 30 0.0006546 0.0006546 0.0006546 0.16% FabArray::mult() 22 0.0006382 0.0006382 0.0006382 0.16% MLPoisson::prepareForSolve() 6 0.0006192 0.0006192 0.0006192 0.16% MLCellLinOp::prepareForSolve() 6 0.0006096 0.0006096 0.0006096 0.15% Castro::check_for_nan() 10 0.0005958 0.0005958 0.0005958 0.15% MultiFab::contains_nan() 10 0.0005901 0.0005901 0.0005901 0.15% MLMG::computeMLResidual() 6 0.0005646 0.0005646 0.0005646 0.14% Gravity::update_max_rhs() 6 0.0004393 0.0004393 0.0004393 0.11% FabArrayBase::CPC::define() 244 0.0004096 0.0004096 0.0004096 0.10% Amr::InitAmr() 1 0.0003901 0.0003901 0.0003901 0.10% Castro::finalize_advance() 5 0.000317 0.000317 0.000317 0.08% FabArrayBase::getFB() 1766 0.0003104 0.0003104 0.0003104 0.08% Gravity::swapTimeLevels() 5 0.0002419 0.0002419 0.0002419 0.06% MLLinOp::define() 6 0.0002051 0.0002051 0.0002051 0.05% MLLinOp::defineGrids() 6 0.0002003 0.0002003 0.0002003 0.05% MLMG::MLResNormInf() 6 0.000152 0.000152 0.000152 0.04% Castro::buildMetrics() 1 0.0001483 0.0001483 0.0001483 0.04% MultiFab::max() 6 0.0001336 0.0001336 0.0001336 0.03% MLMG::MLRhsNormInf() 6 0.0001169 0.0001169 0.0001169 0.03% Castro::create_source_corrector() 5 8.359e-05 8.359e-05 8.359e-05 0.02% FabArrayBase::FB::FB() 26 5.863e-05 5.863e-05 5.863e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.519e-05 3.519e-05 3.519e-05 0.01% Castro::initMFs() 1 2.853e-05 2.853e-05 2.853e-05 0.01% makeSFC 30 2.81e-05 2.81e-05 2.81e-05 0.01% Amr::writeSmallPlotFile() 1 2.77e-05 2.77e-05 2.77e-05 0.01% Castro::swap_state_time_levels() 5 2.215e-05 2.215e-05 2.215e-05 0.01% Castro::finalize_do_advance() 5 1.996e-05 1.996e-05 1.996e-05 0.01% DistributionMapping::Distribute() 31 9.174e-06 9.174e-06 9.174e-06 0.00% Amr::initSubcycle() 1 8.977e-06 8.977e-06 8.977e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.459e-06 4.459e-06 4.459e-06 0.00% Castro::FluxRegCrseInit 5 4.134e-06 4.134e-06 4.134e-06 0.00% Castro::retry_advance_ctu() 5 2.851e-06 2.851e-06 2.851e-06 0.00% Gravity::set_mass_offset() 6 2.153e-06 2.153e-06 2.153e-06 0.00% Castro::FluxRegFineAdd() 5 1.866e-06 1.866e-06 1.866e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.007e-06 1.007e-06 1.007e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 209 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 151 MiB 692 MiB Castro::initMFs() 48 48 60 MiB 68 MiB Castro::swap_state_time_levels() 32 32 49 MiB 55 MiB StateData::restartDoit() 32 32 53 MiB 55 MiB FillPatchIterator::Initialize 160 160 1066 KiB 39 MiB Castro::initialize_do_advance() 40 40 29 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 2373 KiB 28 MiB Castro::initialize_advance() 40 40 18 MiB 23 MiB Castro::buildMetrics() 32 32 14 MiB 15 MiB Castro::post_restart() 48 48 6752 KiB 14 MiB MLMG::prepareForSolve() 361 361 3395 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 194 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 198 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6748 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 14 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2130 B 2048 KiB Gravity::solve_for_phi() 40 40 625 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 16 KiB 2048 KiB BndryData::define() 576 576 312 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 200 KiB 671 KiB Castro::estTimeStep() 10 10 4255 B 480 KiB VisMF::Write(FabArray) 112 112 1244 B 320 KiB Castro::normalize_species() 30 30 7667 B 320 KiB amrex::average_down 469 469 1217 B 257 KiB MLMG::addInterpCorrection() 468 468 1106 B 257 KiB amrex::Dot() 592 592 3258 B 160 KiB FabArray::norminf() 398 398 2283 B 160 KiB check_for_negative_density() 5 5 328 B 160 KiB MultiFab::max() 6 6 51 B 160 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MultiFab::contains_nan() 10 10 29 B 20 KiB MLPoisson::Fsmooth() 60 60 3302 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 49 B 10 KiB FillBoundary_nowait() 336 336 280 B 9648 B MLCellLinOp::applyBC() 3892 3892 214 B 9344 B amrex::Copy() 56 56 5989 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B StateData::FillBoundary(geom) 960 960 41 B 2784 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLCGSolver::bicgstab 324 324 114 B 1472 B MLCellLinOp::defineBC() 36 36 350 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1457 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 114 KiB 8192 KiB VisMF::Write(FabArray) 120 120 248 KiB 3584 KiB VisMF::Read() 24 24 193 KiB 3000 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3302 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 49 B 10 KiB FillBoundary_nowait() 336 336 281 B 9648 B MLCellLinOp::applyBC() 1946 1946 212 B 9328 B amrex::Copy() 56 56 5989 B 8464 B MLCellLinOp::prepareForSolve() 36 36 3 B 7792 B Gravity::get_old_grav_vector() 3 3 2666 B 3072 B Gravity::fill_multipole_BCs() 18 18 4 B 2832 B StateData::FillBoundary(geom) 960 960 41 B 2784 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::prepareForSolve() 7 7 523 B 1296 B amrex::average_down 37 37 225 B 1296 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 23 B 400 B FabArray::norminf() 398 398 16 B 272 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2424 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.06-3-g572f226f4c98) finalized