Initializing CUDA... CUDA initialized with 1 device. AMReX (23.03-8-gaaab804f1438) initialized Starting run at 09:08:11 UTC on 2023-03-14. Successfully read inputs file ... Castro git describe: 23.03-2-g0304294be AMReX git describe: 23.03-8-gaaab804f1 Microphysics git describe: 23.03-12-g1f07dde0 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.05689462 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.032448901 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.042974967 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.046961985 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.054572396 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.054566358 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.076908934 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.05611978 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.060812019 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049491494 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.054977635 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.053811123 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.057381365 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.109658321 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032470792 seconds Ending run at 09:08:12 UTC on 2023-03-14. Run time = 0.888081115 Run time without initialization = 0.751383035 Average number of zones advanced per microsecond: 3.489 Average number of zones advanced per microsecond per rank: 3.489 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8881 ... 0.8881 ... 0.8881 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2276 0.2276 0.2276 25.62% Castro::construct_ctu_hydro_source() 10 0.2072 0.2072 0.2072 23.33% MLCellLinOp::applyBC() 4433 0.0746 0.0746 0.0746 8.40% Amr::checkPoint() 3 0.05715 0.05715 0.05715 6.44% MLPoisson::Fsmooth() 3280 0.03143 0.03143 0.03143 3.54% FillBoundary_nowait() 4023 0.03117 0.03117 0.03117 3.51% StateData::FillBoundary(geom) 328 0.02403 0.02403 0.02403 2.71% amrex::Dot() 1114 0.01985 0.01985 0.01985 2.23% Castro::normalize_species() 62 0.01559 0.01559 0.01559 1.75% amrex::Copy() 1029 0.01454 0.01454 0.01454 1.64% StateDataPhysBCFunct::() 41 0.0139 0.0139 0.0139 1.57% FabArray::norminf() 743 0.01381 0.01381 0.01381 1.55% FabArray::ParallelCopy_nowait() 861 0.01297 0.01297 0.01297 1.46% FabArray::setVal() 1144 0.01281 0.01281 0.01281 1.44% Castro::computeTemp() 63 0.01224 0.01224 0.01224 1.38% MLPoisson::Fapply() 1142 0.01007 0.01007 0.01007 1.13% MLCellLinOp::defineAuxData() 11 0.009552 0.009552 0.009552 1.08% FabArray::Saxpy() 813 0.007929 0.007929 0.007929 0.89% FabArray::Xpay() 821 0.007846 0.007846 0.007846 0.88% MLMG::addInterpCorrection() 410 0.00661 0.00661 0.00661 0.74% Castro::enforce_min_density() 62 0.006067 0.006067 0.006067 0.68% Gravity::fill_multipole_BCs() 11 0.005931 0.005931 0.005931 0.67% amrex::average_down 410 0.005795 0.005795 0.005795 0.65% FabArray::LinComb() 557 0.00434 0.00434 0.00434 0.49% amrex::Add() 164 0.004333 0.004333 0.004333 0.49% Castro::estTimeStep() 21 0.00419 0.00419 0.00419 0.47% Castro::reset_internal_energy(MultiFab) 63 0.003924 0.003924 0.003924 0.44% BndryData::define() 11 0.003522 0.003522 0.003522 0.40% Castro::construct_new_gravity_source() 10 0.00287 0.00287 0.00287 0.32% Amr::writePlotFile() 2 0.00232 0.00232 0.00232 0.26% MLCGSolver::bicgstab 82 0.002253 0.002253 0.002253 0.25% Castro::construct_old_gravity_source() 10 0.002096 0.002096 0.002096 0.24% Castro::do_advance_ctu() 10 0.001902 0.001902 0.001902 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001667 0.001667 0.001667 0.19% Castro::enforce_speed_limit() 62 0.001531 0.001531 0.001531 0.17% Castro::reset_internal_energy(Fab) 504 0.001517 0.001517 0.001517 0.17% Gravity::actual_solve_with_mlmg() 11 0.001468 0.001468 0.001468 0.17% MLCellLinOp::setLevelBC() 11 0.001366 0.001366 0.001366 0.15% FabArray::mult() 43 0.001314 0.001314 0.001314 0.15% FabArray::setDomainBndry() 41 0.001289 0.001289 0.001289 0.15% MultiFab::contains_nan() 20 0.001202 0.001202 0.001202 0.14% Castro::initData() 1 0.001169 0.001169 0.001169 0.13% MLCellLinOp::prepareForSolve() 11 0.0011 0.0011 0.0011 0.12% MLCellLinOp::smooth() 1640 0.001086 0.001086 0.001086 0.12% MLCellLinOp::compGrad() 11 0.0009087 0.0009087 0.0009087 0.10% MLMG::prepareForSolve() 11 0.000846 0.000846 0.000846 0.10% FabArray::FillBoundary() 4023 0.0008041 0.0008041 0.0008041 0.09% FabArrayBase::getCPC() 1323 0.0007438 0.0007438 0.0007438 0.08% FabArrayBase::CPC::define() 454 0.0006758 0.0006758 0.0006758 0.08% Gravity::get_new_grav_vector() 11 0.0006267 0.0006267 0.0006267 0.07% FabArrayBase::getFB() 4023 0.000594 0.000594 0.000594 0.07% Gravity::get_old_grav_vector() 10 0.0005485 0.0005485 0.0005485 0.06% MLCellLinOp::apply() 1142 0.0004596 0.0004596 0.0004596 0.05% Amr::InitAmr() 1 0.0004561 0.0004561 0.0004561 0.05% MLLinOp::defineGrids() 11 0.0004025 0.0004025 0.0004025 0.05% MLMG::mgVcycle() 82 0.0004018 0.0004018 0.0004018 0.05% Amr::coarseTimeStep() 10 0.0003896 0.0003896 0.0003896 0.04% main() 1 0.0002886 0.0002886 0.0002886 0.03% MLCGSolver::ParallelAllReduce 1514 0.0002812 0.0002812 0.0002812 0.03% FillPatchIterator::Initialize 41 0.0002598 0.0002598 0.0002598 0.03% MultiFab::max() 11 0.0002594 0.0002594 0.0002594 0.03% FabArray::ParallelCopy() 861 0.0002472 0.0002472 0.0002472 0.03% MLCellLinOp::correctionResidual() 492 0.0002217 0.0002217 0.0002217 0.02% MLCellLinOp::defineBC() 11 0.0002085 0.0002085 0.0002085 0.02% Amr::timeStep() 10 0.0001788 0.0001788 0.0001788 0.02% StateData::checkPoint() 12 0.000162 0.000162 0.000162 0.02% Castro::subcycle_advance_ctu() 10 0.0001517 0.0001517 0.0001517 0.02% Gravity::update_max_rhs() 11 0.0001198 0.0001198 0.0001198 0.01% Gravity::solve_for_phi() 10 0.0001178 0.0001178 0.0001178 0.01% MLMG:computeResOfCorrection() 410 0.0001152 0.0001152 0.0001152 0.01% AmrLevel::checkPoint() 3 0.0001126 0.0001126 0.0001126 0.01% Castro::finalize_advance() 10 0.0001112 0.0001112 0.0001112 0.01% Castro::advance() 10 0.0001011 0.0001011 0.0001011 0.01% MLMG::mgVcycle_down::0 82 9.341e-05 9.341e-05 9.341e-05 0.01% MLMG::actualBottomSolve() 82 9.339e-05 9.339e-05 9.339e-05 0.01% FabArrayBase::FB::FB() 56 8.675e-05 8.675e-05 8.675e-05 0.01% MLMG::mgVcycle_down::1 82 8.513e-05 8.513e-05 8.513e-05 0.01% Castro::Castro() 1 8.441e-05 8.441e-05 8.441e-05 0.01% MLMG::mgVcycle_down::2 82 8.347e-05 8.347e-05 8.347e-05 0.01% MLMG::solve() 11 8.125e-05 8.125e-05 8.125e-05 0.01% Castro::expand_state() 10 7.973e-05 7.973e-05 7.973e-05 0.01% MLMG::mgVcycle_down::3 82 7.965e-05 7.965e-05 7.965e-05 0.01% MLMG::mgVcycle_down::4 82 7.869e-05 7.869e-05 7.869e-05 0.01% Castro::clean_state() 62 7.75e-05 7.75e-05 7.75e-05 0.01% Castro::initialize_advance() 10 7.32e-05 7.32e-05 7.32e-05 0.01% MLMG::mgVcycle_up::4 82 6.363e-05 6.363e-05 6.363e-05 0.01% MLMG::mgVcycle_up::3 82 5.917e-05 5.917e-05 5.917e-05 0.01% MLMG::oneIter() 82 5.63e-05 5.63e-05 5.63e-05 0.01% MLMG::mgVcycle_up::1 82 5.62e-05 5.62e-05 5.62e-05 0.01% MLMG::mgVcycle_up::0 82 5.584e-05 5.584e-05 5.584e-05 0.01% MLMG::mgVcycle_up::2 82 5.41e-05 5.41e-05 5.41e-05 0.01% Castro::initialize_do_advance() 10 5.086e-05 5.086e-05 5.086e-05 0.01% MLCellLinOp::solutionResidual() 93 4.838e-05 4.838e-05 4.838e-05 0.01% Castro::swap_state_time_levels() 10 4.043e-05 4.043e-05 4.043e-05 0.00% Castro::finalize_do_advance() 10 3.86e-05 3.86e-05 3.86e-05 0.00% MLMG::mgVcycle_bottom 82 3.42e-05 3.42e-05 3.42e-05 0.00% Castro::enforce_consistent_e() 1 3.414e-05 3.414e-05 3.414e-05 0.00% StateData::define() 4 3.402e-05 3.402e-05 3.402e-05 0.00% MLMG::computeResidual() 82 3.354e-05 3.354e-05 3.354e-05 0.00% MLMG::ResNormInf() 93 3.299e-05 3.299e-05 3.299e-05 0.00% FillPatchSingleLevel 41 2.952e-05 2.952e-05 2.952e-05 0.00% makeSFC 55 2.7e-05 2.7e-05 2.7e-05 0.00% Amr::writeSmallPlotFile() 1 2.494e-05 2.494e-05 2.494e-05 0.00% Castro::construct_new_gravity() 10 2.482e-05 2.482e-05 2.482e-05 0.00% MLPoisson::define() 11 2.239e-05 2.239e-05 2.239e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.174e-05 2.174e-05 2.174e-05 0.00% Amr::FinalizeInit() 1 2.088e-05 2.088e-05 2.088e-05 0.00% Castro::initMFs() 1 2.079e-05 2.079e-05 2.079e-05 0.00% Castro::buildMetrics() 1 1.823e-05 1.823e-05 1.823e-05 0.00% Castro::construct_old_source() 50 1.767e-05 1.767e-05 1.767e-05 0.00% Castro::do_new_sources() 10 1.72e-05 1.72e-05 1.72e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.678e-05 1.678e-05 1.678e-05 0.00% DistributionMapping::Distribute() 56 1.636e-05 1.636e-05 1.636e-05 0.00% Castro::do_old_sources() 10 1.597e-05 1.597e-05 1.597e-05 0.00% Castro::construct_new_source() 50 1.592e-05 1.592e-05 1.592e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.582e-05 1.582e-05 1.582e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.551e-05 1.551e-05 1.551e-05 0.00% MLPoisson::prepareForSolve() 11 1.397e-05 1.397e-05 1.397e-05 0.00% Amr::defBaseLevel() 1 1.307e-05 1.307e-05 1.307e-05 0.00% Castro::check_for_nan() 20 1.095e-05 1.095e-05 1.095e-05 0.00% MLLinOp::define() 11 1.063e-05 1.063e-05 1.063e-05 0.00% Castro::apply_source_to_state() 20 1.025e-05 1.025e-05 1.025e-05 0.00% Gravity::swapTimeLevels() 10 9.975e-06 9.975e-06 9.975e-06 0.00% Castro::construct_old_gravity() 10 9.551e-06 9.551e-06 9.551e-06 0.00% Castro::post_timestep() 10 8.482e-06 8.482e-06 8.482e-06 0.00% Gravity::actual_multilevel_solve() 1 8.226e-06 8.226e-06 8.226e-06 0.00% Amr::initSubcycle() 1 8.098e-06 8.098e-06 8.098e-06 0.00% Castro::computeNewDt() 9 7.623e-06 7.623e-06 7.623e-06 0.00% MLMG::computeMLResidual() 11 7.539e-06 7.539e-06 7.539e-06 0.00% MLMG::getGradSolution() 11 5.782e-06 5.782e-06 5.782e-06 0.00% Gravity::set_mass_offset() 11 4.885e-06 4.885e-06 4.885e-06 0.00% AmrLevel::checkPointPost() 3 4.681e-06 4.681e-06 4.681e-06 0.00% Amr::InitializeInit() 1 4.566e-06 4.566e-06 4.566e-06 0.00% Castro::post_init() 1 4.506e-06 4.506e-06 4.506e-06 0.00% MLMG::MLRhsNormInf() 11 3.82e-06 3.82e-06 3.82e-06 0.00% Castro::retry_advance_ctu() 10 3.777e-06 3.777e-06 3.777e-06 0.00% MLMG::MLResNormInf() 11 3.436e-06 3.436e-06 3.436e-06 0.00% AmrLevel::checkPointPre() 3 3.412e-06 3.412e-06 3.412e-06 0.00% Castro::FluxRegCrseInit 10 3.314e-06 3.314e-06 3.314e-06 0.00% Castro::create_source_corrector() 10 3.173e-06 3.173e-06 3.173e-06 0.00% Amr::init() 1 2.821e-06 2.821e-06 2.821e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.664e-06 2.664e-06 2.664e-06 0.00% Castro::computeInitialDt() 2 2.64e-06 2.64e-06 2.64e-06 0.00% Castro::FluxRegFineAdd() 10 2.218e-06 2.218e-06 2.218e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.957e-06 1.957e-06 1.957e-06 0.00% Amr::initialInit() 1 1.064e-06 1.064e-06 1.064e-06 0.00% Castro::post_regrid() 1 9.76e-07 9.76e-07 9.76e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8881 0.8881 0.8881 100.00% Amr::coarseTimeStep() 10 0.7187 0.7187 0.7187 80.92% Amr::timeStep() 10 0.5499 0.5499 0.5499 61.92% Castro::advance() 10 0.5429 0.5429 0.5429 61.14% Castro::subcycle_advance_ctu() 10 0.5314 0.5314 0.5314 59.83% Castro::do_advance_ctu() 10 0.5312 0.5312 0.5312 59.82% Gravity::solve_phi_with_mlmg() 11 0.276 0.276 0.276 31.08% Gravity::actual_solve_with_mlmg() 11 0.2696 0.2696 0.2696 30.36% Castro::construct_new_gravity() 10 0.2521 0.2521 0.2521 28.38% MLMG::solve() 11 0.2493 0.2493 0.2493 28.07% Gravity::solve_for_phi() 10 0.2367 0.2367 0.2367 26.65% MLMG::oneIter() 82 0.2353 0.2353 0.2353 26.50% MLMG::mgVcycle() 82 0.2317 0.2317 0.2317 26.08% VisMF::Write(FabArray) 11 0.2276 0.2276 0.2276 25.62% Amr::checkPoint() 3 0.2228 0.2228 0.2228 25.09% Castro::construct_ctu_hydro_source() 10 0.2072 0.2072 0.2072 23.33% AmrLevel::checkPoint() 3 0.1656 0.1656 0.1656 18.65% StateData::checkPoint() 12 0.1655 0.1655 0.1655 18.64% Amr::init() 1 0.1361 0.1361 0.1361 15.33% MLCellLinOp::smooth() 1640 0.1145 0.1145 0.1145 12.89% MLCellLinOp::applyBC() 4433 0.1073 0.1073 0.1073 12.08% MLMG::mgVcycle_bottom 82 0.07113 0.07113 0.07113 8.01% MLMG::actualBottomSolve() 82 0.07109 0.07109 0.07109 8.00% MLCGSolver::bicgstab 82 0.07041 0.07041 0.07041 7.93% Amr::writePlotFile() 2 0.06504 0.06504 0.06504 7.32% Amr::initialInit() 1 0.04664 0.04664 0.04664 5.25% FillPatchIterator::Initialize 41 0.04359 0.04359 0.04359 4.91% Amr::FinalizeInit() 1 0.04285 0.04285 0.04285 4.83% FillPatchSingleLevel 41 0.04205 0.04205 0.04205 4.73% Castro::post_init() 1 0.04159 0.04159 0.04159 4.68% Castro::clean_state() 62 0.04021 0.04021 0.04021 4.53% Gravity::multilevel_solve_for_new_phi() 1 0.03974 0.03974 0.03974 4.47% Gravity::actual_multilevel_solve() 1 0.03972 0.03972 0.03972 4.47% StateDataPhysBCFunct::() 41 0.03793 0.03793 0.03793 4.27% MLCellLinOp::apply() 1142 0.03536 0.03536 0.03536 3.98% MLMG::mgVcycle_down::0 82 0.0335 0.0335 0.0335 3.77% FabArray::FillBoundary() 4023 0.03266 0.03266 0.03266 3.68% FillBoundary_nowait() 4023 0.03185 0.03185 0.03185 3.59% MLPoisson::Fsmooth() 3280 0.03143 0.03143 0.03143 3.54% MLMG::mgVcycle_up::0 82 0.02544 0.02544 0.02544 2.87% StateData::FillBoundary(geom) 328 0.02403 0.02403 0.02403 2.71% MLCellLinOp::correctionResidual() 492 0.02168 0.02168 0.02168 2.44% amrex::Dot() 1114 0.01985 0.01985 0.01985 2.23% Castro::initialize_do_advance() 10 0.01972 0.01972 0.01972 2.22% MLMG:computeResOfCorrection() 410 0.01912 0.01912 0.01912 2.15% Castro::computeTemp() 63 0.01768 0.01768 0.01768 1.99% Gravity::get_new_grav_vector() 11 0.01691 0.01691 0.01691 1.90% MLPoisson::define() 11 0.01607 0.01607 0.01607 1.81% Castro::normalize_species() 62 0.01559 0.01559 0.01559 1.75% MLMG::mgVcycle_down::1 82 0.01534 0.01534 0.01534 1.73% Castro::construct_old_gravity() 10 0.01515 0.01515 0.01515 1.71% Gravity::get_old_grav_vector() 10 0.01514 0.01514 0.01514 1.71% amrex::Copy() 1029 0.01454 0.01454 0.01454 1.64% MLMG::mgVcycle_down::2 82 0.01427 0.01427 0.01427 1.61% FabArray::ParallelCopy() 861 0.01401 0.01401 0.01401 1.58% MLMG::mgVcycle_down::3 82 0.01392 0.01392 0.01392 1.57% FabArray::norminf() 743 0.01381 0.01381 0.01381 1.55% FabArray::ParallelCopy_nowait() 861 0.01377 0.01377 0.01377 1.55% MLMG::mgVcycle_down::4 82 0.01373 0.01373 0.01373 1.55% Castro::expand_state() 10 0.01294 0.01294 0.01294 1.46% FabArray::setVal() 1144 0.01281 0.01281 0.01281 1.44% MLCGSolver::ParallelAllReduce 1514 0.01191 0.01191 0.01191 1.34% MLMG::addInterpCorrection() 410 0.01157 0.01157 0.01157 1.30% Castro::do_new_sources() 10 0.01142 0.01142 0.01142 1.29% MLMG::mgVcycle_up::4 82 0.01119 0.01119 0.01119 1.26% MLMG::mgVcycle_up::1 82 0.01117 0.01117 0.01117 1.26% MLCellLinOp::defineAuxData() 11 0.01089 0.01089 0.01089 1.23% MLMG::mgVcycle_up::2 82 0.01089 0.01089 0.01089 1.23% Castro::initialize_advance() 10 0.01082 0.01082 0.01082 1.22% amrex::average_down 410 0.01077 0.01077 0.01077 1.21% MLMG::mgVcycle_up::3 82 0.01067 0.01067 0.01067 1.20% MLPoisson::Fapply() 1142 0.01007 0.01007 0.01007 1.13% Castro::do_old_sources() 10 0.0093 0.0093 0.0093 1.05% FabArray::Saxpy() 813 0.007929 0.007929 0.007929 0.89% FabArray::Xpay() 821 0.007846 0.007846 0.007846 0.88% MLCellLinOp::solutionResidual() 93 0.007073 0.007073 0.007073 0.80% Castro::post_timestep() 10 0.0068 0.0068 0.0068 0.77% Gravity::fill_multipole_BCs() 11 0.006203 0.006203 0.006203 0.70% MLMG::computeResidual() 82 0.006094 0.006094 0.006094 0.69% Castro::enforce_min_density() 62 0.006067 0.006067 0.006067 0.68% Castro::reset_internal_energy(MultiFab) 63 0.005441 0.005441 0.005441 0.61% MLCellLinOp::defineBC() 11 0.00468 0.00468 0.00468 0.53% MLMG::prepareForSolve() 11 0.004535 0.004535 0.004535 0.51% BndryData::define() 11 0.004472 0.004472 0.004472 0.50% FabArray::LinComb() 557 0.00434 0.00434 0.00434 0.49% amrex::Add() 164 0.004333 0.004333 0.004333 0.49% Castro::estTimeStep() 21 0.00419 0.00419 0.00419 0.47% Amr::InitializeInit() 1 0.003789 0.003789 0.003789 0.43% Amr::defBaseLevel() 1 0.003784 0.003784 0.003784 0.43% Castro::initData() 1 0.003299 0.003299 0.003299 0.37% Castro::construct_new_source() 50 0.002886 0.002886 0.002886 0.32% Castro::construct_new_gravity_source() 10 0.00287 0.00287 0.00287 0.32% Castro::construct_old_source() 50 0.002114 0.002114 0.002114 0.24% Castro::construct_old_gravity_source() 10 0.002096 0.002096 0.002096 0.24% Castro::computeNewDt() 9 0.00209 0.00209 0.00209 0.24% MLMG::ResNormInf() 93 0.002061 0.002061 0.002061 0.23% Castro::apply_source_to_state() 20 0.001814 0.001814 0.001814 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001667 0.001667 0.001667 0.19% Castro::enforce_speed_limit() 62 0.001531 0.001531 0.001531 0.17% Castro::reset_internal_energy(Fab) 504 0.001517 0.001517 0.001517 0.17% FabArrayBase::getCPC() 1323 0.00142 0.00142 0.00142 0.16% MLMG::getGradSolution() 11 0.00138 0.00138 0.00138 0.16% MLCellLinOp::compGrad() 11 0.001374 0.001374 0.001374 0.15% MLCellLinOp::setLevelBC() 11 0.001366 0.001366 0.001366 0.15% FabArray::mult() 43 0.001314 0.001314 0.001314 0.15% FabArray::setDomainBndry() 41 0.001289 0.001289 0.001289 0.15% Castro::check_for_nan() 20 0.001213 0.001213 0.001213 0.14% MultiFab::contains_nan() 20 0.001202 0.001202 0.001202 0.14% MLPoisson::prepareForSolve() 11 0.001114 0.001114 0.001114 0.13% MLCellLinOp::prepareForSolve() 11 0.0011 0.0011 0.0011 0.12% Castro::post_regrid() 1 0.001092 0.001092 0.001092 0.12% MLMG::computeMLResidual() 11 0.00102 0.00102 0.00102 0.11% Gravity::update_max_rhs() 11 0.0008228 0.0008228 0.0008228 0.09% FabArrayBase::getFB() 4023 0.0006808 0.0006808 0.0006808 0.08% FabArrayBase::CPC::define() 454 0.0006758 0.0006758 0.0006758 0.08% Castro::finalize_advance() 10 0.0006235 0.0006235 0.0006235 0.07% Castro::computeInitialDt() 2 0.0005744 0.0005744 0.0005744 0.06% MLLinOp::define() 11 0.0004729 0.0004729 0.0004729 0.05% Amr::InitAmr() 1 0.0004642 0.0004642 0.0004642 0.05% MLLinOp::defineGrids() 11 0.0004623 0.0004623 0.0004623 0.05% Gravity::swapTimeLevels() 10 0.0004352 0.0004352 0.0004352 0.05% Castro::Castro() 1 0.0004184 0.0004184 0.0004184 0.05% MLMG::MLResNormInf() 11 0.0002838 0.0002838 0.0002838 0.03% MultiFab::max() 11 0.0002594 0.0002594 0.0002594 0.03% MLMG::MLRhsNormInf() 11 0.0002165 0.0002165 0.0002165 0.02% Castro::buildMetrics() 1 0.0001517 0.0001517 0.0001517 0.02% FabArrayBase::FB::FB() 56 8.675e-05 8.675e-05 8.675e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.781e-05 5.781e-05 5.781e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.984e-05 4.984e-05 4.984e-05 0.01% makeSFC 55 4.229e-05 4.229e-05 4.229e-05 0.00% Castro::swap_state_time_levels() 10 4.043e-05 4.043e-05 4.043e-05 0.00% Castro::finalize_do_advance() 10 3.86e-05 3.86e-05 3.86e-05 0.00% Castro::enforce_consistent_e() 1 3.414e-05 3.414e-05 3.414e-05 0.00% StateData::define() 4 3.402e-05 3.402e-05 3.402e-05 0.00% Amr::writeSmallPlotFile() 1 2.494e-05 2.494e-05 2.494e-05 0.00% Castro::initMFs() 1 2.079e-05 2.079e-05 2.079e-05 0.00% DistributionMapping::Distribute() 56 1.636e-05 1.636e-05 1.636e-05 0.00% Amr::initSubcycle() 1 8.098e-06 8.098e-06 8.098e-06 0.00% Gravity::set_mass_offset() 11 4.885e-06 4.885e-06 4.885e-06 0.00% AmrLevel::checkPointPost() 3 4.681e-06 4.681e-06 4.681e-06 0.00% Castro::retry_advance_ctu() 10 3.777e-06 3.777e-06 3.777e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.729e-06 3.729e-06 3.729e-06 0.00% AmrLevel::checkPointPre() 3 3.412e-06 3.412e-06 3.412e-06 0.00% Castro::FluxRegCrseInit 10 3.314e-06 3.314e-06 3.314e-06 0.00% Castro::create_source_corrector() 10 3.173e-06 3.173e-06 3.173e-06 0.00% Castro::FluxRegFineAdd() 10 2.218e-06 2.218e-06 2.218e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.957e-06 1.957e-06 1.957e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 97 MiB 9042 MiB Castro::construct_ctu_hydro_source() 2880 2880 107 MiB 692 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1081 KiB 39 MiB Castro::initialize_do_advance() 80 80 23 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 2057 KiB 28 MiB Castro::initialize_advance() 80 80 14 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7609 KiB 14 MiB MLMG::prepareForSolve() 660 660 3454 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 197 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 174 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7530 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 10 KiB 2053 KiB Gravity::update_max_rhs() 88 88 1775 B 2048 KiB Gravity::solve_for_phi() 80 80 545 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 91 KiB 2048 KiB BndryData::define() 1056 1056 315 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 200 KiB 671 KiB Castro::estTimeStep() 21 21 1858 B 480 KiB VisMF::Write(FabArray) 656 656 3324 B 320 KiB Castro::normalize_species() 62 62 5714 B 320 KiB amrex::average_down 1067 1067 1223 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1119 B 257 KiB amrex::Dot() 1360 1360 3297 B 160 KiB FabArray::norminf() 907 907 2299 B 160 KiB Castro::do_advance_ctu() 10 10 318 B 160 KiB MultiFab::max() 11 11 45 B 160 KiB Castro::initData() 1 1 26 B 160 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MultiFab::contains_nan() 20 20 26 B 20 KiB MLPoisson::Fsmooth() 132 132 3377 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 44 B 10 KiB FillBoundary_nowait() 760 760 308 B 9648 B MLCellLinOp::applyBC() 8866 8866 218 B 9344 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3814 B 6144 B StateData::FillBoundary(geom) 1992 1992 53 B 2976 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCGSolver::bicgstab 738 738 115 B 1472 B MLCellLinOp::defineBC() 66 66 355 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 522 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 49 KiB 8192 KiB VisMF::Write(FabArray) 744 744 596 KiB 3584 KiB FabArray::setVal() 106 106 21 KiB 26 KiB MLPoisson::Fsmooth() 132 132 3377 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 44 B 10 KiB FillBoundary_nowait() 760 760 308 B 9648 B MLCellLinOp::applyBC() 4433 4433 216 B 9328 B MLCellLinOp::prepareForSolve() 66 66 2 B 7792 B amrex::Copy() 100 100 3814 B 6144 B Gravity::get_new_grav_vector() 3 3 2911 B 3072 B StateData::FillBoundary(geom) 1992 1992 53 B 2976 B Gravity::fill_multipole_BCs() 33 33 3 B 2832 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B amrex::average_down 83 83 272 B 1296 B MLMG::addInterpCorrection() 82 82 2 B 1024 B MLMG::prepareForSolve() 11 11 287 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 24 B 400 B FabArray::norminf() 907 907 16 B 272 B Castro::estTimeStep() 21 21 0 B 32 B MultiFab::contains_nan() 20 20 0 B 16 B MultiFab::max() 11 11 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B Castro::do_advance_ctu() 10 10 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.03-8-gaaab804f1438) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.03-8-gaaab804f1438) initialized Starting run at 09:08:13 UTC on 2023-03-14. Successfully read inputs file ... Castro git describe: 23.03-2-g0304294be AMReX git describe: 23.03-8-gaaab804f1 Microphysics git describe: 23.03-12-g1f07dde0 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.468809161 Restart time = 0.046569544 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.047504804 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.0458956 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.047112574 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.048558131 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.059403079 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032262757 seconds Ending run at 09:08:13 UTC on 2023-03-14. Run time = 0.328282502 Run time without initialization = 0.281147077 Average number of zones advanced per microsecond: 4.662 Average number of zones advanced per microsecond per rank: 4.662 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3283 ... 0.3283 ... 0.3283 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0748 0.0748 0.0748 22.78% VisMF::Read() 3 0.04057 0.04057 0.04057 12.36% MLCellLinOp::applyBC() 1946 0.03206 0.03206 0.03206 9.77% VisMF::Write(FabArray) 1 0.03076 0.03076 0.03076 9.37% MLPoisson::Fsmooth() 1440 0.01355 0.01355 0.01355 4.13% FillBoundary_nowait() 1766 0.01257 0.01257 0.01257 3.83% StateData::FillBoundary(geom) 160 0.01189 0.01189 0.01189 3.62% amrex::Dot() 484 0.008472 0.008472 0.008472 2.58% amrex::Copy() 463 0.006783 0.006783 0.006783 2.07% Castro::normalize_species() 30 0.006489 0.006489 0.006489 1.98% Castro::computeTemp() 30 0.0063 0.0063 0.0063 1.92% FabArray::setVal() 537 0.006072 0.006072 0.006072 1.85% FabArray::norminf() 326 0.005988 0.005988 0.005988 1.82% StateDataPhysBCFunct::() 20 0.005895 0.005895 0.005895 1.80% FabArray::ParallelCopy_nowait() 380 0.005854 0.005854 0.005854 1.78% MLCellLinOp::defineAuxData() 6 0.004973 0.004973 0.004973 1.51% MLPoisson::Fapply() 500 0.004324 0.004324 0.004324 1.32% FabArray::Saxpy() 355 0.003531 0.003531 0.003531 1.08% FabArray::Xpay() 361 0.003389 0.003389 0.003389 1.03% Gravity::fill_multipole_BCs() 6 0.003039 0.003039 0.003039 0.93% Castro::estTimeStep() 10 0.002997 0.002997 0.002997 0.91% MLMG::addInterpCorrection() 180 0.002866 0.002866 0.002866 0.87% amrex::average_down 180 0.002519 0.002519 0.002519 0.77% Amr::restart() 1 0.002413 0.002413 0.002413 0.74% Castro::enforce_min_density() 30 0.002291 0.002291 0.002291 0.70% Castro::reset_internal_energy(MultiFab) 30 0.001973 0.001973 0.001973 0.60% BndryData::define() 6 0.00191 0.00191 0.00191 0.58% FabArray::LinComb() 242 0.001849 0.001849 0.001849 0.56% amrex::Add() 72 0.001837 0.001837 0.001837 0.56% Castro::construct_new_gravity_source() 5 0.001635 0.001635 0.001635 0.50% Castro::reset_internal_energy(Fab) 240 0.00138 0.00138 0.00138 0.42% Amr::writePlotFile() 1 0.001332 0.001332 0.001332 0.41% Castro::construct_old_gravity_source() 5 0.001296 0.001296 0.001296 0.39% Castro::do_advance_ctu() 5 0.001132 0.001132 0.001132 0.34% MLCGSolver::bicgstab 36 0.0009911 0.0009911 0.0009911 0.30% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008724 0.0008724 0.0008724 0.27% Gravity::actual_solve_with_mlmg() 6 0.0007788 0.0007788 0.0007788 0.24% MLCellLinOp::setLevelBC() 6 0.0007299 0.0007299 0.0007299 0.22% FabArray::mult() 22 0.0006409 0.0006409 0.0006409 0.20% FabArray::setDomainBndry() 20 0.0006333 0.0006333 0.0006333 0.19% Castro::enforce_speed_limit() 30 0.0006127 0.0006127 0.0006127 0.19% MLCellLinOp::prepareForSolve() 6 0.0005938 0.0005938 0.0005938 0.18% MultiFab::contains_nan() 10 0.0005899 0.0005899 0.0005899 0.18% MLCellLinOp::smooth() 720 0.0005046 0.0005046 0.0005046 0.15% MLCellLinOp::compGrad() 6 0.0004794 0.0004794 0.0004794 0.15% MLMG::prepareForSolve() 6 0.0004616 0.0004616 0.0004616 0.14% FabArrayBase::CPC::define() 244 0.0004107 0.0004107 0.0004107 0.13% Amr::InitAmr() 1 0.0003801 0.0003801 0.0003801 0.12% FabArrayBase::getCPC() 632 0.0003533 0.0003533 0.0003533 0.11% FabArray::FillBoundary() 1766 0.0003471 0.0003471 0.0003471 0.11% Gravity::get_old_grav_vector() 5 0.0002953 0.0002953 0.0002953 0.09% main() 1 0.0002801 0.0002801 0.0002801 0.09% Gravity::get_new_grav_vector() 5 0.0002749 0.0002749 0.0002749 0.08% FabArrayBase::getFB() 1766 0.0002507 0.0002507 0.0002507 0.08% MLCellLinOp::apply() 500 0.000206 0.000206 0.000206 0.06% MLLinOp::defineGrids() 6 0.0001887 0.0001887 0.0001887 0.06% MLMG::mgVcycle() 36 0.0001839 0.0001839 0.0001839 0.06% Amr::coarseTimeStep() 5 0.0001741 0.0001741 0.0001741 0.05% MultiFab::max() 6 0.0001354 0.0001354 0.0001354 0.04% MLCGSolver::ParallelAllReduce 659 0.0001268 0.0001268 0.0001268 0.04% FillPatchIterator::Initialize 20 0.0001195 0.0001195 0.0001195 0.04% MLCellLinOp::defineBC() 6 0.0001157 0.0001157 0.0001157 0.04% FabArray::ParallelCopy() 380 0.0001054 0.0001054 0.0001054 0.03% MLCellLinOp::correctionResidual() 216 0.0001019 0.0001019 0.0001019 0.03% Amr::timeStep() 5 7.737e-05 7.737e-05 7.737e-05 0.02% Castro::subcycle_advance_ctu() 5 7.708e-05 7.708e-05 7.708e-05 0.02% AmrLevel::restart() 1 7.585e-05 7.585e-05 7.585e-05 0.02% StateData::restartDoit() 4 5.979e-05 5.979e-05 5.979e-05 0.02% Gravity::update_max_rhs() 6 5.892e-05 5.892e-05 5.892e-05 0.02% FabArrayBase::FB::FB() 26 5.873e-05 5.873e-05 5.873e-05 0.02% MLMG:computeResOfCorrection() 180 5.722e-05 5.722e-05 5.722e-05 0.02% Castro::finalize_advance() 5 5.552e-05 5.552e-05 5.552e-05 0.02% Gravity::solve_for_phi() 5 5.221e-05 5.221e-05 5.221e-05 0.02% MLMG::actualBottomSolve() 36 4.27e-05 4.27e-05 4.27e-05 0.01% MLMG::mgVcycle_down::0 36 4.125e-05 4.125e-05 4.125e-05 0.01% Castro::expand_state() 5 4.072e-05 4.072e-05 4.072e-05 0.01% MLMG::mgVcycle_down::3 36 3.832e-05 3.832e-05 3.832e-05 0.01% MLMG::mgVcycle_down::1 36 3.718e-05 3.718e-05 3.718e-05 0.01% Castro::clean_state() 30 3.589e-05 3.589e-05 3.589e-05 0.01% MLMG::solve() 6 3.561e-05 3.561e-05 3.561e-05 0.01% MLMG::mgVcycle_down::2 36 3.499e-05 3.499e-05 3.499e-05 0.01% MLMG::mgVcycle_down::4 36 3.478e-05 3.478e-05 3.478e-05 0.01% Castro::initialize_advance() 5 3.468e-05 3.468e-05 3.468e-05 0.01% Castro::advance() 5 3.382e-05 3.382e-05 3.382e-05 0.01% MLMG::mgVcycle_up::4 36 2.886e-05 2.886e-05 2.886e-05 0.01% MLMG::mgVcycle_up::2 36 2.811e-05 2.811e-05 2.811e-05 0.01% Castro::construct_new_source() 25 2.656e-05 2.656e-05 2.656e-05 0.01% Amr::writeSmallPlotFile() 1 2.57e-05 2.57e-05 2.57e-05 0.01% MLMG::oneIter() 36 2.557e-05 2.557e-05 2.557e-05 0.01% Castro::buildMetrics() 1 2.553e-05 2.553e-05 2.553e-05 0.01% Castro::initialize_do_advance() 5 2.506e-05 2.506e-05 2.506e-05 0.01% Castro::swap_state_time_levels() 5 2.483e-05 2.483e-05 2.483e-05 0.01% MLMG::mgVcycle_up::0 36 2.441e-05 2.441e-05 2.441e-05 0.01% Castro::post_restart() 1 2.42e-05 2.42e-05 2.42e-05 0.01% Castro::initMFs() 1 2.37e-05 2.37e-05 2.37e-05 0.01% MLMG::mgVcycle_up::3 36 2.366e-05 2.366e-05 2.366e-05 0.01% MLCellLinOp::solutionResidual() 42 2.356e-05 2.356e-05 2.356e-05 0.01% Castro::finalize_do_advance() 5 2.293e-05 2.293e-05 2.293e-05 0.01% MLMG::mgVcycle_up::1 36 2.269e-05 2.269e-05 2.269e-05 0.01% MLMG::ResNormInf() 42 1.728e-05 1.728e-05 1.728e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.552e-05 1.552e-05 1.552e-05 0.00% MLPoisson::define() 6 1.528e-05 1.528e-05 1.528e-05 0.00% MLMG::computeResidual() 36 1.462e-05 1.462e-05 1.462e-05 0.00% MLMG::mgVcycle_bottom 36 1.446e-05 1.446e-05 1.446e-05 0.00% FillPatchSingleLevel 20 1.413e-05 1.413e-05 1.413e-05 0.00% makeSFC 30 1.341e-05 1.341e-05 1.341e-05 0.00% Castro::construct_new_gravity() 5 1.258e-05 1.258e-05 1.258e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.211e-05 1.211e-05 1.211e-05 0.00% MLPoisson::prepareForSolve() 6 9.852e-06 9.852e-06 9.852e-06 0.00% Castro::construct_old_source() 25 9.831e-06 9.831e-06 9.831e-06 0.00% Castro::do_new_sources() 5 9e-06 9e-06 9e-06 0.00% DistributionMapping::Distribute() 31 8.393e-06 8.393e-06 8.393e-06 0.00% Amr::initSubcycle() 1 8.325e-06 8.325e-06 8.325e-06 0.00% Castro::do_old_sources() 5 7.93e-06 7.93e-06 7.93e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.732e-06 7.732e-06 7.732e-06 0.00% Gravity::actual_multilevel_solve() 1 7.401e-06 7.401e-06 7.401e-06 0.00% Castro::check_for_nan() 10 5.512e-06 5.512e-06 5.512e-06 0.00% MLLinOp::define() 6 5.269e-06 5.269e-06 5.269e-06 0.00% Castro::construct_old_gravity() 5 5.172e-06 5.172e-06 5.172e-06 0.00% Castro::apply_source_to_state() 10 5.167e-06 5.167e-06 5.167e-06 0.00% Castro::post_timestep() 5 4.449e-06 4.449e-06 4.449e-06 0.00% Gravity::swapTimeLevels() 5 4.421e-06 4.421e-06 4.421e-06 0.00% MLMG::computeMLResidual() 6 3.468e-06 3.468e-06 3.468e-06 0.00% Castro::computeNewDt() 5 3.215e-06 3.215e-06 3.215e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.062e-06 3.062e-06 3.062e-06 0.00% MLMG::getGradSolution() 6 3.007e-06 3.007e-06 3.007e-06 0.00% Castro::create_source_corrector() 5 2.477e-06 2.477e-06 2.477e-06 0.00% MLMG::MLRhsNormInf() 6 2.17e-06 2.17e-06 2.17e-06 0.00% MLMG::MLResNormInf() 6 2.128e-06 2.128e-06 2.128e-06 0.00% Gravity::set_mass_offset() 6 2.033e-06 2.033e-06 2.033e-06 0.00% Castro::retry_advance_ctu() 5 1.562e-06 1.562e-06 1.562e-06 0.00% Castro::FluxRegCrseInit 5 1.478e-06 1.478e-06 1.478e-06 0.00% Castro::FluxRegFineAdd() 5 1.373e-06 1.373e-06 1.373e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.064e-06 1.064e-06 1.064e-06 0.00% Amr::init() 1 9e-07 9e-07 9e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3283 0.3283 0.3283 100.00% Amr::coarseTimeStep() 5 0.2486 0.2486 0.2486 75.73% Amr::timeStep() 5 0.2463 0.2463 0.2463 75.01% Castro::advance() 5 0.243 0.243 0.243 74.02% Castro::subcycle_advance_ctu() 5 0.2366 0.2366 0.2366 72.06% Castro::do_advance_ctu() 5 0.2365 0.2365 0.2365 72.04% Castro::construct_new_gravity() 5 0.1259 0.1259 0.1259 38.33% Gravity::solve_phi_with_mlmg() 6 0.121 0.121 0.121 36.86% Gravity::solve_for_phi() 5 0.1181 0.1181 0.1181 35.97% Gravity::actual_solve_with_mlmg() 6 0.1177 0.1177 0.1177 35.86% MLMG::solve() 6 0.107 0.107 0.107 32.59% MLMG::oneIter() 36 0.1003 0.1003 0.1003 30.56% MLMG::mgVcycle() 36 0.09877 0.09877 0.09877 30.09% Castro::construct_ctu_hydro_source() 5 0.0748 0.0748 0.0748 22.78% MLCellLinOp::smooth() 720 0.04843 0.04843 0.04843 14.75% Amr::init() 1 0.04661 0.04661 0.04661 14.20% Amr::restart() 1 0.04661 0.04661 0.04661 14.20% MLCellLinOp::applyBC() 1946 0.04529 0.04529 0.04529 13.80% AmrLevel::restart() 1 0.04076 0.04076 0.04076 12.42% StateData::restartDoit() 4 0.04068 0.04068 0.04068 12.39% VisMF::Read() 3 0.04057 0.04057 0.04057 12.36% Amr::writePlotFile() 1 0.03235 0.03235 0.03235 9.85% VisMF::Write(FabArray) 1 0.03076 0.03076 0.03076 9.37% MLMG::mgVcycle_bottom 36 0.03055 0.03055 0.03055 9.30% MLMG::actualBottomSolve() 36 0.03053 0.03053 0.03053 9.30% MLCGSolver::bicgstab 36 0.03024 0.03024 0.03024 9.21% FillPatchIterator::Initialize 20 0.02058 0.02058 0.02058 6.27% FillPatchSingleLevel 20 0.01983 0.01983 0.01983 6.04% Castro::clean_state() 30 0.01908 0.01908 0.01908 5.81% StateDataPhysBCFunct::() 20 0.01778 0.01778 0.01778 5.42% MLCellLinOp::apply() 500 0.01519 0.01519 0.01519 4.63% MLMG::mgVcycle_down::0 36 0.01398 0.01398 0.01398 4.26% MLPoisson::Fsmooth() 1440 0.01355 0.01355 0.01355 4.13% FabArray::FillBoundary() 1766 0.01323 0.01323 0.01323 4.03% FillBoundary_nowait() 1766 0.01288 0.01288 0.01288 3.92% StateData::FillBoundary(geom) 160 0.01189 0.01189 0.01189 3.62% MLMG::mgVcycle_up::0 36 0.01055 0.01055 0.01055 3.21% Castro::computeTemp() 30 0.009653 0.009653 0.009653 2.94% Castro::initialize_do_advance() 5 0.009558 0.009558 0.009558 2.91% MLCellLinOp::correctionResidual() 216 0.009222 0.009222 0.009222 2.81% MLPoisson::define() 6 0.008496 0.008496 0.008496 2.59% amrex::Dot() 484 0.008472 0.008472 0.008472 2.58% MLMG:computeResOfCorrection() 180 0.008117 0.008117 0.008117 2.47% Gravity::get_new_grav_vector() 5 0.007651 0.007651 0.007651 2.33% Castro::construct_old_gravity() 5 0.007289 0.007289 0.007289 2.22% Gravity::get_old_grav_vector() 5 0.007284 0.007284 0.007284 2.22% amrex::Copy() 463 0.006783 0.006783 0.006783 2.07% Castro::do_new_sources() 5 0.006587 0.006587 0.006587 2.01% MLMG::mgVcycle_down::1 36 0.006566 0.006566 0.006566 2.00% Castro::normalize_species() 30 0.006489 0.006489 0.006489 1.98% FabArray::ParallelCopy() 380 0.006337 0.006337 0.006337 1.93% Castro::expand_state() 5 0.00627 0.00627 0.00627 1.91% FabArray::ParallelCopy_nowait() 380 0.006232 0.006232 0.006232 1.90% MLMG::mgVcycle_down::2 36 0.006154 0.006154 0.006154 1.87% Castro::initialize_advance() 5 0.006087 0.006087 0.006087 1.85% FabArray::setVal() 537 0.006072 0.006072 0.006072 1.85% FabArray::norminf() 326 0.005988 0.005988 0.005988 1.82% MLMG::mgVcycle_down::3 36 0.005983 0.005983 0.005983 1.82% MLMG::mgVcycle_down::4 36 0.005933 0.005933 0.005933 1.81% MLCellLinOp::defineAuxData() 6 0.005679 0.005679 0.005679 1.73% MLCGSolver::ParallelAllReduce 659 0.0051 0.0051 0.0051 1.55% MLMG::addInterpCorrection() 180 0.00501 0.00501 0.00501 1.53% MLMG::mgVcycle_up::4 36 0.004832 0.004832 0.004832 1.47% MLMG::mgVcycle_up::1 36 0.004771 0.004771 0.004771 1.45% MLMG::mgVcycle_up::2 36 0.004685 0.004685 0.004685 1.43% amrex::average_down 180 0.00468 0.00468 0.00468 1.43% MLMG::mgVcycle_up::3 36 0.00459 0.00459 0.00459 1.40% MLPoisson::Fapply() 500 0.004324 0.004324 0.004324 1.32% Castro::do_old_sources() 5 0.004136 0.004136 0.004136 1.26% FabArray::Saxpy() 355 0.003531 0.003531 0.003531 1.08% FabArray::Xpay() 361 0.003389 0.003389 0.003389 1.03% Castro::reset_internal_energy(MultiFab) 30 0.003353 0.003353 0.003353 1.02% Castro::post_restart() 1 0.003265 0.003265 0.003265 0.99% Castro::post_timestep() 5 0.003177 0.003177 0.003177 0.97% Gravity::fill_multipole_BCs() 6 0.003157 0.003157 0.003157 0.96% MLCellLinOp::solutionResidual() 42 0.003154 0.003154 0.003154 0.96% Gravity::multilevel_solve_for_new_phi() 1 0.003153 0.003153 0.003153 0.96% Gravity::actual_multilevel_solve() 1 0.003138 0.003138 0.003138 0.96% Castro::estTimeStep() 10 0.002997 0.002997 0.002997 0.91% MLMG::computeResidual() 36 0.002616 0.002616 0.002616 0.80% MLCellLinOp::defineBC() 6 0.002578 0.002578 0.002578 0.79% BndryData::define() 6 0.002463 0.002463 0.002463 0.75% MLMG::prepareForSolve() 6 0.00244 0.00244 0.00244 0.74% Castro::enforce_min_density() 30 0.002291 0.002291 0.002291 0.70% Castro::computeNewDt() 5 0.002192 0.002192 0.002192 0.67% FabArray::LinComb() 242 0.001849 0.001849 0.001849 0.56% amrex::Add() 72 0.001837 0.001837 0.001837 0.56% Castro::construct_new_source() 25 0.001662 0.001662 0.001662 0.51% Castro::construct_new_gravity_source() 5 0.001635 0.001635 0.001635 0.50% Castro::reset_internal_energy(Fab) 240 0.00138 0.00138 0.00138 0.42% Castro::construct_old_source() 25 0.001306 0.001306 0.001306 0.40% Castro::construct_old_gravity_source() 5 0.001296 0.001296 0.001296 0.39% Castro::apply_source_to_state() 10 0.0009152 0.0009152 0.0009152 0.28% MLMG::ResNormInf() 42 0.0009086 0.0009086 0.0009086 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008724 0.0008724 0.0008724 0.27% FabArrayBase::getCPC() 632 0.0007639 0.0007639 0.0007639 0.23% MLMG::getGradSolution() 6 0.0007426 0.0007426 0.0007426 0.23% MLCellLinOp::compGrad() 6 0.0007396 0.0007396 0.0007396 0.23% MLCellLinOp::setLevelBC() 6 0.0007299 0.0007299 0.0007299 0.22% FabArray::mult() 22 0.0006409 0.0006409 0.0006409 0.20% FabArray::setDomainBndry() 20 0.0006333 0.0006333 0.0006333 0.19% Castro::enforce_speed_limit() 30 0.0006127 0.0006127 0.0006127 0.19% MLPoisson::prepareForSolve() 6 0.0006037 0.0006037 0.0006037 0.18% Castro::check_for_nan() 10 0.0005954 0.0005954 0.0005954 0.18% MLCellLinOp::prepareForSolve() 6 0.0005938 0.0005938 0.0005938 0.18% MultiFab::contains_nan() 10 0.0005899 0.0005899 0.0005899 0.18% MLMG::computeMLResidual() 6 0.0005556 0.0005556 0.0005556 0.17% Gravity::update_max_rhs() 6 0.0004342 0.0004342 0.0004342 0.13% FabArrayBase::CPC::define() 244 0.0004107 0.0004107 0.0004107 0.13% Amr::InitAmr() 1 0.0003884 0.0003884 0.0003884 0.12% Castro::finalize_advance() 5 0.0003126 0.0003126 0.0003126 0.10% FabArrayBase::getFB() 1766 0.0003094 0.0003094 0.0003094 0.09% Gravity::swapTimeLevels() 5 0.0002243 0.0002243 0.0002243 0.07% MLLinOp::define() 6 0.0002233 0.0002233 0.0002233 0.07% MLLinOp::defineGrids() 6 0.0002181 0.0002181 0.0002181 0.07% MLMG::MLResNormInf() 6 0.0001513 0.0001513 0.0001513 0.05% Castro::buildMetrics() 1 0.000143 0.000143 0.000143 0.04% MultiFab::max() 6 0.0001354 0.0001354 0.0001354 0.04% MLMG::MLRhsNormInf() 6 0.0001151 0.0001151 0.0001151 0.04% FabArrayBase::FB::FB() 26 5.873e-05 5.873e-05 5.873e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.829e-05 2.829e-05 2.829e-05 0.01% Amr::writeSmallPlotFile() 1 2.57e-05 2.57e-05 2.57e-05 0.01% Castro::swap_state_time_levels() 5 2.483e-05 2.483e-05 2.483e-05 0.01% Castro::initMFs() 1 2.37e-05 2.37e-05 2.37e-05 0.01% Castro::finalize_do_advance() 5 2.293e-05 2.293e-05 2.293e-05 0.01% makeSFC 30 2.055e-05 2.055e-05 2.055e-05 0.01% DistributionMapping::Distribute() 31 8.393e-06 8.393e-06 8.393e-06 0.00% Amr::initSubcycle() 1 8.325e-06 8.325e-06 8.325e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.314e-06 4.314e-06 4.314e-06 0.00% Castro::create_source_corrector() 5 2.477e-06 2.477e-06 2.477e-06 0.00% Gravity::set_mass_offset() 6 2.033e-06 2.033e-06 2.033e-06 0.00% Castro::retry_advance_ctu() 5 1.562e-06 1.562e-06 1.562e-06 0.00% Castro::FluxRegCrseInit 5 1.478e-06 1.478e-06 1.478e-06 0.00% Castro::FluxRegFineAdd() 5 1.373e-06 1.373e-06 1.373e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.064e-06 1.064e-06 1.064e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 264 MiB 9042 MiB Castro::construct_ctu_hydro_source() 1440 1440 103 MiB 692 MiB Castro::initMFs() 48 48 58 MiB 68 MiB StateData::restartDoit() 32 32 53 MiB 55 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB Castro::initialize_do_advance() 40 40 28 MiB 39 MiB FillPatchIterator::Initialize 160 160 1401 KiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 2756 KiB 28 MiB Castro::initialize_advance() 40 40 17 MiB 23 MiB Castro::buildMetrics() 32 32 13 MiB 15 MiB Castro::post_restart() 48 48 6559 KiB 14 MiB MLMG::prepareForSolve() 361 361 4001 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 229 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 238 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 6554 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 14 KiB 2053 KiB Gravity::update_max_rhs() 48 48 2558 B 2048 KiB Gravity::solve_for_phi() 40 40 734 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 19 KiB 2048 KiB BndryData::define() 576 576 368 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 236 KiB 671 KiB Castro::estTimeStep() 10 10 3183 B 480 KiB VisMF::Write(FabArray) 112 112 1476 B 320 KiB Castro::normalize_species() 30 30 6422 B 320 KiB amrex::average_down 469 469 1349 B 257 KiB MLMG::addInterpCorrection() 468 468 1304 B 257 KiB amrex::Dot() 592 592 3808 B 160 KiB FabArray::norminf() 398 398 2692 B 160 KiB Castro::do_advance_ctu() 5 5 504 B 160 KiB MultiFab::max() 6 6 63 B 160 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MultiFab::contains_nan() 10 10 35 B 20 KiB MLPoisson::Fsmooth() 60 60 3892 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 58 B 10 KiB FillBoundary_nowait() 336 336 333 B 9648 B MLCellLinOp::applyBC() 3892 3892 253 B 9344 B amrex::Copy() 56 56 5879 B 8464 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B StateData::FillBoundary(geom) 960 960 62 B 2688 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLCGSolver::bicgstab 324 324 133 B 1472 B MLCellLinOp::defineBC() 36 36 413 B 1248 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1404 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 134 KiB 8192 KiB VisMF::Write(FabArray) 120 120 288 KiB 3584 KiB VisMF::Read() 24 24 230 KiB 3000 KiB FabArray::setVal() 66 66 18 KiB 23 KiB MLPoisson::Fsmooth() 60 60 3892 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 59 B 10 KiB FillBoundary_nowait() 336 336 333 B 9648 B MLCellLinOp::applyBC() 1946 1946 252 B 9328 B amrex::Copy() 56 56 5879 B 8464 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B Gravity::get_old_grav_vector() 3 3 2576 B 3072 B Gravity::fill_multipole_BCs() 18 18 6 B 2832 B StateData::FillBoundary(geom) 960 960 63 B 2784 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::prepareForSolve() 7 7 566 B 1296 B amrex::average_down 37 37 216 B 1296 B MLMG::addInterpCorrection() 36 36 3 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 27 B 400 B FabArray::norminf() 398 398 19 B 272 B Castro::estTimeStep() 10 10 0 B 32 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::do_advance_ctu() 5 5 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.03-8-gaaab804f1438) finalized