Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-22-g843a7dff2662) initialized Starting run at 08:25:34 UTC on 2022-05-17. Successfully read inputs file ... Castro git describe: 22.05-21-gd98f2bb40 AMReX git describe: 22.05-22-g843a7dff2 Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.043456015 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025060428 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048895521 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.053520962 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.067332996 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.067396467 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.0520152 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.040353677 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.058358031 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.062633652 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.066480876 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060602343 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.055430866 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.040426959 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024730774 seconds Ending run at 08:25:35 UTC on 2022-05-17. Run time = 0.820158877 Run time without initialization = 0.698664322 Average number of zones advanced per microsecond: 3.752 Average number of zones advanced per microsecond per rank: 3.752 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8202 ... 0.8202 ... 0.8202 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1825 0.1825 0.1825 22.25% VisMF::Write(FabArray) 11 0.1672 0.1672 0.1672 20.39% MLCellLinOp::applyBC() 4433 0.08352 0.08352 0.08352 10.18% MLPoisson::Fsmooth() 3280 0.06501 0.06501 0.06501 7.93% MLCGSolver::bicgstab 82 0.02452 0.02452 0.02452 2.99% StateData::FillBoundary(geom) 328 0.02422 0.02422 0.02422 2.95% MultiFab::Dot() 1114 0.02283 0.02283 0.02283 2.78% Castro::normalize_species() 62 0.02014 0.02014 0.02014 2.46% StateDataPhysBCFunct::() 41 0.01698 0.01698 0.01698 2.07% Castro::computeTemp() 63 0.01532 0.01532 0.01532 1.87% MultiFab::LinComb() 1586 0.01469 0.01469 0.01469 1.79% FabArray::setVal() 1144 0.01454 0.01454 0.01454 1.77% FillBoundary_nowait() 4023 0.0143 0.0143 0.0143 1.74% FabArray::ParallelCopy_nowait() 861 0.01343 0.01343 0.01343 1.64% Castro::enforce_min_density() 62 0.01261 0.01261 0.01261 1.54% MLPoisson::Fapply() 1142 0.01202 0.01202 0.01202 1.47% MLCellLinOp::defineAuxData() 11 0.01185 0.01185 0.01185 1.45% Gravity::fill_multipole_BCs() 11 0.008543 0.008543 0.008543 1.04% MLMG::addInterpCorrection() 410 0.007651 0.007651 0.007651 0.93% Castro::estTimeStep() 21 0.007493 0.007493 0.007493 0.91% amrex::average_down 410 0.00704 0.00704 0.00704 0.86% MultiFab::Xpay() 585 0.006764 0.006764 0.006764 0.82% Castro::do_advance_ctu() 10 0.005537 0.005537 0.005537 0.68% Castro::reset_internal_energy(MultiFab) 63 0.004998 0.004998 0.004998 0.61% Amr::checkPoint() 3 0.004332 0.004332 0.004332 0.53% BndryData::define() 11 0.004001 0.004001 0.004001 0.49% Castro::construct_new_gravity_source() 10 0.003057 0.003057 0.003057 0.37% Castro::construct_old_gravity_source() 10 0.002617 0.002617 0.002617 0.32% Amr::writePlotFile() 2 0.002502 0.002502 0.002502 0.31% Castro::enforce_speed_limit() 62 0.002318 0.002318 0.002318 0.28% MLMG::ResNormInf() 93 0.001968 0.001968 0.001968 0.24% Gravity::get_new_grav_vector() 11 0.001935 0.001935 0.001935 0.24% MultiFab::Saxpy() 20 0.001809 0.001809 0.001809 0.22% Gravity::get_old_grav_vector() 10 0.00175 0.00175 0.00175 0.21% Castro::expand_state() 10 0.001733 0.001733 0.001733 0.21% MLMG::oneIter() 82 0.001713 0.001713 0.001713 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001688 0.001688 0.001688 0.21% Castro::reset_internal_energy(Fab) 504 0.001623 0.001623 0.001623 0.20% MLCellLinOp::setLevelBC() 11 0.001577 0.001577 0.001577 0.19% Gravity::actual_solve_with_mlmg() 11 0.001352 0.001352 0.001352 0.16% FabArray::mult() 43 0.001339 0.001339 0.001339 0.16% FabArray::setDomainBndry() 41 0.00133 0.00133 0.00133 0.16% Castro::initData() 1 0.001213 0.001213 0.001213 0.15% MLCellLinOp::prepareForSolve() 11 0.001195 0.001195 0.001195 0.15% MultiFab::contains_nan() 20 0.001182 0.001182 0.001182 0.14% MLCellLinOp::smooth() 1640 0.001151 0.001151 0.001151 0.14% MLMG::prepareForSolve() 11 0.001056 0.001056 0.001056 0.13% MLCellLinOp::compGrad() 11 0.0009156 0.0009156 0.0009156 0.11% FabArrayBase::getCPC() 1323 0.0007771 0.0007771 0.0007771 0.09% FabArray::FillBoundary() 4023 0.0007718 0.0007718 0.0007718 0.09% FabArrayBase::CPC::define() 454 0.0006763 0.0006763 0.0006763 0.08% FabArrayBase::getFB() 4023 0.000611 0.000611 0.000611 0.07% MLCellLinOp::apply() 1142 0.0004829 0.0004829 0.0004829 0.06% Amr::InitAmr() 1 0.0004634 0.0004634 0.0004634 0.06% Gravity::solve_for_phi() 10 0.0004394 0.0004394 0.0004394 0.05% Gravity::update_max_rhs() 11 0.0004089 0.0004089 0.0004089 0.05% CGSolver::sxay() 1586 0.0003497 0.0003497 0.0003497 0.04% FillPatchIterator::Initialize 41 0.0002969 0.0002969 0.0002969 0.04% MLCellLinOp::defineBC() 11 0.0002887 0.0002887 0.0002887 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002703 0.0002703 0.0002703 0.03% FabArray::ParallelCopy() 861 0.0002698 0.0002698 0.0002698 0.03% MultiFab::Copy() 11 0.000265 0.000265 0.000265 0.03% main() 1 0.0002602 0.0002602 0.0002602 0.03% MultiFab::max() 11 0.0002573 0.0002573 0.0002573 0.03% MLCellLinOp::correctionResidual() 492 0.0002341 0.0002341 0.0002341 0.03% MLMG::mgVcycle() 82 0.0002247 0.0002247 0.0002247 0.03% Castro::subcycle_advance_ctu() 10 0.0002137 0.0002137 0.0002137 0.03% Castro::construct_new_gravity() 10 0.0002099 0.0002099 0.0002099 0.03% Amr::coarseTimeStep() 10 0.0002038 0.0002038 0.0002038 0.02% MLMG::MLRhsNormInf() 11 0.0002028 0.0002028 0.0002028 0.02% Amr::timeStep() 10 0.0001864 0.0001864 0.0001864 0.02% MLLinOp::defineGrids() 11 0.0001857 0.0001857 0.0001857 0.02% StateData::checkPoint() 12 0.0001316 0.0001316 0.0001316 0.02% MLMG:computeResOfCorrection() 410 0.0001168 0.0001168 0.0001168 0.01% Castro::advance() 10 0.0001032 0.0001032 0.0001032 0.01% MLMG::actualBottomSolve() 82 9.793e-05 9.793e-05 9.793e-05 0.01% Castro::Castro() 1 9.394e-05 9.394e-05 9.394e-05 0.01% MLMG::mgVcycle_down::0 82 9.122e-05 9.122e-05 9.122e-05 0.01% Castro::initialize_advance() 10 8.543e-05 8.543e-05 8.543e-05 0.01% FabArrayBase::FB::FB() 56 8.347e-05 8.347e-05 8.347e-05 0.01% MLMG::mgVcycle_down::1 82 7.768e-05 7.768e-05 7.768e-05 0.01% MLMG::mgVcycle_down::2 82 7.625e-05 7.625e-05 7.625e-05 0.01% MLMG::solve() 11 7.484e-05 7.484e-05 7.484e-05 0.01% Castro::clean_state() 62 7.376e-05 7.376e-05 7.376e-05 0.01% MLMG::mgVcycle_down::3 82 7.22e-05 7.22e-05 7.22e-05 0.01% AmrLevel::checkPoint() 3 7.151e-05 7.151e-05 7.151e-05 0.01% MLMG::mgVcycle_down::4 82 7.13e-05 7.13e-05 7.13e-05 0.01% Castro::initialize_do_advance() 10 6.421e-05 6.421e-05 6.421e-05 0.01% MLMG::mgVcycle_up::4 82 5.977e-05 5.977e-05 5.977e-05 0.01% Castro::finalize_advance() 10 5.745e-05 5.745e-05 5.745e-05 0.01% MLMG::mgVcycle_up::0 82 5.345e-05 5.345e-05 5.345e-05 0.01% MLCellLinOp::solutionResidual() 93 5.096e-05 5.096e-05 5.096e-05 0.01% MLMG::mgVcycle_up::3 82 5.081e-05 5.081e-05 5.081e-05 0.01% MLMG::mgVcycle_up::1 82 5.043e-05 5.043e-05 5.043e-05 0.01% MLMG::mgVcycle_up::2 82 4.947e-05 4.947e-05 4.947e-05 0.01% StateData::define() 4 4.131e-05 4.131e-05 4.131e-05 0.01% Castro::swap_state_time_levels() 10 3.93e-05 3.93e-05 3.93e-05 0.00% Castro::finalize_do_advance() 10 3.707e-05 3.707e-05 3.707e-05 0.00% Castro::enforce_consistent_e() 1 3.343e-05 3.343e-05 3.343e-05 0.00% Gravity::actual_multilevel_solve() 1 3.238e-05 3.238e-05 3.238e-05 0.00% MLMG::computeResidual() 82 3.201e-05 3.201e-05 3.201e-05 0.00% MLMG::mgVcycle_bottom 82 3.176e-05 3.176e-05 3.176e-05 0.00% Castro::initMFs() 1 3.011e-05 3.011e-05 3.011e-05 0.00% FillPatchSingleLevel 41 2.952e-05 2.952e-05 2.952e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.621e-05 2.621e-05 2.621e-05 0.00% makeSFC 55 2.62e-05 2.62e-05 2.62e-05 0.00% Amr::defBaseLevel() 1 2.526e-05 2.526e-05 2.526e-05 0.00% Amr::writeSmallPlotFile() 1 2.48e-05 2.48e-05 2.48e-05 0.00% Castro::buildMetrics() 1 2.441e-05 2.441e-05 2.441e-05 0.00% MLLinOp::define() 11 2.408e-05 2.408e-05 2.408e-05 0.00% MLPoisson::define() 11 2.227e-05 2.227e-05 2.227e-05 0.00% Amr::FinalizeInit() 1 2.072e-05 2.072e-05 2.072e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.035e-05 2.035e-05 2.035e-05 0.00% Castro::construct_old_source() 50 1.897e-05 1.897e-05 1.897e-05 0.00% Castro::construct_new_source() 50 1.783e-05 1.783e-05 1.783e-05 0.00% DistributionMapping::Distribute() 56 1.674e-05 1.674e-05 1.674e-05 0.00% Castro::do_new_sources() 10 1.603e-05 1.603e-05 1.603e-05 0.00% Castro::do_old_sources() 10 1.599e-05 1.599e-05 1.599e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.33e-05 1.33e-05 1.33e-05 0.00% Castro::check_for_nan() 20 1.23e-05 1.23e-05 1.23e-05 0.00% Castro::apply_source_to_state() 20 1.117e-05 1.117e-05 1.117e-05 0.00% Castro::construct_old_gravity() 10 9.84e-06 9.84e-06 9.84e-06 0.00% MLPoisson::prepareForSolve() 11 9.071e-06 9.071e-06 9.071e-06 0.00% Gravity::swapTimeLevels() 10 8.872e-06 8.872e-06 8.872e-06 0.00% Amr::initSubcycle() 1 8.52e-06 8.52e-06 8.52e-06 0.00% Castro::post_timestep() 10 8.085e-06 8.085e-06 8.085e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.375e-06 7.375e-06 7.375e-06 0.00% Amr::InitializeInit() 1 6.915e-06 6.915e-06 6.915e-06 0.00% MLMG::computeMLResidual() 11 6.532e-06 6.532e-06 6.532e-06 0.00% Castro::computeNewDt() 9 6.186e-06 6.186e-06 6.186e-06 0.00% MLMG::getGradSolution() 11 5.987e-06 5.987e-06 5.987e-06 0.00% MLMG::buildFineMask() 11 5.521e-06 5.521e-06 5.521e-06 0.00% AmrLevel::checkPointPost() 3 5.085e-06 5.085e-06 5.085e-06 0.00% Castro::post_init() 1 4.414e-06 4.414e-06 4.414e-06 0.00% Castro::create_source_corrector() 10 4.35e-06 4.35e-06 4.35e-06 0.00% MLMG::MLResNormInf() 11 4.238e-06 4.238e-06 4.238e-06 0.00% Castro::retry_advance_ctu() 10 3.823e-06 3.823e-06 3.823e-06 0.00% Gravity::set_mass_offset() 11 3.756e-06 3.756e-06 3.756e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.727e-06 3.727e-06 3.727e-06 0.00% Castro::computeInitialDt() 2 2.952e-06 2.952e-06 2.952e-06 0.00% Castro::FluxRegCrseInit 10 2.878e-06 2.878e-06 2.878e-06 0.00% Amr::init() 1 2.863e-06 2.863e-06 2.863e-06 0.00% Castro::FluxRegFineAdd() 10 2.215e-06 2.215e-06 2.215e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.949e-06 1.949e-06 1.949e-06 0.00% AmrLevel::checkPointPre() 3 1.789e-06 1.789e-06 1.789e-06 0.00% Amr::initialInit() 1 1.215e-06 1.215e-06 1.215e-06 0.00% Castro::post_regrid() 1 1.194e-06 1.194e-06 1.194e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8202 0.8202 0.8202 100.00% Amr::coarseTimeStep() 10 0.6737 0.6737 0.6737 82.14% Amr::timeStep() 10 0.5887 0.5887 0.5887 71.78% Castro::advance() 10 0.578 0.578 0.578 70.47% Castro::subcycle_advance_ctu() 10 0.5647 0.5647 0.5647 68.85% Castro::do_advance_ctu() 10 0.5645 0.5645 0.5645 68.83% Gravity::solve_phi_with_mlmg() 11 0.3217 0.3217 0.3217 39.22% Gravity::actual_solve_with_mlmg() 11 0.3129 0.3129 0.3129 38.15% Castro::construct_new_gravity() 10 0.2924 0.2924 0.2924 35.65% MLMG::solve() 11 0.2898 0.2898 0.2898 35.34% Gravity::solve_for_phi() 10 0.2769 0.2769 0.2769 33.76% MLMG::oneIter() 82 0.275 0.275 0.275 33.53% MLMG::mgVcycle() 82 0.2733 0.2733 0.2733 33.32% Castro::construct_ctu_hydro_source() 10 0.1825 0.1825 0.1825 22.25% VisMF::Write(FabArray) 11 0.1672 0.1672 0.1672 20.39% MLCellLinOp::smooth() 1640 0.1401 0.1401 0.1401 17.08% Amr::checkPoint() 3 0.1244 0.1244 0.1244 15.16% Amr::init() 1 0.1209 0.1209 0.1209 14.74% AmrLevel::checkPoint() 3 0.12 0.12 0.12 14.64% StateData::checkPoint() 12 0.12 0.12 0.12 14.63% MLCellLinOp::applyBC() 4433 0.09929 0.09929 0.09929 12.11% MLMG::mgVcycle_bottom 82 0.08397 0.08397 0.08397 10.24% MLMG::actualBottomSolve() 82 0.08394 0.08394 0.08394 10.23% MLCGSolver::bicgstab 82 0.08309 0.08309 0.08309 10.13% MLPoisson::Fsmooth() 3280 0.06501 0.06501 0.06501 7.93% Castro::clean_state() 62 0.05624 0.05624 0.05624 6.86% Amr::initialInit() 1 0.05227 0.05227 0.05227 6.37% Amr::writePlotFile() 2 0.04991 0.04991 0.04991 6.09% Amr::FinalizeInit() 1 0.04825 0.04825 0.04825 5.88% Castro::post_init() 1 0.04696 0.04696 0.04696 5.73% FillPatchIterator::Initialize 41 0.04683 0.04683 0.04683 5.71% Gravity::multilevel_solve_for_new_phi() 1 0.04525 0.04525 0.04525 5.52% Gravity::actual_multilevel_solve() 1 0.04523 0.04523 0.04523 5.51% FillPatchSingleLevel 41 0.0452 0.0452 0.0452 5.51% StateDataPhysBCFunct::() 41 0.04119 0.04119 0.04119 5.02% MLCellLinOp::apply() 1142 0.03735 0.03735 0.03735 4.55% MLMG::mgVcycle_down::0 82 0.03624 0.03624 0.03624 4.42% MLMG::mgVcycle_up::0 82 0.03111 0.03111 0.03111 3.79% StateData::FillBoundary(geom) 328 0.02422 0.02422 0.02422 2.95% MultiFab::Dot() 1114 0.02283 0.02283 0.02283 2.78% Castro::computeTemp() 63 0.02194 0.02194 0.02194 2.67% MLCellLinOp::correctionResidual() 492 0.02191 0.02191 0.02191 2.67% Castro::initialize_do_advance() 10 0.02105 0.02105 0.02105 2.57% Castro::normalize_species() 62 0.02014 0.02014 0.02014 2.46% MLMG:computeResOfCorrection() 410 0.01889 0.01889 0.01889 2.30% MLPoisson::define() 11 0.01874 0.01874 0.01874 2.28% MLMG::mgVcycle_down::1 82 0.01827 0.01827 0.01827 2.23% MLMG::mgVcycle_down::2 82 0.01778 0.01778 0.01778 2.17% MLMG::mgVcycle_down::3 82 0.01691 0.01691 0.01691 2.06% Gravity::get_new_grav_vector() 11 0.01687 0.01687 0.01687 2.06% MLMG::mgVcycle_down::4 82 0.01607 0.01607 0.01607 1.96% FabArray::FillBoundary() 4023 0.01577 0.01577 0.01577 1.92% CGSolver::sxay() 1586 0.01504 0.01504 0.01504 1.83% Castro::construct_old_gravity() 10 0.015 0.015 0.015 1.83% FillBoundary_nowait() 4023 0.015 0.015 0.015 1.83% Gravity::get_old_grav_vector() 10 0.01499 0.01499 0.01499 1.83% MultiFab::LinComb() 1586 0.01469 0.01469 0.01469 1.79% FabArray::setVal() 1144 0.01454 0.01454 0.01454 1.77% FabArray::ParallelCopy() 861 0.01453 0.01453 0.01453 1.77% FabArray::ParallelCopy_nowait() 861 0.01426 0.01426 0.01426 1.74% MLMG::mgVcycle_up::2 82 0.01364 0.01364 0.01364 1.66% MLCGSolver::ParallelAllReduce 1514 0.01354 0.01354 0.01354 1.65% MLMG::mgVcycle_up::1 82 0.01342 0.01342 0.01342 1.64% MLCellLinOp::defineAuxData() 11 0.01321 0.01321 0.01321 1.61% Castro::initialize_advance() 10 0.0131 0.0131 0.0131 1.60% MLMG::addInterpCorrection() 410 0.01291 0.01291 0.01291 1.57% MLMG::mgVcycle_up::3 82 0.01289 0.01289 0.01289 1.57% MLMG::mgVcycle_up::4 82 0.01279 0.01279 0.01279 1.56% Castro::enforce_min_density() 62 0.01261 0.01261 0.01261 1.54% Castro::do_old_sources() 10 0.01249 0.01249 0.01249 1.52% Castro::do_new_sources() 10 0.01249 0.01249 0.01249 1.52% amrex::average_down 410 0.01233 0.01233 0.01233 1.50% MLPoisson::Fapply() 1142 0.01202 0.01202 0.01202 1.47% Castro::expand_state() 10 0.01133 0.01133 0.01133 1.38% Castro::post_timestep() 10 0.01053 0.01053 0.01053 1.28% Gravity::fill_multipole_BCs() 11 0.008543 0.008543 0.008543 1.04% Castro::estTimeStep() 21 0.007493 0.007493 0.007493 0.91% MLCellLinOp::solutionResidual() 93 0.00729 0.00729 0.00729 0.89% MultiFab::Xpay() 585 0.006764 0.006764 0.006764 0.82% Castro::reset_internal_energy(MultiFab) 63 0.006621 0.006621 0.006621 0.81% MLMG::computeResidual() 82 0.006278 0.006278 0.006278 0.77% MLCellLinOp::defineBC() 11 0.005242 0.005242 0.005242 0.64% MLMG::prepareForSolve() 11 0.005228 0.005228 0.005228 0.64% BndryData::define() 11 0.004953 0.004953 0.004953 0.60% Amr::InitializeInit() 1 0.004016 0.004016 0.004016 0.49% Amr::defBaseLevel() 1 0.004009 0.004009 0.004009 0.49% Castro::computeNewDt() 9 0.003479 0.003479 0.003479 0.42% Castro::initData() 1 0.003456 0.003456 0.003456 0.42% Castro::construct_new_source() 50 0.003075 0.003075 0.003075 0.37% Castro::construct_new_gravity_source() 10 0.003057 0.003057 0.003057 0.37% Castro::construct_old_source() 50 0.002636 0.002636 0.002636 0.32% Castro::construct_old_gravity_source() 10 0.002617 0.002617 0.002617 0.32% Castro::enforce_speed_limit() 62 0.002318 0.002318 0.002318 0.28% MLMG::ResNormInf() 93 0.001968 0.001968 0.001968 0.24% Castro::apply_source_to_state() 20 0.00182 0.00182 0.00182 0.22% MultiFab::Saxpy() 20 0.001809 0.001809 0.001809 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001688 0.001688 0.001688 0.21% Castro::reset_internal_energy(Fab) 504 0.001623 0.001623 0.001623 0.20% MLCellLinOp::setLevelBC() 11 0.001577 0.001577 0.001577 0.19% FabArrayBase::getCPC() 1323 0.001453 0.001453 0.001453 0.18% MLMG::getGradSolution() 11 0.00141 0.00141 0.00141 0.17% MLCellLinOp::compGrad() 11 0.001404 0.001404 0.001404 0.17% FabArray::mult() 43 0.001339 0.001339 0.001339 0.16% FabArray::setDomainBndry() 41 0.00133 0.00133 0.00133 0.16% MLPoisson::prepareForSolve() 11 0.001204 0.001204 0.001204 0.15% MLCellLinOp::prepareForSolve() 11 0.001195 0.001195 0.001195 0.15% Castro::check_for_nan() 20 0.001194 0.001194 0.001194 0.15% MultiFab::contains_nan() 20 0.001182 0.001182 0.001182 0.14% Castro::post_regrid() 1 0.001063 0.001063 0.001063 0.13% MLMG::computeMLResidual() 11 0.001051 0.001051 0.001051 0.13% Gravity::update_max_rhs() 11 0.0008158 0.0008158 0.0008158 0.10% FabArrayBase::getFB() 4023 0.0006945 0.0006945 0.0006945 0.08% FabArrayBase::CPC::define() 454 0.0006763 0.0006763 0.0006763 0.08% Castro::computeInitialDt() 2 0.0006756 0.0006756 0.0006756 0.08% Castro::Castro() 1 0.0004742 0.0004742 0.0004742 0.06% Amr::InitAmr() 1 0.0004719 0.0004719 0.0004719 0.06% Gravity::swapTimeLevels() 10 0.0004483 0.0004483 0.0004483 0.05% MLLinOp::define() 11 0.0002666 0.0002666 0.0002666 0.03% MultiFab::Copy() 11 0.000265 0.000265 0.000265 0.03% MLMG::MLResNormInf() 11 0.0002609 0.0002609 0.0002609 0.03% MultiFab::max() 11 0.0002573 0.0002573 0.0002573 0.03% MLLinOp::defineGrids() 11 0.0002425 0.0002425 0.0002425 0.03% MLMG::MLRhsNormInf() 11 0.0002028 0.0002028 0.0002028 0.02% Castro::buildMetrics() 1 0.0001729 0.0001729 0.0001729 0.02% FabArrayBase::FB::FB() 56 8.347e-05 8.347e-05 8.347e-05 0.01% Castro::finalize_advance() 10 6.254e-05 6.254e-05 6.254e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.483e-05 5.483e-05 5.483e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.869e-05 4.869e-05 4.869e-05 0.01% makeSFC 55 4.153e-05 4.153e-05 4.153e-05 0.01% StateData::define() 4 4.131e-05 4.131e-05 4.131e-05 0.01% Castro::swap_state_time_levels() 10 3.93e-05 3.93e-05 3.93e-05 0.00% Castro::finalize_do_advance() 10 3.707e-05 3.707e-05 3.707e-05 0.00% Castro::enforce_consistent_e() 1 3.343e-05 3.343e-05 3.343e-05 0.00% Castro::initMFs() 1 3.011e-05 3.011e-05 3.011e-05 0.00% Amr::writeSmallPlotFile() 1 2.48e-05 2.48e-05 2.48e-05 0.00% DistributionMapping::Distribute() 56 1.674e-05 1.674e-05 1.674e-05 0.00% Amr::initSubcycle() 1 8.52e-06 8.52e-06 8.52e-06 0.00% MLMG::buildFineMask() 11 5.521e-06 5.521e-06 5.521e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.133e-06 5.133e-06 5.133e-06 0.00% AmrLevel::checkPointPost() 3 5.085e-06 5.085e-06 5.085e-06 0.00% Castro::create_source_corrector() 10 4.35e-06 4.35e-06 4.35e-06 0.00% Castro::retry_advance_ctu() 10 3.823e-06 3.823e-06 3.823e-06 0.00% Gravity::set_mass_offset() 11 3.756e-06 3.756e-06 3.756e-06 0.00% Castro::FluxRegCrseInit 10 2.878e-06 2.878e-06 2.878e-06 0.00% Castro::FluxRegFineAdd() 10 2.215e-06 2.215e-06 2.215e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.949e-06 1.949e-06 1.949e-06 0.00% AmrLevel::checkPointPre() 3 1.789e-06 1.789e-06 1.789e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-22-g843a7dff2662) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-22-g843a7dff2662) initialized Starting run at 08:25:36 UTC on 2022-05-17. Successfully read inputs file ... Castro git describe: 22.05-21-gd98f2bb40 AMReX git describe: 22.05-22-g843a7dff2 Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.45096549 Restart time = 0.045677183 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.049803549 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050397112 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.054854963 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.06455336 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.067252079 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026422269 seconds Ending run at 08:25:36 UTC on 2022-05-17. Run time = 0.359887387 Run time without initialization = 0.313619503 Average number of zones advanced per microsecond: 4.179 Average number of zones advanced per microsecond per rank: 4.179 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3599 ... 0.3599 ... 0.3599 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0853 0.0853 0.0853 23.69% VisMF::Read() 3 0.03826 0.03826 0.03826 10.63% MLCellLinOp::applyBC() 1946 0.03506 0.03506 0.03506 9.74% MLPoisson::Fsmooth() 1440 0.02725 0.02725 0.02725 7.57% VisMF::Write(FabArray) 1 0.02496 0.02496 0.02496 6.94% StateData::FillBoundary(geom) 160 0.01122 0.01122 0.01122 3.12% MLCGSolver::bicgstab 36 0.01031 0.01031 0.01031 2.86% MultiFab::Dot() 484 0.009587 0.009587 0.009587 2.66% Castro::normalize_species() 30 0.009047 0.009047 0.009047 2.51% Castro::computeTemp() 30 0.008795 0.008795 0.008795 2.44% FabArray::setVal() 537 0.006764 0.006764 0.006764 1.88% MLCellLinOp::defineAuxData() 6 0.006223 0.006223 0.006223 1.73% FillBoundary_nowait() 1766 0.006206 0.006206 0.006206 1.72% MultiFab::LinComb() 690 0.00611 0.00611 0.00611 1.70% FabArray::ParallelCopy_nowait() 380 0.005954 0.005954 0.005954 1.65% Castro::enforce_min_density() 30 0.005845 0.005845 0.005845 1.62% StateDataPhysBCFunct::() 20 0.005634 0.005634 0.005634 1.57% MLPoisson::Fapply() 500 0.005083 0.005083 0.005083 1.41% Gravity::fill_multipole_BCs() 6 0.004545 0.004545 0.004545 1.26% Castro::estTimeStep() 10 0.004171 0.004171 0.004171 1.16% MLMG::addInterpCorrection() 180 0.003213 0.003213 0.003213 0.89% Amr::restart() 1 0.00309 0.00309 0.00309 0.86% amrex::average_down 180 0.002959 0.002959 0.002959 0.82% MultiFab::Xpay() 258 0.002879 0.002879 0.002879 0.80% Castro::do_advance_ctu() 5 0.002686 0.002686 0.002686 0.75% BndryData::define() 6 0.002128 0.002128 0.002128 0.59% Castro::construct_new_gravity_source() 5 0.001668 0.001668 0.001668 0.46% Amr::writePlotFile() 1 0.001545 0.001545 0.001545 0.43% Castro::reset_internal_energy(MultiFab) 30 0.001509 0.001509 0.001509 0.42% Castro::construct_old_gravity_source() 5 0.001374 0.001374 0.001374 0.38% Castro::enforce_speed_limit() 30 0.001098 0.001098 0.001098 0.31% Castro::reset_internal_energy(Fab) 240 0.0009775 0.0009775 0.0009775 0.27% Gravity::get_old_grav_vector() 5 0.0009695 0.0009695 0.0009695 0.27% Gravity::get_new_grav_vector() 5 0.0009583 0.0009583 0.0009583 0.27% MultiFab::Saxpy() 10 0.0009213 0.0009213 0.0009213 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008891 0.0008891 0.0008891 0.25% Castro::expand_state() 5 0.0008757 0.0008757 0.0008757 0.24% MLMG::ResNormInf() 42 0.0008534 0.0008534 0.0008534 0.24% MLCellLinOp::setLevelBC() 6 0.0008237 0.0008237 0.0008237 0.23% Gravity::actual_solve_with_mlmg() 6 0.0007617 0.0007617 0.0007617 0.21% MLMG::oneIter() 36 0.000746 0.000746 0.000746 0.21% FabArray::mult() 22 0.0006484 0.0006484 0.0006484 0.18% MLCellLinOp::prepareForSolve() 6 0.0006413 0.0006413 0.0006413 0.18% FabArray::setDomainBndry() 20 0.0006353 0.0006353 0.0006353 0.18% MultiFab::contains_nan() 10 0.0006088 0.0006088 0.0006088 0.17% MLMG::prepareForSolve() 6 0.0005629 0.0005629 0.0005629 0.16% MLCellLinOp::smooth() 720 0.0005141 0.0005141 0.0005141 0.14% MLCellLinOp::compGrad() 6 0.0004913 0.0004913 0.0004913 0.14% FabArrayBase::CPC::define() 244 0.0004044 0.0004044 0.0004044 0.11% Amr::InitAmr() 1 0.0003997 0.0003997 0.0003997 0.11% FabArray::FillBoundary() 1766 0.0003699 0.0003699 0.0003699 0.10% FabArrayBase::getCPC() 632 0.000368 0.000368 0.000368 0.10% main() 1 0.000264 0.000264 0.000264 0.07% FabArrayBase::getFB() 1766 0.0002528 0.0002528 0.0002528 0.07% Gravity::update_max_rhs() 6 0.0002285 0.0002285 0.0002285 0.06% Castro::subcycle_advance_ctu() 5 0.000225 0.000225 0.000225 0.06% MLCellLinOp::apply() 500 0.0002191 0.0002191 0.0002191 0.06% Gravity::solve_for_phi() 5 0.0002082 0.0002082 0.0002082 0.06% CGSolver::sxay() 690 0.0001611 0.0001611 0.0001611 0.04% MLCellLinOp::defineBC() 6 0.0001524 0.0001524 0.0001524 0.04% Castro::construct_new_source() 25 0.0001469 0.0001469 0.0001469 0.04% MultiFab::Copy() 6 0.000142 0.000142 0.000142 0.04% MultiFab::max() 6 0.000137 0.000137 0.000137 0.04% Castro::construct_new_gravity() 5 0.0001355 0.0001355 0.0001355 0.04% FillPatchIterator::Initialize 20 0.0001351 0.0001351 0.0001351 0.04% FabArray::ParallelCopy() 380 0.0001303 0.0001303 0.0001303 0.04% Castro::advance() 5 0.0001225 0.0001225 0.0001225 0.03% MLCGSolver::ParallelAllReduce 659 0.000115 0.000115 0.000115 0.03% MLMG::MLRhsNormInf() 6 0.0001064 0.0001064 0.0001064 0.03% Amr::coarseTimeStep() 5 0.0001049 0.0001049 0.0001049 0.03% MLLinOp::defineGrids() 6 0.0001033 0.0001033 0.0001033 0.03% Amr::timeStep() 5 9.761e-05 9.761e-05 9.761e-05 0.03% MLCellLinOp::correctionResidual() 216 9.754e-05 9.754e-05 9.754e-05 0.03% MLMG::mgVcycle() 36 9.3e-05 9.3e-05 9.3e-05 0.03% Gravity::solve_phi_with_mlmg() 6 8.941e-05 8.941e-05 8.941e-05 0.02% Castro::post_timestep() 5 8.262e-05 8.262e-05 8.262e-05 0.02% AmrLevel::restart() 1 7.58e-05 7.58e-05 7.58e-05 0.02% Castro::computeNewDt() 5 7.479e-05 7.479e-05 7.479e-05 0.02% Castro::initialize_advance() 5 7.224e-05 7.224e-05 7.224e-05 0.02% Castro::create_source_corrector() 5 6.705e-05 6.705e-05 6.705e-05 0.02% StateData::restartDoit() 4 6.351e-05 6.351e-05 6.351e-05 0.02% FabArrayBase::FB::FB() 26 5.682e-05 5.682e-05 5.682e-05 0.02% MLMG:computeResOfCorrection() 180 5.674e-05 5.674e-05 5.674e-05 0.02% Castro::finalize_advance() 5 4.962e-05 4.962e-05 4.962e-05 0.01% Castro::initialize_do_advance() 5 4.643e-05 4.643e-05 4.643e-05 0.01% Castro::construct_old_gravity() 5 4.409e-05 4.409e-05 4.409e-05 0.01% MLMG::actualBottomSolve() 36 4.361e-05 4.361e-05 4.361e-05 0.01% MLMG::mgVcycle_down::0 36 3.968e-05 3.968e-05 3.968e-05 0.01% MLMG::mgVcycle_down::1 36 3.957e-05 3.957e-05 3.957e-05 0.01% MLMG::mgVcycle_down::2 36 3.762e-05 3.762e-05 3.762e-05 0.01% Castro::clean_state() 30 3.73e-05 3.73e-05 3.73e-05 0.01% MLMG::mgVcycle_down::4 36 3.529e-05 3.529e-05 3.529e-05 0.01% MLMG::solve() 6 3.431e-05 3.431e-05 3.431e-05 0.01% MLMG::mgVcycle_down::3 36 3.391e-05 3.391e-05 3.391e-05 0.01% Castro::buildMetrics() 1 3.304e-05 3.304e-05 3.304e-05 0.01% Castro::post_restart() 1 3.077e-05 3.077e-05 3.077e-05 0.01% Gravity::actual_multilevel_solve() 1 3.023e-05 3.023e-05 3.023e-05 0.01% Castro::construct_old_source() 25 2.95e-05 2.95e-05 2.95e-05 0.01% MLMG::mgVcycle_up::4 36 2.94e-05 2.94e-05 2.94e-05 0.01% Castro::initMFs() 1 2.801e-05 2.801e-05 2.801e-05 0.01% Castro::swap_state_time_levels() 5 2.656e-05 2.656e-05 2.656e-05 0.01% Amr::writeSmallPlotFile() 1 2.58e-05 2.58e-05 2.58e-05 0.01% MLCellLinOp::solutionResidual() 42 2.5e-05 2.5e-05 2.5e-05 0.01% MLMG::mgVcycle_up::3 36 2.494e-05 2.494e-05 2.494e-05 0.01% MLMG::mgVcycle_up::0 36 2.473e-05 2.473e-05 2.473e-05 0.01% MLMG::mgVcycle_up::2 36 2.46e-05 2.46e-05 2.46e-05 0.01% MLMG::mgVcycle_up::1 36 2.322e-05 2.322e-05 2.322e-05 0.01% MLLinOp::define() 6 2.063e-05 2.063e-05 2.063e-05 0.01% Castro::finalize_do_advance() 5 1.91e-05 1.91e-05 1.91e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.712e-05 1.712e-05 1.712e-05 0.00% MLMG::mgVcycle_bottom 36 1.586e-05 1.586e-05 1.586e-05 0.00% FillPatchSingleLevel 20 1.472e-05 1.472e-05 1.472e-05 0.00% MLPoisson::define() 6 1.459e-05 1.459e-05 1.459e-05 0.00% MLMG::computeResidual() 36 1.426e-05 1.426e-05 1.426e-05 0.00% makeSFC 30 1.402e-05 1.402e-05 1.402e-05 0.00% Castro::do_new_sources() 5 1.034e-05 1.034e-05 1.034e-05 0.00% DistributionMapping::Distribute() 31 9.322e-06 9.322e-06 9.322e-06 0.00% Castro::do_old_sources() 5 8.644e-06 8.644e-06 8.644e-06 0.00% Amr::initSubcycle() 1 8.289e-06 8.289e-06 8.289e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.967e-06 6.967e-06 6.967e-06 0.00% Castro::apply_source_to_state() 10 6.001e-06 6.001e-06 6.001e-06 0.00% Castro::check_for_nan() 10 5.754e-06 5.754e-06 5.754e-06 0.00% Gravity::swapTimeLevels() 5 4.564e-06 4.564e-06 4.564e-06 0.00% MLPoisson::prepareForSolve() 6 4.212e-06 4.212e-06 4.212e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.936e-06 3.936e-06 3.936e-06 0.00% MLMG::computeMLResidual() 6 3.335e-06 3.335e-06 3.335e-06 0.00% MLMG::buildFineMask() 6 3.053e-06 3.053e-06 3.053e-06 0.00% MLMG::getGradSolution() 6 2.976e-06 2.976e-06 2.976e-06 0.00% MLMG::MLResNormInf() 6 2.249e-06 2.249e-06 2.249e-06 0.00% Castro::retry_advance_ctu() 5 2.078e-06 2.078e-06 2.078e-06 0.00% Gravity::set_mass_offset() 6 2.011e-06 2.011e-06 2.011e-06 0.00% Castro::FluxRegCrseInit 5 1.981e-06 1.981e-06 1.981e-06 0.00% Castro::FluxRegFineAdd() 5 1.352e-06 1.352e-06 1.352e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.097e-06 1.097e-06 1.097e-06 0.00% Amr::init() 1 1.006e-06 1.006e-06 1.006e-06 0.00% AmrLevel::AmrLevel() 1 8.44e-07 8.44e-07 8.44e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3599 0.3599 0.3599 100.00% Amr::coarseTimeStep() 5 0.287 0.287 0.287 79.73% Amr::timeStep() 5 0.2851 0.2851 0.2851 79.21% Castro::advance() 5 0.2801 0.2801 0.2801 77.82% Castro::subcycle_advance_ctu() 5 0.2729 0.2729 0.2729 75.83% Castro::do_advance_ctu() 5 0.2727 0.2727 0.2727 75.76% Castro::construct_new_gravity() 5 0.1436 0.1436 0.1436 39.91% Gravity::solve_phi_with_mlmg() 6 0.1398 0.1398 0.1398 38.83% Gravity::solve_for_phi() 5 0.1361 0.1361 0.1361 37.82% Gravity::actual_solve_with_mlmg() 6 0.135 0.135 0.135 37.51% MLMG::solve() 6 0.1227 0.1227 0.1227 34.09% MLMG::oneIter() 36 0.1157 0.1157 0.1157 32.14% MLMG::mgVcycle() 36 0.1149 0.1149 0.1149 31.93% Castro::construct_ctu_hydro_source() 5 0.08527 0.08527 0.08527 23.69% MLCellLinOp::smooth() 720 0.05891 0.05891 0.05891 16.37% Amr::init() 1 0.04574 0.04574 0.04574 12.71% Amr::restart() 1 0.04574 0.04574 0.04574 12.71% MLCellLinOp::applyBC() 1946 0.04195 0.04195 0.04195 11.66% AmrLevel::restart() 1 0.03846 0.03846 0.03846 10.69% StateData::restartDoit() 4 0.03838 0.03838 0.03838 10.66% VisMF::Read() 3 0.03826 0.03826 0.03826 10.63% MLMG::mgVcycle_bottom 36 0.0352 0.0352 0.0352 9.78% MLMG::actualBottomSolve() 36 0.03518 0.03518 0.03518 9.78% MLCGSolver::bicgstab 36 0.03482 0.03482 0.03482 9.68% Castro::clean_state() 30 0.02731 0.02731 0.02731 7.59% MLPoisson::Fsmooth() 1440 0.02725 0.02725 0.02725 7.57% Amr::writePlotFile() 1 0.02651 0.02651 0.02651 7.36% VisMF::Write(FabArray) 1 0.02496 0.02496 0.02496 6.94% FillPatchIterator::Initialize 20 0.01963 0.01963 0.01963 5.46% FillPatchSingleLevel 20 0.01886 0.01886 0.01886 5.24% StateDataPhysBCFunct::() 20 0.01686 0.01686 0.01686 4.68% MLCellLinOp::apply() 500 0.01584 0.01584 0.01584 4.40% MLMG::mgVcycle_down::0 36 0.01535 0.01535 0.01535 4.27% MLMG::mgVcycle_up::0 36 0.01315 0.01315 0.01315 3.65% Castro::computeTemp() 30 0.01128 0.01128 0.01128 3.13% StateData::FillBoundary(geom) 160 0.01122 0.01122 0.01122 3.12% Castro::initialize_do_advance() 5 0.01052 0.01052 0.01052 2.92% MLPoisson::define() 6 0.009968 0.009968 0.009968 2.77% MultiFab::Dot() 484 0.009587 0.009587 0.009587 2.66% MLCellLinOp::correctionResidual() 216 0.009256 0.009256 0.009256 2.57% Castro::normalize_species() 30 0.009047 0.009047 0.009047 2.51% MLMG:computeResOfCorrection() 180 0.007991 0.007991 0.007991 2.22% MLMG::mgVcycle_down::1 36 0.007713 0.007713 0.007713 2.14% Castro::construct_old_gravity() 5 0.00743 0.00743 0.00743 2.06% MLMG::mgVcycle_down::2 36 0.007427 0.007427 0.007427 2.06% Gravity::get_old_grav_vector() 5 0.007386 0.007386 0.007386 2.05% Gravity::get_new_grav_vector() 5 0.007369 0.007369 0.007369 2.05% MLMG::mgVcycle_down::3 36 0.007058 0.007058 0.007058 1.96% Castro::initialize_advance() 5 0.007007 0.007007 0.007007 1.95% MLCellLinOp::defineAuxData() 6 0.006975 0.006975 0.006975 1.94% FabArray::FillBoundary() 1766 0.006886 0.006886 0.006886 1.91% MLMG::mgVcycle_down::4 36 0.006806 0.006806 0.006806 1.89% Castro::do_new_sources() 5 0.006776 0.006776 0.006776 1.88% FabArray::setVal() 537 0.006764 0.006764 0.006764 1.88% Castro::do_old_sources() 5 0.006595 0.006595 0.006595 1.83% FillBoundary_nowait() 1766 0.006516 0.006516 0.006516 1.81% FabArray::ParallelCopy() 380 0.00645 0.00645 0.00645 1.79% FabArray::ParallelCopy_nowait() 380 0.00632 0.00632 0.00632 1.76% CGSolver::sxay() 690 0.006271 0.006271 0.006271 1.74% MultiFab::LinComb() 690 0.00611 0.00611 0.00611 1.70% Castro::enforce_min_density() 30 0.005845 0.005845 0.005845 1.62% MLCGSolver::ParallelAllReduce 659 0.005714 0.005714 0.005714 1.59% MLMG::mgVcycle_up::2 36 0.005702 0.005702 0.005702 1.58% Castro::expand_state() 5 0.005683 0.005683 0.005683 1.58% MLMG::mgVcycle_up::1 36 0.00564 0.00564 0.00564 1.57% MLMG::addInterpCorrection() 180 0.005436 0.005436 0.005436 1.51% MLMG::mgVcycle_up::3 36 0.005401 0.005401 0.005401 1.50% MLMG::mgVcycle_up::4 36 0.00538 0.00538 0.00538 1.49% amrex::average_down 180 0.005196 0.005196 0.005196 1.44% MLPoisson::Fapply() 500 0.005083 0.005083 0.005083 1.41% Castro::post_timestep() 5 0.004888 0.004888 0.004888 1.36% Gravity::fill_multipole_BCs() 6 0.004545 0.004545 0.004545 1.26% Castro::estTimeStep() 10 0.004171 0.004171 0.004171 1.16% Castro::post_restart() 1 0.004007 0.004007 0.004007 1.11% Gravity::multilevel_solve_for_new_phi() 1 0.003884 0.003884 0.003884 1.08% Gravity::actual_multilevel_solve() 1 0.003867 0.003867 0.003867 1.07% MLCellLinOp::solutionResidual() 42 0.003235 0.003235 0.003235 0.90% MultiFab::Xpay() 258 0.002879 0.002879 0.002879 0.80% MLCellLinOp::defineBC() 6 0.002824 0.002824 0.002824 0.78% MLMG::prepareForSolve() 6 0.002782 0.002782 0.002782 0.77% MLMG::computeResidual() 36 0.002674 0.002674 0.002674 0.74% BndryData::define() 6 0.002671 0.002671 0.002671 0.74% Castro::reset_internal_energy(MultiFab) 30 0.002486 0.002486 0.002486 0.69% Castro::construct_new_source() 25 0.001815 0.001815 0.001815 0.50% Castro::computeNewDt() 5 0.001777 0.001777 0.001777 0.49% Castro::construct_new_gravity_source() 5 0.001668 0.001668 0.001668 0.46% Castro::construct_old_source() 25 0.001403 0.001403 0.001403 0.39% Castro::construct_old_gravity_source() 5 0.001374 0.001374 0.001374 0.38% Castro::enforce_speed_limit() 30 0.001098 0.001098 0.001098 0.31% Castro::reset_internal_energy(Fab) 240 0.0009775 0.0009775 0.0009775 0.27% Castro::apply_source_to_state() 10 0.0009273 0.0009273 0.0009273 0.26% MultiFab::Saxpy() 10 0.0009213 0.0009213 0.0009213 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008891 0.0008891 0.0008891 0.25% MLMG::ResNormInf() 42 0.0008534 0.0008534 0.0008534 0.24% MLCellLinOp::setLevelBC() 6 0.0008237 0.0008237 0.0008237 0.23% FabArrayBase::getCPC() 632 0.0007723 0.0007723 0.0007723 0.21% MLMG::getGradSolution() 6 0.0007629 0.0007629 0.0007629 0.21% MLCellLinOp::compGrad() 6 0.0007599 0.0007599 0.0007599 0.21% FabArray::mult() 22 0.0006484 0.0006484 0.0006484 0.18% MLPoisson::prepareForSolve() 6 0.0006455 0.0006455 0.0006455 0.18% MLCellLinOp::prepareForSolve() 6 0.0006413 0.0006413 0.0006413 0.18% FabArray::setDomainBndry() 20 0.0006353 0.0006353 0.0006353 0.18% Castro::check_for_nan() 10 0.0006146 0.0006146 0.0006146 0.17% MultiFab::contains_nan() 10 0.0006088 0.0006088 0.0006088 0.17% MLMG::computeMLResidual() 6 0.0005777 0.0005777 0.0005777 0.16% Gravity::update_max_rhs() 6 0.0004467 0.0004467 0.0004467 0.12% Amr::InitAmr() 1 0.000408 0.000408 0.000408 0.11% FabArrayBase::CPC::define() 244 0.0004044 0.0004044 0.0004044 0.11% FabArrayBase::getFB() 1766 0.0003096 0.0003096 0.0003096 0.09% Gravity::swapTimeLevels() 5 0.0002339 0.0002339 0.0002339 0.06% MLLinOp::define() 6 0.0001538 0.0001538 0.0001538 0.04% Castro::buildMetrics() 1 0.0001531 0.0001531 0.0001531 0.04% MultiFab::Copy() 6 0.000142 0.000142 0.000142 0.04% MultiFab::max() 6 0.000137 0.000137 0.000137 0.04% MLMG::MLResNormInf() 6 0.0001361 0.0001361 0.0001361 0.04% MLLinOp::defineGrids() 6 0.0001332 0.0001332 0.0001332 0.04% MLMG::MLRhsNormInf() 6 0.0001064 0.0001064 0.0001064 0.03% Castro::create_source_corrector() 5 6.705e-05 6.705e-05 6.705e-05 0.02% FabArrayBase::FB::FB() 26 5.682e-05 5.682e-05 5.682e-05 0.02% Castro::finalize_advance() 5 5.296e-05 5.296e-05 5.296e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.884e-05 2.884e-05 2.884e-05 0.01% Castro::initMFs() 1 2.801e-05 2.801e-05 2.801e-05 0.01% Castro::swap_state_time_levels() 5 2.656e-05 2.656e-05 2.656e-05 0.01% Amr::writeSmallPlotFile() 1 2.58e-05 2.58e-05 2.58e-05 0.01% makeSFC 30 2.187e-05 2.187e-05 2.187e-05 0.01% Castro::finalize_do_advance() 5 1.91e-05 1.91e-05 1.91e-05 0.01% DistributionMapping::Distribute() 31 9.322e-06 9.322e-06 9.322e-06 0.00% Amr::initSubcycle() 1 8.289e-06 8.289e-06 8.289e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.408e-06 5.408e-06 5.408e-06 0.00% MLMG::buildFineMask() 6 3.053e-06 3.053e-06 3.053e-06 0.00% Castro::retry_advance_ctu() 5 2.078e-06 2.078e-06 2.078e-06 0.00% Gravity::set_mass_offset() 6 2.011e-06 2.011e-06 2.011e-06 0.00% Castro::FluxRegCrseInit 5 1.981e-06 1.981e-06 1.981e-06 0.00% Castro::FluxRegFineAdd() 5 1.352e-06 1.352e-06 1.352e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.097e-06 1.097e-06 1.097e-06 0.00% AmrLevel::AmrLevel() 1 8.44e-07 8.44e-07 8.44e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-22-g843a7dff2662) finalized