Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-20-g3082028e4287) initialized Starting run at 12:53:50 UTC on 2022-10-26. Successfully read inputs file ... Castro git describe: 22.07-11-g6cd069552 AMReX git describe: 22.10-20-g3082028e4 Microphysics git describe: 22.10-5-g820f45d5 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.05204032 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.029646612 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.049517768 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051096725 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.072431917 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.064428419 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.06251684 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.050467714 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.067466237 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.079276014 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.068238463 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057279554 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.057140187 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.048290482 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029638718 seconds Ending run at 12:53:51 UTC on 2022-10-26. Run time = 0.892380975 Run time without initialization = 0.7584208 Average number of zones advanced per microsecond: 3.456 Average number of zones advanced per microsecond per rank: 3.456 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8924 ... 0.8924 ... 0.8924 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2507 0.2507 0.2507 28.10% VisMF::Write(FabArray) 11 0.201 0.201 0.201 22.53% MLCellLinOp::applyBC() 4433 0.07887 0.07887 0.07887 8.84% MLPoisson::Fsmooth() 3280 0.06233 0.06233 0.06233 6.98% StateData::FillBoundary(geom) 328 0.02493 0.02493 0.02493 2.79% MLCGSolver::bicgstab 82 0.02329 0.02329 0.02329 2.61% MultiFab::Dot() 1114 0.02171 0.02171 0.02171 2.43% StateDataPhysBCFunct::() 41 0.01547 0.01547 0.01547 1.73% FillBoundary_nowait() 4023 0.01406 0.01406 0.01406 1.58% FabArray::setVal() 1144 0.01399 0.01399 0.01399 1.57% MultiFab::LinComb() 1586 0.01394 0.01394 0.01394 1.56% Castro::computeTemp() 63 0.01319 0.01319 0.01319 1.48% FabArray::ParallelCopy_nowait() 861 0.01279 0.01279 0.01279 1.43% Castro::normalize_species() 62 0.01225 0.01225 0.01225 1.37% Castro::enforce_min_density() 62 0.01204 0.01204 0.01204 1.35% MLCellLinOp::defineAuxData() 11 0.01154 0.01154 0.01154 1.29% MLPoisson::Fapply() 1142 0.01149 0.01149 0.01149 1.29% MLMG::addInterpCorrection() 410 0.007735 0.007735 0.007735 0.87% Gravity::fill_multipole_BCs() 11 0.007698 0.007698 0.007698 0.86% amrex::average_down 410 0.006797 0.006797 0.006797 0.76% MultiFab::Xpay() 585 0.006465 0.006465 0.006465 0.72% Amr::checkPoint() 3 0.006194 0.006194 0.006194 0.69% Castro::estTimeStep() 21 0.0052 0.0052 0.0052 0.58% Castro::do_advance_ctu() 10 0.004291 0.004291 0.004291 0.48% Castro::reset_internal_energy(MultiFab) 63 0.003975 0.003975 0.003975 0.45% BndryData::define() 11 0.003744 0.003744 0.003744 0.42% Amr::writePlotFile() 2 0.002882 0.002882 0.002882 0.32% Castro::construct_new_gravity_source() 10 0.002495 0.002495 0.002495 0.28% MLMG::ResNormInf() 93 0.001997 0.001997 0.001997 0.22% Gravity::get_new_grav_vector() 11 0.001931 0.001931 0.001931 0.22% MultiFab::Saxpy() 20 0.001824 0.001824 0.001824 0.20% Castro::expand_state() 10 0.001758 0.001758 0.001758 0.20% Gravity::get_old_grav_vector() 10 0.001732 0.001732 0.001732 0.19% Castro::construct_old_gravity_source() 10 0.001729 0.001729 0.001729 0.19% MultiFab::Add() 82 0.001652 0.001652 0.001652 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001632 0.001632 0.001632 0.18% Castro::reset_internal_energy(Fab) 504 0.001526 0.001526 0.001526 0.17% Gravity::actual_solve_with_mlmg() 11 0.001518 0.001518 0.001518 0.17% MLCellLinOp::setLevelBC() 11 0.001489 0.001489 0.001489 0.17% FabArray::mult() 43 0.001314 0.001314 0.001314 0.15% Castro::initData() 1 0.001291 0.001291 0.001291 0.14% FabArray::setDomainBndry() 41 0.001289 0.001289 0.001289 0.14% MLMG::prepareForSolve() 11 0.001269 0.001269 0.001269 0.14% MultiFab::contains_nan() 20 0.001194 0.001194 0.001194 0.13% Castro::enforce_speed_limit() 62 0.001175 0.001175 0.001175 0.13% MLCellLinOp::prepareForSolve() 11 0.001144 0.001144 0.001144 0.13% MLCellLinOp::smooth() 1640 0.001055 0.001055 0.001055 0.12% MLCellLinOp::compGrad() 11 0.0009204 0.0009204 0.0009204 0.10% FabArray::FillBoundary() 4023 0.000866 0.000866 0.000866 0.10% FabArrayBase::getCPC() 1323 0.0007891 0.0007891 0.0007891 0.09% FabArrayBase::CPC::define() 454 0.0006853 0.0006853 0.0006853 0.08% FabArrayBase::getFB() 4023 0.0006287 0.0006287 0.0006287 0.07% Gravity::solve_for_phi() 10 0.000512 0.000512 0.000512 0.06% Amr::InitAmr() 1 0.0004624 0.0004624 0.0004624 0.05% MLCellLinOp::apply() 1142 0.0004523 0.0004523 0.0004523 0.05% Gravity::update_max_rhs() 11 0.0004219 0.0004219 0.0004219 0.05% CGSolver::sxay() 1586 0.0003546 0.0003546 0.0003546 0.04% Amr::coarseTimeStep() 10 0.0003433 0.0003433 0.0003433 0.04% MultiFab::Copy() 11 0.0003145 0.0003145 0.0003145 0.04% FillPatchIterator::Initialize 41 0.000312 0.000312 0.000312 0.03% MLCGSolver::ParallelAllReduce 1514 0.0002996 0.0002996 0.0002996 0.03% MLCellLinOp::defineBC() 11 0.0002892 0.0002892 0.0002892 0.03% MultiFab::max() 11 0.0002639 0.0002639 0.0002639 0.03% main() 1 0.0002613 0.0002613 0.0002613 0.03% FabArray::ParallelCopy() 861 0.0002547 0.0002547 0.0002547 0.03% MLMG::mgVcycle() 82 0.0002384 0.0002384 0.0002384 0.03% MLCellLinOp::correctionResidual() 492 0.0002324 0.0002324 0.0002324 0.03% Castro::construct_new_gravity() 10 0.0002155 0.0002155 0.0002155 0.02% MLMG::MLRhsNormInf() 11 0.0002133 0.0002133 0.0002133 0.02% Castro::subcycle_advance_ctu() 10 0.0002002 0.0002002 0.0002002 0.02% Amr::timeStep() 10 0.0001833 0.0001833 0.0001833 0.02% MLLinOp::defineGrids() 11 0.0001633 0.0001633 0.0001633 0.02% MLMG:computeResOfCorrection() 410 0.0001596 0.0001596 0.0001596 0.02% StateData::checkPoint() 12 0.0001451 0.0001451 0.0001451 0.02% MLMG::mgVcycle_down::0 82 0.0001138 0.0001138 0.0001138 0.01% MLMG::mgVcycle_down::1 82 0.0001007 0.0001007 0.0001007 0.01% MLMG::actualBottomSolve() 82 9.665e-05 9.665e-05 9.665e-05 0.01% MLMG::mgVcycle_down::2 82 9.512e-05 9.512e-05 9.512e-05 0.01% Castro::clean_state() 62 9.11e-05 9.11e-05 9.11e-05 0.01% Castro::initialize_advance() 10 8.836e-05 8.836e-05 8.836e-05 0.01% MLMG::mgVcycle_down::3 82 8.835e-05 8.835e-05 8.835e-05 0.01% MLMG::mgVcycle_down::4 82 8.809e-05 8.809e-05 8.809e-05 0.01% Castro::Castro() 1 8.711e-05 8.711e-05 8.711e-05 0.01% FabArrayBase::FB::FB() 56 8.164e-05 8.164e-05 8.164e-05 0.01% Castro::advance() 10 7.805e-05 7.805e-05 7.805e-05 0.01% AmrLevel::checkPoint() 3 7.498e-05 7.498e-05 7.498e-05 0.01% Castro::initialize_do_advance() 10 7.243e-05 7.243e-05 7.243e-05 0.01% MLMG::mgVcycle_up::4 82 7.164e-05 7.164e-05 7.164e-05 0.01% MLMG::solve() 11 6.922e-05 6.922e-05 6.922e-05 0.01% MLMG::oneIter() 82 6.785e-05 6.785e-05 6.785e-05 0.01% Castro::finalize_advance() 10 6.133e-05 6.133e-05 6.133e-05 0.01% MLMG::mgVcycle_up::3 82 6.048e-05 6.048e-05 6.048e-05 0.01% MLMG::mgVcycle_up::0 82 5.888e-05 5.888e-05 5.888e-05 0.01% MLMG::mgVcycle_up::2 82 5.872e-05 5.872e-05 5.872e-05 0.01% MLMG::mgVcycle_up::1 82 5.829e-05 5.829e-05 5.829e-05 0.01% MLCellLinOp::solutionResidual() 93 5.381e-05 5.381e-05 5.381e-05 0.01% Castro::swap_state_time_levels() 10 4.859e-05 4.859e-05 4.859e-05 0.01% Castro::finalize_do_advance() 10 4.511e-05 4.511e-05 4.511e-05 0.01% MLMG::computeResidual() 82 4.29e-05 4.29e-05 4.29e-05 0.00% StateData::define() 4 4.15e-05 4.15e-05 4.15e-05 0.00% Castro::buildMetrics() 1 3.428e-05 3.428e-05 3.428e-05 0.00% Castro::enforce_consistent_e() 1 3.379e-05 3.379e-05 3.379e-05 0.00% MLMG::mgVcycle_bottom 82 3.277e-05 3.277e-05 3.277e-05 0.00% MLPoisson::define() 11 3.135e-05 3.135e-05 3.135e-05 0.00% Gravity::actual_multilevel_solve() 1 3.024e-05 3.024e-05 3.024e-05 0.00% FillPatchSingleLevel 41 2.868e-05 2.868e-05 2.868e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.862e-05 2.862e-05 2.862e-05 0.00% Castro::initMFs() 1 2.756e-05 2.756e-05 2.756e-05 0.00% makeSFC 55 2.717e-05 2.717e-05 2.717e-05 0.00% MLLinOp::define() 11 2.552e-05 2.552e-05 2.552e-05 0.00% Amr::writeSmallPlotFile() 1 2.4e-05 2.4e-05 2.4e-05 0.00% Amr::FinalizeInit() 1 2.069e-05 2.069e-05 2.069e-05 0.00% Amr::defBaseLevel() 1 2.028e-05 2.028e-05 2.028e-05 0.00% Castro::construct_old_source() 50 1.958e-05 1.958e-05 1.958e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.906e-05 1.906e-05 1.906e-05 0.00% Castro::construct_new_source() 50 1.82e-05 1.82e-05 1.82e-05 0.00% Castro::do_new_sources() 10 1.773e-05 1.773e-05 1.773e-05 0.00% Castro::do_old_sources() 10 1.753e-05 1.753e-05 1.753e-05 0.00% DistributionMapping::Distribute() 56 1.461e-05 1.461e-05 1.461e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.316e-05 1.316e-05 1.316e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.308e-05 1.308e-05 1.308e-05 0.00% Castro::check_for_nan() 20 1.296e-05 1.296e-05 1.296e-05 0.00% Castro::apply_source_to_state() 20 1.093e-05 1.093e-05 1.093e-05 0.00% MLMG::computeMLResidual() 11 1.02e-05 1.02e-05 1.02e-05 0.00% Castro::construct_old_gravity() 10 9.904e-06 9.904e-06 9.904e-06 0.00% Castro::post_timestep() 10 9.795e-06 9.795e-06 9.795e-06 0.00% MLPoisson::prepareForSolve() 11 8.921e-06 8.921e-06 8.921e-06 0.00% Gravity::swapTimeLevels() 10 8.794e-06 8.794e-06 8.794e-06 0.00% Castro::computeNewDt() 9 8.075e-06 8.075e-06 8.075e-06 0.00% Amr::initSubcycle() 1 8.072e-06 8.072e-06 8.072e-06 0.00% MLMG::getGradSolution() 11 6.842e-06 6.842e-06 6.842e-06 0.00% Castro::retry_advance_ctu() 10 4.8e-06 4.8e-06 4.8e-06 0.00% AmrLevel::checkPointPost() 3 4.73e-06 4.73e-06 4.73e-06 0.00% Amr::InitializeInit() 1 4.57e-06 4.57e-06 4.57e-06 0.00% Gravity::set_mass_offset() 11 4.264e-06 4.264e-06 4.264e-06 0.00% Castro::post_init() 1 3.68e-06 3.68e-06 3.68e-06 0.00% Castro::create_source_corrector() 10 3.619e-06 3.619e-06 3.619e-06 0.00% AmrLevel::checkPointPre() 3 3.573e-06 3.573e-06 3.573e-06 0.00% MLMG::MLResNormInf() 11 3.465e-06 3.465e-06 3.465e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.373e-06 3.373e-06 3.373e-06 0.00% Castro::FluxRegCrseInit 10 3.372e-06 3.372e-06 3.372e-06 0.00% Amr::init() 1 3.025e-06 3.025e-06 3.025e-06 0.00% Castro::FluxRegFineAdd() 10 2.759e-06 2.759e-06 2.759e-06 0.00% Castro::computeInitialDt() 2 2.384e-06 2.384e-06 2.384e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.85e-06 1.85e-06 1.85e-06 0.00% Castro::post_regrid() 1 1.193e-06 1.193e-06 1.193e-06 0.00% Amr::initialInit() 1 1.053e-06 1.053e-06 1.053e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8924 0.8924 0.8924 100.00% Amr::coarseTimeStep() 10 0.7286 0.7286 0.7286 81.64% Amr::timeStep() 10 0.6268 0.6268 0.6268 70.24% Castro::advance() 10 0.62 0.62 0.62 69.48% Castro::subcycle_advance_ctu() 10 0.6083 0.6083 0.6083 68.16% Castro::do_advance_ctu() 10 0.6081 0.6081 0.6081 68.14% Gravity::solve_phi_with_mlmg() 11 0.308 0.308 0.308 34.52% Gravity::actual_solve_with_mlmg() 11 0.3001 0.3001 0.3001 33.63% Castro::construct_new_gravity() 10 0.2816 0.2816 0.2816 31.55% MLMG::solve() 11 0.2776 0.2776 0.2776 31.10% Gravity::solve_for_phi() 10 0.2644 0.2644 0.2644 29.63% MLMG::oneIter() 82 0.2628 0.2628 0.2628 29.45% MLMG::mgVcycle() 82 0.2611 0.2611 0.2611 29.26% Castro::construct_ctu_hydro_source() 10 0.2507 0.2507 0.2507 28.10% VisMF::Write(FabArray) 11 0.201 0.201 0.201 22.53% Amr::checkPoint() 3 0.1509 0.1509 0.1509 16.91% AmrLevel::checkPoint() 3 0.1447 0.1447 0.1447 16.22% StateData::checkPoint() 12 0.1447 0.1447 0.1447 16.21% MLCellLinOp::smooth() 1640 0.1336 0.1336 0.1336 14.97% Amr::init() 1 0.1334 0.1334 0.1334 14.95% MLCellLinOp::applyBC() 4433 0.09451 0.09451 0.09451 10.59% MLMG::mgVcycle_bottom 82 0.0799 0.0799 0.0799 8.95% MLMG::actualBottomSolve() 82 0.07986 0.07986 0.07986 8.95% MLCGSolver::bicgstab 82 0.07906 0.07906 0.07906 8.86% MLPoisson::Fsmooth() 3280 0.06233 0.06233 0.06233 6.98% Amr::writePlotFile() 2 0.05941 0.05941 0.05941 6.66% Amr::initialInit() 1 0.05156 0.05156 0.05156 5.78% Amr::FinalizeInit() 1 0.04748 0.04748 0.04748 5.32% Castro::post_init() 1 0.04607 0.04607 0.04607 5.16% FillPatchIterator::Initialize 41 0.04599 0.04599 0.04599 5.15% FillPatchSingleLevel 41 0.04438 0.04438 0.04438 4.97% Gravity::multilevel_solve_for_new_phi() 1 0.04416 0.04416 0.04416 4.95% Gravity::actual_multilevel_solve() 1 0.04414 0.04414 0.04414 4.95% Castro::clean_state() 62 0.04353 0.04353 0.04353 4.88% StateDataPhysBCFunct::() 41 0.0404 0.0404 0.0404 4.53% MLCellLinOp::apply() 1142 0.03576 0.03576 0.03576 4.01% MLMG::mgVcycle_down::0 82 0.03482 0.03482 0.03482 3.90% MLMG::mgVcycle_up::0 82 0.02981 0.02981 0.02981 3.34% StateData::FillBoundary(geom) 328 0.02493 0.02493 0.02493 2.79% MultiFab::Dot() 1114 0.02171 0.02171 0.02171 2.43% MLCellLinOp::correctionResidual() 492 0.02093 0.02093 0.02093 2.35% Castro::initialize_do_advance() 10 0.01892 0.01892 0.01892 2.12% Gravity::get_new_grav_vector() 11 0.01874 0.01874 0.01874 2.10% Castro::computeTemp() 63 0.01869 0.01869 0.01869 2.09% MLPoisson::define() 11 0.01811 0.01811 0.01811 2.03% MLMG:computeResOfCorrection() 410 0.0181 0.0181 0.0181 2.03% MLMG::mgVcycle_down::1 82 0.01741 0.01741 0.01741 1.95% MLMG::mgVcycle_down::2 82 0.01695 0.01695 0.01695 1.90% Castro::construct_old_gravity() 10 0.01663 0.01663 0.01663 1.86% Gravity::get_old_grav_vector() 10 0.01662 0.01662 0.01662 1.86% MLMG::mgVcycle_down::3 82 0.0161 0.0161 0.0161 1.80% FabArray::FillBoundary() 4023 0.01564 0.01564 0.01564 1.75% MLMG::mgVcycle_down::4 82 0.01535 0.01535 0.01535 1.72% FillBoundary_nowait() 4023 0.01477 0.01477 0.01477 1.66% CGSolver::sxay() 1586 0.0143 0.0143 0.0143 1.60% FabArray::setVal() 1144 0.01399 0.01399 0.01399 1.57% MultiFab::LinComb() 1586 0.01394 0.01394 0.01394 1.56% FabArray::ParallelCopy() 861 0.01389 0.01389 0.01389 1.56% FabArray::ParallelCopy_nowait() 861 0.01364 0.01364 0.01364 1.53% MLMG::mgVcycle_up::2 82 0.01305 0.01305 0.01305 1.46% MLCGSolver::ParallelAllReduce 1514 0.01298 0.01298 0.01298 1.46% MLMG::mgVcycle_up::1 82 0.01287 0.01287 0.01287 1.44% MLCellLinOp::defineAuxData() 11 0.01287 0.01287 0.01287 1.44% MLMG::addInterpCorrection() 410 0.01269 0.01269 0.01269 1.42% MLMG::mgVcycle_up::3 82 0.01239 0.01239 0.01239 1.39% Castro::normalize_species() 62 0.01225 0.01225 0.01225 1.37% MLMG::mgVcycle_up::4 82 0.01223 0.01223 0.01223 1.37% Castro::enforce_min_density() 62 0.01204 0.01204 0.01204 1.35% amrex::average_down 410 0.01179 0.01179 0.01179 1.32% Castro::initialize_advance() 10 0.0116 0.0116 0.0116 1.30% MLPoisson::Fapply() 1142 0.01149 0.01149 0.01149 1.29% Castro::expand_state() 10 0.01127 0.01127 0.01127 1.26% Castro::do_new_sources() 10 0.01066 0.01066 0.01066 1.20% Castro::do_old_sources() 10 0.009536 0.009536 0.009536 1.07% Gravity::fill_multipole_BCs() 11 0.007698 0.007698 0.007698 0.86% MLCellLinOp::solutionResidual() 93 0.00707 0.00707 0.00707 0.79% Castro::post_timestep() 10 0.006606 0.006606 0.006606 0.74% MultiFab::Xpay() 585 0.006465 0.006465 0.006465 0.72% MLMG::computeResidual() 82 0.006076 0.006076 0.006076 0.68% Castro::reset_internal_energy(MultiFab) 63 0.005501 0.005501 0.005501 0.62% MLMG::prepareForSolve() 11 0.005303 0.005303 0.005303 0.59% Castro::estTimeStep() 21 0.0052 0.0052 0.0052 0.58% MLCellLinOp::defineBC() 11 0.004957 0.004957 0.004957 0.56% BndryData::define() 11 0.004668 0.004668 0.004668 0.52% Amr::InitializeInit() 1 0.004077 0.004077 0.004077 0.46% Amr::defBaseLevel() 1 0.004073 0.004073 0.004073 0.46% Castro::initData() 1 0.003529 0.003529 0.003529 0.40% Castro::construct_new_source() 50 0.002513 0.002513 0.002513 0.28% Castro::construct_new_gravity_source() 10 0.002495 0.002495 0.002495 0.28% Castro::computeNewDt() 9 0.002021 0.002021 0.002021 0.23% MLMG::ResNormInf() 93 0.001997 0.001997 0.001997 0.22% Castro::apply_source_to_state() 20 0.001835 0.001835 0.001835 0.21% MultiFab::Saxpy() 20 0.001824 0.001824 0.001824 0.20% Castro::construct_old_source() 50 0.001748 0.001748 0.001748 0.20% Castro::construct_old_gravity_source() 10 0.001729 0.001729 0.001729 0.19% MultiFab::Add() 82 0.001652 0.001652 0.001652 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001632 0.001632 0.001632 0.18% Castro::reset_internal_energy(Fab) 504 0.001526 0.001526 0.001526 0.17% MLCellLinOp::setLevelBC() 11 0.001489 0.001489 0.001489 0.17% FabArrayBase::getCPC() 1323 0.001474 0.001474 0.001474 0.17% MLMG::getGradSolution() 11 0.001422 0.001422 0.001422 0.16% MLCellLinOp::compGrad() 11 0.001416 0.001416 0.001416 0.16% FabArray::mult() 43 0.001314 0.001314 0.001314 0.15% FabArray::setDomainBndry() 41 0.001289 0.001289 0.001289 0.14% Castro::check_for_nan() 20 0.001207 0.001207 0.001207 0.14% MultiFab::contains_nan() 20 0.001194 0.001194 0.001194 0.13% Castro::enforce_speed_limit() 62 0.001175 0.001175 0.001175 0.13% Castro::post_regrid() 1 0.001162 0.001162 0.001162 0.13% MLPoisson::prepareForSolve() 11 0.001153 0.001153 0.001153 0.13% MLCellLinOp::prepareForSolve() 11 0.001144 0.001144 0.001144 0.13% MLMG::computeMLResidual() 11 0.001048 0.001048 0.001048 0.12% Gravity::update_max_rhs() 11 0.0008272 0.0008272 0.0008272 0.09% Castro::computeInitialDt() 2 0.000752 0.000752 0.000752 0.08% FabArrayBase::getFB() 4023 0.0007104 0.0007104 0.0007104 0.08% FabArrayBase::CPC::define() 454 0.0006853 0.0006853 0.0006853 0.08% Amr::InitAmr() 1 0.0004705 0.0004705 0.0004705 0.05% Castro::Castro() 1 0.0004645 0.0004645 0.0004645 0.05% Gravity::swapTimeLevels() 10 0.0004352 0.0004352 0.0004352 0.05% MultiFab::Copy() 11 0.0003145 0.0003145 0.0003145 0.04% MLMG::MLResNormInf() 11 0.0002785 0.0002785 0.0002785 0.03% MultiFab::max() 11 0.0002639 0.0002639 0.0002639 0.03% MLLinOp::define() 11 0.0002443 0.0002443 0.0002443 0.03% MLLinOp::defineGrids() 11 0.0002188 0.0002188 0.0002188 0.02% MLMG::MLRhsNormInf() 11 0.0002133 0.0002133 0.0002133 0.02% Castro::buildMetrics() 1 0.0001866 0.0001866 0.0001866 0.02% FabArrayBase::FB::FB() 56 8.164e-05 8.164e-05 8.164e-05 0.01% Castro::finalize_advance() 10 6.746e-05 6.746e-05 6.746e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.458e-05 5.458e-05 5.458e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.367e-05 5.367e-05 5.367e-05 0.01% Castro::swap_state_time_levels() 10 4.859e-05 4.859e-05 4.859e-05 0.01% Castro::finalize_do_advance() 10 4.511e-05 4.511e-05 4.511e-05 0.01% StateData::define() 4 4.15e-05 4.15e-05 4.15e-05 0.00% makeSFC 55 4.052e-05 4.052e-05 4.052e-05 0.00% Castro::enforce_consistent_e() 1 3.379e-05 3.379e-05 3.379e-05 0.00% Castro::initMFs() 1 2.756e-05 2.756e-05 2.756e-05 0.00% Amr::writeSmallPlotFile() 1 2.4e-05 2.4e-05 2.4e-05 0.00% DistributionMapping::Distribute() 56 1.461e-05 1.461e-05 1.461e-05 0.00% Amr::initSubcycle() 1 8.072e-06 8.072e-06 8.072e-06 0.00% Castro::retry_advance_ctu() 10 4.8e-06 4.8e-06 4.8e-06 0.00% AmrLevel::checkPointPost() 3 4.73e-06 4.73e-06 4.73e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.636e-06 4.636e-06 4.636e-06 0.00% Gravity::set_mass_offset() 11 4.264e-06 4.264e-06 4.264e-06 0.00% Castro::create_source_corrector() 10 3.619e-06 3.619e-06 3.619e-06 0.00% AmrLevel::checkPointPre() 3 3.573e-06 3.573e-06 3.573e-06 0.00% Castro::FluxRegCrseInit 10 3.372e-06 3.372e-06 3.372e-06 0.00% Castro::FluxRegFineAdd() 10 2.759e-06 2.759e-06 2.759e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.85e-06 1.85e-06 1.85e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-20-g3082028e4287) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.10-20-g3082028e4287) initialized Starting run at 12:53:52 UTC on 2022-10-26. Successfully read inputs file ... Castro git describe: 22.07-11-g6cd069552 AMReX git describe: 22.10-20-g3082028e4 Microphysics git describe: 22.10-5-g820f45d5 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.484450794 Restart time = 0.04754276 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.050935056 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050642012 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057382616 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.065305269 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.068471203 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031494009 seconds Ending run at 12:53:52 UTC on 2022-10-26. Run time = 0.372691212 Run time without initialization = 0.324611924 Average number of zones advanced per microsecond: 4.038 Average number of zones advanced per microsecond per rank: 4.038 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3727 ... 0.3727 ... 0.3727 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0950 0.0950 0.0950 25.49% VisMF::Read() 3 0.03973 0.03973 0.03973 10.66% MLCellLinOp::applyBC() 1946 0.03472 0.03472 0.03472 9.31% VisMF::Write(FabArray) 1 0.02981 0.02981 0.02981 8.00% MLPoisson::Fsmooth() 1440 0.02723 0.02723 0.02723 7.31% StateData::FillBoundary(geom) 160 0.01136 0.01136 0.01136 3.05% MLCGSolver::bicgstab 36 0.01012 0.01012 0.01012 2.72% MultiFab::Dot() 484 0.009451 0.009451 0.009451 2.54% Castro::computeTemp() 30 0.007608 0.007608 0.007608 2.04% Castro::normalize_species() 30 0.007604 0.007604 0.007604 2.04% FabArray::setVal() 537 0.006743 0.006743 0.006743 1.81% MLCellLinOp::defineAuxData() 6 0.006228 0.006228 0.006228 1.67% FillBoundary_nowait() 1766 0.006171 0.006171 0.006171 1.66% MultiFab::LinComb() 690 0.006095 0.006095 0.006095 1.64% Castro::enforce_min_density() 30 0.006065 0.006065 0.006065 1.63% FabArray::ParallelCopy_nowait() 380 0.005961 0.005961 0.005961 1.60% StateDataPhysBCFunct::() 20 0.005907 0.005907 0.005907 1.58% MLPoisson::Fapply() 500 0.00503 0.00503 0.00503 1.35% Gravity::fill_multipole_BCs() 6 0.004218 0.004218 0.004218 1.13% Amr::restart() 1 0.003615 0.003615 0.003615 0.97% MLMG::addInterpCorrection() 180 0.003367 0.003367 0.003367 0.90% amrex::average_down 180 0.002965 0.002965 0.002965 0.80% Castro::do_advance_ctu() 5 0.002865 0.002865 0.002865 0.77% MultiFab::Xpay() 258 0.002855 0.002855 0.002855 0.77% Castro::estTimeStep() 10 0.002104 0.002104 0.002104 0.56% BndryData::define() 6 0.00206 0.00206 0.00206 0.55% Castro::construct_new_gravity_source() 5 0.001805 0.001805 0.001805 0.48% Amr::writePlotFile() 1 0.001765 0.001765 0.001765 0.47% Castro::reset_internal_energy(MultiFab) 30 0.001672 0.001672 0.001672 0.45% Castro::construct_old_gravity_source() 5 0.001561 0.001561 0.001561 0.42% Castro::enforce_speed_limit() 30 0.001286 0.001286 0.001286 0.35% Castro::reset_internal_energy(Fab) 240 0.001074 0.001074 0.001074 0.29% Gravity::get_old_grav_vector() 5 0.0009933 0.0009933 0.0009933 0.27% Gravity::get_new_grav_vector() 5 0.0009547 0.0009547 0.0009547 0.26% MultiFab::Saxpy() 10 0.0009214 0.0009214 0.0009214 0.25% MLMG::ResNormInf() 42 0.0008893 0.0008893 0.0008893 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008848 0.0008848 0.0008848 0.24% Castro::expand_state() 5 0.0008704 0.0008704 0.0008704 0.23% MLCellLinOp::setLevelBC() 6 0.0008039 0.0008039 0.0008039 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007974 0.0007974 0.0007974 0.21% MultiFab::Add() 36 0.0007263 0.0007263 0.0007263 0.19% MLMG::prepareForSolve() 6 0.0006513 0.0006513 0.0006513 0.17% FabArray::mult() 22 0.000644 0.000644 0.000644 0.17% FabArray::setDomainBndry() 20 0.0006393 0.0006393 0.0006393 0.17% MLCellLinOp::prepareForSolve() 6 0.0006295 0.0006295 0.0006295 0.17% MultiFab::contains_nan() 10 0.0005864 0.0005864 0.0005864 0.16% MLCellLinOp::compGrad() 6 0.0004972 0.0004972 0.0004972 0.13% MLCellLinOp::smooth() 720 0.0004745 0.0004745 0.0004745 0.13% FabArrayBase::CPC::define() 244 0.0003922 0.0003922 0.0003922 0.11% Amr::InitAmr() 1 0.000388 0.000388 0.000388 0.10% FabArray::FillBoundary() 1766 0.0003703 0.0003703 0.0003703 0.10% FabArrayBase::getCPC() 632 0.0003584 0.0003584 0.0003584 0.10% FabArrayBase::getFB() 1766 0.0002609 0.0002609 0.0002609 0.07% Castro::subcycle_advance_ctu() 5 0.0002439 0.0002439 0.0002439 0.07% main() 1 0.0002314 0.0002314 0.0002314 0.06% Gravity::update_max_rhs() 6 0.0002271 0.0002271 0.0002271 0.06% Gravity::solve_for_phi() 5 0.0001999 0.0001999 0.0001999 0.05% MLCellLinOp::apply() 500 0.0001983 0.0001983 0.0001983 0.05% MultiFab::Copy() 6 0.0001749 0.0001749 0.0001749 0.05% Castro::advance() 5 0.0001711 0.0001711 0.0001711 0.05% Castro::create_source_corrector() 5 0.0001695 0.0001695 0.0001695 0.05% CGSolver::sxay() 690 0.0001627 0.0001627 0.0001627 0.04% Amr::coarseTimeStep() 5 0.0001575 0.0001575 0.0001575 0.04% Castro::construct_new_source() 25 0.0001522 0.0001522 0.0001522 0.04% Castro::construct_new_gravity() 5 0.0001471 0.0001471 0.0001471 0.04% MLCellLinOp::defineBC() 6 0.0001458 0.0001458 0.0001458 0.04% FillPatchIterator::Initialize 20 0.0001369 0.0001369 0.0001369 0.04% MultiFab::max() 6 0.0001355 0.0001355 0.0001355 0.04% MLCGSolver::ParallelAllReduce 659 0.0001165 0.0001165 0.0001165 0.03% MLMG::MLRhsNormInf() 6 0.0001124 0.0001124 0.0001124 0.03% FabArray::ParallelCopy() 380 0.0001123 0.0001123 0.0001123 0.03% MLCellLinOp::correctionResidual() 216 9.557e-05 9.557e-05 9.557e-05 0.03% MLMG::mgVcycle() 36 9.004e-05 9.004e-05 9.004e-05 0.02% MLLinOp::defineGrids() 6 8.989e-05 8.989e-05 8.989e-05 0.02% Castro::construct_old_source() 25 8.515e-05 8.515e-05 8.515e-05 0.02% AmrLevel::restart() 1 8.163e-05 8.163e-05 8.163e-05 0.02% Amr::timeStep() 5 8.159e-05 8.159e-05 8.159e-05 0.02% StateData::restartDoit() 4 7.243e-05 7.243e-05 7.243e-05 0.02% Castro::computeNewDt() 5 7.185e-05 7.185e-05 7.185e-05 0.02% MLMG:computeResOfCorrection() 180 6.891e-05 6.891e-05 6.891e-05 0.02% Castro::finalize_advance() 5 6.735e-05 6.735e-05 6.735e-05 0.02% FabArrayBase::FB::FB() 26 5.393e-05 5.393e-05 5.393e-05 0.01% MLMG::mgVcycle_down::0 36 4.949e-05 4.949e-05 4.949e-05 0.01% Castro::initialize_do_advance() 5 4.692e-05 4.692e-05 4.692e-05 0.01% MLMG::mgVcycle_down::1 36 4.488e-05 4.488e-05 4.488e-05 0.01% MLMG::mgVcycle_down::4 36 4.473e-05 4.473e-05 4.473e-05 0.01% MLMG::mgVcycle_down::2 36 4.466e-05 4.466e-05 4.466e-05 0.01% Castro::initialize_advance() 5 4.059e-05 4.059e-05 4.059e-05 0.01% MLMG::mgVcycle_down::3 36 4.035e-05 4.035e-05 4.035e-05 0.01% Castro::clean_state() 30 3.943e-05 3.943e-05 3.943e-05 0.01% MLMG::actualBottomSolve() 36 3.939e-05 3.939e-05 3.939e-05 0.01% MLMG::mgVcycle_up::4 36 3.29e-05 3.29e-05 3.29e-05 0.01% MLMG::solve() 6 3.192e-05 3.192e-05 3.192e-05 0.01% Castro::buildMetrics() 1 3.192e-05 3.192e-05 3.192e-05 0.01% Gravity::actual_multilevel_solve() 1 2.98e-05 2.98e-05 2.98e-05 0.01% Castro::post_restart() 1 2.822e-05 2.822e-05 2.822e-05 0.01% MLMG::mgVcycle_up::3 36 2.77e-05 2.77e-05 2.77e-05 0.01% MLMG::oneIter() 36 2.754e-05 2.754e-05 2.754e-05 0.01% MLMG::mgVcycle_up::0 36 2.752e-05 2.752e-05 2.752e-05 0.01% Amr::writeSmallPlotFile() 1 2.75e-05 2.75e-05 2.75e-05 0.01% Castro::swap_state_time_levels() 5 2.731e-05 2.731e-05 2.731e-05 0.01% MLMG::mgVcycle_up::2 36 2.664e-05 2.664e-05 2.664e-05 0.01% Castro::initMFs() 1 2.592e-05 2.592e-05 2.592e-05 0.01% MLMG::mgVcycle_up::1 36 2.479e-05 2.479e-05 2.479e-05 0.01% MLCellLinOp::solutionResidual() 42 2.337e-05 2.337e-05 2.337e-05 0.01% MLPoisson::define() 6 2.135e-05 2.135e-05 2.135e-05 0.01% MLLinOp::define() 6 2.059e-05 2.059e-05 2.059e-05 0.01% Castro::post_timestep() 5 2.048e-05 2.048e-05 2.048e-05 0.01% MLMG::computeResidual() 36 1.886e-05 1.886e-05 1.886e-05 0.01% Castro::finalize_do_advance() 5 1.778e-05 1.778e-05 1.778e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.725e-05 1.725e-05 1.725e-05 0.00% makeSFC 30 1.534e-05 1.534e-05 1.534e-05 0.00% MLMG::mgVcycle_bottom 36 1.511e-05 1.511e-05 1.511e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.428e-05 1.428e-05 1.428e-05 0.00% FillPatchSingleLevel 20 1.268e-05 1.268e-05 1.268e-05 0.00% Castro::do_new_sources() 5 1.008e-05 1.008e-05 1.008e-05 0.00% Castro::do_old_sources() 5 8.915e-06 8.915e-06 8.915e-06 0.00% DistributionMapping::Distribute() 31 8.604e-06 8.604e-06 8.604e-06 0.00% Amr::initSubcycle() 1 8.183e-06 8.183e-06 8.183e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.419e-06 7.419e-06 7.419e-06 0.00% Castro::check_for_nan() 10 6.58e-06 6.58e-06 6.58e-06 0.00% Castro::apply_source_to_state() 10 6.263e-06 6.263e-06 6.263e-06 0.00% Castro::construct_old_gravity() 5 5.378e-06 5.378e-06 5.378e-06 0.00% MLMG::computeMLResidual() 6 4.979e-06 4.979e-06 4.979e-06 0.00% MLPoisson::prepareForSolve() 6 4.335e-06 4.335e-06 4.335e-06 0.00% Gravity::swapTimeLevels() 5 4.166e-06 4.166e-06 4.166e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.52e-06 3.52e-06 3.52e-06 0.00% MLMG::getGradSolution() 6 3.076e-06 3.076e-06 3.076e-06 0.00% Gravity::set_mass_offset() 6 2.462e-06 2.462e-06 2.462e-06 0.00% Castro::retry_advance_ctu() 5 2.061e-06 2.061e-06 2.061e-06 0.00% Castro::FluxRegCrseInit 5 1.996e-06 1.996e-06 1.996e-06 0.00% MLMG::MLResNormInf() 6 1.98e-06 1.98e-06 1.98e-06 0.00% Castro::FluxRegFineAdd() 5 1.18e-06 1.18e-06 1.18e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.131e-06 1.131e-06 1.131e-06 0.00% Amr::init() 1 1.121e-06 1.121e-06 1.121e-06 0.00% AmrLevel::AmrLevel() 1 9.94e-07 9.94e-07 9.94e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3727 0.3727 0.3727 100.00% Amr::coarseTimeStep() 5 0.2929 0.2929 0.2929 78.58% Amr::timeStep() 5 0.2915 0.2915 0.2915 78.22% Castro::advance() 5 0.288 0.288 0.288 77.28% Castro::subcycle_advance_ctu() 5 0.2815 0.2815 0.2815 75.52% Castro::do_advance_ctu() 5 0.2812 0.2812 0.2812 75.45% Castro::construct_new_gravity() 5 0.1431 0.1431 0.1431 38.40% Gravity::solve_phi_with_mlmg() 6 0.1387 0.1387 0.1387 37.22% Gravity::solve_for_phi() 5 0.1352 0.1352 0.1352 36.28% Gravity::actual_solve_with_mlmg() 6 0.1344 0.1344 0.1344 36.06% MLMG::solve() 6 0.1222 0.1222 0.1222 32.78% MLMG::oneIter() 36 0.115 0.115 0.115 30.86% MLMG::mgVcycle() 36 0.1143 0.1143 0.1143 30.66% Castro::construct_ctu_hydro_source() 5 0.09502 0.09502 0.09502 25.49% MLCellLinOp::smooth() 720 0.0585 0.0585 0.0585 15.70% Amr::init() 1 0.04758 0.04758 0.04758 12.77% Amr::restart() 1 0.04758 0.04758 0.04758 12.77% MLCellLinOp::applyBC() 1946 0.04157 0.04157 0.04157 11.15% AmrLevel::restart() 1 0.03994 0.03994 0.03994 10.72% StateData::restartDoit() 4 0.03985 0.03985 0.03985 10.69% VisMF::Read() 3 0.03973 0.03973 0.03973 10.66% MLMG::mgVcycle_bottom 36 0.03483 0.03483 0.03483 9.35% MLMG::actualBottomSolve() 36 0.03482 0.03482 0.03482 9.34% MLCGSolver::bicgstab 36 0.03446 0.03446 0.03446 9.25% Amr::writePlotFile() 1 0.03158 0.03158 0.03158 8.47% VisMF::Write(FabArray) 1 0.02981 0.02981 0.02981 8.00% MLPoisson::Fsmooth() 1440 0.02723 0.02723 0.02723 7.31% Castro::clean_state() 30 0.02535 0.02535 0.02535 6.80% FillPatchIterator::Initialize 20 0.02005 0.02005 0.02005 5.38% FillPatchSingleLevel 20 0.01928 0.01928 0.01928 5.17% StateDataPhysBCFunct::() 20 0.01727 0.01727 0.01727 4.63% MLCellLinOp::apply() 500 0.01573 0.01573 0.01573 4.22% MLMG::mgVcycle_down::0 36 0.01529 0.01529 0.01529 4.10% MLMG::mgVcycle_up::0 36 0.01312 0.01312 0.01312 3.52% StateData::FillBoundary(geom) 160 0.01136 0.01136 0.01136 3.05% Castro::initialize_do_advance() 5 0.01118 0.01118 0.01118 3.00% Castro::computeTemp() 30 0.01035 0.01035 0.01035 2.78% MLPoisson::define() 6 0.00985 0.00985 0.00985 2.64% MultiFab::Dot() 484 0.009451 0.009451 0.009451 2.54% MLCellLinOp::correctionResidual() 216 0.009175 0.009175 0.009175 2.46% MLMG:computeResOfCorrection() 180 0.007915 0.007915 0.007915 2.12% Castro::construct_old_gravity() 5 0.007895 0.007895 0.007895 2.12% Gravity::get_old_grav_vector() 5 0.007889 0.007889 0.007889 2.12% Gravity::get_new_grav_vector() 5 0.007747 0.007747 0.007747 2.08% MLMG::mgVcycle_down::1 36 0.007638 0.007638 0.007638 2.05% Castro::normalize_species() 30 0.007604 0.007604 0.007604 2.04% MLMG::mgVcycle_down::2 36 0.007405 0.007405 0.007405 1.99% Castro::do_new_sources() 5 0.007267 0.007267 0.007267 1.95% MLMG::mgVcycle_down::3 36 0.007027 0.007027 0.007027 1.89% MLCellLinOp::defineAuxData() 6 0.006949 0.006949 0.006949 1.86% FabArray::FillBoundary() 1766 0.006856 0.006856 0.006856 1.84% FabArray::setVal() 537 0.006743 0.006743 0.006743 1.81% MLMG::mgVcycle_down::4 36 0.00674 0.00674 0.00674 1.81% FillBoundary_nowait() 1766 0.006486 0.006486 0.006486 1.74% FabArray::ParallelCopy() 380 0.006456 0.006456 0.006456 1.73% FabArray::ParallelCopy_nowait() 380 0.006344 0.006344 0.006344 1.70% Castro::initialize_advance() 5 0.00633 0.00633 0.00633 1.70% CGSolver::sxay() 690 0.006258 0.006258 0.006258 1.68% MultiFab::LinComb() 690 0.006095 0.006095 0.006095 1.64% Castro::do_old_sources() 5 0.006088 0.006088 0.006088 1.63% Castro::enforce_min_density() 30 0.006065 0.006065 0.006065 1.63% MLMG::mgVcycle_up::2 36 0.005738 0.005738 0.005738 1.54% MLCGSolver::ParallelAllReduce 659 0.005649 0.005649 0.005649 1.52% MLMG::mgVcycle_up::1 36 0.005633 0.005633 0.005633 1.51% MLMG::addInterpCorrection() 180 0.005589 0.005589 0.005589 1.50% MLMG::mgVcycle_up::3 36 0.005427 0.005427 0.005427 1.46% Castro::expand_state() 5 0.005354 0.005354 0.005354 1.44% MLMG::mgVcycle_up::4 36 0.005349 0.005349 0.005349 1.44% amrex::average_down 180 0.005206 0.005206 0.005206 1.40% MLPoisson::Fapply() 500 0.00503 0.00503 0.00503 1.35% Gravity::fill_multipole_BCs() 6 0.004218 0.004218 0.004218 1.13% Castro::post_restart() 1 0.003852 0.003852 0.003852 1.03% Gravity::multilevel_solve_for_new_phi() 1 0.003731 0.003731 0.003731 1.00% Gravity::actual_multilevel_solve() 1 0.003714 0.003714 0.003714 1.00% Castro::post_timestep() 5 0.003422 0.003422 0.003422 0.92% MLCellLinOp::solutionResidual() 42 0.0032 0.0032 0.0032 0.86% MLMG::prepareForSolve() 6 0.002865 0.002865 0.002865 0.77% MultiFab::Xpay() 258 0.002855 0.002855 0.002855 0.77% Castro::reset_internal_energy(MultiFab) 30 0.002746 0.002746 0.002746 0.74% MLCellLinOp::defineBC() 6 0.002738 0.002738 0.002738 0.73% MLMG::computeResidual() 36 0.002666 0.002666 0.002666 0.72% BndryData::define() 6 0.002592 0.002592 0.002592 0.70% Castro::estTimeStep() 10 0.002104 0.002104 0.002104 0.56% Castro::construct_new_source() 25 0.001957 0.001957 0.001957 0.53% Castro::construct_new_gravity_source() 5 0.001805 0.001805 0.001805 0.48% Castro::construct_old_source() 25 0.001646 0.001646 0.001646 0.44% Castro::construct_old_gravity_source() 5 0.001561 0.001561 0.001561 0.42% Castro::enforce_speed_limit() 30 0.001286 0.001286 0.001286 0.35% Castro::computeNewDt() 5 0.001192 0.001192 0.001192 0.32% Castro::reset_internal_energy(Fab) 240 0.001074 0.001074 0.001074 0.29% Castro::apply_source_to_state() 10 0.0009277 0.0009277 0.0009277 0.25% MultiFab::Saxpy() 10 0.0009214 0.0009214 0.0009214 0.25% MLMG::ResNormInf() 42 0.0008893 0.0008893 0.0008893 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008848 0.0008848 0.0008848 0.24% MLCellLinOp::setLevelBC() 6 0.0008039 0.0008039 0.0008039 0.22% MLMG::getGradSolution() 6 0.0007748 0.0007748 0.0007748 0.21% MLCellLinOp::compGrad() 6 0.0007718 0.0007718 0.0007718 0.21% FabArrayBase::getCPC() 632 0.0007505 0.0007505 0.0007505 0.20% MultiFab::Add() 36 0.0007263 0.0007263 0.0007263 0.19% FabArray::mult() 22 0.000644 0.000644 0.000644 0.17% FabArray::setDomainBndry() 20 0.0006393 0.0006393 0.0006393 0.17% MLPoisson::prepareForSolve() 6 0.0006339 0.0006339 0.0006339 0.17% MLCellLinOp::prepareForSolve() 6 0.0006295 0.0006295 0.0006295 0.17% Castro::check_for_nan() 10 0.000593 0.000593 0.000593 0.16% MultiFab::contains_nan() 10 0.0005864 0.0005864 0.0005864 0.16% MLMG::computeMLResidual() 6 0.0005582 0.0005582 0.0005582 0.15% Gravity::update_max_rhs() 6 0.0004407 0.0004407 0.0004407 0.12% Amr::InitAmr() 1 0.0003962 0.0003962 0.0003962 0.11% FabArrayBase::CPC::define() 244 0.0003922 0.0003922 0.0003922 0.11% FabArrayBase::getFB() 1766 0.0003148 0.0003148 0.0003148 0.08% Gravity::swapTimeLevels() 5 0.0002413 0.0002413 0.0002413 0.06% MultiFab::Copy() 6 0.0001749 0.0001749 0.0001749 0.05% Castro::create_source_corrector() 5 0.0001695 0.0001695 0.0001695 0.05% Castro::buildMetrics() 1 0.0001485 0.0001485 0.0001485 0.04% MLMG::MLResNormInf() 6 0.0001479 0.0001479 0.0001479 0.04% MLLinOp::define() 6 0.0001417 0.0001417 0.0001417 0.04% MultiFab::max() 6 0.0001355 0.0001355 0.0001355 0.04% MLLinOp::defineGrids() 6 0.0001211 0.0001211 0.0001211 0.03% MLMG::MLRhsNormInf() 6 0.0001124 0.0001124 0.0001124 0.03% Castro::finalize_advance() 5 7.053e-05 7.053e-05 7.053e-05 0.02% FabArrayBase::FB::FB() 26 5.393e-05 5.393e-05 5.393e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.012e-05 3.012e-05 3.012e-05 0.01% Amr::writeSmallPlotFile() 1 2.75e-05 2.75e-05 2.75e-05 0.01% Castro::swap_state_time_levels() 5 2.731e-05 2.731e-05 2.731e-05 0.01% Castro::initMFs() 1 2.592e-05 2.592e-05 2.592e-05 0.01% makeSFC 30 2.27e-05 2.27e-05 2.27e-05 0.01% Castro::finalize_do_advance() 5 1.778e-05 1.778e-05 1.778e-05 0.00% DistributionMapping::Distribute() 31 8.604e-06 8.604e-06 8.604e-06 0.00% Amr::initSubcycle() 1 8.183e-06 8.183e-06 8.183e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.757e-06 4.757e-06 4.757e-06 0.00% Gravity::set_mass_offset() 6 2.462e-06 2.462e-06 2.462e-06 0.00% Castro::retry_advance_ctu() 5 2.061e-06 2.061e-06 2.061e-06 0.00% Castro::FluxRegCrseInit 5 1.996e-06 1.996e-06 1.996e-06 0.00% Castro::FluxRegFineAdd() 5 1.18e-06 1.18e-06 1.18e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.131e-06 1.131e-06 1.131e-06 0.00% AmrLevel::AmrLevel() 1 9.94e-07 9.94e-07 9.94e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.10-20-g3082028e4287) finalized