Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-1-gfb0b31e1439b) initialized Starting run at 08:34:13 UTC on 2022-09-06. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09-1-gfb0b31e14 Microphysics git describe: 22.08-10-g65622313 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.052535401 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030283834 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048577717 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051249637 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.057602979 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.049365662 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.068571446 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.04914519 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.0688312 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.068928861 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.071661123 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.061991019 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.054118234 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.049130575 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.030310056 seconds Ending run at 08:34:14 UTC on 2022-09-06. Run time = 0.862919969 Run time without initialization = 0.73019804 Average number of zones advanced per microsecond: 3.590 Average number of zones advanced per microsecond per rank: 3.590 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8629 ... 0.8629 ... 0.8629 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2057 0.2057 0.2057 23.84% VisMF::Write(FabArray) 11 0.2033 0.2033 0.2033 23.56% MLCellLinOp::applyBC() 4379 0.08048 0.08048 0.08048 9.33% MLPoisson::Fsmooth() 3240 0.06322 0.06322 0.06322 7.33% StateData::FillBoundary(geom) 328 0.02454 0.02454 0.02454 2.84% MLCGSolver::bicgstab 81 0.02398 0.02398 0.02398 2.78% MultiFab::Dot() 1100 0.02236 0.02236 0.02236 2.59% Castro::normalize_species() 62 0.01585 0.01585 0.01585 1.84% FabArray::setVal() 1135 0.01439 0.01439 0.01439 1.67% MultiFab::LinComb() 1566 0.01435 0.01435 0.01435 1.66% FillBoundary_nowait() 3974 0.01426 0.01426 0.01426 1.65% Castro::computeTemp() 63 0.0139 0.0139 0.0139 1.61% FabArray::ParallelCopy_nowait() 851 0.0134 0.0134 0.0134 1.55% MLCellLinOp::defineAuxData() 11 0.01214 0.01214 0.01214 1.41% StateDataPhysBCFunct::() 41 0.01212 0.01212 0.01212 1.40% MLPoisson::Fapply() 1128 0.01178 0.01178 0.01178 1.37% Gravity::fill_multipole_BCs() 11 0.01032 0.01032 0.01032 1.20% Castro::enforce_min_density() 62 0.009882 0.009882 0.009882 1.15% MLMG::addInterpCorrection() 405 0.007884 0.007884 0.007884 0.91% amrex::average_down 405 0.006886 0.006886 0.006886 0.80% MultiFab::Xpay() 578 0.006517 0.006517 0.006517 0.76% Castro::estTimeStep() 21 0.006443 0.006443 0.006443 0.75% Amr::checkPoint() 3 0.005309 0.005309 0.005309 0.62% Castro::do_advance_ctu() 10 0.005278 0.005278 0.005278 0.61% Castro::reset_internal_energy(MultiFab) 63 0.004126 0.004126 0.004126 0.48% BndryData::define() 11 0.004098 0.004098 0.004098 0.47% Castro::construct_new_gravity_source() 10 0.003384 0.003384 0.003384 0.39% Castro::construct_old_gravity_source() 10 0.002881 0.002881 0.002881 0.33% Amr::writePlotFile() 2 0.002827 0.002827 0.002827 0.33% MLMG::ResNormInf() 92 0.00212 0.00212 0.00212 0.25% Gravity::get_new_grav_vector() 11 0.002046 0.002046 0.002046 0.24% MultiFab::Saxpy() 20 0.00182 0.00182 0.00182 0.21% Gravity::get_old_grav_vector() 10 0.00179 0.00179 0.00179 0.21% Castro::expand_state() 10 0.001739 0.001739 0.001739 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001696 0.001696 0.001696 0.20% MultiFab::Add() 81 0.001674 0.001674 0.001674 0.19% Castro::reset_internal_energy(Fab) 504 0.001628 0.001628 0.001628 0.19% MLCellLinOp::setLevelBC() 11 0.001559 0.001559 0.001559 0.18% Gravity::actual_solve_with_mlmg() 11 0.001525 0.001525 0.001525 0.18% FabArray::mult() 43 0.001351 0.001351 0.001351 0.16% FabArray::setDomainBndry() 41 0.001319 0.001319 0.001319 0.15% Castro::initData() 1 0.001308 0.001308 0.001308 0.15% Castro::enforce_speed_limit() 62 0.001294 0.001294 0.001294 0.15% MLMG::prepareForSolve() 11 0.001276 0.001276 0.001276 0.15% MLCellLinOp::prepareForSolve() 11 0.001199 0.001199 0.001199 0.14% MultiFab::contains_nan() 20 0.001193 0.001193 0.001193 0.14% MLCellLinOp::smooth() 1620 0.001157 0.001157 0.001157 0.13% Castro::subcycle_advance_ctu() 10 0.0009406 0.0009406 0.0009406 0.11% MLCellLinOp::compGrad() 11 0.0009359 0.0009359 0.0009359 0.11% FabArrayBase::getCPC() 1313 0.0008823 0.0008823 0.0008823 0.10% FabArray::FillBoundary() 3974 0.0008169 0.0008169 0.0008169 0.09% FabArrayBase::CPC::define() 454 0.0006895 0.0006895 0.0006895 0.08% FabArrayBase::getFB() 3974 0.0005873 0.0005873 0.0005873 0.07% MLCellLinOp::apply() 1128 0.0004945 0.0004945 0.0004945 0.06% Amr::InitAmr() 1 0.0004722 0.0004722 0.0004722 0.05% Gravity::solve_for_phi() 10 0.0004471 0.0004471 0.0004471 0.05% Gravity::update_max_rhs() 11 0.0004283 0.0004283 0.0004283 0.05% Amr::coarseTimeStep() 10 0.0004255 0.0004255 0.0004255 0.05% CGSolver::sxay() 1566 0.0004192 0.0004192 0.0004192 0.05% MultiFab::Copy() 11 0.0003282 0.0003282 0.0003282 0.04% FillPatchIterator::Initialize 41 0.0003144 0.0003144 0.0003144 0.04% MLCellLinOp::defineBC() 11 0.0003043 0.0003043 0.0003043 0.04% MLCGSolver::ParallelAllReduce 1495 0.0002874 0.0002874 0.0002874 0.03% FabArray::ParallelCopy() 851 0.00027 0.00027 0.00027 0.03% MultiFab::max() 11 0.0002633 0.0002633 0.0002633 0.03% main() 1 0.0002571 0.0002571 0.0002571 0.03% MLCellLinOp::correctionResidual() 486 0.0002369 0.0002369 0.0002369 0.03% MLMG::mgVcycle() 81 0.0002238 0.0002238 0.0002238 0.03% MLMG::MLRhsNormInf() 11 0.0002157 0.0002157 0.0002157 0.02% Castro::construct_new_gravity() 10 0.0002134 0.0002134 0.0002134 0.02% Amr::timeStep() 10 0.0001825 0.0001825 0.0001825 0.02% MLLinOp::defineGrids() 11 0.0001483 0.0001483 0.0001483 0.02% MLMG:computeResOfCorrection() 405 0.000148 0.000148 0.000148 0.02% StateData::checkPoint() 12 0.0001289 0.0001289 0.0001289 0.01% MLMG::mgVcycle_down::0 81 0.0001105 0.0001105 0.0001105 0.01% Castro::create_source_corrector() 10 9.603e-05 9.603e-05 9.603e-05 0.01% MLMG::mgVcycle_down::1 81 9.587e-05 9.587e-05 9.587e-05 0.01% Castro::Castro() 1 9.472e-05 9.472e-05 9.472e-05 0.01% Castro::initialize_advance() 10 9.409e-05 9.409e-05 9.409e-05 0.01% MLMG::mgVcycle_down::2 81 9.211e-05 9.211e-05 9.211e-05 0.01% FabArrayBase::FB::FB() 56 9.031e-05 9.031e-05 9.031e-05 0.01% MLMG::mgVcycle_down::3 81 8.759e-05 8.759e-05 8.759e-05 0.01% Castro::clean_state() 62 8.676e-05 8.676e-05 8.676e-05 0.01% MLMG::mgVcycle_down::4 81 8.584e-05 8.584e-05 8.584e-05 0.01% MLMG::actualBottomSolve() 81 8.276e-05 8.276e-05 8.276e-05 0.01% AmrLevel::checkPoint() 3 7.153e-05 7.153e-05 7.153e-05 0.01% MLMG::solve() 11 6.907e-05 6.907e-05 6.907e-05 0.01% MLMG::mgVcycle_up::4 81 6.889e-05 6.889e-05 6.889e-05 0.01% Castro::initialize_do_advance() 10 6.627e-05 6.627e-05 6.627e-05 0.01% MLMG::oneIter() 81 6.064e-05 6.064e-05 6.064e-05 0.01% MLMG::mgVcycle_up::0 81 5.955e-05 5.955e-05 5.955e-05 0.01% MLMG::mgVcycle_up::3 81 5.696e-05 5.696e-05 5.696e-05 0.01% MLMG::mgVcycle_up::1 81 5.683e-05 5.683e-05 5.683e-05 0.01% MLMG::mgVcycle_up::2 81 5.673e-05 5.673e-05 5.673e-05 0.01% Castro::finalize_advance() 10 5.655e-05 5.655e-05 5.655e-05 0.01% MLCellLinOp::solutionResidual() 92 5.415e-05 5.415e-05 5.415e-05 0.01% StateData::define() 4 4.287e-05 4.287e-05 4.287e-05 0.00% Castro::advance() 10 4.111e-05 4.111e-05 4.111e-05 0.00% Castro::swap_state_time_levels() 10 3.979e-05 3.979e-05 3.979e-05 0.00% Castro::finalize_do_advance() 10 3.932e-05 3.932e-05 3.932e-05 0.00% MLMG::computeResidual() 81 3.829e-05 3.829e-05 3.829e-05 0.00% MLMG::mgVcycle_bottom 81 3.62e-05 3.62e-05 3.62e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.543e-05 3.543e-05 3.543e-05 0.00% Castro::enforce_consistent_e() 1 3.373e-05 3.373e-05 3.373e-05 0.00% Gravity::actual_multilevel_solve() 1 3.198e-05 3.198e-05 3.198e-05 0.00% MLPoisson::define() 11 3.173e-05 3.173e-05 3.173e-05 0.00% FillPatchSingleLevel 41 3.033e-05 3.033e-05 3.033e-05 0.00% makeSFC 55 2.854e-05 2.854e-05 2.854e-05 0.00% Amr::defBaseLevel() 1 2.842e-05 2.842e-05 2.842e-05 0.00% Castro::initMFs() 1 2.741e-05 2.741e-05 2.741e-05 0.00% Amr::writeSmallPlotFile() 1 2.583e-05 2.583e-05 2.583e-05 0.00% MLLinOp::define() 11 2.547e-05 2.547e-05 2.547e-05 0.00% Castro::buildMetrics() 1 2.238e-05 2.238e-05 2.238e-05 0.00% Castro::construct_new_source() 50 2.136e-05 2.136e-05 2.136e-05 0.00% Castro::construct_old_source() 50 2.007e-05 2.007e-05 2.007e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.986e-05 1.986e-05 1.986e-05 0.00% Amr::FinalizeInit() 1 1.971e-05 1.971e-05 1.971e-05 0.00% Castro::do_new_sources() 10 1.901e-05 1.901e-05 1.901e-05 0.00% Castro::do_old_sources() 10 1.739e-05 1.739e-05 1.739e-05 0.00% DistributionMapping::Distribute() 56 1.641e-05 1.641e-05 1.641e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.402e-05 1.402e-05 1.402e-05 0.00% Castro::check_for_nan() 20 1.29e-05 1.29e-05 1.29e-05 0.00% Castro::post_timestep() 10 1.201e-05 1.201e-05 1.201e-05 0.00% Castro::apply_source_to_state() 20 1.2e-05 1.2e-05 1.2e-05 0.00% Castro::construct_old_gravity() 10 1.107e-05 1.107e-05 1.107e-05 0.00% MLMG::computeMLResidual() 11 1.05e-05 1.05e-05 1.05e-05 0.00% MLPoisson::prepareForSolve() 11 1.017e-05 1.017e-05 1.017e-05 0.00% Gravity::swapTimeLevels() 10 9.674e-06 9.674e-06 9.674e-06 0.00% Amr::initSubcycle() 1 8.58e-06 8.58e-06 8.58e-06 0.00% Castro::computeNewDt() 9 7.459e-06 7.459e-06 7.459e-06 0.00% MLMG::getGradSolution() 11 6.887e-06 6.887e-06 6.887e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.622e-06 6.622e-06 6.622e-06 0.00% Gravity::set_mass_offset() 11 5.386e-06 5.386e-06 5.386e-06 0.00% Amr::InitializeInit() 1 4.723e-06 4.723e-06 4.723e-06 0.00% Castro::retry_advance_ctu() 10 4.664e-06 4.664e-06 4.664e-06 0.00% AmrLevel::checkPointPost() 3 4.635e-06 4.635e-06 4.635e-06 0.00% Castro::post_init() 1 3.577e-06 3.577e-06 3.577e-06 0.00% MLMG::MLResNormInf() 11 3.479e-06 3.479e-06 3.479e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.31e-06 3.31e-06 3.31e-06 0.00% Castro::FluxRegCrseInit 10 3.174e-06 3.174e-06 3.174e-06 0.00% Castro::computeInitialDt() 2 2.696e-06 2.696e-06 2.696e-06 0.00% Castro::FluxRegFineAdd() 10 2.682e-06 2.682e-06 2.682e-06 0.00% Amr::init() 1 2.653e-06 2.653e-06 2.653e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.157e-06 2.157e-06 2.157e-06 0.00% AmrLevel::checkPointPre() 3 1.746e-06 1.746e-06 1.746e-06 0.00% Castro::post_regrid() 1 1.389e-06 1.389e-06 1.389e-06 0.00% Amr::initialInit() 1 1.115e-06 1.115e-06 1.115e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8629 0.8629 0.8629 100.00% Amr::coarseTimeStep() 10 0.6997 0.6997 0.6997 81.08% Amr::timeStep() 10 0.597 0.597 0.597 69.19% Castro::advance() 10 0.5892 0.5892 0.5892 68.28% Castro::subcycle_advance_ctu() 10 0.5776 0.5776 0.5776 66.94% Castro::do_advance_ctu() 10 0.5767 0.5767 0.5767 66.83% Gravity::solve_phi_with_mlmg() 11 0.318 0.318 0.318 36.85% Gravity::actual_solve_with_mlmg() 11 0.3075 0.3075 0.3075 35.63% Castro::construct_new_gravity() 10 0.2922 0.2922 0.2922 33.86% MLMG::solve() 11 0.2837 0.2837 0.2837 32.87% Gravity::solve_for_phi() 10 0.2767 0.2767 0.2767 32.07% MLMG::oneIter() 81 0.2685 0.2685 0.2685 31.11% MLMG::mgVcycle() 81 0.2667 0.2667 0.2667 30.91% Castro::construct_ctu_hydro_source() 10 0.2057 0.2057 0.2057 23.84% VisMF::Write(FabArray) 11 0.2033 0.2033 0.2033 23.56% Amr::checkPoint() 3 0.1509 0.1509 0.1509 17.49% AmrLevel::checkPoint() 3 0.1456 0.1456 0.1456 16.88% StateData::checkPoint() 12 0.1456 0.1456 0.1456 16.87% MLCellLinOp::smooth() 1620 0.1359 0.1359 0.1359 15.75% Amr::init() 1 0.1321 0.1321 0.1321 15.31% MLCellLinOp::applyBC() 4379 0.09624 0.09624 0.09624 11.15% MLMG::mgVcycle_bottom 81 0.08216 0.08216 0.08216 9.52% MLMG::actualBottomSolve() 81 0.08213 0.08213 0.08213 9.52% MLCGSolver::bicgstab 81 0.08131 0.08131 0.08131 9.42% MLPoisson::Fsmooth() 3240 0.06322 0.06322 0.06322 7.33% Amr::writePlotFile() 2 0.06071 0.06071 0.06071 7.04% Amr::initialInit() 1 0.0492 0.0492 0.0492 5.70% Castro::clean_state() 62 0.04592 0.04592 0.04592 5.32% Amr::FinalizeInit() 1 0.04493 0.04493 0.04493 5.21% Castro::post_init() 1 0.04359 0.04359 0.04359 5.05% FillPatchIterator::Initialize 41 0.04241 0.04241 0.04241 4.91% Gravity::multilevel_solve_for_new_phi() 1 0.0418 0.0418 0.0418 4.84% Gravity::actual_multilevel_solve() 1 0.04178 0.04178 0.04178 4.84% FillPatchSingleLevel 41 0.04078 0.04078 0.04078 4.73% StateDataPhysBCFunct::() 41 0.03666 0.03666 0.03666 4.25% MLCellLinOp::apply() 1128 0.03646 0.03646 0.03646 4.23% MLMG::mgVcycle_down::0 81 0.03506 0.03506 0.03506 4.06% MLMG::mgVcycle_up::0 81 0.03024 0.03024 0.03024 3.50% StateData::FillBoundary(geom) 328 0.02454 0.02454 0.02454 2.84% MultiFab::Dot() 1100 0.02236 0.02236 0.02236 2.59% MLCellLinOp::correctionResidual() 486 0.02127 0.02127 0.02127 2.46% Castro::computeTemp() 63 0.01966 0.01966 0.01966 2.28% MLPoisson::define() 11 0.01923 0.01923 0.01923 2.23% Castro::initialize_do_advance() 10 0.01911 0.01911 0.01911 2.21% MLMG:computeResOfCorrection() 405 0.01836 0.01836 0.01836 2.13% MLMG::mgVcycle_down::1 81 0.0177 0.0177 0.0177 2.05% MLMG::mgVcycle_down::2 81 0.01734 0.01734 0.01734 2.01% Gravity::get_new_grav_vector() 11 0.01696 0.01696 0.01696 1.97% MLMG::mgVcycle_down::3 81 0.01644 0.01644 0.01644 1.91% Castro::normalize_species() 62 0.01585 0.01585 0.01585 1.84% MLMG::mgVcycle_down::4 81 0.0158 0.0158 0.0158 1.83% FabArray::FillBoundary() 3974 0.01575 0.01575 0.01575 1.83% Castro::construct_old_gravity() 10 0.0151 0.0151 0.0151 1.75% Gravity::get_old_grav_vector() 10 0.01508 0.01508 0.01508 1.75% FillBoundary_nowait() 3974 0.01494 0.01494 0.01494 1.73% CGSolver::sxay() 1566 0.01476 0.01476 0.01476 1.71% FabArray::ParallelCopy() 851 0.01452 0.01452 0.01452 1.68% FabArray::setVal() 1135 0.01439 0.01439 0.01439 1.67% MultiFab::LinComb() 1566 0.01435 0.01435 0.01435 1.66% FabArray::ParallelCopy_nowait() 851 0.01425 0.01425 0.01425 1.65% MLCellLinOp::defineAuxData() 11 0.0136 0.0136 0.0136 1.58% MLCGSolver::ParallelAllReduce 1495 0.01338 0.01338 0.01338 1.55% MLMG::mgVcycle_up::2 81 0.01323 0.01323 0.01323 1.53% MLMG::mgVcycle_up::1 81 0.01307 0.01307 0.01307 1.51% MLMG::addInterpCorrection() 405 0.01305 0.01305 0.01305 1.51% MLMG::mgVcycle_up::4 81 0.01274 0.01274 0.01274 1.48% MLMG::mgVcycle_up::3 81 0.01274 0.01274 0.01274 1.48% amrex::average_down 405 0.01215 0.01215 0.01215 1.41% MLPoisson::Fapply() 1128 0.01178 0.01178 0.01178 1.37% Castro::expand_state() 10 0.01159 0.01159 0.01159 1.34% Castro::do_new_sources() 10 0.01155 0.01155 0.01155 1.34% Castro::initialize_advance() 10 0.01152 0.01152 0.01152 1.33% Castro::do_old_sources() 10 0.01148 0.01148 0.01148 1.33% Gravity::fill_multipole_BCs() 11 0.01032 0.01032 0.01032 1.20% Castro::enforce_min_density() 62 0.009882 0.009882 0.009882 1.15% Castro::post_timestep() 10 0.007614 0.007614 0.007614 0.88% MLCellLinOp::solutionResidual() 92 0.00715 0.00715 0.00715 0.83% MultiFab::Xpay() 578 0.006517 0.006517 0.006517 0.76% Castro::estTimeStep() 21 0.006443 0.006443 0.006443 0.75% MLMG::computeResidual() 81 0.006134 0.006134 0.006134 0.71% Castro::reset_internal_energy(MultiFab) 63 0.005755 0.005755 0.005755 0.67% MLMG::prepareForSolve() 11 0.005596 0.005596 0.005596 0.65% MLCellLinOp::defineBC() 11 0.005364 0.005364 0.005364 0.62% BndryData::define() 11 0.005059 0.005059 0.005059 0.59% Amr::InitializeInit() 1 0.004269 0.004269 0.004269 0.49% Amr::defBaseLevel() 1 0.004264 0.004264 0.004264 0.49% Castro::initData() 1 0.003721 0.003721 0.003721 0.43% Castro::construct_new_source() 50 0.003405 0.003405 0.003405 0.39% Castro::construct_new_gravity_source() 10 0.003384 0.003384 0.003384 0.39% Castro::computeNewDt() 9 0.00332 0.00332 0.00332 0.38% Castro::construct_old_source() 50 0.002901 0.002901 0.002901 0.34% Castro::construct_old_gravity_source() 10 0.002881 0.002881 0.002881 0.33% MLMG::ResNormInf() 92 0.00212 0.00212 0.00212 0.25% Castro::apply_source_to_state() 20 0.001832 0.001832 0.001832 0.21% MultiFab::Saxpy() 20 0.00182 0.00182 0.00182 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001696 0.001696 0.001696 0.20% MultiFab::Add() 81 0.001674 0.001674 0.001674 0.19% Castro::reset_internal_energy(Fab) 504 0.001628 0.001628 0.001628 0.19% FabArrayBase::getCPC() 1313 0.001572 0.001572 0.001572 0.18% MLCellLinOp::setLevelBC() 11 0.001559 0.001559 0.001559 0.18% MLMG::getGradSolution() 11 0.001468 0.001468 0.001468 0.17% MLCellLinOp::compGrad() 11 0.001461 0.001461 0.001461 0.17% FabArray::mult() 43 0.001351 0.001351 0.001351 0.16% FabArray::setDomainBndry() 41 0.001319 0.001319 0.001319 0.15% Castro::enforce_speed_limit() 62 0.001294 0.001294 0.001294 0.15% MLPoisson::prepareForSolve() 11 0.001209 0.001209 0.001209 0.14% Castro::check_for_nan() 20 0.001205 0.001205 0.001205 0.14% MLCellLinOp::prepareForSolve() 11 0.001199 0.001199 0.001199 0.14% MultiFab::contains_nan() 20 0.001193 0.001193 0.001193 0.14% Castro::post_regrid() 1 0.001126 0.001126 0.001126 0.13% MLMG::computeMLResidual() 11 0.001064 0.001064 0.001064 0.12% Gravity::update_max_rhs() 11 0.0008511 0.0008511 0.0008511 0.10% Castro::computeInitialDt() 2 0.0007223 0.0007223 0.0007223 0.08% FabArrayBase::CPC::define() 454 0.0006895 0.0006895 0.0006895 0.08% FabArrayBase::getFB() 3974 0.0006777 0.0006777 0.0006777 0.08% Amr::InitAmr() 1 0.0004808 0.0004808 0.0004808 0.06% Castro::Castro() 1 0.0004604 0.0004604 0.0004604 0.05% Gravity::swapTimeLevels() 10 0.0004413 0.0004413 0.0004413 0.05% MultiFab::Copy() 11 0.0003282 0.0003282 0.0003282 0.04% MLMG::MLResNormInf() 11 0.0002816 0.0002816 0.0002816 0.03% MultiFab::max() 11 0.0002633 0.0002633 0.0002633 0.03% MLLinOp::define() 11 0.0002336 0.0002336 0.0002336 0.03% MLMG::MLRhsNormInf() 11 0.0002157 0.0002157 0.0002157 0.02% MLLinOp::defineGrids() 11 0.0002082 0.0002082 0.0002082 0.02% Castro::buildMetrics() 1 0.0001569 0.0001569 0.0001569 0.02% Castro::create_source_corrector() 10 9.603e-05 9.603e-05 9.603e-05 0.01% FabArrayBase::FB::FB() 56 9.031e-05 9.031e-05 9.031e-05 0.01% Castro::finalize_advance() 10 6.241e-05 6.241e-05 6.241e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.769e-05 5.769e-05 5.769e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.95e-05 4.95e-05 4.95e-05 0.01% makeSFC 55 4.367e-05 4.367e-05 4.367e-05 0.01% StateData::define() 4 4.287e-05 4.287e-05 4.287e-05 0.00% Castro::swap_state_time_levels() 10 3.979e-05 3.979e-05 3.979e-05 0.00% Castro::finalize_do_advance() 10 3.932e-05 3.932e-05 3.932e-05 0.00% Castro::enforce_consistent_e() 1 3.373e-05 3.373e-05 3.373e-05 0.00% Castro::initMFs() 1 2.741e-05 2.741e-05 2.741e-05 0.00% Amr::writeSmallPlotFile() 1 2.583e-05 2.583e-05 2.583e-05 0.00% DistributionMapping::Distribute() 56 1.641e-05 1.641e-05 1.641e-05 0.00% Amr::initSubcycle() 1 8.58e-06 8.58e-06 8.58e-06 0.00% Gravity::set_mass_offset() 11 5.386e-06 5.386e-06 5.386e-06 0.00% Castro::retry_advance_ctu() 10 4.664e-06 4.664e-06 4.664e-06 0.00% AmrLevel::checkPointPost() 3 4.635e-06 4.635e-06 4.635e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.59e-06 4.59e-06 4.59e-06 0.00% Castro::FluxRegCrseInit 10 3.174e-06 3.174e-06 3.174e-06 0.00% Castro::FluxRegFineAdd() 10 2.682e-06 2.682e-06 2.682e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.157e-06 2.157e-06 2.157e-06 0.00% AmrLevel::checkPointPre() 3 1.746e-06 1.746e-06 1.746e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-1-gfb0b31e1439b) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-1-gfb0b31e1439b) initialized Starting run at 08:34:15 UTC on 2022-09-06. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09-1-gfb0b31e14 Microphysics git describe: 22.08-10-g65622313 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.45732442 Restart time = 0.048375087 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052517701 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049949758 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.050758556 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.050284651 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.076678132 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031822583 seconds Ending run at 08:34:15 UTC on 2022-09-06. Run time = 0.361379494 Run time without initialization = 0.31239671 Average number of zones advanced per microsecond: 4.196 Average number of zones advanced per microsecond per rank: 4.196 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3614 ... 0.3614 ... 0.3614 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0837 0.0837 0.0837 23.17% VisMF::Read() 3 0.04033 0.04033 0.04033 11.16% MLCellLinOp::applyBC() 1946 0.03498 0.03498 0.03498 9.68% VisMF::Write(FabArray) 1 0.03026 0.03026 0.03026 8.37% MLPoisson::Fsmooth() 1440 0.02742 0.02742 0.02742 7.59% StateData::FillBoundary(geom) 160 0.01194 0.01194 0.01194 3.30% MLCGSolver::bicgstab 36 0.01027 0.01027 0.01027 2.84% MultiFab::Dot() 484 0.009613 0.009613 0.009613 2.66% FabArray::setVal() 537 0.00678 0.00678 0.00678 1.88% Castro::normalize_species() 30 0.006407 0.006407 0.006407 1.77% MLCellLinOp::defineAuxData() 6 0.006391 0.006391 0.006391 1.77% FillBoundary_nowait() 1766 0.006315 0.006315 0.006315 1.75% MultiFab::LinComb() 690 0.006161 0.006161 0.006161 1.70% FabArray::ParallelCopy_nowait() 380 0.006018 0.006018 0.006018 1.67% Castro::enforce_min_density() 30 0.005795 0.005795 0.005795 1.60% Castro::computeTemp() 30 0.005683 0.005683 0.005683 1.57% Gravity::fill_multipole_BCs() 6 0.005647 0.005647 0.005647 1.56% StateDataPhysBCFunct::() 20 0.005241 0.005241 0.005241 1.45% MLPoisson::Fapply() 500 0.005079 0.005079 0.005079 1.41% Amr::restart() 1 0.003813 0.003813 0.003813 1.06% MLMG::addInterpCorrection() 180 0.003403 0.003403 0.003403 0.94% Castro::estTimeStep() 10 0.003153 0.003153 0.003153 0.87% amrex::average_down 180 0.003003 0.003003 0.003003 0.83% MultiFab::Xpay() 258 0.002875 0.002875 0.002875 0.80% BndryData::define() 6 0.002154 0.002154 0.002154 0.60% Castro::reset_internal_energy(MultiFab) 30 0.002142 0.002142 0.002142 0.59% Castro::do_advance_ctu() 5 0.00206 0.00206 0.00206 0.57% Castro::construct_new_gravity_source() 5 0.001667 0.001667 0.001667 0.46% Amr::writePlotFile() 1 0.001643 0.001643 0.001643 0.45% Castro::construct_old_gravity_source() 5 0.001402 0.001402 0.001402 0.39% Castro::enforce_speed_limit() 30 0.00102 0.00102 0.00102 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009225 0.0009225 0.0009225 0.26% MLMG::ResNormInf() 42 0.0009224 0.0009224 0.0009224 0.26% MultiFab::Saxpy() 10 0.0009163 0.0009163 0.0009163 0.25% Gravity::get_old_grav_vector() 5 0.0009065 0.0009065 0.0009065 0.25% Castro::expand_state() 5 0.0008725 0.0008725 0.0008725 0.24% Gravity::get_new_grav_vector() 5 0.0008696 0.0008696 0.0008696 0.24% MLCellLinOp::setLevelBC() 6 0.0008287 0.0008287 0.0008287 0.23% Castro::reset_internal_energy(Fab) 240 0.0007798 0.0007798 0.0007798 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007686 0.0007686 0.0007686 0.21% MultiFab::Add() 36 0.0007221 0.0007221 0.0007221 0.20% FabArray::mult() 22 0.0006658 0.0006658 0.0006658 0.18% MLMG::prepareForSolve() 6 0.000665 0.000665 0.000665 0.18% FabArray::setDomainBndry() 20 0.0006443 0.0006443 0.0006443 0.18% MLCellLinOp::prepareForSolve() 6 0.0006376 0.0006376 0.0006376 0.18% MultiFab::contains_nan() 10 0.0005825 0.0005825 0.0005825 0.16% MLCellLinOp::smooth() 720 0.0005236 0.0005236 0.0005236 0.14% MLCellLinOp::compGrad() 6 0.0004878 0.0004878 0.0004878 0.13% Amr::InitAmr() 1 0.0004295 0.0004295 0.0004295 0.12% FabArrayBase::CPC::define() 244 0.0003955 0.0003955 0.0003955 0.11% FabArray::FillBoundary() 1766 0.0003801 0.0003801 0.0003801 0.11% FabArrayBase::getCPC() 632 0.0003682 0.0003682 0.0003682 0.10% Gravity::update_max_rhs() 6 0.0002685 0.0002685 0.0002685 0.07% FabArrayBase::getFB() 1766 0.0002537 0.0002537 0.0002537 0.07% main() 1 0.0002495 0.0002495 0.0002495 0.07% Gravity::solve_for_phi() 5 0.0002327 0.0002327 0.0002327 0.06% MLCellLinOp::apply() 500 0.0002188 0.0002188 0.0002188 0.06% MultiFab::Copy() 6 0.0001783 0.0001783 0.0001783 0.05% CGSolver::sxay() 690 0.0001738 0.0001738 0.0001738 0.05% Amr::coarseTimeStep() 5 0.0001556 0.0001556 0.0001556 0.04% MLCellLinOp::defineBC() 6 0.0001494 0.0001494 0.0001494 0.04% MultiFab::max() 6 0.0001431 0.0001431 0.0001431 0.04% FillPatchIterator::Initialize 20 0.0001411 0.0001411 0.0001411 0.04% FabArray::ParallelCopy() 380 0.0001256 0.0001256 0.0001256 0.03% MLCGSolver::ParallelAllReduce 659 0.000123 0.000123 0.000123 0.03% Castro::construct_new_gravity() 5 0.0001176 0.0001176 0.0001176 0.03% Castro::subcycle_advance_ctu() 5 0.0001146 0.0001146 0.0001146 0.03% MLMG::MLRhsNormInf() 6 0.0001127 0.0001127 0.0001127 0.03% MLMG::mgVcycle() 36 0.0001077 0.0001077 0.0001077 0.03% MLCellLinOp::correctionResidual() 216 0.000102 0.000102 0.000102 0.03% Castro::create_source_corrector() 5 9.396e-05 9.396e-05 9.396e-05 0.03% Amr::timeStep() 5 8.966e-05 8.966e-05 8.966e-05 0.02% AmrLevel::restart() 1 7.8e-05 7.8e-05 7.8e-05 0.02% StateData::restartDoit() 4 7.615e-05 7.615e-05 7.615e-05 0.02% MLMG:computeResOfCorrection() 180 7.104e-05 7.104e-05 7.104e-05 0.02% MLLinOp::defineGrids() 6 7.078e-05 7.078e-05 7.078e-05 0.02% FabArrayBase::FB::FB() 26 5.661e-05 5.661e-05 5.661e-05 0.02% Castro::finalize_advance() 5 5.415e-05 5.415e-05 5.415e-05 0.01% Castro::construct_new_source() 25 5.16e-05 5.16e-05 5.16e-05 0.01% Castro::initialize_advance() 5 4.937e-05 4.937e-05 4.937e-05 0.01% MLMG::mgVcycle_down::0 36 4.632e-05 4.632e-05 4.632e-05 0.01% Castro::advance() 5 4.582e-05 4.582e-05 4.582e-05 0.01% Castro::construct_old_source() 25 4.579e-05 4.579e-05 4.579e-05 0.01% MLMG::mgVcycle_down::1 36 4.334e-05 4.334e-05 4.334e-05 0.01% MLMG::mgVcycle_down::2 36 4.304e-05 4.304e-05 4.304e-05 0.01% Castro::clean_state() 30 4.116e-05 4.116e-05 4.116e-05 0.01% MLMG::mgVcycle_down::4 36 3.844e-05 3.844e-05 3.844e-05 0.01% MLMG::mgVcycle_down::3 36 3.75e-05 3.75e-05 3.75e-05 0.01% MLMG::actualBottomSolve() 36 3.556e-05 3.556e-05 3.556e-05 0.01% Castro::buildMetrics() 1 3.337e-05 3.337e-05 3.337e-05 0.01% MLMG::mgVcycle_up::4 36 3.26e-05 3.26e-05 3.26e-05 0.01% Castro::initialize_do_advance() 5 3.177e-05 3.177e-05 3.177e-05 0.01% Gravity::actual_multilevel_solve() 1 3.027e-05 3.027e-05 3.027e-05 0.01% MLMG::solve() 6 2.985e-05 2.985e-05 2.985e-05 0.01% Castro::initMFs() 1 2.934e-05 2.934e-05 2.934e-05 0.01% Castro::post_restart() 1 2.884e-05 2.884e-05 2.884e-05 0.01% Castro::swap_state_time_levels() 5 2.787e-05 2.787e-05 2.787e-05 0.01% MLMG::oneIter() 36 2.725e-05 2.725e-05 2.725e-05 0.01% Amr::writeSmallPlotFile() 1 2.667e-05 2.667e-05 2.667e-05 0.01% MLMG::mgVcycle_up::0 36 2.633e-05 2.633e-05 2.633e-05 0.01% MLMG::mgVcycle_up::3 36 2.574e-05 2.574e-05 2.574e-05 0.01% MLMG::mgVcycle_up::2 36 2.439e-05 2.439e-05 2.439e-05 0.01% MLMG::mgVcycle_up::1 36 2.364e-05 2.364e-05 2.364e-05 0.01% MLPoisson::define() 6 2.336e-05 2.336e-05 2.336e-05 0.01% MLCellLinOp::solutionResidual() 42 2.302e-05 2.302e-05 2.302e-05 0.01% MLLinOp::define() 6 2.013e-05 2.013e-05 2.013e-05 0.01% Castro::finalize_do_advance() 5 1.81e-05 1.81e-05 1.81e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.745e-05 1.745e-05 1.745e-05 0.00% MLMG::computeResidual() 36 1.719e-05 1.719e-05 1.719e-05 0.00% MLMG::mgVcycle_bottom 36 1.586e-05 1.586e-05 1.586e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.441e-05 1.441e-05 1.441e-05 0.00% FillPatchSingleLevel 20 1.397e-05 1.397e-05 1.397e-05 0.00% makeSFC 30 1.363e-05 1.363e-05 1.363e-05 0.00% Amr::initSubcycle() 1 9.75e-06 9.75e-06 9.75e-06 0.00% Castro::do_new_sources() 5 8.942e-06 8.942e-06 8.942e-06 0.00% DistributionMapping::Distribute() 31 8.626e-06 8.626e-06 8.626e-06 0.00% Castro::do_old_sources() 5 7.952e-06 7.952e-06 7.952e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.171e-06 7.171e-06 7.171e-06 0.00% Castro::check_for_nan() 10 6.774e-06 6.774e-06 6.774e-06 0.00% Castro::construct_old_gravity() 5 5.728e-06 5.728e-06 5.728e-06 0.00% Castro::apply_source_to_state() 10 5.304e-06 5.304e-06 5.304e-06 0.00% Castro::post_timestep() 5 5.191e-06 5.191e-06 5.191e-06 0.00% MLPoisson::prepareForSolve() 6 5.128e-06 5.128e-06 5.128e-06 0.00% MLMG::computeMLResidual() 6 4.638e-06 4.638e-06 4.638e-06 0.00% Gravity::swapTimeLevels() 5 4.108e-06 4.108e-06 4.108e-06 0.00% Castro::computeNewDt() 5 3.702e-06 3.702e-06 3.702e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.159e-06 3.159e-06 3.159e-06 0.00% MLMG::getGradSolution() 6 3.095e-06 3.095e-06 3.095e-06 0.00% Gravity::set_mass_offset() 6 2.096e-06 2.096e-06 2.096e-06 0.00% MLMG::MLResNormInf() 6 2.05e-06 2.05e-06 2.05e-06 0.00% Castro::retry_advance_ctu() 5 1.94e-06 1.94e-06 1.94e-06 0.00% Castro::FluxRegCrseInit 5 1.773e-06 1.773e-06 1.773e-06 0.00% Amr::init() 1 1.276e-06 1.276e-06 1.276e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.273e-06 1.273e-06 1.273e-06 0.00% Castro::FluxRegFineAdd() 5 1.261e-06 1.261e-06 1.261e-06 0.00% AmrLevel::AmrLevel() 1 9.41e-07 9.41e-07 9.41e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3614 0.3614 0.3614 100.00% Amr::coarseTimeStep() 5 0.2803 0.2803 0.2803 77.57% Amr::timeStep() 5 0.278 0.278 0.278 76.92% Castro::advance() 5 0.2742 0.2742 0.2742 75.88% Castro::subcycle_advance_ctu() 5 0.268 0.268 0.268 74.17% Castro::do_advance_ctu() 5 0.2679 0.2679 0.2679 74.14% Castro::construct_new_gravity() 5 0.1459 0.1459 0.1459 40.38% Gravity::solve_phi_with_mlmg() 6 0.1418 0.1418 0.1418 39.23% Gravity::solve_for_phi() 5 0.1383 0.1383 0.1383 38.26% Gravity::actual_solve_with_mlmg() 6 0.136 0.136 0.136 37.63% MLMG::solve() 6 0.1235 0.1235 0.1235 34.17% MLMG::oneIter() 36 0.1163 0.1163 0.1163 32.18% MLMG::mgVcycle() 36 0.1155 0.1155 0.1155 31.97% Castro::construct_ctu_hydro_source() 5 0.08374 0.08374 0.08374 23.17% MLCellLinOp::smooth() 720 0.05903 0.05903 0.05903 16.33% Amr::init() 1 0.04843 0.04843 0.04843 13.40% Amr::restart() 1 0.04843 0.04843 0.04843 13.40% MLCellLinOp::applyBC() 1946 0.04198 0.04198 0.04198 11.62% AmrLevel::restart() 1 0.04055 0.04055 0.04055 11.22% StateData::restartDoit() 4 0.04046 0.04046 0.04046 11.20% VisMF::Read() 3 0.04033 0.04033 0.04033 11.16% MLMG::mgVcycle_bottom 36 0.03536 0.03536 0.03536 9.78% MLMG::actualBottomSolve() 36 0.03534 0.03534 0.03534 9.78% MLCGSolver::bicgstab 36 0.03499 0.03499 0.03499 9.68% Amr::writePlotFile() 1 0.0319 0.0319 0.0319 8.83% VisMF::Write(FabArray) 1 0.03026 0.03026 0.03026 8.37% MLPoisson::Fsmooth() 1440 0.02742 0.02742 0.02742 7.59% Castro::clean_state() 30 0.02187 0.02187 0.02187 6.05% FillPatchIterator::Initialize 20 0.01999 0.01999 0.01999 5.53% FillPatchSingleLevel 20 0.0192 0.0192 0.0192 5.31% StateDataPhysBCFunct::() 20 0.01718 0.01718 0.01718 4.75% MLCellLinOp::apply() 500 0.01592 0.01592 0.01592 4.40% MLMG::mgVcycle_down::0 36 0.01543 0.01543 0.01543 4.27% MLMG::mgVcycle_up::0 36 0.0132 0.0132 0.0132 3.65% StateData::FillBoundary(geom) 160 0.01194 0.01194 0.01194 3.30% MLPoisson::define() 6 0.01014 0.01014 0.01014 2.81% MultiFab::Dot() 484 0.009613 0.009613 0.009613 2.66% Castro::initialize_do_advance() 5 0.009453 0.009453 0.009453 2.62% MLCellLinOp::correctionResidual() 216 0.009266 0.009266 0.009266 2.56% Castro::computeTemp() 30 0.008604 0.008604 0.008604 2.38% MLMG:computeResOfCorrection() 180 0.007994 0.007994 0.007994 2.21% MLMG::mgVcycle_down::1 36 0.007699 0.007699 0.007699 2.13% Castro::construct_old_gravity() 5 0.00767 0.00767 0.00767 2.12% Gravity::get_old_grav_vector() 5 0.007664 0.007664 0.007664 2.12% Gravity::get_new_grav_vector() 5 0.007537 0.007537 0.007537 2.09% MLMG::mgVcycle_down::2 36 0.007485 0.007485 0.007485 2.07% MLCellLinOp::defineAuxData() 6 0.007157 0.007157 0.007157 1.98% MLMG::mgVcycle_down::3 36 0.0071 0.0071 0.0071 1.96% FabArray::FillBoundary() 1766 0.007005 0.007005 0.007005 1.94% MLMG::mgVcycle_down::4 36 0.00681 0.00681 0.00681 1.88% FabArray::setVal() 537 0.00678 0.00678 0.00678 1.88% Castro::do_new_sources() 5 0.006743 0.006743 0.006743 1.87% FillBoundary_nowait() 1766 0.006625 0.006625 0.006625 1.83% FabArray::ParallelCopy() 380 0.006523 0.006523 0.006523 1.80% Castro::normalize_species() 30 0.006407 0.006407 0.006407 1.77% FabArray::ParallelCopy_nowait() 380 0.006397 0.006397 0.006397 1.77% CGSolver::sxay() 690 0.006334 0.006334 0.006334 1.75% MultiFab::LinComb() 690 0.006161 0.006161 0.006161 1.70% Castro::initialize_advance() 5 0.006072 0.006072 0.006072 1.68% Castro::enforce_min_density() 30 0.005795 0.005795 0.005795 1.60% MLCGSolver::ParallelAllReduce 659 0.005768 0.005768 0.005768 1.60% MLMG::mgVcycle_up::2 36 0.005746 0.005746 0.005746 1.59% MLMG::mgVcycle_up::1 36 0.005665 0.005665 0.005665 1.57% MLMG::addInterpCorrection() 180 0.005652 0.005652 0.005652 1.56% Gravity::fill_multipole_BCs() 6 0.005647 0.005647 0.005647 1.56% MLMG::mgVcycle_up::4 36 0.005474 0.005474 0.005474 1.51% Castro::expand_state() 5 0.005464 0.005464 0.005464 1.51% MLMG::mgVcycle_up::3 36 0.005462 0.005462 0.005462 1.51% amrex::average_down 180 0.005267 0.005267 0.005267 1.46% MLPoisson::Fapply() 500 0.005079 0.005079 0.005079 1.41% Castro::do_old_sources() 5 0.005044 0.005044 0.005044 1.40% Castro::post_restart() 1 0.003889 0.003889 0.003889 1.08% Gravity::multilevel_solve_for_new_phi() 1 0.003766 0.003766 0.003766 1.04% Gravity::actual_multilevel_solve() 1 0.003749 0.003749 0.003749 1.04% Castro::post_timestep() 5 0.003669 0.003669 0.003669 1.02% MLCellLinOp::solutionResidual() 42 0.003215 0.003215 0.003215 0.89% Castro::estTimeStep() 10 0.003153 0.003153 0.003153 0.87% Castro::reset_internal_energy(MultiFab) 30 0.002922 0.002922 0.002922 0.81% MLMG::prepareForSolve() 6 0.002899 0.002899 0.002899 0.80% MultiFab::Xpay() 258 0.002875 0.002875 0.002875 0.80% MLCellLinOp::defineBC() 6 0.002844 0.002844 0.002844 0.79% BndryData::define() 6 0.002695 0.002695 0.002695 0.75% MLMG::computeResidual() 36 0.002667 0.002667 0.002667 0.74% Castro::computeNewDt() 5 0.002196 0.002196 0.002196 0.61% Castro::construct_new_source() 25 0.001718 0.001718 0.001718 0.48% Castro::construct_new_gravity_source() 5 0.001667 0.001667 0.001667 0.46% Castro::construct_old_source() 25 0.001447 0.001447 0.001447 0.40% Castro::construct_old_gravity_source() 5 0.001402 0.001402 0.001402 0.39% Castro::enforce_speed_limit() 30 0.00102 0.00102 0.00102 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009225 0.0009225 0.0009225 0.26% MLMG::ResNormInf() 42 0.0009224 0.0009224 0.0009224 0.26% Castro::apply_source_to_state() 10 0.0009216 0.0009216 0.0009216 0.25% MultiFab::Saxpy() 10 0.0009163 0.0009163 0.0009163 0.25% MLCellLinOp::setLevelBC() 6 0.0008287 0.0008287 0.0008287 0.23% Castro::reset_internal_energy(Fab) 240 0.0007798 0.0007798 0.0007798 0.22% MLMG::getGradSolution() 6 0.0007638 0.0007638 0.0007638 0.21% FabArrayBase::getCPC() 632 0.0007637 0.0007637 0.0007637 0.21% MLCellLinOp::compGrad() 6 0.0007607 0.0007607 0.0007607 0.21% MultiFab::Add() 36 0.0007221 0.0007221 0.0007221 0.20% FabArray::mult() 22 0.0006658 0.0006658 0.0006658 0.18% FabArray::setDomainBndry() 20 0.0006443 0.0006443 0.0006443 0.18% MLPoisson::prepareForSolve() 6 0.0006427 0.0006427 0.0006427 0.18% MLCellLinOp::prepareForSolve() 6 0.0006376 0.0006376 0.0006376 0.18% Castro::check_for_nan() 10 0.0005893 0.0005893 0.0005893 0.16% MultiFab::contains_nan() 10 0.0005825 0.0005825 0.0005825 0.16% MLMG::computeMLResidual() 6 0.0005696 0.0005696 0.0005696 0.16% Gravity::update_max_rhs() 6 0.000497 0.000497 0.000497 0.14% Amr::InitAmr() 1 0.0004392 0.0004392 0.0004392 0.12% FabArrayBase::CPC::define() 244 0.0003955 0.0003955 0.0003955 0.11% FabArrayBase::getFB() 1766 0.0003103 0.0003103 0.0003103 0.09% Gravity::swapTimeLevels() 5 0.0002279 0.0002279 0.0002279 0.06% MultiFab::Copy() 6 0.0001783 0.0001783 0.0001783 0.05% Castro::buildMetrics() 1 0.0001557 0.0001557 0.0001557 0.04% MLMG::MLResNormInf() 6 0.0001478 0.0001478 0.0001478 0.04% MultiFab::max() 6 0.0001431 0.0001431 0.0001431 0.04% MLLinOp::define() 6 0.0001203 0.0001203 0.0001203 0.03% MLMG::MLRhsNormInf() 6 0.0001127 0.0001127 0.0001127 0.03% MLLinOp::defineGrids() 6 0.0001002 0.0001002 0.0001002 0.03% Castro::create_source_corrector() 5 9.396e-05 9.396e-05 9.396e-05 0.03% Castro::finalize_advance() 5 5.718e-05 5.718e-05 5.718e-05 0.02% FabArrayBase::FB::FB() 26 5.661e-05 5.661e-05 5.661e-05 0.02% Castro::initMFs() 1 2.934e-05 2.934e-05 2.934e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.812e-05 2.812e-05 2.812e-05 0.01% Castro::swap_state_time_levels() 5 2.787e-05 2.787e-05 2.787e-05 0.01% Amr::writeSmallPlotFile() 1 2.667e-05 2.667e-05 2.667e-05 0.01% makeSFC 30 2.095e-05 2.095e-05 2.095e-05 0.01% Castro::finalize_do_advance() 5 1.81e-05 1.81e-05 1.81e-05 0.01% Amr::initSubcycle() 1 9.75e-06 9.75e-06 9.75e-06 0.00% DistributionMapping::Distribute() 31 8.626e-06 8.626e-06 8.626e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.463e-06 4.463e-06 4.463e-06 0.00% Gravity::set_mass_offset() 6 2.096e-06 2.096e-06 2.096e-06 0.00% Castro::retry_advance_ctu() 5 1.94e-06 1.94e-06 1.94e-06 0.00% Castro::FluxRegCrseInit 5 1.773e-06 1.773e-06 1.773e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.273e-06 1.273e-06 1.273e-06 0.00% Castro::FluxRegFineAdd() 5 1.261e-06 1.261e-06 1.261e-06 0.00% AmrLevel::AmrLevel() 1 9.41e-07 9.41e-07 9.41e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-1-gfb0b31e1439b) finalized