Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-9-g67384701a808) initialized Starting run at 08:30:54 UTC on 2022-09-12. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09-9-g67384701a Microphysics git describe: 22.08-12-g91211ba4 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.051061452 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.02936625 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.050702244 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051675388 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.050453765 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.049364304 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.068184097 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.047016336 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.073922373 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.059531934 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.060281918 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.051705401 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.055281208 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.047340401 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.02915563 seconds Ending run at 08:30:55 UTC on 2022-09-12. Run time = 0.824945999 Run time without initialization = 0.695244897 Average number of zones advanced per microsecond: 3.771 Average number of zones advanced per microsecond per rank: 3.771 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.825 ... 0.825 ... 0.825 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.1967 0.1967 0.1967 23.84% Castro::construct_ctu_hydro_source() 10 0.1887 0.1887 0.1887 22.87% MLCellLinOp::applyBC() 4379 0.07832 0.07832 0.07832 9.49% MLPoisson::Fsmooth() 3240 0.06252 0.06252 0.06252 7.58% MLCGSolver::bicgstab 81 0.02329 0.02329 0.02329 2.82% StateData::FillBoundary(geom) 328 0.02322 0.02322 0.02322 2.81% MultiFab::Dot() 1100 0.02184 0.02184 0.02184 2.65% StateDataPhysBCFunct::() 41 0.01801 0.01801 0.01801 2.18% FillBoundary_nowait() 3974 0.01402 0.01402 0.01402 1.70% MultiFab::LinComb() 1566 0.01398 0.01398 0.01398 1.69% FabArray::setVal() 1135 0.01395 0.01395 0.01395 1.69% Castro::computeTemp() 63 0.0137 0.0137 0.0137 1.66% Castro::normalize_species() 62 0.01308 0.01308 0.01308 1.59% FabArray::ParallelCopy_nowait() 851 0.01283 0.01283 0.01283 1.55% MLPoisson::Fapply() 1128 0.01148 0.01148 0.01148 1.39% MLCellLinOp::defineAuxData() 11 0.01131 0.01131 0.01131 1.37% Castro::enforce_min_density() 62 0.009785 0.009785 0.009785 1.19% Gravity::fill_multipole_BCs() 11 0.008186 0.008186 0.008186 0.99% MLMG::addInterpCorrection() 405 0.007563 0.007563 0.007563 0.92% amrex::average_down 405 0.006629 0.006629 0.006629 0.80% MultiFab::Xpay() 578 0.006393 0.006393 0.006393 0.77% Castro::estTimeStep() 21 0.004965 0.004965 0.004965 0.60% Amr::checkPoint() 3 0.004699 0.004699 0.004699 0.57% Castro::do_advance_ctu() 10 0.004364 0.004364 0.004364 0.53% Castro::reset_internal_energy(MultiFab) 63 0.004294 0.004294 0.004294 0.52% BndryData::define() 11 0.003751 0.003751 0.003751 0.45% Castro::construct_new_gravity_source() 10 0.003117 0.003117 0.003117 0.38% Amr::writePlotFile() 2 0.002585 0.002585 0.002585 0.31% Castro::construct_old_gravity_source() 10 0.002401 0.002401 0.002401 0.29% MLMG::ResNormInf() 92 0.002053 0.002053 0.002053 0.25% Gravity::get_new_grav_vector() 11 0.001923 0.001923 0.001923 0.23% MultiFab::Saxpy() 20 0.001806 0.001806 0.001806 0.22% Gravity::get_old_grav_vector() 10 0.00173 0.00173 0.00173 0.21% Castro::expand_state() 10 0.001726 0.001726 0.001726 0.21% Castro::enforce_speed_limit() 62 0.001705 0.001705 0.001705 0.21% MultiFab::Add() 81 0.001634 0.001634 0.001634 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001605 0.001605 0.001605 0.19% MLCellLinOp::setLevelBC() 11 0.001516 0.001516 0.001516 0.18% Castro::reset_internal_energy(Fab) 504 0.001441 0.001441 0.001441 0.17% Gravity::actual_solve_with_mlmg() 11 0.001441 0.001441 0.001441 0.17% FabArray::mult() 43 0.001324 0.001324 0.001324 0.16% FabArray::setDomainBndry() 41 0.001281 0.001281 0.001281 0.16% MLMG::prepareForSolve() 11 0.001221 0.001221 0.001221 0.15% Castro::initData() 1 0.00122 0.00122 0.00122 0.15% MultiFab::contains_nan() 20 0.001173 0.001173 0.001173 0.14% MLCellLinOp::prepareForSolve() 11 0.001158 0.001158 0.001158 0.14% MLCellLinOp::smooth() 1620 0.001129 0.001129 0.001129 0.14% MLCellLinOp::compGrad() 11 0.0009119 0.0009119 0.0009119 0.11% FabArray::FillBoundary() 3974 0.0008539 0.0008539 0.0008539 0.10% FabArrayBase::getCPC() 1313 0.000732 0.000732 0.000732 0.09% FabArrayBase::CPC::define() 454 0.0006663 0.0006663 0.0006663 0.08% FabArrayBase::getFB() 3974 0.0005886 0.0005886 0.0005886 0.07% Amr::InitAmr() 1 0.0004594 0.0004594 0.0004594 0.06% Gravity::solve_for_phi() 10 0.0004456 0.0004456 0.0004456 0.05% MLCellLinOp::apply() 1128 0.0004331 0.0004331 0.0004331 0.05% Gravity::update_max_rhs() 11 0.0004069 0.0004069 0.0004069 0.05% CGSolver::sxay() 1566 0.0003651 0.0003651 0.0003651 0.04% Amr::coarseTimeStep() 10 0.0003398 0.0003398 0.0003398 0.04% MultiFab::Copy() 11 0.0003256 0.0003256 0.0003256 0.04% MLCGSolver::ParallelAllReduce 1495 0.000294 0.000294 0.000294 0.04% FillPatchIterator::Initialize 41 0.0002917 0.0002917 0.0002917 0.04% MLCellLinOp::defineBC() 11 0.0002799 0.0002799 0.0002799 0.03% main() 1 0.0002743 0.0002743 0.0002743 0.03% FabArray::ParallelCopy() 851 0.0002598 0.0002598 0.0002598 0.03% MultiFab::max() 11 0.0002542 0.0002542 0.0002542 0.03% Castro::subcycle_advance_ctu() 10 0.000242 0.000242 0.000242 0.03% MLCellLinOp::correctionResidual() 486 0.0002234 0.0002234 0.0002234 0.03% Castro::construct_new_gravity() 10 0.0002175 0.0002175 0.0002175 0.03% MLMG::MLRhsNormInf() 11 0.0002145 0.0002145 0.0002145 0.03% MLMG::mgVcycle() 81 0.0002036 0.0002036 0.0002036 0.02% MLLinOp::defineGrids() 11 0.000159 0.000159 0.000159 0.02% Amr::timeStep() 10 0.0001577 0.0001577 0.0001577 0.02% MLMG:computeResOfCorrection() 405 0.0001506 0.0001506 0.0001506 0.02% StateData::checkPoint() 12 0.0001313 0.0001313 0.0001313 0.02% MLMG::mgVcycle_down::0 81 0.0001064 0.0001064 0.0001064 0.01% Castro::Castro() 1 9.525e-05 9.525e-05 9.525e-05 0.01% MLMG::mgVcycle_down::1 81 8.965e-05 8.965e-05 8.965e-05 0.01% MLMG::mgVcycle_down::2 81 8.597e-05 8.597e-05 8.597e-05 0.01% FabArrayBase::FB::FB() 56 8.577e-05 8.577e-05 8.577e-05 0.01% Castro::initialize_advance() 10 8.488e-05 8.488e-05 8.488e-05 0.01% Castro::clean_state() 62 8.175e-05 8.175e-05 8.175e-05 0.01% MLMG::actualBottomSolve() 81 8.091e-05 8.091e-05 8.091e-05 0.01% MLMG::mgVcycle_down::3 81 7.95e-05 7.95e-05 7.95e-05 0.01% MLMG::mgVcycle_down::4 81 7.728e-05 7.728e-05 7.728e-05 0.01% AmrLevel::checkPoint() 3 7.1e-05 7.1e-05 7.1e-05 0.01% MLMG::mgVcycle_up::4 81 7.016e-05 7.016e-05 7.016e-05 0.01% MLMG::solve() 11 6.906e-05 6.906e-05 6.906e-05 0.01% Castro::finalize_advance() 10 6.397e-05 6.397e-05 6.397e-05 0.01% Castro::initialize_do_advance() 10 6.132e-05 6.132e-05 6.132e-05 0.01% MLMG::mgVcycle_up::0 81 5.883e-05 5.883e-05 5.883e-05 0.01% MLMG::oneIter() 81 5.849e-05 5.849e-05 5.849e-05 0.01% MLMG::mgVcycle_up::1 81 5.72e-05 5.72e-05 5.72e-05 0.01% MLMG::mgVcycle_up::3 81 5.682e-05 5.682e-05 5.682e-05 0.01% MLMG::mgVcycle_up::2 81 5.475e-05 5.475e-05 5.475e-05 0.01% MLCellLinOp::solutionResidual() 92 5.22e-05 5.22e-05 5.22e-05 0.01% StateData::define() 4 4.402e-05 4.402e-05 4.402e-05 0.01% Castro::swap_state_time_levels() 10 4.136e-05 4.136e-05 4.136e-05 0.01% Castro::advance() 10 3.999e-05 3.999e-05 3.999e-05 0.00% MLMG::computeResidual() 81 3.636e-05 3.636e-05 3.636e-05 0.00% Castro::finalize_do_advance() 10 3.438e-05 3.438e-05 3.438e-05 0.00% Castro::enforce_consistent_e() 1 3.299e-05 3.299e-05 3.299e-05 0.00% MLMG::mgVcycle_bottom 81 3.233e-05 3.233e-05 3.233e-05 0.00% Gravity::actual_multilevel_solve() 1 3.149e-05 3.149e-05 3.149e-05 0.00% MLPoisson::define() 11 3.007e-05 3.007e-05 3.007e-05 0.00% Amr::defBaseLevel() 1 2.822e-05 2.822e-05 2.822e-05 0.00% FillPatchSingleLevel 41 2.802e-05 2.802e-05 2.802e-05 0.00% Castro::initMFs() 1 2.654e-05 2.654e-05 2.654e-05 0.00% makeSFC 55 2.598e-05 2.598e-05 2.598e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.593e-05 2.593e-05 2.593e-05 0.00% Amr::writeSmallPlotFile() 1 2.568e-05 2.568e-05 2.568e-05 0.00% MLLinOp::define() 11 2.388e-05 2.388e-05 2.388e-05 0.00% Castro::buildMetrics() 1 2.135e-05 2.135e-05 2.135e-05 0.00% Castro::construct_old_source() 50 2.09e-05 2.09e-05 2.09e-05 0.00% Amr::FinalizeInit() 1 1.976e-05 1.976e-05 1.976e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.903e-05 1.903e-05 1.903e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.779e-05 1.779e-05 1.779e-05 0.00% Castro::do_new_sources() 10 1.74e-05 1.74e-05 1.74e-05 0.00% Castro::construct_new_source() 50 1.68e-05 1.68e-05 1.68e-05 0.00% Castro::do_old_sources() 10 1.612e-05 1.612e-05 1.612e-05 0.00% DistributionMapping::Distribute() 56 1.422e-05 1.422e-05 1.422e-05 0.00% Castro::check_for_nan() 20 1.179e-05 1.179e-05 1.179e-05 0.00% Castro::apply_source_to_state() 20 1.065e-05 1.065e-05 1.065e-05 0.00% Castro::construct_old_gravity() 10 1.046e-05 1.046e-05 1.046e-05 0.00% MLPoisson::prepareForSolve() 11 9.133e-06 9.133e-06 9.133e-06 0.00% MLMG::computeMLResidual() 11 9.046e-06 9.046e-06 9.046e-06 0.00% Castro::post_timestep() 10 8.977e-06 8.977e-06 8.977e-06 0.00% Amr::initSubcycle() 1 8.825e-06 8.825e-06 8.825e-06 0.00% Gravity::swapTimeLevels() 10 8.46e-06 8.46e-06 8.46e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.194e-06 7.194e-06 7.194e-06 0.00% MLMG::getGradSolution() 11 6.52e-06 6.52e-06 6.52e-06 0.00% Castro::computeNewDt() 9 6.039e-06 6.039e-06 6.039e-06 0.00% AmrLevel::checkPointPost() 3 5.606e-06 5.606e-06 5.606e-06 0.00% Amr::InitializeInit() 1 4.905e-06 4.905e-06 4.905e-06 0.00% Gravity::set_mass_offset() 11 4.203e-06 4.203e-06 4.203e-06 0.00% Castro::retry_advance_ctu() 10 3.821e-06 3.821e-06 3.821e-06 0.00% Castro::create_source_corrector() 10 3.523e-06 3.523e-06 3.523e-06 0.00% MLMG::MLResNormInf() 11 3.425e-06 3.425e-06 3.425e-06 0.00% Castro::post_init() 1 3.239e-06 3.239e-06 3.239e-06 0.00% Castro::FluxRegCrseInit 10 2.769e-06 2.769e-06 2.769e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.748e-06 2.748e-06 2.748e-06 0.00% Amr::init() 1 2.734e-06 2.734e-06 2.734e-06 0.00% Castro::computeInitialDt() 2 2.394e-06 2.394e-06 2.394e-06 0.00% Castro::FluxRegFineAdd() 10 2.177e-06 2.177e-06 2.177e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.925e-06 1.925e-06 1.925e-06 0.00% AmrLevel::checkPointPre() 3 1.849e-06 1.849e-06 1.849e-06 0.00% Amr::initialInit() 1 1.104e-06 1.104e-06 1.104e-06 0.00% Castro::post_regrid() 1 1.005e-06 1.005e-06 1.005e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.825 0.825 0.825 100.00% Amr::coarseTimeStep() 10 0.6659 0.6659 0.6659 80.71% Amr::timeStep() 10 0.5686 0.5686 0.5686 68.92% Castro::advance() 10 0.5616 0.5616 0.5616 68.08% Castro::subcycle_advance_ctu() 10 0.5501 0.5501 0.5501 66.69% Castro::do_advance_ctu() 10 0.5499 0.5499 0.5499 66.66% Gravity::solve_phi_with_mlmg() 11 0.3073 0.3073 0.3073 37.25% Gravity::actual_solve_with_mlmg() 11 0.2989 0.2989 0.2989 36.23% Castro::construct_new_gravity() 10 0.2836 0.2836 0.2836 34.38% MLMG::solve() 11 0.2767 0.2767 0.2767 33.54% Gravity::solve_for_phi() 10 0.2672 0.2672 0.2672 32.39% MLMG::oneIter() 81 0.262 0.262 0.262 31.76% MLMG::mgVcycle() 81 0.2603 0.2603 0.2603 31.56% VisMF::Write(FabArray) 11 0.1967 0.1967 0.1967 23.84% Castro::construct_ctu_hydro_source() 10 0.1887 0.1887 0.1887 22.87% Amr::checkPoint() 3 0.1455 0.1455 0.1455 17.64% AmrLevel::checkPoint() 3 0.1408 0.1408 0.1408 17.07% StateData::checkPoint() 12 0.1408 0.1408 0.1408 17.06% MLCellLinOp::smooth() 1620 0.1334 0.1334 0.1334 16.17% Amr::init() 1 0.1291 0.1291 0.1291 15.65% MLCellLinOp::applyBC() 4379 0.09388 0.09388 0.09388 11.38% MLMG::mgVcycle_bottom 81 0.08003 0.08003 0.08003 9.70% MLMG::actualBottomSolve() 81 0.08 0.08 0.08 9.70% MLCGSolver::bicgstab 81 0.0792 0.0792 0.0792 9.60% MLPoisson::Fsmooth() 3240 0.06252 0.06252 0.06252 7.58% Amr::writePlotFile() 2 0.05864 0.05864 0.05864 7.11% Amr::initialInit() 1 0.04857 0.04857 0.04857 5.89% FillPatchIterator::Initialize 41 0.04682 0.04682 0.04682 5.68% FillPatchSingleLevel 41 0.04525 0.04525 0.04525 5.48% Amr::FinalizeInit() 1 0.04443 0.04443 0.04443 5.39% Castro::clean_state() 62 0.04328 0.04328 0.04328 5.25% Castro::post_init() 1 0.04301 0.04301 0.04301 5.21% StateDataPhysBCFunct::() 41 0.04122 0.04122 0.04122 5.00% Gravity::multilevel_solve_for_new_phi() 1 0.04065 0.04065 0.04065 4.93% Gravity::actual_multilevel_solve() 1 0.04064 0.04064 0.04064 4.93% MLCellLinOp::apply() 1128 0.03558 0.03558 0.03558 4.31% MLMG::mgVcycle_down::0 81 0.03456 0.03456 0.03456 4.19% MLMG::mgVcycle_up::0 81 0.0298 0.0298 0.0298 3.61% StateData::FillBoundary(geom) 328 0.02322 0.02322 0.02322 2.81% MultiFab::Dot() 1100 0.02184 0.02184 0.02184 2.65% MLCellLinOp::correctionResidual() 486 0.02078 0.02078 0.02078 2.52% Castro::initialize_do_advance() 10 0.02034 0.02034 0.02034 2.47% Castro::computeTemp() 63 0.01943 0.01943 0.01943 2.36% Gravity::get_new_grav_vector() 11 0.01848 0.01848 0.01848 2.24% MLMG:computeResOfCorrection() 405 0.01796 0.01796 0.01796 2.18% MLPoisson::define() 11 0.01784 0.01784 0.01784 2.16% MLMG::mgVcycle_down::1 81 0.01729 0.01729 0.01729 2.10% MLMG::mgVcycle_down::2 81 0.01684 0.01684 0.01684 2.04% Castro::construct_old_gravity() 10 0.0166 0.0166 0.0166 2.01% Gravity::get_old_grav_vector() 10 0.01659 0.01659 0.01659 2.01% MLMG::mgVcycle_down::3 81 0.01596 0.01596 0.01596 1.94% FabArray::FillBoundary() 3974 0.01555 0.01555 0.01555 1.89% MLMG::mgVcycle_down::4 81 0.01523 0.01523 0.01523 1.85% FillBoundary_nowait() 3974 0.0147 0.0147 0.0147 1.78% CGSolver::sxay() 1566 0.01434 0.01434 0.01434 1.74% MultiFab::LinComb() 1566 0.01398 0.01398 0.01398 1.69% FabArray::setVal() 1135 0.01395 0.01395 0.01395 1.69% FabArray::ParallelCopy() 851 0.01386 0.01386 0.01386 1.68% FabArray::ParallelCopy_nowait() 851 0.0136 0.0136 0.0136 1.65% Castro::normalize_species() 62 0.01308 0.01308 0.01308 1.59% MLMG::mgVcycle_up::2 81 0.01302 0.01302 0.01302 1.58% MLCGSolver::ParallelAllReduce 1495 0.01302 0.01302 0.01302 1.58% MLMG::mgVcycle_up::1 81 0.01278 0.01278 0.01278 1.55% MLCellLinOp::defineAuxData() 11 0.01262 0.01262 0.01262 1.53% MLMG::addInterpCorrection() 405 0.0125 0.0125 0.0125 1.52% MLMG::mgVcycle_up::3 81 0.01232 0.01232 0.01232 1.49% MLMG::mgVcycle_up::4 81 0.0123 0.0123 0.0123 1.49% Castro::expand_state() 10 0.0123 0.0123 0.0123 1.49% amrex::average_down 405 0.01156 0.01156 0.01156 1.40% MLPoisson::Fapply() 1128 0.01148 0.01148 0.01148 1.39% Castro::initialize_advance() 10 0.01135 0.01135 0.01135 1.38% Castro::do_old_sources() 10 0.01059 0.01059 0.01059 1.28% Castro::do_new_sources() 10 0.01045 0.01045 0.01045 1.27% Castro::enforce_min_density() 62 0.009785 0.009785 0.009785 1.19% Gravity::fill_multipole_BCs() 11 0.008186 0.008186 0.008186 0.99% MLCellLinOp::solutionResidual() 92 0.00697 0.00697 0.00697 0.84% Castro::post_timestep() 10 0.006794 0.006794 0.006794 0.82% MultiFab::Xpay() 578 0.006393 0.006393 0.006393 0.77% MLMG::computeResidual() 81 0.005997 0.005997 0.005997 0.73% Castro::reset_internal_energy(MultiFab) 63 0.005736 0.005736 0.005736 0.70% MLMG::prepareForSolve() 11 0.005316 0.005316 0.005316 0.64% Castro::estTimeStep() 21 0.004965 0.004965 0.004965 0.60% MLCellLinOp::defineBC() 11 0.004952 0.004952 0.004952 0.60% BndryData::define() 11 0.004672 0.004672 0.004672 0.57% Amr::InitializeInit() 1 0.004138 0.004138 0.004138 0.50% Amr::defBaseLevel() 1 0.004134 0.004134 0.004134 0.50% Castro::initData() 1 0.003612 0.003612 0.003612 0.44% Castro::construct_new_source() 50 0.003134 0.003134 0.003134 0.38% Castro::construct_new_gravity_source() 10 0.003117 0.003117 0.003117 0.38% Castro::construct_old_source() 50 0.002422 0.002422 0.002422 0.29% Castro::construct_old_gravity_source() 10 0.002401 0.002401 0.002401 0.29% MLMG::ResNormInf() 92 0.002053 0.002053 0.002053 0.25% Castro::computeNewDt() 9 0.002025 0.002025 0.002025 0.25% Castro::apply_source_to_state() 20 0.001816 0.001816 0.001816 0.22% MultiFab::Saxpy() 20 0.001806 0.001806 0.001806 0.22% Castro::enforce_speed_limit() 62 0.001705 0.001705 0.001705 0.21% MultiFab::Add() 81 0.001634 0.001634 0.001634 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001605 0.001605 0.001605 0.19% MLCellLinOp::setLevelBC() 11 0.001516 0.001516 0.001516 0.18% Castro::reset_internal_energy(Fab) 504 0.001441 0.001441 0.001441 0.17% MLMG::getGradSolution() 11 0.001414 0.001414 0.001414 0.17% MLCellLinOp::compGrad() 11 0.001407 0.001407 0.001407 0.17% FabArrayBase::getCPC() 1313 0.001398 0.001398 0.001398 0.17% FabArray::mult() 43 0.001324 0.001324 0.001324 0.16% FabArray::setDomainBndry() 41 0.001281 0.001281 0.001281 0.16% Castro::check_for_nan() 20 0.001184 0.001184 0.001184 0.14% MultiFab::contains_nan() 20 0.001173 0.001173 0.001173 0.14% MLPoisson::prepareForSolve() 11 0.001168 0.001168 0.001168 0.14% Castro::post_regrid() 1 0.00116 0.00116 0.00116 0.14% MLCellLinOp::prepareForSolve() 11 0.001158 0.001158 0.001158 0.14% MLMG::computeMLResidual() 11 0.001018 0.001018 0.001018 0.12% Gravity::update_max_rhs() 11 0.0008138 0.0008138 0.0008138 0.10% Castro::computeInitialDt() 2 0.0007465 0.0007465 0.0007465 0.09% FabArrayBase::getFB() 3974 0.0006744 0.0006744 0.0006744 0.08% FabArrayBase::CPC::define() 454 0.0006663 0.0006663 0.0006663 0.08% Amr::InitAmr() 1 0.0004682 0.0004682 0.0004682 0.06% Castro::Castro() 1 0.0004382 0.0004382 0.0004382 0.05% Gravity::swapTimeLevels() 10 0.0004312 0.0004312 0.0004312 0.05% MultiFab::Copy() 11 0.0003256 0.0003256 0.0003256 0.04% MLMG::MLResNormInf() 11 0.0002809 0.0002809 0.0002809 0.03% MultiFab::max() 11 0.0002542 0.0002542 0.0002542 0.03% MLLinOp::define() 11 0.000243 0.000243 0.000243 0.03% MLLinOp::defineGrids() 11 0.0002191 0.0002191 0.0002191 0.03% MLMG::MLRhsNormInf() 11 0.0002145 0.0002145 0.0002145 0.03% Castro::buildMetrics() 1 0.0001554 0.0001554 0.0001554 0.02% FabArrayBase::FB::FB() 56 8.577e-05 8.577e-05 8.577e-05 0.01% Castro::finalize_advance() 10 6.891e-05 6.891e-05 6.891e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.819e-05 5.819e-05 5.819e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.122e-05 5.122e-05 5.122e-05 0.01% StateData::define() 4 4.402e-05 4.402e-05 4.402e-05 0.01% Castro::swap_state_time_levels() 10 4.136e-05 4.136e-05 4.136e-05 0.01% makeSFC 55 3.916e-05 3.916e-05 3.916e-05 0.00% Castro::finalize_do_advance() 10 3.438e-05 3.438e-05 3.438e-05 0.00% Castro::enforce_consistent_e() 1 3.299e-05 3.299e-05 3.299e-05 0.00% Castro::initMFs() 1 2.654e-05 2.654e-05 2.654e-05 0.00% Amr::writeSmallPlotFile() 1 2.568e-05 2.568e-05 2.568e-05 0.00% DistributionMapping::Distribute() 56 1.422e-05 1.422e-05 1.422e-05 0.00% Amr::initSubcycle() 1 8.825e-06 8.825e-06 8.825e-06 0.00% AmrLevel::checkPointPost() 3 5.606e-06 5.606e-06 5.606e-06 0.00% Gravity::set_mass_offset() 11 4.203e-06 4.203e-06 4.203e-06 0.00% Castro::retry_advance_ctu() 10 3.821e-06 3.821e-06 3.821e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.787e-06 3.787e-06 3.787e-06 0.00% Castro::create_source_corrector() 10 3.523e-06 3.523e-06 3.523e-06 0.00% Castro::FluxRegCrseInit 10 2.769e-06 2.769e-06 2.769e-06 0.00% Castro::FluxRegFineAdd() 10 2.177e-06 2.177e-06 2.177e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.925e-06 1.925e-06 1.925e-06 0.00% AmrLevel::checkPointPre() 3 1.849e-06 1.849e-06 1.849e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-9-g67384701a808) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-9-g67384701a808) initialized Starting run at 08:30:56 UTC on 2022-09-12. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09-9-g67384701a Microphysics git describe: 22.08-12-g91211ba4 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.447113484 Restart time = 0.048169137 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052672966 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.054645794 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.059818644 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.062359154 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.077738798 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031157389 seconds Ending run at 08:30:56 UTC on 2022-09-12. Run time = 0.387512741 Run time without initialization = 0.338789357 Average number of zones advanced per microsecond: 3.869 Average number of zones advanced per microsecond per rank: 3.869 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3875 ... 0.3875 ... 0.3875 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1036 0.1036 0.1036 26.72% VisMF::Read() 3 0.04036 0.04036 0.04036 10.41% MLCellLinOp::applyBC() 1946 0.03431 0.03431 0.03431 8.85% VisMF::Write(FabArray) 1 0.02964 0.02964 0.02964 7.65% MLPoisson::Fsmooth() 1440 0.02698 0.02698 0.02698 6.96% StateData::FillBoundary(geom) 160 0.011 0.011 0.011 2.84% MLCGSolver::bicgstab 36 0.01007 0.01007 0.01007 2.60% MultiFab::Dot() 484 0.009436 0.009436 0.009436 2.43% Castro::computeTemp() 30 0.009003 0.009003 0.009003 2.32% Castro::normalize_species() 30 0.008864 0.008864 0.008864 2.29% StateDataPhysBCFunct::() 20 0.008709 0.008709 0.008709 2.25% FabArray::setVal() 537 0.006638 0.006638 0.006638 1.71% FillBoundary_nowait() 1766 0.006179 0.006179 0.006179 1.59% MultiFab::LinComb() 690 0.006063 0.006063 0.006063 1.56% MLCellLinOp::defineAuxData() 6 0.006011 0.006011 0.006011 1.55% FabArray::ParallelCopy_nowait() 380 0.005922 0.005922 0.005922 1.53% Castro::enforce_min_density() 30 0.005803 0.005803 0.005803 1.50% MLPoisson::Fapply() 500 0.005008 0.005008 0.005008 1.29% Gravity::fill_multipole_BCs() 6 0.004456 0.004456 0.004456 1.15% Amr::restart() 1 0.003602 0.003602 0.003602 0.93% MLMG::addInterpCorrection() 180 0.003305 0.003305 0.003305 0.85% Castro::estTimeStep() 10 0.00292 0.00292 0.00292 0.75% amrex::average_down 180 0.002909 0.002909 0.002909 0.75% MultiFab::Xpay() 258 0.002815 0.002815 0.002815 0.73% Castro::do_advance_ctu() 5 0.002421 0.002421 0.002421 0.62% BndryData::define() 6 0.002055 0.002055 0.002055 0.53% Castro::enforce_speed_limit() 30 0.001943 0.001943 0.001943 0.50% Castro::construct_new_gravity_source() 5 0.001742 0.001742 0.001742 0.45% Castro::reset_internal_energy(MultiFab) 30 0.001704 0.001704 0.001704 0.44% Amr::writePlotFile() 1 0.001604 0.001604 0.001604 0.41% Castro::construct_old_gravity_source() 5 0.001494 0.001494 0.001494 0.39% Castro::subcycle_advance_ctu() 5 0.0009804 0.0009804 0.0009804 0.25% Castro::reset_internal_energy(Fab) 240 0.0009743 0.0009743 0.0009743 0.25% Gravity::get_old_grav_vector() 5 0.0009367 0.0009367 0.0009367 0.24% MultiFab::Saxpy() 10 0.0009205 0.0009205 0.0009205 0.24% MLMG::ResNormInf() 42 0.0009128 0.0009128 0.0009128 0.24% Castro::expand_state() 5 0.0008716 0.0008716 0.0008716 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000867 0.000867 0.000867 0.22% Gravity::get_new_grav_vector() 5 0.0008595 0.0008595 0.0008595 0.22% Castro::create_source_corrector() 5 0.0008496 0.0008496 0.0008496 0.22% MLCellLinOp::setLevelBC() 6 0.0008009 0.0008009 0.0008009 0.21% Gravity::actual_solve_with_mlmg() 6 0.000797 0.000797 0.000797 0.21% MultiFab::Add() 36 0.0007142 0.0007142 0.0007142 0.18% FabArray::mult() 22 0.000647 0.000647 0.000647 0.17% MLMG::prepareForSolve() 6 0.0006451 0.0006451 0.0006451 0.17% FabArray::setDomainBndry() 20 0.00064 0.00064 0.00064 0.17% MLCellLinOp::prepareForSolve() 6 0.0006243 0.0006243 0.0006243 0.16% MultiFab::contains_nan() 10 0.0005919 0.0005919 0.0005919 0.15% MLCellLinOp::smooth() 720 0.0005054 0.0005054 0.0005054 0.13% MLCellLinOp::compGrad() 6 0.0004906 0.0004906 0.0004906 0.13% Amr::InitAmr() 1 0.0003926 0.0003926 0.0003926 0.10% FabArrayBase::CPC::define() 244 0.0003916 0.0003916 0.0003916 0.10% FabArray::FillBoundary() 1766 0.0003913 0.0003913 0.0003913 0.10% FabArrayBase::getCPC() 632 0.0003594 0.0003594 0.0003594 0.09% FabArrayBase::getFB() 1766 0.0002538 0.0002538 0.0002538 0.07% main() 1 0.0002487 0.0002487 0.0002487 0.06% Gravity::update_max_rhs() 6 0.0002267 0.0002267 0.0002267 0.06% Gravity::solve_for_phi() 5 0.0001919 0.0001919 0.0001919 0.05% MLCellLinOp::apply() 500 0.0001887 0.0001887 0.0001887 0.05% MultiFab::Copy() 6 0.0001739 0.0001739 0.0001739 0.04% CGSolver::sxay() 690 0.000165 0.000165 0.000165 0.04% Castro::construct_new_source() 25 0.0001581 0.0001581 0.0001581 0.04% Amr::coarseTimeStep() 5 0.0001574 0.0001574 0.0001574 0.04% Castro::construct_new_gravity() 5 0.0001569 0.0001569 0.0001569 0.04% MLCellLinOp::defineBC() 6 0.0001481 0.0001481 0.0001481 0.04% FillPatchIterator::Initialize 20 0.000135 0.000135 0.000135 0.03% MultiFab::max() 6 0.0001335 0.0001335 0.0001335 0.03% MLCGSolver::ParallelAllReduce 659 0.0001262 0.0001262 0.0001262 0.03% FabArray::ParallelCopy() 380 0.000121 0.000121 0.000121 0.03% MLMG::MLRhsNormInf() 6 0.0001122 0.0001122 0.0001122 0.03% MLLinOp::defineGrids() 6 0.0001082 0.0001082 0.0001082 0.03% MLCellLinOp::correctionResidual() 216 9.317e-05 9.317e-05 9.317e-05 0.02% MLMG::mgVcycle() 36 8.926e-05 8.926e-05 8.926e-05 0.02% Castro::post_timestep() 5 8.719e-05 8.719e-05 8.719e-05 0.02% Castro::advance() 5 8.279e-05 8.279e-05 8.279e-05 0.02% Amr::timeStep() 5 8.017e-05 8.017e-05 8.017e-05 0.02% Castro::construct_old_source() 25 7.831e-05 7.831e-05 7.831e-05 0.02% Castro::computeNewDt() 5 7.6e-05 7.6e-05 7.6e-05 0.02% AmrLevel::restart() 1 7.455e-05 7.455e-05 7.455e-05 0.02% Castro::initialize_do_advance() 5 7.327e-05 7.327e-05 7.327e-05 0.02% StateData::restartDoit() 4 7.239e-05 7.239e-05 7.239e-05 0.02% Castro::finalize_advance() 5 7.092e-05 7.092e-05 7.092e-05 0.02% Castro::initialize_advance() 5 6.946e-05 6.946e-05 6.946e-05 0.02% MLMG:computeResOfCorrection() 180 6.687e-05 6.687e-05 6.687e-05 0.02% FabArrayBase::FB::FB() 26 5.686e-05 5.686e-05 5.686e-05 0.01% Castro::construct_old_gravity() 5 4.594e-05 4.594e-05 4.594e-05 0.01% MLMG::mgVcycle_down::0 36 4.49e-05 4.49e-05 4.49e-05 0.01% MLMG::mgVcycle_down::1 36 4.094e-05 4.094e-05 4.094e-05 0.01% Castro::clean_state() 30 3.845e-05 3.845e-05 3.845e-05 0.01% MLMG::mgVcycle_down::2 36 3.832e-05 3.832e-05 3.832e-05 0.01% MLMG::mgVcycle_down::4 36 3.674e-05 3.674e-05 3.674e-05 0.01% MLMG::solve() 6 3.607e-05 3.607e-05 3.607e-05 0.01% MLMG::mgVcycle_down::3 36 3.512e-05 3.512e-05 3.512e-05 0.01% Castro::buildMetrics() 1 3.364e-05 3.364e-05 3.364e-05 0.01% MLMG::actualBottomSolve() 36 3.362e-05 3.362e-05 3.362e-05 0.01% MLMG::mgVcycle_up::4 36 3.315e-05 3.315e-05 3.315e-05 0.01% Gravity::actual_multilevel_solve() 1 2.989e-05 2.989e-05 2.989e-05 0.01% Castro::post_restart() 1 2.959e-05 2.959e-05 2.959e-05 0.01% Castro::initMFs() 1 2.935e-05 2.935e-05 2.935e-05 0.01% Castro::swap_state_time_levels() 5 2.774e-05 2.774e-05 2.774e-05 0.01% Amr::writeSmallPlotFile() 1 2.726e-05 2.726e-05 2.726e-05 0.01% MLMG::mgVcycle_up::0 36 2.684e-05 2.684e-05 2.684e-05 0.01% MLMG::mgVcycle_up::3 36 2.608e-05 2.608e-05 2.608e-05 0.01% MLMG::oneIter() 36 2.506e-05 2.506e-05 2.506e-05 0.01% MLMG::mgVcycle_up::2 36 2.459e-05 2.459e-05 2.459e-05 0.01% MLMG::mgVcycle_up::1 36 2.365e-05 2.365e-05 2.365e-05 0.01% MLCellLinOp::solutionResidual() 42 2.305e-05 2.305e-05 2.305e-05 0.01% Castro::finalize_do_advance() 5 2.166e-05 2.166e-05 2.166e-05 0.01% MLPoisson::define() 6 2.127e-05 2.127e-05 2.127e-05 0.01% MLLinOp::define() 6 2.001e-05 2.001e-05 2.001e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.817e-05 1.817e-05 1.817e-05 0.00% MLMG::computeResidual() 36 1.71e-05 1.71e-05 1.71e-05 0.00% MLMG::mgVcycle_bottom 36 1.502e-05 1.502e-05 1.502e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.471e-05 1.471e-05 1.471e-05 0.00% FillPatchSingleLevel 20 1.372e-05 1.372e-05 1.372e-05 0.00% makeSFC 30 1.321e-05 1.321e-05 1.321e-05 0.00% Castro::apply_source_to_state() 10 9.861e-06 9.861e-06 9.861e-06 0.00% Castro::do_new_sources() 5 9.66e-06 9.66e-06 9.66e-06 0.00% DistributionMapping::Distribute() 31 8.555e-06 8.555e-06 8.555e-06 0.00% Castro::do_old_sources() 5 8.52e-06 8.52e-06 8.52e-06 0.00% Amr::initSubcycle() 1 8.312e-06 8.312e-06 8.312e-06 0.00% Castro::check_for_nan() 10 8.207e-06 8.207e-06 8.207e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.436e-06 7.436e-06 7.436e-06 0.00% MLPoisson::prepareForSolve() 6 5.077e-06 5.077e-06 5.077e-06 0.00% MLMG::computeMLResidual() 6 4.782e-06 4.782e-06 4.782e-06 0.00% Gravity::swapTimeLevels() 5 4.11e-06 4.11e-06 4.11e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.467e-06 3.467e-06 3.467e-06 0.00% MLMG::getGradSolution() 6 3.052e-06 3.052e-06 3.052e-06 0.00% Gravity::set_mass_offset() 6 2.298e-06 2.298e-06 2.298e-06 0.00% MLMG::MLResNormInf() 6 2.032e-06 2.032e-06 2.032e-06 0.00% Castro::retry_advance_ctu() 5 1.889e-06 1.889e-06 1.889e-06 0.00% Castro::FluxRegCrseInit 5 1.556e-06 1.556e-06 1.556e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.179e-06 1.179e-06 1.179e-06 0.00% Castro::FluxRegFineAdd() 5 1.152e-06 1.152e-06 1.152e-06 0.00% Amr::init() 1 1.145e-06 1.145e-06 1.145e-06 0.00% AmrLevel::AmrLevel() 1 6.67e-07 6.67e-07 6.67e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3875 0.3875 0.3875 100.00% Amr::coarseTimeStep() 5 0.3074 0.3074 0.3074 79.32% Amr::timeStep() 5 0.3056 0.3056 0.3056 78.87% Castro::advance() 5 0.3007 0.3007 0.3007 77.60% Castro::subcycle_advance_ctu() 5 0.2934 0.2934 0.2934 75.70% Castro::do_advance_ctu() 5 0.2924 0.2924 0.2924 75.45% Castro::construct_new_gravity() 5 0.143 0.143 0.143 36.90% Gravity::solve_phi_with_mlmg() 6 0.1377 0.1377 0.1377 35.53% Gravity::solve_for_phi() 5 0.1342 0.1342 0.1342 34.62% Gravity::actual_solve_with_mlmg() 6 0.1331 0.1331 0.1331 34.35% MLMG::solve() 6 0.1211 0.1211 0.1211 31.25% MLMG::oneIter() 36 0.114 0.114 0.114 29.41% MLMG::mgVcycle() 36 0.1133 0.1133 0.1133 29.22% Castro::construct_ctu_hydro_source() 5 0.1036 0.1036 0.1036 26.72% MLCellLinOp::smooth() 720 0.058 0.058 0.058 14.97% Amr::init() 1 0.04822 0.04822 0.04822 12.44% Amr::restart() 1 0.04821 0.04821 0.04821 12.44% MLCellLinOp::applyBC() 1946 0.04119 0.04119 0.04119 10.63% AmrLevel::restart() 1 0.04056 0.04056 0.04056 10.47% StateData::restartDoit() 4 0.04048 0.04048 0.04048 10.45% VisMF::Read() 3 0.04036 0.04036 0.04036 10.41% MLMG::mgVcycle_bottom 36 0.03466 0.03466 0.03466 8.94% MLMG::actualBottomSolve() 36 0.03464 0.03464 0.03464 8.94% MLCGSolver::bicgstab 36 0.0343 0.0343 0.0343 8.85% Amr::writePlotFile() 1 0.03125 0.03125 0.03125 8.06% VisMF::Write(FabArray) 1 0.02964 0.02964 0.02964 7.65% Castro::clean_state() 30 0.02833 0.02833 0.02833 7.31% MLPoisson::Fsmooth() 1440 0.02698 0.02698 0.02698 6.96% FillPatchIterator::Initialize 20 0.02252 0.02252 0.02252 5.81% FillPatchSingleLevel 20 0.02175 0.02175 0.02175 5.61% StateDataPhysBCFunct::() 20 0.01971 0.01971 0.01971 5.09% MLCellLinOp::apply() 500 0.0156 0.0156 0.0156 4.03% MLMG::mgVcycle_down::0 36 0.01516 0.01516 0.01516 3.91% MLMG::mgVcycle_up::0 36 0.01299 0.01299 0.01299 3.35% Castro::initialize_do_advance() 5 0.0119 0.0119 0.0119 3.07% Castro::computeTemp() 30 0.01168 0.01168 0.01168 3.01% StateData::FillBoundary(geom) 160 0.011 0.011 0.011 2.84% MLPoisson::define() 6 0.009649 0.009649 0.009649 2.49% MultiFab::Dot() 484 0.009436 0.009436 0.009436 2.43% MLCellLinOp::correctionResidual() 216 0.009072 0.009072 0.009072 2.34% Castro::normalize_species() 30 0.008864 0.008864 0.008864 2.29% Gravity::get_new_grav_vector() 5 0.008669 0.008669 0.008669 2.24% Castro::construct_old_gravity() 5 0.008172 0.008172 0.008172 2.11% Gravity::get_old_grav_vector() 5 0.008126 0.008126 0.008126 2.10% MLMG:computeResOfCorrection() 180 0.007829 0.007829 0.007829 2.02% MLMG::mgVcycle_down::1 36 0.00753 0.00753 0.00753 1.94% MLMG::mgVcycle_down::2 36 0.007312 0.007312 0.007312 1.89% Castro::initialize_advance() 5 0.007204 0.007204 0.007204 1.86% Castro::do_new_sources() 5 0.007112 0.007112 0.007112 1.84% MLMG::mgVcycle_down::3 36 0.006918 0.006918 0.006918 1.79% FabArray::FillBoundary() 1766 0.006881 0.006881 0.006881 1.78% MLCellLinOp::defineAuxData() 6 0.006746 0.006746 0.006746 1.74% MLMG::mgVcycle_down::4 36 0.006654 0.006654 0.006654 1.72% FabArray::setVal() 537 0.006638 0.006638 0.006638 1.71% FillBoundary_nowait() 1766 0.00649 0.00649 0.00649 1.67% FabArray::ParallelCopy() 380 0.006405 0.006405 0.006405 1.65% Castro::do_old_sources() 5 0.006346 0.006346 0.006346 1.64% FabArray::ParallelCopy_nowait() 380 0.006284 0.006284 0.006284 1.62% CGSolver::sxay() 690 0.006228 0.006228 0.006228 1.61% MultiFab::LinComb() 690 0.006063 0.006063 0.006063 1.56% Castro::expand_state() 5 0.005999 0.005999 0.005999 1.55% Castro::enforce_min_density() 30 0.005803 0.005803 0.005803 1.50% MLCGSolver::ParallelAllReduce 659 0.005675 0.005675 0.005675 1.46% MLMG::mgVcycle_up::2 36 0.00564 0.00564 0.00564 1.46% MLMG::mgVcycle_up::1 36 0.005569 0.005569 0.005569 1.44% MLMG::addInterpCorrection() 180 0.005499 0.005499 0.005499 1.42% MLMG::mgVcycle_up::4 36 0.005376 0.005376 0.005376 1.39% MLMG::mgVcycle_up::3 36 0.005358 0.005358 0.005358 1.38% amrex::average_down 180 0.005096 0.005096 0.005096 1.32% MLPoisson::Fapply() 500 0.005008 0.005008 0.005008 1.29% Castro::post_timestep() 5 0.004826 0.004826 0.004826 1.25% Gravity::fill_multipole_BCs() 6 0.004456 0.004456 0.004456 1.15% Castro::post_restart() 1 0.003871 0.003871 0.003871 1.00% Gravity::multilevel_solve_for_new_phi() 1 0.003749 0.003749 0.003749 0.97% Gravity::actual_multilevel_solve() 1 0.003731 0.003731 0.003731 0.96% MLCellLinOp::solutionResidual() 42 0.003183 0.003183 0.003183 0.82% Castro::estTimeStep() 10 0.00292 0.00292 0.00292 0.75% MLMG::prepareForSolve() 6 0.002838 0.002838 0.002838 0.73% MultiFab::Xpay() 258 0.002815 0.002815 0.002815 0.73% MLCellLinOp::defineBC() 6 0.002725 0.002725 0.002725 0.70% Castro::reset_internal_energy(MultiFab) 30 0.002678 0.002678 0.002678 0.69% MLMG::computeResidual() 36 0.002641 0.002641 0.002641 0.68% BndryData::define() 6 0.002577 0.002577 0.002577 0.66% Castro::enforce_speed_limit() 30 0.001943 0.001943 0.001943 0.50% Castro::construct_new_source() 25 0.0019 0.0019 0.0019 0.49% Castro::construct_new_gravity_source() 5 0.001742 0.001742 0.001742 0.45% Castro::computeNewDt() 5 0.001582 0.001582 0.001582 0.41% Castro::construct_old_source() 25 0.001572 0.001572 0.001572 0.41% Castro::construct_old_gravity_source() 5 0.001494 0.001494 0.001494 0.39% Castro::reset_internal_energy(Fab) 240 0.0009743 0.0009743 0.0009743 0.25% Castro::apply_source_to_state() 10 0.0009304 0.0009304 0.0009304 0.24% MultiFab::Saxpy() 10 0.0009205 0.0009205 0.0009205 0.24% MLMG::ResNormInf() 42 0.0009128 0.0009128 0.0009128 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000867 0.000867 0.000867 0.22% Castro::create_source_corrector() 5 0.0008496 0.0008496 0.0008496 0.22% MLCellLinOp::setLevelBC() 6 0.0008009 0.0008009 0.0008009 0.21% MLMG::getGradSolution() 6 0.0007638 0.0007638 0.0007638 0.20% MLCellLinOp::compGrad() 6 0.0007607 0.0007607 0.0007607 0.20% FabArrayBase::getCPC() 632 0.000751 0.000751 0.000751 0.19% MultiFab::Add() 36 0.0007142 0.0007142 0.0007142 0.18% FabArray::mult() 22 0.000647 0.000647 0.000647 0.17% FabArray::setDomainBndry() 20 0.00064 0.00064 0.00064 0.17% MLPoisson::prepareForSolve() 6 0.0006294 0.0006294 0.0006294 0.16% MLCellLinOp::prepareForSolve() 6 0.0006243 0.0006243 0.0006243 0.16% Castro::check_for_nan() 10 0.0006001 0.0006001 0.0006001 0.15% MultiFab::contains_nan() 10 0.0005919 0.0005919 0.0005919 0.15% MLMG::computeMLResidual() 6 0.0005647 0.0005647 0.0005647 0.15% Gravity::update_max_rhs() 6 0.0004373 0.0004373 0.0004373 0.11% Amr::InitAmr() 1 0.0004009 0.0004009 0.0004009 0.10% FabArrayBase::CPC::define() 244 0.0003916 0.0003916 0.0003916 0.10% FabArrayBase::getFB() 1766 0.0003106 0.0003106 0.0003106 0.08% Gravity::swapTimeLevels() 5 0.0002234 0.0002234 0.0002234 0.06% MultiFab::Copy() 6 0.0001739 0.0001739 0.0001739 0.04% MLLinOp::define() 6 0.0001574 0.0001574 0.0001574 0.04% Castro::buildMetrics() 1 0.0001536 0.0001536 0.0001536 0.04% MLMG::MLResNormInf() 6 0.000146 0.000146 0.000146 0.04% MLLinOp::defineGrids() 6 0.0001374 0.0001374 0.0001374 0.04% MultiFab::max() 6 0.0001335 0.0001335 0.0001335 0.03% MLMG::MLRhsNormInf() 6 0.0001122 0.0001122 0.0001122 0.03% Castro::finalize_advance() 5 7.363e-05 7.363e-05 7.363e-05 0.02% FabArrayBase::FB::FB() 26 5.686e-05 5.686e-05 5.686e-05 0.01% Castro::initMFs() 1 2.935e-05 2.935e-05 2.935e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.803e-05 2.803e-05 2.803e-05 0.01% Castro::swap_state_time_levels() 5 2.774e-05 2.774e-05 2.774e-05 0.01% Amr::writeSmallPlotFile() 1 2.726e-05 2.726e-05 2.726e-05 0.01% Castro::finalize_do_advance() 5 2.166e-05 2.166e-05 2.166e-05 0.01% makeSFC 30 2.059e-05 2.059e-05 2.059e-05 0.01% DistributionMapping::Distribute() 31 8.555e-06 8.555e-06 8.555e-06 0.00% Amr::initSubcycle() 1 8.312e-06 8.312e-06 8.312e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.646e-06 4.646e-06 4.646e-06 0.00% Gravity::set_mass_offset() 6 2.298e-06 2.298e-06 2.298e-06 0.00% Castro::retry_advance_ctu() 5 1.889e-06 1.889e-06 1.889e-06 0.00% Castro::FluxRegCrseInit 5 1.556e-06 1.556e-06 1.556e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.179e-06 1.179e-06 1.179e-06 0.00% Castro::FluxRegFineAdd() 5 1.152e-06 1.152e-06 1.152e-06 0.00% AmrLevel::AmrLevel() 1 6.67e-07 6.67e-07 6.67e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-9-g67384701a808) finalized