Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-1-gfb0b31e1439b) initialized Starting run at 08:42:35 UTC on 2022-09-05. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09-1-gfb0b31e14 Microphysics git describe: 22.08-10-g65622313 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.052690354 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030251656 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.04939048 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052651596 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.054356465 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.069282958 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.066773221 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.04902862 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.073181562 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.071686669 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058722012 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.049363813 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.056257801 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.049332745 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.030990624 seconds Ending run at 08:42:36 UTC on 2022-09-05. Run time = 0.868159607 Run time without initialization = 0.731649287 Average number of zones advanced per microsecond: 3.583 Average number of zones advanced per microsecond per rank: 3.583 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8682 ... 0.8682 ... 0.8682 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2063 0.2063 0.2063 23.76% VisMF::Write(FabArray) 11 0.2042 0.2042 0.2042 23.52% MLCellLinOp::applyBC() 4433 0.08138 0.08138 0.08138 9.37% MLPoisson::Fsmooth() 3280 0.06457 0.06457 0.06457 7.44% MLCGSolver::bicgstab 82 0.0243 0.0243 0.0243 2.80% StateData::FillBoundary(geom) 328 0.02424 0.02424 0.02424 2.79% MultiFab::Dot() 1114 0.02275 0.02275 0.02275 2.62% Castro::computeTemp() 63 0.01543 0.01543 0.01543 1.78% Castro::normalize_species() 62 0.01512 0.01512 0.01512 1.74% MultiFab::LinComb() 1586 0.01464 0.01464 0.01464 1.69% FillBoundary_nowait() 4023 0.01445 0.01445 0.01445 1.66% FabArray::setVal() 1144 0.01439 0.01439 0.01439 1.66% FabArray::ParallelCopy_nowait() 861 0.01332 0.01332 0.01332 1.53% StateDataPhysBCFunct::() 41 0.01282 0.01282 0.01282 1.48% MLCellLinOp::defineAuxData() 11 0.01202 0.01202 0.01202 1.38% MLPoisson::Fapply() 1142 0.01195 0.01195 0.01195 1.38% Gravity::fill_multipole_BCs() 11 0.0104 0.0104 0.0104 1.20% Castro::enforce_min_density() 62 0.01008 0.01008 0.01008 1.16% MLMG::addInterpCorrection() 410 0.007957 0.007957 0.007957 0.92% amrex::average_down 410 0.007031 0.007031 0.007031 0.81% MultiFab::Xpay() 585 0.006648 0.006648 0.006648 0.77% Castro::estTimeStep() 21 0.006087 0.006087 0.006087 0.70% Castro::do_advance_ctu() 10 0.005288 0.005288 0.005288 0.61% Amr::checkPoint() 3 0.005249 0.005249 0.005249 0.60% Castro::reset_internal_energy(MultiFab) 63 0.004354 0.004354 0.004354 0.50% BndryData::define() 11 0.003985 0.003985 0.003985 0.46% Castro::construct_new_gravity_source() 10 0.003416 0.003416 0.003416 0.39% Castro::construct_old_gravity_source() 10 0.002904 0.002904 0.002904 0.33% Amr::writePlotFile() 2 0.00288 0.00288 0.00288 0.33% MLMG::ResNormInf() 93 0.002119 0.002119 0.002119 0.24% Gravity::get_new_grav_vector() 11 0.001952 0.001952 0.001952 0.22% MultiFab::Saxpy() 20 0.001814 0.001814 0.001814 0.21% Castro::expand_state() 10 0.001741 0.001741 0.001741 0.20% Gravity::get_old_grav_vector() 10 0.00174 0.00174 0.00174 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001694 0.001694 0.001694 0.20% MultiFab::Add() 82 0.001684 0.001684 0.001684 0.19% MLCellLinOp::setLevelBC() 11 0.001562 0.001562 0.001562 0.18% Castro::reset_internal_energy(Fab) 504 0.001542 0.001542 0.001542 0.18% Gravity::actual_solve_with_mlmg() 11 0.001474 0.001474 0.001474 0.17% FabArray::mult() 43 0.001338 0.001338 0.001338 0.15% FabArray::setDomainBndry() 41 0.001332 0.001332 0.001332 0.15% Castro::initData() 1 0.001321 0.001321 0.001321 0.15% Castro::enforce_speed_limit() 62 0.001289 0.001289 0.001289 0.15% MLMG::prepareForSolve() 11 0.001273 0.001273 0.001273 0.15% MLCellLinOp::prepareForSolve() 11 0.001196 0.001196 0.001196 0.14% MultiFab::contains_nan() 20 0.001184 0.001184 0.001184 0.14% MLCellLinOp::smooth() 1640 0.001164 0.001164 0.001164 0.13% MLCellLinOp::compGrad() 11 0.0009339 0.0009339 0.0009339 0.11% FabArray::FillBoundary() 4023 0.0008176 0.0008176 0.0008176 0.09% FabArrayBase::getCPC() 1323 0.0007705 0.0007705 0.0007705 0.09% FabArrayBase::CPC::define() 454 0.0006574 0.0006574 0.0006574 0.08% FabArrayBase::getFB() 4023 0.0005814 0.0005814 0.0005814 0.07% MLCellLinOp::apply() 1142 0.0005019 0.0005019 0.0005019 0.06% Amr::InitAmr() 1 0.0004706 0.0004706 0.0004706 0.05% Gravity::solve_for_phi() 10 0.0004258 0.0004258 0.0004258 0.05% Gravity::update_max_rhs() 11 0.000414 0.000414 0.000414 0.05% CGSolver::sxay() 1586 0.0003668 0.0003668 0.0003668 0.04% MultiFab::Copy() 11 0.0003285 0.0003285 0.0003285 0.04% Amr::coarseTimeStep() 10 0.0003178 0.0003178 0.0003178 0.04% FillPatchIterator::Initialize 41 0.0003133 0.0003133 0.0003133 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003026 0.0003026 0.0003026 0.03% MLCellLinOp::defineBC() 11 0.0002846 0.0002846 0.0002846 0.03% FabArray::ParallelCopy() 861 0.0002683 0.0002683 0.0002683 0.03% main() 1 0.0002622 0.0002622 0.0002622 0.03% MultiFab::max() 11 0.0002584 0.0002584 0.0002584 0.03% MLCellLinOp::correctionResidual() 492 0.0002403 0.0002403 0.0002403 0.03% MLMG::mgVcycle() 82 0.0002202 0.0002202 0.0002202 0.03% MLMG::MLRhsNormInf() 11 0.000218 0.000218 0.000218 0.03% Castro::construct_new_gravity() 10 0.0002048 0.0002048 0.0002048 0.02% MLLinOp::defineGrids() 11 0.0001764 0.0001764 0.0001764 0.02% Castro::subcycle_advance_ctu() 10 0.0001749 0.0001749 0.0001749 0.02% MLMG:computeResOfCorrection() 410 0.000158 0.000158 0.000158 0.02% Amr::timeStep() 10 0.000152 0.000152 0.000152 0.02% Castro::create_source_corrector() 10 0.0001495 0.0001495 0.0001495 0.02% StateData::checkPoint() 12 0.0001295 0.0001295 0.0001295 0.01% MLMG::mgVcycle_down::0 82 0.0001106 0.0001106 0.0001106 0.01% MLMG::mgVcycle_down::1 82 9.929e-05 9.929e-05 9.929e-05 0.01% MLMG::mgVcycle_down::2 82 9.578e-05 9.578e-05 9.578e-05 0.01% Castro::Castro() 1 9.365e-05 9.365e-05 9.365e-05 0.01% MLMG::mgVcycle_down::3 82 9.157e-05 9.157e-05 9.157e-05 0.01% MLMG::mgVcycle_down::4 82 9.066e-05 9.066e-05 9.066e-05 0.01% FabArrayBase::FB::FB() 56 8.84e-05 8.84e-05 8.84e-05 0.01% Castro::initialize_advance() 10 8.476e-05 8.476e-05 8.476e-05 0.01% MLMG::actualBottomSolve() 82 8.465e-05 8.465e-05 8.465e-05 0.01% Castro::clean_state() 62 8.306e-05 8.306e-05 8.306e-05 0.01% Castro::construct_new_source() 50 7.984e-05 7.984e-05 7.984e-05 0.01% AmrLevel::checkPoint() 3 7.253e-05 7.253e-05 7.253e-05 0.01% MLMG::mgVcycle_up::4 82 7.048e-05 7.048e-05 7.048e-05 0.01% Castro::initialize_do_advance() 10 6.42e-05 6.42e-05 6.42e-05 0.01% MLMG::solve() 11 6.394e-05 6.394e-05 6.394e-05 0.01% MLMG::mgVcycle_up::0 82 5.94e-05 5.94e-05 5.94e-05 0.01% MLMG::mgVcycle_up::3 82 5.915e-05 5.915e-05 5.915e-05 0.01% MLMG::oneIter() 82 5.837e-05 5.837e-05 5.837e-05 0.01% MLMG::mgVcycle_up::1 82 5.752e-05 5.752e-05 5.752e-05 0.01% MLMG::mgVcycle_up::2 82 5.712e-05 5.712e-05 5.712e-05 0.01% Castro::advance() 10 5.346e-05 5.346e-05 5.346e-05 0.01% Castro::finalize_advance() 10 5.249e-05 5.249e-05 5.249e-05 0.01% MLCellLinOp::solutionResidual() 93 5.121e-05 5.121e-05 5.121e-05 0.01% StateData::define() 4 4.557e-05 4.557e-05 4.557e-05 0.01% MLMG::computeResidual() 82 4.2e-05 4.2e-05 4.2e-05 0.00% Castro::swap_state_time_levels() 10 4.18e-05 4.18e-05 4.18e-05 0.00% Castro::enforce_consistent_e() 1 3.475e-05 3.475e-05 3.475e-05 0.00% Castro::finalize_do_advance() 10 3.462e-05 3.462e-05 3.462e-05 0.00% MLMG::mgVcycle_bottom 82 3.336e-05 3.336e-05 3.336e-05 0.00% Amr::defBaseLevel() 1 3.326e-05 3.326e-05 3.326e-05 0.00% Gravity::actual_multilevel_solve() 1 3.303e-05 3.303e-05 3.303e-05 0.00% DistributionMapping::Distribute() 56 3.143e-05 3.143e-05 3.143e-05 0.00% Castro::initMFs() 1 3.082e-05 3.082e-05 3.082e-05 0.00% FillPatchSingleLevel 41 3.013e-05 3.013e-05 3.013e-05 0.00% MLPoisson::define() 11 2.798e-05 2.798e-05 2.798e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.694e-05 2.694e-05 2.694e-05 0.00% Amr::writeSmallPlotFile() 1 2.672e-05 2.672e-05 2.672e-05 0.00% makeSFC 55 2.654e-05 2.654e-05 2.654e-05 0.00% Castro::buildMetrics() 1 2.518e-05 2.518e-05 2.518e-05 0.00% MLLinOp::define() 11 2.381e-05 2.381e-05 2.381e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.933e-05 1.933e-05 1.933e-05 0.00% Amr::FinalizeInit() 1 1.919e-05 1.919e-05 1.919e-05 0.00% Castro::construct_old_source() 50 1.883e-05 1.883e-05 1.883e-05 0.00% Castro::do_new_sources() 10 1.773e-05 1.773e-05 1.773e-05 0.00% Castro::do_old_sources() 10 1.618e-05 1.618e-05 1.618e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.461e-05 1.461e-05 1.461e-05 0.00% Castro::apply_source_to_state() 20 1.128e-05 1.128e-05 1.128e-05 0.00% MLMG::computeMLResidual() 11 1.078e-05 1.078e-05 1.078e-05 0.00% Castro::check_for_nan() 20 1.047e-05 1.047e-05 1.047e-05 0.00% Castro::construct_old_gravity() 10 1.005e-05 1.005e-05 1.005e-05 0.00% MLPoisson::prepareForSolve() 11 9.566e-06 9.566e-06 9.566e-06 0.00% Gravity::swapTimeLevels() 10 8.8e-06 8.8e-06 8.8e-06 0.00% Castro::post_timestep() 10 8.638e-06 8.638e-06 8.638e-06 0.00% Amr::initSubcycle() 1 8.508e-06 8.508e-06 8.508e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.229e-06 7.229e-06 7.229e-06 0.00% Castro::computeNewDt() 9 6.248e-06 6.248e-06 6.248e-06 0.00% MLMG::getGradSolution() 11 6.193e-06 6.193e-06 6.193e-06 0.00% Amr::InitializeInit() 1 4.582e-06 4.582e-06 4.582e-06 0.00% AmrLevel::checkPointPost() 3 4.193e-06 4.193e-06 4.193e-06 0.00% Castro::post_init() 1 4.024e-06 4.024e-06 4.024e-06 0.00% Castro::retry_advance_ctu() 10 3.831e-06 3.831e-06 3.831e-06 0.00% Gravity::set_mass_offset() 11 3.742e-06 3.742e-06 3.742e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.385e-06 3.385e-06 3.385e-06 0.00% MLMG::MLResNormInf() 11 3.353e-06 3.353e-06 3.353e-06 0.00% Castro::FluxRegCrseInit 10 3.014e-06 3.014e-06 3.014e-06 0.00% Castro::computeInitialDt() 2 2.925e-06 2.925e-06 2.925e-06 0.00% Amr::init() 1 2.83e-06 2.83e-06 2.83e-06 0.00% Castro::FluxRegFineAdd() 10 2.319e-06 2.319e-06 2.319e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.156e-06 2.156e-06 2.156e-06 0.00% AmrLevel::checkPointPre() 3 2.045e-06 2.045e-06 2.045e-06 0.00% Castro::post_regrid() 1 1.246e-06 1.246e-06 1.246e-06 0.00% Amr::initialInit() 1 1.141e-06 1.141e-06 1.141e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8682 0.8682 0.8682 100.00% Amr::coarseTimeStep() 10 0.7004 0.7004 0.7004 80.68% Amr::timeStep() 10 0.5982 0.5982 0.5982 68.90% Castro::advance() 10 0.5908 0.5908 0.5908 68.05% Castro::subcycle_advance_ctu() 10 0.5798 0.5798 0.5798 66.78% Castro::do_advance_ctu() 10 0.5796 0.5796 0.5796 66.76% Gravity::solve_phi_with_mlmg() 11 0.3217 0.3217 0.3217 37.05% Gravity::actual_solve_with_mlmg() 11 0.311 0.311 0.311 35.83% Castro::construct_new_gravity() 10 0.2926 0.2926 0.2926 33.70% MLMG::solve() 11 0.2877 0.2877 0.2877 33.14% Gravity::solve_for_phi() 10 0.2769 0.2769 0.2769 31.90% MLMG::oneIter() 82 0.2726 0.2726 0.2726 31.39% MLMG::mgVcycle() 82 0.2708 0.2708 0.2708 31.19% Castro::construct_ctu_hydro_source() 10 0.2063 0.2063 0.2063 23.76% VisMF::Write(FabArray) 11 0.2042 0.2042 0.2042 23.52% Amr::checkPoint() 3 0.1512 0.1512 0.1512 17.41% AmrLevel::checkPoint() 3 0.1459 0.1459 0.1459 16.81% StateData::checkPoint() 12 0.1459 0.1459 0.1459 16.80% MLCellLinOp::smooth() 1640 0.1381 0.1381 0.1381 15.91% Amr::init() 1 0.1359 0.1359 0.1359 15.66% MLCellLinOp::applyBC() 4433 0.09732 0.09732 0.09732 11.21% MLMG::mgVcycle_bottom 82 0.08346 0.08346 0.08346 9.61% MLMG::actualBottomSolve() 82 0.08342 0.08342 0.08342 9.61% MLCGSolver::bicgstab 82 0.0826 0.0826 0.0826 9.51% MLPoisson::Fsmooth() 3280 0.06457 0.06457 0.06457 7.44% Amr::writePlotFile() 2 0.06138 0.06138 0.06138 7.07% Amr::initialInit() 1 0.05287 0.05287 0.05287 6.09% Amr::FinalizeInit() 1 0.0485 0.0485 0.0485 5.59% Castro::post_init() 1 0.04713 0.04713 0.04713 5.43% Castro::clean_state() 62 0.04701 0.04701 0.04701 5.41% Gravity::multilevel_solve_for_new_phi() 1 0.04522 0.04522 0.04522 5.21% Gravity::actual_multilevel_solve() 1 0.0452 0.0452 0.0452 5.21% FillPatchIterator::Initialize 41 0.04269 0.04269 0.04269 4.92% FillPatchSingleLevel 41 0.04104 0.04104 0.04104 4.73% StateDataPhysBCFunct::() 41 0.03705 0.03705 0.03705 4.27% MLCellLinOp::apply() 1142 0.03687 0.03687 0.03687 4.25% MLMG::mgVcycle_down::0 82 0.03573 0.03573 0.03573 4.12% MLMG::mgVcycle_up::0 82 0.03072 0.03072 0.03072 3.54% StateData::FillBoundary(geom) 328 0.02424 0.02424 0.02424 2.79% MultiFab::Dot() 1114 0.02275 0.02275 0.02275 2.62% MLCellLinOp::correctionResidual() 492 0.02158 0.02158 0.02158 2.49% Castro::computeTemp() 63 0.02132 0.02132 0.02132 2.46% Castro::initialize_do_advance() 10 0.01966 0.01966 0.01966 2.26% MLPoisson::define() 11 0.01887 0.01887 0.01887 2.17% MLMG:computeResOfCorrection() 410 0.01864 0.01864 0.01864 2.15% MLMG::mgVcycle_down::1 82 0.01804 0.01804 0.01804 2.08% MLMG::mgVcycle_down::2 82 0.01752 0.01752 0.01752 2.02% Gravity::get_new_grav_vector() 11 0.01725 0.01725 0.01725 1.99% MLMG::mgVcycle_down::3 82 0.01668 0.01668 0.01668 1.92% FabArray::FillBoundary() 4023 0.01594 0.01594 0.01594 1.84% MLMG::mgVcycle_down::4 82 0.01587 0.01587 0.01587 1.83% Castro::construct_old_gravity() 10 0.01522 0.01522 0.01522 1.75% Gravity::get_old_grav_vector() 10 0.01521 0.01521 0.01521 1.75% Castro::normalize_species() 62 0.01512 0.01512 0.01512 1.74% FillBoundary_nowait() 4023 0.01512 0.01512 0.01512 1.74% CGSolver::sxay() 1586 0.01501 0.01501 0.01501 1.73% MultiFab::LinComb() 1586 0.01464 0.01464 0.01464 1.69% FabArray::ParallelCopy() 861 0.01441 0.01441 0.01441 1.66% FabArray::setVal() 1144 0.01439 0.01439 0.01439 1.66% FabArray::ParallelCopy_nowait() 861 0.01414 0.01414 0.01414 1.63% MLCGSolver::ParallelAllReduce 1514 0.01361 0.01361 0.01361 1.57% MLMG::mgVcycle_up::2 82 0.01354 0.01354 0.01354 1.56% MLCellLinOp::defineAuxData() 11 0.01336 0.01336 0.01336 1.54% MLMG::mgVcycle_up::1 82 0.01333 0.01333 0.01333 1.54% MLMG::addInterpCorrection() 410 0.01321 0.01321 0.01321 1.52% MLMG::mgVcycle_up::3 82 0.01287 0.01287 0.01287 1.48% MLMG::mgVcycle_up::4 82 0.01284 0.01284 0.01284 1.48% Castro::do_new_sources() 10 0.01269 0.01269 0.01269 1.46% amrex::average_down 410 0.01223 0.01223 0.01223 1.41% MLPoisson::Fapply() 1142 0.01195 0.01195 0.01195 1.38% Castro::do_old_sources() 10 0.01171 0.01171 0.01171 1.35% Castro::expand_state() 10 0.01137 0.01137 0.01137 1.31% Castro::initialize_advance() 10 0.01088 0.01088 0.01088 1.25% Gravity::fill_multipole_BCs() 11 0.0104 0.0104 0.0104 1.20% Castro::enforce_min_density() 62 0.01008 0.01008 0.01008 1.16% Castro::post_timestep() 10 0.00728 0.00728 0.00728 0.84% MLCellLinOp::solutionResidual() 93 0.007176 0.007176 0.007176 0.83% MultiFab::Xpay() 585 0.006648 0.006648 0.006648 0.77% MLMG::computeResidual() 82 0.006205 0.006205 0.006205 0.71% Castro::estTimeStep() 21 0.006087 0.006087 0.006087 0.70% Castro::reset_internal_energy(MultiFab) 63 0.005896 0.005896 0.005896 0.68% MLMG::prepareForSolve() 11 0.005487 0.005487 0.005487 0.63% MLCellLinOp::defineBC() 11 0.005228 0.005228 0.005228 0.60% BndryData::define() 11 0.004943 0.004943 0.004943 0.57% Amr::InitializeInit() 1 0.004363 0.004363 0.004363 0.50% Amr::defBaseLevel() 1 0.004359 0.004359 0.004359 0.50% Castro::initData() 1 0.003798 0.003798 0.003798 0.44% Castro::construct_new_source() 50 0.003496 0.003496 0.003496 0.40% Castro::construct_new_gravity_source() 10 0.003416 0.003416 0.003416 0.39% Castro::computeNewDt() 9 0.002939 0.002939 0.002939 0.34% Castro::construct_old_source() 50 0.002923 0.002923 0.002923 0.34% Castro::construct_old_gravity_source() 10 0.002904 0.002904 0.002904 0.33% MLMG::ResNormInf() 93 0.002119 0.002119 0.002119 0.24% Castro::apply_source_to_state() 20 0.001825 0.001825 0.001825 0.21% MultiFab::Saxpy() 20 0.001814 0.001814 0.001814 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001694 0.001694 0.001694 0.20% MultiFab::Add() 82 0.001684 0.001684 0.001684 0.19% MLCellLinOp::setLevelBC() 11 0.001562 0.001562 0.001562 0.18% Castro::reset_internal_energy(Fab) 504 0.001542 0.001542 0.001542 0.18% MLMG::getGradSolution() 11 0.00145 0.00145 0.00145 0.17% MLCellLinOp::compGrad() 11 0.001444 0.001444 0.001444 0.17% FabArrayBase::getCPC() 1323 0.001428 0.001428 0.001428 0.16% FabArray::mult() 43 0.001338 0.001338 0.001338 0.15% FabArray::setDomainBndry() 41 0.001332 0.001332 0.001332 0.15% Castro::enforce_speed_limit() 62 0.001289 0.001289 0.001289 0.15% MLPoisson::prepareForSolve() 11 0.001206 0.001206 0.001206 0.14% MLCellLinOp::prepareForSolve() 11 0.001196 0.001196 0.001196 0.14% Castro::check_for_nan() 20 0.001194 0.001194 0.001194 0.14% MultiFab::contains_nan() 20 0.001184 0.001184 0.001184 0.14% Castro::post_regrid() 1 0.001164 0.001164 0.001164 0.13% MLMG::computeMLResidual() 11 0.001024 0.001024 0.001024 0.12% Gravity::update_max_rhs() 11 0.0008254 0.0008254 0.0008254 0.10% Castro::computeInitialDt() 2 0.0006878 0.0006878 0.0006878 0.08% FabArrayBase::getFB() 4023 0.0006698 0.0006698 0.0006698 0.08% FabArrayBase::CPC::define() 454 0.0006574 0.0006574 0.0006574 0.08% Amr::InitAmr() 1 0.0004791 0.0004791 0.0004791 0.06% Castro::Castro() 1 0.0004545 0.0004545 0.0004545 0.05% Gravity::swapTimeLevels() 10 0.0004435 0.0004435 0.0004435 0.05% MultiFab::Copy() 11 0.0003285 0.0003285 0.0003285 0.04% MLMG::MLResNormInf() 11 0.0002829 0.0002829 0.0002829 0.03% MultiFab::max() 11 0.0002584 0.0002584 0.0002584 0.03% MLLinOp::define() 11 0.0002578 0.0002578 0.0002578 0.03% MLLinOp::defineGrids() 11 0.000234 0.000234 0.000234 0.03% MLMG::MLRhsNormInf() 11 0.000218 0.000218 0.000218 0.03% Castro::buildMetrics() 1 0.0001626 0.0001626 0.0001626 0.02% Castro::create_source_corrector() 10 0.0001495 0.0001495 0.0001495 0.02% FabArrayBase::FB::FB() 56 8.84e-05 8.84e-05 8.84e-05 0.01% Castro::finalize_advance() 10 5.782e-05 5.782e-05 5.782e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.541e-05 5.541e-05 5.541e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.28e-05 5.28e-05 5.28e-05 0.01% StateData::define() 4 4.557e-05 4.557e-05 4.557e-05 0.01% Castro::swap_state_time_levels() 10 4.18e-05 4.18e-05 4.18e-05 0.00% makeSFC 55 4.08e-05 4.08e-05 4.08e-05 0.00% Castro::enforce_consistent_e() 1 3.475e-05 3.475e-05 3.475e-05 0.00% Castro::finalize_do_advance() 10 3.462e-05 3.462e-05 3.462e-05 0.00% DistributionMapping::Distribute() 56 3.143e-05 3.143e-05 3.143e-05 0.00% Castro::initMFs() 1 3.082e-05 3.082e-05 3.082e-05 0.00% Amr::writeSmallPlotFile() 1 2.672e-05 2.672e-05 2.672e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.055e-05 2.055e-05 2.055e-05 0.00% Amr::initSubcycle() 1 8.508e-06 8.508e-06 8.508e-06 0.00% AmrLevel::checkPointPost() 3 4.193e-06 4.193e-06 4.193e-06 0.00% Castro::retry_advance_ctu() 10 3.831e-06 3.831e-06 3.831e-06 0.00% Gravity::set_mass_offset() 11 3.742e-06 3.742e-06 3.742e-06 0.00% Castro::FluxRegCrseInit 10 3.014e-06 3.014e-06 3.014e-06 0.00% Castro::FluxRegFineAdd() 10 2.319e-06 2.319e-06 2.319e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.156e-06 2.156e-06 2.156e-06 0.00% AmrLevel::checkPointPre() 3 2.045e-06 2.045e-06 2.045e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-1-gfb0b31e1439b) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-1-gfb0b31e1439b) initialized Starting run at 08:42:37 UTC on 2022-09-05. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09-1-gfb0b31e14 Microphysics git describe: 22.08-10-g65622313 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.478019122 Restart time = 0.048535102 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.0524746 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.052564533 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.051555782 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.061251431 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.072520073 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032282911 seconds Ending run at 08:42:37 UTC on 2022-09-05. Run time = 0.372200781 Run time without initialization = 0.323059011 Average number of zones advanced per microsecond: 4.057 Average number of zones advanced per microsecond per rank: 4.057 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3722 ... 0.3722 ... 0.3722 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0903 0.0903 0.0903 24.26% VisMF::Read() 3 0.04048 0.04048 0.04048 10.88% MLCellLinOp::applyBC() 1946 0.03616 0.03616 0.03616 9.71% VisMF::Write(FabArray) 1 0.03073 0.03073 0.03073 8.26% MLPoisson::Fsmooth() 1440 0.02757 0.02757 0.02757 7.41% StateData::FillBoundary(geom) 160 0.01181 0.01181 0.01181 3.17% MLCGSolver::bicgstab 36 0.01047 0.01047 0.01047 2.81% MultiFab::Dot() 484 0.009821 0.009821 0.009821 2.64% FabArray::setVal() 537 0.006891 0.006891 0.006891 1.85% Castro::normalize_species() 30 0.0066 0.0066 0.0066 1.77% MLCellLinOp::defineAuxData() 6 0.006487 0.006487 0.006487 1.74% FillBoundary_nowait() 1766 0.006423 0.006423 0.006423 1.73% FabArray::ParallelCopy_nowait() 380 0.006229 0.006229 0.006229 1.67% MultiFab::LinComb() 690 0.006219 0.006219 0.006219 1.67% Castro::enforce_min_density() 30 0.005892 0.005892 0.005892 1.58% Gravity::fill_multipole_BCs() 6 0.005747 0.005747 0.005747 1.54% StateDataPhysBCFunct::() 20 0.00538 0.00538 0.00538 1.45% Castro::computeTemp() 30 0.005299 0.005299 0.005299 1.42% MLPoisson::Fapply() 500 0.005134 0.005134 0.005134 1.38% Amr::restart() 1 0.00365 0.00365 0.00365 0.98% MLMG::addInterpCorrection() 180 0.003491 0.003491 0.003491 0.94% Castro::estTimeStep() 10 0.003278 0.003278 0.003278 0.88% amrex::average_down 180 0.003096 0.003096 0.003096 0.83% MultiFab::Xpay() 258 0.00291 0.00291 0.00291 0.78% Castro::reset_internal_energy(MultiFab) 30 0.002338 0.002338 0.002338 0.63% BndryData::define() 6 0.002191 0.002191 0.002191 0.59% Castro::do_advance_ctu() 5 0.002083 0.002083 0.002083 0.56% Castro::construct_new_gravity_source() 5 0.001674 0.001674 0.001674 0.45% Amr::writePlotFile() 1 0.001652 0.001652 0.001652 0.44% Castro::construct_old_gravity_source() 5 0.001372 0.001372 0.001372 0.37% Castro::subcycle_advance_ctu() 5 0.001241 0.001241 0.001241 0.33% MLMG::ResNormInf() 42 0.0009371 0.0009371 0.0009371 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009274 0.0009274 0.0009274 0.25% Gravity::get_old_grav_vector() 5 0.0009197 0.0009197 0.0009197 0.25% MultiFab::Saxpy() 10 0.0009193 0.0009193 0.0009193 0.25% Castro::expand_state() 5 0.0008768 0.0008768 0.0008768 0.24% Gravity::get_new_grav_vector() 5 0.0008721 0.0008721 0.0008721 0.23% MLCellLinOp::setLevelBC() 6 0.0008392 0.0008392 0.0008392 0.23% Castro::reset_internal_energy(Fab) 240 0.0008174 0.0008174 0.0008174 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007403 0.0007403 0.0007403 0.20% MultiFab::Add() 36 0.0007235 0.0007235 0.0007235 0.19% MLMG::prepareForSolve() 6 0.0006881 0.0006881 0.0006881 0.18% MLCellLinOp::prepareForSolve() 6 0.0006585 0.0006585 0.0006585 0.18% FabArray::mult() 22 0.0006552 0.0006552 0.0006552 0.18% FabArray::setDomainBndry() 20 0.0006427 0.0006427 0.0006427 0.17% MultiFab::contains_nan() 10 0.0005951 0.0005951 0.0005951 0.16% MLCellLinOp::smooth() 720 0.0005698 0.0005698 0.0005698 0.15% Castro::enforce_speed_limit() 30 0.0005334 0.0005334 0.0005334 0.14% MLCellLinOp::compGrad() 6 0.0004954 0.0004954 0.0004954 0.13% Amr::InitAmr() 1 0.0004252 0.0004252 0.0004252 0.11% FabArrayBase::CPC::define() 244 0.0003907 0.0003907 0.0003907 0.10% FabArray::FillBoundary() 1766 0.0003778 0.0003778 0.0003778 0.10% FabArrayBase::getCPC() 632 0.000373 0.000373 0.000373 0.10% main() 1 0.0002681 0.0002681 0.0002681 0.07% FabArrayBase::getFB() 1766 0.0002529 0.0002529 0.0002529 0.07% MLCellLinOp::apply() 500 0.0002359 0.0002359 0.0002359 0.06% Gravity::update_max_rhs() 6 0.0002329 0.0002329 0.0002329 0.06% Gravity::solve_for_phi() 5 0.0002115 0.0002115 0.0002115 0.06% MultiFab::Copy() 6 0.0001785 0.0001785 0.0001785 0.05% CGSolver::sxay() 690 0.0001727 0.0001727 0.0001727 0.05% Amr::coarseTimeStep() 5 0.0001616 0.0001616 0.0001616 0.04% MLCellLinOp::defineBC() 6 0.0001492 0.0001492 0.0001492 0.04% FillPatchIterator::Initialize 20 0.0001438 0.0001438 0.0001438 0.04% MultiFab::max() 6 0.0001359 0.0001359 0.0001359 0.04% MLCGSolver::ParallelAllReduce 659 0.000121 0.000121 0.000121 0.03% FabArray::ParallelCopy() 380 0.0001198 0.0001198 0.0001198 0.03% Castro::construct_new_gravity() 5 0.0001166 0.0001166 0.0001166 0.03% MLMG::MLRhsNormInf() 6 0.0001149 0.0001149 0.0001149 0.03% MLCellLinOp::correctionResidual() 216 0.0001096 0.0001096 0.0001096 0.03% MLMG::mgVcycle() 36 0.0001088 0.0001088 0.0001088 0.03% Amr::timeStep() 5 9.341e-05 9.341e-05 9.341e-05 0.03% Castro::create_source_corrector() 5 8.822e-05 8.822e-05 8.822e-05 0.02% StateData::restartDoit() 4 7.454e-05 7.454e-05 7.454e-05 0.02% AmrLevel::restart() 1 7.44e-05 7.44e-05 7.44e-05 0.02% MLLinOp::defineGrids() 6 7.19e-05 7.19e-05 7.19e-05 0.02% MLMG:computeResOfCorrection() 180 7.045e-05 7.045e-05 7.045e-05 0.02% Castro::finalize_advance() 5 5.838e-05 5.838e-05 5.838e-05 0.02% FabArrayBase::FB::FB() 26 5.814e-05 5.814e-05 5.814e-05 0.02% Castro::construct_new_source() 25 5.537e-05 5.537e-05 5.537e-05 0.01% MLMG::mgVcycle_down::0 36 5.19e-05 5.19e-05 5.19e-05 0.01% MLMG::mgVcycle_down::1 36 4.925e-05 4.925e-05 4.925e-05 0.01% MLMG::mgVcycle_down::3 36 4.852e-05 4.852e-05 4.852e-05 0.01% MLMG::mgVcycle_down::2 36 4.777e-05 4.777e-05 4.777e-05 0.01% MLMG::mgVcycle_down::4 36 4.582e-05 4.582e-05 4.582e-05 0.01% Castro::clean_state() 30 4.343e-05 4.343e-05 4.343e-05 0.01% Castro::initialize_advance() 5 4.262e-05 4.262e-05 4.262e-05 0.01% MLMG::actualBottomSolve() 36 3.729e-05 3.729e-05 3.729e-05 0.01% Castro::initMFs() 1 3.639e-05 3.639e-05 3.639e-05 0.01% MLMG::mgVcycle_up::4 36 3.58e-05 3.58e-05 3.58e-05 0.01% Castro::buildMetrics() 1 3.42e-05 3.42e-05 3.42e-05 0.01% Castro::initialize_do_advance() 5 3.372e-05 3.372e-05 3.372e-05 0.01% MLMG::solve() 6 3.283e-05 3.283e-05 3.283e-05 0.01% Castro::post_restart() 1 3.218e-05 3.218e-05 3.218e-05 0.01% Gravity::actual_multilevel_solve() 1 3.122e-05 3.122e-05 3.122e-05 0.01% MLMG::mgVcycle_up::0 36 2.996e-05 2.996e-05 2.996e-05 0.01% MLMG::mgVcycle_up::3 36 2.927e-05 2.927e-05 2.927e-05 0.01% Castro::advance() 5 2.908e-05 2.908e-05 2.908e-05 0.01% MLMG::mgVcycle_up::2 36 2.835e-05 2.835e-05 2.835e-05 0.01% Castro::swap_state_time_levels() 5 2.82e-05 2.82e-05 2.82e-05 0.01% MLMG::oneIter() 36 2.803e-05 2.803e-05 2.803e-05 0.01% Amr::writeSmallPlotFile() 1 2.773e-05 2.773e-05 2.773e-05 0.01% MLMG::mgVcycle_up::1 36 2.762e-05 2.762e-05 2.762e-05 0.01% Castro::construct_old_source() 25 2.449e-05 2.449e-05 2.449e-05 0.01% MLCellLinOp::solutionResidual() 42 2.312e-05 2.312e-05 2.312e-05 0.01% MLPoisson::define() 6 2.103e-05 2.103e-05 2.103e-05 0.01% MLLinOp::define() 6 2.057e-05 2.057e-05 2.057e-05 0.01% Gravity::solve_phi_with_mlmg() 6 2.011e-05 2.011e-05 2.011e-05 0.01% Castro::finalize_do_advance() 5 1.877e-05 1.877e-05 1.877e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.813e-05 1.813e-05 1.813e-05 0.00% MLMG::computeResidual() 36 1.799e-05 1.799e-05 1.799e-05 0.00% MLMG::mgVcycle_bottom 36 1.567e-05 1.567e-05 1.567e-05 0.00% makeSFC 30 1.483e-05 1.483e-05 1.483e-05 0.00% FillPatchSingleLevel 20 1.404e-05 1.404e-05 1.404e-05 0.00% Castro::do_new_sources() 5 9.861e-06 9.861e-06 9.861e-06 0.00% Amr::initSubcycle() 1 9.625e-06 9.625e-06 9.625e-06 0.00% DistributionMapping::Distribute() 31 9.292e-06 9.292e-06 9.292e-06 0.00% Castro::do_old_sources() 5 8.623e-06 8.623e-06 8.623e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.236e-06 7.236e-06 7.236e-06 0.00% Castro::check_for_nan() 10 7.099e-06 7.099e-06 7.099e-06 0.00% Castro::construct_old_gravity() 5 5.938e-06 5.938e-06 5.938e-06 0.00% Castro::post_timestep() 5 5.772e-06 5.772e-06 5.772e-06 0.00% Castro::apply_source_to_state() 10 5.604e-06 5.604e-06 5.604e-06 0.00% MLPoisson::prepareForSolve() 6 5.254e-06 5.254e-06 5.254e-06 0.00% Gravity::swapTimeLevels() 5 4.663e-06 4.663e-06 4.663e-06 0.00% MLMG::computeMLResidual() 6 4.54e-06 4.54e-06 4.54e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.691e-06 3.691e-06 3.691e-06 0.00% Castro::computeNewDt() 5 3.547e-06 3.547e-06 3.547e-06 0.00% MLMG::getGradSolution() 6 3.085e-06 3.085e-06 3.085e-06 0.00% Castro::retry_advance_ctu() 5 2.857e-06 2.857e-06 2.857e-06 0.00% Gravity::set_mass_offset() 6 2.16e-06 2.16e-06 2.16e-06 0.00% Castro::FluxRegCrseInit 5 2.123e-06 2.123e-06 2.123e-06 0.00% MLMG::MLResNormInf() 6 2.091e-06 2.091e-06 2.091e-06 0.00% Castro::FluxRegFineAdd() 5 1.506e-06 1.506e-06 1.506e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.245e-06 1.245e-06 1.245e-06 0.00% Amr::init() 1 1.178e-06 1.178e-06 1.178e-06 0.00% AmrLevel::AmrLevel() 1 8.41e-07 8.41e-07 8.41e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3722 0.3722 0.3722 100.00% Amr::coarseTimeStep() 5 0.2905 0.2905 0.2905 78.05% Amr::timeStep() 5 0.2882 0.2882 0.2882 77.44% Castro::advance() 5 0.2851 0.2851 0.2851 76.59% Castro::subcycle_advance_ctu() 5 0.2788 0.2788 0.2788 74.89% Castro::do_advance_ctu() 5 0.2775 0.2775 0.2775 74.56% Castro::construct_new_gravity() 5 0.1489 0.1489 0.1489 40.00% Gravity::solve_phi_with_mlmg() 6 0.1447 0.1447 0.1447 38.86% Gravity::solve_for_phi() 5 0.141 0.141 0.141 37.88% Gravity::actual_solve_with_mlmg() 6 0.1388 0.1388 0.1388 37.28% MLMG::solve() 6 0.1261 0.1261 0.1261 33.89% MLMG::oneIter() 36 0.1188 0.1188 0.1188 31.90% MLMG::mgVcycle() 36 0.118 0.118 0.118 31.70% Castro::construct_ctu_hydro_source() 5 0.09029 0.09029 0.09029 24.26% MLCellLinOp::smooth() 720 0.06018 0.06018 0.06018 16.17% Amr::init() 1 0.04858 0.04858 0.04858 13.05% Amr::restart() 1 0.04858 0.04858 0.04858 13.05% MLCellLinOp::applyBC() 1946 0.04327 0.04327 0.04327 11.62% AmrLevel::restart() 1 0.04069 0.04069 0.04069 10.93% StateData::restartDoit() 4 0.04061 0.04061 0.04061 10.91% VisMF::Read() 3 0.04048 0.04048 0.04048 10.88% MLMG::mgVcycle_bottom 36 0.03601 0.03601 0.03601 9.67% MLMG::actualBottomSolve() 36 0.03599 0.03599 0.03599 9.67% MLCGSolver::bicgstab 36 0.03563 0.03563 0.03563 9.57% Amr::writePlotFile() 1 0.03238 0.03238 0.03238 8.70% VisMF::Write(FabArray) 1 0.03073 0.03073 0.03073 8.26% MLPoisson::Fsmooth() 1440 0.02757 0.02757 0.02757 7.41% Castro::clean_state() 30 0.02152 0.02152 0.02152 5.78% FillPatchIterator::Initialize 20 0.02002 0.02002 0.02002 5.38% FillPatchSingleLevel 20 0.01923 0.01923 0.01923 5.17% StateDataPhysBCFunct::() 20 0.01719 0.01719 0.01719 4.62% MLCellLinOp::apply() 500 0.01631 0.01631 0.01631 4.38% MLMG::mgVcycle_down::0 36 0.01576 0.01576 0.01576 4.23% MLMG::mgVcycle_up::0 36 0.01347 0.01347 0.01347 3.62% StateData::FillBoundary(geom) 160 0.01181 0.01181 0.01181 3.17% MLPoisson::define() 6 0.01028 0.01028 0.01028 2.76% MultiFab::Dot() 484 0.009821 0.009821 0.009821 2.64% MLCellLinOp::correctionResidual() 216 0.009487 0.009487 0.009487 2.55% Castro::initialize_do_advance() 5 0.009184 0.009184 0.009184 2.47% Castro::computeTemp() 30 0.008454 0.008454 0.008454 2.27% MLMG:computeResOfCorrection() 180 0.008196 0.008196 0.008196 2.20% MLMG::mgVcycle_down::1 36 0.007885 0.007885 0.007885 2.12% Gravity::get_new_grav_vector() 5 0.007781 0.007781 0.007781 2.09% MLMG::mgVcycle_down::2 36 0.007642 0.007642 0.007642 2.05% Castro::construct_old_gravity() 5 0.007364 0.007364 0.007364 1.98% Gravity::get_old_grav_vector() 5 0.007358 0.007358 0.007358 1.98% MLMG::mgVcycle_down::3 36 0.007297 0.007297 0.007297 1.96% MLCellLinOp::defineAuxData() 6 0.007243 0.007243 0.007243 1.95% FabArray::FillBoundary() 1766 0.007112 0.007112 0.007112 1.91% MLMG::mgVcycle_down::4 36 0.006995 0.006995 0.006995 1.88% FabArray::setVal() 537 0.006891 0.006891 0.006891 1.85% FillBoundary_nowait() 1766 0.006734 0.006734 0.006734 1.81% FabArray::ParallelCopy() 380 0.006732 0.006732 0.006732 1.81% Castro::do_new_sources() 5 0.006636 0.006636 0.006636 1.78% FabArray::ParallelCopy_nowait() 380 0.006613 0.006613 0.006613 1.78% Castro::normalize_species() 30 0.0066 0.0066 0.0066 1.77% CGSolver::sxay() 690 0.006391 0.006391 0.006391 1.72% MultiFab::LinComb() 690 0.006219 0.006219 0.006219 1.67% Castro::initialize_advance() 5 0.006209 0.006209 0.006209 1.67% Castro::enforce_min_density() 30 0.005892 0.005892 0.005892 1.58% MLMG::mgVcycle_up::2 36 0.005878 0.005878 0.005878 1.58% MLCGSolver::ParallelAllReduce 659 0.005878 0.005878 0.005878 1.58% MLMG::addInterpCorrection() 180 0.005844 0.005844 0.005844 1.57% MLMG::mgVcycle_up::1 36 0.005777 0.005777 0.005777 1.55% Gravity::fill_multipole_BCs() 6 0.005747 0.005747 0.005747 1.54% MLMG::mgVcycle_up::4 36 0.005595 0.005595 0.005595 1.50% MLMG::mgVcycle_up::3 36 0.005594 0.005594 0.005594 1.50% Castro::expand_state() 5 0.005481 0.005481 0.005481 1.47% amrex::average_down 180 0.00545 0.00545 0.00545 1.46% MLPoisson::Fapply() 500 0.005134 0.005134 0.005134 1.38% Castro::do_old_sources() 5 0.005067 0.005067 0.005067 1.36% Castro::post_restart() 1 0.004042 0.004042 0.004042 1.09% Gravity::multilevel_solve_for_new_phi() 1 0.003916 0.003916 0.003916 1.05% Gravity::actual_multilevel_solve() 1 0.003898 0.003898 0.003898 1.05% Castro::estTimeStep() 10 0.003278 0.003278 0.003278 0.88% MLCellLinOp::solutionResidual() 42 0.003272 0.003272 0.003272 0.88% Castro::reset_internal_energy(MultiFab) 30 0.003155 0.003155 0.003155 0.85% Castro::post_timestep() 5 0.003076 0.003076 0.003076 0.83% MLMG::prepareForSolve() 6 0.003001 0.003001 0.003001 0.81% MultiFab::Xpay() 258 0.00291 0.00291 0.00291 0.78% MLCellLinOp::defineBC() 6 0.002891 0.002891 0.002891 0.78% BndryData::define() 6 0.002742 0.002742 0.002742 0.74% MLMG::computeResidual() 36 0.002717 0.002717 0.002717 0.73% Castro::computeNewDt() 5 0.002112 0.002112 0.002112 0.57% Castro::construct_new_source() 25 0.001729 0.001729 0.001729 0.46% Castro::construct_new_gravity_source() 5 0.001674 0.001674 0.001674 0.45% Castro::construct_old_source() 25 0.001396 0.001396 0.001396 0.38% Castro::construct_old_gravity_source() 5 0.001372 0.001372 0.001372 0.37% MLMG::ResNormInf() 42 0.0009371 0.0009371 0.0009371 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009274 0.0009274 0.0009274 0.25% Castro::apply_source_to_state() 10 0.0009249 0.0009249 0.0009249 0.25% MultiFab::Saxpy() 10 0.0009193 0.0009193 0.0009193 0.25% MLCellLinOp::setLevelBC() 6 0.0008392 0.0008392 0.0008392 0.23% Castro::reset_internal_energy(Fab) 240 0.0008174 0.0008174 0.0008174 0.22% MLMG::getGradSolution() 6 0.0007821 0.0007821 0.0007821 0.21% MLCellLinOp::compGrad() 6 0.0007791 0.0007791 0.0007791 0.21% FabArrayBase::getCPC() 632 0.0007636 0.0007636 0.0007636 0.21% MultiFab::Add() 36 0.0007235 0.0007235 0.0007235 0.19% MLPoisson::prepareForSolve() 6 0.0006638 0.0006638 0.0006638 0.18% MLCellLinOp::prepareForSolve() 6 0.0006585 0.0006585 0.0006585 0.18% FabArray::mult() 22 0.0006552 0.0006552 0.0006552 0.18% FabArray::setDomainBndry() 20 0.0006427 0.0006427 0.0006427 0.17% Castro::check_for_nan() 10 0.0006022 0.0006022 0.0006022 0.16% MultiFab::contains_nan() 10 0.0005951 0.0005951 0.0005951 0.16% MLMG::computeMLResidual() 6 0.0005776 0.0005776 0.0005776 0.16% Castro::enforce_speed_limit() 30 0.0005334 0.0005334 0.0005334 0.14% Gravity::update_max_rhs() 6 0.0004473 0.0004473 0.0004473 0.12% Amr::InitAmr() 1 0.0004348 0.0004348 0.0004348 0.12% FabArrayBase::CPC::define() 244 0.0003907 0.0003907 0.0003907 0.10% FabArrayBase::getFB() 1766 0.000311 0.000311 0.000311 0.08% Gravity::swapTimeLevels() 5 0.0002359 0.0002359 0.0002359 0.06% MultiFab::Copy() 6 0.0001785 0.0001785 0.0001785 0.05% Castro::buildMetrics() 1 0.0001573 0.0001573 0.0001573 0.04% MLMG::MLResNormInf() 6 0.0001504 0.0001504 0.0001504 0.04% MultiFab::max() 6 0.0001359 0.0001359 0.0001359 0.04% MLLinOp::define() 6 0.0001234 0.0001234 0.0001234 0.03% MLMG::MLRhsNormInf() 6 0.0001149 0.0001149 0.0001149 0.03% MLLinOp::defineGrids() 6 0.0001028 0.0001028 0.0001028 0.03% Castro::create_source_corrector() 5 8.822e-05 8.822e-05 8.822e-05 0.02% Castro::finalize_advance() 5 6.2e-05 6.2e-05 6.2e-05 0.02% FabArrayBase::FB::FB() 26 5.814e-05 5.814e-05 5.814e-05 0.02% Castro::initMFs() 1 3.639e-05 3.639e-05 3.639e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.966e-05 2.966e-05 2.966e-05 0.01% Castro::swap_state_time_levels() 5 2.82e-05 2.82e-05 2.82e-05 0.01% Amr::writeSmallPlotFile() 1 2.773e-05 2.773e-05 2.773e-05 0.01% makeSFC 30 2.243e-05 2.243e-05 2.243e-05 0.01% Castro::finalize_do_advance() 5 1.877e-05 1.877e-05 1.877e-05 0.01% Amr::initSubcycle() 1 9.625e-06 9.625e-06 9.625e-06 0.00% DistributionMapping::Distribute() 31 9.292e-06 9.292e-06 9.292e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.391e-06 5.391e-06 5.391e-06 0.00% Castro::retry_advance_ctu() 5 2.857e-06 2.857e-06 2.857e-06 0.00% Gravity::set_mass_offset() 6 2.16e-06 2.16e-06 2.16e-06 0.00% Castro::FluxRegCrseInit 5 2.123e-06 2.123e-06 2.123e-06 0.00% Castro::FluxRegFineAdd() 5 1.506e-06 1.506e-06 1.506e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.245e-06 1.245e-06 1.245e-06 0.00% AmrLevel::AmrLevel() 1 8.41e-07 8.41e-07 8.41e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-1-gfb0b31e1439b) finalized