Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-5-g6ebf8ffc2689) initialized Starting run at 08:57:55 UTC on 2022-08-05. Successfully read inputs file ... Castro git describe: 22.08-1-g4c16aad67 AMReX git describe: 22.08-5-g6ebf8ffc2 Microphysics git describe: 22.08-1-gffefdd62 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.05326284 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.046576125 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048885807 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050368555 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.058726566 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.050490631 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.072242882 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.052179944 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.063975804 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.071074207 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.063573694 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.061617371 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.058047429 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.073850061 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.030823316 seconds Ending run at 08:57:56 UTC on 2022-08-05. Run time = 0.909404059 Run time without initialization = 0.756533026 Average number of zones advanced per microsecond: 3.465 Average number of zones advanced per microsecond per rank: 3.465 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.9094 ... 0.9094 ... 0.9094 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2236 0.2236 0.2236 24.58% Castro::construct_ctu_hydro_source() 10 0.2059 0.2059 0.2059 22.64% MLCellLinOp::applyBC() 4433 0.08492 0.08492 0.08492 9.34% MLPoisson::Fsmooth() 3280 0.06513 0.06513 0.06513 7.16% Amr::checkPoint() 3 0.03032 0.03032 0.03032 3.33% StateData::FillBoundary(geom) 328 0.02534 0.02534 0.02534 2.79% MLCGSolver::bicgstab 82 0.02449 0.02449 0.02449 2.69% MultiFab::Dot() 1114 0.02293 0.02293 0.02293 2.52% MultiFab::LinComb() 1586 0.01477 0.01477 0.01477 1.62% FabArray::setVal() 1144 0.01453 0.01453 0.01453 1.60% Castro::computeTemp() 63 0.0144 0.0144 0.0144 1.58% FillBoundary_nowait() 4023 0.01433 0.01433 0.01433 1.58% Castro::normalize_species() 62 0.01369 0.01369 0.01369 1.51% FabArray::ParallelCopy_nowait() 861 0.01341 0.01341 0.01341 1.47% MLPoisson::Fapply() 1142 0.0121 0.0121 0.0121 1.33% StateDataPhysBCFunct::() 41 0.012 0.012 0.012 1.32% MLCellLinOp::defineAuxData() 11 0.01174 0.01174 0.01174 1.29% Castro::enforce_min_density() 62 0.009793 0.009793 0.009793 1.08% Gravity::fill_multipole_BCs() 11 0.008655 0.008655 0.008655 0.95% MLMG::addInterpCorrection() 410 0.007879 0.007879 0.007879 0.87% amrex::average_down 410 0.00698 0.00698 0.00698 0.77% MultiFab::Xpay() 585 0.006756 0.006756 0.006756 0.74% Castro::do_advance_ctu() 10 0.005029 0.005029 0.005029 0.55% Castro::estTimeStep() 21 0.004954 0.004954 0.004954 0.54% Castro::reset_internal_energy(MultiFab) 63 0.004047 0.004047 0.004047 0.45% BndryData::define() 11 0.003979 0.003979 0.003979 0.44% Amr::writePlotFile() 2 0.002884 0.002884 0.002884 0.32% Castro::construct_new_gravity_source() 10 0.00269 0.00269 0.00269 0.30% Castro::construct_old_gravity_source() 10 0.002243 0.002243 0.002243 0.25% MLMG::ResNormInf() 93 0.002095 0.002095 0.002095 0.23% Gravity::get_new_grav_vector() 11 0.001961 0.001961 0.001961 0.22% MultiFab::Saxpy() 20 0.001817 0.001817 0.001817 0.20% Gravity::get_old_grav_vector() 10 0.001746 0.001746 0.001746 0.19% Castro::expand_state() 10 0.001736 0.001736 0.001736 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001695 0.001695 0.001695 0.19% MultiFab::Add() 82 0.00169 0.00169 0.00169 0.19% MLCellLinOp::setLevelBC() 11 0.001577 0.001577 0.001577 0.17% Castro::reset_internal_energy(Fab) 504 0.001542 0.001542 0.001542 0.17% Castro::enforce_speed_limit() 62 0.00152 0.00152 0.00152 0.17% Gravity::actual_solve_with_mlmg() 11 0.001396 0.001396 0.001396 0.15% FabArray::mult() 43 0.001336 0.001336 0.001336 0.15% FabArray::setDomainBndry() 41 0.001323 0.001323 0.001323 0.15% MLMG::prepareForSolve() 11 0.00127 0.00127 0.00127 0.14% Castro::initData() 1 0.001234 0.001234 0.001234 0.14% MLCellLinOp::prepareForSolve() 11 0.001196 0.001196 0.001196 0.13% MultiFab::contains_nan() 20 0.001175 0.001175 0.001175 0.13% MLCellLinOp::smooth() 1640 0.001149 0.001149 0.001149 0.13% MLCellLinOp::compGrad() 11 0.0009537 0.0009537 0.0009537 0.10% FabArray::FillBoundary() 4023 0.0008529 0.0008529 0.0008529 0.09% FabArrayBase::getCPC() 1323 0.0007912 0.0007912 0.0007912 0.09% FabArrayBase::CPC::define() 454 0.0006886 0.0006886 0.0006886 0.08% FabArrayBase::getFB() 4023 0.0005771 0.0005771 0.0005771 0.06% Gravity::solve_for_phi() 10 0.0004857 0.0004857 0.0004857 0.05% Amr::InitAmr() 1 0.00048 0.00048 0.00048 0.05% MLCellLinOp::apply() 1142 0.0004531 0.0004531 0.0004531 0.05% Gravity::update_max_rhs() 11 0.0004115 0.0004115 0.0004115 0.05% CGSolver::sxay() 1586 0.0003784 0.0003784 0.0003784 0.04% Amr::coarseTimeStep() 10 0.0003463 0.0003463 0.0003463 0.04% MultiFab::Copy() 11 0.0003357 0.0003357 0.0003357 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003022 0.0003022 0.0003022 0.03% FillPatchIterator::Initialize 41 0.000295 0.000295 0.000295 0.03% MLCellLinOp::defineBC() 11 0.0002823 0.0002823 0.0002823 0.03% main() 1 0.0002761 0.0002761 0.0002761 0.03% MultiFab::max() 11 0.0002668 0.0002668 0.0002668 0.03% FabArray::ParallelCopy() 861 0.0002657 0.0002657 0.0002657 0.03% Castro::subcycle_advance_ctu() 10 0.0002428 0.0002428 0.0002428 0.03% MLMG::MLRhsNormInf() 11 0.0002204 0.0002204 0.0002204 0.02% MLCellLinOp::correctionResidual() 492 0.0002109 0.0002109 0.0002109 0.02% Castro::construct_new_gravity() 10 0.00021 0.00021 0.00021 0.02% Amr::timeStep() 10 0.0002045 0.0002045 0.0002045 0.02% MLMG::mgVcycle() 82 0.0001797 0.0001797 0.0001797 0.02% MLMG:computeResOfCorrection() 410 0.0001559 0.0001559 0.0001559 0.02% MLLinOp::defineGrids() 11 0.0001305 0.0001305 0.0001305 0.01% StateData::checkPoint() 12 0.0001275 0.0001275 0.0001275 0.01% MLMG::mgVcycle_down::0 82 0.0001067 0.0001067 0.0001067 0.01% MLMG::mgVcycle_down::1 82 8.919e-05 8.919e-05 8.919e-05 0.01% Castro::Castro() 1 8.876e-05 8.876e-05 8.876e-05 0.01% MLMG::actualBottomSolve() 82 8.766e-05 8.766e-05 8.766e-05 0.01% FabArrayBase::FB::FB() 56 8.558e-05 8.558e-05 8.558e-05 0.01% MLMG::mgVcycle_down::2 82 8.438e-05 8.438e-05 8.438e-05 0.01% Castro::initialize_advance() 10 8.241e-05 8.241e-05 8.241e-05 0.01% Castro::clean_state() 62 7.999e-05 7.999e-05 7.999e-05 0.01% MLMG::solve() 11 7.869e-05 7.869e-05 7.869e-05 0.01% AmrLevel::checkPoint() 3 7.77e-05 7.77e-05 7.77e-05 0.01% MLMG::mgVcycle_down::4 82 7.595e-05 7.595e-05 7.595e-05 0.01% MLMG::mgVcycle_down::3 82 7.579e-05 7.579e-05 7.579e-05 0.01% MLMG::mgVcycle_up::4 82 6.982e-05 6.982e-05 6.982e-05 0.01% Castro::initialize_do_advance() 10 6.453e-05 6.453e-05 6.453e-05 0.01% MLMG::oneIter() 82 6.17e-05 6.17e-05 6.17e-05 0.01% Castro::construct_new_source() 50 6.037e-05 6.037e-05 6.037e-05 0.01% MLMG::mgVcycle_up::0 82 5.845e-05 5.845e-05 5.845e-05 0.01% MLMG::mgVcycle_up::3 82 5.714e-05 5.714e-05 5.714e-05 0.01% Castro::finalize_advance() 10 5.518e-05 5.518e-05 5.518e-05 0.01% MLMG::mgVcycle_up::1 82 5.442e-05 5.442e-05 5.442e-05 0.01% MLMG::mgVcycle_up::2 82 5.308e-05 5.308e-05 5.308e-05 0.01% Castro::finalize_do_advance() 10 5.012e-05 5.012e-05 5.012e-05 0.01% MLCellLinOp::solutionResidual() 93 4.774e-05 4.774e-05 4.774e-05 0.01% Castro::advance() 10 4.167e-05 4.167e-05 4.167e-05 0.00% Castro::swap_state_time_levels() 10 4.129e-05 4.129e-05 4.129e-05 0.00% MLMG::computeResidual() 82 3.928e-05 3.928e-05 3.928e-05 0.00% StateData::define() 4 3.741e-05 3.741e-05 3.741e-05 0.00% Castro::enforce_consistent_e() 1 3.379e-05 3.379e-05 3.379e-05 0.00% MLMG::mgVcycle_bottom 82 3.345e-05 3.345e-05 3.345e-05 0.00% Gravity::actual_multilevel_solve() 1 3.227e-05 3.227e-05 3.227e-05 0.00% FillPatchSingleLevel 41 2.932e-05 2.932e-05 2.932e-05 0.00% Castro::initMFs() 1 2.762e-05 2.762e-05 2.762e-05 0.00% Amr::defBaseLevel() 1 2.714e-05 2.714e-05 2.714e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.498e-05 2.498e-05 2.498e-05 0.00% makeSFC 55 2.453e-05 2.453e-05 2.453e-05 0.00% Amr::writeSmallPlotFile() 1 2.445e-05 2.445e-05 2.445e-05 0.00% MLPoisson::define() 11 2.356e-05 2.356e-05 2.356e-05 0.00% MLLinOp::define() 11 2.267e-05 2.267e-05 2.267e-05 0.00% Castro::buildMetrics() 1 2.251e-05 2.251e-05 2.251e-05 0.00% Amr::FinalizeInit() 1 2.115e-05 2.115e-05 2.115e-05 0.00% Castro::do_new_sources() 10 1.801e-05 1.801e-05 1.801e-05 0.00% Castro::construct_old_source() 50 1.778e-05 1.778e-05 1.778e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.743e-05 1.743e-05 1.743e-05 0.00% Castro::do_old_sources() 10 1.537e-05 1.537e-05 1.537e-05 0.00% DistributionMapping::Distribute() 56 1.534e-05 1.534e-05 1.534e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.261e-05 1.261e-05 1.261e-05 0.00% Castro::check_for_nan() 20 1.213e-05 1.213e-05 1.213e-05 0.00% Castro::post_timestep() 10 1.204e-05 1.204e-05 1.204e-05 0.00% Castro::apply_source_to_state() 20 1.145e-05 1.145e-05 1.145e-05 0.00% Castro::construct_old_gravity() 10 1.004e-05 1.004e-05 1.004e-05 0.00% MLMG::computeMLResidual() 11 9.46e-06 9.46e-06 9.46e-06 0.00% Gravity::swapTimeLevels() 10 8.962e-06 8.962e-06 8.962e-06 0.00% Amr::initSubcycle() 1 8.582e-06 8.582e-06 8.582e-06 0.00% MLPoisson::prepareForSolve() 11 8.056e-06 8.056e-06 8.056e-06 0.00% Amr::InitializeInit() 1 7.309e-06 7.309e-06 7.309e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.111e-06 7.111e-06 7.111e-06 0.00% Castro::computeNewDt() 9 6.579e-06 6.579e-06 6.579e-06 0.00% MLMG::getGradSolution() 11 6.078e-06 6.078e-06 6.078e-06 0.00% Castro::create_source_corrector() 10 5.218e-06 5.218e-06 5.218e-06 0.00% Castro::retry_advance_ctu() 10 3.856e-06 3.856e-06 3.856e-06 0.00% Castro::post_init() 1 3.812e-06 3.812e-06 3.812e-06 0.00% AmrLevel::checkPointPost() 3 3.743e-06 3.743e-06 3.743e-06 0.00% Gravity::set_mass_offset() 11 3.499e-06 3.499e-06 3.499e-06 0.00% Castro::FluxRegCrseInit 10 3.361e-06 3.361e-06 3.361e-06 0.00% MLMG::MLResNormInf() 11 3.315e-06 3.315e-06 3.315e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.767e-06 2.767e-06 2.767e-06 0.00% AmrLevel::checkPointPre() 3 2.53e-06 2.53e-06 2.53e-06 0.00% Castro::FluxRegFineAdd() 10 2.333e-06 2.333e-06 2.333e-06 0.00% Amr::init() 1 2.199e-06 2.199e-06 2.199e-06 0.00% Castro::computeInitialDt() 2 2.099e-06 2.099e-06 2.099e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.901e-06 1.901e-06 1.901e-06 0.00% Castro::post_regrid() 1 1.339e-06 1.339e-06 1.339e-06 0.00% Amr::initialInit() 1 9.46e-07 9.46e-07 9.46e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9094 0.9094 0.9094 100.00% Amr::coarseTimeStep() 10 0.7255 0.7255 0.7255 79.77% Amr::timeStep() 10 0.5964 0.5964 0.5964 65.58% Castro::advance() 10 0.5897 0.5897 0.5897 64.84% Castro::subcycle_advance_ctu() 10 0.5781 0.5781 0.5781 63.57% Castro::do_advance_ctu() 10 0.5779 0.5779 0.5779 63.54% Gravity::solve_phi_with_mlmg() 11 0.3242 0.3242 0.3242 35.64% Gravity::actual_solve_with_mlmg() 11 0.3153 0.3153 0.3153 34.67% Castro::construct_new_gravity() 10 0.2959 0.2959 0.2959 32.53% MLMG::solve() 11 0.2923 0.2923 0.2923 32.14% Gravity::solve_for_phi() 10 0.2798 0.2798 0.2798 30.77% MLMG::oneIter() 82 0.277 0.277 0.277 30.46% MLMG::mgVcycle() 82 0.2753 0.2753 0.2753 30.27% VisMF::Write(FabArray) 11 0.2236 0.2236 0.2236 24.58% Castro::construct_ctu_hydro_source() 10 0.2059 0.2059 0.2059 22.64% Amr::checkPoint() 3 0.1795 0.1795 0.1795 19.73% Amr::init() 1 0.1523 0.1523 0.1523 16.74% AmrLevel::checkPoint() 3 0.1491 0.1491 0.1491 16.40% StateData::checkPoint() 12 0.149 0.149 0.149 16.39% MLCellLinOp::smooth() 1640 0.1421 0.1421 0.1421 15.63% MLCellLinOp::applyBC() 4433 0.1008 0.1008 0.1008 11.08% MLMG::mgVcycle_bottom 82 0.08393 0.08393 0.08393 9.23% MLMG::actualBottomSolve() 82 0.0839 0.0839 0.0839 9.23% MLCGSolver::bicgstab 82 0.08306 0.08306 0.08306 9.13% Amr::writePlotFile() 2 0.07754 0.07754 0.07754 8.53% MLPoisson::Fsmooth() 3280 0.06513 0.06513 0.06513 7.16% Amr::initialInit() 1 0.05229 0.05229 0.05229 5.75% Amr::FinalizeInit() 1 0.04823 0.04823 0.04823 5.30% Castro::post_init() 1 0.04684 0.04684 0.04684 5.15% Gravity::multilevel_solve_for_new_phi() 1 0.04488 0.04488 0.04488 4.93% Gravity::actual_multilevel_solve() 1 0.04486 0.04486 0.04486 4.93% Castro::clean_state() 62 0.04425 0.04425 0.04425 4.87% FillPatchIterator::Initialize 41 0.04302 0.04302 0.04302 4.73% FillPatchSingleLevel 41 0.0414 0.0414 0.0414 4.55% StateDataPhysBCFunct::() 41 0.03734 0.03734 0.03734 4.11% MLCellLinOp::apply() 1142 0.03695 0.03695 0.03695 4.06% MLMG::mgVcycle_down::0 82 0.03591 0.03591 0.03591 3.95% MLMG::mgVcycle_up::0 82 0.03078 0.03078 0.03078 3.38% StateData::FillBoundary(geom) 328 0.02534 0.02534 0.02534 2.79% MultiFab::Dot() 1114 0.02293 0.02293 0.02293 2.52% MLCellLinOp::correctionResidual() 492 0.02171 0.02171 0.02171 2.39% Castro::computeTemp() 63 0.01999 0.01999 0.01999 2.20% Castro::initialize_do_advance() 10 0.0198 0.0198 0.0198 2.18% MLMG:computeResOfCorrection() 410 0.01875 0.01875 0.01875 2.06% MLPoisson::define() 11 0.01857 0.01857 0.01857 2.04% MLMG::mgVcycle_down::1 82 0.01803 0.01803 0.01803 1.98% Gravity::get_new_grav_vector() 11 0.0177 0.0177 0.0177 1.95% MLMG::mgVcycle_down::2 82 0.01763 0.01763 0.01763 1.94% MLMG::mgVcycle_down::3 82 0.01674 0.01674 0.01674 1.84% MLMG::mgVcycle_up::4 82 0.01642 0.01642 0.01642 1.81% MLMG::mgVcycle_down::4 82 0.01595 0.01595 0.01595 1.75% FabArray::FillBoundary() 4023 0.01584 0.01584 0.01584 1.74% Castro::construct_old_gravity() 10 0.01551 0.01551 0.01551 1.71% Gravity::get_old_grav_vector() 10 0.0155 0.0155 0.0155 1.70% CGSolver::sxay() 1586 0.01515 0.01515 0.01515 1.67% FillBoundary_nowait() 4023 0.01499 0.01499 0.01499 1.65% MultiFab::LinComb() 1586 0.01477 0.01477 0.01477 1.62% FabArray::setVal() 1144 0.01453 0.01453 0.01453 1.60% FabArray::ParallelCopy() 861 0.01452 0.01452 0.01452 1.60% FabArray::ParallelCopy_nowait() 861 0.01425 0.01425 0.01425 1.57% Castro::normalize_species() 62 0.01369 0.01369 0.01369 1.51% MLCGSolver::ParallelAllReduce 1514 0.01368 0.01368 0.01368 1.50% MLMG::mgVcycle_up::2 82 0.01354 0.01354 0.01354 1.49% MLMG::mgVcycle_up::1 82 0.01328 0.01328 0.01328 1.46% MLCellLinOp::defineAuxData() 11 0.01312 0.01312 0.01312 1.44% MLMG::addInterpCorrection() 410 0.01311 0.01311 0.01311 1.44% MLMG::mgVcycle_up::3 82 0.0129 0.0129 0.0129 1.42% amrex::average_down 410 0.01224 0.01224 0.01224 1.35% MLPoisson::Fapply() 1142 0.0121 0.0121 0.0121 1.33% Castro::initialize_advance() 10 0.01148 0.01148 0.01148 1.26% Castro::do_new_sources() 10 0.01125 0.01125 0.01125 1.24% Castro::expand_state() 10 0.01123 0.01123 0.01123 1.23% Castro::do_old_sources() 10 0.01004 0.01004 0.01004 1.10% Castro::enforce_min_density() 62 0.009793 0.009793 0.009793 1.08% Gravity::fill_multipole_BCs() 11 0.008655 0.008655 0.008655 0.95% MLCellLinOp::solutionResidual() 93 0.007242 0.007242 0.007242 0.80% MultiFab::Xpay() 585 0.006756 0.006756 0.006756 0.74% Castro::post_timestep() 10 0.006472 0.006472 0.006472 0.71% MLMG::computeResidual() 82 0.006255 0.006255 0.006255 0.69% Castro::reset_internal_energy(MultiFab) 63 0.005589 0.005589 0.005589 0.61% MLMG::prepareForSolve() 11 0.005546 0.005546 0.005546 0.61% MLCellLinOp::defineBC() 11 0.005225 0.005225 0.005225 0.57% Castro::estTimeStep() 21 0.004954 0.004954 0.004954 0.54% BndryData::define() 11 0.004942 0.004942 0.004942 0.54% Amr::InitializeInit() 1 0.004064 0.004064 0.004064 0.45% Amr::defBaseLevel() 1 0.004056 0.004056 0.004056 0.45% Castro::initData() 1 0.003541 0.003541 0.003541 0.39% Castro::construct_new_source() 50 0.00275 0.00275 0.00275 0.30% Castro::construct_new_gravity_source() 10 0.00269 0.00269 0.00269 0.30% Castro::construct_old_source() 50 0.002261 0.002261 0.002261 0.25% Castro::construct_old_gravity_source() 10 0.002243 0.002243 0.002243 0.25% Castro::computeNewDt() 9 0.002151 0.002151 0.002151 0.24% MLMG::ResNormInf() 93 0.002095 0.002095 0.002095 0.23% Castro::apply_source_to_state() 20 0.001829 0.001829 0.001829 0.20% MultiFab::Saxpy() 20 0.001817 0.001817 0.001817 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001695 0.001695 0.001695 0.19% MultiFab::Add() 82 0.00169 0.00169 0.00169 0.19% MLCellLinOp::setLevelBC() 11 0.001577 0.001577 0.001577 0.17% Castro::reset_internal_energy(Fab) 504 0.001542 0.001542 0.001542 0.17% Castro::enforce_speed_limit() 62 0.00152 0.00152 0.00152 0.17% FabArrayBase::getCPC() 1323 0.00148 0.00148 0.00148 0.16% MLMG::getGradSolution() 11 0.00146 0.00146 0.00146 0.16% MLCellLinOp::compGrad() 11 0.001453 0.001453 0.001453 0.16% FabArray::mult() 43 0.001336 0.001336 0.001336 0.15% FabArray::setDomainBndry() 41 0.001323 0.001323 0.001323 0.15% MLPoisson::prepareForSolve() 11 0.001204 0.001204 0.001204 0.13% MLCellLinOp::prepareForSolve() 11 0.001196 0.001196 0.001196 0.13% Castro::post_regrid() 1 0.001187 0.001187 0.001187 0.13% Castro::check_for_nan() 20 0.001187 0.001187 0.001187 0.13% MultiFab::contains_nan() 20 0.001175 0.001175 0.001175 0.13% MLMG::computeMLResidual() 11 0.001036 0.001036 0.001036 0.11% Gravity::update_max_rhs() 11 0.0008274 0.0008274 0.0008274 0.09% FabArrayBase::CPC::define() 454 0.0006886 0.0006886 0.0006886 0.08% FabArrayBase::getFB() 4023 0.0006627 0.0006627 0.0006627 0.07% Castro::computeInitialDt() 2 0.0006547 0.0006547 0.0006547 0.07% Amr::InitAmr() 1 0.0004886 0.0004886 0.0004886 0.05% Gravity::swapTimeLevels() 10 0.0004485 0.0004485 0.0004485 0.05% Castro::Castro() 1 0.0004402 0.0004402 0.0004402 0.05% MultiFab::Copy() 11 0.0003357 0.0003357 0.0003357 0.04% MLMG::MLResNormInf() 11 0.000283 0.000283 0.000283 0.03% MultiFab::max() 11 0.0002668 0.0002668 0.0002668 0.03% MLMG::MLRhsNormInf() 11 0.0002204 0.0002204 0.0002204 0.02% MLLinOp::define() 11 0.0002063 0.0002063 0.0002063 0.02% MLLinOp::defineGrids() 11 0.0001836 0.0001836 0.0001836 0.02% Castro::buildMetrics() 1 0.000157 0.000157 0.000157 0.02% FabArrayBase::FB::FB() 56 8.558e-05 8.558e-05 8.558e-05 0.01% Castro::finalize_advance() 10 6.087e-05 6.087e-05 6.087e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.12e-05 5.12e-05 5.12e-05 0.01% Castro::finalize_do_advance() 10 5.012e-05 5.012e-05 5.012e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.452e-05 4.452e-05 4.452e-05 0.00% Castro::swap_state_time_levels() 10 4.129e-05 4.129e-05 4.129e-05 0.00% makeSFC 55 3.859e-05 3.859e-05 3.859e-05 0.00% StateData::define() 4 3.741e-05 3.741e-05 3.741e-05 0.00% Castro::enforce_consistent_e() 1 3.379e-05 3.379e-05 3.379e-05 0.00% Castro::initMFs() 1 2.762e-05 2.762e-05 2.762e-05 0.00% Amr::writeSmallPlotFile() 1 2.445e-05 2.445e-05 2.445e-05 0.00% DistributionMapping::Distribute() 56 1.534e-05 1.534e-05 1.534e-05 0.00% Amr::initSubcycle() 1 8.582e-06 8.582e-06 8.582e-06 0.00% Castro::create_source_corrector() 10 5.218e-06 5.218e-06 5.218e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.049e-06 4.049e-06 4.049e-06 0.00% Castro::retry_advance_ctu() 10 3.856e-06 3.856e-06 3.856e-06 0.00% AmrLevel::checkPointPost() 3 3.743e-06 3.743e-06 3.743e-06 0.00% Gravity::set_mass_offset() 11 3.499e-06 3.499e-06 3.499e-06 0.00% Castro::FluxRegCrseInit 10 3.361e-06 3.361e-06 3.361e-06 0.00% AmrLevel::checkPointPre() 3 2.53e-06 2.53e-06 2.53e-06 0.00% Castro::FluxRegFineAdd() 10 2.333e-06 2.333e-06 2.333e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.901e-06 1.901e-06 1.901e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-5-g6ebf8ffc2689) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-5-g6ebf8ffc2689) initialized Starting run at 08:57:57 UTC on 2022-08-05. Successfully read inputs file ... Castro git describe: 22.08-1-g4c16aad67 AMReX git describe: 22.08-5-g6ebf8ffc2 Microphysics git describe: 22.08-1-gffefdd62 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.485809143 Restart time = 0.0479336 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051404048 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051328686 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057728494 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.066458646 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.067524025 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032677161 seconds Ending run at 08:57:57 UTC on 2022-08-05. Run time = 0.376023167 Run time without initialization = 0.327526501 Average number of zones advanced per microsecond: 4.002 Average number of zones advanced per microsecond per rank: 4.002 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3761 ... 0.3761 ... 0.3761 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0968 0.0968 0.0968 25.73% VisMF::Read() 3 0.04019 0.04019 0.04019 10.69% MLCellLinOp::applyBC() 1946 0.0344 0.0344 0.0344 9.15% VisMF::Write(FabArray) 1 0.03095 0.03095 0.03095 8.23% MLPoisson::Fsmooth() 1440 0.02751 0.02751 0.02751 7.31% StateData::FillBoundary(geom) 160 0.01185 0.01185 0.01185 3.15% MLCGSolver::bicgstab 36 0.01023 0.01023 0.01023 2.72% MultiFab::Dot() 484 0.00957 0.00957 0.00957 2.54% Castro::normalize_species() 30 0.007995 0.007995 0.007995 2.13% Castro::computeTemp() 30 0.007561 0.007561 0.007561 2.01% FabArray::setVal() 537 0.006749 0.006749 0.006749 1.79% Castro::enforce_min_density() 30 0.006712 0.006712 0.006712 1.78% FillBoundary_nowait() 1766 0.006236 0.006236 0.006236 1.66% MLCellLinOp::defineAuxData() 6 0.006226 0.006226 0.006226 1.66% MultiFab::LinComb() 690 0.006174 0.006174 0.006174 1.64% FabArray::ParallelCopy_nowait() 380 0.005921 0.005921 0.005921 1.57% StateDataPhysBCFunct::() 20 0.005163 0.005163 0.005163 1.37% MLPoisson::Fapply() 500 0.005108 0.005108 0.005108 1.36% Gravity::fill_multipole_BCs() 6 0.004282 0.004282 0.004282 1.14% Amr::restart() 1 0.003584 0.003584 0.003584 0.95% MLMG::addInterpCorrection() 180 0.003297 0.003297 0.003297 0.88% amrex::average_down 180 0.00294 0.00294 0.00294 0.78% MultiFab::Xpay() 258 0.002892 0.002892 0.002892 0.77% Castro::do_advance_ctu() 5 0.002416 0.002416 0.002416 0.64% BndryData::define() 6 0.002122 0.002122 0.002122 0.56% Amr::writePlotFile() 1 0.001816 0.001816 0.001816 0.48% Castro::reset_internal_energy(MultiFab) 30 0.001705 0.001705 0.001705 0.45% Castro::construct_new_gravity_source() 5 0.001511 0.001511 0.001511 0.40% Castro::estTimeStep() 10 0.001372 0.001372 0.001372 0.36% Castro::construct_old_gravity_source() 5 0.001131 0.001131 0.001131 0.30% Castro::reset_internal_energy(Fab) 240 0.0009717 0.0009717 0.0009717 0.26% Castro::create_source_corrector() 5 0.0009411 0.0009411 0.0009411 0.25% Gravity::get_old_grav_vector() 5 0.0009396 0.0009396 0.0009396 0.25% MultiFab::Saxpy() 10 0.0009236 0.0009236 0.0009236 0.25% Castro::construct_new_source() 25 0.0009186 0.0009186 0.0009186 0.24% MLMG::ResNormInf() 42 0.0009018 0.0009018 0.0009018 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008989 0.0008989 0.0008989 0.24% Castro::expand_state() 5 0.0008719 0.0008719 0.0008719 0.23% Gravity::get_new_grav_vector() 5 0.0008665 0.0008665 0.0008665 0.23% Gravity::actual_solve_with_mlmg() 6 0.0008149 0.0008149 0.0008149 0.22% MLCellLinOp::setLevelBC() 6 0.0008127 0.0008127 0.0008127 0.22% MultiFab::Add() 36 0.0007173 0.0007173 0.0007173 0.19% MLMG::prepareForSolve() 6 0.0006507 0.0006507 0.0006507 0.17% FabArray::mult() 22 0.0006475 0.0006475 0.0006475 0.17% FabArray::setDomainBndry() 20 0.0006332 0.0006332 0.0006332 0.17% MLCellLinOp::prepareForSolve() 6 0.0006253 0.0006253 0.0006253 0.17% MultiFab::contains_nan() 10 0.0005886 0.0005886 0.0005886 0.16% Castro::enforce_speed_limit() 30 0.0005506 0.0005506 0.0005506 0.15% MLCellLinOp::compGrad() 6 0.0004891 0.0004891 0.0004891 0.13% MLCellLinOp::smooth() 720 0.0004755 0.0004755 0.0004755 0.13% Amr::InitAmr() 1 0.0004007 0.0004007 0.0004007 0.11% FabArrayBase::CPC::define() 244 0.0003981 0.0003981 0.0003981 0.11% FabArrayBase::getCPC() 632 0.0003719 0.0003719 0.0003719 0.10% FabArray::FillBoundary() 1766 0.0003661 0.0003661 0.0003661 0.10% FabArrayBase::getFB() 1766 0.0002506 0.0002506 0.0002506 0.07% main() 1 0.0002481 0.0002481 0.0002481 0.07% Castro::subcycle_advance_ctu() 5 0.0002454 0.0002454 0.0002454 0.07% Gravity::update_max_rhs() 6 0.0002329 0.0002329 0.0002329 0.06% Gravity::solve_for_phi() 5 0.0002073 0.0002073 0.0002073 0.06% MLCellLinOp::apply() 500 0.0002009 0.0002009 0.0002009 0.05% MultiFab::Copy() 6 0.0001745 0.0001745 0.0001745 0.05% Amr::coarseTimeStep() 5 0.0001689 0.0001689 0.0001689 0.04% CGSolver::sxay() 690 0.0001651 0.0001651 0.0001651 0.04% MLCellLinOp::defineBC() 6 0.0001452 0.0001452 0.0001452 0.04% Castro::construct_new_gravity() 5 0.0001448 0.0001448 0.0001448 0.04% FillPatchIterator::Initialize 20 0.0001401 0.0001401 0.0001401 0.04% MultiFab::max() 6 0.0001348 0.0001348 0.0001348 0.04% MLCGSolver::ParallelAllReduce 659 0.0001242 0.0001242 0.0001242 0.03% FabArray::ParallelCopy() 380 0.0001137 0.0001137 0.0001137 0.03% Castro::post_timestep() 5 0.0001127 0.0001127 0.0001127 0.03% MLMG::MLRhsNormInf() 6 0.000112 0.000112 0.000112 0.03% Castro::advance() 5 0.0001082 0.0001082 0.0001082 0.03% Amr::timeStep() 5 0.0001038 0.0001038 0.0001038 0.03% Castro::initialize_advance() 5 0.0001027 0.0001027 0.0001027 0.03% Castro::initialize_do_advance() 5 9.711e-05 9.711e-05 9.711e-05 0.03% MLCellLinOp::correctionResidual() 216 9.517e-05 9.517e-05 9.517e-05 0.03% MLLinOp::defineGrids() 6 8.839e-05 8.839e-05 8.839e-05 0.02% MLMG::mgVcycle() 36 7.829e-05 7.829e-05 7.829e-05 0.02% Castro::construct_old_source() 25 7.672e-05 7.672e-05 7.672e-05 0.02% AmrLevel::restart() 1 7.514e-05 7.514e-05 7.514e-05 0.02% MLMG:computeResOfCorrection() 180 7.002e-05 7.002e-05 7.002e-05 0.02% StateData::restartDoit() 4 6.591e-05 6.591e-05 6.591e-05 0.02% FabArrayBase::FB::FB() 26 5.468e-05 5.468e-05 5.468e-05 0.01% Castro::construct_old_gravity() 5 4.988e-05 4.988e-05 4.988e-05 0.01% MLMG::mgVcycle_down::0 36 4.474e-05 4.474e-05 4.474e-05 0.01% MLMG::mgVcycle_down::1 36 3.953e-05 3.953e-05 3.953e-05 0.01% Castro::clean_state() 30 3.887e-05 3.887e-05 3.887e-05 0.01% MLMG::actualBottomSolve() 36 3.823e-05 3.823e-05 3.823e-05 0.01% MLMG::mgVcycle_down::2 36 3.772e-05 3.772e-05 3.772e-05 0.01% MLMG::mgVcycle_down::4 36 3.597e-05 3.597e-05 3.597e-05 0.01% MLMG::mgVcycle_down::3 36 3.542e-05 3.542e-05 3.542e-05 0.01% MLMG::mgVcycle_up::4 36 3.232e-05 3.232e-05 3.232e-05 0.01% Castro::buildMetrics() 1 3.196e-05 3.196e-05 3.196e-05 0.01% Gravity::actual_multilevel_solve() 1 3.066e-05 3.066e-05 3.066e-05 0.01% MLMG::solve() 6 2.972e-05 2.972e-05 2.972e-05 0.01% Castro::post_restart() 1 2.944e-05 2.944e-05 2.944e-05 0.01% Castro::initMFs() 1 2.792e-05 2.792e-05 2.792e-05 0.01% MLMG::oneIter() 36 2.679e-05 2.679e-05 2.679e-05 0.01% Amr::writeSmallPlotFile() 1 2.632e-05 2.632e-05 2.632e-05 0.01% MLMG::mgVcycle_up::0 36 2.63e-05 2.63e-05 2.63e-05 0.01% Castro::finalize_advance() 5 2.539e-05 2.539e-05 2.539e-05 0.01% Castro::swap_state_time_levels() 5 2.525e-05 2.525e-05 2.525e-05 0.01% MLMG::mgVcycle_up::3 36 2.458e-05 2.458e-05 2.458e-05 0.01% MLMG::mgVcycle_up::2 36 2.352e-05 2.352e-05 2.352e-05 0.01% MLMG::mgVcycle_up::1 36 2.206e-05 2.206e-05 2.206e-05 0.01% MLCellLinOp::solutionResidual() 42 2.199e-05 2.199e-05 2.199e-05 0.01% MLLinOp::define() 6 1.982e-05 1.982e-05 1.982e-05 0.01% makeSFC 30 1.905e-05 1.905e-05 1.905e-05 0.01% Castro::finalize_do_advance() 5 1.881e-05 1.881e-05 1.881e-05 0.01% MLMG::computeResidual() 36 1.805e-05 1.805e-05 1.805e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.763e-05 1.763e-05 1.763e-05 0.00% MLMG::mgVcycle_bottom 36 1.542e-05 1.542e-05 1.542e-05 0.00% MLPoisson::define() 6 1.541e-05 1.541e-05 1.541e-05 0.00% FillPatchSingleLevel 20 1.374e-05 1.374e-05 1.374e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.368e-05 1.368e-05 1.368e-05 0.00% Castro::do_new_sources() 5 9.965e-06 9.965e-06 9.965e-06 0.00% DistributionMapping::Distribute() 31 9.923e-06 9.923e-06 9.923e-06 0.00% Amr::initSubcycle() 1 9.015e-06 9.015e-06 9.015e-06 0.00% Castro::do_old_sources() 5 8.447e-06 8.447e-06 8.447e-06 0.00% Castro::check_for_nan() 10 6.998e-06 6.998e-06 6.998e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.844e-06 6.844e-06 6.844e-06 0.00% Castro::apply_source_to_state() 10 5.427e-06 5.427e-06 5.427e-06 0.00% MLMG::computeMLResidual() 6 4.777e-06 4.777e-06 4.777e-06 0.00% MLPoisson::prepareForSolve() 6 4.691e-06 4.691e-06 4.691e-06 0.00% Gravity::swapTimeLevels() 5 3.961e-06 3.961e-06 3.961e-06 0.00% Castro::retry_advance_ctu() 5 3.494e-06 3.494e-06 3.494e-06 0.00% MLMG::getGradSolution() 6 3.077e-06 3.077e-06 3.077e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.982e-06 2.982e-06 2.982e-06 0.00% Castro::computeNewDt() 5 2.85e-06 2.85e-06 2.85e-06 0.00% Gravity::set_mass_offset() 6 2.125e-06 2.125e-06 2.125e-06 0.00% MLMG::MLResNormInf() 6 2.076e-06 2.076e-06 2.076e-06 0.00% Castro::FluxRegFineAdd() 5 1.188e-06 1.188e-06 1.188e-06 0.00% Castro::FluxRegCrseInit 5 1.186e-06 1.186e-06 1.186e-06 0.00% Amr::init() 1 1.126e-06 1.126e-06 1.126e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.074e-06 1.074e-06 1.074e-06 0.00% AmrLevel::AmrLevel() 1 8.78e-07 8.78e-07 8.78e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.376 0.376 0.376 100.00% Amr::coarseTimeStep() 5 0.2946 0.2946 0.2946 78.34% Amr::timeStep() 5 0.2935 0.2935 0.2935 78.05% Castro::advance() 5 0.29 0.29 0.29 77.11% Castro::subcycle_advance_ctu() 5 0.2844 0.2844 0.2844 75.63% Castro::do_advance_ctu() 5 0.2841 0.2841 0.2841 75.56% Castro::construct_new_gravity() 5 0.1435 0.1435 0.1435 38.17% Gravity::solve_phi_with_mlmg() 6 0.1392 0.1392 0.1392 37.01% Gravity::solve_for_phi() 5 0.1357 0.1357 0.1357 36.09% Gravity::actual_solve_with_mlmg() 6 0.1348 0.1348 0.1348 35.84% MLMG::solve() 6 0.1224 0.1224 0.1224 32.56% MLMG::oneIter() 36 0.1153 0.1153 0.1153 30.66% MLMG::mgVcycle() 36 0.1146 0.1146 0.1146 30.46% Castro::construct_ctu_hydro_source() 5 0.09675 0.09675 0.09675 25.73% MLCellLinOp::smooth() 720 0.05859 0.05859 0.05859 15.58% Amr::init() 1 0.04798 0.04798 0.04798 12.76% Amr::restart() 1 0.04798 0.04798 0.04798 12.76% MLCellLinOp::applyBC() 1946 0.04131 0.04131 0.04131 10.98% AmrLevel::restart() 1 0.04039 0.04039 0.04039 10.74% StateData::restartDoit() 4 0.04031 0.04031 0.04031 10.72% VisMF::Read() 3 0.04019 0.04019 0.04019 10.69% MLMG::mgVcycle_bottom 36 0.03517 0.03517 0.03517 9.35% MLMG::actualBottomSolve() 36 0.03516 0.03516 0.03516 9.35% MLCGSolver::bicgstab 36 0.0348 0.0348 0.0348 9.26% Amr::writePlotFile() 1 0.03277 0.03277 0.03277 8.71% VisMF::Write(FabArray) 1 0.03095 0.03095 0.03095 8.23% MLPoisson::Fsmooth() 1440 0.02751 0.02751 0.02751 7.31% Castro::clean_state() 30 0.02553 0.02553 0.02553 6.79% FillPatchIterator::Initialize 20 0.01978 0.01978 0.01978 5.26% FillPatchSingleLevel 20 0.01901 0.01901 0.01901 5.06% StateDataPhysBCFunct::() 20 0.01701 0.01701 0.01701 4.52% MLCellLinOp::apply() 500 0.01574 0.01574 0.01574 4.19% MLMG::mgVcycle_down::0 36 0.01532 0.01532 0.01532 4.07% MLMG::mgVcycle_up::0 36 0.01311 0.01311 0.01311 3.49% StateData::FillBoundary(geom) 160 0.01185 0.01185 0.01185 3.15% Castro::initialize_do_advance() 5 0.01161 0.01161 0.01161 3.09% Castro::computeTemp() 30 0.01024 0.01024 0.01024 2.72% MLPoisson::define() 6 0.009941 0.009941 0.009941 2.64% MultiFab::Dot() 484 0.00957 0.00957 0.00957 2.54% MLCellLinOp::correctionResidual() 216 0.009201 0.009201 0.009201 2.45% Castro::do_new_sources() 5 0.008039 0.008039 0.008039 2.14% Castro::normalize_species() 30 0.007995 0.007995 0.007995 2.13% MLMG:computeResOfCorrection() 180 0.007942 0.007942 0.007942 2.11% Gravity::get_new_grav_vector() 5 0.007644 0.007644 0.007644 2.03% MLMG::mgVcycle_down::1 36 0.007635 0.007635 0.007635 2.03% Castro::construct_old_gravity() 5 0.007472 0.007472 0.007472 1.99% Gravity::get_old_grav_vector() 5 0.007422 0.007422 0.007422 1.97% MLMG::mgVcycle_down::2 36 0.007387 0.007387 0.007387 1.96% MLMG::mgVcycle_down::3 36 0.007046 0.007046 0.007046 1.87% MLCellLinOp::defineAuxData() 6 0.006974 0.006974 0.006974 1.85% FabArray::FillBoundary() 1766 0.006907 0.006907 0.006907 1.84% MLMG::mgVcycle_down::4 36 0.006754 0.006754 0.006754 1.80% FabArray::setVal() 537 0.006749 0.006749 0.006749 1.79% Castro::enforce_min_density() 30 0.006712 0.006712 0.006712 1.78% FillBoundary_nowait() 1766 0.006541 0.006541 0.006541 1.74% FabArray::ParallelCopy() 380 0.006415 0.006415 0.006415 1.71% CGSolver::sxay() 690 0.006339 0.006339 0.006339 1.69% FabArray::ParallelCopy_nowait() 380 0.006301 0.006301 0.006301 1.68% MultiFab::LinComb() 690 0.006174 0.006174 0.006174 1.64% MLCGSolver::ParallelAllReduce 659 0.005727 0.005727 0.005727 1.52% MLMG::mgVcycle_up::2 36 0.005658 0.005658 0.005658 1.50% Castro::do_old_sources() 5 0.005618 0.005618 0.005618 1.49% MLMG::mgVcycle_up::1 36 0.005598 0.005598 0.005598 1.49% MLMG::addInterpCorrection() 180 0.005494 0.005494 0.005494 1.46% Castro::initialize_advance() 5 0.005448 0.005448 0.005448 1.45% MLMG::mgVcycle_up::4 36 0.005399 0.005399 0.005399 1.44% MLMG::mgVcycle_up::3 36 0.00539 0.00539 0.00539 1.43% Castro::expand_state() 5 0.005334 0.005334 0.005334 1.42% amrex::average_down 180 0.005171 0.005171 0.005171 1.38% MLPoisson::Fapply() 500 0.005108 0.005108 0.005108 1.36% Gravity::fill_multipole_BCs() 6 0.004282 0.004282 0.004282 1.14% Castro::post_restart() 1 0.003825 0.003825 0.003825 1.02% Gravity::multilevel_solve_for_new_phi() 1 0.003698 0.003698 0.003698 0.98% Gravity::actual_multilevel_solve() 1 0.00368 0.00368 0.00368 0.98% Castro::post_timestep() 5 0.003434 0.003434 0.003434 0.91% MLCellLinOp::solutionResidual() 42 0.003209 0.003209 0.003209 0.85% MultiFab::Xpay() 258 0.002892 0.002892 0.002892 0.77% MLMG::prepareForSolve() 6 0.00287 0.00287 0.00287 0.76% MLCellLinOp::defineBC() 6 0.002809 0.002809 0.002809 0.75% Castro::reset_internal_energy(MultiFab) 30 0.002677 0.002677 0.002677 0.71% MLMG::computeResidual() 36 0.002667 0.002667 0.002667 0.71% BndryData::define() 6 0.002663 0.002663 0.002663 0.71% Castro::construct_new_source() 25 0.00243 0.00243 0.00243 0.65% Castro::construct_new_gravity_source() 5 0.001511 0.001511 0.001511 0.40% Castro::estTimeStep() 10 0.001372 0.001372 0.001372 0.36% Castro::construct_old_source() 25 0.001207 0.001207 0.001207 0.32% Castro::construct_old_gravity_source() 5 0.001131 0.001131 0.001131 0.30% Castro::reset_internal_energy(Fab) 240 0.0009717 0.0009717 0.0009717 0.26% Castro::create_source_corrector() 5 0.0009411 0.0009411 0.0009411 0.25% Castro::apply_source_to_state() 10 0.000929 0.000929 0.000929 0.25% MultiFab::Saxpy() 10 0.0009236 0.0009236 0.0009236 0.25% Castro::computeNewDt() 5 0.0009205 0.0009205 0.0009205 0.24% MLMG::ResNormInf() 42 0.0009018 0.0009018 0.0009018 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008989 0.0008989 0.0008989 0.24% MLCellLinOp::setLevelBC() 6 0.0008127 0.0008127 0.0008127 0.22% FabArrayBase::getCPC() 632 0.00077 0.00077 0.00077 0.20% MLMG::getGradSolution() 6 0.0007627 0.0007627 0.0007627 0.20% MLCellLinOp::compGrad() 6 0.0007597 0.0007597 0.0007597 0.20% MultiFab::Add() 36 0.0007173 0.0007173 0.0007173 0.19% FabArray::mult() 22 0.0006475 0.0006475 0.0006475 0.17% FabArray::setDomainBndry() 20 0.0006332 0.0006332 0.0006332 0.17% MLPoisson::prepareForSolve() 6 0.00063 0.00063 0.00063 0.17% MLCellLinOp::prepareForSolve() 6 0.0006253 0.0006253 0.0006253 0.17% Castro::check_for_nan() 10 0.0005956 0.0005956 0.0005956 0.16% MultiFab::contains_nan() 10 0.0005886 0.0005886 0.0005886 0.16% MLMG::computeMLResidual() 6 0.0005656 0.0005656 0.0005656 0.15% Castro::enforce_speed_limit() 30 0.0005506 0.0005506 0.0005506 0.15% Gravity::update_max_rhs() 6 0.0004472 0.0004472 0.0004472 0.12% Amr::InitAmr() 1 0.0004097 0.0004097 0.0004097 0.11% FabArrayBase::CPC::define() 244 0.0003981 0.0003981 0.0003981 0.11% FabArrayBase::getFB() 1766 0.0003053 0.0003053 0.0003053 0.08% Gravity::swapTimeLevels() 5 0.0002258 0.0002258 0.0002258 0.06% MultiFab::Copy() 6 0.0001745 0.0001745 0.0001745 0.05% Castro::buildMetrics() 1 0.000153 0.000153 0.000153 0.04% MLMG::MLResNormInf() 6 0.0001463 0.0001463 0.0001463 0.04% MLLinOp::define() 6 0.000143 0.000143 0.000143 0.04% MultiFab::max() 6 0.0001348 0.0001348 0.0001348 0.04% MLLinOp::defineGrids() 6 0.0001232 0.0001232 0.0001232 0.03% MLMG::MLRhsNormInf() 6 0.000112 0.000112 0.000112 0.03% FabArrayBase::FB::FB() 26 5.468e-05 5.468e-05 5.468e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.37e-05 3.37e-05 3.37e-05 0.01% Castro::initMFs() 1 2.792e-05 2.792e-05 2.792e-05 0.01% Castro::finalize_advance() 5 2.776e-05 2.776e-05 2.776e-05 0.01% makeSFC 30 2.686e-05 2.686e-05 2.686e-05 0.01% Amr::writeSmallPlotFile() 1 2.632e-05 2.632e-05 2.632e-05 0.01% Castro::swap_state_time_levels() 5 2.525e-05 2.525e-05 2.525e-05 0.01% Castro::finalize_do_advance() 5 1.881e-05 1.881e-05 1.881e-05 0.01% DistributionMapping::Distribute() 31 9.923e-06 9.923e-06 9.923e-06 0.00% Amr::initSubcycle() 1 9.015e-06 9.015e-06 9.015e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.098e-06 5.098e-06 5.098e-06 0.00% Castro::retry_advance_ctu() 5 3.494e-06 3.494e-06 3.494e-06 0.00% Gravity::set_mass_offset() 6 2.125e-06 2.125e-06 2.125e-06 0.00% Castro::FluxRegFineAdd() 5 1.188e-06 1.188e-06 1.188e-06 0.00% Castro::FluxRegCrseInit 5 1.186e-06 1.186e-06 1.186e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.074e-06 1.074e-06 1.074e-06 0.00% AmrLevel::AmrLevel() 1 8.78e-07 8.78e-07 8.78e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-5-g6ebf8ffc2689) finalized