Initializing CUDA... CUDA initialized with 1 device. AMReX (23.01-15-gf972714bd111) initialized Starting run at 10:10:22 UTC on 2023-01-12. Successfully read inputs file ... Castro git describe: 23.01-14-g78178b78e AMReX git describe: 23.01-15-gf972714bd Microphysics git describe: 23.01-3-g1e475055 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.058295468 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.033495753 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.046222476 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049225073 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.048143248 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.059597386 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.059155829 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.057636038 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.063742773 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.064857221 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.059033592 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.047085898 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.060172281 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.057884018 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033421113 seconds Ending run at 10:10:22 UTC on 2023-01-12. Run time = 0.847666799 Run time without initialization = 0.706835169 Average number of zones advanced per microsecond: 3.709 Average number of zones advanced per microsecond per rank: 3.709 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8477 ... 0.8477 ... 0.8477 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2339 0.2339 0.2339 27.59% Castro::construct_ctu_hydro_source() 10 0.2026 0.2026 0.2026 23.90% MLCellLinOp::applyBC() 4433 0.07505 0.07505 0.07505 8.85% MLPoisson::Fsmooth() 3280 0.03212 0.03212 0.03212 3.79% FillBoundary_nowait() 4023 0.03094 0.03094 0.03094 3.65% StateData::FillBoundary(geom) 328 0.02338 0.02338 0.02338 2.76% amrex::Dot() 1114 0.02034 0.02034 0.02034 2.40% Castro::normalize_species() 62 0.01628 0.01628 0.01628 1.92% amrex::Copy() 1029 0.01479 0.01479 0.01479 1.74% FabArray::norminf() 743 0.01427 0.01427 0.01427 1.68% Castro::computeTemp() 63 0.01426 0.01426 0.01426 1.68% FabArray::setVal() 1144 0.0131 0.0131 0.0131 1.55% FabArray::ParallelCopy_nowait() 861 0.01302 0.01302 0.01302 1.54% StateDataPhysBCFunct::() 41 0.01274 0.01274 0.01274 1.50% MLPoisson::Fapply() 1142 0.01036 0.01036 0.01036 1.22% MLCellLinOp::defineAuxData() 11 0.0096 0.0096 0.0096 1.13% Castro::enforce_min_density() 62 0.008882 0.008882 0.008882 1.05% FabArray::Saxpy() 813 0.008093 0.008093 0.008093 0.95% FabArray::Xpay() 821 0.008034 0.008034 0.008034 0.95% MLMG::addInterpCorrection() 410 0.006526 0.006526 0.006526 0.77% Gravity::fill_multipole_BCs() 11 0.006433 0.006433 0.006433 0.76% amrex::average_down 410 0.005825 0.005825 0.005825 0.69% Castro::estTimeStep() 21 0.005584 0.005584 0.005584 0.66% Castro::reset_internal_energy(MultiFab) 63 0.004721 0.004721 0.004721 0.56% FabArray::LinComb() 557 0.004505 0.004505 0.004505 0.53% amrex::Add() 164 0.004297 0.004297 0.004297 0.51% Amr::checkPoint() 3 0.004009 0.004009 0.004009 0.47% BndryData::define() 11 0.003614 0.003614 0.003614 0.43% Castro::do_advance_ctu() 10 0.003364 0.003364 0.003364 0.40% Castro::construct_new_gravity_source() 10 0.003201 0.003201 0.003201 0.38% Castro::construct_old_gravity_source() 10 0.0026 0.0026 0.0026 0.31% Amr::writePlotFile() 2 0.002396 0.002396 0.002396 0.28% MLCGSolver::bicgstab 82 0.002126 0.002126 0.002126 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00163 0.00163 0.00163 0.19% Castro::reset_internal_energy(Fab) 504 0.001503 0.001503 0.001503 0.18% MLCellLinOp::setLevelBC() 11 0.001393 0.001393 0.001393 0.16% Gravity::actual_solve_with_mlmg() 11 0.001362 0.001362 0.001362 0.16% FabArray::mult() 43 0.001339 0.001339 0.001339 0.16% FabArray::setDomainBndry() 41 0.001288 0.001288 0.001288 0.15% Castro::initData() 1 0.001275 0.001275 0.001275 0.15% MLCellLinOp::smooth() 1640 0.001208 0.001208 0.001208 0.14% MultiFab::contains_nan() 20 0.001171 0.001171 0.001171 0.14% MLCellLinOp::prepareForSolve() 11 0.001091 0.001091 0.001091 0.13% MLCellLinOp::compGrad() 11 0.0009109 0.0009109 0.0009109 0.11% Castro::enforce_speed_limit() 62 0.0009025 0.0009025 0.0009025 0.11% MLMG::prepareForSolve() 11 0.0008407 0.0008407 0.0008407 0.10% FabArray::FillBoundary() 4023 0.0007919 0.0007919 0.0007919 0.09% FabArrayBase::getCPC() 1323 0.0007147 0.0007147 0.0007147 0.08% FabArrayBase::CPC::define() 454 0.0006797 0.0006797 0.0006797 0.08% FabArrayBase::getFB() 4023 0.0006091 0.0006091 0.0006091 0.07% Gravity::get_new_grav_vector() 11 0.0005964 0.0005964 0.0005964 0.07% Gravity::get_old_grav_vector() 10 0.0005332 0.0005332 0.0005332 0.06% Amr::InitAmr() 1 0.0005016 0.0005016 0.0005016 0.06% MLCellLinOp::apply() 1142 0.0004527 0.0004527 0.0004527 0.05% MLMG::mgVcycle() 82 0.0003989 0.0003989 0.0003989 0.05% main() 1 0.0003736 0.0003736 0.0003736 0.04% Amr::coarseTimeStep() 10 0.0003308 0.0003308 0.0003308 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002803 0.0002803 0.0002803 0.03% MultiFab::max() 11 0.0002549 0.0002549 0.0002549 0.03% MLCellLinOp::correctionResidual() 492 0.0002365 0.0002365 0.0002365 0.03% FabArray::ParallelCopy() 861 0.0002343 0.0002343 0.0002343 0.03% MLCellLinOp::defineBC() 11 0.0002105 0.0002105 0.0002105 0.02% FillPatchIterator::Initialize 41 0.0002095 0.0002095 0.0002095 0.02% Castro::subcycle_advance_ctu() 10 0.0001814 0.0001814 0.0001814 0.02% MLLinOp::defineGrids() 11 0.0001776 0.0001776 0.0001776 0.02% Amr::timeStep() 10 0.0001502 0.0001502 0.0001502 0.02% Gravity::solve_for_phi() 10 0.0001439 0.0001439 0.0001439 0.02% StateData::checkPoint() 12 0.0001343 0.0001343 0.0001343 0.02% Castro::advance() 10 0.0001247 0.0001247 0.0001247 0.01% Gravity::update_max_rhs() 11 0.0001094 0.0001094 0.0001094 0.01% MLMG:computeResOfCorrection() 410 0.0001057 0.0001057 0.0001057 0.01% MLMG::mgVcycle_down::0 82 9.138e-05 9.138e-05 9.138e-05 0.01% Castro::expand_state() 10 8.924e-05 8.924e-05 8.924e-05 0.01% Castro::Castro() 1 8.74e-05 8.74e-05 8.74e-05 0.01% MLMG::actualBottomSolve() 82 8.646e-05 8.646e-05 8.646e-05 0.01% FabArrayBase::FB::FB() 56 8.182e-05 8.182e-05 8.182e-05 0.01% Castro::clean_state() 62 7.84e-05 7.84e-05 7.84e-05 0.01% MLMG::mgVcycle_down::1 82 7.72e-05 7.72e-05 7.72e-05 0.01% Castro::finalize_advance() 10 7.475e-05 7.475e-05 7.475e-05 0.01% AmrLevel::checkPoint() 3 7.424e-05 7.424e-05 7.424e-05 0.01% MLMG::mgVcycle_down::2 82 7.225e-05 7.225e-05 7.225e-05 0.01% MLMG::solve() 11 7.065e-05 7.065e-05 7.065e-05 0.01% Castro::initialize_advance() 10 7.011e-05 7.011e-05 7.011e-05 0.01% MLMG::mgVcycle_down::3 82 6.753e-05 6.753e-05 6.753e-05 0.01% MLMG::mgVcycle_down::4 82 6.632e-05 6.632e-05 6.632e-05 0.01% MLMG::mgVcycle_up::4 82 5.538e-05 5.538e-05 5.538e-05 0.01% MLMG::mgVcycle_up::0 82 5.274e-05 5.274e-05 5.274e-05 0.01% MLMG::oneIter() 82 5.077e-05 5.077e-05 5.077e-05 0.01% MLCellLinOp::solutionResidual() 93 4.971e-05 4.971e-05 4.971e-05 0.01% MLMG::mgVcycle_up::1 82 4.751e-05 4.751e-05 4.751e-05 0.01% MLMG::mgVcycle_up::3 82 4.667e-05 4.667e-05 4.667e-05 0.01% MLMG::mgVcycle_up::2 82 4.533e-05 4.533e-05 4.533e-05 0.01% Castro::initialize_do_advance() 10 4.428e-05 4.428e-05 4.428e-05 0.01% Castro::swap_state_time_levels() 10 3.406e-05 3.406e-05 3.406e-05 0.00% Castro::enforce_consistent_e() 1 3.338e-05 3.338e-05 3.338e-05 0.00% Castro::finalize_do_advance() 10 3.322e-05 3.322e-05 3.322e-05 0.00% MLMG::ResNormInf() 93 3.265e-05 3.265e-05 3.265e-05 0.00% FillPatchSingleLevel 41 3.053e-05 3.053e-05 3.053e-05 0.00% MLMG::mgVcycle_bottom 82 3.036e-05 3.036e-05 3.036e-05 0.00% StateData::define() 4 3.016e-05 3.016e-05 3.016e-05 0.00% Castro::initMFs() 1 2.901e-05 2.901e-05 2.901e-05 0.00% MLMG::computeResidual() 82 2.799e-05 2.799e-05 2.799e-05 0.00% makeSFC 55 2.695e-05 2.695e-05 2.695e-05 0.00% Castro::create_source_corrector() 10 2.528e-05 2.528e-05 2.528e-05 0.00% Amr::writeSmallPlotFile() 1 2.517e-05 2.517e-05 2.517e-05 0.00% Amr::defBaseLevel() 1 2.476e-05 2.476e-05 2.476e-05 0.00% MLPoisson::define() 11 2.428e-05 2.428e-05 2.428e-05 0.00% Castro::construct_new_gravity() 10 2.415e-05 2.415e-05 2.415e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.304e-05 2.304e-05 2.304e-05 0.00% Amr::FinalizeInit() 1 1.882e-05 1.882e-05 1.882e-05 0.00% Castro::buildMetrics() 1 1.721e-05 1.721e-05 1.721e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.711e-05 1.711e-05 1.711e-05 0.00% Castro::construct_old_source() 50 1.704e-05 1.704e-05 1.704e-05 0.00% Castro::construct_new_source() 50 1.678e-05 1.678e-05 1.678e-05 0.00% Castro::do_new_sources() 10 1.635e-05 1.635e-05 1.635e-05 0.00% Castro::do_old_sources() 10 1.536e-05 1.536e-05 1.536e-05 0.00% DistributionMapping::Distribute() 56 1.52e-05 1.52e-05 1.52e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.486e-05 1.486e-05 1.486e-05 0.00% Castro::post_init() 1 1.291e-05 1.291e-05 1.291e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.275e-05 1.275e-05 1.275e-05 0.00% Castro::check_for_nan() 20 1.22e-05 1.22e-05 1.22e-05 0.00% Amr::InitializeInit() 1 1.203e-05 1.203e-05 1.203e-05 0.00% Castro::construct_old_gravity() 10 9.899e-06 9.899e-06 9.899e-06 0.00% Castro::post_timestep() 10 9.677e-06 9.677e-06 9.677e-06 0.00% Castro::apply_source_to_state() 20 9.455e-06 9.455e-06 9.455e-06 0.00% Gravity::swapTimeLevels() 10 9.272e-06 9.272e-06 9.272e-06 0.00% MLLinOp::define() 11 9.238e-06 9.238e-06 9.238e-06 0.00% Amr::initSubcycle() 1 8.667e-06 8.667e-06 8.667e-06 0.00% MLPoisson::prepareForSolve() 11 8.115e-06 8.115e-06 8.115e-06 0.00% Gravity::actual_multilevel_solve() 1 7.54e-06 7.54e-06 7.54e-06 0.00% Castro::computeNewDt() 9 7.369e-06 7.369e-06 7.369e-06 0.00% MLMG::computeMLResidual() 11 7.315e-06 7.315e-06 7.315e-06 0.00% MLMG::getGradSolution() 11 6.099e-06 6.099e-06 6.099e-06 0.00% AmrLevel::checkPointPost() 3 5.35e-06 5.35e-06 5.35e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.258e-06 4.258e-06 4.258e-06 0.00% Castro::retry_advance_ctu() 10 4.174e-06 4.174e-06 4.174e-06 0.00% MLMG::MLRhsNormInf() 11 3.801e-06 3.801e-06 3.801e-06 0.00% Gravity::set_mass_offset() 11 3.629e-06 3.629e-06 3.629e-06 0.00% MLMG::MLResNormInf() 11 3.363e-06 3.363e-06 3.363e-06 0.00% Castro::FluxRegCrseInit 10 2.864e-06 2.864e-06 2.864e-06 0.00% Amr::init() 1 2.573e-06 2.573e-06 2.573e-06 0.00% Castro::computeInitialDt() 2 2.443e-06 2.443e-06 2.443e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.101e-06 2.101e-06 2.101e-06 0.00% Castro::FluxRegFineAdd() 10 2.06e-06 2.06e-06 2.06e-06 0.00% AmrLevel::checkPointPre() 3 1.721e-06 1.721e-06 1.721e-06 0.00% Castro::post_regrid() 1 1.286e-06 1.286e-06 1.286e-06 0.00% Amr::initialInit() 1 1.054e-06 1.054e-06 1.054e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8477 0.8477 0.8477 100.00% Amr::coarseTimeStep() 10 0.6732 0.6732 0.6732 79.41% Amr::timeStep() 10 0.5534 0.5534 0.5534 65.28% Castro::advance() 10 0.5462 0.5462 0.5462 64.43% Castro::subcycle_advance_ctu() 10 0.5338 0.5338 0.5338 62.97% Castro::do_advance_ctu() 10 0.5336 0.5336 0.5336 62.95% Gravity::solve_phi_with_mlmg() 11 0.2794 0.2794 0.2794 32.96% Gravity::actual_solve_with_mlmg() 11 0.2725 0.2725 0.2725 32.15% Castro::construct_new_gravity() 10 0.2545 0.2545 0.2545 30.02% MLMG::solve() 11 0.2524 0.2524 0.2524 29.78% Gravity::solve_for_phi() 10 0.2394 0.2394 0.2394 28.24% MLMG::oneIter() 82 0.2383 0.2383 0.2383 28.11% MLMG::mgVcycle() 82 0.2347 0.2347 0.2347 27.69% VisMF::Write(FabArray) 11 0.2339 0.2339 0.2339 27.59% Castro::construct_ctu_hydro_source() 10 0.2026 0.2026 0.2026 23.90% Amr::checkPoint() 3 0.174 0.174 0.174 20.52% AmrLevel::checkPoint() 3 0.1699 0.1699 0.1699 20.05% StateData::checkPoint() 12 0.1699 0.1699 0.1699 20.04% Amr::init() 1 0.1401 0.1401 0.1401 16.53% MLCellLinOp::smooth() 1640 0.1153 0.1153 0.1153 13.60% MLCellLinOp::applyBC() 4433 0.1075 0.1075 0.1075 12.68% MLMG::mgVcycle_bottom 82 0.07302 0.07302 0.07302 8.61% MLMG::actualBottomSolve() 82 0.07299 0.07299 0.07299 8.61% MLCGSolver::bicgstab 82 0.07229 0.07229 0.07229 8.53% Amr::writePlotFile() 2 0.06707 0.06707 0.06707 7.91% Amr::initialInit() 1 0.04819 0.04819 0.04819 5.69% Castro::clean_state() 62 0.04566 0.04566 0.04566 5.39% Amr::FinalizeInit() 1 0.04378 0.04378 0.04378 5.16% Castro::post_init() 1 0.04246 0.04246 0.04246 5.01% FillPatchIterator::Initialize 41 0.04162 0.04162 0.04162 4.91% Gravity::multilevel_solve_for_new_phi() 1 0.04049 0.04049 0.04049 4.78% Gravity::actual_multilevel_solve() 1 0.04048 0.04048 0.04048 4.78% FillPatchSingleLevel 41 0.04012 0.04012 0.04012 4.73% StateDataPhysBCFunct::() 41 0.03612 0.03612 0.03612 4.26% MLCellLinOp::apply() 1142 0.03583 0.03583 0.03583 4.23% MLMG::mgVcycle_down::0 82 0.03343 0.03343 0.03343 3.94% FabArray::FillBoundary() 4023 0.03242 0.03242 0.03242 3.83% MLPoisson::Fsmooth() 3280 0.03212 0.03212 0.03212 3.79% FillBoundary_nowait() 4023 0.03163 0.03163 0.03163 3.73% MLMG::mgVcycle_up::0 82 0.02528 0.02528 0.02528 2.98% StateData::FillBoundary(geom) 328 0.02338 0.02338 0.02338 2.76% MLCellLinOp::correctionResidual() 492 0.02191 0.02191 0.02191 2.58% Castro::computeTemp() 63 0.02048 0.02048 0.02048 2.42% amrex::Dot() 1114 0.02034 0.02034 0.02034 2.40% MLMG:computeResOfCorrection() 410 0.01932 0.01932 0.01932 2.28% Castro::initialize_do_advance() 10 0.01895 0.01895 0.01895 2.24% Gravity::get_new_grav_vector() 11 0.01682 0.01682 0.01682 1.98% Castro::normalize_species() 62 0.01628 0.01628 0.01628 1.92% MLPoisson::define() 11 0.01592 0.01592 0.01592 1.88% MLMG::mgVcycle_down::1 82 0.01549 0.01549 0.01549 1.83% amrex::Copy() 1029 0.01479 0.01479 0.01479 1.74% Castro::construct_old_gravity() 10 0.01466 0.01466 0.01466 1.73% Gravity::get_old_grav_vector() 10 0.01465 0.01465 0.01465 1.73% MLMG::mgVcycle_down::2 82 0.01451 0.01451 0.01451 1.71% FabArray::norminf() 743 0.01427 0.01427 0.01427 1.68% MLMG::mgVcycle_down::3 82 0.0141 0.0141 0.0141 1.66% FabArray::ParallelCopy() 861 0.01405 0.01405 0.01405 1.66% MLMG::mgVcycle_down::4 82 0.01395 0.01395 0.01395 1.65% FabArray::ParallelCopy_nowait() 861 0.01382 0.01382 0.01382 1.63% FabArray::setVal() 1144 0.0131 0.0131 0.0131 1.55% Castro::do_new_sources() 10 0.01294 0.01294 0.01294 1.53% MLCGSolver::ParallelAllReduce 1514 0.01217 0.01217 0.01217 1.44% Castro::initialize_advance() 10 0.01165 0.01165 0.01165 1.37% MLMG::addInterpCorrection() 410 0.01159 0.01159 0.01159 1.37% Castro::expand_state() 10 0.01147 0.01147 0.01147 1.35% MLMG::mgVcycle_up::4 82 0.01139 0.01139 0.01139 1.34% MLMG::mgVcycle_up::1 82 0.01129 0.01129 0.01129 1.33% MLMG::mgVcycle_up::2 82 0.01104 0.01104 0.01104 1.30% MLCellLinOp::defineAuxData() 11 0.0109 0.0109 0.0109 1.29% amrex::average_down 410 0.01085 0.01085 0.01085 1.28% MLMG::mgVcycle_up::3 82 0.01083 0.01083 0.01083 1.28% MLPoisson::Fapply() 1142 0.01036 0.01036 0.01036 1.22% Castro::do_old_sources() 10 0.01029 0.01029 0.01029 1.21% Castro::enforce_min_density() 62 0.008882 0.008882 0.008882 1.05% FabArray::Saxpy() 813 0.008093 0.008093 0.008093 0.95% FabArray::Xpay() 821 0.008034 0.008034 0.008034 0.95% MLCellLinOp::solutionResidual() 93 0.007106 0.007106 0.007106 0.84% Castro::post_timestep() 10 0.007038 0.007038 0.007038 0.83% Gravity::fill_multipole_BCs() 11 0.006688 0.006688 0.006688 0.79% Castro::reset_internal_energy(MultiFab) 63 0.006223 0.006223 0.006223 0.73% MLMG::computeResidual() 82 0.006139 0.006139 0.006139 0.72% Castro::estTimeStep() 21 0.005584 0.005584 0.005584 0.66% MLCellLinOp::defineBC() 11 0.004755 0.004755 0.004755 0.56% MLMG::prepareForSolve() 11 0.004573 0.004573 0.004573 0.54% BndryData::define() 11 0.004544 0.004544 0.004544 0.54% FabArray::LinComb() 557 0.004505 0.004505 0.004505 0.53% Amr::InitializeInit() 1 0.004414 0.004414 0.004414 0.52% Amr::defBaseLevel() 1 0.004402 0.004402 0.004402 0.52% amrex::Add() 164 0.004297 0.004297 0.004297 0.51% Castro::initData() 1 0.003868 0.003868 0.003868 0.46% Castro::construct_new_source() 50 0.003218 0.003218 0.003218 0.38% Castro::construct_new_gravity_source() 10 0.003201 0.003201 0.003201 0.38% Castro::computeNewDt() 9 0.003124 0.003124 0.003124 0.37% Castro::construct_old_source() 50 0.002617 0.002617 0.002617 0.31% Castro::construct_old_gravity_source() 10 0.0026 0.0026 0.0026 0.31% MLMG::ResNormInf() 93 0.00212 0.00212 0.00212 0.25% Castro::apply_source_to_state() 20 0.001817 0.001817 0.001817 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00163 0.00163 0.00163 0.19% Castro::reset_internal_energy(Fab) 504 0.001503 0.001503 0.001503 0.18% FabArrayBase::getCPC() 1323 0.001394 0.001394 0.001394 0.16% MLCellLinOp::setLevelBC() 11 0.001393 0.001393 0.001393 0.16% MLMG::getGradSolution() 11 0.001378 0.001378 0.001378 0.16% MLCellLinOp::compGrad() 11 0.001372 0.001372 0.001372 0.16% FabArray::mult() 43 0.001339 0.001339 0.001339 0.16% FabArray::setDomainBndry() 41 0.001288 0.001288 0.001288 0.15% Castro::check_for_nan() 20 0.001183 0.001183 0.001183 0.14% MultiFab::contains_nan() 20 0.001171 0.001171 0.001171 0.14% Castro::post_regrid() 1 0.001136 0.001136 0.001136 0.13% MLPoisson::prepareForSolve() 11 0.001099 0.001099 0.001099 0.13% MLCellLinOp::prepareForSolve() 11 0.001091 0.001091 0.001091 0.13% MLMG::computeMLResidual() 11 0.001002 0.001002 0.001002 0.12% Castro::enforce_speed_limit() 62 0.0009025 0.0009025 0.0009025 0.11% Castro::computeInitialDt() 2 0.0008742 0.0008742 0.0008742 0.10% Gravity::update_max_rhs() 11 0.0008248 0.0008248 0.0008248 0.10% FabArrayBase::getFB() 4023 0.0006909 0.0006909 0.0006909 0.08% FabArrayBase::CPC::define() 454 0.0006797 0.0006797 0.0006797 0.08% Castro::finalize_advance() 10 0.0005886 0.0005886 0.0005886 0.07% Amr::InitAmr() 1 0.0005102 0.0005102 0.0005102 0.06% Castro::Castro() 1 0.000457 0.000457 0.000457 0.05% Gravity::swapTimeLevels() 10 0.0004307 0.0004307 0.0004307 0.05% MLMG::MLResNormInf() 11 0.0002899 0.0002899 0.0002899 0.03% MultiFab::max() 11 0.0002549 0.0002549 0.0002549 0.03% MLLinOp::define() 11 0.0002426 0.0002426 0.0002426 0.03% MLLinOp::defineGrids() 11 0.0002334 0.0002334 0.0002334 0.03% MLMG::MLRhsNormInf() 11 0.0002187 0.0002187 0.0002187 0.03% Castro::buildMetrics() 1 0.0001609 0.0001609 0.0001609 0.02% FabArrayBase::FB::FB() 56 8.182e-05 8.182e-05 8.182e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.367e-05 5.367e-05 5.367e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.726e-05 4.726e-05 4.726e-05 0.01% makeSFC 55 4.091e-05 4.091e-05 4.091e-05 0.00% Castro::swap_state_time_levels() 10 3.406e-05 3.406e-05 3.406e-05 0.00% Castro::enforce_consistent_e() 1 3.338e-05 3.338e-05 3.338e-05 0.00% Castro::finalize_do_advance() 10 3.322e-05 3.322e-05 3.322e-05 0.00% StateData::define() 4 3.016e-05 3.016e-05 3.016e-05 0.00% Castro::initMFs() 1 2.901e-05 2.901e-05 2.901e-05 0.00% Castro::create_source_corrector() 10 2.528e-05 2.528e-05 2.528e-05 0.00% Amr::writeSmallPlotFile() 1 2.517e-05 2.517e-05 2.517e-05 0.00% DistributionMapping::Distribute() 56 1.52e-05 1.52e-05 1.52e-05 0.00% Amr::initSubcycle() 1 8.667e-06 8.667e-06 8.667e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.491e-06 5.491e-06 5.491e-06 0.00% AmrLevel::checkPointPost() 3 5.35e-06 5.35e-06 5.35e-06 0.00% Castro::retry_advance_ctu() 10 4.174e-06 4.174e-06 4.174e-06 0.00% Gravity::set_mass_offset() 11 3.629e-06 3.629e-06 3.629e-06 0.00% Castro::FluxRegCrseInit 10 2.864e-06 2.864e-06 2.864e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.101e-06 2.101e-06 2.101e-06 0.00% Castro::FluxRegFineAdd() 10 2.06e-06 2.06e-06 2.06e-06 0.00% AmrLevel::checkPointPre() 3 1.721e-06 1.721e-06 1.721e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-15-gf972714bd111) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.01-15-gf972714bd111) initialized Starting run at 10:10:23 UTC on 2023-01-12. Successfully read inputs file ... Castro git describe: 23.01-14-g78178b78e AMReX git describe: 23.01-15-gf972714bd Microphysics git describe: 23.01-3-g1e475055 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.460722668 Restart time = 0.072871329 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.050589268 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049411687 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057501109 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.056995018 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062957484 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.038032278 seconds Ending run at 10:10:24 UTC on 2023-01-12. Run time = 0.389481443 Run time without initialization = 0.315983706 Average number of zones advanced per microsecond: 4.148 Average number of zones advanced per microsecond per rank: 4.148 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3895 ... 0.3895 ... 0.3895 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0957 0.0957 0.0957 24.57% VisMF::Read() 3 0.04123 0.04123 0.04123 10.58% VisMF::Write(FabArray) 1 0.03641 0.03641 0.03641 9.35% MLCellLinOp::applyBC() 1946 0.03268 0.03268 0.03268 8.39% Amr::restart() 1 0.02796 0.02796 0.02796 7.18% MLPoisson::Fsmooth() 1440 0.01407 0.01407 0.01407 3.61% FillBoundary_nowait() 1766 0.01262 0.01262 0.01262 3.24% StateData::FillBoundary(geom) 160 0.01179 0.01179 0.01179 3.03% amrex::Dot() 484 0.008726 0.008726 0.008726 2.24% Castro::normalize_species() 30 0.008722 0.008722 0.008722 2.24% amrex::Copy() 463 0.007013 0.007013 0.007013 1.80% Castro::computeTemp() 30 0.006534 0.006534 0.006534 1.68% FabArray::setVal() 537 0.006259 0.006259 0.006259 1.61% FabArray::norminf() 326 0.006188 0.006188 0.006188 1.59% Castro::enforce_min_density() 30 0.006054 0.006054 0.006054 1.55% FabArray::ParallelCopy_nowait() 380 0.005933 0.005933 0.005933 1.52% MLCellLinOp::defineAuxData() 6 0.005321 0.005321 0.005321 1.37% StateDataPhysBCFunct::() 20 0.00514 0.00514 0.00514 1.32% MLPoisson::Fapply() 500 0.00451 0.00451 0.00451 1.16% FabArray::Saxpy() 355 0.003628 0.003628 0.003628 0.93% FabArray::Xpay() 361 0.003526 0.003526 0.003526 0.91% MLMG::addInterpCorrection() 180 0.002821 0.002821 0.002821 0.72% amrex::average_down 180 0.002526 0.002526 0.002526 0.65% Gravity::fill_multipole_BCs() 6 0.002501 0.002501 0.002501 0.64% Castro::estTimeStep() 10 0.002261 0.002261 0.002261 0.58% BndryData::define() 6 0.002043 0.002043 0.002043 0.52% FabArray::LinComb() 242 0.001948 0.001948 0.001948 0.50% Castro::reset_internal_energy(MultiFab) 30 0.001898 0.001898 0.001898 0.49% amrex::Add() 72 0.001841 0.001841 0.001841 0.47% Castro::construct_new_gravity_source() 5 0.001681 0.001681 0.001681 0.43% Castro::do_advance_ctu() 5 0.001598 0.001598 0.001598 0.41% Amr::writePlotFile() 1 0.001465 0.001465 0.001465 0.38% Castro::construct_old_gravity_source() 5 0.001178 0.001178 0.001178 0.30% Castro::reset_internal_energy(Fab) 240 0.0009517 0.0009517 0.0009517 0.24% MLCGSolver::bicgstab 36 0.0009327 0.0009327 0.0009327 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008901 0.0008901 0.0008901 0.23% Castro::computeNewDt() 5 0.0007747 0.0007747 0.0007747 0.20% MLCellLinOp::setLevelBC() 6 0.0007366 0.0007366 0.0007366 0.19% Gravity::actual_solve_with_mlmg() 6 0.0007295 0.0007295 0.0007295 0.19% FabArray::mult() 22 0.0006587 0.0006587 0.0006587 0.17% FabArray::setDomainBndry() 20 0.0006502 0.0006502 0.0006502 0.17% MLCellLinOp::prepareForSolve() 6 0.0005957 0.0005957 0.0005957 0.15% MultiFab::contains_nan() 10 0.0005918 0.0005918 0.0005918 0.15% Castro::enforce_speed_limit() 30 0.0005818 0.0005818 0.0005818 0.15% MLCellLinOp::smooth() 720 0.0005504 0.0005504 0.0005504 0.14% MLCellLinOp::compGrad() 6 0.0004881 0.0004881 0.0004881 0.13% MLMG::prepareForSolve() 6 0.0004709 0.0004709 0.0004709 0.12% Amr::InitAmr() 1 0.0004186 0.0004186 0.0004186 0.11% FabArrayBase::CPC::define() 244 0.0004024 0.0004024 0.0004024 0.10% Gravity::get_old_grav_vector() 5 0.0003699 0.0003699 0.0003699 0.09% FabArrayBase::getCPC() 632 0.000352 0.000352 0.000352 0.09% FabArray::FillBoundary() 1766 0.0003484 0.0003484 0.0003484 0.09% main() 1 0.0003348 0.0003348 0.0003348 0.09% Gravity::get_new_grav_vector() 5 0.0002966 0.0002966 0.0002966 0.08% FabArrayBase::getFB() 1766 0.000257 0.000257 0.000257 0.07% MLCellLinOp::apply() 500 0.0002116 0.0002116 0.0002116 0.05% Amr::coarseTimeStep() 5 0.000201 0.000201 0.000201 0.05% MLMG::mgVcycle() 36 0.0001728 0.0001728 0.0001728 0.04% MultiFab::max() 6 0.0001345 0.0001345 0.0001345 0.03% Castro::advance() 5 0.0001325 0.0001325 0.0001325 0.03% Castro::subcycle_advance_ctu() 5 0.0001324 0.0001324 0.0001324 0.03% MLCGSolver::ParallelAllReduce 659 0.0001278 0.0001278 0.0001278 0.03% Castro::construct_new_source() 25 0.0001177 0.0001177 0.0001177 0.03% FillPatchIterator::Initialize 20 0.0001123 0.0001123 0.0001123 0.03% FabArray::ParallelCopy() 380 0.0001105 0.0001105 0.0001105 0.03% MLCellLinOp::defineBC() 6 0.0001102 0.0001102 0.0001102 0.03% MLLinOp::defineGrids() 6 0.0001098 0.0001098 0.0001098 0.03% MLCellLinOp::correctionResidual() 216 0.000102 0.000102 0.000102 0.03% Gravity::solve_for_phi() 5 8.196e-05 8.196e-05 8.196e-05 0.02% Amr::timeStep() 5 7.747e-05 7.747e-05 7.747e-05 0.02% Castro::construct_new_gravity() 5 7.29e-05 7.29e-05 7.29e-05 0.02% AmrLevel::restart() 1 6.753e-05 6.753e-05 6.753e-05 0.02% Castro::finalize_advance() 5 5.894e-05 5.894e-05 5.894e-05 0.02% FabArrayBase::FB::FB() 26 5.77e-05 5.77e-05 5.77e-05 0.01% StateData::restartDoit() 4 5.642e-05 5.642e-05 5.642e-05 0.01% Gravity::update_max_rhs() 6 5.475e-05 5.475e-05 5.475e-05 0.01% MLMG:computeResOfCorrection() 180 4.904e-05 4.904e-05 4.904e-05 0.01% Castro::post_timestep() 5 4.511e-05 4.511e-05 4.511e-05 0.01% Castro::clean_state() 30 4.419e-05 4.419e-05 4.419e-05 0.01% Castro::initialize_do_advance() 5 4.061e-05 4.061e-05 4.061e-05 0.01% MLMG::actualBottomSolve() 36 3.91e-05 3.91e-05 3.91e-05 0.01% MLMG::mgVcycle_down::0 36 3.868e-05 3.868e-05 3.868e-05 0.01% Castro::expand_state() 5 3.81e-05 3.81e-05 3.81e-05 0.01% MLMG::mgVcycle_down::1 36 3.561e-05 3.561e-05 3.561e-05 0.01% MLMG::solve() 6 3.311e-05 3.311e-05 3.311e-05 0.01% MLMG::mgVcycle_down::2 36 3.244e-05 3.244e-05 3.244e-05 0.01% MLMG::mgVcycle_down::4 36 3.131e-05 3.131e-05 3.131e-05 0.01% MLMG::mgVcycle_down::3 36 3.043e-05 3.043e-05 3.043e-05 0.01% Castro::initialize_advance() 5 3.008e-05 3.008e-05 3.008e-05 0.01% Amr::writeSmallPlotFile() 1 2.771e-05 2.771e-05 2.771e-05 0.01% MLMG::mgVcycle_up::4 36 2.75e-05 2.75e-05 2.75e-05 0.01% Castro::buildMetrics() 1 2.531e-05 2.531e-05 2.531e-05 0.01% MLMG::oneIter() 36 2.353e-05 2.353e-05 2.353e-05 0.01% Castro::create_source_corrector() 5 2.304e-05 2.304e-05 2.304e-05 0.01% MLMG::mgVcycle_up::0 36 2.291e-05 2.291e-05 2.291e-05 0.01% MLCellLinOp::solutionResidual() 42 2.194e-05 2.194e-05 2.194e-05 0.01% Castro::post_restart() 1 2.133e-05 2.133e-05 2.133e-05 0.01% Castro::initMFs() 1 2.073e-05 2.073e-05 2.073e-05 0.01% MLMG::mgVcycle_up::3 36 2.07e-05 2.07e-05 2.07e-05 0.01% MLMG::mgVcycle_up::2 36 2.042e-05 2.042e-05 2.042e-05 0.01% Castro::swap_state_time_levels() 5 2.018e-05 2.018e-05 2.018e-05 0.01% MLMG::mgVcycle_up::1 36 1.959e-05 1.959e-05 1.959e-05 0.01% Castro::finalize_do_advance() 5 1.844e-05 1.844e-05 1.844e-05 0.00% FillPatchSingleLevel 20 1.835e-05 1.835e-05 1.835e-05 0.00% MLMG::ResNormInf() 42 1.739e-05 1.739e-05 1.739e-05 0.00% MLMG::mgVcycle_bottom 36 1.438e-05 1.438e-05 1.438e-05 0.00% MLPoisson::define() 6 1.384e-05 1.384e-05 1.384e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.325e-05 1.325e-05 1.325e-05 0.00% makeSFC 30 1.304e-05 1.304e-05 1.304e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.276e-05 1.276e-05 1.276e-05 0.00% MLMG::computeResidual() 36 1.252e-05 1.252e-05 1.252e-05 0.00% Castro::do_new_sources() 5 1.069e-05 1.069e-05 1.069e-05 0.00% Castro::construct_old_source() 25 9.805e-06 9.805e-06 9.805e-06 0.00% DistributionMapping::Distribute() 31 8.637e-06 8.637e-06 8.637e-06 0.00% Amr::initSubcycle() 1 8.349e-06 8.349e-06 8.349e-06 0.00% Castro::do_old_sources() 5 8.01e-06 8.01e-06 8.01e-06 0.00% Castro::check_for_nan() 10 6.772e-06 6.772e-06 6.772e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.74e-06 6.74e-06 6.74e-06 0.00% Gravity::actual_multilevel_solve() 1 6.567e-06 6.567e-06 6.567e-06 0.00% Castro::apply_source_to_state() 10 5.761e-06 5.761e-06 5.761e-06 0.00% MLLinOp::define() 6 5.591e-06 5.591e-06 5.591e-06 0.00% Castro::construct_old_gravity() 5 4.797e-06 4.797e-06 4.797e-06 0.00% MLPoisson::prepareForSolve() 6 4.192e-06 4.192e-06 4.192e-06 0.00% Gravity::swapTimeLevels() 5 4.099e-06 4.099e-06 4.099e-06 0.00% MLMG::computeMLResidual() 6 3.488e-06 3.488e-06 3.488e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.26e-06 3.26e-06 3.26e-06 0.00% MLMG::getGradSolution() 6 3.057e-06 3.057e-06 3.057e-06 0.00% MLMG::MLResNormInf() 6 2.249e-06 2.249e-06 2.249e-06 0.00% MLMG::MLRhsNormInf() 6 2.163e-06 2.163e-06 2.163e-06 0.00% Gravity::set_mass_offset() 6 1.897e-06 1.897e-06 1.897e-06 0.00% Castro::retry_advance_ctu() 5 1.703e-06 1.703e-06 1.703e-06 0.00% Castro::FluxRegCrseInit 5 1.432e-06 1.432e-06 1.432e-06 0.00% Castro::FluxRegFineAdd() 5 1.211e-06 1.211e-06 1.211e-06 0.00% AmrLevel::AmrLevel() 1 1.183e-06 1.183e-06 1.183e-06 0.00% Amr::init() 1 1.123e-06 1.123e-06 1.123e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.108e-06 1.108e-06 1.108e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3895 0.3895 0.3895 100.00% Amr::coarseTimeStep() 5 0.2776 0.2776 0.2776 71.28% Amr::timeStep() 5 0.2754 0.2754 0.2754 70.72% Castro::advance() 5 0.2699 0.2699 0.2699 69.30% Castro::subcycle_advance_ctu() 5 0.2641 0.2641 0.2641 67.79% Castro::do_advance_ctu() 5 0.2639 0.2639 0.2639 67.76% Castro::construct_new_gravity() 5 0.1282 0.1282 0.1282 32.92% Gravity::solve_phi_with_mlmg() 6 0.1234 0.1234 0.1234 31.67% Gravity::actual_solve_with_mlmg() 6 0.1206 0.1206 0.1206 30.97% Gravity::solve_for_phi() 5 0.1204 0.1204 0.1204 30.90% MLMG::solve() 6 0.1095 0.1095 0.1095 28.11% MLMG::oneIter() 36 0.1027 0.1027 0.1027 26.36% MLMG::mgVcycle() 36 0.1012 0.1012 0.1012 25.97% Castro::construct_ctu_hydro_source() 5 0.09571 0.09571 0.09571 24.57% Amr::init() 1 0.07293 0.07293 0.07293 18.72% Amr::restart() 1 0.07293 0.07293 0.07293 18.72% MLCellLinOp::smooth() 720 0.04956 0.04956 0.04956 12.72% MLCellLinOp::applyBC() 1946 0.04596 0.04596 0.04596 11.80% AmrLevel::restart() 1 0.04141 0.04141 0.04141 10.63% StateData::restartDoit() 4 0.04134 0.04134 0.04134 10.61% VisMF::Read() 3 0.04123 0.04123 0.04123 10.58% Amr::writePlotFile() 1 0.03813 0.03813 0.03813 9.79% VisMF::Write(FabArray) 1 0.03641 0.03641 0.03641 9.35% MLMG::mgVcycle_bottom 36 0.03152 0.03152 0.03152 8.09% MLMG::actualBottomSolve() 36 0.03151 0.03151 0.03151 8.09% MLCGSolver::bicgstab 36 0.0312 0.0312 0.0312 8.01% Castro::clean_state() 30 0.02478 0.02478 0.02478 6.36% FillPatchIterator::Initialize 20 0.01968 0.01968 0.01968 5.05% FillPatchSingleLevel 20 0.01892 0.01892 0.01892 4.86% StateDataPhysBCFunct::() 20 0.01693 0.01693 0.01693 4.35% MLCellLinOp::apply() 500 0.01549 0.01549 0.01549 3.98% MLPoisson::Fsmooth() 1440 0.01407 0.01407 0.01407 3.61% MLMG::mgVcycle_down::0 36 0.01406 0.01406 0.01406 3.61% FabArray::FillBoundary() 1766 0.01328 0.01328 0.01328 3.41% FillBoundary_nowait() 1766 0.01293 0.01293 0.01293 3.32% StateData::FillBoundary(geom) 160 0.01179 0.01179 0.01179 3.03% MLMG::mgVcycle_up::0 36 0.01067 0.01067 0.01067 2.74% Castro::initialize_do_advance() 5 0.009558 0.009558 0.009558 2.45% MLCellLinOp::correctionResidual() 216 0.009423 0.009423 0.009423 2.42% Castro::computeTemp() 30 0.009384 0.009384 0.009384 2.41% MLPoisson::define() 6 0.008908 0.008908 0.008908 2.29% amrex::Dot() 484 0.008726 0.008726 0.008726 2.24% Castro::normalize_species() 30 0.008722 0.008722 0.008722 2.24% MLMG:computeResOfCorrection() 180 0.008275 0.008275 0.008275 2.12% Gravity::get_new_grav_vector() 5 0.007701 0.007701 0.007701 1.98% Castro::do_new_sources() 5 0.007443 0.007443 0.007443 1.91% Castro::construct_old_gravity() 5 0.007346 0.007346 0.007346 1.89% Gravity::get_old_grav_vector() 5 0.007342 0.007342 0.007342 1.88% amrex::Copy() 463 0.007013 0.007013 0.007013 1.80% MLMG::mgVcycle_down::1 36 0.006815 0.006815 0.006815 1.75% FabArray::ParallelCopy() 380 0.006411 0.006411 0.006411 1.65% FabArray::ParallelCopy_nowait() 380 0.0063 0.0063 0.0063 1.62% MLMG::mgVcycle_down::2 36 0.006297 0.006297 0.006297 1.62% FabArray::setVal() 537 0.006259 0.006259 0.006259 1.61% FabArray::norminf() 326 0.006188 0.006188 0.006188 1.59% MLMG::mgVcycle_down::3 36 0.00614 0.00614 0.00614 1.58% MLMG::mgVcycle_down::4 36 0.006078 0.006078 0.006078 1.56% Castro::enforce_min_density() 30 0.006054 0.006054 0.006054 1.55% MLCellLinOp::defineAuxData() 6 0.006052 0.006052 0.006052 1.55% Castro::do_old_sources() 5 0.005511 0.005511 0.005511 1.41% Castro::post_timestep() 5 0.005442 0.005442 0.005442 1.40% Castro::initialize_advance() 5 0.005413 0.005413 0.005413 1.39% Castro::expand_state() 5 0.005413 0.005413 0.005413 1.39% MLCGSolver::ParallelAllReduce 659 0.005268 0.005268 0.005268 1.35% MLMG::addInterpCorrection() 180 0.005042 0.005042 0.005042 1.29% MLMG::mgVcycle_up::4 36 0.004934 0.004934 0.004934 1.27% MLMG::mgVcycle_up::1 36 0.004923 0.004923 0.004923 1.26% MLMG::mgVcycle_up::2 36 0.004818 0.004818 0.004818 1.24% amrex::average_down 180 0.004741 0.004741 0.004741 1.22% MLMG::mgVcycle_up::3 36 0.004727 0.004727 0.004727 1.21% MLPoisson::Fapply() 500 0.00451 0.00451 0.00451 1.16% FabArray::Saxpy() 355 0.003628 0.003628 0.003628 0.93% FabArray::Xpay() 361 0.003526 0.003526 0.003526 0.91% Castro::post_restart() 1 0.003388 0.003388 0.003388 0.87% Gravity::multilevel_solve_for_new_phi() 1 0.003273 0.003273 0.003273 0.84% Gravity::actual_multilevel_solve() 1 0.00326 0.00326 0.00326 0.84% MLCellLinOp::solutionResidual() 42 0.003216 0.003216 0.003216 0.83% Castro::reset_internal_energy(MultiFab) 30 0.00285 0.00285 0.00285 0.73% MLCellLinOp::defineBC() 6 0.002698 0.002698 0.002698 0.69% MLMG::computeResidual() 36 0.002665 0.002665 0.002665 0.68% Gravity::fill_multipole_BCs() 6 0.002626 0.002626 0.002626 0.67% BndryData::define() 6 0.002588 0.002588 0.002588 0.66% MLMG::prepareForSolve() 6 0.002493 0.002493 0.002493 0.64% Castro::estTimeStep() 10 0.002261 0.002261 0.002261 0.58% Castro::computeNewDt() 5 0.001998 0.001998 0.001998 0.51% FabArray::LinComb() 242 0.001948 0.001948 0.001948 0.50% amrex::Add() 72 0.001841 0.001841 0.001841 0.47% Castro::construct_new_source() 25 0.001799 0.001799 0.001799 0.46% Castro::construct_new_gravity_source() 5 0.001681 0.001681 0.001681 0.43% Castro::construct_old_source() 25 0.001188 0.001188 0.001188 0.30% Castro::construct_old_gravity_source() 5 0.001178 0.001178 0.001178 0.30% Castro::reset_internal_energy(Fab) 240 0.0009517 0.0009517 0.0009517 0.24% MLMG::ResNormInf() 42 0.0009348 0.0009348 0.0009348 0.24% Castro::apply_source_to_state() 10 0.0009211 0.0009211 0.0009211 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008901 0.0008901 0.0008901 0.23% FabArrayBase::getCPC() 632 0.0007544 0.0007544 0.0007544 0.19% MLMG::getGradSolution() 6 0.0007529 0.0007529 0.0007529 0.19% MLCellLinOp::compGrad() 6 0.0007498 0.0007498 0.0007498 0.19% MLCellLinOp::setLevelBC() 6 0.0007366 0.0007366 0.0007366 0.19% FabArray::mult() 22 0.0006587 0.0006587 0.0006587 0.17% FabArray::setDomainBndry() 20 0.0006502 0.0006502 0.0006502 0.17% MLPoisson::prepareForSolve() 6 0.0005999 0.0005999 0.0005999 0.15% Castro::check_for_nan() 10 0.0005986 0.0005986 0.0005986 0.15% MLCellLinOp::prepareForSolve() 6 0.0005957 0.0005957 0.0005957 0.15% MultiFab::contains_nan() 10 0.0005918 0.0005918 0.0005918 0.15% Castro::enforce_speed_limit() 30 0.0005818 0.0005818 0.0005818 0.15% MLMG::computeMLResidual() 6 0.0005666 0.0005666 0.0005666 0.15% Gravity::update_max_rhs() 6 0.0004446 0.0004446 0.0004446 0.11% Amr::InitAmr() 1 0.0004269 0.0004269 0.0004269 0.11% FabArrayBase::CPC::define() 244 0.0004024 0.0004024 0.0004024 0.10% Castro::finalize_advance() 5 0.0003228 0.0003228 0.0003228 0.08% FabArrayBase::getFB() 1766 0.0003147 0.0003147 0.0003147 0.08% Gravity::swapTimeLevels() 5 0.0002242 0.0002242 0.0002242 0.06% MLMG::MLResNormInf() 6 0.000149 0.000149 0.000149 0.04% Castro::buildMetrics() 1 0.0001481 0.0001481 0.0001481 0.04% MLLinOp::define() 6 0.0001435 0.0001435 0.0001435 0.04% MLLinOp::defineGrids() 6 0.0001379 0.0001379 0.0001379 0.04% MultiFab::max() 6 0.0001345 0.0001345 0.0001345 0.03% MLMG::MLRhsNormInf() 6 0.0001161 0.0001161 0.0001161 0.03% FabArrayBase::FB::FB() 26 5.77e-05 5.77e-05 5.77e-05 0.01% Amr::writeSmallPlotFile() 1 2.771e-05 2.771e-05 2.771e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.698e-05 2.698e-05 2.698e-05 0.01% Castro::create_source_corrector() 5 2.304e-05 2.304e-05 2.304e-05 0.01% Castro::initMFs() 1 2.073e-05 2.073e-05 2.073e-05 0.01% makeSFC 30 2.024e-05 2.024e-05 2.024e-05 0.01% Castro::swap_state_time_levels() 5 2.018e-05 2.018e-05 2.018e-05 0.01% Castro::finalize_do_advance() 5 1.844e-05 1.844e-05 1.844e-05 0.00% DistributionMapping::Distribute() 31 8.637e-06 8.637e-06 8.637e-06 0.00% Amr::initSubcycle() 1 8.349e-06 8.349e-06 8.349e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.696e-06 4.696e-06 4.696e-06 0.00% Gravity::set_mass_offset() 6 1.897e-06 1.897e-06 1.897e-06 0.00% Castro::retry_advance_ctu() 5 1.703e-06 1.703e-06 1.703e-06 0.00% Castro::FluxRegCrseInit 5 1.432e-06 1.432e-06 1.432e-06 0.00% Castro::FluxRegFineAdd() 5 1.211e-06 1.211e-06 1.211e-06 0.00% AmrLevel::AmrLevel() 1 1.183e-06 1.183e-06 1.183e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.108e-06 1.108e-06 1.108e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-15-gf972714bd111) finalized