Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-26-g0d136ea53aed) initialized Starting run at 08:24:23 UTC on 2022-04-19. Successfully read inputs file ... Castro git describe: 22.04-12-g78961ce2c AMReX git describe: 22.04-26-g0d136ea53 Microphysics git describe: 22.04-3-g3c498521 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.041871576 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025387563 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.112188245 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.070059712 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.061155455 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.070209362 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.070195737 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.041772084 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052559832 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049000907 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.061700502 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.073045032 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.069251815 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.039192913 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024204588 seconds Ending run at 08:24:24 UTC on 2022-04-19. Run time = 0.924670239 Run time without initialization = 0.795023938 Average number of zones advanced per microsecond: 3.297 Average number of zones advanced per microsecond per rank: 3.297 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.9247 ... 0.9247 ... 0.9247 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2530 0.2530 0.2530 27.36% VisMF::Write(FabArray) 11 0.1644 0.1644 0.1644 17.78% MLCellLinOp::applyBC() 4433 0.09545 0.09545 0.09545 10.32% MLPoisson::Fsmooth() 3280 0.06085 0.06085 0.06085 6.58% StateDataPhysBCFunct::() 41 0.03411 0.03411 0.03411 3.69% FabArray::setVal() 1144 0.02679 0.02679 0.02679 2.90% StateData::FillBoundary(geom) 328 0.0237 0.0237 0.0237 2.56% MLCGSolver::bicgstab 82 0.0222 0.0222 0.0222 2.40% MultiFab::Dot() 1114 0.02198 0.02198 0.02198 2.38% FillBoundary_nowait() 4023 0.01781 0.01781 0.01781 1.93% FabArray::ParallelCopy_nowait() 861 0.01647 0.01647 0.01647 1.78% Castro::computeTemp() 63 0.01402 0.01402 0.01402 1.52% Castro::enforce_min_density() 62 0.01336 0.01336 0.01336 1.44% MultiFab::LinComb() 1586 0.01254 0.01254 0.01254 1.36% Gravity::fill_multipole_BCs() 11 0.01238 0.01238 0.01238 1.34% MLPoisson::Fapply() 1142 0.01082 0.01082 0.01082 1.17% MLCellLinOp::defineAuxData() 11 0.01047 0.01047 0.01047 1.13% MLMG::addInterpCorrection() 410 0.009941 0.009941 0.009941 1.08% amrex::average_down 410 0.006686 0.006686 0.006686 0.72% Castro::estTimeStep() 21 0.006438 0.006438 0.006438 0.70% Castro::reset_internal_energy() 63 0.006251 0.006251 0.006251 0.68% MultiFab::Xpay() 585 0.006238 0.006238 0.006238 0.67% FabArray::setDomainBndry() 41 0.005643 0.005643 0.005643 0.61% Castro::enforce_speed_limit() 62 0.005608 0.005608 0.005608 0.61% Amr::checkPoint() 3 0.004816 0.004816 0.004816 0.52% Castro::expand_state() 10 0.004728 0.004728 0.004728 0.51% Castro::normalize_species() 62 0.004417 0.004417 0.004417 0.48% BndryData::define() 11 0.003831 0.003831 0.003831 0.41% Castro::do_advance_ctu() 10 0.003329 0.003329 0.003329 0.36% Amr::writePlotFile() 2 0.003233 0.003233 0.003233 0.35% Castro::construct_new_gravity_source() 10 0.003017 0.003017 0.003017 0.33% Gravity::get_new_grav_vector() 11 0.002869 0.002869 0.002869 0.31% Castro::construct_old_gravity_source() 10 0.002699 0.002699 0.002699 0.29% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002311 0.002311 0.002311 0.25% MLCellLinOp::compGrad() 11 0.002054 0.002054 0.002054 0.22% MLMG::ResNormInf() 93 0.00197 0.00197 0.00197 0.21% Gravity::get_old_grav_vector() 10 0.001931 0.001931 0.001931 0.21% MultiFab::Saxpy() 20 0.00182 0.00182 0.00182 0.20% MLMG::oneIter() 82 0.001726 0.001726 0.001726 0.19% MLCellLinOp::setLevelBC() 11 0.001569 0.001569 0.001569 0.17% Gravity::actual_solve_with_mlmg() 11 0.001426 0.001426 0.001426 0.15% MLCellLinOp::prepareForSolve() 11 0.00137 0.00137 0.00137 0.15% FabArray::mult() 43 0.001345 0.001345 0.001345 0.15% Castro::initData() 1 0.001305 0.001305 0.001305 0.14% MLCellLinOp::smooth() 1640 0.001179 0.001179 0.001179 0.13% MultiFab::contains_nan() 20 0.001173 0.001173 0.001173 0.13% FabArrayBase::getCPC() 1323 0.0008836 0.0008836 0.0008836 0.10% FabArray::FillBoundary() 4023 0.0008376 0.0008376 0.0008376 0.09% MLMG::prepareForSolve() 11 0.0007832 0.0007832 0.0007832 0.08% FabArrayBase::CPC::define() 454 0.0007129 0.0007129 0.0007129 0.08% FabArrayBase::getFB() 4023 0.0006693 0.0006693 0.0006693 0.07% Gravity::update_max_rhs() 11 0.0005513 0.0005513 0.0005513 0.06% MultiFab::Copy() 11 0.0005333 0.0005333 0.0005333 0.06% MLCellLinOp::apply() 1142 0.0004892 0.0004892 0.0004892 0.05% CGSolver::sxay() 1586 0.0004616 0.0004616 0.0004616 0.05% Amr::InitAmr() 1 0.0004589 0.0004589 0.0004589 0.05% Gravity::solve_for_phi() 10 0.0004399 0.0004399 0.0004399 0.05% MLMG::mgVcycle() 82 0.000423 0.000423 0.000423 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003317 0.0003317 0.0003317 0.04% MultiFab::min() 10 0.0003304 0.0003304 0.0003304 0.04% FabArray::ParallelCopy() 861 0.0002972 0.0002972 0.0002972 0.03% main() 1 0.0002863 0.0002863 0.0002863 0.03% Castro::construct_new_gravity() 10 0.0002666 0.0002666 0.0002666 0.03% MultiFab::max() 11 0.0002609 0.0002609 0.0002609 0.03% FillPatchIterator::Initialize 41 0.0002577 0.0002577 0.0002577 0.03% Gravity::actual_multilevel_solve() 1 0.0002355 0.0002355 0.0002355 0.03% MLCellLinOp::correctionResidual() 492 0.0002221 0.0002221 0.0002221 0.02% Amr::coarseTimeStep() 10 0.000212 0.000212 0.000212 0.02% MLMG::MLRhsNormInf() 11 0.0002027 0.0002027 0.0002027 0.02% Amr::timeStep() 10 0.0001939 0.0001939 0.0001939 0.02% MLCellLinOp::defineBC() 11 0.0001877 0.0001877 0.0001877 0.02% MLLinOp::defineGrids() 11 0.0001732 0.0001732 0.0001732 0.02% Castro::subcycle_advance_ctu() 10 0.0001562 0.0001562 0.0001562 0.02% Amr::defBaseLevel() 1 0.0001509 0.0001509 0.0001509 0.02% MLMG:computeResOfCorrection() 410 0.0001455 0.0001455 0.0001455 0.02% StateData::checkPoint() 12 0.0001278 0.0001278 0.0001278 0.01% MLMG::actualBottomSolve() 82 0.0001068 0.0001068 0.0001068 0.01% FabArrayBase::FB::FB() 56 9.123e-05 9.123e-05 9.123e-05 0.01% MLMG::mgVcycle_down::0 82 9.029e-05 9.029e-05 9.029e-05 0.01% MLMG::mgVcycle_down::1 82 8.416e-05 8.416e-05 8.416e-05 0.01% MLMG::solve() 11 8.062e-05 8.062e-05 8.062e-05 0.01% MLMG::mgVcycle_down::2 82 8.031e-05 8.031e-05 8.031e-05 0.01% MLMG::mgVcycle_down::4 82 7.805e-05 7.805e-05 7.805e-05 0.01% Castro::initialize_advance() 10 7.623e-05 7.623e-05 7.623e-05 0.01% MLMG::mgVcycle_down::3 82 7.557e-05 7.557e-05 7.557e-05 0.01% Castro::clean_state() 62 7.002e-05 7.002e-05 7.002e-05 0.01% AmrLevel::checkPoint() 3 6.908e-05 6.908e-05 6.908e-05 0.01% MLMG::mgVcycle_up::4 82 6.727e-05 6.727e-05 6.727e-05 0.01% Castro::initialize_do_advance() 10 6.456e-05 6.456e-05 6.456e-05 0.01% Castro::advance() 10 6.28e-05 6.28e-05 6.28e-05 0.01% Castro::finalize_advance() 10 5.836e-05 5.836e-05 5.836e-05 0.01% MLCellLinOp::solutionResidual() 93 5.214e-05 5.214e-05 5.214e-05 0.01% MLMG::mgVcycle_up::0 82 5.15e-05 5.15e-05 5.15e-05 0.01% MLMG::mgVcycle_up::2 82 5.146e-05 5.146e-05 5.146e-05 0.01% MLMG::mgVcycle_up::1 82 5.094e-05 5.094e-05 5.094e-05 0.01% MLMG::mgVcycle_up::3 82 4.965e-05 4.965e-05 4.965e-05 0.01% StateData::define() 4 4.396e-05 4.396e-05 4.396e-05 0.00% Castro::construct_new_source() 50 4.388e-05 4.388e-05 4.388e-05 0.00% Castro::finalize_do_advance() 10 3.884e-05 3.884e-05 3.884e-05 0.00% Castro::swap_state_time_levels() 10 3.844e-05 3.844e-05 3.844e-05 0.00% MLMG::computeResidual() 82 3.609e-05 3.609e-05 3.609e-05 0.00% MLMG::mgVcycle_bottom 82 3.531e-05 3.531e-05 3.531e-05 0.00% Castro::enforce_consistent_e() 1 3.515e-05 3.515e-05 3.515e-05 0.00% MLLinOp::define() 11 2.994e-05 2.994e-05 2.994e-05 0.00% FillPatchSingleLevel 41 2.965e-05 2.965e-05 2.965e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.785e-05 2.785e-05 2.785e-05 0.00% makeSFC 55 2.561e-05 2.561e-05 2.561e-05 0.00% MLPoisson::define() 11 2.518e-05 2.518e-05 2.518e-05 0.00% Amr::writeSmallPlotFile() 1 2.406e-05 2.406e-05 2.406e-05 0.00% Castro::construct_old_source() 50 2.191e-05 2.191e-05 2.191e-05 0.00% Castro::do_new_sources() 10 2.055e-05 2.055e-05 2.055e-05 0.00% Amr::FinalizeInit() 1 2.012e-05 2.012e-05 2.012e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.771e-05 1.771e-05 1.771e-05 0.00% Castro::do_old_sources() 10 1.633e-05 1.633e-05 1.633e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.623e-05 1.623e-05 1.623e-05 0.00% DistributionMapping::Distribute() 56 1.555e-05 1.555e-05 1.555e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.518e-05 1.518e-05 1.518e-05 0.00% Castro::check_for_nan() 20 1.346e-05 1.346e-05 1.346e-05 0.00% Castro::apply_source_to_state() 20 1.137e-05 1.137e-05 1.137e-05 0.00% Castro::construct_old_gravity() 10 1.116e-05 1.116e-05 1.116e-05 0.00% Amr::initSubcycle() 1 9.806e-06 9.806e-06 9.806e-06 0.00% Gravity::swapTimeLevels() 10 9.437e-06 9.437e-06 9.437e-06 0.00% MLPoisson::prepareForSolve() 11 9.185e-06 9.185e-06 9.185e-06 0.00% Castro::create_source_corrector() 10 7.316e-06 7.316e-06 7.316e-06 0.00% MLMG::computeMLResidual() 11 7.191e-06 7.191e-06 7.191e-06 0.00% Castro::post_timestep() 10 7.164e-06 7.164e-06 7.164e-06 0.00% Amr::InitializeInit() 1 6.569e-06 6.569e-06 6.569e-06 0.00% Castro::computeNewDt() 9 6.369e-06 6.369e-06 6.369e-06 0.00% MLMG::buildFineMask() 11 5.698e-06 5.698e-06 5.698e-06 0.00% MLMG::getGradSolution() 11 5.382e-06 5.382e-06 5.382e-06 0.00% MLMG::MLResNormInf() 11 4.769e-06 4.769e-06 4.769e-06 0.00% Gravity::set_mass_offset() 11 4.604e-06 4.604e-06 4.604e-06 0.00% AmrLevel::checkPointPost() 3 4.577e-06 4.577e-06 4.577e-06 0.00% Castro::post_init() 1 4.263e-06 4.263e-06 4.263e-06 0.00% Castro::retry_advance_ctu() 10 3.949e-06 3.949e-06 3.949e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.361e-06 3.361e-06 3.361e-06 0.00% Castro::FluxRegFineAdd() 10 2.714e-06 2.714e-06 2.714e-06 0.00% Amr::init() 1 2.425e-06 2.425e-06 2.425e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.388e-06 2.388e-06 2.388e-06 0.00% Castro::computeInitialDt() 2 2.109e-06 2.109e-06 2.109e-06 0.00% AmrLevel::checkPointPre() 3 2e-06 2e-06 2e-06 0.00% Castro::post_regrid() 1 1.238e-06 1.238e-06 1.238e-06 0.00% Amr::initialInit() 1 9.06e-07 9.06e-07 9.06e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9247 0.9247 0.9247 100.00% Amr::coarseTimeStep() 10 0.7706 0.7706 0.7706 83.34% Amr::timeStep() 10 0.6862 0.6862 0.6862 74.21% Castro::advance() 10 0.6762 0.6762 0.6762 73.12% Castro::subcycle_advance_ctu() 10 0.661 0.661 0.661 71.48% Castro::do_advance_ctu() 10 0.6608 0.6608 0.6608 71.46% Gravity::solve_phi_with_mlmg() 11 0.3353 0.3353 0.3353 36.26% Gravity::actual_solve_with_mlmg() 11 0.3227 0.3227 0.3227 34.90% Castro::construct_new_gravity() 10 0.3225 0.3225 0.3225 34.87% MLMG::solve() 11 0.2993 0.2993 0.2993 32.37% Gravity::solve_for_phi() 10 0.2867 0.2867 0.2867 31.00% MLMG::oneIter() 82 0.2832 0.2832 0.2832 30.62% MLMG::mgVcycle() 82 0.2814 0.2814 0.2814 30.43% Castro::construct_ctu_hydro_source() 10 0.253 0.253 0.253 27.36% VisMF::Write(FabArray) 11 0.1644 0.1644 0.1644 17.78% MLCellLinOp::smooth() 1640 0.1472 0.1472 0.1472 15.92% Amr::init() 1 0.1291 0.1291 0.1291 13.96% Amr::checkPoint() 3 0.123 0.123 0.123 13.30% AmrLevel::checkPoint() 3 0.1181 0.1181 0.1181 12.77% StateData::checkPoint() 12 0.1181 0.1181 0.1181 12.77% MLCellLinOp::applyBC() 4433 0.1149 0.1149 0.1149 12.42% MLMG::mgVcycle_bottom 82 0.07949 0.07949 0.07949 8.60% MLMG::actualBottomSolve() 82 0.07946 0.07946 0.07946 8.59% MLCGSolver::bicgstab 82 0.07871 0.07871 0.07871 8.51% FillPatchIterator::Initialize 41 0.06794 0.06794 0.06794 7.35% FillPatchSingleLevel 41 0.06204 0.06204 0.06204 6.71% Amr::initialInit() 1 0.06169 0.06169 0.06169 6.67% MLPoisson::Fsmooth() 3280 0.06085 0.06085 0.06085 6.58% StateDataPhysBCFunct::() 41 0.05781 0.05781 0.05781 6.25% Amr::FinalizeInit() 1 0.05252 0.05252 0.05252 5.68% Castro::post_init() 1 0.05179 0.05179 0.05179 5.60% Amr::writePlotFile() 2 0.0497 0.0497 0.0497 5.37% Gravity::multilevel_solve_for_new_phi() 1 0.04935 0.04935 0.04935 5.34% Gravity::actual_multilevel_solve() 1 0.04933 0.04933 0.04933 5.34% Castro::clean_state() 62 0.04288 0.04288 0.04288 4.64% MLCellLinOp::apply() 1142 0.04041 0.04041 0.04041 4.37% MLMG::mgVcycle_down::0 82 0.03878 0.03878 0.03878 4.19% Gravity::get_new_grav_vector() 11 0.03782 0.03782 0.03782 4.09% MLMG::mgVcycle_up::0 82 0.03324 0.03324 0.03324 3.59% FabArray::setVal() 1144 0.02679 0.02679 0.02679 2.90% Castro::initialize_do_advance() 10 0.02501 0.02501 0.02501 2.70% StateData::FillBoundary(geom) 328 0.0237 0.0237 0.0237 2.56% MLCellLinOp::correctionResidual() 492 0.02276 0.02276 0.02276 2.46% MultiFab::Dot() 1114 0.02198 0.02198 0.02198 2.38% Castro::computeTemp() 63 0.02027 0.02027 0.02027 2.19% MLMG:computeResOfCorrection() 410 0.01971 0.01971 0.01971 2.13% FabArray::FillBoundary() 4023 0.01941 0.01941 0.01941 2.10% MLMG::mgVcycle_down::1 82 0.01896 0.01896 0.01896 2.05% FillBoundary_nowait() 4023 0.01857 0.01857 0.01857 2.01% Castro::expand_state() 10 0.01854 0.01854 0.01854 2.01% MLMG::mgVcycle_down::2 82 0.01832 0.01832 0.01832 1.98% MLPoisson::define() 11 0.01779 0.01779 0.01779 1.92% FabArray::ParallelCopy() 861 0.01766 0.01766 0.01766 1.91% MLMG::mgVcycle_down::3 82 0.01745 0.01745 0.01745 1.89% FabArray::ParallelCopy_nowait() 861 0.01736 0.01736 0.01736 1.88% Castro::construct_old_gravity() 10 0.01674 0.01674 0.01674 1.81% Gravity::get_old_grav_vector() 10 0.01673 0.01673 0.01673 1.81% MLMG::addInterpCorrection() 410 0.01669 0.01669 0.01669 1.81% MLMG::mgVcycle_down::4 82 0.01666 0.01666 0.01666 1.80% MLMG::mgVcycle_up::4 82 0.01632 0.01632 0.01632 1.76% Castro::initialize_advance() 10 0.0151 0.0151 0.0151 1.63% MLMG::mgVcycle_up::2 82 0.01422 0.01422 0.01422 1.54% MLMG::mgVcycle_up::1 82 0.014 0.014 0.014 1.51% MLMG::mgVcycle_up::3 82 0.01356 0.01356 0.01356 1.47% amrex::average_down 410 0.0134 0.0134 0.0134 1.45% Castro::enforce_min_density() 62 0.01336 0.01336 0.01336 1.44% MLCGSolver::ParallelAllReduce 1514 0.01322 0.01322 0.01322 1.43% CGSolver::sxay() 1586 0.01301 0.01301 0.01301 1.41% MultiFab::LinComb() 1586 0.01254 0.01254 0.01254 1.36% Gravity::fill_multipole_BCs() 11 0.01238 0.01238 0.01238 1.34% Castro::do_new_sources() 10 0.01225 0.01225 0.01225 1.32% MLCellLinOp::defineAuxData() 11 0.01218 0.01218 0.01218 1.32% MLPoisson::Fapply() 1142 0.01082 0.01082 0.01082 1.17% Castro::do_old_sources() 10 0.009957 0.009957 0.009957 1.08% Castro::post_timestep() 10 0.009829 0.009829 0.009829 1.06% Amr::InitializeInit() 1 0.009163 0.009163 0.009163 0.99% Amr::defBaseLevel() 1 0.009156 0.009156 0.009156 0.99% MLCellLinOp::solutionResidual() 93 0.007927 0.007927 0.007927 0.86% MLMG::computeResidual() 82 0.006856 0.006856 0.006856 0.74% Castro::estTimeStep() 21 0.006438 0.006438 0.006438 0.70% Castro::reset_internal_energy() 63 0.006251 0.006251 0.006251 0.68% MultiFab::Xpay() 585 0.006238 0.006238 0.006238 0.67% MLMG::prepareForSolve() 11 0.005914 0.005914 0.005914 0.64% FabArray::setDomainBndry() 41 0.005643 0.005643 0.005643 0.61% Castro::enforce_speed_limit() 62 0.005608 0.005608 0.005608 0.61% MLCellLinOp::defineBC() 11 0.005324 0.005324 0.005324 0.58% BndryData::define() 11 0.005136 0.005136 0.005136 0.56% Castro::normalize_species() 62 0.004417 0.004417 0.004417 0.48% Castro::initData() 1 0.003587 0.003587 0.003587 0.39% Castro::construct_new_source() 50 0.003061 0.003061 0.003061 0.33% Castro::construct_new_gravity_source() 10 0.003017 0.003017 0.003017 0.33% Castro::construct_old_source() 50 0.002721 0.002721 0.002721 0.29% Castro::construct_old_gravity_source() 10 0.002699 0.002699 0.002699 0.29% Castro::computeNewDt() 9 0.002678 0.002678 0.002678 0.29% MLMG::getGradSolution() 11 0.002614 0.002614 0.002614 0.28% MLCellLinOp::compGrad() 11 0.002608 0.002608 0.002608 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002311 0.002311 0.002311 0.25% MLMG::ResNormInf() 93 0.00197 0.00197 0.00197 0.21% Castro::apply_source_to_state() 20 0.001832 0.001832 0.001832 0.20% MultiFab::Saxpy() 20 0.00182 0.00182 0.00182 0.20% FabArrayBase::getCPC() 1323 0.001597 0.001597 0.001597 0.17% MLCellLinOp::setLevelBC() 11 0.001569 0.001569 0.001569 0.17% MLPoisson::prepareForSolve() 11 0.001379 0.001379 0.001379 0.15% MLCellLinOp::prepareForSolve() 11 0.00137 0.00137 0.00137 0.15% FabArray::mult() 43 0.001345 0.001345 0.001345 0.15% Castro::check_for_nan() 20 0.001186 0.001186 0.001186 0.13% MultiFab::contains_nan() 20 0.001173 0.001173 0.001173 0.13% Gravity::swapTimeLevels() 10 0.001158 0.001158 0.001158 0.13% MLMG::computeMLResidual() 11 0.001114 0.001114 0.001114 0.12% Gravity::update_max_rhs() 11 0.0009652 0.0009652 0.0009652 0.10% FabArrayBase::getFB() 4023 0.0007605 0.0007605 0.0007605 0.08% FabArrayBase::CPC::define() 454 0.0007129 0.0007129 0.0007129 0.08% Castro::computeInitialDt() 2 0.0006798 0.0006798 0.0006798 0.07% MultiFab::Copy() 11 0.0005333 0.0005333 0.0005333 0.06% Castro::post_regrid() 1 0.000501 0.000501 0.000501 0.05% Amr::InitAmr() 1 0.0004687 0.0004687 0.0004687 0.05% MultiFab::min() 10 0.0003304 0.0003304 0.0003304 0.04% MLLinOp::define() 11 0.0002615 0.0002615 0.0002615 0.03% MultiFab::max() 11 0.0002609 0.0002609 0.0002609 0.03% MLMG::MLResNormInf() 11 0.000253 0.000253 0.000253 0.03% MLLinOp::defineGrids() 11 0.0002315 0.0002315 0.0002315 0.03% MLMG::MLRhsNormInf() 11 0.0002027 0.0002027 0.0002027 0.02% FabArrayBase::FB::FB() 56 9.123e-05 9.123e-05 9.123e-05 0.01% Castro::finalize_advance() 10 6.107e-05 6.107e-05 6.107e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.914e-05 5.914e-05 5.914e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.598e-05 5.598e-05 5.598e-05 0.01% StateData::define() 4 4.396e-05 4.396e-05 4.396e-05 0.00% makeSFC 55 3.975e-05 3.975e-05 3.975e-05 0.00% Castro::finalize_do_advance() 10 3.884e-05 3.884e-05 3.884e-05 0.00% Castro::swap_state_time_levels() 10 3.844e-05 3.844e-05 3.844e-05 0.00% Castro::enforce_consistent_e() 1 3.515e-05 3.515e-05 3.515e-05 0.00% Amr::writeSmallPlotFile() 1 2.406e-05 2.406e-05 2.406e-05 0.00% DistributionMapping::Distribute() 56 1.555e-05 1.555e-05 1.555e-05 0.00% Amr::initSubcycle() 1 9.806e-06 9.806e-06 9.806e-06 0.00% Castro::create_source_corrector() 10 7.316e-06 7.316e-06 7.316e-06 0.00% MLMG::buildFineMask() 11 5.698e-06 5.698e-06 5.698e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.766e-06 4.766e-06 4.766e-06 0.00% Gravity::set_mass_offset() 11 4.604e-06 4.604e-06 4.604e-06 0.00% AmrLevel::checkPointPost() 3 4.577e-06 4.577e-06 4.577e-06 0.00% Castro::retry_advance_ctu() 10 3.949e-06 3.949e-06 3.949e-06 0.00% Castro::FluxRegFineAdd() 10 2.714e-06 2.714e-06 2.714e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.388e-06 2.388e-06 2.388e-06 0.00% AmrLevel::checkPointPre() 3 2e-06 2e-06 2e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-26-g0d136ea53aed) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-26-g0d136ea53aed) initialized Starting run at 08:24:25 UTC on 2022-04-19. Successfully read inputs file ... Castro git describe: 22.04-12-g78961ce2c AMReX git describe: 22.04-26-g0d136ea53 Microphysics git describe: 22.04-3-g3c498521 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.555167476 Restart time = 0.056110173 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.113502181 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048846395 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.051921083 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.058717938 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.066996429 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024726174 seconds Ending run at 08:24:25 UTC on 2022-04-19. Run time = 0.421734389 Run time without initialization = 0.365044417 Average number of zones advanced per microsecond: 3.591 Average number of zones advanced per microsecond per rank: 3.591 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4218 ... 0.4218 ... 0.4218 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1307 0.1307 0.1307 30.99% VisMF::Read() 3 0.04407 0.04407 0.04407 10.45% MLCellLinOp::applyBC() 1946 0.03823 0.03823 0.03823 9.06% MLPoisson::Fsmooth() 1440 0.02586 0.02586 0.02586 6.13% VisMF::Write(FabArray) 1 0.02333 0.02333 0.02333 5.53% FabArray::setVal() 537 0.01703 0.01703 0.01703 4.04% StateData::FillBoundary(geom) 160 0.01163 0.01163 0.01163 2.76% MLCGSolver::bicgstab 36 0.009508 0.009508 0.009508 2.25% MultiFab::Dot() 484 0.009419 0.009419 0.009419 2.23% StateDataPhysBCFunct::() 20 0.00745 0.00745 0.00745 1.77% FillBoundary_nowait() 1766 0.007233 0.007233 0.007233 1.71% FabArray::ParallelCopy_nowait() 380 0.007198 0.007198 0.007198 1.71% Gravity::fill_multipole_BCs() 6 0.006076 0.006076 0.006076 1.44% Castro::computeTemp() 30 0.005907 0.005907 0.005907 1.40% MLCellLinOp::defineAuxData() 6 0.005639 0.005639 0.005639 1.34% MultiFab::LinComb() 690 0.005405 0.005405 0.005405 1.28% MLPoisson::Fapply() 500 0.004711 0.004711 0.004711 1.12% Castro::enforce_min_density() 30 0.004604 0.004604 0.004604 1.09% Castro::expand_state() 5 0.004145 0.004145 0.004145 0.98% FabArray::setDomainBndry() 20 0.003765 0.003765 0.003765 0.89% Amr::restart() 1 0.003087 0.003087 0.003087 0.73% MLMG::addInterpCorrection() 180 0.003028 0.003028 0.003028 0.72% amrex::average_down 180 0.002816 0.002816 0.002816 0.67% MultiFab::Xpay() 258 0.002721 0.002721 0.002721 0.65% Castro::estTimeStep() 10 0.002674 0.002674 0.002674 0.63% Castro::do_advance_ctu() 5 0.002458 0.002458 0.002458 0.58% Castro::reset_internal_energy() 30 0.002448 0.002448 0.002448 0.58% Castro::normalize_species() 30 0.002249 0.002249 0.002249 0.53% BndryData::define() 6 0.002062 0.002062 0.002062 0.49% Gravity::get_new_grav_vector() 5 0.001865 0.001865 0.001865 0.44% Castro::construct_new_gravity_source() 5 0.001537 0.001537 0.001537 0.36% MLCellLinOp::compGrad() 6 0.001488 0.001488 0.001488 0.35% Amr::writePlotFile() 1 0.001474 0.001474 0.001474 0.35% Castro::subcycle_advance_ctu() 5 0.001431 0.001431 0.001431 0.34% Castro::construct_old_gravity_source() 5 0.00133 0.00133 0.00133 0.32% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001228 0.001228 0.001228 0.29% Gravity::get_old_grav_vector() 5 0.001064 0.001064 0.001064 0.25% MultiFab::Saxpy() 10 0.0009224 0.0009224 0.0009224 0.22% MLMG::ResNormInf() 42 0.0008653 0.0008653 0.0008653 0.21% MLCellLinOp::setLevelBC() 6 0.0008364 0.0008364 0.0008364 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007711 0.0007711 0.0007711 0.18% MLMG::oneIter() 36 0.0007535 0.0007535 0.0007535 0.18% MLCellLinOp::prepareForSolve() 6 0.0007042 0.0007042 0.0007042 0.17% FabArray::mult() 22 0.0006596 0.0006596 0.0006596 0.16% MultiFab::contains_nan() 10 0.0005833 0.0005833 0.0005833 0.14% Castro::enforce_speed_limit() 30 0.0005556 0.0005556 0.0005556 0.13% Gravity::update_max_rhs() 6 0.0005386 0.0005386 0.0005386 0.13% MLCellLinOp::smooth() 720 0.0005105 0.0005105 0.0005105 0.12% MLMG::prepareForSolve() 6 0.0004269 0.0004269 0.0004269 0.10% FabArrayBase::getCPC() 632 0.0004268 0.0004268 0.0004268 0.10% FabArrayBase::CPC::define() 244 0.0004245 0.0004245 0.0004245 0.10% Amr::InitAmr() 1 0.0004051 0.0004051 0.0004051 0.10% FabArray::FillBoundary() 1766 0.0004032 0.0004032 0.0004032 0.10% MultiFab::Copy() 6 0.0003721 0.0003721 0.0003721 0.09% FabArrayBase::getFB() 1766 0.000315 0.000315 0.000315 0.07% main() 1 0.0002763 0.0002763 0.0002763 0.07% Castro::construct_new_gravity() 5 0.0002546 0.0002546 0.0002546 0.06% MLCellLinOp::apply() 500 0.0002343 0.0002343 0.0002343 0.06% Gravity::actual_multilevel_solve() 1 0.0002273 0.0002273 0.0002273 0.05% Gravity::solve_for_phi() 5 0.0002202 0.0002202 0.0002202 0.05% CGSolver::sxay() 690 0.0002116 0.0002116 0.0002116 0.05% MLMG::mgVcycle() 36 0.0001764 0.0001764 0.0001764 0.04% MultiFab::min() 5 0.0001642 0.0001642 0.0001642 0.04% FabArray::ParallelCopy() 380 0.0001452 0.0001452 0.0001452 0.03% MLCGSolver::ParallelAllReduce 659 0.0001438 0.0001438 0.0001438 0.03% MultiFab::max() 6 0.0001356 0.0001356 0.0001356 0.03% FillPatchIterator::Initialize 20 0.0001218 0.0001218 0.0001218 0.03% MLCellLinOp::correctionResidual() 216 0.0001092 0.0001092 0.0001092 0.03% MLMG::MLRhsNormInf() 6 0.000107 0.000107 0.000107 0.03% Castro::advance() 5 0.000106 0.000106 0.000106 0.03% Amr::coarseTimeStep() 5 0.0001051 0.0001051 0.0001051 0.02% Castro::construct_new_source() 25 0.0001016 0.0001016 0.0001016 0.02% MLCellLinOp::defineBC() 6 0.0001011 0.0001011 0.0001011 0.02% Amr::timeStep() 5 9.594e-05 9.594e-05 9.594e-05 0.02% MLLinOp::defineGrids() 6 9.359e-05 9.359e-05 9.359e-05 0.02% Castro::create_source_corrector() 5 7.538e-05 7.538e-05 7.538e-05 0.02% AmrLevel::restart() 1 7.499e-05 7.499e-05 7.499e-05 0.02% StateData::restartDoit() 4 7.217e-05 7.217e-05 7.217e-05 0.02% MLMG:computeResOfCorrection() 180 6.664e-05 6.664e-05 6.664e-05 0.02% FabArrayBase::FB::FB() 26 5.917e-05 5.917e-05 5.917e-05 0.01% MLMG::actualBottomSolve() 36 4.743e-05 4.743e-05 4.743e-05 0.01% MLMG::mgVcycle_down::0 36 4.042e-05 4.042e-05 4.042e-05 0.01% MLMG::mgVcycle_down::2 36 3.917e-05 3.917e-05 3.917e-05 0.01% MLMG::mgVcycle_down::1 36 3.808e-05 3.808e-05 3.808e-05 0.01% Castro::initialize_advance() 5 3.745e-05 3.745e-05 3.745e-05 0.01% MLMG::solve() 6 3.592e-05 3.592e-05 3.592e-05 0.01% Castro::clean_state() 30 3.553e-05 3.553e-05 3.553e-05 0.01% MLMG::mgVcycle_down::4 36 3.37e-05 3.37e-05 3.37e-05 0.01% MLMG::mgVcycle_down::3 36 3.319e-05 3.319e-05 3.319e-05 0.01% Castro::initialize_do_advance() 5 2.991e-05 2.991e-05 2.991e-05 0.01% Castro::construct_old_source() 25 2.937e-05 2.937e-05 2.937e-05 0.01% MLMG::mgVcycle_up::4 36 2.854e-05 2.854e-05 2.854e-05 0.01% MLLinOp::define() 6 2.639e-05 2.639e-05 2.639e-05 0.01% Amr::writeSmallPlotFile() 1 2.586e-05 2.586e-05 2.586e-05 0.01% Castro::swap_state_time_levels() 5 2.505e-05 2.505e-05 2.505e-05 0.01% Castro::finalize_advance() 5 2.475e-05 2.475e-05 2.475e-05 0.01% MLCellLinOp::solutionResidual() 42 2.457e-05 2.457e-05 2.457e-05 0.01% MLMG::mgVcycle_up::3 36 2.384e-05 2.384e-05 2.384e-05 0.01% MLMG::mgVcycle_up::0 36 2.373e-05 2.373e-05 2.373e-05 0.01% MLMG::mgVcycle_up::2 36 2.365e-05 2.365e-05 2.365e-05 0.01% Castro::post_restart() 1 2.321e-05 2.321e-05 2.321e-05 0.01% MLMG::mgVcycle_up::1 36 2.255e-05 2.255e-05 2.255e-05 0.01% Castro::post_timestep() 5 2.172e-05 2.172e-05 2.172e-05 0.01% Castro::finalize_do_advance() 5 1.975e-05 1.975e-05 1.975e-05 0.00% MLMG::mgVcycle_bottom 36 1.715e-05 1.715e-05 1.715e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.586e-05 1.586e-05 1.586e-05 0.00% MLMG::computeResidual() 36 1.564e-05 1.564e-05 1.564e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.551e-05 1.551e-05 1.551e-05 0.00% MLPoisson::define() 6 1.526e-05 1.526e-05 1.526e-05 0.00% FillPatchSingleLevel 20 1.496e-05 1.496e-05 1.496e-05 0.00% makeSFC 30 1.383e-05 1.383e-05 1.383e-05 0.00% DistributionMapping::Distribute() 31 9.542e-06 9.542e-06 9.542e-06 0.00% Amr::initSubcycle() 1 9.508e-06 9.508e-06 9.508e-06 0.00% Castro::do_new_sources() 5 9.262e-06 9.262e-06 9.262e-06 0.00% Castro::do_old_sources() 5 8.333e-06 8.333e-06 8.333e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.892e-06 7.892e-06 7.892e-06 0.00% Castro::apply_source_to_state() 10 6.445e-06 6.445e-06 6.445e-06 0.00% Castro::check_for_nan() 10 6.179e-06 6.179e-06 6.179e-06 0.00% Castro::construct_old_gravity() 5 5.517e-06 5.517e-06 5.517e-06 0.00% MLPoisson::prepareForSolve() 6 4.735e-06 4.735e-06 4.735e-06 0.00% Gravity::swapTimeLevels() 5 4.554e-06 4.554e-06 4.554e-06 0.00% Castro::computeNewDt() 5 3.481e-06 3.481e-06 3.481e-06 0.00% MLMG::computeMLResidual() 6 3.262e-06 3.262e-06 3.262e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.22e-06 3.22e-06 3.22e-06 0.00% MLMG::buildFineMask() 6 3.028e-06 3.028e-06 3.028e-06 0.00% MLMG::getGradSolution() 6 2.843e-06 2.843e-06 2.843e-06 0.00% MLMG::MLResNormInf() 6 2.588e-06 2.588e-06 2.588e-06 0.00% Gravity::set_mass_offset() 6 2.343e-06 2.343e-06 2.343e-06 0.00% Castro::retry_advance_ctu() 5 1.836e-06 1.836e-06 1.836e-06 0.00% Castro::FluxRegFineAdd() 5 1.82e-06 1.82e-06 1.82e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.106e-06 1.106e-06 1.106e-06 0.00% Amr::init() 1 1.013e-06 1.013e-06 1.013e-06 0.00% AmrLevel::AmrLevel() 1 9.82e-07 9.82e-07 9.82e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4217 0.4217 0.4217 99.99% Amr::coarseTimeStep() 5 0.3401 0.3401 0.3401 80.63% Amr::timeStep() 5 0.3384 0.3384 0.3384 80.23% Castro::advance() 5 0.3353 0.3353 0.3353 79.50% Castro::subcycle_advance_ctu() 5 0.3245 0.3245 0.3245 76.93% Castro::do_advance_ctu() 5 0.323 0.323 0.323 76.59% Castro::construct_new_gravity() 5 0.1485 0.1485 0.1485 35.20% Gravity::solve_phi_with_mlmg() 6 0.1443 0.1443 0.1443 34.20% Gravity::solve_for_phi() 5 0.1384 0.1384 0.1384 32.82% Gravity::actual_solve_with_mlmg() 6 0.1381 0.1381 0.1381 32.73% Castro::construct_ctu_hydro_source() 5 0.1307 0.1307 0.1307 30.99% MLMG::solve() 6 0.125 0.125 0.125 29.65% MLMG::oneIter() 36 0.117 0.117 0.117 27.73% MLMG::mgVcycle() 36 0.1162 0.1162 0.1162 27.55% MLCellLinOp::smooth() 720 0.06048 0.06048 0.06048 14.34% Amr::init() 1 0.05614 0.05614 0.05614 13.31% Amr::restart() 1 0.05614 0.05614 0.05614 13.31% MLCellLinOp::applyBC() 1946 0.04624 0.04624 0.04624 10.96% AmrLevel::restart() 1 0.04506 0.04506 0.04506 10.68% StateData::restartDoit() 4 0.04498 0.04498 0.04498 10.67% VisMF::Read() 3 0.04407 0.04407 0.04407 10.45% MLMG::mgVcycle_bottom 36 0.03383 0.03383 0.03383 8.02% MLMG::actualBottomSolve() 36 0.03381 0.03381 0.03381 8.02% MLCGSolver::bicgstab 36 0.03348 0.03348 0.03348 7.94% MLPoisson::Fsmooth() 1440 0.02586 0.02586 0.02586 6.13% FillPatchIterator::Initialize 20 0.02506 0.02506 0.02506 5.94% Amr::writePlotFile() 1 0.0248 0.0248 0.0248 5.88% VisMF::Write(FabArray) 1 0.02333 0.02333 0.02333 5.53% FillPatchSingleLevel 20 0.02117 0.02117 0.02117 5.02% StateDataPhysBCFunct::() 20 0.01908 0.01908 0.01908 4.52% FabArray::setVal() 537 0.01703 0.01703 0.01703 4.04% MLCellLinOp::apply() 500 0.01678 0.01678 0.01678 3.98% MLMG::mgVcycle_down::0 36 0.01608 0.01608 0.01608 3.81% Castro::clean_state() 30 0.0158 0.0158 0.0158 3.75% Castro::initialize_do_advance() 5 0.01388 0.01388 0.01388 3.29% MLMG::mgVcycle_up::0 36 0.0137 0.0137 0.0137 3.25% Castro::expand_state() 5 0.01163 0.01163 0.01163 2.76% StateData::FillBoundary(geom) 160 0.01163 0.01163 0.01163 2.76% Castro::initialize_advance() 5 0.0107 0.0107 0.0107 2.54% Gravity::get_new_grav_vector() 5 0.00978 0.00978 0.00978 2.32% MLPoisson::define() 6 0.009625 0.009625 0.009625 2.28% MLCellLinOp::correctionResidual() 216 0.009509 0.009509 0.009509 2.25% MultiFab::Dot() 484 0.009419 0.009419 0.009419 2.23% Castro::construct_old_gravity() 5 0.008531 0.008531 0.008531 2.02% Gravity::get_old_grav_vector() 5 0.008525 0.008525 0.008525 2.02% Castro::computeTemp() 30 0.008355 0.008355 0.008355 1.98% MLMG:computeResOfCorrection() 180 0.008231 0.008231 0.008231 1.95% FabArray::FillBoundary() 1766 0.00801 0.00801 0.00801 1.90% MLMG::mgVcycle_down::1 36 0.007878 0.007878 0.007878 1.87% FabArray::ParallelCopy() 380 0.007765 0.007765 0.007765 1.84% FabArray::ParallelCopy_nowait() 380 0.00762 0.00762 0.00762 1.81% FillBoundary_nowait() 1766 0.007607 0.007607 0.007607 1.80% MLMG::mgVcycle_down::2 36 0.007603 0.007603 0.007603 1.80% Castro::do_new_sources() 5 0.007317 0.007317 0.007317 1.73% MLMG::mgVcycle_down::3 36 0.007205 0.007205 0.007205 1.71% MLMG::mgVcycle_down::4 36 0.006937 0.006937 0.006937 1.64% Castro::post_restart() 1 0.006697 0.006697 0.006697 1.59% MLCellLinOp::defineAuxData() 6 0.006578 0.006578 0.006578 1.56% Gravity::multilevel_solve_for_new_phi() 1 0.006301 0.006301 0.006301 1.49% Gravity::actual_multilevel_solve() 1 0.006285 0.006285 0.006285 1.49% Gravity::fill_multipole_BCs() 6 0.006076 0.006076 0.006076 1.44% MLMG::mgVcycle_up::2 36 0.005881 0.005881 0.005881 1.39% MLMG::addInterpCorrection() 180 0.005869 0.005869 0.005869 1.39% MLMG::mgVcycle_up::1 36 0.005834 0.005834 0.005834 1.38% MLCGSolver::ParallelAllReduce 659 0.005696 0.005696 0.005696 1.35% amrex::average_down 180 0.005663 0.005663 0.005663 1.34% CGSolver::sxay() 690 0.005617 0.005617 0.005617 1.33% MLMG::mgVcycle_up::3 36 0.005568 0.005568 0.005568 1.32% MLMG::mgVcycle_up::4 36 0.005526 0.005526 0.005526 1.31% MultiFab::LinComb() 690 0.005405 0.005405 0.005405 1.28% MLPoisson::Fapply() 500 0.004711 0.004711 0.004711 1.12% Castro::enforce_min_density() 30 0.004604 0.004604 0.004604 1.09% Castro::do_old_sources() 5 0.003793 0.003793 0.003793 0.90% FabArray::setDomainBndry() 20 0.003765 0.003765 0.003765 0.89% MLMG::prepareForSolve() 6 0.003608 0.003608 0.003608 0.86% MLCellLinOp::solutionResidual() 42 0.003443 0.003443 0.003443 0.82% Castro::post_timestep() 5 0.002973 0.002973 0.002973 0.70% MLCellLinOp::defineBC() 6 0.002881 0.002881 0.002881 0.68% MLMG::computeResidual() 36 0.002854 0.002854 0.002854 0.68% BndryData::define() 6 0.00278 0.00278 0.00278 0.66% MultiFab::Xpay() 258 0.002721 0.002721 0.002721 0.65% Castro::estTimeStep() 10 0.002674 0.002674 0.002674 0.63% Castro::reset_internal_energy() 30 0.002448 0.002448 0.002448 0.58% Castro::normalize_species() 30 0.002249 0.002249 0.002249 0.53% MLMG::getGradSolution() 6 0.001779 0.001779 0.001779 0.42% MLCellLinOp::compGrad() 6 0.001776 0.001776 0.001776 0.42% Castro::construct_new_source() 25 0.001639 0.001639 0.001639 0.39% Castro::computeNewDt() 5 0.0016 0.0016 0.0016 0.38% Castro::construct_new_gravity_source() 5 0.001537 0.001537 0.001537 0.36% Castro::construct_old_source() 25 0.00136 0.00136 0.00136 0.32% Castro::construct_old_gravity_source() 5 0.00133 0.00133 0.00133 0.32% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001228 0.001228 0.001228 0.29% Castro::apply_source_to_state() 10 0.0009288 0.0009288 0.0009288 0.22% MultiFab::Saxpy() 10 0.0009224 0.0009224 0.0009224 0.22% Gravity::swapTimeLevels() 5 0.0009033 0.0009033 0.0009033 0.21% MLMG::ResNormInf() 42 0.0008653 0.0008653 0.0008653 0.21% FabArrayBase::getCPC() 632 0.0008513 0.0008513 0.0008513 0.20% MLCellLinOp::setLevelBC() 6 0.0008364 0.0008364 0.0008364 0.20% Gravity::update_max_rhs() 6 0.0007529 0.0007529 0.0007529 0.18% MLPoisson::prepareForSolve() 6 0.0007089 0.0007089 0.0007089 0.17% MLCellLinOp::prepareForSolve() 6 0.0007042 0.0007042 0.0007042 0.17% FabArray::mult() 22 0.0006596 0.0006596 0.0006596 0.16% MLMG::computeMLResidual() 6 0.0006081 0.0006081 0.0006081 0.14% Castro::check_for_nan() 10 0.0005894 0.0005894 0.0005894 0.14% MultiFab::contains_nan() 10 0.0005833 0.0005833 0.0005833 0.14% Castro::enforce_speed_limit() 30 0.0005556 0.0005556 0.0005556 0.13% FabArrayBase::CPC::define() 244 0.0004245 0.0004245 0.0004245 0.10% Amr::InitAmr() 1 0.0004146 0.0004146 0.0004146 0.10% FabArrayBase::getFB() 1766 0.0003742 0.0003742 0.0003742 0.09% MultiFab::Copy() 6 0.0003721 0.0003721 0.0003721 0.09% MultiFab::min() 5 0.0001642 0.0001642 0.0001642 0.04% MLLinOp::define() 6 0.0001507 0.0001507 0.0001507 0.04% MultiFab::max() 6 0.0001356 0.0001356 0.0001356 0.03% MLMG::MLResNormInf() 6 0.000135 0.000135 0.000135 0.03% MLLinOp::defineGrids() 6 0.0001243 0.0001243 0.0001243 0.03% MLMG::MLRhsNormInf() 6 0.000107 0.000107 0.000107 0.03% Castro::create_source_corrector() 5 7.538e-05 7.538e-05 7.538e-05 0.02% FabArrayBase::FB::FB() 26 5.917e-05 5.917e-05 5.917e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.962e-05 2.962e-05 2.962e-05 0.01% Castro::finalize_advance() 5 2.657e-05 2.657e-05 2.657e-05 0.01% Amr::writeSmallPlotFile() 1 2.586e-05 2.586e-05 2.586e-05 0.01% Castro::swap_state_time_levels() 5 2.505e-05 2.505e-05 2.505e-05 0.01% makeSFC 30 2.173e-05 2.173e-05 2.173e-05 0.01% Castro::finalize_do_advance() 5 1.975e-05 1.975e-05 1.975e-05 0.00% DistributionMapping::Distribute() 31 9.542e-06 9.542e-06 9.542e-06 0.00% Amr::initSubcycle() 1 9.508e-06 9.508e-06 9.508e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.868e-06 4.868e-06 4.868e-06 0.00% MLMG::buildFineMask() 6 3.028e-06 3.028e-06 3.028e-06 0.00% Gravity::set_mass_offset() 6 2.343e-06 2.343e-06 2.343e-06 0.00% Castro::retry_advance_ctu() 5 1.836e-06 1.836e-06 1.836e-06 0.00% Castro::FluxRegFineAdd() 5 1.82e-06 1.82e-06 1.82e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.106e-06 1.106e-06 1.106e-06 0.00% AmrLevel::AmrLevel() 1 9.82e-07 9.82e-07 9.82e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-26-g0d136ea53aed) finalized