Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.11-14-ge0afc82f63ce) initialized Starting run at 09:55:01 UTC on 2022-11-11. Successfully read inputs file ... Castro git describe: 22.11-7-g14ecd000b AMReX git describe: 22.11-14-ge0afc82f6 Microphysics git describe: 22.11-8-g0d57093f reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.052320828 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.029815891 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.050940822 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.054654487 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.061620222 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.064548212 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.07842728 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.050704836 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.072203133 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051495647 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.062640948 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.063514016 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.067390436 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.048364027 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029677096 seconds Ending run at 09:55:02 UTC on 2022-11-11. Run time = 0.890726625 Run time without initialization = 0.75687196 Average number of zones advanced per microsecond: 3.464 Average number of zones advanced per microsecond per rank: 3.464 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8908 ... 0.8908 ... 0.8908 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2186 0.2186 0.2186 24.54% VisMF::Write(FabArray) 11 0.2018 0.2018 0.2018 22.66% MLCellLinOp::applyBC() 4379 0.08511 0.08511 0.08511 9.56% MLPoisson::Fsmooth() 3240 0.06477 0.06477 0.06477 7.27% StateData::FillBoundary(geom) 328 0.03809 0.03809 0.03809 4.28% MLCGSolver::bicgstab 81 0.02468 0.02468 0.02468 2.77% MultiFab::Dot() 1100 0.0228 0.0228 0.0228 2.56% FillBoundary_nowait() 3974 0.01486 0.01486 0.01486 1.67% FabArray::setVal() 1135 0.01478 0.01478 0.01478 1.66% MultiFab::LinComb() 1566 0.0147 0.0147 0.0147 1.65% Castro::normalize_species() 62 0.01409 0.01409 0.01409 1.58% FabArray::ParallelCopy_nowait() 851 0.01407 0.01407 0.01407 1.58% Castro::computeTemp() 63 0.01267 0.01267 0.01267 1.42% MLCellLinOp::defineAuxData() 11 0.01238 0.01238 0.01238 1.39% MLPoisson::Fapply() 1128 0.01222 0.01222 0.01222 1.37% Castro::enforce_min_density() 62 0.009783 0.009783 0.009783 1.10% StateDataPhysBCFunct::() 41 0.008905 0.008905 0.008905 1.00% Gravity::fill_multipole_BCs() 11 0.008348 0.008348 0.008348 0.94% MLMG::addInterpCorrection() 405 0.008272 0.008272 0.008272 0.93% amrex::average_down 405 0.00737 0.00737 0.00737 0.83% MultiFab::Xpay() 578 0.006691 0.006691 0.006691 0.75% Amr::checkPoint() 3 0.006157 0.006157 0.006157 0.69% Castro::do_advance_ctu() 10 0.00486 0.00486 0.00486 0.55% BndryData::define() 11 0.004019 0.004019 0.004019 0.45% Castro::estTimeStep() 21 0.003981 0.003981 0.003981 0.45% Castro::reset_internal_energy(MultiFab) 63 0.003905 0.003905 0.003905 0.44% Castro::construct_new_gravity_source() 10 0.003346 0.003346 0.003346 0.38% Amr::writePlotFile() 2 0.002891 0.002891 0.002891 0.32% Castro::construct_old_gravity_source() 10 0.002646 0.002646 0.002646 0.30% MLMG::ResNormInf() 92 0.002096 0.002096 0.002096 0.24% Gravity::get_new_grav_vector() 11 0.002092 0.002092 0.002092 0.23% Castro::reset_internal_energy(Fab) 504 0.001913 0.001913 0.001913 0.21% Gravity::get_old_grav_vector() 10 0.00188 0.00188 0.00188 0.21% MultiFab::Saxpy() 20 0.001851 0.001851 0.001851 0.21% Castro::expand_state() 10 0.001824 0.001824 0.001824 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001816 0.001816 0.001816 0.20% Castro::enforce_speed_limit() 62 0.001736 0.001736 0.001736 0.19% MultiFab::Add() 81 0.001672 0.001672 0.001672 0.19% MLCellLinOp::smooth() 1620 0.001606 0.001606 0.001606 0.18% MLCellLinOp::setLevelBC() 11 0.001565 0.001565 0.001565 0.18% Gravity::actual_solve_with_mlmg() 11 0.001435 0.001435 0.001435 0.16% MLMG::prepareForSolve() 11 0.00141 0.00141 0.00141 0.16% FabArray::setDomainBndry() 41 0.001358 0.001358 0.001358 0.15% FabArray::mult() 43 0.001355 0.001355 0.001355 0.15% MLCellLinOp::prepareForSolve() 11 0.001257 0.001257 0.001257 0.14% Castro::initData() 1 0.00122 0.00122 0.00122 0.14% MultiFab::contains_nan() 20 0.001215 0.001215 0.001215 0.14% FabArray::FillBoundary() 3974 0.0009505 0.0009505 0.0009505 0.11% MLCellLinOp::compGrad() 11 0.0009232 0.0009232 0.0009232 0.10% FabArrayBase::getCPC() 1313 0.0008462 0.0008462 0.0008462 0.10% FabArrayBase::CPC::define() 454 0.0007263 0.0007263 0.0007263 0.08% FabArrayBase::getFB() 3974 0.000666 0.000666 0.000666 0.07% Castro::finalize_advance() 10 0.0006193 0.0006193 0.0006193 0.07% MLCellLinOp::apply() 1128 0.0005933 0.0005933 0.0005933 0.07% Amr::InitAmr() 1 0.0005373 0.0005373 0.0005373 0.06% Gravity::solve_for_phi() 10 0.0005033 0.0005033 0.0005033 0.06% Gravity::update_max_rhs() 11 0.0004712 0.0004712 0.0004712 0.05% CGSolver::sxay() 1566 0.0004606 0.0004606 0.0004606 0.05% Amr::coarseTimeStep() 10 0.0003781 0.0003781 0.0003781 0.04% FillPatchIterator::Initialize 41 0.0003501 0.0003501 0.0003501 0.04% MultiFab::Copy() 11 0.0003436 0.0003436 0.0003436 0.04% MLCGSolver::ParallelAllReduce 1495 0.0003126 0.0003126 0.0003126 0.04% FabArray::ParallelCopy() 851 0.0002981 0.0002981 0.0002981 0.03% MLCellLinOp::defineBC() 11 0.0002926 0.0002926 0.0002926 0.03% MLCellLinOp::correctionResidual() 486 0.0002913 0.0002913 0.0002913 0.03% main() 1 0.0002754 0.0002754 0.0002754 0.03% MultiFab::max() 11 0.0002721 0.0002721 0.0002721 0.03% MLMG::mgVcycle() 81 0.0002517 0.0002517 0.0002517 0.03% Castro::construct_new_gravity() 10 0.0002325 0.0002325 0.0002325 0.03% MLMG::MLRhsNormInf() 11 0.0002249 0.0002249 0.0002249 0.03% Amr::timeStep() 10 0.0002023 0.0002023 0.0002023 0.02% MLMG:computeResOfCorrection() 405 0.0001715 0.0001715 0.0001715 0.02% StateData::checkPoint() 12 0.0001591 0.0001591 0.0001591 0.02% MLLinOp::defineGrids() 11 0.0001558 0.0001558 0.0001558 0.02% Castro::subcycle_advance_ctu() 10 0.0001514 0.0001514 0.0001514 0.02% MLMG::mgVcycle_down::0 81 0.0001499 0.0001499 0.0001499 0.02% MLMG::mgVcycle_down::1 81 0.0001356 0.0001356 0.0001356 0.02% MLMG::mgVcycle_down::2 81 0.0001351 0.0001351 0.0001351 0.02% MLMG::mgVcycle_down::3 81 0.0001281 0.0001281 0.0001281 0.01% MLMG::mgVcycle_down::4 81 0.000128 0.000128 0.000128 0.01% Castro::clean_state() 62 0.0001128 0.0001128 0.0001128 0.01% MLMG::actualBottomSolve() 81 0.0001058 0.0001058 0.0001058 0.01% AmrLevel::checkPoint() 3 0.0001057 0.0001057 0.0001057 0.01% FabArrayBase::FB::FB() 56 0.0001018 0.0001018 0.0001018 0.01% Castro::initialize_advance() 10 0.0001017 0.0001017 0.0001017 0.01% Castro::Castro() 1 9.378e-05 9.378e-05 9.378e-05 0.01% MLMG::mgVcycle_up::4 81 9.015e-05 9.015e-05 9.015e-05 0.01% MLMG::solve() 11 8.091e-05 8.091e-05 8.091e-05 0.01% MLMG::oneIter() 81 8.006e-05 8.006e-05 8.006e-05 0.01% MLMG::mgVcycle_up::3 81 7.919e-05 7.919e-05 7.919e-05 0.01% MLMG::mgVcycle_up::1 81 7.711e-05 7.711e-05 7.711e-05 0.01% MLMG::mgVcycle_up::0 81 7.604e-05 7.604e-05 7.604e-05 0.01% MLMG::mgVcycle_up::2 81 7.58e-05 7.58e-05 7.58e-05 0.01% Castro::initialize_do_advance() 10 7.449e-05 7.449e-05 7.449e-05 0.01% MLCellLinOp::solutionResidual() 92 6.299e-05 6.299e-05 6.299e-05 0.01% Castro::advance() 10 6.163e-05 6.163e-05 6.163e-05 0.01% StateData::define() 4 4.643e-05 4.643e-05 4.643e-05 0.01% MLMG::computeResidual() 81 4.304e-05 4.304e-05 4.304e-05 0.00% Castro::swap_state_time_levels() 10 4.243e-05 4.243e-05 4.243e-05 0.00% FillPatchSingleLevel 41 4.092e-05 4.092e-05 4.092e-05 0.00% Castro::finalize_do_advance() 10 3.815e-05 3.815e-05 3.815e-05 0.00% Castro::enforce_consistent_e() 1 3.68e-05 3.68e-05 3.68e-05 0.00% MLPoisson::define() 11 3.576e-05 3.576e-05 3.576e-05 0.00% MLMG::mgVcycle_bottom 81 3.571e-05 3.571e-05 3.571e-05 0.00% Gravity::actual_multilevel_solve() 1 3.263e-05 3.263e-05 3.263e-05 0.00% Castro::do_new_sources() 10 3.232e-05 3.232e-05 3.232e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.231e-05 3.231e-05 3.231e-05 0.00% Castro::do_old_sources() 10 3.127e-05 3.127e-05 3.127e-05 0.00% Castro::construct_new_source() 50 2.95e-05 2.95e-05 2.95e-05 0.00% Castro::initMFs() 1 2.902e-05 2.902e-05 2.902e-05 0.00% makeSFC 55 2.814e-05 2.814e-05 2.814e-05 0.00% Castro::construct_old_source() 50 2.786e-05 2.786e-05 2.786e-05 0.00% MLLinOp::define() 11 2.73e-05 2.73e-05 2.73e-05 0.00% Amr::writeSmallPlotFile() 1 2.727e-05 2.727e-05 2.727e-05 0.00% Castro::buildMetrics() 1 2.525e-05 2.525e-05 2.525e-05 0.00% Amr::FinalizeInit() 1 2.464e-05 2.464e-05 2.464e-05 0.00% Amr::defBaseLevel() 1 2.259e-05 2.259e-05 2.259e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.191e-05 2.191e-05 2.191e-05 0.00% Castro::apply_source_to_state() 20 1.927e-05 1.927e-05 1.927e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.643e-05 1.643e-05 1.643e-05 0.00% DistributionMapping::Distribute() 56 1.485e-05 1.485e-05 1.485e-05 0.00% Castro::check_for_nan() 20 1.372e-05 1.372e-05 1.372e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.348e-05 1.348e-05 1.348e-05 0.00% Castro::construct_old_gravity() 10 1.339e-05 1.339e-05 1.339e-05 0.00% MLMG::computeMLResidual() 11 1.24e-05 1.24e-05 1.24e-05 0.00% Gravity::swapTimeLevels() 10 1.215e-05 1.215e-05 1.215e-05 0.00% Castro::post_timestep() 10 1.062e-05 1.062e-05 1.062e-05 0.00% MLPoisson::prepareForSolve() 11 9.275e-06 9.275e-06 9.275e-06 0.00% Amr::initSubcycle() 1 9.193e-06 9.193e-06 9.193e-06 0.00% MLMG::getGradSolution() 11 7.625e-06 7.625e-06 7.625e-06 0.00% Castro::computeNewDt() 9 7.48e-06 7.48e-06 7.48e-06 0.00% AmrLevel::checkPointPost() 3 6.128e-06 6.128e-06 6.128e-06 0.00% Amr::InitializeInit() 1 6.079e-06 6.079e-06 6.079e-06 0.00% Gravity::set_mass_offset() 11 5.288e-06 5.288e-06 5.288e-06 0.00% Castro::post_init() 1 5.061e-06 5.061e-06 5.061e-06 0.00% Castro::retry_advance_ctu() 10 4.618e-06 4.618e-06 4.618e-06 0.00% Castro::create_source_corrector() 10 4.301e-06 4.301e-06 4.301e-06 0.00% MLMG::MLResNormInf() 11 3.806e-06 3.806e-06 3.806e-06 0.00% Castro::FluxRegCrseInit 10 3.105e-06 3.105e-06 3.105e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.936e-06 2.936e-06 2.936e-06 0.00% Castro::computeInitialDt() 2 2.911e-06 2.911e-06 2.911e-06 0.00% Amr::init() 1 2.907e-06 2.907e-06 2.907e-06 0.00% AmrLevel::checkPointPre() 3 2.676e-06 2.676e-06 2.676e-06 0.00% Castro::FluxRegFineAdd() 10 2.005e-06 2.005e-06 2.005e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.801e-06 1.801e-06 1.801e-06 0.00% Castro::post_regrid() 1 1.568e-06 1.568e-06 1.568e-06 0.00% Amr::initialInit() 1 1.199e-06 1.199e-06 1.199e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8907 0.8907 0.8907 100.00% Amr::coarseTimeStep() 10 0.7269 0.7269 0.7269 81.61% Amr::timeStep() 10 0.6251 0.6251 0.6251 70.18% Castro::advance() 10 0.6179 0.6179 0.6179 69.36% Castro::subcycle_advance_ctu() 10 0.6053 0.6053 0.6053 67.95% Castro::do_advance_ctu() 10 0.6051 0.6051 0.6051 67.93% Gravity::solve_phi_with_mlmg() 11 0.3285 0.3285 0.3285 36.88% Gravity::actual_solve_with_mlmg() 11 0.3199 0.3199 0.3199 35.92% Castro::construct_new_gravity() 10 0.3055 0.3055 0.3055 34.30% MLMG::solve() 11 0.296 0.296 0.296 33.23% Gravity::solve_for_phi() 10 0.2859 0.2859 0.2859 32.09% MLMG::oneIter() 81 0.2803 0.2803 0.2803 31.47% MLMG::mgVcycle() 81 0.2786 0.2786 0.2786 31.27% Castro::construct_ctu_hydro_source() 10 0.2186 0.2186 0.2186 24.54% VisMF::Write(FabArray) 11 0.2018 0.2018 0.2018 22.66% Amr::checkPoint() 3 0.1515 0.1515 0.1515 17.01% AmrLevel::checkPoint() 3 0.1453 0.1453 0.1453 16.32% StateData::checkPoint() 12 0.1452 0.1452 0.1452 16.31% MLCellLinOp::smooth() 1620 0.142 0.142 0.142 15.94% Amr::init() 1 0.1332 0.1332 0.1332 14.95% MLCellLinOp::applyBC() 4379 0.1017 0.1017 0.1017 11.42% MLMG::mgVcycle_bottom 81 0.08479 0.08479 0.08479 9.52% MLMG::actualBottomSolve() 81 0.08475 0.08475 0.08475 9.51% MLCGSolver::bicgstab 81 0.08387 0.08387 0.08387 9.42% MLPoisson::Fsmooth() 3240 0.06477 0.06477 0.06477 7.27% Amr::writePlotFile() 2 0.05963 0.05963 0.05963 6.69% FillPatchIterator::Initialize 41 0.05277 0.05277 0.05277 5.92% FillPatchSingleLevel 41 0.05107 0.05107 0.05107 5.73% Amr::initialInit() 1 0.05094 0.05094 0.05094 5.72% StateDataPhysBCFunct::() 41 0.04699 0.04699 0.04699 5.28% Amr::FinalizeInit() 1 0.0469 0.0469 0.0469 5.26% Castro::post_init() 1 0.0456 0.0456 0.0456 5.12% Castro::clean_state() 62 0.04344 0.04344 0.04344 4.88% Gravity::multilevel_solve_for_new_phi() 1 0.04323 0.04323 0.04323 4.85% Gravity::actual_multilevel_solve() 1 0.04321 0.04321 0.04321 4.85% MLCellLinOp::apply() 1128 0.03834 0.03834 0.03834 4.30% StateData::FillBoundary(geom) 328 0.03809 0.03809 0.03809 4.28% MLMG::mgVcycle_down::0 81 0.03634 0.03634 0.03634 4.08% MLMG::mgVcycle_up::0 81 0.03121 0.03121 0.03121 3.50% MultiFab::Dot() 1100 0.0228 0.0228 0.0228 2.56% MLCellLinOp::correctionResidual() 486 0.02237 0.02237 0.02237 2.51% Gravity::get_new_grav_vector() 11 0.02169 0.02169 0.02169 2.44% MLPoisson::define() 11 0.01949 0.01949 0.01949 2.19% Castro::construct_old_gravity() 10 0.01947 0.01947 0.01947 2.19% Gravity::get_old_grav_vector() 10 0.01946 0.01946 0.01946 2.18% MLMG:computeResOfCorrection() 405 0.01931 0.01931 0.01931 2.17% Castro::initialize_do_advance() 10 0.0193 0.0193 0.0193 2.17% MLMG::mgVcycle_down::1 81 0.01878 0.01878 0.01878 2.11% Castro::computeTemp() 63 0.01849 0.01849 0.01849 2.08% MLMG::mgVcycle_down::2 81 0.01829 0.01829 0.01829 2.05% MLMG::mgVcycle_down::3 81 0.01751 0.01751 0.01751 1.97% MLMG::mgVcycle_down::4 81 0.01667 0.01667 0.01667 1.87% FabArray::FillBoundary() 3974 0.01658 0.01658 0.01658 1.86% FillBoundary_nowait() 3974 0.01563 0.01563 0.01563 1.75% FabArray::ParallelCopy() 851 0.01523 0.01523 0.01523 1.71% CGSolver::sxay() 1566 0.01516 0.01516 0.01516 1.70% FabArray::ParallelCopy_nowait() 851 0.01493 0.01493 0.01493 1.68% FabArray::setVal() 1135 0.01478 0.01478 0.01478 1.66% MultiFab::LinComb() 1566 0.0147 0.0147 0.0147 1.65% MLMG::mgVcycle_up::2 81 0.01409 0.01409 0.01409 1.58% Castro::normalize_species() 62 0.01409 0.01409 0.01409 1.58% MLMG::mgVcycle_up::1 81 0.01395 0.01395 0.01395 1.57% MLCellLinOp::defineAuxData() 11 0.01387 0.01387 0.01387 1.56% MLMG::addInterpCorrection() 405 0.01385 0.01385 0.01385 1.55% MLCGSolver::ParallelAllReduce 1495 0.01362 0.01362 0.01362 1.53% MLMG::mgVcycle_up::3 81 0.01341 0.01341 0.01341 1.51% MLMG::mgVcycle_up::4 81 0.01326 0.01326 0.01326 1.49% amrex::average_down 405 0.01299 0.01299 0.01299 1.46% MLPoisson::Fapply() 1128 0.01222 0.01222 0.01222 1.37% Castro::initialize_advance() 10 0.01192 0.01192 0.01192 1.34% Castro::expand_state() 10 0.01184 0.01184 0.01184 1.33% Castro::do_new_sources() 10 0.01157 0.01157 0.01157 1.30% Castro::do_old_sources() 10 0.01026 0.01026 0.01026 1.15% Castro::enforce_min_density() 62 0.009783 0.009783 0.009783 1.10% Gravity::fill_multipole_BCs() 11 0.008348 0.008348 0.008348 0.94% MLCellLinOp::solutionResidual() 92 0.007401 0.007401 0.007401 0.83% Castro::post_timestep() 10 0.007028 0.007028 0.007028 0.79% MultiFab::Xpay() 578 0.006691 0.006691 0.006691 0.75% MLMG::computeResidual() 81 0.006357 0.006357 0.006357 0.71% Castro::reset_internal_energy(MultiFab) 63 0.005818 0.005818 0.005818 0.65% MLMG::prepareForSolve() 11 0.005808 0.005808 0.005808 0.65% MLCellLinOp::defineBC() 11 0.005339 0.005339 0.005339 0.60% BndryData::define() 11 0.005046 0.005046 0.005046 0.57% Amr::InitializeInit() 1 0.004044 0.004044 0.004044 0.45% Amr::defBaseLevel() 1 0.004038 0.004038 0.004038 0.45% Castro::estTimeStep() 21 0.003981 0.003981 0.003981 0.45% Castro::initData() 1 0.003491 0.003491 0.003491 0.39% Castro::construct_new_source() 50 0.003375 0.003375 0.003375 0.38% Castro::construct_new_gravity_source() 10 0.003346 0.003346 0.003346 0.38% Castro::construct_old_source() 50 0.002674 0.002674 0.002674 0.30% Castro::construct_old_gravity_source() 10 0.002646 0.002646 0.002646 0.30% MLMG::ResNormInf() 92 0.002096 0.002096 0.002096 0.24% Castro::reset_internal_energy(Fab) 504 0.001913 0.001913 0.001913 0.21% Castro::apply_source_to_state() 20 0.00187 0.00187 0.00187 0.21% MultiFab::Saxpy() 20 0.001851 0.001851 0.001851 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001816 0.001816 0.001816 0.20% Castro::enforce_speed_limit() 62 0.001736 0.001736 0.001736 0.19% MultiFab::Add() 81 0.001672 0.001672 0.001672 0.19% FabArrayBase::getCPC() 1313 0.001573 0.001573 0.001573 0.18% MLCellLinOp::setLevelBC() 11 0.001565 0.001565 0.001565 0.18% Castro::computeNewDt() 9 0.001515 0.001515 0.001515 0.17% MLMG::getGradSolution() 11 0.001462 0.001462 0.001462 0.16% MLCellLinOp::compGrad() 11 0.001454 0.001454 0.001454 0.16% FabArray::setDomainBndry() 41 0.001358 0.001358 0.001358 0.15% FabArray::mult() 43 0.001355 0.001355 0.001355 0.15% MLPoisson::prepareForSolve() 11 0.001266 0.001266 0.001266 0.14% MLCellLinOp::prepareForSolve() 11 0.001257 0.001257 0.001257 0.14% Castro::check_for_nan() 20 0.001228 0.001228 0.001228 0.14% MultiFab::contains_nan() 20 0.001215 0.001215 0.001215 0.14% Castro::post_regrid() 1 0.001122 0.001122 0.001122 0.13% MLMG::computeMLResidual() 11 0.001099 0.001099 0.001099 0.12% Castro::computeInitialDt() 2 0.000953 0.000953 0.000953 0.11% Gravity::update_max_rhs() 11 0.0008939 0.0008939 0.0008939 0.10% FabArrayBase::getFB() 3974 0.0007678 0.0007678 0.0007678 0.09% FabArrayBase::CPC::define() 454 0.0007263 0.0007263 0.0007263 0.08% Castro::finalize_advance() 10 0.0006244 0.0006244 0.0006244 0.07% Amr::InitAmr() 1 0.0005465 0.0005465 0.0005465 0.06% Gravity::swapTimeLevels() 10 0.0004635 0.0004635 0.0004635 0.05% Castro::Castro() 1 0.0004604 0.0004604 0.0004604 0.05% MultiFab::Copy() 11 0.0003436 0.0003436 0.0003436 0.04% MLMG::MLResNormInf() 11 0.0002951 0.0002951 0.0002951 0.03% MultiFab::max() 11 0.0002721 0.0002721 0.0002721 0.03% MLLinOp::define() 11 0.0002431 0.0002431 0.0002431 0.03% MLMG::MLRhsNormInf() 11 0.0002249 0.0002249 0.0002249 0.03% MLLinOp::defineGrids() 11 0.0002158 0.0002158 0.0002158 0.02% Castro::buildMetrics() 1 0.0001687 0.0001687 0.0001687 0.02% FabArrayBase::FB::FB() 56 0.0001018 0.0001018 0.0001018 0.01% AmrLevel::AmrLevel(dm) 1 5.99e-05 5.99e-05 5.99e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.823e-05 5.823e-05 5.823e-05 0.01% StateData::define() 4 4.643e-05 4.643e-05 4.643e-05 0.01% Castro::swap_state_time_levels() 10 4.243e-05 4.243e-05 4.243e-05 0.00% makeSFC 55 4.18e-05 4.18e-05 4.18e-05 0.00% Castro::finalize_do_advance() 10 3.815e-05 3.815e-05 3.815e-05 0.00% Castro::enforce_consistent_e() 1 3.68e-05 3.68e-05 3.68e-05 0.00% Castro::initMFs() 1 2.902e-05 2.902e-05 2.902e-05 0.00% Amr::writeSmallPlotFile() 1 2.727e-05 2.727e-05 2.727e-05 0.00% DistributionMapping::Distribute() 56 1.485e-05 1.485e-05 1.485e-05 0.00% Amr::initSubcycle() 1 9.193e-06 9.193e-06 9.193e-06 0.00% AmrLevel::checkPointPost() 3 6.128e-06 6.128e-06 6.128e-06 0.00% Gravity::set_mass_offset() 11 5.288e-06 5.288e-06 5.288e-06 0.00% Castro::retry_advance_ctu() 10 4.618e-06 4.618e-06 4.618e-06 0.00% Castro::create_source_corrector() 10 4.301e-06 4.301e-06 4.301e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.12e-06 4.12e-06 4.12e-06 0.00% Castro::FluxRegCrseInit 10 3.105e-06 3.105e-06 3.105e-06 0.00% AmrLevel::checkPointPre() 3 2.676e-06 2.676e-06 2.676e-06 0.00% Castro::FluxRegFineAdd() 10 2.005e-06 2.005e-06 2.005e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.801e-06 1.801e-06 1.801e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.11-14-ge0afc82f63ce) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.11-14-ge0afc82f63ce) initialized Starting run at 09:55:03 UTC on 2022-11-11. Successfully read inputs file ... Castro git describe: 22.11-7-g14ecd000b AMReX git describe: 22.11-14-ge0afc82f6 Microphysics git describe: 22.11-8-g0d57093f reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.494790539 Restart time = 0.048176364 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052575541 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.058345452 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057970401 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.05978161 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.074682231 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.03170189 seconds Ending run at 09:55:03 UTC on 2022-11-11. Run time = 0.384282123 Run time without initialization = 0.335571007 Average number of zones advanced per microsecond: 3.906 Average number of zones advanced per microsecond per rank: 3.906 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3843 ... 0.3843 ... 0.3843 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1093 0.1093 0.1093 28.44% VisMF::Read() 3 0.04032 0.04032 0.04032 10.49% MLCellLinOp::applyBC() 1946 0.03414 0.03414 0.03414 8.88% VisMF::Write(FabArray) 1 0.03008 0.03008 0.03008 7.83% MLPoisson::Fsmooth() 1440 0.02675 0.02675 0.02675 6.96% StateData::FillBoundary(geom) 160 0.01139 0.01139 0.01139 2.96% MLCGSolver::bicgstab 36 0.009965 0.009965 0.009965 2.59% Castro::normalize_species() 30 0.009541 0.009541 0.009541 2.48% MultiFab::Dot() 484 0.009229 0.009229 0.009229 2.40% Castro::computeTemp() 30 0.008084 0.008084 0.008084 2.10% FabArray::setVal() 537 0.006568 0.006568 0.006568 1.71% MLCellLinOp::defineAuxData() 6 0.006185 0.006185 0.006185 1.61% FillBoundary_nowait() 1766 0.006173 0.006173 0.006173 1.61% MultiFab::LinComb() 690 0.005897 0.005897 0.005897 1.53% FabArray::ParallelCopy_nowait() 380 0.005834 0.005834 0.005834 1.52% StateDataPhysBCFunct::() 20 0.00549 0.00549 0.00549 1.43% MLPoisson::Fapply() 500 0.004946 0.004946 0.004946 1.29% Gravity::fill_multipole_BCs() 6 0.004366 0.004366 0.004366 1.14% Amr::restart() 1 0.003577 0.003577 0.003577 0.93% MLMG::addInterpCorrection() 180 0.003249 0.003249 0.003249 0.85% amrex::average_down 180 0.00287 0.00287 0.00287 0.75% MultiFab::Xpay() 258 0.002791 0.002791 0.002791 0.73% Castro::enforce_min_density() 30 0.002742 0.002742 0.002742 0.71% Castro::estTimeStep() 10 0.002477 0.002477 0.002477 0.64% Castro::do_advance_ctu() 5 0.002359 0.002359 0.002359 0.61% BndryData::define() 6 0.002023 0.002023 0.002023 0.53% Castro::construct_new_gravity_source() 5 0.001713 0.001713 0.001713 0.45% Amr::writePlotFile() 1 0.0017 0.0017 0.0017 0.44% Castro::reset_internal_energy(MultiFab) 30 0.001643 0.001643 0.001643 0.43% Castro::construct_old_gravity_source() 5 0.001496 0.001496 0.001496 0.39% Castro::reset_internal_energy(Fab) 240 0.001144 0.001144 0.001144 0.30% Gravity::get_old_grav_vector() 5 0.001004 0.001004 0.001004 0.26% Castro::subcycle_advance_ctu() 5 0.0009613 0.0009613 0.0009613 0.25% Gravity::get_new_grav_vector() 5 0.0009508 0.0009508 0.0009508 0.25% MultiFab::Saxpy() 10 0.0009153 0.0009153 0.0009153 0.24% MLMG::ResNormInf() 42 0.0008956 0.0008956 0.0008956 0.23% Castro::expand_state() 5 0.000871 0.000871 0.000871 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008706 0.0008706 0.0008706 0.23% MLCellLinOp::setLevelBC() 6 0.0007954 0.0007954 0.0007954 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007803 0.0007803 0.0007803 0.20% MultiFab::Add() 36 0.0007055 0.0007055 0.0007055 0.18% MLMG::prepareForSolve() 6 0.0006513 0.0006513 0.0006513 0.17% FabArray::setDomainBndry() 20 0.0006489 0.0006489 0.0006489 0.17% FabArray::mult() 22 0.0006448 0.0006448 0.0006448 0.17% MLCellLinOp::prepareForSolve() 6 0.0006154 0.0006154 0.0006154 0.16% MultiFab::contains_nan() 10 0.0005836 0.0005836 0.0005836 0.15% MLCellLinOp::smooth() 720 0.0004994 0.0004994 0.0004994 0.13% MLCellLinOp::compGrad() 6 0.0004791 0.0004791 0.0004791 0.12% Castro::enforce_speed_limit() 30 0.000451 0.000451 0.000451 0.12% Amr::InitAmr() 1 0.0003867 0.0003867 0.0003867 0.10% FabArrayBase::CPC::define() 244 0.0003829 0.0003829 0.0003829 0.10% Castro::finalize_advance() 5 0.0003814 0.0003814 0.0003814 0.10% FabArray::FillBoundary() 1766 0.0003759 0.0003759 0.0003759 0.10% FabArrayBase::getCPC() 632 0.0003685 0.0003685 0.0003685 0.10% Amr::coarseTimeStep() 5 0.0002941 0.0002941 0.0002941 0.08% FabArrayBase::getFB() 1766 0.0002485 0.0002485 0.0002485 0.06% Gravity::solve_for_phi() 5 0.0002416 0.0002416 0.0002416 0.06% Gravity::update_max_rhs() 6 0.0002329 0.0002329 0.0002329 0.06% main() 1 0.0002329 0.0002329 0.0002329 0.06% MLCellLinOp::apply() 500 0.0001994 0.0001994 0.0001994 0.05% MultiFab::Copy() 6 0.0001723 0.0001723 0.0001723 0.04% CGSolver::sxay() 690 0.0001637 0.0001637 0.0001637 0.04% Castro::construct_new_gravity() 5 0.0001626 0.0001626 0.0001626 0.04% MLCellLinOp::defineBC() 6 0.0001415 0.0001415 0.0001415 0.04% Castro::post_timestep() 5 0.0001382 0.0001382 0.0001382 0.04% Castro::advance() 5 0.0001336 0.0001336 0.0001336 0.03% MultiFab::max() 6 0.0001327 0.0001327 0.0001327 0.03% FillPatchIterator::Initialize 20 0.0001326 0.0001326 0.0001326 0.03% MLCGSolver::ParallelAllReduce 659 0.0001232 0.0001232 0.0001232 0.03% FabArray::ParallelCopy() 380 0.0001171 0.0001171 0.0001171 0.03% MLMG::MLRhsNormInf() 6 0.0001126 0.0001126 0.0001126 0.03% Castro::construct_new_source() 25 0.0001117 0.0001117 0.0001117 0.03% Castro::initialize_do_advance() 5 9.783e-05 9.783e-05 9.783e-05 0.03% Amr::timeStep() 5 9.647e-05 9.647e-05 9.647e-05 0.03% MLCellLinOp::correctionResidual() 216 9.366e-05 9.366e-05 9.366e-05 0.02% MLMG::mgVcycle() 36 8.952e-05 8.952e-05 8.952e-05 0.02% MLLinOp::defineGrids() 6 8.897e-05 8.897e-05 8.897e-05 0.02% AmrLevel::restart() 1 8.53e-05 8.53e-05 8.53e-05 0.02% Castro::create_source_corrector() 5 8.271e-05 8.271e-05 8.271e-05 0.02% MLMG:computeResOfCorrection() 180 7.316e-05 7.316e-05 7.316e-05 0.02% StateData::restartDoit() 4 7.158e-05 7.158e-05 7.158e-05 0.02% FabArrayBase::FB::FB() 26 5.605e-05 5.605e-05 5.605e-05 0.01% Castro::construct_old_source() 25 5.295e-05 5.295e-05 5.295e-05 0.01% Castro::initialize_advance() 5 5.01e-05 5.01e-05 5.01e-05 0.01% MLMG::mgVcycle_down::0 36 5.002e-05 5.002e-05 5.002e-05 0.01% Castro::clean_state() 30 4.549e-05 4.549e-05 4.549e-05 0.01% MLMG::mgVcycle_down::1 36 4.487e-05 4.487e-05 4.487e-05 0.01% MLMG::mgVcycle_down::2 36 4.184e-05 4.184e-05 4.184e-05 0.01% MLMG::mgVcycle_down::4 36 4.026e-05 4.026e-05 4.026e-05 0.01% MLMG::mgVcycle_down::3 36 3.885e-05 3.885e-05 3.885e-05 0.01% MLMG::actualBottomSolve() 36 3.78e-05 3.78e-05 3.78e-05 0.01% MLMG::solve() 6 3.775e-05 3.775e-05 3.775e-05 0.01% MLMG::mgVcycle_up::4 36 3.585e-05 3.585e-05 3.585e-05 0.01% Castro::buildMetrics() 1 3.041e-05 3.041e-05 3.041e-05 0.01% Gravity::actual_multilevel_solve() 1 2.922e-05 2.922e-05 2.922e-05 0.01% MLMG::oneIter() 36 2.848e-05 2.848e-05 2.848e-05 0.01% Castro::post_restart() 1 2.75e-05 2.75e-05 2.75e-05 0.01% MLMG::mgVcycle_up::0 36 2.699e-05 2.699e-05 2.699e-05 0.01% Amr::writeSmallPlotFile() 1 2.687e-05 2.687e-05 2.687e-05 0.01% Castro::swap_state_time_levels() 5 2.684e-05 2.684e-05 2.684e-05 0.01% Castro::initMFs() 1 2.652e-05 2.652e-05 2.652e-05 0.01% MLMG::mgVcycle_up::3 36 2.643e-05 2.643e-05 2.643e-05 0.01% MLMG::mgVcycle_up::2 36 2.578e-05 2.578e-05 2.578e-05 0.01% Castro::do_old_sources() 5 2.54e-05 2.54e-05 2.54e-05 0.01% MLMG::mgVcycle_up::1 36 2.419e-05 2.419e-05 2.419e-05 0.01% MLPoisson::define() 6 2.355e-05 2.355e-05 2.355e-05 0.01% MLCellLinOp::solutionResidual() 42 2.193e-05 2.193e-05 2.193e-05 0.01% Castro::construct_old_gravity() 5 2.094e-05 2.094e-05 2.094e-05 0.01% MLLinOp::define() 6 2.072e-05 2.072e-05 2.072e-05 0.01% Castro::computeNewDt() 5 1.977e-05 1.977e-05 1.977e-05 0.01% MLMG::computeResidual() 36 1.922e-05 1.922e-05 1.922e-05 0.01% Castro::finalize_do_advance() 5 1.767e-05 1.767e-05 1.767e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.726e-05 1.726e-05 1.726e-05 0.00% MLMG::mgVcycle_bottom 36 1.482e-05 1.482e-05 1.482e-05 0.00% makeSFC 30 1.469e-05 1.469e-05 1.469e-05 0.00% FillPatchSingleLevel 20 1.334e-05 1.334e-05 1.334e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.315e-05 1.315e-05 1.315e-05 0.00% Castro::do_new_sources() 5 9.609e-06 9.609e-06 9.609e-06 0.00% DistributionMapping::Distribute() 31 8.443e-06 8.443e-06 8.443e-06 0.00% Amr::initSubcycle() 1 8.194e-06 8.194e-06 8.194e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.238e-06 7.238e-06 7.238e-06 0.00% Castro::check_for_nan() 10 6.435e-06 6.435e-06 6.435e-06 0.00% Castro::apply_source_to_state() 10 5.911e-06 5.911e-06 5.911e-06 0.00% MLMG::computeMLResidual() 6 4.754e-06 4.754e-06 4.754e-06 0.00% Gravity::set_mass_offset() 6 4.449e-06 4.449e-06 4.449e-06 0.00% MLPoisson::prepareForSolve() 6 4.423e-06 4.423e-06 4.423e-06 0.00% Gravity::swapTimeLevels() 5 4.216e-06 4.216e-06 4.216e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.05e-06 3.05e-06 3.05e-06 0.00% MLMG::getGradSolution() 6 3.022e-06 3.022e-06 3.022e-06 0.00% Castro::FluxRegCrseInit 5 2.67e-06 2.67e-06 2.67e-06 0.00% Castro::retry_advance_ctu() 5 2.521e-06 2.521e-06 2.521e-06 0.00% MLMG::MLResNormInf() 6 2.083e-06 2.083e-06 2.083e-06 0.00% Castro::FluxRegFineAdd() 5 1.49e-06 1.49e-06 1.49e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.17e-06 1.17e-06 1.17e-06 0.00% Amr::init() 1 9.87e-07 9.87e-07 9.87e-07 0.00% AmrLevel::AmrLevel() 1 9.03e-07 9.03e-07 9.03e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3843 0.3843 0.3843 100.00% Amr::coarseTimeStep() 5 0.3036 0.3036 0.3036 79.01% Amr::timeStep() 5 0.3019 0.3019 0.3019 78.57% Castro::advance() 5 0.2963 0.2963 0.2963 77.10% Castro::subcycle_advance_ctu() 5 0.2912 0.2912 0.2912 75.78% Castro::do_advance_ctu() 5 0.2903 0.2903 0.2903 75.53% Castro::construct_new_gravity() 5 0.1406 0.1406 0.1406 36.59% Gravity::solve_phi_with_mlmg() 6 0.1365 0.1365 0.1365 35.51% Gravity::solve_for_phi() 5 0.133 0.133 0.133 34.60% Gravity::actual_solve_with_mlmg() 6 0.132 0.132 0.132 34.35% MLMG::solve() 6 0.1199 0.1199 0.1199 31.20% MLMG::oneIter() 36 0.1128 0.1128 0.1128 29.36% MLMG::mgVcycle() 36 0.1121 0.1121 0.1121 29.17% Castro::construct_ctu_hydro_source() 5 0.1093 0.1093 0.1093 28.44% MLCellLinOp::smooth() 720 0.05765 0.05765 0.05765 15.00% Amr::init() 1 0.04822 0.04822 0.04822 12.55% Amr::restart() 1 0.04822 0.04822 0.04822 12.55% MLCellLinOp::applyBC() 1946 0.041 0.041 0.041 10.67% AmrLevel::restart() 1 0.04053 0.04053 0.04053 10.55% StateData::restartDoit() 4 0.04044 0.04044 0.04044 10.52% VisMF::Read() 3 0.04032 0.04032 0.04032 10.49% MLMG::mgVcycle_bottom 36 0.03404 0.03404 0.03404 8.86% MLMG::actualBottomSolve() 36 0.03402 0.03402 0.03402 8.85% MLCGSolver::bicgstab 36 0.03368 0.03368 0.03368 8.76% Amr::writePlotFile() 1 0.03178 0.03178 0.03178 8.27% VisMF::Write(FabArray) 1 0.03008 0.03008 0.03008 7.83% MLPoisson::Fsmooth() 1440 0.02675 0.02675 0.02675 6.96% Castro::clean_state() 30 0.02365 0.02365 0.02365 6.15% FillPatchIterator::Initialize 20 0.01967 0.01967 0.01967 5.12% FillPatchSingleLevel 20 0.01889 0.01889 0.01889 4.91% StateDataPhysBCFunct::() 20 0.01688 0.01688 0.01688 4.39% MLCellLinOp::apply() 500 0.01547 0.01547 0.01547 4.03% MLMG::mgVcycle_down::0 36 0.01514 0.01514 0.01514 3.94% MLMG::mgVcycle_up::0 36 0.01294 0.01294 0.01294 3.37% StateData::FillBoundary(geom) 160 0.01139 0.01139 0.01139 2.96% Castro::computeTemp() 30 0.01087 0.01087 0.01087 2.83% MLPoisson::define() 6 0.009756 0.009756 0.009756 2.54% Castro::initialize_do_advance() 5 0.009559 0.009559 0.009559 2.49% Castro::normalize_species() 30 0.009541 0.009541 0.009541 2.48% MultiFab::Dot() 484 0.009229 0.009229 0.009229 2.40% MLCellLinOp::correctionResidual() 216 0.009033 0.009033 0.009033 2.35% MLMG:computeResOfCorrection() 180 0.007812 0.007812 0.007812 2.03% Castro::do_new_sources() 5 0.0076 0.0076 0.0076 1.98% Gravity::get_new_grav_vector() 5 0.007502 0.007502 0.007502 1.95% MLMG::mgVcycle_down::1 36 0.0075 0.0075 0.0075 1.95% Castro::construct_old_gravity() 5 0.007439 0.007439 0.007439 1.94% Gravity::get_old_grav_vector() 5 0.007418 0.007418 0.007418 1.93% MLMG::mgVcycle_down::2 36 0.007249 0.007249 0.007249 1.89% MLCellLinOp::defineAuxData() 6 0.00691 0.00691 0.00691 1.80% MLMG::mgVcycle_down::3 36 0.006892 0.006892 0.006892 1.79% FabArray::FillBoundary() 1766 0.006853 0.006853 0.006853 1.78% MLMG::mgVcycle_down::4 36 0.006587 0.006587 0.006587 1.71% FabArray::setVal() 537 0.006568 0.006568 0.006568 1.71% FillBoundary_nowait() 1766 0.006478 0.006478 0.006478 1.69% FabArray::ParallelCopy() 380 0.006329 0.006329 0.006329 1.65% FabArray::ParallelCopy_nowait() 380 0.006212 0.006212 0.006212 1.62% Castro::do_old_sources() 5 0.006094 0.006094 0.006094 1.59% CGSolver::sxay() 690 0.00606 0.00606 0.00606 1.58% MultiFab::LinComb() 690 0.005897 0.005897 0.005897 1.53% MLMG::mgVcycle_up::2 36 0.005614 0.005614 0.005614 1.46% Castro::expand_state() 5 0.005588 0.005588 0.005588 1.45% Castro::post_timestep() 5 0.005562 0.005562 0.005562 1.45% MLCGSolver::ParallelAllReduce 659 0.005519 0.005519 0.005519 1.44% MLMG::mgVcycle_up::1 36 0.005513 0.005513 0.005513 1.43% MLMG::addInterpCorrection() 180 0.005397 0.005397 0.005397 1.40% MLMG::mgVcycle_up::3 36 0.00529 0.00529 0.00529 1.38% MLMG::mgVcycle_up::4 36 0.005253 0.005253 0.005253 1.37% amrex::average_down 180 0.005057 0.005057 0.005057 1.32% MLPoisson::Fapply() 500 0.004946 0.004946 0.004946 1.29% Castro::initialize_advance() 5 0.004536 0.004536 0.004536 1.18% Gravity::fill_multipole_BCs() 6 0.004366 0.004366 0.004366 1.14% Castro::post_restart() 1 0.003935 0.003935 0.003935 1.02% Gravity::multilevel_solve_for_new_phi() 1 0.003815 0.003815 0.003815 0.99% Gravity::actual_multilevel_solve() 1 0.003798 0.003798 0.003798 0.99% MLCellLinOp::solutionResidual() 42 0.003183 0.003183 0.003183 0.83% MLMG::prepareForSolve() 6 0.002821 0.002821 0.002821 0.73% MultiFab::Xpay() 258 0.002791 0.002791 0.002791 0.73% Castro::reset_internal_energy(MultiFab) 30 0.002788 0.002788 0.002788 0.73% Castro::enforce_min_density() 30 0.002742 0.002742 0.002742 0.71% MLCellLinOp::defineBC() 6 0.002683 0.002683 0.002683 0.70% MLMG::computeResidual() 36 0.002647 0.002647 0.002647 0.69% BndryData::define() 6 0.002541 0.002541 0.002541 0.66% Castro::estTimeStep() 10 0.002477 0.002477 0.002477 0.64% Castro::construct_new_source() 25 0.001824 0.001824 0.001824 0.47% Castro::construct_new_gravity_source() 5 0.001713 0.001713 0.001713 0.45% Castro::construct_old_source() 25 0.001549 0.001549 0.001549 0.40% Castro::construct_old_gravity_source() 5 0.001496 0.001496 0.001496 0.39% Castro::computeNewDt() 5 0.001398 0.001398 0.001398 0.36% Castro::reset_internal_energy(Fab) 240 0.001144 0.001144 0.001144 0.30% Castro::apply_source_to_state() 10 0.0009212 0.0009212 0.0009212 0.24% MultiFab::Saxpy() 10 0.0009153 0.0009153 0.0009153 0.24% MLMG::ResNormInf() 42 0.0008956 0.0008956 0.0008956 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008706 0.0008706 0.0008706 0.23% MLCellLinOp::setLevelBC() 6 0.0007954 0.0007954 0.0007954 0.21% FabArrayBase::getCPC() 632 0.0007514 0.0007514 0.0007514 0.20% MLMG::getGradSolution() 6 0.000751 0.000751 0.000751 0.20% MLCellLinOp::compGrad() 6 0.0007479 0.0007479 0.0007479 0.19% MultiFab::Add() 36 0.0007055 0.0007055 0.0007055 0.18% FabArray::setDomainBndry() 20 0.0006489 0.0006489 0.0006489 0.17% FabArray::mult() 22 0.0006448 0.0006448 0.0006448 0.17% MLPoisson::prepareForSolve() 6 0.0006198 0.0006198 0.0006198 0.16% MLCellLinOp::prepareForSolve() 6 0.0006154 0.0006154 0.0006154 0.16% Castro::check_for_nan() 10 0.0005901 0.0005901 0.0005901 0.15% MultiFab::contains_nan() 10 0.0005836 0.0005836 0.0005836 0.15% MLMG::computeMLResidual() 6 0.0005599 0.0005599 0.0005599 0.15% Castro::enforce_speed_limit() 30 0.000451 0.000451 0.000451 0.12% Gravity::update_max_rhs() 6 0.0004432 0.0004432 0.0004432 0.12% Amr::InitAmr() 1 0.0003949 0.0003949 0.0003949 0.10% Castro::finalize_advance() 5 0.0003855 0.0003855 0.0003855 0.10% FabArrayBase::CPC::define() 244 0.0003829 0.0003829 0.0003829 0.10% FabArrayBase::getFB() 1766 0.0003045 0.0003045 0.0003045 0.08% Gravity::swapTimeLevels() 5 0.0002207 0.0002207 0.0002207 0.06% MultiFab::Copy() 6 0.0001723 0.0001723 0.0001723 0.04% MLMG::MLResNormInf() 6 0.000147 0.000147 0.000147 0.04% Castro::buildMetrics() 1 0.0001459 0.0001459 0.0001459 0.04% MLLinOp::define() 6 0.0001401 0.0001401 0.0001401 0.04% MultiFab::max() 6 0.0001327 0.0001327 0.0001327 0.03% MLLinOp::defineGrids() 6 0.0001193 0.0001193 0.0001193 0.03% MLMG::MLRhsNormInf() 6 0.0001126 0.0001126 0.0001126 0.03% Castro::create_source_corrector() 5 8.271e-05 8.271e-05 8.271e-05 0.02% FabArrayBase::FB::FB() 26 5.605e-05 5.605e-05 5.605e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.92e-05 2.92e-05 2.92e-05 0.01% Amr::writeSmallPlotFile() 1 2.687e-05 2.687e-05 2.687e-05 0.01% Castro::swap_state_time_levels() 5 2.684e-05 2.684e-05 2.684e-05 0.01% Castro::initMFs() 1 2.652e-05 2.652e-05 2.652e-05 0.01% makeSFC 30 2.196e-05 2.196e-05 2.196e-05 0.01% Castro::finalize_do_advance() 5 1.767e-05 1.767e-05 1.767e-05 0.00% DistributionMapping::Distribute() 31 8.443e-06 8.443e-06 8.443e-06 0.00% Amr::initSubcycle() 1 8.194e-06 8.194e-06 8.194e-06 0.00% Gravity::set_mass_offset() 6 4.449e-06 4.449e-06 4.449e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.224e-06 4.224e-06 4.224e-06 0.00% Castro::FluxRegCrseInit 5 2.67e-06 2.67e-06 2.67e-06 0.00% Castro::retry_advance_ctu() 5 2.521e-06 2.521e-06 2.521e-06 0.00% Castro::FluxRegFineAdd() 5 1.49e-06 1.49e-06 1.49e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.17e-06 1.17e-06 1.17e-06 0.00% AmrLevel::AmrLevel() 1 9.03e-07 9.03e-07 9.03e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.11-14-ge0afc82f63ce) finalized