Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-13-g17c94cc196d7) initialized Starting run at 08:31:04 UTC on 2022-09-15. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09-13-g17c94cc19 Microphysics git describe: 22.08-13-g1adf1bdb reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.051743706 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.029427902 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.047862106 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052442149 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.062352869 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.059072918 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.071833269 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.047491407 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.065286884 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050474323 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.047791134 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.056540361 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.060285157 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.04750753 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029397666 seconds Ending run at 08:31:04 UTC on 2022-09-15. Run time = 0.828656697 Run time without initialization = 0.698958626 Average number of zones advanced per microsecond: 3.750 Average number of zones advanced per microsecond per rank: 3.750 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8287 ... 0.8287 ... 0.8287 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.1982 0.1982 0.1982 23.92% Castro::construct_ctu_hydro_source() 10 0.1967 0.1967 0.1967 23.74% MLCellLinOp::applyBC() 4379 0.07834 0.07834 0.07834 9.45% MLPoisson::Fsmooth() 3240 0.06205 0.06205 0.06205 7.49% StateData::FillBoundary(geom) 328 0.0235 0.0235 0.0235 2.84% MLCGSolver::bicgstab 81 0.02316 0.02316 0.02316 2.79% MultiFab::Dot() 1100 0.02169 0.02169 0.02169 2.62% Castro::normalize_species() 62 0.01407 0.01407 0.01407 1.70% MultiFab::LinComb() 1566 0.01394 0.01394 0.01394 1.68% FillBoundary_nowait() 3974 0.01391 0.01391 0.01391 1.68% FabArray::setVal() 1135 0.01385 0.01385 0.01385 1.67% Castro::computeTemp() 63 0.01313 0.01313 0.01313 1.59% FabArray::ParallelCopy_nowait() 851 0.01287 0.01287 0.01287 1.55% StateDataPhysBCFunct::() 41 0.01257 0.01257 0.01257 1.52% MLCellLinOp::defineAuxData() 11 0.01144 0.01144 0.01144 1.38% MLPoisson::Fapply() 1128 0.01142 0.01142 0.01142 1.38% Gravity::fill_multipole_BCs() 11 0.008131 0.008131 0.008131 0.98% Castro::enforce_min_density() 62 0.007881 0.007881 0.007881 0.95% MLMG::addInterpCorrection() 405 0.007646 0.007646 0.007646 0.92% amrex::average_down 405 0.006665 0.006665 0.006665 0.80% MultiFab::Xpay() 578 0.006353 0.006353 0.006353 0.77% Castro::estTimeStep() 21 0.005394 0.005394 0.005394 0.65% Castro::do_advance_ctu() 10 0.005334 0.005334 0.005334 0.64% Amr::checkPoint() 3 0.00483 0.00483 0.00483 0.58% Castro::reset_internal_energy(MultiFab) 63 0.003914 0.003914 0.003914 0.47% BndryData::define() 11 0.003732 0.003732 0.003732 0.45% Castro::construct_new_gravity_source() 10 0.003268 0.003268 0.003268 0.39% Castro::construct_old_gravity_source() 10 0.002591 0.002591 0.002591 0.31% Amr::writePlotFile() 2 0.002571 0.002571 0.002571 0.31% MLMG::ResNormInf() 92 0.002036 0.002036 0.002036 0.25% Gravity::get_new_grav_vector() 11 0.001911 0.001911 0.001911 0.23% MultiFab::Saxpy() 20 0.001806 0.001806 0.001806 0.22% Castro::expand_state() 10 0.001734 0.001734 0.001734 0.21% Gravity::get_old_grav_vector() 10 0.001724 0.001724 0.001724 0.21% MultiFab::Add() 81 0.001635 0.001635 0.001635 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001614 0.001614 0.001614 0.19% Castro::reset_internal_energy(Fab) 504 0.001541 0.001541 0.001541 0.19% MLCellLinOp::setLevelBC() 11 0.001499 0.001499 0.001499 0.18% Gravity::actual_solve_with_mlmg() 11 0.001445 0.001445 0.001445 0.17% FabArray::mult() 43 0.001331 0.001331 0.001331 0.16% FabArray::setDomainBndry() 41 0.001313 0.001313 0.001313 0.16% Castro::initData() 1 0.001253 0.001253 0.001253 0.15% MLMG::prepareForSolve() 11 0.001232 0.001232 0.001232 0.15% MultiFab::contains_nan() 20 0.001176 0.001176 0.001176 0.14% MLCellLinOp::prepareForSolve() 11 0.001153 0.001153 0.001153 0.14% MLCellLinOp::smooth() 1620 0.00114 0.00114 0.00114 0.14% Castro::enforce_speed_limit() 62 0.001085 0.001085 0.001085 0.13% MLCellLinOp::compGrad() 11 0.0009032 0.0009032 0.0009032 0.11% FabArray::FillBoundary() 3974 0.0008112 0.0008112 0.0008112 0.10% Castro::subcycle_advance_ctu() 10 0.0007763 0.0007763 0.0007763 0.09% FabArrayBase::getCPC() 1313 0.0007532 0.0007532 0.0007532 0.09% FabArrayBase::CPC::define() 454 0.0006578 0.0006578 0.0006578 0.08% FabArrayBase::getFB() 3974 0.0005815 0.0005815 0.0005815 0.07% Amr::InitAmr() 1 0.0004818 0.0004818 0.0004818 0.06% MLCellLinOp::apply() 1128 0.000464 0.000464 0.000464 0.06% Gravity::solve_for_phi() 10 0.0004504 0.0004504 0.0004504 0.05% Gravity::update_max_rhs() 11 0.0004132 0.0004132 0.0004132 0.05% CGSolver::sxay() 1566 0.000338 0.000338 0.000338 0.04% FillPatchIterator::Initialize 41 0.000334 0.000334 0.000334 0.04% Amr::coarseTimeStep() 10 0.000331 0.000331 0.000331 0.04% MultiFab::Copy() 11 0.0003213 0.0003213 0.0003213 0.04% MLCellLinOp::defineBC() 11 0.0002892 0.0002892 0.0002892 0.03% MLCGSolver::ParallelAllReduce 1495 0.0002877 0.0002877 0.0002877 0.03% main() 1 0.0002798 0.0002798 0.0002798 0.03% FabArray::ParallelCopy() 851 0.0002586 0.0002586 0.0002586 0.03% MultiFab::max() 11 0.0002565 0.0002565 0.0002565 0.03% MLCellLinOp::correctionResidual() 486 0.0002194 0.0002194 0.0002194 0.03% MLMG::MLRhsNormInf() 11 0.0002133 0.0002133 0.0002133 0.03% MLMG::mgVcycle() 81 0.0002121 0.0002121 0.0002121 0.03% Castro::construct_new_gravity() 10 0.0002025 0.0002025 0.0002025 0.02% Amr::timeStep() 10 0.0001707 0.0001707 0.0001707 0.02% MLLinOp::defineGrids() 11 0.0001596 0.0001596 0.0001596 0.02% MLMG:computeResOfCorrection() 405 0.0001437 0.0001437 0.0001437 0.02% StateData::checkPoint() 12 0.0001328 0.0001328 0.0001328 0.02% MLMG::mgVcycle_down::0 81 0.000109 0.000109 0.000109 0.01% Castro::advance() 10 9.35e-05 9.35e-05 9.35e-05 0.01% Castro::Castro() 1 9.136e-05 9.136e-05 9.136e-05 0.01% MLMG::mgVcycle_down::1 81 8.638e-05 8.638e-05 8.638e-05 0.01% FabArrayBase::FB::FB() 56 8.33e-05 8.33e-05 8.33e-05 0.01% Castro::clean_state() 62 8.326e-05 8.326e-05 8.326e-05 0.01% MLMG::mgVcycle_down::2 81 8.217e-05 8.217e-05 8.217e-05 0.01% Castro::initialize_advance() 10 8.212e-05 8.212e-05 8.212e-05 0.01% MLMG::actualBottomSolve() 81 8.118e-05 8.118e-05 8.118e-05 0.01% MLMG::mgVcycle_down::3 81 7.549e-05 7.549e-05 7.549e-05 0.01% MLMG::mgVcycle_down::4 81 7.492e-05 7.492e-05 7.492e-05 0.01% Castro::finalize_advance() 10 7.28e-05 7.28e-05 7.28e-05 0.01% AmrLevel::checkPoint() 3 7.196e-05 7.196e-05 7.196e-05 0.01% MLMG::solve() 11 7.077e-05 7.077e-05 7.077e-05 0.01% MLMG::mgVcycle_up::4 81 6.624e-05 6.624e-05 6.624e-05 0.01% Castro::initialize_do_advance() 10 6.581e-05 6.581e-05 6.581e-05 0.01% MLMG::mgVcycle_up::2 81 5.717e-05 5.717e-05 5.717e-05 0.01% MLMG::oneIter() 81 5.561e-05 5.561e-05 5.561e-05 0.01% MLMG::mgVcycle_up::0 81 5.557e-05 5.557e-05 5.557e-05 0.01% MLMG::mgVcycle_up::1 81 5.513e-05 5.513e-05 5.513e-05 0.01% MLMG::mgVcycle_up::3 81 5.332e-05 5.332e-05 5.332e-05 0.01% MLCellLinOp::solutionResidual() 92 5.013e-05 5.013e-05 5.013e-05 0.01% StateData::define() 4 4.354e-05 4.354e-05 4.354e-05 0.01% Castro::swap_state_time_levels() 10 4.027e-05 4.027e-05 4.027e-05 0.00% MLMG::computeResidual() 81 3.946e-05 3.946e-05 3.946e-05 0.00% Castro::finalize_do_advance() 10 3.476e-05 3.476e-05 3.476e-05 0.00% Castro::enforce_consistent_e() 1 3.407e-05 3.407e-05 3.407e-05 0.00% MLMG::mgVcycle_bottom 81 3.347e-05 3.347e-05 3.347e-05 0.00% Gravity::actual_multilevel_solve() 1 3.165e-05 3.165e-05 3.165e-05 0.00% Castro::initMFs() 1 2.939e-05 2.939e-05 2.939e-05 0.00% FillPatchSingleLevel 41 2.874e-05 2.874e-05 2.874e-05 0.00% MLPoisson::define() 11 2.842e-05 2.842e-05 2.842e-05 0.00% Amr::defBaseLevel() 1 2.752e-05 2.752e-05 2.752e-05 0.00% makeSFC 55 2.663e-05 2.663e-05 2.663e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.591e-05 2.591e-05 2.591e-05 0.00% Amr::writeSmallPlotFile() 1 2.578e-05 2.578e-05 2.578e-05 0.00% MLLinOp::define() 11 2.511e-05 2.511e-05 2.511e-05 0.00% Castro::buildMetrics() 1 2.448e-05 2.448e-05 2.448e-05 0.00% Castro::create_source_corrector() 10 2.114e-05 2.114e-05 2.114e-05 0.00% Amr::FinalizeInit() 1 2.039e-05 2.039e-05 2.039e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.993e-05 1.993e-05 1.993e-05 0.00% Castro::construct_old_source() 50 1.821e-05 1.821e-05 1.821e-05 0.00% Castro::construct_new_source() 50 1.714e-05 1.714e-05 1.714e-05 0.00% Castro::do_new_sources() 10 1.641e-05 1.641e-05 1.641e-05 0.00% Castro::do_old_sources() 10 1.605e-05 1.605e-05 1.605e-05 0.00% DistributionMapping::Distribute() 56 1.539e-05 1.539e-05 1.539e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.43e-05 1.43e-05 1.43e-05 0.00% Castro::check_for_nan() 20 1.233e-05 1.233e-05 1.233e-05 0.00% Castro::apply_source_to_state() 20 1.161e-05 1.161e-05 1.161e-05 0.00% Castro::construct_old_gravity() 10 9.777e-06 9.777e-06 9.777e-06 0.00% MLMG::computeMLResidual() 11 9.03e-06 9.03e-06 9.03e-06 0.00% MLPoisson::prepareForSolve() 11 8.916e-06 8.916e-06 8.916e-06 0.00% Amr::initSubcycle() 1 8.554e-06 8.554e-06 8.554e-06 0.00% Gravity::swapTimeLevels() 10 8.553e-06 8.553e-06 8.553e-06 0.00% Castro::post_timestep() 10 8.547e-06 8.547e-06 8.547e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.504e-06 6.504e-06 6.504e-06 0.00% MLMG::getGradSolution() 11 6.046e-06 6.046e-06 6.046e-06 0.00% Castro::computeNewDt() 9 5.79e-06 5.79e-06 5.79e-06 0.00% AmrLevel::checkPointPost() 3 5.645e-06 5.645e-06 5.645e-06 0.00% Castro::retry_advance_ctu() 10 4.864e-06 4.864e-06 4.864e-06 0.00% Amr::InitializeInit() 1 4.758e-06 4.758e-06 4.758e-06 0.00% Castro::post_init() 1 4.729e-06 4.729e-06 4.729e-06 0.00% Gravity::set_mass_offset() 11 4.364e-06 4.364e-06 4.364e-06 0.00% MLMG::MLResNormInf() 11 3.453e-06 3.453e-06 3.453e-06 0.00% Castro::FluxRegCrseInit 10 3.046e-06 3.046e-06 3.046e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.892e-06 2.892e-06 2.892e-06 0.00% Castro::FluxRegFineAdd() 10 2.759e-06 2.759e-06 2.759e-06 0.00% Castro::computeInitialDt() 2 2.717e-06 2.717e-06 2.717e-06 0.00% Amr::init() 1 2.474e-06 2.474e-06 2.474e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.92e-06 1.92e-06 1.92e-06 0.00% AmrLevel::checkPointPre() 3 1.85e-06 1.85e-06 1.85e-06 0.00% Castro::post_regrid() 1 1.283e-06 1.283e-06 1.283e-06 0.00% Amr::initialInit() 1 1.132e-06 1.132e-06 1.132e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8287 0.8287 0.8287 100.00% Amr::coarseTimeStep() 10 0.6693 0.6693 0.6693 80.77% Amr::timeStep() 10 0.5709 0.5709 0.5709 68.89% Castro::advance() 10 0.5646 0.5646 0.5646 68.14% Castro::subcycle_advance_ctu() 10 0.5539 0.5539 0.5539 66.84% Castro::do_advance_ctu() 10 0.5531 0.5531 0.5531 66.74% Gravity::solve_phi_with_mlmg() 11 0.3064 0.3064 0.3064 36.97% Gravity::actual_solve_with_mlmg() 11 0.298 0.298 0.298 35.96% Castro::construct_new_gravity() 10 0.2812 0.2812 0.2812 33.94% MLMG::solve() 11 0.2757 0.2757 0.2757 33.27% Gravity::solve_for_phi() 10 0.2662 0.2662 0.2662 32.13% MLMG::oneIter() 81 0.2611 0.2611 0.2611 31.51% MLMG::mgVcycle() 81 0.2594 0.2594 0.2594 31.30% VisMF::Write(FabArray) 11 0.1982 0.1982 0.1982 23.92% Castro::construct_ctu_hydro_source() 10 0.1967 0.1967 0.1967 23.74% Amr::checkPoint() 3 0.1469 0.1469 0.1469 17.72% AmrLevel::checkPoint() 3 0.142 0.142 0.142 17.14% StateData::checkPoint() 12 0.142 0.142 0.142 17.13% MLCellLinOp::smooth() 1620 0.1329 0.1329 0.1329 16.03% Amr::init() 1 0.1291 0.1291 0.1291 15.58% MLCellLinOp::applyBC() 4379 0.09372 0.09372 0.09372 11.31% MLMG::mgVcycle_bottom 81 0.07954 0.07954 0.07954 9.60% MLMG::actualBottomSolve() 81 0.07951 0.07951 0.07951 9.59% MLCGSolver::bicgstab 81 0.07872 0.07872 0.07872 9.50% MLPoisson::Fsmooth() 3240 0.06205 0.06205 0.06205 7.49% Amr::writePlotFile() 2 0.05896 0.05896 0.05896 7.11% Amr::initialInit() 1 0.04778 0.04778 0.04778 5.77% Amr::FinalizeInit() 1 0.04367 0.04367 0.04367 5.27% Castro::post_init() 1 0.04234 0.04234 0.04234 5.11% FillPatchIterator::Initialize 41 0.04174 0.04174 0.04174 5.04% Castro::clean_state() 62 0.0409 0.0409 0.0409 4.94% Gravity::multilevel_solve_for_new_phi() 1 0.04063 0.04063 0.04063 4.90% Gravity::actual_multilevel_solve() 1 0.04061 0.04061 0.04061 4.90% FillPatchSingleLevel 41 0.0401 0.0401 0.0401 4.84% StateDataPhysBCFunct::() 41 0.03607 0.03607 0.03607 4.35% MLCellLinOp::apply() 1128 0.03544 0.03544 0.03544 4.28% MLMG::mgVcycle_down::0 81 0.03442 0.03442 0.03442 4.15% MLMG::mgVcycle_up::0 81 0.02962 0.02962 0.02962 3.57% StateData::FillBoundary(geom) 328 0.0235 0.0235 0.0235 2.84% MultiFab::Dot() 1100 0.02169 0.02169 0.02169 2.62% MLCellLinOp::correctionResidual() 486 0.0207 0.0207 0.0207 2.50% Castro::computeTemp() 63 0.01859 0.01859 0.01859 2.24% Castro::initialize_do_advance() 10 0.01828 0.01828 0.01828 2.21% MLPoisson::define() 11 0.01795 0.01795 0.01795 2.17% MLMG:computeResOfCorrection() 405 0.01789 0.01789 0.01789 2.16% MLMG::mgVcycle_down::1 81 0.01727 0.01727 0.01727 2.08% MLMG::mgVcycle_down::2 81 0.01678 0.01678 0.01678 2.02% Gravity::get_new_grav_vector() 11 0.0164 0.0164 0.0164 1.98% MLMG::mgVcycle_down::3 81 0.01593 0.01593 0.01593 1.92% FabArray::FillBoundary() 3974 0.01538 0.01538 0.01538 1.86% MLMG::mgVcycle_down::4 81 0.01524 0.01524 0.01524 1.84% Castro::construct_old_gravity() 10 0.01474 0.01474 0.01474 1.78% Gravity::get_old_grav_vector() 10 0.01473 0.01473 0.01473 1.78% FillBoundary_nowait() 3974 0.01457 0.01457 0.01457 1.76% CGSolver::sxay() 1566 0.01427 0.01427 0.01427 1.72% Castro::normalize_species() 62 0.01407 0.01407 0.01407 1.70% MultiFab::LinComb() 1566 0.01394 0.01394 0.01394 1.68% FabArray::ParallelCopy() 851 0.01393 0.01393 0.01393 1.68% FabArray::setVal() 1135 0.01385 0.01385 0.01385 1.67% FabArray::ParallelCopy_nowait() 851 0.01367 0.01367 0.01367 1.65% MLMG::mgVcycle_up::2 81 0.013 0.013 0.013 1.57% MLCGSolver::ParallelAllReduce 1495 0.01297 0.01297 0.01297 1.56% MLMG::mgVcycle_up::1 81 0.01276 0.01276 0.01276 1.54% MLCellLinOp::defineAuxData() 11 0.01275 0.01275 0.01275 1.54% Castro::do_new_sources() 10 0.01264 0.01264 0.01264 1.53% MLMG::addInterpCorrection() 405 0.01261 0.01261 0.01261 1.52% MLMG::mgVcycle_up::3 81 0.01233 0.01233 0.01233 1.49% MLMG::mgVcycle_up::4 81 0.01229 0.01229 0.01229 1.48% amrex::average_down 405 0.01163 0.01163 0.01163 1.40% Castro::expand_state() 10 0.01145 0.01145 0.01145 1.38% MLPoisson::Fapply() 1128 0.01142 0.01142 0.01142 1.38% Castro::initialize_advance() 10 0.01058 0.01058 0.01058 1.28% Castro::do_old_sources() 10 0.009472 0.009472 0.009472 1.14% Gravity::fill_multipole_BCs() 11 0.008131 0.008131 0.008131 0.98% Castro::enforce_min_density() 62 0.007881 0.007881 0.007881 0.95% MLCellLinOp::solutionResidual() 92 0.006948 0.006948 0.006948 0.84% MultiFab::Xpay() 578 0.006353 0.006353 0.006353 0.77% Castro::post_timestep() 10 0.00607 0.00607 0.00607 0.73% MLMG::computeResidual() 81 0.005989 0.005989 0.005989 0.72% Castro::reset_internal_energy(MultiFab) 63 0.005456 0.005456 0.005456 0.66% Castro::estTimeStep() 21 0.005394 0.005394 0.005394 0.65% MLMG::prepareForSolve() 11 0.005301 0.005301 0.005301 0.64% MLCellLinOp::defineBC() 11 0.004938 0.004938 0.004938 0.60% BndryData::define() 11 0.004649 0.004649 0.004649 0.56% Amr::InitializeInit() 1 0.004107 0.004107 0.004107 0.50% Amr::defBaseLevel() 1 0.004102 0.004102 0.004102 0.49% Castro::initData() 1 0.003573 0.003573 0.003573 0.43% Castro::construct_new_source() 50 0.003285 0.003285 0.003285 0.40% Castro::construct_new_gravity_source() 10 0.003268 0.003268 0.003268 0.39% Castro::construct_old_source() 50 0.002609 0.002609 0.002609 0.31% Castro::construct_old_gravity_source() 10 0.002591 0.002591 0.002591 0.31% Castro::computeNewDt() 9 0.002541 0.002541 0.002541 0.31% MLMG::ResNormInf() 92 0.002036 0.002036 0.002036 0.25% Castro::apply_source_to_state() 20 0.001818 0.001818 0.001818 0.22% MultiFab::Saxpy() 20 0.001806 0.001806 0.001806 0.22% MultiFab::Add() 81 0.001635 0.001635 0.001635 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001614 0.001614 0.001614 0.19% Castro::reset_internal_energy(Fab) 504 0.001541 0.001541 0.001541 0.19% MLCellLinOp::setLevelBC() 11 0.001499 0.001499 0.001499 0.18% FabArrayBase::getCPC() 1313 0.001411 0.001411 0.001411 0.17% MLMG::getGradSolution() 11 0.001406 0.001406 0.001406 0.17% MLCellLinOp::compGrad() 11 0.0014 0.0014 0.0014 0.17% FabArray::mult() 43 0.001331 0.001331 0.001331 0.16% FabArray::setDomainBndry() 41 0.001313 0.001313 0.001313 0.16% Castro::check_for_nan() 20 0.001189 0.001189 0.001189 0.14% MultiFab::contains_nan() 20 0.001176 0.001176 0.001176 0.14% MLPoisson::prepareForSolve() 11 0.001161 0.001161 0.001161 0.14% MLCellLinOp::prepareForSolve() 11 0.001153 0.001153 0.001153 0.14% Castro::enforce_speed_limit() 62 0.001085 0.001085 0.001085 0.13% Castro::post_regrid() 1 0.001082 0.001082 0.001082 0.13% MLMG::computeMLResidual() 11 0.001008 0.001008 0.001008 0.12% Gravity::update_max_rhs() 11 0.0008209 0.0008209 0.0008209 0.10% Castro::computeInitialDt() 2 0.0007439 0.0007439 0.0007439 0.09% FabArrayBase::getFB() 3974 0.0006648 0.0006648 0.0006648 0.08% FabArrayBase::CPC::define() 454 0.0006578 0.0006578 0.0006578 0.08% Amr::InitAmr() 1 0.0004903 0.0004903 0.0004903 0.06% Castro::Castro() 1 0.0004473 0.0004473 0.0004473 0.05% Gravity::swapTimeLevels() 10 0.0004321 0.0004321 0.0004321 0.05% MultiFab::Copy() 11 0.0003213 0.0003213 0.0003213 0.04% MLMG::MLResNormInf() 11 0.000275 0.000275 0.000275 0.03% MultiFab::max() 11 0.0002565 0.0002565 0.0002565 0.03% MLLinOp::define() 11 0.0002416 0.0002416 0.0002416 0.03% MLLinOp::defineGrids() 11 0.0002165 0.0002165 0.0002165 0.03% MLMG::MLRhsNormInf() 11 0.0002133 0.0002133 0.0002133 0.03% Castro::buildMetrics() 1 0.0001615 0.0001615 0.0001615 0.02% FabArrayBase::FB::FB() 56 8.33e-05 8.33e-05 8.33e-05 0.01% Castro::finalize_advance() 10 7.861e-05 7.861e-05 7.861e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.496e-05 5.496e-05 5.496e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.004e-05 5.004e-05 5.004e-05 0.01% StateData::define() 4 4.354e-05 4.354e-05 4.354e-05 0.01% makeSFC 55 4.066e-05 4.066e-05 4.066e-05 0.00% Castro::swap_state_time_levels() 10 4.027e-05 4.027e-05 4.027e-05 0.00% Castro::finalize_do_advance() 10 3.476e-05 3.476e-05 3.476e-05 0.00% Castro::enforce_consistent_e() 1 3.407e-05 3.407e-05 3.407e-05 0.00% Castro::initMFs() 1 2.939e-05 2.939e-05 2.939e-05 0.00% Amr::writeSmallPlotFile() 1 2.578e-05 2.578e-05 2.578e-05 0.00% Castro::create_source_corrector() 10 2.114e-05 2.114e-05 2.114e-05 0.00% DistributionMapping::Distribute() 56 1.539e-05 1.539e-05 1.539e-05 0.00% Amr::initSubcycle() 1 8.554e-06 8.554e-06 8.554e-06 0.00% AmrLevel::checkPointPost() 3 5.645e-06 5.645e-06 5.645e-06 0.00% Castro::retry_advance_ctu() 10 4.864e-06 4.864e-06 4.864e-06 0.00% Gravity::set_mass_offset() 11 4.364e-06 4.364e-06 4.364e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.251e-06 4.251e-06 4.251e-06 0.00% Castro::FluxRegCrseInit 10 3.046e-06 3.046e-06 3.046e-06 0.00% Castro::FluxRegFineAdd() 10 2.759e-06 2.759e-06 2.759e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.92e-06 1.92e-06 1.92e-06 0.00% AmrLevel::checkPointPre() 3 1.85e-06 1.85e-06 1.85e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-13-g17c94cc196d7) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-13-g17c94cc196d7) initialized Starting run at 08:31:05 UTC on 2022-09-15. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09-13-g17c94cc19 Microphysics git describe: 22.08-13-g1adf1bdb reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.470760674 Restart time = 0.048713365 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.053865632 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048354663 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058689643 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060654019 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.064089721 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031405099 seconds Ending run at 08:31:06 UTC on 2022-09-15. Run time = 0.366755767 Run time without initialization = 0.317499316 Average number of zones advanced per microsecond: 4.128 Average number of zones advanced per microsecond per rank: 4.128 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3668 ... 0.3668 ... 0.3668 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0911 0.0911 0.0911 24.84% VisMF::Read() 3 0.04101 0.04101 0.04101 11.18% MLCellLinOp::applyBC() 1946 0.03379 0.03379 0.03379 9.21% VisMF::Write(FabArray) 1 0.02988 0.02988 0.02988 8.15% MLPoisson::Fsmooth() 1440 0.0269 0.0269 0.0269 7.33% StateData::FillBoundary(geom) 160 0.01161 0.01161 0.01161 3.16% MLCGSolver::bicgstab 36 0.009888 0.009888 0.009888 2.70% MultiFab::Dot() 484 0.009211 0.009211 0.009211 2.51% Castro::normalize_species() 30 0.008157 0.008157 0.008157 2.22% Castro::computeTemp() 30 0.007412 0.007412 0.007412 2.02% FabArray::setVal() 537 0.006529 0.006529 0.006529 1.78% FillBoundary_nowait() 1766 0.006145 0.006145 0.006145 1.68% MLCellLinOp::defineAuxData() 6 0.006092 0.006092 0.006092 1.66% MultiFab::LinComb() 690 0.005926 0.005926 0.005926 1.62% FabArray::ParallelCopy_nowait() 380 0.005774 0.005774 0.005774 1.57% Castro::enforce_min_density() 30 0.005759 0.005759 0.005759 1.57% StateDataPhysBCFunct::() 20 0.005263 0.005263 0.005263 1.43% MLPoisson::Fapply() 500 0.004902 0.004902 0.004902 1.34% Gravity::fill_multipole_BCs() 6 0.004038 0.004038 0.004038 1.10% Amr::restart() 1 0.00361 0.00361 0.00361 0.98% MLMG::addInterpCorrection() 180 0.003297 0.003297 0.003297 0.90% Castro::do_advance_ctu() 5 0.003187 0.003187 0.003187 0.87% amrex::average_down 180 0.002906 0.002906 0.002906 0.79% MultiFab::Xpay() 258 0.002766 0.002766 0.002766 0.75% Castro::estTimeStep() 10 0.002508 0.002508 0.002508 0.68% BndryData::define() 6 0.002024 0.002024 0.002024 0.55% Castro::reset_internal_energy(MultiFab) 30 0.001771 0.001771 0.001771 0.48% Castro::construct_new_gravity_source() 5 0.001685 0.001685 0.001685 0.46% Amr::writePlotFile() 1 0.001613 0.001613 0.001613 0.44% Castro::construct_old_gravity_source() 5 0.001255 0.001255 0.001255 0.34% Castro::enforce_speed_limit() 30 0.001199 0.001199 0.001199 0.33% Gravity::get_old_grav_vector() 5 0.001002 0.001002 0.001002 0.27% Gravity::get_new_grav_vector() 5 0.0009493 0.0009493 0.0009493 0.26% MultiFab::Saxpy() 10 0.0009161 0.0009161 0.0009161 0.25% MLMG::ResNormInf() 42 0.0009145 0.0009145 0.0009145 0.25% Castro::reset_internal_energy(Fab) 240 0.0008731 0.0008731 0.0008731 0.24% Castro::expand_state() 5 0.0008724 0.0008724 0.0008724 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008591 0.0008591 0.0008591 0.23% Gravity::actual_solve_with_mlmg() 6 0.0008079 0.0008079 0.0008079 0.22% MLCellLinOp::setLevelBC() 6 0.0007943 0.0007943 0.0007943 0.22% MultiFab::Add() 36 0.0007138 0.0007138 0.0007138 0.19% MLMG::prepareForSolve() 6 0.0006559 0.0006559 0.0006559 0.18% FabArray::setDomainBndry() 20 0.0006507 0.0006507 0.0006507 0.18% FabArray::mult() 22 0.0006466 0.0006466 0.0006466 0.18% MLCellLinOp::prepareForSolve() 6 0.0006187 0.0006187 0.0006187 0.17% MultiFab::contains_nan() 10 0.0005831 0.0005831 0.0005831 0.16% MLCellLinOp::smooth() 720 0.0005693 0.0005693 0.0005693 0.16% MLCellLinOp::compGrad() 6 0.0004843 0.0004843 0.0004843 0.13% FabArrayBase::CPC::define() 244 0.0003973 0.0003973 0.0003973 0.11% Amr::InitAmr() 1 0.0003802 0.0003802 0.0003802 0.10% FabArray::FillBoundary() 1766 0.0003758 0.0003758 0.0003758 0.10% FabArrayBase::getCPC() 632 0.0003683 0.0003683 0.0003683 0.10% FabArrayBase::getFB() 1766 0.0002788 0.0002788 0.0002788 0.08% main() 1 0.0002524 0.0002524 0.0002524 0.07% Gravity::update_max_rhs() 6 0.0002267 0.0002267 0.0002267 0.06% MLCellLinOp::apply() 500 0.0002236 0.0002236 0.0002236 0.06% Gravity::solve_for_phi() 5 0.0002146 0.0002146 0.0002146 0.06% Amr::coarseTimeStep() 5 0.0002032 0.0002032 0.0002032 0.06% Castro::construct_new_gravity() 5 0.000178 0.000178 0.000178 0.05% CGSolver::sxay() 690 0.0001766 0.0001766 0.0001766 0.05% MultiFab::Copy() 6 0.0001701 0.0001701 0.0001701 0.05% MLCellLinOp::defineBC() 6 0.0001471 0.0001471 0.0001471 0.04% FillPatchIterator::Initialize 20 0.0001444 0.0001444 0.0001444 0.04% Castro::subcycle_advance_ctu() 5 0.0001444 0.0001444 0.0001444 0.04% MultiFab::max() 6 0.0001347 0.0001347 0.0001347 0.04% MLCGSolver::ParallelAllReduce 659 0.0001256 0.0001256 0.0001256 0.03% FabArray::ParallelCopy() 380 0.000124 0.000124 0.000124 0.03% Castro::advance() 5 0.0001212 0.0001212 0.0001212 0.03% Castro::construct_new_source() 25 0.0001196 0.0001196 0.0001196 0.03% MLMG::MLRhsNormInf() 6 0.0001115 0.0001115 0.0001115 0.03% MLMG::mgVcycle() 36 0.0001053 0.0001053 0.0001053 0.03% MLCellLinOp::correctionResidual() 216 9.936e-05 9.936e-05 9.936e-05 0.03% Castro::initialize_do_advance() 5 9.414e-05 9.414e-05 9.414e-05 0.03% Castro::post_timestep() 5 8.795e-05 8.795e-05 8.795e-05 0.02% MLLinOp::defineGrids() 6 8.691e-05 8.691e-05 8.691e-05 0.02% Amr::timeStep() 5 8.471e-05 8.471e-05 8.471e-05 0.02% StateData::restartDoit() 4 7.884e-05 7.884e-05 7.884e-05 0.02% Castro::create_source_corrector() 5 7.844e-05 7.844e-05 7.844e-05 0.02% AmrLevel::restart() 1 7.582e-05 7.582e-05 7.582e-05 0.02% MLMG:computeResOfCorrection() 180 6.505e-05 6.505e-05 6.505e-05 0.02% FabArrayBase::FB::FB() 26 5.699e-05 5.699e-05 5.699e-05 0.02% Castro::finalize_advance() 5 5.567e-05 5.567e-05 5.567e-05 0.02% Castro::construct_old_source() 25 5.123e-05 5.123e-05 5.123e-05 0.01% MLMG::mgVcycle_down::0 36 4.916e-05 4.916e-05 4.916e-05 0.01% Castro::clean_state() 30 4.349e-05 4.349e-05 4.349e-05 0.01% MLMG::mgVcycle_down::1 36 4.127e-05 4.127e-05 4.127e-05 0.01% Castro::initialize_advance() 5 4.08e-05 4.08e-05 4.08e-05 0.01% MLMG::mgVcycle_down::2 36 3.872e-05 3.872e-05 3.872e-05 0.01% MLMG::actualBottomSolve() 36 3.86e-05 3.86e-05 3.86e-05 0.01% MLMG::mgVcycle_down::4 36 3.627e-05 3.627e-05 3.627e-05 0.01% Castro::computeNewDt() 5 3.559e-05 3.559e-05 3.559e-05 0.01% MLMG::mgVcycle_down::3 36 3.55e-05 3.55e-05 3.55e-05 0.01% MLMG::solve() 6 3.341e-05 3.341e-05 3.341e-05 0.01% Castro::buildMetrics() 1 3.255e-05 3.255e-05 3.255e-05 0.01% MLMG::mgVcycle_up::4 36 3.135e-05 3.135e-05 3.135e-05 0.01% Castro::post_restart() 1 3.026e-05 3.026e-05 3.026e-05 0.01% Castro::initMFs() 1 3.003e-05 3.003e-05 3.003e-05 0.01% Gravity::actual_multilevel_solve() 1 2.952e-05 2.952e-05 2.952e-05 0.01% Amr::writeSmallPlotFile() 1 2.74e-05 2.74e-05 2.74e-05 0.01% MLMG::mgVcycle_up::0 36 2.733e-05 2.733e-05 2.733e-05 0.01% Castro::swap_state_time_levels() 5 2.729e-05 2.729e-05 2.729e-05 0.01% MLMG::oneIter() 36 2.667e-05 2.667e-05 2.667e-05 0.01% MLMG::mgVcycle_up::3 36 2.538e-05 2.538e-05 2.538e-05 0.01% MLMG::mgVcycle_up::2 36 2.501e-05 2.501e-05 2.501e-05 0.01% MLMG::mgVcycle_up::1 36 2.362e-05 2.362e-05 2.362e-05 0.01% MLCellLinOp::solutionResidual() 42 2.359e-05 2.359e-05 2.359e-05 0.01% MLMG::computeResidual() 36 2.234e-05 2.234e-05 2.234e-05 0.01% MLPoisson::define() 6 2.121e-05 2.121e-05 2.121e-05 0.01% Castro::construct_old_gravity() 5 2.079e-05 2.079e-05 2.079e-05 0.01% MLLinOp::define() 6 2.065e-05 2.065e-05 2.065e-05 0.01% Castro::finalize_do_advance() 5 1.967e-05 1.967e-05 1.967e-05 0.01% Gravity::solve_phi_with_mlmg() 6 1.742e-05 1.742e-05 1.742e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.734e-05 1.734e-05 1.734e-05 0.00% MLMG::mgVcycle_bottom 36 1.551e-05 1.551e-05 1.551e-05 0.00% makeSFC 30 1.51e-05 1.51e-05 1.51e-05 0.00% FillPatchSingleLevel 20 1.352e-05 1.352e-05 1.352e-05 0.00% Castro::do_new_sources() 5 1.033e-05 1.033e-05 1.033e-05 0.00% DistributionMapping::Distribute() 31 9.519e-06 9.519e-06 9.519e-06 0.00% Castro::do_old_sources() 5 9.165e-06 9.165e-06 9.165e-06 0.00% Amr::initSubcycle() 1 7.929e-06 7.929e-06 7.929e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.657e-06 7.657e-06 7.657e-06 0.00% Castro::check_for_nan() 10 6.6e-06 6.6e-06 6.6e-06 0.00% Castro::apply_source_to_state() 10 5.854e-06 5.854e-06 5.854e-06 0.00% MLPoisson::prepareForSolve() 6 4.971e-06 4.971e-06 4.971e-06 0.00% MLMG::computeMLResidual() 6 4.819e-06 4.819e-06 4.819e-06 0.00% Gravity::swapTimeLevels() 5 4.705e-06 4.705e-06 4.705e-06 0.00% MLMG::getGradSolution() 6 3.179e-06 3.179e-06 3.179e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.065e-06 3.065e-06 3.065e-06 0.00% MLMG::MLResNormInf() 6 2.238e-06 2.238e-06 2.238e-06 0.00% Gravity::set_mass_offset() 6 2.105e-06 2.105e-06 2.105e-06 0.00% Castro::retry_advance_ctu() 5 2.05e-06 2.05e-06 2.05e-06 0.00% Castro::FluxRegCrseInit 5 1.777e-06 1.777e-06 1.777e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.549e-06 1.549e-06 1.549e-06 0.00% Castro::FluxRegFineAdd() 5 1.208e-06 1.208e-06 1.208e-06 0.00% Amr::init() 1 1.173e-06 1.173e-06 1.173e-06 0.00% AmrLevel::AmrLevel() 1 9.37e-07 9.37e-07 9.37e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3668 0.3668 0.3668 100.00% Amr::coarseTimeStep() 5 0.2859 0.2859 0.2859 77.93% Amr::timeStep() 5 0.2844 0.2844 0.2844 77.53% Castro::advance() 5 0.2792 0.2792 0.2792 76.12% Castro::subcycle_advance_ctu() 5 0.2737 0.2737 0.2737 74.63% Castro::do_advance_ctu() 5 0.2736 0.2736 0.2736 74.59% Castro::construct_new_gravity() 5 0.1402 0.1402 0.1402 38.23% Gravity::solve_phi_with_mlmg() 6 0.1359 0.1359 0.1359 37.06% Gravity::solve_for_phi() 5 0.1326 0.1326 0.1326 36.14% Gravity::actual_solve_with_mlmg() 6 0.1318 0.1318 0.1318 35.92% MLMG::solve() 6 0.1197 0.1197 0.1197 32.64% MLMG::oneIter() 36 0.1127 0.1127 0.1127 30.72% MLMG::mgVcycle() 36 0.1119 0.1119 0.1119 30.52% Castro::construct_ctu_hydro_source() 5 0.09109 0.09109 0.09109 24.84% MLCellLinOp::smooth() 720 0.05762 0.05762 0.05762 15.71% Amr::init() 1 0.04876 0.04876 0.04876 13.29% Amr::restart() 1 0.04876 0.04876 0.04876 13.29% AmrLevel::restart() 1 0.04122 0.04122 0.04122 11.24% StateData::restartDoit() 4 0.04114 0.04114 0.04114 11.22% VisMF::Read() 3 0.04101 0.04101 0.04101 11.18% MLCellLinOp::applyBC() 1946 0.04065 0.04065 0.04065 11.08% MLMG::mgVcycle_bottom 36 0.03393 0.03393 0.03393 9.25% MLMG::actualBottomSolve() 36 0.03392 0.03392 0.03392 9.25% MLCGSolver::bicgstab 36 0.03358 0.03358 0.03358 9.15% Amr::writePlotFile() 1 0.03149 0.03149 0.03149 8.59% VisMF::Write(FabArray) 1 0.02988 0.02988 0.02988 8.15% MLPoisson::Fsmooth() 1440 0.0269 0.0269 0.0269 7.33% Castro::clean_state() 30 0.02521 0.02521 0.02521 6.87% FillPatchIterator::Initialize 20 0.01967 0.01967 0.01967 5.36% FillPatchSingleLevel 20 0.01887 0.01887 0.01887 5.15% StateDataPhysBCFunct::() 20 0.01687 0.01687 0.01687 4.60% MLCellLinOp::apply() 500 0.01535 0.01535 0.01535 4.19% MLMG::mgVcycle_down::0 36 0.01511 0.01511 0.01511 4.12% MLMG::mgVcycle_up::0 36 0.01289 0.01289 0.01289 3.51% StateData::FillBoundary(geom) 160 0.01161 0.01161 0.01161 3.16% Castro::initialize_do_advance() 5 0.01042 0.01042 0.01042 2.84% Castro::computeTemp() 30 0.01006 0.01006 0.01006 2.74% MLPoisson::define() 6 0.009665 0.009665 0.009665 2.64% MultiFab::Dot() 484 0.009211 0.009211 0.009211 2.51% MLCellLinOp::correctionResidual() 216 0.008948 0.008948 0.008948 2.44% Castro::normalize_species() 30 0.008157 0.008157 0.008157 2.22% MLMG:computeResOfCorrection() 180 0.007728 0.007728 0.007728 2.11% Gravity::get_new_grav_vector() 5 0.007505 0.007505 0.007505 2.05% Castro::construct_old_gravity() 5 0.007488 0.007488 0.007488 2.04% MLMG::mgVcycle_down::1 36 0.007469 0.007469 0.007469 2.04% Gravity::get_old_grav_vector() 5 0.007468 0.007468 0.007468 2.04% MLMG::mgVcycle_down::2 36 0.007242 0.007242 0.007242 1.97% Castro::do_new_sources() 5 0.007074 0.007074 0.007074 1.93% MLMG::mgVcycle_down::3 36 0.006877 0.006877 0.006877 1.87% FabArray::FillBoundary() 1766 0.006857 0.006857 0.006857 1.87% MLCellLinOp::defineAuxData() 6 0.00681 0.00681 0.00681 1.86% MLMG::mgVcycle_down::4 36 0.006571 0.006571 0.006571 1.79% FabArray::setVal() 537 0.006529 0.006529 0.006529 1.78% FillBoundary_nowait() 1766 0.006481 0.006481 0.006481 1.77% FabArray::ParallelCopy() 380 0.006281 0.006281 0.006281 1.71% FabArray::ParallelCopy_nowait() 380 0.006157 0.006157 0.006157 1.68% CGSolver::sxay() 690 0.006103 0.006103 0.006103 1.66% MultiFab::LinComb() 690 0.005926 0.005926 0.005926 1.62% Castro::do_old_sources() 5 0.005895 0.005895 0.005895 1.61% Castro::enforce_min_density() 30 0.005759 0.005759 0.005759 1.57% MLMG::mgVcycle_up::2 36 0.005615 0.005615 0.005615 1.53% MLCGSolver::ParallelAllReduce 659 0.005546 0.005546 0.005546 1.51% Castro::expand_state() 5 0.005524 0.005524 0.005524 1.51% MLMG::mgVcycle_up::1 36 0.005513 0.005513 0.005513 1.50% MLMG::addInterpCorrection() 180 0.00543 0.00543 0.00543 1.48% MLMG::mgVcycle_up::3 36 0.005303 0.005303 0.005303 1.45% MLMG::mgVcycle_up::4 36 0.005302 0.005302 0.005302 1.45% Castro::initialize_advance() 5 0.005269 0.005269 0.005269 1.44% Castro::post_timestep() 5 0.005095 0.005095 0.005095 1.39% amrex::average_down 180 0.005066 0.005066 0.005066 1.38% MLPoisson::Fapply() 500 0.004902 0.004902 0.004902 1.34% Gravity::fill_multipole_BCs() 6 0.004038 0.004038 0.004038 1.10% Castro::post_restart() 1 0.003745 0.003745 0.003745 1.02% Gravity::multilevel_solve_for_new_phi() 1 0.003621 0.003621 0.003621 0.99% Gravity::actual_multilevel_solve() 1 0.003604 0.003604 0.003604 0.98% MLCellLinOp::solutionResidual() 42 0.003157 0.003157 0.003157 0.86% MLMG::prepareForSolve() 6 0.002811 0.002811 0.002811 0.77% MultiFab::Xpay() 258 0.002766 0.002766 0.002766 0.75% MLCellLinOp::defineBC() 6 0.002694 0.002694 0.002694 0.73% Castro::reset_internal_energy(MultiFab) 30 0.002644 0.002644 0.002644 0.72% MLMG::computeResidual() 36 0.002627 0.002627 0.002627 0.72% BndryData::define() 6 0.002547 0.002547 0.002547 0.69% Castro::estTimeStep() 10 0.002508 0.002508 0.002508 0.68% Castro::construct_new_source() 25 0.001805 0.001805 0.001805 0.49% Castro::construct_new_gravity_source() 5 0.001685 0.001685 0.001685 0.46% Castro::construct_old_source() 25 0.001307 0.001307 0.001307 0.36% Castro::computeNewDt() 5 0.001285 0.001285 0.001285 0.35% Castro::construct_old_gravity_source() 5 0.001255 0.001255 0.001255 0.34% Castro::enforce_speed_limit() 30 0.001199 0.001199 0.001199 0.33% Castro::apply_source_to_state() 10 0.000922 0.000922 0.000922 0.25% MultiFab::Saxpy() 10 0.0009161 0.0009161 0.0009161 0.25% MLMG::ResNormInf() 42 0.0009145 0.0009145 0.0009145 0.25% Castro::reset_internal_energy(Fab) 240 0.0008731 0.0008731 0.0008731 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008591 0.0008591 0.0008591 0.23% MLCellLinOp::setLevelBC() 6 0.0007943 0.0007943 0.0007943 0.22% FabArrayBase::getCPC() 632 0.0007656 0.0007656 0.0007656 0.21% MLMG::getGradSolution() 6 0.0007582 0.0007582 0.0007582 0.21% MLCellLinOp::compGrad() 6 0.000755 0.000755 0.000755 0.21% MultiFab::Add() 36 0.0007138 0.0007138 0.0007138 0.19% FabArray::setDomainBndry() 20 0.0006507 0.0006507 0.0006507 0.18% FabArray::mult() 22 0.0006466 0.0006466 0.0006466 0.18% MLPoisson::prepareForSolve() 6 0.0006237 0.0006237 0.0006237 0.17% MLCellLinOp::prepareForSolve() 6 0.0006187 0.0006187 0.0006187 0.17% Castro::check_for_nan() 10 0.0005897 0.0005897 0.0005897 0.16% MultiFab::contains_nan() 10 0.0005831 0.0005831 0.0005831 0.16% MLMG::computeMLResidual() 6 0.0005578 0.0005578 0.0005578 0.15% Gravity::update_max_rhs() 6 0.0004405 0.0004405 0.0004405 0.12% FabArrayBase::CPC::define() 244 0.0003973 0.0003973 0.0003973 0.11% Amr::InitAmr() 1 0.0003881 0.0003881 0.0003881 0.11% FabArrayBase::getFB() 1766 0.0003358 0.0003358 0.0003358 0.09% Gravity::swapTimeLevels() 5 0.0002235 0.0002235 0.0002235 0.06% MultiFab::Copy() 6 0.0001701 0.0001701 0.0001701 0.05% MLMG::MLResNormInf() 6 0.0001533 0.0001533 0.0001533 0.04% Castro::buildMetrics() 1 0.0001521 0.0001521 0.0001521 0.04% MLLinOp::define() 6 0.0001397 0.0001397 0.0001397 0.04% MultiFab::max() 6 0.0001347 0.0001347 0.0001347 0.04% MLLinOp::defineGrids() 6 0.000119 0.000119 0.000119 0.03% MLMG::MLRhsNormInf() 6 0.0001115 0.0001115 0.0001115 0.03% Castro::create_source_corrector() 5 7.844e-05 7.844e-05 7.844e-05 0.02% Castro::finalize_advance() 5 5.866e-05 5.866e-05 5.866e-05 0.02% FabArrayBase::FB::FB() 26 5.699e-05 5.699e-05 5.699e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 3.057e-05 3.057e-05 3.057e-05 0.01% Castro::initMFs() 1 3.003e-05 3.003e-05 3.003e-05 0.01% Amr::writeSmallPlotFile() 1 2.74e-05 2.74e-05 2.74e-05 0.01% Castro::swap_state_time_levels() 5 2.729e-05 2.729e-05 2.729e-05 0.01% makeSFC 30 2.292e-05 2.292e-05 2.292e-05 0.01% Castro::finalize_do_advance() 5 1.967e-05 1.967e-05 1.967e-05 0.01% DistributionMapping::Distribute() 31 9.519e-06 9.519e-06 9.519e-06 0.00% Amr::initSubcycle() 1 7.929e-06 7.929e-06 7.929e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.768e-06 4.768e-06 4.768e-06 0.00% Gravity::set_mass_offset() 6 2.105e-06 2.105e-06 2.105e-06 0.00% Castro::retry_advance_ctu() 5 2.05e-06 2.05e-06 2.05e-06 0.00% Castro::FluxRegCrseInit 5 1.777e-06 1.777e-06 1.777e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.549e-06 1.549e-06 1.549e-06 0.00% Castro::FluxRegFineAdd() 5 1.208e-06 1.208e-06 1.208e-06 0.00% AmrLevel::AmrLevel() 1 9.37e-07 9.37e-07 9.37e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-13-g17c94cc196d7) finalized