Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-24-g994073b931f1) initialized Starting run at 08:26:46 UTC on 2022-05-19. Successfully read inputs file ... Castro git describe: 22.05-24-g1b7bd38a3 AMReX git describe: 22.05-24-g994073b93 Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.042962072 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.024903755 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.04917006 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051210212 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.063095329 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.064409097 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.062211815 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.040352401 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.050245606 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049708411 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.063662363 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.064231936 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.066862943 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.0408182 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024888531 seconds Ending run at 08:26:47 UTC on 2022-05-19. Run time = 0.811990773 Run time without initialization = 0.69138106 Average number of zones advanced per microsecond: 3.792 Average number of zones advanced per microsecond per rank: 3.792 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.812 ... 0.812 ... 0.812 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1804 0.1804 0.1804 22.22% VisMF::Write(FabArray) 11 0.1672 0.1672 0.1672 20.59% MLCellLinOp::applyBC() 4433 0.08235 0.08235 0.08235 10.14% MLPoisson::Fsmooth() 3280 0.06455 0.06455 0.06455 7.95% StateData::FillBoundary(geom) 328 0.02427 0.02427 0.02427 2.99% MLCGSolver::bicgstab 82 0.02423 0.02423 0.02423 2.98% MultiFab::Dot() 1114 0.02251 0.02251 0.02251 2.77% Castro::normalize_species() 62 0.02082 0.02082 0.02082 2.56% Castro::computeTemp() 63 0.0159 0.0159 0.0159 1.96% MultiFab::LinComb() 1586 0.0146 0.0146 0.0146 1.80% FabArray::setVal() 1144 0.0144 0.0144 0.0144 1.77% FillBoundary_nowait() 4023 0.01425 0.01425 0.01425 1.75% Castro::enforce_min_density() 62 0.01381 0.01381 0.01381 1.70% FabArray::ParallelCopy_nowait() 861 0.01323 0.01323 0.01323 1.63% MLCellLinOp::defineAuxData() 11 0.0119 0.0119 0.0119 1.47% MLPoisson::Fapply() 1142 0.01189 0.01189 0.01189 1.46% StateDataPhysBCFunct::() 41 0.01144 0.01144 0.01144 1.41% Gravity::fill_multipole_BCs() 11 0.008365 0.008365 0.008365 1.03% MLMG::addInterpCorrection() 410 0.007536 0.007536 0.007536 0.93% Castro::estTimeStep() 21 0.00741 0.00741 0.00741 0.91% amrex::average_down 410 0.0069 0.0069 0.0069 0.85% MultiFab::Xpay() 585 0.006696 0.006696 0.006696 0.82% Castro::reset_internal_energy(MultiFab) 63 0.005837 0.005837 0.005837 0.72% Castro::do_advance_ctu() 10 0.005319 0.005319 0.005319 0.66% Amr::checkPoint() 3 0.004271 0.004271 0.004271 0.53% BndryData::define() 11 0.003919 0.003919 0.003919 0.48% Castro::construct_new_gravity_source() 10 0.003166 0.003166 0.003166 0.39% Castro::construct_old_gravity_source() 10 0.002582 0.002582 0.002582 0.32% Amr::writePlotFile() 2 0.002499 0.002499 0.002499 0.31% Gravity::get_new_grav_vector() 11 0.001939 0.001939 0.001939 0.24% MLMG::ResNormInf() 93 0.001873 0.001873 0.001873 0.23% MultiFab::Saxpy() 20 0.001815 0.001815 0.001815 0.22% Gravity::get_old_grav_vector() 10 0.001738 0.001738 0.001738 0.21% Castro::expand_state() 10 0.001729 0.001729 0.001729 0.21% Castro::enforce_speed_limit() 62 0.001725 0.001725 0.001725 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001664 0.001664 0.001664 0.20% MLMG::oneIter() 82 0.00164 0.00164 0.00164 0.20% Castro::reset_internal_energy(Fab) 504 0.001603 0.001603 0.001603 0.20% MLCellLinOp::setLevelBC() 11 0.001554 0.001554 0.001554 0.19% Gravity::actual_solve_with_mlmg() 11 0.001358 0.001358 0.001358 0.17% FabArray::mult() 43 0.00132 0.00132 0.00132 0.16% FabArray::setDomainBndry() 41 0.001316 0.001316 0.001316 0.16% Castro::initData() 1 0.00125 0.00125 0.00125 0.15% MLCellLinOp::prepareForSolve() 11 0.001183 0.001183 0.001183 0.15% MultiFab::contains_nan() 20 0.00118 0.00118 0.00118 0.15% MLCellLinOp::smooth() 1640 0.001147 0.001147 0.001147 0.14% MLMG::prepareForSolve() 11 0.001029 0.001029 0.001029 0.13% Castro::subcycle_advance_ctu() 10 0.0009436 0.0009436 0.0009436 0.12% MLCellLinOp::compGrad() 11 0.0009248 0.0009248 0.0009248 0.11% FabArray::FillBoundary() 4023 0.0007554 0.0007554 0.0007554 0.09% FabArrayBase::getCPC() 1323 0.000746 0.000746 0.000746 0.09% FabArrayBase::CPC::define() 454 0.0006689 0.0006689 0.0006689 0.08% FabArrayBase::getFB() 4023 0.0006436 0.0006436 0.0006436 0.08% Amr::InitAmr() 1 0.0004811 0.0004811 0.0004811 0.06% MLCellLinOp::apply() 1142 0.0004701 0.0004701 0.0004701 0.06% Gravity::solve_for_phi() 10 0.0004337 0.0004337 0.0004337 0.05% Gravity::update_max_rhs() 11 0.0004075 0.0004075 0.0004075 0.05% CGSolver::sxay() 1586 0.0003628 0.0003628 0.0003628 0.04% FillPatchIterator::Initialize 41 0.0003055 0.0003055 0.0003055 0.04% MLCellLinOp::defineBC() 11 0.0002892 0.0002892 0.0002892 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002782 0.0002782 0.0002782 0.03% FabArray::ParallelCopy() 861 0.0002725 0.0002725 0.0002725 0.03% main() 1 0.0002689 0.0002689 0.0002689 0.03% MultiFab::Copy() 11 0.0002651 0.0002651 0.0002651 0.03% MultiFab::max() 11 0.0002548 0.0002548 0.0002548 0.03% MLCellLinOp::correctionResidual() 492 0.0002094 0.0002094 0.0002094 0.03% Castro::construct_new_gravity() 10 0.0002058 0.0002058 0.0002058 0.03% MLMG::MLRhsNormInf() 11 0.000204 0.000204 0.000204 0.03% Amr::coarseTimeStep() 10 0.0002035 0.0002035 0.0002035 0.03% MLMG::mgVcycle() 82 0.0001954 0.0001954 0.0001954 0.02% MLLinOp::defineGrids() 11 0.00019 0.00019 0.00019 0.02% Amr::timeStep() 10 0.000185 0.000185 0.000185 0.02% StateData::checkPoint() 12 0.0001334 0.0001334 0.0001334 0.02% MLMG:computeResOfCorrection() 410 0.000121 0.000121 0.000121 0.01% MLMG::actualBottomSolve() 82 9.592e-05 9.592e-05 9.592e-05 0.01% Castro::advance() 10 9.444e-05 9.444e-05 9.444e-05 0.01% MLMG::mgVcycle_down::0 82 9.119e-05 9.119e-05 9.119e-05 0.01% Castro::Castro() 1 8.674e-05 8.674e-05 8.674e-05 0.01% Castro::initialize_advance() 10 8.546e-05 8.546e-05 8.546e-05 0.01% FabArrayBase::FB::FB() 56 8.258e-05 8.258e-05 8.258e-05 0.01% AmrLevel::checkPoint() 3 7.776e-05 7.776e-05 7.776e-05 0.01% MLMG::mgVcycle_down::1 82 7.718e-05 7.718e-05 7.718e-05 0.01% MLMG::mgVcycle_down::2 82 7.611e-05 7.611e-05 7.611e-05 0.01% MLMG::solve() 11 7.396e-05 7.396e-05 7.396e-05 0.01% MLMG::mgVcycle_down::4 82 7.339e-05 7.339e-05 7.339e-05 0.01% Castro::clean_state() 62 7.272e-05 7.272e-05 7.272e-05 0.01% MLMG::mgVcycle_down::3 82 7.176e-05 7.176e-05 7.176e-05 0.01% Castro::finalize_advance() 10 6.981e-05 6.981e-05 6.981e-05 0.01% Castro::initialize_do_advance() 10 6.304e-05 6.304e-05 6.304e-05 0.01% MLMG::mgVcycle_up::4 82 5.796e-05 5.796e-05 5.796e-05 0.01% MLCellLinOp::solutionResidual() 93 5.261e-05 5.261e-05 5.261e-05 0.01% MLMG::mgVcycle_up::3 82 4.95e-05 4.95e-05 4.95e-05 0.01% MLMG::mgVcycle_up::0 82 4.851e-05 4.851e-05 4.851e-05 0.01% MLMG::mgVcycle_up::1 82 4.746e-05 4.746e-05 4.746e-05 0.01% MLMG::mgVcycle_up::2 82 4.666e-05 4.666e-05 4.666e-05 0.01% Castro::swap_state_time_levels() 10 3.989e-05 3.989e-05 3.989e-05 0.00% StateData::define() 4 3.731e-05 3.731e-05 3.731e-05 0.00% Castro::finalize_do_advance() 10 3.558e-05 3.558e-05 3.558e-05 0.00% Castro::enforce_consistent_e() 1 3.53e-05 3.53e-05 3.53e-05 0.00% MLMG::mgVcycle_bottom 82 3.162e-05 3.162e-05 3.162e-05 0.00% Gravity::actual_multilevel_solve() 1 3.118e-05 3.118e-05 3.118e-05 0.00% MLMG::computeResidual() 82 2.949e-05 2.949e-05 2.949e-05 0.00% Castro::post_timestep() 10 2.94e-05 2.94e-05 2.94e-05 0.00% FillPatchSingleLevel 41 2.864e-05 2.864e-05 2.864e-05 0.00% Castro::initMFs() 1 2.812e-05 2.812e-05 2.812e-05 0.00% makeSFC 55 2.747e-05 2.747e-05 2.747e-05 0.00% Amr::writeSmallPlotFile() 1 2.653e-05 2.653e-05 2.653e-05 0.00% MLLinOp::define() 11 2.588e-05 2.588e-05 2.588e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.524e-05 2.524e-05 2.524e-05 0.00% Amr::defBaseLevel() 1 2.346e-05 2.346e-05 2.346e-05 0.00% Castro::buildMetrics() 1 2.315e-05 2.315e-05 2.315e-05 0.00% MLPoisson::define() 11 2.307e-05 2.307e-05 2.307e-05 0.00% Amr::FinalizeInit() 1 2.184e-05 2.184e-05 2.184e-05 0.00% Castro::construct_old_gravity() 10 2.054e-05 2.054e-05 2.054e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.881e-05 1.881e-05 1.881e-05 0.00% Castro::construct_old_source() 50 1.78e-05 1.78e-05 1.78e-05 0.00% Castro::construct_new_source() 50 1.751e-05 1.751e-05 1.751e-05 0.00% Castro::do_new_sources() 10 1.637e-05 1.637e-05 1.637e-05 0.00% Castro::do_old_sources() 10 1.559e-05 1.559e-05 1.559e-05 0.00% DistributionMapping::Distribute() 56 1.515e-05 1.515e-05 1.515e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.328e-05 1.328e-05 1.328e-05 0.00% Castro::apply_source_to_state() 20 1.248e-05 1.248e-05 1.248e-05 0.00% Castro::check_for_nan() 20 1.207e-05 1.207e-05 1.207e-05 0.00% Gravity::swapTimeLevels() 10 9.222e-06 9.222e-06 9.222e-06 0.00% MLPoisson::prepareForSolve() 11 8.626e-06 8.626e-06 8.626e-06 0.00% Amr::initSubcycle() 1 8.273e-06 8.273e-06 8.273e-06 0.00% MLMG::computeMLResidual() 11 6.733e-06 6.733e-06 6.733e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.685e-06 6.685e-06 6.685e-06 0.00% Castro::computeNewDt() 9 5.652e-06 5.652e-06 5.652e-06 0.00% MLMG::getGradSolution() 11 5.634e-06 5.634e-06 5.634e-06 0.00% Amr::InitializeInit() 1 5.632e-06 5.632e-06 5.632e-06 0.00% MLMG::buildFineMask() 11 5.614e-06 5.614e-06 5.614e-06 0.00% MLMG::MLResNormInf() 11 4.445e-06 4.445e-06 4.445e-06 0.00% Castro::retry_advance_ctu() 10 4.365e-06 4.365e-06 4.365e-06 0.00% AmrLevel::checkPointPost() 3 4.276e-06 4.276e-06 4.276e-06 0.00% Castro::create_source_corrector() 10 4.238e-06 4.238e-06 4.238e-06 0.00% Gravity::set_mass_offset() 11 4.072e-06 4.072e-06 4.072e-06 0.00% Castro::post_init() 1 3.806e-06 3.806e-06 3.806e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.574e-06 3.574e-06 3.574e-06 0.00% Castro::FluxRegCrseInit 10 2.901e-06 2.901e-06 2.901e-06 0.00% Castro::computeInitialDt() 2 2.564e-06 2.564e-06 2.564e-06 0.00% Amr::init() 1 2.322e-06 2.322e-06 2.322e-06 0.00% Castro::FluxRegFineAdd() 10 2.221e-06 2.221e-06 2.221e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.895e-06 1.895e-06 1.895e-06 0.00% AmrLevel::checkPointPre() 3 1.78e-06 1.78e-06 1.78e-06 0.00% Castro::post_regrid() 1 1.353e-06 1.353e-06 1.353e-06 0.00% Amr::initialInit() 1 1.181e-06 1.181e-06 1.181e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.812 0.812 0.812 100.00% Amr::coarseTimeStep() 10 0.6663 0.6663 0.6663 82.05% Amr::timeStep() 10 0.5807 0.5807 0.5807 71.52% Castro::advance() 10 0.57 0.57 0.57 70.19% Castro::subcycle_advance_ctu() 10 0.5562 0.5562 0.5562 68.49% Castro::do_advance_ctu() 10 0.5552 0.5552 0.5552 68.38% Gravity::solve_phi_with_mlmg() 11 0.318 0.318 0.318 39.17% Gravity::actual_solve_with_mlmg() 11 0.3094 0.3094 0.3094 38.11% Castro::construct_new_gravity() 10 0.289 0.289 0.289 35.59% MLMG::solve() 11 0.2864 0.2864 0.2864 35.27% Gravity::solve_for_phi() 10 0.2738 0.2738 0.2738 33.72% MLMG::oneIter() 82 0.2718 0.2718 0.2718 33.47% MLMG::mgVcycle() 82 0.2701 0.2701 0.2701 33.27% Castro::construct_ctu_hydro_source() 10 0.1804 0.1804 0.1804 22.22% VisMF::Write(FabArray) 11 0.1672 0.1672 0.1672 20.59% MLCellLinOp::smooth() 1640 0.1387 0.1387 0.1387 17.09% Amr::checkPoint() 3 0.1243 0.1243 0.1243 15.30% Amr::init() 1 0.12 0.12 0.12 14.78% AmrLevel::checkPoint() 3 0.12 0.12 0.12 14.78% StateData::checkPoint() 12 0.1199 0.1199 0.1199 14.77% MLCellLinOp::applyBC() 4433 0.09809 0.09809 0.09809 12.08% MLMG::mgVcycle_bottom 82 0.08293 0.08293 0.08293 10.21% MLMG::actualBottomSolve() 82 0.0829 0.0829 0.0829 10.21% MLCGSolver::bicgstab 82 0.08206 0.08206 0.08206 10.11% MLPoisson::Fsmooth() 3280 0.06455 0.06455 0.06455 7.95% Castro::clean_state() 62 0.05902 0.05902 0.05902 7.27% Amr::initialInit() 1 0.05203 0.05203 0.05203 6.41% Amr::writePlotFile() 2 0.04992 0.04992 0.04992 6.15% Amr::FinalizeInit() 1 0.04795 0.04795 0.04795 5.91% Castro::post_init() 1 0.04661 0.04661 0.04661 5.74% Gravity::multilevel_solve_for_new_phi() 1 0.04472 0.04472 0.04472 5.51% Gravity::actual_multilevel_solve() 1 0.0447 0.0447 0.0447 5.50% FillPatchIterator::Initialize 41 0.04132 0.04132 0.04132 5.09% FillPatchSingleLevel 41 0.0397 0.0397 0.0397 4.89% MLCellLinOp::apply() 1142 0.03692 0.03692 0.03692 4.55% MLMG::mgVcycle_down::0 82 0.0361 0.0361 0.0361 4.45% StateDataPhysBCFunct::() 41 0.03571 0.03571 0.03571 4.40% MLMG::mgVcycle_up::0 82 0.03094 0.03094 0.03094 3.81% StateData::FillBoundary(geom) 328 0.02427 0.02427 0.02427 2.99% Castro::computeTemp() 63 0.02334 0.02334 0.02334 2.87% MultiFab::Dot() 1114 0.02251 0.02251 0.02251 2.77% MLCellLinOp::correctionResidual() 492 0.02167 0.02167 0.02167 2.67% Castro::initialize_do_advance() 10 0.02151 0.02151 0.02151 2.65% Castro::normalize_species() 62 0.02082 0.02082 0.02082 2.56% MLMG:computeResOfCorrection() 410 0.01869 0.01869 0.01869 2.30% MLPoisson::define() 11 0.01868 0.01868 0.01868 2.30% MLMG::mgVcycle_down::1 82 0.01803 0.01803 0.01803 2.22% MLMG::mgVcycle_down::2 82 0.01754 0.01754 0.01754 2.16% Gravity::get_new_grav_vector() 11 0.01677 0.01677 0.01677 2.07% MLMG::mgVcycle_down::3 82 0.01662 0.01662 0.01662 2.05% MLMG::mgVcycle_down::4 82 0.01578 0.01578 0.01578 1.94% FabArray::FillBoundary() 4023 0.01573 0.01573 0.01573 1.94% FillBoundary_nowait() 4023 0.01498 0.01498 0.01498 1.84% CGSolver::sxay() 1586 0.01496 0.01496 0.01496 1.84% MultiFab::LinComb() 1586 0.0146 0.0146 0.0146 1.80% Castro::construct_old_gravity() 10 0.01458 0.01458 0.01458 1.80% Gravity::get_old_grav_vector() 10 0.01456 0.01456 0.01456 1.79% FabArray::setVal() 1144 0.0144 0.0144 0.0144 1.77% FabArray::ParallelCopy() 861 0.0143 0.0143 0.0143 1.76% FabArray::ParallelCopy_nowait() 861 0.01403 0.01403 0.01403 1.73% Castro::enforce_min_density() 62 0.01381 0.01381 0.01381 1.70% Castro::initialize_advance() 10 0.0136 0.0136 0.0136 1.68% MLMG::mgVcycle_up::2 82 0.01348 0.01348 0.01348 1.66% Castro::do_new_sources() 10 0.01341 0.01341 0.01341 1.65% MLCGSolver::ParallelAllReduce 1514 0.01336 0.01336 0.01336 1.65% MLMG::mgVcycle_up::1 82 0.01327 0.01327 0.01327 1.63% MLCellLinOp::defineAuxData() 11 0.01323 0.01323 0.01323 1.63% Castro::do_old_sources() 10 0.01277 0.01277 0.01277 1.57% MLMG::mgVcycle_up::3 82 0.01275 0.01275 0.01275 1.57% MLMG::addInterpCorrection() 410 0.0127 0.0127 0.0127 1.56% MLMG::mgVcycle_up::4 82 0.0125 0.0125 0.0125 1.54% amrex::average_down 410 0.01208 0.01208 0.01208 1.49% MLPoisson::Fapply() 1142 0.01189 0.01189 0.01189 1.46% Castro::expand_state() 10 0.01151 0.01151 0.01151 1.42% Castro::post_timestep() 10 0.0106 0.0106 0.0106 1.31% Gravity::fill_multipole_BCs() 11 0.008365 0.008365 0.008365 1.03% Castro::reset_internal_energy(MultiFab) 63 0.00744 0.00744 0.00744 0.92% Castro::estTimeStep() 21 0.00741 0.00741 0.00741 0.91% MLCellLinOp::solutionResidual() 93 0.007268 0.007268 0.007268 0.90% MultiFab::Xpay() 585 0.006696 0.006696 0.006696 0.82% MLMG::computeResidual() 82 0.006266 0.006266 0.006266 0.77% MLMG::prepareForSolve() 11 0.005189 0.005189 0.005189 0.64% MLCellLinOp::defineBC() 11 0.005155 0.005155 0.005155 0.63% BndryData::define() 11 0.004865 0.004865 0.004865 0.60% Amr::InitializeInit() 1 0.004075 0.004075 0.004075 0.50% Amr::defBaseLevel() 1 0.004069 0.004069 0.004069 0.50% Castro::initData() 1 0.003563 0.003563 0.003563 0.44% Castro::computeNewDt() 9 0.003562 0.003562 0.003562 0.44% Castro::construct_new_source() 50 0.003184 0.003184 0.003184 0.39% Castro::construct_new_gravity_source() 10 0.003166 0.003166 0.003166 0.39% Castro::construct_old_source() 50 0.0026 0.0026 0.0026 0.32% Castro::construct_old_gravity_source() 10 0.002582 0.002582 0.002582 0.32% MLMG::ResNormInf() 93 0.001873 0.001873 0.001873 0.23% Castro::apply_source_to_state() 20 0.001827 0.001827 0.001827 0.23% MultiFab::Saxpy() 20 0.001815 0.001815 0.001815 0.22% Castro::enforce_speed_limit() 62 0.001725 0.001725 0.001725 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001664 0.001664 0.001664 0.20% Castro::reset_internal_energy(Fab) 504 0.001603 0.001603 0.001603 0.20% MLCellLinOp::setLevelBC() 11 0.001554 0.001554 0.001554 0.19% FabArrayBase::getCPC() 1323 0.001415 0.001415 0.001415 0.17% MLMG::getGradSolution() 11 0.001412 0.001412 0.001412 0.17% MLCellLinOp::compGrad() 11 0.001406 0.001406 0.001406 0.17% FabArray::mult() 43 0.00132 0.00132 0.00132 0.16% FabArray::setDomainBndry() 41 0.001316 0.001316 0.001316 0.16% Castro::check_for_nan() 20 0.001192 0.001192 0.001192 0.15% MLPoisson::prepareForSolve() 11 0.001191 0.001191 0.001191 0.15% MLCellLinOp::prepareForSolve() 11 0.001183 0.001183 0.001183 0.15% MultiFab::contains_nan() 20 0.00118 0.00118 0.00118 0.15% Castro::post_regrid() 1 0.001107 0.001107 0.001107 0.14% MLMG::computeMLResidual() 11 0.001038 0.001038 0.001038 0.13% Gravity::update_max_rhs() 11 0.0008124 0.0008124 0.0008124 0.10% FabArrayBase::getFB() 4023 0.0007261 0.0007261 0.0007261 0.09% Castro::computeInitialDt() 2 0.0007012 0.0007012 0.0007012 0.09% FabArrayBase::CPC::define() 454 0.0006689 0.0006689 0.0006689 0.08% Amr::InitAmr() 1 0.0004893 0.0004893 0.0004893 0.06% Gravity::swapTimeLevels() 10 0.0004435 0.0004435 0.0004435 0.05% Castro::Castro() 1 0.0004348 0.0004348 0.0004348 0.05% MLLinOp::define() 11 0.0002726 0.0002726 0.0002726 0.03% MultiFab::Copy() 11 0.0002651 0.0002651 0.0002651 0.03% MLMG::MLResNormInf() 11 0.0002612 0.0002612 0.0002612 0.03% MultiFab::max() 11 0.0002548 0.0002548 0.0002548 0.03% MLLinOp::defineGrids() 11 0.0002467 0.0002467 0.0002467 0.03% MLMG::MLRhsNormInf() 11 0.000204 0.000204 0.000204 0.03% Castro::buildMetrics() 1 0.0001564 0.0001564 0.0001564 0.02% FabArrayBase::FB::FB() 56 8.258e-05 8.258e-05 8.258e-05 0.01% Castro::finalize_advance() 10 7.493e-05 7.493e-05 7.493e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.489e-05 5.489e-05 5.489e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.4e-05 4.4e-05 4.4e-05 0.01% makeSFC 55 4.161e-05 4.161e-05 4.161e-05 0.01% Castro::swap_state_time_levels() 10 3.989e-05 3.989e-05 3.989e-05 0.00% StateData::define() 4 3.731e-05 3.731e-05 3.731e-05 0.00% Castro::finalize_do_advance() 10 3.558e-05 3.558e-05 3.558e-05 0.00% Castro::enforce_consistent_e() 1 3.53e-05 3.53e-05 3.53e-05 0.00% Castro::initMFs() 1 2.812e-05 2.812e-05 2.812e-05 0.00% Amr::writeSmallPlotFile() 1 2.653e-05 2.653e-05 2.653e-05 0.00% DistributionMapping::Distribute() 56 1.515e-05 1.515e-05 1.515e-05 0.00% Amr::initSubcycle() 1 8.273e-06 8.273e-06 8.273e-06 0.00% MLMG::buildFineMask() 11 5.614e-06 5.614e-06 5.614e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.588e-06 4.588e-06 4.588e-06 0.00% Castro::retry_advance_ctu() 10 4.365e-06 4.365e-06 4.365e-06 0.00% AmrLevel::checkPointPost() 3 4.276e-06 4.276e-06 4.276e-06 0.00% Castro::create_source_corrector() 10 4.238e-06 4.238e-06 4.238e-06 0.00% Gravity::set_mass_offset() 11 4.072e-06 4.072e-06 4.072e-06 0.00% Castro::FluxRegCrseInit 10 2.901e-06 2.901e-06 2.901e-06 0.00% Castro::FluxRegFineAdd() 10 2.221e-06 2.221e-06 2.221e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.895e-06 1.895e-06 1.895e-06 0.00% AmrLevel::checkPointPre() 3 1.78e-06 1.78e-06 1.78e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-24-g994073b931f1) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-24-g994073b931f1) initialized Starting run at 08:26:47 UTC on 2022-05-19. Successfully read inputs file ... Castro git describe: 22.05-24-g1b7bd38a3 AMReX git describe: 22.05-24-g994073b93 Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.451003822 Restart time = 0.045868474 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052218203 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050972997 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.063910341 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.062897138 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.064161076 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026026714 seconds Ending run at 08:26:48 UTC on 2022-05-19. Run time = 0.366998867 Run time without initialization = 0.320526455 Average number of zones advanced per microsecond: 4.089 Average number of zones advanced per microsecond per rank: 4.089 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.367 ... 0.367 ... 0.367 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0915 0.0915 0.0915 24.93% VisMF::Read() 3 0.03844 0.03844 0.03844 10.47% MLCellLinOp::applyBC() 1946 0.03462 0.03462 0.03462 9.43% MLPoisson::Fsmooth() 1440 0.02705 0.02705 0.02705 7.37% VisMF::Write(FabArray) 1 0.02466 0.02466 0.02466 6.72% StateData::FillBoundary(geom) 160 0.01174 0.01174 0.01174 3.20% Castro::normalize_species() 30 0.01044 0.01044 0.01044 2.84% MLCGSolver::bicgstab 36 0.01021 0.01021 0.01021 2.78% MultiFab::Dot() 484 0.00947 0.00947 0.00947 2.58% Castro::computeTemp() 30 0.008893 0.008893 0.008893 2.42% Castro::enforce_min_density() 30 0.007864 0.007864 0.007864 2.14% FabArray::setVal() 537 0.006677 0.006677 0.006677 1.82% MLCellLinOp::defineAuxData() 6 0.006248 0.006248 0.006248 1.70% FillBoundary_nowait() 1766 0.006185 0.006185 0.006185 1.69% MultiFab::LinComb() 690 0.00607 0.00607 0.00607 1.65% FabArray::ParallelCopy_nowait() 380 0.005855 0.005855 0.005855 1.60% StateDataPhysBCFunct::() 20 0.005451 0.005451 0.005451 1.49% MLPoisson::Fapply() 500 0.005068 0.005068 0.005068 1.38% Gravity::fill_multipole_BCs() 6 0.004772 0.004772 0.004772 1.30% Castro::estTimeStep() 10 0.00394 0.00394 0.00394 1.07% MLMG::addInterpCorrection() 180 0.003195 0.003195 0.003195 0.87% Amr::restart() 1 0.003112 0.003112 0.003112 0.85% amrex::average_down 180 0.002921 0.002921 0.002921 0.80% MultiFab::Xpay() 258 0.002846 0.002846 0.002846 0.78% Castro::do_advance_ctu() 5 0.002484 0.002484 0.002484 0.68% BndryData::define() 6 0.00213 0.00213 0.00213 0.58% Castro::reset_internal_energy(MultiFab) 30 0.002129 0.002129 0.002129 0.58% Castro::construct_new_gravity_source() 5 0.001624 0.001624 0.001624 0.44% Amr::writePlotFile() 1 0.001455 0.001455 0.001455 0.40% Castro::construct_old_gravity_source() 5 0.001372 0.001372 0.001372 0.37% MultiFab::Saxpy() 10 0.0009147 0.0009147 0.0009147 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008896 0.0008896 0.0008896 0.24% Castro::expand_state() 5 0.0008827 0.0008827 0.0008827 0.24% Gravity::get_old_grav_vector() 5 0.0008766 0.0008766 0.0008766 0.24% Gravity::get_new_grav_vector() 5 0.0008607 0.0008607 0.0008607 0.23% MLMG::ResNormInf() 42 0.0008263 0.0008263 0.0008263 0.23% MLCellLinOp::setLevelBC() 6 0.0008198 0.0008198 0.0008198 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007269 0.0007269 0.0007269 0.20% MLMG::oneIter() 36 0.0007123 0.0007123 0.0007123 0.19% Castro::reset_internal_energy(Fab) 240 0.0007014 0.0007014 0.0007014 0.19% FabArray::setDomainBndry() 20 0.0006541 0.0006541 0.0006541 0.18% FabArray::mult() 22 0.0006523 0.0006523 0.0006523 0.18% MLCellLinOp::prepareForSolve() 6 0.0006325 0.0006325 0.0006325 0.17% MultiFab::contains_nan() 10 0.0005863 0.0005863 0.0005863 0.16% MLMG::prepareForSolve() 6 0.0005555 0.0005555 0.0005555 0.15% MLCellLinOp::smooth() 720 0.0005272 0.0005272 0.0005272 0.14% MLCellLinOp::compGrad() 6 0.0004898 0.0004898 0.0004898 0.13% Amr::InitAmr() 1 0.0004133 0.0004133 0.0004133 0.11% FabArrayBase::CPC::define() 244 0.0003985 0.0003985 0.0003985 0.11% FabArray::FillBoundary() 1766 0.0003596 0.0003596 0.0003596 0.10% FabArrayBase::getCPC() 632 0.00035 0.00035 0.00035 0.10% Castro::enforce_speed_limit() 30 0.0003168 0.0003168 0.0003168 0.09% main() 1 0.0002687 0.0002687 0.0002687 0.07% FabArrayBase::getFB() 1766 0.0002605 0.0002605 0.0002605 0.07% Gravity::solve_for_phi() 5 0.0002309 0.0002309 0.0002309 0.06% Gravity::update_max_rhs() 6 0.0002229 0.0002229 0.0002229 0.06% MLCellLinOp::apply() 500 0.0002155 0.0002155 0.0002155 0.06% CGSolver::sxay() 690 0.0001794 0.0001794 0.0001794 0.05% MLCellLinOp::defineBC() 6 0.0001519 0.0001519 0.0001519 0.04% Castro::subcycle_advance_ctu() 5 0.0001511 0.0001511 0.0001511 0.04% MultiFab::Copy() 6 0.0001372 0.0001372 0.0001372 0.04% MultiFab::max() 6 0.000134 0.000134 0.000134 0.04% FillPatchIterator::Initialize 20 0.0001309 0.0001309 0.0001309 0.04% FabArray::ParallelCopy() 380 0.0001264 0.0001264 0.0001264 0.03% MLCGSolver::ParallelAllReduce 659 0.0001145 0.0001145 0.0001145 0.03% MLMG::MLRhsNormInf() 6 0.000109 0.000109 0.000109 0.03% Amr::coarseTimeStep() 5 0.0001053 0.0001053 0.0001053 0.03% MLLinOp::defineGrids() 6 0.0001035 0.0001035 0.0001035 0.03% Castro::construct_new_gravity() 5 0.0001032 0.0001032 0.0001032 0.03% MLMG::mgVcycle() 36 9.677e-05 9.677e-05 9.677e-05 0.03% Amr::timeStep() 5 9.516e-05 9.516e-05 9.516e-05 0.03% MLCellLinOp::correctionResidual() 216 9.352e-05 9.352e-05 9.352e-05 0.03% AmrLevel::restart() 1 8.186e-05 8.186e-05 8.186e-05 0.02% Castro::advance() 5 8.069e-05 8.069e-05 8.069e-05 0.02% StateData::restartDoit() 4 6.581e-05 6.581e-05 6.581e-05 0.02% FabArrayBase::FB::FB() 26 5.721e-05 5.721e-05 5.721e-05 0.02% MLMG:computeResOfCorrection() 180 5.544e-05 5.544e-05 5.544e-05 0.02% Castro::initialize_do_advance() 5 4.509e-05 4.509e-05 4.509e-05 0.01% MLMG::actualBottomSolve() 36 4.28e-05 4.28e-05 4.28e-05 0.01% Castro::create_source_corrector() 5 4.217e-05 4.217e-05 4.217e-05 0.01% Castro::initialize_advance() 5 4.033e-05 4.033e-05 4.033e-05 0.01% MLMG::mgVcycle_down::0 36 3.846e-05 3.846e-05 3.846e-05 0.01% MLMG::solve() 6 3.608e-05 3.608e-05 3.608e-05 0.01% Castro::clean_state() 30 3.571e-05 3.571e-05 3.571e-05 0.01% MLMG::mgVcycle_down::1 36 3.375e-05 3.375e-05 3.375e-05 0.01% MLMG::mgVcycle_down::3 36 3.316e-05 3.316e-05 3.316e-05 0.01% Castro::construct_old_source() 25 3.241e-05 3.241e-05 3.241e-05 0.01% Castro::buildMetrics() 1 3.222e-05 3.222e-05 3.222e-05 0.01% MLMG::mgVcycle_down::2 36 3.213e-05 3.213e-05 3.213e-05 0.01% Gravity::actual_multilevel_solve() 1 3.124e-05 3.124e-05 3.124e-05 0.01% Castro::post_restart() 1 3.11e-05 3.11e-05 3.11e-05 0.01% MLMG::mgVcycle_down::4 36 3.086e-05 3.086e-05 3.086e-05 0.01% Castro::swap_state_time_levels() 5 2.839e-05 2.839e-05 2.839e-05 0.01% MLMG::mgVcycle_up::4 36 2.781e-05 2.781e-05 2.781e-05 0.01% Castro::initMFs() 1 2.73e-05 2.73e-05 2.73e-05 0.01% Amr::writeSmallPlotFile() 1 2.587e-05 2.587e-05 2.587e-05 0.01% Castro::finalize_advance() 5 2.563e-05 2.563e-05 2.563e-05 0.01% MLMG::mgVcycle_up::0 36 2.527e-05 2.527e-05 2.527e-05 0.01% MLMG::mgVcycle_up::3 36 2.425e-05 2.425e-05 2.425e-05 0.01% MLMG::mgVcycle_up::2 36 2.405e-05 2.405e-05 2.405e-05 0.01% MLCellLinOp::solutionResidual() 42 2.389e-05 2.389e-05 2.389e-05 0.01% MLMG::mgVcycle_up::1 36 2.203e-05 2.203e-05 2.203e-05 0.01% MLLinOp::define() 6 2.097e-05 2.097e-05 2.097e-05 0.01% Castro::finalize_do_advance() 5 1.85e-05 1.85e-05 1.85e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.742e-05 1.742e-05 1.742e-05 0.00% MLMG::mgVcycle_bottom 36 1.595e-05 1.595e-05 1.595e-05 0.00% MLPoisson::define() 6 1.509e-05 1.509e-05 1.509e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.411e-05 1.411e-05 1.411e-05 0.00% MLMG::computeResidual() 36 1.403e-05 1.403e-05 1.403e-05 0.00% makeSFC 30 1.37e-05 1.37e-05 1.37e-05 0.00% FillPatchSingleLevel 20 1.336e-05 1.336e-05 1.336e-05 0.00% Castro::construct_new_source() 25 1.008e-05 1.008e-05 1.008e-05 0.00% Amr::initSubcycle() 1 8.877e-06 8.877e-06 8.877e-06 0.00% DistributionMapping::Distribute() 31 8.563e-06 8.563e-06 8.563e-06 0.00% Castro::do_new_sources() 5 8.397e-06 8.397e-06 8.397e-06 0.00% Castro::do_old_sources() 5 8.033e-06 8.033e-06 8.033e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.356e-06 7.356e-06 7.356e-06 0.00% Castro::apply_source_to_state() 10 6.657e-06 6.657e-06 6.657e-06 0.00% Castro::check_for_nan() 10 5.952e-06 5.952e-06 5.952e-06 0.00% Castro::construct_old_gravity() 5 5.208e-06 5.208e-06 5.208e-06 0.00% Castro::post_timestep() 5 4.657e-06 4.657e-06 4.657e-06 0.00% Gravity::swapTimeLevels() 5 4.488e-06 4.488e-06 4.488e-06 0.00% MLPoisson::prepareForSolve() 6 4.336e-06 4.336e-06 4.336e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.08e-06 4.08e-06 4.08e-06 0.00% Castro::computeNewDt() 5 3.361e-06 3.361e-06 3.361e-06 0.00% MLMG::computeMLResidual() 6 3.011e-06 3.011e-06 3.011e-06 0.00% MLMG::buildFineMask() 6 2.957e-06 2.957e-06 2.957e-06 0.00% MLMG::getGradSolution() 6 2.901e-06 2.901e-06 2.901e-06 0.00% MLMG::MLResNormInf() 6 2.278e-06 2.278e-06 2.278e-06 0.00% Gravity::set_mass_offset() 6 2.109e-06 2.109e-06 2.109e-06 0.00% Castro::retry_advance_ctu() 5 2.046e-06 2.046e-06 2.046e-06 0.00% Castro::FluxRegCrseInit 5 1.757e-06 1.757e-06 1.757e-06 0.00% Castro::FluxRegFineAdd() 5 1.153e-06 1.153e-06 1.153e-06 0.00% Amr::init() 1 1.094e-06 1.094e-06 1.094e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.048e-06 1.048e-06 1.048e-06 0.00% AmrLevel::AmrLevel() 1 8.46e-07 8.46e-07 8.46e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.367 0.367 0.367 100.00% Amr::coarseTimeStep() 5 0.2943 0.2943 0.2943 80.17% Amr::timeStep() 5 0.2919 0.2919 0.2919 79.53% Castro::advance() 5 0.2865 0.2865 0.2865 78.05% Castro::subcycle_advance_ctu() 5 0.2788 0.2788 0.2788 75.95% Castro::do_advance_ctu() 5 0.2786 0.2786 0.2786 75.91% Castro::construct_new_gravity() 5 0.1428 0.1428 0.1428 38.90% Gravity::solve_phi_with_mlmg() 6 0.1386 0.1386 0.1386 37.77% Gravity::solve_for_phi() 5 0.1351 0.1351 0.1351 36.80% Gravity::actual_solve_with_mlmg() 6 0.1337 0.1337 0.1337 36.44% MLMG::solve() 6 0.1215 0.1215 0.1215 33.09% MLMG::oneIter() 36 0.1145 0.1145 0.1145 31.19% MLMG::mgVcycle() 36 0.1138 0.1138 0.1138 31.00% Castro::construct_ctu_hydro_source() 5 0.09152 0.09152 0.09152 24.93% MLCellLinOp::smooth() 720 0.05832 0.05832 0.05832 15.89% Amr::init() 1 0.04592 0.04592 0.04592 12.51% Amr::restart() 1 0.04592 0.04592 0.04592 12.51% MLCellLinOp::applyBC() 1946 0.04149 0.04149 0.04149 11.30% AmrLevel::restart() 1 0.03865 0.03865 0.03865 10.53% StateData::restartDoit() 4 0.03856 0.03856 0.03856 10.51% VisMF::Read() 3 0.03844 0.03844 0.03844 10.47% MLMG::mgVcycle_bottom 36 0.03489 0.03489 0.03489 9.51% MLMG::actualBottomSolve() 36 0.03488 0.03488 0.03488 9.50% MLCGSolver::bicgstab 36 0.03452 0.03452 0.03452 9.41% Castro::clean_state() 30 0.03038 0.03038 0.03038 8.28% MLPoisson::Fsmooth() 1440 0.02705 0.02705 0.02705 7.37% Amr::writePlotFile() 1 0.02611 0.02611 0.02611 7.11% VisMF::Write(FabArray) 1 0.02466 0.02466 0.02466 6.72% FillPatchIterator::Initialize 20 0.01996 0.01996 0.01996 5.44% FillPatchSingleLevel 20 0.01917 0.01917 0.01917 5.22% StateDataPhysBCFunct::() 20 0.01719 0.01719 0.01719 4.68% MLCellLinOp::apply() 500 0.01576 0.01576 0.01576 4.29% MLMG::mgVcycle_down::0 36 0.01532 0.01532 0.01532 4.18% MLMG::mgVcycle_up::0 36 0.01309 0.01309 0.01309 3.57% Castro::initialize_do_advance() 5 0.01193 0.01193 0.01193 3.25% StateData::FillBoundary(geom) 160 0.01174 0.01174 0.01174 3.20% Castro::computeTemp() 30 0.01172 0.01172 0.01172 3.19% Castro::normalize_species() 30 0.01044 0.01044 0.01044 2.84% MLPoisson::define() 6 0.009973 0.009973 0.009973 2.72% MultiFab::Dot() 484 0.00947 0.00947 0.00947 2.58% MLCellLinOp::correctionResidual() 216 0.00918 0.00918 0.00918 2.50% MLMG:computeResOfCorrection() 180 0.007917 0.007917 0.007917 2.16% Castro::enforce_min_density() 30 0.007864 0.007864 0.007864 2.14% MLMG::mgVcycle_down::1 36 0.007616 0.007616 0.007616 2.08% Gravity::get_new_grav_vector() 5 0.007603 0.007603 0.007603 2.07% Castro::initialize_advance() 5 0.00758 0.00758 0.00758 2.07% MLMG::mgVcycle_down::2 36 0.007341 0.007341 0.007341 2.00% Castro::construct_old_gravity() 5 0.007266 0.007266 0.007266 1.98% Gravity::get_old_grav_vector() 5 0.00726 0.00726 0.00726 1.98% Castro::do_new_sources() 5 0.007249 0.007249 0.007249 1.98% MLCellLinOp::defineAuxData() 6 0.006981 0.006981 0.006981 1.90% MLMG::mgVcycle_down::3 36 0.006969 0.006969 0.006969 1.90% FabArray::FillBoundary() 1766 0.006862 0.006862 0.006862 1.87% MLMG::mgVcycle_down::4 36 0.006677 0.006677 0.006677 1.82% FabArray::setVal() 537 0.006677 0.006677 0.006677 1.82% Castro::do_old_sources() 5 0.006596 0.006596 0.006596 1.80% FillBoundary_nowait() 1766 0.006502 0.006502 0.006502 1.77% FabArray::ParallelCopy() 380 0.006346 0.006346 0.006346 1.73% CGSolver::sxay() 690 0.00625 0.00625 0.00625 1.70% FabArray::ParallelCopy_nowait() 380 0.006219 0.006219 0.006219 1.69% MultiFab::LinComb() 690 0.00607 0.00607 0.00607 1.65% Castro::expand_state() 5 0.00583 0.00583 0.00583 1.59% MLCGSolver::ParallelAllReduce 659 0.005649 0.005649 0.005649 1.54% MLMG::mgVcycle_up::2 36 0.00562 0.00562 0.00562 1.53% MLMG::mgVcycle_up::1 36 0.00556 0.00556 0.00556 1.51% MLMG::addInterpCorrection() 180 0.005382 0.005382 0.005382 1.47% Castro::post_timestep() 5 0.00535 0.00535 0.00535 1.46% MLMG::mgVcycle_up::3 36 0.005331 0.005331 0.005331 1.45% MLMG::mgVcycle_up::4 36 0.005257 0.005257 0.005257 1.43% amrex::average_down 180 0.005115 0.005115 0.005115 1.39% MLPoisson::Fapply() 500 0.005068 0.005068 0.005068 1.38% Gravity::fill_multipole_BCs() 6 0.004772 0.004772 0.004772 1.30% Castro::post_restart() 1 0.003979 0.003979 0.003979 1.08% Castro::estTimeStep() 10 0.00394 0.00394 0.00394 1.07% Gravity::multilevel_solve_for_new_phi() 1 0.003857 0.003857 0.003857 1.05% Gravity::actual_multilevel_solve() 1 0.00384 0.00384 0.00384 1.05% MLCellLinOp::solutionResidual() 42 0.003247 0.003247 0.003247 0.88% MultiFab::Xpay() 258 0.002846 0.002846 0.002846 0.78% Castro::reset_internal_energy(MultiFab) 30 0.00283 0.00283 0.00283 0.77% MLCellLinOp::defineBC() 6 0.002823 0.002823 0.002823 0.77% MLMG::prepareForSolve() 6 0.002729 0.002729 0.002729 0.74% MLMG::computeResidual() 36 0.002672 0.002672 0.002672 0.73% BndryData::define() 6 0.002671 0.002671 0.002671 0.73% Castro::computeNewDt() 5 0.002252 0.002252 0.002252 0.61% Castro::construct_new_source() 25 0.001634 0.001634 0.001634 0.45% Castro::construct_new_gravity_source() 5 0.001624 0.001624 0.001624 0.44% Castro::construct_old_source() 25 0.001404 0.001404 0.001404 0.38% Castro::construct_old_gravity_source() 5 0.001372 0.001372 0.001372 0.37% Castro::apply_source_to_state() 10 0.0009213 0.0009213 0.0009213 0.25% MultiFab::Saxpy() 10 0.0009147 0.0009147 0.0009147 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008896 0.0008896 0.0008896 0.24% MLMG::ResNormInf() 42 0.0008263 0.0008263 0.0008263 0.23% MLCellLinOp::setLevelBC() 6 0.0008198 0.0008198 0.0008198 0.22% MLMG::getGradSolution() 6 0.0007589 0.0007589 0.0007589 0.21% MLCellLinOp::compGrad() 6 0.000756 0.000756 0.000756 0.21% FabArrayBase::getCPC() 632 0.0007484 0.0007484 0.0007484 0.20% Castro::reset_internal_energy(Fab) 240 0.0007014 0.0007014 0.0007014 0.19% FabArray::setDomainBndry() 20 0.0006541 0.0006541 0.0006541 0.18% FabArray::mult() 22 0.0006523 0.0006523 0.0006523 0.18% MLPoisson::prepareForSolve() 6 0.0006368 0.0006368 0.0006368 0.17% MLCellLinOp::prepareForSolve() 6 0.0006325 0.0006325 0.0006325 0.17% Castro::check_for_nan() 10 0.0005922 0.0005922 0.0005922 0.16% MLMG::computeMLResidual() 6 0.0005922 0.0005922 0.0005922 0.16% MultiFab::contains_nan() 10 0.0005863 0.0005863 0.0005863 0.16% Gravity::update_max_rhs() 6 0.0004343 0.0004343 0.0004343 0.12% Amr::InitAmr() 1 0.0004222 0.0004222 0.0004222 0.12% FabArrayBase::CPC::define() 244 0.0003985 0.0003985 0.0003985 0.11% FabArrayBase::getFB() 1766 0.0003177 0.0003177 0.0003177 0.09% Castro::enforce_speed_limit() 30 0.0003168 0.0003168 0.0003168 0.09% Gravity::swapTimeLevels() 5 0.0002287 0.0002287 0.0002287 0.06% MLLinOp::define() 6 0.0001538 0.0001538 0.0001538 0.04% Castro::buildMetrics() 1 0.0001524 0.0001524 0.0001524 0.04% MultiFab::Copy() 6 0.0001372 0.0001372 0.0001372 0.04% MLMG::MLResNormInf() 6 0.0001352 0.0001352 0.0001352 0.04% MultiFab::max() 6 0.000134 0.000134 0.000134 0.04% MLLinOp::defineGrids() 6 0.0001329 0.0001329 0.0001329 0.04% MLMG::MLRhsNormInf() 6 0.000109 0.000109 0.000109 0.03% FabArrayBase::FB::FB() 26 5.721e-05 5.721e-05 5.721e-05 0.02% Castro::create_source_corrector() 5 4.217e-05 4.217e-05 4.217e-05 0.01% Castro::finalize_advance() 5 2.854e-05 2.854e-05 2.854e-05 0.01% Castro::swap_state_time_levels() 5 2.839e-05 2.839e-05 2.839e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.833e-05 2.833e-05 2.833e-05 0.01% Castro::initMFs() 1 2.73e-05 2.73e-05 2.73e-05 0.01% Amr::writeSmallPlotFile() 1 2.587e-05 2.587e-05 2.587e-05 0.01% makeSFC 30 2.097e-05 2.097e-05 2.097e-05 0.01% Castro::finalize_do_advance() 5 1.85e-05 1.85e-05 1.85e-05 0.01% Amr::initSubcycle() 1 8.877e-06 8.877e-06 8.877e-06 0.00% DistributionMapping::Distribute() 31 8.563e-06 8.563e-06 8.563e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.367e-06 5.367e-06 5.367e-06 0.00% MLMG::buildFineMask() 6 2.957e-06 2.957e-06 2.957e-06 0.00% Gravity::set_mass_offset() 6 2.109e-06 2.109e-06 2.109e-06 0.00% Castro::retry_advance_ctu() 5 2.046e-06 2.046e-06 2.046e-06 0.00% Castro::FluxRegCrseInit 5 1.757e-06 1.757e-06 1.757e-06 0.00% Castro::FluxRegFineAdd() 5 1.153e-06 1.153e-06 1.153e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.048e-06 1.048e-06 1.048e-06 0.00% AmrLevel::AmrLevel() 1 8.46e-07 8.46e-07 8.46e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-24-g994073b931f1) finalized