Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-4-gef52970b80e3) initialized Starting run at 08:24:29 UTC on 2022-05-04. Successfully read inputs file ... Castro git describe: 22.05-8-gf5a7de0a1 AMReX git describe: 22.05-4-gef52970b8 Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.038419622 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022443847 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.045972367 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049138694 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.066022556 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.057859845 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.048304186 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.035429189 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.050921252 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.057692318 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057976357 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.056767364 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.0517762 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.035422864 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.022360624 seconds Ending run at 08:24:30 UTC on 2022-05-04. Run time = 0.747026307 Run time without initialization = 0.636142508 Average number of zones advanced per microsecond: 4.121 Average number of zones advanced per microsecond per rank: 4.121 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.7471 ... 0.7471 ... 0.7471 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1634 0.1634 0.1634 21.88% VisMF::Write(FabArray) 11 0.1476 0.1476 0.1476 19.76% MLCellLinOp::applyBC() 4433 0.07253 0.07253 0.07253 9.71% MLPoisson::Fsmooth() 3280 0.05959 0.05959 0.05959 7.98% StateData::FillBoundary(geom) 328 0.02334 0.02334 0.02334 3.12% MLCGSolver::bicgstab 82 0.02133 0.02133 0.02133 2.86% MultiFab::Dot() 1114 0.02068 0.02068 0.02068 2.77% Castro::normalize_species() 62 0.02009 0.02009 0.02009 2.69% StateDataPhysBCFunct::() 41 0.0184 0.0184 0.0184 2.46% Castro::computeTemp() 63 0.01445 0.01445 0.01445 1.93% FillBoundary_nowait() 4023 0.01438 0.01438 0.01438 1.92% FabArray::setVal() 1144 0.01306 0.01306 0.01306 1.75% FabArray::ParallelCopy_nowait() 861 0.01297 0.01297 0.01297 1.74% MultiFab::LinComb() 1586 0.01228 0.01228 0.01228 1.64% Castro::enforce_min_density() 62 0.01176 0.01176 0.01176 1.57% MLPoisson::Fapply() 1142 0.0106 0.0106 0.0106 1.42% MLCellLinOp::defineAuxData() 11 0.009626 0.009626 0.009626 1.29% Gravity::fill_multipole_BCs() 11 0.008433 0.008433 0.008433 1.13% Castro::estTimeStep() 21 0.007115 0.007115 0.007115 0.95% MLMG::addInterpCorrection() 410 0.006857 0.006857 0.006857 0.92% amrex::average_down 410 0.00613 0.00613 0.00613 0.82% MultiFab::Xpay() 585 0.006018 0.006018 0.006018 0.81% Castro::reset_internal_energy(MultiFab) 63 0.005019 0.005019 0.005019 0.67% Castro::do_advance_ctu() 10 0.004863 0.004863 0.004863 0.65% Amr::checkPoint() 3 0.004107 0.004107 0.004107 0.55% BndryData::define() 11 0.003574 0.003574 0.003574 0.48% Castro::construct_new_gravity_source() 10 0.003189 0.003189 0.003189 0.43% Castro::construct_old_gravity_source() 10 0.002633 0.002633 0.002633 0.35% Amr::writePlotFile() 2 0.002384 0.002384 0.002384 0.32% Castro::enforce_speed_limit() 62 0.00219 0.00219 0.00219 0.29% MLMG::ResNormInf() 93 0.001967 0.001967 0.001967 0.26% Gravity::get_new_grav_vector() 11 0.001918 0.001918 0.001918 0.26% MultiFab::Saxpy() 20 0.001813 0.001813 0.001813 0.24% Castro::expand_state() 10 0.001733 0.001733 0.001733 0.23% Gravity::get_old_grav_vector() 10 0.001716 0.001716 0.001716 0.23% MLMG::oneIter() 82 0.001704 0.001704 0.001704 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001632 0.001632 0.001632 0.22% Castro::reset_internal_energy(Fab) 504 0.001583 0.001583 0.001583 0.21% MLCellLinOp::smooth() 1640 0.001518 0.001518 0.001518 0.20% Gravity::actual_solve_with_mlmg() 11 0.001441 0.001441 0.001441 0.19% FabArray::mult() 43 0.00133 0.00133 0.00133 0.18% MLCellLinOp::setLevelBC() 11 0.001297 0.001297 0.001297 0.17% FabArray::setDomainBndry() 41 0.001283 0.001283 0.001283 0.17% Castro::initData() 1 0.001218 0.001218 0.001218 0.16% MultiFab::contains_nan() 20 0.001166 0.001166 0.001166 0.16% MLCellLinOp::prepareForSolve() 11 0.001056 0.001056 0.001056 0.14% MLCellLinOp::compGrad() 11 0.0008937 0.0008937 0.0008937 0.12% FabArray::FillBoundary() 4023 0.0008804 0.0008804 0.0008804 0.12% FabArrayBase::getCPC() 1323 0.0008691 0.0008691 0.0008691 0.12% MLMG::prepareForSolve() 11 0.0007887 0.0007887 0.0007887 0.11% FabArrayBase::getFB() 4023 0.0007205 0.0007205 0.0007205 0.10% FabArrayBase::CPC::define() 454 0.000695 0.000695 0.000695 0.09% MLCellLinOp::apply() 1142 0.0006414 0.0006414 0.0006414 0.09% CGSolver::sxay() 1586 0.0005288 0.0005288 0.0005288 0.07% Amr::InitAmr() 1 0.0004463 0.0004463 0.0004463 0.06% Gravity::update_max_rhs() 11 0.0004118 0.0004118 0.0004118 0.06% MLMG::mgVcycle() 82 0.0004028 0.0004028 0.0004028 0.05% MLLinOp::defineGrids() 11 0.0003882 0.0003882 0.0003882 0.05% MLCGSolver::ParallelAllReduce 1514 0.00038 0.00038 0.00038 0.05% Gravity::solve_for_phi() 10 0.0003712 0.0003712 0.0003712 0.05% FabArray::ParallelCopy() 861 0.0003335 0.0003335 0.0003335 0.04% main() 1 0.0003078 0.0003078 0.0003078 0.04% MLCellLinOp::correctionResidual() 492 0.0002687 0.0002687 0.0002687 0.04% MultiFab::Copy() 11 0.0002628 0.0002628 0.0002628 0.04% FillPatchIterator::Initialize 41 0.000259 0.000259 0.000259 0.03% MultiFab::max() 11 0.0002561 0.0002561 0.0002561 0.03% Amr::coarseTimeStep() 10 0.00021 0.00021 0.00021 0.03% Castro::construct_new_gravity() 10 0.000205 0.000205 0.000205 0.03% Amr::timeStep() 10 0.0001981 0.0001981 0.0001981 0.03% MLMG::MLRhsNormInf() 11 0.000198 0.000198 0.000198 0.03% Castro::subcycle_advance_ctu() 10 0.0001971 0.0001971 0.0001971 0.03% MLCellLinOp::defineBC() 11 0.0001853 0.0001853 0.0001853 0.02% MLMG:computeResOfCorrection() 410 0.0001635 0.0001635 0.0001635 0.02% StateData::checkPoint() 12 0.0001346 0.0001346 0.0001346 0.02% MLMG::actualBottomSolve() 82 0.0001179 0.0001179 0.0001179 0.02% Castro::advance() 10 9.916e-05 9.916e-05 9.916e-05 0.01% MLMG::mgVcycle_down::0 82 9.401e-05 9.401e-05 9.401e-05 0.01% FabArrayBase::FB::FB() 56 8.807e-05 8.807e-05 8.807e-05 0.01% Castro::clean_state() 62 8.728e-05 8.728e-05 8.728e-05 0.01% MLMG::solve() 11 8.619e-05 8.619e-05 8.619e-05 0.01% MLMG::mgVcycle_down::1 82 8.549e-05 8.549e-05 8.549e-05 0.01% Castro::construct_new_source() 50 8.508e-05 8.508e-05 8.508e-05 0.01% MLMG::mgVcycle_down::2 82 8.312e-05 8.312e-05 8.312e-05 0.01% Castro::Castro() 1 8.119e-05 8.119e-05 8.119e-05 0.01% Castro::initialize_advance() 10 8.016e-05 8.016e-05 8.016e-05 0.01% MLMG::mgVcycle_down::4 82 7.923e-05 7.923e-05 7.923e-05 0.01% MLMG::mgVcycle_down::3 82 7.9e-05 7.9e-05 7.9e-05 0.01% AmrLevel::checkPoint() 3 7.562e-05 7.562e-05 7.562e-05 0.01% Castro::initialize_do_advance() 10 6.364e-05 6.364e-05 6.364e-05 0.01% MLMG::mgVcycle_up::4 82 6.237e-05 6.237e-05 6.237e-05 0.01% Castro::finalize_advance() 10 5.686e-05 5.686e-05 5.686e-05 0.01% MLCellLinOp::solutionResidual() 93 5.416e-05 5.416e-05 5.416e-05 0.01% MLMG::mgVcycle_up::0 82 5.405e-05 5.405e-05 5.405e-05 0.01% MLMG::mgVcycle_up::1 82 5.367e-05 5.367e-05 5.367e-05 0.01% MLMG::mgVcycle_up::2 82 5.115e-05 5.115e-05 5.115e-05 0.01% MLMG::mgVcycle_up::3 82 5.096e-05 5.096e-05 5.096e-05 0.01% StateData::define() 4 4.31e-05 4.31e-05 4.31e-05 0.01% MLMG::computeResidual() 82 4.091e-05 4.091e-05 4.091e-05 0.01% Castro::swap_state_time_levels() 10 3.836e-05 3.836e-05 3.836e-05 0.01% MLMG::mgVcycle_bottom 82 3.787e-05 3.787e-05 3.787e-05 0.01% Castro::finalize_do_advance() 10 3.671e-05 3.671e-05 3.671e-05 0.00% Castro::enforce_consistent_e() 1 3.513e-05 3.513e-05 3.513e-05 0.00% FillPatchSingleLevel 41 3.425e-05 3.425e-05 3.425e-05 0.00% Gravity::actual_multilevel_solve() 1 3.035e-05 3.035e-05 3.035e-05 0.00% MLLinOp::define() 11 2.485e-05 2.485e-05 2.485e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.472e-05 2.472e-05 2.472e-05 0.00% Amr::writeSmallPlotFile() 1 2.47e-05 2.47e-05 2.47e-05 0.00% makeSFC 55 2.411e-05 2.411e-05 2.411e-05 0.00% Castro::initMFs() 1 2.299e-05 2.299e-05 2.299e-05 0.00% MLPoisson::define() 11 2.294e-05 2.294e-05 2.294e-05 0.00% Amr::FinalizeInit() 1 2.198e-05 2.198e-05 2.198e-05 0.00% Castro::buildMetrics() 1 2.137e-05 2.137e-05 2.137e-05 0.00% Castro::construct_old_source() 50 2.007e-05 2.007e-05 2.007e-05 0.00% Castro::do_new_sources() 10 1.778e-05 1.778e-05 1.778e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.726e-05 1.726e-05 1.726e-05 0.00% Castro::do_old_sources() 10 1.722e-05 1.722e-05 1.722e-05 0.00% Amr::defBaseLevel() 1 1.588e-05 1.588e-05 1.588e-05 0.00% DistributionMapping::Distribute() 56 1.571e-05 1.571e-05 1.571e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.447e-05 1.447e-05 1.447e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.412e-05 1.412e-05 1.412e-05 0.00% Castro::check_for_nan() 20 1.316e-05 1.316e-05 1.316e-05 0.00% Castro::apply_source_to_state() 20 1.271e-05 1.271e-05 1.271e-05 0.00% Gravity::swapTimeLevels() 10 1.096e-05 1.096e-05 1.096e-05 0.00% Castro::construct_old_gravity() 10 1.061e-05 1.061e-05 1.061e-05 0.00% Amr::initSubcycle() 1 9.223e-06 9.223e-06 9.223e-06 0.00% MLPoisson::prepareForSolve() 11 9.004e-06 9.004e-06 9.004e-06 0.00% Castro::post_timestep() 10 7.173e-06 7.173e-06 7.173e-06 0.00% MLMG::computeMLResidual() 11 6.836e-06 6.836e-06 6.836e-06 0.00% Castro::computeNewDt() 9 6.712e-06 6.712e-06 6.712e-06 0.00% MLMG::getGradSolution() 11 6.623e-06 6.623e-06 6.623e-06 0.00% Amr::InitializeInit() 1 6.319e-06 6.319e-06 6.319e-06 0.00% MLMG::MLResNormInf() 11 5.845e-06 5.845e-06 5.845e-06 0.00% AmrLevel::checkPointPost() 3 5.384e-06 5.384e-06 5.384e-06 0.00% MLMG::buildFineMask() 11 5.299e-06 5.299e-06 5.299e-06 0.00% Castro::create_source_corrector() 10 5.149e-06 5.149e-06 5.149e-06 0.00% Gravity::set_mass_offset() 11 5.144e-06 5.144e-06 5.144e-06 0.00% Castro::retry_advance_ctu() 10 4.015e-06 4.015e-06 4.015e-06 0.00% Castro::post_init() 1 3.663e-06 3.663e-06 3.663e-06 0.00% Castro::FluxRegCrseInit 10 3.234e-06 3.234e-06 3.234e-06 0.00% Castro::FluxRegFineAdd() 10 2.819e-06 2.819e-06 2.819e-06 0.00% Castro::computeInitialDt() 2 2.716e-06 2.716e-06 2.716e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.565e-06 2.565e-06 2.565e-06 0.00% Amr::init() 1 2.469e-06 2.469e-06 2.469e-06 0.00% AmrLevel::checkPointPre() 3 2.176e-06 2.176e-06 2.176e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.746e-06 1.746e-06 1.746e-06 0.00% Castro::post_regrid() 1 1.174e-06 1.174e-06 1.174e-06 0.00% Amr::initialInit() 1 9.31e-07 9.31e-07 9.31e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.747 0.747 0.747 100.00% Amr::coarseTimeStep() 10 0.6136 0.6136 0.6136 82.13% Amr::timeStep() 10 0.5386 0.5386 0.5386 72.10% Castro::advance() 10 0.5284 0.5284 0.5284 70.74% Castro::subcycle_advance_ctu() 10 0.5151 0.5151 0.5151 68.95% Castro::do_advance_ctu() 10 0.5149 0.5149 0.5149 68.92% Gravity::solve_phi_with_mlmg() 11 0.2901 0.2901 0.2901 38.83% Gravity::actual_solve_with_mlmg() 11 0.2814 0.2814 0.2814 37.67% Castro::construct_new_gravity() 10 0.2631 0.2631 0.2631 35.22% MLMG::solve() 11 0.2611 0.2611 0.2611 34.95% Gravity::solve_for_phi() 10 0.2483 0.2483 0.2483 33.24% MLMG::oneIter() 82 0.2473 0.2473 0.2473 33.11% MLMG::mgVcycle() 82 0.2456 0.2456 0.2456 32.88% Castro::construct_ctu_hydro_source() 10 0.1634 0.1634 0.1634 21.88% VisMF::Write(FabArray) 11 0.1476 0.1476 0.1476 19.76% MLCellLinOp::smooth() 1640 0.1267 0.1267 0.1267 16.96% Amr::init() 1 0.1103 0.1103 0.1103 14.76% Amr::checkPoint() 3 0.1094 0.1094 0.1094 14.64% AmrLevel::checkPoint() 3 0.1053 0.1053 0.1053 14.09% StateData::checkPoint() 12 0.1052 0.1052 0.1052 14.08% MLCellLinOp::applyBC() 4433 0.0886 0.0886 0.0886 11.86% MLMG::mgVcycle_bottom 82 0.07378 0.07378 0.07378 9.88% MLMG::actualBottomSolve() 82 0.07374 0.07374 0.07374 9.87% MLCGSolver::bicgstab 82 0.073 0.073 0.073 9.77% MLPoisson::Fsmooth() 3280 0.05959 0.05959 0.05959 7.98% Castro::clean_state() 62 0.05442 0.05442 0.05442 7.28% Amr::initialInit() 1 0.04931 0.04931 0.04931 6.60% FillPatchIterator::Initialize 41 0.04725 0.04725 0.04725 6.32% FillPatchSingleLevel 41 0.0457 0.0457 0.0457 6.12% Amr::FinalizeInit() 1 0.04522 0.04522 0.04522 6.05% Amr::writePlotFile() 2 0.04492 0.04492 0.04492 6.01% Castro::post_init() 1 0.0439 0.0439 0.0439 5.88% Gravity::multilevel_solve_for_new_phi() 1 0.04215 0.04215 0.04215 5.64% Gravity::actual_multilevel_solve() 1 0.04214 0.04214 0.04214 5.64% StateDataPhysBCFunct::() 41 0.04174 0.04174 0.04174 5.59% MLMG::mgVcycle_down::0 82 0.03459 0.03459 0.03459 4.63% MLCellLinOp::apply() 1142 0.03378 0.03378 0.03378 4.52% MLMG::mgVcycle_up::0 82 0.02958 0.02958 0.02958 3.96% Castro::initialize_do_advance() 10 0.0275 0.0275 0.0275 3.68% StateData::FillBoundary(geom) 328 0.02334 0.02334 0.02334 3.12% Castro::computeTemp() 63 0.02105 0.02105 0.02105 2.82% MultiFab::Dot() 1114 0.02068 0.02068 0.02068 2.77% Castro::normalize_species() 62 0.02009 0.02009 0.02009 2.69% MLCellLinOp::correctionResidual() 492 0.01968 0.01968 0.01968 2.63% Castro::expand_state() 10 0.01822 0.01822 0.01822 2.44% MLMG:computeResOfCorrection() 410 0.01712 0.01712 0.01712 2.29% MLMG::mgVcycle_down::1 82 0.01625 0.01625 0.01625 2.18% Gravity::get_new_grav_vector() 11 0.01621 0.01621 0.01621 2.17% MLPoisson::define() 11 0.01618 0.01618 0.01618 2.17% FabArray::FillBoundary() 4023 0.01607 0.01607 0.01607 2.15% MLMG::mgVcycle_down::2 82 0.01569 0.01569 0.01569 2.10% FillBoundary_nowait() 4023 0.01519 0.01519 0.01519 2.03% MLMG::mgVcycle_down::3 82 0.0148 0.0148 0.0148 1.98% Castro::construct_old_gravity() 10 0.01438 0.01438 0.01438 1.92% Gravity::get_old_grav_vector() 10 0.01437 0.01437 0.01437 1.92% FabArray::ParallelCopy() 861 0.01419 0.01419 0.01419 1.90% MLMG::mgVcycle_down::4 82 0.014 0.014 0.014 1.87% FabArray::ParallelCopy_nowait() 861 0.01386 0.01386 0.01386 1.86% Castro::do_new_sources() 10 0.01319 0.01319 0.01319 1.76% Castro::initialize_advance() 10 0.01317 0.01317 0.01317 1.76% FabArray::setVal() 1144 0.01306 0.01306 0.01306 1.75% CGSolver::sxay() 1586 0.01281 0.01281 0.01281 1.71% MLCGSolver::ParallelAllReduce 1514 0.01247 0.01247 0.01247 1.67% MultiFab::LinComb() 1586 0.01228 0.01228 0.01228 1.64% MLMG::mgVcycle_up::2 82 0.01211 0.01211 0.01211 1.62% MLMG::addInterpCorrection() 410 0.01195 0.01195 0.01195 1.60% MLMG::mgVcycle_up::1 82 0.0119 0.0119 0.0119 1.59% Castro::enforce_min_density() 62 0.01176 0.01176 0.01176 1.57% MLMG::mgVcycle_up::3 82 0.01137 0.01137 0.01137 1.52% Castro::do_old_sources() 10 0.01131 0.01131 0.01131 1.51% amrex::average_down 410 0.0113 0.0113 0.0113 1.51% MLMG::mgVcycle_up::4 82 0.01116 0.01116 0.01116 1.49% MLCellLinOp::defineAuxData() 11 0.011 0.011 0.011 1.47% MLPoisson::Fapply() 1142 0.0106 0.0106 0.0106 1.42% Castro::post_timestep() 10 0.009958 0.009958 0.009958 1.33% Gravity::fill_multipole_BCs() 11 0.008433 0.008433 0.008433 1.13% Castro::estTimeStep() 21 0.007115 0.007115 0.007115 0.95% MLCellLinOp::solutionResidual() 93 0.007031 0.007031 0.007031 0.94% Castro::reset_internal_energy(MultiFab) 63 0.006602 0.006602 0.006602 0.88% MLMG::computeResidual() 82 0.006062 0.006062 0.006062 0.81% MultiFab::Xpay() 585 0.006018 0.006018 0.006018 0.81% MLCellLinOp::defineBC() 11 0.004689 0.004689 0.004689 0.63% BndryData::define() 11 0.004503 0.004503 0.004503 0.60% MLMG::prepareForSolve() 11 0.00446 0.00446 0.00446 0.60% Amr::InitializeInit() 1 0.004091 0.004091 0.004091 0.55% Amr::defBaseLevel() 1 0.004085 0.004085 0.004085 0.55% Castro::initData() 1 0.003587 0.003587 0.003587 0.48% Castro::computeNewDt() 9 0.003352 0.003352 0.003352 0.45% Castro::construct_new_source() 50 0.003274 0.003274 0.003274 0.44% Castro::construct_new_gravity_source() 10 0.003189 0.003189 0.003189 0.43% Castro::construct_old_source() 50 0.002653 0.002653 0.002653 0.36% Castro::construct_old_gravity_source() 10 0.002633 0.002633 0.002633 0.35% Castro::enforce_speed_limit() 62 0.00219 0.00219 0.00219 0.29% MLMG::ResNormInf() 93 0.001967 0.001967 0.001967 0.26% Castro::apply_source_to_state() 20 0.001825 0.001825 0.001825 0.24% MultiFab::Saxpy() 20 0.001813 0.001813 0.001813 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001632 0.001632 0.001632 0.22% Castro::reset_internal_energy(Fab) 504 0.001583 0.001583 0.001583 0.21% FabArrayBase::getCPC() 1323 0.001564 0.001564 0.001564 0.21% MLMG::getGradSolution() 11 0.001358 0.001358 0.001358 0.18% MLCellLinOp::compGrad() 11 0.001352 0.001352 0.001352 0.18% FabArray::mult() 43 0.00133 0.00133 0.00133 0.18% MLCellLinOp::setLevelBC() 11 0.001297 0.001297 0.001297 0.17% FabArray::setDomainBndry() 41 0.001283 0.001283 0.001283 0.17% Castro::check_for_nan() 20 0.001179 0.001179 0.001179 0.16% MultiFab::contains_nan() 20 0.001166 0.001166 0.001166 0.16% Castro::post_regrid() 1 0.001082 0.001082 0.001082 0.14% MLPoisson::prepareForSolve() 11 0.001065 0.001065 0.001065 0.14% MLCellLinOp::prepareForSolve() 11 0.001056 0.001056 0.001056 0.14% MLMG::computeMLResidual() 11 0.001017 0.001017 0.001017 0.14% Gravity::update_max_rhs() 11 0.0008144 0.0008144 0.0008144 0.11% FabArrayBase::getFB() 4023 0.0008086 0.0008086 0.0008086 0.11% FabArrayBase::CPC::define() 454 0.000695 0.000695 0.000695 0.09% Castro::computeInitialDt() 2 0.0006815 0.0006815 0.0006815 0.09% MLLinOp::define() 11 0.0004676 0.0004676 0.0004676 0.06% Amr::InitAmr() 1 0.0004555 0.0004555 0.0004555 0.06% Gravity::swapTimeLevels() 10 0.0004446 0.0004446 0.0004446 0.06% MLLinOp::defineGrids() 11 0.0004428 0.0004428 0.0004428 0.06% Castro::Castro() 1 0.000421 0.000421 0.000421 0.06% MultiFab::Copy() 11 0.0002628 0.0002628 0.0002628 0.04% MultiFab::max() 11 0.0002561 0.0002561 0.0002561 0.03% MLMG::MLResNormInf() 11 0.0002555 0.0002555 0.0002555 0.03% MLMG::MLRhsNormInf() 11 0.000198 0.000198 0.000198 0.03% Castro::buildMetrics() 1 0.0001534 0.0001534 0.0001534 0.02% FabArrayBase::FB::FB() 56 8.807e-05 8.807e-05 8.807e-05 0.01% Castro::finalize_advance() 10 6.291e-05 6.291e-05 6.291e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.757e-05 5.757e-05 5.757e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.282e-05 5.282e-05 5.282e-05 0.01% StateData::define() 4 4.31e-05 4.31e-05 4.31e-05 0.01% makeSFC 55 3.87e-05 3.87e-05 3.87e-05 0.01% Castro::swap_state_time_levels() 10 3.836e-05 3.836e-05 3.836e-05 0.01% Castro::finalize_do_advance() 10 3.671e-05 3.671e-05 3.671e-05 0.00% Castro::enforce_consistent_e() 1 3.513e-05 3.513e-05 3.513e-05 0.00% Amr::writeSmallPlotFile() 1 2.47e-05 2.47e-05 2.47e-05 0.00% Castro::initMFs() 1 2.299e-05 2.299e-05 2.299e-05 0.00% DistributionMapping::Distribute() 56 1.571e-05 1.571e-05 1.571e-05 0.00% Amr::initSubcycle() 1 9.223e-06 9.223e-06 9.223e-06 0.00% AmrLevel::checkPointPost() 3 5.384e-06 5.384e-06 5.384e-06 0.00% MLMG::buildFineMask() 11 5.299e-06 5.299e-06 5.299e-06 0.00% Castro::create_source_corrector() 10 5.149e-06 5.149e-06 5.149e-06 0.00% Gravity::set_mass_offset() 11 5.144e-06 5.144e-06 5.144e-06 0.00% Castro::retry_advance_ctu() 10 4.015e-06 4.015e-06 4.015e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.692e-06 3.692e-06 3.692e-06 0.00% Castro::FluxRegCrseInit 10 3.234e-06 3.234e-06 3.234e-06 0.00% Castro::FluxRegFineAdd() 10 2.819e-06 2.819e-06 2.819e-06 0.00% AmrLevel::checkPointPre() 3 2.176e-06 2.176e-06 2.176e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.746e-06 1.746e-06 1.746e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-4-gef52970b80e3) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-4-gef52970b80e3) initialized Starting run at 08:24:30 UTC on 2022-05-04. Successfully read inputs file ... Castro git describe: 22.05-8-gf5a7de0a1 AMReX git describe: 22.05-4-gef52970b8 Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.413544009 Restart time = 0.046092565 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.049533318 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048564013 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.049042835 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.056582346 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062507397 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.023655279 seconds Ending run at 08:24:31 UTC on 2022-05-04. Run time = 0.336881989 Run time without initialization = 0.290246653 Average number of zones advanced per microsecond: 4.516 Average number of zones advanced per microsecond per rank: 4.516 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3369 ... 0.3369 ... 0.3369 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0798 0.0798 0.0798 23.68% VisMF::Read() 3 0.03907 0.03907 0.03907 11.60% MLCellLinOp::applyBC() 1946 0.03135 0.03135 0.03135 9.31% MLPoisson::Fsmooth() 1440 0.02544 0.02544 0.02544 7.55% VisMF::Write(FabArray) 1 0.02224 0.02224 0.02224 6.60% StateData::FillBoundary(geom) 160 0.01127 0.01127 0.01127 3.35% MLCGSolver::bicgstab 36 0.009111 0.009111 0.009111 2.70% Castro::normalize_species() 30 0.009019 0.009019 0.009019 2.68% Castro::computeTemp() 30 0.00894 0.00894 0.00894 2.65% MultiFab::Dot() 484 0.00872 0.00872 0.00872 2.59% FabArray::setVal() 537 0.006216 0.006216 0.006216 1.84% FillBoundary_nowait() 1766 0.00619 0.00619 0.00619 1.84% FabArray::ParallelCopy_nowait() 380 0.005857 0.005857 0.005857 1.74% MLCellLinOp::defineAuxData() 6 0.005328 0.005328 0.005328 1.58% MultiFab::LinComb() 690 0.005251 0.005251 0.005251 1.56% StateDataPhysBCFunct::() 20 0.004826 0.004826 0.004826 1.43% MLPoisson::Fapply() 500 0.00454 0.00454 0.00454 1.35% Gravity::fill_multipole_BCs() 6 0.004477 0.004477 0.004477 1.33% Castro::enforce_min_density() 30 0.003995 0.003995 0.003995 1.19% Castro::estTimeStep() 10 0.003768 0.003768 0.003768 1.12% Amr::restart() 1 0.002943 0.002943 0.002943 0.87% MLMG::addInterpCorrection() 180 0.002844 0.002844 0.002844 0.84% amrex::average_down 180 0.002666 0.002666 0.002666 0.79% MultiFab::Xpay() 258 0.002632 0.002632 0.002632 0.78% Castro::do_advance_ctu() 5 0.002401 0.002401 0.002401 0.71% BndryData::define() 6 0.001999 0.001999 0.001999 0.59% Castro::construct_new_gravity_source() 5 0.001738 0.001738 0.001738 0.52% Castro::reset_internal_energy(MultiFab) 30 0.001615 0.001615 0.001615 0.48% Amr::writePlotFile() 1 0.001517 0.001517 0.001517 0.45% Castro::construct_old_gravity_source() 5 0.001377 0.001377 0.001377 0.41% Gravity::get_old_grav_vector() 5 0.0009294 0.0009294 0.0009294 0.28% MultiFab::Saxpy() 10 0.0009222 0.0009222 0.0009222 0.27% Castro::reset_internal_energy(Fab) 240 0.0009085 0.0009085 0.0009085 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008789 0.0008789 0.0008789 0.26% Castro::expand_state() 5 0.000871 0.000871 0.000871 0.26% Gravity::get_new_grav_vector() 5 0.0008642 0.0008642 0.0008642 0.26% MLMG::ResNormInf() 42 0.0008582 0.0008582 0.0008582 0.25% Gravity::actual_solve_with_mlmg() 6 0.0008216 0.0008216 0.0008216 0.24% MLMG::oneIter() 36 0.0007369 0.0007369 0.0007369 0.22% MLCellLinOp::setLevelBC() 6 0.0007168 0.0007168 0.0007168 0.21% FabArray::mult() 22 0.0006511 0.0006511 0.0006511 0.19% FabArray::setDomainBndry() 20 0.0006263 0.0006263 0.0006263 0.19% MLCellLinOp::prepareForSolve() 6 0.0005924 0.0005924 0.0005924 0.18% MultiFab::contains_nan() 10 0.0005852 0.0005852 0.0005852 0.17% MLCellLinOp::smooth() 720 0.0005057 0.0005057 0.0005057 0.15% MLCellLinOp::compGrad() 6 0.0004781 0.0004781 0.0004781 0.14% FabArrayBase::CPC::define() 244 0.0004255 0.0004255 0.0004255 0.13% MLMG::prepareForSolve() 6 0.0004241 0.0004241 0.0004241 0.13% FabArrayBase::getCPC() 632 0.0003965 0.0003965 0.0003965 0.12% FabArray::FillBoundary() 1766 0.0003776 0.0003776 0.0003776 0.11% Castro::enforce_speed_limit() 30 0.0003756 0.0003756 0.0003756 0.11% Amr::InitAmr() 1 0.0003695 0.0003695 0.0003695 0.11% FabArrayBase::getFB() 1766 0.0002995 0.0002995 0.0002995 0.09% main() 1 0.0002804 0.0002804 0.0002804 0.08% MLCellLinOp::apply() 500 0.0002426 0.0002426 0.0002426 0.07% Gravity::solve_for_phi() 5 0.0002389 0.0002389 0.0002389 0.07% Gravity::update_max_rhs() 6 0.0002314 0.0002314 0.0002314 0.07% Castro::subcycle_advance_ctu() 5 0.0002153 0.0002153 0.0002153 0.06% CGSolver::sxay() 690 0.0002029 0.0002029 0.0002029 0.06% MLLinOp::defineGrids() 6 0.0001806 0.0001806 0.0001806 0.05% Castro::construct_new_gravity() 5 0.0001645 0.0001645 0.0001645 0.05% MLMG::mgVcycle() 36 0.0001612 0.0001612 0.0001612 0.05% Castro::initialize_advance() 5 0.0001597 0.0001597 0.0001597 0.05% Castro::post_timestep() 5 0.0001589 0.0001589 0.0001589 0.05% MLCGSolver::ParallelAllReduce 659 0.0001575 0.0001575 0.0001575 0.05% MultiFab::Copy() 6 0.0001426 0.0001426 0.0001426 0.04% Castro::create_source_corrector() 5 0.0001367 0.0001367 0.0001367 0.04% MultiFab::max() 6 0.0001356 0.0001356 0.0001356 0.04% FabArray::ParallelCopy() 380 0.0001337 0.0001337 0.0001337 0.04% Castro::construct_new_source() 25 0.0001313 0.0001313 0.0001313 0.04% FillPatchIterator::Initialize 20 0.0001252 0.0001252 0.0001252 0.04% Castro::advance() 5 0.0001107 0.0001107 0.0001107 0.03% MLCellLinOp::correctionResidual() 216 0.0001065 0.0001065 0.0001065 0.03% Amr::coarseTimeStep() 5 0.000106 0.000106 0.000106 0.03% MLMG::MLRhsNormInf() 6 0.0001055 0.0001055 0.0001055 0.03% Amr::timeStep() 5 0.0001008 0.0001008 0.0001008 0.03% MLCellLinOp::defineBC() 6 0.0001008 0.0001008 0.0001008 0.03% Castro::finalize_advance() 5 9.163e-05 9.163e-05 9.163e-05 0.03% AmrLevel::restart() 1 7.574e-05 7.574e-05 7.574e-05 0.02% StateData::restartDoit() 4 6.983e-05 6.983e-05 6.983e-05 0.02% Castro::initialize_do_advance() 5 6.832e-05 6.832e-05 6.832e-05 0.02% FabArrayBase::FB::FB() 26 6.069e-05 6.069e-05 6.069e-05 0.02% MLMG:computeResOfCorrection() 180 5.766e-05 5.766e-05 5.766e-05 0.02% Castro::construct_old_source() 25 5.359e-05 5.359e-05 5.359e-05 0.02% MLMG::actualBottomSolve() 36 4.644e-05 4.644e-05 4.644e-05 0.01% MLMG::mgVcycle_down::0 36 3.712e-05 3.712e-05 3.712e-05 0.01% MLMG::solve() 6 3.689e-05 3.689e-05 3.689e-05 0.01% Castro::clean_state() 30 3.662e-05 3.662e-05 3.662e-05 0.01% MLMG::mgVcycle_down::1 36 3.555e-05 3.555e-05 3.555e-05 0.01% Castro::construct_old_gravity() 5 3.383e-05 3.383e-05 3.383e-05 0.01% MLMG::mgVcycle_down::2 36 3.334e-05 3.334e-05 3.334e-05 0.01% MLMG::mgVcycle_down::4 36 3.216e-05 3.216e-05 3.216e-05 0.01% MLMG::mgVcycle_down::3 36 3.163e-05 3.163e-05 3.163e-05 0.01% Castro::buildMetrics() 1 3.151e-05 3.151e-05 3.151e-05 0.01% Gravity::actual_multilevel_solve() 1 2.948e-05 2.948e-05 2.948e-05 0.01% Castro::swap_state_time_levels() 5 2.757e-05 2.757e-05 2.757e-05 0.01% Castro::post_restart() 1 2.722e-05 2.722e-05 2.722e-05 0.01% MLMG::mgVcycle_up::4 36 2.654e-05 2.654e-05 2.654e-05 0.01% Amr::writeSmallPlotFile() 1 2.545e-05 2.545e-05 2.545e-05 0.01% Castro::initMFs() 1 2.482e-05 2.482e-05 2.482e-05 0.01% MLLinOp::define() 6 2.245e-05 2.245e-05 2.245e-05 0.01% MLCellLinOp::solutionResidual() 42 2.239e-05 2.239e-05 2.239e-05 0.01% MLMG::mgVcycle_up::0 36 2.105e-05 2.105e-05 2.105e-05 0.01% MLMG::mgVcycle_up::3 36 2.101e-05 2.101e-05 2.101e-05 0.01% Castro::finalize_do_advance() 5 2.048e-05 2.048e-05 2.048e-05 0.01% MLMG::mgVcycle_up::2 36 1.995e-05 1.995e-05 1.995e-05 0.01% MLMG::mgVcycle_up::1 36 1.885e-05 1.885e-05 1.885e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.688e-05 1.688e-05 1.688e-05 0.01% MLMG::mgVcycle_bottom 36 1.616e-05 1.616e-05 1.616e-05 0.00% MLMG::computeResidual() 36 1.562e-05 1.562e-05 1.562e-05 0.00% MLPoisson::define() 6 1.491e-05 1.491e-05 1.491e-05 0.00% FillPatchSingleLevel 20 1.395e-05 1.395e-05 1.395e-05 0.00% makeSFC 30 1.373e-05 1.373e-05 1.373e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.362e-05 1.362e-05 1.362e-05 0.00% Castro::do_new_sources() 5 9.194e-06 9.194e-06 9.194e-06 0.00% Amr::initSubcycle() 1 8.974e-06 8.974e-06 8.974e-06 0.00% Castro::do_old_sources() 5 8.525e-06 8.525e-06 8.525e-06 0.00% DistributionMapping::Distribute() 31 8.496e-06 8.496e-06 8.496e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.61e-06 7.61e-06 7.61e-06 0.00% Castro::check_for_nan() 10 6.218e-06 6.218e-06 6.218e-06 0.00% Castro::apply_source_to_state() 10 5.699e-06 5.699e-06 5.699e-06 0.00% MLPoisson::prepareForSolve() 6 4.434e-06 4.434e-06 4.434e-06 0.00% Gravity::swapTimeLevels() 5 4.161e-06 4.161e-06 4.161e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.758e-06 3.758e-06 3.758e-06 0.00% MLMG::computeMLResidual() 6 3.203e-06 3.203e-06 3.203e-06 0.00% MLMG::getGradSolution() 6 3.15e-06 3.15e-06 3.15e-06 0.00% Castro::computeNewDt() 5 3.076e-06 3.076e-06 3.076e-06 0.00% MLMG::buildFineMask() 6 2.826e-06 2.826e-06 2.826e-06 0.00% MLMG::MLResNormInf() 6 2.684e-06 2.684e-06 2.684e-06 0.00% Gravity::set_mass_offset() 6 2.633e-06 2.633e-06 2.633e-06 0.00% Castro::retry_advance_ctu() 5 2.14e-06 2.14e-06 2.14e-06 0.00% Castro::FluxRegCrseInit 5 1.635e-06 1.635e-06 1.635e-06 0.00% Castro::FluxRegFineAdd() 5 1.238e-06 1.238e-06 1.238e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.235e-06 1.235e-06 1.235e-06 0.00% AmrLevel::AmrLevel() 1 1.16e-06 1.16e-06 1.16e-06 0.00% Amr::init() 1 1.035e-06 1.035e-06 1.035e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3369 0.3369 0.3369 100.00% Amr::coarseTimeStep() 5 0.2663 0.2663 0.2663 79.05% Amr::timeStep() 5 0.2647 0.2647 0.2647 78.56% Castro::advance() 5 0.2595 0.2595 0.2595 77.01% Castro::subcycle_advance_ctu() 5 0.2536 0.2536 0.2536 75.28% Castro::do_advance_ctu() 5 0.2534 0.2534 0.2534 75.21% Castro::construct_new_gravity() 5 0.132 0.132 0.132 39.19% Gravity::solve_phi_with_mlmg() 6 0.128 0.128 0.128 38.00% Gravity::solve_for_phi() 5 0.1247 0.1247 0.1247 37.01% Gravity::actual_solve_with_mlmg() 6 0.1234 0.1234 0.1234 36.63% MLMG::solve() 6 0.1122 0.1122 0.1122 33.30% MLMG::oneIter() 36 0.1056 0.1056 0.1056 31.33% MLMG::mgVcycle() 36 0.1048 0.1048 0.1048 31.11% Castro::construct_ctu_hydro_source() 5 0.07978 0.07978 0.07978 23.68% MLCellLinOp::smooth() 720 0.05421 0.05421 0.05421 16.09% Amr::init() 1 0.04613 0.04613 0.04613 13.69% Amr::restart() 1 0.04613 0.04613 0.04613 13.69% AmrLevel::restart() 1 0.03927 0.03927 0.03927 11.66% StateData::restartDoit() 4 0.03919 0.03919 0.03919 11.63% VisMF::Read() 3 0.03907 0.03907 0.03907 11.60% MLCellLinOp::applyBC() 1946 0.03828 0.03828 0.03828 11.36% MLMG::mgVcycle_bottom 36 0.03136 0.03136 0.03136 9.31% MLMG::actualBottomSolve() 36 0.03135 0.03135 0.03135 9.30% MLCGSolver::bicgstab 36 0.03103 0.03103 0.03103 9.21% MLPoisson::Fsmooth() 1440 0.02544 0.02544 0.02544 7.55% Castro::clean_state() 30 0.02489 0.02489 0.02489 7.39% Amr::writePlotFile() 1 0.02375 0.02375 0.02375 7.05% VisMF::Write(FabArray) 1 0.02224 0.02224 0.02224 6.60% FillPatchIterator::Initialize 20 0.01887 0.01887 0.01887 5.60% FillPatchSingleLevel 20 0.01812 0.01812 0.01812 5.38% StateDataPhysBCFunct::() 20 0.0161 0.0161 0.0161 4.78% MLMG::mgVcycle_down::0 36 0.01479 0.01479 0.01479 4.39% MLCellLinOp::apply() 500 0.01454 0.01454 0.01454 4.32% MLMG::mgVcycle_up::0 36 0.01258 0.01258 0.01258 3.73% Castro::computeTemp() 30 0.01146 0.01146 0.01146 3.40% StateData::FillBoundary(geom) 160 0.01127 0.01127 0.01127 3.35% Castro::initialize_do_advance() 5 0.009523 0.009523 0.009523 2.83% Castro::normalize_species() 30 0.009019 0.009019 0.009019 2.68% MLPoisson::define() 6 0.008973 0.008973 0.008973 2.66% MultiFab::Dot() 484 0.00872 0.00872 0.00872 2.59% MLCellLinOp::correctionResidual() 216 0.008448 0.008448 0.008448 2.51% Castro::construct_old_gravity() 5 0.007357 0.007357 0.007357 2.18% MLMG:computeResOfCorrection() 180 0.007331 0.007331 0.007331 2.18% Gravity::get_old_grav_vector() 5 0.007324 0.007324 0.007324 2.17% Gravity::get_new_grav_vector() 5 0.007167 0.007167 0.007167 2.13% MLMG::mgVcycle_down::1 36 0.00697 0.00697 0.00697 2.07% FabArray::FillBoundary() 1766 0.006928 0.006928 0.006928 2.06% MLMG::mgVcycle_down::2 36 0.006717 0.006717 0.006717 1.99% FillBoundary_nowait() 1766 0.00655 0.00655 0.00655 1.94% Castro::do_old_sources() 5 0.006538 0.006538 0.006538 1.94% FabArray::ParallelCopy() 380 0.006394 0.006394 0.006394 1.90% MLMG::mgVcycle_down::3 36 0.006334 0.006334 0.006334 1.88% Castro::do_new_sources() 5 0.006278 0.006278 0.006278 1.86% FabArray::ParallelCopy_nowait() 380 0.006261 0.006261 0.006261 1.86% FabArray::setVal() 537 0.006216 0.006216 0.006216 1.84% MLCellLinOp::defineAuxData() 6 0.006087 0.006087 0.006087 1.81% MLMG::mgVcycle_down::4 36 0.006026 0.006026 0.006026 1.79% Castro::initialize_advance() 5 0.005657 0.005657 0.005657 1.68% CGSolver::sxay() 690 0.005454 0.005454 0.005454 1.62% MLCGSolver::ParallelAllReduce 659 0.005284 0.005284 0.005284 1.57% MultiFab::LinComb() 690 0.005251 0.005251 0.005251 1.56% MLMG::mgVcycle_up::2 36 0.005161 0.005161 0.005161 1.53% Castro::expand_state() 5 0.005139 0.005139 0.005139 1.53% Castro::post_timestep() 5 0.005103 0.005103 0.005103 1.51% MLMG::mgVcycle_up::1 36 0.005079 0.005079 0.005079 1.51% MLMG::addInterpCorrection() 180 0.005019 0.005019 0.005019 1.49% amrex::average_down 180 0.00488 0.00488 0.00488 1.45% MLMG::mgVcycle_up::3 36 0.004855 0.004855 0.004855 1.44% MLMG::mgVcycle_up::4 36 0.0048 0.0048 0.0048 1.42% MLPoisson::Fapply() 500 0.00454 0.00454 0.00454 1.35% Gravity::fill_multipole_BCs() 6 0.004477 0.004477 0.004477 1.33% Castro::enforce_min_density() 30 0.003995 0.003995 0.003995 1.19% Castro::estTimeStep() 10 0.003768 0.003768 0.003768 1.12% Castro::post_restart() 1 0.003725 0.003725 0.003725 1.11% Gravity::multilevel_solve_for_new_phi() 1 0.003604 0.003604 0.003604 1.07% Gravity::actual_multilevel_solve() 1 0.003587 0.003587 0.003587 1.06% MLCellLinOp::solutionResidual() 42 0.003162 0.003162 0.003162 0.94% MLCellLinOp::defineBC() 6 0.002639 0.002639 0.002639 0.78% MultiFab::Xpay() 258 0.002632 0.002632 0.002632 0.78% MLMG::computeResidual() 36 0.002622 0.002622 0.002622 0.78% BndryData::define() 6 0.002538 0.002538 0.002538 0.75% Castro::reset_internal_energy(MultiFab) 30 0.002524 0.002524 0.002524 0.75% MLMG::prepareForSolve() 6 0.002432 0.002432 0.002432 0.72% Castro::construct_new_source() 25 0.001869 0.001869 0.001869 0.55% Castro::construct_new_gravity_source() 5 0.001738 0.001738 0.001738 0.52% Castro::computeNewDt() 5 0.001546 0.001546 0.001546 0.46% Castro::construct_old_source() 25 0.00143 0.00143 0.00143 0.42% Castro::construct_old_gravity_source() 5 0.001377 0.001377 0.001377 0.41% Castro::apply_source_to_state() 10 0.0009279 0.0009279 0.0009279 0.28% MultiFab::Saxpy() 10 0.0009222 0.0009222 0.0009222 0.27% Castro::reset_internal_energy(Fab) 240 0.0009085 0.0009085 0.0009085 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008789 0.0008789 0.0008789 0.26% MLMG::ResNormInf() 42 0.0008582 0.0008582 0.0008582 0.25% FabArrayBase::getCPC() 632 0.000822 0.000822 0.000822 0.24% MLMG::getGradSolution() 6 0.0007324 0.0007324 0.0007324 0.22% MLCellLinOp::compGrad() 6 0.0007292 0.0007292 0.0007292 0.22% MLCellLinOp::setLevelBC() 6 0.0007168 0.0007168 0.0007168 0.21% FabArray::mult() 22 0.0006511 0.0006511 0.0006511 0.19% FabArray::setDomainBndry() 20 0.0006263 0.0006263 0.0006263 0.19% MLPoisson::prepareForSolve() 6 0.0005968 0.0005968 0.0005968 0.18% MLCellLinOp::prepareForSolve() 6 0.0005924 0.0005924 0.0005924 0.18% Castro::check_for_nan() 10 0.0005914 0.0005914 0.0005914 0.18% MultiFab::contains_nan() 10 0.0005852 0.0005852 0.0005852 0.17% MLMG::computeMLResidual() 6 0.0005588 0.0005588 0.0005588 0.17% Gravity::update_max_rhs() 6 0.0004449 0.0004449 0.0004449 0.13% FabArrayBase::CPC::define() 244 0.0004255 0.0004255 0.0004255 0.13% Amr::InitAmr() 1 0.0003785 0.0003785 0.0003785 0.11% Castro::enforce_speed_limit() 30 0.0003756 0.0003756 0.0003756 0.11% FabArrayBase::getFB() 1766 0.0003602 0.0003602 0.0003602 0.11% MLLinOp::define() 6 0.0002325 0.0002325 0.0002325 0.07% Gravity::swapTimeLevels() 5 0.0002255 0.0002255 0.0002255 0.07% MLLinOp::defineGrids() 6 0.0002101 0.0002101 0.0002101 0.06% Castro::buildMetrics() 1 0.0001578 0.0001578 0.0001578 0.05% MultiFab::Copy() 6 0.0001426 0.0001426 0.0001426 0.04% Castro::create_source_corrector() 5 0.0001367 0.0001367 0.0001367 0.04% MultiFab::max() 6 0.0001356 0.0001356 0.0001356 0.04% MLMG::MLResNormInf() 6 0.0001344 0.0001344 0.0001344 0.04% MLMG::MLRhsNormInf() 6 0.0001055 0.0001055 0.0001055 0.03% Castro::finalize_advance() 5 9.45e-05 9.45e-05 9.45e-05 0.03% FabArrayBase::FB::FB() 26 6.069e-05 6.069e-05 6.069e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.826e-05 2.826e-05 2.826e-05 0.01% Castro::swap_state_time_levels() 5 2.757e-05 2.757e-05 2.757e-05 0.01% Amr::writeSmallPlotFile() 1 2.545e-05 2.545e-05 2.545e-05 0.01% Castro::initMFs() 1 2.482e-05 2.482e-05 2.482e-05 0.01% makeSFC 30 2.065e-05 2.065e-05 2.065e-05 0.01% Castro::finalize_do_advance() 5 2.048e-05 2.048e-05 2.048e-05 0.01% Amr::initSubcycle() 1 8.974e-06 8.974e-06 8.974e-06 0.00% DistributionMapping::Distribute() 31 8.496e-06 8.496e-06 8.496e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.336e-06 5.336e-06 5.336e-06 0.00% MLMG::buildFineMask() 6 2.826e-06 2.826e-06 2.826e-06 0.00% Gravity::set_mass_offset() 6 2.633e-06 2.633e-06 2.633e-06 0.00% Castro::retry_advance_ctu() 5 2.14e-06 2.14e-06 2.14e-06 0.00% Castro::FluxRegCrseInit 5 1.635e-06 1.635e-06 1.635e-06 0.00% Castro::FluxRegFineAdd() 5 1.238e-06 1.238e-06 1.238e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.235e-06 1.235e-06 1.235e-06 0.00% AmrLevel::AmrLevel() 1 1.16e-06 1.16e-06 1.16e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-4-gef52970b80e3) finalized