Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-39-g50466d11e7b2) initialized Starting run at 20:36:31 UTC on 2022-04-29. Successfully read inputs file ... Castro git describe: 22.04-49-g759123458 AMReX git describe: 22.04-39-g50466d11e Microphysics git describe: 22.04-3-g3c498521 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.0383166 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022197077 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.044900717 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.04727075 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.061920378 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.057382051 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.056870702 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.035176502 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.048182409 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049000898 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.055893821 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057348925 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.059411119 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.035373632 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.022122432 seconds Ending run at 20:36:32 UTC on 2022-04-29. Run time = 0.738373849 Run time without initialization = 0.631347904 Average number of zones advanced per microsecond: 4.152 Average number of zones advanced per microsecond per rank: 4.152 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.7384 ... 0.7384 ... 0.7384 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1739 0.1739 0.1739 23.55% VisMF::Write(FabArray) 11 0.1468 0.1468 0.1468 19.87% MLCellLinOp::applyBC() 4379 0.07136 0.07136 0.07136 9.66% MLPoisson::Fsmooth() 3240 0.05909 0.05909 0.05909 8.00% StateData::FillBoundary(geom) 328 0.02289 0.02289 0.02289 3.10% MLCGSolver::bicgstab 81 0.02068 0.02068 0.02068 2.80% MultiFab::Dot() 1100 0.02 0.02 0.02 2.71% Castro::normalize_species() 62 0.01852 0.01852 0.01852 2.51% FillBoundary_nowait() 3974 0.01412 0.01412 0.01412 1.91% Castro::computeTemp() 63 0.01407 0.01407 0.01407 1.91% Castro::enforce_min_density() 62 0.01338 0.01338 0.01338 1.81% FabArray::setVal() 1135 0.01296 0.01296 0.01296 1.76% FabArray::ParallelCopy_nowait() 851 0.01288 0.01288 0.01288 1.74% MultiFab::LinComb() 1566 0.01202 0.01202 0.01202 1.63% StateDataPhysBCFunct::() 41 0.01119 0.01119 0.01119 1.51% MLPoisson::Fapply() 1128 0.01029 0.01029 0.01029 1.39% MLCellLinOp::defineAuxData() 11 0.009615 0.009615 0.009615 1.30% Gravity::fill_multipole_BCs() 11 0.008215 0.008215 0.008215 1.11% MLMG::addInterpCorrection() 405 0.006529 0.006529 0.006529 0.88% Castro::estTimeStep() 21 0.00624 0.00624 0.00624 0.85% amrex::average_down 405 0.006066 0.006066 0.006066 0.82% MultiFab::Xpay() 578 0.005861 0.005861 0.005861 0.79% Castro::reset_internal_energy(MultiFab) 63 0.004745 0.004745 0.004745 0.64% Castro::do_advance_ctu() 10 0.00415 0.00415 0.00415 0.56% Amr::checkPoint() 3 0.004085 0.004085 0.004085 0.55% BndryData::define() 11 0.003576 0.003576 0.003576 0.48% Castro::construct_new_gravity_source() 10 0.002544 0.002544 0.002544 0.34% Amr::writePlotFile() 2 0.002411 0.002411 0.002411 0.33% Gravity::get_new_grav_vector() 11 0.00192 0.00192 0.00192 0.26% Castro::enforce_speed_limit() 62 0.0019 0.0019 0.0019 0.26% MLMG::ResNormInf() 92 0.001876 0.001876 0.001876 0.25% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.25% Castro::expand_state() 10 0.001729 0.001729 0.001729 0.23% Gravity::get_old_grav_vector() 10 0.001716 0.001716 0.001716 0.23% MLMG::oneIter() 81 0.001658 0.001658 0.001658 0.22% Castro::construct_old_gravity_source() 10 0.001641 0.001641 0.001641 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001616 0.001616 0.001616 0.22% Castro::reset_internal_energy(Fab) 504 0.001536 0.001536 0.001536 0.21% Gravity::actual_solve_with_mlmg() 11 0.001425 0.001425 0.001425 0.19% FabArray::mult() 43 0.001324 0.001324 0.001324 0.18% MLCellLinOp::setLevelBC() 11 0.001302 0.001302 0.001302 0.18% FabArray::setDomainBndry() 41 0.001278 0.001278 0.001278 0.17% MultiFab::contains_nan() 20 0.001163 0.001163 0.001163 0.16% Castro::initData() 1 0.001134 0.001134 0.001134 0.15% MLCellLinOp::prepareForSolve() 11 0.001068 0.001068 0.001068 0.14% MLCellLinOp::smooth() 1620 0.001036 0.001036 0.001036 0.14% MLCellLinOp::compGrad() 11 0.0008904 0.0008904 0.0008904 0.12% FabArrayBase::getCPC() 1313 0.000842 0.000842 0.000842 0.11% FabArray::FillBoundary() 3974 0.0008134 0.0008134 0.0008134 0.11% MLMG::prepareForSolve() 11 0.0007741 0.0007741 0.0007741 0.10% FabArrayBase::CPC::define() 454 0.0007027 0.0007027 0.0007027 0.10% FabArrayBase::getFB() 3974 0.0007015 0.0007015 0.0007015 0.10% MLCellLinOp::apply() 1128 0.0004819 0.0004819 0.0004819 0.07% Amr::InitAmr() 1 0.0004227 0.0004227 0.0004227 0.06% CGSolver::sxay() 1566 0.0004126 0.0004126 0.0004126 0.06% Gravity::update_max_rhs() 11 0.0004042 0.0004042 0.0004042 0.05% MLMG::mgVcycle() 81 0.0003691 0.0003691 0.0003691 0.05% Gravity::solve_for_phi() 10 0.000369 0.000369 0.000369 0.05% MLLinOp::defineGrids() 11 0.0003584 0.0003584 0.0003584 0.05% MLCGSolver::ParallelAllReduce 1495 0.0003157 0.0003157 0.0003157 0.04% FabArray::ParallelCopy() 851 0.0002824 0.0002824 0.0002824 0.04% main() 1 0.0002752 0.0002752 0.0002752 0.04% MultiFab::Copy() 11 0.0002607 0.0002607 0.0002607 0.04% FillPatchIterator::Initialize 41 0.0002605 0.0002605 0.0002605 0.04% MultiFab::max() 11 0.0002533 0.0002533 0.0002533 0.03% MLCellLinOp::correctionResidual() 486 0.000221 0.000221 0.000221 0.03% Castro::construct_new_gravity() 10 0.0002178 0.0002178 0.0002178 0.03% Amr::coarseTimeStep() 10 0.0002016 0.0002016 0.0002016 0.03% MLMG::MLRhsNormInf() 11 0.0001964 0.0001964 0.0001964 0.03% Amr::timeStep() 10 0.0001924 0.0001924 0.0001924 0.03% MLCellLinOp::defineBC() 11 0.0001873 0.0001873 0.0001873 0.03% Castro::subcycle_advance_ctu() 10 0.0001797 0.0001797 0.0001797 0.02% StateData::checkPoint() 12 0.0001302 0.0001302 0.0001302 0.02% MLMG:computeResOfCorrection() 405 0.0001282 0.0001282 0.0001282 0.02% Castro::Castro() 1 0.0001109 0.0001109 0.0001109 0.02% MLMG::actualBottomSolve() 81 0.0001105 0.0001105 0.0001105 0.01% Castro::advance() 10 9.251e-05 9.251e-05 9.251e-05 0.01% FabArrayBase::FB::FB() 56 8.628e-05 8.628e-05 8.628e-05 0.01% MLMG::mgVcycle_down::0 81 8.481e-05 8.481e-05 8.481e-05 0.01% MLMG::solve() 11 7.957e-05 7.957e-05 7.957e-05 0.01% Castro::initialize_advance() 10 7.713e-05 7.713e-05 7.713e-05 0.01% MLMG::mgVcycle_down::2 81 7.632e-05 7.632e-05 7.632e-05 0.01% MLMG::mgVcycle_down::1 81 7.507e-05 7.507e-05 7.507e-05 0.01% AmrLevel::checkPoint() 3 7.343e-05 7.343e-05 7.343e-05 0.01% MLMG::mgVcycle_down::3 81 7.32e-05 7.32e-05 7.32e-05 0.01% Castro::clean_state() 62 7.263e-05 7.263e-05 7.263e-05 0.01% MLMG::mgVcycle_down::4 81 7.144e-05 7.144e-05 7.144e-05 0.01% MLMG::mgVcycle_up::4 81 5.734e-05 5.734e-05 5.734e-05 0.01% Castro::initialize_do_advance() 10 5.526e-05 5.526e-05 5.526e-05 0.01% Castro::finalize_advance() 10 5.193e-05 5.193e-05 5.193e-05 0.01% MLCellLinOp::solutionResidual() 92 4.913e-05 4.913e-05 4.913e-05 0.01% MLMG::mgVcycle_up::0 81 4.824e-05 4.824e-05 4.824e-05 0.01% MLMG::mgVcycle_up::1 81 4.484e-05 4.484e-05 4.484e-05 0.01% MLMG::mgVcycle_up::2 81 4.472e-05 4.472e-05 4.472e-05 0.01% MLMG::mgVcycle_up::3 81 4.453e-05 4.453e-05 4.453e-05 0.01% StateData::define() 4 4.149e-05 4.149e-05 4.149e-05 0.01% Castro::enforce_consistent_e() 1 3.853e-05 3.853e-05 3.853e-05 0.01% Castro::construct_new_source() 50 3.828e-05 3.828e-05 3.828e-05 0.01% Castro::swap_state_time_levels() 10 3.818e-05 3.818e-05 3.818e-05 0.01% Castro::finalize_do_advance() 10 3.721e-05 3.721e-05 3.721e-05 0.01% MLMG::mgVcycle_bottom 81 3.531e-05 3.531e-05 3.531e-05 0.00% MLMG::computeResidual() 81 3.264e-05 3.264e-05 3.264e-05 0.00% Gravity::actual_multilevel_solve() 1 3.162e-05 3.162e-05 3.162e-05 0.00% Castro::post_timestep() 10 2.986e-05 2.986e-05 2.986e-05 0.00% FillPatchSingleLevel 41 2.885e-05 2.885e-05 2.885e-05 0.00% Amr::writeSmallPlotFile() 1 2.644e-05 2.644e-05 2.644e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.563e-05 2.563e-05 2.563e-05 0.00% makeSFC 55 2.445e-05 2.445e-05 2.445e-05 0.00% MLLinOp::define() 11 2.416e-05 2.416e-05 2.416e-05 0.00% Castro::initMFs() 1 2.329e-05 2.329e-05 2.329e-05 0.00% MLPoisson::define() 11 2.307e-05 2.307e-05 2.307e-05 0.00% Amr::FinalizeInit() 1 2.111e-05 2.111e-05 2.111e-05 0.00% Castro::buildMetrics() 1 2.007e-05 2.007e-05 2.007e-05 0.00% Castro::construct_old_source() 50 1.87e-05 1.87e-05 1.87e-05 0.00% Castro::do_new_sources() 10 1.702e-05 1.702e-05 1.702e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.676e-05 1.676e-05 1.676e-05 0.00% Amr::defBaseLevel() 1 1.603e-05 1.603e-05 1.603e-05 0.00% Castro::do_old_sources() 10 1.569e-05 1.569e-05 1.569e-05 0.00% DistributionMapping::Distribute() 56 1.433e-05 1.433e-05 1.433e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.427e-05 1.427e-05 1.427e-05 0.00% Castro::apply_source_to_state() 20 1.121e-05 1.121e-05 1.121e-05 0.00% Castro::check_for_nan() 20 1.047e-05 1.047e-05 1.047e-05 0.00% AmrLevel::AmrLevel(dm) 1 9.722e-06 9.722e-06 9.722e-06 0.00% Amr::initSubcycle() 1 9.668e-06 9.668e-06 9.668e-06 0.00% Gravity::swapTimeLevels() 10 9.186e-06 9.186e-06 9.186e-06 0.00% Castro::construct_old_gravity() 10 9.121e-06 9.121e-06 9.121e-06 0.00% MLPoisson::prepareForSolve() 11 8.629e-06 8.629e-06 8.629e-06 0.00% MLMG::computeMLResidual() 11 6.611e-06 6.611e-06 6.611e-06 0.00% Castro::computeNewDt() 9 6.347e-06 6.347e-06 6.347e-06 0.00% Amr::InitializeInit() 1 5.876e-06 5.876e-06 5.876e-06 0.00% MLMG::getGradSolution() 11 5.76e-06 5.76e-06 5.76e-06 0.00% MLMG::buildFineMask() 11 5.756e-06 5.756e-06 5.756e-06 0.00% AmrLevel::checkPointPost() 3 5.263e-06 5.263e-06 5.263e-06 0.00% MLMG::MLResNormInf() 11 4.468e-06 4.468e-06 4.468e-06 0.00% Castro::retry_advance_ctu() 10 3.992e-06 3.992e-06 3.992e-06 0.00% Gravity::set_mass_offset() 11 3.805e-06 3.805e-06 3.805e-06 0.00% Castro::post_init() 1 3.613e-06 3.613e-06 3.613e-06 0.00% Castro::create_source_corrector() 10 3.5e-06 3.5e-06 3.5e-06 0.00% Castro::FluxRegCrseInit 10 3.096e-06 3.096e-06 3.096e-06 0.00% Amr::init() 1 2.733e-06 2.733e-06 2.733e-06 0.00% Castro::computeInitialDt() 2 2.733e-06 2.733e-06 2.733e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.621e-06 2.621e-06 2.621e-06 0.00% Castro::FluxRegFineAdd() 10 2.585e-06 2.585e-06 2.585e-06 0.00% AmrLevel::checkPointPre() 3 2.231e-06 2.231e-06 2.231e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.968e-06 1.968e-06 1.968e-06 0.00% Castro::post_regrid() 1 1.167e-06 1.167e-06 1.167e-06 0.00% Amr::initialInit() 1 9.01e-07 9.01e-07 9.01e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.7384 0.7384 0.7384 100.00% Amr::coarseTimeStep() 10 0.609 0.609 0.609 82.48% Amr::timeStep() 10 0.5346 0.5346 0.5346 72.40% Castro::advance() 10 0.5254 0.5254 0.5254 71.15% Castro::subcycle_advance_ctu() 10 0.5126 0.5126 0.5126 69.42% Castro::do_advance_ctu() 10 0.5124 0.5124 0.5124 69.40% Gravity::solve_phi_with_mlmg() 11 0.2839 0.2839 0.2839 38.44% Gravity::actual_solve_with_mlmg() 11 0.2754 0.2754 0.2754 37.30% Castro::construct_new_gravity() 10 0.2601 0.2601 0.2601 35.23% MLMG::solve() 11 0.2552 0.2552 0.2552 34.57% Gravity::solve_for_phi() 10 0.2457 0.2457 0.2457 33.27% MLMG::oneIter() 81 0.2417 0.2417 0.2417 32.73% MLMG::mgVcycle() 81 0.2401 0.2401 0.2401 32.51% Castro::construct_ctu_hydro_source() 10 0.1739 0.1739 0.1739 23.55% VisMF::Write(FabArray) 11 0.1468 0.1468 0.1468 19.87% MLCellLinOp::smooth() 1620 0.1247 0.1247 0.1247 16.89% Amr::checkPoint() 3 0.109 0.109 0.109 14.76% Amr::init() 1 0.1065 0.1065 0.1065 14.42% AmrLevel::checkPoint() 3 0.1049 0.1049 0.1049 14.21% StateData::checkPoint() 12 0.1048 0.1048 0.1048 14.20% MLCellLinOp::applyBC() 4379 0.08708 0.08708 0.08708 11.79% MLMG::mgVcycle_bottom 81 0.07141 0.07141 0.07141 9.67% MLMG::actualBottomSolve() 81 0.07138 0.07138 0.07138 9.67% MLCGSolver::bicgstab 81 0.07065 0.07065 0.07065 9.57% MLPoisson::Fsmooth() 3240 0.05909 0.05909 0.05909 8.00% Castro::clean_state() 62 0.0535 0.0535 0.0535 7.25% Amr::initialInit() 1 0.04582 0.04582 0.04582 6.21% Amr::writePlotFile() 2 0.04445 0.04445 0.04445 6.02% Amr::FinalizeInit() 1 0.0418 0.0418 0.0418 5.66% Castro::post_init() 1 0.0404 0.0404 0.0404 5.47% FillPatchIterator::Initialize 41 0.03961 0.03961 0.03961 5.36% Gravity::multilevel_solve_for_new_phi() 1 0.03862 0.03862 0.03862 5.23% Gravity::actual_multilevel_solve() 1 0.03861 0.03861 0.03861 5.23% FillPatchSingleLevel 41 0.03807 0.03807 0.03807 5.16% StateDataPhysBCFunct::() 41 0.03408 0.03408 0.03408 4.62% MLMG::mgVcycle_down::0 81 0.03386 0.03386 0.03386 4.59% MLCellLinOp::apply() 1128 0.03282 0.03282 0.03282 4.45% MLMG::mgVcycle_up::0 81 0.02904 0.02904 0.02904 3.93% StateData::FillBoundary(geom) 328 0.02289 0.02289 0.02289 3.10% Castro::computeTemp() 63 0.02035 0.02035 0.02035 2.76% Castro::initialize_do_advance() 10 0.02029 0.02029 0.02029 2.75% MultiFab::Dot() 1100 0.02 0.02 0.02 2.71% MLCellLinOp::correctionResidual() 486 0.01916 0.01916 0.01916 2.60% Castro::normalize_species() 62 0.01852 0.01852 0.01852 2.51% MLMG:computeResOfCorrection() 405 0.01663 0.01663 0.01663 2.25% MLPoisson::define() 11 0.0161 0.0161 0.0161 2.18% MLMG::mgVcycle_down::1 81 0.01599 0.01599 0.01599 2.17% Gravity::get_new_grav_vector() 11 0.01596 0.01596 0.01596 2.16% FabArray::FillBoundary() 3974 0.01572 0.01572 0.01572 2.13% MLMG::mgVcycle_down::2 81 0.01539 0.01539 0.01539 2.08% FillBoundary_nowait() 3974 0.01491 0.01491 0.01491 2.02% MLMG::mgVcycle_down::3 81 0.01452 0.01452 0.01452 1.97% Castro::construct_old_gravity() 10 0.01434 0.01434 0.01434 1.94% Gravity::get_old_grav_vector() 10 0.01433 0.01433 0.01433 1.94% FabArray::ParallelCopy() 851 0.01406 0.01406 0.01406 1.90% FabArray::ParallelCopy_nowait() 851 0.01378 0.01378 0.01378 1.87% MLMG::mgVcycle_down::4 81 0.01372 0.01372 0.01372 1.86% Castro::enforce_min_density() 62 0.01338 0.01338 0.01338 1.81% FabArray::setVal() 1135 0.01296 0.01296 0.01296 1.76% Castro::initialize_advance() 10 0.0126 0.0126 0.0126 1.71% CGSolver::sxay() 1566 0.01243 0.01243 0.01243 1.68% Castro::do_new_sources() 10 0.01214 0.01214 0.01214 1.64% MultiFab::LinComb() 1566 0.01202 0.01202 0.01202 1.63% MLCGSolver::ParallelAllReduce 1495 0.012 0.012 0.012 1.62% MLMG::mgVcycle_up::2 81 0.01187 0.01187 0.01187 1.61% MLMG::mgVcycle_up::1 81 0.01169 0.01169 0.01169 1.58% MLMG::addInterpCorrection() 405 0.01157 0.01157 0.01157 1.57% MLMG::mgVcycle_up::3 81 0.01114 0.01114 0.01114 1.51% amrex::average_down 405 0.01112 0.01112 0.01112 1.51% MLMG::mgVcycle_up::4 81 0.01105 0.01105 0.01105 1.50% MLCellLinOp::defineAuxData() 11 0.01095 0.01095 0.01095 1.48% Castro::expand_state() 10 0.01077 0.01077 0.01077 1.46% Castro::do_old_sources() 10 0.01055 0.01055 0.01055 1.43% MLPoisson::Fapply() 1128 0.01029 0.01029 0.01029 1.39% Castro::post_timestep() 10 0.009035 0.009035 0.009035 1.22% Gravity::fill_multipole_BCs() 11 0.008215 0.008215 0.008215 1.11% MLCellLinOp::solutionResidual() 92 0.006875 0.006875 0.006875 0.93% Castro::reset_internal_energy(MultiFab) 63 0.006281 0.006281 0.006281 0.85% Castro::estTimeStep() 21 0.00624 0.00624 0.00624 0.85% MLMG::computeResidual() 81 0.005911 0.005911 0.005911 0.80% MultiFab::Xpay() 578 0.005861 0.005861 0.005861 0.79% MLCellLinOp::defineBC() 11 0.004688 0.004688 0.004688 0.63% BndryData::define() 11 0.004501 0.004501 0.004501 0.61% MLMG::prepareForSolve() 11 0.004447 0.004447 0.004447 0.60% Amr::InitializeInit() 1 0.004022 0.004022 0.004022 0.54% Amr::defBaseLevel() 1 0.004016 0.004016 0.004016 0.54% Castro::initData() 1 0.003496 0.003496 0.003496 0.47% Castro::computeNewDt() 9 0.003195 0.003195 0.003195 0.43% Castro::construct_new_source() 50 0.002582 0.002582 0.002582 0.35% Castro::construct_new_gravity_source() 10 0.002544 0.002544 0.002544 0.34% Castro::enforce_speed_limit() 62 0.0019 0.0019 0.0019 0.26% MLMG::ResNormInf() 92 0.001876 0.001876 0.001876 0.25% Castro::apply_source_to_state() 20 0.001822 0.001822 0.001822 0.25% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.25% Castro::construct_old_source() 50 0.00166 0.00166 0.00166 0.22% Castro::construct_old_gravity_source() 10 0.001641 0.001641 0.001641 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001616 0.001616 0.001616 0.22% FabArrayBase::getCPC() 1313 0.001545 0.001545 0.001545 0.21% Castro::reset_internal_energy(Fab) 504 0.001536 0.001536 0.001536 0.21% MLMG::getGradSolution() 11 0.001355 0.001355 0.001355 0.18% MLCellLinOp::compGrad() 11 0.001349 0.001349 0.001349 0.18% FabArray::mult() 43 0.001324 0.001324 0.001324 0.18% MLCellLinOp::setLevelBC() 11 0.001302 0.001302 0.001302 0.18% FabArray::setDomainBndry() 41 0.001278 0.001278 0.001278 0.17% Castro::check_for_nan() 20 0.001174 0.001174 0.001174 0.16% MultiFab::contains_nan() 20 0.001163 0.001163 0.001163 0.16% Castro::post_regrid() 1 0.001147 0.001147 0.001147 0.16% MLPoisson::prepareForSolve() 11 0.001077 0.001077 0.001077 0.15% MLCellLinOp::prepareForSolve() 11 0.001068 0.001068 0.001068 0.14% MLMG::computeMLResidual() 11 0.001004 0.001004 0.001004 0.14% Gravity::update_max_rhs() 11 0.0008035 0.0008035 0.0008035 0.11% FabArrayBase::getFB() 3974 0.0007878 0.0007878 0.0007878 0.11% FabArrayBase::CPC::define() 454 0.0007027 0.0007027 0.0007027 0.10% Castro::computeInitialDt() 2 0.0006037 0.0006037 0.0006037 0.08% Castro::Castro() 1 0.0004484 0.0004484 0.0004484 0.06% MLLinOp::define() 11 0.0004363 0.0004363 0.0004363 0.06% Gravity::swapTimeLevels() 10 0.000436 0.000436 0.000436 0.06% Amr::InitAmr() 1 0.0004324 0.0004324 0.0004324 0.06% MLLinOp::defineGrids() 11 0.0004122 0.0004122 0.0004122 0.06% MultiFab::Copy() 11 0.0002607 0.0002607 0.0002607 0.04% MultiFab::max() 11 0.0002533 0.0002533 0.0002533 0.03% MLMG::MLResNormInf() 11 0.0002522 0.0002522 0.0002522 0.03% MLMG::MLRhsNormInf() 11 0.0001964 0.0001964 0.0001964 0.03% Castro::buildMetrics() 1 0.0001513 0.0001513 0.0001513 0.02% FabArrayBase::FB::FB() 56 8.628e-05 8.628e-05 8.628e-05 0.01% Castro::finalize_advance() 10 5.761e-05 5.761e-05 5.761e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.179e-05 5.179e-05 5.179e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.121e-05 5.121e-05 5.121e-05 0.01% StateData::define() 4 4.149e-05 4.149e-05 4.149e-05 0.01% Castro::enforce_consistent_e() 1 3.853e-05 3.853e-05 3.853e-05 0.01% Castro::swap_state_time_levels() 10 3.818e-05 3.818e-05 3.818e-05 0.01% makeSFC 55 3.753e-05 3.753e-05 3.753e-05 0.01% Castro::finalize_do_advance() 10 3.721e-05 3.721e-05 3.721e-05 0.01% Amr::writeSmallPlotFile() 1 2.644e-05 2.644e-05 2.644e-05 0.00% Castro::initMFs() 1 2.329e-05 2.329e-05 2.329e-05 0.00% DistributionMapping::Distribute() 56 1.433e-05 1.433e-05 1.433e-05 0.00% Amr::initSubcycle() 1 9.668e-06 9.668e-06 9.668e-06 0.00% MLMG::buildFineMask() 11 5.756e-06 5.756e-06 5.756e-06 0.00% AmrLevel::checkPointPost() 3 5.263e-06 5.263e-06 5.263e-06 0.00% Castro::retry_advance_ctu() 10 3.992e-06 3.992e-06 3.992e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.876e-06 3.876e-06 3.876e-06 0.00% Gravity::set_mass_offset() 11 3.805e-06 3.805e-06 3.805e-06 0.00% Castro::create_source_corrector() 10 3.5e-06 3.5e-06 3.5e-06 0.00% Castro::FluxRegCrseInit 10 3.096e-06 3.096e-06 3.096e-06 0.00% Castro::FluxRegFineAdd() 10 2.585e-06 2.585e-06 2.585e-06 0.00% AmrLevel::checkPointPre() 3 2.231e-06 2.231e-06 2.231e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.968e-06 1.968e-06 1.968e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-39-g50466d11e7b2) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-39-g50466d11e7b2) initialized Starting run at 20:36:33 UTC on 2022-04-29. Successfully read inputs file ... Castro git describe: 22.04-49-g759123458 AMReX git describe: 22.04-39-g50466d11e Microphysics git describe: 22.04-3-g3c498521 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.410483748 Restart time = 0.045639117 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.047894172 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.044498581 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058893104 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059987369 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.05982023 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.051356086 seconds Ending run at 20:36:33 UTC on 2022-04-29. Run time = 0.369024395 Run time without initialization = 0.322821594 Average number of zones advanced per microsecond: 4.060 Average number of zones advanced per microsecond per rank: 4.060 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3691 ... 0.3691 ... 0.3691 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0841 0.0841 0.0841 22.78% VisMF::Read() 3 0.0387 0.0387 0.0387 10.49% MLCellLinOp::applyBC() 1946 0.03141 0.03141 0.03141 8.51% Amr::writePlotFile() 1 0.02868 0.02868 0.02868 7.77% MLPoisson::Fsmooth() 1440 0.02554 0.02554 0.02554 6.92% VisMF::Write(FabArray) 1 0.02277 0.02277 0.02277 6.17% StateData::FillBoundary(geom) 160 0.01108 0.01108 0.01108 3.00% MLCGSolver::bicgstab 36 0.009033 0.009033 0.009033 2.45% MultiFab::Dot() 484 0.008761 0.008761 0.008761 2.37% Castro::normalize_species() 30 0.008669 0.008669 0.008669 2.35% Castro::computeTemp() 30 0.008548 0.008548 0.008548 2.32% Castro::enforce_min_density() 30 0.006897 0.006897 0.006897 1.87% FillBoundary_nowait() 1766 0.006267 0.006267 0.006267 1.70% FabArray::setVal() 537 0.006239 0.006239 0.006239 1.69% FabArray::ParallelCopy_nowait() 380 0.005894 0.005894 0.005894 1.60% StateDataPhysBCFunct::() 20 0.005638 0.005638 0.005638 1.53% MultiFab::LinComb() 690 0.005218 0.005218 0.005218 1.41% MLCellLinOp::defineAuxData() 6 0.005209 0.005209 0.005209 1.41% Gravity::fill_multipole_BCs() 6 0.004619 0.004619 0.004619 1.25% MLPoisson::Fapply() 500 0.004549 0.004549 0.004549 1.23% Amr::restart() 1 0.003009 0.003009 0.003009 0.82% Castro::estTimeStep() 10 0.002985 0.002985 0.002985 0.81% MLMG::addInterpCorrection() 180 0.002854 0.002854 0.002854 0.77% amrex::average_down 180 0.002657 0.002657 0.002657 0.72% MultiFab::Xpay() 258 0.002604 0.002604 0.002604 0.71% Castro::do_advance_ctu() 5 0.002159 0.002159 0.002159 0.58% Castro::reset_internal_energy(MultiFab) 30 0.001969 0.001969 0.001969 0.53% BndryData::define() 6 0.001945 0.001945 0.001945 0.53% Castro::construct_new_gravity_source() 5 0.001249 0.001249 0.001249 0.34% MultiFab::Saxpy() 10 0.0009234 0.0009234 0.0009234 0.25% Gravity::get_old_grav_vector() 5 0.0008788 0.0008788 0.0008788 0.24% Castro::expand_state() 5 0.0008725 0.0008725 0.0008725 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008724 0.0008724 0.0008724 0.24% Gravity::get_new_grav_vector() 5 0.0008605 0.0008605 0.0008605 0.23% MLMG::ResNormInf() 42 0.0008489 0.0008489 0.0008489 0.23% Castro::construct_old_gravity_source() 5 0.0008134 0.0008134 0.0008134 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007298 0.0007298 0.0007298 0.20% MLMG::oneIter() 36 0.0007255 0.0007255 0.0007255 0.20% MLCellLinOp::setLevelBC() 6 0.0007005 0.0007005 0.0007005 0.19% Castro::reset_internal_energy(Fab) 240 0.000697 0.000697 0.000697 0.19% Castro::enforce_speed_limit() 30 0.0006766 0.0006766 0.0006766 0.18% FabArray::mult() 22 0.0006493 0.0006493 0.0006493 0.18% FabArray::setDomainBndry() 20 0.0006288 0.0006288 0.0006288 0.17% MultiFab::contains_nan() 10 0.000589 0.000589 0.000589 0.16% MLCellLinOp::prepareForSolve() 6 0.000573 0.000573 0.000573 0.16% MLCellLinOp::smooth() 720 0.0005381 0.0005381 0.0005381 0.15% MLCellLinOp::compGrad() 6 0.0004762 0.0004762 0.0004762 0.13% MLMG::prepareForSolve() 6 0.0004277 0.0004277 0.0004277 0.12% FabArrayBase::CPC::define() 244 0.0004215 0.0004215 0.0004215 0.11% FabArrayBase::getCPC() 632 0.0004037 0.0004037 0.0004037 0.11% FabArray::FillBoundary() 1766 0.0003891 0.0003891 0.0003891 0.11% Amr::InitAmr() 1 0.0003782 0.0003782 0.0003782 0.10% FabArrayBase::getFB() 1766 0.0002967 0.0002967 0.0002967 0.08% main() 1 0.0002717 0.0002717 0.0002717 0.07% MLCellLinOp::apply() 500 0.0002426 0.0002426 0.0002426 0.07% Gravity::update_max_rhs() 6 0.0002231 0.0002231 0.0002231 0.06% CGSolver::sxay() 690 0.0002175 0.0002175 0.0002175 0.06% Gravity::solve_for_phi() 5 0.0001927 0.0001927 0.0001927 0.05% MLLinOp::defineGrids() 6 0.0001923 0.0001923 0.0001923 0.05% MLMG::mgVcycle() 36 0.0001729 0.0001729 0.0001729 0.05% MLCGSolver::ParallelAllReduce 659 0.000169 0.000169 0.000169 0.05% FabArray::ParallelCopy() 380 0.0001565 0.0001565 0.0001565 0.04% Castro::subcycle_advance_ctu() 5 0.0001396 0.0001396 0.0001396 0.04% MultiFab::Copy() 6 0.0001393 0.0001393 0.0001393 0.04% MultiFab::max() 6 0.0001342 0.0001342 0.0001342 0.04% FillPatchIterator::Initialize 20 0.0001214 0.0001214 0.0001214 0.03% Amr::coarseTimeStep() 5 0.0001078 0.0001078 0.0001078 0.03% Amr::timeStep() 5 0.0001073 0.0001073 0.0001073 0.03% MLMG::MLRhsNormInf() 6 0.0001064 0.0001064 0.0001064 0.03% Castro::construct_new_gravity() 5 0.0001059 0.0001059 0.0001059 0.03% MLCellLinOp::correctionResidual() 216 0.0001047 0.0001047 0.0001047 0.03% MLCellLinOp::defineBC() 6 0.0001014 0.0001014 0.0001014 0.03% Castro::advance() 5 8.394e-05 8.394e-05 8.394e-05 0.02% StateData::restartDoit() 4 7.227e-05 7.227e-05 7.227e-05 0.02% AmrLevel::restart() 1 6.918e-05 6.918e-05 6.918e-05 0.02% MLMG:computeResOfCorrection() 180 6.422e-05 6.422e-05 6.422e-05 0.02% FabArrayBase::FB::FB() 26 6.005e-05 6.005e-05 6.005e-05 0.02% MLMG::actualBottomSolve() 36 4.658e-05 4.658e-05 4.658e-05 0.01% Castro::initialize_do_advance() 5 4.084e-05 4.084e-05 4.084e-05 0.01% Castro::create_source_corrector() 5 4.075e-05 4.075e-05 4.075e-05 0.01% Castro::construct_new_source() 25 4.008e-05 4.008e-05 4.008e-05 0.01% MLMG::solve() 6 3.946e-05 3.946e-05 3.946e-05 0.01% Castro::clean_state() 30 3.808e-05 3.808e-05 3.808e-05 0.01% MLMG::mgVcycle_down::0 36 3.707e-05 3.707e-05 3.707e-05 0.01% MLMG::mgVcycle_down::1 36 3.59e-05 3.59e-05 3.59e-05 0.01% Castro::initialize_advance() 5 3.543e-05 3.543e-05 3.543e-05 0.01% MLMG::mgVcycle_down::2 36 3.467e-05 3.467e-05 3.467e-05 0.01% MLMG::mgVcycle_down::4 36 3.397e-05 3.397e-05 3.397e-05 0.01% Castro::post_restart() 1 3.239e-05 3.239e-05 3.239e-05 0.01% MLMG::mgVcycle_down::3 36 3.233e-05 3.233e-05 3.233e-05 0.01% Castro::construct_old_source() 25 3.127e-05 3.127e-05 3.127e-05 0.01% Castro::buildMetrics() 1 3.047e-05 3.047e-05 3.047e-05 0.01% MLMG::mgVcycle_up::4 36 2.918e-05 2.918e-05 2.918e-05 0.01% Amr::writeSmallPlotFile() 1 2.903e-05 2.903e-05 2.903e-05 0.01% Gravity::actual_multilevel_solve() 1 2.844e-05 2.844e-05 2.844e-05 0.01% Castro::swap_state_time_levels() 5 2.632e-05 2.632e-05 2.632e-05 0.01% Castro::finalize_advance() 5 2.515e-05 2.515e-05 2.515e-05 0.01% Castro::initMFs() 1 2.512e-05 2.512e-05 2.512e-05 0.01% MLMG::mgVcycle_up::0 36 2.305e-05 2.305e-05 2.305e-05 0.01% MLCellLinOp::solutionResidual() 42 2.293e-05 2.293e-05 2.293e-05 0.01% MLMG::mgVcycle_up::3 36 2.24e-05 2.24e-05 2.24e-05 0.01% MLMG::mgVcycle_up::2 36 2.175e-05 2.175e-05 2.175e-05 0.01% MLLinOp::define() 6 2.166e-05 2.166e-05 2.166e-05 0.01% MLMG::mgVcycle_up::1 36 2.137e-05 2.137e-05 2.137e-05 0.01% Castro::finalize_do_advance() 5 1.902e-05 1.902e-05 1.902e-05 0.01% MLMG::mgVcycle_bottom 36 1.813e-05 1.813e-05 1.813e-05 0.00% Amr::initSubcycle() 1 1.733e-05 1.733e-05 1.733e-05 0.00% MLMG::computeResidual() 36 1.608e-05 1.608e-05 1.608e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.573e-05 1.573e-05 1.573e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.484e-05 1.484e-05 1.484e-05 0.00% FillPatchSingleLevel 20 1.483e-05 1.483e-05 1.483e-05 0.00% MLPoisson::define() 6 1.439e-05 1.439e-05 1.439e-05 0.00% makeSFC 30 1.366e-05 1.366e-05 1.366e-05 0.00% DistributionMapping::Distribute() 31 1.258e-05 1.258e-05 1.258e-05 0.00% Castro::do_new_sources() 5 9.364e-06 9.364e-06 9.364e-06 0.00% Castro::do_old_sources() 5 8.487e-06 8.487e-06 8.487e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.781e-06 7.781e-06 7.781e-06 0.00% Castro::apply_source_to_state() 10 6.422e-06 6.422e-06 6.422e-06 0.00% Castro::construct_old_gravity() 5 6.023e-06 6.023e-06 6.023e-06 0.00% Castro::check_for_nan() 10 5.907e-06 5.907e-06 5.907e-06 0.00% Castro::post_timestep() 5 4.697e-06 4.697e-06 4.697e-06 0.00% MLPoisson::prepareForSolve() 6 4.447e-06 4.447e-06 4.447e-06 0.00% Gravity::swapTimeLevels() 5 4.376e-06 4.376e-06 4.376e-06 0.00% MLMG::buildFineMask() 6 3.563e-06 3.563e-06 3.563e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.463e-06 3.463e-06 3.463e-06 0.00% MLMG::getGradSolution() 6 3.261e-06 3.261e-06 3.261e-06 0.00% MLMG::computeMLResidual() 6 3.18e-06 3.18e-06 3.18e-06 0.00% Castro::computeNewDt() 5 3.055e-06 3.055e-06 3.055e-06 0.00% MLMG::MLResNormInf() 6 2.553e-06 2.553e-06 2.553e-06 0.00% Castro::retry_advance_ctu() 5 2.377e-06 2.377e-06 2.377e-06 0.00% Gravity::set_mass_offset() 6 2.27e-06 2.27e-06 2.27e-06 0.00% Castro::FluxRegCrseInit 5 1.688e-06 1.688e-06 1.688e-06 0.00% Castro::FluxRegFineAdd() 5 1.352e-06 1.352e-06 1.352e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.276e-06 1.276e-06 1.276e-06 0.00% Amr::init() 1 9.83e-07 9.83e-07 9.83e-07 0.00% AmrLevel::AmrLevel() 1 8.47e-07 8.47e-07 8.47e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.369 0.369 0.369 100.00% Amr::coarseTimeStep() 5 0.2712 0.2712 0.2712 73.48% Amr::timeStep() 5 0.2693 0.2693 0.2693 72.97% Castro::advance() 5 0.2642 0.2642 0.2642 71.59% Castro::subcycle_advance_ctu() 5 0.2576 0.2576 0.2576 69.80% Castro::do_advance_ctu() 5 0.2575 0.2575 0.2575 69.76% Castro::construct_new_gravity() 5 0.1325 0.1325 0.1325 35.89% Gravity::solve_phi_with_mlmg() 6 0.1282 0.1282 0.1282 34.74% Gravity::solve_for_phi() 5 0.125 0.125 0.125 33.87% Gravity::actual_solve_with_mlmg() 6 0.1235 0.1235 0.1235 33.46% MLMG::solve() 6 0.1125 0.1125 0.1125 30.48% MLMG::oneIter() 36 0.1059 0.1059 0.1059 28.70% MLMG::mgVcycle() 36 0.1052 0.1052 0.1052 28.50% Castro::construct_ctu_hydro_source() 5 0.08406 0.08406 0.08406 22.78% MLCellLinOp::smooth() 720 0.05444 0.05444 0.05444 14.75% Amr::writePlotFile() 1 0.05146 0.05146 0.05146 13.94% Amr::init() 1 0.04569 0.04569 0.04569 12.38% Amr::restart() 1 0.04569 0.04569 0.04569 12.38% AmrLevel::restart() 1 0.03891 0.03891 0.03891 10.54% StateData::restartDoit() 4 0.03883 0.03883 0.03883 10.52% VisMF::Read() 3 0.0387 0.0387 0.0387 10.49% MLCellLinOp::applyBC() 1946 0.03842 0.03842 0.03842 10.41% MLMG::mgVcycle_bottom 36 0.03134 0.03134 0.03134 8.49% MLMG::actualBottomSolve() 36 0.03132 0.03132 0.03132 8.49% MLCGSolver::bicgstab 36 0.031 0.031 0.031 8.40% Castro::clean_state() 30 0.02749 0.02749 0.02749 7.45% MLPoisson::Fsmooth() 1440 0.02554 0.02554 0.02554 6.92% VisMF::Write(FabArray) 1 0.02277 0.02277 0.02277 6.17% FillPatchIterator::Initialize 20 0.01948 0.01948 0.01948 5.28% FillPatchSingleLevel 20 0.01873 0.01873 0.01873 5.08% StateDataPhysBCFunct::() 20 0.01672 0.01672 0.01672 4.53% MLMG::mgVcycle_down::0 36 0.01478 0.01478 0.01478 4.00% MLCellLinOp::apply() 500 0.01459 0.01459 0.01459 3.95% MLMG::mgVcycle_up::0 36 0.0126 0.0126 0.0126 3.41% Castro::initialize_do_advance() 5 0.01128 0.01128 0.01128 3.06% Castro::computeTemp() 30 0.01121 0.01121 0.01121 3.04% StateData::FillBoundary(geom) 160 0.01108 0.01108 0.01108 3.00% MLPoisson::define() 6 0.008799 0.008799 0.008799 2.38% MultiFab::Dot() 484 0.008761 0.008761 0.008761 2.37% Castro::normalize_species() 30 0.008669 0.008669 0.008669 2.35% MLCellLinOp::correctionResidual() 216 0.008471 0.008471 0.008471 2.30% Gravity::get_new_grav_vector() 5 0.007368 0.007368 0.007368 2.00% MLMG:computeResOfCorrection() 180 0.007355 0.007355 0.007355 1.99% Castro::construct_old_gravity() 5 0.007098 0.007098 0.007098 1.92% Gravity::get_old_grav_vector() 5 0.007092 0.007092 0.007092 1.92% FabArray::FillBoundary() 1766 0.007013 0.007013 0.007013 1.90% MLMG::mgVcycle_down::1 36 0.007009 0.007009 0.007009 1.90% Castro::enforce_min_density() 30 0.006897 0.006897 0.006897 1.87% MLMG::mgVcycle_down::2 36 0.006747 0.006747 0.006747 1.83% FillBoundary_nowait() 1766 0.006624 0.006624 0.006624 1.79% Castro::initialize_advance() 5 0.006471 0.006471 0.006471 1.75% FabArray::ParallelCopy() 380 0.006467 0.006467 0.006467 1.75% MLMG::mgVcycle_down::3 36 0.006384 0.006384 0.006384 1.73% FabArray::ParallelCopy_nowait() 380 0.00631 0.00631 0.00631 1.71% FabArray::setVal() 537 0.006239 0.006239 0.006239 1.69% Castro::do_new_sources() 5 0.006144 0.006144 0.006144 1.66% MLMG::mgVcycle_down::4 36 0.006096 0.006096 0.006096 1.65% MLCellLinOp::defineAuxData() 6 0.005959 0.005959 0.005959 1.61% Castro::expand_state() 5 0.005761 0.005761 0.005761 1.56% CGSolver::sxay() 690 0.005436 0.005436 0.005436 1.47% Castro::do_old_sources() 5 0.005415 0.005415 0.005415 1.47% MLCGSolver::ParallelAllReduce 659 0.005318 0.005318 0.005318 1.44% MultiFab::LinComb() 690 0.005218 0.005218 0.005218 1.41% MLMG::mgVcycle_up::2 36 0.005193 0.005193 0.005193 1.41% MLMG::mgVcycle_up::1 36 0.005109 0.005109 0.005109 1.38% MLMG::addInterpCorrection() 180 0.005082 0.005082 0.005082 1.38% Castro::post_timestep() 5 0.004998 0.004998 0.004998 1.35% amrex::average_down 180 0.004904 0.004904 0.004904 1.33% MLMG::mgVcycle_up::4 36 0.00488 0.00488 0.00488 1.32% MLMG::mgVcycle_up::3 36 0.004879 0.004879 0.004879 1.32% Gravity::fill_multipole_BCs() 6 0.004619 0.004619 0.004619 1.25% MLPoisson::Fapply() 500 0.004549 0.004549 0.004549 1.23% Castro::post_restart() 1 0.00359 0.00359 0.00359 0.97% Gravity::multilevel_solve_for_new_phi() 1 0.003466 0.003466 0.003466 0.94% Gravity::actual_multilevel_solve() 1 0.00345 0.00345 0.00345 0.93% MLCellLinOp::solutionResidual() 42 0.003162 0.003162 0.003162 0.86% Castro::estTimeStep() 10 0.002985 0.002985 0.002985 0.81% Castro::reset_internal_energy(MultiFab) 30 0.002666 0.002666 0.002666 0.72% MLMG::computeResidual() 36 0.002618 0.002618 0.002618 0.71% MultiFab::Xpay() 258 0.002604 0.002604 0.002604 0.71% MLCellLinOp::defineBC() 6 0.002578 0.002578 0.002578 0.70% BndryData::define() 6 0.002477 0.002477 0.002477 0.67% MLMG::prepareForSolve() 6 0.002408 0.002408 0.002408 0.65% Castro::computeNewDt() 5 0.001786 0.001786 0.001786 0.48% Castro::construct_new_source() 25 0.001289 0.001289 0.001289 0.35% Castro::construct_new_gravity_source() 5 0.001249 0.001249 0.001249 0.34% Castro::apply_source_to_state() 10 0.0009299 0.0009299 0.0009299 0.25% MultiFab::Saxpy() 10 0.0009234 0.0009234 0.0009234 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008724 0.0008724 0.0008724 0.24% MLMG::ResNormInf() 42 0.0008489 0.0008489 0.0008489 0.23% Castro::construct_old_source() 25 0.0008446 0.0008446 0.0008446 0.23% FabArrayBase::getCPC() 632 0.0008252 0.0008252 0.0008252 0.22% Castro::construct_old_gravity_source() 5 0.0008134 0.0008134 0.0008134 0.22% MLMG::getGradSolution() 6 0.0007391 0.0007391 0.0007391 0.20% MLCellLinOp::compGrad() 6 0.0007358 0.0007358 0.0007358 0.20% MLCellLinOp::setLevelBC() 6 0.0007005 0.0007005 0.0007005 0.19% Castro::reset_internal_energy(Fab) 240 0.000697 0.000697 0.000697 0.19% Castro::enforce_speed_limit() 30 0.0006766 0.0006766 0.0006766 0.18% FabArray::mult() 22 0.0006493 0.0006493 0.0006493 0.18% FabArray::setDomainBndry() 20 0.0006288 0.0006288 0.0006288 0.17% Castro::check_for_nan() 10 0.0005949 0.0005949 0.0005949 0.16% MultiFab::contains_nan() 10 0.000589 0.000589 0.000589 0.16% MLPoisson::prepareForSolve() 6 0.0005774 0.0005774 0.0005774 0.16% MLCellLinOp::prepareForSolve() 6 0.000573 0.000573 0.000573 0.16% MLMG::computeMLResidual() 6 0.0005634 0.0005634 0.0005634 0.15% Gravity::update_max_rhs() 6 0.0004353 0.0004353 0.0004353 0.12% FabArrayBase::CPC::define() 244 0.0004215 0.0004215 0.0004215 0.11% Amr::InitAmr() 1 0.0003955 0.0003955 0.0003955 0.11% FabArrayBase::getFB() 1766 0.0003567 0.0003567 0.0003567 0.10% MLLinOp::define() 6 0.000248 0.000248 0.000248 0.07% MLLinOp::defineGrids() 6 0.0002263 0.0002263 0.0002263 0.06% Gravity::swapTimeLevels() 5 0.000225 0.000225 0.000225 0.06% Castro::buildMetrics() 1 0.0001582 0.0001582 0.0001582 0.04% MultiFab::Copy() 6 0.0001393 0.0001393 0.0001393 0.04% MLMG::MLResNormInf() 6 0.0001346 0.0001346 0.0001346 0.04% MultiFab::max() 6 0.0001342 0.0001342 0.0001342 0.04% MLMG::MLRhsNormInf() 6 0.0001064 0.0001064 0.0001064 0.03% FabArrayBase::FB::FB() 26 6.005e-05 6.005e-05 6.005e-05 0.02% Castro::create_source_corrector() 5 4.075e-05 4.075e-05 4.075e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.27e-05 3.27e-05 3.27e-05 0.01% Amr::writeSmallPlotFile() 1 2.903e-05 2.903e-05 2.903e-05 0.01% Castro::finalize_advance() 5 2.819e-05 2.819e-05 2.819e-05 0.01% Castro::swap_state_time_levels() 5 2.632e-05 2.632e-05 2.632e-05 0.01% Castro::initMFs() 1 2.512e-05 2.512e-05 2.512e-05 0.01% makeSFC 30 2.492e-05 2.492e-05 2.492e-05 0.01% Castro::finalize_do_advance() 5 1.902e-05 1.902e-05 1.902e-05 0.01% Amr::initSubcycle() 1 1.733e-05 1.733e-05 1.733e-05 0.00% DistributionMapping::Distribute() 31 1.258e-05 1.258e-05 1.258e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.782e-06 4.782e-06 4.782e-06 0.00% MLMG::buildFineMask() 6 3.563e-06 3.563e-06 3.563e-06 0.00% Castro::retry_advance_ctu() 5 2.377e-06 2.377e-06 2.377e-06 0.00% Gravity::set_mass_offset() 6 2.27e-06 2.27e-06 2.27e-06 0.00% Castro::FluxRegCrseInit 5 1.688e-06 1.688e-06 1.688e-06 0.00% Castro::FluxRegFineAdd() 5 1.352e-06 1.352e-06 1.352e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.276e-06 1.276e-06 1.276e-06 0.00% AmrLevel::AmrLevel() 1 8.47e-07 8.47e-07 8.47e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-39-g50466d11e7b2) finalized