Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.03-12-g806f2373b68e) initialized Starting run at 08:18:05 UTC on 2022-03-14. Successfully read inputs file ... Castro git describe: 22.02-5-g5116b7268 AMReX git describe: 22.03-12-g806f2373b Microphysics git describe: 22.03-6-g41c4af20 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.03782799 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022518018 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.107631876 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.048144153 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.049757866 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.057332031 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.059398367 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.034680816 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051651834 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048366208 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.048036338 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057348521 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062504297 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.034711899 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.02171184 seconds Ending run at 08:18:05 UTC on 2022-03-14. Run time = 0.803514019 Run time without initialization = 0.681751259 Average number of zones advanced per microsecond: 3.845 Average number of zones advanced per microsecond per rank: 3.845 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8036 ... 0.8036 ... 0.8036 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1974 0.1974 0.1974 24.56% VisMF::Write(FabArray) 11 0.1446 0.1446 0.1446 18.00% MLCellLinOp::applyBC() 4433 0.09318 0.09318 0.09318 11.60% MLPoisson::Fsmooth() 3280 0.05895 0.05895 0.05895 7.34% FabArray::setVal() 1144 0.02588 0.02588 0.02588 3.22% StateData::FillBoundary(geom) 328 0.02382 0.02382 0.02382 2.96% MLCGSolver::bicgstab 82 0.02139 0.02139 0.02139 2.66% MultiFab::Dot() 1114 0.02104 0.02104 0.02104 2.62% FillBoundary_nowait() 4023 0.01788 0.01788 0.01788 2.23% FabArray::ParallelCopy_nowait() 861 0.0159 0.0159 0.0159 1.98% StateDataPhysBCFunct::() 41 0.015 0.015 0.015 1.87% Castro::computeTemp() 63 0.01343 0.01343 0.01343 1.67% MultiFab::LinComb() 1586 0.01216 0.01216 0.01216 1.51% Gravity::fill_multipole_BCs() 11 0.01193 0.01193 0.01193 1.48% Castro::enforce_min_density() 62 0.01073 0.01073 0.01073 1.34% MLPoisson::Fapply() 1142 0.01042 0.01042 0.01042 1.30% MLCellLinOp::defineAuxData() 11 0.01015 0.01015 0.01015 1.26% MLMG::addInterpCorrection() 410 0.006977 0.006977 0.006977 0.87% amrex::average_down 410 0.00654 0.00654 0.00654 0.81% MultiFab::Xpay() 585 0.005999 0.005999 0.005999 0.75% FabArray::setDomainBndry() 41 0.005526 0.005526 0.005526 0.69% Castro::reset_internal_energy() 63 0.005164 0.005164 0.005164 0.64% Castro::expand_state() 10 0.004747 0.004747 0.004747 0.59% Castro::estTimeStep() 21 0.004595 0.004595 0.004595 0.57% Amr::checkPoint() 3 0.003808 0.003808 0.003808 0.47% BndryData::define() 11 0.003559 0.003559 0.003559 0.44% Castro::do_advance_ctu() 10 0.003217 0.003217 0.003217 0.40% Amr::writePlotFile() 2 0.003058 0.003058 0.003058 0.38% Castro::construct_new_gravity_source() 10 0.003008 0.003008 0.003008 0.37% Castro::enforce_speed_limit() 62 0.002913 0.002913 0.002913 0.36% Gravity::get_new_grav_vector() 11 0.002721 0.002721 0.002721 0.34% Castro::normalize_species() 62 0.002617 0.002617 0.002617 0.33% Castro::construct_old_gravity_source() 10 0.002505 0.002505 0.002505 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002228 0.002228 0.002228 0.28% MLCellLinOp::compGrad() 11 0.001995 0.001995 0.001995 0.25% MLMG::ResNormInf() 93 0.00188 0.00188 0.00188 0.23% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.23% Gravity::get_old_grav_vector() 10 0.001761 0.001761 0.001761 0.22% MLMG::oneIter() 82 0.001645 0.001645 0.001645 0.20% MLCellLinOp::setLevelBC() 11 0.001514 0.001514 0.001514 0.19% Gravity::actual_solve_with_mlmg() 11 0.001435 0.001435 0.001435 0.18% MLCellLinOp::prepareForSolve() 11 0.001341 0.001341 0.001341 0.17% FabArray::mult() 43 0.001325 0.001325 0.001325 0.16% Castro::initData() 1 0.001177 0.001177 0.001177 0.15% MLCellLinOp::smooth() 1640 0.001141 0.001141 0.001141 0.14% FabArray::FillBoundary() 4023 0.0008364 0.0008364 0.0008364 0.10% FabArrayBase::getCPC() 1323 0.0008324 0.0008324 0.0008324 0.10% MLMG::prepareForSolve() 11 0.0007705 0.0007705 0.0007705 0.10% FabArrayBase::CPC::define() 454 0.0007204 0.0007204 0.0007204 0.09% FabArrayBase::getFB() 4023 0.0006786 0.0006786 0.0006786 0.08% Gravity::update_max_rhs() 11 0.0005421 0.0005421 0.0005421 0.07% MLCellLinOp::apply() 1142 0.0005241 0.0005241 0.0005241 0.07% MultiFab::Copy() 11 0.0005134 0.0005134 0.0005134 0.06% CGSolver::sxay() 1586 0.0004834 0.0004834 0.0004834 0.06% Gravity::solve_for_phi() 10 0.0004466 0.0004466 0.0004466 0.06% Amr::InitAmr() 1 0.0004423 0.0004423 0.0004423 0.06% MLMG::mgVcycle() 82 0.0003903 0.0003903 0.0003903 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003394 0.0003394 0.0003394 0.04% MultiFab::min() 10 0.0003236 0.0003236 0.0003236 0.04% main() 1 0.0003043 0.0003043 0.0003043 0.04% FabArray::ParallelCopy() 861 0.0002897 0.0002897 0.0002897 0.04% Castro::construct_new_gravity() 10 0.0002627 0.0002627 0.0002627 0.03% FillPatchIterator::Initialize 41 0.0002584 0.0002584 0.0002584 0.03% MultiFab::max() 11 0.0002545 0.0002545 0.0002545 0.03% MLCellLinOp::correctionResidual() 492 0.0002293 0.0002293 0.0002293 0.03% Gravity::actual_multilevel_solve() 1 0.0002144 0.0002144 0.0002144 0.03% Amr::coarseTimeStep() 10 0.0002052 0.0002052 0.0002052 0.03% MLMG::MLRhsNormInf() 11 0.0001965 0.0001965 0.0001965 0.02% Amr::timeStep() 10 0.0001931 0.0001931 0.0001931 0.02% MLCellLinOp::defineBC() 11 0.0001867 0.0001867 0.0001867 0.02% MLLinOp::defineGrids() 11 0.0001728 0.0001728 0.0001728 0.02% MLMG:computeResOfCorrection() 410 0.0001491 0.0001491 0.0001491 0.02% Amr::defBaseLevel() 1 0.0001465 0.0001465 0.0001465 0.02% Castro::subcycle_advance_ctu() 10 0.0001414 0.0001414 0.0001414 0.02% StateData::checkPoint() 12 0.0001323 0.0001323 0.0001323 0.02% MLMG::actualBottomSolve() 82 0.000107 0.000107 0.000107 0.01% MLMG::mgVcycle_down::0 82 9.894e-05 9.894e-05 9.894e-05 0.01% FabArrayBase::FB::FB() 56 9.548e-05 9.548e-05 9.548e-05 0.01% MLMG::mgVcycle_down::1 82 9.088e-05 9.088e-05 9.088e-05 0.01% Castro::post_timestep() 10 8.849e-05 8.849e-05 8.849e-05 0.01% Castro::advance() 10 8.57e-05 8.57e-05 8.57e-05 0.01% MLMG::mgVcycle_down::2 82 8.34e-05 8.34e-05 8.34e-05 0.01% MLMG::mgVcycle_down::3 82 8.072e-05 8.072e-05 8.072e-05 0.01% MLMG::mgVcycle_down::4 82 8.06e-05 8.06e-05 8.06e-05 0.01% Castro::initialize_advance() 10 7.771e-05 7.771e-05 7.771e-05 0.01% MLMG::solve() 11 7.683e-05 7.683e-05 7.683e-05 0.01% Castro::clean_state() 62 7.349e-05 7.349e-05 7.349e-05 0.01% AmrLevel::checkPoint() 3 7.283e-05 7.283e-05 7.283e-05 0.01% Castro::construct_new_source() 50 6.775e-05 6.775e-05 6.775e-05 0.01% MLMG::mgVcycle_up::4 82 6.667e-05 6.667e-05 6.667e-05 0.01% MLMG::mgVcycle_up::0 82 6.03e-05 6.03e-05 6.03e-05 0.01% Castro::finalize_advance() 10 5.797e-05 5.797e-05 5.797e-05 0.01% Castro::initialize_do_advance() 10 5.747e-05 5.747e-05 5.747e-05 0.01% MLMG::mgVcycle_up::3 82 5.701e-05 5.701e-05 5.701e-05 0.01% MLMG::mgVcycle_up::1 82 5.615e-05 5.615e-05 5.615e-05 0.01% MLMG::mgVcycle_up::2 82 5.379e-05 5.379e-05 5.379e-05 0.01% MLCellLinOp::solutionResidual() 93 4.691e-05 4.691e-05 4.691e-05 0.01% Castro::construct_old_source() 50 4.398e-05 4.398e-05 4.398e-05 0.01% StateData::define() 4 4.337e-05 4.337e-05 4.337e-05 0.01% Castro::swap_state_time_levels() 10 3.879e-05 3.879e-05 3.879e-05 0.00% Castro::finalize_do_advance() 10 3.815e-05 3.815e-05 3.815e-05 0.00% MLMG::mgVcycle_bottom 82 3.554e-05 3.554e-05 3.554e-05 0.00% MLMG::computeResidual() 82 3.463e-05 3.463e-05 3.463e-05 0.00% MLLinOp::define() 11 3.406e-05 3.406e-05 3.406e-05 0.00% Castro::enforce_consistent_e() 1 3.357e-05 3.357e-05 3.357e-05 0.00% FillPatchSingleLevel 41 3.137e-05 3.137e-05 3.137e-05 0.00% makeSFC 55 2.997e-05 2.997e-05 2.997e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.712e-05 2.712e-05 2.712e-05 0.00% Amr::writeSmallPlotFile() 1 2.44e-05 2.44e-05 2.44e-05 0.00% MLPoisson::define() 11 2.428e-05 2.428e-05 2.428e-05 0.00% Amr::FinalizeInit() 1 2.097e-05 2.097e-05 2.097e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.722e-05 1.722e-05 1.722e-05 0.00% Castro::do_new_sources() 10 1.702e-05 1.702e-05 1.702e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.652e-05 1.652e-05 1.652e-05 0.00% Castro::do_old_sources() 10 1.567e-05 1.567e-05 1.567e-05 0.00% DistributionMapping::Distribute() 56 1.549e-05 1.549e-05 1.549e-05 0.00% Castro::construct_old_gravity() 10 1.198e-05 1.198e-05 1.198e-05 0.00% Castro::apply_source_to_state() 20 1.151e-05 1.151e-05 1.151e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.049e-05 1.049e-05 1.049e-05 0.00% Gravity::swapTimeLevels() 10 9.141e-06 9.141e-06 9.141e-06 0.00% Amr::initSubcycle() 1 9.022e-06 9.022e-06 9.022e-06 0.00% MLPoisson::prepareForSolve() 11 8.537e-06 8.537e-06 8.537e-06 0.00% MLMG::computeMLResidual() 11 8.205e-06 8.205e-06 8.205e-06 0.00% Amr::InitializeInit() 1 6.359e-06 6.359e-06 6.359e-06 0.00% MLMG::buildFineMask() 11 6.231e-06 6.231e-06 6.231e-06 0.00% Castro::computeNewDt() 9 6.135e-06 6.135e-06 6.135e-06 0.00% AmrLevel::checkPointPost() 3 6.095e-06 6.095e-06 6.095e-06 0.00% MLMG::getGradSolution() 11 5.999e-06 5.999e-06 5.999e-06 0.00% Castro::create_source_corrector() 10 5.501e-06 5.501e-06 5.501e-06 0.00% MLMG::MLResNormInf() 11 4.826e-06 4.826e-06 4.826e-06 0.00% Gravity::set_mass_offset() 11 4.425e-06 4.425e-06 4.425e-06 0.00% Castro::retry_advance_ctu() 10 4.386e-06 4.386e-06 4.386e-06 0.00% Castro::post_init() 1 3.703e-06 3.703e-06 3.703e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.266e-06 3.266e-06 3.266e-06 0.00% Castro::FluxRegFineAdd() 10 3.024e-06 3.024e-06 3.024e-06 0.00% Castro::computeInitialDt() 2 2.825e-06 2.825e-06 2.825e-06 0.00% Amr::init() 1 2.824e-06 2.824e-06 2.824e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.132e-06 2.132e-06 2.132e-06 0.00% AmrLevel::checkPointPre() 3 1.572e-06 1.572e-06 1.572e-06 0.00% Castro::post_regrid() 1 1.194e-06 1.194e-06 1.194e-06 0.00% Amr::initialInit() 1 9.1e-07 9.1e-07 9.1e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8035 0.8035 0.8035 100.00% Amr::coarseTimeStep() 10 0.6598 0.6598 0.6598 82.11% Amr::timeStep() 10 0.5877 0.5877 0.5877 73.14% Castro::advance() 10 0.5816 0.5816 0.5816 72.37% Castro::subcycle_advance_ctu() 10 0.5674 0.5674 0.5674 70.62% Castro::do_advance_ctu() 10 0.5673 0.5673 0.5673 70.60% Gravity::solve_phi_with_mlmg() 11 0.3229 0.3229 0.3229 40.19% Gravity::actual_solve_with_mlmg() 11 0.3108 0.3108 0.3108 38.67% Castro::construct_new_gravity() 10 0.2931 0.2931 0.2931 36.48% MLMG::solve() 11 0.2882 0.2882 0.2882 35.86% Gravity::solve_for_phi() 10 0.2753 0.2753 0.2753 34.26% MLMG::oneIter() 82 0.2723 0.2723 0.2723 33.89% MLMG::mgVcycle() 82 0.2707 0.2707 0.2707 33.69% Castro::construct_ctu_hydro_source() 10 0.1974 0.1974 0.1974 24.56% VisMF::Write(FabArray) 11 0.1446 0.1446 0.1446 18.00% MLCellLinOp::smooth() 1640 0.1437 0.1437 0.1437 17.88% Amr::init() 1 0.1212 0.1212 0.1212 15.08% MLCellLinOp::applyBC() 4433 0.1127 0.1127 0.1127 14.02% Amr::checkPoint() 3 0.1073 0.1073 0.1073 13.36% AmrLevel::checkPoint() 3 0.1035 0.1035 0.1035 12.88% StateData::checkPoint() 12 0.1035 0.1035 0.1035 12.87% MLMG::mgVcycle_bottom 82 0.07674 0.07674 0.07674 9.55% MLMG::actualBottomSolve() 82 0.07671 0.07671 0.07671 9.55% MLCGSolver::bicgstab 82 0.07598 0.07598 0.07598 9.46% Amr::initialInit() 1 0.06071 0.06071 0.06071 7.56% MLPoisson::Fsmooth() 3280 0.05895 0.05895 0.05895 7.34% Amr::FinalizeInit() 1 0.05158 0.05158 0.05158 6.42% Castro::post_init() 1 0.05088 0.05088 0.05088 6.33% FillPatchIterator::Initialize 41 0.04884 0.04884 0.04884 6.08% Gravity::multilevel_solve_for_new_phi() 1 0.04831 0.04831 0.04831 6.01% Gravity::actual_multilevel_solve() 1 0.04829 0.04829 0.04829 6.01% Amr::writePlotFile() 2 0.04434 0.04434 0.04434 5.52% FillPatchSingleLevel 41 0.04305 0.04305 0.04305 5.36% MLCellLinOp::apply() 1142 0.03951 0.03951 0.03951 4.92% StateDataPhysBCFunct::() 41 0.03883 0.03883 0.03883 4.83% MLMG::mgVcycle_down::0 82 0.03824 0.03824 0.03824 4.76% Castro::clean_state() 62 0.03422 0.03422 0.03422 4.26% MLMG::mgVcycle_up::0 82 0.03278 0.03278 0.03278 4.08% FabArray::setVal() 1144 0.02588 0.02588 0.02588 3.22% StateData::FillBoundary(geom) 328 0.02382 0.02382 0.02382 2.96% Castro::initialize_do_advance() 10 0.02355 0.02355 0.02355 2.93% MLCellLinOp::correctionResidual() 492 0.02213 0.02213 0.02213 2.75% MultiFab::Dot() 1114 0.02104 0.02104 0.02104 2.62% Gravity::get_new_grav_vector() 11 0.01996 0.01996 0.01996 2.48% FabArray::FillBoundary() 4023 0.01949 0.01949 0.01949 2.43% MLMG:computeResOfCorrection() 410 0.01918 0.01918 0.01918 2.39% FillBoundary_nowait() 4023 0.01866 0.01866 0.01866 2.32% Castro::computeTemp() 63 0.0186 0.0186 0.0186 2.31% MLMG::mgVcycle_down::1 82 0.01829 0.01829 0.01829 2.28% Castro::expand_state() 10 0.01812 0.01812 0.01812 2.26% MLMG::mgVcycle_down::2 82 0.01774 0.01774 0.01774 2.21% MLPoisson::define() 11 0.0171 0.0171 0.0171 2.13% FabArray::ParallelCopy() 861 0.01706 0.01706 0.01706 2.12% MLMG::mgVcycle_down::3 82 0.01696 0.01696 0.01696 2.11% FabArray::ParallelCopy_nowait() 861 0.01677 0.01677 0.01677 2.09% MLMG::mgVcycle_down::4 82 0.01611 0.01611 0.01611 2.01% Castro::construct_old_gravity() 10 0.01595 0.01595 0.01595 1.99% Gravity::get_old_grav_vector() 10 0.01594 0.01594 0.01594 1.98% Castro::initialize_advance() 10 0.01396 0.01396 0.01396 1.74% MLMG::mgVcycle_up::2 82 0.01381 0.01381 0.01381 1.72% MLMG::mgVcycle_up::1 82 0.01357 0.01357 0.01357 1.69% MLMG::addInterpCorrection() 410 0.01342 0.01342 0.01342 1.67% MLMG::mgVcycle_up::3 82 0.01312 0.01312 0.01312 1.63% amrex::average_down 410 0.01296 0.01296 0.01296 1.61% MLMG::mgVcycle_up::4 82 0.01294 0.01294 0.01294 1.61% MLCGSolver::ParallelAllReduce 1514 0.01271 0.01271 0.01271 1.58% CGSolver::sxay() 1586 0.01264 0.01264 0.01264 1.57% MultiFab::LinComb() 1586 0.01216 0.01216 0.01216 1.51% Gravity::fill_multipole_BCs() 11 0.01193 0.01193 0.01193 1.48% Castro::do_new_sources() 10 0.01187 0.01187 0.01187 1.48% MLCellLinOp::defineAuxData() 11 0.0118 0.0118 0.0118 1.47% Castro::enforce_min_density() 62 0.01073 0.01073 0.01073 1.34% MLPoisson::Fapply() 1142 0.01042 0.01042 0.01042 1.30% Amr::InitializeInit() 1 0.009138 0.009138 0.009138 1.14% Amr::defBaseLevel() 1 0.009132 0.009132 0.009132 1.14% Castro::do_old_sources() 10 0.007919 0.007919 0.007919 0.99% MLCellLinOp::solutionResidual() 93 0.007844 0.007844 0.007844 0.98% MLMG::computeResidual() 82 0.006782 0.006782 0.006782 0.84% MultiFab::Xpay() 585 0.005999 0.005999 0.005999 0.75% Castro::post_timestep() 10 0.005992 0.005992 0.005992 0.75% MLMG::prepareForSolve() 11 0.005769 0.005769 0.005769 0.72% FabArray::setDomainBndry() 41 0.005526 0.005526 0.005526 0.69% Castro::reset_internal_energy() 63 0.005164 0.005164 0.005164 0.64% MLCellLinOp::defineBC() 11 0.005008 0.005008 0.005008 0.62% BndryData::define() 11 0.004821 0.004821 0.004821 0.60% Castro::estTimeStep() 21 0.004595 0.004595 0.004595 0.57% Castro::initData() 1 0.003345 0.003345 0.003345 0.42% Castro::construct_new_source() 50 0.003076 0.003076 0.003076 0.38% Castro::construct_new_gravity_source() 10 0.003008 0.003008 0.003008 0.37% Castro::enforce_speed_limit() 62 0.002913 0.002913 0.002913 0.36% Castro::normalize_species() 62 0.002617 0.002617 0.002617 0.33% MLMG::getGradSolution() 11 0.002556 0.002556 0.002556 0.32% MLCellLinOp::compGrad() 11 0.00255 0.00255 0.00255 0.32% Castro::construct_old_source() 50 0.002549 0.002549 0.002549 0.32% Castro::construct_old_gravity_source() 10 0.002505 0.002505 0.002505 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002228 0.002228 0.002228 0.28% Castro::computeNewDt() 9 0.001925 0.001925 0.001925 0.24% MLMG::ResNormInf() 93 0.00188 0.00188 0.00188 0.23% Castro::apply_source_to_state() 20 0.001822 0.001822 0.001822 0.23% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.23% FabArrayBase::getCPC() 1323 0.001553 0.001553 0.001553 0.19% MLCellLinOp::setLevelBC() 11 0.001514 0.001514 0.001514 0.19% MLPoisson::prepareForSolve() 11 0.001349 0.001349 0.001349 0.17% MLCellLinOp::prepareForSolve() 11 0.001341 0.001341 0.001341 0.17% FabArray::mult() 43 0.001325 0.001325 0.001325 0.16% MLMG::computeMLResidual() 11 0.001105 0.001105 0.001105 0.14% Gravity::swapTimeLevels() 10 0.001033 0.001033 0.001033 0.13% Gravity::update_max_rhs() 11 0.0009391 0.0009391 0.0009391 0.12% FabArrayBase::getFB() 4023 0.0007741 0.0007741 0.0007741 0.10% FabArrayBase::CPC::define() 454 0.0007204 0.0007204 0.0007204 0.09% Castro::computeInitialDt() 2 0.0006993 0.0006993 0.0006993 0.09% MultiFab::Copy() 11 0.0005134 0.0005134 0.0005134 0.06% Castro::post_regrid() 1 0.0004693 0.0004693 0.0004693 0.06% Amr::InitAmr() 1 0.0004513 0.0004513 0.0004513 0.06% MultiFab::min() 10 0.0003236 0.0003236 0.0003236 0.04% MLLinOp::define() 11 0.0002698 0.0002698 0.0002698 0.03% MultiFab::max() 11 0.0002545 0.0002545 0.0002545 0.03% MLMG::MLResNormInf() 11 0.0002511 0.0002511 0.0002511 0.03% MLLinOp::defineGrids() 11 0.0002358 0.0002358 0.0002358 0.03% MLMG::MLRhsNormInf() 11 0.0001965 0.0001965 0.0001965 0.02% FabArrayBase::FB::FB() 56 9.548e-05 9.548e-05 9.548e-05 0.01% Castro::finalize_advance() 10 6.099e-05 6.099e-05 6.099e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 6.081e-05 6.081e-05 6.081e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.386e-05 5.386e-05 5.386e-05 0.01% makeSFC 55 4.429e-05 4.429e-05 4.429e-05 0.01% StateData::define() 4 4.337e-05 4.337e-05 4.337e-05 0.01% Castro::swap_state_time_levels() 10 3.879e-05 3.879e-05 3.879e-05 0.00% Castro::finalize_do_advance() 10 3.815e-05 3.815e-05 3.815e-05 0.00% Castro::enforce_consistent_e() 1 3.357e-05 3.357e-05 3.357e-05 0.00% Amr::writeSmallPlotFile() 1 2.44e-05 2.44e-05 2.44e-05 0.00% DistributionMapping::Distribute() 56 1.549e-05 1.549e-05 1.549e-05 0.00% Amr::initSubcycle() 1 9.022e-06 9.022e-06 9.022e-06 0.00% MLMG::buildFineMask() 11 6.231e-06 6.231e-06 6.231e-06 0.00% AmrLevel::checkPointPost() 3 6.095e-06 6.095e-06 6.095e-06 0.00% Castro::create_source_corrector() 10 5.501e-06 5.501e-06 5.501e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.438e-06 4.438e-06 4.438e-06 0.00% Gravity::set_mass_offset() 11 4.425e-06 4.425e-06 4.425e-06 0.00% Castro::retry_advance_ctu() 10 4.386e-06 4.386e-06 4.386e-06 0.00% Castro::FluxRegFineAdd() 10 3.024e-06 3.024e-06 3.024e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.132e-06 2.132e-06 2.132e-06 0.00% AmrLevel::checkPointPre() 3 1.572e-06 1.572e-06 1.572e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.03-12-g806f2373b68e) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.03-12-g806f2373b68e) initialized Starting run at 08:18:06 UTC on 2022-03-14. Successfully read inputs file ... Castro git describe: 22.02-5-g5116b7268 AMReX git describe: 22.03-12-g806f2373b Microphysics git describe: 22.03-6-g41c4af20 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.478634705 Restart time = 0.054429435 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.113826212 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048912565 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.054516024 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.056966581 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.061450848 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.04099286 seconds Ending run at 08:18:06 UTC on 2022-03-14. Run time = 0.431990719 Run time without initialization = 0.377015764 Average number of zones advanced per microsecond: 3.477 Average number of zones advanced per microsecond per rank: 3.477 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.432 ... 0.432 ... 0.432 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1300 0.1300 0.1300 30.08% VisMF::Read() 3 0.04297 0.04297 0.04297 9.95% VisMF::Write(FabArray) 1 0.03975 0.03975 0.03975 9.20% MLCellLinOp::applyBC() 1946 0.03722 0.03722 0.03722 8.62% MLPoisson::Fsmooth() 1440 0.0253 0.0253 0.0253 5.86% FabArray::setVal() 537 0.01653 0.01653 0.01653 3.83% StateData::FillBoundary(geom) 160 0.01104 0.01104 0.01104 2.56% MLCGSolver::bicgstab 36 0.00915 0.00915 0.00915 2.12% MultiFab::Dot() 484 0.009016 0.009016 0.009016 2.09% StateDataPhysBCFunct::() 20 0.007405 0.007405 0.007405 1.71% FillBoundary_nowait() 1766 0.007065 0.007065 0.007065 1.64% FabArray::ParallelCopy_nowait() 380 0.006983 0.006983 0.006983 1.62% Castro::computeTemp() 30 0.006886 0.006886 0.006886 1.59% Gravity::fill_multipole_BCs() 6 0.006332 0.006332 0.006332 1.47% MLCellLinOp::defineAuxData() 6 0.005469 0.005469 0.005469 1.27% MultiFab::LinComb() 690 0.005169 0.005169 0.005169 1.20% Castro::enforce_min_density() 30 0.004856 0.004856 0.004856 1.12% MLPoisson::Fapply() 500 0.004542 0.004542 0.004542 1.05% Castro::expand_state() 5 0.004149 0.004149 0.004149 0.96% FabArray::setDomainBndry() 20 0.004048 0.004048 0.004048 0.94% MLMG::addInterpCorrection() 180 0.002974 0.002974 0.002974 0.69% Amr::restart() 1 0.002918 0.002918 0.002918 0.68% amrex::average_down 180 0.002765 0.002765 0.002765 0.64% Castro::do_advance_ctu() 5 0.00264 0.00264 0.00264 0.61% MultiFab::Xpay() 258 0.002607 0.002607 0.002607 0.60% Castro::reset_internal_energy() 30 0.00242 0.00242 0.00242 0.56% Castro::estTimeStep() 10 0.002384 0.002384 0.002384 0.55% Castro::enforce_speed_limit() 30 0.002026 0.002026 0.002026 0.47% BndryData::define() 6 0.00192 0.00192 0.00192 0.44% Gravity::get_new_grav_vector() 5 0.001833 0.001833 0.001833 0.42% Castro::construct_new_gravity_source() 5 0.001539 0.001539 0.001539 0.36% MLCellLinOp::compGrad() 6 0.001352 0.001352 0.001352 0.31% Amr::writePlotFile() 1 0.001333 0.001333 0.001333 0.31% Castro::construct_old_gravity_source() 5 0.001319 0.001319 0.001319 0.31% Castro::normalize_species() 30 0.001293 0.001293 0.001293 0.30% Castro::subcycle_advance_ctu() 5 0.001286 0.001286 0.001286 0.30% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001184 0.001184 0.001184 0.27% MultiFab::Saxpy() 10 0.0009207 0.0009207 0.0009207 0.21% Gravity::get_old_grav_vector() 5 0.000909 0.000909 0.000909 0.21% MLMG::ResNormInf() 42 0.000829 0.000829 0.000829 0.19% MLCellLinOp::setLevelBC() 6 0.0008155 0.0008155 0.0008155 0.19% Gravity::actual_solve_with_mlmg() 6 0.0007322 0.0007322 0.0007322 0.17% MLMG::oneIter() 36 0.000719 0.000719 0.000719 0.17% MLCellLinOp::prepareForSolve() 6 0.0006932 0.0006932 0.0006932 0.16% FabArray::mult() 22 0.0006497 0.0006497 0.0006497 0.15% Gravity::update_max_rhs() 6 0.0005395 0.0005395 0.0005395 0.12% MLCellLinOp::smooth() 720 0.0004987 0.0004987 0.0004987 0.12% FabArrayBase::CPC::define() 244 0.000469 0.000469 0.000469 0.11% MLMG::prepareForSolve() 6 0.0004285 0.0004285 0.0004285 0.10% FabArrayBase::getCPC() 632 0.0004168 0.0004168 0.0004168 0.10% FabArray::FillBoundary() 1766 0.0003975 0.0003975 0.0003975 0.09% MultiFab::Copy() 6 0.0003601 0.0003601 0.0003601 0.08% Amr::InitAmr() 1 0.0003498 0.0003498 0.0003498 0.08% FabArrayBase::getFB() 1766 0.0003064 0.0003064 0.0003064 0.07% main() 1 0.0002946 0.0002946 0.0002946 0.07% Gravity::actual_multilevel_solve() 1 0.0002281 0.0002281 0.0002281 0.05% MLCellLinOp::apply() 500 0.0002206 0.0002206 0.0002206 0.05% Gravity::solve_for_phi() 5 0.0002136 0.0002136 0.0002136 0.05% CGSolver::sxay() 690 0.0002011 0.0002011 0.0002011 0.05% Castro::construct_new_gravity() 5 0.000201 0.000201 0.000201 0.05% MLMG::mgVcycle() 36 0.0001636 0.0001636 0.0001636 0.04% MultiFab::min() 5 0.0001609 0.0001609 0.0001609 0.04% MLCGSolver::ParallelAllReduce 659 0.0001545 0.0001545 0.0001545 0.04% FabArray::ParallelCopy() 380 0.0001469 0.0001469 0.0001469 0.03% MultiFab::max() 6 0.0001312 0.0001312 0.0001312 0.03% FillPatchIterator::Initialize 20 0.0001233 0.0001233 0.0001233 0.03% Amr::coarseTimeStep() 5 0.0001154 0.0001154 0.0001154 0.03% MLMG::MLRhsNormInf() 6 0.0001045 0.0001045 0.0001045 0.02% Amr::timeStep() 5 9.892e-05 9.892e-05 9.892e-05 0.02% MLLinOp::defineGrids() 6 9.736e-05 9.736e-05 9.736e-05 0.02% Castro::construct_new_source() 25 9.646e-05 9.646e-05 9.646e-05 0.02% MLCellLinOp::defineBC() 6 9.644e-05 9.644e-05 9.644e-05 0.02% MLCellLinOp::correctionResidual() 216 9.581e-05 9.581e-05 9.581e-05 0.02% AmrLevel::restart() 1 7.475e-05 7.475e-05 7.475e-05 0.02% StateData::restartDoit() 4 7.446e-05 7.446e-05 7.446e-05 0.02% FabArrayBase::FB::FB() 26 6.801e-05 6.801e-05 6.801e-05 0.02% MLMG:computeResOfCorrection() 180 6.741e-05 6.741e-05 6.741e-05 0.02% Castro::construct_old_source() 25 5.208e-05 5.208e-05 5.208e-05 0.01% MLMG::actualBottomSolve() 36 5.133e-05 5.133e-05 5.133e-05 0.01% Castro::advance() 5 5.041e-05 5.041e-05 5.041e-05 0.01% Castro::create_source_corrector() 5 4.639e-05 4.639e-05 4.639e-05 0.01% MLMG::mgVcycle_down::0 36 4.168e-05 4.168e-05 4.168e-05 0.01% MLMG::mgVcycle_down::1 36 4.096e-05 4.096e-05 4.096e-05 0.01% MLMG::solve() 6 3.948e-05 3.948e-05 3.948e-05 0.01% Castro::post_timestep() 5 3.829e-05 3.829e-05 3.829e-05 0.01% MLMG::mgVcycle_down::2 36 3.823e-05 3.823e-05 3.823e-05 0.01% Castro::initialize_advance() 5 3.676e-05 3.676e-05 3.676e-05 0.01% Castro::clean_state() 30 3.641e-05 3.641e-05 3.641e-05 0.01% MLMG::mgVcycle_down::3 36 3.572e-05 3.572e-05 3.572e-05 0.01% MLMG::mgVcycle_down::4 36 3.569e-05 3.569e-05 3.569e-05 0.01% Castro::do_new_sources() 5 3.4e-05 3.4e-05 3.4e-05 0.01% Castro::initialize_do_advance() 5 2.919e-05 2.919e-05 2.919e-05 0.01% MLMG::mgVcycle_up::4 36 2.899e-05 2.899e-05 2.899e-05 0.01% MLLinOp::define() 6 2.814e-05 2.814e-05 2.814e-05 0.01% Castro::swap_state_time_levels() 5 2.625e-05 2.625e-05 2.625e-05 0.01% MLMG::mgVcycle_up::0 36 2.613e-05 2.613e-05 2.613e-05 0.01% Amr::writeSmallPlotFile() 1 2.59e-05 2.59e-05 2.59e-05 0.01% Castro::finalize_advance() 5 2.526e-05 2.526e-05 2.526e-05 0.01% MLMG::mgVcycle_up::3 36 2.452e-05 2.452e-05 2.452e-05 0.01% MLMG::mgVcycle_up::2 36 2.431e-05 2.431e-05 2.431e-05 0.01% Castro::post_restart() 1 2.34e-05 2.34e-05 2.34e-05 0.01% MLMG::mgVcycle_up::1 36 2.313e-05 2.313e-05 2.313e-05 0.01% MLCellLinOp::solutionResidual() 42 2.214e-05 2.214e-05 2.214e-05 0.01% Castro::finalize_do_advance() 5 2.016e-05 2.016e-05 2.016e-05 0.00% makeSFC 30 1.682e-05 1.682e-05 1.682e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.677e-05 1.677e-05 1.677e-05 0.00% MLMG::mgVcycle_bottom 36 1.645e-05 1.645e-05 1.645e-05 0.00% FillPatchSingleLevel 20 1.575e-05 1.575e-05 1.575e-05 0.00% MLPoisson::define() 6 1.537e-05 1.537e-05 1.537e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.535e-05 1.535e-05 1.535e-05 0.00% MLMG::computeResidual() 36 1.497e-05 1.497e-05 1.497e-05 0.00% DistributionMapping::Distribute() 31 9.716e-06 9.716e-06 9.716e-06 0.00% Amr::initSubcycle() 1 9.071e-06 9.071e-06 9.071e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.744e-06 8.744e-06 8.744e-06 0.00% Castro::do_old_sources() 5 8.659e-06 8.659e-06 8.659e-06 0.00% MLMG::computeMLResidual() 6 8.036e-06 8.036e-06 8.036e-06 0.00% Castro::apply_source_to_state() 10 5.879e-06 5.879e-06 5.879e-06 0.00% Castro::construct_old_gravity() 5 5.697e-06 5.697e-06 5.697e-06 0.00% Gravity::swapTimeLevels() 5 4.41e-06 4.41e-06 4.41e-06 0.00% MLPoisson::prepareForSolve() 6 4.284e-06 4.284e-06 4.284e-06 0.00% MLMG::buildFineMask() 6 3.667e-06 3.667e-06 3.667e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.434e-06 3.434e-06 3.434e-06 0.00% Castro::computeNewDt() 5 3.083e-06 3.083e-06 3.083e-06 0.00% MLMG::getGradSolution() 6 3.078e-06 3.078e-06 3.078e-06 0.00% MLMG::MLResNormInf() 6 2.574e-06 2.574e-06 2.574e-06 0.00% Gravity::set_mass_offset() 6 2.485e-06 2.485e-06 2.485e-06 0.00% Castro::retry_advance_ctu() 5 1.982e-06 1.982e-06 1.982e-06 0.00% Castro::FluxRegFineAdd() 5 1.828e-06 1.828e-06 1.828e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.408e-06 1.408e-06 1.408e-06 0.00% Amr::init() 1 1.036e-06 1.036e-06 1.036e-06 0.00% AmrLevel::AmrLevel() 1 9.58e-07 9.58e-07 9.58e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.432 0.432 0.432 100.00% Amr::coarseTimeStep() 5 0.3358 0.3358 0.3358 77.72% Amr::timeStep() 5 0.3345 0.3345 0.3345 77.42% Castro::advance() 5 0.332 0.332 0.332 76.86% Castro::subcycle_advance_ctu() 5 0.3207 0.3207 0.3207 74.22% Castro::do_advance_ctu() 5 0.3194 0.3194 0.3194 73.92% Castro::construct_new_gravity() 5 0.1443 0.1443 0.1443 33.40% Gravity::solve_phi_with_mlmg() 6 0.1402 0.1402 0.1402 32.45% Gravity::solve_for_phi() 5 0.1346 0.1346 0.1346 31.16% Gravity::actual_solve_with_mlmg() 6 0.1337 0.1337 0.1337 30.96% Castro::construct_ctu_hydro_source() 5 0.13 0.13 0.13 30.08% MLMG::solve() 6 0.1212 0.1212 0.1212 28.07% MLMG::oneIter() 36 0.1135 0.1135 0.1135 26.27% MLMG::mgVcycle() 36 0.1128 0.1128 0.1128 26.10% MLCellLinOp::smooth() 720 0.05906 0.05906 0.05906 13.67% Amr::init() 1 0.05447 0.05447 0.05447 12.61% Amr::restart() 1 0.05446 0.05446 0.05446 12.61% MLCellLinOp::applyBC() 1946 0.04506 0.04506 0.04506 10.43% AmrLevel::restart() 1 0.04388 0.04388 0.04388 10.16% StateData::restartDoit() 4 0.0438 0.0438 0.0438 10.14% VisMF::Read() 3 0.04297 0.04297 0.04297 9.95% Amr::writePlotFile() 1 0.04108 0.04108 0.04108 9.51% VisMF::Write(FabArray) 1 0.03975 0.03975 0.03975 9.20% MLMG::mgVcycle_bottom 36 0.03249 0.03249 0.03249 7.52% MLMG::actualBottomSolve() 36 0.03248 0.03248 0.03248 7.52% MLCGSolver::bicgstab 36 0.03216 0.03216 0.03216 7.45% MLPoisson::Fsmooth() 1440 0.0253 0.0253 0.0253 5.86% FillPatchIterator::Initialize 20 0.02472 0.02472 0.02472 5.72% FillPatchSingleLevel 20 0.02055 0.02055 0.02055 4.76% StateDataPhysBCFunct::() 20 0.01845 0.01845 0.01845 4.27% Castro::clean_state() 30 0.01752 0.01752 0.01752 4.05% FabArray::setVal() 537 0.01653 0.01653 0.01653 3.83% MLCellLinOp::apply() 500 0.01628 0.01628 0.01628 3.77% MLMG::mgVcycle_down::0 36 0.01576 0.01576 0.01576 3.65% Castro::initialize_do_advance() 5 0.01483 0.01483 0.01483 3.43% MLMG::mgVcycle_up::0 36 0.01345 0.01345 0.01345 3.11% Castro::expand_state() 5 0.01208 0.01208 0.01208 2.80% Castro::initialize_advance() 5 0.01131 0.01131 0.01131 2.62% StateData::FillBoundary(geom) 160 0.01104 0.01104 0.01104 2.56% Gravity::get_new_grav_vector() 5 0.009478 0.009478 0.009478 2.19% Castro::computeTemp() 30 0.009306 0.009306 0.009306 2.15% MLPoisson::define() 6 0.009294 0.009294 0.009294 2.15% MLCellLinOp::correctionResidual() 216 0.009184 0.009184 0.009184 2.13% MultiFab::Dot() 484 0.009016 0.009016 0.009016 2.09% MLMG:computeResOfCorrection() 180 0.007954 0.007954 0.007954 1.84% Castro::construct_old_gravity() 5 0.007872 0.007872 0.007872 1.82% Gravity::get_old_grav_vector() 5 0.007866 0.007866 0.007866 1.82% FabArray::FillBoundary() 1766 0.007837 0.007837 0.007837 1.81% MLMG::mgVcycle_down::1 36 0.007633 0.007633 0.007633 1.77% FabArray::ParallelCopy() 380 0.007568 0.007568 0.007568 1.75% Castro::do_new_sources() 5 0.007496 0.007496 0.007496 1.74% FillBoundary_nowait() 1766 0.007439 0.007439 0.007439 1.72% FabArray::ParallelCopy_nowait() 380 0.007421 0.007421 0.007421 1.72% MLMG::mgVcycle_down::2 36 0.007344 0.007344 0.007344 1.70% MLMG::mgVcycle_down::3 36 0.007004 0.007004 0.007004 1.62% MLMG::mgVcycle_down::4 36 0.006722 0.006722 0.006722 1.56% Castro::post_restart() 1 0.006461 0.006461 0.006461 1.50% MLCellLinOp::defineAuxData() 6 0.006393 0.006393 0.006393 1.48% Gravity::fill_multipole_BCs() 6 0.006332 0.006332 0.006332 1.47% Gravity::multilevel_solve_for_new_phi() 1 0.006055 0.006055 0.006055 1.40% Gravity::actual_multilevel_solve() 1 0.006039 0.006039 0.006039 1.40% MLMG::mgVcycle_up::2 36 0.005734 0.005734 0.005734 1.33% MLMG::addInterpCorrection() 180 0.00572 0.00572 0.00572 1.32% MLMG::mgVcycle_up::1 36 0.005651 0.005651 0.005651 1.31% amrex::average_down 180 0.005498 0.005498 0.005498 1.27% MLCGSolver::ParallelAllReduce 659 0.005485 0.005485 0.005485 1.27% MLMG::mgVcycle_up::3 36 0.005436 0.005436 0.005436 1.26% MLMG::mgVcycle_up::4 36 0.005373 0.005373 0.005373 1.24% CGSolver::sxay() 690 0.00537 0.00537 0.00537 1.24% MultiFab::LinComb() 690 0.005169 0.005169 0.005169 1.20% Castro::enforce_min_density() 30 0.004856 0.004856 0.004856 1.12% MLPoisson::Fapply() 500 0.004542 0.004542 0.004542 1.05% Castro::do_old_sources() 5 0.0044 0.0044 0.0044 1.02% FabArray::setDomainBndry() 20 0.004048 0.004048 0.004048 0.94% MLMG::prepareForSolve() 6 0.003406 0.003406 0.003406 0.79% MLCellLinOp::solutionResidual() 42 0.003362 0.003362 0.003362 0.78% MLMG::computeResidual() 36 0.002788 0.002788 0.002788 0.65% MLCellLinOp::defineBC() 6 0.002725 0.002725 0.002725 0.63% BndryData::define() 6 0.002629 0.002629 0.002629 0.61% MultiFab::Xpay() 258 0.002607 0.002607 0.002607 0.60% Castro::reset_internal_energy() 30 0.00242 0.00242 0.00242 0.56% Castro::estTimeStep() 10 0.002384 0.002384 0.002384 0.55% Castro::post_timestep() 5 0.002328 0.002328 0.002328 0.54% Castro::enforce_speed_limit() 30 0.002026 0.002026 0.002026 0.47% MLMG::getGradSolution() 6 0.001646 0.001646 0.001646 0.38% MLCellLinOp::compGrad() 6 0.001643 0.001643 0.001643 0.38% Castro::construct_new_source() 25 0.001636 0.001636 0.001636 0.38% Castro::construct_new_gravity_source() 5 0.001539 0.001539 0.001539 0.36% Castro::construct_old_source() 25 0.001371 0.001371 0.001371 0.32% Castro::construct_old_gravity_source() 5 0.001319 0.001319 0.001319 0.31% Castro::normalize_species() 30 0.001293 0.001293 0.001293 0.30% Castro::computeNewDt() 5 0.001194 0.001194 0.001194 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001184 0.001184 0.001184 0.27% Castro::apply_source_to_state() 10 0.0009266 0.0009266 0.0009266 0.21% MultiFab::Saxpy() 10 0.0009207 0.0009207 0.0009207 0.21% Gravity::swapTimeLevels() 5 0.0009093 0.0009093 0.0009093 0.21% FabArrayBase::getCPC() 632 0.0008857 0.0008857 0.0008857 0.21% MLMG::ResNormInf() 42 0.000829 0.000829 0.000829 0.19% MLCellLinOp::setLevelBC() 6 0.0008155 0.0008155 0.0008155 0.19% Gravity::update_max_rhs() 6 0.0007468 0.0007468 0.0007468 0.17% MLPoisson::prepareForSolve() 6 0.0006975 0.0006975 0.0006975 0.16% MLCellLinOp::prepareForSolve() 6 0.0006932 0.0006932 0.0006932 0.16% FabArray::mult() 22 0.0006497 0.0006497 0.0006497 0.15% MLMG::computeMLResidual() 6 0.0005977 0.0005977 0.0005977 0.14% FabArrayBase::CPC::define() 244 0.000469 0.000469 0.000469 0.11% FabArrayBase::getFB() 1766 0.0003744 0.0003744 0.0003744 0.09% MultiFab::Copy() 6 0.0003601 0.0003601 0.0003601 0.08% Amr::InitAmr() 1 0.0003588 0.0003588 0.0003588 0.08% MultiFab::min() 5 0.0001609 0.0001609 0.0001609 0.04% MLLinOp::define() 6 0.0001608 0.0001608 0.0001608 0.04% MLMG::MLResNormInf() 6 0.0001335 0.0001335 0.0001335 0.03% MLLinOp::defineGrids() 6 0.0001327 0.0001327 0.0001327 0.03% MultiFab::max() 6 0.0001312 0.0001312 0.0001312 0.03% MLMG::MLRhsNormInf() 6 0.0001045 0.0001045 0.0001045 0.02% FabArrayBase::FB::FB() 26 6.801e-05 6.801e-05 6.801e-05 0.02% Castro::create_source_corrector() 5 4.639e-05 4.639e-05 4.639e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.39e-05 3.39e-05 3.39e-05 0.01% Castro::finalize_advance() 5 2.709e-05 2.709e-05 2.709e-05 0.01% Castro::swap_state_time_levels() 5 2.625e-05 2.625e-05 2.625e-05 0.01% Amr::writeSmallPlotFile() 1 2.59e-05 2.59e-05 2.59e-05 0.01% makeSFC 30 2.515e-05 2.515e-05 2.515e-05 0.01% Castro::finalize_do_advance() 5 2.016e-05 2.016e-05 2.016e-05 0.00% DistributionMapping::Distribute() 31 9.716e-06 9.716e-06 9.716e-06 0.00% Amr::initSubcycle() 1 9.071e-06 9.071e-06 9.071e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.816e-06 4.816e-06 4.816e-06 0.00% MLMG::buildFineMask() 6 3.667e-06 3.667e-06 3.667e-06 0.00% Gravity::set_mass_offset() 6 2.485e-06 2.485e-06 2.485e-06 0.00% Castro::retry_advance_ctu() 5 1.982e-06 1.982e-06 1.982e-06 0.00% Castro::FluxRegFineAdd() 5 1.828e-06 1.828e-06 1.828e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.408e-06 1.408e-06 1.408e-06 0.00% AmrLevel::AmrLevel() 1 9.58e-07 9.58e-07 9.58e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.03-12-g806f2373b68e) finalized