Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.02-14-gaa0dc39f50f7) initialized Starting run at 08:54:39 UTC on 2022-02-21. Successfully read inputs file ... Castro git describe: 22.02-4-g0d942c92d AMReX git describe: 22.02-14-gaa0dc39f5 Microphysics git describe: 22.02-4-ga894d649 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.04085256 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.024649245 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.110573446 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.055900978 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.049500625 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.058993521 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.069225034 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.038391188 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.066235401 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.067877647 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.063790801 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.063103332 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.064670859 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.085177992 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024198087 seconds Ending run at 08:54:40 UTC on 2022-02-21. Run time = 0.94644385 Run time without initialization = 0.818118019 Average number of zones advanced per microsecond: 3.204 Average number of zones advanced per microsecond per rank: 3.204 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.9465 ... 0.9465 ... 0.9465 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2482 0.2482 0.2482 26.23% VisMF::Write(FabArray) 11 0.1601 0.1601 0.1601 16.91% MLCellLinOp::applyBC() 4433 0.0952 0.0952 0.0952 10.06% MLPoisson::Fsmooth() 3280 0.06071 0.06071 0.06071 6.41% Amr::checkPoint() 3 0.05019 0.05019 0.05019 5.30% StateDataPhysBCFunct::() 41 0.02847 0.02847 0.02847 3.01% FabArray::setVal() 1144 0.02709 0.02709 0.02709 2.86% StateData::FillBoundary(geom) 328 0.0239 0.0239 0.0239 2.52% MultiFab::Dot() 1114 0.02226 0.02226 0.02226 2.35% MLCGSolver::bicgstab 82 0.02224 0.02224 0.02224 2.35% FillBoundary_nowait() 4023 0.01814 0.01814 0.01814 1.92% FabArray::ParallelCopy_nowait() 861 0.01632 0.01632 0.01632 1.72% Castro::computeTemp() 63 0.01342 0.01342 0.01342 1.42% Castro::enforce_min_density() 62 0.0134 0.0134 0.0134 1.42% MultiFab::LinComb() 1586 0.0128 0.0128 0.0128 1.35% Gravity::fill_multipole_BCs() 11 0.01197 0.01197 0.01197 1.26% MLPoisson::Fapply() 1142 0.01087 0.01087 0.01087 1.15% MLCellLinOp::defineAuxData() 11 0.01042 0.01042 0.01042 1.10% MLMG::addInterpCorrection() 410 0.007238 0.007238 0.007238 0.76% Castro::estTimeStep() 21 0.007043 0.007043 0.007043 0.74% amrex::average_down 410 0.006666 0.006666 0.006666 0.70% MultiFab::Xpay() 585 0.006221 0.006221 0.006221 0.66% Castro::reset_internal_energy() 63 0.005976 0.005976 0.005976 0.63% FabArray::setDomainBndry() 41 0.00535 0.00535 0.00535 0.57% Castro::expand_state() 10 0.004798 0.004798 0.004798 0.51% BndryData::define() 11 0.003798 0.003798 0.003798 0.40% Castro::do_advance_ctu() 10 0.003255 0.003255 0.003255 0.34% Castro::construct_new_gravity_source() 10 0.003068 0.003068 0.003068 0.32% Amr::writePlotFile() 2 0.002971 0.002971 0.002971 0.31% Gravity::get_new_grav_vector() 11 0.002862 0.002862 0.002862 0.30% Castro::enforce_speed_limit() 62 0.002679 0.002679 0.002679 0.28% Castro::normalize_species() 62 0.002635 0.002635 0.002635 0.28% Castro::construct_old_gravity_source() 10 0.002544 0.002544 0.002544 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002358 0.002358 0.002358 0.25% MLCellLinOp::compGrad() 11 0.001934 0.001934 0.001934 0.20% Gravity::get_old_grav_vector() 10 0.001926 0.001926 0.001926 0.20% MLMG::ResNormInf() 93 0.001919 0.001919 0.001919 0.20% MultiFab::Saxpy() 20 0.00182 0.00182 0.00182 0.19% MLMG::oneIter() 82 0.001658 0.001658 0.001658 0.18% MLCellLinOp::setLevelBC() 11 0.001553 0.001553 0.001553 0.16% Gravity::actual_solve_with_mlmg() 11 0.001435 0.001435 0.001435 0.15% MLCellLinOp::prepareForSolve() 11 0.001372 0.001372 0.001372 0.14% FabArray::mult() 43 0.001342 0.001342 0.001342 0.14% Castro::initData() 1 0.00123 0.00123 0.00123 0.13% MLCellLinOp::smooth() 1640 0.001172 0.001172 0.001172 0.12% FabArrayBase::getCPC() 1323 0.0008997 0.0008997 0.0008997 0.10% FabArray::FillBoundary() 4023 0.0008291 0.0008291 0.0008291 0.09% MLMG::prepareForSolve() 11 0.0007866 0.0007866 0.0007866 0.08% FabArrayBase::getFB() 4023 0.0007287 0.0007287 0.0007287 0.08% FabArrayBase::CPC::define() 454 0.0007251 0.0007251 0.0007251 0.08% Gravity::update_max_rhs() 11 0.0005493 0.0005493 0.0005493 0.06% MultiFab::Copy() 11 0.0005313 0.0005313 0.0005313 0.06% Castro::advance() 10 0.0005143 0.0005143 0.0005143 0.05% CGSolver::sxay() 1586 0.0004885 0.0004885 0.0004885 0.05% Amr::InitAmr() 1 0.0004828 0.0004828 0.0004828 0.05% MLCellLinOp::apply() 1142 0.0004697 0.0004697 0.0004697 0.05% Gravity::solve_for_phi() 10 0.0004484 0.0004484 0.0004484 0.05% MLMG::mgVcycle() 82 0.0003673 0.0003673 0.0003673 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003415 0.0003415 0.0003415 0.04% MultiFab::min() 10 0.0003165 0.0003165 0.0003165 0.03% FabArray::ParallelCopy() 861 0.0002946 0.0002946 0.0002946 0.03% main() 1 0.0002924 0.0002924 0.0002924 0.03% Castro::construct_new_gravity() 10 0.0002607 0.0002607 0.0002607 0.03% MultiFab::max() 11 0.000255 0.000255 0.000255 0.03% FillPatchIterator::Initialize 41 0.00025 0.00025 0.00025 0.03% MLCellLinOp::correctionResidual() 492 0.0002291 0.0002291 0.0002291 0.02% Gravity::actual_multilevel_solve() 1 0.0002198 0.0002198 0.0002198 0.02% Amr::coarseTimeStep() 10 0.0002121 0.0002121 0.0002121 0.02% Castro::subcycle_advance_ctu() 10 0.000211 0.000211 0.000211 0.02% Amr::timeStep() 10 0.0002052 0.0002052 0.0002052 0.02% MLMG::MLRhsNormInf() 11 0.0002018 0.0002018 0.0002018 0.02% MLCellLinOp::defineBC() 11 0.0001953 0.0001953 0.0001953 0.02% MLLinOp::defineGrids() 11 0.0001752 0.0001752 0.0001752 0.02% StateData::checkPoint() 12 0.0001529 0.0001529 0.0001529 0.02% Amr::defBaseLevel() 1 0.0001516 0.0001516 0.0001516 0.02% MLMG:computeResOfCorrection() 410 0.0001456 0.0001456 0.0001456 0.02% AmrLevel::checkPoint() 3 0.0001106 0.0001106 0.0001106 0.01% MLMG::actualBottomSolve() 82 0.0001104 0.0001104 0.0001104 0.01% FabArrayBase::FB::FB() 56 9.103e-05 9.103e-05 9.103e-05 0.01% MLMG::mgVcycle_down::0 82 8.927e-05 8.927e-05 8.927e-05 0.01% Castro::initialize_advance() 10 8.119e-05 8.119e-05 8.119e-05 0.01% MLMG::mgVcycle_down::1 82 7.847e-05 7.847e-05 7.847e-05 0.01% MLMG::mgVcycle_down::2 82 7.587e-05 7.587e-05 7.587e-05 0.01% MLMG::mgVcycle_down::4 82 7.449e-05 7.449e-05 7.449e-05 0.01% MLMG::mgVcycle_down::3 82 7.37e-05 7.37e-05 7.37e-05 0.01% MLMG::solve() 11 7.119e-05 7.119e-05 7.119e-05 0.01% Castro::clean_state() 62 6.793e-05 6.793e-05 6.793e-05 0.01% MLMG::mgVcycle_up::4 82 6.131e-05 6.131e-05 6.131e-05 0.01% Castro::finalize_advance() 10 5.957e-05 5.957e-05 5.957e-05 0.01% Castro::initialize_do_advance() 10 5.638e-05 5.638e-05 5.638e-05 0.01% MLCellLinOp::solutionResidual() 93 5.568e-05 5.568e-05 5.568e-05 0.01% MLMG::mgVcycle_up::1 82 5.137e-05 5.137e-05 5.137e-05 0.01% MLMG::mgVcycle_up::0 82 5.098e-05 5.098e-05 5.098e-05 0.01% MLMG::mgVcycle_up::3 82 5.02e-05 5.02e-05 5.02e-05 0.01% MLMG::mgVcycle_up::2 82 4.904e-05 4.904e-05 4.904e-05 0.01% StateData::define() 4 4.174e-05 4.174e-05 4.174e-05 0.00% Castro::construct_new_source() 50 4.092e-05 4.092e-05 4.092e-05 0.00% Castro::swap_state_time_levels() 10 3.732e-05 3.732e-05 3.732e-05 0.00% Castro::finalize_do_advance() 10 3.642e-05 3.642e-05 3.642e-05 0.00% Castro::enforce_consistent_e() 1 3.596e-05 3.596e-05 3.596e-05 0.00% MLMG::computeResidual() 82 3.549e-05 3.549e-05 3.549e-05 0.00% MLMG::mgVcycle_bottom 82 3.197e-05 3.197e-05 3.197e-05 0.00% MLLinOp::define() 11 2.956e-05 2.956e-05 2.956e-05 0.00% FillPatchSingleLevel 41 2.864e-05 2.864e-05 2.864e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.759e-05 2.759e-05 2.759e-05 0.00% makeSFC 55 2.535e-05 2.535e-05 2.535e-05 0.00% Amr::writeSmallPlotFile() 1 2.227e-05 2.227e-05 2.227e-05 0.00% MLPoisson::define() 11 2.193e-05 2.193e-05 2.193e-05 0.00% Amr::FinalizeInit() 1 1.907e-05 1.907e-05 1.907e-05 0.00% Castro::do_new_sources() 10 1.693e-05 1.693e-05 1.693e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.671e-05 1.671e-05 1.671e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.636e-05 1.636e-05 1.636e-05 0.00% Castro::do_old_sources() 10 1.59e-05 1.59e-05 1.59e-05 0.00% Castro::construct_old_source() 50 1.577e-05 1.577e-05 1.577e-05 0.00% DistributionMapping::Distribute() 56 1.378e-05 1.378e-05 1.378e-05 0.00% Castro::apply_source_to_state() 20 1.247e-05 1.247e-05 1.247e-05 0.00% Castro::construct_old_gravity() 10 1.079e-05 1.079e-05 1.079e-05 0.00% Gravity::swapTimeLevels() 10 1.02e-05 1.02e-05 1.02e-05 0.00% AmrLevel::AmrLevel(dm) 1 9.986e-06 9.986e-06 9.986e-06 0.00% Amr::initSubcycle() 1 9.985e-06 9.985e-06 9.985e-06 0.00% MLPoisson::prepareForSolve() 11 8.221e-06 8.221e-06 8.221e-06 0.00% Castro::post_timestep() 10 7.46e-06 7.46e-06 7.46e-06 0.00% Amr::InitializeInit() 1 7.183e-06 7.183e-06 7.183e-06 0.00% MLMG::computeMLResidual() 11 6.248e-06 6.248e-06 6.248e-06 0.00% Castro::create_source_corrector() 10 6.063e-06 6.063e-06 6.063e-06 0.00% MLMG::getGradSolution() 11 5.831e-06 5.831e-06 5.831e-06 0.00% Castro::computeNewDt() 9 5.717e-06 5.717e-06 5.717e-06 0.00% MLMG::buildFineMask() 11 5.562e-06 5.562e-06 5.562e-06 0.00% MLMG::MLResNormInf() 11 4.988e-06 4.988e-06 4.988e-06 0.00% AmrLevel::checkPointPre() 3 4.801e-06 4.801e-06 4.801e-06 0.00% Gravity::set_mass_offset() 11 4.6e-06 4.6e-06 4.6e-06 0.00% AmrLevel::checkPointPost() 3 4.407e-06 4.407e-06 4.407e-06 0.00% Castro::FluxRegFineAdd() 10 4.166e-06 4.166e-06 4.166e-06 0.00% Castro::post_init() 1 4.105e-06 4.105e-06 4.105e-06 0.00% Castro::retry_advance_ctu() 10 4.013e-06 4.013e-06 4.013e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.045e-06 3.045e-06 3.045e-06 0.00% Castro::computeInitialDt() 2 2.731e-06 2.731e-06 2.731e-06 0.00% Amr::init() 1 2.705e-06 2.705e-06 2.705e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.238e-06 2.238e-06 2.238e-06 0.00% Castro::post_regrid() 1 1.187e-06 1.187e-06 1.187e-06 0.00% Amr::initialInit() 1 1.025e-06 1.025e-06 1.025e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9465 0.9465 0.9465 100.00% Amr::coarseTimeStep() 10 0.7937 0.7937 0.7937 83.86% Amr::timeStep() 10 0.6663 0.6663 0.6663 70.40% Castro::advance() 10 0.6577 0.6577 0.6577 69.49% Castro::subcycle_advance_ctu() 10 0.6428 0.6428 0.6428 67.91% Castro::do_advance_ctu() 10 0.6425 0.6425 0.6425 67.89% Gravity::solve_phi_with_mlmg() 11 0.3323 0.3323 0.3323 35.11% Gravity::actual_solve_with_mlmg() 11 0.3201 0.3201 0.3201 33.82% Castro::construct_new_gravity() 10 0.3018 0.3018 0.3018 31.89% MLMG::solve() 11 0.2969 0.2969 0.2969 31.37% Gravity::solve_for_phi() 10 0.2835 0.2835 0.2835 29.96% MLMG::oneIter() 82 0.2807 0.2807 0.2807 29.66% MLMG::mgVcycle() 82 0.2791 0.2791 0.2791 29.49% Castro::construct_ctu_hydro_source() 10 0.2482 0.2482 0.2482 26.23% Amr::checkPoint() 3 0.1645 0.1645 0.1645 17.38% VisMF::Write(FabArray) 11 0.1601 0.1601 0.1601 16.91% MLCellLinOp::smooth() 1640 0.1473 0.1473 0.1473 15.57% Amr::init() 1 0.1277 0.1277 0.1277 13.49% MLCellLinOp::applyBC() 4433 0.115 0.115 0.115 12.15% AmrLevel::checkPoint() 3 0.1143 0.1143 0.1143 12.08% StateData::checkPoint() 12 0.1142 0.1142 0.1142 12.07% MLMG::mgVcycle_bottom 82 0.08009 0.08009 0.08009 8.46% MLMG::actualBottomSolve() 82 0.08006 0.08006 0.08006 8.46% MLCGSolver::bicgstab 82 0.07931 0.07931 0.07931 8.38% FillPatchIterator::Initialize 41 0.0622 0.0622 0.0622 6.57% Amr::initialInit() 1 0.06208 0.06208 0.06208 6.56% MLPoisson::Fsmooth() 3280 0.06071 0.06071 0.06071 6.41% FillPatchSingleLevel 41 0.0566 0.0566 0.0566 5.98% Amr::FinalizeInit() 1 0.05268 0.05268 0.05268 5.57% StateDataPhysBCFunct::() 41 0.05237 0.05237 0.05237 5.53% Castro::post_init() 1 0.05197 0.05197 0.05197 5.49% Gravity::multilevel_solve_for_new_phi() 1 0.04947 0.04947 0.04947 5.23% Gravity::actual_multilevel_solve() 1 0.04945 0.04945 0.04945 5.23% Amr::writePlotFile() 2 0.04895 0.04895 0.04895 5.17% MLCellLinOp::apply() 1142 0.04034 0.04034 0.04034 4.26% MLMG::mgVcycle_down::0 82 0.03872 0.03872 0.03872 4.09% Castro::clean_state() 62 0.0375 0.0375 0.0375 3.96% MLMG::mgVcycle_up::0 82 0.03336 0.03336 0.03336 3.52% Castro::initialize_do_advance() 10 0.03021 0.03021 0.03021 3.19% FabArray::setVal() 1144 0.02709 0.02709 0.02709 2.86% Castro::expand_state() 10 0.02453 0.02453 0.02453 2.59% StateData::FillBoundary(geom) 328 0.0239 0.0239 0.0239 2.52% Castro::construct_old_gravity() 10 0.02275 0.02275 0.02275 2.40% Gravity::get_old_grav_vector() 10 0.02274 0.02274 0.02274 2.40% MLCellLinOp::correctionResidual() 492 0.02272 0.02272 0.02272 2.40% MultiFab::Dot() 1114 0.02226 0.02226 0.02226 2.35% Gravity::get_new_grav_vector() 11 0.02033 0.02033 0.02033 2.15% FabArray::FillBoundary() 4023 0.01979 0.01979 0.01979 2.09% MLMG:computeResOfCorrection() 410 0.01966 0.01966 0.01966 2.08% Castro::computeTemp() 63 0.0194 0.0194 0.0194 2.05% FillBoundary_nowait() 4023 0.01896 0.01896 0.01896 2.00% MLMG::mgVcycle_down::1 82 0.01885 0.01885 0.01885 1.99% MLMG::mgVcycle_down::2 82 0.01827 0.01827 0.01827 1.93% MLPoisson::define() 11 0.01776 0.01776 0.01776 1.88% FabArray::ParallelCopy() 861 0.01753 0.01753 0.01753 1.85% MLMG::mgVcycle_down::3 82 0.01748 0.01748 0.01748 1.85% FabArray::ParallelCopy_nowait() 861 0.01724 0.01724 0.01724 1.82% MLMG::mgVcycle_down::4 82 0.01668 0.01668 0.01668 1.76% Castro::initialize_advance() 10 0.01436 0.01436 0.01436 1.52% MLMG::mgVcycle_up::2 82 0.01423 0.01423 0.01423 1.50% MLMG::mgVcycle_up::1 82 0.01403 0.01403 0.01403 1.48% MLMG::addInterpCorrection() 410 0.0139 0.0139 0.0139 1.47% MLMG::mgVcycle_up::3 82 0.01358 0.01358 0.01358 1.43% MLMG::mgVcycle_up::4 82 0.01341 0.01341 0.01341 1.42% MLCGSolver::ParallelAllReduce 1514 0.0134 0.0134 0.0134 1.42% Castro::enforce_min_density() 62 0.0134 0.0134 0.0134 1.42% amrex::average_down 410 0.01333 0.01333 0.01333 1.41% CGSolver::sxay() 1586 0.01329 0.01329 0.01329 1.40% MultiFab::LinComb() 1586 0.0128 0.0128 0.0128 1.35% MLCellLinOp::defineAuxData() 11 0.01217 0.01217 0.01217 1.29% Gravity::fill_multipole_BCs() 11 0.01197 0.01197 0.01197 1.26% Castro::do_new_sources() 10 0.01155 0.01155 0.01155 1.22% MLPoisson::Fapply() 1142 0.01087 0.01087 0.01087 1.15% Amr::InitializeInit() 1 0.009405 0.009405 0.009405 0.99% Amr::defBaseLevel() 1 0.009398 0.009398 0.009398 0.99% Castro::do_old_sources() 10 0.009174 0.009174 0.009174 0.97% Castro::post_timestep() 10 0.008369 0.008369 0.008369 0.88% MLCellLinOp::solutionResidual() 93 0.007905 0.007905 0.007905 0.84% Castro::estTimeStep() 21 0.007043 0.007043 0.007043 0.74% MLMG::computeResidual() 82 0.006836 0.006836 0.006836 0.72% MultiFab::Xpay() 585 0.006221 0.006221 0.006221 0.66% MLMG::prepareForSolve() 11 0.005998 0.005998 0.005998 0.63% Castro::reset_internal_energy() 63 0.005976 0.005976 0.005976 0.63% FabArray::setDomainBndry() 41 0.00535 0.00535 0.00535 0.57% MLCellLinOp::defineBC() 11 0.005311 0.005311 0.005311 0.56% BndryData::define() 11 0.005115 0.005115 0.005115 0.54% Castro::initData() 1 0.003204 0.003204 0.003204 0.34% Castro::construct_new_source() 50 0.003109 0.003109 0.003109 0.33% Castro::computeNewDt() 9 0.003108 0.003108 0.003108 0.33% Castro::construct_new_gravity_source() 10 0.003068 0.003068 0.003068 0.32% Castro::enforce_speed_limit() 62 0.002679 0.002679 0.002679 0.28% Castro::normalize_species() 62 0.002635 0.002635 0.002635 0.28% Castro::construct_old_source() 50 0.002559 0.002559 0.002559 0.27% Castro::construct_old_gravity_source() 10 0.002544 0.002544 0.002544 0.27% MLMG::getGradSolution() 11 0.002498 0.002498 0.002498 0.26% MLCellLinOp::compGrad() 11 0.002493 0.002493 0.002493 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002358 0.002358 0.002358 0.25% MLMG::ResNormInf() 93 0.001919 0.001919 0.001919 0.20% Castro::apply_source_to_state() 20 0.001832 0.001832 0.001832 0.19% MultiFab::Saxpy() 20 0.00182 0.00182 0.00182 0.19% FabArrayBase::getCPC() 1323 0.001625 0.001625 0.001625 0.17% MLCellLinOp::setLevelBC() 11 0.001553 0.001553 0.001553 0.16% MLPoisson::prepareForSolve() 11 0.00138 0.00138 0.00138 0.15% MLCellLinOp::prepareForSolve() 11 0.001372 0.001372 0.001372 0.14% FabArray::mult() 43 0.001342 0.001342 0.001342 0.14% MLMG::computeMLResidual() 11 0.001111 0.001111 0.001111 0.12% Gravity::swapTimeLevels() 10 0.00104 0.00104 0.00104 0.11% Gravity::update_max_rhs() 11 0.0009548 0.0009548 0.0009548 0.10% FabArrayBase::getFB() 4023 0.0008197 0.0008197 0.0008197 0.09% FabArrayBase::CPC::define() 454 0.0007251 0.0007251 0.0007251 0.08% Castro::computeInitialDt() 2 0.0006782 0.0006782 0.0006782 0.07% MultiFab::Copy() 11 0.0005313 0.0005313 0.0005313 0.06% Amr::InitAmr() 1 0.0004927 0.0004927 0.0004927 0.05% Castro::post_regrid() 1 0.0004748 0.0004748 0.0004748 0.05% MultiFab::min() 10 0.0003165 0.0003165 0.0003165 0.03% MLLinOp::define() 11 0.0002614 0.0002614 0.0002614 0.03% MLMG::MLResNormInf() 11 0.0002572 0.0002572 0.0002572 0.03% MultiFab::max() 11 0.000255 0.000255 0.000255 0.03% MLLinOp::defineGrids() 11 0.0002318 0.0002318 0.0002318 0.02% MLMG::MLRhsNormInf() 11 0.0002018 0.0002018 0.0002018 0.02% FabArrayBase::FB::FB() 56 9.103e-05 9.103e-05 9.103e-05 0.01% Castro::finalize_advance() 10 6.373e-05 6.373e-05 6.373e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.444e-05 5.444e-05 5.444e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.173e-05 5.173e-05 5.173e-05 0.01% StateData::define() 4 4.174e-05 4.174e-05 4.174e-05 0.00% makeSFC 55 3.808e-05 3.808e-05 3.808e-05 0.00% Castro::swap_state_time_levels() 10 3.732e-05 3.732e-05 3.732e-05 0.00% Castro::finalize_do_advance() 10 3.642e-05 3.642e-05 3.642e-05 0.00% Castro::enforce_consistent_e() 1 3.596e-05 3.596e-05 3.596e-05 0.00% Amr::writeSmallPlotFile() 1 2.227e-05 2.227e-05 2.227e-05 0.00% DistributionMapping::Distribute() 56 1.378e-05 1.378e-05 1.378e-05 0.00% Amr::initSubcycle() 1 9.985e-06 9.985e-06 9.985e-06 0.00% Castro::create_source_corrector() 10 6.063e-06 6.063e-06 6.063e-06 0.00% MLMG::buildFineMask() 11 5.562e-06 5.562e-06 5.562e-06 0.00% AmrLevel::checkPointPre() 3 4.801e-06 4.801e-06 4.801e-06 0.00% Gravity::set_mass_offset() 11 4.6e-06 4.6e-06 4.6e-06 0.00% AmrLevel::checkPointPost() 3 4.407e-06 4.407e-06 4.407e-06 0.00% Castro::FluxRegFineAdd() 10 4.166e-06 4.166e-06 4.166e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.09e-06 4.09e-06 4.09e-06 0.00% Castro::retry_advance_ctu() 10 4.013e-06 4.013e-06 4.013e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.238e-06 2.238e-06 2.238e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.02-14-gaa0dc39f50f7) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.02-14-gaa0dc39f50f7) initialized Starting run at 08:54:41 UTC on 2022-02-21. Successfully read inputs file ... Castro git describe: 22.02-4-g0d942c92d AMReX git describe: 22.02-14-gaa0dc39f5 Microphysics git describe: 22.02-4-ga894d649 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.51083164 Restart time = 0.05562174 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.095386131 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049072475 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.048952616 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.067902016 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.073402924 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024644528 seconds Ending run at 08:54:41 UTC on 2022-02-21. Run time = 0.415911099 Run time without initialization = 0.359700621 Average number of zones advanced per microsecond: 3.644 Average number of zones advanced per microsecond per rank: 3.644 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4159 ... 0.4159 ... 0.4159 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1290 0.1290 0.1290 31.00% VisMF::Read() 3 0.04356 0.04356 0.04356 10.47% MLCellLinOp::applyBC() 1946 0.03831 0.03831 0.03831 9.21% MLPoisson::Fsmooth() 1440 0.02604 0.02604 0.02604 6.26% VisMF::Write(FabArray) 1 0.02335 0.02335 0.02335 5.61% FabArray::setVal() 537 0.01623 0.01623 0.01623 3.90% StateData::FillBoundary(geom) 160 0.01151 0.01151 0.01151 2.77% MLCGSolver::bicgstab 36 0.009515 0.009515 0.009515 2.29% MultiFab::Dot() 484 0.009485 0.009485 0.009485 2.28% StateDataPhysBCFunct::() 20 0.007274 0.007274 0.007274 1.75% FillBoundary_nowait() 1766 0.007238 0.007238 0.007238 1.74% FabArray::ParallelCopy_nowait() 380 0.007162 0.007162 0.007162 1.72% Gravity::fill_multipole_BCs() 6 0.006182 0.006182 0.006182 1.49% Castro::computeTemp() 30 0.005655 0.005655 0.005655 1.36% MLCellLinOp::defineAuxData() 6 0.005567 0.005567 0.005567 1.34% Castro::enforce_min_density() 30 0.005413 0.005413 0.005413 1.30% MultiFab::LinComb() 690 0.005397 0.005397 0.005397 1.30% Castro::estTimeStep() 10 0.005027 0.005027 0.005027 1.21% MLPoisson::Fapply() 500 0.004722 0.004722 0.004722 1.14% FabArray::setDomainBndry() 20 0.00394 0.00394 0.00394 0.95% MLMG::addInterpCorrection() 180 0.003064 0.003064 0.003064 0.74% Amr::restart() 1 0.002941 0.002941 0.002941 0.71% amrex::average_down 180 0.002853 0.002853 0.002853 0.69% MultiFab::Xpay() 258 0.002697 0.002697 0.002697 0.65% Castro::expand_state() 5 0.002385 0.002385 0.002385 0.57% Castro::reset_internal_energy() 30 0.002337 0.002337 0.002337 0.56% BndryData::define() 6 0.002056 0.002056 0.002056 0.49% Castro::do_advance_ctu() 5 0.001835 0.001835 0.001835 0.44% Gravity::get_new_grav_vector() 5 0.001789 0.001789 0.001789 0.43% MLCellLinOp::compGrad() 6 0.001537 0.001537 0.001537 0.37% Amr::writePlotFile() 1 0.001372 0.001372 0.001372 0.33% Castro::normalize_species() 30 0.001305 0.001305 0.001305 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.00124 0.00124 0.00124 0.30% Castro::construct_new_gravity_source() 5 0.001157 0.001157 0.001157 0.28% Gravity::get_old_grav_vector() 5 0.0009911 0.0009911 0.0009911 0.24% MultiFab::Saxpy() 10 0.0009224 0.0009224 0.0009224 0.22% MLMG::ResNormInf() 42 0.0008525 0.0008525 0.0008525 0.20% Castro::construct_old_gravity_source() 5 0.0008423 0.0008423 0.0008423 0.20% MLCellLinOp::setLevelBC() 6 0.0008341 0.0008341 0.0008341 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007424 0.0007424 0.0007424 0.18% MLMG::oneIter() 36 0.0007388 0.0007388 0.0007388 0.18% MLCellLinOp::prepareForSolve() 6 0.0007033 0.0007033 0.0007033 0.17% Castro::subcycle_advance_ctu() 5 0.0006914 0.0006914 0.0006914 0.17% FabArray::mult() 22 0.0006609 0.0006609 0.0006609 0.16% Gravity::update_max_rhs() 6 0.0005695 0.0005695 0.0005695 0.14% MLCellLinOp::smooth() 720 0.0005223 0.0005223 0.0005223 0.13% FabArrayBase::getCPC() 632 0.0004612 0.0004612 0.0004612 0.11% Castro::enforce_speed_limit() 30 0.0004483 0.0004483 0.0004483 0.11% MLMG::prepareForSolve() 6 0.0004431 0.0004431 0.0004431 0.11% FabArrayBase::CPC::define() 244 0.000426 0.000426 0.000426 0.10% FabArray::FillBoundary() 1766 0.0004085 0.0004085 0.0004085 0.10% Amr::InitAmr() 1 0.0003979 0.0003979 0.0003979 0.10% MultiFab::Copy() 6 0.0003617 0.0003617 0.0003617 0.09% FabArrayBase::getFB() 1766 0.0003191 0.0003191 0.0003191 0.08% main() 1 0.0002774 0.0002774 0.0002774 0.07% Gravity::actual_multilevel_solve() 1 0.0002332 0.0002332 0.0002332 0.06% MLCellLinOp::apply() 500 0.0002215 0.0002215 0.0002215 0.05% Gravity::solve_for_phi() 5 0.000221 0.000221 0.000221 0.05% CGSolver::sxay() 690 0.0002072 0.0002072 0.0002072 0.05% MLMG::mgVcycle() 36 0.0001847 0.0001847 0.0001847 0.04% Castro::construct_new_gravity() 5 0.0001835 0.0001835 0.0001835 0.04% MultiFab::min() 5 0.0001786 0.0001786 0.0001786 0.04% Castro::construct_new_source() 25 0.000159 0.000159 0.000159 0.04% MLCGSolver::ParallelAllReduce 659 0.0001515 0.0001515 0.0001515 0.04% FabArray::ParallelCopy() 380 0.000145 0.000145 0.000145 0.03% MultiFab::max() 6 0.0001343 0.0001343 0.0001343 0.03% FillPatchIterator::Initialize 20 0.0001228 0.0001228 0.0001228 0.03% Castro::advance() 5 0.0001147 0.0001147 0.0001147 0.03% MLMG::MLRhsNormInf() 6 0.0001114 0.0001114 0.0001114 0.03% Amr::coarseTimeStep() 5 0.0001098 0.0001098 0.0001098 0.03% Castro::create_source_corrector() 5 0.0001043 0.0001043 0.0001043 0.03% MLCellLinOp::defineBC() 6 0.0001041 0.0001041 0.0001041 0.03% Amr::timeStep() 5 0.0001033 0.0001033 0.0001033 0.02% MLCellLinOp::correctionResidual() 216 9.492e-05 9.492e-05 9.492e-05 0.02% MLLinOp::defineGrids() 6 9.331e-05 9.331e-05 9.331e-05 0.02% StateData::restartDoit() 4 8.421e-05 8.421e-05 8.421e-05 0.02% AmrLevel::restart() 1 7.84e-05 7.84e-05 7.84e-05 0.02% MLMG:computeResOfCorrection() 180 6.426e-05 6.426e-05 6.426e-05 0.02% FabArrayBase::FB::FB() 26 6.274e-05 6.274e-05 6.274e-05 0.02% MLMG::actualBottomSolve() 36 4.827e-05 4.827e-05 4.827e-05 0.01% MLMG::mgVcycle_down::0 36 4.344e-05 4.344e-05 4.344e-05 0.01% MLMG::mgVcycle_down::1 36 3.907e-05 3.907e-05 3.907e-05 0.01% Castro::initialize_advance() 5 3.808e-05 3.808e-05 3.808e-05 0.01% Castro::do_new_sources() 5 3.714e-05 3.714e-05 3.714e-05 0.01% Castro::construct_old_source() 25 3.561e-05 3.561e-05 3.561e-05 0.01% MLMG::mgVcycle_down::2 36 3.537e-05 3.537e-05 3.537e-05 0.01% MLMG::solve() 6 3.455e-05 3.455e-05 3.455e-05 0.01% Castro::clean_state() 30 3.393e-05 3.393e-05 3.393e-05 0.01% MLMG::mgVcycle_down::4 36 3.372e-05 3.372e-05 3.372e-05 0.01% MLMG::mgVcycle_down::3 36 3.317e-05 3.317e-05 3.317e-05 0.01% Castro::initialize_do_advance() 5 2.976e-05 2.976e-05 2.976e-05 0.01% MLLinOp::define() 6 2.859e-05 2.859e-05 2.859e-05 0.01% Castro::swap_state_time_levels() 5 2.672e-05 2.672e-05 2.672e-05 0.01% MLMG::mgVcycle_up::4 36 2.651e-05 2.651e-05 2.651e-05 0.01% Castro::post_restart() 1 2.621e-05 2.621e-05 2.621e-05 0.01% Castro::finalize_advance() 5 2.603e-05 2.603e-05 2.603e-05 0.01% MLCellLinOp::solutionResidual() 42 2.491e-05 2.491e-05 2.491e-05 0.01% MLMG::mgVcycle_up::1 36 2.431e-05 2.431e-05 2.431e-05 0.01% MLMG::mgVcycle_up::0 36 2.412e-05 2.412e-05 2.412e-05 0.01% Amr::writeSmallPlotFile() 1 2.318e-05 2.318e-05 2.318e-05 0.01% MLMG::mgVcycle_up::3 36 2.276e-05 2.276e-05 2.276e-05 0.01% Castro::finalize_do_advance() 5 2.187e-05 2.187e-05 2.187e-05 0.01% MLMG::mgVcycle_up::2 36 2.129e-05 2.129e-05 2.129e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.622e-05 1.622e-05 1.622e-05 0.00% MLMG::computeResidual() 36 1.596e-05 1.596e-05 1.596e-05 0.00% MLMG::mgVcycle_bottom 36 1.534e-05 1.534e-05 1.534e-05 0.00% FillPatchSingleLevel 20 1.503e-05 1.503e-05 1.503e-05 0.00% MLPoisson::define() 6 1.471e-05 1.471e-05 1.471e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.455e-05 1.455e-05 1.455e-05 0.00% makeSFC 30 1.386e-05 1.386e-05 1.386e-05 0.00% Amr::initSubcycle() 1 9.159e-06 9.159e-06 9.159e-06 0.00% Castro::do_old_sources() 5 8.567e-06 8.567e-06 8.567e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.422e-06 8.422e-06 8.422e-06 0.00% DistributionMapping::Distribute() 31 8.285e-06 8.285e-06 8.285e-06 0.00% Castro::apply_source_to_state() 10 6.263e-06 6.263e-06 6.263e-06 0.00% Castro::construct_old_gravity() 5 5.678e-06 5.678e-06 5.678e-06 0.00% Castro::post_timestep() 5 4.779e-06 4.779e-06 4.779e-06 0.00% Gravity::swapTimeLevels() 5 4.602e-06 4.602e-06 4.602e-06 0.00% MLPoisson::prepareForSolve() 6 4.45e-06 4.45e-06 4.45e-06 0.00% MLMG::buildFineMask() 6 3.606e-06 3.606e-06 3.606e-06 0.00% Castro::computeNewDt() 5 3.465e-06 3.465e-06 3.465e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.366e-06 3.366e-06 3.366e-06 0.00% MLMG::getGradSolution() 6 3.126e-06 3.126e-06 3.126e-06 0.00% MLMG::computeMLResidual() 6 3.003e-06 3.003e-06 3.003e-06 0.00% MLMG::MLResNormInf() 6 2.511e-06 2.511e-06 2.511e-06 0.00% Gravity::set_mass_offset() 6 2.329e-06 2.329e-06 2.329e-06 0.00% Castro::FluxRegFineAdd() 5 1.751e-06 1.751e-06 1.751e-06 0.00% Castro::retry_advance_ctu() 5 1.607e-06 1.607e-06 1.607e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.152e-06 1.152e-06 1.152e-06 0.00% Amr::init() 1 1.069e-06 1.069e-06 1.069e-06 0.00% AmrLevel::AmrLevel() 1 8.73e-07 8.73e-07 8.73e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4159 0.4159 0.4159 100.00% Amr::coarseTimeStep() 5 0.3348 0.3348 0.3348 80.50% Amr::timeStep() 5 0.3318 0.3318 0.3318 79.78% Castro::advance() 5 0.3291 0.3291 0.3291 79.11% Castro::subcycle_advance_ctu() 5 0.3182 0.3182 0.3182 76.51% Castro::do_advance_ctu() 5 0.3176 0.3176 0.3176 76.35% Castro::construct_new_gravity() 5 0.1487 0.1487 0.1487 35.75% Gravity::solve_phi_with_mlmg() 6 0.1447 0.1447 0.1447 34.80% Gravity::solve_for_phi() 5 0.1387 0.1387 0.1387 33.35% Gravity::actual_solve_with_mlmg() 6 0.1384 0.1384 0.1384 33.28% Castro::construct_ctu_hydro_source() 5 0.129 0.129 0.129 31.00% MLMG::solve() 6 0.1254 0.1254 0.1254 30.15% MLMG::oneIter() 36 0.1173 0.1173 0.1173 28.20% MLMG::mgVcycle() 36 0.1166 0.1166 0.1166 28.02% MLCellLinOp::smooth() 720 0.06079 0.06079 0.06079 14.61% Amr::init() 1 0.05567 0.05567 0.05567 13.38% Amr::restart() 1 0.05567 0.05567 0.05567 13.38% MLCellLinOp::applyBC() 1946 0.04634 0.04634 0.04634 11.14% AmrLevel::restart() 1 0.04444 0.04444 0.04444 10.68% StateData::restartDoit() 4 0.04436 0.04436 0.04436 10.66% VisMF::Read() 3 0.04356 0.04356 0.04356 10.47% MLMG::mgVcycle_bottom 36 0.03389 0.03389 0.03389 8.15% MLMG::actualBottomSolve() 36 0.03388 0.03388 0.03388 8.14% MLCGSolver::bicgstab 36 0.03355 0.03355 0.03355 8.07% MLPoisson::Fsmooth() 1440 0.02604 0.02604 0.02604 6.26% FillPatchIterator::Initialize 20 0.02493 0.02493 0.02493 5.99% Amr::writePlotFile() 1 0.02473 0.02473 0.02473 5.94% VisMF::Write(FabArray) 1 0.02335 0.02335 0.02335 5.61% FillPatchSingleLevel 20 0.02087 0.02087 0.02087 5.02% StateDataPhysBCFunct::() 20 0.01879 0.01879 0.01879 4.52% MLCellLinOp::apply() 500 0.01676 0.01676 0.01676 4.03% FabArray::setVal() 537 0.01623 0.01623 0.01623 3.90% MLMG::mgVcycle_down::0 36 0.01613 0.01613 0.01613 3.88% Castro::clean_state() 30 0.01519 0.01519 0.01519 3.65% MLMG::mgVcycle_up::0 36 0.01378 0.01378 0.01378 3.31% Castro::initialize_do_advance() 5 0.01258 0.01258 0.01258 3.02% StateData::FillBoundary(geom) 160 0.01151 0.01151 0.01151 2.77% Castro::initialize_advance() 5 0.01066 0.01066 0.01066 2.56% Castro::expand_state() 5 0.01005 0.01005 0.01005 2.42% Gravity::get_new_grav_vector() 5 0.009799 0.009799 0.009799 2.36% MLPoisson::define() 6 0.009591 0.009591 0.009591 2.31% MultiFab::Dot() 484 0.009485 0.009485 0.009485 2.28% MLCellLinOp::correctionResidual() 216 0.009446 0.009446 0.009446 2.27% MLMG:computeResOfCorrection() 180 0.008172 0.008172 0.008172 1.96% Castro::construct_old_gravity() 5 0.008122 0.008122 0.008122 1.95% Gravity::get_old_grav_vector() 5 0.008116 0.008116 0.008116 1.95% FabArray::FillBoundary() 1766 0.008028 0.008028 0.008028 1.93% Castro::computeTemp() 30 0.007992 0.007992 0.007992 1.92% MLMG::mgVcycle_down::1 36 0.00786 0.00786 0.00786 1.89% FabArray::ParallelCopy() 380 0.007737 0.007737 0.007737 1.86% MLMG::mgVcycle_down::2 36 0.007626 0.007626 0.007626 1.83% FillBoundary_nowait() 1766 0.00762 0.00762 0.00762 1.83% FabArray::ParallelCopy_nowait() 380 0.007592 0.007592 0.007592 1.83% MLMG::mgVcycle_down::3 36 0.007244 0.007244 0.007244 1.74% MLMG::mgVcycle_down::4 36 0.006963 0.006963 0.006963 1.67% Castro::post_restart() 1 0.006913 0.006913 0.006913 1.66% MLCellLinOp::defineAuxData() 6 0.006531 0.006531 0.006531 1.57% Gravity::multilevel_solve_for_new_phi() 1 0.00648 0.00648 0.00648 1.56% Gravity::actual_multilevel_solve() 1 0.006464 0.006464 0.006464 1.55% Gravity::fill_multipole_BCs() 6 0.006182 0.006182 0.006182 1.49% MLMG::mgVcycle_up::2 36 0.005894 0.005894 0.005894 1.42% MLMG::addInterpCorrection() 180 0.005881 0.005881 0.005881 1.41% MLMG::mgVcycle_up::1 36 0.005818 0.005818 0.005818 1.40% MLCGSolver::ParallelAllReduce 659 0.00575 0.00575 0.00575 1.38% amrex::average_down 180 0.005703 0.005703 0.005703 1.37% Castro::do_new_sources() 5 0.00563 0.00563 0.00563 1.35% CGSolver::sxay() 690 0.005604 0.005604 0.005604 1.35% MLMG::mgVcycle_up::3 36 0.005598 0.005598 0.005598 1.35% MLMG::mgVcycle_up::4 36 0.005569 0.005569 0.005569 1.34% Castro::enforce_min_density() 30 0.005413 0.005413 0.005413 1.30% MultiFab::LinComb() 690 0.005397 0.005397 0.005397 1.30% Castro::estTimeStep() 10 0.005027 0.005027 0.005027 1.21% MLPoisson::Fapply() 500 0.004722 0.004722 0.004722 1.14% FabArray::setDomainBndry() 20 0.00394 0.00394 0.00394 0.95% MLMG::prepareForSolve() 6 0.003664 0.003664 0.003664 0.88% Castro::do_old_sources() 5 0.003566 0.003566 0.003566 0.86% MLCellLinOp::solutionResidual() 42 0.003438 0.003438 0.003438 0.83% MLCellLinOp::defineBC() 6 0.002892 0.002892 0.002892 0.70% Castro::computeNewDt() 5 0.002888 0.002888 0.002888 0.69% MLMG::computeResidual() 36 0.002848 0.002848 0.002848 0.68% BndryData::define() 6 0.002788 0.002788 0.002788 0.67% MultiFab::Xpay() 258 0.002697 0.002697 0.002697 0.65% Castro::post_timestep() 5 0.002667 0.002667 0.002667 0.64% Castro::reset_internal_energy() 30 0.002337 0.002337 0.002337 0.56% MLMG::getGradSolution() 6 0.001836 0.001836 0.001836 0.44% MLCellLinOp::compGrad() 6 0.001833 0.001833 0.001833 0.44% Castro::construct_new_source() 25 0.001316 0.001316 0.001316 0.32% Castro::normalize_species() 30 0.001305 0.001305 0.001305 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.00124 0.00124 0.00124 0.30% Castro::construct_new_gravity_source() 5 0.001157 0.001157 0.001157 0.28% Gravity::swapTimeLevels() 5 0.0009439 0.0009439 0.0009439 0.23% Castro::apply_source_to_state() 10 0.0009286 0.0009286 0.0009286 0.22% MultiFab::Saxpy() 10 0.0009224 0.0009224 0.0009224 0.22% FabArrayBase::getCPC() 632 0.0008872 0.0008872 0.0008872 0.21% Castro::construct_old_source() 25 0.0008779 0.0008779 0.0008779 0.21% MLMG::ResNormInf() 42 0.0008525 0.0008525 0.0008525 0.20% Castro::construct_old_gravity_source() 5 0.0008423 0.0008423 0.0008423 0.20% MLCellLinOp::setLevelBC() 6 0.0008341 0.0008341 0.0008341 0.20% Gravity::update_max_rhs() 6 0.0007833 0.0007833 0.0007833 0.19% MLPoisson::prepareForSolve() 6 0.0007077 0.0007077 0.0007077 0.17% MLCellLinOp::prepareForSolve() 6 0.0007033 0.0007033 0.0007033 0.17% FabArray::mult() 22 0.0006609 0.0006609 0.0006609 0.16% MLMG::computeMLResidual() 6 0.0006094 0.0006094 0.0006094 0.15% Castro::enforce_speed_limit() 30 0.0004483 0.0004483 0.0004483 0.11% FabArrayBase::CPC::define() 244 0.000426 0.000426 0.000426 0.10% Amr::InitAmr() 1 0.0004071 0.0004071 0.0004071 0.10% FabArrayBase::getFB() 1766 0.0003818 0.0003818 0.0003818 0.09% MultiFab::Copy() 6 0.0003617 0.0003617 0.0003617 0.09% MultiFab::min() 5 0.0001786 0.0001786 0.0001786 0.04% MLLinOp::define() 6 0.0001525 0.0001525 0.0001525 0.04% MLMG::MLResNormInf() 6 0.0001402 0.0001402 0.0001402 0.03% MultiFab::max() 6 0.0001343 0.0001343 0.0001343 0.03% MLLinOp::defineGrids() 6 0.0001239 0.0001239 0.0001239 0.03% MLMG::MLRhsNormInf() 6 0.0001114 0.0001114 0.0001114 0.03% Castro::create_source_corrector() 5 0.0001043 0.0001043 0.0001043 0.03% FabArrayBase::FB::FB() 26 6.274e-05 6.274e-05 6.274e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.948e-05 2.948e-05 2.948e-05 0.01% Castro::finalize_advance() 5 2.778e-05 2.778e-05 2.778e-05 0.01% Castro::swap_state_time_levels() 5 2.672e-05 2.672e-05 2.672e-05 0.01% Amr::writeSmallPlotFile() 1 2.318e-05 2.318e-05 2.318e-05 0.01% Castro::finalize_do_advance() 5 2.187e-05 2.187e-05 2.187e-05 0.01% makeSFC 30 2.106e-05 2.106e-05 2.106e-05 0.01% Amr::initSubcycle() 1 9.159e-06 9.159e-06 9.159e-06 0.00% DistributionMapping::Distribute() 31 8.285e-06 8.285e-06 8.285e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.457e-06 4.457e-06 4.457e-06 0.00% MLMG::buildFineMask() 6 3.606e-06 3.606e-06 3.606e-06 0.00% Gravity::set_mass_offset() 6 2.329e-06 2.329e-06 2.329e-06 0.00% Castro::FluxRegFineAdd() 5 1.751e-06 1.751e-06 1.751e-06 0.00% Castro::retry_advance_ctu() 5 1.607e-06 1.607e-06 1.607e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.152e-06 1.152e-06 1.152e-06 0.00% AmrLevel::AmrLevel() 1 8.73e-07 8.73e-07 8.73e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.02-14-gaa0dc39f50f7) finalized