Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.03-8-g6bbcbbb78299) initialized Starting run at 08:53:55 UTC on 2022-03-08. Successfully read inputs file ... Castro git describe: 22.02-4-g0d942c92d AMReX git describe: 22.03-8-g6bbcbbb78 Microphysics git describe: 22.03-2-ge2419c42 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.037764035 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022609508 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.107395146 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049189482 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.055317574 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.056465281 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.057805414 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.03527138 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.060961551 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049719764 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.049022429 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057460562 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.071865913 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.0352476 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.022002525 seconds Ending run at 08:53:56 UTC on 2022-03-08. Run time = 0.830212952 Run time without initialization = 0.708210498 Average number of zones advanced per microsecond: 3.701 Average number of zones advanced per microsecond per rank: 3.701 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8303 ... 0.8303 ... 0.8303 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2219 0.2219 0.2219 26.72% VisMF::Write(FabArray) 11 0.1462 0.1462 0.1462 17.61% MLCellLinOp::applyBC() 4433 0.09439 0.09439 0.09439 11.37% MLPoisson::Fsmooth() 3280 0.05988 0.05988 0.05988 7.21% FabArray::setVal() 1144 0.02569 0.02569 0.02569 3.09% StateData::FillBoundary(geom) 328 0.02341 0.02341 0.02341 2.82% MultiFab::Dot() 1114 0.02177 0.02177 0.02177 2.62% MLCGSolver::bicgstab 82 0.02151 0.02151 0.02151 2.59% FillBoundary_nowait() 4023 0.01827 0.01827 0.01827 2.20% FabArray::ParallelCopy_nowait() 861 0.0161 0.0161 0.0161 1.94% StateDataPhysBCFunct::() 41 0.01605 0.01605 0.01605 1.93% Castro::computeTemp() 63 0.01303 0.01303 0.01303 1.57% MultiFab::LinComb() 1586 0.01226 0.01226 0.01226 1.48% Gravity::fill_multipole_BCs() 11 0.0112 0.0112 0.0112 1.35% MLPoisson::Fapply() 1142 0.01052 0.01052 0.01052 1.27% Castro::enforce_min_density() 62 0.01026 0.01026 0.01026 1.24% MLCellLinOp::defineAuxData() 11 0.01004 0.01004 0.01004 1.21% MLMG::addInterpCorrection() 410 0.007115 0.007115 0.007115 0.86% amrex::average_down 410 0.006519 0.006519 0.006519 0.79% MultiFab::Xpay() 585 0.006016 0.006016 0.006016 0.72% Castro::estTimeStep() 21 0.005232 0.005232 0.005232 0.63% Castro::reset_internal_energy() 63 0.00514 0.00514 0.00514 0.62% FabArray::setDomainBndry() 41 0.005105 0.005105 0.005105 0.61% Castro::expand_state() 10 0.004611 0.004611 0.004611 0.56% Amr::checkPoint() 3 0.003791 0.003791 0.003791 0.46% BndryData::define() 11 0.003567 0.003567 0.003567 0.43% Castro::do_advance_ctu() 10 0.00321 0.00321 0.00321 0.39% Amr::writePlotFile() 2 0.002933 0.002933 0.002933 0.35% Castro::normalize_species() 62 0.002609 0.002609 0.002609 0.31% Castro::construct_new_gravity_source() 10 0.002562 0.002562 0.002562 0.31% Gravity::get_new_grav_vector() 11 0.002303 0.002303 0.002303 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002267 0.002267 0.002267 0.27% Castro::enforce_speed_limit() 62 0.00217 0.00217 0.00217 0.26% Castro::construct_old_gravity_source() 10 0.002006 0.002006 0.002006 0.24% MLMG::ResNormInf() 93 0.001906 0.001906 0.001906 0.23% MLCellLinOp::compGrad() 11 0.001834 0.001834 0.001834 0.22% MultiFab::Saxpy() 20 0.001814 0.001814 0.001814 0.22% Gravity::get_old_grav_vector() 10 0.001759 0.001759 0.001759 0.21% MLMG::oneIter() 82 0.001636 0.001636 0.001636 0.20% MLCellLinOp::setLevelBC() 11 0.001538 0.001538 0.001538 0.19% Gravity::actual_solve_with_mlmg() 11 0.001476 0.001476 0.001476 0.18% MLCellLinOp::prepareForSolve() 11 0.001342 0.001342 0.001342 0.16% FabArray::mult() 43 0.001324 0.001324 0.001324 0.16% Castro::initData() 1 0.001216 0.001216 0.001216 0.15% MLCellLinOp::smooth() 1640 0.00112 0.00112 0.00112 0.13% FabArrayBase::getCPC() 1323 0.0009645 0.0009645 0.0009645 0.12% FabArray::FillBoundary() 4023 0.0008247 0.0008247 0.0008247 0.10% MLMG::prepareForSolve() 11 0.0007674 0.0007674 0.0007674 0.09% FabArrayBase::getFB() 4023 0.0007433 0.0007433 0.0007433 0.09% FabArrayBase::CPC::define() 454 0.0007287 0.0007287 0.0007287 0.09% Gravity::update_max_rhs() 11 0.0005501 0.0005501 0.0005501 0.07% MLCellLinOp::apply() 1142 0.0005155 0.0005155 0.0005155 0.06% MultiFab::Copy() 11 0.0005154 0.0005154 0.0005154 0.06% CGSolver::sxay() 1586 0.0004748 0.0004748 0.0004748 0.06% Gravity::solve_for_phi() 10 0.0004671 0.0004671 0.0004671 0.06% Amr::InitAmr() 1 0.0004587 0.0004587 0.0004587 0.06% MLMG::mgVcycle() 82 0.0004193 0.0004193 0.0004193 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003534 0.0003534 0.0003534 0.04% MultiFab::min() 10 0.000323 0.000323 0.000323 0.04% FabArray::ParallelCopy() 861 0.0003028 0.0003028 0.0003028 0.04% main() 1 0.0002947 0.0002947 0.0002947 0.04% MultiFab::max() 11 0.0002558 0.0002558 0.0002558 0.03% FillPatchIterator::Initialize 41 0.0002503 0.0002503 0.0002503 0.03% MLCellLinOp::correctionResidual() 492 0.0002463 0.0002463 0.0002463 0.03% Castro::construct_new_gravity() 10 0.0002376 0.0002376 0.0002376 0.03% Gravity::actual_multilevel_solve() 1 0.0002259 0.0002259 0.0002259 0.03% Amr::coarseTimeStep() 10 0.0002126 0.0002126 0.0002126 0.03% MLMG::MLRhsNormInf() 11 0.0002012 0.0002012 0.0002012 0.02% Amr::timeStep() 10 0.0001982 0.0001982 0.0001982 0.02% MLCellLinOp::defineBC() 11 0.0001941 0.0001941 0.0001941 0.02% MLLinOp::defineGrids() 11 0.0001869 0.0001869 0.0001869 0.02% Amr::defBaseLevel() 1 0.0001542 0.0001542 0.0001542 0.02% Castro::subcycle_advance_ctu() 10 0.0001516 0.0001516 0.0001516 0.02% MLMG:computeResOfCorrection() 410 0.0001436 0.0001436 0.0001436 0.02% StateData::checkPoint() 12 0.0001389 0.0001389 0.0001389 0.02% MLMG::actualBottomSolve() 82 0.0001044 0.0001044 0.0001044 0.01% Castro::advance() 10 0.0001041 0.0001041 0.0001041 0.01% FabArrayBase::FB::FB() 56 9.424e-05 9.424e-05 9.424e-05 0.01% Castro::construct_new_source() 50 9.06e-05 9.06e-05 9.06e-05 0.01% MLMG::mgVcycle_down::0 82 8.912e-05 8.912e-05 8.912e-05 0.01% MLMG::mgVcycle_down::1 82 8.21e-05 8.21e-05 8.21e-05 0.01% Castro::initialize_advance() 10 8.204e-05 8.204e-05 8.204e-05 0.01% MLMG::mgVcycle_down::2 82 7.899e-05 7.899e-05 7.899e-05 0.01% MLMG::mgVcycle_down::4 82 7.648e-05 7.648e-05 7.648e-05 0.01% MLMG::mgVcycle_down::3 82 7.567e-05 7.567e-05 7.567e-05 0.01% AmrLevel::checkPoint() 3 7.505e-05 7.505e-05 7.505e-05 0.01% Castro::clean_state() 62 7.171e-05 7.171e-05 7.171e-05 0.01% MLMG::solve() 11 6.487e-05 6.487e-05 6.487e-05 0.01% MLMG::mgVcycle_up::4 82 6.212e-05 6.212e-05 6.212e-05 0.01% Castro::initialize_do_advance() 10 5.829e-05 5.829e-05 5.829e-05 0.01% Castro::finalize_advance() 10 5.789e-05 5.789e-05 5.789e-05 0.01% MLMG::mgVcycle_up::0 82 5.354e-05 5.354e-05 5.354e-05 0.01% MLCellLinOp::solutionResidual() 93 5.086e-05 5.086e-05 5.086e-05 0.01% Castro::post_timestep() 10 4.848e-05 4.848e-05 4.848e-05 0.01% MLMG::mgVcycle_up::3 82 4.832e-05 4.832e-05 4.832e-05 0.01% MLMG::mgVcycle_up::1 82 4.765e-05 4.765e-05 4.765e-05 0.01% MLMG::mgVcycle_up::2 82 4.64e-05 4.64e-05 4.64e-05 0.01% StateData::define() 4 4.075e-05 4.075e-05 4.075e-05 0.00% Castro::swap_state_time_levels() 10 3.937e-05 3.937e-05 3.937e-05 0.00% Castro::finalize_do_advance() 10 3.674e-05 3.674e-05 3.674e-05 0.00% Castro::enforce_consistent_e() 1 3.404e-05 3.404e-05 3.404e-05 0.00% MLMG::mgVcycle_bottom 82 3.397e-05 3.397e-05 3.397e-05 0.00% MLMG::computeResidual() 82 3.255e-05 3.255e-05 3.255e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.924e-05 2.924e-05 2.924e-05 0.00% FillPatchSingleLevel 41 2.873e-05 2.873e-05 2.873e-05 0.00% MLLinOp::define() 11 2.619e-05 2.619e-05 2.619e-05 0.00% makeSFC 55 2.588e-05 2.588e-05 2.588e-05 0.00% MLPoisson::define() 11 2.501e-05 2.501e-05 2.501e-05 0.00% Amr::writeSmallPlotFile() 1 2.276e-05 2.276e-05 2.276e-05 0.00% Amr::FinalizeInit() 1 2.165e-05 2.165e-05 2.165e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.726e-05 1.726e-05 1.726e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.688e-05 1.688e-05 1.688e-05 0.00% Castro::construct_old_source() 50 1.683e-05 1.683e-05 1.683e-05 0.00% Castro::do_new_sources() 10 1.682e-05 1.682e-05 1.682e-05 0.00% Castro::do_old_sources() 10 1.61e-05 1.61e-05 1.61e-05 0.00% DistributionMapping::Distribute() 56 1.397e-05 1.397e-05 1.397e-05 0.00% Castro::apply_source_to_state() 20 1.24e-05 1.24e-05 1.24e-05 0.00% Castro::construct_old_gravity() 10 1.127e-05 1.127e-05 1.127e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.078e-05 1.078e-05 1.078e-05 0.00% Gravity::swapTimeLevels() 10 1.033e-05 1.033e-05 1.033e-05 0.00% MLMG::MLResNormInf() 11 9.795e-06 9.795e-06 9.795e-06 0.00% Amr::initSubcycle() 1 9.707e-06 9.707e-06 9.707e-06 0.00% MLPoisson::prepareForSolve() 11 7.701e-06 7.701e-06 7.701e-06 0.00% MLMG::computeMLResidual() 11 6.823e-06 6.823e-06 6.823e-06 0.00% MLMG::getGradSolution() 11 6.419e-06 6.419e-06 6.419e-06 0.00% Amr::InitializeInit() 1 6.257e-06 6.257e-06 6.257e-06 0.00% MLMG::buildFineMask() 11 6.19e-06 6.19e-06 6.19e-06 0.00% AmrLevel::checkPointPost() 3 5.984e-06 5.984e-06 5.984e-06 0.00% Castro::computeNewDt() 9 5.715e-06 5.715e-06 5.715e-06 0.00% Castro::create_source_corrector() 10 5.281e-06 5.281e-06 5.281e-06 0.00% Gravity::set_mass_offset() 11 4.288e-06 4.288e-06 4.288e-06 0.00% Castro::retry_advance_ctu() 10 3.938e-06 3.938e-06 3.938e-06 0.00% Castro::post_init() 1 3.806e-06 3.806e-06 3.806e-06 0.00% Castro::FluxRegFineAdd() 10 3.31e-06 3.31e-06 3.31e-06 0.00% Amr::init() 1 2.858e-06 2.858e-06 2.858e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.753e-06 2.753e-06 2.753e-06 0.00% Castro::computeInitialDt() 2 2.279e-06 2.279e-06 2.279e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.253e-06 2.253e-06 2.253e-06 0.00% AmrLevel::checkPointPre() 3 2.082e-06 2.082e-06 2.082e-06 0.00% Amr::initialInit() 1 1.38e-06 1.38e-06 1.38e-06 0.00% Castro::post_regrid() 1 1.363e-06 1.363e-06 1.363e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8302 0.8302 0.8302 100.00% Amr::coarseTimeStep() 10 0.686 0.686 0.686 82.63% Amr::timeStep() 10 0.6125 0.6125 0.6125 73.77% Castro::advance() 10 0.6069 0.6069 0.6069 73.09% Castro::subcycle_advance_ctu() 10 0.593 0.593 0.593 71.42% Castro::do_advance_ctu() 10 0.5928 0.5928 0.5928 71.40% Gravity::solve_phi_with_mlmg() 11 0.3262 0.3262 0.3262 39.29% Gravity::actual_solve_with_mlmg() 11 0.3148 0.3148 0.3148 37.91% Castro::construct_new_gravity() 10 0.2966 0.2966 0.2966 35.73% MLMG::solve() 11 0.2922 0.2922 0.2922 35.20% Gravity::solve_for_phi() 10 0.279 0.279 0.279 33.60% MLMG::oneIter() 82 0.2763 0.2763 0.2763 33.28% MLMG::mgVcycle() 82 0.2746 0.2746 0.2746 33.08% Castro::construct_ctu_hydro_source() 10 0.2219 0.2219 0.2219 26.72% VisMF::Write(FabArray) 11 0.1462 0.1462 0.1462 17.61% MLCellLinOp::smooth() 1640 0.1458 0.1458 0.1458 17.57% Amr::init() 1 0.1214 0.1214 0.1214 14.62% MLCellLinOp::applyBC() 4433 0.1143 0.1143 0.1143 13.77% Amr::checkPoint() 3 0.1084 0.1084 0.1084 13.06% AmrLevel::checkPoint() 3 0.1046 0.1046 0.1046 12.60% StateData::checkPoint() 12 0.1045 0.1045 0.1045 12.59% MLMG::mgVcycle_bottom 82 0.07792 0.07792 0.07792 9.38% MLMG::actualBottomSolve() 82 0.07788 0.07788 0.07788 9.38% MLCGSolver::bicgstab 82 0.07716 0.07716 0.07716 9.29% Amr::initialInit() 1 0.06092 0.06092 0.06092 7.34% MLPoisson::Fsmooth() 3280 0.05988 0.05988 0.05988 7.21% Amr::FinalizeInit() 1 0.05176 0.05176 0.05176 6.23% Castro::post_init() 1 0.05106 0.05106 0.05106 6.15% FillPatchIterator::Initialize 41 0.04901 0.04901 0.04901 5.90% Gravity::multilevel_solve_for_new_phi() 1 0.04796 0.04796 0.04796 5.78% Gravity::actual_multilevel_solve() 1 0.04794 0.04794 0.04794 5.77% Amr::writePlotFile() 2 0.04472 0.04472 0.04472 5.39% FillPatchSingleLevel 41 0.04366 0.04366 0.04366 5.26% MLCellLinOp::apply() 1142 0.03995 0.03995 0.03995 4.81% StateDataPhysBCFunct::() 41 0.03946 0.03946 0.03946 4.75% MLMG::mgVcycle_down::0 82 0.03884 0.03884 0.03884 4.68% MLMG::mgVcycle_up::0 82 0.03342 0.03342 0.03342 4.03% Castro::clean_state() 62 0.03255 0.03255 0.03255 3.92% FabArray::setVal() 1144 0.02569 0.02569 0.02569 3.09% StateData::FillBoundary(geom) 328 0.02341 0.02341 0.02341 2.82% Castro::initialize_do_advance() 10 0.02237 0.02237 0.02237 2.69% MLCellLinOp::correctionResidual() 492 0.02236 0.02236 0.02236 2.69% MultiFab::Dot() 1114 0.02177 0.02177 0.02177 2.62% Gravity::get_new_grav_vector() 11 0.02032 0.02032 0.02032 2.45% FabArray::FillBoundary() 4023 0.01993 0.01993 0.01993 2.40% MLMG:computeResOfCorrection() 410 0.01936 0.01936 0.01936 2.33% FillBoundary_nowait() 4023 0.01911 0.01911 0.01911 2.30% MLMG::mgVcycle_down::1 82 0.01852 0.01852 0.01852 2.23% Castro::computeTemp() 63 0.01817 0.01817 0.01817 2.19% MLMG::mgVcycle_down::2 82 0.01792 0.01792 0.01792 2.16% FabArray::ParallelCopy() 861 0.01734 0.01734 0.01734 2.09% Castro::expand_state() 10 0.01719 0.01719 0.01719 2.07% MLPoisson::define() 11 0.01712 0.01712 0.01712 2.06% MLMG::mgVcycle_down::3 82 0.01712 0.01712 0.01712 2.06% FabArray::ParallelCopy_nowait() 861 0.01704 0.01704 0.01704 2.05% MLMG::mgVcycle_down::4 82 0.01633 0.01633 0.01633 1.97% Castro::construct_old_gravity() 10 0.01612 0.01612 0.01612 1.94% Gravity::get_old_grav_vector() 10 0.01611 0.01611 0.01611 1.94% MLMG::mgVcycle_up::2 82 0.01401 0.01401 0.01401 1.69% MLMG::mgVcycle_up::1 82 0.01374 0.01374 0.01374 1.65% Castro::initialize_advance() 10 0.01372 0.01372 0.01372 1.65% MLMG::addInterpCorrection() 410 0.0137 0.0137 0.0137 1.65% MLMG::mgVcycle_up::3 82 0.0133 0.0133 0.0133 1.60% MLMG::mgVcycle_up::4 82 0.01312 0.01312 0.01312 1.58% amrex::average_down 410 0.01311 0.01311 0.01311 1.58% MLCGSolver::ParallelAllReduce 1514 0.01301 0.01301 0.01301 1.57% CGSolver::sxay() 1586 0.01274 0.01274 0.01274 1.53% MultiFab::LinComb() 1586 0.01226 0.01226 0.01226 1.48% Castro::do_new_sources() 10 0.01202 0.01202 0.01202 1.45% MLCellLinOp::defineAuxData() 11 0.01178 0.01178 0.01178 1.42% Gravity::fill_multipole_BCs() 11 0.0112 0.0112 0.0112 1.35% MLPoisson::Fapply() 1142 0.01052 0.01052 0.01052 1.27% Castro::enforce_min_density() 62 0.01026 0.01026 0.01026 1.24% Amr::InitializeInit() 1 0.009161 0.009161 0.009161 1.10% Amr::defBaseLevel() 1 0.009154 0.009154 0.009154 1.10% MLCellLinOp::solutionResidual() 93 0.007936 0.007936 0.007936 0.96% Castro::do_old_sources() 10 0.006888 0.006888 0.006888 0.83% MLMG::computeResidual() 82 0.006858 0.006858 0.006858 0.83% MultiFab::Xpay() 585 0.006016 0.006016 0.006016 0.72% MLMG::prepareForSolve() 11 0.005807 0.005807 0.005807 0.70% Castro::post_timestep() 10 0.005441 0.005441 0.005441 0.66% Castro::estTimeStep() 21 0.005232 0.005232 0.005232 0.63% Castro::reset_internal_energy() 63 0.00514 0.00514 0.00514 0.62% FabArray::setDomainBndry() 41 0.005105 0.005105 0.005105 0.61% MLCellLinOp::defineBC() 11 0.005043 0.005043 0.005043 0.61% BndryData::define() 11 0.004849 0.004849 0.004849 0.58% Castro::initData() 1 0.003332 0.003332 0.003332 0.40% Castro::construct_new_source() 50 0.002652 0.002652 0.002652 0.32% Castro::normalize_species() 62 0.002609 0.002609 0.002609 0.31% Castro::construct_new_gravity_source() 10 0.002562 0.002562 0.002562 0.31% MLMG::getGradSolution() 11 0.002406 0.002406 0.002406 0.29% MLCellLinOp::compGrad() 11 0.0024 0.0024 0.0024 0.29% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002267 0.002267 0.002267 0.27% Castro::computeNewDt() 9 0.002264 0.002264 0.002264 0.27% Castro::enforce_speed_limit() 62 0.00217 0.00217 0.00217 0.26% Castro::construct_old_source() 50 0.002022 0.002022 0.002022 0.24% Castro::construct_old_gravity_source() 10 0.002006 0.002006 0.002006 0.24% MLMG::ResNormInf() 93 0.001906 0.001906 0.001906 0.23% Castro::apply_source_to_state() 20 0.001826 0.001826 0.001826 0.22% MultiFab::Saxpy() 20 0.001814 0.001814 0.001814 0.22% FabArrayBase::getCPC() 1323 0.001693 0.001693 0.001693 0.20% MLCellLinOp::setLevelBC() 11 0.001538 0.001538 0.001538 0.19% MLPoisson::prepareForSolve() 11 0.00135 0.00135 0.00135 0.16% MLCellLinOp::prepareForSolve() 11 0.001342 0.001342 0.001342 0.16% FabArray::mult() 43 0.001324 0.001324 0.001324 0.16% MLMG::computeMLResidual() 11 0.001118 0.001118 0.001118 0.13% Gravity::swapTimeLevels() 10 0.001047 0.001047 0.001047 0.13% Gravity::update_max_rhs() 11 0.0009495 0.0009495 0.0009495 0.11% FabArrayBase::getFB() 4023 0.0008375 0.0008375 0.0008375 0.10% FabArrayBase::CPC::define() 454 0.0007287 0.0007287 0.0007287 0.09% Castro::computeInitialDt() 2 0.0006316 0.0006316 0.0006316 0.08% MultiFab::Copy() 11 0.0005154 0.0005154 0.0005154 0.06% Amr::InitAmr() 1 0.0004684 0.0004684 0.0004684 0.06% Castro::post_regrid() 1 0.0004617 0.0004617 0.0004617 0.06% MultiFab::min() 10 0.000323 0.000323 0.000323 0.04% MLLinOp::define() 11 0.0002711 0.0002711 0.0002711 0.03% MLMG::MLResNormInf() 11 0.0002616 0.0002616 0.0002616 0.03% MultiFab::max() 11 0.0002558 0.0002558 0.0002558 0.03% MLLinOp::defineGrids() 11 0.000245 0.000245 0.000245 0.03% MLMG::MLRhsNormInf() 11 0.0002012 0.0002012 0.0002012 0.02% FabArrayBase::FB::FB() 56 9.424e-05 9.424e-05 9.424e-05 0.01% Castro::finalize_advance() 10 6.12e-05 6.12e-05 6.12e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.582e-05 5.582e-05 5.582e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.153e-05 5.153e-05 5.153e-05 0.01% StateData::define() 4 4.075e-05 4.075e-05 4.075e-05 0.00% Castro::swap_state_time_levels() 10 3.937e-05 3.937e-05 3.937e-05 0.00% makeSFC 55 3.894e-05 3.894e-05 3.894e-05 0.00% Castro::finalize_do_advance() 10 3.674e-05 3.674e-05 3.674e-05 0.00% Castro::enforce_consistent_e() 1 3.404e-05 3.404e-05 3.404e-05 0.00% Amr::writeSmallPlotFile() 1 2.276e-05 2.276e-05 2.276e-05 0.00% DistributionMapping::Distribute() 56 1.397e-05 1.397e-05 1.397e-05 0.00% Amr::initSubcycle() 1 9.707e-06 9.707e-06 9.707e-06 0.00% MLMG::buildFineMask() 11 6.19e-06 6.19e-06 6.19e-06 0.00% AmrLevel::checkPointPost() 3 5.984e-06 5.984e-06 5.984e-06 0.00% Castro::create_source_corrector() 10 5.281e-06 5.281e-06 5.281e-06 0.00% Gravity::set_mass_offset() 11 4.288e-06 4.288e-06 4.288e-06 0.00% Castro::retry_advance_ctu() 10 3.938e-06 3.938e-06 3.938e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.67e-06 3.67e-06 3.67e-06 0.00% Castro::FluxRegFineAdd() 10 3.31e-06 3.31e-06 3.31e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.253e-06 2.253e-06 2.253e-06 0.00% AmrLevel::checkPointPre() 3 2.082e-06 2.082e-06 2.082e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.03-8-g6bbcbbb78299) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.03-8-g6bbcbbb78299) initialized Starting run at 08:53:57 UTC on 2022-03-08. Successfully read inputs file ... Castro git describe: 22.02-4-g0d942c92d AMReX git describe: 22.03-8-g6bbcbbb78 Microphysics git describe: 22.03-2-ge2419c42 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.483389146 Restart time = 0.053458801 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.112932632 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.04945792 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.053577934 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.058532543 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.061484612 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.023152312 seconds Ending run at 08:53:57 UTC on 2022-03-08. Run time = 0.413541009 Run time without initialization = 0.359479161 Average number of zones advanced per microsecond: 3.646 Average number of zones advanced per microsecond per rank: 3.646 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4136 ... 0.4136 ... 0.4136 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1311 0.1311 0.1311 31.71% VisMF::Read() 3 0.0417 0.0417 0.0417 10.08% MLCellLinOp::applyBC() 1946 0.03741 0.03741 0.03741 9.05% MLPoisson::Fsmooth() 1440 0.02556 0.02556 0.02556 6.18% VisMF::Write(FabArray) 1 0.02184 0.02184 0.02184 5.28% FabArray::setVal() 537 0.01535 0.01535 0.01535 3.71% StateData::FillBoundary(geom) 160 0.01103 0.01103 0.01103 2.67% MLCGSolver::bicgstab 36 0.009221 0.009221 0.009221 2.23% MultiFab::Dot() 484 0.00914 0.00914 0.00914 2.21% Castro::computeTemp() 30 0.007425 0.007425 0.007425 1.80% FillBoundary_nowait() 1766 0.007125 0.007125 0.007125 1.72% FabArray::ParallelCopy_nowait() 380 0.00705 0.00705 0.00705 1.70% StateDataPhysBCFunct::() 20 0.007043 0.007043 0.007043 1.70% Gravity::fill_multipole_BCs() 6 0.006555 0.006555 0.006555 1.58% Castro::enforce_min_density() 30 0.005979 0.005979 0.005979 1.45% MLCellLinOp::defineAuxData() 6 0.005488 0.005488 0.005488 1.33% MultiFab::LinComb() 690 0.005267 0.005267 0.005267 1.27% MLPoisson::Fapply() 500 0.004543 0.004543 0.004543 1.10% Castro::expand_state() 5 0.004116 0.004116 0.004116 1.00% FabArray::setDomainBndry() 20 0.003844 0.003844 0.003844 0.93% Amr::restart() 1 0.003107 0.003107 0.003107 0.75% MLMG::addInterpCorrection() 180 0.003 0.003 0.003 0.73% amrex::average_down 180 0.002805 0.002805 0.002805 0.68% MultiFab::Xpay() 258 0.002624 0.002624 0.002624 0.63% Castro::do_advance_ctu() 5 0.002536 0.002536 0.002536 0.61% Castro::reset_internal_energy() 30 0.002384 0.002384 0.002384 0.58% Castro::estTimeStep() 10 0.002294 0.002294 0.002294 0.55% BndryData::define() 6 0.001931 0.001931 0.001931 0.47% MLCellLinOp::compGrad() 6 0.001463 0.001463 0.001463 0.35% Amr::writePlotFile() 1 0.001402 0.001402 0.001402 0.34% Castro::construct_new_gravity_source() 5 0.001363 0.001363 0.001363 0.33% Castro::normalize_species() 30 0.001302 0.001302 0.001302 0.31% Gravity::get_new_grav_vector() 5 0.001229 0.001229 0.001229 0.30% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001209 0.001209 0.001209 0.29% Castro::construct_old_gravity_source() 5 0.001028 0.001028 0.001028 0.25% MultiFab::Saxpy() 10 0.0009235 0.0009235 0.0009235 0.22% Gravity::get_old_grav_vector() 5 0.000905 0.000905 0.000905 0.22% MLCellLinOp::setLevelBC() 6 0.0008376 0.0008376 0.0008376 0.20% MLMG::ResNormInf() 42 0.0008325 0.0008325 0.0008325 0.20% Castro::subcycle_advance_ctu() 5 0.0008132 0.0008132 0.0008132 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007455 0.0007455 0.0007455 0.18% MLMG::oneIter() 36 0.0007214 0.0007214 0.0007214 0.17% MLCellLinOp::prepareForSolve() 6 0.0006841 0.0006841 0.0006841 0.17% FabArray::mult() 22 0.0006536 0.0006536 0.0006536 0.16% Castro::construct_new_source() 25 0.000647 0.000647 0.000647 0.16% Castro::post_timestep() 5 0.000631 0.000631 0.000631 0.15% Castro::enforce_speed_limit() 30 0.0006204 0.0006204 0.0006204 0.15% Gravity::update_max_rhs() 6 0.0005266 0.0005266 0.0005266 0.13% MLCellLinOp::smooth() 720 0.0005018 0.0005018 0.0005018 0.12% FabArrayBase::getCPC() 632 0.0004726 0.0004726 0.0004726 0.11% FabArrayBase::CPC::define() 244 0.0004703 0.0004703 0.0004703 0.11% MLMG::prepareForSolve() 6 0.0004368 0.0004368 0.0004368 0.11% Amr::InitAmr() 1 0.0004231 0.0004231 0.0004231 0.10% FabArray::FillBoundary() 1766 0.0003902 0.0003902 0.0003902 0.09% FabArrayBase::getFB() 1766 0.0003205 0.0003205 0.0003205 0.08% MultiFab::Copy() 6 0.0003122 0.0003122 0.0003122 0.08% main() 1 0.0002816 0.0002816 0.0002816 0.07% MLCellLinOp::apply() 500 0.0002335 0.0002335 0.0002335 0.06% Gravity::actual_multilevel_solve() 1 0.0002266 0.0002266 0.0002266 0.05% CGSolver::sxay() 690 0.0002241 0.0002241 0.0002241 0.05% Gravity::solve_for_phi() 5 0.000215 0.000215 0.000215 0.05% MultiFab::min() 5 0.0001823 0.0001823 0.0001823 0.04% Castro::construct_new_gravity() 5 0.0001818 0.0001818 0.0001818 0.04% MLMG::mgVcycle() 36 0.0001625 0.0001625 0.0001625 0.04% FabArray::ParallelCopy() 380 0.000146 0.000146 0.000146 0.04% MLCGSolver::ParallelAllReduce 659 0.0001434 0.0001434 0.0001434 0.03% MultiFab::max() 6 0.0001325 0.0001325 0.0001325 0.03% FillPatchIterator::Initialize 20 0.000132 0.000132 0.000132 0.03% MLCellLinOp::defineBC() 6 0.0001088 0.0001088 0.0001088 0.03% Amr::coarseTimeStep() 5 0.0001084 0.0001084 0.0001084 0.03% MLMG::MLRhsNormInf() 6 0.0001051 0.0001051 0.0001051 0.03% Amr::timeStep() 5 0.0001038 0.0001038 0.0001038 0.03% MLLinOp::defineGrids() 6 9.779e-05 9.779e-05 9.779e-05 0.02% MLCellLinOp::correctionResidual() 216 9.479e-05 9.479e-05 9.479e-05 0.02% StateData::restartDoit() 4 8.208e-05 8.208e-05 8.208e-05 0.02% AmrLevel::restart() 1 8.011e-05 8.011e-05 8.011e-05 0.02% FabArrayBase::FB::FB() 26 6.596e-05 6.596e-05 6.596e-05 0.02% MLMG:computeResOfCorrection() 180 6.428e-05 6.428e-05 6.428e-05 0.02% Castro::construct_old_source() 25 6.129e-05 6.129e-05 6.129e-05 0.01% Castro::initialize_advance() 5 5.629e-05 5.629e-05 5.629e-05 0.01% Castro::advance() 5 4.748e-05 4.748e-05 4.748e-05 0.01% MLMG::actualBottomSolve() 36 4.722e-05 4.722e-05 4.722e-05 0.01% MLMG::mgVcycle_down::0 36 3.771e-05 3.771e-05 3.771e-05 0.01% MLMG::mgVcycle_down::1 36 3.508e-05 3.508e-05 3.508e-05 0.01% Castro::clean_state() 30 3.392e-05 3.392e-05 3.392e-05 0.01% Castro::do_new_sources() 5 3.367e-05 3.367e-05 3.367e-05 0.01% MLMG::mgVcycle_down::2 36 3.319e-05 3.319e-05 3.319e-05 0.01% MLMG::solve() 6 3.225e-05 3.225e-05 3.225e-05 0.01% MLMG::mgVcycle_down::4 36 3.223e-05 3.223e-05 3.223e-05 0.01% Castro::post_restart() 1 3.116e-05 3.116e-05 3.116e-05 0.01% MLMG::mgVcycle_down::3 36 3.075e-05 3.075e-05 3.075e-05 0.01% Castro::initialize_do_advance() 5 2.895e-05 2.895e-05 2.895e-05 0.01% MLLinOp::define() 6 2.798e-05 2.798e-05 2.798e-05 0.01% Castro::swap_state_time_levels() 5 2.718e-05 2.718e-05 2.718e-05 0.01% Castro::finalize_advance() 5 2.706e-05 2.706e-05 2.706e-05 0.01% MLMG::mgVcycle_up::4 36 2.562e-05 2.562e-05 2.562e-05 0.01% Amr::writeSmallPlotFile() 1 2.331e-05 2.331e-05 2.331e-05 0.01% MLCellLinOp::solutionResidual() 42 2.3e-05 2.3e-05 2.3e-05 0.01% MLMG::mgVcycle_up::0 36 2.226e-05 2.226e-05 2.226e-05 0.01% MLMG::mgVcycle_up::3 36 2.177e-05 2.177e-05 2.177e-05 0.01% MLMG::mgVcycle_up::2 36 2.12e-05 2.12e-05 2.12e-05 0.01% Castro::finalize_do_advance() 5 2.108e-05 2.108e-05 2.108e-05 0.01% MLMG::mgVcycle_up::1 36 2.032e-05 2.032e-05 2.032e-05 0.00% Castro::create_source_corrector() 5 1.94e-05 1.94e-05 1.94e-05 0.00% MLPoisson::define() 6 1.733e-05 1.733e-05 1.733e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.684e-05 1.684e-05 1.684e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.623e-05 1.623e-05 1.623e-05 0.00% MLMG::computeResidual() 36 1.465e-05 1.465e-05 1.465e-05 0.00% MLMG::mgVcycle_bottom 36 1.419e-05 1.419e-05 1.419e-05 0.00% FillPatchSingleLevel 20 1.401e-05 1.401e-05 1.401e-05 0.00% makeSFC 30 1.392e-05 1.392e-05 1.392e-05 0.00% Amr::initSubcycle() 1 9.54e-06 9.54e-06 9.54e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.809e-06 8.809e-06 8.809e-06 0.00% DistributionMapping::Distribute() 31 8.699e-06 8.699e-06 8.699e-06 0.00% Castro::do_old_sources() 5 7.975e-06 7.975e-06 7.975e-06 0.00% Castro::apply_source_to_state() 10 6.256e-06 6.256e-06 6.256e-06 0.00% Castro::construct_old_gravity() 5 5.647e-06 5.647e-06 5.647e-06 0.00% Gravity::swapTimeLevels() 5 4.97e-06 4.97e-06 4.97e-06 0.00% MLPoisson::prepareForSolve() 6 4.775e-06 4.775e-06 4.775e-06 0.00% MLMG::buildFineMask() 6 3.745e-06 3.745e-06 3.745e-06 0.00% Castro::computeNewDt() 5 3.646e-06 3.646e-06 3.646e-06 0.00% MLMG::getGradSolution() 6 3.242e-06 3.242e-06 3.242e-06 0.00% MLMG::computeMLResidual() 6 3.223e-06 3.223e-06 3.223e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.017e-06 3.017e-06 3.017e-06 0.00% Gravity::set_mass_offset() 6 2.749e-06 2.749e-06 2.749e-06 0.00% MLMG::MLResNormInf() 6 2.654e-06 2.654e-06 2.654e-06 0.00% Castro::retry_advance_ctu() 5 1.686e-06 1.686e-06 1.686e-06 0.00% Castro::FluxRegFineAdd() 5 1.368e-06 1.368e-06 1.368e-06 0.00% AmrLevel::AmrLevel() 1 1.123e-06 1.123e-06 1.123e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.069e-06 1.069e-06 1.069e-06 0.00% Amr::init() 1 1.058e-06 1.058e-06 1.058e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4136 0.4136 0.4136 100.00% Amr::coarseTimeStep() 5 0.3361 0.3361 0.3361 81.26% Amr::timeStep() 5 0.3347 0.3347 0.3347 80.92% Castro::advance() 5 0.3314 0.3314 0.3314 80.13% Castro::subcycle_advance_ctu() 5 0.3205 0.3205 0.3205 77.49% Castro::do_advance_ctu() 5 0.3197 0.3197 0.3197 77.29% Castro::construct_new_gravity() 5 0.1449 0.1449 0.1449 35.05% Gravity::solve_phi_with_mlmg() 6 0.1415 0.1415 0.1415 34.21% Gravity::solve_for_phi() 5 0.1358 0.1358 0.1358 32.83% Gravity::actual_solve_with_mlmg() 6 0.1348 0.1348 0.1348 32.60% Castro::construct_ctu_hydro_source() 5 0.1311 0.1311 0.1311 31.71% MLMG::solve() 6 0.1221 0.1221 0.1221 29.52% MLMG::oneIter() 36 0.1145 0.1145 0.1145 27.68% MLMG::mgVcycle() 36 0.1137 0.1137 0.1137 27.50% MLCellLinOp::smooth() 720 0.05951 0.05951 0.05951 14.39% Amr::init() 1 0.05349 0.05349 0.05349 12.93% Amr::restart() 1 0.05349 0.05349 0.05349 12.93% MLCellLinOp::applyBC() 1946 0.04531 0.04531 0.04531 10.96% AmrLevel::restart() 1 0.04259 0.04259 0.04259 10.30% StateData::restartDoit() 4 0.0425 0.0425 0.0425 10.28% VisMF::Read() 3 0.0417 0.0417 0.0417 10.08% MLMG::mgVcycle_bottom 36 0.03289 0.03289 0.03289 7.95% MLMG::actualBottomSolve() 36 0.03287 0.03287 0.03287 7.95% MLCGSolver::bicgstab 36 0.03256 0.03256 0.03256 7.87% MLPoisson::Fsmooth() 1440 0.02556 0.02556 0.02556 6.18% FillPatchIterator::Initialize 20 0.02416 0.02416 0.02416 5.84% Amr::writePlotFile() 1 0.02324 0.02324 0.02324 5.62% VisMF::Write(FabArray) 1 0.02184 0.02184 0.02184 5.28% FillPatchSingleLevel 20 0.02018 0.02018 0.02018 4.88% StateDataPhysBCFunct::() 20 0.01807 0.01807 0.01807 4.37% Castro::clean_state() 30 0.01774 0.01774 0.01774 4.29% MLCellLinOp::apply() 500 0.01635 0.01635 0.01635 3.95% MLMG::mgVcycle_down::0 36 0.01585 0.01585 0.01585 3.83% FabArray::setVal() 537 0.01535 0.01535 0.01535 3.71% Castro::initialize_do_advance() 5 0.01485 0.01485 0.01485 3.59% MLMG::mgVcycle_up::0 36 0.01349 0.01349 0.01349 3.26% Castro::expand_state() 5 0.01164 0.01164 0.01164 2.81% StateData::FillBoundary(geom) 160 0.01103 0.01103 0.01103 2.67% Castro::initialize_advance() 5 0.01086 0.01086 0.01086 2.63% Castro::computeTemp() 30 0.009809 0.009809 0.009809 2.37% MLPoisson::define() 6 0.0094 0.0094 0.0094 2.27% MLCellLinOp::correctionResidual() 216 0.009209 0.009209 0.009209 2.23% MultiFab::Dot() 484 0.00914 0.00914 0.00914 2.21% Gravity::get_new_grav_vector() 5 0.008995 0.008995 0.008995 2.17% MLMG:computeResOfCorrection() 180 0.007973 0.007973 0.007973 1.93% FabArray::FillBoundary() 1766 0.007901 0.007901 0.007901 1.91% MLMG::mgVcycle_down::1 36 0.0077 0.0077 0.0077 1.86% FabArray::ParallelCopy() 380 0.007651 0.007651 0.007651 1.85% Castro::construct_old_gravity() 5 0.007603 0.007603 0.007603 1.84% Gravity::get_old_grav_vector() 5 0.007597 0.007597 0.007597 1.84% FillBoundary_nowait() 1766 0.007511 0.007511 0.007511 1.82% FabArray::ParallelCopy_nowait() 380 0.007505 0.007505 0.007505 1.81% MLMG::mgVcycle_down::2 36 0.007409 0.007409 0.007409 1.79% MLMG::mgVcycle_down::3 36 0.007088 0.007088 0.007088 1.71% MLMG::mgVcycle_down::4 36 0.006774 0.006774 0.006774 1.64% Castro::post_restart() 1 0.006578 0.006578 0.006578 1.59% Gravity::fill_multipole_BCs() 6 0.006555 0.006555 0.006555 1.58% MLCellLinOp::defineAuxData() 6 0.006479 0.006479 0.006479 1.57% Castro::do_new_sources() 5 0.006448 0.006448 0.006448 1.56% Gravity::multilevel_solve_for_new_phi() 1 0.006183 0.006183 0.006183 1.49% Gravity::actual_multilevel_solve() 1 0.006166 0.006166 0.006166 1.49% Castro::enforce_min_density() 30 0.005979 0.005979 0.005979 1.45% MLMG::addInterpCorrection() 180 0.005769 0.005769 0.005769 1.39% MLMG::mgVcycle_up::2 36 0.005759 0.005759 0.005759 1.39% MLMG::mgVcycle_up::1 36 0.005708 0.005708 0.005708 1.38% amrex::average_down 180 0.005591 0.005591 0.005591 1.35% MLCGSolver::ParallelAllReduce 659 0.005529 0.005529 0.005529 1.34% CGSolver::sxay() 690 0.005491 0.005491 0.005491 1.33% MLMG::mgVcycle_up::3 36 0.005489 0.005489 0.005489 1.33% MLMG::mgVcycle_up::4 36 0.005426 0.005426 0.005426 1.31% MultiFab::LinComb() 690 0.005267 0.005267 0.005267 1.27% Castro::do_old_sources() 5 0.004723 0.004723 0.004723 1.14% MLPoisson::Fapply() 500 0.004543 0.004543 0.004543 1.10% FabArray::setDomainBndry() 20 0.003844 0.003844 0.003844 0.93% MLCellLinOp::solutionResidual() 42 0.003361 0.003361 0.003361 0.81% MLMG::prepareForSolve() 6 0.00325 0.00325 0.00325 0.79% Castro::post_timestep() 5 0.003144 0.003144 0.003144 0.76% MLMG::computeResidual() 36 0.00278 0.00278 0.00278 0.67% MLCellLinOp::defineBC() 6 0.002747 0.002747 0.002747 0.66% BndryData::define() 6 0.002639 0.002639 0.002639 0.64% MultiFab::Xpay() 258 0.002624 0.002624 0.002624 0.63% Castro::reset_internal_energy() 30 0.002384 0.002384 0.002384 0.58% Castro::estTimeStep() 10 0.002294 0.002294 0.002294 0.55% Castro::construct_new_source() 25 0.00201 0.00201 0.00201 0.49% MLMG::getGradSolution() 6 0.001755 0.001755 0.001755 0.42% MLCellLinOp::compGrad() 6 0.001752 0.001752 0.001752 0.42% Castro::construct_new_gravity_source() 5 0.001363 0.001363 0.001363 0.33% Castro::computeNewDt() 5 0.001318 0.001318 0.001318 0.32% Castro::normalize_species() 30 0.001302 0.001302 0.001302 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001209 0.001209 0.001209 0.29% Castro::construct_old_source() 25 0.00109 0.00109 0.00109 0.26% Castro::construct_old_gravity_source() 5 0.001028 0.001028 0.001028 0.25% FabArrayBase::getCPC() 632 0.0009429 0.0009429 0.0009429 0.23% Castro::apply_source_to_state() 10 0.0009298 0.0009298 0.0009298 0.22% MultiFab::Saxpy() 10 0.0009235 0.0009235 0.0009235 0.22% Gravity::swapTimeLevels() 5 0.0008651 0.0008651 0.0008651 0.21% MLCellLinOp::setLevelBC() 6 0.0008376 0.0008376 0.0008376 0.20% MLMG::ResNormInf() 42 0.0008325 0.0008325 0.0008325 0.20% Gravity::update_max_rhs() 6 0.000735 0.000735 0.000735 0.18% MLPoisson::prepareForSolve() 6 0.0006889 0.0006889 0.0006889 0.17% MLCellLinOp::prepareForSolve() 6 0.0006841 0.0006841 0.0006841 0.17% FabArray::mult() 22 0.0006536 0.0006536 0.0006536 0.16% Castro::enforce_speed_limit() 30 0.0006204 0.0006204 0.0006204 0.15% MLMG::computeMLResidual() 6 0.0005992 0.0005992 0.0005992 0.14% FabArrayBase::CPC::define() 244 0.0004703 0.0004703 0.0004703 0.11% Amr::InitAmr() 1 0.0004327 0.0004327 0.0004327 0.10% FabArrayBase::getFB() 1766 0.0003865 0.0003865 0.0003865 0.09% MultiFab::Copy() 6 0.0003122 0.0003122 0.0003122 0.08% MultiFab::min() 5 0.0001823 0.0001823 0.0001823 0.04% MLLinOp::define() 6 0.000157 0.000157 0.000157 0.04% MLMG::MLResNormInf() 6 0.0001353 0.0001353 0.0001353 0.03% MultiFab::max() 6 0.0001325 0.0001325 0.0001325 0.03% MLLinOp::defineGrids() 6 0.000129 0.000129 0.000129 0.03% MLMG::MLRhsNormInf() 6 0.0001051 0.0001051 0.0001051 0.03% FabArrayBase::FB::FB() 26 6.596e-05 6.596e-05 6.596e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 3.018e-05 3.018e-05 3.018e-05 0.01% Castro::finalize_advance() 5 2.843e-05 2.843e-05 2.843e-05 0.01% Castro::swap_state_time_levels() 5 2.718e-05 2.718e-05 2.718e-05 0.01% Amr::writeSmallPlotFile() 1 2.331e-05 2.331e-05 2.331e-05 0.01% makeSFC 30 2.137e-05 2.137e-05 2.137e-05 0.01% Castro::finalize_do_advance() 5 2.108e-05 2.108e-05 2.108e-05 0.01% Castro::create_source_corrector() 5 1.94e-05 1.94e-05 1.94e-05 0.00% Amr::initSubcycle() 1 9.54e-06 9.54e-06 9.54e-06 0.00% DistributionMapping::Distribute() 31 8.699e-06 8.699e-06 8.699e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.27e-06 4.27e-06 4.27e-06 0.00% MLMG::buildFineMask() 6 3.745e-06 3.745e-06 3.745e-06 0.00% Gravity::set_mass_offset() 6 2.749e-06 2.749e-06 2.749e-06 0.00% Castro::retry_advance_ctu() 5 1.686e-06 1.686e-06 1.686e-06 0.00% Castro::FluxRegFineAdd() 5 1.368e-06 1.368e-06 1.368e-06 0.00% AmrLevel::AmrLevel() 1 1.123e-06 1.123e-06 1.123e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.069e-06 1.069e-06 1.069e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.03-8-g6bbcbbb78299) finalized