Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-7-gec648a92a89f) initialized Starting run at 08:24:16 UTC on 2022-04-06. Successfully read inputs file ... Castro git describe: 22.04-1-gabf8dcbcc AMReX git describe: 22.04-7-gec648a92a Microphysics git describe: 22.04-1-gef526ccd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.040788331 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.024663793 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.109485706 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049973509 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.049205247 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.064541387 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.060860699 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.038259829 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051957322 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050074129 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058954382 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060234491 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063975586 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.038441499 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024292403 seconds Ending run at 08:24:16 UTC on 2022-04-06. Run time = 0.848524285 Run time without initialization = 0.720732017 Average number of zones advanced per microsecond: 3.637 Average number of zones advanced per microsecond per rank: 3.637 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8486 ... 0.8486 ... 0.8486 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2075 0.2075 0.2075 24.45% VisMF::Write(FabArray) 11 0.1592 0.1592 0.1592 18.76% MLCellLinOp::applyBC() 4433 0.09527 0.09527 0.09527 11.23% MLPoisson::Fsmooth() 3280 0.06034 0.06034 0.06034 7.11% FabArray::setVal() 1144 0.02644 0.02644 0.02644 3.12% StateData::FillBoundary(geom) 328 0.02396 0.02396 0.02396 2.82% MLCGSolver::bicgstab 82 0.02225 0.02225 0.02225 2.62% MultiFab::Dot() 1114 0.02192 0.02192 0.02192 2.58% StateDataPhysBCFunct::() 41 0.0216 0.0216 0.0216 2.55% FillBoundary_nowait() 4023 0.01802 0.01802 0.01802 2.12% FabArray::ParallelCopy_nowait() 861 0.01638 0.01638 0.01638 1.93% Castro::computeTemp() 63 0.01322 0.01322 0.01322 1.56% MultiFab::LinComb() 1586 0.01251 0.01251 0.01251 1.47% Gravity::fill_multipole_BCs() 11 0.01182 0.01182 0.01182 1.39% Castro::enforce_min_density() 62 0.01149 0.01149 0.01149 1.35% MLPoisson::Fapply() 1142 0.01077 0.01077 0.01077 1.27% MLCellLinOp::defineAuxData() 11 0.01056 0.01056 0.01056 1.24% MLMG::addInterpCorrection() 410 0.007202 0.007202 0.007202 0.85% amrex::average_down 410 0.006725 0.006725 0.006725 0.79% MultiFab::Xpay() 585 0.006157 0.006157 0.006157 0.73% Castro::estTimeStep() 21 0.005661 0.005661 0.005661 0.67% Castro::reset_internal_energy() 63 0.005571 0.005571 0.005571 0.66% FabArray::setDomainBndry() 41 0.005372 0.005372 0.005372 0.63% Castro::expand_state() 10 0.00476 0.00476 0.00476 0.56% Castro::normalize_species() 62 0.004461 0.004461 0.004461 0.53% Amr::checkPoint() 3 0.003995 0.003995 0.003995 0.47% BndryData::define() 11 0.003831 0.003831 0.003831 0.45% Castro::enforce_speed_limit() 62 0.003499 0.003499 0.003499 0.41% Amr::writePlotFile() 2 0.003289 0.003289 0.003289 0.39% Castro::do_advance_ctu() 10 0.00328 0.00328 0.00328 0.39% Castro::construct_new_gravity_source() 10 0.003027 0.003027 0.003027 0.36% Gravity::get_new_grav_vector() 11 0.002661 0.002661 0.002661 0.31% Castro::construct_old_gravity_source() 10 0.002423 0.002423 0.002423 0.29% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002342 0.002342 0.002342 0.28% MLCellLinOp::compGrad() 11 0.001959 0.001959 0.001959 0.23% MLMG::ResNormInf() 93 0.001945 0.001945 0.001945 0.23% MultiFab::Saxpy() 20 0.001807 0.001807 0.001807 0.21% Gravity::get_old_grav_vector() 10 0.001775 0.001775 0.001775 0.21% MLMG::oneIter() 82 0.001711 0.001711 0.001711 0.20% MLCellLinOp::setLevelBC() 11 0.00157 0.00157 0.00157 0.18% Gravity::actual_solve_with_mlmg() 11 0.001396 0.001396 0.001396 0.16% MLCellLinOp::prepareForSolve() 11 0.001383 0.001383 0.001383 0.16% FabArray::mult() 43 0.001324 0.001324 0.001324 0.16% MLCellLinOp::smooth() 1640 0.001235 0.001235 0.001235 0.15% Castro::initData() 1 0.001219 0.001219 0.001219 0.14% FabArrayBase::getCPC() 1323 0.0008803 0.0008803 0.0008803 0.10% FabArray::FillBoundary() 4023 0.0008275 0.0008275 0.0008275 0.10% MLMG::prepareForSolve() 11 0.0007949 0.0007949 0.0007949 0.09% FabArrayBase::CPC::define() 454 0.0007459 0.0007459 0.0007459 0.09% FabArrayBase::getFB() 4023 0.0007177 0.0007177 0.0007177 0.08% MLCellLinOp::apply() 1142 0.0005575 0.0005575 0.0005575 0.07% Gravity::update_max_rhs() 11 0.0005494 0.0005494 0.0005494 0.06% CGSolver::sxay() 1586 0.0005243 0.0005243 0.0005243 0.06% MultiFab::Copy() 11 0.0005209 0.0005209 0.0005209 0.06% Gravity::solve_for_phi() 10 0.0004698 0.0004698 0.0004698 0.06% Amr::InitAmr() 1 0.0004638 0.0004638 0.0004638 0.05% MLMG::mgVcycle() 82 0.0003809 0.0003809 0.0003809 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003281 0.0003281 0.0003281 0.04% MultiFab::min() 10 0.0003261 0.0003261 0.0003261 0.04% main() 1 0.0003045 0.0003045 0.0003045 0.04% FabArray::ParallelCopy() 861 0.0003023 0.0003023 0.0003023 0.04% Castro::construct_new_gravity() 10 0.0002686 0.0002686 0.0002686 0.03% MultiFab::max() 11 0.0002573 0.0002573 0.0002573 0.03% FillPatchIterator::Initialize 41 0.0002558 0.0002558 0.0002558 0.03% MLCellLinOp::correctionResidual() 492 0.000249 0.000249 0.000249 0.03% Gravity::actual_multilevel_solve() 1 0.0002178 0.0002178 0.0002178 0.03% Amr::coarseTimeStep() 10 0.0002042 0.0002042 0.0002042 0.02% MLMG::MLRhsNormInf() 11 0.0001998 0.0001998 0.0001998 0.02% MLCellLinOp::defineBC() 11 0.0001847 0.0001847 0.0001847 0.02% Amr::timeStep() 10 0.0001816 0.0001816 0.0001816 0.02% MLLinOp::defineGrids() 11 0.0001716 0.0001716 0.0001716 0.02% MLMG:computeResOfCorrection() 410 0.0001549 0.0001549 0.0001549 0.02% Amr::defBaseLevel() 1 0.000149 0.000149 0.000149 0.02% Castro::subcycle_advance_ctu() 10 0.000147 0.000147 0.000147 0.02% StateData::checkPoint() 12 0.0001329 0.0001329 0.0001329 0.02% MLMG::actualBottomSolve() 82 0.0001166 0.0001166 0.0001166 0.01% MLMG::mgVcycle_down::0 82 0.0001025 0.0001025 0.0001025 0.01% FabArrayBase::FB::FB() 56 9.11e-05 9.11e-05 9.11e-05 0.01% MLMG::mgVcycle_down::1 82 9.038e-05 9.038e-05 9.038e-05 0.01% MLMG::mgVcycle_down::2 82 8.377e-05 8.377e-05 8.377e-05 0.01% MLMG::mgVcycle_down::4 82 8.076e-05 8.076e-05 8.076e-05 0.01% MLMG::mgVcycle_down::3 82 7.973e-05 7.973e-05 7.973e-05 0.01% Castro::post_timestep() 10 7.916e-05 7.916e-05 7.916e-05 0.01% Castro::initialize_advance() 10 7.905e-05 7.905e-05 7.905e-05 0.01% MLMG::solve() 11 7.564e-05 7.564e-05 7.564e-05 0.01% Castro::clean_state() 62 7.274e-05 7.274e-05 7.274e-05 0.01% AmrLevel::checkPoint() 3 6.957e-05 6.957e-05 6.957e-05 0.01% Castro::advance() 10 6.474e-05 6.474e-05 6.474e-05 0.01% MLMG::mgVcycle_up::4 82 6.415e-05 6.415e-05 6.415e-05 0.01% Castro::construct_new_source() 50 6.106e-05 6.106e-05 6.106e-05 0.01% Castro::finalize_advance() 10 5.879e-05 5.879e-05 5.879e-05 0.01% Castro::initialize_do_advance() 10 5.84e-05 5.84e-05 5.84e-05 0.01% MLMG::mgVcycle_up::1 82 5.562e-05 5.562e-05 5.562e-05 0.01% MLMG::mgVcycle_up::0 82 5.504e-05 5.504e-05 5.504e-05 0.01% MLMG::mgVcycle_up::3 82 5.484e-05 5.484e-05 5.484e-05 0.01% MLMG::mgVcycle_up::2 82 5.282e-05 5.282e-05 5.282e-05 0.01% MLCellLinOp::solutionResidual() 93 5.104e-05 5.104e-05 5.104e-05 0.01% StateData::define() 4 4.321e-05 4.321e-05 4.321e-05 0.01% Castro::finalize_do_advance() 10 3.881e-05 3.881e-05 3.881e-05 0.00% Castro::swap_state_time_levels() 10 3.657e-05 3.657e-05 3.657e-05 0.00% Castro::enforce_consistent_e() 1 3.534e-05 3.534e-05 3.534e-05 0.00% MLMG::computeResidual() 82 3.454e-05 3.454e-05 3.454e-05 0.00% MLMG::mgVcycle_bottom 82 3.432e-05 3.432e-05 3.432e-05 0.00% makeSFC 55 2.972e-05 2.972e-05 2.972e-05 0.00% FillPatchSingleLevel 41 2.958e-05 2.958e-05 2.958e-05 0.00% MLLinOp::define() 11 2.843e-05 2.843e-05 2.843e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.675e-05 2.675e-05 2.675e-05 0.00% Amr::writeSmallPlotFile() 1 2.528e-05 2.528e-05 2.528e-05 0.00% MLPoisson::define() 11 2.352e-05 2.352e-05 2.352e-05 0.00% Castro::construct_old_source() 50 1.958e-05 1.958e-05 1.958e-05 0.00% Amr::FinalizeInit() 1 1.856e-05 1.856e-05 1.856e-05 0.00% Castro::do_new_sources() 10 1.736e-05 1.736e-05 1.736e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.634e-05 1.634e-05 1.634e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.623e-05 1.623e-05 1.623e-05 0.00% Castro::do_old_sources() 10 1.609e-05 1.609e-05 1.609e-05 0.00% DistributionMapping::Distribute() 56 1.553e-05 1.553e-05 1.553e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.478e-05 1.478e-05 1.478e-05 0.00% Castro::apply_source_to_state() 20 1.169e-05 1.169e-05 1.169e-05 0.00% Amr::initSubcycle() 1 1.08e-05 1.08e-05 1.08e-05 0.00% Castro::construct_old_gravity() 10 1.042e-05 1.042e-05 1.042e-05 0.00% Gravity::swapTimeLevels() 10 9.959e-06 9.959e-06 9.959e-06 0.00% MLPoisson::prepareForSolve() 11 7.471e-06 7.471e-06 7.471e-06 0.00% MLMG::computeMLResidual() 11 6.868e-06 6.868e-06 6.868e-06 0.00% MLMG::getGradSolution() 11 6.269e-06 6.269e-06 6.269e-06 0.00% Castro::create_source_corrector() 10 6.137e-06 6.137e-06 6.137e-06 0.00% Amr::InitializeInit() 1 6.088e-06 6.088e-06 6.088e-06 0.00% Castro::computeNewDt() 9 6.083e-06 6.083e-06 6.083e-06 0.00% MLMG::buildFineMask() 11 5.478e-06 5.478e-06 5.478e-06 0.00% AmrLevel::checkPointPost() 3 5.135e-06 5.135e-06 5.135e-06 0.00% Gravity::set_mass_offset() 11 4.973e-06 4.973e-06 4.973e-06 0.00% MLMG::MLResNormInf() 11 4.703e-06 4.703e-06 4.703e-06 0.00% Castro::post_init() 1 4.314e-06 4.314e-06 4.314e-06 0.00% Castro::retry_advance_ctu() 10 3.979e-06 3.979e-06 3.979e-06 0.00% Castro::FluxRegFineAdd() 10 3.096e-06 3.096e-06 3.096e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.724e-06 2.724e-06 2.724e-06 0.00% Castro::computeInitialDt() 2 2.555e-06 2.555e-06 2.555e-06 0.00% Amr::init() 1 2.511e-06 2.511e-06 2.511e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.077e-06 2.077e-06 2.077e-06 0.00% AmrLevel::checkPointPre() 3 1.928e-06 1.928e-06 1.928e-06 0.00% Castro::post_regrid() 1 1.503e-06 1.503e-06 1.503e-06 0.00% Amr::initialInit() 1 1.016e-06 1.016e-06 1.016e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8485 0.8485 0.8485 100.00% Amr::coarseTimeStep() 10 0.6962 0.6962 0.6962 82.05% Amr::timeStep() 10 0.6164 0.6164 0.6164 72.64% Castro::advance() 10 0.6095 0.6095 0.6095 71.83% Castro::subcycle_advance_ctu() 10 0.5954 0.5954 0.5954 70.17% Castro::do_advance_ctu() 10 0.5953 0.5953 0.5953 70.15% Gravity::solve_phi_with_mlmg() 11 0.3315 0.3315 0.3315 39.07% Gravity::actual_solve_with_mlmg() 11 0.3195 0.3195 0.3195 37.65% Castro::construct_new_gravity() 10 0.3067 0.3067 0.3067 36.14% MLMG::solve() 11 0.2961 0.2961 0.2961 34.89% Gravity::solve_for_phi() 10 0.2829 0.2829 0.2829 33.34% MLMG::oneIter() 82 0.28 0.28 0.28 33.00% MLMG::mgVcycle() 82 0.2783 0.2783 0.2783 32.79% Castro::construct_ctu_hydro_source() 10 0.2075 0.2075 0.2075 24.45% VisMF::Write(FabArray) 11 0.1592 0.1592 0.1592 18.76% MLCellLinOp::smooth() 1640 0.1469 0.1469 0.1469 17.32% Amr::init() 1 0.1272 0.1272 0.1272 14.99% Amr::checkPoint() 3 0.1176 0.1176 0.1176 13.86% MLCellLinOp::applyBC() 4433 0.1149 0.1149 0.1149 13.54% AmrLevel::checkPoint() 3 0.1136 0.1136 0.1136 13.39% StateData::checkPoint() 12 0.1135 0.1135 0.1135 13.38% MLMG::mgVcycle_bottom 82 0.07949 0.07949 0.07949 9.37% MLMG::actualBottomSolve() 82 0.07945 0.07945 0.07945 9.36% MLCGSolver::bicgstab 82 0.07869 0.07869 0.07869 9.27% Amr::initialInit() 1 0.06158 0.06158 0.06158 7.26% MLPoisson::Fsmooth() 3280 0.06034 0.06034 0.06034 7.11% FillPatchIterator::Initialize 41 0.05545 0.05545 0.05545 6.53% Amr::FinalizeInit() 1 0.05253 0.05253 0.05253 6.19% Castro::post_init() 1 0.0518 0.0518 0.0518 6.10% FillPatchSingleLevel 41 0.04983 0.04983 0.04983 5.87% Gravity::multilevel_solve_for_new_phi() 1 0.04938 0.04938 0.04938 5.82% Gravity::actual_multilevel_solve() 1 0.04936 0.04936 0.04936 5.82% Amr::writePlotFile() 2 0.04909 0.04909 0.04909 5.79% StateDataPhysBCFunct::() 41 0.04556 0.04556 0.04556 5.37% MLCellLinOp::apply() 1142 0.04033 0.04033 0.04033 4.75% MLMG::mgVcycle_down::0 82 0.0387 0.0387 0.0387 4.56% Castro::clean_state() 62 0.0376 0.0376 0.0376 4.43% MLMG::mgVcycle_up::0 82 0.0331 0.0331 0.0331 3.90% FabArray::setVal() 1144 0.02644 0.02644 0.02644 3.12% Gravity::get_new_grav_vector() 11 0.02578 0.02578 0.02578 3.04% Castro::initialize_do_advance() 10 0.02444 0.02444 0.02444 2.88% StateData::FillBoundary(geom) 328 0.02396 0.02396 0.02396 2.82% MLCellLinOp::correctionResidual() 492 0.02272 0.02272 0.02272 2.68% MultiFab::Dot() 1114 0.02192 0.02192 0.02192 2.58% MLMG:computeResOfCorrection() 410 0.01967 0.01967 0.01967 2.32% FabArray::FillBoundary() 4023 0.01965 0.01965 0.01965 2.32% MLMG::mgVcycle_down::1 82 0.01894 0.01894 0.01894 2.23% FillBoundary_nowait() 4023 0.01883 0.01883 0.01883 2.22% Castro::computeTemp() 63 0.01879 0.01879 0.01879 2.21% MLMG::mgVcycle_down::2 82 0.01831 0.01831 0.01831 2.16% Castro::expand_state() 10 0.01824 0.01824 0.01824 2.15% MLPoisson::define() 11 0.0179 0.0179 0.0179 2.11% FabArray::ParallelCopy() 861 0.01761 0.01761 0.01761 2.07% MLMG::mgVcycle_down::3 82 0.01748 0.01748 0.01748 2.06% FabArray::ParallelCopy_nowait() 861 0.0173 0.0173 0.0173 2.04% MLMG::mgVcycle_down::4 82 0.01666 0.01666 0.01666 1.96% Castro::construct_old_gravity() 10 0.01628 0.01628 0.01628 1.92% Gravity::get_old_grav_vector() 10 0.01627 0.01627 0.01627 1.92% MLMG::mgVcycle_up::2 82 0.01424 0.01424 0.01424 1.68% MLMG::mgVcycle_up::1 82 0.01401 0.01401 0.01401 1.65% Castro::initialize_advance() 10 0.01398 0.01398 0.01398 1.65% MLMG::addInterpCorrection() 410 0.0139 0.0139 0.0139 1.64% MLMG::mgVcycle_up::3 82 0.01357 0.01357 0.01357 1.60% MLMG::mgVcycle_up::4 82 0.0134 0.0134 0.0134 1.58% amrex::average_down 410 0.0134 0.0134 0.0134 1.58% MLCGSolver::ParallelAllReduce 1514 0.01319 0.01319 0.01319 1.55% CGSolver::sxay() 1586 0.01303 0.01303 0.01303 1.54% MultiFab::LinComb() 1586 0.01251 0.01251 0.01251 1.47% MLCellLinOp::defineAuxData() 11 0.01229 0.01229 0.01229 1.45% Gravity::fill_multipole_BCs() 11 0.01182 0.01182 0.01182 1.39% Castro::do_new_sources() 10 0.01173 0.01173 0.01173 1.38% Castro::enforce_min_density() 62 0.01149 0.01149 0.01149 1.35% MLPoisson::Fapply() 1142 0.01077 0.01077 0.01077 1.27% Amr::InitializeInit() 1 0.009046 0.009046 0.009046 1.07% Amr::defBaseLevel() 1 0.00904 0.00904 0.00904 1.07% Castro::do_old_sources() 10 0.008556 0.008556 0.008556 1.01% MLCellLinOp::solutionResidual() 93 0.007888 0.007888 0.007888 0.93% MLMG::computeResidual() 82 0.006827 0.006827 0.006827 0.80% Castro::post_timestep() 10 0.006659 0.006659 0.006659 0.78% MultiFab::Xpay() 585 0.006157 0.006157 0.006157 0.73% MLMG::prepareForSolve() 11 0.005954 0.005954 0.005954 0.70% Castro::estTimeStep() 21 0.005661 0.005661 0.005661 0.67% Castro::reset_internal_energy() 63 0.005571 0.005571 0.005571 0.66% FabArray::setDomainBndry() 41 0.005372 0.005372 0.005372 0.63% MLCellLinOp::defineBC() 11 0.005327 0.005327 0.005327 0.63% BndryData::define() 11 0.005142 0.005142 0.005142 0.61% Castro::normalize_species() 62 0.004461 0.004461 0.004461 0.53% Castro::enforce_speed_limit() 62 0.003499 0.003499 0.003499 0.41% Castro::initData() 1 0.003245 0.003245 0.003245 0.38% Castro::construct_new_source() 50 0.003088 0.003088 0.003088 0.36% Castro::construct_new_gravity_source() 10 0.003027 0.003027 0.003027 0.36% MLMG::getGradSolution() 11 0.002519 0.002519 0.002519 0.30% MLCellLinOp::compGrad() 11 0.002513 0.002513 0.002513 0.30% Castro::construct_old_source() 50 0.002443 0.002443 0.002443 0.29% Castro::construct_old_gravity_source() 10 0.002423 0.002423 0.002423 0.29% Castro::computeNewDt() 9 0.002403 0.002403 0.002403 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002342 0.002342 0.002342 0.28% MLMG::ResNormInf() 93 0.001945 0.001945 0.001945 0.23% Castro::apply_source_to_state() 20 0.001819 0.001819 0.001819 0.21% MultiFab::Saxpy() 20 0.001807 0.001807 0.001807 0.21% FabArrayBase::getCPC() 1323 0.001626 0.001626 0.001626 0.19% MLCellLinOp::setLevelBC() 11 0.00157 0.00157 0.00157 0.18% MLPoisson::prepareForSolve() 11 0.00139 0.00139 0.00139 0.16% MLCellLinOp::prepareForSolve() 11 0.001383 0.001383 0.001383 0.16% FabArray::mult() 43 0.001324 0.001324 0.001324 0.16% Gravity::swapTimeLevels() 10 0.001155 0.001155 0.001155 0.14% MLMG::computeMLResidual() 11 0.001103 0.001103 0.001103 0.13% Gravity::update_max_rhs() 11 0.0009505 0.0009505 0.0009505 0.11% FabArrayBase::getFB() 4023 0.0008088 0.0008088 0.0008088 0.10% FabArrayBase::CPC::define() 454 0.0007459 0.0007459 0.0007459 0.09% Castro::computeInitialDt() 2 0.00068 0.00068 0.00068 0.08% MultiFab::Copy() 11 0.0005209 0.0005209 0.0005209 0.06% Castro::post_regrid() 1 0.0004994 0.0004994 0.0004994 0.06% Amr::InitAmr() 1 0.0004746 0.0004746 0.0004746 0.06% MultiFab::min() 10 0.0003261 0.0003261 0.0003261 0.04% MLLinOp::define() 11 0.0002624 0.0002624 0.0002624 0.03% MultiFab::max() 11 0.0002573 0.0002573 0.0002573 0.03% MLMG::MLResNormInf() 11 0.0002517 0.0002517 0.0002517 0.03% MLLinOp::defineGrids() 11 0.0002339 0.0002339 0.0002339 0.03% MLMG::MLRhsNormInf() 11 0.0001998 0.0001998 0.0001998 0.02% FabArrayBase::FB::FB() 56 9.11e-05 9.11e-05 9.11e-05 0.01% Castro::finalize_advance() 10 6.188e-05 6.188e-05 6.188e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 6.023e-05 6.023e-05 6.023e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.799e-05 5.799e-05 5.799e-05 0.01% makeSFC 55 4.388e-05 4.388e-05 4.388e-05 0.01% StateData::define() 4 4.321e-05 4.321e-05 4.321e-05 0.01% Castro::finalize_do_advance() 10 3.881e-05 3.881e-05 3.881e-05 0.00% Castro::swap_state_time_levels() 10 3.657e-05 3.657e-05 3.657e-05 0.00% Castro::enforce_consistent_e() 1 3.534e-05 3.534e-05 3.534e-05 0.00% Amr::writeSmallPlotFile() 1 2.528e-05 2.528e-05 2.528e-05 0.00% DistributionMapping::Distribute() 56 1.553e-05 1.553e-05 1.553e-05 0.00% Amr::initSubcycle() 1 1.08e-05 1.08e-05 1.08e-05 0.00% Castro::create_source_corrector() 10 6.137e-06 6.137e-06 6.137e-06 0.00% MLMG::buildFineMask() 11 5.478e-06 5.478e-06 5.478e-06 0.00% AmrLevel::checkPointPost() 3 5.135e-06 5.135e-06 5.135e-06 0.00% Gravity::set_mass_offset() 11 4.973e-06 4.973e-06 4.973e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.087e-06 4.087e-06 4.087e-06 0.00% Castro::retry_advance_ctu() 10 3.979e-06 3.979e-06 3.979e-06 0.00% Castro::FluxRegFineAdd() 10 3.096e-06 3.096e-06 3.096e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.077e-06 2.077e-06 2.077e-06 0.00% AmrLevel::checkPointPre() 3 1.928e-06 1.928e-06 1.928e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-7-gec648a92a89f) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-7-gec648a92a89f) initialized Starting run at 08:24:17 UTC on 2022-04-06. Successfully read inputs file ... Castro git describe: 22.04-1-gabf8dcbcc AMReX git describe: 22.04-7-gec648a92a Microphysics git describe: 22.04-1-gef526ccd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.500033016 Restart time = 0.055927366 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.11246132 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048190181 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.050474675 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.05982021 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065570446 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.058041931 seconds Ending run at 08:24:17 UTC on 2022-04-06. Run time = 0.451418776 Run time without initialization = 0.394922679 Average number of zones advanced per microsecond: 3.319 Average number of zones advanced per microsecond per rank: 3.319 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4514 ... 0.4514 ... 0.4514 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1267 0.1267 0.1267 28.07% VisMF::Read() 3 0.04413 0.04413 0.04413 9.77% MLCellLinOp::applyBC() 1946 0.03792 0.03792 0.03792 8.40% Amr::writePlotFile() 1 0.03442 0.03442 0.03442 7.62% MLPoisson::Fsmooth() 1440 0.02576 0.02576 0.02576 5.71% VisMF::Write(FabArray) 1 0.0237 0.0237 0.0237 5.25% FabArray::setVal() 537 0.01637 0.01637 0.01637 3.63% StateData::FillBoundary(geom) 160 0.01168 0.01168 0.01168 2.59% MLCGSolver::bicgstab 36 0.009555 0.009555 0.009555 2.12% MultiFab::Dot() 484 0.009354 0.009354 0.009354 2.07% Castro::computeTemp() 30 0.007626 0.007626 0.007626 1.69% FillBoundary_nowait() 1766 0.007296 0.007296 0.007296 1.62% FabArray::ParallelCopy_nowait() 380 0.007181 0.007181 0.007181 1.59% StateDataPhysBCFunct::() 20 0.007129 0.007129 0.007129 1.58% Gravity::fill_multipole_BCs() 6 0.006509 0.006509 0.006509 1.44% Castro::enforce_min_density() 30 0.00579 0.00579 0.00579 1.28% MLCellLinOp::defineAuxData() 6 0.005619 0.005619 0.005619 1.24% MultiFab::LinComb() 690 0.00538 0.00538 0.00538 1.19% MLPoisson::Fapply() 500 0.004651 0.004651 0.004651 1.03% FabArray::setDomainBndry() 20 0.004024 0.004024 0.004024 0.89% Castro::expand_state() 5 0.003978 0.003978 0.003978 0.88% MLMG::addInterpCorrection() 180 0.00302 0.00302 0.00302 0.67% Amr::restart() 1 0.002989 0.002989 0.002989 0.66% amrex::average_down 180 0.002836 0.002836 0.002836 0.63% MultiFab::Xpay() 258 0.00269 0.00269 0.00269 0.60% Castro::estTimeStep() 10 0.002453 0.002453 0.002453 0.54% Castro::do_advance_ctu() 5 0.002442 0.002442 0.002442 0.54% Castro::reset_internal_energy() 30 0.002416 0.002416 0.002416 0.54% Castro::normalize_species() 30 0.002216 0.002216 0.002216 0.49% BndryData::define() 6 0.002069 0.002069 0.002069 0.46% Gravity::get_new_grav_vector() 5 0.001938 0.001938 0.001938 0.43% Castro::construct_new_gravity_source() 5 0.00151 0.00151 0.00151 0.33% MLCellLinOp::compGrad() 6 0.001425 0.001425 0.001425 0.32% Castro::construct_old_gravity_source() 5 0.001245 0.001245 0.001245 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.00122 0.00122 0.00122 0.27% Castro::enforce_speed_limit() 30 0.001134 0.001134 0.001134 0.25% MultiFab::Saxpy() 10 0.0009196 0.0009196 0.0009196 0.20% Gravity::get_old_grav_vector() 5 0.00091 0.00091 0.00091 0.20% MLMG::ResNormInf() 42 0.0008621 0.0008621 0.0008621 0.19% MLCellLinOp::setLevelBC() 6 0.0008415 0.0008415 0.0008415 0.19% MLMG::oneIter() 36 0.0007513 0.0007513 0.0007513 0.17% Gravity::actual_solve_with_mlmg() 6 0.0007246 0.0007246 0.0007246 0.16% MLCellLinOp::prepareForSolve() 6 0.0006988 0.0006988 0.0006988 0.15% FabArray::mult() 22 0.0006606 0.0006606 0.0006606 0.15% MLCellLinOp::smooth() 720 0.0005555 0.0005555 0.0005555 0.12% Gravity::update_max_rhs() 6 0.000529 0.000529 0.000529 0.12% Castro::subcycle_advance_ctu() 5 0.0004768 0.0004768 0.0004768 0.11% MLMG::prepareForSolve() 6 0.0004288 0.0004288 0.0004288 0.09% FabArrayBase::CPC::define() 244 0.0004212 0.0004212 0.0004212 0.09% FabArrayBase::getCPC() 632 0.0004145 0.0004145 0.0004145 0.09% FabArray::FillBoundary() 1766 0.0004112 0.0004112 0.0004112 0.09% Amr::InitAmr() 1 0.0003918 0.0003918 0.0003918 0.09% MultiFab::Copy() 6 0.0003707 0.0003707 0.0003707 0.08% FabArrayBase::getFB() 1766 0.0003088 0.0003088 0.0003088 0.07% main() 1 0.0003056 0.0003056 0.0003056 0.07% Gravity::solve_for_phi() 5 0.0002368 0.0002368 0.0002368 0.05% MLCellLinOp::apply() 500 0.0002298 0.0002298 0.0002298 0.05% Gravity::actual_multilevel_solve() 1 0.0002274 0.0002274 0.0002274 0.05% CGSolver::sxay() 690 0.0002088 0.0002088 0.0002088 0.05% Castro::construct_new_gravity() 5 0.0001892 0.0001892 0.0001892 0.04% MLMG::mgVcycle() 36 0.0001651 0.0001651 0.0001651 0.04% MultiFab::min() 5 0.0001638 0.0001638 0.0001638 0.04% FabArray::ParallelCopy() 380 0.0001459 0.0001459 0.0001459 0.03% MLCGSolver::ParallelAllReduce 659 0.0001399 0.0001399 0.0001399 0.03% MultiFab::max() 6 0.0001341 0.0001341 0.0001341 0.03% FillPatchIterator::Initialize 20 0.0001225 0.0001225 0.0001225 0.03% Castro::construct_new_source() 25 0.0001121 0.0001121 0.0001121 0.02% MLMG::MLRhsNormInf() 6 0.0001064 0.0001064 0.0001064 0.02% Amr::coarseTimeStep() 5 0.0001056 0.0001056 0.0001056 0.02% Amr::timeStep() 5 0.0001026 0.0001026 0.0001026 0.02% MLCellLinOp::defineBC() 6 9.673e-05 9.673e-05 9.673e-05 0.02% MLCellLinOp::correctionResidual() 216 9.342e-05 9.342e-05 9.342e-05 0.02% MLLinOp::defineGrids() 6 9.283e-05 9.283e-05 9.283e-05 0.02% AmrLevel::restart() 1 7.602e-05 7.602e-05 7.602e-05 0.02% StateData::restartDoit() 4 7.28e-05 7.28e-05 7.28e-05 0.02% MLMG:computeResOfCorrection() 180 6.294e-05 6.294e-05 6.294e-05 0.01% FabArrayBase::FB::FB() 26 6.13e-05 6.13e-05 6.13e-05 0.01% Castro::post_timestep() 5 6.001e-05 6.001e-05 6.001e-05 0.01% Castro::advance() 5 5.582e-05 5.582e-05 5.582e-05 0.01% MLMG::actualBottomSolve() 36 5.099e-05 5.099e-05 5.099e-05 0.01% Castro::create_source_corrector() 5 4.266e-05 4.266e-05 4.266e-05 0.01% MLMG::mgVcycle_down::0 36 4.138e-05 4.138e-05 4.138e-05 0.01% MLMG::mgVcycle_down::1 36 3.809e-05 3.809e-05 3.809e-05 0.01% MLMG::solve() 6 3.733e-05 3.733e-05 3.733e-05 0.01% Castro::do_new_sources() 5 3.715e-05 3.715e-05 3.715e-05 0.01% Castro::construct_old_source() 25 3.713e-05 3.713e-05 3.713e-05 0.01% Castro::clean_state() 30 3.645e-05 3.645e-05 3.645e-05 0.01% MLMG::mgVcycle_down::2 36 3.566e-05 3.566e-05 3.566e-05 0.01% Castro::initialize_advance() 5 3.476e-05 3.476e-05 3.476e-05 0.01% MLMG::mgVcycle_down::3 36 3.374e-05 3.374e-05 3.374e-05 0.01% MLMG::mgVcycle_down::4 36 3.368e-05 3.368e-05 3.368e-05 0.01% MLMG::mgVcycle_up::4 36 2.906e-05 2.906e-05 2.906e-05 0.01% Castro::initialize_do_advance() 5 2.827e-05 2.827e-05 2.827e-05 0.01% MLLinOp::define() 6 2.815e-05 2.815e-05 2.815e-05 0.01% Castro::finalize_advance() 5 2.659e-05 2.659e-05 2.659e-05 0.01% Amr::writeSmallPlotFile() 1 2.55e-05 2.55e-05 2.55e-05 0.01% MLMG::mgVcycle_up::0 36 2.493e-05 2.493e-05 2.493e-05 0.01% Castro::swap_state_time_levels() 5 2.477e-05 2.477e-05 2.477e-05 0.01% MLCellLinOp::solutionResidual() 42 2.462e-05 2.462e-05 2.462e-05 0.01% Castro::post_restart() 1 2.456e-05 2.456e-05 2.456e-05 0.01% MLMG::mgVcycle_up::3 36 2.327e-05 2.327e-05 2.327e-05 0.01% MLMG::mgVcycle_up::2 36 2.244e-05 2.244e-05 2.244e-05 0.00% MLMG::mgVcycle_up::1 36 2.174e-05 2.174e-05 2.174e-05 0.00% Castro::finalize_do_advance() 5 1.97e-05 1.97e-05 1.97e-05 0.00% MLMG::mgVcycle_bottom 36 1.743e-05 1.743e-05 1.743e-05 0.00% MLMG::computeResidual() 36 1.59e-05 1.59e-05 1.59e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.569e-05 1.569e-05 1.569e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.551e-05 1.551e-05 1.551e-05 0.00% MLPoisson::define() 6 1.463e-05 1.463e-05 1.463e-05 0.00% FillPatchSingleLevel 20 1.43e-05 1.43e-05 1.43e-05 0.00% makeSFC 30 1.4e-05 1.4e-05 1.4e-05 0.00% DistributionMapping::Distribute() 31 9.906e-06 9.906e-06 9.906e-06 0.00% Amr::initSubcycle() 1 9.168e-06 9.168e-06 9.168e-06 0.00% Castro::do_old_sources() 5 7.909e-06 7.909e-06 7.909e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.792e-06 7.792e-06 7.792e-06 0.00% Castro::apply_source_to_state() 10 6.398e-06 6.398e-06 6.398e-06 0.00% Castro::construct_old_gravity() 5 5.318e-06 5.318e-06 5.318e-06 0.00% Gravity::swapTimeLevels() 5 4.45e-06 4.45e-06 4.45e-06 0.00% MLPoisson::prepareForSolve() 6 4.445e-06 4.445e-06 4.445e-06 0.00% MLMG::buildFineMask() 6 3.461e-06 3.461e-06 3.461e-06 0.00% Castro::computeNewDt() 5 3.288e-06 3.288e-06 3.288e-06 0.00% MLMG::computeMLResidual() 6 3.239e-06 3.239e-06 3.239e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.013e-06 3.013e-06 3.013e-06 0.00% MLMG::getGradSolution() 6 2.976e-06 2.976e-06 2.976e-06 0.00% MLMG::MLResNormInf() 6 2.627e-06 2.627e-06 2.627e-06 0.00% Gravity::set_mass_offset() 6 2.284e-06 2.284e-06 2.284e-06 0.00% Castro::retry_advance_ctu() 5 1.895e-06 1.895e-06 1.895e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.526e-06 1.526e-06 1.526e-06 0.00% Castro::FluxRegFineAdd() 5 1.372e-06 1.372e-06 1.372e-06 0.00% Amr::init() 1 1.206e-06 1.206e-06 1.206e-06 0.00% AmrLevel::AmrLevel() 1 7.64e-07 7.64e-07 7.64e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4514 0.4514 0.4514 100.00% Amr::coarseTimeStep() 5 0.3366 0.3366 0.3366 74.56% Amr::timeStep() 5 0.3354 0.3354 0.3354 74.28% Castro::advance() 5 0.3312 0.3312 0.3312 73.36% Castro::subcycle_advance_ctu() 5 0.3202 0.3202 0.3202 70.92% Castro::do_advance_ctu() 5 0.3197 0.3197 0.3197 70.82% Castro::construct_new_gravity() 5 0.1478 0.1478 0.1478 32.74% Gravity::solve_phi_with_mlmg() 6 0.144 0.144 0.144 31.89% Gravity::solve_for_phi() 5 0.1382 0.1382 0.1382 30.60% Gravity::actual_solve_with_mlmg() 6 0.1373 0.1373 0.1373 30.42% Castro::construct_ctu_hydro_source() 5 0.1267 0.1267 0.1267 28.07% MLMG::solve() 6 0.1245 0.1245 0.1245 27.57% MLMG::oneIter() 36 0.1165 0.1165 0.1165 25.80% MLMG::mgVcycle() 36 0.1157 0.1157 0.1157 25.63% MLCellLinOp::smooth() 720 0.06029 0.06029 0.06029 13.35% Amr::writePlotFile() 1 0.05812 0.05812 0.05812 12.87% Amr::init() 1 0.05596 0.05596 0.05596 12.40% Amr::restart() 1 0.05596 0.05596 0.05596 12.40% MLCellLinOp::applyBC() 1946 0.046 0.046 0.046 10.19% AmrLevel::restart() 1 0.04503 0.04503 0.04503 9.97% StateData::restartDoit() 4 0.04495 0.04495 0.04495 9.96% VisMF::Read() 3 0.04413 0.04413 0.04413 9.77% MLMG::mgVcycle_bottom 36 0.03366 0.03366 0.03366 7.46% MLMG::actualBottomSolve() 36 0.03364 0.03364 0.03364 7.45% MLCGSolver::bicgstab 36 0.03331 0.03331 0.03331 7.38% MLPoisson::Fsmooth() 1440 0.02576 0.02576 0.02576 5.71% FillPatchIterator::Initialize 20 0.02505 0.02505 0.02505 5.55% VisMF::Write(FabArray) 1 0.0237 0.0237 0.0237 5.25% FillPatchSingleLevel 20 0.0209 0.0209 0.0209 4.63% Castro::clean_state() 30 0.01922 0.01922 0.01922 4.26% StateDataPhysBCFunct::() 20 0.01881 0.01881 0.01881 4.17% MLCellLinOp::apply() 500 0.01662 0.01662 0.01662 3.68% FabArray::setVal() 537 0.01637 0.01637 0.01637 3.63% MLMG::mgVcycle_down::0 36 0.01605 0.01605 0.01605 3.56% Castro::initialize_do_advance() 5 0.01479 0.01479 0.01479 3.28% MLMG::mgVcycle_up::0 36 0.01371 0.01371 0.01371 3.04% Castro::expand_state() 5 0.01195 0.01195 0.01195 2.65% StateData::FillBoundary(geom) 160 0.01168 0.01168 0.01168 2.59% Castro::initialize_advance() 5 0.0109 0.0109 0.0109 2.41% Castro::computeTemp() 30 0.01004 0.01004 0.01004 2.22% MLPoisson::define() 6 0.009584 0.009584 0.009584 2.12% Gravity::get_new_grav_vector() 5 0.009481 0.009481 0.009481 2.10% MLCellLinOp::correctionResidual() 216 0.009395 0.009395 0.009395 2.08% MultiFab::Dot() 484 0.009354 0.009354 0.009354 2.07% Castro::construct_old_gravity() 5 0.008162 0.008162 0.008162 1.81% Gravity::get_old_grav_vector() 5 0.008157 0.008157 0.008157 1.81% MLMG:computeResOfCorrection() 180 0.008138 0.008138 0.008138 1.80% FabArray::FillBoundary() 1766 0.008077 0.008077 0.008077 1.79% MLMG::mgVcycle_down::1 36 0.007853 0.007853 0.007853 1.74% FabArray::ParallelCopy() 380 0.00775 0.00775 0.00775 1.72% FillBoundary_nowait() 1766 0.007666 0.007666 0.007666 1.70% FabArray::ParallelCopy_nowait() 380 0.007604 0.007604 0.007604 1.68% MLMG::mgVcycle_down::2 36 0.007527 0.007527 0.007527 1.67% MLMG::mgVcycle_down::3 36 0.007173 0.007173 0.007173 1.59% Castro::do_new_sources() 5 0.006954 0.006954 0.006954 1.54% MLMG::mgVcycle_down::4 36 0.006882 0.006882 0.006882 1.52% Castro::post_restart() 1 0.006694 0.006694 0.006694 1.48% MLCellLinOp::defineAuxData() 6 0.006534 0.006534 0.006534 1.45% Gravity::fill_multipole_BCs() 6 0.006509 0.006509 0.006509 1.44% Gravity::multilevel_solve_for_new_phi() 1 0.006302 0.006302 0.006302 1.40% Gravity::actual_multilevel_solve() 1 0.006286 0.006286 0.006286 1.39% MLMG::mgVcycle_up::2 36 0.00585 0.00585 0.00585 1.30% MLMG::addInterpCorrection() 180 0.005849 0.005849 0.005849 1.30% MLMG::mgVcycle_up::1 36 0.005798 0.005798 0.005798 1.28% Castro::enforce_min_density() 30 0.00579 0.00579 0.00579 1.28% amrex::average_down 180 0.005682 0.005682 0.005682 1.26% MLCGSolver::ParallelAllReduce 659 0.005646 0.005646 0.005646 1.25% CGSolver::sxay() 690 0.005589 0.005589 0.005589 1.24% MLMG::mgVcycle_up::3 36 0.005548 0.005548 0.005548 1.23% MLMG::mgVcycle_up::4 36 0.005494 0.005494 0.005494 1.22% MultiFab::LinComb() 690 0.00538 0.00538 0.00538 1.19% MLPoisson::Fapply() 500 0.004651 0.004651 0.004651 1.03% Castro::do_old_sources() 5 0.004145 0.004145 0.004145 0.92% Castro::post_timestep() 5 0.004077 0.004077 0.004077 0.90% FabArray::setDomainBndry() 20 0.004024 0.004024 0.004024 0.89% MLMG::prepareForSolve() 6 0.00355 0.00355 0.00355 0.79% MLCellLinOp::solutionResidual() 42 0.003429 0.003429 0.003429 0.76% MLCellLinOp::defineBC() 6 0.002884 0.002884 0.002884 0.64% MLMG::computeResidual() 36 0.002846 0.002846 0.002846 0.63% BndryData::define() 6 0.002787 0.002787 0.002787 0.62% MultiFab::Xpay() 258 0.00269 0.00269 0.00269 0.60% Castro::estTimeStep() 10 0.002453 0.002453 0.002453 0.54% Castro::reset_internal_energy() 30 0.002416 0.002416 0.002416 0.54% Castro::normalize_species() 30 0.002216 0.002216 0.002216 0.49% MLMG::getGradSolution() 6 0.001718 0.001718 0.001718 0.38% MLCellLinOp::compGrad() 6 0.001715 0.001715 0.001715 0.38% Castro::construct_new_source() 25 0.001622 0.001622 0.001622 0.36% Castro::construct_new_gravity_source() 5 0.00151 0.00151 0.00151 0.33% Castro::construct_old_source() 25 0.001282 0.001282 0.001282 0.28% Castro::construct_old_gravity_source() 5 0.001245 0.001245 0.001245 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.00122 0.00122 0.00122 0.27% Castro::computeNewDt() 5 0.001156 0.001156 0.001156 0.26% Castro::enforce_speed_limit() 30 0.001134 0.001134 0.001134 0.25% Castro::apply_source_to_state() 10 0.000926 0.000926 0.000926 0.21% MultiFab::Saxpy() 10 0.0009196 0.0009196 0.0009196 0.20% Gravity::swapTimeLevels() 5 0.0008798 0.0008798 0.0008798 0.19% MLMG::ResNormInf() 42 0.0008621 0.0008621 0.0008621 0.19% MLCellLinOp::setLevelBC() 6 0.0008415 0.0008415 0.0008415 0.19% FabArrayBase::getCPC() 632 0.0008357 0.0008357 0.0008357 0.19% Gravity::update_max_rhs() 6 0.0007419 0.0007419 0.0007419 0.16% MLPoisson::prepareForSolve() 6 0.0007032 0.0007032 0.0007032 0.16% MLCellLinOp::prepareForSolve() 6 0.0006988 0.0006988 0.0006988 0.15% FabArray::mult() 22 0.0006606 0.0006606 0.0006606 0.15% MLMG::computeMLResidual() 6 0.0006026 0.0006026 0.0006026 0.13% FabArrayBase::CPC::define() 244 0.0004212 0.0004212 0.0004212 0.09% Amr::InitAmr() 1 0.000401 0.000401 0.000401 0.09% MultiFab::Copy() 6 0.0003707 0.0003707 0.0003707 0.08% FabArrayBase::getFB() 1766 0.00037 0.00037 0.00037 0.08% MultiFab::min() 5 0.0001638 0.0001638 0.0001638 0.04% MLLinOp::define() 6 0.0001521 0.0001521 0.0001521 0.03% MLMG::MLResNormInf() 6 0.0001347 0.0001347 0.0001347 0.03% MultiFab::max() 6 0.0001341 0.0001341 0.0001341 0.03% MLLinOp::defineGrids() 6 0.0001239 0.0001239 0.0001239 0.03% MLMG::MLRhsNormInf() 6 0.0001064 0.0001064 0.0001064 0.02% FabArrayBase::FB::FB() 26 6.13e-05 6.13e-05 6.13e-05 0.01% Castro::create_source_corrector() 5 4.266e-05 4.266e-05 4.266e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.958e-05 2.958e-05 2.958e-05 0.01% Castro::finalize_advance() 5 2.797e-05 2.797e-05 2.797e-05 0.01% Amr::writeSmallPlotFile() 1 2.55e-05 2.55e-05 2.55e-05 0.01% Castro::swap_state_time_levels() 5 2.477e-05 2.477e-05 2.477e-05 0.01% makeSFC 30 2.179e-05 2.179e-05 2.179e-05 0.00% Castro::finalize_do_advance() 5 1.97e-05 1.97e-05 1.97e-05 0.00% DistributionMapping::Distribute() 31 9.906e-06 9.906e-06 9.906e-06 0.00% Amr::initSubcycle() 1 9.168e-06 9.168e-06 9.168e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.129e-06 5.129e-06 5.129e-06 0.00% MLMG::buildFineMask() 6 3.461e-06 3.461e-06 3.461e-06 0.00% Gravity::set_mass_offset() 6 2.284e-06 2.284e-06 2.284e-06 0.00% Castro::retry_advance_ctu() 5 1.895e-06 1.895e-06 1.895e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.526e-06 1.526e-06 1.526e-06 0.00% Castro::FluxRegFineAdd() 5 1.372e-06 1.372e-06 1.372e-06 0.00% AmrLevel::AmrLevel() 1 7.64e-07 7.64e-07 7.64e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-7-gec648a92a89f) finalized