Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.02-10-g76d08651adb9) initialized Starting run at 08:53:55 UTC on 2022-02-16. Successfully read inputs file ... Castro git describe: 22.02-4-g0d942c92d AMReX git describe: 22.02-10-g76d08651a Microphysics git describe: 22.02-2-gf51cc451 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.040548243 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.024571599 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.093998375 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050635481 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.059995038 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.063045408 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.066403778 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.038457101 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.050337196 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048987737 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.050665479 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.066831651 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.068450065 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.063498458 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024076512 seconds Ending run at 08:53:56 UTC on 2022-02-16. Run time = 0.869548108 Run time without initialization = 0.745855254 Average number of zones advanced per microsecond: 3.515 Average number of zones advanced per microsecond per rank: 3.515 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8696 ... 0.8696 ... 0.8696 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2181 0.2181 0.2181 25.08% VisMF::Write(FabArray) 11 0.1593 0.1593 0.1593 18.32% MLCellLinOp::applyBC() 4379 0.09416 0.09416 0.09416 10.83% MLPoisson::Fsmooth() 3240 0.06003 0.06003 0.06003 6.90% Amr::checkPoint() 3 0.02873 0.02873 0.02873 3.30% FabArray::setVal() 1135 0.0253 0.0253 0.0253 2.91% StateData::FillBoundary(geom) 328 0.02424 0.02424 0.02424 2.79% MLCGSolver::bicgstab 81 0.02193 0.02193 0.02193 2.52% MultiFab::Dot() 1100 0.02188 0.02188 0.02188 2.52% FillBoundary_nowait() 3974 0.01761 0.01761 0.01761 2.03% FabArray::ParallelCopy_nowait() 851 0.01613 0.01613 0.01613 1.86% StateDataPhysBCFunct::() 41 0.01574 0.01574 0.01574 1.81% Castro::computeTemp() 63 0.01559 0.01559 0.01559 1.79% MultiFab::LinComb() 1566 0.01257 0.01257 0.01257 1.45% Gravity::fill_multipole_BCs() 11 0.01184 0.01184 0.01184 1.36% MLPoisson::Fapply() 1128 0.01079 0.01079 0.01079 1.24% Castro::enforce_min_density() 62 0.01064 0.01064 0.01064 1.22% MLCellLinOp::defineAuxData() 11 0.01044 0.01044 0.01044 1.20% MLMG::addInterpCorrection() 405 0.007135 0.007135 0.007135 0.82% amrex::average_down 405 0.006567 0.006567 0.006567 0.76% MultiFab::Xpay() 578 0.00614 0.00614 0.00614 0.71% Castro::estTimeStep() 21 0.006008 0.006008 0.006008 0.69% FabArray::setDomainBndry() 41 0.005985 0.005985 0.005985 0.69% Castro::reset_internal_energy() 63 0.005421 0.005421 0.005421 0.62% BndryData::define() 11 0.003784 0.003784 0.003784 0.44% Castro::expand_state() 10 0.003174 0.003174 0.003174 0.36% Amr::writePlotFile() 2 0.003069 0.003069 0.003069 0.35% Castro::do_advance_ctu() 10 0.002801 0.002801 0.002801 0.32% Castro::normalize_species() 62 0.002604 0.002604 0.002604 0.30% Gravity::get_new_grav_vector() 11 0.002571 0.002571 0.002571 0.30% Castro::enforce_speed_limit() 62 0.002419 0.002419 0.002419 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002368 0.002368 0.002368 0.27% Castro::construct_new_gravity_source() 10 0.002258 0.002258 0.002258 0.26% MLCellLinOp::compGrad() 11 0.001974 0.001974 0.001974 0.23% MLMG::ResNormInf() 92 0.001909 0.001909 0.001909 0.22% MultiFab::Saxpy() 20 0.00182 0.00182 0.00182 0.21% Gravity::get_old_grav_vector() 10 0.001775 0.001775 0.001775 0.20% MLMG::oneIter() 81 0.001635 0.001635 0.001635 0.19% Castro::construct_old_gravity_source() 10 0.001624 0.001624 0.001624 0.19% MLCellLinOp::setLevelBC() 11 0.001594 0.001594 0.001594 0.18% Gravity::actual_solve_with_mlmg() 11 0.001435 0.001435 0.001435 0.17% MLCellLinOp::prepareForSolve() 11 0.001372 0.001372 0.001372 0.16% FabArray::mult() 43 0.001332 0.001332 0.001332 0.15% Castro::initData() 1 0.001238 0.001238 0.001238 0.14% MLCellLinOp::smooth() 1620 0.001112 0.001112 0.001112 0.13% FabArrayBase::getCPC() 1313 0.0008483 0.0008483 0.0008483 0.10% FabArray::FillBoundary() 3974 0.0008172 0.0008172 0.0008172 0.09% MLMG::prepareForSolve() 11 0.0007797 0.0007797 0.0007797 0.09% FabArrayBase::CPC::define() 454 0.0007314 0.0007314 0.0007314 0.08% FabArrayBase::getFB() 3974 0.000709 0.000709 0.000709 0.08% Gravity::update_max_rhs() 11 0.0005559 0.0005559 0.0005559 0.06% MultiFab::Copy() 11 0.0005247 0.0005247 0.0005247 0.06% CGSolver::sxay() 1566 0.0005108 0.0005108 0.0005108 0.06% MLCellLinOp::apply() 1128 0.0004985 0.0004985 0.0004985 0.06% Amr::InitAmr() 1 0.0004604 0.0004604 0.0004604 0.05% Gravity::solve_for_phi() 10 0.0004445 0.0004445 0.0004445 0.05% MLMG::mgVcycle() 81 0.000369 0.000369 0.000369 0.04% MultiFab::min() 10 0.0003461 0.0003461 0.0003461 0.04% MLCGSolver::ParallelAllReduce 1495 0.0003211 0.0003211 0.0003211 0.04% FabArray::ParallelCopy() 851 0.000305 0.000305 0.000305 0.04% main() 1 0.0002995 0.0002995 0.0002995 0.03% FillPatchIterator::Initialize 41 0.0002611 0.0002611 0.0002611 0.03% MultiFab::max() 11 0.0002583 0.0002583 0.0002583 0.03% Castro::construct_new_gravity() 10 0.000237 0.000237 0.000237 0.03% Gravity::actual_multilevel_solve() 1 0.000229 0.000229 0.000229 0.03% MLCellLinOp::correctionResidual() 486 0.0002166 0.0002166 0.0002166 0.02% Amr::coarseTimeStep() 10 0.0002086 0.0002086 0.0002086 0.02% MLMG::MLRhsNormInf() 11 0.0002018 0.0002018 0.0002018 0.02% MLCellLinOp::defineBC() 11 0.0001962 0.0001962 0.0001962 0.02% Amr::timeStep() 10 0.0001841 0.0001841 0.0001841 0.02% MLLinOp::defineGrids() 11 0.0001704 0.0001704 0.0001704 0.02% StateData::checkPoint() 12 0.0001532 0.0001532 0.0001532 0.02% Castro::subcycle_advance_ctu() 10 0.0001496 0.0001496 0.0001496 0.02% Amr::defBaseLevel() 1 0.0001494 0.0001494 0.0001494 0.02% MLMG:computeResOfCorrection() 405 0.000138 0.000138 0.000138 0.02% AmrLevel::checkPoint() 3 0.0001112 0.0001112 0.0001112 0.01% MLMG::actualBottomSolve() 81 0.0001012 0.0001012 0.0001012 0.01% FabArrayBase::FB::FB() 56 9.234e-05 9.234e-05 9.234e-05 0.01% MLMG::mgVcycle_down::0 81 9.131e-05 9.131e-05 9.131e-05 0.01% MLMG::mgVcycle_down::3 81 8.458e-05 8.458e-05 8.458e-05 0.01% Castro::initialize_advance() 10 8.081e-05 8.081e-05 8.081e-05 0.01% MLMG::mgVcycle_down::1 81 8.068e-05 8.068e-05 8.068e-05 0.01% MLMG::mgVcycle_down::2 81 8.009e-05 8.009e-05 8.009e-05 0.01% MLMG::mgVcycle_down::4 81 7.768e-05 7.768e-05 7.768e-05 0.01% Castro::clean_state() 62 6.853e-05 6.853e-05 6.853e-05 0.01% MLMG::solve() 11 6.72e-05 6.72e-05 6.72e-05 0.01% Castro::advance() 10 6.143e-05 6.143e-05 6.143e-05 0.01% MLMG::mgVcycle_up::4 81 6.099e-05 6.099e-05 6.099e-05 0.01% Castro::initialize_do_advance() 10 5.756e-05 5.756e-05 5.756e-05 0.01% Castro::finalize_advance() 10 5.618e-05 5.618e-05 5.618e-05 0.01% Castro::post_timestep() 10 5.451e-05 5.451e-05 5.451e-05 0.01% MLMG::mgVcycle_up::0 81 5.112e-05 5.112e-05 5.112e-05 0.01% MLCellLinOp::solutionResidual() 92 4.983e-05 4.983e-05 4.983e-05 0.01% MLMG::mgVcycle_up::1 81 4.894e-05 4.894e-05 4.894e-05 0.01% MLMG::mgVcycle_up::3 81 4.845e-05 4.845e-05 4.845e-05 0.01% MLMG::mgVcycle_up::2 81 4.727e-05 4.727e-05 4.727e-05 0.01% StateData::define() 4 4.303e-05 4.303e-05 4.303e-05 0.00% Castro::swap_state_time_levels() 10 4.242e-05 4.242e-05 4.242e-05 0.00% MLMG::computeResidual() 81 3.622e-05 3.622e-05 3.622e-05 0.00% Castro::construct_new_source() 50 3.599e-05 3.599e-05 3.599e-05 0.00% Castro::finalize_do_advance() 10 3.583e-05 3.583e-05 3.583e-05 0.00% Castro::enforce_consistent_e() 1 3.384e-05 3.384e-05 3.384e-05 0.00% MLMG::mgVcycle_bottom 81 3.164e-05 3.164e-05 3.164e-05 0.00% MLLinOp::define() 11 2.913e-05 2.913e-05 2.913e-05 0.00% FillPatchSingleLevel 41 2.652e-05 2.652e-05 2.652e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.625e-05 2.625e-05 2.625e-05 0.00% makeSFC 55 2.517e-05 2.517e-05 2.517e-05 0.00% Amr::writeSmallPlotFile() 1 2.254e-05 2.254e-05 2.254e-05 0.00% MLPoisson::define() 11 2.188e-05 2.188e-05 2.188e-05 0.00% Amr::FinalizeInit() 1 2.037e-05 2.037e-05 2.037e-05 0.00% Castro::do_new_sources() 10 1.701e-05 1.701e-05 1.701e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.681e-05 1.681e-05 1.681e-05 0.00% Castro::construct_old_source() 50 1.673e-05 1.673e-05 1.673e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.57e-05 1.57e-05 1.57e-05 0.00% Castro::do_old_sources() 10 1.53e-05 1.53e-05 1.53e-05 0.00% DistributionMapping::Distribute() 56 1.416e-05 1.416e-05 1.416e-05 0.00% Castro::apply_source_to_state() 20 1.198e-05 1.198e-05 1.198e-05 0.00% Amr::initSubcycle() 1 1.022e-05 1.022e-05 1.022e-05 0.00% Castro::construct_old_gravity() 10 1.019e-05 1.019e-05 1.019e-05 0.00% Gravity::swapTimeLevels() 10 9.629e-06 9.629e-06 9.629e-06 0.00% AmrLevel::AmrLevel(dm) 1 9.487e-06 9.487e-06 9.487e-06 0.00% MLPoisson::prepareForSolve() 11 7.787e-06 7.787e-06 7.787e-06 0.00% MLMG::computeMLResidual() 11 6.942e-06 6.942e-06 6.942e-06 0.00% MLMG::getGradSolution() 11 6.445e-06 6.445e-06 6.445e-06 0.00% Amr::InitializeInit() 1 6.034e-06 6.034e-06 6.034e-06 0.00% Castro::computeNewDt() 9 5.992e-06 5.992e-06 5.992e-06 0.00% Castro::create_source_corrector() 10 5.826e-06 5.826e-06 5.826e-06 0.00% MLMG::buildFineMask() 11 5.455e-06 5.455e-06 5.455e-06 0.00% Castro::post_init() 1 4.87e-06 4.87e-06 4.87e-06 0.00% AmrLevel::checkPointPost() 3 4.662e-06 4.662e-06 4.662e-06 0.00% MLMG::MLResNormInf() 11 4.561e-06 4.561e-06 4.561e-06 0.00% AmrLevel::checkPointPre() 3 4.353e-06 4.353e-06 4.353e-06 0.00% Gravity::set_mass_offset() 11 3.963e-06 3.963e-06 3.963e-06 0.00% Castro::retry_advance_ctu() 10 3.59e-06 3.59e-06 3.59e-06 0.00% Castro::FluxRegFineAdd() 10 3.204e-06 3.204e-06 3.204e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.003e-06 3.003e-06 3.003e-06 0.00% Castro::computeInitialDt() 2 2.211e-06 2.211e-06 2.211e-06 0.00% Amr::init() 1 2.183e-06 2.183e-06 2.183e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.016e-06 2.016e-06 2.016e-06 0.00% Castro::post_regrid() 1 1.412e-06 1.412e-06 1.412e-06 0.00% Amr::initialInit() 1 1.072e-06 1.072e-06 1.072e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8696 0.8696 0.8696 100.00% Amr::coarseTimeStep() 10 0.7216 0.7216 0.7216 82.98% Amr::timeStep() 10 0.6162 0.6162 0.6162 70.86% Castro::advance() 10 0.6092 0.6092 0.6092 70.06% Castro::subcycle_advance_ctu() 10 0.595 0.595 0.595 68.42% Castro::do_advance_ctu() 10 0.5948 0.5948 0.5948 68.40% Gravity::solve_phi_with_mlmg() 11 0.3283 0.3283 0.3283 37.76% Gravity::actual_solve_with_mlmg() 11 0.3163 0.3163 0.3163 36.37% Castro::construct_new_gravity() 10 0.3012 0.3012 0.3012 34.64% MLMG::solve() 11 0.2929 0.2929 0.2929 33.69% Gravity::solve_for_phi() 10 0.2831 0.2831 0.2831 32.56% MLMG::oneIter() 81 0.277 0.277 0.277 31.85% MLMG::mgVcycle() 81 0.2754 0.2754 0.2754 31.67% Castro::construct_ctu_hydro_source() 10 0.2181 0.2181 0.2181 25.08% VisMF::Write(FabArray) 11 0.1593 0.1593 0.1593 18.32% MLCellLinOp::smooth() 1620 0.1453 0.1453 0.1453 16.71% Amr::checkPoint() 3 0.1426 0.1426 0.1426 16.40% Amr::init() 1 0.1231 0.1231 0.1231 14.15% AmrLevel::checkPoint() 3 0.1139 0.1139 0.1139 13.10% StateData::checkPoint() 12 0.1138 0.1138 0.1138 13.08% MLCellLinOp::applyBC() 4379 0.1134 0.1134 0.1134 13.04% MLMG::mgVcycle_bottom 81 0.07897 0.07897 0.07897 9.08% MLMG::actualBottomSolve() 81 0.07894 0.07894 0.07894 9.08% MLCGSolver::bicgstab 81 0.07819 0.07819 0.07819 8.99% MLPoisson::Fsmooth() 3240 0.06003 0.06003 0.06003 6.90% Amr::initialInit() 1 0.05784 0.05784 0.05784 6.65% FillPatchIterator::Initialize 41 0.05044 0.05044 0.05044 5.80% Amr::FinalizeInit() 1 0.04911 0.04911 0.04911 5.65% Amr::writePlotFile() 2 0.04876 0.04876 0.04876 5.61% Castro::post_init() 1 0.0484 0.0484 0.0484 5.57% Gravity::multilevel_solve_for_new_phi() 1 0.0459 0.0459 0.0459 5.28% Gravity::actual_multilevel_solve() 1 0.04588 0.04588 0.04588 5.28% FillPatchSingleLevel 41 0.04419 0.04419 0.04419 5.08% StateDataPhysBCFunct::() 41 0.03998 0.03998 0.03998 4.60% MLCellLinOp::apply() 1128 0.03995 0.03995 0.03995 4.59% MLMG::mgVcycle_down::0 81 0.03806 0.03806 0.03806 4.38% Castro::clean_state() 62 0.03601 0.03601 0.03601 4.14% MLMG::mgVcycle_up::0 81 0.03271 0.03271 0.03271 3.76% FabArray::setVal() 1135 0.0253 0.0253 0.0253 2.91% StateData::FillBoundary(geom) 328 0.02424 0.02424 0.02424 2.79% Castro::initialize_do_advance() 10 0.02288 0.02288 0.02288 2.63% MLCellLinOp::correctionResidual() 486 0.02251 0.02251 0.02251 2.59% MultiFab::Dot() 1100 0.02188 0.02188 0.02188 2.52% Castro::computeTemp() 63 0.02101 0.02101 0.02101 2.42% Gravity::get_new_grav_vector() 11 0.02018 0.02018 0.02018 2.32% MLMG:computeResOfCorrection() 405 0.01945 0.01945 0.01945 2.24% FabArray::FillBoundary() 3974 0.01923 0.01923 0.01923 2.21% MLMG::mgVcycle_down::1 81 0.01867 0.01867 0.01867 2.15% FillBoundary_nowait() 3974 0.01841 0.01841 0.01841 2.12% MLMG::mgVcycle_down::2 81 0.01814 0.01814 0.01814 2.09% MLPoisson::define() 11 0.01776 0.01776 0.01776 2.04% FabArray::ParallelCopy() 851 0.01733 0.01733 0.01733 1.99% MLMG::mgVcycle_down::3 81 0.01731 0.01731 0.01731 1.99% Castro::expand_state() 10 0.01724 0.01724 0.01724 1.98% FabArray::ParallelCopy_nowait() 851 0.01702 0.01702 0.01702 1.96% MLMG::mgVcycle_down::4 81 0.01654 0.01654 0.01654 1.90% Castro::construct_old_gravity() 10 0.01615 0.01615 0.01615 1.86% Gravity::get_old_grav_vector() 10 0.01614 0.01614 0.01614 1.86% Castro::initialize_advance() 10 0.01415 0.01415 0.01415 1.63% MLMG::mgVcycle_up::2 81 0.01412 0.01412 0.01412 1.62% MLMG::mgVcycle_up::1 81 0.01388 0.01388 0.01388 1.60% MLMG::addInterpCorrection() 405 0.01371 0.01371 0.01371 1.58% MLMG::mgVcycle_up::3 81 0.01341 0.01341 0.01341 1.54% MLMG::mgVcycle_up::4 81 0.01319 0.01319 0.01319 1.52% MLCGSolver::ParallelAllReduce 1495 0.01317 0.01317 0.01317 1.51% amrex::average_down 405 0.01314 0.01314 0.01314 1.51% CGSolver::sxay() 1566 0.01308 0.01308 0.01308 1.50% MultiFab::LinComb() 1566 0.01257 0.01257 0.01257 1.45% MLCellLinOp::defineAuxData() 11 0.01218 0.01218 0.01218 1.40% Gravity::fill_multipole_BCs() 11 0.01184 0.01184 0.01184 1.36% MLPoisson::Fapply() 1128 0.01079 0.01079 0.01079 1.24% Castro::enforce_min_density() 62 0.01064 0.01064 0.01064 1.22% Castro::do_new_sources() 10 0.0106 0.0106 0.0106 1.22% Amr::InitializeInit() 1 0.008728 0.008728 0.008728 1.00% Amr::defBaseLevel() 1 0.008722 0.008722 0.008722 1.00% MLCellLinOp::solutionResidual() 92 0.007791 0.007791 0.007791 0.90% Castro::do_old_sources() 10 0.007162 0.007162 0.007162 0.82% Castro::post_timestep() 10 0.006791 0.006791 0.006791 0.78% MLMG::computeResidual() 81 0.006722 0.006722 0.006722 0.77% MultiFab::Xpay() 578 0.00614 0.00614 0.00614 0.71% Castro::estTimeStep() 21 0.006008 0.006008 0.006008 0.69% FabArray::setDomainBndry() 41 0.005985 0.005985 0.005985 0.69% MLMG::prepareForSolve() 11 0.005919 0.005919 0.005919 0.68% Castro::reset_internal_energy() 63 0.005421 0.005421 0.005421 0.62% MLCellLinOp::defineBC() 11 0.005304 0.005304 0.005304 0.61% BndryData::define() 11 0.005108 0.005108 0.005108 0.59% Castro::initData() 1 0.00319 0.00319 0.00319 0.37% Castro::computeNewDt() 9 0.002658 0.002658 0.002658 0.31% Castro::normalize_species() 62 0.002604 0.002604 0.002604 0.30% MLMG::getGradSolution() 11 0.002534 0.002534 0.002534 0.29% MLCellLinOp::compGrad() 11 0.002528 0.002528 0.002528 0.29% Castro::enforce_speed_limit() 62 0.002419 0.002419 0.002419 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002368 0.002368 0.002368 0.27% Castro::construct_new_source() 50 0.002294 0.002294 0.002294 0.26% Castro::construct_new_gravity_source() 10 0.002258 0.002258 0.002258 0.26% MLMG::ResNormInf() 92 0.001909 0.001909 0.001909 0.22% Castro::apply_source_to_state() 20 0.001832 0.001832 0.001832 0.21% MultiFab::Saxpy() 20 0.00182 0.00182 0.00182 0.21% Castro::construct_old_source() 50 0.00164 0.00164 0.00164 0.19% Castro::construct_old_gravity_source() 10 0.001624 0.001624 0.001624 0.19% MLCellLinOp::setLevelBC() 11 0.001594 0.001594 0.001594 0.18% FabArrayBase::getCPC() 1313 0.00158 0.00158 0.00158 0.18% MLPoisson::prepareForSolve() 11 0.00138 0.00138 0.00138 0.16% MLCellLinOp::prepareForSolve() 11 0.001372 0.001372 0.001372 0.16% FabArray::mult() 43 0.001332 0.001332 0.001332 0.15% MLMG::computeMLResidual() 11 0.001113 0.001113 0.001113 0.13% Gravity::swapTimeLevels() 10 0.001078 0.001078 0.001078 0.12% Gravity::update_max_rhs() 11 0.0009598 0.0009598 0.0009598 0.11% FabArrayBase::getFB() 3974 0.0008013 0.0008013 0.0008013 0.09% FabArrayBase::CPC::define() 454 0.0007314 0.0007314 0.0007314 0.08% Castro::computeInitialDt() 2 0.0006652 0.0006652 0.0006652 0.08% MultiFab::Copy() 11 0.0005247 0.0005247 0.0005247 0.06% Castro::post_regrid() 1 0.0004844 0.0004844 0.0004844 0.06% Amr::InitAmr() 1 0.0004706 0.0004706 0.0004706 0.05% MultiFab::min() 10 0.0003461 0.0003461 0.0003461 0.04% MultiFab::max() 11 0.0002583 0.0002583 0.0002583 0.03% MLMG::MLResNormInf() 11 0.0002573 0.0002573 0.0002573 0.03% MLLinOp::define() 11 0.0002557 0.0002557 0.0002557 0.03% MLLinOp::defineGrids() 11 0.0002266 0.0002266 0.0002266 0.03% MLMG::MLRhsNormInf() 11 0.0002018 0.0002018 0.0002018 0.02% FabArrayBase::FB::FB() 56 9.234e-05 9.234e-05 9.234e-05 0.01% Castro::finalize_advance() 10 5.938e-05 5.938e-05 5.938e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.416e-05 5.416e-05 5.416e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.252e-05 5.252e-05 5.252e-05 0.01% StateData::define() 4 4.303e-05 4.303e-05 4.303e-05 0.00% Castro::swap_state_time_levels() 10 4.242e-05 4.242e-05 4.242e-05 0.00% makeSFC 55 3.846e-05 3.846e-05 3.846e-05 0.00% Castro::finalize_do_advance() 10 3.583e-05 3.583e-05 3.583e-05 0.00% Castro::enforce_consistent_e() 1 3.384e-05 3.384e-05 3.384e-05 0.00% Amr::writeSmallPlotFile() 1 2.254e-05 2.254e-05 2.254e-05 0.00% DistributionMapping::Distribute() 56 1.416e-05 1.416e-05 1.416e-05 0.00% Amr::initSubcycle() 1 1.022e-05 1.022e-05 1.022e-05 0.00% Castro::create_source_corrector() 10 5.826e-06 5.826e-06 5.826e-06 0.00% MLMG::buildFineMask() 11 5.455e-06 5.455e-06 5.455e-06 0.00% AmrLevel::checkPointPost() 3 4.662e-06 4.662e-06 4.662e-06 0.00% AmrLevel::checkPointPre() 3 4.353e-06 4.353e-06 4.353e-06 0.00% Gravity::set_mass_offset() 11 3.963e-06 3.963e-06 3.963e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.871e-06 3.871e-06 3.871e-06 0.00% Castro::retry_advance_ctu() 10 3.59e-06 3.59e-06 3.59e-06 0.00% Castro::FluxRegFineAdd() 10 3.204e-06 3.204e-06 3.204e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.016e-06 2.016e-06 2.016e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.02-10-g76d08651adb9) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.02-10-g76d08651adb9) initialized Starting run at 08:53:56 UTC on 2022-02-16. Successfully read inputs file ... Castro git describe: 22.02-4-g0d942c92d AMReX git describe: 22.02-10-g76d08651a Microphysics git describe: 22.02-2-gf51cc451 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.496141453 Restart time = 0.054236692 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.114479558 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049168586 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.049503869 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.058739174 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.066030157 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024404337 seconds Ending run at 08:53:57 UTC on 2022-02-16. Run time = 0.417473101 Run time without initialization = 0.362662129 Average number of zones advanced per microsecond: 3.614 Average number of zones advanced per microsecond per rank: 3.614 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4175 ... 0.4175 ... 0.4175 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1252 0.1252 0.1252 29.98% VisMF::Read() 3 0.04255 0.04255 0.04255 10.19% MLCellLinOp::applyBC() 1946 0.03833 0.03833 0.03833 9.18% MLPoisson::Fsmooth() 1440 0.02599 0.02599 0.02599 6.23% VisMF::Write(FabArray) 1 0.0231 0.0231 0.0231 5.53% FabArray::setVal() 537 0.01666 0.01666 0.01666 3.99% StateData::FillBoundary(geom) 160 0.01135 0.01135 0.01135 2.72% MLCGSolver::bicgstab 36 0.009538 0.009538 0.009538 2.28% MultiFab::Dot() 484 0.009485 0.009485 0.009485 2.27% StateDataPhysBCFunct::() 20 0.008049 0.008049 0.008049 1.93% Castro::computeTemp() 30 0.007831 0.007831 0.007831 1.88% FillBoundary_nowait() 1766 0.007301 0.007301 0.007301 1.75% FabArray::ParallelCopy_nowait() 380 0.007184 0.007184 0.007184 1.72% Gravity::fill_multipole_BCs() 6 0.0062 0.0062 0.0062 1.48% MLCellLinOp::defineAuxData() 6 0.005652 0.005652 0.005652 1.35% MultiFab::LinComb() 690 0.005482 0.005482 0.005482 1.31% Castro::enforce_min_density() 30 0.005392 0.005392 0.005392 1.29% MLPoisson::Fapply() 500 0.004755 0.004755 0.004755 1.14% Castro::expand_state() 5 0.003953 0.003953 0.003953 0.95% FabArray::setDomainBndry() 20 0.003889 0.003889 0.003889 0.93% MLMG::addInterpCorrection() 180 0.00307 0.00307 0.00307 0.74% Castro::estTimeStep() 10 0.002998 0.002998 0.002998 0.72% Amr::restart() 1 0.002945 0.002945 0.002945 0.71% amrex::average_down 180 0.002846 0.002846 0.002846 0.68% MultiFab::Xpay() 258 0.002707 0.002707 0.002707 0.65% Castro::do_advance_ctu() 5 0.002576 0.002576 0.002576 0.62% Castro::reset_internal_energy() 30 0.002419 0.002419 0.002419 0.58% BndryData::define() 6 0.002044 0.002044 0.002044 0.49% Gravity::get_new_grav_vector() 5 0.001884 0.001884 0.001884 0.45% Castro::normalize_species() 30 0.001612 0.001612 0.001612 0.39% Castro::construct_new_gravity_source() 5 0.001583 0.001583 0.001583 0.38% MLCellLinOp::compGrad() 6 0.001478 0.001478 0.001478 0.35% Amr::writePlotFile() 1 0.001389 0.001389 0.001389 0.33% Castro::construct_old_gravity_source() 5 0.001356 0.001356 0.001356 0.32% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001236 0.001236 0.001236 0.30% Castro::enforce_speed_limit() 30 0.001122 0.001122 0.001122 0.27% Gravity::get_old_grav_vector() 5 0.001019 0.001019 0.001019 0.24% MultiFab::Saxpy() 10 0.0009258 0.0009258 0.0009258 0.22% Castro::construct_new_source() 25 0.0008927 0.0008927 0.0008927 0.21% MLMG::ResNormInf() 42 0.000847 0.000847 0.000847 0.20% MLCellLinOp::setLevelBC() 6 0.0008425 0.0008425 0.0008425 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007744 0.0007744 0.0007744 0.19% MLMG::oneIter() 36 0.000737 0.000737 0.000737 0.18% MLCellLinOp::prepareForSolve() 6 0.0007075 0.0007075 0.0007075 0.17% Castro::subcycle_advance_ctu() 5 0.0006765 0.0006765 0.0006765 0.16% FabArray::mult() 22 0.0006601 0.0006601 0.0006601 0.16% Gravity::update_max_rhs() 6 0.0005154 0.0005154 0.0005154 0.12% MLCellLinOp::smooth() 720 0.0004965 0.0004965 0.0004965 0.12% FabArrayBase::CPC::define() 244 0.000456 0.000456 0.000456 0.11% FabArrayBase::getCPC() 632 0.0004355 0.0004355 0.0004355 0.10% MLMG::prepareForSolve() 6 0.0004348 0.0004348 0.0004348 0.10% FabArray::FillBoundary() 1766 0.0004051 0.0004051 0.0004051 0.10% Amr::InitAmr() 1 0.0003839 0.0003839 0.0003839 0.09% FabArrayBase::getFB() 1766 0.0003685 0.0003685 0.0003685 0.09% MultiFab::Copy() 6 0.0003555 0.0003555 0.0003555 0.09% main() 1 0.0002697 0.0002697 0.0002697 0.06% Castro::construct_new_gravity() 5 0.0002563 0.0002563 0.0002563 0.06% Gravity::solve_for_phi() 5 0.0002431 0.0002431 0.0002431 0.06% CGSolver::sxay() 690 0.0002188 0.0002188 0.0002188 0.05% MLCellLinOp::apply() 500 0.0002172 0.0002172 0.0002172 0.05% Gravity::actual_multilevel_solve() 1 0.0002147 0.0002147 0.0002147 0.05% MLMG::mgVcycle() 36 0.0001801 0.0001801 0.0001801 0.04% MultiFab::min() 5 0.0001621 0.0001621 0.0001621 0.04% MLCGSolver::ParallelAllReduce 659 0.0001536 0.0001536 0.0001536 0.04% FabArray::ParallelCopy() 380 0.0001457 0.0001457 0.0001457 0.03% MultiFab::max() 6 0.0001337 0.0001337 0.0001337 0.03% FillPatchIterator::Initialize 20 0.0001231 0.0001231 0.0001231 0.03% MLMG::MLRhsNormInf() 6 0.0001088 0.0001088 0.0001088 0.03% Amr::coarseTimeStep() 5 0.0001084 0.0001084 0.0001084 0.03% MLCellLinOp::defineBC() 6 0.0001062 0.0001062 0.0001062 0.03% Castro::advance() 5 9.818e-05 9.818e-05 9.818e-05 0.02% Amr::timeStep() 5 9.662e-05 9.662e-05 9.662e-05 0.02% MLCellLinOp::correctionResidual() 216 9.621e-05 9.621e-05 9.621e-05 0.02% MLLinOp::defineGrids() 6 9.255e-05 9.255e-05 9.255e-05 0.02% Castro::initialize_advance() 5 8.131e-05 8.131e-05 8.131e-05 0.02% Castro::post_timestep() 5 7.84e-05 7.84e-05 7.84e-05 0.02% AmrLevel::restart() 1 7.551e-05 7.551e-05 7.551e-05 0.02% StateData::restartDoit() 4 7.243e-05 7.243e-05 7.243e-05 0.02% MLMG:computeResOfCorrection() 180 6.891e-05 6.891e-05 6.891e-05 0.02% FabArrayBase::FB::FB() 26 6.41e-05 6.41e-05 6.41e-05 0.02% Castro::construct_old_source() 25 5.425e-05 5.425e-05 5.425e-05 0.01% Castro::create_source_corrector() 5 5.373e-05 5.373e-05 5.373e-05 0.01% MLMG::actualBottomSolve() 36 4.733e-05 4.733e-05 4.733e-05 0.01% Castro::initialize_do_advance() 5 4.226e-05 4.226e-05 4.226e-05 0.01% MLMG::mgVcycle_down::0 36 4.032e-05 4.032e-05 4.032e-05 0.01% MLMG::mgVcycle_down::1 36 3.741e-05 3.741e-05 3.741e-05 0.01% Castro::clean_state() 30 3.581e-05 3.581e-05 3.581e-05 0.01% MLMG::mgVcycle_down::2 36 3.461e-05 3.461e-05 3.461e-05 0.01% MLMG::mgVcycle_down::4 36 3.418e-05 3.418e-05 3.418e-05 0.01% MLMG::solve() 6 3.335e-05 3.335e-05 3.335e-05 0.01% MLMG::mgVcycle_down::3 36 3.314e-05 3.314e-05 3.314e-05 0.01% MLLinOp::define() 6 2.772e-05 2.772e-05 2.772e-05 0.01% Castro::finalize_advance() 5 2.712e-05 2.712e-05 2.712e-05 0.01% Castro::post_restart() 1 2.695e-05 2.695e-05 2.695e-05 0.01% Castro::swap_state_time_levels() 5 2.641e-05 2.641e-05 2.641e-05 0.01% Castro::construct_old_gravity() 5 2.612e-05 2.612e-05 2.612e-05 0.01% MLMG::mgVcycle_up::4 36 2.531e-05 2.531e-05 2.531e-05 0.01% MLCellLinOp::solutionResidual() 42 2.429e-05 2.429e-05 2.429e-05 0.01% MLMG::mgVcycle_up::0 36 2.322e-05 2.322e-05 2.322e-05 0.01% Amr::writeSmallPlotFile() 1 2.317e-05 2.317e-05 2.317e-05 0.01% MLMG::mgVcycle_up::3 36 2.198e-05 2.198e-05 2.198e-05 0.01% MLMG::mgVcycle_up::2 36 2.19e-05 2.19e-05 2.19e-05 0.01% MLMG::mgVcycle_up::1 36 2.054e-05 2.054e-05 2.054e-05 0.00% Castro::finalize_do_advance() 5 2.044e-05 2.044e-05 2.044e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.618e-05 1.618e-05 1.618e-05 0.00% MLMG::computeResidual() 36 1.615e-05 1.615e-05 1.615e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.61e-05 1.61e-05 1.61e-05 0.00% MLMG::mgVcycle_bottom 36 1.484e-05 1.484e-05 1.484e-05 0.00% MLPoisson::define() 6 1.471e-05 1.471e-05 1.471e-05 0.00% makeSFC 30 1.436e-05 1.436e-05 1.436e-05 0.00% FillPatchSingleLevel 20 1.345e-05 1.345e-05 1.345e-05 0.00% Castro::do_new_sources() 5 9.685e-06 9.685e-06 9.685e-06 0.00% Amr::initSubcycle() 1 9.214e-06 9.214e-06 9.214e-06 0.00% Castro::do_old_sources() 5 8.47e-06 8.47e-06 8.47e-06 0.00% DistributionMapping::Distribute() 31 8.372e-06 8.372e-06 8.372e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.585e-06 7.585e-06 7.585e-06 0.00% Castro::apply_source_to_state() 10 5.668e-06 5.668e-06 5.668e-06 0.00% Gravity::swapTimeLevels() 5 4.926e-06 4.926e-06 4.926e-06 0.00% MLPoisson::prepareForSolve() 6 4.552e-06 4.552e-06 4.552e-06 0.00% Castro::computeNewDt() 5 3.609e-06 3.609e-06 3.609e-06 0.00% MLMG::buildFineMask() 6 3.594e-06 3.594e-06 3.594e-06 0.00% MLMG::computeMLResidual() 6 3.363e-06 3.363e-06 3.363e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.301e-06 3.301e-06 3.301e-06 0.00% MLMG::getGradSolution() 6 3e-06 3e-06 3e-06 0.00% Gravity::set_mass_offset() 6 2.519e-06 2.519e-06 2.519e-06 0.00% MLMG::MLResNormInf() 6 2.445e-06 2.445e-06 2.445e-06 0.00% Castro::retry_advance_ctu() 5 1.65e-06 1.65e-06 1.65e-06 0.00% Castro::FluxRegFineAdd() 5 1.495e-06 1.495e-06 1.495e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.182e-06 1.182e-06 1.182e-06 0.00% Amr::init() 1 1.049e-06 1.049e-06 1.049e-06 0.00% AmrLevel::AmrLevel() 1 9.51e-07 9.51e-07 9.51e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4175 0.4175 0.4175 100.00% Amr::coarseTimeStep() 5 0.338 0.338 0.338 80.96% Amr::timeStep() 5 0.3366 0.3366 0.3366 80.62% Castro::advance() 5 0.3339 0.3339 0.3339 79.97% Castro::subcycle_advance_ctu() 5 0.322 0.322 0.322 77.13% Castro::do_advance_ctu() 5 0.3214 0.3214 0.3214 76.97% Castro::construct_new_gravity() 5 0.1491 0.1491 0.1491 35.71% Gravity::solve_phi_with_mlmg() 6 0.1449 0.1449 0.1449 34.70% Gravity::solve_for_phi() 5 0.139 0.139 0.139 33.30% Gravity::actual_solve_with_mlmg() 6 0.1385 0.1385 0.1385 33.18% MLMG::solve() 6 0.1255 0.1255 0.1255 30.05% Castro::construct_ctu_hydro_source() 5 0.1252 0.1252 0.1252 29.98% MLMG::oneIter() 36 0.1175 0.1175 0.1175 28.15% MLMG::mgVcycle() 36 0.1168 0.1168 0.1168 27.97% MLCellLinOp::smooth() 720 0.06081 0.06081 0.06081 14.57% Amr::init() 1 0.05429 0.05429 0.05429 13.00% Amr::restart() 1 0.05429 0.05429 0.05429 13.00% MLCellLinOp::applyBC() 1946 0.04647 0.04647 0.04647 11.13% AmrLevel::restart() 1 0.04342 0.04342 0.04342 10.40% StateData::restartDoit() 4 0.04334 0.04334 0.04334 10.38% VisMF::Read() 3 0.04255 0.04255 0.04255 10.19% MLMG::mgVcycle_bottom 36 0.03403 0.03403 0.03403 8.15% MLMG::actualBottomSolve() 36 0.03402 0.03402 0.03402 8.15% MLCGSolver::bicgstab 36 0.03369 0.03369 0.03369 8.07% MLPoisson::Fsmooth() 1440 0.02599 0.02599 0.02599 6.23% FillPatchIterator::Initialize 20 0.02552 0.02552 0.02552 6.11% Amr::writePlotFile() 1 0.02449 0.02449 0.02449 5.87% VisMF::Write(FabArray) 1 0.0231 0.0231 0.0231 5.53% FillPatchSingleLevel 20 0.02151 0.02151 0.02151 5.15% StateDataPhysBCFunct::() 20 0.0194 0.0194 0.0194 4.65% Castro::clean_state() 30 0.01841 0.01841 0.01841 4.41% MLCellLinOp::apply() 500 0.01683 0.01683 0.01683 4.03% FabArray::setVal() 537 0.01666 0.01666 0.01666 3.99% MLMG::mgVcycle_down::0 36 0.01613 0.01613 0.01613 3.86% Castro::initialize_do_advance() 5 0.01477 0.01477 0.01477 3.54% MLMG::mgVcycle_up::0 36 0.01377 0.01377 0.01377 3.30% Castro::expand_state() 5 0.01196 0.01196 0.01196 2.86% Castro::initialize_advance() 5 0.01171 0.01171 0.01171 2.80% StateData::FillBoundary(geom) 160 0.01135 0.01135 0.01135 2.72% Castro::computeTemp() 30 0.01025 0.01025 0.01025 2.46% Gravity::get_new_grav_vector() 5 0.009817 0.009817 0.009817 2.35% MLPoisson::define() 6 0.009661 0.009661 0.009661 2.31% MLCellLinOp::correctionResidual() 216 0.009502 0.009502 0.009502 2.28% MultiFab::Dot() 484 0.009485 0.009485 0.009485 2.27% Castro::do_new_sources() 5 0.008661 0.008661 0.008661 2.07% Castro::construct_old_gravity() 5 0.008364 0.008364 0.008364 2.00% Gravity::get_old_grav_vector() 5 0.008338 0.008338 0.008338 2.00% MLMG:computeResOfCorrection() 180 0.008229 0.008229 0.008229 1.97% FabArray::FillBoundary() 1766 0.008139 0.008139 0.008139 1.95% MLMG::mgVcycle_down::1 36 0.007884 0.007884 0.007884 1.89% FabArray::ParallelCopy() 380 0.007765 0.007765 0.007765 1.86% FillBoundary_nowait() 1766 0.007734 0.007734 0.007734 1.85% MLMG::mgVcycle_down::2 36 0.007637 0.007637 0.007637 1.83% FabArray::ParallelCopy_nowait() 380 0.007619 0.007619 0.007619 1.82% MLMG::mgVcycle_down::3 36 0.00727 0.00727 0.00727 1.74% MLMG::mgVcycle_down::4 36 0.006972 0.006972 0.006972 1.67% Castro::post_restart() 1 0.006677 0.006677 0.006677 1.60% MLCellLinOp::defineAuxData() 6 0.006615 0.006615 0.006615 1.58% Gravity::multilevel_solve_for_new_phi() 1 0.0063 0.0063 0.0063 1.51% Gravity::actual_multilevel_solve() 1 0.006284 0.006284 0.006284 1.51% Gravity::fill_multipole_BCs() 6 0.0062 0.0062 0.0062 1.48% MLMG::addInterpCorrection() 180 0.005897 0.005897 0.005897 1.41% MLMG::mgVcycle_up::2 36 0.005895 0.005895 0.005895 1.41% MLMG::mgVcycle_up::1 36 0.005848 0.005848 0.005848 1.40% MLCGSolver::ParallelAllReduce 659 0.005752 0.005752 0.005752 1.38% CGSolver::sxay() 690 0.005701 0.005701 0.005701 1.37% amrex::average_down 180 0.005694 0.005694 0.005694 1.36% MLMG::mgVcycle_up::3 36 0.005606 0.005606 0.005606 1.34% MLMG::mgVcycle_up::4 36 0.005553 0.005553 0.005553 1.33% MultiFab::LinComb() 690 0.005482 0.005482 0.005482 1.31% Castro::enforce_min_density() 30 0.005392 0.005392 0.005392 1.29% MLPoisson::Fapply() 500 0.004755 0.004755 0.004755 1.14% Castro::do_old_sources() 5 0.004194 0.004194 0.004194 1.00% FabArray::setDomainBndry() 20 0.003889 0.003889 0.003889 0.93% MLMG::prepareForSolve() 6 0.003504 0.003504 0.003504 0.84% MLCellLinOp::solutionResidual() 42 0.003441 0.003441 0.003441 0.82% Castro::estTimeStep() 10 0.002998 0.002998 0.002998 0.72% MLCellLinOp::defineBC() 6 0.002881 0.002881 0.002881 0.69% MLMG::computeResidual() 36 0.002856 0.002856 0.002856 0.68% BndryData::define() 6 0.002774 0.002774 0.002774 0.66% MultiFab::Xpay() 258 0.002707 0.002707 0.002707 0.65% Castro::post_timestep() 5 0.002642 0.002642 0.002642 0.63% Castro::construct_new_source() 25 0.002475 0.002475 0.002475 0.59% Castro::reset_internal_energy() 30 0.002419 0.002419 0.002419 0.58% MLMG::getGradSolution() 6 0.001771 0.001771 0.001771 0.42% MLCellLinOp::compGrad() 6 0.001768 0.001768 0.001768 0.42% Castro::normalize_species() 30 0.001612 0.001612 0.001612 0.39% Castro::construct_new_gravity_source() 5 0.001583 0.001583 0.001583 0.38% Castro::construct_old_source() 25 0.00141 0.00141 0.00141 0.34% Castro::construct_old_gravity_source() 5 0.001356 0.001356 0.001356 0.32% Castro::computeNewDt() 5 0.001304 0.001304 0.001304 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001236 0.001236 0.001236 0.30% Castro::enforce_speed_limit() 30 0.001122 0.001122 0.001122 0.27% Castro::apply_source_to_state() 10 0.0009314 0.0009314 0.0009314 0.22% MultiFab::Saxpy() 10 0.0009258 0.0009258 0.0009258 0.22% FabArrayBase::getCPC() 632 0.0008915 0.0008915 0.0008915 0.21% Gravity::swapTimeLevels() 5 0.0008845 0.0008845 0.0008845 0.21% MLMG::ResNormInf() 42 0.000847 0.000847 0.000847 0.20% MLCellLinOp::setLevelBC() 6 0.0008425 0.0008425 0.0008425 0.20% Gravity::update_max_rhs() 6 0.0007285 0.0007285 0.0007285 0.17% MLPoisson::prepareForSolve() 6 0.000712 0.000712 0.000712 0.17% MLCellLinOp::prepareForSolve() 6 0.0007075 0.0007075 0.0007075 0.17% FabArray::mult() 22 0.0006601 0.0006601 0.0006601 0.16% MLMG::computeMLResidual() 6 0.0006047 0.0006047 0.0006047 0.14% FabArrayBase::CPC::define() 244 0.000456 0.000456 0.000456 0.11% FabArrayBase::getFB() 1766 0.0004326 0.0004326 0.0004326 0.10% Amr::InitAmr() 1 0.0003931 0.0003931 0.0003931 0.09% MultiFab::Copy() 6 0.0003555 0.0003555 0.0003555 0.09% MultiFab::min() 5 0.0001621 0.0001621 0.0001621 0.04% MLLinOp::define() 6 0.0001507 0.0001507 0.0001507 0.04% MLMG::MLResNormInf() 6 0.0001354 0.0001354 0.0001354 0.03% MultiFab::max() 6 0.0001337 0.0001337 0.0001337 0.03% MLLinOp::defineGrids() 6 0.000123 0.000123 0.000123 0.03% MLMG::MLRhsNormInf() 6 0.0001088 0.0001088 0.0001088 0.03% FabArrayBase::FB::FB() 26 6.41e-05 6.41e-05 6.41e-05 0.02% Castro::create_source_corrector() 5 5.373e-05 5.373e-05 5.373e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.927e-05 2.927e-05 2.927e-05 0.01% Castro::finalize_advance() 5 2.862e-05 2.862e-05 2.862e-05 0.01% Castro::swap_state_time_levels() 5 2.641e-05 2.641e-05 2.641e-05 0.01% Amr::writeSmallPlotFile() 1 2.317e-05 2.317e-05 2.317e-05 0.01% makeSFC 30 2.169e-05 2.169e-05 2.169e-05 0.01% Castro::finalize_do_advance() 5 2.044e-05 2.044e-05 2.044e-05 0.00% Amr::initSubcycle() 1 9.214e-06 9.214e-06 9.214e-06 0.00% DistributionMapping::Distribute() 31 8.372e-06 8.372e-06 8.372e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.347e-06 4.347e-06 4.347e-06 0.00% MLMG::buildFineMask() 6 3.594e-06 3.594e-06 3.594e-06 0.00% Gravity::set_mass_offset() 6 2.519e-06 2.519e-06 2.519e-06 0.00% Castro::retry_advance_ctu() 5 1.65e-06 1.65e-06 1.65e-06 0.00% Castro::FluxRegFineAdd() 5 1.495e-06 1.495e-06 1.495e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.182e-06 1.182e-06 1.182e-06 0.00% AmrLevel::AmrLevel() 1 9.51e-07 9.51e-07 9.51e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.02-10-g76d08651adb9) finalized