Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.01-18-g405b490356e5) initialized Starting run at 09:05:46 UTC on 2022-01-12. Successfully read inputs file ... Castro git describe: 21.12-23-g727e39bb6 AMReX git describe: 22.01-18-g405b49035 Microphysics git describe: 22.01-14-geeef682d reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.042309857 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025129163 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.138588564 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.073763394 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.072629262 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.085635971 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.086775187 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.039874133 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.074636024 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.07320764 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.082587618 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.088316613 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.091618218 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.042148148 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.025201413 seconds Ending run at 09:05:47 UTC on 2022-01-12. Run time = 1.143161857 Run time without initialization = 0.975615106 Average number of zones advanced per microsecond: 2.687 Average number of zones advanced per microsecond per rank: 2.687 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489334272 TinyProfiler total time across processes [min...avg...max]: 1.143 ... 1.143 ... 1.143 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2383 0.2383 0.2383 20.85% VisMF::Write(FabArray) 11 0.1665 0.1665 0.1665 14.56% MLCellLinOp::applyBC() 4433 0.1178 0.1178 0.1178 10.30% MLPoisson::Fsmooth() 3280 0.07524 0.07524 0.07524 6.58% FillBoundary_nowait() 4023 0.04004 0.04004 0.04004 3.50% MLCGSolver::bicgstab 82 0.03812 0.03812 0.03812 3.33% FabArray::setVal() 1140 0.03053 0.03053 0.03053 2.67% MultiFab::Dot() 1114 0.02806 0.02806 0.02806 2.45% StateData::FillBoundary(geom) 328 0.02534 0.02534 0.02534 2.22% StateDataPhysBCFunct::() 41 0.02528 0.02528 0.02528 2.21% MLCellLinOp::smooth() 1640 0.02397 0.02397 0.02397 2.10% FabArray::FillBoundary() 4023 0.02302 0.02302 0.02302 2.01% FabArray::ParallelCopy_nowait() 861 0.02192 0.02192 0.02192 1.92% MultiFab::LinComb() 1586 0.02018 0.02018 0.02018 1.76% MLPoisson::Fapply() 1142 0.01612 0.01612 0.01612 1.41% Castro::computeTemp() 63 0.01464 0.01464 0.01464 1.28% MLCellLinOp::defineAuxData() 11 0.01404 0.01404 0.01404 1.23% Gravity::fill_multipole_BCs() 11 0.01338 0.01338 0.01338 1.17% FabArrayBase::getFB() 4023 0.01192 0.01192 0.01192 1.04% Castro::enforce_min_density() 62 0.0111 0.0111 0.0111 0.97% MLMG::addInterpCorrection() 410 0.009945 0.009945 0.009945 0.87% MLCellLinOp::apply() 1142 0.009931 0.009931 0.009931 0.87% CGSolver::sxay() 1586 0.009184 0.009184 0.009184 0.80% amrex::average_down 410 0.009137 0.009137 0.009137 0.80% MultiFab::Xpay() 585 0.009016 0.009016 0.009016 0.79% Castro::reset_internal_energy() 63 0.006829 0.006829 0.006829 0.60% MLCGSolver::ParallelAllReduce 1514 0.00623 0.00623 0.00623 0.54% Castro::estTimeStep() 21 0.006154 0.006154 0.006154 0.54% FabArray::setDomainBndry() 41 0.005991 0.005991 0.005991 0.52% FabArrayBase::getCPC() 1323 0.005785 0.005785 0.005785 0.51% Castro::enforce_speed_limit() 62 0.005131 0.005131 0.005131 0.45% FabArray::ParallelCopy() 861 0.005032 0.005032 0.005032 0.44% Amr::checkPoint() 3 0.004731 0.004731 0.004731 0.41% Castro::expand_state() 10 0.004673 0.004673 0.004673 0.41% BndryData::define() 11 0.004538 0.004538 0.004538 0.40% MLCellLinOp::correctionResidual() 492 0.004287 0.004287 0.004287 0.37% Castro::do_advance_ctu() 10 0.003792 0.003792 0.003792 0.33% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.003671 0.003671 0.003671 0.32% Amr::writePlotFile() 2 0.003371 0.003371 0.003371 0.29% Castro::construct_new_gravity_source() 10 0.003181 0.003181 0.003181 0.28% MLMG::mgVcycle() 82 0.003072 0.003072 0.003072 0.27% Gravity::get_new_grav_vector() 11 0.003036 0.003036 0.003036 0.27% Castro::normalize_species() 62 0.002934 0.002934 0.002934 0.26% Castro::construct_old_gravity_source() 10 0.002745 0.002745 0.002745 0.24% MLMG:computeResOfCorrection() 410 0.002387 0.002387 0.002387 0.21% MLMG::ResNormInf() 93 0.00235 0.00235 0.00235 0.21% MLMG::oneIter() 82 0.002294 0.002294 0.002294 0.20% Amr::defBaseLevel() 1 0.002156 0.002156 0.002156 0.19% Gravity::get_old_grav_vector() 10 0.002069 0.002069 0.002069 0.18% MLCellLinOp::compGrad() 11 0.002053 0.002053 0.002053 0.18% FabArrayBase::CPC::define() 454 0.001967 0.001967 0.001967 0.17% MultiFab::Saxpy() 20 0.001922 0.001922 0.001922 0.17% MLCellLinOp::setLevelBC() 11 0.001682 0.001682 0.001682 0.15% MLMG::prepareForSolve() 11 0.001642 0.001642 0.001642 0.14% Gravity::actual_solve_with_mlmg() 11 0.001614 0.001614 0.001614 0.14% FabArray::mult() 43 0.001522 0.001522 0.001522 0.13% MLMG::mgVcycle_down::1 82 0.001458 0.001458 0.001458 0.13% MLMG::mgVcycle_down::2 82 0.001457 0.001457 0.001457 0.13% MLMG::mgVcycle_down::4 82 0.001436 0.001436 0.001436 0.13% MLMG::mgVcycle_down::0 82 0.001433 0.001433 0.001433 0.13% MLMG::mgVcycle_down::3 82 0.001409 0.001409 0.001409 0.12% MLCellLinOp::prepareForSolve() 11 0.001392 0.001392 0.001392 0.12% Castro::initData() 1 0.001376 0.001376 0.001376 0.12% MLMG::actualBottomSolve() 82 0.001254 0.001254 0.001254 0.11% MLMG::solve() 11 0.001023 0.001023 0.001023 0.09% MLMG::mgVcycle_up::4 82 0.0009641 0.0009641 0.0009641 0.08% MLMG::mgVcycle_up::0 82 0.0009557 0.0009557 0.0009557 0.08% MLMG::mgVcycle_up::1 82 0.0009517 0.0009517 0.0009517 0.08% MLMG::mgVcycle_up::2 82 0.000949 0.000949 0.000949 0.08% MLMG::mgVcycle_up::3 82 0.0009371 0.0009371 0.0009371 0.08% Castro::clean_state() 62 0.0009183 0.0009183 0.0009183 0.08% MLCellLinOp::solutionResidual() 93 0.0008055 0.0008055 0.0008055 0.07% Gravity::update_max_rhs() 11 0.0007077 0.0007077 0.0007077 0.06% FillPatchIterator::Initialize 41 0.0005895 0.0005895 0.0005895 0.05% MultiFab::Copy() 11 0.0005736 0.0005736 0.0005736 0.05% Gravity::solve_for_phi() 10 0.0005705 0.0005705 0.0005705 0.05% Amr::InitAmr() 1 0.0004877 0.0004877 0.0004877 0.04% MLMG::mgVcycle_bottom 82 0.0004779 0.0004779 0.0004779 0.04% MLMG::computeResidual() 82 0.0004762 0.0004762 0.0004762 0.04% Castro::subcycle_advance_ctu() 10 0.0004411 0.0004411 0.0004411 0.04% FillPatchSingleLevel 41 0.0004073 0.0004073 0.0004073 0.04% Castro::initialize_advance() 10 0.000398 0.000398 0.000398 0.03% Amr::coarseTimeStep() 10 0.0003967 0.0003967 0.0003967 0.03% MultiFab::min() 10 0.0003753 0.0003753 0.0003753 0.03% Castro::construct_new_gravity() 10 0.000348 0.000348 0.000348 0.03% main() 1 0.0003372 0.0003372 0.0003372 0.03% makeSFC 55 0.0003268 0.0003268 0.0003268 0.03% Castro::do_old_sources() 10 0.0003108 0.0003108 0.0003108 0.03% Castro::advance() 10 0.0003081 0.0003081 0.0003081 0.03% MultiFab::max() 11 0.0002998 0.0002998 0.0002998 0.03% MLLinOp::defineGrids() 11 0.0002962 0.0002962 0.0002962 0.03% Castro::construct_new_source() 50 0.0002948 0.0002948 0.0002948 0.03% Castro::do_new_sources() 10 0.0002908 0.0002908 0.0002908 0.03% Amr::timeStep() 10 0.0002644 0.0002644 0.0002644 0.02% FabArrayBase::FB::FB() 56 0.0002634 0.0002634 0.0002634 0.02% MLCellLinOp::defineBC() 11 0.0002629 0.0002629 0.0002629 0.02% MLMG::MLRhsNormInf() 11 0.0002553 0.0002553 0.0002553 0.02% Gravity::actual_multilevel_solve() 1 0.0002383 0.0002383 0.0002383 0.02% Castro::construct_old_source() 50 0.0002153 0.0002153 0.0002153 0.02% MLLinOp::makeAgglomeratedDMap 11 0.0001947 0.0001947 0.0001947 0.02% StateData::checkPoint() 12 0.0001903 0.0001903 0.0001903 0.02% DistributionMapping::Distribute() 56 0.0001657 0.0001657 0.0001657 0.01% Gravity::solve_phi_with_mlmg() 11 0.0001467 0.0001467 0.0001467 0.01% MLPoisson::define() 11 0.0001462 0.0001462 0.0001462 0.01% Castro::initialize_do_advance() 10 0.0001394 0.0001394 0.0001394 0.01% Castro::finalize_advance() 10 0.000133 0.000133 0.000133 0.01% Castro::apply_source_to_state() 20 0.0001218 0.0001218 0.0001218 0.01% Gravity::swapTimeLevels() 10 0.0001183 0.0001183 0.0001183 0.01% AmrLevel::checkPoint() 3 0.0001139 0.0001139 0.0001139 0.01% Castro::post_timestep() 10 9.525e-05 9.525e-05 9.525e-05 0.01% MLLinOp::define() 11 7.388e-05 7.388e-05 7.388e-05 0.01% Castro::swap_state_time_levels() 10 6.646e-05 6.646e-05 6.646e-05 0.01% MLPoisson::prepareForSolve() 11 6.642e-05 6.642e-05 6.642e-05 0.01% MLMG::computeMLResidual() 11 6.638e-05 6.638e-05 6.638e-05 0.01% Castro::construct_old_gravity() 10 6.579e-05 6.579e-05 6.579e-05 0.01% MLMG::getGradSolution() 11 6.571e-05 6.571e-05 6.571e-05 0.01% MLMG::MLResNormInf() 11 6.542e-05 6.542e-05 6.542e-05 0.01% Castro::finalize_do_advance() 10 6.409e-05 6.409e-05 6.409e-05 0.01% Castro::computeNewDt() 9 5.441e-05 5.441e-05 5.441e-05 0.00% StateData::define() 4 4.908e-05 4.908e-05 4.908e-05 0.00% Amr::FinalizeInit() 1 4.242e-05 4.242e-05 4.242e-05 0.00% Castro::enforce_consistent_e() 1 3.892e-05 3.892e-05 3.892e-05 0.00% MLMG::buildFineMask() 11 3.521e-05 3.521e-05 3.521e-05 0.00% Gravity::set_mass_offset() 11 3.425e-05 3.425e-05 3.425e-05 0.00% Castro::create_source_corrector() 10 3.321e-05 3.321e-05 3.321e-05 0.00% MLLinOp::makeSubCommunicator() 11 3.209e-05 3.209e-05 3.209e-05 0.00% Castro::retry_advance_ctu() 10 3.117e-05 3.117e-05 3.117e-05 0.00% Castro::FluxRegFineAdd() 10 3.021e-05 3.021e-05 3.021e-05 0.00% AmrLevel::AmrLevel(dm) 1 2.467e-05 2.467e-05 2.467e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.192e-05 2.192e-05 2.192e-05 0.00% Castro::post_init() 1 1.75e-05 1.75e-05 1.75e-05 0.00% Amr::writeSmallPlotFile() 1 1.45e-05 1.45e-05 1.45e-05 0.00% AmrLevel::checkPointPost() 3 1.408e-05 1.408e-05 1.408e-05 0.00% Amr::init() 1 1.408e-05 1.408e-05 1.408e-05 0.00% Castro::computeInitialDt() 2 1.229e-05 1.229e-05 1.229e-05 0.00% Amr::InitializeInit() 1 1.192e-05 1.192e-05 1.192e-05 0.00% AmrLevel::checkPointPre() 3 1.046e-05 1.046e-05 1.046e-05 0.00% Amr::initialInit() 1 9.334e-06 9.334e-06 9.334e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 8.666e-06 8.666e-06 8.666e-06 0.00% Amr::initSubcycle() 1 8.583e-06 8.583e-06 8.583e-06 0.00% Castro::post_regrid() 1 6.833e-06 6.833e-06 6.833e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 1.143 1.143 1.143 100.00% Amr::coarseTimeStep() 10 0.9502 0.9502 0.9502 83.11% Amr::timeStep() 10 0.8643 0.8643 0.8643 75.60% Castro::advance() 10 0.8566 0.8566 0.8566 74.93% Castro::subcycle_advance_ctu() 10 0.8412 0.8412 0.8412 73.58% Castro::do_advance_ctu() 10 0.8407 0.8407 0.8407 73.54% Gravity::solve_phi_with_mlmg() 11 0.5697 0.5697 0.5697 49.84% Gravity::actual_solve_with_mlmg() 11 0.5559 0.5559 0.5559 48.63% MLMG::solve() 11 0.5222 0.5222 0.5222 45.68% Castro::construct_new_gravity() 10 0.5071 0.5071 0.5071 44.36% MLMG::oneIter() 82 0.4975 0.4975 0.4975 43.52% MLMG::mgVcycle() 82 0.4952 0.4952 0.4952 43.32% Gravity::solve_for_phi() 10 0.4853 0.4853 0.4853 42.46% MLCellLinOp::smooth() 1640 0.2407 0.2407 0.2407 21.05% Castro::construct_ctu_hydro_source() 10 0.2383 0.2383 0.2383 20.85% MLCellLinOp::applyBC() 4433 0.193 0.193 0.193 16.89% Amr::init() 1 0.1669 0.1669 0.1669 14.60% VisMF::Write(FabArray) 11 0.1665 0.1665 0.1665 14.56% MLMG::mgVcycle_bottom 82 0.1486 0.1486 0.1486 13.00% MLMG::actualBottomSolve() 82 0.1481 0.1481 0.1481 12.96% MLCGSolver::bicgstab 82 0.1458 0.1458 0.1458 12.75% Amr::checkPoint() 3 0.1245 0.1245 0.1245 10.89% AmrLevel::checkPoint() 3 0.1197 0.1197 0.1197 10.47% StateData::checkPoint() 12 0.1196 0.1196 0.1196 10.46% Amr::initialInit() 1 0.09936 0.09936 0.09936 8.69% Amr::FinalizeInit() 1 0.089 0.089 0.089 7.78% Castro::post_init() 1 0.0882 0.0882 0.0882 7.72% Gravity::multilevel_solve_for_new_phi() 1 0.08521 0.08521 0.08521 7.45% Gravity::actual_multilevel_solve() 1 0.08519 0.08519 0.08519 7.45% MLCellLinOp::apply() 1142 0.07687 0.07687 0.07687 6.72% FabArray::FillBoundary() 4023 0.07525 0.07525 0.07525 6.58% MLPoisson::Fsmooth() 3280 0.07524 0.07524 0.07524 6.58% FillPatchIterator::Initialize 41 0.0624 0.0624 0.0624 5.46% FillPatchSingleLevel 41 0.05582 0.05582 0.05582 4.88% MLMG::mgVcycle_down::0 82 0.05476 0.05476 0.05476 4.79% FillBoundary_nowait() 4023 0.05222 0.05222 0.05222 4.57% StateDataPhysBCFunct::() 41 0.05062 0.05062 0.05062 4.43% Amr::writePlotFile() 2 0.05045 0.05045 0.05045 4.41% MLMG::mgVcycle_up::0 82 0.04578 0.04578 0.04578 4.00% MLCellLinOp::correctionResidual() 492 0.04477 0.04477 0.04477 3.92% Castro::clean_state() 62 0.04059 0.04059 0.04059 3.55% MLMG:computeResOfCorrection() 410 0.0402 0.0402 0.0402 3.52% MLMG::mgVcycle_down::1 82 0.03512 0.03512 0.03512 3.07% MLMG::mgVcycle_down::2 82 0.03486 0.03486 0.03486 3.05% MLMG::mgVcycle_down::3 82 0.03392 0.03392 0.03392 2.97% MLMG::mgVcycle_down::4 82 0.03293 0.03293 0.03293 2.88% FabArray::ParallelCopy() 861 0.03085 0.03085 0.03085 2.70% FabArray::setVal() 1140 0.03053 0.03053 0.03053 2.67% CGSolver::sxay() 1586 0.02936 0.02936 0.02936 2.57% MultiFab::Dot() 1114 0.02806 0.02806 0.02806 2.45% MLPoisson::define() 11 0.0276 0.0276 0.0276 2.41% MLMG::mgVcycle_up::2 82 0.02709 0.02709 0.02709 2.37% MLMG::mgVcycle_up::1 82 0.02696 0.02696 0.02696 2.36% Castro::initialize_do_advance() 10 0.02673 0.02673 0.02673 2.34% MLMG::mgVcycle_up::3 82 0.02615 0.02615 0.02615 2.29% MLMG::mgVcycle_up::4 82 0.02594 0.02594 0.02594 2.27% FabArray::ParallelCopy_nowait() 861 0.02582 0.02582 0.02582 2.26% StateData::FillBoundary(geom) 328 0.02534 0.02534 0.02534 2.22% Gravity::get_new_grav_vector() 11 0.02416 0.02416 0.02416 2.11% MLMG::addInterpCorrection() 410 0.02294 0.02294 0.02294 2.01% Castro::construct_old_gravity() 10 0.0229 0.0229 0.0229 2.00% Gravity::get_old_grav_vector() 10 0.02284 0.02284 0.02284 2.00% MLCGSolver::ParallelAllReduce 1514 0.02264 0.02264 0.02264 1.98% amrex::average_down 410 0.02219 0.02219 0.02219 1.94% Castro::computeTemp() 63 0.02147 0.02147 0.02147 1.88% MultiFab::LinComb() 1586 0.02018 0.02018 0.02018 1.76% Castro::expand_state() 10 0.0199 0.0199 0.0199 1.74% MLCellLinOp::defineAuxData() 11 0.01983 0.01983 0.01983 1.73% MLPoisson::Fapply() 1142 0.01612 0.01612 0.01612 1.41% Castro::initialize_advance() 10 0.01489 0.01489 0.01489 1.30% Gravity::fill_multipole_BCs() 11 0.01338 0.01338 0.01338 1.17% Castro::do_new_sources() 10 0.01306 0.01306 0.01306 1.14% FabArrayBase::getFB() 4023 0.01218 0.01218 0.01218 1.07% MLCellLinOp::solutionResidual() 93 0.01218 0.01218 0.01218 1.07% Castro::enforce_min_density() 62 0.0111 0.0111 0.0111 0.97% MLMG::computeResidual() 82 0.01103 0.01103 0.01103 0.96% Amr::InitializeInit() 1 0.01035 0.01035 0.01035 0.91% Amr::defBaseLevel() 1 0.01034 0.01034 0.01034 0.90% Castro::do_old_sources() 10 0.009952 0.009952 0.009952 0.87% MultiFab::Xpay() 585 0.009016 0.009016 0.009016 0.79% MLMG::prepareForSolve() 11 0.008276 0.008276 0.008276 0.72% FabArrayBase::getCPC() 1323 0.007752 0.007752 0.007752 0.68% Castro::post_timestep() 10 0.007459 0.007459 0.007459 0.65% Castro::reset_internal_energy() 63 0.006829 0.006829 0.006829 0.60% MLCellLinOp::defineBC() 11 0.006541 0.006541 0.006541 0.57% BndryData::define() 11 0.006278 0.006278 0.006278 0.55% Castro::estTimeStep() 21 0.006154 0.006154 0.006154 0.54% FabArray::setDomainBndry() 41 0.005991 0.005991 0.005991 0.52% Castro::enforce_speed_limit() 62 0.005131 0.005131 0.005131 0.45% Castro::initData() 1 0.00379 0.00379 0.00379 0.33% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.003671 0.003671 0.003671 0.32% Castro::construct_new_source() 50 0.003476 0.003476 0.003476 0.30% Castro::construct_new_gravity_source() 10 0.003181 0.003181 0.003181 0.28% Castro::construct_old_source() 50 0.00296 0.00296 0.00296 0.26% Castro::normalize_species() 62 0.002934 0.002934 0.002934 0.26% MLMG::getGradSolution() 11 0.002875 0.002875 0.002875 0.25% Castro::computeNewDt() 9 0.002826 0.002826 0.002826 0.25% MLCellLinOp::compGrad() 11 0.00281 0.00281 0.00281 0.25% Castro::construct_old_gravity_source() 10 0.002745 0.002745 0.002745 0.24% MLMG::ResNormInf() 93 0.00235 0.00235 0.00235 0.21% Castro::apply_source_to_state() 20 0.002044 0.002044 0.002044 0.18% FabArrayBase::CPC::define() 454 0.001967 0.001967 0.001967 0.17% MultiFab::Saxpy() 20 0.001922 0.001922 0.001922 0.17% MLMG::computeMLResidual() 11 0.001693 0.001693 0.001693 0.15% MLCellLinOp::setLevelBC() 11 0.001682 0.001682 0.001682 0.15% FabArray::mult() 43 0.001522 0.001522 0.001522 0.13% MLPoisson::prepareForSolve() 11 0.001459 0.001459 0.001459 0.13% MLCellLinOp::prepareForSolve() 11 0.001392 0.001392 0.001392 0.12% Gravity::swapTimeLevels() 10 0.001391 0.001391 0.001391 0.12% Gravity::update_max_rhs() 11 0.001196 0.001196 0.001196 0.10% MLLinOp::define() 11 0.001086 0.001086 0.001086 0.09% MLLinOp::defineGrids() 11 0.001012 0.001012 0.001012 0.09% Castro::computeInitialDt() 2 0.0007074 0.0007074 0.0007074 0.06% MLLinOp::makeAgglomeratedDMap 11 0.0006836 0.0006836 0.0006836 0.06% MultiFab::Copy() 11 0.0005736 0.0005736 0.0005736 0.05% Castro::post_regrid() 1 0.0005284 0.0005284 0.0005284 0.05% Amr::InitAmr() 1 0.0004963 0.0004963 0.0004963 0.04% makeSFC 55 0.0004888 0.0004888 0.0004888 0.04% MultiFab::min() 10 0.0003753 0.0003753 0.0003753 0.03% MLMG::MLResNormInf() 11 0.0003737 0.0003737 0.0003737 0.03% MultiFab::max() 11 0.0002998 0.0002998 0.0002998 0.03% FabArrayBase::FB::FB() 56 0.0002634 0.0002634 0.0002634 0.02% MLMG::MLRhsNormInf() 11 0.0002553 0.0002553 0.0002553 0.02% DistributionMapping::Distribute() 56 0.0001657 0.0001657 0.0001657 0.01% Castro::finalize_advance() 10 0.0001632 0.0001632 0.0001632 0.01% AmrLevel::AmrLevel(dm) 1 7.375e-05 7.375e-05 7.375e-05 0.01% Castro::swap_state_time_levels() 10 6.646e-05 6.646e-05 6.646e-05 0.01% Castro::finalize_do_advance() 10 6.409e-05 6.409e-05 6.409e-05 0.01% StateData::define() 4 4.908e-05 4.908e-05 4.908e-05 0.00% Castro::enforce_consistent_e() 1 3.892e-05 3.892e-05 3.892e-05 0.00% MLMG::buildFineMask() 11 3.521e-05 3.521e-05 3.521e-05 0.00% Gravity::set_mass_offset() 11 3.425e-05 3.425e-05 3.425e-05 0.00% Castro::create_source_corrector() 10 3.321e-05 3.321e-05 3.321e-05 0.00% MLLinOp::makeSubCommunicator() 11 3.209e-05 3.209e-05 3.209e-05 0.00% Castro::retry_advance_ctu() 10 3.117e-05 3.117e-05 3.117e-05 0.00% Castro::FluxRegFineAdd() 10 3.021e-05 3.021e-05 3.021e-05 0.00% Amr::writeSmallPlotFile() 1 1.45e-05 1.45e-05 1.45e-05 0.00% AmrLevel::checkPointPost() 3 1.408e-05 1.408e-05 1.408e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 1.233e-05 1.233e-05 1.233e-05 0.00% AmrLevel::checkPointPre() 3 1.046e-05 1.046e-05 1.046e-05 0.00% Amr::initSubcycle() 1 8.583e-06 8.583e-06 8.583e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10622 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.01-18-g405b490356e5) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.01-18-g405b490356e5) initialized Starting run at 09:05:48 UTC on 2022-01-12. Successfully read inputs file ... Castro git describe: 21.12-23-g727e39bb6 AMReX git describe: 22.01-18-g405b49035 Microphysics git describe: 22.01-14-geeef682d reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.664774427 Restart time = 0.059669469 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.143147385 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.073986729 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.074685608 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.086559894 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.092634016 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.063636829 seconds Ending run at 09:05:48 UTC on 2022-01-12. Run time = 0.595321562 Run time without initialization = 0.535026421 Average number of zones advanced per microsecond: 2.450 Average number of zones advanced per microsecond per rank: 2.450 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489334272 TinyProfiler total time across processes [min...avg...max]: 0.5954 ... 0.5954 ... 0.5954 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1372 0.1372 0.1372 23.05% MLCellLinOp::applyBC() 1946 0.05468 0.05468 0.05468 9.18% VisMF::Read() 3 0.04503 0.04503 0.04503 7.56% Amr::writePlotFile() 1 0.03783 0.03783 0.03783 6.35% MLPoisson::Fsmooth() 1440 0.0325 0.0325 0.0325 5.46% VisMF::Write(FabArray) 1 0.02589 0.02589 0.02589 4.35% FabArray::setVal() 533 0.01869 0.01869 0.01869 3.14% FillBoundary_nowait() 1766 0.0186 0.0186 0.0186 3.12% MLCGSolver::bicgstab 36 0.01662 0.01662 0.01662 2.79% StateData::FillBoundary(geom) 160 0.01242 0.01242 0.01242 2.09% MultiFab::Dot() 484 0.01216 0.01216 0.01216 2.04% MLCellLinOp::smooth() 720 0.01032 0.01032 0.01032 1.73% FabArray::ParallelCopy_nowait() 380 0.01025 0.01025 0.01025 1.72% StateDataPhysBCFunct::() 20 0.01021 0.01021 0.01021 1.71% FabArray::FillBoundary() 1766 0.01013 0.01013 0.01013 1.70% MultiFab::LinComb() 690 0.008666 0.008666 0.008666 1.46% Castro::computeTemp() 30 0.007861 0.007861 0.007861 1.32% MLCellLinOp::defineAuxData() 6 0.007553 0.007553 0.007553 1.27% Gravity::fill_multipole_BCs() 6 0.007273 0.007273 0.007273 1.22% MLPoisson::Fapply() 500 0.006945 0.006945 0.006945 1.17% Castro::enforce_min_density() 30 0.005427 0.005427 0.005427 0.91% FabArrayBase::getFB() 1766 0.00528 0.00528 0.00528 0.89% Amr::restart() 1 0.004926 0.004926 0.004926 0.83% MLMG::addInterpCorrection() 180 0.004483 0.004483 0.004483 0.75% MLCellLinOp::apply() 500 0.004327 0.004327 0.004327 0.73% Castro::expand_state() 5 0.004236 0.004236 0.004236 0.71% amrex::average_down 180 0.004085 0.004085 0.004085 0.69% FabArray::setDomainBndry() 20 0.004057 0.004057 0.004057 0.68% CGSolver::sxay() 690 0.003983 0.003983 0.003983 0.67% MultiFab::Xpay() 258 0.003924 0.003924 0.003924 0.66% Castro::reset_internal_energy() 30 0.003528 0.003528 0.003528 0.59% Castro::do_advance_ctu() 5 0.002863 0.002863 0.002863 0.48% FabArrayBase::getCPC() 632 0.002812 0.002812 0.002812 0.47% MLCGSolver::ParallelAllReduce 659 0.002718 0.002718 0.002718 0.46% Castro::estTimeStep() 10 0.002498 0.002498 0.002498 0.42% BndryData::define() 6 0.00245 0.00245 0.00245 0.41% FabArray::ParallelCopy() 380 0.002246 0.002246 0.002246 0.38% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001964 0.001964 0.001964 0.33% Gravity::get_new_grav_vector() 5 0.001906 0.001906 0.001906 0.32% MLCellLinOp::correctionResidual() 216 0.001842 0.001842 0.001842 0.31% Castro::construct_new_gravity_source() 5 0.001643 0.001643 0.001643 0.28% MLCellLinOp::compGrad() 6 0.001561 0.001561 0.001561 0.26% Castro::normalize_species() 30 0.001498 0.001498 0.001498 0.25% Castro::construct_old_gravity_source() 5 0.001423 0.001423 0.001423 0.24% MLMG::mgVcycle() 36 0.001336 0.001336 0.001336 0.22% Castro::enforce_speed_limit() 30 0.001314 0.001314 0.001314 0.22% FabArrayBase::CPC::define() 244 0.001118 0.001118 0.001118 0.19% MLMG::ResNormInf() 42 0.001061 0.001061 0.001061 0.18% MLMG:computeResOfCorrection() 180 0.001046 0.001046 0.001046 0.18% MLMG::oneIter() 36 0.001029 0.001029 0.001029 0.17% Gravity::get_old_grav_vector() 5 0.001008 0.001008 0.001008 0.17% MultiFab::Saxpy() 10 0.0009595 0.0009595 0.0009595 0.16% MLCellLinOp::setLevelBC() 6 0.0009249 0.0009249 0.0009249 0.16% MLMG::prepareForSolve() 6 0.0008933 0.0008933 0.0008933 0.15% MLCellLinOp::prepareForSolve() 6 0.0008125 0.0008125 0.0008125 0.14% Gravity::actual_solve_with_mlmg() 6 0.0007951 0.0007951 0.0007951 0.13% FabArray::mult() 22 0.0007753 0.0007753 0.0007753 0.13% Castro::subcycle_advance_ctu() 5 0.0007751 0.0007751 0.0007751 0.13% MLMG::mgVcycle_down::0 36 0.0006662 0.0006662 0.0006662 0.11% Gravity::update_max_rhs() 6 0.0006559 0.0006559 0.0006559 0.11% MLMG::mgVcycle_down::1 36 0.0006315 0.0006315 0.0006315 0.11% MLMG::mgVcycle_down::2 36 0.0006169 0.0006169 0.0006169 0.10% MLMG::mgVcycle_down::4 36 0.0006148 0.0006148 0.0006148 0.10% MLMG::mgVcycle_down::3 36 0.000614 0.000614 0.000614 0.10% MLMG::actualBottomSolve() 36 0.0005533 0.0005533 0.0005533 0.09% MLMG::solve() 6 0.0004645 0.0004645 0.0004645 0.08% Castro::clean_state() 30 0.0004638 0.0004638 0.0004638 0.08% MLMG::mgVcycle_up::3 36 0.0004252 0.0004252 0.0004252 0.07% MLMG::mgVcycle_up::2 36 0.0004232 0.0004232 0.0004232 0.07% Amr::InitAmr() 1 0.0004175 0.0004175 0.0004175 0.07% MultiFab::Copy() 6 0.0004166 0.0004166 0.0004166 0.07% MLMG::mgVcycle_up::4 36 0.0004162 0.0004162 0.0004162 0.07% MLMG::mgVcycle_up::0 36 0.0004112 0.0004112 0.0004112 0.07% MLMG::mgVcycle_up::1 36 0.0004085 0.0004085 0.0004085 0.07% Castro::construct_new_source() 25 0.0003842 0.0003842 0.0003842 0.06% MLCellLinOp::solutionResidual() 42 0.0003648 0.0003648 0.0003648 0.06% Castro::construct_new_gravity() 5 0.0003095 0.0003095 0.0003095 0.05% main() 1 0.0003042 0.0003042 0.0003042 0.05% FillPatchIterator::Initialize 20 0.0002868 0.0002868 0.0002868 0.05% Gravity::solve_for_phi() 5 0.0002863 0.0002863 0.0002863 0.05% Gravity::actual_multilevel_solve() 1 0.0002336 0.0002336 0.0002336 0.04% Castro::construct_old_source() 25 0.0002292 0.0002292 0.0002292 0.04% Castro::do_new_sources() 5 0.0002159 0.0002159 0.0002159 0.04% MLMG::mgVcycle_bottom 36 0.000209 0.000209 0.000209 0.04% MLMG::computeResidual() 36 0.0002071 0.0002071 0.0002071 0.03% Castro::initialize_advance() 5 0.0001987 0.0001987 0.0001987 0.03% Amr::coarseTimeStep() 5 0.000186 0.000186 0.000186 0.03% MultiFab::min() 5 0.0001838 0.0001838 0.0001838 0.03% makeSFC 30 0.0001794 0.0001794 0.0001794 0.03% FillPatchSingleLevel 20 0.0001781 0.0001781 0.0001781 0.03% MultiFab::max() 6 0.0001622 0.0001622 0.0001622 0.03% Amr::timeStep() 5 0.0001549 0.0001549 0.0001549 0.03% Castro::advance() 5 0.0001533 0.0001533 0.0001533 0.03% Castro::do_old_sources() 5 0.0001442 0.0001442 0.0001442 0.02% MLLinOp::defineGrids() 6 0.000141 0.000141 0.000141 0.02% MLCellLinOp::defineBC() 6 0.0001404 0.0001404 0.0001404 0.02% FabArrayBase::FB::FB() 26 0.0001349 0.0001349 0.0001349 0.02% MLMG::MLRhsNormInf() 6 0.0001321 0.0001321 0.0001321 0.02% Castro::create_source_corrector() 5 0.0001264 0.0001264 0.0001264 0.02% Castro::post_timestep() 5 0.0001242 0.0001242 0.0001242 0.02% MLLinOp::makeAgglomeratedDMap 6 0.0001061 0.0001061 0.0001061 0.02% MLPoisson::define() 6 9.95e-05 9.95e-05 9.95e-05 0.02% AmrLevel::restart() 1 9.58e-05 9.58e-05 9.58e-05 0.02% DistributionMapping::Distribute() 31 9.362e-05 9.362e-05 9.362e-05 0.02% StateData::restartDoit() 4 9.354e-05 9.354e-05 9.354e-05 0.02% Gravity::solve_phi_with_mlmg() 6 8.042e-05 8.042e-05 8.042e-05 0.01% Castro::finalize_advance() 5 7.75e-05 7.75e-05 7.75e-05 0.01% Castro::initialize_do_advance() 5 7.042e-05 7.042e-05 7.042e-05 0.01% Castro::apply_source_to_state() 10 6.075e-05 6.075e-05 6.075e-05 0.01% Gravity::swapTimeLevels() 5 5.963e-05 5.963e-05 5.963e-05 0.01% Castro::finalize_do_advance() 5 5.2e-05 5.2e-05 5.2e-05 0.01% MLLinOp::define() 6 4.263e-05 4.263e-05 4.263e-05 0.01% Castro::swap_state_time_levels() 5 3.954e-05 3.954e-05 3.954e-05 0.01% Castro::construct_old_gravity() 5 3.821e-05 3.821e-05 3.821e-05 0.01% MLPoisson::prepareForSolve() 6 3.7e-05 3.7e-05 3.7e-05 0.01% MLMG::computeMLResidual() 6 3.571e-05 3.571e-05 3.571e-05 0.01% MLMG::MLResNormInf() 6 3.521e-05 3.521e-05 3.521e-05 0.01% MLMG::getGradSolution() 6 3.513e-05 3.513e-05 3.513e-05 0.01% Castro::post_restart() 1 3.462e-05 3.462e-05 3.462e-05 0.01% Castro::computeNewDt() 5 3.017e-05 3.017e-05 3.017e-05 0.01% Amr::init() 1 2.334e-05 2.334e-05 2.334e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.108e-05 2.108e-05 2.108e-05 0.00% MLMG::buildFineMask() 6 2.054e-05 2.054e-05 2.054e-05 0.00% Gravity::set_mass_offset() 6 1.842e-05 1.842e-05 1.842e-05 0.00% MLLinOp::makeSubCommunicator() 6 1.754e-05 1.754e-05 1.754e-05 0.00% Castro::retry_advance_ctu() 5 1.746e-05 1.746e-05 1.746e-05 0.00% Castro::FluxRegFineAdd() 5 1.642e-05 1.642e-05 1.642e-05 0.00% Amr::writeSmallPlotFile() 1 1.367e-05 1.367e-05 1.367e-05 0.00% Amr::initSubcycle() 1 8.667e-06 8.667e-06 8.667e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 8.624e-06 8.624e-06 8.624e-06 0.00% AmrLevel::AmrLevel() 1 3.75e-06 3.75e-06 3.75e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.5953 0.5953 0.5953 100.00% Amr::coarseTimeStep() 5 0.4711 0.4711 0.4711 79.14% Amr::timeStep() 5 0.4696 0.4696 0.4696 78.87% Castro::advance() 5 0.4668 0.4668 0.4668 78.41% Castro::subcycle_advance_ctu() 5 0.4544 0.4544 0.4544 76.33% Castro::do_advance_ctu() 5 0.4536 0.4536 0.4536 76.19% Castro::construct_new_gravity() 5 0.2636 0.2636 0.2636 44.28% Gravity::solve_phi_with_mlmg() 6 0.2599 0.2599 0.2599 43.66% Gravity::actual_solve_with_mlmg() 6 0.2524 0.2524 0.2524 42.40% Gravity::solve_for_phi() 5 0.2524 0.2524 0.2524 42.39% MLMG::solve() 6 0.2337 0.2337 0.2337 39.26% MLMG::oneIter() 36 0.221 0.221 0.221 37.13% MLMG::mgVcycle() 36 0.22 0.22 0.22 36.95% Castro::construct_ctu_hydro_source() 5 0.1372 0.1372 0.1372 23.05% MLCellLinOp::smooth() 720 0.1077 0.1077 0.1077 18.08% MLCellLinOp::applyBC() 1946 0.08884 0.08884 0.08884 14.92% MLMG::mgVcycle_bottom 36 0.06504 0.06504 0.06504 10.92% MLMG::actualBottomSolve() 36 0.06483 0.06483 0.06483 10.89% MLCGSolver::bicgstab 36 0.0638 0.0638 0.0638 10.72% Amr::writePlotFile() 1 0.06373 0.06373 0.06373 10.70% Amr::init() 1 0.05973 0.05973 0.05973 10.03% Amr::restart() 1 0.05971 0.05971 0.05971 10.03% AmrLevel::restart() 1 0.04617 0.04617 0.04617 7.76% StateData::restartDoit() 4 0.04606 0.04606 0.04606 7.74% VisMF::Read() 3 0.04503 0.04503 0.04503 7.56% MLCellLinOp::apply() 500 0.0348 0.0348 0.0348 5.85% FabArray::FillBoundary() 1766 0.03415 0.03415 0.03415 5.74% MLPoisson::Fsmooth() 1440 0.0325 0.0325 0.0325 5.46% FillPatchIterator::Initialize 20 0.02961 0.02961 0.02961 4.97% VisMF::Write(FabArray) 1 0.02589 0.02589 0.02589 4.35% FillPatchSingleLevel 20 0.02527 0.02527 0.02527 4.24% MLMG::mgVcycle_down::0 36 0.02493 0.02493 0.02493 4.19% FillBoundary_nowait() 1766 0.02402 0.02402 0.02402 4.03% StateDataPhysBCFunct::() 20 0.02262 0.02262 0.02262 3.80% MLMG::mgVcycle_up::0 36 0.021 0.021 0.021 3.53% Castro::clean_state() 30 0.02009 0.02009 0.02009 3.37% MLCellLinOp::correctionResidual() 216 0.01999 0.01999 0.01999 3.36% FabArray::setVal() 533 0.01869 0.01869 0.01869 3.14% MLMG:computeResOfCorrection() 180 0.01793 0.01793 0.01793 3.01% Castro::initialize_do_advance() 5 0.01679 0.01679 0.01679 2.82% MLMG::mgVcycle_down::1 36 0.01551 0.01551 0.01551 2.61% MLMG::mgVcycle_down::2 36 0.01544 0.01544 0.01544 2.59% MLPoisson::define() 6 0.01495 0.01495 0.01495 2.51% MLMG::mgVcycle_down::3 36 0.01494 0.01494 0.01494 2.51% MLMG::mgVcycle_down::4 36 0.01483 0.01483 0.01483 2.49% FabArray::ParallelCopy() 380 0.01426 0.01426 0.01426 2.40% Castro::expand_state() 5 0.01304 0.01304 0.01304 2.19% CGSolver::sxay() 690 0.01265 0.01265 0.01265 2.12% StateData::FillBoundary(geom) 160 0.01242 0.01242 0.01242 2.09% MultiFab::Dot() 484 0.01216 0.01216 0.01216 2.04% Castro::initialize_advance() 5 0.01213 0.01213 0.01213 2.04% MLMG::mgVcycle_up::2 36 0.01202 0.01202 0.01202 2.02% FabArray::ParallelCopy_nowait() 380 0.01202 0.01202 0.01202 2.02% MLMG::mgVcycle_up::1 36 0.01173 0.01173 0.01173 1.97% MLMG::mgVcycle_up::4 36 0.0117 0.0117 0.0117 1.96% MLMG::mgVcycle_up::3 36 0.01154 0.01154 0.01154 1.94% Castro::computeTemp() 30 0.01139 0.01139 0.01139 1.91% Gravity::get_new_grav_vector() 5 0.01098 0.01098 0.01098 1.84% MLCellLinOp::defineAuxData() 6 0.01072 0.01072 0.01072 1.80% MLMG::addInterpCorrection() 180 0.01035 0.01035 0.01035 1.74% amrex::average_down 180 0.01002 0.01002 0.01002 1.68% MLCGSolver::ParallelAllReduce 659 0.00986 0.00986 0.00986 1.66% Castro::construct_old_gravity() 5 0.009831 0.009831 0.009831 1.65% Gravity::get_old_grav_vector() 5 0.009793 0.009793 0.009793 1.64% MultiFab::LinComb() 690 0.008666 0.008666 0.008666 1.46% Castro::post_restart() 1 0.008607 0.008607 0.008607 1.45% Castro::do_new_sources() 5 0.008174 0.008174 0.008174 1.37% Gravity::multilevel_solve_for_new_phi() 1 0.00812 0.00812 0.00812 1.36% Gravity::actual_multilevel_solve() 1 0.008099 0.008099 0.008099 1.36% Gravity::fill_multipole_BCs() 6 0.007273 0.007273 0.007273 1.22% MLPoisson::Fapply() 500 0.006945 0.006945 0.006945 1.17% MLCellLinOp::solutionResidual() 42 0.005689 0.005689 0.005689 0.96% Castro::do_old_sources() 5 0.005533 0.005533 0.005533 0.93% Castro::enforce_min_density() 30 0.005427 0.005427 0.005427 0.91% FabArrayBase::getFB() 1766 0.005415 0.005415 0.005415 0.91% MLMG::prepareForSolve() 6 0.005058 0.005058 0.005058 0.85% MLMG::computeResidual() 36 0.004981 0.004981 0.004981 0.84% FabArray::setDomainBndry() 20 0.004057 0.004057 0.004057 0.68% FabArrayBase::getCPC() 632 0.003929 0.003929 0.003929 0.66% MultiFab::Xpay() 258 0.003924 0.003924 0.003924 0.66% MLCellLinOp::defineBC() 6 0.003556 0.003556 0.003556 0.60% Castro::reset_internal_energy() 30 0.003528 0.003528 0.003528 0.59% BndryData::define() 6 0.003416 0.003416 0.003416 0.57% Castro::post_timestep() 5 0.002615 0.002615 0.002615 0.44% Castro::estTimeStep() 10 0.002498 0.002498 0.002498 0.42% MLMG::getGradSolution() 6 0.002058 0.002058 0.002058 0.35% Castro::construct_new_source() 25 0.002027 0.002027 0.002027 0.34% MLCellLinOp::compGrad() 6 0.002023 0.002023 0.002023 0.34% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001964 0.001964 0.001964 0.33% Castro::construct_old_source() 25 0.001653 0.001653 0.001653 0.28% Castro::construct_new_gravity_source() 5 0.001643 0.001643 0.001643 0.28% Castro::normalize_species() 30 0.001498 0.001498 0.001498 0.25% Castro::construct_old_gravity_source() 5 0.001423 0.001423 0.001423 0.24% Castro::computeNewDt() 5 0.001374 0.001374 0.001374 0.23% Castro::enforce_speed_limit() 30 0.001314 0.001314 0.001314 0.22% FabArrayBase::CPC::define() 244 0.001118 0.001118 0.001118 0.19% MLMG::ResNormInf() 42 0.001061 0.001061 0.001061 0.18% Castro::apply_source_to_state() 10 0.00102 0.00102 0.00102 0.17% Gravity::swapTimeLevels() 5 0.001012 0.001012 0.001012 0.17% MultiFab::Saxpy() 10 0.0009595 0.0009595 0.0009595 0.16% MLMG::computeMLResidual() 6 0.0009508 0.0009508 0.0009508 0.16% MLCellLinOp::setLevelBC() 6 0.0009249 0.0009249 0.0009249 0.16% Gravity::update_max_rhs() 6 0.0009221 0.0009221 0.0009221 0.15% MLPoisson::prepareForSolve() 6 0.0008495 0.0008495 0.0008495 0.14% MLCellLinOp::prepareForSolve() 6 0.0008125 0.0008125 0.0008125 0.14% FabArray::mult() 22 0.0007753 0.0007753 0.0007753 0.13% MLLinOp::define() 6 0.0005763 0.0005763 0.0005763 0.10% MLLinOp::defineGrids() 6 0.0005337 0.0005337 0.0005337 0.09% Amr::InitAmr() 1 0.0004261 0.0004261 0.0004261 0.07% MultiFab::Copy() 6 0.0004166 0.0004166 0.0004166 0.07% MLLinOp::makeAgglomeratedDMap 6 0.0003752 0.0003752 0.0003752 0.06% makeSFC 30 0.0002691 0.0002691 0.0002691 0.05% MLMG::MLResNormInf() 6 0.0002105 0.0002105 0.0002105 0.04% MultiFab::min() 5 0.0001838 0.0001838 0.0001838 0.03% MultiFab::max() 6 0.0001622 0.0001622 0.0001622 0.03% FabArrayBase::FB::FB() 26 0.0001349 0.0001349 0.0001349 0.02% MLMG::MLRhsNormInf() 6 0.0001321 0.0001321 0.0001321 0.02% Castro::create_source_corrector() 5 0.0001264 0.0001264 0.0001264 0.02% Castro::finalize_advance() 5 9.392e-05 9.392e-05 9.392e-05 0.02% DistributionMapping::Distribute() 31 9.362e-05 9.362e-05 9.362e-05 0.02% Castro::finalize_do_advance() 5 5.2e-05 5.2e-05 5.2e-05 0.01% Castro::swap_state_time_levels() 5 3.954e-05 3.954e-05 3.954e-05 0.01% MLMG::buildFineMask() 6 2.054e-05 2.054e-05 2.054e-05 0.00% Gravity::set_mass_offset() 6 1.842e-05 1.842e-05 1.842e-05 0.00% MLLinOp::makeSubCommunicator() 6 1.754e-05 1.754e-05 1.754e-05 0.00% Castro::retry_advance_ctu() 5 1.746e-05 1.746e-05 1.746e-05 0.00% Castro::FluxRegFineAdd() 5 1.642e-05 1.642e-05 1.642e-05 0.00% Amr::writeSmallPlotFile() 1 1.367e-05 1.367e-05 1.367e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 1.258e-05 1.258e-05 1.258e-05 0.00% Amr::initSubcycle() 1 8.667e-06 8.667e-06 8.667e-06 0.00% AmrLevel::AmrLevel() 1 3.75e-06 3.75e-06 3.75e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10622 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.01-18-g405b490356e5) finalized