Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.03-12-g806f2373b68e) initialized Starting run at 08:18:10 UTC on 2022-03-15. Successfully read inputs file ... Castro git describe: 22.02-5-g5116b7268 AMReX git describe: 22.03-12-g806f2373b Microphysics git describe: 22.03-6-g41c4af20 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.038164037 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022805435 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.125269273 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051078768 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.053585702 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.056538433 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.059836889 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.035189537 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052532139 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048031203 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.047173106 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.047940338 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.051176426 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.035056387 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.021949652 seconds Ending run at 08:18:10 UTC on 2022-03-15. Run time = 0.80492482 Run time without initialization = 0.685858583 Average number of zones advanced per microsecond: 3.822 Average number of zones advanced per microsecond per rank: 3.822 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.805 ... 0.805 ... 0.805 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2001 0.2001 0.2001 24.86% VisMF::Write(FabArray) 11 0.1462 0.1462 0.1462 18.17% MLCellLinOp::applyBC() 4379 0.09057 0.09057 0.09057 11.25% MLPoisson::Fsmooth() 3240 0.05787 0.05787 0.05787 7.19% FabArray::setVal() 1135 0.02647 0.02647 0.02647 3.29% StateData::FillBoundary(geom) 328 0.02311 0.02311 0.02311 2.87% StateDataPhysBCFunct::() 41 0.02283 0.02283 0.02283 2.84% MLCGSolver::bicgstab 81 0.02092 0.02092 0.02092 2.60% MultiFab::Dot() 1100 0.02083 0.02083 0.02083 2.59% FillBoundary_nowait() 3974 0.01669 0.01669 0.01669 2.07% FabArray::ParallelCopy_nowait() 851 0.01576 0.01576 0.01576 1.96% MultiFab::LinComb() 1566 0.01199 0.01199 0.01199 1.49% Gravity::fill_multipole_BCs() 11 0.01181 0.01181 0.01181 1.47% Castro::computeTemp() 63 0.01075 0.01075 0.01075 1.34% MLPoisson::Fapply() 1128 0.01024 0.01024 0.01024 1.27% MLCellLinOp::defineAuxData() 11 0.009967 0.009967 0.009967 1.24% Castro::enforce_min_density() 62 0.008676 0.008676 0.008676 1.08% MLMG::addInterpCorrection() 405 0.006833 0.006833 0.006833 0.85% amrex::average_down 405 0.006328 0.006328 0.006328 0.79% MultiFab::Xpay() 578 0.005863 0.005863 0.005863 0.73% FabArray::setDomainBndry() 41 0.005753 0.005753 0.005753 0.71% Castro::reset_internal_energy() 63 0.005275 0.005275 0.005275 0.66% Castro::expand_state() 10 0.004826 0.004826 0.004826 0.60% Castro::estTimeStep() 21 0.004671 0.004671 0.004671 0.58% Amr::checkPoint() 3 0.003864 0.003864 0.003864 0.48% BndryData::define() 11 0.003597 0.003597 0.003597 0.45% Castro::enforce_speed_limit() 62 0.003325 0.003325 0.003325 0.41% Castro::do_advance_ctu() 10 0.003193 0.003193 0.003193 0.40% Amr::writePlotFile() 2 0.003089 0.003089 0.003089 0.38% Castro::construct_new_gravity_source() 10 0.00307 0.00307 0.00307 0.38% Gravity::get_new_grav_vector() 11 0.002743 0.002743 0.002743 0.34% Castro::normalize_species() 62 0.002609 0.002609 0.002609 0.32% Castro::construct_old_gravity_source() 10 0.002535 0.002535 0.002535 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002238 0.002238 0.002238 0.28% MLCellLinOp::compGrad() 11 0.001982 0.001982 0.001982 0.25% MLMG::ResNormInf() 92 0.001849 0.001849 0.001849 0.23% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.23% Gravity::get_old_grav_vector() 10 0.001762 0.001762 0.001762 0.22% MLMG::oneIter() 81 0.001609 0.001609 0.001609 0.20% MLCellLinOp::setLevelBC() 11 0.001519 0.001519 0.001519 0.19% Gravity::actual_solve_with_mlmg() 11 0.001365 0.001365 0.001365 0.17% MLCellLinOp::prepareForSolve() 11 0.001319 0.001319 0.001319 0.16% FabArray::mult() 43 0.001316 0.001316 0.001316 0.16% Castro::initData() 1 0.001246 0.001246 0.001246 0.15% MLCellLinOp::smooth() 1620 0.001179 0.001179 0.001179 0.15% FabArrayBase::getCPC() 1313 0.0008243 0.0008243 0.0008243 0.10% FabArray::FillBoundary() 3974 0.0008221 0.0008221 0.0008221 0.10% MLMG::prepareForSolve() 11 0.0007518 0.0007518 0.0007518 0.09% FabArrayBase::CPC::define() 454 0.0007038 0.0007038 0.0007038 0.09% FabArrayBase::getFB() 3974 0.0006949 0.0006949 0.0006949 0.09% Gravity::update_max_rhs() 11 0.0005567 0.0005567 0.0005567 0.07% MultiFab::Copy() 11 0.0005083 0.0005083 0.0005083 0.06% MLCellLinOp::apply() 1128 0.000504 0.000504 0.000504 0.06% CGSolver::sxay() 1566 0.0004923 0.0004923 0.0004923 0.06% Gravity::solve_for_phi() 10 0.0004383 0.0004383 0.0004383 0.05% Amr::InitAmr() 1 0.0004329 0.0004329 0.0004329 0.05% MLMG::mgVcycle() 81 0.0003875 0.0003875 0.0003875 0.05% MLCGSolver::ParallelAllReduce 1495 0.0003373 0.0003373 0.0003373 0.04% MultiFab::min() 10 0.0003162 0.0003162 0.0003162 0.04% main() 1 0.0002897 0.0002897 0.0002897 0.04% FabArray::ParallelCopy() 851 0.0002833 0.0002833 0.0002833 0.04% Castro::construct_new_gravity() 10 0.000261 0.000261 0.000261 0.03% MultiFab::max() 11 0.0002505 0.0002505 0.0002505 0.03% FillPatchIterator::Initialize 41 0.0002471 0.0002471 0.0002471 0.03% Gravity::actual_multilevel_solve() 1 0.000226 0.000226 0.000226 0.03% MLCellLinOp::correctionResidual() 486 0.0002193 0.0002193 0.0002193 0.03% Amr::coarseTimeStep() 10 0.0002117 0.0002117 0.0002117 0.03% Amr::timeStep() 10 0.0002019 0.0002019 0.0002019 0.03% MLMG::MLRhsNormInf() 11 0.0001939 0.0001939 0.0001939 0.02% Castro::subcycle_advance_ctu() 10 0.0001854 0.0001854 0.0001854 0.02% MLCellLinOp::defineBC() 11 0.0001796 0.0001796 0.0001796 0.02% MLLinOp::defineGrids() 11 0.0001713 0.0001713 0.0001713 0.02% Amr::defBaseLevel() 1 0.0001405 0.0001405 0.0001405 0.02% StateData::checkPoint() 12 0.0001333 0.0001333 0.0001333 0.02% MLMG:computeResOfCorrection() 405 0.0001307 0.0001307 0.0001307 0.02% Castro::advance() 10 0.0001171 0.0001171 0.0001171 0.01% MLMG::actualBottomSolve() 81 0.0001038 0.0001038 0.0001038 0.01% MLMG::mgVcycle_down::0 81 9.847e-05 9.847e-05 9.847e-05 0.01% FabArrayBase::FB::FB() 56 8.517e-05 8.517e-05 8.517e-05 0.01% MLMG::mgVcycle_down::1 81 8.25e-05 8.25e-05 8.25e-05 0.01% Castro::initialize_advance() 10 8.032e-05 8.032e-05 8.032e-05 0.01% MLMG::mgVcycle_down::2 81 7.938e-05 7.938e-05 7.938e-05 0.01% MLMG::solve() 11 7.872e-05 7.872e-05 7.872e-05 0.01% MLMG::mgVcycle_down::4 81 7.671e-05 7.671e-05 7.671e-05 0.01% Castro::clean_state() 62 7.604e-05 7.604e-05 7.604e-05 0.01% MLMG::mgVcycle_down::3 81 7.595e-05 7.595e-05 7.595e-05 0.01% AmrLevel::checkPoint() 3 7.308e-05 7.308e-05 7.308e-05 0.01% MLMG::mgVcycle_up::4 81 5.98e-05 5.98e-05 5.98e-05 0.01% Castro::finalize_advance() 10 5.902e-05 5.902e-05 5.902e-05 0.01% Castro::initialize_do_advance() 10 5.764e-05 5.764e-05 5.764e-05 0.01% MLMG::mgVcycle_up::0 81 5.592e-05 5.592e-05 5.592e-05 0.01% Castro::post_timestep() 10 5.228e-05 5.228e-05 5.228e-05 0.01% MLMG::mgVcycle_up::1 81 4.871e-05 4.871e-05 4.871e-05 0.01% MLCellLinOp::solutionResidual() 92 4.831e-05 4.831e-05 4.831e-05 0.01% MLMG::mgVcycle_up::3 81 4.797e-05 4.797e-05 4.797e-05 0.01% MLMG::mgVcycle_up::2 81 4.64e-05 4.64e-05 4.64e-05 0.01% StateData::define() 4 4.246e-05 4.246e-05 4.246e-05 0.01% Castro::construct_new_source() 50 4.091e-05 4.091e-05 4.091e-05 0.01% Castro::finalize_do_advance() 10 4.029e-05 4.029e-05 4.029e-05 0.01% Castro::construct_old_source() 50 3.928e-05 3.928e-05 3.928e-05 0.00% Castro::swap_state_time_levels() 10 3.784e-05 3.784e-05 3.784e-05 0.00% Castro::enforce_consistent_e() 1 3.445e-05 3.445e-05 3.445e-05 0.00% MLMG::computeResidual() 81 3.419e-05 3.419e-05 3.419e-05 0.00% MLMG::mgVcycle_bottom 81 3.324e-05 3.324e-05 3.324e-05 0.00% FillPatchSingleLevel 41 3.063e-05 3.063e-05 3.063e-05 0.00% makeSFC 55 2.955e-05 2.955e-05 2.955e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.776e-05 2.776e-05 2.776e-05 0.00% MLLinOp::define() 11 2.752e-05 2.752e-05 2.752e-05 0.00% Amr::writeSmallPlotFile() 1 2.511e-05 2.511e-05 2.511e-05 0.00% MLPoisson::define() 11 2.313e-05 2.313e-05 2.313e-05 0.00% Amr::FinalizeInit() 1 2.272e-05 2.272e-05 2.272e-05 0.00% Castro::do_new_sources() 10 1.74e-05 1.74e-05 1.74e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.627e-05 1.627e-05 1.627e-05 0.00% Castro::do_old_sources() 10 1.622e-05 1.622e-05 1.622e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.597e-05 1.597e-05 1.597e-05 0.00% DistributionMapping::Distribute() 56 1.582e-05 1.582e-05 1.582e-05 0.00% Castro::apply_source_to_state() 20 1.274e-05 1.274e-05 1.274e-05 0.00% Castro::construct_old_gravity() 10 1.034e-05 1.034e-05 1.034e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.024e-05 1.024e-05 1.024e-05 0.00% Gravity::swapTimeLevels() 10 9.719e-06 9.719e-06 9.719e-06 0.00% Amr::initSubcycle() 1 9.343e-06 9.343e-06 9.343e-06 0.00% MLPoisson::prepareForSolve() 11 9.156e-06 9.156e-06 9.156e-06 0.00% MLMG::computeMLResidual() 11 7.751e-06 7.751e-06 7.751e-06 0.00% Castro::computeNewDt() 9 6.76e-06 6.76e-06 6.76e-06 0.00% MLMG::buildFineMask() 11 6.317e-06 6.317e-06 6.317e-06 0.00% MLMG::getGradSolution() 11 6.132e-06 6.132e-06 6.132e-06 0.00% Amr::InitializeInit() 1 5.832e-06 5.832e-06 5.832e-06 0.00% Castro::create_source_corrector() 10 5.714e-06 5.714e-06 5.714e-06 0.00% AmrLevel::checkPointPost() 3 5.202e-06 5.202e-06 5.202e-06 0.00% MLMG::MLResNormInf() 11 4.745e-06 4.745e-06 4.745e-06 0.00% Gravity::set_mass_offset() 11 4.341e-06 4.341e-06 4.341e-06 0.00% Castro::retry_advance_ctu() 10 3.92e-06 3.92e-06 3.92e-06 0.00% Castro::FluxRegFineAdd() 10 3.698e-06 3.698e-06 3.698e-06 0.00% Castro::post_init() 1 3.579e-06 3.579e-06 3.579e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.131e-06 3.131e-06 3.131e-06 0.00% Amr::init() 1 2.453e-06 2.453e-06 2.453e-06 0.00% Castro::computeInitialDt() 2 2.431e-06 2.431e-06 2.431e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.214e-06 2.214e-06 2.214e-06 0.00% AmrLevel::checkPointPre() 3 1.928e-06 1.928e-06 1.928e-06 0.00% Castro::post_regrid() 1 1.134e-06 1.134e-06 1.134e-06 0.00% Amr::initialInit() 1 1.007e-06 1.007e-06 1.007e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8049 0.8049 0.8049 100.00% Amr::coarseTimeStep() 10 0.6637 0.6637 0.6637 82.45% Amr::timeStep() 10 0.5906 0.5906 0.5906 73.37% Castro::advance() 10 0.5851 0.5851 0.5851 72.68% Castro::subcycle_advance_ctu() 10 0.5713 0.5713 0.5713 70.98% Castro::do_advance_ctu() 10 0.5711 0.5711 0.5711 70.95% Gravity::solve_phi_with_mlmg() 11 0.3156 0.3156 0.3156 39.21% Gravity::actual_solve_with_mlmg() 11 0.3036 0.3036 0.3036 37.72% Castro::construct_new_gravity() 10 0.2962 0.2962 0.2962 36.80% MLMG::solve() 11 0.2813 0.2813 0.2813 34.94% Gravity::solve_for_phi() 10 0.2715 0.2715 0.2715 33.73% MLMG::oneIter() 81 0.2659 0.2659 0.2659 33.03% MLMG::mgVcycle() 81 0.2643 0.2643 0.2643 32.83% Castro::construct_ctu_hydro_source() 10 0.2001 0.2001 0.2001 24.86% VisMF::Write(FabArray) 11 0.1462 0.1462 0.1462 18.17% MLCellLinOp::smooth() 1620 0.1398 0.1398 0.1398 17.37% Amr::init() 1 0.1185 0.1185 0.1185 14.72% MLCellLinOp::applyBC() 4379 0.1089 0.1089 0.1089 13.52% Amr::checkPoint() 3 0.1085 0.1085 0.1085 13.48% AmrLevel::checkPoint() 3 0.1047 0.1047 0.1047 13.00% StateData::checkPoint() 12 0.1046 0.1046 0.1046 12.99% MLMG::mgVcycle_bottom 81 0.07535 0.07535 0.07535 9.36% MLMG::actualBottomSolve() 81 0.07532 0.07532 0.07532 9.36% MLCGSolver::bicgstab 81 0.0746 0.0746 0.0746 9.27% MLPoisson::Fsmooth() 3240 0.05787 0.05787 0.05787 7.19% Amr::initialInit() 1 0.0574 0.0574 0.0574 7.13% FillPatchIterator::Initialize 41 0.05624 0.05624 0.05624 6.99% FillPatchSingleLevel 41 0.05024 0.05024 0.05024 6.24% Amr::FinalizeInit() 1 0.04786 0.04786 0.04786 5.95% Castro::post_init() 1 0.04713 0.04713 0.04713 5.86% StateDataPhysBCFunct::() 41 0.04595 0.04595 0.04595 5.71% Amr::writePlotFile() 2 0.04488 0.04488 0.04488 5.58% Gravity::multilevel_solve_for_new_phi() 1 0.04482 0.04482 0.04482 5.57% Gravity::actual_multilevel_solve() 1 0.0448 0.0448 0.0448 5.57% MLCellLinOp::apply() 1128 0.03831 0.03831 0.03831 4.76% MLMG::mgVcycle_down::0 81 0.03676 0.03676 0.03676 4.57% MLMG::mgVcycle_up::0 81 0.03155 0.03155 0.03155 3.92% Castro::clean_state() 62 0.03001 0.03001 0.03001 3.73% Gravity::get_new_grav_vector() 11 0.02657 0.02657 0.02657 3.30% FabArray::setVal() 1135 0.02647 0.02647 0.02647 3.29% Castro::initialize_do_advance() 10 0.02463 0.02463 0.02463 3.06% StateData::FillBoundary(geom) 328 0.02311 0.02311 0.02311 2.87% MLCellLinOp::correctionResidual() 486 0.02159 0.02159 0.02159 2.68% MultiFab::Dot() 1100 0.02083 0.02083 0.02083 2.59% Castro::expand_state() 10 0.01902 0.01902 0.01902 2.36% MLMG:computeResOfCorrection() 405 0.01869 0.01869 0.01869 2.32% FabArray::FillBoundary() 3974 0.0183 0.0183 0.0183 2.27% MLMG::mgVcycle_down::1 81 0.01794 0.01794 0.01794 2.23% FillBoundary_nowait() 3974 0.01747 0.01747 0.01747 2.17% MLMG::mgVcycle_down::2 81 0.01741 0.01741 0.01741 2.16% MLPoisson::define() 11 0.01694 0.01694 0.01694 2.10% FabArray::ParallelCopy() 851 0.0169 0.0169 0.0169 2.10% FabArray::ParallelCopy_nowait() 851 0.01662 0.01662 0.01662 2.06% MLMG::mgVcycle_down::3 81 0.0166 0.0166 0.0166 2.06% Castro::computeTemp() 63 0.01602 0.01602 0.01602 1.99% MLMG::mgVcycle_down::4 81 0.01583 0.01583 0.01583 1.97% Castro::construct_old_gravity() 10 0.01577 0.01577 0.01577 1.96% Gravity::get_old_grav_vector() 10 0.01576 0.01576 0.01576 1.96% Castro::initialize_advance() 10 0.01356 0.01356 0.01356 1.68% MLMG::mgVcycle_up::2 81 0.01355 0.01355 0.01355 1.68% MLMG::mgVcycle_up::1 81 0.0133 0.0133 0.0133 1.65% MLMG::addInterpCorrection() 405 0.01316 0.01316 0.01316 1.63% MLMG::mgVcycle_up::3 81 0.01287 0.01287 0.01287 1.60% MLMG::mgVcycle_up::4 81 0.0127 0.0127 0.0127 1.58% amrex::average_down 405 0.01264 0.01264 0.01264 1.57% MLCGSolver::ParallelAllReduce 1495 0.01255 0.01255 0.01255 1.56% CGSolver::sxay() 1566 0.01249 0.01249 0.01249 1.55% MultiFab::LinComb() 1566 0.01199 0.01199 0.01199 1.49% Gravity::fill_multipole_BCs() 11 0.01181 0.01181 0.01181 1.47% MLCellLinOp::defineAuxData() 11 0.01163 0.01163 0.01163 1.44% MLPoisson::Fapply() 1128 0.01024 0.01024 0.01024 1.27% Castro::do_new_sources() 10 0.01005 0.01005 0.01005 1.25% Amr::InitializeInit() 1 0.009546 0.009546 0.009546 1.19% Amr::defBaseLevel() 1 0.00954 0.00954 0.00954 1.19% Castro::enforce_min_density() 62 0.008676 0.008676 0.008676 1.08% Castro::do_old_sources() 10 0.007817 0.007817 0.007817 0.97% MLCellLinOp::solutionResidual() 92 0.007498 0.007498 0.007498 0.93% MLMG::computeResidual() 81 0.006473 0.006473 0.006473 0.80% MultiFab::Xpay() 578 0.005863 0.005863 0.005863 0.73% FabArray::setDomainBndry() 41 0.005753 0.005753 0.005753 0.71% MLMG::prepareForSolve() 11 0.005713 0.005713 0.005713 0.71% Castro::post_timestep() 10 0.005304 0.005304 0.005304 0.66% Castro::reset_internal_energy() 63 0.005275 0.005275 0.005275 0.66% MLCellLinOp::defineBC() 11 0.005025 0.005025 0.005025 0.62% BndryData::define() 11 0.004846 0.004846 0.004846 0.60% Castro::estTimeStep() 21 0.004671 0.004671 0.004671 0.58% Castro::initData() 1 0.003463 0.003463 0.003463 0.43% Castro::enforce_speed_limit() 62 0.003325 0.003325 0.003325 0.41% Castro::construct_new_source() 50 0.003111 0.003111 0.003111 0.39% Castro::construct_new_gravity_source() 10 0.00307 0.00307 0.00307 0.38% Castro::normalize_species() 62 0.002609 0.002609 0.002609 0.32% Castro::construct_old_source() 50 0.002574 0.002574 0.002574 0.32% Castro::construct_old_gravity_source() 10 0.002535 0.002535 0.002535 0.31% MLMG::getGradSolution() 11 0.002517 0.002517 0.002517 0.31% MLCellLinOp::compGrad() 11 0.002511 0.002511 0.002511 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002238 0.002238 0.002238 0.28% Castro::computeNewDt() 9 0.002029 0.002029 0.002029 0.25% MLMG::ResNormInf() 92 0.001849 0.001849 0.001849 0.23% Castro::apply_source_to_state() 20 0.001824 0.001824 0.001824 0.23% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.23% FabArrayBase::getCPC() 1313 0.001528 0.001528 0.001528 0.19% MLCellLinOp::setLevelBC() 11 0.001519 0.001519 0.001519 0.19% MLPoisson::prepareForSolve() 11 0.001328 0.001328 0.001328 0.16% MLCellLinOp::prepareForSolve() 11 0.001319 0.001319 0.001319 0.16% FabArray::mult() 43 0.001316 0.001316 0.001316 0.16% Gravity::swapTimeLevels() 10 0.001077 0.001077 0.001077 0.13% MLMG::computeMLResidual() 11 0.001067 0.001067 0.001067 0.13% Gravity::update_max_rhs() 11 0.0009496 0.0009496 0.0009496 0.12% FabArrayBase::getFB() 3974 0.0007801 0.0007801 0.0007801 0.10% Castro::computeInitialDt() 2 0.000758 0.000758 0.000758 0.09% FabArrayBase::CPC::define() 454 0.0007038 0.0007038 0.0007038 0.09% MultiFab::Copy() 11 0.0005083 0.0005083 0.0005083 0.06% Castro::post_regrid() 1 0.0004869 0.0004869 0.0004869 0.06% Amr::InitAmr() 1 0.0004422 0.0004422 0.0004422 0.05% MultiFab::min() 10 0.0003162 0.0003162 0.0003162 0.04% MLLinOp::define() 11 0.0002615 0.0002615 0.0002615 0.03% MLMG::MLResNormInf() 11 0.0002506 0.0002506 0.0002506 0.03% MultiFab::max() 11 0.0002505 0.0002505 0.0002505 0.03% MLLinOp::defineGrids() 11 0.0002339 0.0002339 0.0002339 0.03% MLMG::MLRhsNormInf() 11 0.0001939 0.0001939 0.0001939 0.02% FabArrayBase::FB::FB() 56 8.517e-05 8.517e-05 8.517e-05 0.01% Castro::finalize_advance() 10 6.272e-05 6.272e-05 6.272e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 6.046e-05 6.046e-05 6.046e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.271e-05 5.271e-05 5.271e-05 0.01% makeSFC 55 4.418e-05 4.418e-05 4.418e-05 0.01% StateData::define() 4 4.246e-05 4.246e-05 4.246e-05 0.01% Castro::finalize_do_advance() 10 4.029e-05 4.029e-05 4.029e-05 0.01% Castro::swap_state_time_levels() 10 3.784e-05 3.784e-05 3.784e-05 0.00% Castro::enforce_consistent_e() 1 3.445e-05 3.445e-05 3.445e-05 0.00% Amr::writeSmallPlotFile() 1 2.511e-05 2.511e-05 2.511e-05 0.00% DistributionMapping::Distribute() 56 1.582e-05 1.582e-05 1.582e-05 0.00% Amr::initSubcycle() 1 9.343e-06 9.343e-06 9.343e-06 0.00% MLMG::buildFineMask() 11 6.317e-06 6.317e-06 6.317e-06 0.00% Castro::create_source_corrector() 10 5.714e-06 5.714e-06 5.714e-06 0.00% AmrLevel::checkPointPost() 3 5.202e-06 5.202e-06 5.202e-06 0.00% Gravity::set_mass_offset() 11 4.341e-06 4.341e-06 4.341e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.315e-06 4.315e-06 4.315e-06 0.00% Castro::retry_advance_ctu() 10 3.92e-06 3.92e-06 3.92e-06 0.00% Castro::FluxRegFineAdd() 10 3.698e-06 3.698e-06 3.698e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.214e-06 2.214e-06 2.214e-06 0.00% AmrLevel::checkPointPre() 3 1.928e-06 1.928e-06 1.928e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.03-12-g806f2373b68e) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.03-12-g806f2373b68e) initialized Starting run at 08:18:11 UTC on 2022-03-15. Successfully read inputs file ... Castro git describe: 22.02-5-g5116b7268 AMReX git describe: 22.03-12-g806f2373b Microphysics git describe: 22.03-6-g41c4af20 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.500506506 Restart time = 0.055104942 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.114418678 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048623056 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.04742197 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.049008974 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.061243816 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.067841857 seconds Ending run at 08:18:11 UTC on 2022-03-15. Run time = 0.444558708 Run time without initialization = 0.388904867 Average number of zones advanced per microsecond: 3.370 Average number of zones advanced per microsecond per rank: 3.370 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4446 ... 0.4446 ... 0.4446 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1171 0.1171 0.1171 26.34% VisMF::Write(FabArray) 1 0.06654 0.06654 0.06654 14.97% VisMF::Read() 3 0.04363 0.04363 0.04363 9.81% MLCellLinOp::applyBC() 1946 0.03781 0.03781 0.03781 8.51% MLPoisson::Fsmooth() 1440 0.02539 0.02539 0.02539 5.71% FabArray::setVal() 537 0.01628 0.01628 0.01628 3.66% StateData::FillBoundary(geom) 160 0.01136 0.01136 0.01136 2.55% MLCGSolver::bicgstab 36 0.009169 0.009169 0.009169 2.06% MultiFab::Dot() 484 0.009099 0.009099 0.009099 2.05% FillBoundary_nowait() 1766 0.007102 0.007102 0.007102 1.60% Castro::computeTemp() 30 0.007074 0.007074 0.007074 1.59% FabArray::ParallelCopy_nowait() 380 0.007063 0.007063 0.007063 1.59% StateDataPhysBCFunct::() 20 0.006507 0.006507 0.006507 1.46% Gravity::fill_multipole_BCs() 6 0.005914 0.005914 0.005914 1.33% MLCellLinOp::defineAuxData() 6 0.005373 0.005373 0.005373 1.21% MultiFab::LinComb() 690 0.005206 0.005206 0.005206 1.17% MLPoisson::Fapply() 500 0.004533 0.004533 0.004533 1.02% Castro::expand_state() 5 0.004246 0.004246 0.004246 0.96% FabArray::setDomainBndry() 20 0.003877 0.003877 0.003877 0.87% Castro::enforce_min_density() 30 0.003501 0.003501 0.003501 0.79% MLMG::addInterpCorrection() 180 0.002999 0.002999 0.002999 0.67% Amr::restart() 1 0.002977 0.002977 0.002977 0.67% amrex::average_down 180 0.002803 0.002803 0.002803 0.63% Castro::do_advance_ctu() 5 0.002648 0.002648 0.002648 0.60% MultiFab::Xpay() 258 0.002608 0.002608 0.002608 0.59% Castro::estTimeStep() 10 0.00248 0.00248 0.00248 0.56% Castro::reset_internal_energy() 30 0.002391 0.002391 0.002391 0.54% Castro::enforce_speed_limit() 30 0.002026 0.002026 0.002026 0.46% BndryData::define() 6 0.001954 0.001954 0.001954 0.44% Gravity::get_new_grav_vector() 5 0.001881 0.001881 0.001881 0.42% Castro::construct_new_gravity_source() 5 0.001624 0.001624 0.001624 0.37% Castro::normalize_species() 30 0.001443 0.001443 0.001443 0.32% Amr::writePlotFile() 1 0.001384 0.001384 0.001384 0.31% MLCellLinOp::compGrad() 6 0.001358 0.001358 0.001358 0.31% Castro::construct_old_gravity_source() 5 0.001307 0.001307 0.001307 0.29% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.00119 0.00119 0.00119 0.27% Gravity::get_old_grav_vector() 5 0.001043 0.001043 0.001043 0.23% MultiFab::Saxpy() 10 0.0009132 0.0009132 0.0009132 0.21% MLMG::ResNormInf() 42 0.0008317 0.0008317 0.0008317 0.19% MLCellLinOp::setLevelBC() 6 0.0008216 0.0008216 0.0008216 0.18% Gravity::actual_solve_with_mlmg() 6 0.000786 0.000786 0.000786 0.18% MLMG::oneIter() 36 0.0007199 0.0007199 0.0007199 0.16% MLCellLinOp::prepareForSolve() 6 0.0007006 0.0007006 0.0007006 0.16% FabArray::mult() 22 0.0006508 0.0006508 0.0006508 0.15% Gravity::update_max_rhs() 6 0.0005167 0.0005167 0.0005167 0.12% MLCellLinOp::smooth() 720 0.0004996 0.0004996 0.0004996 0.11% FabArrayBase::CPC::define() 244 0.000429 0.000429 0.000429 0.10% MLMG::prepareForSolve() 6 0.0004258 0.0004258 0.0004258 0.10% FabArrayBase::getCPC() 632 0.0004151 0.0004151 0.0004151 0.09% FabArray::FillBoundary() 1766 0.0003915 0.0003915 0.0003915 0.09% Amr::InitAmr() 1 0.0003767 0.0003767 0.0003767 0.08% MultiFab::Copy() 6 0.0003752 0.0003752 0.0003752 0.08% FabArrayBase::getFB() 1766 0.000302 0.000302 0.000302 0.07% main() 1 0.0002658 0.0002658 0.0002658 0.06% Castro::construct_new_gravity() 5 0.0002617 0.0002617 0.0002617 0.06% MLCellLinOp::apply() 500 0.0002239 0.0002239 0.0002239 0.05% Gravity::solve_for_phi() 5 0.0002183 0.0002183 0.0002183 0.05% CGSolver::sxay() 690 0.0002152 0.0002152 0.0002152 0.05% Gravity::actual_multilevel_solve() 1 0.0002099 0.0002099 0.0002099 0.05% MLMG::mgVcycle() 36 0.0001844 0.0001844 0.0001844 0.04% MultiFab::min() 5 0.0001623 0.0001623 0.0001623 0.04% MLCGSolver::ParallelAllReduce 659 0.0001496 0.0001496 0.0001496 0.03% FabArray::ParallelCopy() 380 0.0001438 0.0001438 0.0001438 0.03% MultiFab::max() 6 0.0001324 0.0001324 0.0001324 0.03% FillPatchIterator::Initialize 20 0.000129 0.000129 0.000129 0.03% Amr::coarseTimeStep() 5 0.0001137 0.0001137 0.0001137 0.03% Castro::advance() 5 0.0001129 0.0001129 0.0001129 0.03% Castro::construct_new_source() 25 0.0001115 0.0001115 0.0001115 0.03% MLMG::MLRhsNormInf() 6 0.000105 0.000105 0.000105 0.02% Amr::timeStep() 5 0.0001041 0.0001041 0.0001041 0.02% MLCellLinOp::defineBC() 6 0.0001031 0.0001031 0.0001031 0.02% MLCellLinOp::correctionResidual() 216 9.906e-05 9.906e-05 9.906e-05 0.02% Castro::subcycle_advance_ctu() 5 9.436e-05 9.436e-05 9.436e-05 0.02% MLLinOp::defineGrids() 6 9.155e-05 9.155e-05 9.155e-05 0.02% StateData::restartDoit() 4 8.505e-05 8.505e-05 8.505e-05 0.02% AmrLevel::restart() 1 7.503e-05 7.503e-05 7.503e-05 0.02% MLMG:computeResOfCorrection() 180 6.12e-05 6.12e-05 6.12e-05 0.01% FabArrayBase::FB::FB() 26 6.018e-05 6.018e-05 6.018e-05 0.01% Castro::create_source_corrector() 5 5.981e-05 5.981e-05 5.981e-05 0.01% Castro::post_timestep() 5 5.949e-05 5.949e-05 5.949e-05 0.01% MLMG::actualBottomSolve() 36 5.222e-05 5.222e-05 5.222e-05 0.01% MLMG::mgVcycle_down::0 36 4.097e-05 4.097e-05 4.097e-05 0.01% MLMG::mgVcycle_down::1 36 3.913e-05 3.913e-05 3.913e-05 0.01% Castro::clean_state() 30 3.768e-05 3.768e-05 3.768e-05 0.01% Castro::initialize_advance() 5 3.767e-05 3.767e-05 3.767e-05 0.01% MLMG::solve() 6 3.701e-05 3.701e-05 3.701e-05 0.01% MLMG::mgVcycle_down::2 36 3.474e-05 3.474e-05 3.474e-05 0.01% MLMG::mgVcycle_down::4 36 3.448e-05 3.448e-05 3.448e-05 0.01% MLMG::mgVcycle_down::3 36 3.343e-05 3.343e-05 3.343e-05 0.01% Castro::initialize_do_advance() 5 3.045e-05 3.045e-05 3.045e-05 0.01% MLLinOp::define() 6 2.776e-05 2.776e-05 2.776e-05 0.01% Castro::finalize_advance() 5 2.673e-05 2.673e-05 2.673e-05 0.01% Castro::swap_state_time_levels() 5 2.63e-05 2.63e-05 2.63e-05 0.01% MLMG::mgVcycle_up::4 36 2.619e-05 2.619e-05 2.619e-05 0.01% MLMG::mgVcycle_up::0 36 2.563e-05 2.563e-05 2.563e-05 0.01% Amr::writeSmallPlotFile() 1 2.481e-05 2.481e-05 2.481e-05 0.01% MLMG::mgVcycle_up::3 36 2.256e-05 2.256e-05 2.256e-05 0.01% MLCellLinOp::solutionResidual() 42 2.255e-05 2.255e-05 2.255e-05 0.01% Castro::post_restart() 1 2.231e-05 2.231e-05 2.231e-05 0.01% MLMG::mgVcycle_up::2 36 2.161e-05 2.161e-05 2.161e-05 0.00% MLMG::mgVcycle_up::1 36 2.147e-05 2.147e-05 2.147e-05 0.00% Castro::finalize_do_advance() 5 2.023e-05 2.023e-05 2.023e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.974e-05 1.974e-05 1.974e-05 0.00% FillPatchSingleLevel 20 1.578e-05 1.578e-05 1.578e-05 0.00% makeSFC 30 1.575e-05 1.575e-05 1.575e-05 0.00% MLMG::mgVcycle_bottom 36 1.558e-05 1.558e-05 1.558e-05 0.00% MLPoisson::define() 6 1.541e-05 1.541e-05 1.541e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.529e-05 1.529e-05 1.529e-05 0.00% MLMG::computeResidual() 36 1.473e-05 1.473e-05 1.473e-05 0.00% Castro::construct_old_source() 25 1.058e-05 1.058e-05 1.058e-05 0.00% Castro::do_new_sources() 5 9.735e-06 9.735e-06 9.735e-06 0.00% DistributionMapping::Distribute() 31 9.506e-06 9.506e-06 9.506e-06 0.00% Amr::initSubcycle() 1 9.1e-06 9.1e-06 9.1e-06 0.00% Castro::do_old_sources() 5 8.104e-06 8.104e-06 8.104e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.789e-06 7.789e-06 7.789e-06 0.00% Castro::apply_source_to_state() 10 5.998e-06 5.998e-06 5.998e-06 0.00% Castro::construct_old_gravity() 5 5.788e-06 5.788e-06 5.788e-06 0.00% MLPoisson::prepareForSolve() 6 4.883e-06 4.883e-06 4.883e-06 0.00% Gravity::swapTimeLevels() 5 4.368e-06 4.368e-06 4.368e-06 0.00% MLMG::buildFineMask() 6 3.708e-06 3.708e-06 3.708e-06 0.00% Castro::computeNewDt() 5 3.631e-06 3.631e-06 3.631e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.595e-06 3.595e-06 3.595e-06 0.00% MLMG::computeMLResidual() 6 3.001e-06 3.001e-06 3.001e-06 0.00% MLMG::getGradSolution() 6 2.98e-06 2.98e-06 2.98e-06 0.00% MLMG::MLResNormInf() 6 2.704e-06 2.704e-06 2.704e-06 0.00% Gravity::set_mass_offset() 6 2.328e-06 2.328e-06 2.328e-06 0.00% Castro::retry_advance_ctu() 5 2.03e-06 2.03e-06 2.03e-06 0.00% Castro::FluxRegFineAdd() 5 1.644e-06 1.644e-06 1.644e-06 0.00% Amr::init() 1 1.183e-06 1.183e-06 1.183e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.146e-06 1.146e-06 1.146e-06 0.00% AmrLevel::AmrLevel() 1 7.88e-07 7.88e-07 7.88e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4446 0.4446 0.4446 100.00% Amr::coarseTimeStep() 5 0.3208 0.3208 0.3208 72.16% Amr::timeStep() 5 0.3192 0.3192 0.3192 71.79% Castro::advance() 5 0.3152 0.3152 0.3152 70.89% Castro::subcycle_advance_ctu() 5 0.3045 0.3045 0.3045 68.50% Castro::do_advance_ctu() 5 0.3044 0.3044 0.3044 68.47% Castro::construct_new_gravity() 5 0.1446 0.1446 0.1446 32.53% Gravity::solve_phi_with_mlmg() 6 0.1408 0.1408 0.1408 31.67% Gravity::solve_for_phi() 5 0.1352 0.1352 0.1352 30.42% Gravity::actual_solve_with_mlmg() 6 0.1348 0.1348 0.1348 30.31% MLMG::solve() 6 0.1223 0.1223 0.1223 27.50% Castro::construct_ctu_hydro_source() 5 0.1171 0.1171 0.1171 26.34% MLMG::oneIter() 36 0.1144 0.1144 0.1144 25.74% MLMG::mgVcycle() 36 0.1137 0.1137 0.1137 25.58% Amr::writePlotFile() 1 0.06793 0.06793 0.06793 15.28% VisMF::Write(FabArray) 1 0.06654 0.06654 0.06654 14.97% MLCellLinOp::smooth() 720 0.05956 0.05956 0.05956 13.40% Amr::init() 1 0.05515 0.05515 0.05515 12.40% Amr::restart() 1 0.05514 0.05514 0.05514 12.40% MLCellLinOp::applyBC() 1946 0.04567 0.04567 0.04567 10.27% AmrLevel::restart() 1 0.04457 0.04457 0.04457 10.03% StateData::restartDoit() 4 0.04449 0.04449 0.04449 10.01% VisMF::Read() 3 0.04363 0.04363 0.04363 9.81% MLMG::mgVcycle_bottom 36 0.03274 0.03274 0.03274 7.36% MLMG::actualBottomSolve() 36 0.03272 0.03272 0.03272 7.36% MLCGSolver::bicgstab 36 0.0324 0.0324 0.0324 7.29% MLPoisson::Fsmooth() 1440 0.02539 0.02539 0.02539 5.71% FillPatchIterator::Initialize 20 0.02396 0.02396 0.02396 5.39% FillPatchSingleLevel 20 0.01995 0.01995 0.01995 4.49% StateDataPhysBCFunct::() 20 0.01787 0.01787 0.01787 4.02% Castro::clean_state() 30 0.01647 0.01647 0.01647 3.71% MLCellLinOp::apply() 500 0.01646 0.01646 0.01646 3.70% FabArray::setVal() 537 0.01628 0.01628 0.01628 3.66% MLMG::mgVcycle_down::0 36 0.01583 0.01583 0.01583 3.56% Castro::initialize_do_advance() 5 0.01451 0.01451 0.01451 3.26% MLMG::mgVcycle_up::0 36 0.01353 0.01353 0.01353 3.04% Castro::expand_state() 5 0.01222 0.01222 0.01222 2.75% StateData::FillBoundary(geom) 160 0.01136 0.01136 0.01136 2.55% Castro::initialize_advance() 5 0.0105 0.0105 0.0105 2.36% Castro::computeTemp() 30 0.009465 0.009465 0.009465 2.13% MLCellLinOp::correctionResidual() 216 0.009272 0.009272 0.009272 2.09% MLPoisson::define() 6 0.009216 0.009216 0.009216 2.07% Gravity::get_new_grav_vector() 5 0.009139 0.009139 0.009139 2.06% MultiFab::Dot() 484 0.009099 0.009099 0.009099 2.05% MLMG:computeResOfCorrection() 180 0.008029 0.008029 0.008029 1.81% FabArray::FillBoundary() 1766 0.007855 0.007855 0.007855 1.77% Castro::construct_old_gravity() 5 0.007745 0.007745 0.007745 1.74% Gravity::get_old_grav_vector() 5 0.00774 0.00774 0.00774 1.74% MLMG::mgVcycle_down::1 36 0.007704 0.007704 0.007704 1.73% FabArray::ParallelCopy() 380 0.007622 0.007622 0.007622 1.71% FabArray::ParallelCopy_nowait() 380 0.007478 0.007478 0.007478 1.68% FillBoundary_nowait() 1766 0.007464 0.007464 0.007464 1.68% MLMG::mgVcycle_down::2 36 0.007425 0.007425 0.007425 1.67% MLMG::mgVcycle_down::3 36 0.007114 0.007114 0.007114 1.60% Castro::do_new_sources() 5 0.006866 0.006866 0.006866 1.54% MLMG::mgVcycle_down::4 36 0.006788 0.006788 0.006788 1.53% Castro::post_restart() 1 0.006384 0.006384 0.006384 1.44% MLCellLinOp::defineAuxData() 6 0.006288 0.006288 0.006288 1.41% Gravity::multilevel_solve_for_new_phi() 1 0.006008 0.006008 0.006008 1.35% Gravity::actual_multilevel_solve() 1 0.005989 0.005989 0.005989 1.35% Gravity::fill_multipole_BCs() 6 0.005914 0.005914 0.005914 1.33% MLMG::mgVcycle_up::2 36 0.005791 0.005791 0.005791 1.30% MLMG::addInterpCorrection() 180 0.005773 0.005773 0.005773 1.30% MLMG::mgVcycle_up::1 36 0.005691 0.005691 0.005691 1.28% amrex::average_down 180 0.00558 0.00558 0.00558 1.26% MLCGSolver::ParallelAllReduce 659 0.005514 0.005514 0.005514 1.24% MLMG::mgVcycle_up::3 36 0.00551 0.00551 0.00551 1.24% CGSolver::sxay() 690 0.005421 0.005421 0.005421 1.22% MLMG::mgVcycle_up::4 36 0.005418 0.005418 0.005418 1.22% MultiFab::LinComb() 690 0.005206 0.005206 0.005206 1.17% MLPoisson::Fapply() 500 0.004533 0.004533 0.004533 1.02% Castro::do_old_sources() 5 0.004244 0.004244 0.004244 0.95% Castro::post_timestep() 5 0.003914 0.003914 0.003914 0.88% FabArray::setDomainBndry() 20 0.003877 0.003877 0.003877 0.87% Castro::enforce_min_density() 30 0.003501 0.003501 0.003501 0.79% MLMG::prepareForSolve() 6 0.003458 0.003458 0.003458 0.78% MLCellLinOp::solutionResidual() 42 0.003382 0.003382 0.003382 0.76% MLMG::computeResidual() 36 0.002805 0.002805 0.002805 0.63% MLCellLinOp::defineBC() 6 0.00276 0.00276 0.00276 0.62% BndryData::define() 6 0.002657 0.002657 0.002657 0.60% MultiFab::Xpay() 258 0.002608 0.002608 0.002608 0.59% Castro::estTimeStep() 10 0.00248 0.00248 0.00248 0.56% Castro::reset_internal_energy() 30 0.002391 0.002391 0.002391 0.54% Castro::enforce_speed_limit() 30 0.002026 0.002026 0.002026 0.46% Castro::construct_new_source() 25 0.001736 0.001736 0.001736 0.39% MLMG::getGradSolution() 6 0.001651 0.001651 0.001651 0.37% MLCellLinOp::compGrad() 6 0.001648 0.001648 0.001648 0.37% Castro::construct_new_gravity_source() 5 0.001624 0.001624 0.001624 0.37% Castro::computeNewDt() 5 0.00152 0.00152 0.00152 0.34% Castro::normalize_species() 30 0.001443 0.001443 0.001443 0.32% Castro::construct_old_source() 25 0.001318 0.001318 0.001318 0.30% Castro::construct_old_gravity_source() 5 0.001307 0.001307 0.001307 0.29% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.00119 0.00119 0.00119 0.27% Castro::apply_source_to_state() 10 0.0009192 0.0009192 0.0009192 0.21% MultiFab::Saxpy() 10 0.0009132 0.0009132 0.0009132 0.21% Gravity::swapTimeLevels() 5 0.0008778 0.0008778 0.0008778 0.20% FabArrayBase::getCPC() 632 0.0008441 0.0008441 0.0008441 0.19% MLMG::ResNormInf() 42 0.0008317 0.0008317 0.0008317 0.19% MLCellLinOp::setLevelBC() 6 0.0008216 0.0008216 0.0008216 0.18% Gravity::update_max_rhs() 6 0.0007266 0.0007266 0.0007266 0.16% MLPoisson::prepareForSolve() 6 0.0007055 0.0007055 0.0007055 0.16% MLCellLinOp::prepareForSolve() 6 0.0007006 0.0007006 0.0007006 0.16% FabArray::mult() 22 0.0006508 0.0006508 0.0006508 0.15% MLMG::computeMLResidual() 6 0.0005952 0.0005952 0.0005952 0.13% FabArrayBase::CPC::define() 244 0.000429 0.000429 0.000429 0.10% Amr::InitAmr() 1 0.0003858 0.0003858 0.0003858 0.09% MultiFab::Copy() 6 0.0003752 0.0003752 0.0003752 0.08% FabArrayBase::getFB() 1766 0.0003622 0.0003622 0.0003622 0.08% MultiFab::min() 5 0.0001623 0.0001623 0.0001623 0.04% MLLinOp::define() 6 0.0001521 0.0001521 0.0001521 0.03% MLMG::MLResNormInf() 6 0.0001341 0.0001341 0.0001341 0.03% MultiFab::max() 6 0.0001324 0.0001324 0.0001324 0.03% MLLinOp::defineGrids() 6 0.0001244 0.0001244 0.0001244 0.03% MLMG::MLRhsNormInf() 6 0.000105 0.000105 0.000105 0.02% FabArrayBase::FB::FB() 26 6.018e-05 6.018e-05 6.018e-05 0.01% Castro::create_source_corrector() 5 5.981e-05 5.981e-05 5.981e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.167e-05 3.167e-05 3.167e-05 0.01% Castro::finalize_advance() 5 2.838e-05 2.838e-05 2.838e-05 0.01% Castro::swap_state_time_levels() 5 2.63e-05 2.63e-05 2.63e-05 0.01% Amr::writeSmallPlotFile() 1 2.481e-05 2.481e-05 2.481e-05 0.01% makeSFC 30 2.388e-05 2.388e-05 2.388e-05 0.01% Castro::finalize_do_advance() 5 2.023e-05 2.023e-05 2.023e-05 0.00% DistributionMapping::Distribute() 31 9.506e-06 9.506e-06 9.506e-06 0.00% Amr::initSubcycle() 1 9.1e-06 9.1e-06 9.1e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.971e-06 4.971e-06 4.971e-06 0.00% MLMG::buildFineMask() 6 3.708e-06 3.708e-06 3.708e-06 0.00% Gravity::set_mass_offset() 6 2.328e-06 2.328e-06 2.328e-06 0.00% Castro::retry_advance_ctu() 5 2.03e-06 2.03e-06 2.03e-06 0.00% Castro::FluxRegFineAdd() 5 1.644e-06 1.644e-06 1.644e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.146e-06 1.146e-06 1.146e-06 0.00% AmrLevel::AmrLevel() 1 7.88e-07 7.88e-07 7.88e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.03-12-g806f2373b68e) finalized