Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.07-9-g40b3d2176b17) initialized Starting run at 08:44:07 UTC on 2022-07-14. Successfully read inputs file ... Castro git describe: 22.07-4-g31cce7d5a AMReX git describe: 22.07-9-g40b3d2176 Microphysics git describe: 22.07-10-g6c6c6fcd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.043341218 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025171817 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.049887194 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050408148 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.053026339 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.052215038 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.076937208 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.040341892 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.078624647 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048311522 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.051499345 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.053480007 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.069454505 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.040366827 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024928698 seconds Ending run at 08:44:08 UTC on 2022-07-14. Run time = 0.811470084 Run time without initialization = 0.690081886 Average number of zones advanced per microsecond: 3.799 Average number of zones advanced per microsecond per rank: 3.799 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8115 ... 0.8115 ... 0.8115 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1902 0.1902 0.1902 23.44% VisMF::Write(FabArray) 11 0.1674 0.1674 0.1674 20.63% MLCellLinOp::applyBC() 4433 0.08197 0.08197 0.08197 10.10% MLPoisson::Fsmooth() 3280 0.06471 0.06471 0.06471 7.97% MLCGSolver::bicgstab 82 0.02435 0.02435 0.02435 3.00% StateData::FillBoundary(geom) 328 0.02421 0.02421 0.02421 2.98% MultiFab::Dot() 1114 0.02271 0.02271 0.02271 2.80% Castro::normalize_species() 62 0.01837 0.01837 0.01837 2.26% MultiFab::LinComb() 1586 0.01462 0.01462 0.01462 1.80% FillBoundary_nowait() 4023 0.01441 0.01441 0.01441 1.78% FabArray::setVal() 1144 0.0144 0.0144 0.0144 1.77% Castro::computeTemp() 63 0.01408 0.01408 0.01408 1.73% FabArray::ParallelCopy_nowait() 861 0.01333 0.01333 0.01333 1.64% MLPoisson::Fapply() 1142 0.01195 0.01195 0.01195 1.47% MLCellLinOp::defineAuxData() 11 0.01158 0.01158 0.01158 1.43% StateDataPhysBCFunct::() 41 0.01139 0.01139 0.01139 1.40% Gravity::fill_multipole_BCs() 11 0.009569 0.009569 0.009569 1.18% Castro::estTimeStep() 21 0.008442 0.008442 0.008442 1.04% Castro::enforce_min_density() 62 0.008142 0.008142 0.008142 1.00% MLMG::addInterpCorrection() 410 0.007553 0.007553 0.007553 0.93% amrex::average_down 410 0.006924 0.006924 0.006924 0.85% MultiFab::Xpay() 585 0.006681 0.006681 0.006681 0.82% Castro::do_advance_ctu() 10 0.004811 0.004811 0.004811 0.59% Amr::checkPoint() 3 0.004362 0.004362 0.004362 0.54% BndryData::define() 11 0.003921 0.003921 0.003921 0.48% Castro::reset_internal_energy(MultiFab) 63 0.003835 0.003835 0.003835 0.47% Castro::construct_new_gravity_source() 10 0.003074 0.003074 0.003074 0.38% Castro::construct_old_gravity_source() 10 0.002812 0.002812 0.002812 0.35% Amr::writePlotFile() 2 0.002504 0.002504 0.002504 0.31% MLMG::ResNormInf() 93 0.001963 0.001963 0.001963 0.24% Gravity::get_new_grav_vector() 11 0.001935 0.001935 0.001935 0.24% MultiFab::Saxpy() 20 0.001821 0.001821 0.001821 0.22% Gravity::get_old_grav_vector() 10 0.001741 0.001741 0.001741 0.21% Castro::expand_state() 10 0.001736 0.001736 0.001736 0.21% MLMG::oneIter() 82 0.001707 0.001707 0.001707 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001669 0.001669 0.001669 0.21% MLCellLinOp::setLevelBC() 11 0.001559 0.001559 0.001559 0.19% Castro::reset_internal_energy(Fab) 504 0.001548 0.001548 0.001548 0.19% MLCellLinOp::smooth() 1640 0.001359 0.001359 0.001359 0.17% Gravity::actual_solve_with_mlmg() 11 0.001358 0.001358 0.001358 0.17% FabArray::mult() 43 0.001315 0.001315 0.001315 0.16% FabArray::setDomainBndry() 41 0.001312 0.001312 0.001312 0.16% Castro::initData() 1 0.001202 0.001202 0.001202 0.15% MultiFab::contains_nan() 20 0.001178 0.001178 0.001178 0.15% MLCellLinOp::prepareForSolve() 11 0.001175 0.001175 0.001175 0.14% MLMG::prepareForSolve() 11 0.001038 0.001038 0.001038 0.13% Castro::enforce_speed_limit() 62 0.001018 0.001018 0.001018 0.13% MLCellLinOp::compGrad() 11 0.0009242 0.0009242 0.0009242 0.11% FabArray::FillBoundary() 4023 0.0008098 0.0008098 0.0008098 0.10% FabArrayBase::getCPC() 1323 0.0007805 0.0007805 0.0007805 0.10% FabArrayBase::CPC::define() 454 0.0006683 0.0006683 0.0006683 0.08% FabArrayBase::getFB() 4023 0.0006375 0.0006375 0.0006375 0.08% MLCellLinOp::apply() 1142 0.0005321 0.0005321 0.0005321 0.07% Amr::InitAmr() 1 0.0005175 0.0005175 0.0005175 0.06% Gravity::solve_for_phi() 10 0.0004433 0.0004433 0.0004433 0.05% CGSolver::sxay() 1586 0.0004188 0.0004188 0.0004188 0.05% Gravity::update_max_rhs() 11 0.0004126 0.0004126 0.0004126 0.05% FillPatchIterator::Initialize 41 0.0003216 0.0003216 0.0003216 0.04% Amr::coarseTimeStep() 10 0.0003026 0.0003026 0.0003026 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003011 0.0003011 0.0003011 0.04% FabArray::ParallelCopy() 861 0.0002855 0.0002855 0.0002855 0.04% MLCellLinOp::defineBC() 11 0.0002741 0.0002741 0.0002741 0.03% MultiFab::Copy() 11 0.000263 0.000263 0.000263 0.03% MultiFab::max() 11 0.0002586 0.0002586 0.0002586 0.03% main() 1 0.0002529 0.0002529 0.0002529 0.03% MLCellLinOp::correctionResidual() 492 0.0002439 0.0002439 0.0002439 0.03% MLMG::mgVcycle() 82 0.0002169 0.0002169 0.0002169 0.03% Castro::construct_new_gravity() 10 0.000205 0.000205 0.000205 0.03% MLMG::MLRhsNormInf() 11 0.0002034 0.0002034 0.0002034 0.03% Amr::timeStep() 10 0.0001951 0.0001951 0.0001951 0.02% MLLinOp::defineGrids() 11 0.000194 0.000194 0.000194 0.02% Castro::subcycle_advance_ctu() 10 0.000188 0.000188 0.000188 0.02% Castro::create_source_corrector() 10 0.0001322 0.0001322 0.0001322 0.02% StateData::checkPoint() 12 0.0001295 0.0001295 0.0001295 0.02% MLMG:computeResOfCorrection() 410 0.0001185 0.0001185 0.0001185 0.01% MLMG::mgVcycle_down::0 82 0.0001145 0.0001145 0.0001145 0.01% MLMG::actualBottomSolve() 82 0.0001033 0.0001033 0.0001033 0.01% MLMG::mgVcycle_down::1 82 9.369e-05 9.369e-05 9.369e-05 0.01% Castro::Castro() 1 9.124e-05 9.124e-05 9.124e-05 0.01% Castro::initialize_advance() 10 8.742e-05 8.742e-05 8.742e-05 0.01% MLMG::mgVcycle_down::2 82 8.588e-05 8.588e-05 8.588e-05 0.01% FabArrayBase::FB::FB() 56 8.419e-05 8.419e-05 8.419e-05 0.01% MLMG::mgVcycle_down::3 82 8.016e-05 8.016e-05 8.016e-05 0.01% MLMG::mgVcycle_down::4 82 8.015e-05 8.015e-05 8.015e-05 0.01% MLMG::solve() 11 8.014e-05 8.014e-05 8.014e-05 0.01% Castro::clean_state() 62 7.571e-05 7.571e-05 7.571e-05 0.01% AmrLevel::checkPoint() 3 7.384e-05 7.384e-05 7.384e-05 0.01% Castro::initialize_do_advance() 10 6.483e-05 6.483e-05 6.483e-05 0.01% MLMG::mgVcycle_up::4 82 6.227e-05 6.227e-05 6.227e-05 0.01% MLMG::mgVcycle_up::1 82 5.487e-05 5.487e-05 5.487e-05 0.01% MLMG::mgVcycle_up::2 82 5.415e-05 5.415e-05 5.415e-05 0.01% MLMG::mgVcycle_up::3 82 5.337e-05 5.337e-05 5.337e-05 0.01% MLCellLinOp::solutionResidual() 93 5.319e-05 5.319e-05 5.319e-05 0.01% MLMG::mgVcycle_up::0 82 5.263e-05 5.263e-05 5.263e-05 0.01% Castro::finalize_advance() 10 5.022e-05 5.022e-05 5.022e-05 0.01% Castro::construct_new_source() 50 4.268e-05 4.268e-05 4.268e-05 0.01% Castro::swap_state_time_levels() 10 3.944e-05 3.944e-05 3.944e-05 0.00% StateData::define() 4 3.922e-05 3.922e-05 3.922e-05 0.00% Castro::advance() 10 3.879e-05 3.879e-05 3.879e-05 0.00% Castro::post_timestep() 10 3.789e-05 3.789e-05 3.789e-05 0.00% Castro::finalize_do_advance() 10 3.614e-05 3.614e-05 3.614e-05 0.00% Castro::enforce_consistent_e() 1 3.431e-05 3.431e-05 3.431e-05 0.00% MLMG::mgVcycle_bottom 82 3.39e-05 3.39e-05 3.39e-05 0.00% Gravity::actual_multilevel_solve() 1 3.181e-05 3.181e-05 3.181e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.143e-05 3.143e-05 3.143e-05 0.00% MLMG::computeResidual() 82 3.113e-05 3.113e-05 3.113e-05 0.00% FillPatchSingleLevel 41 2.967e-05 2.967e-05 2.967e-05 0.00% Castro::initMFs() 1 2.924e-05 2.924e-05 2.924e-05 0.00% makeSFC 55 2.587e-05 2.587e-05 2.587e-05 0.00% Amr::writeSmallPlotFile() 1 2.516e-05 2.516e-05 2.516e-05 0.00% Castro::buildMetrics() 1 2.391e-05 2.391e-05 2.391e-05 0.00% MLPoisson::define() 11 2.299e-05 2.299e-05 2.299e-05 0.00% Amr::defBaseLevel() 1 2.254e-05 2.254e-05 2.254e-05 0.00% MLLinOp::define() 11 2.246e-05 2.246e-05 2.246e-05 0.00% Amr::FinalizeInit() 1 2.23e-05 2.23e-05 2.23e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.958e-05 1.958e-05 1.958e-05 0.00% Castro::construct_old_source() 50 1.849e-05 1.849e-05 1.849e-05 0.00% Castro::do_new_sources() 10 1.791e-05 1.791e-05 1.791e-05 0.00% Castro::do_old_sources() 10 1.725e-05 1.725e-05 1.725e-05 0.00% DistributionMapping::Distribute() 56 1.516e-05 1.516e-05 1.516e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.271e-05 1.271e-05 1.271e-05 0.00% Castro::apply_source_to_state() 20 1.232e-05 1.232e-05 1.232e-05 0.00% Castro::construct_old_gravity() 10 1.154e-05 1.154e-05 1.154e-05 0.00% Castro::check_for_nan() 20 1.098e-05 1.098e-05 1.098e-05 0.00% Gravity::swapTimeLevels() 10 9.699e-06 9.699e-06 9.699e-06 0.00% Amr::initSubcycle() 1 8.876e-06 8.876e-06 8.876e-06 0.00% MLPoisson::prepareForSolve() 11 8.352e-06 8.352e-06 8.352e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.871e-06 7.871e-06 7.871e-06 0.00% MLMG::computeMLResidual() 11 6.859e-06 6.859e-06 6.859e-06 0.00% Castro::computeNewDt() 9 6.508e-06 6.508e-06 6.508e-06 0.00% Amr::InitializeInit() 1 6.437e-06 6.437e-06 6.437e-06 0.00% MLMG::getGradSolution() 11 6.227e-06 6.227e-06 6.227e-06 0.00% MLMG::buildFineMask() 11 5.813e-06 5.813e-06 5.813e-06 0.00% AmrLevel::checkPointPost() 3 4.604e-06 4.604e-06 4.604e-06 0.00% MLMG::MLResNormInf() 11 4.516e-06 4.516e-06 4.516e-06 0.00% Gravity::set_mass_offset() 11 4.169e-06 4.169e-06 4.169e-06 0.00% Castro::post_init() 1 3.916e-06 3.916e-06 3.916e-06 0.00% Castro::retry_advance_ctu() 10 3.637e-06 3.637e-06 3.637e-06 0.00% Castro::computeInitialDt() 2 3.35e-06 3.35e-06 3.35e-06 0.00% Castro::FluxRegCrseInit 10 3.277e-06 3.277e-06 3.277e-06 0.00% Amr::init() 1 3.204e-06 3.204e-06 3.204e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.082e-06 3.082e-06 3.082e-06 0.00% Castro::FluxRegFineAdd() 10 2.462e-06 2.462e-06 2.462e-06 0.00% AmrLevel::checkPointPre() 3 2.044e-06 2.044e-06 2.044e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.774e-06 1.774e-06 1.774e-06 0.00% Castro::post_regrid() 1 1.349e-06 1.349e-06 1.349e-06 0.00% Amr::initialInit() 1 1.176e-06 1.176e-06 1.176e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8115 0.8115 0.8115 100.00% Amr::coarseTimeStep() 10 0.6649 0.6649 0.6649 81.94% Amr::timeStep() 10 0.5779 0.5779 0.5779 71.22% Castro::advance() 10 0.5707 0.5707 0.5707 70.33% Castro::subcycle_advance_ctu() 10 0.5608 0.5608 0.5608 69.11% Castro::do_advance_ctu() 10 0.5606 0.5606 0.5606 69.09% Gravity::solve_phi_with_mlmg() 11 0.3201 0.3201 0.3201 39.45% Gravity::actual_solve_with_mlmg() 11 0.3103 0.3103 0.3103 38.24% Castro::construct_new_gravity() 10 0.2908 0.2908 0.2908 35.83% MLMG::solve() 11 0.2876 0.2876 0.2876 35.45% Gravity::solve_for_phi() 10 0.2756 0.2756 0.2756 33.97% MLMG::oneIter() 82 0.2729 0.2729 0.2729 33.63% MLMG::mgVcycle() 82 0.2712 0.2712 0.2712 33.42% Castro::construct_ctu_hydro_source() 10 0.1902 0.1902 0.1902 23.44% VisMF::Write(FabArray) 11 0.1674 0.1674 0.1674 20.63% MLCellLinOp::smooth() 1640 0.139 0.139 0.139 17.13% Amr::checkPoint() 3 0.1242 0.1242 0.1242 15.31% Amr::init() 1 0.1208 0.1208 0.1208 14.88% AmrLevel::checkPoint() 3 0.1198 0.1198 0.1198 14.77% StateData::checkPoint() 12 0.1198 0.1198 0.1198 14.76% MLCellLinOp::applyBC() 4433 0.09791 0.09791 0.09791 12.07% MLMG::mgVcycle_bottom 82 0.08343 0.08343 0.08343 10.28% MLMG::actualBottomSolve() 82 0.0834 0.0834 0.0834 10.28% MLCGSolver::bicgstab 82 0.08256 0.08256 0.08256 10.17% MLPoisson::Fsmooth() 3280 0.06471 0.06471 0.06471 7.97% Amr::initialInit() 1 0.05209 0.05209 0.05209 6.42% Amr::writePlotFile() 2 0.05024 0.05024 0.05024 6.19% Amr::FinalizeInit() 1 0.04817 0.04817 0.04817 5.94% Castro::post_init() 1 0.04684 0.04684 0.04684 5.77% Castro::clean_state() 62 0.04633 0.04633 0.04633 5.71% Gravity::multilevel_solve_for_new_phi() 1 0.04499 0.04499 0.04499 5.54% Gravity::actual_multilevel_solve() 1 0.04497 0.04497 0.04497 5.54% FillPatchIterator::Initialize 41 0.04119 0.04119 0.04119 5.08% FillPatchSingleLevel 41 0.03956 0.03956 0.03956 4.87% MLCellLinOp::apply() 1142 0.03694 0.03694 0.03694 4.55% MLMG::mgVcycle_down::0 82 0.03601 0.03601 0.03601 4.44% StateDataPhysBCFunct::() 41 0.03559 0.03559 0.03559 4.39% MLMG::mgVcycle_up::0 82 0.03081 0.03081 0.03081 3.80% StateData::FillBoundary(geom) 328 0.02421 0.02421 0.02421 2.98% MultiFab::Dot() 1114 0.02271 0.02271 0.02271 2.80% MLCellLinOp::correctionResidual() 492 0.02168 0.02168 0.02168 2.67% Castro::initialize_do_advance() 10 0.02108 0.02108 0.02108 2.60% Castro::computeTemp() 63 0.01946 0.01946 0.01946 2.40% MLMG:computeResOfCorrection() 410 0.01869 0.01869 0.01869 2.30% Castro::normalize_species() 62 0.01837 0.01837 0.01837 2.26% MLPoisson::define() 11 0.01836 0.01836 0.01836 2.26% MLMG::mgVcycle_down::1 82 0.01802 0.01802 0.01802 2.22% MLMG::mgVcycle_down::2 82 0.01767 0.01767 0.01767 2.18% MLMG::mgVcycle_down::3 82 0.01675 0.01675 0.01675 2.06% Gravity::get_new_grav_vector() 11 0.0167 0.0167 0.0167 2.06% FabArray::FillBoundary() 4023 0.01594 0.01594 0.01594 1.96% MLMG::mgVcycle_down::4 82 0.01592 0.01592 0.01592 1.96% FillBoundary_nowait() 4023 0.01513 0.01513 0.01513 1.86% CGSolver::sxay() 1586 0.01504 0.01504 0.01504 1.85% Castro::construct_old_gravity() 10 0.01483 0.01483 0.01483 1.83% Gravity::get_old_grav_vector() 10 0.01482 0.01482 0.01482 1.83% MultiFab::LinComb() 1586 0.01462 0.01462 0.01462 1.80% FabArray::ParallelCopy() 861 0.01444 0.01444 0.01444 1.78% FabArray::setVal() 1144 0.0144 0.0144 0.0144 1.77% FabArray::ParallelCopy_nowait() 861 0.01416 0.01416 0.01416 1.74% MLCGSolver::ParallelAllReduce 1514 0.01354 0.01354 0.01354 1.67% MLMG::mgVcycle_up::2 82 0.01353 0.01353 0.01353 1.67% Castro::do_new_sources() 10 0.01347 0.01347 0.01347 1.66% MLMG::mgVcycle_up::1 82 0.01331 0.01331 0.01331 1.64% MLCellLinOp::defineAuxData() 11 0.01292 0.01292 0.01292 1.59% MLMG::mgVcycle_up::3 82 0.01286 0.01286 0.01286 1.58% MLMG::addInterpCorrection() 410 0.01278 0.01278 0.01278 1.57% MLMG::mgVcycle_up::4 82 0.0127 0.0127 0.0127 1.57% amrex::average_down 410 0.01221 0.01221 0.01221 1.50% MLPoisson::Fapply() 1142 0.01195 0.01195 0.01195 1.47% Castro::expand_state() 10 0.0112 0.0112 0.0112 1.38% Castro::do_old_sources() 10 0.01029 0.01029 0.01029 1.27% Castro::initialize_advance() 10 0.0098 0.0098 0.0098 1.21% Gravity::fill_multipole_BCs() 11 0.009569 0.009569 0.009569 1.18% Castro::estTimeStep() 21 0.008442 0.008442 0.008442 1.04% Castro::enforce_min_density() 62 0.008142 0.008142 0.008142 1.00% MLCellLinOp::solutionResidual() 93 0.007255 0.007255 0.007255 0.89% Castro::post_timestep() 10 0.006985 0.006985 0.006985 0.86% MultiFab::Xpay() 585 0.006681 0.006681 0.006681 0.82% MLMG::computeResidual() 82 0.006252 0.006252 0.006252 0.77% Castro::computeNewDt() 9 0.005446 0.005446 0.005446 0.67% Castro::reset_internal_energy(MultiFab) 63 0.005383 0.005383 0.005383 0.66% MLMG::prepareForSolve() 11 0.00516 0.00516 0.00516 0.64% MLCellLinOp::defineBC() 11 0.005146 0.005146 0.005146 0.63% BndryData::define() 11 0.004872 0.004872 0.004872 0.60% Amr::InitializeInit() 1 0.003921 0.003921 0.003921 0.48% Amr::defBaseLevel() 1 0.003915 0.003915 0.003915 0.48% Castro::initData() 1 0.003392 0.003392 0.003392 0.42% Castro::construct_new_source() 50 0.003117 0.003117 0.003117 0.38% Castro::construct_new_gravity_source() 10 0.003074 0.003074 0.003074 0.38% Castro::construct_old_source() 50 0.00283 0.00283 0.00283 0.35% Castro::construct_old_gravity_source() 10 0.002812 0.002812 0.002812 0.35% MLMG::ResNormInf() 93 0.001963 0.001963 0.001963 0.24% Castro::apply_source_to_state() 20 0.001833 0.001833 0.001833 0.23% MultiFab::Saxpy() 20 0.001821 0.001821 0.001821 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001669 0.001669 0.001669 0.21% MLCellLinOp::setLevelBC() 11 0.001559 0.001559 0.001559 0.19% Castro::reset_internal_energy(Fab) 504 0.001548 0.001548 0.001548 0.19% FabArrayBase::getCPC() 1323 0.001449 0.001449 0.001449 0.18% MLMG::getGradSolution() 11 0.001416 0.001416 0.001416 0.17% MLCellLinOp::compGrad() 11 0.00141 0.00141 0.00141 0.17% FabArray::mult() 43 0.001315 0.001315 0.001315 0.16% FabArray::setDomainBndry() 41 0.001312 0.001312 0.001312 0.16% Castro::check_for_nan() 20 0.001189 0.001189 0.001189 0.15% MLPoisson::prepareForSolve() 11 0.001184 0.001184 0.001184 0.15% MultiFab::contains_nan() 20 0.001178 0.001178 0.001178 0.15% MLCellLinOp::prepareForSolve() 11 0.001175 0.001175 0.001175 0.14% Castro::post_regrid() 1 0.001131 0.001131 0.001131 0.14% MLMG::computeMLResidual() 11 0.001041 0.001041 0.001041 0.13% Castro::enforce_speed_limit() 62 0.001018 0.001018 0.001018 0.13% Gravity::update_max_rhs() 11 0.0008204 0.0008204 0.0008204 0.10% FabArrayBase::getFB() 4023 0.0007217 0.0007217 0.0007217 0.09% FabArrayBase::CPC::define() 454 0.0006683 0.0006683 0.0006683 0.08% Castro::computeInitialDt() 2 0.0006409 0.0006409 0.0006409 0.08% Amr::InitAmr() 1 0.0005264 0.0005264 0.0005264 0.06% Gravity::swapTimeLevels() 10 0.0004659 0.0004659 0.0004659 0.06% Castro::Castro() 1 0.0004484 0.0004484 0.0004484 0.06% MLLinOp::define() 11 0.0002708 0.0002708 0.0002708 0.03% MultiFab::Copy() 11 0.000263 0.000263 0.000263 0.03% MultiFab::max() 11 0.0002586 0.0002586 0.0002586 0.03% MLMG::MLResNormInf() 11 0.0002572 0.0002572 0.0002572 0.03% MLLinOp::defineGrids() 11 0.0002483 0.0002483 0.0002483 0.03% MLMG::MLRhsNormInf() 11 0.0002034 0.0002034 0.0002034 0.03% Castro::buildMetrics() 1 0.000161 0.000161 0.000161 0.02% Castro::create_source_corrector() 10 0.0001322 0.0001322 0.0001322 0.02% FabArrayBase::FB::FB() 56 8.419e-05 8.419e-05 8.419e-05 0.01% Castro::finalize_advance() 10 5.596e-05 5.596e-05 5.596e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.253e-05 5.253e-05 5.253e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.709e-05 4.709e-05 4.709e-05 0.01% makeSFC 55 3.982e-05 3.982e-05 3.982e-05 0.00% Castro::swap_state_time_levels() 10 3.944e-05 3.944e-05 3.944e-05 0.00% StateData::define() 4 3.922e-05 3.922e-05 3.922e-05 0.00% Castro::finalize_do_advance() 10 3.614e-05 3.614e-05 3.614e-05 0.00% Castro::enforce_consistent_e() 1 3.431e-05 3.431e-05 3.431e-05 0.00% Castro::initMFs() 1 2.924e-05 2.924e-05 2.924e-05 0.00% Amr::writeSmallPlotFile() 1 2.516e-05 2.516e-05 2.516e-05 0.00% DistributionMapping::Distribute() 56 1.516e-05 1.516e-05 1.516e-05 0.00% Amr::initSubcycle() 1 8.876e-06 8.876e-06 8.876e-06 0.00% MLMG::buildFineMask() 11 5.813e-06 5.813e-06 5.813e-06 0.00% AmrLevel::checkPointPost() 3 4.604e-06 4.604e-06 4.604e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.299e-06 4.299e-06 4.299e-06 0.00% Gravity::set_mass_offset() 11 4.169e-06 4.169e-06 4.169e-06 0.00% Castro::retry_advance_ctu() 10 3.637e-06 3.637e-06 3.637e-06 0.00% Castro::FluxRegCrseInit 10 3.277e-06 3.277e-06 3.277e-06 0.00% Castro::FluxRegFineAdd() 10 2.462e-06 2.462e-06 2.462e-06 0.00% AmrLevel::checkPointPre() 3 2.044e-06 2.044e-06 2.044e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.774e-06 1.774e-06 1.774e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.07-9-g40b3d2176b17) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.07-9-g40b3d2176b17) initialized Starting run at 08:44:09 UTC on 2022-07-14. Successfully read inputs file ... Castro git describe: 22.07-4-g31cce7d5a AMReX git describe: 22.07-9-g40b3d2176 Microphysics git describe: 22.07-10-g6c6c6fcd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.444228797 Restart time = 0.046420172 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.050974131 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051513042 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.065724456 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.064838558 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.068144133 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026358446 seconds Ending run at 08:44:09 UTC on 2022-07-14. Run time = 0.374919512 Run time without initialization = 0.327928583 Average number of zones advanced per microsecond: 3.997 Average number of zones advanced per microsecond per rank: 3.997 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3749 ... 0.3749 ... 0.3749 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1045 0.1045 0.1045 27.86% VisMF::Read() 3 0.03914 0.03914 0.03914 10.44% MLCellLinOp::applyBC() 1946 0.03474 0.03474 0.03474 9.27% MLPoisson::Fsmooth() 1440 0.02704 0.02704 0.02704 7.21% VisMF::Write(FabArray) 1 0.025 0.025 0.025 6.67% StateData::FillBoundary(geom) 160 0.0117 0.0117 0.0117 3.12% MLCGSolver::bicgstab 36 0.01027 0.01027 0.01027 2.74% MultiFab::Dot() 484 0.009548 0.009548 0.009548 2.55% Castro::enforce_min_density() 30 0.007461 0.007461 0.007461 1.99% Castro::normalize_species() 30 0.007336 0.007336 0.007336 1.96% Castro::computeTemp() 30 0.006798 0.006798 0.006798 1.81% FabArray::setVal() 537 0.006716 0.006716 0.006716 1.79% FillBoundary_nowait() 1766 0.006175 0.006175 0.006175 1.65% MLCellLinOp::defineAuxData() 6 0.006162 0.006162 0.006162 1.64% MultiFab::LinComb() 690 0.006119 0.006119 0.006119 1.63% FabArray::ParallelCopy_nowait() 380 0.005912 0.005912 0.005912 1.58% StateDataPhysBCFunct::() 20 0.005475 0.005475 0.005475 1.46% Gravity::fill_multipole_BCs() 6 0.005355 0.005355 0.005355 1.43% MLPoisson::Fapply() 500 0.005012 0.005012 0.005012 1.34% MLMG::addInterpCorrection() 180 0.003195 0.003195 0.003195 0.85% Amr::restart() 1 0.003042 0.003042 0.003042 0.81% Castro::estTimeStep() 10 0.003024 0.003024 0.003024 0.81% amrex::average_down 180 0.002968 0.002968 0.002968 0.79% MultiFab::Xpay() 258 0.002863 0.002863 0.002863 0.76% BndryData::define() 6 0.002109 0.002109 0.002109 0.56% Castro::do_advance_ctu() 5 0.002023 0.002023 0.002023 0.54% Castro::reset_internal_energy(MultiFab) 30 0.001647 0.001647 0.001647 0.44% Castro::construct_new_gravity_source() 5 0.001491 0.001491 0.001491 0.40% Amr::writePlotFile() 1 0.001442 0.001442 0.001442 0.38% Castro::construct_old_gravity_source() 5 0.001434 0.001434 0.001434 0.38% MultiFab::Saxpy() 10 0.0009217 0.0009217 0.0009217 0.25% Castro::enforce_speed_limit() 30 0.0009167 0.0009167 0.0009167 0.24% Gravity::get_old_grav_vector() 5 0.0009084 0.0009084 0.0009084 0.24% Castro::expand_state() 5 0.0008912 0.0008912 0.0008912 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008857 0.0008857 0.0008857 0.24% Gravity::get_new_grav_vector() 5 0.0008712 0.0008712 0.0008712 0.23% MLMG::ResNormInf() 42 0.0008549 0.0008549 0.0008549 0.23% MLCellLinOp::setLevelBC() 6 0.0008297 0.0008297 0.0008297 0.22% Castro::reset_internal_energy(Fab) 240 0.0007804 0.0007804 0.0007804 0.21% MLMG::oneIter() 36 0.0007431 0.0007431 0.0007431 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007025 0.0007025 0.0007025 0.19% FabArray::mult() 22 0.0006459 0.0006459 0.0006459 0.17% FabArray::setDomainBndry() 20 0.0006409 0.0006409 0.0006409 0.17% MLCellLinOp::prepareForSolve() 6 0.0006341 0.0006341 0.0006341 0.17% MultiFab::contains_nan() 10 0.0005849 0.0005849 0.0005849 0.16% MLMG::prepareForSolve() 6 0.0005718 0.0005718 0.0005718 0.15% MLCellLinOp::smooth() 720 0.0005435 0.0005435 0.0005435 0.14% MLCellLinOp::compGrad() 6 0.0004911 0.0004911 0.0004911 0.13% Amr::InitAmr() 1 0.0004104 0.0004104 0.0004104 0.11% FabArrayBase::CPC::define() 244 0.0003893 0.0003893 0.0003893 0.10% FabArrayBase::getCPC() 632 0.0003783 0.0003783 0.0003783 0.10% FabArray::FillBoundary() 1766 0.0003466 0.0003466 0.0003466 0.09% FabArrayBase::getFB() 1766 0.0002552 0.0002552 0.0002552 0.07% main() 1 0.0002373 0.0002373 0.0002373 0.06% Gravity::update_max_rhs() 6 0.0002266 0.0002266 0.0002266 0.06% MLCellLinOp::apply() 500 0.0002096 0.0002096 0.0002096 0.06% Gravity::solve_for_phi() 5 0.0002067 0.0002067 0.0002067 0.06% CGSolver::sxay() 690 0.0001689 0.0001689 0.0001689 0.05% Amr::coarseTimeStep() 5 0.0001519 0.0001519 0.0001519 0.04% MLCellLinOp::defineBC() 6 0.0001486 0.0001486 0.0001486 0.04% MultiFab::Copy() 6 0.0001404 0.0001404 0.0001404 0.04% FillPatchIterator::Initialize 20 0.0001364 0.0001364 0.0001364 0.04% MultiFab::max() 6 0.0001359 0.0001359 0.0001359 0.04% FabArray::ParallelCopy() 380 0.0001264 0.0001264 0.0001264 0.03% MLCGSolver::ParallelAllReduce 659 0.0001246 0.0001246 0.0001246 0.03% Castro::construct_new_gravity() 5 0.0001102 0.0001102 0.0001102 0.03% MLMG::MLRhsNormInf() 6 0.0001075 0.0001075 0.0001075 0.03% Castro::subcycle_advance_ctu() 5 9.831e-05 9.831e-05 9.831e-05 0.03% Amr::timeStep() 5 9.72e-05 9.72e-05 9.72e-05 0.03% MLCellLinOp::correctionResidual() 216 9.389e-05 9.389e-05 9.389e-05 0.03% MLMG::mgVcycle() 36 9.028e-05 9.028e-05 9.028e-05 0.02% Castro::create_source_corrector() 5 8.668e-05 8.668e-05 8.668e-05 0.02% AmrLevel::restart() 1 7.534e-05 7.534e-05 7.534e-05 0.02% StateData::restartDoit() 4 7.187e-05 7.187e-05 7.187e-05 0.02% MLLinOp::defineGrids() 6 6.848e-05 6.848e-05 6.848e-05 0.02% FabArrayBase::FB::FB() 26 5.774e-05 5.774e-05 5.774e-05 0.02% MLMG:computeResOfCorrection() 180 5.771e-05 5.771e-05 5.771e-05 0.02% MLMG::actualBottomSolve() 36 4.245e-05 4.245e-05 4.245e-05 0.01% Castro::initialize_advance() 5 4.057e-05 4.057e-05 4.057e-05 0.01% MLMG::mgVcycle_down::0 36 3.892e-05 3.892e-05 3.892e-05 0.01% Castro::clean_state() 30 3.867e-05 3.867e-05 3.867e-05 0.01% MLMG::mgVcycle_down::1 36 3.725e-05 3.725e-05 3.725e-05 0.01% MLMG::mgVcycle_down::2 36 3.596e-05 3.596e-05 3.596e-05 0.01% MLMG::mgVcycle_down::4 36 3.422e-05 3.422e-05 3.422e-05 0.01% MLMG::solve() 6 3.402e-05 3.402e-05 3.402e-05 0.01% Castro::buildMetrics() 1 3.352e-05 3.352e-05 3.352e-05 0.01% Castro::construct_new_source() 25 3.332e-05 3.332e-05 3.332e-05 0.01% MLMG::mgVcycle_down::3 36 3.325e-05 3.325e-05 3.325e-05 0.01% Castro::initialize_do_advance() 5 3.212e-05 3.212e-05 3.212e-05 0.01% Castro::post_restart() 1 3.035e-05 3.035e-05 3.035e-05 0.01% Gravity::actual_multilevel_solve() 1 3.029e-05 3.029e-05 3.029e-05 0.01% MLMG::mgVcycle_up::4 36 3.007e-05 3.007e-05 3.007e-05 0.01% Castro::advance() 5 2.995e-05 2.995e-05 2.995e-05 0.01% Castro::initMFs() 1 2.702e-05 2.702e-05 2.702e-05 0.01% Castro::swap_state_time_levels() 5 2.633e-05 2.633e-05 2.633e-05 0.01% Castro::finalize_advance() 5 2.611e-05 2.611e-05 2.611e-05 0.01% Amr::writeSmallPlotFile() 1 2.571e-05 2.571e-05 2.571e-05 0.01% MLMG::mgVcycle_up::3 36 2.504e-05 2.504e-05 2.504e-05 0.01% MLMG::mgVcycle_up::2 36 2.464e-05 2.464e-05 2.464e-05 0.01% MLMG::mgVcycle_up::0 36 2.454e-05 2.454e-05 2.454e-05 0.01% Castro::construct_old_source() 25 2.352e-05 2.352e-05 2.352e-05 0.01% MLCellLinOp::solutionResidual() 42 2.343e-05 2.343e-05 2.343e-05 0.01% MLMG::mgVcycle_up::1 36 2.264e-05 2.264e-05 2.264e-05 0.01% MLLinOp::define() 6 1.915e-05 1.915e-05 1.915e-05 0.01% Castro::finalize_do_advance() 5 1.907e-05 1.907e-05 1.907e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.735e-05 1.735e-05 1.735e-05 0.00% MLMG::computeResidual() 36 1.53e-05 1.53e-05 1.53e-05 0.00% MLMG::mgVcycle_bottom 36 1.478e-05 1.478e-05 1.478e-05 0.00% MLPoisson::define() 6 1.437e-05 1.437e-05 1.437e-05 0.00% makeSFC 30 1.389e-05 1.389e-05 1.389e-05 0.00% FillPatchSingleLevel 20 1.351e-05 1.351e-05 1.351e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.256e-05 1.256e-05 1.256e-05 0.00% Amr::initSubcycle() 1 9.046e-06 9.046e-06 9.046e-06 0.00% Castro::do_old_sources() 5 8.948e-06 8.948e-06 8.948e-06 0.00% Castro::do_new_sources() 5 8.49e-06 8.49e-06 8.49e-06 0.00% DistributionMapping::Distribute() 31 8.146e-06 8.146e-06 8.146e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.701e-06 6.701e-06 6.701e-06 0.00% Castro::check_for_nan() 10 6.101e-06 6.101e-06 6.101e-06 0.00% Castro::apply_source_to_state() 10 5.892e-06 5.892e-06 5.892e-06 0.00% Castro::construct_old_gravity() 5 5.48e-06 5.48e-06 5.48e-06 0.00% Gravity::swapTimeLevels() 5 5.364e-06 5.364e-06 5.364e-06 0.00% Castro::post_timestep() 5 5.228e-06 5.228e-06 5.228e-06 0.00% MLPoisson::prepareForSolve() 6 4.095e-06 4.095e-06 4.095e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.407e-06 3.407e-06 3.407e-06 0.00% MLMG::buildFineMask() 6 3.294e-06 3.294e-06 3.294e-06 0.00% MLMG::getGradSolution() 6 3.166e-06 3.166e-06 3.166e-06 0.00% MLMG::computeMLResidual() 6 3.152e-06 3.152e-06 3.152e-06 0.00% Castro::computeNewDt() 5 3.073e-06 3.073e-06 3.073e-06 0.00% MLMG::MLResNormInf() 6 2.315e-06 2.315e-06 2.315e-06 0.00% Gravity::set_mass_offset() 6 2.141e-06 2.141e-06 2.141e-06 0.00% Castro::retry_advance_ctu() 5 1.881e-06 1.881e-06 1.881e-06 0.00% Castro::FluxRegCrseInit 5 1.573e-06 1.573e-06 1.573e-06 0.00% Amr::init() 1 1.537e-06 1.537e-06 1.537e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.131e-06 1.131e-06 1.131e-06 0.00% Castro::FluxRegFineAdd() 5 1.129e-06 1.129e-06 1.129e-06 0.00% AmrLevel::AmrLevel() 1 9.58e-07 9.58e-07 9.58e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3749 0.3749 0.3749 100.00% Amr::coarseTimeStep() 5 0.3013 0.3013 0.3013 80.37% Amr::timeStep() 5 0.2992 0.2992 0.2992 79.79% Castro::advance() 5 0.2956 0.2956 0.2956 78.84% Castro::subcycle_advance_ctu() 5 0.2903 0.2903 0.2903 77.43% Castro::do_advance_ctu() 5 0.2902 0.2902 0.2902 77.41% Castro::construct_new_gravity() 5 0.1436 0.1436 0.1436 38.30% Gravity::solve_phi_with_mlmg() 6 0.1395 0.1395 0.1395 37.21% Gravity::solve_for_phi() 5 0.136 0.136 0.136 36.27% Gravity::actual_solve_with_mlmg() 6 0.134 0.134 0.134 35.75% MLMG::solve() 6 0.1219 0.1219 0.1219 32.52% MLMG::oneIter() 36 0.1149 0.1149 0.1149 30.66% MLMG::mgVcycle() 36 0.1142 0.1142 0.1142 30.46% Castro::construct_ctu_hydro_source() 5 0.1045 0.1045 0.1045 27.86% MLCellLinOp::smooth() 720 0.05845 0.05845 0.05845 15.59% Amr::init() 1 0.04647 0.04647 0.04647 12.39% Amr::restart() 1 0.04647 0.04647 0.04647 12.39% MLCellLinOp::applyBC() 1946 0.04158 0.04158 0.04158 11.09% AmrLevel::restart() 1 0.03935 0.03935 0.03935 10.49% StateData::restartDoit() 4 0.03927 0.03927 0.03927 10.47% VisMF::Read() 3 0.03914 0.03914 0.03914 10.44% MLMG::mgVcycle_bottom 36 0.03508 0.03508 0.03508 9.36% MLMG::actualBottomSolve() 36 0.03507 0.03507 0.03507 9.35% MLCGSolver::bicgstab 36 0.03471 0.03471 0.03471 9.26% MLPoisson::Fsmooth() 1440 0.02704 0.02704 0.02704 7.21% Amr::writePlotFile() 1 0.02644 0.02644 0.02644 7.05% VisMF::Write(FabArray) 1 0.025 0.025 0.025 6.67% Castro::clean_state() 30 0.02498 0.02498 0.02498 6.66% FillPatchIterator::Initialize 20 0.01994 0.01994 0.01994 5.32% FillPatchSingleLevel 20 0.01916 0.01916 0.01916 5.11% StateDataPhysBCFunct::() 20 0.01718 0.01718 0.01718 4.58% MLCellLinOp::apply() 500 0.01567 0.01567 0.01567 4.18% MLMG::mgVcycle_down::0 36 0.0153 0.0153 0.0153 4.08% MLMG::mgVcycle_up::0 36 0.01305 0.01305 0.01305 3.48% StateData::FillBoundary(geom) 160 0.0117 0.0117 0.0117 3.12% Castro::initialize_do_advance() 5 0.01166 0.01166 0.01166 3.11% MLPoisson::define() 6 0.009816 0.009816 0.009816 2.62% MultiFab::Dot() 484 0.009548 0.009548 0.009548 2.55% Castro::computeTemp() 30 0.009225 0.009225 0.009225 2.46% MLCellLinOp::correctionResidual() 216 0.009154 0.009154 0.009154 2.44% MLMG:computeResOfCorrection() 180 0.007891 0.007891 0.007891 2.10% Castro::do_new_sources() 5 0.007814 0.007814 0.007814 2.08% MLMG::mgVcycle_down::1 36 0.007595 0.007595 0.007595 2.03% Gravity::get_new_grav_vector() 5 0.007494 0.007494 0.007494 2.00% Castro::construct_old_gravity() 5 0.007478 0.007478 0.007478 1.99% Gravity::get_old_grav_vector() 5 0.007472 0.007472 0.007472 1.99% Castro::enforce_min_density() 30 0.007461 0.007461 0.007461 1.99% MLMG::mgVcycle_down::2 36 0.007423 0.007423 0.007423 1.98% Castro::normalize_species() 30 0.007336 0.007336 0.007336 1.96% MLMG::mgVcycle_down::3 36 0.006995 0.006995 0.006995 1.87% MLCellLinOp::defineAuxData() 6 0.006892 0.006892 0.006892 1.84% FabArray::FillBoundary() 1766 0.006835 0.006835 0.006835 1.82% FabArray::setVal() 537 0.006716 0.006716 0.006716 1.79% MLMG::mgVcycle_down::4 36 0.006704 0.006704 0.006704 1.79% FillBoundary_nowait() 1766 0.006488 0.006488 0.006488 1.73% FabArray::ParallelCopy() 380 0.006425 0.006425 0.006425 1.71% FabArray::ParallelCopy_nowait() 380 0.006299 0.006299 0.006299 1.68% CGSolver::sxay() 690 0.006288 0.006288 0.006288 1.68% MultiFab::LinComb() 690 0.006119 0.006119 0.006119 1.63% MLCGSolver::ParallelAllReduce 659 0.005736 0.005736 0.005736 1.53% Castro::do_old_sources() 5 0.005671 0.005671 0.005671 1.51% MLMG::mgVcycle_up::2 36 0.005666 0.005666 0.005666 1.51% Castro::expand_state() 5 0.005625 0.005625 0.005625 1.50% MLMG::mgVcycle_up::1 36 0.005584 0.005584 0.005584 1.49% MLMG::addInterpCorrection() 180 0.005407 0.005407 0.005407 1.44% MLMG::mgVcycle_up::3 36 0.005376 0.005376 0.005376 1.43% Gravity::fill_multipole_BCs() 6 0.005355 0.005355 0.005355 1.43% MLMG::mgVcycle_up::4 36 0.005338 0.005338 0.005338 1.42% Castro::initialize_advance() 5 0.005234 0.005234 0.005234 1.40% amrex::average_down 180 0.005205 0.005205 0.005205 1.39% MLPoisson::Fapply() 500 0.005012 0.005012 0.005012 1.34% Castro::post_restart() 1 0.003896 0.003896 0.003896 1.04% Gravity::multilevel_solve_for_new_phi() 1 0.003773 0.003773 0.003773 1.01% Gravity::actual_multilevel_solve() 1 0.003756 0.003756 0.003756 1.00% Castro::post_timestep() 5 0.00343 0.00343 0.00343 0.91% MLCellLinOp::solutionResidual() 42 0.003196 0.003196 0.003196 0.85% Castro::estTimeStep() 10 0.003024 0.003024 0.003024 0.81% MultiFab::Xpay() 258 0.002863 0.002863 0.002863 0.76% MLCellLinOp::defineBC() 6 0.002794 0.002794 0.002794 0.75% MLMG::prepareForSolve() 6 0.002772 0.002772 0.002772 0.74% MLMG::computeResidual() 36 0.002652 0.002652 0.002652 0.71% BndryData::define() 6 0.002646 0.002646 0.002646 0.71% Castro::reset_internal_energy(MultiFab) 30 0.002427 0.002427 0.002427 0.65% Castro::computeNewDt() 5 0.002033 0.002033 0.002033 0.54% Castro::construct_new_source() 25 0.001525 0.001525 0.001525 0.41% Castro::construct_new_gravity_source() 5 0.001491 0.001491 0.001491 0.40% Castro::construct_old_source() 25 0.001457 0.001457 0.001457 0.39% Castro::construct_old_gravity_source() 5 0.001434 0.001434 0.001434 0.38% Castro::apply_source_to_state() 10 0.0009276 0.0009276 0.0009276 0.25% MultiFab::Saxpy() 10 0.0009217 0.0009217 0.0009217 0.25% Castro::enforce_speed_limit() 30 0.0009167 0.0009167 0.0009167 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008857 0.0008857 0.0008857 0.24% MLMG::ResNormInf() 42 0.0008549 0.0008549 0.0008549 0.23% MLCellLinOp::setLevelBC() 6 0.0008297 0.0008297 0.0008297 0.22% Castro::reset_internal_energy(Fab) 240 0.0007804 0.0007804 0.0007804 0.21% FabArrayBase::getCPC() 632 0.0007675 0.0007675 0.0007675 0.20% MLMG::getGradSolution() 6 0.0007617 0.0007617 0.0007617 0.20% MLCellLinOp::compGrad() 6 0.0007585 0.0007585 0.0007585 0.20% FabArray::mult() 22 0.0006459 0.0006459 0.0006459 0.17% FabArray::setDomainBndry() 20 0.0006409 0.0006409 0.0006409 0.17% MLPoisson::prepareForSolve() 6 0.0006382 0.0006382 0.0006382 0.17% MLCellLinOp::prepareForSolve() 6 0.0006341 0.0006341 0.0006341 0.17% Castro::check_for_nan() 10 0.000591 0.000591 0.000591 0.16% MultiFab::contains_nan() 10 0.0005849 0.0005849 0.0005849 0.16% MLMG::computeMLResidual() 6 0.0005626 0.0005626 0.0005626 0.15% Gravity::update_max_rhs() 6 0.0004405 0.0004405 0.0004405 0.12% Amr::InitAmr() 1 0.0004194 0.0004194 0.0004194 0.11% FabArrayBase::CPC::define() 244 0.0003893 0.0003893 0.0003893 0.10% FabArrayBase::getFB() 1766 0.000313 0.000313 0.000313 0.08% Gravity::swapTimeLevels() 5 0.0002266 0.0002266 0.0002266 0.06% Castro::buildMetrics() 1 0.0001519 0.0001519 0.0001519 0.04% MultiFab::Copy() 6 0.0001404 0.0001404 0.0001404 0.04% MultiFab::max() 6 0.0001359 0.0001359 0.0001359 0.04% MLMG::MLResNormInf() 6 0.0001351 0.0001351 0.0001351 0.04% MLLinOp::define() 6 0.000116 0.000116 0.000116 0.03% MLMG::MLRhsNormInf() 6 0.0001075 0.0001075 0.0001075 0.03% MLLinOp::defineGrids() 6 9.686e-05 9.686e-05 9.686e-05 0.03% Castro::create_source_corrector() 5 8.668e-05 8.668e-05 8.668e-05 0.02% FabArrayBase::FB::FB() 26 5.774e-05 5.774e-05 5.774e-05 0.02% Castro::finalize_advance() 5 2.881e-05 2.881e-05 2.881e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.725e-05 2.725e-05 2.725e-05 0.01% Castro::initMFs() 1 2.702e-05 2.702e-05 2.702e-05 0.01% Castro::swap_state_time_levels() 5 2.633e-05 2.633e-05 2.633e-05 0.01% Amr::writeSmallPlotFile() 1 2.571e-05 2.571e-05 2.571e-05 0.01% makeSFC 30 2.055e-05 2.055e-05 2.055e-05 0.01% Castro::finalize_do_advance() 5 1.907e-05 1.907e-05 1.907e-05 0.01% Amr::initSubcycle() 1 9.046e-06 9.046e-06 9.046e-06 0.00% DistributionMapping::Distribute() 31 8.146e-06 8.146e-06 8.146e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.898e-06 4.898e-06 4.898e-06 0.00% MLMG::buildFineMask() 6 3.294e-06 3.294e-06 3.294e-06 0.00% Gravity::set_mass_offset() 6 2.141e-06 2.141e-06 2.141e-06 0.00% Castro::retry_advance_ctu() 5 1.881e-06 1.881e-06 1.881e-06 0.00% Castro::FluxRegCrseInit 5 1.573e-06 1.573e-06 1.573e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.131e-06 1.131e-06 1.131e-06 0.00% Castro::FluxRegFineAdd() 5 1.129e-06 1.129e-06 1.129e-06 0.00% AmrLevel::AmrLevel() 1 9.58e-07 9.58e-07 9.58e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.07-9-g40b3d2176b17) finalized