Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.11-17-gd8bc97f92a1a) initialized Starting run at 09:51:31 UTC on 2022-11-15. Successfully read inputs file ... Castro git describe: 22.11-11-gcd7cb4550 AMReX git describe: 22.11-17-gd8bc97f92 Microphysics git describe: 22.11-21-g6bc0ff40 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.053744537 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.03085877 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.050478792 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.054445841 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.059487709 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.060722091 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.061059947 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.049515995 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.067513987 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.06955246 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058363839 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.051028252 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063320228 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.084099096 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031380798 seconds Ending run at 09:51:32 UTC on 2022-11-15. Run time = 0.895672428 Run time without initialization = 0.761590927 Average number of zones advanced per microsecond: 3.442 Average number of zones advanced per microsecond per rank: 3.442 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8957 ... 0.8957 ... 0.8957 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2079 0.2079 0.2079 23.22% Castro::construct_ctu_hydro_source() 10 0.207 0.207 0.207 23.11% MLCellLinOp::applyBC() 4379 0.08008 0.08008 0.08008 8.94% MLPoisson::Fsmooth() 3240 0.06365 0.06365 0.06365 7.11% Amr::checkPoint() 3 0.03883 0.03883 0.03883 4.34% MLCGSolver::bicgstab 81 0.02405 0.02405 0.02405 2.68% StateData::FillBoundary(geom) 328 0.02373 0.02373 0.02373 2.65% MultiFab::Dot() 1100 0.02247 0.02247 0.02247 2.51% Castro::normalize_species() 62 0.01582 0.01582 0.01582 1.77% Castro::computeTemp() 63 0.01542 0.01542 0.01542 1.72% MultiFab::LinComb() 1566 0.0144 0.0144 0.0144 1.61% FabArray::setVal() 1135 0.01429 0.01429 0.01429 1.60% FillBoundary_nowait() 3974 0.01409 0.01409 0.01409 1.57% FabArray::ParallelCopy_nowait() 851 0.01331 0.01331 0.01331 1.49% StateDataPhysBCFunct::() 41 0.01256 0.01256 0.01256 1.40% MLPoisson::Fapply() 1128 0.01181 0.01181 0.01181 1.32% MLCellLinOp::defineAuxData() 11 0.01175 0.01175 0.01175 1.31% Gravity::fill_multipole_BCs() 11 0.008662 0.008662 0.008662 0.97% MLMG::addInterpCorrection() 405 0.007747 0.007747 0.007747 0.86% Castro::enforce_min_density() 62 0.00752 0.00752 0.00752 0.84% amrex::average_down 405 0.006771 0.006771 0.006771 0.76% MultiFab::Xpay() 578 0.006624 0.006624 0.006624 0.74% Castro::estTimeStep() 21 0.005254 0.005254 0.005254 0.59% Castro::do_advance_ctu() 10 0.005209 0.005209 0.005209 0.58% Castro::reset_internal_energy(MultiFab) 63 0.004304 0.004304 0.004304 0.48% BndryData::define() 11 0.004 0.004 0.004 0.45% Castro::construct_new_gravity_source() 10 0.003299 0.003299 0.003299 0.37% Amr::writePlotFile() 2 0.002815 0.002815 0.002815 0.31% Castro::construct_old_gravity_source() 10 0.002601 0.002601 0.002601 0.29% MLMG::ResNormInf() 92 0.002071 0.002071 0.002071 0.23% Gravity::get_new_grav_vector() 11 0.001939 0.001939 0.001939 0.22% MultiFab::Saxpy() 20 0.0018 0.0018 0.0018 0.20% Gravity::get_old_grav_vector() 10 0.001737 0.001737 0.001737 0.19% Castro::expand_state() 10 0.001735 0.001735 0.001735 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00169 0.00169 0.00169 0.19% Castro::reset_internal_energy(Fab) 504 0.001682 0.001682 0.001682 0.19% MultiFab::Add() 81 0.001667 0.001667 0.001667 0.19% MLCellLinOp::setLevelBC() 11 0.001546 0.001546 0.001546 0.17% Gravity::actual_solve_with_mlmg() 11 0.001363 0.001363 0.001363 0.15% Castro::initData() 1 0.001335 0.001335 0.001335 0.15% FabArray::mult() 43 0.001331 0.001331 0.001331 0.15% FabArray::setDomainBndry() 41 0.001318 0.001318 0.001318 0.15% MLMG::prepareForSolve() 11 0.001221 0.001221 0.001221 0.14% MultiFab::contains_nan() 20 0.001187 0.001187 0.001187 0.13% MLCellLinOp::prepareForSolve() 11 0.001181 0.001181 0.001181 0.13% MLCellLinOp::smooth() 1620 0.001053 0.001053 0.001053 0.12% Castro::enforce_speed_limit() 62 0.0009736 0.0009736 0.0009736 0.11% MLCellLinOp::compGrad() 11 0.0009238 0.0009238 0.0009238 0.10% FabArray::FillBoundary() 3974 0.0008194 0.0008194 0.0008194 0.09% FabArrayBase::getCPC() 1313 0.0007507 0.0007507 0.0007507 0.08% FabArrayBase::CPC::define() 454 0.0006437 0.0006437 0.0006437 0.07% FabArrayBase::getFB() 3974 0.0006027 0.0006027 0.0006027 0.07% Castro::finalize_advance() 10 0.0005608 0.0005608 0.0005608 0.06% Amr::InitAmr() 1 0.00046 0.00046 0.00046 0.05% Gravity::solve_for_phi() 10 0.0004462 0.0004462 0.0004462 0.05% MLCellLinOp::apply() 1128 0.0004356 0.0004356 0.0004356 0.05% Gravity::update_max_rhs() 11 0.0004102 0.0004102 0.0004102 0.05% CGSolver::sxay() 1566 0.0003704 0.0003704 0.0003704 0.04% Amr::coarseTimeStep() 10 0.0003562 0.0003562 0.0003562 0.04% MultiFab::Copy() 11 0.0003358 0.0003358 0.0003358 0.04% MLCGSolver::ParallelAllReduce 1495 0.0002927 0.0002927 0.0002927 0.03% FillPatchIterator::Initialize 41 0.000284 0.000284 0.000284 0.03% MLCellLinOp::defineBC() 11 0.0002726 0.0002726 0.0002726 0.03% MultiFab::max() 11 0.0002564 0.0002564 0.0002564 0.03% FabArray::ParallelCopy() 851 0.0002499 0.0002499 0.0002499 0.03% main() 1 0.0002484 0.0002484 0.0002484 0.03% MLCellLinOp::correctionResidual() 486 0.000224 0.000224 0.000224 0.03% MLMG::MLRhsNormInf() 11 0.0002229 0.0002229 0.0002229 0.02% MLMG::mgVcycle() 81 0.0002067 0.0002067 0.0002067 0.02% Castro::construct_new_gravity() 10 0.0002041 0.0002041 0.0002041 0.02% Castro::subcycle_advance_ctu() 10 0.0001781 0.0001781 0.0001781 0.02% MLLinOp::defineGrids() 11 0.0001612 0.0001612 0.0001612 0.02% MLMG:computeResOfCorrection() 405 0.0001509 0.0001509 0.0001509 0.02% Amr::timeStep() 10 0.0001439 0.0001439 0.0001439 0.02% StateData::checkPoint() 12 0.0001433 0.0001433 0.0001433 0.02% MLMG::mgVcycle_down::0 81 0.0001192 0.0001192 0.0001192 0.01% Castro::advance() 10 0.000119 0.000119 0.000119 0.01% MLMG::mgVcycle_down::1 81 0.0001064 0.0001064 0.0001064 0.01% MLMG::mgVcycle_down::2 81 0.000103 0.000103 0.000103 0.01% MLMG::mgVcycle_down::3 81 9.529e-05 9.529e-05 9.529e-05 0.01% MLMG::mgVcycle_down::4 81 9.345e-05 9.345e-05 9.345e-05 0.01% Castro::Castro() 1 8.792e-05 8.792e-05 8.792e-05 0.01% FabArrayBase::FB::FB() 56 8.422e-05 8.422e-05 8.422e-05 0.01% MLMG::actualBottomSolve() 81 8.346e-05 8.346e-05 8.346e-05 0.01% AmrLevel::checkPoint() 3 8.276e-05 8.276e-05 8.276e-05 0.01% Castro::clean_state() 62 7.974e-05 7.974e-05 7.974e-05 0.01% Castro::initialize_advance() 10 7.92e-05 7.92e-05 7.92e-05 0.01% MLMG::mgVcycle_up::4 81 7.853e-05 7.853e-05 7.853e-05 0.01% MLMG::solve() 11 6.653e-05 6.653e-05 6.653e-05 0.01% MLMG::oneIter() 81 6.457e-05 6.457e-05 6.457e-05 0.01% MLMG::mgVcycle_up::3 81 6.416e-05 6.416e-05 6.416e-05 0.01% MLMG::mgVcycle_up::0 81 6.308e-05 6.308e-05 6.308e-05 0.01% MLMG::mgVcycle_up::1 81 6.273e-05 6.273e-05 6.273e-05 0.01% Castro::initialize_do_advance() 10 6.221e-05 6.221e-05 6.221e-05 0.01% MLMG::mgVcycle_up::2 81 6.123e-05 6.123e-05 6.123e-05 0.01% Castro::create_source_corrector() 10 4.754e-05 4.754e-05 4.754e-05 0.01% MLCellLinOp::solutionResidual() 92 4.636e-05 4.636e-05 4.636e-05 0.01% StateData::define() 4 4.212e-05 4.212e-05 4.212e-05 0.00% MLMG::computeResidual() 81 3.989e-05 3.989e-05 3.989e-05 0.00% Castro::swap_state_time_levels() 10 3.895e-05 3.895e-05 3.895e-05 0.00% Castro::enforce_consistent_e() 1 3.442e-05 3.442e-05 3.442e-05 0.00% MLMG::mgVcycle_bottom 81 3.329e-05 3.329e-05 3.329e-05 0.00% Castro::finalize_do_advance() 10 3.192e-05 3.192e-05 3.192e-05 0.00% Gravity::actual_multilevel_solve() 1 3.136e-05 3.136e-05 3.136e-05 0.00% FillPatchSingleLevel 41 2.721e-05 2.721e-05 2.721e-05 0.00% Amr::defBaseLevel() 1 2.606e-05 2.606e-05 2.606e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.489e-05 2.489e-05 2.489e-05 0.00% Amr::writeSmallPlotFile() 1 2.482e-05 2.482e-05 2.482e-05 0.00% makeSFC 55 2.447e-05 2.447e-05 2.447e-05 0.00% MLPoisson::define() 11 2.444e-05 2.444e-05 2.444e-05 0.00% Castro::initMFs() 1 2.318e-05 2.318e-05 2.318e-05 0.00% Castro::buildMetrics() 1 2.312e-05 2.312e-05 2.312e-05 0.00% MLLinOp::define() 11 2.136e-05 2.136e-05 2.136e-05 0.00% Amr::FinalizeInit() 1 1.872e-05 1.872e-05 1.872e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.751e-05 1.751e-05 1.751e-05 0.00% Castro::construct_old_source() 50 1.697e-05 1.697e-05 1.697e-05 0.00% Castro::construct_new_source() 50 1.692e-05 1.692e-05 1.692e-05 0.00% Castro::do_new_sources() 10 1.585e-05 1.585e-05 1.585e-05 0.00% Castro::do_old_sources() 10 1.507e-05 1.507e-05 1.507e-05 0.00% DistributionMapping::Distribute() 56 1.451e-05 1.451e-05 1.451e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.273e-05 1.273e-05 1.273e-05 0.00% Castro::check_for_nan() 20 1.169e-05 1.169e-05 1.169e-05 0.00% Castro::apply_source_to_state() 20 1.018e-05 1.018e-05 1.018e-05 0.00% Castro::post_timestep() 10 1.016e-05 1.016e-05 1.016e-05 0.00% MLMG::computeMLResidual() 11 1.004e-05 1.004e-05 1.004e-05 0.00% Castro::construct_old_gravity() 10 9.609e-06 9.609e-06 9.609e-06 0.00% MLPoisson::prepareForSolve() 11 8.919e-06 8.919e-06 8.919e-06 0.00% Gravity::swapTimeLevels() 10 8.065e-06 8.065e-06 8.065e-06 0.00% Amr::initSubcycle() 1 7.875e-06 7.875e-06 7.875e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.195e-06 7.195e-06 7.195e-06 0.00% MLMG::getGradSolution() 11 6.684e-06 6.684e-06 6.684e-06 0.00% Castro::computeNewDt() 9 6.487e-06 6.487e-06 6.487e-06 0.00% Amr::InitializeInit() 1 4.673e-06 4.673e-06 4.673e-06 0.00% AmrLevel::checkPointPost() 3 4.603e-06 4.603e-06 4.603e-06 0.00% Gravity::set_mass_offset() 11 4.016e-06 4.016e-06 4.016e-06 0.00% Castro::retry_advance_ctu() 10 3.643e-06 3.643e-06 3.643e-06 0.00% MLMG::MLResNormInf() 11 3.519e-06 3.519e-06 3.519e-06 0.00% Castro::post_init() 1 3.4e-06 3.4e-06 3.4e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.029e-06 3.029e-06 3.029e-06 0.00% Amr::init() 1 2.885e-06 2.885e-06 2.885e-06 0.00% Castro::computeInitialDt() 2 2.801e-06 2.801e-06 2.801e-06 0.00% Castro::FluxRegCrseInit 10 2.599e-06 2.599e-06 2.599e-06 0.00% AmrLevel::checkPointPre() 3 2.388e-06 2.388e-06 2.388e-06 0.00% Castro::FluxRegFineAdd() 10 2.161e-06 2.161e-06 2.161e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.58e-06 1.58e-06 1.58e-06 0.00% Castro::post_regrid() 1 9.88e-07 9.88e-07 9.88e-07 0.00% Amr::initialInit() 1 8.55e-07 8.55e-07 8.55e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8957 0.8957 0.8957 100.00% Amr::coarseTimeStep() 10 0.73 0.73 0.73 81.50% Amr::timeStep() 10 0.5925 0.5925 0.5925 66.14% Castro::advance() 10 0.5854 0.5854 0.5854 65.36% Castro::subcycle_advance_ctu() 10 0.5741 0.5741 0.5741 64.09% Castro::do_advance_ctu() 10 0.5739 0.5739 0.5739 64.07% Gravity::solve_phi_with_mlmg() 11 0.3151 0.3151 0.3151 35.17% Gravity::actual_solve_with_mlmg() 11 0.3062 0.3062 0.3062 34.18% Castro::construct_new_gravity() 10 0.2894 0.2894 0.2894 32.31% MLMG::solve() 11 0.2833 0.2833 0.2833 31.62% Gravity::solve_for_phi() 10 0.274 0.274 0.274 30.59% MLMG::oneIter() 81 0.2683 0.2683 0.2683 29.95% MLMG::mgVcycle() 81 0.2665 0.2665 0.2665 29.76% VisMF::Write(FabArray) 11 0.2079 0.2079 0.2079 23.22% Castro::construct_ctu_hydro_source() 10 0.207 0.207 0.207 23.11% Amr::checkPoint() 3 0.1875 0.1875 0.1875 20.93% AmrLevel::checkPoint() 3 0.1486 0.1486 0.1486 16.59% StateData::checkPoint() 12 0.1486 0.1486 0.1486 16.59% MLCellLinOp::smooth() 1620 0.1358 0.1358 0.1358 15.17% Amr::init() 1 0.1335 0.1335 0.1335 14.91% MLCellLinOp::applyBC() 4379 0.09567 0.09567 0.09567 10.68% MLMG::mgVcycle_bottom 81 0.08222 0.08222 0.08222 9.18% MLMG::actualBottomSolve() 81 0.08219 0.08219 0.08219 9.18% MLCGSolver::bicgstab 81 0.08138 0.08138 0.08138 9.09% MLPoisson::Fsmooth() 3240 0.06365 0.06365 0.06365 7.11% Amr::writePlotFile() 2 0.06235 0.06235 0.06235 6.96% Amr::initialInit() 1 0.04879 0.04879 0.04879 5.45% Castro::clean_state() 62 0.04499 0.04499 0.04499 5.02% Amr::FinalizeInit() 1 0.04463 0.04463 0.04463 4.98% Castro::post_init() 1 0.0433 0.0433 0.0433 4.83% FillPatchIterator::Initialize 41 0.04192 0.04192 0.04192 4.68% Gravity::multilevel_solve_for_new_phi() 1 0.04152 0.04152 0.04152 4.63% Gravity::actual_multilevel_solve() 1 0.0415 0.0415 0.0415 4.63% FillPatchSingleLevel 41 0.04032 0.04032 0.04032 4.50% StateDataPhysBCFunct::() 41 0.03629 0.03629 0.03629 4.05% MLCellLinOp::apply() 1128 0.03628 0.03628 0.03628 4.05% MLMG::mgVcycle_down::0 81 0.03517 0.03517 0.03517 3.93% MLMG::mgVcycle_up::0 81 0.03024 0.03024 0.03024 3.38% StateData::FillBoundary(geom) 328 0.02373 0.02373 0.02373 2.65% MultiFab::Dot() 1100 0.02247 0.02247 0.02247 2.51% Castro::computeTemp() 63 0.02141 0.02141 0.02141 2.39% MLCellLinOp::correctionResidual() 486 0.02136 0.02136 0.02136 2.39% Castro::initialize_do_advance() 10 0.01929 0.01929 0.01929 2.15% MLPoisson::define() 11 0.01857 0.01857 0.01857 2.07% MLMG:computeResOfCorrection() 405 0.01844 0.01844 0.01844 2.06% MLMG::mgVcycle_down::1 81 0.0178 0.0178 0.0178 1.99% MLMG::mgVcycle_down::2 81 0.0173 0.0173 0.0173 1.93% Gravity::get_new_grav_vector() 11 0.01684 0.01684 0.01684 1.88% MLMG::mgVcycle_down::3 81 0.01642 0.01642 0.01642 1.83% Castro::normalize_species() 62 0.01582 0.01582 0.01582 1.77% MLMG::mgVcycle_down::4 81 0.01565 0.01565 0.01565 1.75% FabArray::FillBoundary() 3974 0.0156 0.0156 0.0156 1.74% Castro::construct_old_gravity() 10 0.01479 0.01479 0.01479 1.65% Gravity::get_old_grav_vector() 10 0.01478 0.01478 0.01478 1.65% FillBoundary_nowait() 3974 0.01478 0.01478 0.01478 1.65% CGSolver::sxay() 1566 0.01477 0.01477 0.01477 1.65% MultiFab::LinComb() 1566 0.0144 0.0144 0.0144 1.61% FabArray::ParallelCopy() 851 0.01436 0.01436 0.01436 1.60% FabArray::setVal() 1135 0.01429 0.01429 0.01429 1.60% FabArray::ParallelCopy_nowait() 851 0.01411 0.01411 0.01411 1.57% Castro::do_new_sources() 10 0.01358 0.01358 0.01358 1.52% MLCGSolver::ParallelAllReduce 1495 0.01338 0.01338 0.01338 1.49% MLMG::mgVcycle_up::2 81 0.01328 0.01328 0.01328 1.48% MLMG::mgVcycle_up::1 81 0.01311 0.01311 0.01311 1.46% MLCellLinOp::defineAuxData() 11 0.01307 0.01307 0.01307 1.46% MLMG::addInterpCorrection() 405 0.01292 0.01292 0.01292 1.44% MLMG::mgVcycle_up::3 81 0.01262 0.01262 0.01262 1.41% MLMG::mgVcycle_up::4 81 0.01249 0.01249 0.01249 1.39% amrex::average_down 405 0.01195 0.01195 0.01195 1.33% MLPoisson::Fapply() 1128 0.01181 0.01181 0.01181 1.32% Castro::expand_state() 10 0.01162 0.01162 0.01162 1.30% Castro::initialize_advance() 10 0.01062 0.01062 0.01062 1.19% Castro::do_old_sources() 10 0.01015 0.01015 0.01015 1.13% Gravity::fill_multipole_BCs() 11 0.008662 0.008662 0.008662 0.97% Castro::enforce_min_density() 62 0.00752 0.00752 0.00752 0.84% MLCellLinOp::solutionResidual() 92 0.007133 0.007133 0.007133 0.80% Castro::post_timestep() 10 0.006917 0.006917 0.006917 0.77% MultiFab::Xpay() 578 0.006624 0.006624 0.006624 0.74% MLMG::computeResidual() 81 0.006134 0.006134 0.006134 0.68% Castro::reset_internal_energy(MultiFab) 63 0.005985 0.005985 0.005985 0.67% MLMG::prepareForSolve() 11 0.005448 0.005448 0.005448 0.61% Castro::estTimeStep() 21 0.005254 0.005254 0.005254 0.59% MLCellLinOp::defineBC() 11 0.005239 0.005239 0.005239 0.58% BndryData::define() 11 0.004966 0.004966 0.004966 0.55% Amr::InitializeInit() 1 0.004165 0.004165 0.004165 0.46% Amr::defBaseLevel() 1 0.00416 0.00416 0.00416 0.46% Castro::initData() 1 0.00364 0.00364 0.00364 0.41% Castro::construct_new_source() 50 0.003316 0.003316 0.003316 0.37% Castro::construct_new_gravity_source() 10 0.003299 0.003299 0.003299 0.37% Castro::computeNewDt() 9 0.002794 0.002794 0.002794 0.31% Castro::construct_old_source() 50 0.002618 0.002618 0.002618 0.29% Castro::construct_old_gravity_source() 10 0.002601 0.002601 0.002601 0.29% MLMG::ResNormInf() 92 0.002071 0.002071 0.002071 0.23% Castro::apply_source_to_state() 20 0.00181 0.00181 0.00181 0.20% MultiFab::Saxpy() 20 0.0018 0.0018 0.0018 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00169 0.00169 0.00169 0.19% Castro::reset_internal_energy(Fab) 504 0.001682 0.001682 0.001682 0.19% MultiFab::Add() 81 0.001667 0.001667 0.001667 0.19% MLCellLinOp::setLevelBC() 11 0.001546 0.001546 0.001546 0.17% MLMG::getGradSolution() 11 0.001434 0.001434 0.001434 0.16% MLCellLinOp::compGrad() 11 0.001427 0.001427 0.001427 0.16% FabArrayBase::getCPC() 1313 0.001394 0.001394 0.001394 0.16% FabArray::mult() 43 0.001331 0.001331 0.001331 0.15% FabArray::setDomainBndry() 41 0.001318 0.001318 0.001318 0.15% Castro::check_for_nan() 20 0.001198 0.001198 0.001198 0.13% MLPoisson::prepareForSolve() 11 0.00119 0.00119 0.00119 0.13% MultiFab::contains_nan() 20 0.001187 0.001187 0.001187 0.13% MLCellLinOp::prepareForSolve() 11 0.001181 0.001181 0.001181 0.13% Castro::post_regrid() 1 0.001144 0.001144 0.001144 0.13% MLMG::computeMLResidual() 11 0.001049 0.001049 0.001049 0.12% Castro::enforce_speed_limit() 62 0.0009736 0.0009736 0.0009736 0.11% Castro::computeInitialDt() 2 0.0008665 0.0008665 0.0008665 0.10% Gravity::update_max_rhs() 11 0.0008133 0.0008133 0.0008133 0.09% FabArrayBase::getFB() 3974 0.0006869 0.0006869 0.0006869 0.08% FabArrayBase::CPC::define() 454 0.0006437 0.0006437 0.0006437 0.07% Castro::finalize_advance() 10 0.0005656 0.0005656 0.0005656 0.06% Amr::InitAmr() 1 0.0004679 0.0004679 0.0004679 0.05% Castro::Castro() 1 0.0004411 0.0004411 0.0004411 0.05% Gravity::swapTimeLevels() 10 0.0004371 0.0004371 0.0004371 0.05% MultiFab::Copy() 11 0.0003358 0.0003358 0.0003358 0.04% MLMG::MLResNormInf() 11 0.0002802 0.0002802 0.0002802 0.03% MultiFab::max() 11 0.0002564 0.0002564 0.0002564 0.03% MLLinOp::define() 11 0.0002347 0.0002347 0.0002347 0.03% MLMG::MLRhsNormInf() 11 0.0002229 0.0002229 0.0002229 0.02% MLLinOp::defineGrids() 11 0.0002133 0.0002133 0.0002133 0.02% Castro::buildMetrics() 1 0.0001661 0.0001661 0.0001661 0.02% FabArrayBase::FB::FB() 56 8.422e-05 8.422e-05 8.422e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.05e-05 5.05e-05 5.05e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.932e-05 4.932e-05 4.932e-05 0.01% Castro::create_source_corrector() 10 4.754e-05 4.754e-05 4.754e-05 0.01% StateData::define() 4 4.212e-05 4.212e-05 4.212e-05 0.00% Castro::swap_state_time_levels() 10 3.895e-05 3.895e-05 3.895e-05 0.00% makeSFC 55 3.777e-05 3.777e-05 3.777e-05 0.00% Castro::enforce_consistent_e() 1 3.442e-05 3.442e-05 3.442e-05 0.00% Castro::finalize_do_advance() 10 3.192e-05 3.192e-05 3.192e-05 0.00% Amr::writeSmallPlotFile() 1 2.482e-05 2.482e-05 2.482e-05 0.00% Castro::initMFs() 1 2.318e-05 2.318e-05 2.318e-05 0.00% DistributionMapping::Distribute() 56 1.451e-05 1.451e-05 1.451e-05 0.00% Amr::initSubcycle() 1 7.875e-06 7.875e-06 7.875e-06 0.00% AmrLevel::checkPointPost() 3 4.603e-06 4.603e-06 4.603e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.24e-06 4.24e-06 4.24e-06 0.00% Gravity::set_mass_offset() 11 4.016e-06 4.016e-06 4.016e-06 0.00% Castro::retry_advance_ctu() 10 3.643e-06 3.643e-06 3.643e-06 0.00% Castro::FluxRegCrseInit 10 2.599e-06 2.599e-06 2.599e-06 0.00% AmrLevel::checkPointPre() 3 2.388e-06 2.388e-06 2.388e-06 0.00% Castro::FluxRegFineAdd() 10 2.161e-06 2.161e-06 2.161e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.58e-06 1.58e-06 1.58e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.11-17-gd8bc97f92a1a) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.11-17-gd8bc97f92a1a) initialized Starting run at 09:51:32 UTC on 2022-11-15. Successfully read inputs file ... Castro git describe: 22.11-11-gcd7cb4550 AMReX git describe: 22.11-17-gd8bc97f92 Microphysics git describe: 22.11-21-g6bc0ff40 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.469811437 Restart time = 0.047994527 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.056093786 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.059076177 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.064372927 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.065911289 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065743646 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.033844111 seconds Ending run at 09:51:33 UTC on 2022-11-15. Run time = 0.394080731 Run time without initialization = 0.345549771 Average number of zones advanced per microsecond: 3.793 Average number of zones advanced per microsecond per rank: 3.793 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3941 ... 0.3941 ... 0.3941 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1049 0.1049 0.1049 26.61% VisMF::Read() 3 0.03999 0.03999 0.03999 10.15% MLCellLinOp::applyBC() 1946 0.03614 0.03614 0.03614 9.17% VisMF::Write(FabArray) 1 0.03217 0.03217 0.03217 8.16% MLPoisson::Fsmooth() 1440 0.02793 0.02793 0.02793 7.09% StateData::FillBoundary(geom) 160 0.01177 0.01177 0.01177 2.99% MLCGSolver::bicgstab 36 0.01063 0.01063 0.01063 2.70% MultiFab::Dot() 484 0.009876 0.009876 0.009876 2.51% Castro::normalize_species() 30 0.008907 0.008907 0.008907 2.26% FabArray::setVal() 537 0.006959 0.006959 0.006959 1.77% Castro::enforce_min_density() 30 0.006956 0.006956 0.006956 1.76% Castro::computeTemp() 30 0.006886 0.006886 0.006886 1.75% MLCellLinOp::defineAuxData() 6 0.006541 0.006541 0.006541 1.66% FillBoundary_nowait() 1766 0.006414 0.006414 0.006414 1.63% MultiFab::LinComb() 690 0.006318 0.006318 0.006318 1.60% FabArray::ParallelCopy_nowait() 380 0.006183 0.006183 0.006183 1.57% StateDataPhysBCFunct::() 20 0.005476 0.005476 0.005476 1.39% MLPoisson::Fapply() 500 0.005249 0.005249 0.005249 1.33% Gravity::fill_multipole_BCs() 6 0.004531 0.004531 0.004531 1.15% Castro::do_advance_ctu() 5 0.003649 0.003649 0.003649 0.93% Amr::restart() 1 0.00358 0.00358 0.00358 0.91% MLMG::addInterpCorrection() 180 0.003508 0.003508 0.003508 0.89% amrex::average_down 180 0.003078 0.003078 0.003078 0.78% MultiFab::Xpay() 258 0.00295 0.00295 0.00295 0.75% Castro::estTimeStep() 10 0.002479 0.002479 0.002479 0.63% BndryData::define() 6 0.002234 0.002234 0.002234 0.57% Castro::reset_internal_energy(MultiFab) 30 0.001791 0.001791 0.001791 0.45% Castro::construct_new_gravity_source() 5 0.001766 0.001766 0.001766 0.45% Amr::writePlotFile() 1 0.001764 0.001764 0.001764 0.45% Castro::construct_old_gravity_source() 5 0.001369 0.001369 0.001369 0.35% Castro::enforce_speed_limit() 30 0.001292 0.001292 0.001292 0.33% Gravity::get_old_grav_vector() 5 0.001006 0.001006 0.001006 0.26% Gravity::get_new_grav_vector() 5 0.001002 0.001002 0.001002 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009369 0.0009369 0.0009369 0.24% MLMG::ResNormInf() 42 0.0009353 0.0009353 0.0009353 0.24% Castro::reset_internal_energy(Fab) 240 0.0009342 0.0009342 0.0009342 0.24% MultiFab::Saxpy() 10 0.000924 0.000924 0.000924 0.23% Castro::expand_state() 5 0.0008755 0.0008755 0.0008755 0.22% Gravity::actual_solve_with_mlmg() 6 0.0008666 0.0008666 0.0008666 0.22% MLCellLinOp::setLevelBC() 6 0.0008584 0.0008584 0.0008584 0.22% MultiFab::Add() 36 0.0007412 0.0007412 0.0007412 0.19% MLMG::prepareForSolve() 6 0.0007196 0.0007196 0.0007196 0.18% FabArray::mult() 22 0.0006743 0.0006743 0.0006743 0.17% MLCellLinOp::prepareForSolve() 6 0.0006721 0.0006721 0.0006721 0.17% FabArray::setDomainBndry() 20 0.0006617 0.0006617 0.0006617 0.17% MultiFab::contains_nan() 10 0.0005984 0.0005984 0.0005984 0.15% MLCellLinOp::smooth() 720 0.0005594 0.0005594 0.0005594 0.14% MLCellLinOp::compGrad() 6 0.000501 0.000501 0.000501 0.13% FabArrayBase::CPC::define() 244 0.0004263 0.0004263 0.0004263 0.11% FabArray::FillBoundary() 1766 0.0004216 0.0004216 0.0004216 0.11% FabArrayBase::getCPC() 632 0.0003985 0.0003985 0.0003985 0.10% Amr::InitAmr() 1 0.0003796 0.0003796 0.0003796 0.10% Castro::finalize_advance() 5 0.0003239 0.0003239 0.0003239 0.08% FabArrayBase::getFB() 1766 0.0002846 0.0002846 0.0002846 0.07% Gravity::update_max_rhs() 6 0.0002676 0.0002676 0.0002676 0.07% main() 1 0.0002522 0.0002522 0.0002522 0.06% Gravity::solve_for_phi() 5 0.0002501 0.0002501 0.0002501 0.06% Amr::coarseTimeStep() 5 0.0002473 0.0002473 0.0002473 0.06% MLCellLinOp::apply() 500 0.0002301 0.0002301 0.0002301 0.06% CGSolver::sxay() 690 0.0002081 0.0002081 0.0002081 0.05% Castro::construct_new_gravity() 5 0.0001982 0.0001982 0.0001982 0.05% MultiFab::Copy() 6 0.0001834 0.0001834 0.0001834 0.05% MLCellLinOp::defineBC() 6 0.0001632 0.0001632 0.0001632 0.04% FillPatchIterator::Initialize 20 0.0001567 0.0001567 0.0001567 0.04% Castro::advance() 5 0.0001399 0.0001399 0.0001399 0.04% MultiFab::max() 6 0.0001399 0.0001399 0.0001399 0.04% Amr::timeStep() 5 0.0001284 0.0001284 0.0001284 0.03% MLCGSolver::ParallelAllReduce 659 0.0001274 0.0001274 0.0001274 0.03% Castro::construct_new_source() 25 0.0001233 0.0001233 0.0001233 0.03% FabArray::ParallelCopy() 380 0.0001222 0.0001222 0.0001222 0.03% Castro::subcycle_advance_ctu() 5 0.0001188 0.0001188 0.0001188 0.03% MLLinOp::defineGrids() 6 0.0001188 0.0001188 0.0001188 0.03% MLMG::MLRhsNormInf() 6 0.0001165 0.0001165 0.0001165 0.03% MLCellLinOp::correctionResidual() 216 0.000115 0.000115 0.000115 0.03% MLMG::mgVcycle() 36 9.972e-05 9.972e-05 9.972e-05 0.03% AmrLevel::restart() 1 8.285e-05 8.285e-05 8.285e-05 0.02% MLMG:computeResOfCorrection() 180 7.998e-05 7.998e-05 7.998e-05 0.02% StateData::restartDoit() 4 7.401e-05 7.401e-05 7.401e-05 0.02% FabArrayBase::FB::FB() 26 6.486e-05 6.486e-05 6.486e-05 0.02% MLMG::mgVcycle_down::0 36 5.81e-05 5.81e-05 5.81e-05 0.01% MLMG::mgVcycle_down::1 36 5.166e-05 5.166e-05 5.166e-05 0.01% Castro::initialize_do_advance() 5 4.911e-05 4.911e-05 4.911e-05 0.01% MLMG::mgVcycle_down::2 36 4.903e-05 4.903e-05 4.903e-05 0.01% Castro::clean_state() 30 4.849e-05 4.849e-05 4.849e-05 0.01% MLMG::mgVcycle_down::4 36 4.652e-05 4.652e-05 4.652e-05 0.01% MLMG::mgVcycle_down::3 36 4.545e-05 4.545e-05 4.545e-05 0.01% Castro::post_timestep() 5 4.471e-05 4.471e-05 4.471e-05 0.01% Amr::writeSmallPlotFile() 1 4.442e-05 4.442e-05 4.442e-05 0.01% Castro::do_new_sources() 5 4.37e-05 4.37e-05 4.37e-05 0.01% Castro::initialize_advance() 5 4.294e-05 4.294e-05 4.294e-05 0.01% MLMG::actualBottomSolve() 36 4.258e-05 4.258e-05 4.258e-05 0.01% Castro::create_source_corrector() 5 4.255e-05 4.255e-05 4.255e-05 0.01% MLMG::mgVcycle_up::4 36 3.981e-05 3.981e-05 3.981e-05 0.01% MLMG::solve() 6 3.778e-05 3.778e-05 3.778e-05 0.01% Castro::buildMetrics() 1 3.345e-05 3.345e-05 3.345e-05 0.01% MLMG::oneIter() 36 3.191e-05 3.191e-05 3.191e-05 0.01% Castro::construct_old_gravity() 5 3.135e-05 3.135e-05 3.135e-05 0.01% MLMG::mgVcycle_up::3 36 3.134e-05 3.134e-05 3.134e-05 0.01% MLMG::mgVcycle_up::0 36 3.134e-05 3.134e-05 3.134e-05 0.01% Castro::post_restart() 1 3.044e-05 3.044e-05 3.044e-05 0.01% Gravity::actual_multilevel_solve() 1 3.02e-05 3.02e-05 3.02e-05 0.01% MLMG::mgVcycle_up::2 36 2.943e-05 2.943e-05 2.943e-05 0.01% Castro::computeNewDt() 5 2.932e-05 2.932e-05 2.932e-05 0.01% MLMG::mgVcycle_up::1 36 2.765e-05 2.765e-05 2.765e-05 0.01% Castro::swap_state_time_levels() 5 2.759e-05 2.759e-05 2.759e-05 0.01% Castro::initMFs() 1 2.671e-05 2.671e-05 2.671e-05 0.01% MLPoisson::define() 6 2.573e-05 2.573e-05 2.573e-05 0.01% MLCellLinOp::solutionResidual() 42 2.469e-05 2.469e-05 2.469e-05 0.01% MLLinOp::define() 6 2.312e-05 2.312e-05 2.312e-05 0.01% makeSFC 30 2.182e-05 2.182e-05 2.182e-05 0.01% MLMG::computeResidual() 36 1.999e-05 1.999e-05 1.999e-05 0.01% Castro::finalize_do_advance() 5 1.974e-05 1.974e-05 1.974e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.761e-05 1.761e-05 1.761e-05 0.00% MLMG::mgVcycle_bottom 36 1.672e-05 1.672e-05 1.672e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.577e-05 1.577e-05 1.577e-05 0.00% FillPatchSingleLevel 20 1.426e-05 1.426e-05 1.426e-05 0.00% Castro::construct_old_source() 25 1.138e-05 1.138e-05 1.138e-05 0.00% DistributionMapping::Distribute() 31 9.57e-06 9.57e-06 9.57e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 9.233e-06 9.233e-06 9.233e-06 0.00% Castro::do_old_sources() 5 8.651e-06 8.651e-06 8.651e-06 0.00% Amr::initSubcycle() 1 7.843e-06 7.843e-06 7.843e-06 0.00% Castro::check_for_nan() 10 6.98e-06 6.98e-06 6.98e-06 0.00% MLMG::computeMLResidual() 6 6.696e-06 6.696e-06 6.696e-06 0.00% Castro::apply_source_to_state() 10 6.614e-06 6.614e-06 6.614e-06 0.00% MLPoisson::prepareForSolve() 6 6.524e-06 6.524e-06 6.524e-06 0.00% Gravity::swapTimeLevels() 5 5.672e-06 5.672e-06 5.672e-06 0.00% MLMG::getGradSolution() 6 3.754e-06 3.754e-06 3.754e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.553e-06 3.553e-06 3.553e-06 0.00% Gravity::set_mass_offset() 6 3.379e-06 3.379e-06 3.379e-06 0.00% Castro::retry_advance_ctu() 5 2.264e-06 2.264e-06 2.264e-06 0.00% MLMG::MLResNormInf() 6 2.264e-06 2.264e-06 2.264e-06 0.00% Castro::FluxRegCrseInit 5 1.799e-06 1.799e-06 1.799e-06 0.00% Amr::init() 1 1.168e-06 1.168e-06 1.168e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.127e-06 1.127e-06 1.127e-06 0.00% Castro::FluxRegFineAdd() 5 1.074e-06 1.074e-06 1.074e-06 0.00% AmrLevel::AmrLevel() 1 8.4e-07 8.4e-07 8.4e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3941 0.3941 0.3941 100.00% Amr::coarseTimeStep() 5 0.3114 0.3114 0.3114 79.02% Amr::timeStep() 5 0.3099 0.3099 0.3099 78.64% Castro::advance() 5 0.3052 0.3052 0.3052 77.44% Castro::subcycle_advance_ctu() 5 0.298 0.298 0.298 75.62% Castro::do_advance_ctu() 5 0.2979 0.2979 0.2979 75.59% Castro::construct_new_gravity() 5 0.1494 0.1494 0.1494 37.90% Gravity::solve_phi_with_mlmg() 6 0.1449 0.1449 0.1449 36.77% Gravity::solve_for_phi() 5 0.1413 0.1413 0.1413 35.85% Gravity::actual_solve_with_mlmg() 6 0.1403 0.1403 0.1403 35.59% MLMG::solve() 6 0.1273 0.1273 0.1273 32.29% MLMG::oneIter() 36 0.1197 0.1197 0.1197 30.38% MLMG::mgVcycle() 36 0.1189 0.1189 0.1189 30.18% Castro::construct_ctu_hydro_source() 5 0.1049 0.1049 0.1049 26.61% MLCellLinOp::smooth() 720 0.06057 0.06057 0.06057 15.37% Amr::init() 1 0.04804 0.04804 0.04804 12.19% Amr::restart() 1 0.04804 0.04804 0.04804 12.19% MLCellLinOp::applyBC() 1946 0.04332 0.04332 0.04332 10.99% AmrLevel::restart() 1 0.04021 0.04021 0.04021 10.20% StateData::restartDoit() 4 0.04012 0.04012 0.04012 10.18% VisMF::Read() 3 0.03999 0.03999 0.03999 10.15% MLMG::mgVcycle_bottom 36 0.03642 0.03642 0.03642 9.24% MLMG::actualBottomSolve() 36 0.0364 0.0364 0.0364 9.24% MLCGSolver::bicgstab 36 0.03604 0.03604 0.03604 9.14% Amr::writePlotFile() 1 0.03394 0.03394 0.03394 8.61% VisMF::Write(FabArray) 1 0.03217 0.03217 0.03217 8.16% MLPoisson::Fsmooth() 1440 0.02793 0.02793 0.02793 7.09% Castro::clean_state() 30 0.02681 0.02681 0.02681 6.80% FillPatchIterator::Initialize 20 0.02009 0.02009 0.02009 5.10% FillPatchSingleLevel 20 0.01927 0.01927 0.01927 4.89% StateDataPhysBCFunct::() 20 0.01724 0.01724 0.01724 4.38% MLCellLinOp::apply() 500 0.01643 0.01643 0.01643 4.17% MLMG::mgVcycle_down::0 36 0.01586 0.01586 0.01586 4.02% MLMG::mgVcycle_up::0 36 0.01352 0.01352 0.01352 3.43% StateData::FillBoundary(geom) 160 0.01177 0.01177 0.01177 2.99% MLPoisson::define() 6 0.01049 0.01049 0.01049 2.66% Castro::initialize_do_advance() 5 0.01045 0.01045 0.01045 2.65% MultiFab::Dot() 484 0.009876 0.009876 0.009876 2.51% MLCellLinOp::correctionResidual() 216 0.009614 0.009614 0.009614 2.44% Castro::computeTemp() 30 0.009612 0.009612 0.009612 2.44% Castro::normalize_species() 30 0.008907 0.008907 0.008907 2.26% MLMG:computeResOfCorrection() 180 0.008298 0.008298 0.008298 2.11% Castro::do_new_sources() 5 0.008233 0.008233 0.008233 2.09% MLMG::mgVcycle_down::1 36 0.007965 0.007965 0.007965 2.02% Gravity::get_new_grav_vector() 5 0.007901 0.007901 0.007901 2.00% MLMG::mgVcycle_down::2 36 0.007679 0.007679 0.007679 1.95% Castro::construct_old_gravity() 5 0.007522 0.007522 0.007522 1.91% Gravity::get_old_grav_vector() 5 0.007491 0.007491 0.007491 1.90% MLCellLinOp::defineAuxData() 6 0.007326 0.007326 0.007326 1.86% MLMG::mgVcycle_down::3 36 0.007321 0.007321 0.007321 1.86% FabArray::FillBoundary() 1766 0.007185 0.007185 0.007185 1.82% MLMG::mgVcycle_down::4 36 0.007007 0.007007 0.007007 1.78% FabArray::setVal() 537 0.006959 0.006959 0.006959 1.77% Castro::enforce_min_density() 30 0.006956 0.006956 0.006956 1.76% FillBoundary_nowait() 1766 0.006763 0.006763 0.006763 1.72% Castro::initialize_advance() 5 0.006722 0.006722 0.006722 1.71% FabArray::ParallelCopy() 380 0.00672 0.00672 0.00672 1.71% FabArray::ParallelCopy_nowait() 380 0.006598 0.006598 0.006598 1.67% CGSolver::sxay() 690 0.006526 0.006526 0.006526 1.66% MultiFab::LinComb() 690 0.006318 0.006318 0.006318 1.60% Castro::do_old_sources() 5 0.006084 0.006084 0.006084 1.54% MLMG::mgVcycle_up::2 36 0.005935 0.005935 0.005935 1.51% MLCGSolver::ParallelAllReduce 659 0.005924 0.005924 0.005924 1.50% MLMG::addInterpCorrection() 180 0.005848 0.005848 0.005848 1.48% MLMG::mgVcycle_up::1 36 0.005836 0.005836 0.005836 1.48% MLMG::mgVcycle_up::4 36 0.005652 0.005652 0.005652 1.43% MLMG::mgVcycle_up::3 36 0.005647 0.005647 0.005647 1.43% Castro::expand_state() 5 0.005606 0.005606 0.005606 1.42% amrex::average_down 180 0.005449 0.005449 0.005449 1.38% MLPoisson::Fapply() 500 0.005249 0.005249 0.005249 1.33% Castro::post_timestep() 5 0.00457 0.00457 0.00457 1.16% Gravity::fill_multipole_BCs() 6 0.004531 0.004531 0.004531 1.15% Castro::post_restart() 1 0.004065 0.004065 0.004065 1.03% Gravity::multilevel_solve_for_new_phi() 1 0.003938 0.003938 0.003938 1.00% Gravity::actual_multilevel_solve() 1 0.00392 0.00392 0.00392 0.99% MLCellLinOp::solutionResidual() 42 0.003329 0.003329 0.003329 0.84% MLMG::prepareForSolve() 6 0.00309 0.00309 0.00309 0.78% MLCellLinOp::defineBC() 6 0.002958 0.002958 0.002958 0.75% MultiFab::Xpay() 258 0.00295 0.00295 0.00295 0.75% BndryData::define() 6 0.002795 0.002795 0.002795 0.71% MLMG::computeResidual() 36 0.002753 0.002753 0.002753 0.70% Castro::reset_internal_energy(MultiFab) 30 0.002725 0.002725 0.002725 0.69% Castro::estTimeStep() 10 0.002479 0.002479 0.002479 0.63% Castro::construct_new_source() 25 0.001889 0.001889 0.001889 0.48% Castro::construct_new_gravity_source() 5 0.001766 0.001766 0.001766 0.45% Castro::construct_old_source() 25 0.00138 0.00138 0.00138 0.35% Castro::construct_old_gravity_source() 5 0.001369 0.001369 0.001369 0.35% Castro::enforce_speed_limit() 30 0.001292 0.001292 0.001292 0.33% Castro::computeNewDt() 5 0.001275 0.001275 0.001275 0.32% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009369 0.0009369 0.0009369 0.24% MLMG::ResNormInf() 42 0.0009353 0.0009353 0.0009353 0.24% Castro::reset_internal_energy(Fab) 240 0.0009342 0.0009342 0.0009342 0.24% Castro::apply_source_to_state() 10 0.0009306 0.0009306 0.0009306 0.24% MultiFab::Saxpy() 10 0.000924 0.000924 0.000924 0.23% MLCellLinOp::setLevelBC() 6 0.0008584 0.0008584 0.0008584 0.22% FabArrayBase::getCPC() 632 0.0008248 0.0008248 0.0008248 0.21% MLMG::getGradSolution() 6 0.0007904 0.0007904 0.0007904 0.20% MLCellLinOp::compGrad() 6 0.0007866 0.0007866 0.0007866 0.20% MultiFab::Add() 36 0.0007412 0.0007412 0.0007412 0.19% MLPoisson::prepareForSolve() 6 0.0006786 0.0006786 0.0006786 0.17% FabArray::mult() 22 0.0006743 0.0006743 0.0006743 0.17% MLCellLinOp::prepareForSolve() 6 0.0006721 0.0006721 0.0006721 0.17% FabArray::setDomainBndry() 20 0.0006617 0.0006617 0.0006617 0.17% Castro::check_for_nan() 10 0.0006053 0.0006053 0.0006053 0.15% MLMG::computeMLResidual() 6 0.0006028 0.0006028 0.0006028 0.15% MultiFab::contains_nan() 10 0.0005984 0.0005984 0.0005984 0.15% Gravity::update_max_rhs() 6 0.0004894 0.0004894 0.0004894 0.12% FabArrayBase::CPC::define() 244 0.0004263 0.0004263 0.0004263 0.11% Amr::InitAmr() 1 0.0003874 0.0003874 0.0003874 0.10% FabArrayBase::getFB() 1766 0.0003495 0.0003495 0.0003495 0.09% Castro::finalize_advance() 5 0.0003268 0.0003268 0.0003268 0.08% Gravity::swapTimeLevels() 5 0.0002303 0.0002303 0.0002303 0.06% MultiFab::Copy() 6 0.0001834 0.0001834 0.0001834 0.05% MLLinOp::define() 6 0.0001819 0.0001819 0.0001819 0.05% MLLinOp::defineGrids() 6 0.0001588 0.0001588 0.0001588 0.04% MLMG::MLResNormInf() 6 0.0001568 0.0001568 0.0001568 0.04% Castro::buildMetrics() 1 0.0001555 0.0001555 0.0001555 0.04% MultiFab::max() 6 0.0001399 0.0001399 0.0001399 0.04% MLMG::MLRhsNormInf() 6 0.0001165 0.0001165 0.0001165 0.03% FabArrayBase::FB::FB() 26 6.486e-05 6.486e-05 6.486e-05 0.02% Amr::writeSmallPlotFile() 1 4.442e-05 4.442e-05 4.442e-05 0.01% Castro::create_source_corrector() 5 4.255e-05 4.255e-05 4.255e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.88e-05 3.88e-05 3.88e-05 0.01% makeSFC 30 2.957e-05 2.957e-05 2.957e-05 0.01% Castro::swap_state_time_levels() 5 2.759e-05 2.759e-05 2.759e-05 0.01% Castro::initMFs() 1 2.671e-05 2.671e-05 2.671e-05 0.01% Castro::finalize_do_advance() 5 1.974e-05 1.974e-05 1.974e-05 0.01% DistributionMapping::Distribute() 31 9.57e-06 9.57e-06 9.57e-06 0.00% Amr::initSubcycle() 1 7.843e-06 7.843e-06 7.843e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.374e-06 5.374e-06 5.374e-06 0.00% Gravity::set_mass_offset() 6 3.379e-06 3.379e-06 3.379e-06 0.00% Castro::retry_advance_ctu() 5 2.264e-06 2.264e-06 2.264e-06 0.00% Castro::FluxRegCrseInit 5 1.799e-06 1.799e-06 1.799e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.127e-06 1.127e-06 1.127e-06 0.00% Castro::FluxRegFineAdd() 5 1.074e-06 1.074e-06 1.074e-06 0.00% AmrLevel::AmrLevel() 1 8.4e-07 8.4e-07 8.4e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.11-17-gd8bc97f92a1a) finalized