Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-1-g94693291667b) initialized Starting run at 08:36:59 UTC on 2022-08-03. Successfully read inputs file ... Castro git describe: 22.08 AMReX git describe: 22.08-1-g946932916 Microphysics git describe: 22.08 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.05184959 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.029675973 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048122894 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051562599 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.051094381 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.062654358 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.062479591 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.047757293 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.075569769 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.060313509 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.061380364 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.063379897 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.064056984 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.047564236 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029460207 seconds Ending run at 08:37:00 UTC on 2022-08-03. Run time = 0.859058611 Run time without initialization = 0.726047079 Average number of zones advanced per microsecond: 3.611 Average number of zones advanced per microsecond per rank: 3.611 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8591 ... 0.8591 ... 0.8591 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2165 0.2165 0.2165 25.20% VisMF::Write(FabArray) 11 0.1983 0.1983 0.1983 23.09% MLCellLinOp::applyBC() 4433 0.07951 0.07951 0.07951 9.25% MLPoisson::Fsmooth() 3280 0.06301 0.06301 0.06301 7.33% StateData::FillBoundary(geom) 328 0.02412 0.02412 0.02412 2.81% MLCGSolver::bicgstab 82 0.02353 0.02353 0.02353 2.74% MultiFab::Dot() 1114 0.02197 0.02197 0.02197 2.56% Castro::normalize_species() 62 0.01664 0.01664 0.01664 1.94% FillBoundary_nowait() 4023 0.01417 0.01417 0.01417 1.65% MultiFab::LinComb() 1586 0.01409 0.01409 0.01409 1.64% FabArray::setVal() 1144 0.01397 0.01397 0.01397 1.63% FabArray::ParallelCopy_nowait() 861 0.01289 0.01289 0.01289 1.50% Castro::computeTemp() 63 0.01281 0.01281 0.01281 1.49% MLPoisson::Fapply() 1142 0.01158 0.01158 0.01158 1.35% MLCellLinOp::defineAuxData() 11 0.01151 0.01151 0.01151 1.34% StateDataPhysBCFunct::() 41 0.01136 0.01136 0.01136 1.32% Castro::enforce_min_density() 62 0.01102 0.01102 0.01102 1.28% Gravity::fill_multipole_BCs() 11 0.009572 0.009572 0.009572 1.11% MLMG::addInterpCorrection() 410 0.00758 0.00758 0.00758 0.88% amrex::average_down 410 0.006774 0.006774 0.006774 0.79% MultiFab::Xpay() 585 0.006514 0.006514 0.006514 0.76% Castro::estTimeStep() 21 0.006304 0.006304 0.006304 0.73% Amr::checkPoint() 3 0.005201 0.005201 0.005201 0.61% Castro::do_advance_ctu() 10 0.004768 0.004768 0.004768 0.56% Castro::reset_internal_energy(MultiFab) 63 0.003816 0.003816 0.003816 0.44% BndryData::define() 11 0.003684 0.003684 0.003684 0.43% Castro::construct_new_gravity_source() 10 0.003219 0.003219 0.003219 0.37% Castro::construct_old_gravity_source() 10 0.002888 0.002888 0.002888 0.34% Amr::writePlotFile() 2 0.002835 0.002835 0.002835 0.33% MLMG::ResNormInf() 93 0.002028 0.002028 0.002028 0.24% Gravity::get_new_grav_vector() 11 0.001912 0.001912 0.001912 0.22% MultiFab::Saxpy() 20 0.00181 0.00181 0.00181 0.21% Gravity::get_old_grav_vector() 10 0.001734 0.001734 0.001734 0.20% Castro::expand_state() 10 0.001732 0.001732 0.001732 0.20% MultiFab::Add() 82 0.001643 0.001643 0.001643 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00163 0.00163 0.00163 0.19% MLCellLinOp::setLevelBC() 11 0.001504 0.001504 0.001504 0.18% Castro::reset_internal_energy(Fab) 504 0.001442 0.001442 0.001442 0.17% Gravity::actual_solve_with_mlmg() 11 0.001429 0.001429 0.001429 0.17% FabArray::mult() 43 0.001308 0.001308 0.001308 0.15% FabArray::setDomainBndry() 41 0.001279 0.001279 0.001279 0.15% MLMG::prepareForSolve() 11 0.00122 0.00122 0.00122 0.14% MLCellLinOp::smooth() 1640 0.001198 0.001198 0.001198 0.14% MultiFab::contains_nan() 20 0.001191 0.001191 0.001191 0.14% MLCellLinOp::prepareForSolve() 11 0.001157 0.001157 0.001157 0.13% Castro::enforce_speed_limit() 62 0.001143 0.001143 0.001143 0.13% Castro::initData() 1 0.001097 0.001097 0.001097 0.13% MLCellLinOp::compGrad() 11 0.0008908 0.0008908 0.0008908 0.10% FabArray::FillBoundary() 4023 0.000812 0.000812 0.000812 0.09% FabArrayBase::getCPC() 1323 0.0007815 0.0007815 0.0007815 0.09% FabArrayBase::CPC::define() 454 0.0006642 0.0006642 0.0006642 0.08% FabArrayBase::getFB() 4023 0.00059 0.00059 0.00059 0.07% MLCellLinOp::apply() 1142 0.000495 0.000495 0.000495 0.06% Amr::InitAmr() 1 0.000478 0.000478 0.000478 0.06% Gravity::solve_for_phi() 10 0.0004627 0.0004627 0.0004627 0.05% Gravity::update_max_rhs() 11 0.0004114 0.0004114 0.0004114 0.05% CGSolver::sxay() 1586 0.0003926 0.0003926 0.0003926 0.05% Amr::coarseTimeStep() 10 0.0003515 0.0003515 0.0003515 0.04% MultiFab::Copy() 11 0.0003232 0.0003232 0.0003232 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002953 0.0002953 0.0002953 0.03% FillPatchIterator::Initialize 41 0.0002863 0.0002863 0.0002863 0.03% MLCellLinOp::defineBC() 11 0.0002772 0.0002772 0.0002772 0.03% FabArray::ParallelCopy() 861 0.0002688 0.0002688 0.0002688 0.03% main() 1 0.0002664 0.0002664 0.0002664 0.03% MultiFab::max() 11 0.0002543 0.0002543 0.0002543 0.03% MLCellLinOp::correctionResidual() 492 0.0002382 0.0002382 0.0002382 0.03% Amr::timeStep() 10 0.0002266 0.0002266 0.0002266 0.03% MLMG::MLRhsNormInf() 11 0.0002133 0.0002133 0.0002133 0.02% Castro::construct_new_gravity() 10 0.0002109 0.0002109 0.0002109 0.02% MLMG::mgVcycle() 82 0.0002056 0.0002056 0.0002056 0.02% Castro::create_source_corrector() 10 0.0001359 0.0001359 0.0001359 0.02% MLLinOp::defineGrids() 11 0.0001358 0.0001358 0.0001358 0.02% MLMG:computeResOfCorrection() 410 0.0001358 0.0001358 0.0001358 0.02% Castro::subcycle_advance_ctu() 10 0.0001341 0.0001341 0.0001341 0.02% StateData::checkPoint() 12 0.0001292 0.0001292 0.0001292 0.02% MLMG::mgVcycle_down::0 82 0.0001066 0.0001066 0.0001066 0.01% MLMG::mgVcycle_down::1 82 9.521e-05 9.521e-05 9.521e-05 0.01% MLMG::mgVcycle_down::2 82 8.96e-05 8.96e-05 8.96e-05 0.01% Castro::Castro() 1 8.639e-05 8.639e-05 8.639e-05 0.01% Castro::initialize_advance() 10 8.589e-05 8.589e-05 8.589e-05 0.01% MLMG::mgVcycle_down::3 82 8.288e-05 8.288e-05 8.288e-05 0.01% MLMG::mgVcycle_down::4 82 8.218e-05 8.218e-05 8.218e-05 0.01% FabArrayBase::FB::FB() 56 8.133e-05 8.133e-05 8.133e-05 0.01% MLMG::actualBottomSolve() 82 8.021e-05 8.021e-05 8.021e-05 0.01% Castro::clean_state() 62 8.003e-05 8.003e-05 8.003e-05 0.01% AmrLevel::checkPoint() 3 7.295e-05 7.295e-05 7.295e-05 0.01% MLMG::solve() 11 7.122e-05 7.122e-05 7.122e-05 0.01% MLMG::mgVcycle_up::4 82 6.883e-05 6.883e-05 6.883e-05 0.01% Castro::initialize_do_advance() 10 6.597e-05 6.597e-05 6.597e-05 0.01% MLMG::oneIter() 82 6.05e-05 6.05e-05 6.05e-05 0.01% Castro::finalize_advance() 10 5.944e-05 5.944e-05 5.944e-05 0.01% Castro::advance() 10 5.731e-05 5.731e-05 5.731e-05 0.01% MLMG::mgVcycle_up::2 82 5.679e-05 5.679e-05 5.679e-05 0.01% MLMG::mgVcycle_up::0 82 5.653e-05 5.653e-05 5.653e-05 0.01% MLMG::mgVcycle_up::3 82 5.477e-05 5.477e-05 5.477e-05 0.01% MLMG::mgVcycle_up::1 82 5.309e-05 5.309e-05 5.309e-05 0.01% MLCellLinOp::solutionResidual() 93 5.039e-05 5.039e-05 5.039e-05 0.01% Castro::construct_new_source() 50 4.033e-05 4.033e-05 4.033e-05 0.00% Castro::swap_state_time_levels() 10 4.021e-05 4.021e-05 4.021e-05 0.00% MLMG::computeResidual() 82 3.778e-05 3.778e-05 3.778e-05 0.00% StateData::define() 4 3.681e-05 3.681e-05 3.681e-05 0.00% Castro::finalize_do_advance() 10 3.502e-05 3.502e-05 3.502e-05 0.00% FillPatchSingleLevel 41 3.486e-05 3.486e-05 3.486e-05 0.00% Castro::enforce_consistent_e() 1 3.387e-05 3.387e-05 3.387e-05 0.00% MLMG::mgVcycle_bottom 82 3.339e-05 3.339e-05 3.339e-05 0.00% Gravity::actual_multilevel_solve() 1 3.077e-05 3.077e-05 3.077e-05 0.00% MLPoisson::define() 11 3.069e-05 3.069e-05 3.069e-05 0.00% makeSFC 55 2.769e-05 2.769e-05 2.769e-05 0.00% Castro::initMFs() 1 2.747e-05 2.747e-05 2.747e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.724e-05 2.724e-05 2.724e-05 0.00% Amr::defBaseLevel() 1 2.703e-05 2.703e-05 2.703e-05 0.00% Amr::writeSmallPlotFile() 1 2.453e-05 2.453e-05 2.453e-05 0.00% MLLinOp::define() 11 2.421e-05 2.421e-05 2.421e-05 0.00% Amr::FinalizeInit() 1 2.316e-05 2.316e-05 2.316e-05 0.00% Castro::buildMetrics() 1 2.17e-05 2.17e-05 2.17e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.918e-05 1.918e-05 1.918e-05 0.00% Castro::construct_old_source() 50 1.849e-05 1.849e-05 1.849e-05 0.00% Castro::do_new_sources() 10 1.7e-05 1.7e-05 1.7e-05 0.00% Castro::do_old_sources() 10 1.607e-05 1.607e-05 1.607e-05 0.00% DistributionMapping::Distribute() 56 1.458e-05 1.458e-05 1.458e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.363e-05 1.363e-05 1.363e-05 0.00% Castro::check_for_nan() 20 1.153e-05 1.153e-05 1.153e-05 0.00% Castro::apply_source_to_state() 20 1.151e-05 1.151e-05 1.151e-05 0.00% Castro::construct_old_gravity() 10 1.019e-05 1.019e-05 1.019e-05 0.00% Gravity::swapTimeLevels() 10 8.93e-06 8.93e-06 8.93e-06 0.00% Castro::post_timestep() 10 8.71e-06 8.71e-06 8.71e-06 0.00% MLPoisson::prepareForSolve() 11 8.3e-06 8.3e-06 8.3e-06 0.00% MLMG::computeMLResidual() 11 8.288e-06 8.288e-06 8.288e-06 0.00% Amr::initSubcycle() 1 8.278e-06 8.278e-06 8.278e-06 0.00% Amr::InitializeInit() 1 6.947e-06 6.947e-06 6.947e-06 0.00% MLMG::getGradSolution() 11 6.215e-06 6.215e-06 6.215e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.126e-06 6.126e-06 6.126e-06 0.00% Castro::computeNewDt() 9 5.96e-06 5.96e-06 5.96e-06 0.00% AmrLevel::checkPointPost() 3 5.699e-06 5.699e-06 5.699e-06 0.00% Gravity::set_mass_offset() 11 3.961e-06 3.961e-06 3.961e-06 0.00% MLMG::MLResNormInf() 11 3.667e-06 3.667e-06 3.667e-06 0.00% Castro::retry_advance_ctu() 10 3.665e-06 3.665e-06 3.665e-06 0.00% Castro::post_init() 1 3.642e-06 3.642e-06 3.642e-06 0.00% Castro::FluxRegCrseInit 10 3.094e-06 3.094e-06 3.094e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.918e-06 2.918e-06 2.918e-06 0.00% Castro::FluxRegFineAdd() 10 2.702e-06 2.702e-06 2.702e-06 0.00% Amr::init() 1 2.408e-06 2.408e-06 2.408e-06 0.00% Castro::computeInitialDt() 2 2.368e-06 2.368e-06 2.368e-06 0.00% AmrLevel::checkPointPre() 3 1.9e-06 1.9e-06 1.9e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.793e-06 1.793e-06 1.793e-06 0.00% Castro::post_regrid() 1 1.33e-06 1.33e-06 1.33e-06 0.00% Amr::initialInit() 1 9.64e-07 9.64e-07 9.64e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8591 0.8591 0.8591 100.00% Amr::coarseTimeStep() 10 0.6964 0.6964 0.6964 81.06% Amr::timeStep() 10 0.5968 0.5968 0.5968 69.47% Castro::advance() 10 0.59 0.59 0.59 68.68% Castro::subcycle_advance_ctu() 10 0.5798 0.5798 0.5798 67.49% Castro::do_advance_ctu() 10 0.5797 0.5797 0.5797 67.47% Gravity::solve_phi_with_mlmg() 11 0.3117 0.3117 0.3117 36.28% Gravity::actual_solve_with_mlmg() 11 0.3019 0.3019 0.3019 35.14% Castro::construct_new_gravity() 10 0.2831 0.2831 0.2831 32.96% MLMG::solve() 11 0.2796 0.2796 0.2796 32.55% Gravity::solve_for_phi() 10 0.2681 0.2681 0.2681 31.20% MLMG::oneIter() 82 0.2649 0.2649 0.2649 30.84% MLMG::mgVcycle() 82 0.2632 0.2632 0.2632 30.64% Castro::construct_ctu_hydro_source() 10 0.2165 0.2165 0.2165 25.20% VisMF::Write(FabArray) 11 0.1983 0.1983 0.1983 23.09% Amr::checkPoint() 3 0.1473 0.1473 0.1473 17.15% AmrLevel::checkPoint() 3 0.1421 0.1421 0.1421 16.54% StateData::checkPoint() 12 0.142 0.142 0.142 16.53% MLCellLinOp::smooth() 1640 0.1348 0.1348 0.1348 15.69% Amr::init() 1 0.1324 0.1324 0.1324 15.41% MLCellLinOp::applyBC() 4433 0.09516 0.09516 0.09516 11.08% MLMG::mgVcycle_bottom 82 0.08095 0.08095 0.08095 9.42% MLMG::actualBottomSolve() 82 0.08092 0.08092 0.08092 9.42% MLCGSolver::bicgstab 82 0.08013 0.08013 0.08013 9.33% MLPoisson::Fsmooth() 3280 0.06301 0.06301 0.06301 7.33% Amr::writePlotFile() 2 0.05926 0.05926 0.05926 6.90% Amr::initialInit() 1 0.05075 0.05075 0.05075 5.91% Amr::FinalizeInit() 1 0.04724 0.04724 0.04724 5.50% Castro::clean_state() 62 0.04643 0.04643 0.04643 5.40% Castro::post_init() 1 0.046 0.046 0.046 5.35% Gravity::multilevel_solve_for_new_phi() 1 0.04416 0.04416 0.04416 5.14% Gravity::actual_multilevel_solve() 1 0.04414 0.04414 0.04414 5.14% FillPatchIterator::Initialize 41 0.04106 0.04106 0.04106 4.78% FillPatchSingleLevel 41 0.03949 0.03949 0.03949 4.60% MLCellLinOp::apply() 1142 0.03619 0.03619 0.03619 4.21% StateDataPhysBCFunct::() 41 0.03548 0.03548 0.03548 4.13% MLMG::mgVcycle_down::0 82 0.03501 0.03501 0.03501 4.08% MLMG::mgVcycle_up::0 82 0.02997 0.02997 0.02997 3.49% StateData::FillBoundary(geom) 328 0.02412 0.02412 0.02412 2.81% MultiFab::Dot() 1114 0.02197 0.02197 0.02197 2.56% Castro::initialize_do_advance() 10 0.0214 0.0214 0.0214 2.49% MLCellLinOp::correctionResidual() 492 0.02102 0.02102 0.02102 2.45% MLMG:computeResOfCorrection() 410 0.01817 0.01817 0.01817 2.11% Castro::computeTemp() 63 0.01807 0.01807 0.01807 2.10% MLPoisson::define() 11 0.01796 0.01796 0.01796 2.09% MLMG::mgVcycle_down::1 82 0.0175 0.0175 0.0175 2.04% MLMG::mgVcycle_down::2 82 0.01709 0.01709 0.01709 1.99% Castro::normalize_species() 62 0.01664 0.01664 0.01664 1.94% Gravity::get_new_grav_vector() 11 0.01663 0.01663 0.01663 1.94% MLMG::mgVcycle_down::3 82 0.01619 0.01619 0.01619 1.88% FabArray::FillBoundary() 4023 0.01566 0.01566 0.01566 1.82% MLMG::mgVcycle_down::4 82 0.01546 0.01546 0.01546 1.80% Castro::construct_old_gravity() 10 0.01489 0.01489 0.01489 1.73% Gravity::get_old_grav_vector() 10 0.01488 0.01488 0.01488 1.73% FillBoundary_nowait() 4023 0.01484 0.01484 0.01484 1.73% CGSolver::sxay() 1586 0.01448 0.01448 0.01448 1.69% MultiFab::LinComb() 1586 0.01409 0.01409 0.01409 1.64% FabArray::ParallelCopy() 861 0.01399 0.01399 0.01399 1.63% FabArray::setVal() 1144 0.01397 0.01397 0.01397 1.63% FabArray::ParallelCopy_nowait() 861 0.01372 0.01372 0.01372 1.60% Castro::do_new_sources() 10 0.01319 0.01319 0.01319 1.54% MLCGSolver::ParallelAllReduce 1514 0.0131 0.0131 0.0131 1.52% MLMG::mgVcycle_up::2 82 0.01309 0.01309 0.01309 1.52% MLMG::mgVcycle_up::1 82 0.01293 0.01293 0.01293 1.51% MLCellLinOp::defineAuxData() 11 0.01282 0.01282 0.01282 1.49% MLMG::addInterpCorrection() 410 0.01255 0.01255 0.01255 1.46% MLMG::mgVcycle_up::3 82 0.01245 0.01245 0.01245 1.45% MLMG::mgVcycle_up::4 82 0.01238 0.01238 0.01238 1.44% amrex::average_down 410 0.01181 0.01181 0.01181 1.37% MLPoisson::Fapply() 1142 0.01158 0.01158 0.01158 1.35% Castro::enforce_min_density() 62 0.01102 0.01102 0.01102 1.28% Castro::expand_state() 10 0.01074 0.01074 0.01074 1.25% Castro::do_old_sources() 10 0.01059 0.01059 0.01059 1.23% Castro::initialize_advance() 10 0.0101 0.0101 0.0101 1.18% Gravity::fill_multipole_BCs() 11 0.009572 0.009572 0.009572 1.11% MLCellLinOp::solutionResidual() 93 0.007077 0.007077 0.007077 0.82% Castro::post_timestep() 10 0.006518 0.006518 0.006518 0.76% MultiFab::Xpay() 585 0.006514 0.006514 0.006514 0.76% Castro::estTimeStep() 21 0.006304 0.006304 0.006304 0.73% MLMG::computeResidual() 82 0.00612 0.00612 0.00612 0.71% MLMG::prepareForSolve() 11 0.005268 0.005268 0.005268 0.61% Castro::reset_internal_energy(MultiFab) 63 0.005259 0.005259 0.005259 0.61% MLCellLinOp::defineBC() 11 0.004895 0.004895 0.004895 0.57% BndryData::define() 11 0.004618 0.004618 0.004618 0.54% Amr::InitializeInit() 1 0.003511 0.003511 0.003511 0.41% Amr::defBaseLevel() 1 0.003504 0.003504 0.003504 0.41% Castro::computeNewDt() 9 0.003314 0.003314 0.003314 0.39% Castro::construct_new_source() 50 0.00326 0.00326 0.00326 0.38% Castro::construct_new_gravity_source() 10 0.003219 0.003219 0.003219 0.37% Castro::initData() 1 0.003007 0.003007 0.003007 0.35% Castro::construct_old_source() 50 0.002907 0.002907 0.002907 0.34% Castro::construct_old_gravity_source() 10 0.002888 0.002888 0.002888 0.34% MLMG::ResNormInf() 93 0.002028 0.002028 0.002028 0.24% Castro::apply_source_to_state() 20 0.001822 0.001822 0.001822 0.21% MultiFab::Saxpy() 20 0.00181 0.00181 0.00181 0.21% MultiFab::Add() 82 0.001643 0.001643 0.001643 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00163 0.00163 0.00163 0.19% MLCellLinOp::setLevelBC() 11 0.001504 0.001504 0.001504 0.18% FabArrayBase::getCPC() 1323 0.001446 0.001446 0.001446 0.17% Castro::reset_internal_energy(Fab) 504 0.001442 0.001442 0.001442 0.17% MLMG::getGradSolution() 11 0.001391 0.001391 0.001391 0.16% MLCellLinOp::compGrad() 11 0.001385 0.001385 0.001385 0.16% FabArray::mult() 43 0.001308 0.001308 0.001308 0.15% FabArray::setDomainBndry() 41 0.001279 0.001279 0.001279 0.15% Castro::check_for_nan() 20 0.001202 0.001202 0.001202 0.14% MultiFab::contains_nan() 20 0.001191 0.001191 0.001191 0.14% MLPoisson::prepareForSolve() 11 0.001165 0.001165 0.001165 0.14% MLCellLinOp::prepareForSolve() 11 0.001157 0.001157 0.001157 0.13% Castro::enforce_speed_limit() 62 0.001143 0.001143 0.001143 0.13% Castro::post_regrid() 1 0.001073 0.001073 0.001073 0.12% MLMG::computeMLResidual() 11 0.001003 0.001003 0.001003 0.12% Gravity::update_max_rhs() 11 0.0008146 0.0008146 0.0008146 0.09% FabArrayBase::getFB() 4023 0.0006714 0.0006714 0.0006714 0.08% FabArrayBase::CPC::define() 454 0.0006642 0.0006642 0.0006642 0.08% Castro::computeInitialDt() 2 0.0006516 0.0006516 0.0006516 0.08% Amr::InitAmr() 1 0.0004863 0.0004863 0.0004863 0.06% Gravity::swapTimeLevels() 10 0.0004312 0.0004312 0.0004312 0.05% Castro::Castro() 1 0.0004233 0.0004233 0.0004233 0.05% MultiFab::Copy() 11 0.0003232 0.0003232 0.0003232 0.04% MLMG::MLResNormInf() 11 0.000273 0.000273 0.000273 0.03% MultiFab::max() 11 0.0002543 0.0002543 0.0002543 0.03% MLLinOp::define() 11 0.0002164 0.0002164 0.0002164 0.03% MLMG::MLRhsNormInf() 11 0.0002133 0.0002133 0.0002133 0.02% MLLinOp::defineGrids() 11 0.0001922 0.0001922 0.0001922 0.02% Castro::buildMetrics() 1 0.000148 0.000148 0.000148 0.02% Castro::create_source_corrector() 10 0.0001359 0.0001359 0.0001359 0.02% FabArrayBase::FB::FB() 56 8.133e-05 8.133e-05 8.133e-05 0.01% Castro::finalize_advance() 10 6.524e-05 6.524e-05 6.524e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.462e-05 5.462e-05 5.462e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.294e-05 4.294e-05 4.294e-05 0.00% makeSFC 55 4.099e-05 4.099e-05 4.099e-05 0.00% Castro::swap_state_time_levels() 10 4.021e-05 4.021e-05 4.021e-05 0.00% StateData::define() 4 3.681e-05 3.681e-05 3.681e-05 0.00% Castro::finalize_do_advance() 10 3.502e-05 3.502e-05 3.502e-05 0.00% Castro::enforce_consistent_e() 1 3.387e-05 3.387e-05 3.387e-05 0.00% Castro::initMFs() 1 2.747e-05 2.747e-05 2.747e-05 0.00% Amr::writeSmallPlotFile() 1 2.453e-05 2.453e-05 2.453e-05 0.00% DistributionMapping::Distribute() 56 1.458e-05 1.458e-05 1.458e-05 0.00% Amr::initSubcycle() 1 8.278e-06 8.278e-06 8.278e-06 0.00% AmrLevel::checkPointPost() 3 5.699e-06 5.699e-06 5.699e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.201e-06 4.201e-06 4.201e-06 0.00% Gravity::set_mass_offset() 11 3.961e-06 3.961e-06 3.961e-06 0.00% Castro::retry_advance_ctu() 10 3.665e-06 3.665e-06 3.665e-06 0.00% Castro::FluxRegCrseInit 10 3.094e-06 3.094e-06 3.094e-06 0.00% Castro::FluxRegFineAdd() 10 2.702e-06 2.702e-06 2.702e-06 0.00% AmrLevel::checkPointPre() 3 1.9e-06 1.9e-06 1.9e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.793e-06 1.793e-06 1.793e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-1-g94693291667b) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-1-g94693291667b) initialized Starting run at 08:37:01 UTC on 2022-08-03. Successfully read inputs file ... Castro git describe: 22.08 AMReX git describe: 22.08-1-g946932916 Microphysics git describe: 22.08 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.456723389 Restart time = 0.047965406 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051020932 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049456466 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.063502109 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.062446331 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065051241 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031246125 seconds Ending run at 08:37:01 UTC on 2022-08-03. Run time = 0.371664693 Run time without initialization = 0.323141859 Average number of zones advanced per microsecond: 4.056 Average number of zones advanced per microsecond per rank: 4.056 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3717 ... 0.3717 ... 0.3717 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0990 0.0990 0.0990 26.64% VisMF::Read() 3 0.04026 0.04026 0.04026 10.83% MLCellLinOp::applyBC() 1946 0.03393 0.03393 0.03393 9.13% VisMF::Write(FabArray) 1 0.02967 0.02967 0.02967 7.98% MLPoisson::Fsmooth() 1440 0.02658 0.02658 0.02658 7.15% StateData::FillBoundary(geom) 160 0.01156 0.01156 0.01156 3.11% MLCGSolver::bicgstab 36 0.009954 0.009954 0.009954 2.68% MultiFab::Dot() 484 0.009197 0.009197 0.009197 2.47% Castro::normalize_species() 30 0.00785 0.00785 0.00785 2.11% FabArray::setVal() 537 0.00654 0.00654 0.00654 1.76% Castro::computeTemp() 30 0.006446 0.006446 0.006446 1.73% FillBoundary_nowait() 1766 0.006196 0.006196 0.006196 1.67% MLCellLinOp::defineAuxData() 6 0.006086 0.006086 0.006086 1.64% Castro::enforce_min_density() 30 0.006049 0.006049 0.006049 1.63% StateDataPhysBCFunct::() 20 0.005965 0.005965 0.005965 1.60% MultiFab::LinComb() 690 0.005949 0.005949 0.005949 1.60% FabArray::ParallelCopy_nowait() 380 0.005829 0.005829 0.005829 1.57% Gravity::fill_multipole_BCs() 6 0.005353 0.005353 0.005353 1.44% MLPoisson::Fapply() 500 0.004919 0.004919 0.004919 1.32% Amr::restart() 1 0.003582 0.003582 0.003582 0.96% MLMG::addInterpCorrection() 180 0.003248 0.003248 0.003248 0.87% Castro::estTimeStep() 10 0.00304 0.00304 0.00304 0.82% amrex::average_down 180 0.00288 0.00288 0.00288 0.77% MultiFab::Xpay() 258 0.002791 0.002791 0.002791 0.75% BndryData::define() 6 0.002014 0.002014 0.002014 0.54% Castro::do_advance_ctu() 5 0.00199 0.00199 0.00199 0.54% Amr::writePlotFile() 1 0.001678 0.001678 0.001678 0.45% Castro::reset_internal_energy(MultiFab) 30 0.001667 0.001667 0.001667 0.45% Castro::construct_new_gravity_source() 5 0.001313 0.001313 0.001313 0.35% Castro::construct_old_gravity_source() 5 0.001125 0.001125 0.001125 0.30% MultiFab::Saxpy() 10 0.0009123 0.0009123 0.0009123 0.25% Gravity::get_old_grav_vector() 5 0.0008921 0.0008921 0.0008921 0.24% MLMG::ResNormInf() 42 0.0008914 0.0008914 0.0008914 0.24% Castro::expand_state() 5 0.0008701 0.0008701 0.0008701 0.23% Gravity::get_new_grav_vector() 5 0.0008572 0.0008572 0.0008572 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008567 0.0008567 0.0008567 0.23% MLCellLinOp::setLevelBC() 6 0.0007954 0.0007954 0.0007954 0.21% Castro::reset_internal_energy(Fab) 240 0.0007429 0.0007429 0.0007429 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007349 0.0007349 0.0007349 0.20% MultiFab::Add() 36 0.0007054 0.0007054 0.0007054 0.19% FabArray::mult() 22 0.0006478 0.0006478 0.0006478 0.17% MLMG::prepareForSolve() 6 0.000645 0.000645 0.000645 0.17% FabArray::setDomainBndry() 20 0.0006324 0.0006324 0.0006324 0.17% MLCellLinOp::prepareForSolve() 6 0.0006163 0.0006163 0.0006163 0.17% Castro::enforce_speed_limit() 30 0.0006019 0.0006019 0.0006019 0.16% MultiFab::contains_nan() 10 0.0005839 0.0005839 0.0005839 0.16% MLCellLinOp::smooth() 720 0.0005208 0.0005208 0.0005208 0.14% MLCellLinOp::compGrad() 6 0.0004795 0.0004795 0.0004795 0.13% Amr::InitAmr() 1 0.0003911 0.0003911 0.0003911 0.11% FabArrayBase::CPC::define() 244 0.0003865 0.0003865 0.0003865 0.10% FabArray::FillBoundary() 1766 0.0003736 0.0003736 0.0003736 0.10% FabArrayBase::getCPC() 632 0.0003712 0.0003712 0.0003712 0.10% FabArrayBase::getFB() 1766 0.0002482 0.0002482 0.0002482 0.07% main() 1 0.0002467 0.0002467 0.0002467 0.07% Gravity::update_max_rhs() 6 0.0002257 0.0002257 0.0002257 0.06% Gravity::solve_for_phi() 5 0.0002201 0.0002201 0.0002201 0.06% MLCellLinOp::apply() 500 0.0002122 0.0002122 0.0002122 0.06% CGSolver::sxay() 690 0.0001837 0.0001837 0.0001837 0.05% Amr::coarseTimeStep() 5 0.0001746 0.0001746 0.0001746 0.05% MultiFab::Copy() 6 0.0001704 0.0001704 0.0001704 0.05% MLCellLinOp::defineBC() 6 0.0001452 0.0001452 0.0001452 0.04% FillPatchIterator::Initialize 20 0.0001374 0.0001374 0.0001374 0.04% MultiFab::max() 6 0.0001344 0.0001344 0.0001344 0.04% MLCGSolver::ParallelAllReduce 659 0.0001226 0.0001226 0.0001226 0.03% Castro::construct_new_gravity() 5 0.0001224 0.0001224 0.0001224 0.03% FabArray::ParallelCopy() 380 0.0001162 0.0001162 0.0001162 0.03% MLMG::MLRhsNormInf() 6 0.0001113 0.0001113 0.0001113 0.03% Amr::timeStep() 5 0.0001082 0.0001082 0.0001082 0.03% MLCellLinOp::correctionResidual() 216 0.0001074 0.0001074 0.0001074 0.03% Castro::subcycle_advance_ctu() 5 9.613e-05 9.613e-05 9.613e-05 0.03% MLMG::mgVcycle() 36 9.136e-05 9.136e-05 9.136e-05 0.02% Castro::create_source_corrector() 5 8.389e-05 8.389e-05 8.389e-05 0.02% AmrLevel::restart() 1 7.368e-05 7.368e-05 7.368e-05 0.02% MLLinOp::defineGrids() 6 7.291e-05 7.291e-05 7.291e-05 0.02% StateData::restartDoit() 4 6.643e-05 6.643e-05 6.643e-05 0.02% MLMG:computeResOfCorrection() 180 6.542e-05 6.542e-05 6.542e-05 0.02% FabArrayBase::FB::FB() 26 5.407e-05 5.407e-05 5.407e-05 0.01% Castro::clean_state() 30 5.358e-05 5.358e-05 5.358e-05 0.01% MLMG::mgVcycle_down::0 36 4.281e-05 4.281e-05 4.281e-05 0.01% MLMG::mgVcycle_down::1 36 4.062e-05 4.062e-05 4.062e-05 0.01% Castro::initialize_advance() 5 4.013e-05 4.013e-05 4.013e-05 0.01% Castro::buildMetrics() 1 3.907e-05 3.907e-05 3.907e-05 0.01% MLMG::mgVcycle_down::2 36 3.821e-05 3.821e-05 3.821e-05 0.01% MLMG::mgVcycle_down::3 36 3.579e-05 3.579e-05 3.579e-05 0.01% MLMG::mgVcycle_down::4 36 3.563e-05 3.563e-05 3.563e-05 0.01% MLMG::actualBottomSolve() 36 3.412e-05 3.412e-05 3.412e-05 0.01% MLMG::solve() 6 3.204e-05 3.204e-05 3.204e-05 0.01% Castro::construct_new_source() 25 3.175e-05 3.175e-05 3.175e-05 0.01% Castro::initialize_do_advance() 5 3.171e-05 3.171e-05 3.171e-05 0.01% MLMG::mgVcycle_up::4 36 3.108e-05 3.108e-05 3.108e-05 0.01% Castro::post_restart() 1 3.042e-05 3.042e-05 3.042e-05 0.01% Gravity::actual_multilevel_solve() 1 2.997e-05 2.997e-05 2.997e-05 0.01% Castro::initMFs() 1 2.854e-05 2.854e-05 2.854e-05 0.01% Castro::advance() 5 2.689e-05 2.689e-05 2.689e-05 0.01% Amr::writeSmallPlotFile() 1 2.648e-05 2.648e-05 2.648e-05 0.01% Castro::swap_state_time_levels() 5 2.648e-05 2.648e-05 2.648e-05 0.01% MLMG::oneIter() 36 2.627e-05 2.627e-05 2.627e-05 0.01% MLMG::mgVcycle_up::0 36 2.524e-05 2.524e-05 2.524e-05 0.01% Castro::construct_old_source() 25 2.512e-05 2.512e-05 2.512e-05 0.01% MLPoisson::define() 6 2.397e-05 2.397e-05 2.397e-05 0.01% Castro::finalize_advance() 5 2.378e-05 2.378e-05 2.378e-05 0.01% MLMG::mgVcycle_up::3 36 2.305e-05 2.305e-05 2.305e-05 0.01% MLMG::mgVcycle_up::2 36 2.28e-05 2.28e-05 2.28e-05 0.01% MLCellLinOp::solutionResidual() 42 2.193e-05 2.193e-05 2.193e-05 0.01% MLMG::mgVcycle_up::1 36 2.095e-05 2.095e-05 2.095e-05 0.01% MLLinOp::define() 6 2.077e-05 2.077e-05 2.077e-05 0.01% Castro::finalize_do_advance() 5 1.85e-05 1.85e-05 1.85e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.747e-05 1.747e-05 1.747e-05 0.00% MLMG::computeResidual() 36 1.657e-05 1.657e-05 1.657e-05 0.00% MLMG::mgVcycle_bottom 36 1.494e-05 1.494e-05 1.494e-05 0.00% makeSFC 30 1.416e-05 1.416e-05 1.416e-05 0.00% FillPatchSingleLevel 20 1.401e-05 1.401e-05 1.401e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.32e-05 1.32e-05 1.32e-05 0.00% Castro::do_new_sources() 5 8.941e-06 8.941e-06 8.941e-06 0.00% DistributionMapping::Distribute() 31 8.74e-06 8.74e-06 8.74e-06 0.00% Castro::do_old_sources() 5 8.634e-06 8.634e-06 8.634e-06 0.00% Amr::initSubcycle() 1 8.23e-06 8.23e-06 8.23e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.83e-06 7.83e-06 7.83e-06 0.00% Castro::check_for_nan() 10 6.209e-06 6.209e-06 6.209e-06 0.00% Castro::apply_source_to_state() 10 5.916e-06 5.916e-06 5.916e-06 0.00% Castro::construct_old_gravity() 5 5.735e-06 5.735e-06 5.735e-06 0.00% Gravity::swapTimeLevels() 5 4.955e-06 4.955e-06 4.955e-06 0.00% Castro::post_timestep() 5 4.799e-06 4.799e-06 4.799e-06 0.00% MLPoisson::prepareForSolve() 6 4.709e-06 4.709e-06 4.709e-06 0.00% MLMG::computeMLResidual() 6 4.361e-06 4.361e-06 4.361e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.292e-06 3.292e-06 3.292e-06 0.00% MLMG::getGradSolution() 6 3.094e-06 3.094e-06 3.094e-06 0.00% Castro::computeNewDt() 5 2.58e-06 2.58e-06 2.58e-06 0.00% Gravity::set_mass_offset() 6 2.152e-06 2.152e-06 2.152e-06 0.00% MLMG::MLResNormInf() 6 2.147e-06 2.147e-06 2.147e-06 0.00% Castro::retry_advance_ctu() 5 1.975e-06 1.975e-06 1.975e-06 0.00% Castro::FluxRegCrseInit 5 1.631e-06 1.631e-06 1.631e-06 0.00% Castro::FluxRegFineAdd() 5 1.245e-06 1.245e-06 1.245e-06 0.00% Amr::init() 1 1.111e-06 1.111e-06 1.111e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.089e-06 1.089e-06 1.089e-06 0.00% AmrLevel::AmrLevel() 1 8.52e-07 8.52e-07 8.52e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3717 0.3717 0.3717 100.00% Amr::coarseTimeStep() 5 0.2916 0.2916 0.2916 78.46% Amr::timeStep() 5 0.2894 0.2894 0.2894 77.86% Castro::advance() 5 0.2867 0.2867 0.2867 77.13% Castro::subcycle_advance_ctu() 5 0.2819 0.2819 0.2819 75.84% Castro::do_advance_ctu() 5 0.2818 0.2818 0.2818 75.81% Castro::construct_new_gravity() 5 0.1413 0.1413 0.1413 38.01% Gravity::solve_phi_with_mlmg() 6 0.1369 0.1369 0.1369 36.83% Gravity::solve_for_phi() 5 0.1335 0.1335 0.1335 35.92% Gravity::actual_solve_with_mlmg() 6 0.1314 0.1314 0.1314 35.36% MLMG::solve() 6 0.1195 0.1195 0.1195 32.15% MLMG::oneIter() 36 0.1125 0.1125 0.1125 30.27% MLMG::mgVcycle() 36 0.1118 0.1118 0.1118 30.07% Castro::construct_ctu_hydro_source() 5 0.09903 0.09903 0.09903 26.64% MLCellLinOp::smooth() 720 0.05736 0.05736 0.05736 15.43% Amr::init() 1 0.04802 0.04802 0.04802 12.92% Amr::restart() 1 0.04801 0.04801 0.04801 12.92% MLCellLinOp::applyBC() 1946 0.0408 0.0408 0.0408 10.98% AmrLevel::restart() 1 0.04045 0.04045 0.04045 10.88% StateData::restartDoit() 4 0.04038 0.04038 0.04038 10.86% VisMF::Read() 3 0.04026 0.04026 0.04026 10.83% MLMG::mgVcycle_bottom 36 0.03405 0.03405 0.03405 9.16% MLMG::actualBottomSolve() 36 0.03404 0.03404 0.03404 9.16% MLCGSolver::bicgstab 36 0.0337 0.0337 0.0337 9.07% Amr::writePlotFile() 1 0.03135 0.03135 0.03135 8.43% VisMF::Write(FabArray) 1 0.02967 0.02967 0.02967 7.98% MLPoisson::Fsmooth() 1440 0.02658 0.02658 0.02658 7.15% Castro::clean_state() 30 0.02341 0.02341 0.02341 6.30% FillPatchIterator::Initialize 20 0.0203 0.0203 0.0203 5.46% FillPatchSingleLevel 20 0.01953 0.01953 0.01953 5.25% StateDataPhysBCFunct::() 20 0.01753 0.01753 0.01753 4.72% MLCellLinOp::apply() 500 0.0154 0.0154 0.0154 4.14% MLMG::mgVcycle_down::0 36 0.01506 0.01506 0.01506 4.05% Castro::initialize_do_advance() 5 0.01316 0.01316 0.01316 3.54% MLMG::mgVcycle_up::0 36 0.01284 0.01284 0.01284 3.45% StateData::FillBoundary(geom) 160 0.01156 0.01156 0.01156 3.11% MLPoisson::define() 6 0.009633 0.009633 0.009633 2.59% MultiFab::Dot() 484 0.009197 0.009197 0.009197 2.47% MLCellLinOp::correctionResidual() 216 0.008984 0.008984 0.008984 2.42% Castro::computeTemp() 30 0.008856 0.008856 0.008856 2.38% Castro::normalize_species() 30 0.00785 0.00785 0.00785 2.11% MLMG:computeResOfCorrection() 180 0.007751 0.007751 0.007751 2.09% Gravity::get_new_grav_vector() 5 0.007643 0.007643 0.007643 2.06% Castro::construct_old_gravity() 5 0.007457 0.007457 0.007457 2.01% Gravity::get_old_grav_vector() 5 0.007451 0.007451 0.007451 2.00% MLMG::mgVcycle_down::1 36 0.00745 0.00745 0.00745 2.00% MLMG::mgVcycle_down::2 36 0.007224 0.007224 0.007224 1.94% FabArray::FillBoundary() 1766 0.006871 0.006871 0.006871 1.85% MLMG::mgVcycle_down::3 36 0.006854 0.006854 0.006854 1.84% MLCellLinOp::defineAuxData() 6 0.006807 0.006807 0.006807 1.83% MLMG::mgVcycle_down::4 36 0.006584 0.006584 0.006584 1.77% Castro::do_new_sources() 5 0.006583 0.006583 0.006583 1.77% FabArray::setVal() 537 0.00654 0.00654 0.00654 1.76% FillBoundary_nowait() 1766 0.006498 0.006498 0.006498 1.75% FabArray::ParallelCopy() 380 0.00632 0.00632 0.00632 1.70% FabArray::ParallelCopy_nowait() 380 0.006204 0.006204 0.006204 1.67% CGSolver::sxay() 690 0.006132 0.006132 0.006132 1.65% Castro::enforce_min_density() 30 0.006049 0.006049 0.006049 1.63% MultiFab::LinComb() 690 0.005949 0.005949 0.005949 1.60% Castro::expand_state() 5 0.005858 0.005858 0.005858 1.58% MLMG::mgVcycle_up::2 36 0.005566 0.005566 0.005566 1.50% MLCGSolver::ParallelAllReduce 659 0.005518 0.005518 0.005518 1.48% MLMG::mgVcycle_up::1 36 0.005488 0.005488 0.005488 1.48% MLMG::addInterpCorrection() 180 0.005401 0.005401 0.005401 1.45% Gravity::fill_multipole_BCs() 6 0.005353 0.005353 0.005353 1.44% MLMG::mgVcycle_up::4 36 0.005282 0.005282 0.005282 1.42% MLMG::mgVcycle_up::3 36 0.005269 0.005269 0.005269 1.42% amrex::average_down 180 0.005058 0.005058 0.005058 1.36% MLPoisson::Fapply() 500 0.004919 0.004919 0.004919 1.32% Castro::do_old_sources() 5 0.004914 0.004914 0.004914 1.32% Castro::initialize_advance() 5 0.004752 0.004752 0.004752 1.28% Castro::post_restart() 1 0.003796 0.003796 0.003796 1.02% Gravity::multilevel_solve_for_new_phi() 1 0.003672 0.003672 0.003672 0.99% Gravity::actual_multilevel_solve() 1 0.003654 0.003654 0.003654 0.98% MLCellLinOp::solutionResidual() 42 0.003166 0.003166 0.003166 0.85% Castro::estTimeStep() 10 0.00304 0.00304 0.00304 0.82% MLMG::prepareForSolve() 6 0.002795 0.002795 0.002795 0.75% MultiFab::Xpay() 258 0.002791 0.002791 0.002791 0.75% MLCellLinOp::defineBC() 6 0.002679 0.002679 0.002679 0.72% MLMG::computeResidual() 36 0.002631 0.002631 0.002631 0.71% Castro::post_timestep() 5 0.002614 0.002614 0.002614 0.70% BndryData::define() 6 0.002534 0.002534 0.002534 0.68% Castro::reset_internal_energy(MultiFab) 30 0.00241 0.00241 0.00241 0.65% Castro::computeNewDt() 5 0.00205 0.00205 0.00205 0.55% Castro::construct_new_source() 25 0.001344 0.001344 0.001344 0.36% Castro::construct_new_gravity_source() 5 0.001313 0.001313 0.001313 0.35% Castro::construct_old_source() 25 0.00115 0.00115 0.00115 0.31% Castro::construct_old_gravity_source() 5 0.001125 0.001125 0.001125 0.30% Castro::apply_source_to_state() 10 0.0009182 0.0009182 0.0009182 0.25% MultiFab::Saxpy() 10 0.0009123 0.0009123 0.0009123 0.25% MLMG::ResNormInf() 42 0.0008914 0.0008914 0.0008914 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008567 0.0008567 0.0008567 0.23% MLCellLinOp::setLevelBC() 6 0.0007954 0.0007954 0.0007954 0.21% MLMG::getGradSolution() 6 0.0007594 0.0007594 0.0007594 0.20% FabArrayBase::getCPC() 632 0.0007577 0.0007577 0.0007577 0.20% MLCellLinOp::compGrad() 6 0.0007563 0.0007563 0.0007563 0.20% Castro::reset_internal_energy(Fab) 240 0.0007429 0.0007429 0.0007429 0.20% MultiFab::Add() 36 0.0007054 0.0007054 0.0007054 0.19% FabArray::mult() 22 0.0006478 0.0006478 0.0006478 0.17% FabArray::setDomainBndry() 20 0.0006324 0.0006324 0.0006324 0.17% MLPoisson::prepareForSolve() 6 0.000621 0.000621 0.000621 0.17% MLCellLinOp::prepareForSolve() 6 0.0006163 0.0006163 0.0006163 0.17% Castro::enforce_speed_limit() 30 0.0006019 0.0006019 0.0006019 0.16% Castro::check_for_nan() 10 0.0005901 0.0005901 0.0005901 0.16% MultiFab::contains_nan() 10 0.0005839 0.0005839 0.0005839 0.16% MLMG::computeMLResidual() 6 0.0005568 0.0005568 0.0005568 0.15% Gravity::update_max_rhs() 6 0.0004361 0.0004361 0.0004361 0.12% Amr::InitAmr() 1 0.0003993 0.0003993 0.0003993 0.11% FabArrayBase::CPC::define() 244 0.0003865 0.0003865 0.0003865 0.10% FabArrayBase::getFB() 1766 0.0003022 0.0003022 0.0003022 0.08% Gravity::swapTimeLevels() 5 0.0002208 0.0002208 0.0002208 0.06% MultiFab::Copy() 6 0.0001704 0.0001704 0.0001704 0.05% Castro::buildMetrics() 1 0.0001539 0.0001539 0.0001539 0.04% MLMG::MLResNormInf() 6 0.0001462 0.0001462 0.0001462 0.04% MultiFab::max() 6 0.0001344 0.0001344 0.0001344 0.04% MLLinOp::define() 6 0.000124 0.000124 0.000124 0.03% MLMG::MLRhsNormInf() 6 0.0001113 0.0001113 0.0001113 0.03% MLLinOp::defineGrids() 6 0.0001032 0.0001032 0.0001032 0.03% Castro::create_source_corrector() 5 8.389e-05 8.389e-05 8.389e-05 0.02% FabArrayBase::FB::FB() 26 5.407e-05 5.407e-05 5.407e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.921e-05 2.921e-05 2.921e-05 0.01% Castro::initMFs() 1 2.854e-05 2.854e-05 2.854e-05 0.01% Castro::finalize_advance() 5 2.666e-05 2.666e-05 2.666e-05 0.01% Amr::writeSmallPlotFile() 1 2.648e-05 2.648e-05 2.648e-05 0.01% Castro::swap_state_time_levels() 5 2.648e-05 2.648e-05 2.648e-05 0.01% makeSFC 30 2.138e-05 2.138e-05 2.138e-05 0.01% Castro::finalize_do_advance() 5 1.85e-05 1.85e-05 1.85e-05 0.00% DistributionMapping::Distribute() 31 8.74e-06 8.74e-06 8.74e-06 0.00% Amr::initSubcycle() 1 8.23e-06 8.23e-06 8.23e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.807e-06 4.807e-06 4.807e-06 0.00% Gravity::set_mass_offset() 6 2.152e-06 2.152e-06 2.152e-06 0.00% Castro::retry_advance_ctu() 5 1.975e-06 1.975e-06 1.975e-06 0.00% Castro::FluxRegCrseInit 5 1.631e-06 1.631e-06 1.631e-06 0.00% Castro::FluxRegFineAdd() 5 1.245e-06 1.245e-06 1.245e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.089e-06 1.089e-06 1.089e-06 0.00% AmrLevel::AmrLevel() 1 8.52e-07 8.52e-07 8.52e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-1-g94693291667b) finalized