Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-24-gb84d7c069cef) initialized Starting run at 08:34:30 UTC on 2022-09-27. Successfully read inputs file ... Castro git describe: 22.09-1-g65b273ad0 AMReX git describe: 22.09-24-gb84d7c069 Microphysics git describe: 22.09-13-gb685b7d3 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.053003646 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030492837 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.049015365 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052664071 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.049928012 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.052550952 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.076321903 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.049336087 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.065144702 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051543357 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.053660216 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.053596531 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065245605 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.069894471 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.030508749 seconds Ending run at 08:34:31 UTC on 2022-09-27. Run time = 0.856673377 Run time without initialization = 0.720135182 Average number of zones advanced per microsecond: 3.640 Average number of zones advanced per microsecond per rank: 3.640 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8567 ... 0.8567 ... 0.8567 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2060 0.2060 0.2060 24.05% Castro::construct_ctu_hydro_source() 10 0.1842 0.1842 0.1842 21.50% MLCellLinOp::applyBC() 4433 0.0809 0.0809 0.0809 9.44% MLPoisson::Fsmooth() 3280 0.06469 0.06469 0.06469 7.55% Amr::checkPoint() 3 0.02467 0.02467 0.02467 2.88% MLCGSolver::bicgstab 82 0.02447 0.02447 0.02447 2.86% StateData::FillBoundary(geom) 328 0.02373 0.02373 0.02373 2.77% MultiFab::Dot() 1114 0.0227 0.0227 0.0227 2.65% MultiFab::LinComb() 1586 0.01469 0.01469 0.01469 1.71% FabArray::setVal() 1144 0.01448 0.01448 0.01448 1.69% FillBoundary_nowait() 4023 0.01433 0.01433 0.01433 1.67% FabArray::ParallelCopy_nowait() 861 0.01352 0.01352 0.01352 1.58% Castro::normalize_species() 62 0.01336 0.01336 0.01336 1.56% Castro::computeTemp() 63 0.01332 0.01332 0.01332 1.56% StateDataPhysBCFunct::() 41 0.01247 0.01247 0.01247 1.46% MLCellLinOp::defineAuxData() 11 0.012 0.012 0.012 1.40% MLPoisson::Fapply() 1142 0.01188 0.01188 0.01188 1.39% Castro::enforce_min_density() 62 0.008678 0.008678 0.008678 1.01% Gravity::fill_multipole_BCs() 11 0.008413 0.008413 0.008413 0.98% MLMG::addInterpCorrection() 410 0.00779 0.00779 0.00779 0.91% amrex::average_down 410 0.006927 0.006927 0.006927 0.81% MultiFab::Xpay() 585 0.006618 0.006618 0.006618 0.77% Castro::estTimeStep() 21 0.005083 0.005083 0.005083 0.59% Castro::do_advance_ctu() 10 0.004833 0.004833 0.004833 0.56% Castro::reset_internal_energy(MultiFab) 63 0.004113 0.004113 0.004113 0.48% BndryData::define() 11 0.003937 0.003937 0.003937 0.46% Castro::construct_new_gravity_source() 10 0.00326 0.00326 0.00326 0.38% Amr::writePlotFile() 2 0.002604 0.002604 0.002604 0.30% Castro::construct_old_gravity_source() 10 0.002495 0.002495 0.002495 0.29% MLMG::ResNormInf() 93 0.002094 0.002094 0.002094 0.24% Gravity::get_new_grav_vector() 11 0.001937 0.001937 0.001937 0.23% Castro::enforce_speed_limit() 62 0.001826 0.001826 0.001826 0.21% MultiFab::Saxpy() 20 0.001816 0.001816 0.001816 0.21% Gravity::get_old_grav_vector() 10 0.001753 0.001753 0.001753 0.20% Castro::expand_state() 10 0.001735 0.001735 0.001735 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00167 0.00167 0.00167 0.19% MultiFab::Add() 82 0.001664 0.001664 0.001664 0.19% Castro::reset_internal_energy(Fab) 504 0.001618 0.001618 0.001618 0.19% MLCellLinOp::setLevelBC() 11 0.001552 0.001552 0.001552 0.18% Gravity::actual_solve_with_mlmg() 11 0.001413 0.001413 0.001413 0.16% FabArray::setDomainBndry() 41 0.00134 0.00134 0.00134 0.16% FabArray::mult() 43 0.001334 0.001334 0.001334 0.16% Castro::initData() 1 0.00128 0.00128 0.00128 0.15% MLMG::prepareForSolve() 11 0.00126 0.00126 0.00126 0.15% MLCellLinOp::smooth() 1640 0.001209 0.001209 0.001209 0.14% MLCellLinOp::prepareForSolve() 11 0.001204 0.001204 0.001204 0.14% MultiFab::contains_nan() 20 0.001179 0.001179 0.001179 0.14% MLCellLinOp::compGrad() 11 0.0009221 0.0009221 0.0009221 0.11% FabArray::FillBoundary() 4023 0.0008472 0.0008472 0.0008472 0.10% FabArrayBase::getCPC() 1323 0.0007713 0.0007713 0.0007713 0.09% FabArrayBase::CPC::define() 454 0.0006683 0.0006683 0.0006683 0.08% FabArrayBase::getFB() 4023 0.0005926 0.0005926 0.0005926 0.07% MLCellLinOp::apply() 1142 0.0004948 0.0004948 0.0004948 0.06% Amr::InitAmr() 1 0.000473 0.000473 0.000473 0.06% Gravity::solve_for_phi() 10 0.0004716 0.0004716 0.0004716 0.06% Amr::coarseTimeStep() 10 0.0004331 0.0004331 0.0004331 0.05% Gravity::update_max_rhs() 11 0.0004103 0.0004103 0.0004103 0.05% CGSolver::sxay() 1586 0.000387 0.000387 0.000387 0.05% MultiFab::Copy() 11 0.0003256 0.0003256 0.0003256 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003103 0.0003103 0.0003103 0.04% FillPatchIterator::Initialize 41 0.0002949 0.0002949 0.0002949 0.03% FabArray::ParallelCopy() 861 0.000287 0.000287 0.000287 0.03% MLCellLinOp::defineBC() 11 0.000278 0.000278 0.000278 0.03% main() 1 0.0002592 0.0002592 0.0002592 0.03% MultiFab::max() 11 0.000256 0.000256 0.000256 0.03% MLMG::mgVcycle() 82 0.0002482 0.0002482 0.0002482 0.03% MLCellLinOp::correctionResidual() 492 0.0002403 0.0002403 0.0002403 0.03% Castro::subcycle_advance_ctu() 10 0.0002234 0.0002234 0.0002234 0.03% MLMG::MLRhsNormInf() 11 0.0002172 0.0002172 0.0002172 0.03% Castro::construct_new_gravity() 10 0.0002102 0.0002102 0.0002102 0.02% Amr::timeStep() 10 0.0001583 0.0001583 0.0001583 0.02% MLMG:computeResOfCorrection() 410 0.0001506 0.0001506 0.0001506 0.02% MLLinOp::defineGrids() 11 0.0001417 0.0001417 0.0001417 0.02% StateData::checkPoint() 12 0.0001271 0.0001271 0.0001271 0.01% MLMG::mgVcycle_down::0 82 0.0001182 0.0001182 0.0001182 0.01% MLMG::mgVcycle_down::1 82 0.0001021 0.0001021 0.0001021 0.01% MLMG::mgVcycle_down::2 82 9.904e-05 9.904e-05 9.904e-05 0.01% MLMG::mgVcycle_down::3 82 9.294e-05 9.294e-05 9.294e-05 0.01% MLMG::mgVcycle_down::4 82 9.101e-05 9.101e-05 9.101e-05 0.01% MLMG::actualBottomSolve() 82 8.97e-05 8.97e-05 8.97e-05 0.01% Castro::Castro() 1 8.822e-05 8.822e-05 8.822e-05 0.01% Castro::initialize_advance() 10 8.594e-05 8.594e-05 8.594e-05 0.01% FabArrayBase::FB::FB() 56 8.552e-05 8.552e-05 8.552e-05 0.01% Castro::clean_state() 62 8.514e-05 8.514e-05 8.514e-05 0.01% AmrLevel::checkPoint() 3 7.759e-05 7.759e-05 7.759e-05 0.01% MLMG::mgVcycle_up::4 82 7.085e-05 7.085e-05 7.085e-05 0.01% MLMG::solve() 11 7.068e-05 7.068e-05 7.068e-05 0.01% Castro::finalize_advance() 10 7.046e-05 7.046e-05 7.046e-05 0.01% MLMG::oneIter() 82 6.575e-05 6.575e-05 6.575e-05 0.01% Castro::initialize_do_advance() 10 6.348e-05 6.348e-05 6.348e-05 0.01% MLMG::mgVcycle_up::3 82 5.79e-05 5.79e-05 5.79e-05 0.01% MLMG::mgVcycle_up::0 82 5.726e-05 5.726e-05 5.726e-05 0.01% MLMG::mgVcycle_up::1 82 5.712e-05 5.712e-05 5.712e-05 0.01% MLMG::mgVcycle_up::2 82 5.551e-05 5.551e-05 5.551e-05 0.01% MLCellLinOp::solutionResidual() 93 5.47e-05 5.47e-05 5.47e-05 0.01% StateData::define() 4 4.395e-05 4.395e-05 4.395e-05 0.01% Castro::advance() 10 4.186e-05 4.186e-05 4.186e-05 0.00% Castro::swap_state_time_levels() 10 3.98e-05 3.98e-05 3.98e-05 0.00% MLMG::computeResidual() 82 3.753e-05 3.753e-05 3.753e-05 0.00% Castro::enforce_consistent_e() 1 3.409e-05 3.409e-05 3.409e-05 0.00% Castro::finalize_do_advance() 10 3.366e-05 3.366e-05 3.366e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.284e-05 3.284e-05 3.284e-05 0.00% Gravity::actual_multilevel_solve() 1 3.228e-05 3.228e-05 3.228e-05 0.00% MLMG::mgVcycle_bottom 82 3.062e-05 3.062e-05 3.062e-05 0.00% FillPatchSingleLevel 41 2.756e-05 2.756e-05 2.756e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.59e-05 2.59e-05 2.59e-05 0.00% makeSFC 55 2.584e-05 2.584e-05 2.584e-05 0.00% Castro::initMFs() 1 2.57e-05 2.57e-05 2.57e-05 0.00% Amr::writeSmallPlotFile() 1 2.471e-05 2.471e-05 2.471e-05 0.00% MLLinOp::define() 11 2.396e-05 2.396e-05 2.396e-05 0.00% Castro::buildMetrics() 1 2.382e-05 2.382e-05 2.382e-05 0.00% Amr::defBaseLevel() 1 1.998e-05 1.998e-05 1.998e-05 0.00% MLPoisson::define() 11 1.974e-05 1.974e-05 1.974e-05 0.00% Amr::FinalizeInit() 1 1.906e-05 1.906e-05 1.906e-05 0.00% Castro::construct_new_source() 50 1.795e-05 1.795e-05 1.795e-05 0.00% Castro::construct_old_source() 50 1.794e-05 1.794e-05 1.794e-05 0.00% Castro::do_new_sources() 10 1.703e-05 1.703e-05 1.703e-05 0.00% Castro::do_old_sources() 10 1.625e-05 1.625e-05 1.625e-05 0.00% DistributionMapping::Distribute() 56 1.586e-05 1.586e-05 1.586e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.383e-05 1.383e-05 1.383e-05 0.00% Castro::check_for_nan() 20 1.307e-05 1.307e-05 1.307e-05 0.00% Castro::apply_source_to_state() 20 1.094e-05 1.094e-05 1.094e-05 0.00% Castro::post_timestep() 10 1.058e-05 1.058e-05 1.058e-05 0.00% MLMG::computeMLResidual() 11 9.919e-06 9.919e-06 9.919e-06 0.00% Gravity::swapTimeLevels() 10 9.405e-06 9.405e-06 9.405e-06 0.00% Castro::construct_old_gravity() 10 9.357e-06 9.357e-06 9.357e-06 0.00% MLPoisson::prepareForSolve() 11 9.064e-06 9.064e-06 9.064e-06 0.00% Amr::initSubcycle() 1 8.679e-06 8.679e-06 8.679e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.045e-06 7.045e-06 7.045e-06 0.00% MLMG::getGradSolution() 11 7.042e-06 7.042e-06 7.042e-06 0.00% Castro::computeNewDt() 9 7.009e-06 7.009e-06 7.009e-06 0.00% Amr::InitializeInit() 1 5.2e-06 5.2e-06 5.2e-06 0.00% Gravity::set_mass_offset() 11 4.895e-06 4.895e-06 4.895e-06 0.00% AmrLevel::checkPointPost() 3 4.04e-06 4.04e-06 4.04e-06 0.00% Castro::post_init() 1 3.869e-06 3.869e-06 3.869e-06 0.00% Castro::create_source_corrector() 10 3.812e-06 3.812e-06 3.812e-06 0.00% Castro::retry_advance_ctu() 10 3.774e-06 3.774e-06 3.774e-06 0.00% MLMG::MLResNormInf() 11 3.588e-06 3.588e-06 3.588e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.305e-06 3.305e-06 3.305e-06 0.00% Amr::init() 1 2.867e-06 2.867e-06 2.867e-06 0.00% Castro::FluxRegCrseInit 10 2.819e-06 2.819e-06 2.819e-06 0.00% Castro::FluxRegFineAdd() 10 2.531e-06 2.531e-06 2.531e-06 0.00% Castro::computeInitialDt() 2 2.5e-06 2.5e-06 2.5e-06 0.00% AmrLevel::checkPointPre() 3 2.381e-06 2.381e-06 2.381e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.936e-06 1.936e-06 1.936e-06 0.00% Amr::initialInit() 1 1.375e-06 1.375e-06 1.375e-06 0.00% Castro::post_regrid() 1 1.206e-06 1.206e-06 1.206e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8567 0.8567 0.8567 100.00% Amr::coarseTimeStep() 10 0.6894 0.6894 0.6894 80.47% Amr::timeStep() 10 0.567 0.567 0.567 66.19% Castro::advance() 10 0.5598 0.5598 0.5598 65.34% Castro::subcycle_advance_ctu() 10 0.5482 0.5482 0.5482 63.99% Castro::do_advance_ctu() 10 0.548 0.548 0.548 63.97% Gravity::solve_phi_with_mlmg() 11 0.3191 0.3191 0.3191 37.25% Gravity::actual_solve_with_mlmg() 11 0.3105 0.3105 0.3105 36.24% Castro::construct_new_gravity() 10 0.29 0.29 0.29 33.85% MLMG::solve() 11 0.2873 0.2873 0.2873 33.54% Gravity::solve_for_phi() 10 0.2746 0.2746 0.2746 32.05% MLMG::oneIter() 82 0.2723 0.2723 0.2723 31.78% MLMG::mgVcycle() 82 0.2705 0.2705 0.2705 31.58% VisMF::Write(FabArray) 11 0.206 0.206 0.206 24.05% Castro::construct_ctu_hydro_source() 10 0.1842 0.1842 0.1842 21.50% Amr::checkPoint() 3 0.1724 0.1724 0.1724 20.12% AmrLevel::checkPoint() 3 0.1477 0.1477 0.1477 17.24% StateData::checkPoint() 12 0.1476 0.1476 0.1476 17.23% MLCellLinOp::smooth() 1640 0.1379 0.1379 0.1379 16.09% Amr::init() 1 0.136 0.136 0.136 15.87% MLCellLinOp::applyBC() 4433 0.09675 0.09675 0.09675 11.29% MLMG::mgVcycle_bottom 82 0.08352 0.08352 0.08352 9.75% MLMG::actualBottomSolve() 82 0.08349 0.08349 0.08349 9.74% MLCGSolver::bicgstab 82 0.08265 0.08265 0.08265 9.65% MLPoisson::Fsmooth() 3280 0.06469 0.06469 0.06469 7.55% Amr::writePlotFile() 2 0.06112 0.06112 0.06112 7.13% Amr::initialInit() 1 0.05232 0.05232 0.05232 6.11% Amr::FinalizeInit() 1 0.04824 0.04824 0.04824 5.63% Castro::post_init() 1 0.04683 0.04683 0.04683 5.47% Gravity::multilevel_solve_for_new_phi() 1 0.04503 0.04503 0.04503 5.26% Gravity::actual_multilevel_solve() 1 0.04501 0.04501 0.04501 5.25% Castro::clean_state() 62 0.04231 0.04231 0.04231 4.94% FillPatchIterator::Initialize 41 0.04188 0.04188 0.04188 4.89% FillPatchSingleLevel 41 0.04024 0.04024 0.04024 4.70% MLCellLinOp::apply() 1142 0.03665 0.03665 0.03665 4.28% StateDataPhysBCFunct::() 41 0.0362 0.0362 0.0362 4.23% MLMG::mgVcycle_down::0 82 0.03572 0.03572 0.03572 4.17% MLMG::mgVcycle_up::0 82 0.03072 0.03072 0.03072 3.59% StateData::FillBoundary(geom) 328 0.02373 0.02373 0.02373 2.77% MultiFab::Dot() 1114 0.0227 0.0227 0.0227 2.65% MLCellLinOp::correctionResidual() 492 0.0215 0.0215 0.0215 2.51% Castro::computeTemp() 63 0.01905 0.01905 0.01905 2.22% MLPoisson::define() 11 0.01875 0.01875 0.01875 2.19% Castro::initialize_do_advance() 10 0.0186 0.0186 0.0186 2.17% MLMG:computeResOfCorrection() 410 0.01854 0.01854 0.01854 2.16% MLMG::mgVcycle_down::1 82 0.01798 0.01798 0.01798 2.10% MLMG::mgVcycle_down::2 82 0.01755 0.01755 0.01755 2.05% Gravity::get_new_grav_vector() 11 0.01687 0.01687 0.01687 1.97% MLMG::mgVcycle_down::3 82 0.01664 0.01664 0.01664 1.94% MLMG::mgVcycle_down::4 82 0.01591 0.01591 0.01591 1.86% FabArray::FillBoundary() 4023 0.01586 0.01586 0.01586 1.85% CGSolver::sxay() 1586 0.01507 0.01507 0.01507 1.76% FillBoundary_nowait() 4023 0.01501 0.01501 0.01501 1.75% Castro::construct_old_gravity() 10 0.01484 0.01484 0.01484 1.73% Gravity::get_old_grav_vector() 10 0.01483 0.01483 0.01483 1.73% MultiFab::LinComb() 1586 0.01469 0.01469 0.01469 1.71% FabArray::ParallelCopy() 861 0.01461 0.01461 0.01461 1.71% FabArray::setVal() 1144 0.01448 0.01448 0.01448 1.69% FabArray::ParallelCopy_nowait() 861 0.01433 0.01433 0.01433 1.67% MLCGSolver::ParallelAllReduce 1514 0.01353 0.01353 0.01353 1.58% MLMG::mgVcycle_up::2 82 0.01348 0.01348 0.01348 1.57% Castro::normalize_species() 62 0.01336 0.01336 0.01336 1.56% MLCellLinOp::defineAuxData() 11 0.01335 0.01335 0.01335 1.56% MLMG::mgVcycle_up::1 82 0.01327 0.01327 0.01327 1.55% MLMG::addInterpCorrection() 410 0.01307 0.01307 0.01307 1.53% MLMG::mgVcycle_up::3 82 0.01283 0.01283 0.01283 1.50% MLMG::mgVcycle_up::4 82 0.01265 0.01265 0.01265 1.48% amrex::average_down 410 0.01225 0.01225 0.01225 1.43% MLPoisson::Fapply() 1142 0.01188 0.01188 0.01188 1.39% Castro::expand_state() 10 0.01146 0.01146 0.01146 1.34% Castro::initialize_advance() 10 0.01142 0.01142 0.01142 1.33% Castro::do_new_sources() 10 0.01076 0.01076 0.01076 1.26% Castro::do_old_sources() 10 0.0104 0.0104 0.0104 1.21% Castro::enforce_min_density() 62 0.008678 0.008678 0.008678 1.01% Gravity::fill_multipole_BCs() 11 0.008413 0.008413 0.008413 0.98% MLCellLinOp::solutionResidual() 93 0.007149 0.007149 0.007149 0.83% Castro::post_timestep() 10 0.007085 0.007085 0.007085 0.83% MultiFab::Xpay() 585 0.006618 0.006618 0.006618 0.77% MLMG::computeResidual() 82 0.006167 0.006167 0.006167 0.72% Castro::reset_internal_energy(MultiFab) 63 0.005731 0.005731 0.005731 0.67% MLMG::prepareForSolve() 11 0.005477 0.005477 0.005477 0.64% MLCellLinOp::defineBC() 11 0.005158 0.005158 0.005158 0.60% Castro::estTimeStep() 21 0.005083 0.005083 0.005083 0.59% BndryData::define() 11 0.004881 0.004881 0.004881 0.57% Amr::InitializeInit() 1 0.004079 0.004079 0.004079 0.48% Amr::defBaseLevel() 1 0.004074 0.004074 0.004074 0.48% Castro::initData() 1 0.00356 0.00356 0.00356 0.42% Castro::construct_new_source() 50 0.003278 0.003278 0.003278 0.38% Castro::construct_new_gravity_source() 10 0.00326 0.00326 0.00326 0.38% Castro::construct_old_source() 50 0.002513 0.002513 0.002513 0.29% Castro::construct_old_gravity_source() 10 0.002495 0.002495 0.002495 0.29% Castro::computeNewDt() 9 0.002125 0.002125 0.002125 0.25% MLMG::ResNormInf() 93 0.002094 0.002094 0.002094 0.24% Castro::apply_source_to_state() 20 0.001827 0.001827 0.001827 0.21% Castro::enforce_speed_limit() 62 0.001826 0.001826 0.001826 0.21% MultiFab::Saxpy() 20 0.001816 0.001816 0.001816 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00167 0.00167 0.00167 0.19% MultiFab::Add() 82 0.001664 0.001664 0.001664 0.19% Castro::reset_internal_energy(Fab) 504 0.001618 0.001618 0.001618 0.19% MLCellLinOp::setLevelBC() 11 0.001552 0.001552 0.001552 0.18% FabArrayBase::getCPC() 1323 0.00144 0.00144 0.00144 0.17% MLMG::getGradSolution() 11 0.001431 0.001431 0.001431 0.17% MLCellLinOp::compGrad() 11 0.001424 0.001424 0.001424 0.17% FabArray::setDomainBndry() 41 0.00134 0.00134 0.00134 0.16% FabArray::mult() 43 0.001334 0.001334 0.001334 0.16% MLPoisson::prepareForSolve() 11 0.001213 0.001213 0.001213 0.14% MLCellLinOp::prepareForSolve() 11 0.001204 0.001204 0.001204 0.14% Castro::check_for_nan() 20 0.001192 0.001192 0.001192 0.14% MultiFab::contains_nan() 20 0.001179 0.001179 0.001179 0.14% Castro::post_regrid() 1 0.00116 0.00116 0.00116 0.14% MLMG::computeMLResidual() 11 0.001029 0.001029 0.001029 0.12% Gravity::update_max_rhs() 11 0.0008185 0.0008185 0.0008185 0.10% Castro::computeInitialDt() 2 0.0007364 0.0007364 0.0007364 0.09% FabArrayBase::getFB() 4023 0.0006781 0.0006781 0.0006781 0.08% FabArrayBase::CPC::define() 454 0.0006683 0.0006683 0.0006683 0.08% Amr::InitAmr() 1 0.0004817 0.0004817 0.0004817 0.06% Gravity::swapTimeLevels() 10 0.0004434 0.0004434 0.0004434 0.05% Castro::Castro() 1 0.0004386 0.0004386 0.0004386 0.05% MultiFab::Copy() 11 0.0003256 0.0003256 0.0003256 0.04% MLMG::MLResNormInf() 11 0.0002816 0.0002816 0.0002816 0.03% MultiFab::max() 11 0.000256 0.000256 0.000256 0.03% MLLinOp::define() 11 0.000222 0.000222 0.000222 0.03% MLMG::MLRhsNormInf() 11 0.0002172 0.0002172 0.0002172 0.03% MLLinOp::defineGrids() 11 0.0001981 0.0001981 0.0001981 0.02% Castro::buildMetrics() 1 0.0001602 0.0001602 0.0001602 0.02% FabArrayBase::FB::FB() 56 8.552e-05 8.552e-05 8.552e-05 0.01% Castro::finalize_advance() 10 7.581e-05 7.581e-05 7.581e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.438e-05 5.438e-05 5.438e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.1e-05 5.1e-05 5.1e-05 0.01% StateData::define() 4 4.395e-05 4.395e-05 4.395e-05 0.01% makeSFC 55 4.056e-05 4.056e-05 4.056e-05 0.00% Castro::swap_state_time_levels() 10 3.98e-05 3.98e-05 3.98e-05 0.00% Castro::enforce_consistent_e() 1 3.409e-05 3.409e-05 3.409e-05 0.00% Castro::finalize_do_advance() 10 3.366e-05 3.366e-05 3.366e-05 0.00% Castro::initMFs() 1 2.57e-05 2.57e-05 2.57e-05 0.00% Amr::writeSmallPlotFile() 1 2.471e-05 2.471e-05 2.471e-05 0.00% DistributionMapping::Distribute() 56 1.586e-05 1.586e-05 1.586e-05 0.00% Amr::initSubcycle() 1 8.679e-06 8.679e-06 8.679e-06 0.00% Gravity::set_mass_offset() 11 4.895e-06 4.895e-06 4.895e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.448e-06 4.448e-06 4.448e-06 0.00% AmrLevel::checkPointPost() 3 4.04e-06 4.04e-06 4.04e-06 0.00% Castro::create_source_corrector() 10 3.812e-06 3.812e-06 3.812e-06 0.00% Castro::retry_advance_ctu() 10 3.774e-06 3.774e-06 3.774e-06 0.00% Castro::FluxRegCrseInit 10 2.819e-06 2.819e-06 2.819e-06 0.00% Castro::FluxRegFineAdd() 10 2.531e-06 2.531e-06 2.531e-06 0.00% AmrLevel::checkPointPre() 3 2.381e-06 2.381e-06 2.381e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.936e-06 1.936e-06 1.936e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-24-gb84d7c069cef) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-24-gb84d7c069cef) initialized Starting run at 08:34:31 UTC on 2022-09-27. Successfully read inputs file ... Castro git describe: 22.09-1-g65b273ad0 AMReX git describe: 22.09-24-gb84d7c069 Microphysics git describe: 22.09-13-gb685b7d3 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.466399527 Restart time = 0.048341888 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.05172161 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.05179642 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.060043936 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060765068 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.064882252 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032086227 seconds Ending run at 08:34:32 UTC on 2022-09-27. Run time = 0.370694221 Run time without initialization = 0.321752787 Average number of zones advanced per microsecond: 4.074 Average number of zones advanced per microsecond per rank: 4.074 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3707 ... 0.3707 ... 0.3707 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0908 0.0908 0.0908 24.48% VisMF::Read() 3 0.04034 0.04034 0.04034 10.88% MLCellLinOp::applyBC() 1946 0.0347 0.0347 0.0347 9.36% VisMF::Write(FabArray) 1 0.03057 0.03057 0.03057 8.25% MLPoisson::Fsmooth() 1440 0.0271 0.0271 0.0271 7.31% StateData::FillBoundary(geom) 160 0.01182 0.01182 0.01182 3.19% MLCGSolver::bicgstab 36 0.01029 0.01029 0.01029 2.78% MultiFab::Dot() 484 0.009555 0.009555 0.009555 2.58% Castro::computeTemp() 30 0.007975 0.007975 0.007975 2.15% FabArray::setVal() 537 0.00674 0.00674 0.00674 1.82% Castro::normalize_species() 30 0.006729 0.006729 0.006729 1.82% MLCellLinOp::defineAuxData() 6 0.006318 0.006318 0.006318 1.70% Castro::enforce_min_density() 30 0.006279 0.006279 0.006279 1.69% FillBoundary_nowait() 1766 0.006239 0.006239 0.006239 1.68% MultiFab::LinComb() 690 0.00616 0.00616 0.00616 1.66% FabArray::ParallelCopy_nowait() 380 0.005931 0.005931 0.005931 1.60% StateDataPhysBCFunct::() 20 0.005358 0.005358 0.005358 1.45% MLPoisson::Fapply() 500 0.005051 0.005051 0.005051 1.36% Gravity::fill_multipole_BCs() 6 0.004404 0.004404 0.004404 1.19% Amr::restart() 1 0.003669 0.003669 0.003669 0.99% MLMG::addInterpCorrection() 180 0.00333 0.00333 0.00333 0.90% amrex::average_down 180 0.002952 0.002952 0.002952 0.80% MultiFab::Xpay() 258 0.002846 0.002846 0.002846 0.77% Castro::do_advance_ctu() 5 0.002541 0.002541 0.002541 0.69% Castro::estTimeStep() 10 0.002498 0.002498 0.002498 0.67% BndryData::define() 6 0.002134 0.002134 0.002134 0.58% Castro::construct_new_gravity_source() 5 0.00174 0.00174 0.00174 0.47% Castro::reset_internal_energy(MultiFab) 30 0.001708 0.001708 0.001708 0.46% Amr::writePlotFile() 1 0.001597 0.001597 0.001597 0.43% Castro::construct_old_gravity_source() 5 0.001424 0.001424 0.001424 0.38% Castro::enforce_speed_limit() 30 0.001372 0.001372 0.001372 0.37% Gravity::get_old_grav_vector() 5 0.0009614 0.0009614 0.0009614 0.26% MultiFab::Saxpy() 10 0.0009201 0.0009201 0.0009201 0.25% MLMG::ResNormInf() 42 0.0009106 0.0009106 0.0009106 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000894 0.000894 0.000894 0.24% Castro::reset_internal_energy(Fab) 240 0.0008857 0.0008857 0.0008857 0.24% Castro::post_timestep() 5 0.0008777 0.0008777 0.0008777 0.24% Castro::expand_state() 5 0.0008731 0.0008731 0.0008731 0.24% Gravity::get_new_grav_vector() 5 0.0008658 0.0008658 0.0008658 0.23% MLCellLinOp::setLevelBC() 6 0.0008253 0.0008253 0.0008253 0.22% Gravity::actual_solve_with_mlmg() 6 0.0008134 0.0008134 0.0008134 0.22% MultiFab::Add() 36 0.0007079 0.0007079 0.0007079 0.19% MLMG::prepareForSolve() 6 0.0006604 0.0006604 0.0006604 0.18% FabArray::mult() 22 0.0006474 0.0006474 0.0006474 0.17% FabArray::setDomainBndry() 20 0.0006402 0.0006402 0.0006402 0.17% MLCellLinOp::prepareForSolve() 6 0.0006354 0.0006354 0.0006354 0.17% MultiFab::contains_nan() 10 0.0005876 0.0005876 0.0005876 0.16% MLCellLinOp::smooth() 720 0.0005047 0.0005047 0.0005047 0.14% MLCellLinOp::compGrad() 6 0.0004894 0.0004894 0.0004894 0.13% Amr::InitAmr() 1 0.0004154 0.0004154 0.0004154 0.11% FabArrayBase::CPC::define() 244 0.0003987 0.0003987 0.0003987 0.11% FabArray::FillBoundary() 1766 0.000378 0.000378 0.000378 0.10% FabArrayBase::getCPC() 632 0.0003716 0.0003716 0.0003716 0.10% main() 1 0.0002919 0.0002919 0.0002919 0.08% Castro::subcycle_advance_ctu() 5 0.0002573 0.0002573 0.0002573 0.07% FabArrayBase::getFB() 1766 0.0002557 0.0002557 0.0002557 0.07% Gravity::update_max_rhs() 6 0.0002281 0.0002281 0.0002281 0.06% Amr::coarseTimeStep() 5 0.0002094 0.0002094 0.0002094 0.06% Gravity::solve_for_phi() 5 0.0002009 0.0002009 0.0002009 0.05% MLCellLinOp::apply() 500 0.0001931 0.0001931 0.0001931 0.05% Castro::construct_new_gravity() 5 0.0001803 0.0001803 0.0001803 0.05% Castro::create_source_corrector() 5 0.000178 0.000178 0.000178 0.05% MultiFab::Copy() 6 0.0001778 0.0001778 0.0001778 0.05% CGSolver::sxay() 690 0.0001673 0.0001673 0.0001673 0.05% Castro::construct_new_source() 25 0.0001648 0.0001648 0.0001648 0.04% MLCellLinOp::defineBC() 6 0.0001486 0.0001486 0.0001486 0.04% FillPatchIterator::Initialize 20 0.0001427 0.0001427 0.0001427 0.04% MultiFab::max() 6 0.0001357 0.0001357 0.0001357 0.04% MLCGSolver::ParallelAllReduce 659 0.0001288 0.0001288 0.0001288 0.03% FabArray::ParallelCopy() 380 0.0001234 0.0001234 0.0001234 0.03% MLMG::MLRhsNormInf() 6 0.0001129 0.0001129 0.0001129 0.03% MLMG::mgVcycle() 36 0.0001044 0.0001044 0.0001044 0.03% MLCellLinOp::correctionResidual() 216 0.0001019 0.0001019 0.0001019 0.03% Castro::initialize_do_advance() 5 9.951e-05 9.951e-05 9.951e-05 0.03% Castro::advance() 5 9.574e-05 9.574e-05 9.574e-05 0.03% MLLinOp::defineGrids() 6 8.883e-05 8.883e-05 8.883e-05 0.02% Amr::timeStep() 5 7.83e-05 7.83e-05 7.83e-05 0.02% AmrLevel::restart() 1 7.828e-05 7.828e-05 7.828e-05 0.02% StateData::restartDoit() 4 7.547e-05 7.547e-05 7.547e-05 0.02% Castro::finalize_advance() 5 6.878e-05 6.878e-05 6.878e-05 0.02% MLMG:computeResOfCorrection() 180 6.703e-05 6.703e-05 6.703e-05 0.02% Castro::initialize_advance() 5 5.688e-05 5.688e-05 5.688e-05 0.02% FabArrayBase::FB::FB() 26 5.522e-05 5.522e-05 5.522e-05 0.01% Castro::computeNewDt() 5 5.497e-05 5.497e-05 5.497e-05 0.01% Castro::construct_old_source() 25 5.425e-05 5.425e-05 5.425e-05 0.01% MLMG::mgVcycle_down::0 36 4.797e-05 4.797e-05 4.797e-05 0.01% Castro::clean_state() 30 4.552e-05 4.552e-05 4.552e-05 0.01% MLMG::mgVcycle_down::1 36 4.288e-05 4.288e-05 4.288e-05 0.01% MLMG::mgVcycle_down::4 36 4.238e-05 4.238e-05 4.238e-05 0.01% MLMG::mgVcycle_down::2 36 4.008e-05 4.008e-05 4.008e-05 0.01% MLMG::actualBottomSolve() 36 3.776e-05 3.776e-05 3.776e-05 0.01% MLMG::mgVcycle_down::3 36 3.687e-05 3.687e-05 3.687e-05 0.01% MLMG::solve() 6 3.289e-05 3.289e-05 3.289e-05 0.01% Castro::buildMetrics() 1 3.262e-05 3.262e-05 3.262e-05 0.01% MLMG::mgVcycle_up::4 36 3.12e-05 3.12e-05 3.12e-05 0.01% Castro::post_restart() 1 2.987e-05 2.987e-05 2.987e-05 0.01% Gravity::actual_multilevel_solve() 1 2.974e-05 2.974e-05 2.974e-05 0.01% MLMG::oneIter() 36 2.963e-05 2.963e-05 2.963e-05 0.01% Castro::initMFs() 1 2.784e-05 2.784e-05 2.784e-05 0.01% Castro::swap_state_time_levels() 5 2.727e-05 2.727e-05 2.727e-05 0.01% MLLinOp::define() 6 2.726e-05 2.726e-05 2.726e-05 0.01% Amr::writeSmallPlotFile() 1 2.597e-05 2.597e-05 2.597e-05 0.01% MLMG::mgVcycle_up::0 36 2.518e-05 2.518e-05 2.518e-05 0.01% MLMG::mgVcycle_up::3 36 2.504e-05 2.504e-05 2.504e-05 0.01% Castro::construct_old_gravity() 5 2.491e-05 2.491e-05 2.491e-05 0.01% MLMG::mgVcycle_up::2 36 2.377e-05 2.377e-05 2.377e-05 0.01% MLCellLinOp::solutionResidual() 42 2.281e-05 2.281e-05 2.281e-05 0.01% MLMG::mgVcycle_up::1 36 2.272e-05 2.272e-05 2.272e-05 0.01% MLMG::computeResidual() 36 1.859e-05 1.859e-05 1.859e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.822e-05 1.822e-05 1.822e-05 0.00% Castro::finalize_do_advance() 5 1.775e-05 1.775e-05 1.775e-05 0.00% makeSFC 30 1.467e-05 1.467e-05 1.467e-05 0.00% MLMG::mgVcycle_bottom 36 1.46e-05 1.46e-05 1.46e-05 0.00% FillPatchSingleLevel 20 1.351e-05 1.351e-05 1.351e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.327e-05 1.327e-05 1.327e-05 0.00% Castro::do_new_sources() 5 1.019e-05 1.019e-05 1.019e-05 0.00% MLPoisson::define() 6 9.871e-06 9.871e-06 9.871e-06 0.00% Amr::initSubcycle() 1 8.815e-06 8.815e-06 8.815e-06 0.00% DistributionMapping::Distribute() 31 8.594e-06 8.594e-06 8.594e-06 0.00% Castro::do_old_sources() 5 8.523e-06 8.523e-06 8.523e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.618e-06 7.618e-06 7.618e-06 0.00% Castro::check_for_nan() 10 6.575e-06 6.575e-06 6.575e-06 0.00% Castro::apply_source_to_state() 10 5.639e-06 5.639e-06 5.639e-06 0.00% MLPoisson::prepareForSolve() 6 4.836e-06 4.836e-06 4.836e-06 0.00% MLMG::computeMLResidual() 6 4.591e-06 4.591e-06 4.591e-06 0.00% Gravity::swapTimeLevels() 5 4.286e-06 4.286e-06 4.286e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.276e-06 3.276e-06 3.276e-06 0.00% MLMG::getGradSolution() 6 3.173e-06 3.173e-06 3.173e-06 0.00% Gravity::set_mass_offset() 6 2.693e-06 2.693e-06 2.693e-06 0.00% MLMG::MLResNormInf() 6 2.454e-06 2.454e-06 2.454e-06 0.00% Castro::retry_advance_ctu() 5 2.106e-06 2.106e-06 2.106e-06 0.00% Castro::FluxRegCrseInit 5 1.624e-06 1.624e-06 1.624e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.201e-06 1.201e-06 1.201e-06 0.00% Castro::FluxRegFineAdd() 5 1.125e-06 1.125e-06 1.125e-06 0.00% AmrLevel::AmrLevel() 1 1.064e-06 1.064e-06 1.064e-06 0.00% Amr::init() 1 9.09e-07 9.09e-07 9.09e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3707 0.3707 0.3707 100.00% Amr::coarseTimeStep() 5 0.2894 0.2894 0.2894 78.07% Amr::timeStep() 5 0.2879 0.2879 0.2879 77.67% Castro::advance() 5 0.2829 0.2829 0.2829 76.32% Castro::subcycle_advance_ctu() 5 0.2768 0.2768 0.2768 74.66% Castro::do_advance_ctu() 5 0.2765 0.2765 0.2765 74.59% Castro::construct_new_gravity() 5 0.1436 0.1436 0.1436 38.73% Gravity::solve_phi_with_mlmg() 6 0.1394 0.1394 0.1394 37.61% Gravity::solve_for_phi() 5 0.1358 0.1358 0.1358 36.64% Gravity::actual_solve_with_mlmg() 6 0.1349 0.1349 0.1349 36.39% MLMG::solve() 6 0.1225 0.1225 0.1225 33.04% MLMG::oneIter() 36 0.1153 0.1153 0.1153 31.10% MLMG::mgVcycle() 36 0.1146 0.1146 0.1146 30.90% Castro::construct_ctu_hydro_source() 5 0.09076 0.09076 0.09076 24.48% MLCellLinOp::smooth() 720 0.05847 0.05847 0.05847 15.77% Amr::init() 1 0.04838 0.04838 0.04838 13.05% Amr::restart() 1 0.04838 0.04838 0.04838 13.05% MLCellLinOp::applyBC() 1946 0.04162 0.04162 0.04162 11.23% AmrLevel::restart() 1 0.04054 0.04054 0.04054 10.94% StateData::restartDoit() 4 0.04046 0.04046 0.04046 10.91% VisMF::Read() 3 0.04034 0.04034 0.04034 10.88% MLMG::mgVcycle_bottom 36 0.03522 0.03522 0.03522 9.50% MLMG::actualBottomSolve() 36 0.0352 0.0352 0.0352 9.50% MLCGSolver::bicgstab 36 0.03484 0.03484 0.03484 9.40% Amr::writePlotFile() 1 0.03217 0.03217 0.03217 8.68% VisMF::Write(FabArray) 1 0.03057 0.03057 0.03057 8.25% MLPoisson::Fsmooth() 1440 0.0271 0.0271 0.0271 7.31% Castro::clean_state() 30 0.02499 0.02499 0.02499 6.74% FillPatchIterator::Initialize 20 0.01995 0.01995 0.01995 5.38% FillPatchSingleLevel 20 0.01917 0.01917 0.01917 5.17% StateDataPhysBCFunct::() 20 0.01717 0.01717 0.01717 4.63% MLCellLinOp::apply() 500 0.01573 0.01573 0.01573 4.24% MLMG::mgVcycle_down::0 36 0.01526 0.01526 0.01526 4.12% MLMG::mgVcycle_up::0 36 0.01305 0.01305 0.01305 3.52% StateData::FillBoundary(geom) 160 0.01182 0.01182 0.01182 3.19% Castro::computeTemp() 30 0.01057 0.01057 0.01057 2.85% Castro::initialize_do_advance() 5 0.01012 0.01012 0.01012 2.73% MLPoisson::define() 6 0.01003 0.01003 0.01003 2.71% MultiFab::Dot() 484 0.009555 0.009555 0.009555 2.58% MLCellLinOp::correctionResidual() 216 0.009162 0.009162 0.009162 2.47% MLMG:computeResOfCorrection() 180 0.007909 0.007909 0.007909 2.13% MLMG::mgVcycle_down::1 36 0.00761 0.00761 0.00761 2.05% Gravity::get_new_grav_vector() 5 0.007591 0.007591 0.007591 2.05% Castro::construct_old_gravity() 5 0.007529 0.007529 0.007529 2.03% Gravity::get_old_grav_vector() 5 0.007504 0.007504 0.007504 2.02% MLMG::mgVcycle_down::2 36 0.007406 0.007406 0.007406 2.00% Castro::do_new_sources() 5 0.007149 0.007149 0.007149 1.93% MLCellLinOp::defineAuxData() 6 0.007057 0.007057 0.007057 1.90% MLMG::mgVcycle_down::3 36 0.007014 0.007014 0.007014 1.89% FabArray::FillBoundary() 1766 0.006927 0.006927 0.006927 1.87% MLMG::mgVcycle_down::4 36 0.006774 0.006774 0.006774 1.83% FabArray::setVal() 537 0.00674 0.00674 0.00674 1.82% Castro::normalize_species() 30 0.006729 0.006729 0.006729 1.82% FillBoundary_nowait() 1766 0.006549 0.006549 0.006549 1.77% Castro::do_old_sources() 5 0.006482 0.006482 0.006482 1.75% FabArray::ParallelCopy() 380 0.006442 0.006442 0.006442 1.74% CGSolver::sxay() 690 0.006327 0.006327 0.006327 1.71% FabArray::ParallelCopy_nowait() 380 0.006319 0.006319 0.006319 1.70% Castro::enforce_min_density() 30 0.006279 0.006279 0.006279 1.69% MultiFab::LinComb() 690 0.00616 0.00616 0.00616 1.66% Castro::initialize_advance() 5 0.006002 0.006002 0.006002 1.62% MLCGSolver::ParallelAllReduce 659 0.005731 0.005731 0.005731 1.55% MLMG::mgVcycle_up::2 36 0.005712 0.005712 0.005712 1.54% MLMG::mgVcycle_up::1 36 0.005619 0.005619 0.005619 1.52% Castro::expand_state() 5 0.005572 0.005572 0.005572 1.50% MLMG::addInterpCorrection() 180 0.005544 0.005544 0.005544 1.50% MLMG::mgVcycle_up::3 36 0.005428 0.005428 0.005428 1.46% MLMG::mgVcycle_up::4 36 0.005378 0.005378 0.005378 1.45% amrex::average_down 180 0.005203 0.005203 0.005203 1.40% MLPoisson::Fapply() 500 0.005051 0.005051 0.005051 1.36% Castro::post_timestep() 5 0.004922 0.004922 0.004922 1.33% Gravity::fill_multipole_BCs() 6 0.004404 0.004404 0.004404 1.19% Castro::post_restart() 1 0.003991 0.003991 0.003991 1.08% Gravity::multilevel_solve_for_new_phi() 1 0.003866 0.003866 0.003866 1.04% Gravity::actual_multilevel_solve() 1 0.003848 0.003848 0.003848 1.04% MLCellLinOp::solutionResidual() 42 0.003181 0.003181 0.003181 0.86% MLMG::prepareForSolve() 6 0.00291 0.00291 0.00291 0.78% MultiFab::Xpay() 258 0.002846 0.002846 0.002846 0.77% MLCellLinOp::defineBC() 6 0.00282 0.00282 0.00282 0.76% BndryData::define() 6 0.002671 0.002671 0.002671 0.72% MLMG::computeResidual() 36 0.00264 0.00264 0.00264 0.71% Castro::reset_internal_energy(MultiFab) 30 0.002594 0.002594 0.002594 0.70% Castro::estTimeStep() 10 0.002498 0.002498 0.002498 0.67% Castro::construct_new_source() 25 0.001904 0.001904 0.001904 0.51% Castro::construct_new_gravity_source() 5 0.00174 0.00174 0.00174 0.47% Castro::construct_old_source() 25 0.001479 0.001479 0.001479 0.40% Castro::construct_old_gravity_source() 5 0.001424 0.001424 0.001424 0.38% Castro::enforce_speed_limit() 30 0.001372 0.001372 0.001372 0.37% Castro::computeNewDt() 5 0.001249 0.001249 0.001249 0.34% Castro::apply_source_to_state() 10 0.0009257 0.0009257 0.0009257 0.25% MultiFab::Saxpy() 10 0.0009201 0.0009201 0.0009201 0.25% MLMG::ResNormInf() 42 0.0009106 0.0009106 0.0009106 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000894 0.000894 0.000894 0.24% Castro::reset_internal_energy(Fab) 240 0.0008857 0.0008857 0.0008857 0.24% MLCellLinOp::setLevelBC() 6 0.0008253 0.0008253 0.0008253 0.22% FabArrayBase::getCPC() 632 0.0007703 0.0007703 0.0007703 0.21% MLMG::getGradSolution() 6 0.0007666 0.0007666 0.0007666 0.21% MLCellLinOp::compGrad() 6 0.0007635 0.0007635 0.0007635 0.21% MultiFab::Add() 36 0.0007079 0.0007079 0.0007079 0.19% FabArray::mult() 22 0.0006474 0.0006474 0.0006474 0.17% MLPoisson::prepareForSolve() 6 0.0006403 0.0006403 0.0006403 0.17% FabArray::setDomainBndry() 20 0.0006402 0.0006402 0.0006402 0.17% MLCellLinOp::prepareForSolve() 6 0.0006354 0.0006354 0.0006354 0.17% Castro::check_for_nan() 10 0.0005942 0.0005942 0.0005942 0.16% MultiFab::contains_nan() 10 0.0005876 0.0005876 0.0005876 0.16% MLMG::computeMLResidual() 6 0.0005638 0.0005638 0.0005638 0.15% Gravity::update_max_rhs() 6 0.0004424 0.0004424 0.0004424 0.12% Amr::InitAmr() 1 0.0004243 0.0004243 0.0004243 0.11% FabArrayBase::CPC::define() 244 0.0003987 0.0003987 0.0003987 0.11% FabArrayBase::getFB() 1766 0.0003109 0.0003109 0.0003109 0.08% Gravity::swapTimeLevels() 5 0.000231 0.000231 0.000231 0.06% Castro::create_source_corrector() 5 0.000178 0.000178 0.000178 0.05% MultiFab::Copy() 6 0.0001778 0.0001778 0.0001778 0.05% Castro::buildMetrics() 1 0.0001506 0.0001506 0.0001506 0.04% MLLinOp::define() 6 0.0001468 0.0001468 0.0001468 0.04% MLMG::MLResNormInf() 6 0.0001468 0.0001468 0.0001468 0.04% MultiFab::max() 6 0.0001357 0.0001357 0.0001357 0.04% MLLinOp::defineGrids() 6 0.0001196 0.0001196 0.0001196 0.03% MLMG::MLRhsNormInf() 6 0.0001129 0.0001129 0.0001129 0.03% Castro::finalize_advance() 5 7.153e-05 7.153e-05 7.153e-05 0.02% FabArrayBase::FB::FB() 26 5.522e-05 5.522e-05 5.522e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.955e-05 2.955e-05 2.955e-05 0.01% Castro::initMFs() 1 2.784e-05 2.784e-05 2.784e-05 0.01% Castro::swap_state_time_levels() 5 2.727e-05 2.727e-05 2.727e-05 0.01% Amr::writeSmallPlotFile() 1 2.597e-05 2.597e-05 2.597e-05 0.01% makeSFC 30 2.194e-05 2.194e-05 2.194e-05 0.01% Castro::finalize_do_advance() 5 1.775e-05 1.775e-05 1.775e-05 0.00% Amr::initSubcycle() 1 8.815e-06 8.815e-06 8.815e-06 0.00% DistributionMapping::Distribute() 31 8.594e-06 8.594e-06 8.594e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.599e-06 4.599e-06 4.599e-06 0.00% Gravity::set_mass_offset() 6 2.693e-06 2.693e-06 2.693e-06 0.00% Castro::retry_advance_ctu() 5 2.106e-06 2.106e-06 2.106e-06 0.00% Castro::FluxRegCrseInit 5 1.624e-06 1.624e-06 1.624e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.201e-06 1.201e-06 1.201e-06 0.00% Castro::FluxRegFineAdd() 5 1.125e-06 1.125e-06 1.125e-06 0.00% AmrLevel::AmrLevel() 1 1.064e-06 1.064e-06 1.064e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-24-gb84d7c069cef) finalized