Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.11-6-g6044e9c2bb9a) initialized Starting run at 08:49:38 UTC on 2022-11-04. Successfully read inputs file ... Castro git describe: 22.11-1-gb27642d86 AMReX git describe: 22.11-6-g6044e9c2b Microphysics git describe: 22.11-4-gcfe7aa24 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.053666102 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030881071 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.049349127 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050918471 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.051774743 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.06282404 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.077121498 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.049394313 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.062869407 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.061552062 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.062060413 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.061810569 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065414458 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.061657834 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031314201 seconds Ending run at 08:49:39 UTC on 2022-11-04. Run time = 0.882683245 Run time without initialization = 0.748696662 Average number of zones advanced per microsecond: 3.501 Average number of zones advanced per microsecond per rank: 3.501 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8827 ... 0.8827 ... 0.8827 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2189 0.2189 0.2189 24.80% Castro::construct_ctu_hydro_source() 10 0.214 0.214 0.214 24.24% MLCellLinOp::applyBC() 4379 0.08109 0.08109 0.08109 9.19% MLPoisson::Fsmooth() 3240 0.06429 0.06429 0.06429 7.28% MLCGSolver::bicgstab 81 0.02408 0.02408 0.02408 2.73% StateData::FillBoundary(geom) 328 0.02348 0.02348 0.02348 2.66% MultiFab::Dot() 1100 0.02255 0.02255 0.02255 2.55% Castro::normalize_species() 62 0.01541 0.01541 0.01541 1.75% MultiFab::LinComb() 1566 0.01441 0.01441 0.01441 1.63% FabArray::setVal() 1135 0.01432 0.01432 0.01432 1.62% FillBoundary_nowait() 3974 0.01416 0.01416 0.01416 1.60% Castro::computeTemp() 63 0.0141 0.0141 0.0141 1.60% FabArray::ParallelCopy_nowait() 851 0.01335 0.01335 0.01335 1.51% StateDataPhysBCFunct::() 41 0.01294 0.01294 0.01294 1.47% MLPoisson::Fapply() 1128 0.01193 0.01193 0.01193 1.35% MLCellLinOp::defineAuxData() 11 0.01174 0.01174 0.01174 1.33% Castro::enforce_min_density() 62 0.01041 0.01041 0.01041 1.18% Gravity::fill_multipole_BCs() 11 0.008568 0.008568 0.008568 0.97% MLMG::addInterpCorrection() 405 0.007768 0.007768 0.007768 0.88% amrex::average_down 405 0.006831 0.006831 0.006831 0.77% MultiFab::Xpay() 578 0.006612 0.006612 0.006612 0.75% Amr::checkPoint() 3 0.005208 0.005208 0.005208 0.59% Castro::do_advance_ctu() 10 0.005129 0.005129 0.005129 0.58% Castro::reset_internal_energy(MultiFab) 63 0.004115 0.004115 0.004115 0.47% BndryData::define() 11 0.004024 0.004024 0.004024 0.46% Castro::estTimeStep() 21 0.003728 0.003728 0.003728 0.42% Castro::construct_new_gravity_source() 10 0.00321 0.00321 0.00321 0.36% Amr::writePlotFile() 2 0.002861 0.002861 0.002861 0.32% Castro::construct_old_gravity_source() 10 0.002527 0.002527 0.002527 0.29% MLMG::ResNormInf() 92 0.002062 0.002062 0.002062 0.23% Gravity::get_new_grav_vector() 11 0.001937 0.001937 0.001937 0.22% MultiFab::Saxpy() 20 0.001807 0.001807 0.001807 0.20% Gravity::get_old_grav_vector() 10 0.001762 0.001762 0.001762 0.20% Castro::expand_state() 10 0.001743 0.001743 0.001743 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001696 0.001696 0.001696 0.19% MultiFab::Add() 81 0.001635 0.001635 0.001635 0.19% Castro::reset_internal_energy(Fab) 504 0.001617 0.001617 0.001617 0.18% MLCellLinOp::setLevelBC() 11 0.001551 0.001551 0.001551 0.18% Gravity::actual_solve_with_mlmg() 11 0.0014 0.0014 0.0014 0.16% Castro::subcycle_advance_ctu() 10 0.001392 0.001392 0.001392 0.16% FabArray::setDomainBndry() 41 0.001347 0.001347 0.001347 0.15% FabArray::mult() 43 0.001326 0.001326 0.001326 0.15% Castro::initData() 1 0.001271 0.001271 0.001271 0.14% MLMG::prepareForSolve() 11 0.001229 0.001229 0.001229 0.14% MLCellLinOp::prepareForSolve() 11 0.001194 0.001194 0.001194 0.14% MultiFab::contains_nan() 20 0.001193 0.001193 0.001193 0.14% MLCellLinOp::smooth() 1620 0.001105 0.001105 0.001105 0.13% Castro::enforce_speed_limit() 62 0.001011 0.001011 0.001011 0.11% MLCellLinOp::compGrad() 11 0.0009264 0.0009264 0.0009264 0.10% FabArray::FillBoundary() 3974 0.0008383 0.0008383 0.0008383 0.09% FabArrayBase::getCPC() 1313 0.0007741 0.0007741 0.0007741 0.09% FabArrayBase::CPC::define() 454 0.0006832 0.0006832 0.0006832 0.08% Castro::finalize_advance() 10 0.0006379 0.0006379 0.0006379 0.07% FabArrayBase::getFB() 3974 0.0005795 0.0005795 0.0005795 0.07% Amr::InitAmr() 1 0.0004776 0.0004776 0.0004776 0.05% Gravity::solve_for_phi() 10 0.0004533 0.0004533 0.0004533 0.05% MLCellLinOp::apply() 1128 0.0004449 0.0004449 0.0004449 0.05% Gravity::update_max_rhs() 11 0.0004257 0.0004257 0.0004257 0.05% Amr::coarseTimeStep() 10 0.0003386 0.0003386 0.0003386 0.04% CGSolver::sxay() 1566 0.0003333 0.0003333 0.0003333 0.04% MultiFab::Copy() 11 0.0003308 0.0003308 0.0003308 0.04% main() 1 0.0002911 0.0002911 0.0002911 0.03% FillPatchIterator::Initialize 41 0.0002851 0.0002851 0.0002851 0.03% MLCGSolver::ParallelAllReduce 1495 0.0002845 0.0002845 0.0002845 0.03% MLCellLinOp::defineBC() 11 0.0002729 0.0002729 0.0002729 0.03% FabArray::ParallelCopy() 851 0.0002588 0.0002588 0.0002588 0.03% MultiFab::max() 11 0.0002561 0.0002561 0.0002561 0.03% MLMG::MLRhsNormInf() 11 0.0002214 0.0002214 0.0002214 0.03% MLCellLinOp::correctionResidual() 486 0.0002099 0.0002099 0.0002099 0.02% MLMG::mgVcycle() 81 0.0002043 0.0002043 0.0002043 0.02% Castro::construct_new_gravity() 10 0.0002042 0.0002042 0.0002042 0.02% Amr::timeStep() 10 0.0001598 0.0001598 0.0001598 0.02% MLMG:computeResOfCorrection() 405 0.0001546 0.0001546 0.0001546 0.02% MLLinOp::defineGrids() 11 0.0001304 0.0001304 0.0001304 0.01% StateData::checkPoint() 12 0.0001288 0.0001288 0.0001288 0.01% MLMG::mgVcycle_down::0 81 0.0001185 0.0001185 0.0001185 0.01% MLMG::mgVcycle_down::1 81 0.0001017 0.0001017 0.0001017 0.01% MLMG::mgVcycle_down::2 81 9.515e-05 9.515e-05 9.515e-05 0.01% MLMG::mgVcycle_down::3 81 8.907e-05 8.907e-05 8.907e-05 0.01% MLMG::actualBottomSolve() 81 8.86e-05 8.86e-05 8.86e-05 0.01% MLMG::mgVcycle_down::4 81 8.71e-05 8.71e-05 8.71e-05 0.01% Castro::Castro() 1 8.63e-05 8.63e-05 8.63e-05 0.01% FabArrayBase::FB::FB() 56 8.471e-05 8.471e-05 8.471e-05 0.01% Castro::initialize_advance() 10 8.268e-05 8.268e-05 8.268e-05 0.01% Castro::clean_state() 62 7.912e-05 7.912e-05 7.912e-05 0.01% AmrLevel::checkPoint() 3 7.364e-05 7.364e-05 7.364e-05 0.01% MLMG::mgVcycle_up::4 81 6.894e-05 6.894e-05 6.894e-05 0.01% MLMG::solve() 11 6.663e-05 6.663e-05 6.663e-05 0.01% MLMG::oneIter() 81 6.502e-05 6.502e-05 6.502e-05 0.01% Castro::initialize_do_advance() 10 6.275e-05 6.275e-05 6.275e-05 0.01% Castro::advance() 10 6.147e-05 6.147e-05 6.147e-05 0.01% MLMG::mgVcycle_up::0 81 5.865e-05 5.865e-05 5.865e-05 0.01% MLMG::mgVcycle_up::3 81 5.677e-05 5.677e-05 5.677e-05 0.01% MLMG::mgVcycle_up::1 81 5.668e-05 5.668e-05 5.668e-05 0.01% MLMG::mgVcycle_up::2 81 5.506e-05 5.506e-05 5.506e-05 0.01% MLCellLinOp::solutionResidual() 92 5.128e-05 5.128e-05 5.128e-05 0.01% StateData::define() 4 4.295e-05 4.295e-05 4.295e-05 0.00% MLMG::computeResidual() 81 4.08e-05 4.08e-05 4.08e-05 0.00% Castro::swap_state_time_levels() 10 3.763e-05 3.763e-05 3.763e-05 0.00% Castro::initMFs() 1 3.511e-05 3.511e-05 3.511e-05 0.00% Castro::enforce_consistent_e() 1 3.395e-05 3.395e-05 3.395e-05 0.00% MLMG::mgVcycle_bottom 81 3.28e-05 3.28e-05 3.28e-05 0.00% Castro::finalize_do_advance() 10 3.227e-05 3.227e-05 3.227e-05 0.00% Gravity::actual_multilevel_solve() 1 3.108e-05 3.108e-05 3.108e-05 0.00% Castro::post_timestep() 10 3.086e-05 3.086e-05 3.086e-05 0.00% MLPoisson::define() 11 3.012e-05 3.012e-05 3.012e-05 0.00% FillPatchSingleLevel 41 2.693e-05 2.693e-05 2.693e-05 0.00% Amr::writeSmallPlotFile() 1 2.608e-05 2.608e-05 2.608e-05 0.00% makeSFC 55 2.605e-05 2.605e-05 2.605e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.417e-05 2.417e-05 2.417e-05 0.00% Castro::buildMetrics() 1 2.386e-05 2.386e-05 2.386e-05 0.00% Amr::defBaseLevel() 1 2.167e-05 2.167e-05 2.167e-05 0.00% MLLinOp::define() 11 2.146e-05 2.146e-05 2.146e-05 0.00% Amr::FinalizeInit() 1 1.849e-05 1.849e-05 1.849e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.777e-05 1.777e-05 1.777e-05 0.00% Castro::construct_new_source() 50 1.732e-05 1.732e-05 1.732e-05 0.00% Castro::do_new_sources() 10 1.696e-05 1.696e-05 1.696e-05 0.00% Castro::construct_old_source() 50 1.585e-05 1.585e-05 1.585e-05 0.00% Castro::do_old_sources() 10 1.526e-05 1.526e-05 1.526e-05 0.00% MLMG::computeMLResidual() 11 1.487e-05 1.487e-05 1.487e-05 0.00% DistributionMapping::Distribute() 56 1.422e-05 1.422e-05 1.422e-05 0.00% Gravity::swapTimeLevels() 10 1.325e-05 1.325e-05 1.325e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.295e-05 1.295e-05 1.295e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.226e-05 1.226e-05 1.226e-05 0.00% Castro::check_for_nan() 20 1.215e-05 1.215e-05 1.215e-05 0.00% Castro::apply_source_to_state() 20 1.091e-05 1.091e-05 1.091e-05 0.00% Castro::construct_old_gravity() 10 1.005e-05 1.005e-05 1.005e-05 0.00% Amr::initSubcycle() 1 8.618e-06 8.618e-06 8.618e-06 0.00% MLPoisson::prepareForSolve() 11 8.543e-06 8.543e-06 8.543e-06 0.00% Castro::computeNewDt() 9 7.3e-06 7.3e-06 7.3e-06 0.00% MLMG::getGradSolution() 11 6.696e-06 6.696e-06 6.696e-06 0.00% Amr::InitializeInit() 1 5.061e-06 5.061e-06 5.061e-06 0.00% Castro::retry_advance_ctu() 10 4.602e-06 4.602e-06 4.602e-06 0.00% AmrLevel::checkPointPost() 3 4.582e-06 4.582e-06 4.582e-06 0.00% Castro::create_source_corrector() 10 4.144e-06 4.144e-06 4.144e-06 0.00% MLMG::MLResNormInf() 11 3.578e-06 3.578e-06 3.578e-06 0.00% Castro::post_init() 1 3.573e-06 3.573e-06 3.573e-06 0.00% Gravity::set_mass_offset() 11 3.486e-06 3.486e-06 3.486e-06 0.00% Castro::FluxRegCrseInit 10 3.421e-06 3.421e-06 3.421e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.311e-06 3.311e-06 3.311e-06 0.00% Castro::FluxRegFineAdd() 10 2.645e-06 2.645e-06 2.645e-06 0.00% AmrLevel::checkPointPre() 3 2.534e-06 2.534e-06 2.534e-06 0.00% Amr::init() 1 2.437e-06 2.437e-06 2.437e-06 0.00% Castro::computeInitialDt() 2 2.397e-06 2.397e-06 2.397e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.744e-06 1.744e-06 1.744e-06 0.00% Amr::initialInit() 1 1.257e-06 1.257e-06 1.257e-06 0.00% Castro::post_regrid() 1 1.206e-06 1.206e-06 1.206e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8827 0.8827 0.8827 100.00% Amr::coarseTimeStep() 10 0.7172 0.7172 0.7172 81.24% Amr::timeStep() 10 0.6035 0.6035 0.6035 68.37% Castro::advance() 10 0.5966 0.5966 0.5966 67.58% Castro::subcycle_advance_ctu() 10 0.5846 0.5846 0.5846 66.23% Castro::do_advance_ctu() 10 0.5832 0.5832 0.5832 66.07% Gravity::solve_phi_with_mlmg() 11 0.3171 0.3171 0.3171 35.92% Gravity::actual_solve_with_mlmg() 11 0.3083 0.3083 0.3083 34.93% Castro::construct_new_gravity() 10 0.2917 0.2917 0.2917 33.05% MLMG::solve() 11 0.2853 0.2853 0.2853 32.32% Gravity::solve_for_phi() 10 0.2762 0.2762 0.2762 31.29% MLMG::oneIter() 81 0.2703 0.2703 0.2703 30.62% MLMG::mgVcycle() 81 0.2686 0.2686 0.2686 30.43% VisMF::Write(FabArray) 11 0.2189 0.2189 0.2189 24.80% Castro::construct_ctu_hydro_source() 10 0.214 0.214 0.214 24.24% Amr::checkPoint() 3 0.1648 0.1648 0.1648 18.67% AmrLevel::checkPoint() 3 0.1596 0.1596 0.1596 18.08% StateData::checkPoint() 12 0.1596 0.1596 0.1596 18.08% MLCellLinOp::smooth() 1620 0.1374 0.1374 0.1374 15.57% Amr::init() 1 0.1334 0.1334 0.1334 15.11% MLCellLinOp::applyBC() 4379 0.09676 0.09676 0.09676 10.96% MLMG::mgVcycle_bottom 81 0.08247 0.08247 0.08247 9.34% MLMG::actualBottomSolve() 81 0.08244 0.08244 0.08244 9.34% MLCGSolver::bicgstab 81 0.08161 0.08161 0.08161 9.25% MLPoisson::Fsmooth() 3240 0.06429 0.06429 0.06429 7.28% Amr::writePlotFile() 2 0.06235 0.06235 0.06235 7.06% Amr::initialInit() 1 0.04868 0.04868 0.04868 5.51% Castro::clean_state() 62 0.0459 0.0459 0.0459 5.20% Amr::FinalizeInit() 1 0.04455 0.04455 0.04455 5.05% Castro::post_init() 1 0.04324 0.04324 0.04324 4.90% FillPatchIterator::Initialize 41 0.04205 0.04205 0.04205 4.76% Gravity::multilevel_solve_for_new_phi() 1 0.04141 0.04141 0.04141 4.69% Gravity::actual_multilevel_solve() 1 0.04139 0.04139 0.04139 4.69% FillPatchSingleLevel 41 0.04042 0.04042 0.04042 4.58% MLCellLinOp::apply() 1128 0.03662 0.03662 0.03662 4.15% StateDataPhysBCFunct::() 41 0.03642 0.03642 0.03642 4.13% MLMG::mgVcycle_down::0 81 0.0355 0.0355 0.0355 4.02% MLMG::mgVcycle_up::0 81 0.03057 0.03057 0.03057 3.46% StateData::FillBoundary(geom) 328 0.02348 0.02348 0.02348 2.66% MultiFab::Dot() 1100 0.02255 0.02255 0.02255 2.55% MLCellLinOp::correctionResidual() 486 0.0215 0.0215 0.0215 2.44% Castro::computeTemp() 63 0.01983 0.01983 0.01983 2.25% Castro::initialize_do_advance() 10 0.01938 0.01938 0.01938 2.20% MLPoisson::define() 11 0.0186 0.0186 0.0186 2.11% MLMG:computeResOfCorrection() 405 0.01858 0.01858 0.01858 2.10% MLMG::mgVcycle_down::1 81 0.01792 0.01792 0.01792 2.03% MLMG::mgVcycle_down::2 81 0.01751 0.01751 0.01751 1.98% Gravity::get_new_grav_vector() 11 0.01706 0.01706 0.01706 1.93% MLMG::mgVcycle_down::3 81 0.0166 0.0166 0.0166 1.88% MLMG::mgVcycle_down::4 81 0.01577 0.01577 0.01577 1.79% FabArray::FillBoundary() 3974 0.01567 0.01567 0.01567 1.77% Castro::normalize_species() 62 0.01541 0.01541 0.01541 1.75% FillBoundary_nowait() 3974 0.01483 0.01483 0.01483 1.68% Castro::construct_old_gravity() 10 0.01483 0.01483 0.01483 1.68% Gravity::get_old_grav_vector() 10 0.01482 0.01482 0.01482 1.68% CGSolver::sxay() 1566 0.01475 0.01475 0.01475 1.67% FabArray::ParallelCopy() 851 0.01444 0.01444 0.01444 1.64% MultiFab::LinComb() 1566 0.01441 0.01441 0.01441 1.63% FabArray::setVal() 1135 0.01432 0.01432 0.01432 1.62% FabArray::ParallelCopy_nowait() 851 0.01418 0.01418 0.01418 1.61% MLMG::mgVcycle_up::2 81 0.01347 0.01347 0.01347 1.53% MLCGSolver::ParallelAllReduce 1495 0.01344 0.01344 0.01344 1.52% MLMG::mgVcycle_up::1 81 0.01328 0.01328 0.01328 1.50% MLCellLinOp::defineAuxData() 11 0.01311 0.01311 0.01311 1.49% MLMG::addInterpCorrection() 405 0.01299 0.01299 0.01299 1.47% MLMG::mgVcycle_up::3 81 0.01276 0.01276 0.01276 1.45% MLMG::mgVcycle_up::4 81 0.01254 0.01254 0.01254 1.42% Castro::do_new_sources() 10 0.01212 0.01212 0.01212 1.37% amrex::average_down 405 0.01208 0.01208 0.01208 1.37% MLPoisson::Fapply() 1128 0.01193 0.01193 0.01193 1.35% Castro::expand_state() 10 0.01137 0.01137 0.01137 1.29% Castro::initialize_advance() 10 0.01121 0.01121 0.01121 1.27% Castro::do_old_sources() 10 0.01065 0.01065 0.01065 1.21% Castro::enforce_min_density() 62 0.01041 0.01041 0.01041 1.18% Gravity::fill_multipole_BCs() 11 0.008568 0.008568 0.008568 0.97% MLCellLinOp::solutionResidual() 92 0.007152 0.007152 0.007152 0.81% Castro::post_timestep() 10 0.006754 0.006754 0.006754 0.77% MultiFab::Xpay() 578 0.006612 0.006612 0.006612 0.75% MLMG::computeResidual() 81 0.006174 0.006174 0.006174 0.70% Castro::reset_internal_energy(MultiFab) 63 0.005732 0.005732 0.005732 0.65% MLMG::prepareForSolve() 11 0.005469 0.005469 0.005469 0.62% MLCellLinOp::defineBC() 11 0.005259 0.005259 0.005259 0.60% BndryData::define() 11 0.004986 0.004986 0.004986 0.56% Amr::InitializeInit() 1 0.004132 0.004132 0.004132 0.47% Amr::defBaseLevel() 1 0.004127 0.004127 0.004127 0.47% Castro::estTimeStep() 21 0.003728 0.003728 0.003728 0.42% Castro::initData() 1 0.003595 0.003595 0.003595 0.41% Castro::construct_new_source() 50 0.003227 0.003227 0.003227 0.37% Castro::construct_new_gravity_source() 10 0.00321 0.00321 0.00321 0.36% Castro::construct_old_source() 50 0.002543 0.002543 0.002543 0.29% Castro::construct_old_gravity_source() 10 0.002527 0.002527 0.002527 0.29% MLMG::ResNormInf() 92 0.002062 0.002062 0.002062 0.23% Castro::apply_source_to_state() 20 0.001818 0.001818 0.001818 0.21% MultiFab::Saxpy() 20 0.001807 0.001807 0.001807 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001696 0.001696 0.001696 0.19% MultiFab::Add() 81 0.001635 0.001635 0.001635 0.19% Castro::reset_internal_energy(Fab) 504 0.001617 0.001617 0.001617 0.18% MLCellLinOp::setLevelBC() 11 0.001551 0.001551 0.001551 0.18% Castro::computeNewDt() 9 0.001516 0.001516 0.001516 0.17% FabArrayBase::getCPC() 1313 0.001457 0.001457 0.001457 0.17% MLMG::getGradSolution() 11 0.001444 0.001444 0.001444 0.16% MLCellLinOp::compGrad() 11 0.001437 0.001437 0.001437 0.16% FabArray::setDomainBndry() 41 0.001347 0.001347 0.001347 0.15% FabArray::mult() 43 0.001326 0.001326 0.001326 0.15% Castro::check_for_nan() 20 0.001205 0.001205 0.001205 0.14% MLPoisson::prepareForSolve() 11 0.001203 0.001203 0.001203 0.14% MLCellLinOp::prepareForSolve() 11 0.001194 0.001194 0.001194 0.14% MultiFab::contains_nan() 20 0.001193 0.001193 0.001193 0.14% Castro::post_regrid() 1 0.001129 0.001129 0.001129 0.13% MLMG::computeMLResidual() 11 0.001034 0.001034 0.001034 0.12% Castro::enforce_speed_limit() 62 0.001011 0.001011 0.001011 0.11% Castro::computeInitialDt() 2 0.0008555 0.0008555 0.0008555 0.10% Gravity::update_max_rhs() 11 0.0008313 0.0008313 0.0008313 0.09% FabArrayBase::CPC::define() 454 0.0006832 0.0006832 0.0006832 0.08% FabArrayBase::getFB() 3974 0.0006642 0.0006642 0.0006642 0.08% Castro::finalize_advance() 10 0.0006439 0.0006439 0.0006439 0.07% Amr::InitAmr() 1 0.0004862 0.0004862 0.0004862 0.06% Gravity::swapTimeLevels() 10 0.0004534 0.0004534 0.0004534 0.05% Castro::Castro() 1 0.0004501 0.0004501 0.0004501 0.05% MultiFab::Copy() 11 0.0003308 0.0003308 0.0003308 0.04% MLMG::MLResNormInf() 11 0.0002831 0.0002831 0.0002831 0.03% MultiFab::max() 11 0.0002561 0.0002561 0.0002561 0.03% MLMG::MLRhsNormInf() 11 0.0002214 0.0002214 0.0002214 0.03% MLLinOp::define() 11 0.000205 0.000205 0.000205 0.02% MLLinOp::defineGrids() 11 0.0001835 0.0001835 0.0001835 0.02% Castro::buildMetrics() 1 0.0001628 0.0001628 0.0001628 0.02% FabArrayBase::FB::FB() 56 8.471e-05 8.471e-05 8.471e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.59e-05 5.59e-05 5.59e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.14e-05 5.14e-05 5.14e-05 0.01% StateData::define() 4 4.295e-05 4.295e-05 4.295e-05 0.00% makeSFC 55 3.914e-05 3.914e-05 3.914e-05 0.00% Castro::swap_state_time_levels() 10 3.763e-05 3.763e-05 3.763e-05 0.00% Castro::initMFs() 1 3.511e-05 3.511e-05 3.511e-05 0.00% Castro::enforce_consistent_e() 1 3.395e-05 3.395e-05 3.395e-05 0.00% Castro::finalize_do_advance() 10 3.227e-05 3.227e-05 3.227e-05 0.00% Amr::writeSmallPlotFile() 1 2.608e-05 2.608e-05 2.608e-05 0.00% DistributionMapping::Distribute() 56 1.422e-05 1.422e-05 1.422e-05 0.00% Amr::initSubcycle() 1 8.618e-06 8.618e-06 8.618e-06 0.00% Castro::retry_advance_ctu() 10 4.602e-06 4.602e-06 4.602e-06 0.00% AmrLevel::checkPointPost() 3 4.582e-06 4.582e-06 4.582e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.441e-06 4.441e-06 4.441e-06 0.00% Castro::create_source_corrector() 10 4.144e-06 4.144e-06 4.144e-06 0.00% Gravity::set_mass_offset() 11 3.486e-06 3.486e-06 3.486e-06 0.00% Castro::FluxRegCrseInit 10 3.421e-06 3.421e-06 3.421e-06 0.00% Castro::FluxRegFineAdd() 10 2.645e-06 2.645e-06 2.645e-06 0.00% AmrLevel::checkPointPre() 3 2.534e-06 2.534e-06 2.534e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.744e-06 1.744e-06 1.744e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.11-6-g6044e9c2bb9a) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.11-6-g6044e9c2bb9a) initialized Starting run at 08:49:39 UTC on 2022-11-04. Successfully read inputs file ... Castro git describe: 22.11-1-gb27642d86 AMReX git describe: 22.11-6-g6044e9c2b Microphysics git describe: 22.11-4-gcfe7aa24 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.475373156 Restart time = 0.049009276 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.055660366 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050702486 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.060974946 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.077663292 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.080157983 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032559054 seconds Ending run at 08:49:40 UTC on 2022-11-04. Run time = 0.407741787 Run time without initialization = 0.358136178 Average number of zones advanced per microsecond: 3.660 Average number of zones advanced per microsecond per rank: 3.660 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4078 ... 0.4078 ... 0.4078 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1252 0.1252 0.1252 30.71% VisMF::Read() 3 0.04104 0.04104 0.04104 10.06% MLCellLinOp::applyBC() 1946 0.03494 0.03494 0.03494 8.57% VisMF::Write(FabArray) 1 0.03093 0.03093 0.03093 7.59% MLPoisson::Fsmooth() 1440 0.02718 0.02718 0.02718 6.66% StateData::FillBoundary(geom) 160 0.01147 0.01147 0.01147 2.81% MLCGSolver::bicgstab 36 0.01022 0.01022 0.01022 2.51% Castro::computeTemp() 30 0.009634 0.009634 0.009634 2.36% MultiFab::Dot() 484 0.009549 0.009549 0.009549 2.34% Castro::normalize_species() 30 0.008588 0.008588 0.008588 2.11% FabArray::setVal() 537 0.006742 0.006742 0.006742 1.65% FillBoundary_nowait() 1766 0.006304 0.006304 0.006304 1.55% MLCellLinOp::defineAuxData() 6 0.006145 0.006145 0.006145 1.51% MultiFab::LinComb() 690 0.006073 0.006073 0.006073 1.49% FabArray::ParallelCopy_nowait() 380 0.00597 0.00597 0.00597 1.46% StateDataPhysBCFunct::() 20 0.005572 0.005572 0.005572 1.37% MLPoisson::Fapply() 500 0.005091 0.005091 0.005091 1.25% Gravity::fill_multipole_BCs() 6 0.004675 0.004675 0.004675 1.15% Amr::restart() 1 0.003656 0.003656 0.003656 0.90% Castro::do_advance_ctu() 5 0.003478 0.003478 0.003478 0.85% MLMG::addInterpCorrection() 180 0.00332 0.00332 0.00332 0.81% Castro::enforce_min_density() 30 0.003283 0.003283 0.003283 0.81% amrex::average_down 180 0.002935 0.002935 0.002935 0.72% MultiFab::Xpay() 258 0.002875 0.002875 0.002875 0.71% Castro::estTimeStep() 10 0.002572 0.002572 0.002572 0.63% BndryData::define() 6 0.002111 0.002111 0.002111 0.52% Amr::writePlotFile() 1 0.001726 0.001726 0.001726 0.42% Castro::reset_internal_energy(MultiFab) 30 0.001703 0.001703 0.001703 0.42% Castro::construct_new_gravity_source() 5 0.001531 0.001531 0.001531 0.38% Castro::construct_old_gravity_source() 5 0.001328 0.001328 0.001328 0.33% Castro::enforce_speed_limit() 30 0.001305 0.001305 0.001305 0.32% Gravity::get_old_grav_vector() 5 0.00101 0.00101 0.00101 0.25% Castro::reset_internal_energy(Fab) 240 0.000974 0.000974 0.000974 0.24% Gravity::get_new_grav_vector() 5 0.0009649 0.0009649 0.0009649 0.24% MultiFab::Saxpy() 10 0.0009173 0.0009173 0.0009173 0.22% MLMG::ResNormInf() 42 0.000913 0.000913 0.000913 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008972 0.0008972 0.0008972 0.22% Castro::expand_state() 5 0.0008736 0.0008736 0.0008736 0.21% Castro::subcycle_advance_ctu() 5 0.0008263 0.0008263 0.0008263 0.20% MLCellLinOp::setLevelBC() 6 0.0008113 0.0008113 0.0008113 0.20% Gravity::actual_solve_with_mlmg() 6 0.000792 0.000792 0.000792 0.19% MultiFab::Add() 36 0.0007149 0.0007149 0.0007149 0.18% MLMG::prepareForSolve() 6 0.000664 0.000664 0.000664 0.16% FabArray::setDomainBndry() 20 0.0006499 0.0006499 0.0006499 0.16% FabArray::mult() 22 0.0006493 0.0006493 0.0006493 0.16% MLCellLinOp::prepareForSolve() 6 0.0006349 0.0006349 0.0006349 0.16% MultiFab::contains_nan() 10 0.0005904 0.0005904 0.0005904 0.14% MLCellLinOp::compGrad() 6 0.0004908 0.0004908 0.0004908 0.12% MLCellLinOp::smooth() 720 0.0004836 0.0004836 0.0004836 0.12% Amr::InitAmr() 1 0.0004045 0.0004045 0.0004045 0.10% FabArrayBase::CPC::define() 244 0.0003881 0.0003881 0.0003881 0.10% FabArray::FillBoundary() 1766 0.0003866 0.0003866 0.0003866 0.09% FabArrayBase::getCPC() 632 0.0003617 0.0003617 0.0003617 0.09% Castro::finalize_advance() 5 0.000351 0.000351 0.000351 0.09% main() 1 0.0002615 0.0002615 0.0002615 0.06% FabArrayBase::getFB() 1766 0.0002607 0.0002607 0.0002607 0.06% Gravity::update_max_rhs() 6 0.000234 0.000234 0.000234 0.06% Gravity::solve_for_phi() 5 0.0002101 0.0002101 0.0002101 0.05% MLCellLinOp::apply() 500 0.0002011 0.0002011 0.0002011 0.05% Amr::coarseTimeStep() 5 0.0001856 0.0001856 0.0001856 0.05% Castro::construct_new_gravity() 5 0.0001828 0.0001828 0.0001828 0.04% MultiFab::Copy() 6 0.0001772 0.0001772 0.0001772 0.04% CGSolver::sxay() 690 0.0001704 0.0001704 0.0001704 0.04% MLCellLinOp::defineBC() 6 0.0001613 0.0001613 0.0001613 0.04% Castro::advance() 5 0.0001424 0.0001424 0.0001424 0.03% FillPatchIterator::Initialize 20 0.0001349 0.0001349 0.0001349 0.03% MLCGSolver::ParallelAllReduce 659 0.0001336 0.0001336 0.0001336 0.03% MultiFab::max() 6 0.0001332 0.0001332 0.0001332 0.03% FabArray::ParallelCopy() 380 0.0001238 0.0001238 0.0001238 0.03% MLMG::MLRhsNormInf() 6 0.0001175 0.0001175 0.0001175 0.03% Amr::timeStep() 5 9.525e-05 9.525e-05 9.525e-05 0.02% MLCellLinOp::correctionResidual() 216 9.203e-05 9.203e-05 9.203e-05 0.02% AmrLevel::restart() 1 8.99e-05 8.99e-05 8.99e-05 0.02% MLMG::mgVcycle() 36 8.879e-05 8.879e-05 8.879e-05 0.02% StateData::restartDoit() 4 7.7e-05 7.7e-05 7.7e-05 0.02% MLMG:computeResOfCorrection() 180 7.45e-05 7.45e-05 7.45e-05 0.02% MLLinOp::defineGrids() 6 7.103e-05 7.103e-05 7.103e-05 0.02% Castro::construct_new_source() 25 6.648e-05 6.648e-05 6.648e-05 0.02% Castro::post_timestep() 5 6.296e-05 6.296e-05 6.296e-05 0.02% Castro::initialize_do_advance() 5 6.111e-05 6.111e-05 6.111e-05 0.01% FabArrayBase::FB::FB() 26 5.674e-05 5.674e-05 5.674e-05 0.01% MLMG::mgVcycle_down::0 36 5.34e-05 5.34e-05 5.34e-05 0.01% MLMG::mgVcycle_down::1 36 4.783e-05 4.783e-05 4.783e-05 0.01% MLMG::mgVcycle_down::2 36 4.361e-05 4.361e-05 4.361e-05 0.01% Castro::construct_old_gravity() 5 4.245e-05 4.245e-05 4.245e-05 0.01% MLMG::mgVcycle_down::4 36 4.135e-05 4.135e-05 4.135e-05 0.01% MLMG::mgVcycle_down::3 36 4.042e-05 4.042e-05 4.042e-05 0.01% Castro::clean_state() 30 4.028e-05 4.028e-05 4.028e-05 0.01% MLMG::actualBottomSolve() 36 4.028e-05 4.028e-05 4.028e-05 0.01% Castro::create_source_corrector() 5 3.98e-05 3.98e-05 3.98e-05 0.01% Castro::initialize_advance() 5 3.874e-05 3.874e-05 3.874e-05 0.01% MLMG::mgVcycle_up::4 36 3.428e-05 3.428e-05 3.428e-05 0.01% Castro::buildMetrics() 1 3.288e-05 3.288e-05 3.288e-05 0.01% MLMG::solve() 6 3.268e-05 3.268e-05 3.268e-05 0.01% Castro::construct_old_source() 25 3.057e-05 3.057e-05 3.057e-05 0.01% Gravity::actual_multilevel_solve() 1 3.05e-05 3.05e-05 3.05e-05 0.01% MLMG::oneIter() 36 2.976e-05 2.976e-05 2.976e-05 0.01% Castro::post_restart() 1 2.881e-05 2.881e-05 2.881e-05 0.01% Castro::do_old_sources() 5 2.814e-05 2.814e-05 2.814e-05 0.01% MLMG::mgVcycle_up::3 36 2.748e-05 2.748e-05 2.748e-05 0.01% Castro::swap_state_time_levels() 5 2.745e-05 2.745e-05 2.745e-05 0.01% Amr::writeSmallPlotFile() 1 2.737e-05 2.737e-05 2.737e-05 0.01% MLMG::mgVcycle_up::0 36 2.704e-05 2.704e-05 2.704e-05 0.01% MLMG::mgVcycle_up::2 36 2.594e-05 2.594e-05 2.594e-05 0.01% Castro::initMFs() 1 2.523e-05 2.523e-05 2.523e-05 0.01% MLMG::mgVcycle_up::1 36 2.497e-05 2.497e-05 2.497e-05 0.01% MLCellLinOp::solutionResidual() 42 2.3e-05 2.3e-05 2.3e-05 0.01% Castro::computeNewDt() 5 2.238e-05 2.238e-05 2.238e-05 0.01% MLPoisson::define() 6 2.11e-05 2.11e-05 2.11e-05 0.01% MLLinOp::define() 6 1.984e-05 1.984e-05 1.984e-05 0.00% MLMG::computeResidual() 36 1.936e-05 1.936e-05 1.936e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.773e-05 1.773e-05 1.773e-05 0.00% Castro::finalize_do_advance() 5 1.717e-05 1.717e-05 1.717e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.683e-05 1.683e-05 1.683e-05 0.00% MLMG::mgVcycle_bottom 36 1.597e-05 1.597e-05 1.597e-05 0.00% makeSFC 30 1.393e-05 1.393e-05 1.393e-05 0.00% FillPatchSingleLevel 20 1.268e-05 1.268e-05 1.268e-05 0.00% Castro::do_new_sources() 5 9.762e-06 9.762e-06 9.762e-06 0.00% Amr::initSubcycle() 1 8.655e-06 8.655e-06 8.655e-06 0.00% DistributionMapping::Distribute() 31 8.329e-06 8.329e-06 8.329e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.099e-06 7.099e-06 7.099e-06 0.00% Castro::check_for_nan() 10 7.061e-06 7.061e-06 7.061e-06 0.00% Castro::apply_source_to_state() 10 5.657e-06 5.657e-06 5.657e-06 0.00% MLMG::computeMLResidual() 6 5.259e-06 5.259e-06 5.259e-06 0.00% MLPoisson::prepareForSolve() 6 4.597e-06 4.597e-06 4.597e-06 0.00% Gravity::swapTimeLevels() 5 4.177e-06 4.177e-06 4.177e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.323e-06 3.323e-06 3.323e-06 0.00% MLMG::getGradSolution() 6 3.1e-06 3.1e-06 3.1e-06 0.00% Castro::retry_advance_ctu() 5 2.683e-06 2.683e-06 2.683e-06 0.00% Castro::FluxRegCrseInit 5 2.261e-06 2.261e-06 2.261e-06 0.00% MLMG::MLResNormInf() 6 2.2e-06 2.2e-06 2.2e-06 0.00% Gravity::set_mass_offset() 6 2.076e-06 2.076e-06 2.076e-06 0.00% Castro::FluxRegFineAdd() 5 1.324e-06 1.324e-06 1.324e-06 0.00% Amr::init() 1 1.247e-06 1.247e-06 1.247e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.169e-06 1.169e-06 1.169e-06 0.00% AmrLevel::AmrLevel() 1 8.26e-07 8.26e-07 8.26e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4078 0.4078 0.4078 100.00% Amr::coarseTimeStep() 5 0.3253 0.3253 0.3253 79.78% Amr::timeStep() 5 0.3238 0.3238 0.3238 79.40% Castro::advance() 5 0.3186 0.3186 0.3186 78.13% Castro::subcycle_advance_ctu() 5 0.3126 0.3126 0.3126 76.66% Castro::do_advance_ctu() 5 0.3118 0.3118 0.3118 76.45% Castro::construct_new_gravity() 5 0.1439 0.1439 0.1439 35.30% Gravity::solve_phi_with_mlmg() 6 0.1398 0.1398 0.1398 34.28% Gravity::solve_for_phi() 5 0.1362 0.1362 0.1362 33.40% Gravity::actual_solve_with_mlmg() 6 0.135 0.135 0.135 33.11% Castro::construct_ctu_hydro_source() 5 0.1252 0.1252 0.1252 30.71% MLMG::solve() 6 0.1228 0.1228 0.1228 30.11% MLMG::oneIter() 36 0.1156 0.1156 0.1156 28.34% MLMG::mgVcycle() 36 0.1148 0.1148 0.1148 28.16% MLCellLinOp::smooth() 720 0.05878 0.05878 0.05878 14.41% Amr::init() 1 0.04907 0.04907 0.04907 12.03% Amr::restart() 1 0.04907 0.04907 0.04907 12.03% MLCellLinOp::applyBC() 1946 0.04195 0.04195 0.04195 10.29% AmrLevel::restart() 1 0.04126 0.04126 0.04126 10.12% StateData::restartDoit() 4 0.04117 0.04117 0.04117 10.10% VisMF::Read() 3 0.04104 0.04104 0.04104 10.06% MLMG::mgVcycle_bottom 36 0.03505 0.03505 0.03505 8.60% MLMG::actualBottomSolve() 36 0.03504 0.03504 0.03504 8.59% MLCGSolver::bicgstab 36 0.03468 0.03468 0.03468 8.51% Amr::writePlotFile() 1 0.03266 0.03266 0.03266 8.01% VisMF::Write(FabArray) 1 0.03093 0.03093 0.03093 7.59% MLPoisson::Fsmooth() 1440 0.02718 0.02718 0.02718 6.66% Castro::clean_state() 30 0.02553 0.02553 0.02553 6.26% FillPatchIterator::Initialize 20 0.01982 0.01982 0.01982 4.86% FillPatchSingleLevel 20 0.01903 0.01903 0.01903 4.67% StateDataPhysBCFunct::() 20 0.01705 0.01705 0.01705 4.18% MLCellLinOp::apply() 500 0.01585 0.01585 0.01585 3.89% MLMG::mgVcycle_down::0 36 0.01544 0.01544 0.01544 3.79% MLMG::mgVcycle_up::0 36 0.01316 0.01316 0.01316 3.23% Castro::computeTemp() 30 0.01231 0.01231 0.01231 3.02% StateData::FillBoundary(geom) 160 0.01147 0.01147 0.01147 2.81% Castro::initialize_do_advance() 5 0.009968 0.009968 0.009968 2.44% MLPoisson::define() 6 0.009836 0.009836 0.009836 2.41% MultiFab::Dot() 484 0.009549 0.009549 0.009549 2.34% MLCellLinOp::correctionResidual() 216 0.009276 0.009276 0.009276 2.27% Castro::normalize_species() 30 0.008588 0.008588 0.008588 2.11% MLMG:computeResOfCorrection() 180 0.008025 0.008025 0.008025 1.97% MLMG::mgVcycle_down::1 36 0.007663 0.007663 0.007663 1.88% Gravity::get_new_grav_vector() 5 0.007545 0.007545 0.007545 1.85% Castro::construct_old_gravity() 5 0.007538 0.007538 0.007538 1.85% Gravity::get_old_grav_vector() 5 0.007496 0.007496 0.007496 1.84% MLMG::mgVcycle_down::2 36 0.007448 0.007448 0.007448 1.83% MLMG::mgVcycle_down::3 36 0.007069 0.007069 0.007069 1.73% FabArray::FillBoundary() 1766 0.007008 0.007008 0.007008 1.72% MLCellLinOp::defineAuxData() 6 0.006878 0.006878 0.006878 1.69% Castro::do_new_sources() 5 0.006861 0.006861 0.006861 1.68% MLMG::mgVcycle_down::4 36 0.006751 0.006751 0.006751 1.66% FabArray::setVal() 537 0.006742 0.006742 0.006742 1.65% FillBoundary_nowait() 1766 0.006621 0.006621 0.006621 1.62% FabArray::ParallelCopy() 380 0.006464 0.006464 0.006464 1.59% Castro::do_old_sources() 5 0.006452 0.006452 0.006452 1.58% FabArray::ParallelCopy_nowait() 380 0.00634 0.00634 0.00634 1.55% CGSolver::sxay() 690 0.006243 0.006243 0.006243 1.53% MultiFab::LinComb() 690 0.006073 0.006073 0.006073 1.49% MLCGSolver::ParallelAllReduce 659 0.005734 0.005734 0.005734 1.41% MLMG::mgVcycle_up::2 36 0.005695 0.005695 0.005695 1.40% MLMG::mgVcycle_up::1 36 0.005623 0.005623 0.005623 1.38% MLMG::addInterpCorrection() 180 0.005523 0.005523 0.005523 1.35% Castro::initialize_advance() 5 0.005516 0.005516 0.005516 1.35% Castro::expand_state() 5 0.005507 0.005507 0.005507 1.35% MLMG::mgVcycle_up::4 36 0.005429 0.005429 0.005429 1.33% MLMG::mgVcycle_up::3 36 0.005401 0.005401 0.005401 1.32% amrex::average_down 180 0.005221 0.005221 0.005221 1.28% MLPoisson::Fapply() 500 0.005091 0.005091 0.005091 1.25% Castro::post_timestep() 5 0.005088 0.005088 0.005088 1.25% Gravity::fill_multipole_BCs() 6 0.004675 0.004675 0.004675 1.15% Castro::post_restart() 1 0.003967 0.003967 0.003967 0.97% Gravity::multilevel_solve_for_new_phi() 1 0.003844 0.003844 0.003844 0.94% Gravity::actual_multilevel_solve() 1 0.003827 0.003827 0.003827 0.94% Castro::enforce_min_density() 30 0.003283 0.003283 0.003283 0.81% MLCellLinOp::solutionResidual() 42 0.003238 0.003238 0.003238 0.79% MLMG::prepareForSolve() 6 0.002897 0.002897 0.002897 0.71% MultiFab::Xpay() 258 0.002875 0.002875 0.002875 0.71% MLCellLinOp::defineBC() 6 0.002817 0.002817 0.002817 0.69% MLMG::computeResidual() 36 0.002688 0.002688 0.002688 0.66% Castro::reset_internal_energy(MultiFab) 30 0.002677 0.002677 0.002677 0.66% BndryData::define() 6 0.002656 0.002656 0.002656 0.65% Castro::estTimeStep() 10 0.002572 0.002572 0.002572 0.63% Castro::construct_new_source() 25 0.001598 0.001598 0.001598 0.39% Castro::construct_new_gravity_source() 5 0.001531 0.001531 0.001531 0.38% Castro::construct_old_source() 25 0.001359 0.001359 0.001359 0.33% Castro::computeNewDt() 5 0.001357 0.001357 0.001357 0.33% Castro::construct_old_gravity_source() 5 0.001328 0.001328 0.001328 0.33% Castro::enforce_speed_limit() 30 0.001305 0.001305 0.001305 0.32% Castro::reset_internal_energy(Fab) 240 0.000974 0.000974 0.000974 0.24% Castro::apply_source_to_state() 10 0.000923 0.000923 0.000923 0.23% MultiFab::Saxpy() 10 0.0009173 0.0009173 0.0009173 0.22% MLMG::ResNormInf() 42 0.000913 0.000913 0.000913 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008972 0.0008972 0.0008972 0.22% MLCellLinOp::setLevelBC() 6 0.0008113 0.0008113 0.0008113 0.20% MLMG::getGradSolution() 6 0.0007669 0.0007669 0.0007669 0.19% MLCellLinOp::compGrad() 6 0.0007638 0.0007638 0.0007638 0.19% FabArrayBase::getCPC() 632 0.0007498 0.0007498 0.0007498 0.18% MultiFab::Add() 36 0.0007149 0.0007149 0.0007149 0.18% FabArray::setDomainBndry() 20 0.0006499 0.0006499 0.0006499 0.16% FabArray::mult() 22 0.0006493 0.0006493 0.0006493 0.16% MLPoisson::prepareForSolve() 6 0.0006395 0.0006395 0.0006395 0.16% MLCellLinOp::prepareForSolve() 6 0.0006349 0.0006349 0.0006349 0.16% Castro::check_for_nan() 10 0.0005975 0.0005975 0.0005975 0.15% MultiFab::contains_nan() 10 0.0005904 0.0005904 0.0005904 0.14% MLMG::computeMLResidual() 6 0.0005748 0.0005748 0.0005748 0.14% Gravity::update_max_rhs() 6 0.0004456 0.0004456 0.0004456 0.11% Amr::InitAmr() 1 0.0004132 0.0004132 0.0004132 0.10% FabArrayBase::CPC::define() 244 0.0003881 0.0003881 0.0003881 0.10% Castro::finalize_advance() 5 0.0003545 0.0003545 0.0003545 0.09% FabArrayBase::getFB() 1766 0.0003175 0.0003175 0.0003175 0.08% Gravity::swapTimeLevels() 5 0.0002331 0.0002331 0.0002331 0.06% MultiFab::Copy() 6 0.0001772 0.0001772 0.0001772 0.04% Castro::buildMetrics() 1 0.000153 0.000153 0.000153 0.04% MLMG::MLResNormInf() 6 0.0001501 0.0001501 0.0001501 0.04% MultiFab::max() 6 0.0001332 0.0001332 0.0001332 0.03% MLLinOp::define() 6 0.0001201 0.0001201 0.0001201 0.03% MLMG::MLRhsNormInf() 6 0.0001175 0.0001175 0.0001175 0.03% MLLinOp::defineGrids() 6 0.0001002 0.0001002 0.0001002 0.02% FabArrayBase::FB::FB() 26 5.674e-05 5.674e-05 5.674e-05 0.01% Castro::create_source_corrector() 5 3.98e-05 3.98e-05 3.98e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.803e-05 2.803e-05 2.803e-05 0.01% Castro::swap_state_time_levels() 5 2.745e-05 2.745e-05 2.745e-05 0.01% Amr::writeSmallPlotFile() 1 2.737e-05 2.737e-05 2.737e-05 0.01% Castro::initMFs() 1 2.523e-05 2.523e-05 2.523e-05 0.01% makeSFC 30 2.093e-05 2.093e-05 2.093e-05 0.01% Castro::finalize_do_advance() 5 1.717e-05 1.717e-05 1.717e-05 0.00% Amr::initSubcycle() 1 8.655e-06 8.655e-06 8.655e-06 0.00% DistributionMapping::Distribute() 31 8.329e-06 8.329e-06 8.329e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.652e-06 4.652e-06 4.652e-06 0.00% Castro::retry_advance_ctu() 5 2.683e-06 2.683e-06 2.683e-06 0.00% Castro::FluxRegCrseInit 5 2.261e-06 2.261e-06 2.261e-06 0.00% Gravity::set_mass_offset() 6 2.076e-06 2.076e-06 2.076e-06 0.00% Castro::FluxRegFineAdd() 5 1.324e-06 1.324e-06 1.324e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.169e-06 1.169e-06 1.169e-06 0.00% AmrLevel::AmrLevel() 1 8.26e-07 8.26e-07 8.26e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2545 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.11-6-g6044e9c2bb9a) finalized