Initializing CUDA... CUDA initialized with 1 device. AMReX (23.01-19-ga5063a5d705b) initialized Starting run at 10:15:40 UTC on 2023-01-19. Successfully read inputs file ... Castro git describe: 23.01-15-gd760f9747 AMReX git describe: 23.01-19-ga5063a5d7 Microphysics git describe: 23.01-4-gd64aa25b reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.056788042 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.032282278 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.04525524 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.046592339 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.04971542 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.051716662 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.072428961 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.055931835 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.062506909 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047577644 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.047535463 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.049198341 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.051155725 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.056021246 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032280296 seconds Ending run at 10:15:41 UTC on 2023-01-19. Run time = 0.80482171 Run time without initialization = 0.668525613 Average number of zones advanced per microsecond: 3.921 Average number of zones advanced per microsecond per rank: 3.921 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8049 ... 0.8049 ... 0.8049 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2266 0.2266 0.2266 28.16% Castro::construct_ctu_hydro_source() 10 0.1792 0.1792 0.1792 22.27% MLCellLinOp::applyBC() 4433 0.07374 0.07374 0.07374 9.16% FillBoundary_nowait() 4023 0.03228 0.03228 0.03228 4.01% MLPoisson::Fsmooth() 3280 0.03155 0.03155 0.03155 3.92% StateData::FillBoundary(geom) 328 0.02185 0.02185 0.02185 2.71% amrex::Dot() 1114 0.01967 0.01967 0.01967 2.44% Castro::normalize_species() 62 0.01473 0.01473 0.01473 1.83% amrex::Copy() 1029 0.01448 0.01448 0.01448 1.80% FabArray::norminf() 743 0.01389 0.01389 0.01389 1.73% Castro::computeTemp() 63 0.01342 0.01342 0.01342 1.67% StateDataPhysBCFunct::() 41 0.01311 0.01311 0.01311 1.63% FabArray::setVal() 1144 0.01276 0.01276 0.01276 1.59% FabArray::ParallelCopy_nowait() 861 0.01273 0.01273 0.01273 1.58% MLPoisson::Fapply() 1142 0.01013 0.01013 0.01013 1.26% MLCellLinOp::defineAuxData() 11 0.00926 0.00926 0.00926 1.15% FabArray::Saxpy() 813 0.007901 0.007901 0.007901 0.98% FabArray::Xpay() 821 0.007845 0.007845 0.007845 0.97% Castro::enforce_min_density() 62 0.00718 0.00718 0.00718 0.89% MLMG::addInterpCorrection() 410 0.006418 0.006418 0.006418 0.80% Gravity::fill_multipole_BCs() 11 0.00617 0.00617 0.00617 0.77% amrex::average_down 410 0.005636 0.005636 0.005636 0.70% Castro::estTimeStep() 21 0.004966 0.004966 0.004966 0.62% Castro::reset_internal_energy(MultiFab) 63 0.004528 0.004528 0.004528 0.56% FabArray::LinComb() 557 0.004367 0.004367 0.004367 0.54% amrex::Add() 164 0.004297 0.004297 0.004297 0.53% Amr::checkPoint() 3 0.003879 0.003879 0.003879 0.48% BndryData::define() 11 0.003445 0.003445 0.003445 0.43% Castro::construct_new_gravity_source() 10 0.003149 0.003149 0.003149 0.39% Castro::construct_old_gravity_source() 10 0.002609 0.002609 0.002609 0.32% Amr::writePlotFile() 2 0.002298 0.002298 0.002298 0.29% MLCGSolver::bicgstab 82 0.002083 0.002083 0.002083 0.26% Castro::do_advance_ctu() 10 0.002082 0.002082 0.002082 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001613 0.001613 0.001613 0.20% Castro::reset_internal_energy(Fab) 504 0.001428 0.001428 0.001428 0.18% Gravity::actual_solve_with_mlmg() 11 0.001369 0.001369 0.001369 0.17% MLCellLinOp::setLevelBC() 11 0.001342 0.001342 0.001342 0.17% FabArray::mult() 43 0.001319 0.001319 0.001319 0.16% FabArray::setDomainBndry() 41 0.001311 0.001311 0.001311 0.16% Castro::initData() 1 0.001239 0.001239 0.001239 0.15% Castro::enforce_speed_limit() 62 0.001203 0.001203 0.001203 0.15% MultiFab::contains_nan() 20 0.001176 0.001176 0.001176 0.15% MLCellLinOp::prepareForSolve() 11 0.001063 0.001063 0.001063 0.13% MLCellLinOp::smooth() 1640 0.001043 0.001043 0.001043 0.13% MLCellLinOp::compGrad() 11 0.0008964 0.0008964 0.0008964 0.11% FabArray::FillBoundary() 4023 0.0008194 0.0008194 0.0008194 0.10% MLMG::prepareForSolve() 11 0.0008119 0.0008119 0.0008119 0.10% FabArrayBase::getCPC() 1323 0.0007097 0.0007097 0.0007097 0.09% FabArrayBase::CPC::define() 454 0.0006681 0.0006681 0.0006681 0.08% Gravity::get_new_grav_vector() 11 0.0005987 0.0005987 0.0005987 0.07% FabArrayBase::getFB() 4023 0.000576 0.000576 0.000576 0.07% Gravity::get_old_grav_vector() 10 0.0005362 0.0005362 0.0005362 0.07% Amr::InitAmr() 1 0.0004654 0.0004654 0.0004654 0.06% MLCellLinOp::apply() 1142 0.0004406 0.0004406 0.0004406 0.05% MLMG::mgVcycle() 82 0.0004154 0.0004154 0.0004154 0.05% Amr::coarseTimeStep() 10 0.0003313 0.0003313 0.0003313 0.04% main() 1 0.0002881 0.0002881 0.0002881 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002795 0.0002795 0.0002795 0.03% MultiFab::max() 11 0.0002561 0.0002561 0.0002561 0.03% Castro::subcycle_advance_ctu() 10 0.0002398 0.0002398 0.0002398 0.03% FabArray::ParallelCopy() 861 0.0002329 0.0002329 0.0002329 0.03% MLCellLinOp::correctionResidual() 492 0.0002175 0.0002175 0.0002175 0.03% MLCellLinOp::defineBC() 11 0.0001967 0.0001967 0.0001967 0.02% FillPatchIterator::Initialize 41 0.0001948 0.0001948 0.0001948 0.02% Gravity::solve_for_phi() 10 0.000181 0.000181 0.000181 0.02% Amr::timeStep() 10 0.0001581 0.0001581 0.0001581 0.02% MLLinOp::defineGrids() 11 0.0001522 0.0001522 0.0001522 0.02% StateData::checkPoint() 12 0.0001349 0.0001349 0.0001349 0.02% Gravity::update_max_rhs() 11 0.000106 0.000106 0.000106 0.01% MLMG:computeResOfCorrection() 410 0.0001046 0.0001046 0.0001046 0.01% MLMG::mgVcycle_down::0 82 0.0001015 0.0001015 0.0001015 0.01% Castro::finalize_advance() 10 9.064e-05 9.064e-05 9.064e-05 0.01% MLMG::actualBottomSolve() 82 8.601e-05 8.601e-05 8.601e-05 0.01% MLMG::mgVcycle_down::1 82 8.469e-05 8.469e-05 8.469e-05 0.01% Castro::clean_state() 62 8.374e-05 8.374e-05 8.374e-05 0.01% FabArrayBase::FB::FB() 56 8.172e-05 8.172e-05 8.172e-05 0.01% Castro::advance() 10 7.753e-05 7.753e-05 7.753e-05 0.01% Castro::Castro() 1 7.595e-05 7.595e-05 7.595e-05 0.01% MLMG::mgVcycle_down::2 82 7.487e-05 7.487e-05 7.487e-05 0.01% AmrLevel::checkPoint() 3 7.449e-05 7.449e-05 7.449e-05 0.01% Castro::expand_state() 10 7.38e-05 7.38e-05 7.38e-05 0.01% MLMG::mgVcycle_down::3 82 7.134e-05 7.134e-05 7.134e-05 0.01% MLMG::mgVcycle_down::4 82 7.114e-05 7.114e-05 7.114e-05 0.01% MLMG::solve() 11 6.927e-05 6.927e-05 6.927e-05 0.01% Castro::initialize_advance() 10 6.324e-05 6.324e-05 6.324e-05 0.01% MLMG::mgVcycle_up::4 82 5.594e-05 5.594e-05 5.594e-05 0.01% Amr::InitializeInit() 1 5.576e-05 5.576e-05 5.576e-05 0.01% MLMG::oneIter() 82 5.299e-05 5.299e-05 5.299e-05 0.01% MLMG::mgVcycle_up::0 82 5.11e-05 5.11e-05 5.11e-05 0.01% MLMG::mgVcycle_up::1 82 4.864e-05 4.864e-05 4.864e-05 0.01% MLCellLinOp::solutionResidual() 93 4.853e-05 4.853e-05 4.853e-05 0.01% MLMG::mgVcycle_up::3 82 4.782e-05 4.782e-05 4.782e-05 0.01% MLMG::mgVcycle_up::2 82 4.72e-05 4.72e-05 4.72e-05 0.01% Castro::initialize_do_advance() 10 4.27e-05 4.27e-05 4.27e-05 0.01% Castro::construct_new_source() 50 4.051e-05 4.051e-05 4.051e-05 0.01% Castro::finalize_do_advance() 10 3.428e-05 3.428e-05 3.428e-05 0.00% Castro::enforce_consistent_e() 1 3.38e-05 3.38e-05 3.38e-05 0.00% Castro::swap_state_time_levels() 10 3.296e-05 3.296e-05 3.296e-05 0.00% MLMG::mgVcycle_bottom 82 3.186e-05 3.186e-05 3.186e-05 0.00% MLMG::ResNormInf() 93 3.17e-05 3.17e-05 3.17e-05 0.00% FillPatchSingleLevel 41 2.899e-05 2.899e-05 2.899e-05 0.00% MLMG::computeResidual() 82 2.891e-05 2.891e-05 2.891e-05 0.00% StateData::define() 4 2.682e-05 2.682e-05 2.682e-05 0.00% makeSFC 55 2.556e-05 2.556e-05 2.556e-05 0.00% Castro::construct_new_gravity() 10 2.515e-05 2.515e-05 2.515e-05 0.00% Amr::writeSmallPlotFile() 1 2.402e-05 2.402e-05 2.402e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.231e-05 2.231e-05 2.231e-05 0.00% MLPoisson::define() 11 2.182e-05 2.182e-05 2.182e-05 0.00% Amr::FinalizeInit() 1 2.083e-05 2.083e-05 2.083e-05 0.00% Castro::initMFs() 1 1.907e-05 1.907e-05 1.907e-05 0.00% Amr::defBaseLevel() 1 1.864e-05 1.864e-05 1.864e-05 0.00% Castro::construct_old_source() 50 1.852e-05 1.852e-05 1.852e-05 0.00% Castro::do_new_sources() 10 1.675e-05 1.675e-05 1.675e-05 0.00% Castro::buildMetrics() 1 1.589e-05 1.589e-05 1.589e-05 0.00% Castro::do_old_sources() 10 1.552e-05 1.552e-05 1.552e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.496e-05 1.496e-05 1.496e-05 0.00% DistributionMapping::Distribute() 56 1.479e-05 1.479e-05 1.479e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.365e-05 1.365e-05 1.365e-05 0.00% Castro::check_for_nan() 20 1.261e-05 1.261e-05 1.261e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.234e-05 1.234e-05 1.234e-05 0.00% Castro::apply_source_to_state() 20 1.097e-05 1.097e-05 1.097e-05 0.00% Castro::post_timestep() 10 9.879e-06 9.879e-06 9.879e-06 0.00% Castro::construct_old_gravity() 10 9.72e-06 9.72e-06 9.72e-06 0.00% Castro::post_init() 1 9.584e-06 9.584e-06 9.584e-06 0.00% MLLinOp::define() 11 9.492e-06 9.492e-06 9.492e-06 0.00% Gravity::swapTimeLevels() 10 9.131e-06 9.131e-06 9.131e-06 0.00% Amr::initSubcycle() 1 8.565e-06 8.565e-06 8.565e-06 0.00% MLMG::computeMLResidual() 11 7.774e-06 7.774e-06 7.774e-06 0.00% MLPoisson::prepareForSolve() 11 7.603e-06 7.603e-06 7.603e-06 0.00% Gravity::actual_multilevel_solve() 1 6.699e-06 6.699e-06 6.699e-06 0.00% Castro::computeNewDt() 9 5.949e-06 5.949e-06 5.949e-06 0.00% MLMG::getGradSolution() 11 5.7e-06 5.7e-06 5.7e-06 0.00% AmrLevel::checkPointPost() 3 5.036e-06 5.036e-06 5.036e-06 0.00% Castro::retry_advance_ctu() 10 3.982e-06 3.982e-06 3.982e-06 0.00% Gravity::set_mass_offset() 11 3.827e-06 3.827e-06 3.827e-06 0.00% MLMG::MLResNormInf() 11 3.817e-06 3.817e-06 3.817e-06 0.00% MLMG::MLRhsNormInf() 11 3.706e-06 3.706e-06 3.706e-06 0.00% Castro::create_source_corrector() 10 3.344e-06 3.344e-06 3.344e-06 0.00% Castro::FluxRegCrseInit 10 3.068e-06 3.068e-06 3.068e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.635e-06 2.635e-06 2.635e-06 0.00% Amr::init() 1 2.545e-06 2.545e-06 2.545e-06 0.00% Castro::computeInitialDt() 2 2.544e-06 2.544e-06 2.544e-06 0.00% Castro::FluxRegFineAdd() 10 2.321e-06 2.321e-06 2.321e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.951e-06 1.951e-06 1.951e-06 0.00% AmrLevel::checkPointPre() 3 1.883e-06 1.883e-06 1.883e-06 0.00% Amr::initialInit() 1 1.294e-06 1.294e-06 1.294e-06 0.00% Castro::post_regrid() 1 9.46e-07 9.46e-07 9.46e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8048 0.8048 0.8048 100.00% Amr::coarseTimeStep() 10 0.636 0.636 0.636 79.02% Amr::timeStep() 10 0.5209 0.5209 0.5209 64.72% Castro::advance() 10 0.5135 0.5135 0.5135 63.80% Castro::subcycle_advance_ctu() 10 0.5021 0.5021 0.5021 62.39% Castro::do_advance_ctu() 10 0.5019 0.5019 0.5019 62.36% Gravity::solve_phi_with_mlmg() 11 0.2748 0.2748 0.2748 34.15% Gravity::actual_solve_with_mlmg() 11 0.2682 0.2682 0.2682 33.32% Castro::construct_new_gravity() 10 0.2507 0.2507 0.2507 31.15% MLMG::solve() 11 0.2488 0.2488 0.2488 30.91% Gravity::solve_for_phi() 10 0.2361 0.2361 0.2361 29.33% MLMG::oneIter() 82 0.2349 0.2349 0.2349 29.19% MLMG::mgVcycle() 82 0.2313 0.2313 0.2313 28.74% VisMF::Write(FabArray) 11 0.2266 0.2266 0.2266 28.16% Castro::construct_ctu_hydro_source() 10 0.1792 0.1792 0.1792 22.27% Amr::checkPoint() 3 0.1689 0.1689 0.1689 20.98% AmrLevel::checkPoint() 3 0.165 0.165 0.165 20.50% StateData::checkPoint() 12 0.1649 0.1649 0.1649 20.49% Amr::init() 1 0.1357 0.1357 0.1357 16.86% MLCellLinOp::smooth() 1640 0.1149 0.1149 0.1149 14.27% MLCellLinOp::applyBC() 4433 0.1075 0.1075 0.1075 13.36% MLMG::mgVcycle_bottom 82 0.07081 0.07081 0.07081 8.80% MLMG::actualBottomSolve() 82 0.07077 0.07077 0.07077 8.79% MLCGSolver::bicgstab 82 0.07009 0.07009 0.07009 8.71% Amr::writePlotFile() 2 0.06469 0.06469 0.06469 8.04% Amr::initialInit() 1 0.04649 0.04649 0.04649 5.78% Amr::FinalizeInit() 1 0.04229 0.04229 0.04229 5.25% Castro::clean_state() 62 0.04176 0.04176 0.04176 5.19% Castro::post_init() 1 0.04097 0.04097 0.04097 5.09% FillPatchIterator::Initialize 41 0.04043 0.04043 0.04043 5.02% Gravity::multilevel_solve_for_new_phi() 1 0.03923 0.03923 0.03923 4.87% Gravity::actual_multilevel_solve() 1 0.03922 0.03922 0.03922 4.87% FillPatchSingleLevel 41 0.03893 0.03893 0.03893 4.84% MLCellLinOp::apply() 1142 0.03534 0.03534 0.03534 4.39% StateDataPhysBCFunct::() 41 0.03496 0.03496 0.03496 4.34% MLMG::mgVcycle_down::0 82 0.03389 0.03389 0.03389 4.21% FabArray::FillBoundary() 4023 0.03376 0.03376 0.03376 4.19% FillBoundary_nowait() 4023 0.03294 0.03294 0.03294 4.09% MLPoisson::Fsmooth() 3280 0.03155 0.03155 0.03155 3.92% MLMG::mgVcycle_up::0 82 0.02573 0.02573 0.02573 3.20% StateData::FillBoundary(geom) 328 0.02185 0.02185 0.02185 2.71% MLCellLinOp::correctionResidual() 492 0.02176 0.02176 0.02176 2.70% amrex::Dot() 1114 0.01967 0.01967 0.01967 2.44% Castro::computeTemp() 63 0.01937 0.01937 0.01937 2.41% Castro::initialize_do_advance() 10 0.01929 0.01929 0.01929 2.40% MLMG:computeResOfCorrection() 410 0.01922 0.01922 0.01922 2.39% Gravity::get_new_grav_vector() 11 0.01602 0.01602 0.01602 1.99% MLPoisson::define() 11 0.01535 0.01535 0.01535 1.91% MLMG::mgVcycle_down::1 82 0.0152 0.0152 0.0152 1.89% Castro::normalize_species() 62 0.01473 0.01473 0.01473 1.83% amrex::Copy() 1029 0.01448 0.01448 0.01448 1.80% Castro::construct_old_gravity() 10 0.01417 0.01417 0.01417 1.76% Gravity::get_old_grav_vector() 10 0.01417 0.01417 0.01417 1.76% MLMG::mgVcycle_down::2 82 0.01416 0.01416 0.01416 1.76% FabArray::norminf() 743 0.01389 0.01389 0.01389 1.73% MLMG::mgVcycle_down::3 82 0.0138 0.0138 0.0138 1.72% FabArray::ParallelCopy() 861 0.01374 0.01374 0.01374 1.71% MLMG::mgVcycle_down::4 82 0.01364 0.01364 0.01364 1.70% FabArray::ParallelCopy_nowait() 861 0.01351 0.01351 0.01351 1.68% FabArray::setVal() 1144 0.01276 0.01276 0.01276 1.59% MLCGSolver::ParallelAllReduce 1514 0.01178 0.01178 0.01178 1.46% MLMG::addInterpCorrection() 410 0.01135 0.01135 0.01135 1.41% Castro::expand_state() 10 0.01123 0.01123 0.01123 1.40% MLMG::mgVcycle_up::4 82 0.01114 0.01114 0.01114 1.38% MLMG::mgVcycle_up::1 82 0.0111 0.0111 0.0111 1.38% MLMG::mgVcycle_up::2 82 0.01081 0.01081 0.01081 1.34% Castro::initialize_advance() 10 0.01069 0.01069 0.01069 1.33% Castro::do_old_sources() 10 0.01063 0.01063 0.01063 1.32% MLMG::mgVcycle_up::3 82 0.0106 0.0106 0.0106 1.32% MLCellLinOp::defineAuxData() 11 0.01055 0.01055 0.01055 1.31% amrex::average_down 410 0.0105 0.0105 0.0105 1.30% MLPoisson::Fapply() 1142 0.01013 0.01013 0.01013 1.26% Castro::do_new_sources() 10 0.01009 0.01009 0.01009 1.25% FabArray::Saxpy() 813 0.007901 0.007901 0.007901 0.98% FabArray::Xpay() 821 0.007845 0.007845 0.007845 0.97% Castro::post_timestep() 10 0.007225 0.007225 0.007225 0.90% Castro::enforce_min_density() 62 0.00718 0.00718 0.00718 0.89% MLCellLinOp::solutionResidual() 93 0.007013 0.007013 0.007013 0.87% Gravity::fill_multipole_BCs() 11 0.006426 0.006426 0.006426 0.80% MLMG::computeResidual() 82 0.006053 0.006053 0.006053 0.75% Castro::reset_internal_energy(MultiFab) 63 0.005956 0.005956 0.005956 0.74% Castro::estTimeStep() 21 0.004966 0.004966 0.004966 0.62% MLCellLinOp::defineBC() 11 0.004562 0.004562 0.004562 0.57% MLMG::prepareForSolve() 11 0.00444 0.00444 0.00444 0.55% FabArray::LinComb() 557 0.004367 0.004367 0.004367 0.54% BndryData::define() 11 0.004365 0.004365 0.004365 0.54% amrex::Add() 164 0.004297 0.004297 0.004297 0.53% Amr::InitializeInit() 1 0.004207 0.004207 0.004207 0.52% Amr::defBaseLevel() 1 0.004152 0.004152 0.004152 0.52% Castro::initData() 1 0.003681 0.003681 0.003681 0.46% Castro::construct_new_source() 50 0.003189 0.003189 0.003189 0.40% Castro::construct_new_gravity_source() 10 0.003149 0.003149 0.003149 0.39% Castro::construct_old_source() 50 0.002628 0.002628 0.002628 0.33% Castro::construct_old_gravity_source() 10 0.002609 0.002609 0.002609 0.32% MLMG::ResNormInf() 93 0.002091 0.002091 0.002091 0.26% Castro::computeNewDt() 9 0.002079 0.002079 0.002079 0.26% Castro::apply_source_to_state() 20 0.001812 0.001812 0.001812 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001613 0.001613 0.001613 0.20% Castro::reset_internal_energy(Fab) 504 0.001428 0.001428 0.001428 0.18% FabArrayBase::getCPC() 1323 0.001378 0.001378 0.001378 0.17% MLMG::getGradSolution() 11 0.001361 0.001361 0.001361 0.17% MLCellLinOp::compGrad() 11 0.001355 0.001355 0.001355 0.17% MLCellLinOp::setLevelBC() 11 0.001342 0.001342 0.001342 0.17% FabArray::mult() 43 0.001319 0.001319 0.001319 0.16% FabArray::setDomainBndry() 41 0.001311 0.001311 0.001311 0.16% Castro::enforce_speed_limit() 62 0.001203 0.001203 0.001203 0.15% Castro::check_for_nan() 20 0.001188 0.001188 0.001188 0.15% MultiFab::contains_nan() 20 0.001176 0.001176 0.001176 0.15% Castro::post_regrid() 1 0.001138 0.001138 0.001138 0.14% MLPoisson::prepareForSolve() 11 0.001071 0.001071 0.001071 0.13% MLCellLinOp::prepareForSolve() 11 0.001063 0.001063 0.001063 0.13% MLMG::computeMLResidual() 11 0.0009962 0.0009962 0.0009962 0.12% Castro::computeInitialDt() 2 0.000843 0.000843 0.000843 0.10% Gravity::update_max_rhs() 11 0.0007992 0.0007992 0.0007992 0.10% FabArrayBase::CPC::define() 454 0.0006681 0.0006681 0.0006681 0.08% FabArrayBase::getFB() 4023 0.0006577 0.0006577 0.0006577 0.08% Castro::finalize_advance() 10 0.0006071 0.0006071 0.0006071 0.08% Amr::InitAmr() 1 0.000474 0.000474 0.000474 0.06% Gravity::swapTimeLevels() 10 0.0004372 0.0004372 0.0004372 0.05% Castro::Castro() 1 0.000406 0.000406 0.000406 0.05% MLMG::MLResNormInf() 11 0.0002796 0.0002796 0.0002796 0.03% MultiFab::max() 11 0.0002561 0.0002561 0.0002561 0.03% MLMG::MLRhsNormInf() 11 0.0002157 0.0002157 0.0002157 0.03% MLLinOp::define() 11 0.0002154 0.0002154 0.0002154 0.03% MLLinOp::defineGrids() 11 0.0002059 0.0002059 0.0002059 0.03% Castro::buildMetrics() 1 0.0001516 0.0001516 0.0001516 0.02% FabArrayBase::FB::FB() 56 8.172e-05 8.172e-05 8.172e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.169e-05 5.169e-05 5.169e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.179e-05 4.179e-05 4.179e-05 0.01% makeSFC 55 3.935e-05 3.935e-05 3.935e-05 0.00% Castro::finalize_do_advance() 10 3.428e-05 3.428e-05 3.428e-05 0.00% Castro::enforce_consistent_e() 1 3.38e-05 3.38e-05 3.38e-05 0.00% Castro::swap_state_time_levels() 10 3.296e-05 3.296e-05 3.296e-05 0.00% StateData::define() 4 2.682e-05 2.682e-05 2.682e-05 0.00% Amr::writeSmallPlotFile() 1 2.402e-05 2.402e-05 2.402e-05 0.00% Castro::initMFs() 1 1.907e-05 1.907e-05 1.907e-05 0.00% DistributionMapping::Distribute() 56 1.479e-05 1.479e-05 1.479e-05 0.00% Amr::initSubcycle() 1 8.565e-06 8.565e-06 8.565e-06 0.00% AmrLevel::checkPointPost() 3 5.036e-06 5.036e-06 5.036e-06 0.00% Castro::retry_advance_ctu() 10 3.982e-06 3.982e-06 3.982e-06 0.00% Gravity::set_mass_offset() 11 3.827e-06 3.827e-06 3.827e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.636e-06 3.636e-06 3.636e-06 0.00% Castro::create_source_corrector() 10 3.344e-06 3.344e-06 3.344e-06 0.00% Castro::FluxRegCrseInit 10 3.068e-06 3.068e-06 3.068e-06 0.00% Castro::FluxRegFineAdd() 10 2.321e-06 2.321e-06 2.321e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.951e-06 1.951e-06 1.951e-06 0.00% AmrLevel::checkPointPre() 3 1.883e-06 1.883e-06 1.883e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-19-ga5063a5d705b) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.01-19-ga5063a5d705b) initialized Starting run at 10:15:42 UTC on 2023-01-19. Successfully read inputs file ... Castro git describe: 23.01-15-gd760f9747 AMReX git describe: 23.01-19-ga5063a5d7 Microphysics git describe: 23.01-4-gd64aa25b reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.457941934 Restart time = 0.046626801 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.046495316 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.045413519 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.048789505 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.048854548 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.051819278 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032240488 seconds Ending run at 10:15:42 UTC on 2023-01-19. Run time = 0.321236855 Run time without initialization = 0.273988532 Average number of zones advanced per microsecond: 4.784 Average number of zones advanced per microsecond per rank: 4.784 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3213 ... 0.3213 ... 0.3213 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0726 0.0726 0.0726 22.61% VisMF::Read() 3 0.04057 0.04057 0.04057 12.63% MLCellLinOp::applyBC() 1946 0.03138 0.03138 0.03138 9.77% VisMF::Write(FabArray) 1 0.03077 0.03077 0.03077 9.58% MLPoisson::Fsmooth() 1440 0.01344 0.01344 0.01344 4.18% FillBoundary_nowait() 1766 0.01271 0.01271 0.01271 3.96% StateData::FillBoundary(geom) 160 0.01094 0.01094 0.01094 3.41% amrex::Dot() 484 0.008274 0.008274 0.008274 2.58% Castro::normalize_species() 30 0.007084 0.007084 0.007084 2.21% amrex::Copy() 463 0.006769 0.006769 0.006769 2.11% FabArray::setVal() 537 0.006011 0.006011 0.006011 1.87% FabArray::norminf() 326 0.005912 0.005912 0.005912 1.84% StateDataPhysBCFunct::() 20 0.005711 0.005711 0.005711 1.78% FabArray::ParallelCopy_nowait() 380 0.005698 0.005698 0.005698 1.77% Castro::computeTemp() 30 0.005207 0.005207 0.005207 1.62% MLCellLinOp::defineAuxData() 6 0.004938 0.004938 0.004938 1.54% MLPoisson::Fapply() 500 0.004323 0.004323 0.004323 1.35% FabArray::Saxpy() 355 0.003489 0.003489 0.003489 1.09% FabArray::Xpay() 361 0.003378 0.003378 0.003378 1.05% Gravity::fill_multipole_BCs() 6 0.002994 0.002994 0.002994 0.93% MLMG::addInterpCorrection() 180 0.002737 0.002737 0.002737 0.85% Castro::enforce_min_density() 30 0.002733 0.002733 0.002733 0.85% Amr::restart() 1 0.002456 0.002456 0.002456 0.76% amrex::average_down 180 0.002429 0.002429 0.002429 0.76% Castro::estTimeStep() 10 0.002394 0.002394 0.002394 0.75% Castro::reset_internal_energy(MultiFab) 30 0.001941 0.001941 0.001941 0.60% BndryData::define() 6 0.001872 0.001872 0.001872 0.58% FabArray::LinComb() 242 0.001825 0.001825 0.001825 0.57% amrex::Add() 72 0.001806 0.001806 0.001806 0.56% Castro::construct_new_gravity_source() 5 0.001604 0.001604 0.001604 0.50% Amr::writePlotFile() 1 0.001307 0.001307 0.001307 0.41% Castro::construct_old_gravity_source() 5 0.001158 0.001158 0.001158 0.36% Castro::do_advance_ctu() 5 0.0009845 0.0009845 0.0009845 0.31% MLCGSolver::bicgstab 36 0.000939 0.000939 0.000939 0.29% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008408 0.0008408 0.0008408 0.26% Gravity::actual_solve_with_mlmg() 6 0.0007266 0.0007266 0.0007266 0.23% MLCellLinOp::setLevelBC() 6 0.0007128 0.0007128 0.0007128 0.22% Castro::reset_internal_energy(Fab) 240 0.0006604 0.0006604 0.0006604 0.21% FabArray::mult() 22 0.000657 0.000657 0.000657 0.20% FabArray::setDomainBndry() 20 0.0006293 0.0006293 0.0006293 0.20% MultiFab::contains_nan() 10 0.0005828 0.0005828 0.0005828 0.18% MLCellLinOp::prepareForSolve() 6 0.0005736 0.0005736 0.0005736 0.18% Castro::enforce_speed_limit() 30 0.0005609 0.0005609 0.0005609 0.17% MLCellLinOp::smooth() 720 0.0004932 0.0004932 0.0004932 0.15% MLCellLinOp::compGrad() 6 0.0004763 0.0004763 0.0004763 0.15% MLMG::prepareForSolve() 6 0.0004417 0.0004417 0.0004417 0.14% Amr::InitAmr() 1 0.0004303 0.0004303 0.0004303 0.13% FabArrayBase::CPC::define() 244 0.0004063 0.0004063 0.0004063 0.13% FabArray::FillBoundary() 1766 0.0003649 0.0003649 0.0003649 0.11% FabArrayBase::getCPC() 632 0.0003448 0.0003448 0.0003448 0.11% main() 1 0.000289 0.000289 0.000289 0.09% Gravity::get_old_grav_vector() 5 0.000288 0.000288 0.000288 0.09% Gravity::get_new_grav_vector() 5 0.0002654 0.0002654 0.0002654 0.08% FabArrayBase::getFB() 1766 0.0002581 0.0002581 0.0002581 0.08% MLCellLinOp::apply() 500 0.0002036 0.0002036 0.0002036 0.06% MLMG::mgVcycle() 36 0.0001774 0.0001774 0.0001774 0.06% Amr::coarseTimeStep() 5 0.0001533 0.0001533 0.0001533 0.05% MultiFab::max() 6 0.0001336 0.0001336 0.0001336 0.04% MLCGSolver::ParallelAllReduce 659 0.000127 0.000127 0.000127 0.04% Castro::subcycle_advance_ctu() 5 0.0001243 0.0001243 0.0001243 0.04% FabArray::ParallelCopy() 380 0.0001073 0.0001073 0.0001073 0.03% MLCellLinOp::defineBC() 6 0.0001047 0.0001047 0.0001047 0.03% FillPatchIterator::Initialize 20 9.788e-05 9.788e-05 9.788e-05 0.03% MLCellLinOp::correctionResidual() 216 9.476e-05 9.476e-05 9.476e-05 0.03% Gravity::solve_for_phi() 5 8.931e-05 8.931e-05 8.931e-05 0.03% MLLinOp::defineGrids() 6 8.346e-05 8.346e-05 8.346e-05 0.03% Amr::timeStep() 5 7.811e-05 7.811e-05 7.811e-05 0.02% AmrLevel::restart() 1 6.886e-05 6.886e-05 6.886e-05 0.02% StateData::restartDoit() 4 5.849e-05 5.849e-05 5.849e-05 0.02% FabArrayBase::FB::FB() 26 5.848e-05 5.848e-05 5.848e-05 0.02% Gravity::update_max_rhs() 6 5.344e-05 5.344e-05 5.344e-05 0.02% Castro::expand_state() 5 4.598e-05 4.598e-05 4.598e-05 0.01% MLMG:computeResOfCorrection() 180 4.569e-05 4.569e-05 4.569e-05 0.01% Castro::clean_state() 30 4.359e-05 4.359e-05 4.359e-05 0.01% Castro::finalize_advance() 5 4.271e-05 4.271e-05 4.271e-05 0.01% MLMG::mgVcycle_down::0 36 3.873e-05 3.873e-05 3.873e-05 0.01% MLMG::actualBottomSolve() 36 3.844e-05 3.844e-05 3.844e-05 0.01% MLMG::mgVcycle_down::1 36 3.594e-05 3.594e-05 3.594e-05 0.01% MLMG::solve() 6 3.208e-05 3.208e-05 3.208e-05 0.01% MLMG::mgVcycle_down::2 36 3.173e-05 3.173e-05 3.173e-05 0.01% MLMG::mgVcycle_up::4 36 3.124e-05 3.124e-05 3.124e-05 0.01% MLMG::mgVcycle_down::4 36 3.089e-05 3.089e-05 3.089e-05 0.01% MLMG::mgVcycle_down::3 36 2.997e-05 2.997e-05 2.997e-05 0.01% Castro::initialize_advance() 5 2.879e-05 2.879e-05 2.879e-05 0.01% MLMG::oneIter() 36 2.688e-05 2.688e-05 2.688e-05 0.01% Amr::writeSmallPlotFile() 1 2.572e-05 2.572e-05 2.572e-05 0.01% Castro::buildMetrics() 1 2.356e-05 2.356e-05 2.356e-05 0.01% Castro::initialize_do_advance() 5 2.256e-05 2.256e-05 2.256e-05 0.01% MLMG::mgVcycle_up::0 36 2.232e-05 2.232e-05 2.232e-05 0.01% MLCellLinOp::solutionResidual() 42 2.209e-05 2.209e-05 2.209e-05 0.01% Castro::post_restart() 1 2.176e-05 2.176e-05 2.176e-05 0.01% MLMG::mgVcycle_up::3 36 2.145e-05 2.145e-05 2.145e-05 0.01% MLMG::mgVcycle_up::2 36 2.108e-05 2.108e-05 2.108e-05 0.01% MLMG::mgVcycle_up::1 36 2.037e-05 2.037e-05 2.037e-05 0.01% Castro::swap_state_time_levels() 5 2.022e-05 2.022e-05 2.022e-05 0.01% Castro::initMFs() 1 1.923e-05 1.923e-05 1.923e-05 0.01% Castro::finalize_do_advance() 5 1.883e-05 1.883e-05 1.883e-05 0.01% MLMG::ResNormInf() 42 1.63e-05 1.63e-05 1.63e-05 0.01% MLMG::mgVcycle_bottom 36 1.485e-05 1.485e-05 1.485e-05 0.00% MLPoisson::define() 6 1.457e-05 1.457e-05 1.457e-05 0.00% FillPatchSingleLevel 20 1.383e-05 1.383e-05 1.383e-05 0.00% Castro::advance() 5 1.377e-05 1.377e-05 1.377e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.327e-05 1.327e-05 1.327e-05 0.00% makeSFC 30 1.298e-05 1.298e-05 1.298e-05 0.00% MLMG::computeResidual() 36 1.285e-05 1.285e-05 1.285e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.234e-05 1.234e-05 1.234e-05 0.00% Castro::construct_new_gravity() 5 1.16e-05 1.16e-05 1.16e-05 0.00% Castro::construct_new_source() 25 1.038e-05 1.038e-05 1.038e-05 0.00% Castro::construct_old_source() 25 1.028e-05 1.028e-05 1.028e-05 0.00% DistributionMapping::Distribute() 31 8.628e-06 8.628e-06 8.628e-06 0.00% Castro::do_new_sources() 5 8.272e-06 8.272e-06 8.272e-06 0.00% Amr::initSubcycle() 1 7.893e-06 7.893e-06 7.893e-06 0.00% Castro::do_old_sources() 5 7.732e-06 7.732e-06 7.732e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.907e-06 6.907e-06 6.907e-06 0.00% Gravity::actual_multilevel_solve() 1 6.158e-06 6.158e-06 6.158e-06 0.00% Castro::check_for_nan() 10 5.795e-06 5.795e-06 5.795e-06 0.00% Castro::apply_source_to_state() 10 5.533e-06 5.533e-06 5.533e-06 0.00% Castro::post_timestep() 5 5.311e-06 5.311e-06 5.311e-06 0.00% Castro::construct_old_gravity() 5 5.065e-06 5.065e-06 5.065e-06 0.00% MLLinOp::define() 6 4.867e-06 4.867e-06 4.867e-06 0.00% MLPoisson::prepareForSolve() 6 4.232e-06 4.232e-06 4.232e-06 0.00% Gravity::swapTimeLevels() 5 4.148e-06 4.148e-06 4.148e-06 0.00% Castro::computeNewDt() 5 3.463e-06 3.463e-06 3.463e-06 0.00% MLMG::computeMLResidual() 6 3.318e-06 3.318e-06 3.318e-06 0.00% MLMG::getGradSolution() 6 2.977e-06 2.977e-06 2.977e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.861e-06 2.861e-06 2.861e-06 0.00% MLMG::MLRhsNormInf() 6 2.145e-06 2.145e-06 2.145e-06 0.00% MLMG::MLResNormInf() 6 2.087e-06 2.087e-06 2.087e-06 0.00% Castro::retry_advance_ctu() 5 2.015e-06 2.015e-06 2.015e-06 0.00% Gravity::set_mass_offset() 6 1.83e-06 1.83e-06 1.83e-06 0.00% Castro::create_source_corrector() 5 1.744e-06 1.744e-06 1.744e-06 0.00% Castro::FluxRegFineAdd() 5 1.321e-06 1.321e-06 1.321e-06 0.00% Castro::FluxRegCrseInit 5 1.216e-06 1.216e-06 1.216e-06 0.00% Amr::init() 1 1.184e-06 1.184e-06 1.184e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.019e-06 1.019e-06 1.019e-06 0.00% AmrLevel::AmrLevel() 1 6.04e-07 6.04e-07 6.04e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3213 0.3213 0.3213 99.99% Amr::coarseTimeStep() 5 0.2415 0.2415 0.2415 75.17% Amr::timeStep() 5 0.2399 0.2399 0.2399 74.68% Castro::advance() 5 0.2372 0.2372 0.2372 73.82% Castro::subcycle_advance_ctu() 5 0.2304 0.2304 0.2304 71.71% Castro::do_advance_ctu() 5 0.2303 0.2303 0.2303 71.68% Castro::construct_new_gravity() 5 0.1238 0.1238 0.1238 38.54% Gravity::solve_phi_with_mlmg() 6 0.1192 0.1192 0.1192 37.09% Gravity::solve_for_phi() 5 0.1162 0.1162 0.1162 36.18% Gravity::actual_solve_with_mlmg() 6 0.1159 0.1159 0.1159 36.08% MLMG::solve() 6 0.1055 0.1055 0.1055 32.83% MLMG::oneIter() 36 0.09888 0.09888 0.09888 30.78% MLMG::mgVcycle() 36 0.09736 0.09736 0.09736 30.30% Castro::construct_ctu_hydro_source() 5 0.07263 0.07263 0.07263 22.61% MLCellLinOp::smooth() 720 0.04799 0.04799 0.04799 14.94% Amr::init() 1 0.04667 0.04667 0.04667 14.53% Amr::restart() 1 0.04667 0.04667 0.04667 14.53% MLCellLinOp::applyBC() 1946 0.04477 0.04477 0.04477 13.94% AmrLevel::restart() 1 0.04075 0.04075 0.04075 12.68% StateData::restartDoit() 4 0.04068 0.04068 0.04068 12.66% VisMF::Read() 3 0.04057 0.04057 0.04057 12.63% Amr::writePlotFile() 1 0.03232 0.03232 0.03232 10.06% VisMF::Write(FabArray) 1 0.03077 0.03077 0.03077 9.58% MLMG::mgVcycle_bottom 36 0.02999 0.02999 0.02999 9.34% MLMG::actualBottomSolve() 36 0.02998 0.02998 0.02998 9.33% MLCGSolver::bicgstab 36 0.02969 0.02969 0.02969 9.24% FillPatchIterator::Initialize 20 0.01935 0.01935 0.01935 6.02% FillPatchSingleLevel 20 0.01862 0.01862 0.01862 5.80% Castro::clean_state() 30 0.01823 0.01823 0.01823 5.67% StateDataPhysBCFunct::() 20 0.01665 0.01665 0.01665 5.18% MLCellLinOp::apply() 500 0.01498 0.01498 0.01498 4.66% MLMG::mgVcycle_down::0 36 0.0139 0.0139 0.0139 4.33% MLPoisson::Fsmooth() 1440 0.01344 0.01344 0.01344 4.18% FabArray::FillBoundary() 1766 0.01339 0.01339 0.01339 4.17% FillBoundary_nowait() 1766 0.01303 0.01303 0.01303 4.05% StateData::FillBoundary(geom) 160 0.01094 0.01094 0.01094 3.41% MLMG::mgVcycle_up::0 36 0.01052 0.01052 0.01052 3.27% MLCellLinOp::correctionResidual() 216 0.009131 0.009131 0.009131 2.84% Castro::initialize_do_advance() 5 0.008891 0.008891 0.008891 2.77% amrex::Dot() 484 0.008274 0.008274 0.008274 2.58% MLPoisson::define() 6 0.00827 0.00827 0.00827 2.57% MLMG:computeResOfCorrection() 180 0.008031 0.008031 0.008031 2.50% Castro::computeTemp() 30 0.007809 0.007809 0.007809 2.43% Gravity::get_new_grav_vector() 5 0.007454 0.007454 0.007454 2.32% Castro::normalize_species() 30 0.007084 0.007084 0.007084 2.21% Castro::construct_old_gravity() 5 0.007062 0.007062 0.007062 2.20% Gravity::get_old_grav_vector() 5 0.007057 0.007057 0.007057 2.20% amrex::Copy() 463 0.006769 0.006769 0.006769 2.11% MLMG::mgVcycle_down::1 36 0.006488 0.006488 0.006488 2.02% Castro::initialize_advance() 5 0.00645 0.00645 0.00645 2.01% FabArray::ParallelCopy() 380 0.006173 0.006173 0.006173 1.92% FabArray::ParallelCopy_nowait() 380 0.006066 0.006066 0.006066 1.89% MLMG::mgVcycle_down::2 36 0.006036 0.006036 0.006036 1.88% FabArray::setVal() 537 0.006011 0.006011 0.006011 1.87% FabArray::norminf() 326 0.005912 0.005912 0.005912 1.84% MLMG::mgVcycle_down::3 36 0.00589 0.00589 0.00589 1.83% MLMG::mgVcycle_down::4 36 0.005802 0.005802 0.005802 1.81% MLCellLinOp::defineAuxData() 6 0.005653 0.005653 0.005653 1.76% Castro::do_new_sources() 5 0.005523 0.005523 0.005523 1.72% Castro::expand_state() 5 0.005479 0.005479 0.005479 1.71% MLCGSolver::ParallelAllReduce 659 0.004988 0.004988 0.004988 1.55% MLMG::addInterpCorrection() 180 0.004852 0.004852 0.004852 1.51% MLMG::mgVcycle_up::4 36 0.004733 0.004733 0.004733 1.47% MLMG::mgVcycle_up::1 36 0.004714 0.004714 0.004714 1.47% MLMG::mgVcycle_up::2 36 0.004605 0.004605 0.004605 1.43% amrex::average_down 180 0.004528 0.004528 0.004528 1.41% MLMG::mgVcycle_up::3 36 0.004498 0.004498 0.004498 1.40% MLPoisson::Fapply() 500 0.004323 0.004323 0.004323 1.35% Castro::do_old_sources() 5 0.00414 0.00414 0.00414 1.29% FabArray::Saxpy() 355 0.003489 0.003489 0.003489 1.09% FabArray::Xpay() 361 0.003378 0.003378 0.003378 1.05% Castro::post_restart() 1 0.003299 0.003299 0.003299 1.03% Gravity::multilevel_solve_for_new_phi() 1 0.003184 0.003184 0.003184 0.99% Gravity::actual_multilevel_solve() 1 0.003171 0.003171 0.003171 0.99% Gravity::fill_multipole_BCs() 6 0.003137 0.003137 0.003137 0.98% MLCellLinOp::solutionResidual() 42 0.003134 0.003134 0.003134 0.98% Castro::enforce_min_density() 30 0.002733 0.002733 0.002733 0.85% Castro::post_timestep() 5 0.002677 0.002677 0.002677 0.83% Castro::reset_internal_energy(MultiFab) 30 0.002602 0.002602 0.002602 0.81% MLMG::computeResidual() 36 0.002598 0.002598 0.002598 0.81% MLCellLinOp::defineBC() 6 0.002485 0.002485 0.002485 0.77% Castro::estTimeStep() 10 0.002394 0.002394 0.002394 0.75% BndryData::define() 6 0.002381 0.002381 0.002381 0.74% MLMG::prepareForSolve() 6 0.002369 0.002369 0.002369 0.74% FabArray::LinComb() 242 0.001825 0.001825 0.001825 0.57% amrex::Add() 72 0.001806 0.001806 0.001806 0.56% Castro::construct_new_source() 25 0.001615 0.001615 0.001615 0.50% Castro::construct_new_gravity_source() 5 0.001604 0.001604 0.001604 0.50% Castro::computeNewDt() 5 0.001437 0.001437 0.001437 0.45% Castro::construct_old_source() 25 0.001168 0.001168 0.001168 0.36% Castro::construct_old_gravity_source() 5 0.001158 0.001158 0.001158 0.36% MLMG::ResNormInf() 42 0.000921 0.000921 0.000921 0.29% Castro::apply_source_to_state() 10 0.0009125 0.0009125 0.0009125 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008408 0.0008408 0.0008408 0.26% FabArrayBase::getCPC() 632 0.0007511 0.0007511 0.0007511 0.23% MLMG::getGradSolution() 6 0.0007345 0.0007345 0.0007345 0.23% MLCellLinOp::compGrad() 6 0.0007315 0.0007315 0.0007315 0.23% MLCellLinOp::setLevelBC() 6 0.0007128 0.0007128 0.0007128 0.22% Castro::reset_internal_energy(Fab) 240 0.0006604 0.0006604 0.0006604 0.21% FabArray::mult() 22 0.000657 0.000657 0.000657 0.20% FabArray::setDomainBndry() 20 0.0006293 0.0006293 0.0006293 0.20% Castro::check_for_nan() 10 0.0005886 0.0005886 0.0005886 0.18% MultiFab::contains_nan() 10 0.0005828 0.0005828 0.0005828 0.18% MLPoisson::prepareForSolve() 6 0.0005778 0.0005778 0.0005778 0.18% MLCellLinOp::prepareForSolve() 6 0.0005736 0.0005736 0.0005736 0.18% Castro::enforce_speed_limit() 30 0.0005609 0.0005609 0.0005609 0.17% MLMG::computeMLResidual() 6 0.0005523 0.0005523 0.0005523 0.17% Amr::InitAmr() 1 0.0004382 0.0004382 0.0004382 0.14% Gravity::update_max_rhs() 6 0.0004283 0.0004283 0.0004283 0.13% FabArrayBase::CPC::define() 244 0.0004063 0.0004063 0.0004063 0.13% FabArrayBase::getFB() 1766 0.0003166 0.0003166 0.0003166 0.10% Castro::finalize_advance() 5 0.0003026 0.0003026 0.0003026 0.09% Gravity::swapTimeLevels() 5 0.0002201 0.0002201 0.0002201 0.07% MLMG::MLResNormInf() 6 0.0001497 0.0001497 0.0001497 0.05% Castro::buildMetrics() 1 0.0001402 0.0001402 0.0001402 0.04% MultiFab::max() 6 0.0001336 0.0001336 0.0001336 0.04% MLLinOp::define() 6 0.0001165 0.0001165 0.0001165 0.04% MLMG::MLRhsNormInf() 6 0.0001129 0.0001129 0.0001129 0.04% MLLinOp::defineGrids() 6 0.0001116 0.0001116 0.0001116 0.03% FabArrayBase::FB::FB() 26 5.848e-05 5.848e-05 5.848e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.712e-05 2.712e-05 2.712e-05 0.01% Amr::writeSmallPlotFile() 1 2.572e-05 2.572e-05 2.572e-05 0.01% Castro::swap_state_time_levels() 5 2.022e-05 2.022e-05 2.022e-05 0.01% makeSFC 30 2.021e-05 2.021e-05 2.021e-05 0.01% Castro::initMFs() 1 1.923e-05 1.923e-05 1.923e-05 0.01% Castro::finalize_do_advance() 5 1.883e-05 1.883e-05 1.883e-05 0.01% DistributionMapping::Distribute() 31 8.628e-06 8.628e-06 8.628e-06 0.00% Amr::initSubcycle() 1 7.893e-06 7.893e-06 7.893e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.255e-06 4.255e-06 4.255e-06 0.00% Castro::retry_advance_ctu() 5 2.015e-06 2.015e-06 2.015e-06 0.00% Gravity::set_mass_offset() 6 1.83e-06 1.83e-06 1.83e-06 0.00% Castro::create_source_corrector() 5 1.744e-06 1.744e-06 1.744e-06 0.00% Castro::FluxRegFineAdd() 5 1.321e-06 1.321e-06 1.321e-06 0.00% Castro::FluxRegCrseInit 5 1.216e-06 1.216e-06 1.216e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.019e-06 1.019e-06 1.019e-06 0.00% AmrLevel::AmrLevel() 1 6.04e-07 6.04e-07 6.04e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-19-ga5063a5d705b) finalized