Initializing CUDA...
CUDA initialized with 1 GPU
AMReX (22.04-31-gc768c36d6e08) initialized

Starting run at 14:50:47 UTC on 2022-04-27.
Successfully read inputs file ... 

Castro git describe: 22.04-37-g780f24a28
AMReX git describe: 22.04-31-gc768c36d6
Microphysics git describe: 22.04-3-g3c498521

reading extern runtime parameters ...
3 Species: 
C12  O16  Mg24  
Successfully read inputs file ... 
INITIAL GRIDS 
  Level 0   8 grids  262144 cells  100 % of domain
            smallest grid: 32 x 32 x 32  biggest grid: 32 x 32 x 32

CHECKPOINT: file = dustcollapse-restart_chk00000
checkPoint() time = 0.037758127 secs.
PLOTFILE: file = dustcollapse-restart_plt00000
Write plotfile time = 0.047116434  seconds

[Level 0 step 1] ADVANCE with dt = 4.541742215e-05
[Level 0 step 1] Advanced 262144 cells

[STEP 1] Coarse TimeStep time: 0.04629337
[STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05

[Level 0 step 2] ADVANCE with dt = 4.768829326e-05
[Level 0 step 2] Advanced 262144 cells

[STEP 2] Coarse TimeStep time: 0.049616354
[STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05

[Level 0 step 3] ADVANCE with dt = 5.007270792e-05
[Level 0 step 3] Advanced 262144 cells

[STEP 3] Coarse TimeStep time: 0.059334225
[STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05

[Level 0 step 4] ADVANCE with dt = 5.257634331e-05
[Level 0 step 4] Advanced 262144 cells

[STEP 4] Coarse TimeStep time: 0.062513116
[STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05

[Level 0 step 5] ADVANCE with dt = 5.520516048e-05
[Level 0 step 5] Advanced 262144 cells

[STEP 5] Coarse TimeStep time: 0.050498661
[STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05

CHECKPOINT: file = dustcollapse-restart_chk00005
checkPoint() time = 0.037024725 secs.
[Level 0 step 6] ADVANCE with dt = 5.79654185e-05
[Level 0 step 6] Advanced 262144 cells

[STEP 6] Coarse TimeStep time: 0.055835937
[STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05

[Level 0 step 7] ADVANCE with dt = 6.086368943e-05
[Level 0 step 7] Advanced 262144 cells

[STEP 7] Coarse TimeStep time: 0.056783013
[STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05

[Level 0 step 8] ADVANCE with dt = 6.39068739e-05
[Level 0 step 8] Advanced 262144 cells

[STEP 8] Coarse TimeStep time: 0.058036944
[STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05

[Level 0 step 9] ADVANCE with dt = 6.71022176e-05
[Level 0 step 9] Advanced 262144 cells

[STEP 9] Coarse TimeStep time: 0.056943074
[STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05

[Level 0 step 10] ADVANCE with dt = 7.045732848e-05
[Level 0 step 10] Advanced 262144 cells

[STEP 10] Coarse TimeStep time: 0.055767233
[STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05

CHECKPOINT: file = dustcollapse-restart_chk00010
checkPoint() time = 0.034560973 secs.
PLOTFILE: file = dustcollapse-restart_plt00010
Write plotfile time = 0.021919769  seconds


Ending run at 14:50:48 UTC on 2022-04-27.
Run time = 0.77996696
Run time without initialization = 0.645638877

  Average number of zones advanced per microsecond: 4.060
  Average number of zones advanced per microsecond per rank: 4.060

CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728


TinyProfiler total time across processes [min...avg...max]: 0.78 ... 0.78 ... 0.78

--------------------------------------------------------------------------------------------
Name                                         NCalls  Excl. Min  Excl. Avg  Excl. Max   Max %
--------------------------------------------------------------------------------------------
Castro::construct_ctu_hydro_source()             10     0.1717     0.1717     0.1717  22.01%
VisMF::Write(FabArray)                           11      0.171      0.171      0.171  21.92%
MLCellLinOp::applyBC()                         4433    0.07226    0.07226    0.07226   9.26%
MLPoisson::Fsmooth()                           3280    0.06019    0.06019    0.06019   7.72%
StateData::FillBoundary(geom)                   328    0.02359    0.02359    0.02359   3.02%
MLCGSolver::bicgstab                             82    0.02108    0.02108    0.02108   2.70%
MultiFab::Dot()                                1114    0.02055    0.02055    0.02055   2.63%
StateDataPhysBCFunct::()                         41    0.02003    0.02003    0.02003   2.57%
Castro::normalize_species()                      62    0.01866    0.01866    0.01866   2.39%
MultiFab::LinComb()                            1586    0.01574    0.01574    0.01574   2.02%
FillBoundary_nowait()                          4023    0.01429    0.01429    0.01429   1.83%
Castro::computeTemp()                            63    0.01365    0.01365    0.01365   1.75%
FabArray::setVal()                             1144     0.0131     0.0131     0.0131   1.68%
FabArray::ParallelCopy_nowait()                 861    0.01301    0.01301    0.01301   1.67%
MLPoisson::Fapply()                            1142    0.01055    0.01055    0.01055   1.35%
Castro::enforce_min_density()                    62     0.0105     0.0105     0.0105   1.35%
MLCellLinOp::defineAuxData()                     11   0.009846   0.009846   0.009846   1.26%
Gravity::fill_multipole_BCs()                    11   0.008003   0.008003   0.008003   1.03%
Castro::estTimeStep()                            21   0.007203   0.007203   0.007203   0.92%
MLMG::addInterpCorrection()                     410   0.006698   0.006698   0.006698   0.86%
Castro::reset_internal_energy()                  63    0.00664    0.00664    0.00664   0.85%
MultiFab::Xpay()                                585   0.006072   0.006072   0.006072   0.78%
amrex::average_down                             410   0.006057   0.006057   0.006057   0.78%
Amr::checkPoint()                                 3   0.004928   0.004928   0.004928   0.63%
Castro::do_advance_ctu()                         10   0.004765   0.004765   0.004765   0.61%
BndryData::define()                              11    0.00357    0.00357    0.00357   0.46%
Castro::construct_new_gravity_source()           10   0.003124   0.003124   0.003124   0.40%
Castro::enforce_speed_limit()                    62   0.002625   0.002625   0.002625   0.34%
Amr::writePlotFile()                              2   0.002493   0.002493   0.002493   0.32%
Castro::construct_old_gravity_source()           10   0.002491   0.002491   0.002491   0.32%
Gravity::get_new_grav_vector()                   11   0.001931   0.001931   0.001931   0.25%
MLMG::ResNormInf()                               93   0.001901   0.001901   0.001901   0.24%
MultiFab::Saxpy()                                20   0.001828   0.001828   0.001828   0.23%
Castro::expand_state()                           10   0.001729   0.001729   0.001729   0.22%
Gravity::get_old_grav_vector()                   10   0.001719   0.001719   0.001719   0.22%
MLMG::oneIter()                                  82   0.001641   0.001641   0.001641   0.21%
FabArray::setVal(val, thecmd, scomp, ncomp)     462   0.001624   0.001624   0.001624   0.21%
Gravity::actual_solve_with_mlmg()                11   0.001371   0.001371   0.001371   0.18%
FabArray::mult()                                 43   0.001341   0.001341   0.001341   0.17%
MLCellLinOp::setLevelBC()                        11   0.001313   0.001313   0.001313   0.17%
FabArray::setDomainBndry()                       41   0.001291   0.001291   0.001291   0.17%
MultiFab::contains_nan()                         20   0.001174   0.001174   0.001174   0.15%
Castro::initData()                                1   0.001124   0.001124   0.001124   0.14%
MLCellLinOp::smooth()                          1640   0.001083   0.001083   0.001083   0.14%
MLCellLinOp::prepareForSolve()                   11   0.001072   0.001072   0.001072   0.14%
MLCellLinOp::compGrad()                          11  0.0009029  0.0009029  0.0009029   0.12%
FabArray::FillBoundary()                       4023  0.0008453  0.0008453  0.0008453   0.11%
FabArrayBase::getCPC()                         1323  0.0008212  0.0008212  0.0008212   0.11%
MLMG::prepareForSolve()                          11  0.0007735  0.0007735  0.0007735   0.10%
FabArrayBase::CPC::define()                     454     0.0007     0.0007     0.0007   0.09%
FabArrayBase::getFB()                          4023  0.0006429  0.0006429  0.0006429   0.08%
MLCellLinOp::apply()                           1142  0.0004719  0.0004719  0.0004719   0.06%
CGSolver::sxay()                               1586   0.000468   0.000468   0.000468   0.06%
Amr::InitAmr()                                    1  0.0004307  0.0004307  0.0004307   0.06%
Gravity::update_max_rhs()                        11   0.000409   0.000409   0.000409   0.05%
MLMG::mgVcycle()                                 82  0.0003958  0.0003958  0.0003958   0.05%
Gravity::solve_for_phi()                         10  0.0003884  0.0003884  0.0003884   0.05%
MLLinOp::defineGrids()                           11  0.0003827  0.0003827  0.0003827   0.05%
MLCGSolver::ParallelAllReduce                  1514   0.000354   0.000354   0.000354   0.05%
main()                                            1  0.0003247  0.0003247  0.0003247   0.04%
FabArray::ParallelCopy()                        861  0.0002942  0.0002942  0.0002942   0.04%
MultiFab::max()                                  11  0.0002687  0.0002687  0.0002687   0.03%
MultiFab::Copy()                                 11  0.0002558  0.0002558  0.0002558   0.03%
FillPatchIterator::Initialize                    41  0.0002554  0.0002554  0.0002554   0.03%
MLCellLinOp::correctionResidual()               492  0.0002412  0.0002412  0.0002412   0.03%
Castro::construct_new_gravity()                  10  0.0002348  0.0002348  0.0002348   0.03%
Amr::coarseTimeStep()                            10  0.0002188  0.0002188  0.0002188   0.03%
Amr::timeStep()                                  10   0.000206   0.000206   0.000206   0.03%
MLMG::MLRhsNormInf()                             11  0.0002004  0.0002004  0.0002004   0.03%
MLCellLinOp::defineBC()                          11  0.0001835  0.0001835  0.0001835   0.02%
Castro::subcycle_advance_ctu()                   10  0.0001835  0.0001835  0.0001835   0.02%
Amr::defBaseLevel()                               1  0.0001436  0.0001436  0.0001436   0.02%
StateData::checkPoint()                          12  0.0001354  0.0001354  0.0001354   0.02%
MLMG:computeResOfCorrection()                   410  0.0001329  0.0001329  0.0001329   0.02%
MLMG::actualBottomSolve()                        82  0.0001065  0.0001065  0.0001065   0.01%
MLMG::mgVcycle_down::0                           82  9.437e-05  9.437e-05  9.437e-05   0.01%
FabArrayBase::FB::FB()                           56  8.663e-05  8.663e-05  8.663e-05   0.01%
Castro::finalize_advance()                       10  8.258e-05  8.258e-05  8.258e-05   0.01%
MLMG::mgVcycle_down::1                           82  8.111e-05  8.111e-05  8.111e-05   0.01%
Castro::initialize_advance()                     10  8.036e-05  8.036e-05  8.036e-05   0.01%
Castro::advance()                                10  8.008e-05  8.008e-05  8.008e-05   0.01%
Castro::clean_state()                            62  7.935e-05  7.935e-05  7.935e-05   0.01%
MLMG::solve()                                    11  7.903e-05  7.903e-05  7.903e-05   0.01%
MLMG::mgVcycle_down::2                           82  7.587e-05  7.587e-05  7.587e-05   0.01%
MLMG::mgVcycle_down::4                           82   7.31e-05   7.31e-05   7.31e-05   0.01%
MLMG::mgVcycle_down::3                           82  7.264e-05  7.264e-05  7.264e-05   0.01%
AmrLevel::checkPoint()                            3  7.068e-05  7.068e-05  7.068e-05   0.01%
MLMG::mgVcycle_up::4                             82  6.398e-05  6.398e-05  6.398e-05   0.01%
Castro::initialize_do_advance()                  10  5.425e-05  5.425e-05  5.425e-05   0.01%
MLMG::mgVcycle_up::0                             82   5.23e-05   5.23e-05   5.23e-05   0.01%
MLCellLinOp::solutionResidual()                  93  5.061e-05  5.061e-05  5.061e-05   0.01%
MLMG::mgVcycle_up::3                             82  4.946e-05  4.946e-05  4.946e-05   0.01%
MLMG::mgVcycle_up::1                             82  4.673e-05  4.673e-05  4.673e-05   0.01%
MLMG::mgVcycle_up::2                             82  4.595e-05  4.595e-05  4.595e-05   0.01%
StateData::define()                               4  4.119e-05  4.119e-05  4.119e-05   0.01%
Castro::swap_state_time_levels()                 10  3.863e-05  3.863e-05  3.863e-05   0.00%
Castro::finalize_do_advance()                    10  3.693e-05  3.693e-05  3.693e-05   0.00%
Castro::enforce_consistent_e()                    1  3.639e-05  3.639e-05  3.639e-05   0.00%
MLMG::mgVcycle_bottom                            82  3.593e-05  3.593e-05  3.593e-05   0.00%
MLMG::computeResidual()                          82  3.249e-05  3.249e-05  3.249e-05   0.00%
Castro::post_timestep()                          10  3.241e-05  3.241e-05  3.241e-05   0.00%
Gravity::actual_multilevel_solve()                1  3.118e-05  3.118e-05  3.118e-05   0.00%
Amr::writeSmallPlotFile()                         1  3.059e-05  3.059e-05  3.059e-05   0.00%
FillPatchSingleLevel                             41  2.862e-05  2.862e-05  2.862e-05   0.00%
makeSFC                                          55  2.743e-05  2.743e-05  2.743e-05   0.00%
Gravity::solve_phi_with_mlmg()                   11   2.49e-05   2.49e-05   2.49e-05   0.00%
MLLinOp::define()                                11  2.442e-05  2.442e-05  2.442e-05   0.00%
MLPoisson::define()                              11   2.27e-05   2.27e-05   2.27e-05   0.00%
Amr::FinalizeInit()                               1  2.112e-05  2.112e-05  2.112e-05   0.00%
Castro::construct_old_source()                   50   1.96e-05   1.96e-05   1.96e-05   0.00%
Castro::construct_new_source()                   50  1.902e-05  1.902e-05  1.902e-05   0.00%
Gravity::multilevel_solve_for_new_phi()           1  1.755e-05  1.755e-05  1.755e-05   0.00%
Castro::do_new_sources()                         10   1.61e-05   1.61e-05   1.61e-05   0.00%
Castro::do_old_sources()                         10  1.577e-05  1.577e-05  1.577e-05   0.00%
MLLinOp::makeAgglomeratedDMap                    11  1.541e-05  1.541e-05  1.541e-05   0.00%
DistributionMapping::Distribute()                56  1.517e-05  1.517e-05  1.517e-05   0.00%
Amr::InitializeInit()                             1  1.113e-05  1.113e-05  1.113e-05   0.00%
Castro::check_for_nan()                          20  1.098e-05  1.098e-05  1.098e-05   0.00%
Castro::apply_source_to_state()                  20   1.08e-05   1.08e-05   1.08e-05   0.00%
Castro::construct_old_gravity()                  10  1.072e-05  1.072e-05  1.072e-05   0.00%
Gravity::swapTimeLevels()                        10  1.006e-05  1.006e-05  1.006e-05   0.00%
Amr::initSubcycle()                               1  9.633e-06  9.633e-06  9.633e-06   0.00%
AmrLevel::AmrLevel(dm)                            1  9.313e-06  9.313e-06  9.313e-06   0.00%
MLPoisson::prepareForSolve()                     11  9.119e-06  9.119e-06  9.119e-06   0.00%
MLMG::computeMLResidual()                        11  7.679e-06  7.679e-06  7.679e-06   0.00%
MLMG::getGradSolution()                          11  6.163e-06  6.163e-06  6.163e-06   0.00%
Castro::computeNewDt()                            9  5.848e-06  5.848e-06  5.848e-06   0.00%
MLMG::buildFineMask()                            11   5.72e-06   5.72e-06   5.72e-06   0.00%
Castro::create_source_corrector()                10  5.597e-06  5.597e-06  5.597e-06   0.00%
AmrLevel::checkPointPost()                        3  5.455e-06  5.455e-06  5.455e-06   0.00%
MLMG::MLResNormInf()                             11  5.197e-06  5.197e-06  5.197e-06   0.00%
Gravity::set_mass_offset()                       11  4.587e-06  4.587e-06  4.587e-06   0.00%
Castro::post_init()                               1  4.131e-06  4.131e-06  4.131e-06   0.00%
DistributionMapping::SFCProcessorMapDoIt()        1  3.494e-06  3.494e-06  3.494e-06   0.00%
Castro::retry_advance_ctu()                      10  3.347e-06  3.347e-06  3.347e-06   0.00%
Castro::FluxRegFineAdd()                         10  2.746e-06  2.746e-06  2.746e-06   0.00%
Amr::init()                                       1  2.483e-06  2.483e-06  2.483e-06   0.00%
Castro::computeInitialDt()                        2  2.375e-06  2.375e-06  2.375e-06   0.00%
MLLinOp::makeSubCommunicator()                   11  2.032e-06  2.032e-06  2.032e-06   0.00%
AmrLevel::checkPointPre()                         3  1.984e-06  1.984e-06  1.984e-06   0.00%
Castro::post_regrid()                             1  1.024e-06  1.024e-06  1.024e-06   0.00%
Amr::initialInit()                                1   8.92e-07   8.92e-07   8.92e-07   0.00%
--------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------
Name                                         NCalls  Incl. Min  Incl. Avg  Incl. Max   Max %
--------------------------------------------------------------------------------------------
main()                                            1       0.78       0.78       0.78 100.00%
Amr::coarseTimeStep()                            10     0.6235     0.6235     0.6235  79.93%
Amr::timeStep()                                  10     0.5481     0.5481     0.5481  70.27%
Castro::advance()                                10     0.5382     0.5382     0.5382  69.00%
Castro::subcycle_advance_ctu()                   10     0.5254     0.5254     0.5254  67.36%
Castro::do_advance_ctu()                         10     0.5252     0.5252     0.5252  67.33%
Gravity::solve_phi_with_mlmg()                   11     0.2918     0.2918     0.2918  37.41%
Gravity::actual_solve_with_mlmg()                11     0.2835     0.2835     0.2835  36.35%
Castro::construct_new_gravity()                  10     0.2739     0.2739     0.2739  35.12%
MLMG::solve()                                    11     0.2631     0.2631     0.2631  33.73%
Gravity::solve_for_phi()                         10     0.2504     0.2504     0.2504  32.10%
MLMG::oneIter()                                  82     0.2494     0.2494     0.2494  31.97%
MLMG::mgVcycle()                                 82     0.2478     0.2478     0.2478  31.76%
Castro::construct_ctu_hydro_source()             10     0.1717     0.1717     0.1717  22.01%
VisMF::Write(FabArray)                           11      0.171      0.171      0.171  21.92%
Amr::init()                                       1     0.1337     0.1337     0.1337  17.15%
MLCellLinOp::smooth()                          1640     0.1265     0.1265     0.1265  16.22%
Amr::checkPoint()                                 3     0.1095     0.1095     0.1095  14.03%
AmrLevel::checkPoint()                            3     0.1045     0.1045     0.1045  13.40%
StateData::checkPoint()                          12     0.1045     0.1045     0.1045  13.39%
MLCellLinOp::applyBC()                         4433    0.08813    0.08813    0.08813  11.30%
MLMG::mgVcycle_bottom                            82    0.07662    0.07662    0.07662   9.82%
MLMG::actualBottomSolve()                        82    0.07658    0.07658    0.07658   9.82%
MLCGSolver::bicgstab                             82    0.07585    0.07585    0.07585   9.72%
Amr::writePlotFile()                              2    0.06916    0.06916    0.06916   8.87%
MLPoisson::Fsmooth()                           3280    0.06019    0.06019    0.06019   7.72%
Castro::clean_state()                            62    0.05161    0.05161    0.05161   6.62%
FillPatchIterator::Initialize                    41    0.04921    0.04921    0.04921   6.31%
Amr::initialInit()                                1    0.04873    0.04873    0.04873   6.25%
FillPatchSingleLevel                             41    0.04766    0.04766    0.04766   6.11%
Amr::FinalizeInit()                               1    0.04502    0.04502    0.04502   5.77%
Castro::post_init()                               1    0.04368    0.04368    0.04368   5.60%
StateDataPhysBCFunct::()                         41    0.04361    0.04361    0.04361   5.59%
Gravity::multilevel_solve_for_new_phi()           1    0.04185    0.04185    0.04185   5.37%
Gravity::actual_multilevel_solve()                1    0.04184    0.04184    0.04184   5.36%
MLMG::mgVcycle_down::0                           82     0.0345     0.0345     0.0345   4.42%
MLCellLinOp::apply()                           1142    0.03342    0.03342    0.03342   4.29%
MLMG::mgVcycle_up::0                             82    0.02946    0.02946    0.02946   3.78%
Gravity::get_new_grav_vector()                   11    0.02508    0.02508    0.02508   3.22%
StateData::FillBoundary(geom)                   328    0.02359    0.02359    0.02359   3.02%
MultiFab::Dot()                                1114    0.02055    0.02055    0.02055   2.63%
Castro::computeTemp()                            63    0.02029    0.02029    0.02029   2.60%
MLCellLinOp::correctionResidual()               492    0.01952    0.01952    0.01952   2.50%
Castro::initialize_do_advance()                  10    0.01925    0.01925    0.01925   2.47%
Castro::normalize_species()                      62    0.01866    0.01866    0.01866   2.39%
MLMG:computeResOfCorrection()                   410    0.01692    0.01692    0.01692   2.17%
MLPoisson::define()                              11    0.01636    0.01636    0.01636   2.10%
CGSolver::sxay()                               1586    0.01621    0.01621    0.01621   2.08%
MLMG::mgVcycle_down::1                           82    0.01616    0.01616    0.01616   2.07%
FabArray::FillBoundary()                       4023    0.01587    0.01587    0.01587   2.03%
MultiFab::LinComb()                            1586    0.01574    0.01574    0.01574   2.02%
MLMG::mgVcycle_down::2                           82     0.0156     0.0156     0.0156   2.00%
FillBoundary_nowait()                          4023    0.01502    0.01502    0.01502   1.93%
MLMG::mgVcycle_down::3                           82    0.01469    0.01469    0.01469   1.88%
Castro::construct_old_gravity()                  10    0.01439    0.01439    0.01439   1.85%
Gravity::get_old_grav_vector()                   10    0.01438    0.01438    0.01438   1.84%
FabArray::ParallelCopy()                        861    0.01418    0.01418    0.01418   1.82%
MLMG::mgVcycle_down::4                           82     0.0139     0.0139     0.0139   1.78%
FabArray::ParallelCopy_nowait()                 861    0.01389    0.01389    0.01389   1.78%
FabArray::setVal()                             1144     0.0131     0.0131     0.0131   1.68%
Castro::initialize_advance()                     10    0.01264    0.01264    0.01264   1.62%
Castro::do_new_sources()                         10    0.01239    0.01239    0.01239   1.59%
MLCGSolver::ParallelAllReduce                  1514    0.01237    0.01237    0.01237   1.59%
MLMG::mgVcycle_up::2                             82    0.01206    0.01206    0.01206   1.55%
MLMG::mgVcycle_up::1                             82    0.01183    0.01183    0.01183   1.52%
MLMG::addInterpCorrection()                     410    0.01178    0.01178    0.01178   1.51%
MLMG::mgVcycle_up::3                             82    0.01129    0.01129    0.01129   1.45%
MLMG::mgVcycle_up::4                             82    0.01126    0.01126    0.01126   1.44%
MLCellLinOp::defineAuxData()                     11    0.01119    0.01119    0.01119   1.44%
amrex::average_down                             410    0.01114    0.01114    0.01114   1.43%
Castro::expand_state()                           10    0.01107    0.01107    0.01107   1.42%
Castro::do_old_sources()                         10    0.01071    0.01071    0.01071   1.37%
MLPoisson::Fapply()                            1142    0.01055    0.01055    0.01055   1.35%
Castro::enforce_min_density()                    62     0.0105     0.0105     0.0105   1.35%
Castro::post_timestep()                          10   0.009664   0.009664   0.009664   1.24%
Gravity::fill_multipole_BCs()                    11   0.008003   0.008003   0.008003   1.03%
Castro::estTimeStep()                            21   0.007203   0.007203   0.007203   0.92%
MLCellLinOp::solutionResidual()                  93   0.007034   0.007034   0.007034   0.90%
Castro::reset_internal_energy()                  63    0.00664    0.00664    0.00664   0.85%
MultiFab::Xpay()                                585   0.006072   0.006072   0.006072   0.78%
MLMG::computeResidual()                          82    0.00607    0.00607    0.00607   0.78%
MLCellLinOp::defineBC()                          11   0.004675   0.004675   0.004675   0.60%
BndryData::define()                              11   0.004491   0.004491   0.004491   0.58%
MLMG::prepareForSolve()                          11    0.00446    0.00446    0.00446   0.57%
Amr::InitializeInit()                             1   0.003715   0.003715   0.003715   0.48%
Amr::defBaseLevel()                               1   0.003704   0.003704   0.003704   0.47%
Castro::initData()                                1    0.00321    0.00321    0.00321   0.41%
Castro::construct_new_source()                   50   0.003143   0.003143   0.003143   0.40%
Castro::construct_new_gravity_source()           10   0.003124   0.003124   0.003124   0.40%
Castro::computeNewDt()                            9   0.003089   0.003089   0.003089   0.40%
Castro::enforce_speed_limit()                    62   0.002625   0.002625   0.002625   0.34%
Castro::construct_old_source()                   50   0.002511   0.002511   0.002511   0.32%
Castro::construct_old_gravity_source()           10   0.002491   0.002491   0.002491   0.32%
MLMG::ResNormInf()                               93   0.001901   0.001901   0.001901   0.24%
Castro::apply_source_to_state()                  20   0.001839   0.001839   0.001839   0.24%
MultiFab::Saxpy()                                20   0.001828   0.001828   0.001828   0.23%
FabArray::setVal(val, thecmd, scomp, ncomp)     462   0.001624   0.001624   0.001624   0.21%
FabArrayBase::getCPC()                         1323   0.001521   0.001521   0.001521   0.20%
MLMG::getGradSolution()                          11   0.001366   0.001366   0.001366   0.18%
MLCellLinOp::compGrad()                          11   0.001359   0.001359   0.001359   0.17%
FabArray::mult()                                 43   0.001341   0.001341   0.001341   0.17%
MLCellLinOp::setLevelBC()                        11   0.001313   0.001313   0.001313   0.17%
FabArray::setDomainBndry()                       41   0.001291   0.001291   0.001291   0.17%
Castro::check_for_nan()                          20   0.001185   0.001185   0.001185   0.15%
MultiFab::contains_nan()                         20   0.001174   0.001174   0.001174   0.15%
Castro::post_regrid()                             1   0.001105   0.001105   0.001105   0.14%
MLPoisson::prepareForSolve()                     11   0.001081   0.001081   0.001081   0.14%
MLCellLinOp::prepareForSolve()                   11   0.001072   0.001072   0.001072   0.14%
MLMG::computeMLResidual()                        11   0.001004   0.001004   0.001004   0.13%
Gravity::update_max_rhs()                        11  0.0008241  0.0008241  0.0008241   0.11%
FabArrayBase::getFB()                          4023  0.0007296  0.0007296  0.0007296   0.09%
FabArrayBase::CPC::define()                     454     0.0007     0.0007     0.0007   0.09%
Castro::computeInitialDt()                        2  0.0006538  0.0006538  0.0006538   0.08%
MLLinOp::define()                                11  0.0004658  0.0004658  0.0004658   0.06%
MLLinOp::defineGrids()                           11  0.0004414  0.0004414  0.0004414   0.06%
Gravity::swapTimeLevels()                        10  0.0004408  0.0004408  0.0004408   0.06%
Amr::InitAmr()                                    1  0.0004403  0.0004403  0.0004403   0.06%
MultiFab::max()                                  11  0.0002687  0.0002687  0.0002687   0.03%
MLMG::MLResNormInf()                             11  0.0002569  0.0002569  0.0002569   0.03%
MultiFab::Copy()                                 11  0.0002558  0.0002558  0.0002558   0.03%
MLMG::MLRhsNormInf()                             11  0.0002004  0.0002004  0.0002004   0.03%
FabArrayBase::FB::FB()                           56  8.663e-05  8.663e-05  8.663e-05   0.01%
Castro::finalize_advance()                       10  8.532e-05  8.532e-05  8.532e-05   0.01%
MLLinOp::makeAgglomeratedDMap                    11   5.66e-05   5.66e-05   5.66e-05   0.01%
AmrLevel::AmrLevel(dm)                            1   5.05e-05   5.05e-05   5.05e-05   0.01%
makeSFC                                          55  4.119e-05  4.119e-05  4.119e-05   0.01%
StateData::define()                               4  4.119e-05  4.119e-05  4.119e-05   0.01%
Castro::swap_state_time_levels()                 10  3.863e-05  3.863e-05  3.863e-05   0.00%
Castro::finalize_do_advance()                    10  3.693e-05  3.693e-05  3.693e-05   0.00%
Castro::enforce_consistent_e()                    1  3.639e-05  3.639e-05  3.639e-05   0.00%
Amr::writeSmallPlotFile()                         1  3.059e-05  3.059e-05  3.059e-05   0.00%
DistributionMapping::Distribute()                56  1.517e-05  1.517e-05  1.517e-05   0.00%
Amr::initSubcycle()                               1  9.633e-06  9.633e-06  9.633e-06   0.00%
MLMG::buildFineMask()                            11   5.72e-06   5.72e-06   5.72e-06   0.00%
Castro::create_source_corrector()                10  5.597e-06  5.597e-06  5.597e-06   0.00%
AmrLevel::checkPointPost()                        3  5.455e-06  5.455e-06  5.455e-06   0.00%
DistributionMapping::SFCProcessorMapDoIt()        1  4.905e-06  4.905e-06  4.905e-06   0.00%
Gravity::set_mass_offset()                       11  4.587e-06  4.587e-06  4.587e-06   0.00%
Castro::retry_advance_ctu()                      10  3.347e-06  3.347e-06  3.347e-06   0.00%
Castro::FluxRegFineAdd()                         10  2.746e-06  2.746e-06  2.746e-06   0.00%
MLLinOp::makeSubCommunicator()                   11  2.032e-06  2.032e-06  2.032e-06   0.00%
AmrLevel::checkPointPre()                         3  1.984e-06  1.984e-06  1.984e-06   0.00%
--------------------------------------------------------------------------------------------

Unused ParmParse Variables:
  [TOP]::amr.ref_ratio(nvals = 4)  :: [2, 2, 2, 2]
  [TOP]::amr.regrid_int(nvals = 4)  :: [2, 2, 2, 2]

Total GPU global memory (MB): 12066
Free  GPU global memory (MB): 2465
[The         Arena] space allocated (MB): 9049
[The         Arena] space used      (MB): 0
[The Managed Arena] space allocated (MB): 8
[The Managed Arena] space used      (MB): 0
[The  Pinned Arena] space allocated (MB): 8
[The  Pinned Arena] space used      (MB): 0
AMReX (22.04-31-gc768c36d6e08) finalized
Initializing CUDA...
CUDA initialized with 1 GPU
AMReX (22.04-31-gc768c36d6e08) initialized

Starting run at 14:50:48 UTC on 2022-04-27.
Successfully read inputs file ... 

Castro git describe: 22.04-37-g780f24a28
AMReX git describe: 22.04-31-gc768c36d6
Microphysics git describe: 22.04-3-g3c498521

reading extern runtime parameters ...
3 Species: 
C12  O16  Mg24  
restarting calculation from file: dustcollapse-restart_chk00005
Successfully read inputs file ... 
read CPU time: 0.439417021
Restart time = 0.044536395 seconds.
[Level 0 step 6] ADVANCE with dt = 5.79654185e-05
[Level 0 step 6] Advanced 262144 cells

[STEP 6] Coarse TimeStep time: 0.048239798
[STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05

[Level 0 step 7] ADVANCE with dt = 6.086368943e-05
[Level 0 step 7] Advanced 262144 cells

[STEP 7] Coarse TimeStep time: 0.045593029
[STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05

[Level 0 step 8] ADVANCE with dt = 6.39068739e-05
[Level 0 step 8] Advanced 262144 cells

[STEP 8] Coarse TimeStep time: 0.05765088
[STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05

[Level 0 step 9] ADVANCE with dt = 6.71022176e-05
[Level 0 step 9] Advanced 262144 cells

[STEP 9] Coarse TimeStep time: 0.058501891
[STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05

[Level 0 step 10] ADVANCE with dt = 7.045732848e-05
[Level 0 step 10] Advanced 262144 cells

[STEP 10] Coarse TimeStep time: 0.058805947
[STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638]

STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05

PLOTFILE: file = dustcollapse-restart_plt00010
Write plotfile time = 0.022884751  seconds


Ending run at 14:50:49 UTC on 2022-04-27.
Run time = 0.337127554
Run time without initialization = 0.292042256

  Average number of zones advanced per microsecond: 4.488
  Average number of zones advanced per microsecond per rank: 4.488

CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728


TinyProfiler total time across processes [min...avg...max]: 0.3372 ... 0.3372 ... 0.3372

--------------------------------------------------------------------------------------------
Name                                         NCalls  Excl. Min  Excl. Avg  Excl. Max   Max %
--------------------------------------------------------------------------------------------
Castro::construct_ctu_hydro_source()              5     0.0815     0.0815     0.0815  24.18%
VisMF::Read()                                     3     0.0377     0.0377     0.0377  11.18%
MLCellLinOp::applyBC()                         1946    0.03155    0.03155    0.03155   9.36%
MLPoisson::Fsmooth()                           1440    0.02546    0.02546    0.02546   7.55%
VisMF::Write(FabArray)                            1    0.02163    0.02163    0.02163   6.42%
StateData::FillBoundary(geom)                   160    0.01169    0.01169    0.01169   3.47%
Castro::normalize_species()                      30   0.009261   0.009261   0.009261   2.75%
MLCGSolver::bicgstab                             36   0.009061   0.009061   0.009061   2.69%
MultiFab::Dot()                                 484   0.008816   0.008816   0.008816   2.61%
Castro::computeTemp()                            30   0.006665   0.006665   0.006665   1.98%
FillBoundary_nowait()                          1766   0.006306   0.006306   0.006306   1.87%
FabArray::setVal()                              537    0.00625    0.00625    0.00625   1.85%
Castro::enforce_min_density()                    30   0.006045   0.006045   0.006045   1.79%
FabArray::ParallelCopy_nowait()                 380   0.005902   0.005902   0.005902   1.75%
StateDataPhysBCFunct::()                         20   0.005715   0.005715   0.005715   1.70%
MLCellLinOp::defineAuxData()                      6   0.005326   0.005326   0.005326   1.58%
MultiFab::LinComb()                             690   0.005223   0.005223   0.005223   1.55%
MLPoisson::Fapply()                             500   0.004531   0.004531   0.004531   1.34%
Gravity::fill_multipole_BCs()                     6    0.00448    0.00448    0.00448   1.33%
Castro::estTimeStep()                            10   0.003302   0.003302   0.003302   0.98%
Amr::restart()                                    1   0.002932   0.002932   0.002932   0.87%
MLMG::addInterpCorrection()                     180   0.002853   0.002853   0.002853   0.85%
amrex::average_down                             180    0.00265    0.00265    0.00265   0.79%
MultiFab::Xpay()                                258   0.002639   0.002639   0.002639   0.78%
Castro::reset_internal_energy()                  30   0.002543   0.002543   0.002543   0.75%
Castro::do_advance_ctu()                          5    0.00246    0.00246    0.00246   0.73%
BndryData::define()                               6   0.001979   0.001979   0.001979   0.59%
Castro::construct_new_gravity_source()            5   0.001563   0.001563   0.001563   0.46%
Amr::writePlotFile()                              1   0.001359   0.001359   0.001359   0.40%
Castro::construct_old_gravity_source()            5   0.001264   0.001264   0.001264   0.37%
Castro::enforce_speed_limit()                    30   0.001237   0.001237   0.001237   0.37%
MultiFab::Saxpy()                                10   0.000922   0.000922   0.000922   0.27%
Gravity::get_old_grav_vector()                    5  0.0008774  0.0008774  0.0008774   0.26%
Castro::expand_state()                            5  0.0008769  0.0008769  0.0008769   0.26%
FabArray::setVal(val, thecmd, scomp, ncomp)     252  0.0008764  0.0008764  0.0008764   0.26%
Gravity::get_new_grav_vector()                    5  0.0008581  0.0008581  0.0008581   0.25%
MLMG::ResNormInf()                               42  0.0008403  0.0008403  0.0008403   0.25%
Gravity::actual_solve_with_mlmg()                 6   0.000727   0.000727   0.000727   0.22%
MLMG::oneIter()                                  36  0.0007169  0.0007169  0.0007169   0.21%
MLCellLinOp::setLevelBC()                         6  0.0007006  0.0007006  0.0007006   0.21%
FabArray::mult()                                 22  0.0006506  0.0006506  0.0006506   0.19%
FabArray::setDomainBndry()                       20  0.0006302  0.0006302  0.0006302   0.19%
MultiFab::contains_nan()                         10  0.0005815  0.0005815  0.0005815   0.17%
MLCellLinOp::prepareForSolve()                    6  0.0005807  0.0005807  0.0005807   0.17%
MLCellLinOp::compGrad()                           6  0.0004846  0.0004846  0.0004846   0.14%
MLCellLinOp::smooth()                           720  0.0004826  0.0004826  0.0004826   0.14%
MLMG::prepareForSolve()                           6  0.0004196  0.0004196  0.0004196   0.12%
FabArrayBase::CPC::define()                     244  0.0004154  0.0004154  0.0004154   0.12%
FabArrayBase::getCPC()                          632   0.000404   0.000404   0.000404   0.12%
FabArray::FillBoundary()                       1766  0.0003986  0.0003986  0.0003986   0.12%
Amr::InitAmr()                                    1  0.0003692  0.0003692  0.0003692   0.11%
FabArrayBase::getFB()                          1766  0.0003085  0.0003085  0.0003085   0.09%
main()                                            1  0.0002834  0.0002834  0.0002834   0.08%
Gravity::update_max_rhs()                         6  0.0002262  0.0002262  0.0002262   0.07%
MLLinOp::defineGrids()                            6  0.0002045  0.0002045  0.0002045   0.06%
MLCellLinOp::apply()                            500  0.0002012  0.0002012  0.0002012   0.06%
Gravity::solve_for_phi()                          5  0.0001974  0.0001974  0.0001974   0.06%
MLMG::mgVcycle()                                 36  0.0001857  0.0001857  0.0001857   0.06%
CGSolver::sxay()                                690  0.0001803  0.0001803  0.0001803   0.05%
MultiFab::Copy()                                  6  0.0001402  0.0001402  0.0001402   0.04%
FabArray::ParallelCopy()                        380  0.0001373  0.0001373  0.0001373   0.04%
MLCGSolver::ParallelAllReduce                   659  0.0001357  0.0001357  0.0001357   0.04%
MultiFab::max()                                   6  0.0001352  0.0001352  0.0001352   0.04%
FillPatchIterator::Initialize                    20  0.0001234  0.0001234  0.0001234   0.04%
Amr::coarseTimeStep()                             5  0.0001088  0.0001088  0.0001088   0.03%
MLMG::MLRhsNormInf()                              6  0.0001056  0.0001056  0.0001056   0.03%
Castro::construct_new_gravity()                   5  0.0001052  0.0001052  0.0001052   0.03%
Amr::timeStep()                                   5  0.0001034  0.0001034  0.0001034   0.03%
MLCellLinOp::correctionResidual()               216  0.0001024  0.0001024  0.0001024   0.03%
MLCellLinOp::defineBC()                           6  9.869e-05  9.869e-05  9.869e-05   0.03%
Castro::subcycle_advance_ctu()                    5  9.664e-05  9.664e-05  9.664e-05   0.03%
AmrLevel::restart()                               1  7.893e-05  7.893e-05  7.893e-05   0.02%
StateData::restartDoit()                          4  7.133e-05  7.133e-05  7.133e-05   0.02%
FabArrayBase::FB::FB()                           26  6.094e-05  6.094e-05  6.094e-05   0.02%
MLMG:computeResOfCorrection()                   180  5.514e-05  5.514e-05  5.514e-05   0.02%
Castro::advance()                                 5  5.358e-05  5.358e-05  5.358e-05   0.02%
MLMG::actualBottomSolve()                        36  4.573e-05  4.573e-05  4.573e-05   0.01%
MLMG::mgVcycle_down::0                           36  4.247e-05  4.247e-05  4.247e-05   0.01%
Castro::clean_state()                            30  4.088e-05  4.088e-05  4.088e-05   0.01%
Castro::initialize_advance()                      5  3.679e-05  3.679e-05  3.679e-05   0.01%
MLMG::mgVcycle_down::1                           36   3.63e-05   3.63e-05   3.63e-05   0.01%
MLMG::solve()                                     6   3.56e-05   3.56e-05   3.56e-05   0.01%
MLMG::mgVcycle_down::2                           36  3.279e-05  3.279e-05  3.279e-05   0.01%
Castro::post_restart()                            1  3.211e-05  3.211e-05  3.211e-05   0.01%
MLMG::mgVcycle_down::4                           36  3.149e-05  3.149e-05  3.149e-05   0.01%
MLMG::mgVcycle_down::3                           36  3.128e-05  3.128e-05  3.128e-05   0.01%
Castro::initialize_do_advance()                   5  2.914e-05  2.914e-05  2.914e-05   0.01%
Gravity::actual_multilevel_solve()                1  2.914e-05  2.914e-05  2.914e-05   0.01%
Amr::writeSmallPlotFile()                         1  2.749e-05  2.749e-05  2.749e-05   0.01%
Castro::swap_state_time_levels()                  5  2.695e-05  2.695e-05  2.695e-05   0.01%
MLMG::mgVcycle_up::4                             36  2.589e-05  2.589e-05  2.589e-05   0.01%
Castro::finalize_advance()                        5  2.459e-05  2.459e-05  2.459e-05   0.01%
MLCellLinOp::solutionResidual()                  42  2.331e-05  2.331e-05  2.331e-05   0.01%
MLMG::mgVcycle_up::0                             36  2.295e-05  2.295e-05  2.295e-05   0.01%
MLLinOp::define()                                 6  2.234e-05  2.234e-05  2.234e-05   0.01%
MLMG::mgVcycle_up::3                             36  2.086e-05  2.086e-05  2.086e-05   0.01%
Castro::finalize_do_advance()                     5  2.028e-05  2.028e-05  2.028e-05   0.01%
MLMG::mgVcycle_up::2                             36  2.021e-05  2.021e-05  2.021e-05   0.01%
MLMG::mgVcycle_up::1                             36  1.936e-05  1.936e-05  1.936e-05   0.01%
MLMG::mgVcycle_bottom                            36  1.646e-05  1.646e-05  1.646e-05   0.00%
Gravity::multilevel_solve_for_new_phi()           1  1.563e-05  1.563e-05  1.563e-05   0.00%
MLMG::computeResidual()                          36  1.551e-05  1.551e-05  1.551e-05   0.00%
makeSFC                                          30  1.484e-05  1.484e-05  1.484e-05   0.00%
FillPatchSingleLevel                             20   1.44e-05   1.44e-05   1.44e-05   0.00%
MLPoisson::define()                               6  1.432e-05  1.432e-05  1.432e-05   0.00%
Gravity::solve_phi_with_mlmg()                    6   1.43e-05   1.43e-05   1.43e-05   0.00%
Amr::initSubcycle()                               1  1.009e-05  1.009e-05  1.009e-05   0.00%
Castro::construct_old_source()                   25   9.17e-06   9.17e-06   9.17e-06   0.00%
Castro::construct_new_source()                   25  8.976e-06  8.976e-06  8.976e-06   0.00%
DistributionMapping::Distribute()                31  8.721e-06  8.721e-06  8.721e-06   0.00%
Castro::do_new_sources()                          5  8.628e-06  8.628e-06  8.628e-06   0.00%
Castro::do_old_sources()                          5  8.368e-06  8.368e-06  8.368e-06   0.00%
MLLinOp::makeAgglomeratedDMap                     6  7.636e-06  7.636e-06  7.636e-06   0.00%
Castro::check_for_nan()                          10   5.64e-06   5.64e-06   5.64e-06   0.00%
Castro::apply_source_to_state()                  10  5.571e-06  5.571e-06  5.571e-06   0.00%
Castro::post_timestep()                           5  5.379e-06  5.379e-06  5.379e-06   0.00%
Castro::construct_old_gravity()                   5  5.094e-06  5.094e-06  5.094e-06   0.00%
MLPoisson::prepareForSolve()                      6  5.039e-06  5.039e-06  5.039e-06   0.00%
Gravity::swapTimeLevels()                         5  4.281e-06  4.281e-06  4.281e-06   0.00%
MLMG::buildFineMask()                             6  3.239e-06  3.239e-06  3.239e-06   0.00%
DistributionMapping::SFCProcessorMapDoIt()        1  3.181e-06  3.181e-06  3.181e-06   0.00%
MLMG::computeMLResidual()                         6  3.105e-06  3.105e-06  3.105e-06   0.00%
Castro::computeNewDt()                            5   3.07e-06   3.07e-06   3.07e-06   0.00%
MLMG::getGradSolution()                           6  2.791e-06  2.791e-06  2.791e-06   0.00%
MLMG::MLResNormInf()                              6  2.657e-06  2.657e-06  2.657e-06   0.00%
Gravity::set_mass_offset()                        6  2.265e-06  2.265e-06  2.265e-06   0.00%
Castro::create_source_corrector()                 5    2.2e-06    2.2e-06    2.2e-06   0.00%
Castro::FluxRegFineAdd()                          5  2.154e-06  2.154e-06  2.154e-06   0.00%
Castro::retry_advance_ctu()                       5  1.824e-06  1.824e-06  1.824e-06   0.00%
Amr::init()                                       1  1.164e-06  1.164e-06  1.164e-06   0.00%
AmrLevel::AmrLevel()                              1  1.114e-06  1.114e-06  1.114e-06   0.00%
MLLinOp::makeSubCommunicator()                    6   9.92e-07   9.92e-07   9.92e-07   0.00%
--------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------
Name                                         NCalls  Incl. Min  Incl. Avg  Incl. Max   Max %
--------------------------------------------------------------------------------------------
main()                                            1     0.3371     0.3371     0.3371 100.00%
Amr::coarseTimeStep()                             5     0.2689     0.2689     0.2689  79.75%
Amr::timeStep()                                   5      0.267      0.267      0.267  79.19%
Castro::advance()                                 5     0.2622     0.2622     0.2622  77.77%
Castro::subcycle_advance_ctu()                    5     0.2557     0.2557     0.2557  75.83%
Castro::do_advance_ctu()                          5     0.2556     0.2556     0.2556  75.80%
Castro::construct_new_gravity()                   5     0.1326     0.1326     0.1326  39.34%
Gravity::solve_phi_with_mlmg()                    6     0.1282     0.1282     0.1282  38.04%
Gravity::solve_for_phi()                          5      0.125      0.125      0.125  37.08%
Gravity::actual_solve_with_mlmg()                 6     0.1236     0.1236     0.1236  36.67%
MLMG::solve()                                     6     0.1125     0.1125     0.1125  33.37%
MLMG::oneIter()                                  36     0.1059     0.1059     0.1059  31.42%
MLMG::mgVcycle()                                 36     0.1052     0.1052     0.1052  31.21%
Castro::construct_ctu_hydro_source()              5    0.08153    0.08153    0.08153  24.18%
MLCellLinOp::smooth()                           720    0.05448    0.05448    0.05448  16.16%
Amr::init()                                       1    0.04457    0.04457    0.04457  13.22%
Amr::restart()                                    1    0.04457    0.04457    0.04457  13.22%
MLCellLinOp::applyBC()                         1946    0.03862    0.03862    0.03862  11.45%
AmrLevel::restart()                               1    0.03791    0.03791    0.03791  11.25%
StateData::restartDoit()                          4    0.03783    0.03783    0.03783  11.22%
VisMF::Read()                                     3     0.0377     0.0377     0.0377  11.18%
MLMG::mgVcycle_bottom                            36     0.0314     0.0314     0.0314   9.31%
MLMG::actualBottomSolve()                        36    0.03139    0.03139    0.03139   9.31%
MLCGSolver::bicgstab                             36    0.03106    0.03106    0.03106   9.21%
Castro::clean_state()                            30    0.02579    0.02579    0.02579   7.65%
MLPoisson::Fsmooth()                           1440    0.02546    0.02546    0.02546   7.55%
Amr::writePlotFile()                              1    0.02299    0.02299    0.02299   6.82%
VisMF::Write(FabArray)                            1    0.02163    0.02163    0.02163   6.42%
FillPatchIterator::Initialize                    20    0.02019    0.02019    0.02019   5.99%
FillPatchSingleLevel                             20    0.01943    0.01943    0.01943   5.76%
StateDataPhysBCFunct::()                         20    0.01741    0.01741    0.01741   5.16%
MLMG::mgVcycle_down::0                           36    0.01479    0.01479    0.01479   4.39%
MLCellLinOp::apply()                            500    0.01457    0.01457    0.01457   4.32%
MLMG::mgVcycle_up::0                             36    0.01258    0.01258    0.01258   3.73%
StateData::FillBoundary(geom)                   160    0.01169    0.01169    0.01169   3.47%
Castro::initialize_do_advance()                   5    0.01088    0.01088    0.01088   3.23%
Castro::normalize_species()                      30   0.009261   0.009261   0.009261   2.75%
Castro::computeTemp()                            30   0.009209   0.009209   0.009209   2.73%
MLPoisson::define()                               6   0.008955   0.008955   0.008955   2.66%
MultiFab::Dot()                                 484   0.008816   0.008816   0.008816   2.61%
MLCellLinOp::correctionResidual()               216   0.008467   0.008467   0.008467   2.51%
Gravity::get_new_grav_vector()                    5   0.007521   0.007521   0.007521   2.23%
MLMG:computeResOfCorrection()                   180   0.007322   0.007322   0.007322   2.17%
Castro::construct_old_gravity()                   5   0.007321   0.007321   0.007321   2.17%
Gravity::get_old_grav_vector()                    5   0.007315   0.007315   0.007315   2.17%
FabArray::FillBoundary()                       1766   0.007074   0.007074   0.007074   2.10%
MLMG::mgVcycle_down::1                           36   0.006961   0.006961   0.006961   2.06%
MLMG::mgVcycle_down::2                           36   0.006746   0.006746   0.006746   2.00%
FillBoundary_nowait()                          1766   0.006675   0.006675   0.006675   1.98%
Castro::initialize_advance()                      5   0.006468   0.006468   0.006468   1.92%
FabArray::ParallelCopy()                        380   0.006456   0.006456   0.006456   1.91%
MLMG::mgVcycle_down::3                           36   0.006386   0.006386   0.006386   1.89%
FabArray::ParallelCopy_nowait()                 380   0.006319   0.006319   0.006319   1.87%
FabArray::setVal()                              537    0.00625    0.00625    0.00625   1.85%
Castro::do_new_sources()                          5   0.006188   0.006188   0.006188   1.84%
MLMG::mgVcycle_down::4                           36   0.006086   0.006086   0.006086   1.81%
MLCellLinOp::defineAuxData()                      6   0.006072   0.006072   0.006072   1.80%
Castro::expand_state()                            5   0.006052   0.006052   0.006052   1.79%
Castro::enforce_min_density()                    30   0.006045   0.006045   0.006045   1.79%
Castro::do_old_sources()                          5   0.005654   0.005654   0.005654   1.68%
CGSolver::sxay()                                690   0.005403   0.005403   0.005403   1.60%
MLCGSolver::ParallelAllReduce                   659   0.005331   0.005331   0.005331   1.58%
MultiFab::LinComb()                             690   0.005223   0.005223   0.005223   1.55%
MLMG::mgVcycle_up::2                             36   0.005216   0.005216   0.005216   1.55%
MLMG::mgVcycle_up::1                             36   0.005097   0.005097   0.005097   1.51%
MLMG::addInterpCorrection()                     180   0.005067   0.005067   0.005067   1.50%
MLMG::mgVcycle_up::3                             36   0.004896   0.004896   0.004896   1.45%
amrex::average_down                             180   0.004879   0.004879   0.004879   1.45%
MLMG::mgVcycle_up::4                             36   0.004871   0.004871   0.004871   1.44%
Castro::post_timestep()                           5   0.004677   0.004677   0.004677   1.39%
MLPoisson::Fapply()                             500   0.004531   0.004531   0.004531   1.34%
Gravity::fill_multipole_BCs()                     6    0.00448    0.00448    0.00448   1.33%
Castro::post_restart()                            1   0.003599   0.003599   0.003599   1.07%
Gravity::multilevel_solve_for_new_phi()           1   0.003477   0.003477   0.003477   1.03%
Gravity::actual_multilevel_solve()                1   0.003462   0.003462   0.003462   1.03%
Castro::estTimeStep()                            10   0.003302   0.003302   0.003302   0.98%
MLCellLinOp::solutionResidual()                  42   0.003158   0.003158   0.003158   0.94%
MultiFab::Xpay()                                258   0.002639   0.002639   0.002639   0.78%
MLMG::computeResidual()                          36   0.002623   0.002623   0.002623   0.78%
MLCellLinOp::defineBC()                           6   0.002611   0.002611   0.002611   0.77%
Castro::reset_internal_energy()                  30   0.002543   0.002543   0.002543   0.75%
BndryData::define()                               6   0.002513   0.002513   0.002513   0.75%
MLMG::prepareForSolve()                           6   0.002418   0.002418   0.002418   0.72%
Castro::computeNewDt()                            5   0.001802   0.001802   0.001802   0.53%
Castro::construct_new_source()                   25   0.001572   0.001572   0.001572   0.47%
Castro::construct_new_gravity_source()            5   0.001563   0.001563   0.001563   0.46%
Castro::construct_old_source()                   25   0.001273   0.001273   0.001273   0.38%
Castro::construct_old_gravity_source()            5   0.001264   0.001264   0.001264   0.37%
Castro::enforce_speed_limit()                    30   0.001237   0.001237   0.001237   0.37%
Castro::apply_source_to_state()                  10  0.0009275  0.0009275  0.0009275   0.28%
MultiFab::Saxpy()                                10   0.000922   0.000922   0.000922   0.27%
FabArray::setVal(val, thecmd, scomp, ncomp)     252  0.0008764  0.0008764  0.0008764   0.26%
MLMG::ResNormInf()                               42  0.0008403  0.0008403  0.0008403   0.25%
FabArrayBase::getCPC()                          632  0.0008194  0.0008194  0.0008194   0.24%
MLMG::getGradSolution()                           6  0.0007386  0.0007386  0.0007386   0.22%
MLCellLinOp::compGrad()                           6  0.0007358  0.0007358  0.0007358   0.22%
MLCellLinOp::setLevelBC()                         6  0.0007006  0.0007006  0.0007006   0.21%
FabArray::mult()                                 22  0.0006506  0.0006506  0.0006506   0.19%
FabArray::setDomainBndry()                       20  0.0006302  0.0006302  0.0006302   0.19%
Castro::check_for_nan()                          10  0.0005871  0.0005871  0.0005871   0.17%
MLPoisson::prepareForSolve()                      6  0.0005857  0.0005857  0.0005857   0.17%
MultiFab::contains_nan()                         10  0.0005815  0.0005815  0.0005815   0.17%
MLCellLinOp::prepareForSolve()                    6  0.0005807  0.0005807  0.0005807   0.17%
MLMG::computeMLResidual()                         6   0.000554   0.000554   0.000554   0.16%
Gravity::update_max_rhs()                         6  0.0004381  0.0004381  0.0004381   0.13%
FabArrayBase::CPC::define()                     244  0.0004154  0.0004154  0.0004154   0.12%
Amr::InitAmr()                                    1  0.0003793  0.0003793  0.0003793   0.11%
FabArrayBase::getFB()                          1766  0.0003694  0.0003694  0.0003694   0.11%
MLLinOp::define()                                 6  0.0002578  0.0002578  0.0002578   0.08%
MLLinOp::defineGrids()                            6  0.0002354  0.0002354  0.0002354   0.07%
Gravity::swapTimeLevels()                         5  0.0002276  0.0002276  0.0002276   0.07%
MultiFab::Copy()                                  6  0.0001402  0.0001402  0.0001402   0.04%
MultiFab::max()                                   6  0.0001352  0.0001352  0.0001352   0.04%
MLMG::MLResNormInf()                              6  0.0001335  0.0001335  0.0001335   0.04%
MLMG::MLRhsNormInf()                              6  0.0001056  0.0001056  0.0001056   0.03%
FabArrayBase::FB::FB()                           26  6.094e-05  6.094e-05  6.094e-05   0.02%
MLLinOp::makeAgglomeratedDMap                     6  2.998e-05  2.998e-05  2.998e-05   0.01%
Amr::writeSmallPlotFile()                         1  2.749e-05  2.749e-05  2.749e-05   0.01%
Castro::swap_state_time_levels()                  5  2.695e-05  2.695e-05  2.695e-05   0.01%
Castro::finalize_advance()                        5  2.674e-05  2.674e-05  2.674e-05   0.01%
makeSFC                                          30  2.234e-05  2.234e-05  2.234e-05   0.01%
Castro::finalize_do_advance()                     5  2.028e-05  2.028e-05  2.028e-05   0.01%
Amr::initSubcycle()                               1  1.009e-05  1.009e-05  1.009e-05   0.00%
DistributionMapping::Distribute()                31  8.721e-06  8.721e-06  8.721e-06   0.00%
DistributionMapping::SFCProcessorMapDoIt()        1    4.4e-06    4.4e-06    4.4e-06   0.00%
MLMG::buildFineMask()                             6  3.239e-06  3.239e-06  3.239e-06   0.00%
Gravity::set_mass_offset()                        6  2.265e-06  2.265e-06  2.265e-06   0.00%
Castro::create_source_corrector()                 5    2.2e-06    2.2e-06    2.2e-06   0.00%
Castro::FluxRegFineAdd()                          5  2.154e-06  2.154e-06  2.154e-06   0.00%
Castro::retry_advance_ctu()                       5  1.824e-06  1.824e-06  1.824e-06   0.00%
AmrLevel::AmrLevel()                              1  1.114e-06  1.114e-06  1.114e-06   0.00%
MLLinOp::makeSubCommunicator()                    6   9.92e-07   9.92e-07   9.92e-07   0.00%
--------------------------------------------------------------------------------------------

Unused ParmParse Variables:
  [TOP]::amr.ref_ratio(nvals = 4)  :: [2, 2, 2, 2]
  [TOP]::amr.regrid_int(nvals = 4)  :: [2, 2, 2, 2]

Total GPU global memory (MB): 12066
Free  GPU global memory (MB): 2465
[The         Arena] space allocated (MB): 9049
[The         Arena] space used      (MB): 0
[The Managed Arena] space allocated (MB): 8
[The Managed Arena] space used      (MB): 0
[The  Pinned Arena] space allocated (MB): 8
[The  Pinned Arena] space used      (MB): 0
AMReX (22.04-31-gc768c36d6e08) finalized