MPI initialized with 4 MPI processes MPI initialized with thread support level 3 OMP initialized with 4 OMP threads AMReX (23.07-44-g49dd703558c5) initialized Starting run at 06:26:28 UTC on 2023-07-27. Successfully read inputs file ... Castro git describe: 23.07-20-g4174f17eb AMReX git describe: 23.07-44-g49dd70355 Microphysics git describe: 23.07-48-g588beb9c reading extern runtime parameters ... 13 Species: He4 C12 O16 Ne20 Mg24 Si28 S32 Ar36 Ca40 Ti44 Cr48 Fe52 Ni56 Successfully read inputs file ... Generated initial model for primary WD of mass 0.9 solar masses, central density 1.574e+07 g cm**-3, and radius 6.747e+08 cm. Generated initial model for secondary WD of mass 0.9 solar masses, central density 1.574e+07 g cm**-3, and radius 6.747e+08 cm. Generated binary orbit of distance 1.781e+09 cm = 1.190e-04 AU. The primary orbits the center of mass at distance -8.903e+08 cm = -5.951e-05 AU. The secondary orbits the center of mass at distance 8.903e+08 cm = 5.951e-05 AU. The initial orbital period is 30.543687 s. The initial orbital speed of the primary is -1.831e+08 cm/s. The initial orbital speed of the secondary is 1.831e+08 cm/s. Castro::numpts_1d at level 0 is 174 Setting the gravity type to PoissonGrav Initializing the data at level 0 Done initializing the level 0 data ...estimated hydro-limited timestep at level 0: 0.05426072637 ...hydro-limited CFL timestep constrained at (i,j,k) = (64,0,0) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.05426072637 grid_places() time: 0.034310586 new finest: 0 Now regridding at level lbase = 0 grid_places() time: 0.036302213 new finest: 0 STEP = 0 TIME = 0 : REGRID with lbase = 0 Gravity::fill_multipole_BCs() time = 0.365073387 Gravity::multilevel_solve_for_new_phi() time = 1.729612921 Gravity::fill_multipole_BCs() time = 0.361650142 ... norm of composite residual at level 0 3.509358608e-10 INITIAL GRIDS Level 0 27 grids 884736 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 PLOTFILE: file = wdmerger_3D_plt00000 Write plotfile time = 0.599100797 seconds ...estimated hydro-limited timestep at level 0: 0.05426072637 ...hydro-limited CFL timestep constrained at (i,j,k) = (64,0,0) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.05426072637 [Level 0 step 1] ADVANCE at time 0 with dt = 0.005426072637  Beginning subcycle 1 starting at time 0 with dt = 0.005426072637 Estimated number of subcycles remaining: 1 Castro::construct_old_gravity() time = 0.899901185 on level 0 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: -3.287624606e+47 ener added: 6.129070894e+49 Contributions to the state from old-time sources: mass added: 0 xmom added: 1.44936293e+25 ymom added: -1.046793351e+26 zmom added: -4.133074948e+24 eint added: 0 ener added: -1.270533196e+50 Castro::do_old_sources() time = 2.438403092 on level 0 ... Entering construct_ctu_hydro_source() on level 0 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 9.00547245e+47 ener added: 1.651139801e+46 ... Leaving construct_ctu_hydro_source() on level 0 Castro::construct_ctu_hydro_source() time = 20.85502862 on level 0 ... new-time composite Poisson gravity solve from level 0 to level 0 Gravity::fill_multipole_BCs() time = 0.684687123 Gravity::multilevel_solve_for_new_phi() time = 1.731238695 Castro::construct_new_gravity() time = 1.846552324 on level 0 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 6.440206272e+49 ener added: 2.911791643e+45 Contributions to the state from new-time sources: mass added: 0 xmom added: 1.451207927e+26 ymom added: 1.075144428e+26 zmom added: -5.613375295e+26 eint added: 0 ener added: 1.260457978e+50 Castro::do_new_sources() time = 2.404058983 on level 0 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 2.184461852e+27 ener added: 2.243834889e+27 ...estimated hydro-limited timestep at level 0: 0.05461684804 ...hydro-limited CFL timestep constrained at (i,j,k) = (0,29,94) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.05461684804  Subcycle completed Subcycling complete Zones advanced per microsecond at this level: 0.02460056836 [Level 0 step 1] Advanced 884736 cells Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 5.190352593e+26 ener added: 6.213783704e+26 Updating the rotational period from 30.54368673 s to 30.45939751 s. [STEP 1] Coarse TimeStep time: 38.19039274 [STEP 1] FAB kilobyte spread across MPI nodes: [679617 ... 761594] STEP = 1 TIME = 0.005426072637 DT = 0.005426072637 ...estimated hydro-limited timestep at level 0: 0.05461684804 ...hydro-limited CFL timestep constrained at (i,j,k) = (0,29,94) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.05461684804 Castro::compute_new_dt : limiting dt at level 0 ... new dt computed: 0.05461684804 ... but limiting to: 0.0059686799 = 1.1 * 0.005426072637 Now regridding at level lbase = 0 grid_places() time: 0.038284466 new finest: 1 Castro::numpts_1d at level 1 is 340 Gravity::fill_multipole_BCs() time = 0.431405983 Gravity::multilevel_solve_for_new_phi() time = 1.963588413 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 2.306135866e+46 ener added: 9.211440366e+51 STEP = 1 TIME = 0.005426072637 : REGRID with lbase = 0 Level 1 4 grids 65536 cells 0.9259259259 % of domain smallest grid: 32 x 32 x 16 biggest grid: 32 x 32 x 16 ...estimated hydro-limited timestep at level 0: 0.05461684804 ...hydro-limited CFL timestep constrained at (i,j,k) = (0,29,94) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.05461684804 ...estimated hydro-limited timestep at level 1: 2.526135286e-09 ...hydro-limited CFL timestep constrained at (i,j,k) = (87,103,92) Castro::estTimeStep (hydro-limited) at level 1: estdt = 2.526135286e-09 [Level 0 step 2] ADVANCE at time 0.005426072637 with dt = 2.526135286e-09 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 2.849569753e+31 ener added: 2.785937322e+31  Beginning subcycle 1 starting at time 0.005426072637 with dt = 2.526135286e-09 Estimated number of subcycles remaining: 1 Castro::construct_old_gravity() time = 0.879531266 on level 0 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: -1.200514105e+40 ener added: 6.433158191e+33 Contributions to the state from old-time sources: mass added: 0 xmom added: 5.533896722e+24 ymom added: 4.756222073e+26 zmom added: -2.866993882e+23 eint added: 0 ener added: -1.721443002e+45 Castro::do_old_sources() time = 2.140089131 on level 0 ... Entering construct_ctu_hydro_source() on level 0 ... Leaving construct_ctu_hydro_source() on level 0 Castro::construct_ctu_hydro_source() time = 21.07338575 on level 0 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 1.147330623e+31 ener added: 1.147330701e+31 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 4.551111111e+23 ener added: 6.281481481e+23 Castro::construct_old_gravity() time = 0.039990733 on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: -1.197253775e+40 ener added: 1.79168512e+42 Contributions to the state from old-time sources: mass added: 0 xmom added: 7.574244488e+22 ymom added: 1.475739526e+22 zmom added: 6.442237137e+21 eint added: 0 ener added: -2.586789965e+47 Castro::do_old_sources() time = 0.202664379 on level 1 ... Entering construct_ctu_hydro_source() on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 9.204912576e+51 ener added: 6.747117479e+48 ... Leaving construct_ctu_hydro_source() on level 1 Castro::construct_ctu_hydro_source() time = 2.811047982 on level 1 Castro::reflux() at level 0 : time = 2.315851878 ... new-time composite Poisson gravity solve from level 0 to level 1 Gravity::fill_multipole_BCs() time = 0.670759289 Gravity::multilevel_solve_for_new_phi() time = 1.621956549 Castro::construct_new_gravity() time = 1.736562695 on level 0 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: -3.46375568e+41 ener added: 7.485558923e+35 Contributions to the state from new-time sources: mass added: 0 xmom added: -4.864072325e+21 ymom added: -6.244635709e+21 zmom added: -1.246148803e+20 eint added: 0 ener added: -3.223710603e+41 Castro::do_new_sources() time = 2.143139622 on level 0 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: -1.14873037e+26 ener added: 4.251851852e+22 ...estimated hydro-limited timestep at level 0: 0.05461716025 ...hydro-limited CFL timestep constrained at (i,j,k) = (0,29,94) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.05461716025 Castro::construct_new_gravity() time = 0.028828 on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 2.564837764e+47 ener added: 4.559472528e+41 Contributions to the state from new-time sources: mass added: 0 xmom added: 5.112126331e+18 ymom added: -1.460517359e+19 zmom added: -8.220975454e+19 eint added: 0 ener added: 2.564611685e+47 Castro::do_new_sources() time = 0.15149105 on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 3.281876859e+29 ener added: 3.281875295e+29 ...estimated hydro-limited timestep at level 1: 6.287037368e-07 ...hydro-limited CFL timestep constrained at (i,j,k) = (121,90,90) Castro::estTimeStep (hydro-limited) at level 1: estdt = 6.287037368e-07  Subcycle completed Subcycling complete Zones advanced per microsecond from level 0 to level 1: 0.02276412096 [Level 0 step 2] Advanced 884736 cells [Level 1 step 2] ADVANCE at time 0.005426072637 with dt = 2.526135286e-09 Advance at this level has already been completed. [Level 1 step 2] Advanced 65536 cells Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 6.036240741e+24 ener added: 5.917722222e+24 Updating the rotational period from 30.45939751 s to 30.52749735 s. [STEP 2] Coarse TimeStep time: 50.11075698 [STEP 2] FAB kilobyte spread across MPI nodes: [833440 ... 934735] STEP = 2 TIME = 0.005426075163 DT = 2.526135286e-09 ...estimated hydro-limited timestep at level 0: 0.0008131491259 ...hydro-limited CFL timestep constrained at (i,j,k) = (37,49,42) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.0008131491259 ...estimated hydro-limited timestep at level 1: 6.287037368e-07 ...hydro-limited CFL timestep constrained at (i,j,k) = (121,90,90) Castro::estTimeStep (hydro-limited) at level 1: estdt = 6.287037368e-07 Castro::compute_new_dt : limiting dt at level 0 ... new dt computed: 0.0008131491259 ... but limiting to: 2.778748814e-09 = 1.1 * 2.526135286e-09 Castro::compute_new_dt : limiting dt at level 1 ... new dt computed: 6.287037368e-07 ... but limiting to: 2.778748814e-09 = 1.1 * 2.526135286e-09 Now regridding at level lbase = 0 grid_places() time: 0.031738098 new finest: 1 Castro::numpts_1d at level 1 is 340 Gravity::fill_multipole_BCs() time = 0.406313243 Gravity::multilevel_solve_for_new_phi() time = 0.534021688 STEP = 2 TIME = 0.005426075163 : REGRID with lbase = 0 Level 1 4 grids 65536 cells 0.9259259259 % of domain smallest grid: 32 x 32 x 16 biggest grid: 32 x 32 x 16 ...estimated hydro-limited timestep at level 0: 0.0008131491259 ...hydro-limited CFL timestep constrained at (i,j,k) = (37,49,42) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.0008131491259 ...estimated hydro-limited timestep at level 1: 6.287037368e-07 ...hydro-limited CFL timestep constrained at (i,j,k) = (121,90,90) Castro::estTimeStep (hydro-limited) at level 1: estdt = 6.287037368e-07 [Level 0 step 3] ADVANCE at time 0.005426075163 with dt = 2.778748814e-09  Beginning subcycle 1 starting at time 0.005426075163 with dt = 2.778748814e-09 Estimated number of subcycles remaining: 1 Castro::construct_old_gravity() time = 0.881383947 on level 0 Contributions to the state from old-time sources: mass added: 0 xmom added: 6.083120164e+24 ymom added: 5.231862796e+26 zmom added: -3.157842738e+23 eint added: 0 ener added: -1.937110674e+45 Castro::do_old_sources() time = 2.138814796 on level 0 ... Entering construct_ctu_hydro_source() on level 0 ... Leaving construct_ctu_hydro_source() on level 0 Castro::construct_ctu_hydro_source() time = 21.40953526 on level 0 Castro::construct_old_gravity() time = 0.040546131 on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: -1.818691488e+40 ener added: 1.212217726e+36 Contributions to the state from old-time sources: mass added: 0 xmom added: 8.441961473e+22 ymom added: 3.836922767e+21 zmom added: 7.10314939e+21 eint added: 0 ener added: -2.344787748e+45 Castro::do_old_sources() time = 0.205540066 on level 1 ... Entering construct_ctu_hydro_source() on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 1.83673455e+48 ener added: 2.077900047e+42 ... Leaving construct_ctu_hydro_source() on level 1 Castro::construct_ctu_hydro_source() time = 2.794808144 on level 1 Castro::reflux() at level 0 : time = 2.276396447 ... new-time composite Poisson gravity solve from level 0 to level 1 Gravity::fill_multipole_BCs() time = 0.579112015 Gravity::multilevel_solve_for_new_phi() time = 1.157562877 Castro::construct_new_gravity() time = 1.27239882 on level 0 Contributions to the state from new-time sources: mass added: 0 xmom added: -2.108598872e+21 ymom added: -7.90284783e+21 zmom added: 1.10682106e+20 eint added: 0 ener added: -3.427512073e+41 Castro::do_new_sources() time = 2.095849003 on level 0 ...estimated hydro-limited timestep at level 0: 0.0008131520194 ...hydro-limited CFL timestep constrained at (i,j,k) = (37,49,42) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.0008131520194 Castro::construct_new_gravity() time = 0.031651687 on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 5.244373444e+43 ener added: 1.74625952e+38 Contributions to the state from new-time sources: mass added: 0 xmom added: -5.604455924e+17 ymom added: 2.1743379e+19 zmom added: 9.538642682e+19 eint added: 0 ener added: 5.303047905e+43 Castro::do_new_sources() time = 0.162551217 on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 1.495038611e+26 ener added: 1.500090463e+26 ...estimated hydro-limited timestep at level 1: 6.564309766e-07 ...hydro-limited CFL timestep constrained at (i,j,k) = (121,90,90) Castro::estTimeStep (hydro-limited) at level 1: estdt = 6.564309766e-07  Subcycle completed Subcycling complete Zones advanced per microsecond from level 0 to level 1: 0.02285260299 [Level 0 step 3] Advanced 884736 cells [Level 1 step 3] ADVANCE at time 0.005426075163 with dt = 2.778748814e-09 Advance at this level has already been completed. [Level 1 step 3] Advanced 65536 cells Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 6.283851852e+24 ener added: 6.674962963e+24 Updating the rotational period from 30.52749735 s to 30.52751553 s. [STEP 3] Coarse TimeStep time: 48.03768003 [STEP 3] FAB kilobyte spread across MPI nodes: [833440 ... 934735] STEP = 3 TIME = 0.005426077942 DT = 2.778748814e-09 ...estimated hydro-limited timestep at level 0: 0.0008130329735 ...hydro-limited CFL timestep constrained at (i,j,k) = (37,49,42) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.0008130329735 ...estimated hydro-limited timestep at level 1: 6.564309766e-07 ...hydro-limited CFL timestep constrained at (i,j,k) = (121,90,90) Castro::estTimeStep (hydro-limited) at level 1: estdt = 6.564309766e-07 Castro::compute_new_dt : limiting dt at level 0 ... new dt computed: 0.0008130329735 ... but limiting to: 3.056623696e-09 = 1.1 * 2.778748814e-09 Castro::compute_new_dt : limiting dt at level 1 ... new dt computed: 6.564309766e-07 ... but limiting to: 3.056623696e-09 = 1.1 * 2.778748814e-09 Now regridding at level lbase = 0 grid_places() time: 0.032455905 new finest: 1 Castro::numpts_1d at level 1 is 340 Gravity::fill_multipole_BCs() time = 0.402196387 Gravity::multilevel_solve_for_new_phi() time = 0.535115609 STEP = 3 TIME = 0.005426077942 : REGRID with lbase = 0 Level 1 4 grids 65536 cells 0.9259259259 % of domain smallest grid: 32 x 32 x 16 biggest grid: 32 x 32 x 16 ...estimated hydro-limited timestep at level 0: 0.0008130329735 ...hydro-limited CFL timestep constrained at (i,j,k) = (37,49,42) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.0008130329735 ...estimated hydro-limited timestep at level 1: 6.564309766e-07 ...hydro-limited CFL timestep constrained at (i,j,k) = (121,90,90) Castro::estTimeStep (hydro-limited) at level 1: estdt = 6.564309766e-07 [Level 0 step 4] ADVANCE at time 0.005426077942 with dt = 3.056623696e-09  Beginning subcycle 1 starting at time 0.005426077942 with dt = 3.056623695e-09 Estimated number of subcycles remaining: 1 Castro::construct_old_gravity() time = 0.899889296 on level 0 Contributions to the state from old-time sources: mass added: 0 xmom added: 6.694655893e+24 ymom added: 5.755038681e+26 zmom added: -3.471615689e+23 eint added: 0 ener added: -2.130751807e+45 Castro::do_old_sources() time = 2.161228289 on level 0 ... Entering construct_ctu_hydro_source() on level 0 ... Leaving construct_ctu_hydro_source() on level 0 Castro::construct_ctu_hydro_source() time = 21.48807329 on level 0 Castro::construct_old_gravity() time = 0.046692337 on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: -2.150425501e+40 ener added: 1.373884257e+36 Contributions to the state from old-time sources: mass added: 0 xmom added: 9.201587e+22 ymom added: 1.180591621e+22 zmom added: 8.023757211e+21 eint added: 0 ener added: -2.520808161e+45 Castro::do_old_sources() time = 0.202750435 on level 1 ... Entering construct_ctu_hydro_source() on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 1.318881554e+48 ener added: 1.675623034e+42 ... Leaving construct_ctu_hydro_source() on level 1 Castro::construct_ctu_hydro_source() time = 2.815861096 on level 1 Castro::reflux() at level 0 : time = 2.314049287 ... new-time composite Poisson gravity solve from level 0 to level 1 Gravity::fill_multipole_BCs() time = 0.588267361 Gravity::multilevel_solve_for_new_phi() time = 1.149569161 Castro::construct_new_gravity() time = 1.265403449 on level 0 Contributions to the state from new-time sources: mass added: 0 xmom added: -2.405469507e+21 ymom added: -9.735672131e+21 zmom added: -7.437548959e+18 eint added: 0 ener added: -3.723007452e+41 Castro::do_new_sources() time = 2.134730507 on level 0 ...estimated hydro-limited timestep at level 0: 0.000813036171 ...hydro-limited CFL timestep constrained at (i,j,k) = (37,49,42) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.000813036171 Castro::construct_new_gravity() time = 0.030015547 on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 3.983601425e+43 ener added: 1.617265925e+38 Contributions to the state from new-time sources: mass added: 0 xmom added: 3.521743097e+18 ymom added: -2.667031699e+19 zmom added: -1.338681259e+19 eint added: 0 ener added: 4.042918443e+43 Castro::do_new_sources() time = 0.146647369 on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 2.851958796e+25 ener added: 2.794684722e+25 ...estimated hydro-limited timestep at level 1: 6.863488975e-07 ...hydro-limited CFL timestep constrained at (i,j,k) = (121,90,90) Castro::estTimeStep (hydro-limited) at level 1: estdt = 6.863488975e-07  Subcycle completed Subcycling complete Zones advanced per microsecond from level 0 to level 1: 0.02278067482 [Level 0 step 4] Advanced 884736 cells [Level 1 step 4] ADVANCE at time 0.005426077942 with dt = 3.056623696e-09 Advance at this level has already been completed. [Level 1 step 4] Advanced 65536 cells Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: -3.365462963e+23 ener added: 4.62962963e+19 Updating the rotational period from 30.52751553 s to 30.52751552 s. [STEP 4] Coarse TimeStep time: 48.05617319 [STEP 4] FAB kilobyte spread across MPI nodes: [833440 ... 934735] STEP = 4 TIME = 0.005426080998 DT = 3.056623696e-09 ...estimated hydro-limited timestep at level 0: 0.0002506596503 ...hydro-limited CFL timestep constrained at (i,j,k) = (57,41,46) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.0002506596503 ...estimated hydro-limited timestep at level 1: 6.863488975e-07 ...hydro-limited CFL timestep constrained at (i,j,k) = (121,90,90) Castro::estTimeStep (hydro-limited) at level 1: estdt = 6.863488975e-07 Castro::compute_new_dt : limiting dt at level 0 ... new dt computed: 0.0002506596503 ... but limiting to: 3.362286065e-09 = 1.1 * 3.056623696e-09 Castro::compute_new_dt : limiting dt at level 1 ... new dt computed: 6.863488975e-07 ... but limiting to: 3.362286065e-09 = 1.1 * 3.056623696e-09 Now regridding at level lbase = 0 grid_places() time: 0.032308201 new finest: 1 Castro::numpts_1d at level 1 is 340 Gravity::fill_multipole_BCs() time = 0.407132528 Gravity::multilevel_solve_for_new_phi() time = 0.541907642 STEP = 4 TIME = 0.005426080998 : REGRID with lbase = 0 Level 1 4 grids 65536 cells 0.9259259259 % of domain smallest grid: 32 x 32 x 16 biggest grid: 32 x 32 x 16 ...estimated hydro-limited timestep at level 0: 0.0002506596503 ...hydro-limited CFL timestep constrained at (i,j,k) = (57,41,46) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.0002506596503 ...estimated hydro-limited timestep at level 1: 6.863488975e-07 ...hydro-limited CFL timestep constrained at (i,j,k) = (121,90,90) Castro::estTimeStep (hydro-limited) at level 1: estdt = 6.863488975e-07 [Level 0 step 5] ADVANCE at time 0.005426080998 with dt = 3.362286065e-09  Beginning subcycle 1 starting at time 0.005426080998 with dt = 3.362286065e-09 Estimated number of subcycles remaining: 1 Castro::construct_old_gravity() time = 0.843994255 on level 0 Contributions to the state from old-time sources: mass added: 0 xmom added: 7.366517551e+24 ymom added: 6.330531998e+26 zmom added: -3.819051789e+23 eint added: 0 ener added: -2.343742535e+45 Castro::do_old_sources() time = 2.129190384 on level 0 ... Entering construct_ctu_hydro_source() on level 0 ... Leaving construct_ctu_hydro_source() on level 0 Castro::construct_ctu_hydro_source() time = 21.24275617 on level 0 Castro::construct_old_gravity() time = 0.036091773 on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: -2.560109508e+40 ener added: 1.662186972e+36 Contributions to the state from old-time sources: mass added: 0 xmom added: 1.013522137e+23 ymom added: 1.829917012e+22 zmom added: 8.796935195e+21 eint added: 0 ener added: -2.728291723e+45 Castro::do_old_sources() time = 0.202613098 on level 1 ... Entering construct_ctu_hydro_source() on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 1.05099824e+48 ener added: 1.448876286e+42 ... Leaving construct_ctu_hydro_source() on level 1 Castro::construct_ctu_hydro_source() time = 2.748299127 on level 1 Castro::reflux() at level 0 : time = 2.262530969 ... new-time composite Poisson gravity solve from level 0 to level 1 Gravity::fill_multipole_BCs() time = 0.585679251 Gravity::multilevel_solve_for_new_phi() time = 1.139315855 Castro::construct_new_gravity() time = 1.250635677 on level 0 Contributions to the state from new-time sources: mass added: 0 xmom added: -2.798425632e+21 ymom added: -1.178643014e+22 zmom added: -5.143756067e+18 eint added: 0 ener added: -4.148475158e+41 Castro::do_new_sources() time = 2.094247495 on level 0 ...estimated hydro-limited timestep at level 0: 0.0006828000901 ...hydro-limited CFL timestep constrained at (i,j,k) = (57,41,46) Castro::estTimeStep (hydro-limited) at level 0: estdt = 0.0006828000901 Castro::construct_new_gravity() time = 0.030747607 on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 3.326552265e+43 ener added: 1.577214716e+38 Contributions to the state from new-time sources: mass added: 0 xmom added: 2.013307822e+18 ymom added: -2.460766836e+19 zmom added: -1.244728843e+19 eint added: 0 ener added: 3.390623254e+43 Castro::do_new_sources() time = 0.158264812 on level 1 Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: 4.005277778e+24 ener added: 3.882013889e+24 ...estimated hydro-limited timestep at level 1: 7.169281148e-07 ...hydro-limited CFL timestep constrained at (i,j,k) = (121,90,90) Castro::estTimeStep (hydro-limited) at level 1: estdt = 7.169281148e-07  Subcycle completed Subcycling complete Zones advanced per microsecond from level 0 to level 1: 0.02304969113 [Level 0 step 5] Advanced 884736 cells [Level 1 step 5] ADVANCE at time 0.005426080998 with dt = 3.362286065e-09 Advance at this level has already been completed. [Level 1 step 5] Advanced 65536 cells Contributions to the state from negative energy resets: mass added: 0 xmom added: 0 ymom added: 0 zmom added: 0 eint added: -2.814813006e+22 ener added: 2.962962963e+20 Updating the rotational period from 30.52751552 s to 30.52751551 s. [STEP 5] Coarse TimeStep time: 47.54622903 [STEP 5] FAB kilobyte spread across MPI nodes: [833440 ... 934735] STEP = 5 TIME = 0.005426084361 DT = 3.362286065e-09 PLOTFILE: file = wdmerger_3D_plt00005 Write plotfile time = 0.580064076 seconds Ending run at 06:30:33 UTC on 2023-07-27. Run time = 244.3511427 Run time without initialization = 232.5234937 Average number of zones advanced per microsecond: 0.021 Average number of zones advanced per microsecond per rank: 0.005 TinyProfiler total time across processes [min...avg...max]: 244.4 ... 244.4 ... 244.4 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 9 93.5484 102.7006 106.2212 43.47% Castro::computeTemp() 88 14.86 16.68 17.47 7.15% MultiFab::AddProduct() 4205 10.31 12.67 15.07 6.17% amrex::Copy() 736 12.97 14.35 14.94 6.11% check_for_negative_density() 9 0.713 4.28 13.52 5.53% Castro::normalize_species() 87 9.867 10.99 11.45 4.69% FabArray::ParallelCopy_finish() 830 1.68 4.219 10.83 4.43% Castro::estTimeStep() 26 4.085 5.709 9.841 4.03% Castro::reset_internal_energy(Fab) 1476 8.528 9.317 9.62 3.94% MultiFab::Subtract() 175 7.014 7.802 8.108 3.32% FabArray::setVal() 5078 6.984 7.758 8.105 3.32% Castro::reset_internal_energy(MultiFab) 88 3.761 4.858 6.482 2.65% Gravity::fill_multipole_BCs() 11 4.309 4.633 5.402 2.21% Castro::enforce_min_density() 87 1.772 2.508 3.979 1.63% FabArray::sum() 4205 3.227 3.568 3.719 1.52% FluxRegister::Reflux() 24 3.04 3.4 3.542 1.45% FabArray::ParallelCopy_nowait() 830 2.076 2.295 2.666 1.09% MLPoisson::Fsmooth() 1376 2.099 2.341 2.429 0.99% Castro::wd_update() 6 0.4634 0.9462 2.019 0.83% Castro::FluxRegCrseInit 9 1.596 1.764 1.85 0.76% Castro::reflux() 4 1.553 1.739 1.814 0.74% FabArray::Saxpy() 125 1.39 1.536 1.594 0.65% FillPatchInterp(MF) 30 1.235 1.325 1.425 0.58% Castro::construct_old_gravity_source() 18 0.9279 1.072 1.197 0.49% amrex::Add() 143 1.013 1.118 1.187 0.49% Amr::InitAmr() 1 1.184 1.184 1.184 0.48% Castro::initMFs() 5 1.086 1.143 1.173 0.48% MLPoisson::Fapply() 326 0.8877 1.022 1.092 0.45% FillBoundary_finish() 1561 0.3987 0.604 1.06 0.43% Castro::problem_post_timestep() 10 0.8005 0.9358 1.05 0.43% Castro::construct_new_gravity_source() 9 0.7701 0.8804 0.9998 0.41% FillPatchIterator::Initialize 62 0.8056 0.8839 0.9242 0.38% VisMF::Write(FabArray) 3 0.6426 0.7297 0.805 0.33% Castro::volInBoundary() 42 0.5831 0.688 0.7315 0.30% StateData::FillBoundary(geom) 50 0.6066 0.6166 0.6259 0.26% Castro::construct_new_source() 63 0.4878 0.5377 0.5767 0.24% MultiFab::max() 18 0.02913 0.1632 0.5384 0.22% MultiFab::contains_nan() 18 0.4042 0.4514 0.4686 0.19% MLCellLinOp::applyBC() 1750 0.3544 0.4178 0.4485 0.18% FabArray::Xpay() 291 0.352 0.3922 0.4076 0.17% FillBoundary_nowait() 1561 0.3281 0.3642 0.4043 0.17% Castro::initialize_do_advance() 9 0.2427 0.2707 0.285 0.12% Amr::writePlotFile() 2 0.08313 0.1582 0.2744 0.11% FabArray::setDomainBndry() 92 0.1999 0.2216 0.2351 0.10% Castro::construct_old_source() 63 0.1864 0.2022 0.2203 0.09% amrex::average_down 233 0.1945 0.2026 0.2061 0.08% Castro::fill_hybrid_hydro_source() 27 0.1724 0.1904 0.1985 0.08% MLCellLinOp::defineAuxData() 11 0.1573 0.1697 0.1747 0.07% Castro::initData() 1 0.1326 0.1459 0.1554 0.06% MLPoisson::FFlux() 150 0.004617 0.04204 0.154 0.06% StateData::define() 25 0.1214 0.1355 0.1479 0.06% MLCellLinOp::compGrad() 18 0.1232 0.1352 0.141 0.06% FabArray::mult() 56 0.1119 0.1249 0.1297 0.05% Castro::evaluate_source_change() 193 0.115 0.12 0.1259 0.05% Castro::swap_state_time_levels() 9 0.09355 0.112 0.1243 0.05% MLMG::addInterpCorrection() 155 0.1034 0.1171 0.1228 0.05% Castro::initialize_advance() 9 0.0828 0.09239 0.1032 0.04% Gravity::get_old_grav_vector() 9 0.08089 0.09008 0.09865 0.04% Gravity::get_new_grav_vector() 10 0.08724 0.09473 0.09791 0.04% AmrLevel::FillPatch() 62 0.07866 0.08421 0.09376 0.04% MLMG::prepareForSolve() 10 0.08113 0.08538 0.09051 0.04% MLMG::solve() 10 0.003929 0.04361 0.06851 0.03% MLCellLinOp::updateSolBC() 25 0.06703 0.06752 0.06827 0.03% FabArray::norminf(mask) 21 0.05354 0.05969 0.06239 0.03% Castro::apply_tagging_restrictions() 6 0.04768 0.05503 0.0608 0.02% MLMG::ResNormInf() 54 0.000921 0.02447 0.05732 0.02% AmrMesh::MakeNewGrids() 6 0.04685 0.04868 0.05285 0.02% Gravity::multilevel_solve_for_new_phi() 10 0.04084 0.04429 0.04862 0.02% Castro::do_new_sources() 9 0.0006082 0.01245 0.04789 0.02% BndryData::define() 27 0.04539 0.04602 0.04714 0.02% MLCellLinOp::updateCorBC() 17 0.0454 0.04559 0.04593 0.02% FabArray::norminf() 117 0.0378 0.03919 0.04067 0.02% Castro::do_old_sources() 9 0.0006098 0.01011 0.03836 0.02% MLCellLinOp::reflux() 30 0.004184 0.01199 0.03487 0.01% MLMG::MLResNormInf() 15 9.342e-05 0.02352 0.03402 0.01% Castro::FluxRegFineAdd() 9 0.02985 0.03055 0.03151 0.01% FabArray::setVal(val, thecmd, scomp, ncomp) 622 0.01874 0.02277 0.02843 0.01% MultiFab::Multiply() 8 0.02253 0.02582 0.02738 0.01% MLCellLinOp::defineBC() 11 0.02617 0.02643 0.02685 0.01% StateDataPhysBCFunct::() 95 0.002086 0.00784 0.02381 0.01% Castro::build_fine_mask() 44 0.02015 0.02143 0.02338 0.01% FillPatchTwoLevels 31 0.01389 0.01646 0.02078 0.01% FabArrayBase::CPC::define() 416 0.01447 0.01661 0.02014 0.01% Castro::Castro() 5 0.01634 0.01814 0.01941 0.01% MLMG::interpCorrection_1 17 0.01827 0.01862 0.01922 0.01% Castro::buildMetrics() 5 0.01546 0.01732 0.01858 0.01% Castro::linear_to_hybrid_momentum() 1 0.01385 0.01603 0.01747 0.01% MLLinOp::makeSubCommunicator() 11 0.0004437 0.008171 0.0161 0.01% DistributionMapping::LeastUsedCPUs() 5 0.001201 0.008506 0.01562 0.01% Castro::construct_old_gravity() 9 0.0003944 0.006035 0.01421 0.01% Gravity::actual_multilevel_solve() 10 0.01182 0.01312 0.01397 0.01% MLCellLinOp::prepareForSolve() 11 0.01157 0.01168 0.01175 0.00% Gravity::update_max_rhs() 10 0.008383 0.009726 0.01165 0.00% FabArrayBase::getCPC() 1440 0.01018 0.01071 0.01122 0.00% Castro::enforce_consistent_e() 1 0.009165 0.01062 0.01121 0.00% Castro::finalize_do_advance() 9 0.008162 0.009365 0.01038 0.00% AMRErrorTag::operator() 6 0.008587 0.009762 0.01036 0.00% Gravity::actual_solve_with_mlmg() 11 0.009408 0.009774 0.01026 0.00% Castro::apply_problem_tags() 6 0.006339 0.007086 0.007352 0.00% AmrLevel::derive() 6 0.005304 0.006214 0.007183 0.00% Castro::construct_new_gravity() 9 0.0004671 0.002113 0.00683 0.00% average_down_faces 48 0.005775 0.005806 0.005856 0.00% MLCellLinOp::setLevelBC() 19 0.003877 0.004592 0.005336 0.00% MLCGSolver::bicgstab 7 0 0.001304 0.005218 0.00% FabArray::FillBoundary() 1561 0.004442 0.004658 0.005044 0.00% MLCellLinOp::smooth() 688 0.003837 0.003917 0.003981 0.00% FabArrayBase::getFB() 1561 0.003146 0.003284 0.003627 0.00% FabArrayBase::FB::FB() 19 0.002795 0.003176 0.003581 0.00% FabArray::ParallelCopy() 830 0.003119 0.003211 0.003428 0.00% amrex::Dot() 115 0 0.0008266 0.003306 0.00% Gravity::test_composite_phi() 1 0.002064 0.002394 0.002627 0.00% main() 1 0.001616 0.00186 0.002018 0.00% AmrLevel::FillCoarsePatch() 5 0.0009083 0.001194 0.001967 0.00% FabArray::LinComb() 57 0 0.0004754 0.001902 0.00% MLLinOp::defineGrids() 11 0.001365 0.001574 0.001741 0.00% Amr::coarseTimeStep() 5 0.0005795 0.001332 0.001594 0.00% Amr::regrid() 5 0.001104 0.001281 0.001587 0.00% MLCellLinOp::apply() 326 0.0008896 0.001071 0.001582 0.00% Castro::clean_state() 87 0.001219 0.001326 0.001413 0.00% TagBoxArray::collate() 6 0.0005456 0.0009465 0.001299 0.00% MLCGSolver::ParallelAllReduce 153 0 0.0002704 0.001082 0.00% Gravity::average_fine_ec_onto_crse_ec() 8 0.0009277 0.0009429 0.0009628 0.00% Castro::finalize_advance() 9 0.000447 0.0006287 0.0009454 0.00% OwnerMask() 6 0.0007649 0.0008759 0.0009375 0.00% MLCellLinOp::correctionResidual() 196 0.0007436 0.0007716 0.0008381 0.00% Castro::do_advance_ctu() 5 0.0006219 0.0006574 0.0007014 0.00% FillPatchSingleLevel 92 0.0005464 0.0005737 0.0006117 0.00% MLMG::mgVcycle() 65 0.0005809 0.0005979 0.0006065 0.00% Amr::grid_places() 6 0.0004006 0.0004797 0.0006025 0.00% FillPatchIterator::FillFromLevel0() 31 0.0004083 0.0004537 0.000504 0.00% TagBoxArray::mapPRD 6 0.0004514 0.0004661 0.0004763 0.00% MLMG:computeResOfCorrection() 155 0.000382 0.0003969 0.0004088 0.00% MLCellLinOp::solutionResidual() 72 0.0003874 0.000397 0.0004022 0.00% MLMG::oneIter() 31 0.000354 0.0003647 0.0003749 0.00% Castro::subcycle_advance_ctu() 5 8.352e-05 0.0001631 0.0003688 0.00% MLMG::actualBottomSolve() 31 4.065e-05 0.0001235 0.0003673 0.00% FillPatchIterator::FillFromTwoLevels() 31 0.000304 0.0003232 0.0003632 0.00% MLMG::mgVcycle_down::0 31 0.0002881 0.0002972 0.0003093 0.00% Castro::computeNewDt() 8 0.0001282 0.0001738 0.0002991 0.00% Castro::post_regrid() 10 0.0001911 0.0002354 0.0002965 0.00% Gravity::solve_phi_with_mlmg() 11 0.0002396 0.0002618 0.0002822 0.00% Amr::defBaseLevel() 1 0.0002526 0.0002652 0.0002796 0.00% Amr::timeStep() 9 0.0002099 0.0002346 0.0002653 0.00% Castro::enforce_speed_limit() 87 0.0002448 0.0002512 0.0002633 0.00% MLMG::mgVcycle_down::1 31 0.0002367 0.0002484 0.0002563 0.00% MLMG::mgVcycle_bottom 65 0.000217 0.000224 0.0002324 0.00% FabArrayBase::TheFPinfo() 25 0.0001866 0.000199 0.0002321 0.00% AmrLevel::AmrLevel(dm) 5 0.0002088 0.0002188 0.0002295 0.00% makeSFC 33 0.0001878 0.0002003 0.000218 0.00% MLMG::mgVcycle_down::2 31 0.0001997 0.0002084 0.0002177 0.00% MLMG::mgVcycle_down::3 31 0.0001999 0.0002055 0.0002159 0.00% MLMG::mgVcycle_up::0 31 0.0001927 0.0002004 0.0002116 0.00% Castro::check_for_nan() 18 0.0001498 0.0001739 0.0001999 0.00% MLMG::mgVcycle_down::4 31 0.0001761 0.0001811 0.0001894 0.00% Castro::react_state() 18 0.0001355 0.0001575 0.0001817 0.00% Amr::writeSmallPlotFile() 1 9.402e-05 0.0001132 0.0001599 0.00% MLMG::mgVcycle_up::1 31 0.0001377 0.0001443 0.0001535 0.00% FPinfo::FPinfo() 3 0.0001256 0.0001329 0.0001466 0.00% MLMG::computeResWithCrseSolFineCor() 17 0.0001337 0.0001399 0.0001461 0.00% MLMG::mgVcycle_up::2 31 0.0001342 0.0001384 0.0001423 0.00% MLLinOp::makeAgglomeratedDMap 11 0.0001205 0.0001266 0.0001391 0.00% MLMG::mgVcycle_up::4 31 0.0001185 0.0001248 0.0001361 0.00% MLMG::mgVcycle_up::3 31 0.0001279 0.0001325 0.000136 0.00% DistributionMapping::Distribute() 35 0.0001197 0.0001259 0.0001333 0.00% Castro::expand_state() 13 0.0001014 0.000117 0.0001313 0.00% MLPoisson::define() 11 0.0001061 0.0001145 0.0001242 0.00% Castro::errorEst() 6 0.0001108 0.0001165 0.0001221 0.00% FabArray::RecvLayoutMask() 8 9.788e-05 0.0001065 0.0001177 0.00% Castro::apply_source_to_state() 18 0.0001053 0.0001112 0.0001167 0.00% Castro::advance() 5 8.855e-05 0.000101 0.0001148 0.00% ClusterList::intersect() 1 0 2.545e-05 0.0001018 0.00% Castro::post_timestep() 9 8.08e-05 8.674e-05 9.443e-05 0.00% MLMG::computeResidual() 31 8.637e-05 8.898e-05 9.284e-05 0.00% Gravity::swapTimeLevels() 9 8.276e-05 8.573e-05 8.86e-05 0.00% MLMG::computeMLResidual() 15 8.145e-05 8.345e-05 8.71e-05 0.00% Amr::FinalizeInit() 1 3.061e-05 5.62e-05 8.433e-05 0.00% Castro::avgDown(state_indx) 20 7.297e-05 7.835e-05 8.418e-05 0.00% MLMG::miniCycle() 34 7.066e-05 7.214e-05 7.389e-05 0.00% knapsack() 3 6.686e-05 7.095e-05 7.348e-05 0.00% MLLinOp::define() 11 5.965e-05 6.744e-05 7.223e-05 0.00% Gravity::set_mass_offset() 6 4.679e-05 5.87e-05 6.97e-05 0.00% MLMG::computeResWithCrseCorFineCor() 17 6.493e-05 6.594e-05 6.769e-05 0.00% MLMG::getGradSolution() 10 6.395e-05 6.546e-05 6.733e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 2 4.892e-05 5.569e-05 6.727e-05 0.00% Castro::create_source_corrector() 9 5.662e-05 6.148e-05 6.581e-05 0.00% amrex::average_down_w_geom 20 5.483e-05 5.583e-05 5.684e-05 0.00% MLPoisson::prepareForSolve() 11 4.656e-05 5.07e-05 5.612e-05 0.00% Castro::init(old) 3 4.504e-05 4.824e-05 5.318e-05 0.00% Amr::initSubcycle() 1 5.154e-05 5.168e-05 5.183e-05 0.00% Castro::construct_new_hybrid_source() 9 4.555e-05 4.891e-05 5.161e-05 0.00% Castro::construct_old_hybrid_source() 9 3.316e-05 4.233e-05 5.005e-05 0.00% BoxList::complementIn 12 2.64e-05 3.198e-05 4.533e-05 0.00% BoxList::parallelComplementIn() 12 2.728e-05 3.268e-05 4.392e-05 0.00% AmrMesh-cluster 1 0 1.04e-05 4.16e-05 0.00% Castro::derive() 6 2.023e-05 2.708e-05 4.07e-05 0.00% DistributionMapping::KnapSackDoIt() 3 3.136e-05 3.338e-05 3.584e-05 0.00% MLMG::MLRhsNormInf() 10 3.445e-05 3.532e-05 3.584e-05 0.00% Castro::post_init() 1 2.624e-05 2.924e-05 3.481e-05 0.00% Castro::avgDown() 4 2.841e-05 3.158e-05 3.361e-05 0.00% Castro::retry_advance_ctu() 5 2.336e-05 2.679e-05 3.229e-05 0.00% Castro::construct_new_rotation_source() 9 2.04e-05 2.477e-05 3.225e-05 0.00% MLMG::compResidual() 1 2.775e-05 2.933e-05 3.155e-05 0.00% Castro::construct_old_rotation_source() 9 2.131e-05 2.435e-05 3.041e-05 0.00% Castro::init() 1 1.728e-05 2.055e-05 2.429e-05 0.00% Amr::bldFineLevels() 1 2.153e-05 2.261e-05 2.356e-05 0.00% Castro::computeInitialDt() 2 1.904e-05 2.055e-05 2.248e-05 0.00% ClusterList::chop() 1 0 2.908e-06 1.163e-05 0.00% Amr::init() 1 6.452e-06 7.552e-06 1.014e-05 0.00% Amr::InitializeInit() 1 8.126e-06 8.582e-06 9.701e-06 0.00% Amr::initialInit() 1 4.714e-06 6.376e-06 8.143e-06 0.00% knapsack()swap 1 3.588e-06 3.881e-06 4.014e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 244.4 244.4 244.4 100.00% Amr::coarseTimeStep() 5 231.9 231.9 231.9 94.92% Amr::timeStep() 9 230.6 230.6 230.6 94.36% Castro::advance() 5 202.2 202.2 202.2 82.76% Castro::subcycle_advance_ctu() 5 193.4 193.8 194.9 79.74% Castro::do_advance_ctu() 5 193.4 193.8 194.9 79.74% Castro::construct_ctu_hydro_source() 9 116.9 117.1 117.2 47.94% Castro::clean_state() 87 71.81 82.14 86.23 35.29% Castro::computeTemp() 88 45.89 52.25 54.88 22.46% Castro::reset_internal_energy(MultiFab) 88 31.03 35.56 37.41 15.31% Castro::evaluate_source_change() 193 17.21 20.3 23.01 9.42% Castro::enforce_min_density() 87 16.86 19.82 21.06 8.62% Castro::post_regrid() 10 16.02 17.8 18.52 7.58% Amr::regrid() 5 15.93 17.54 18.19 7.44% Castro::initialize_do_advance() 9 14.94 15.32 15.46 6.33% MultiFab::AddProduct() 4205 10.31 12.67 15.07 6.17% AmrLevel::FillPatch() 62 7.928 10.03 14.95 6.12% amrex::Copy() 736 12.97 14.35 14.94 6.11% check_for_negative_density() 9 0.713 4.28 13.52 5.53% FillPatchIterator::Initialize 62 6.166 8.354 13.42 5.49% FabArray::ParallelCopy() 830 3.82 6.579 13.08 5.35% Gravity::solve_phi_with_mlmg() 11 11.3 11.62 12.39 5.07% Gravity::multilevel_solve_for_new_phi() 10 11.02 11.35 12.1 4.95% Gravity::actual_multilevel_solve() 10 10.98 11.3 12.06 4.93% Castro::do_old_sources() 9 11.69 11.75 11.82 4.84% Castro::do_new_sources() 9 9.662 10.95 11.47 4.69% Castro::normalize_species() 87 9.867 10.99 11.45 4.69% FillPatchSingleLevel 92 3.907 6.147 11.36 4.65% FabArray::ParallelCopy_finish() 830 1.68 4.219 10.83 4.43% FillPatchIterator::FillFromLevel0() 31 3.435 5.668 10.77 4.41% Amr::init() 1 10.64 10.64 10.64 4.36% Castro::post_timestep() 9 10.61 10.61 10.62 4.34% Amr::initialInit() 1 10.04 10.04 10.04 4.11% Castro::estTimeStep() 26 4.085 5.709 9.841 4.03% Castro::reset_internal_energy(Fab) 1476 8.528 9.317 9.62 3.94% Castro::reflux() 4 8.43 8.957 9.164 3.75% Castro::initialize_advance() 9 7.378 8.405 8.845 3.62% MultiFab::Subtract() 175 7.014 7.802 8.108 3.32% FabArray::setVal() 5078 6.984 7.758 8.105 3.32% Castro::construct_new_gravity() 9 6.446 6.751 7.492 3.07% Gravity::actual_solve_with_mlmg() 11 6.879 6.886 6.895 2.82% Castro::expand_state() 13 5.198 5.727 6.687 2.74% MLMG::solve() 10 6.372 6.382 6.397 2.62% Amr::FinalizeInit() 1 5.742 5.912 6.363 2.60% Gravity::fill_multipole_BCs() 11 4.395 4.717 5.478 2.24% MLMG::oneIter() 31 5.129 5.147 5.164 2.11% Castro::construct_old_gravity() 9 0.7071 1.801 4.564 1.87% Gravity::get_old_grav_vector() 9 0.6929 1.795 4.564 1.87% Castro::finalize_do_advance() 9 1.394 2.268 4.525 1.85% FluxRegister::Reflux() 24 3.819 4.306 4.507 1.84% MLMG::mgVcycle() 65 4.342 4.397 4.423 1.81% Amr::InitializeInit() 1 3.681 4.132 4.301 1.76% Amr::defBaseLevel() 1 3.681 4.132 4.301 1.76% Castro::problem_post_timestep() 10 2.874 3.314 4.281 1.75% Castro::computeNewDt() 8 2.206 2.793 4.216 1.73% FabArray::sum() 4205 3.227 3.568 3.719 1.52% MLCellLinOp::smooth() 688 3.278 3.349 3.421 1.40% Castro::post_init() 1 2.437 2.568 2.932 1.20% Castro::FluxRegCrseInit 9 1.815 2.139 2.859 1.17% Castro::initData() 1 2.317 2.661 2.826 1.16% FabArray::ParallelCopy_nowait() 830 2.137 2.357 2.731 1.12% Castro::wd_update() 6 1.193 1.634 2.602 1.06% Castro::construct_new_source() 63 2.07 2.317 2.468 1.01% MLPoisson::Fsmooth() 1376 2.099 2.341 2.429 0.99% MLMG::mgVcycle_down::0 31 2.005 2.046 2.1 0.86% MLCellLinOp::applyBC() 1750 1.217 1.397 1.83 0.75% FillPatchIterator::FillFromTwoLevels() 31 1.502 1.581 1.649 0.67% FillPatchTwoLevels 31 1.502 1.58 1.649 0.67% FabArray::Saxpy() 125 1.39 1.536 1.594 0.65% Amr::bldFineLevels() 1 1.334 1.516 1.578 0.65% FabArray::FillBoundary() 1561 0.7962 0.9793 1.476 0.60% Castro::Castro() 5 1.328 1.428 1.472 0.60% MLMG::mgVcycle_up::0 31 1.382 1.405 1.43 0.59% FillPatchInterp(MF) 30 1.235 1.325 1.425 0.58% MLCellLinOp::apply() 326 1.292 1.344 1.399 0.57% Castro::construct_old_source() 63 1.065 1.171 1.237 0.51% Castro::construct_old_gravity_source() 18 0.9279 1.072 1.197 0.49% amrex::Add() 143 1.013 1.118 1.187 0.49% Amr::InitAmr() 1 1.184 1.184 1.184 0.48% Amr::writePlotFile() 2 1.179 1.18 1.18 0.48% Castro::initMFs() 5 1.091 1.148 1.178 0.48% Castro::computeInitialDt() 2 0.4883 0.6576 1.108 0.45% MLPoisson::Fapply() 326 0.8877 1.022 1.092 0.45% MLCellLinOp::solutionResidual() 72 1.001 1.055 1.086 0.44% FillBoundary_finish() 1561 0.3987 0.604 1.06 0.43% Castro::construct_new_gravity_source() 9 0.7701 0.8804 0.9998 0.41% MLCellLinOp::correctionResidual() 196 0.8178 0.8297 0.8397 0.34% Gravity::get_new_grav_vector() 10 0.7788 0.7918 0.8106 0.33% VisMF::Write(FabArray) 3 0.6426 0.7297 0.805 0.33% Castro::volInBoundary() 42 0.5831 0.6881 0.7315 0.30% MLMG:computeResOfCorrection() 155 0.6498 0.6942 0.7181 0.29% Castro::apply_source_to_state() 18 0.5544 0.6131 0.6348 0.26% StateDataPhysBCFunct::() 95 0.6087 0.6245 0.6338 0.26% StateData::FillBoundary(geom) 50 0.6066 0.6166 0.6259 0.26% Gravity::update_max_rhs() 10 0.09044 0.2232 0.5911 0.24% MLMG::computeResWithCrseSolFineCor() 17 0.4739 0.5057 0.5881 0.24% MultiFab::max() 18 0.02913 0.1632 0.5384 0.22% Castro::init() 1 0.4985 0.5133 0.5194 0.21% AmrLevel::FillCoarsePatch() 5 0.4985 0.5133 0.5194 0.21% MLMG::computeMLResidual() 15 0.4408 0.4671 0.516 0.21% Castro::check_for_nan() 18 0.4044 0.4516 0.4688 0.19% MultiFab::contains_nan() 18 0.4042 0.4514 0.4686 0.19% FillBoundary_nowait() 1561 0.3341 0.3706 0.4111 0.17% FabArray::Xpay() 291 0.352 0.3922 0.4076 0.17% Gravity::test_composite_phi() 1 0.4039 0.4046 0.4061 0.17% MLMG::mgVcycle_down::1 31 0.3012 0.3387 0.4025 0.16% Castro::init(old) 3 0.366 0.3713 0.3806 0.16% MLMG::computeResidual() 31 0.347 0.3589 0.3674 0.15% MLCellLinOp::reflux() 30 0.1766 0.2303 0.3579 0.15% MLPoisson::define() 11 0.2916 0.3 0.3124 0.13% amrex::average_down 233 0.2652 0.2781 0.2923 0.12% Castro::react_state() 18 0.2227 0.2476 0.2627 0.11% FabArray::setDomainBndry() 92 0.1999 0.2216 0.2351 0.10% MLMG::prepareForSolve() 10 0.2146 0.2217 0.2267 0.09% MLMG::mgVcycle_up::1 31 0.2064 0.2116 0.2137 0.09% MLCellLinOp::defineAuxData() 11 0.1843 0.1993 0.2088 0.09% Amr::grid_places() 6 0.2053 0.2055 0.2055 0.08% AmrMesh::MakeNewGrids() 6 0.2049 0.205 0.2051 0.08% Castro::fill_hybrid_hydro_source() 27 0.1724 0.1904 0.1985 0.08% MLMG::addInterpCorrection() 155 0.128 0.1663 0.1858 0.08% MLMG::mgVcycle_bottom 65 0.1498 0.1582 0.1821 0.07% MLMG::getGradSolution() 10 0.1679 0.1706 0.1716 0.07% MLCellLinOp::compGrad() 18 0.1678 0.1705 0.1716 0.07% Castro::avgDown() 4 0.1258 0.1375 0.163 0.07% Castro::avgDown(state_indx) 20 0.1258 0.1374 0.163 0.07% amrex::average_down_w_geom 20 0.1257 0.1374 0.1629 0.07% MLPoisson::FFlux() 150 0.004617 0.04204 0.154 0.06% MLMG::miniCycle() 34 0.1496 0.1501 0.1507 0.06% AmrLevel::AmrLevel(dm) 5 0.1217 0.1357 0.1481 0.06% StateData::define() 25 0.1214 0.1355 0.1479 0.06% Castro::construct_new_hybrid_source() 9 0.116 0.1274 0.1323 0.05% Castro::errorEst() 6 0.1061 0.1214 0.1299 0.05% FabArray::mult() 56 0.1119 0.1249 0.1297 0.05% Castro::swap_state_time_levels() 9 0.09355 0.112 0.1243 0.05% MLCellLinOp::updateSolBC() 25 0.07692 0.09882 0.1239 0.05% MLMG::ResNormInf() 54 0.07019 0.09218 0.1182 0.05% MLCellLinOp::defineBC() 11 0.08774 0.09031 0.09168 0.04% MLMG::computeResWithCrseCorFineCor() 17 0.08707 0.08813 0.08942 0.04% MLMG::mgVcycle_up::3 31 0.02483 0.06768 0.08212 0.03% MLMG::mgVcycle_down::2 31 0.06734 0.07156 0.08068 0.03% MLMG::MLResNormInf() 15 0.04267 0.06519 0.07332 0.03% Castro::construct_old_hybrid_source() 9 0.05655 0.06317 0.0676 0.03% BndryData::define() 27 0.06036 0.06268 0.06398 0.03% FabArray::norminf(mask) 21 0.05354 0.05969 0.06239 0.03% MLMG::interpCorrection_1 17 0.0257 0.04175 0.06191 0.03% Castro::apply_tagging_restrictions() 6 0.04768 0.05503 0.0608 0.02% Castro::derive() 6 0.04339 0.04945 0.05274 0.02% AmrLevel::derive() 6 0.04337 0.04943 0.0527 0.02% MLMG::mgVcycle_up::2 31 0.05064 0.05132 0.05217 0.02% MLCellLinOp::updateCorBC() 17 0.05022 0.05058 0.05098 0.02% Castro::buildMetrics() 5 0.03655 0.04281 0.04714 0.02% FabArray::norminf() 117 0.0378 0.03919 0.04067 0.02% TagBoxArray::mapPRD 6 0.01209 0.0225 0.03985 0.02% Gravity::swapTimeLevels() 9 0.03213 0.03548 0.03768 0.02% MLMG::mgVcycle_down::3 31 0.03069 0.03163 0.03392 0.01% MLMG::actualBottomSolve() 31 4.065e-05 0.008196 0.03266 0.01% MLCGSolver::bicgstab 7 0 0.007993 0.03197 0.01% Castro::FluxRegFineAdd() 9 0.02985 0.03055 0.03151 0.01% FabArrayBase::getCPC() 1440 0.02527 0.02731 0.03077 0.01% MLMG::MLRhsNormInf() 10 0.02453 0.02728 0.02851 0.01% FabArray::setVal(val, thecmd, scomp, ncomp) 622 0.01874 0.02277 0.02843 0.01% MultiFab::Multiply() 8 0.02253 0.02582 0.02738 0.01% Castro::build_fine_mask() 44 0.02015 0.02143 0.02338 0.01% Gravity::average_fine_ec_onto_crse_ec() 8 0.01466 0.01678 0.02145 0.01% MLMG::compResidual() 1 0.01758 0.01972 0.02122 0.01% FabArrayBase::CPC::define() 416 0.01447 0.01661 0.02014 0.01% MLLinOp::define() 11 0.002424 0.01025 0.01834 0.01% MLMG::mgVcycle_down::4 31 0.0053 0.0086 0.01828 0.01% MLLinOp::defineGrids() 11 0.002352 0.01018 0.01827 0.01% Castro::linear_to_hybrid_momentum() 1 0.01385 0.01603 0.01747 0.01% MLLinOp::makeSubCommunicator() 11 0.0004437 0.008171 0.0161 0.01% FabArrayBase::TheFPinfo() 25 0.001567 0.00887 0.016 0.01% FPinfo::FPinfo() 3 0.001379 0.008671 0.01576 0.01% DistributionMapping::LeastUsedCPUs() 5 0.001201 0.008506 0.01562 0.01% DistributionMapping::KnapSackDoIt() 3 0.001253 0.008538 0.01562 0.01% MLMG::mgVcycle_up::4 31 0.004402 0.006803 0.01388 0.01% MLPoisson::prepareForSolve() 11 0.01162 0.01173 0.0118 0.00% MLCellLinOp::prepareForSolve() 11 0.01157 0.01168 0.01175 0.00% Castro::enforce_consistent_e() 1 0.009165 0.01062 0.01121 0.00% AMRErrorTag::operator() 6 0.008587 0.009762 0.01036 0.00% average_down_faces 48 0.009055 0.009676 0.0103 0.00% Castro::apply_problem_tags() 6 0.006339 0.007086 0.007352 0.00% FabArrayBase::getFB() 1561 0.005985 0.00646 0.006896 0.00% MLCellLinOp::setLevelBC() 19 0.003877 0.004592 0.005336 0.00% FabArrayBase::FB::FB() 19 0.002795 0.003176 0.003581 0.00% amrex::Dot() 115 0 0.0008266 0.003306 0.00% MLCGSolver::ParallelAllReduce 153 0 0.0007103 0.002841 0.00% FabArray::LinComb() 57 0 0.0004754 0.001902 0.00% TagBoxArray::collate() 6 0.0005456 0.0009465 0.001299 0.00% Castro::finalize_advance() 9 0.000447 0.0006287 0.0009454 0.00% OwnerMask() 6 0.0007649 0.0008759 0.0009375 0.00% MLLinOp::makeAgglomeratedDMap 11 0.0004166 0.00044 0.0004758 0.00% makeSFC 33 0.0002961 0.0003134 0.0003367 0.00% Castro::enforce_speed_limit() 87 0.0002448 0.0002512 0.0002633 0.00% DistributionMapping::SFCProcessorMapDoIt() 2 0.0001285 0.0001442 0.0001844 0.00% Amr::writeSmallPlotFile() 1 9.402e-05 0.0001132 0.0001599 0.00% AmrMesh-cluster 1 0 3.876e-05 0.000155 0.00% DistributionMapping::Distribute() 35 0.0001197 0.0001259 0.0001333 0.00% FabArray::RecvLayoutMask() 8 9.788e-05 0.0001065 0.0001177 0.00% ClusterList::intersect() 1 0 2.545e-05 0.0001018 0.00% knapsack() 3 7.045e-05 7.483e-05 7.748e-05 0.00% BoxList::parallelComplementIn() 12 5.488e-05 6.466e-05 7.479e-05 0.00% Gravity::set_mass_offset() 6 4.679e-05 5.87e-05 6.97e-05 0.00% Castro::create_source_corrector() 9 5.662e-05 6.148e-05 6.581e-05 0.00% Amr::initSubcycle() 1 5.154e-05 5.168e-05 5.183e-05 0.00% BoxList::complementIn 12 2.64e-05 3.198e-05 4.533e-05 0.00% Castro::retry_advance_ctu() 5 2.336e-05 2.679e-05 3.229e-05 0.00% Castro::construct_new_rotation_source() 9 2.04e-05 2.477e-05 3.225e-05 0.00% Castro::construct_old_rotation_source() 9 2.131e-05 2.435e-05 3.041e-05 0.00% ClusterList::chop() 1 0 2.908e-06 1.163e-05 0.00% knapsack()swap 1 3.588e-06 3.881e-06 4.014e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::mg.v(nvals = 1) :: [2] Pinned Memory Usage: --------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max --------------------------------------------------------------------------------------------------------------------- The_Pinned_Arena::Initialize() 4 4 0 B 0 B 0 B 8192 KiB 8192 KiB 8192 KiB --------------------------------------------------------------------------------------------------------------------- AMReX (23.07-44-g49dd703558c5) finalized