Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-14-gd70f70e2f56e) initialized Starting run at 08:25:22 UTC on 2022-04-11. Successfully read inputs file ... Castro git describe: 22.04-2-gb74783602 AMReX git describe: 22.04-14-gd70f70e2f Microphysics git describe: 22.04-1-gef526ccd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.038196521 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022797339 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.110427335 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052262013 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.0566415 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.059410934 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.075796839 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.035380013 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.050074847 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050976517 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.056635649 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057298204 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063033804 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.035270494 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.022041184 seconds Ending run at 08:25:23 UTC on 2022-04-11. Run time = 0.848382255 Run time without initialization = 0.725743396 Average number of zones advanced per microsecond: 3.612 Average number of zones advanced per microsecond per rank: 3.612 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8484 ... 0.8484 ... 0.8484 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2290 0.2290 0.2290 26.99% VisMF::Write(FabArray) 11 0.1463 0.1463 0.1463 17.25% MLCellLinOp::applyBC() 4433 0.0942 0.0942 0.0942 11.10% MLPoisson::Fsmooth() 3280 0.06014 0.06014 0.06014 7.09% FabArray::setVal() 1144 0.02601 0.02601 0.02601 3.07% StateData::FillBoundary(geom) 328 0.02275 0.02275 0.02275 2.68% MLCGSolver::bicgstab 82 0.02163 0.02163 0.02163 2.55% MultiFab::Dot() 1114 0.02128 0.02128 0.02128 2.51% FillBoundary_nowait() 4023 0.01823 0.01823 0.01823 2.15% StateDataPhysBCFunct::() 41 0.01691 0.01691 0.01691 1.99% FabArray::ParallelCopy_nowait() 861 0.0161 0.0161 0.0161 1.90% Castro::computeTemp() 63 0.01555 0.01555 0.01555 1.83% MultiFab::LinComb() 1586 0.01218 0.01218 0.01218 1.44% Gravity::fill_multipole_BCs() 11 0.01194 0.01194 0.01194 1.41% Castro::enforce_min_density() 62 0.01112 0.01112 0.01112 1.31% MLPoisson::Fapply() 1142 0.01051 0.01051 0.01051 1.24% MLCellLinOp::defineAuxData() 11 0.01037 0.01037 0.01037 1.22% MLMG::addInterpCorrection() 410 0.007034 0.007034 0.007034 0.83% amrex::average_down 410 0.006577 0.006577 0.006577 0.78% MultiFab::Xpay() 585 0.006036 0.006036 0.006036 0.71% Castro::reset_internal_energy() 63 0.005744 0.005744 0.005744 0.68% FabArray::setDomainBndry() 41 0.005603 0.005603 0.005603 0.66% Castro::estTimeStep() 21 0.004947 0.004947 0.004947 0.58% Castro::expand_state() 10 0.004783 0.004783 0.004783 0.56% Castro::normalize_species() 62 0.004482 0.004482 0.004482 0.53% Amr::checkPoint() 3 0.004174 0.004174 0.004174 0.49% BndryData::define() 11 0.003671 0.003671 0.003671 0.43% Castro::enforce_speed_limit() 62 0.003548 0.003548 0.003548 0.42% Castro::do_advance_ctu() 10 0.003193 0.003193 0.003193 0.38% Amr::writePlotFile() 2 0.003188 0.003188 0.003188 0.38% Castro::construct_new_gravity_source() 10 0.003101 0.003101 0.003101 0.37% Gravity::get_new_grav_vector() 11 0.002686 0.002686 0.002686 0.32% Castro::construct_old_gravity_source() 10 0.002565 0.002565 0.002565 0.30% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002252 0.002252 0.002252 0.27% MLMG::ResNormInf() 93 0.001931 0.001931 0.001931 0.23% MLCellLinOp::compGrad() 11 0.00191 0.00191 0.00191 0.23% MultiFab::Saxpy() 20 0.001806 0.001806 0.001806 0.21% Gravity::get_old_grav_vector() 10 0.001746 0.001746 0.001746 0.21% MLMG::oneIter() 82 0.001691 0.001691 0.001691 0.20% MLCellLinOp::setLevelBC() 11 0.001547 0.001547 0.001547 0.18% Gravity::actual_solve_with_mlmg() 11 0.001379 0.001379 0.001379 0.16% MLCellLinOp::prepareForSolve() 11 0.001352 0.001352 0.001352 0.16% FabArray::mult() 43 0.001319 0.001319 0.001319 0.16% Castro::initData() 1 0.001204 0.001204 0.001204 0.14% MLCellLinOp::smooth() 1640 0.001133 0.001133 0.001133 0.13% FabArrayBase::getCPC() 1323 0.0008716 0.0008716 0.0008716 0.10% FabArray::FillBoundary() 4023 0.0008666 0.0008666 0.0008666 0.10% MLMG::prepareForSolve() 11 0.0007847 0.0007847 0.0007847 0.09% FabArrayBase::CPC::define() 454 0.0007222 0.0007222 0.0007222 0.09% FabArrayBase::getFB() 4023 0.00069 0.00069 0.00069 0.08% Gravity::update_max_rhs() 11 0.0005608 0.0005608 0.0005608 0.07% MultiFab::Copy() 11 0.0005051 0.0005051 0.0005051 0.06% MLCellLinOp::apply() 1142 0.0004907 0.0004907 0.0004907 0.06% Gravity::solve_for_phi() 10 0.0004724 0.0004724 0.0004724 0.06% CGSolver::sxay() 1586 0.000463 0.000463 0.000463 0.05% Amr::InitAmr() 1 0.0004356 0.0004356 0.0004356 0.05% MLMG::mgVcycle() 82 0.0004195 0.0004195 0.0004195 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003426 0.0003426 0.0003426 0.04% MultiFab::min() 10 0.0003258 0.0003258 0.0003258 0.04% FabArray::ParallelCopy() 861 0.0002954 0.0002954 0.0002954 0.03% main() 1 0.0002912 0.0002912 0.0002912 0.03% FillPatchIterator::Initialize 41 0.0002618 0.0002618 0.0002618 0.03% MultiFab::max() 11 0.0002541 0.0002541 0.0002541 0.03% Castro::construct_new_gravity() 10 0.0002384 0.0002384 0.0002384 0.03% Gravity::actual_multilevel_solve() 1 0.0002362 0.0002362 0.0002362 0.03% MLCellLinOp::correctionResidual() 492 0.0002197 0.0002197 0.0002197 0.03% Amr::coarseTimeStep() 10 0.0002132 0.0002132 0.0002132 0.03% Amr::timeStep() 10 0.0002043 0.0002043 0.0002043 0.02% MLMG::MLRhsNormInf() 11 0.0001973 0.0001973 0.0001973 0.02% MLCellLinOp::defineBC() 11 0.0001866 0.0001866 0.0001866 0.02% MLLinOp::defineGrids() 11 0.0001761 0.0001761 0.0001761 0.02% Amr::defBaseLevel() 1 0.0001503 0.0001503 0.0001503 0.02% Castro::subcycle_advance_ctu() 10 0.000141 0.000141 0.000141 0.02% MLMG:computeResOfCorrection() 410 0.0001334 0.0001334 0.0001334 0.02% StateData::checkPoint() 12 0.0001305 0.0001305 0.0001305 0.02% MLMG::actualBottomSolve() 82 0.0001077 0.0001077 0.0001077 0.01% MLMG::mgVcycle_down::0 82 9.452e-05 9.452e-05 9.452e-05 0.01% FabArrayBase::FB::FB() 56 9.19e-05 9.19e-05 9.19e-05 0.01% Castro::construct_new_source() 50 8.269e-05 8.269e-05 8.269e-05 0.01% MLMG::mgVcycle_down::1 82 8.224e-05 8.224e-05 8.224e-05 0.01% Castro::initialize_advance() 10 8.026e-05 8.026e-05 8.026e-05 0.01% MLMG::solve() 11 7.892e-05 7.892e-05 7.892e-05 0.01% MLMG::mgVcycle_down::3 82 7.827e-05 7.827e-05 7.827e-05 0.01% MLMG::mgVcycle_down::2 82 7.816e-05 7.816e-05 7.816e-05 0.01% Castro::advance() 10 7.731e-05 7.731e-05 7.731e-05 0.01% MLMG::mgVcycle_down::4 82 7.653e-05 7.653e-05 7.653e-05 0.01% Castro::post_timestep() 10 7.509e-05 7.509e-05 7.509e-05 0.01% Castro::clean_state() 62 7.449e-05 7.449e-05 7.449e-05 0.01% AmrLevel::checkPoint() 3 7.2e-05 7.2e-05 7.2e-05 0.01% MLMG::mgVcycle_up::4 82 6.024e-05 6.024e-05 6.024e-05 0.01% Castro::finalize_advance() 10 5.953e-05 5.953e-05 5.953e-05 0.01% Castro::initialize_do_advance() 10 5.657e-05 5.657e-05 5.657e-05 0.01% MLMG::mgVcycle_up::0 82 5.306e-05 5.306e-05 5.306e-05 0.01% MLMG::mgVcycle_up::1 82 5.286e-05 5.286e-05 5.286e-05 0.01% MLMG::mgVcycle_up::3 82 5.106e-05 5.106e-05 5.106e-05 0.01% MLMG::mgVcycle_up::2 82 5.049e-05 5.049e-05 5.049e-05 0.01% MLCellLinOp::solutionResidual() 93 4.889e-05 4.889e-05 4.889e-05 0.01% StateData::define() 4 4.134e-05 4.134e-05 4.134e-05 0.00% Castro::swap_state_time_levels() 10 3.887e-05 3.887e-05 3.887e-05 0.00% Castro::finalize_do_advance() 10 3.871e-05 3.871e-05 3.871e-05 0.00% Castro::enforce_consistent_e() 1 3.503e-05 3.503e-05 3.503e-05 0.00% MLMG::computeResidual() 82 3.469e-05 3.469e-05 3.469e-05 0.00% MLMG::mgVcycle_bottom 82 3.396e-05 3.396e-05 3.396e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.849e-05 2.849e-05 2.849e-05 0.00% MLLinOp::define() 11 2.818e-05 2.818e-05 2.818e-05 0.00% FillPatchSingleLevel 41 2.778e-05 2.778e-05 2.778e-05 0.00% makeSFC 55 2.652e-05 2.652e-05 2.652e-05 0.00% Amr::writeSmallPlotFile() 1 2.553e-05 2.553e-05 2.553e-05 0.00% MLPoisson::define() 11 2.516e-05 2.516e-05 2.516e-05 0.00% Amr::FinalizeInit() 1 2.2e-05 2.2e-05 2.2e-05 0.00% Castro::construct_old_source() 50 2.168e-05 2.168e-05 2.168e-05 0.00% DistributionMapping::Distribute() 56 1.676e-05 1.676e-05 1.676e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.664e-05 1.664e-05 1.664e-05 0.00% Castro::do_new_sources() 10 1.633e-05 1.633e-05 1.633e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.609e-05 1.609e-05 1.609e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.533e-05 1.533e-05 1.533e-05 0.00% Castro::do_old_sources() 10 1.503e-05 1.503e-05 1.503e-05 0.00% Castro::construct_old_gravity() 10 1.369e-05 1.369e-05 1.369e-05 0.00% Castro::apply_source_to_state() 20 1.179e-05 1.179e-05 1.179e-05 0.00% Amr::initSubcycle() 1 1.008e-05 1.008e-05 1.008e-05 0.00% Gravity::swapTimeLevels() 10 9.883e-06 9.883e-06 9.883e-06 0.00% MLPoisson::prepareForSolve() 11 9.136e-06 9.136e-06 9.136e-06 0.00% MLMG::computeMLResidual() 11 6.917e-06 6.917e-06 6.917e-06 0.00% AmrLevel::checkPointPost() 3 6.579e-06 6.579e-06 6.579e-06 0.00% Amr::InitializeInit() 1 6.203e-06 6.203e-06 6.203e-06 0.00% Castro::computeNewDt() 9 6.081e-06 6.081e-06 6.081e-06 0.00% MLMG::getGradSolution() 11 6.053e-06 6.053e-06 6.053e-06 0.00% MLMG::buildFineMask() 11 5.855e-06 5.855e-06 5.855e-06 0.00% Castro::create_source_corrector() 10 5.009e-06 5.009e-06 5.009e-06 0.00% Gravity::set_mass_offset() 11 4.965e-06 4.965e-06 4.965e-06 0.00% MLMG::MLResNormInf() 11 4.527e-06 4.527e-06 4.527e-06 0.00% Castro::post_init() 1 3.953e-06 3.953e-06 3.953e-06 0.00% Castro::retry_advance_ctu() 10 3.624e-06 3.624e-06 3.624e-06 0.00% Castro::FluxRegFineAdd() 10 3.166e-06 3.166e-06 3.166e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.628e-06 2.628e-06 2.628e-06 0.00% Castro::computeInitialDt() 2 2.605e-06 2.605e-06 2.605e-06 0.00% Amr::init() 1 2.339e-06 2.339e-06 2.339e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.307e-06 2.307e-06 2.307e-06 0.00% AmrLevel::checkPointPre() 3 2.059e-06 2.059e-06 2.059e-06 0.00% Castro::post_regrid() 1 1.455e-06 1.455e-06 1.455e-06 0.00% Amr::initialInit() 1 8.87e-07 8.87e-07 8.87e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8484 0.8484 0.8484 100.00% Amr::coarseTimeStep() 10 0.7035 0.7035 0.7035 82.92% Amr::timeStep() 10 0.63 0.63 0.63 74.26% Castro::advance() 10 0.623 0.623 0.623 73.43% Castro::subcycle_advance_ctu() 10 0.6082 0.6082 0.6082 71.69% Castro::do_advance_ctu() 10 0.6081 0.6081 0.6081 71.67% Gravity::solve_phi_with_mlmg() 11 0.3268 0.3268 0.3268 38.52% Gravity::actual_solve_with_mlmg() 11 0.3147 0.3147 0.3147 37.09% Castro::construct_new_gravity() 10 0.2971 0.2971 0.2971 35.02% MLMG::solve() 11 0.2918 0.2918 0.2918 34.39% Gravity::solve_for_phi() 10 0.2791 0.2791 0.2791 32.89% MLMG::oneIter() 82 0.2758 0.2758 0.2758 32.50% MLMG::mgVcycle() 82 0.2741 0.2741 0.2741 32.30% Castro::construct_ctu_hydro_source() 10 0.229 0.229 0.229 26.99% VisMF::Write(FabArray) 11 0.1463 0.1463 0.1463 17.25% MLCellLinOp::smooth() 1640 0.146 0.146 0.146 17.20% Amr::init() 1 0.1221 0.1221 0.1221 14.39% MLCellLinOp::applyBC() 4433 0.1141 0.1141 0.1141 13.45% Amr::checkPoint() 3 0.109 0.109 0.109 12.84% AmrLevel::checkPoint() 3 0.1048 0.1048 0.1048 12.35% StateData::checkPoint() 12 0.1047 0.1047 0.1047 12.34% MLMG::mgVcycle_bottom 82 0.07734 0.07734 0.07734 9.12% MLMG::actualBottomSolve() 82 0.0773 0.0773 0.0773 9.11% MLCGSolver::bicgstab 82 0.07657 0.07657 0.07657 9.02% Amr::initialInit() 1 0.06096 0.06096 0.06096 7.18% MLPoisson::Fsmooth() 3280 0.06014 0.06014 0.06014 7.09% Amr::FinalizeInit() 1 0.05176 0.05176 0.05176 6.10% Castro::post_init() 1 0.05103 0.05103 0.05103 6.01% FillPatchIterator::Initialize 41 0.0498 0.0498 0.0498 5.87% Gravity::multilevel_solve_for_new_phi() 1 0.04849 0.04849 0.04849 5.72% Gravity::actual_multilevel_solve() 1 0.04848 0.04848 0.04848 5.71% Amr::writePlotFile() 2 0.04495 0.04495 0.04495 5.30% FillPatchSingleLevel 41 0.04394 0.04394 0.04394 5.18% MLCellLinOp::apply() 1142 0.03983 0.03983 0.03983 4.69% Castro::clean_state() 62 0.03977 0.03977 0.03977 4.69% StateDataPhysBCFunct::() 41 0.03966 0.03966 0.03966 4.67% MLMG::mgVcycle_down::0 82 0.0389 0.0389 0.0389 4.59% MLMG::mgVcycle_up::0 82 0.03331 0.03331 0.03331 3.93% FabArray::setVal() 1144 0.02601 0.02601 0.02601 3.07% Castro::initialize_do_advance() 10 0.02491 0.02491 0.02491 2.94% StateData::FillBoundary(geom) 328 0.02275 0.02275 0.02275 2.68% MLCellLinOp::correctionResidual() 492 0.02231 0.02231 0.02231 2.63% Castro::computeTemp() 63 0.02129 0.02129 0.02129 2.51% MultiFab::Dot() 1114 0.02128 0.02128 0.02128 2.51% Gravity::get_new_grav_vector() 11 0.02018 0.02018 0.02018 2.38% FabArray::FillBoundary() 4023 0.01987 0.01987 0.01987 2.34% MLMG:computeResOfCorrection() 410 0.01934 0.01934 0.01934 2.28% FillBoundary_nowait() 4023 0.01901 0.01901 0.01901 2.24% Castro::expand_state() 10 0.01873 0.01873 0.01873 2.21% MLMG::mgVcycle_down::1 82 0.0187 0.0187 0.0187 2.20% MLMG::mgVcycle_down::2 82 0.01794 0.01794 0.01794 2.11% MLPoisson::define() 11 0.01746 0.01746 0.01746 2.06% FabArray::ParallelCopy() 861 0.0173 0.0173 0.0173 2.04% MLMG::mgVcycle_down::3 82 0.01709 0.01709 0.01709 2.01% FabArray::ParallelCopy_nowait() 861 0.017 0.017 0.017 2.00% MLMG::mgVcycle_down::4 82 0.01629 0.01629 0.01629 1.92% Castro::construct_old_gravity() 10 0.01591 0.01591 0.01591 1.88% Gravity::get_old_grav_vector() 10 0.0159 0.0159 0.0159 1.87% Castro::initialize_advance() 10 0.01459 0.01459 0.01459 1.72% MLMG::mgVcycle_up::2 82 0.01395 0.01395 0.01395 1.64% MLMG::mgVcycle_up::1 82 0.01374 0.01374 0.01374 1.62% MLMG::addInterpCorrection() 410 0.01356 0.01356 0.01356 1.60% MLMG::mgVcycle_up::3 82 0.01328 0.01328 0.01328 1.57% Castro::do_new_sources() 10 0.01315 0.01315 0.01315 1.55% MLMG::mgVcycle_up::4 82 0.01312 0.01312 0.01312 1.55% amrex::average_down 410 0.0131 0.0131 0.0131 1.54% MLCGSolver::ParallelAllReduce 1514 0.01284 0.01284 0.01284 1.51% CGSolver::sxay() 1586 0.01264 0.01264 0.01264 1.49% MultiFab::LinComb() 1586 0.01218 0.01218 0.01218 1.44% MLCellLinOp::defineAuxData() 11 0.01205 0.01205 0.01205 1.42% Gravity::fill_multipole_BCs() 11 0.01194 0.01194 0.01194 1.41% Castro::enforce_min_density() 62 0.01112 0.01112 0.01112 1.31% MLPoisson::Fapply() 1142 0.01051 0.01051 0.01051 1.24% Amr::InitializeInit() 1 0.009192 0.009192 0.009192 1.08% Amr::defBaseLevel() 1 0.009186 0.009186 0.009186 1.08% Castro::do_old_sources() 10 0.008672 0.008672 0.008672 1.02% MLCellLinOp::solutionResidual() 93 0.007951 0.007951 0.007951 0.94% MLMG::computeResidual() 82 0.00688 0.00688 0.00688 0.81% Castro::post_timestep() 10 0.00685 0.00685 0.00685 0.81% MultiFab::Xpay() 585 0.006036 0.006036 0.006036 0.71% MLMG::prepareForSolve() 11 0.005815 0.005815 0.005815 0.69% Castro::reset_internal_energy() 63 0.005744 0.005744 0.005744 0.68% FabArray::setDomainBndry() 41 0.005603 0.005603 0.005603 0.66% MLCellLinOp::defineBC() 11 0.00512 0.00512 0.00512 0.60% Castro::estTimeStep() 21 0.004947 0.004947 0.004947 0.58% BndryData::define() 11 0.004933 0.004933 0.004933 0.58% Castro::normalize_species() 62 0.004482 0.004482 0.004482 0.53% Castro::enforce_speed_limit() 62 0.003548 0.003548 0.003548 0.42% Castro::initData() 1 0.003343 0.003343 0.003343 0.39% Castro::construct_new_source() 50 0.003184 0.003184 0.003184 0.38% Castro::construct_new_gravity_source() 10 0.003101 0.003101 0.003101 0.37% Castro::construct_old_source() 50 0.002586 0.002586 0.002586 0.30% Castro::construct_old_gravity_source() 10 0.002565 0.002565 0.002565 0.30% MLMG::getGradSolution() 11 0.00248 0.00248 0.00248 0.29% MLCellLinOp::compGrad() 11 0.002474 0.002474 0.002474 0.29% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002252 0.002252 0.002252 0.27% Castro::computeNewDt() 9 0.002065 0.002065 0.002065 0.24% MLMG::ResNormInf() 93 0.001931 0.001931 0.001931 0.23% Castro::apply_source_to_state() 20 0.001818 0.001818 0.001818 0.21% MultiFab::Saxpy() 20 0.001806 0.001806 0.001806 0.21% FabArrayBase::getCPC() 1323 0.001594 0.001594 0.001594 0.19% MLCellLinOp::setLevelBC() 11 0.001547 0.001547 0.001547 0.18% MLPoisson::prepareForSolve() 11 0.001361 0.001361 0.001361 0.16% MLCellLinOp::prepareForSolve() 11 0.001352 0.001352 0.001352 0.16% FabArray::mult() 43 0.001319 0.001319 0.001319 0.16% MLMG::computeMLResidual() 11 0.001113 0.001113 0.001113 0.13% Gravity::swapTimeLevels() 10 0.001026 0.001026 0.001026 0.12% Gravity::update_max_rhs() 11 0.0009579 0.0009579 0.0009579 0.11% FabArrayBase::getFB() 4023 0.0007819 0.0007819 0.0007819 0.09% FabArrayBase::CPC::define() 454 0.0007222 0.0007222 0.0007222 0.09% Castro::computeInitialDt() 2 0.0006742 0.0006742 0.0006742 0.08% MultiFab::Copy() 11 0.0005051 0.0005051 0.0005051 0.06% Castro::post_regrid() 1 0.0004933 0.0004933 0.0004933 0.06% Amr::InitAmr() 1 0.0004457 0.0004457 0.0004457 0.05% MultiFab::min() 10 0.0003258 0.0003258 0.0003258 0.04% MLLinOp::define() 11 0.0002646 0.0002646 0.0002646 0.03% MultiFab::max() 11 0.0002541 0.0002541 0.0002541 0.03% MLMG::MLResNormInf() 11 0.0002509 0.0002509 0.0002509 0.03% MLLinOp::defineGrids() 11 0.0002365 0.0002365 0.0002365 0.03% MLMG::MLRhsNormInf() 11 0.0001973 0.0001973 0.0001973 0.02% FabArrayBase::FB::FB() 56 9.19e-05 9.19e-05 9.19e-05 0.01% Castro::finalize_advance() 10 6.269e-05 6.269e-05 6.269e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.804e-05 5.804e-05 5.804e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.666e-05 5.666e-05 5.666e-05 0.01% makeSFC 55 4.195e-05 4.195e-05 4.195e-05 0.00% StateData::define() 4 4.134e-05 4.134e-05 4.134e-05 0.00% Castro::swap_state_time_levels() 10 3.887e-05 3.887e-05 3.887e-05 0.00% Castro::finalize_do_advance() 10 3.871e-05 3.871e-05 3.871e-05 0.00% Castro::enforce_consistent_e() 1 3.503e-05 3.503e-05 3.503e-05 0.00% Amr::writeSmallPlotFile() 1 2.553e-05 2.553e-05 2.553e-05 0.00% DistributionMapping::Distribute() 56 1.676e-05 1.676e-05 1.676e-05 0.00% Amr::initSubcycle() 1 1.008e-05 1.008e-05 1.008e-05 0.00% AmrLevel::checkPointPost() 3 6.579e-06 6.579e-06 6.579e-06 0.00% MLMG::buildFineMask() 11 5.855e-06 5.855e-06 5.855e-06 0.00% Castro::create_source_corrector() 10 5.009e-06 5.009e-06 5.009e-06 0.00% Gravity::set_mass_offset() 11 4.965e-06 4.965e-06 4.965e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.959e-06 3.959e-06 3.959e-06 0.00% Castro::retry_advance_ctu() 10 3.624e-06 3.624e-06 3.624e-06 0.00% Castro::FluxRegFineAdd() 10 3.166e-06 3.166e-06 3.166e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.307e-06 2.307e-06 2.307e-06 0.00% AmrLevel::checkPointPre() 3 2.059e-06 2.059e-06 2.059e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-14-gd70f70e2f56e) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-14-gd70f70e2f56e) initialized Starting run at 08:25:24 UTC on 2022-04-11. Successfully read inputs file ... Castro git describe: 22.04-2-gb74783602 AMReX git describe: 22.04-14-gd70f70e2f Microphysics git describe: 22.04-1-gef526ccd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.512491404 Restart time = 0.055252229 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.11567199 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048204517 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.048646686 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059445341 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062047582 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.0232241 seconds Ending run at 08:25:24 UTC on 2022-04-11. Run time = 0.413455608 Run time without initialization = 0.357614518 Average number of zones advanced per microsecond: 3.665 Average number of zones advanced per microsecond per rank: 3.665 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4135 ... 0.4135 ... 0.4135 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1257 0.1257 0.1257 30.40% VisMF::Read() 3 0.04358 0.04358 0.04358 10.54% MLCellLinOp::applyBC() 1946 0.03729 0.03729 0.03729 9.02% MLPoisson::Fsmooth() 1440 0.02544 0.02544 0.02544 6.15% VisMF::Write(FabArray) 1 0.0219 0.0219 0.0219 5.30% FabArray::setVal() 537 0.01669 0.01669 0.01669 4.04% StateData::FillBoundary(geom) 160 0.0112 0.0112 0.0112 2.71% MLCGSolver::bicgstab 36 0.009224 0.009224 0.009224 2.23% MultiFab::Dot() 484 0.009072 0.009072 0.009072 2.19% StateDataPhysBCFunct::() 20 0.007706 0.007706 0.007706 1.86% FillBoundary_nowait() 1766 0.007162 0.007162 0.007162 1.73% Castro::computeTemp() 30 0.007144 0.007144 0.007144 1.73% FabArray::ParallelCopy_nowait() 380 0.007045 0.007045 0.007045 1.70% Gravity::fill_multipole_BCs() 6 0.006317 0.006317 0.006317 1.53% Castro::enforce_min_density() 30 0.006191 0.006191 0.006191 1.50% MLCellLinOp::defineAuxData() 6 0.005492 0.005492 0.005492 1.33% MultiFab::LinComb() 690 0.005228 0.005228 0.005228 1.26% MLPoisson::Fapply() 500 0.004556 0.004556 0.004556 1.10% FabArray::setDomainBndry() 20 0.004129 0.004129 0.004129 1.00% Castro::expand_state() 5 0.004065 0.004065 0.004065 0.98% Amr::restart() 1 0.003045 0.003045 0.003045 0.74% MLMG::addInterpCorrection() 180 0.00296 0.00296 0.00296 0.72% Castro::do_advance_ctu() 5 0.002834 0.002834 0.002834 0.69% amrex::average_down 180 0.002762 0.002762 0.002762 0.67% MultiFab::Xpay() 258 0.002629 0.002629 0.002629 0.64% Castro::reset_internal_energy() 30 0.002454 0.002454 0.002454 0.59% Castro::estTimeStep() 10 0.002309 0.002309 0.002309 0.56% Castro::normalize_species() 30 0.002194 0.002194 0.002194 0.53% BndryData::define() 6 0.001984 0.001984 0.001984 0.48% Gravity::get_new_grav_vector() 5 0.001781 0.001781 0.001781 0.43% Castro::construct_new_gravity_source() 5 0.001554 0.001554 0.001554 0.38% Amr::writePlotFile() 1 0.001427 0.001427 0.001427 0.35% MLCellLinOp::compGrad() 6 0.001366 0.001366 0.001366 0.33% Castro::construct_old_gravity_source() 5 0.001345 0.001345 0.001345 0.33% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001199 0.001199 0.001199 0.29% Castro::enforce_speed_limit() 30 0.001065 0.001065 0.001065 0.26% MultiFab::Saxpy() 10 0.0009186 0.0009186 0.0009186 0.22% Gravity::get_old_grav_vector() 5 0.0009072 0.0009072 0.0009072 0.22% MLMG::ResNormInf() 42 0.0008587 0.0008587 0.0008587 0.21% MLCellLinOp::setLevelBC() 6 0.0008173 0.0008173 0.0008173 0.20% MLMG::oneIter() 36 0.0007446 0.0007446 0.0007446 0.18% Gravity::actual_solve_with_mlmg() 6 0.0007167 0.0007167 0.0007167 0.17% MLCellLinOp::prepareForSolve() 6 0.000691 0.000691 0.000691 0.17% Castro::subcycle_advance_ctu() 5 0.0006816 0.0006816 0.0006816 0.16% FabArray::mult() 22 0.0006571 0.0006571 0.0006571 0.16% MLCellLinOp::smooth() 720 0.0006033 0.0006033 0.0006033 0.15% Gravity::update_max_rhs() 6 0.0005229 0.0005229 0.0005229 0.13% FabArrayBase::getCPC() 632 0.0004381 0.0004381 0.0004381 0.11% MLMG::prepareForSolve() 6 0.0004262 0.0004262 0.0004262 0.10% FabArrayBase::CPC::define() 244 0.0004256 0.0004256 0.0004256 0.10% FabArray::FillBoundary() 1766 0.0004193 0.0004193 0.0004193 0.10% Amr::InitAmr() 1 0.0003874 0.0003874 0.0003874 0.09% MultiFab::Copy() 6 0.00037 0.00037 0.00037 0.09% FabArrayBase::getFB() 1766 0.0003239 0.0003239 0.0003239 0.08% main() 1 0.0003073 0.0003073 0.0003073 0.07% MLCellLinOp::apply() 500 0.0002468 0.0002468 0.0002468 0.06% Gravity::actual_multilevel_solve() 1 0.0002274 0.0002274 0.0002274 0.05% CGSolver::sxay() 690 0.0002214 0.0002214 0.0002214 0.05% Gravity::solve_for_phi() 5 0.0002205 0.0002205 0.0002205 0.05% MultiFab::min() 5 0.0001934 0.0001934 0.0001934 0.05% Castro::construct_new_gravity() 5 0.0001827 0.0001827 0.0001827 0.04% Castro::construct_new_source() 25 0.0001822 0.0001822 0.0001822 0.04% MLMG::mgVcycle() 36 0.0001734 0.0001734 0.0001734 0.04% MLCGSolver::ParallelAllReduce 659 0.0001491 0.0001491 0.0001491 0.04% FabArray::ParallelCopy() 380 0.0001466 0.0001466 0.0001466 0.04% MultiFab::max() 6 0.0001336 0.0001336 0.0001336 0.03% FillPatchIterator::Initialize 20 0.0001241 0.0001241 0.0001241 0.03% Amr::coarseTimeStep() 5 0.0001101 0.0001101 0.0001101 0.03% MLMG::MLRhsNormInf() 6 0.0001054 0.0001054 0.0001054 0.03% Amr::timeStep() 5 0.0001035 0.0001035 0.0001035 0.03% MLCellLinOp::correctionResidual() 216 0.0001034 0.0001034 0.0001034 0.03% MLLinOp::defineGrids() 6 0.0001013 0.0001013 0.0001013 0.02% MLCellLinOp::defineBC() 6 0.0001009 0.0001009 0.0001009 0.02% AmrLevel::restart() 1 7.743e-05 7.743e-05 7.743e-05 0.02% StateData::restartDoit() 4 7.265e-05 7.265e-05 7.265e-05 0.02% MLMG:computeResOfCorrection() 180 6.768e-05 6.768e-05 6.768e-05 0.02% Castro::initialize_advance() 5 6.034e-05 6.034e-05 6.034e-05 0.01% FabArrayBase::FB::FB() 26 5.997e-05 5.997e-05 5.997e-05 0.01% Castro::post_timestep() 5 5.814e-05 5.814e-05 5.814e-05 0.01% MLMG::actualBottomSolve() 36 4.982e-05 4.982e-05 4.982e-05 0.01% Castro::initialize_do_advance() 5 4.883e-05 4.883e-05 4.883e-05 0.01% Castro::advance() 5 4.556e-05 4.556e-05 4.556e-05 0.01% MLMG::mgVcycle_down::1 36 4.183e-05 4.183e-05 4.183e-05 0.01% MLMG::mgVcycle_down::0 36 4.122e-05 4.122e-05 4.122e-05 0.01% MLMG::solve() 6 4.1e-05 4.1e-05 4.1e-05 0.01% MLMG::mgVcycle_down::2 36 3.835e-05 3.835e-05 3.835e-05 0.01% MLMG::mgVcycle_down::3 36 3.721e-05 3.721e-05 3.721e-05 0.01% Castro::clean_state() 30 3.702e-05 3.702e-05 3.702e-05 0.01% MLMG::mgVcycle_down::4 36 3.674e-05 3.674e-05 3.674e-05 0.01% Castro::do_new_sources() 5 3.57e-05 3.57e-05 3.57e-05 0.01% MLMG::mgVcycle_up::4 36 3.142e-05 3.142e-05 3.142e-05 0.01% Amr::writeSmallPlotFile() 1 2.834e-05 2.834e-05 2.834e-05 0.01% MLMG::mgVcycle_up::0 36 2.779e-05 2.779e-05 2.779e-05 0.01% Castro::swap_state_time_levels() 5 2.699e-05 2.699e-05 2.699e-05 0.01% MLMG::mgVcycle_up::3 36 2.676e-05 2.676e-05 2.676e-05 0.01% Castro::finalize_advance() 5 2.675e-05 2.675e-05 2.675e-05 0.01% MLLinOp::define() 6 2.634e-05 2.634e-05 2.634e-05 0.01% MLMG::mgVcycle_up::2 36 2.617e-05 2.617e-05 2.617e-05 0.01% MLMG::mgVcycle_up::1 36 2.548e-05 2.548e-05 2.548e-05 0.01% Castro::post_restart() 1 2.449e-05 2.449e-05 2.449e-05 0.01% MLCellLinOp::solutionResidual() 42 2.446e-05 2.446e-05 2.446e-05 0.01% Castro::finalize_do_advance() 5 2.155e-05 2.155e-05 2.155e-05 0.01% Castro::create_source_corrector() 5 2.145e-05 2.145e-05 2.145e-05 0.01% MLMG::mgVcycle_bottom 36 1.706e-05 1.706e-05 1.706e-05 0.00% MLPoisson::define() 6 1.596e-05 1.596e-05 1.596e-05 0.00% MLMG::computeResidual() 36 1.572e-05 1.572e-05 1.572e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.56e-05 1.56e-05 1.56e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.537e-05 1.537e-05 1.537e-05 0.00% FillPatchSingleLevel 20 1.527e-05 1.527e-05 1.527e-05 0.00% makeSFC 30 1.455e-05 1.455e-05 1.455e-05 0.00% Castro::construct_old_source() 25 1.244e-05 1.244e-05 1.244e-05 0.00% Amr::initSubcycle() 1 1.014e-05 1.014e-05 1.014e-05 0.00% DistributionMapping::Distribute() 31 8.999e-06 8.999e-06 8.999e-06 0.00% Castro::do_old_sources() 5 8.07e-06 8.07e-06 8.07e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.049e-06 8.049e-06 8.049e-06 0.00% Castro::construct_old_gravity() 5 6.563e-06 6.563e-06 6.563e-06 0.00% Castro::apply_source_to_state() 10 5.562e-06 5.562e-06 5.562e-06 0.00% MLPoisson::prepareForSolve() 6 5.336e-06 5.336e-06 5.336e-06 0.00% Gravity::swapTimeLevels() 5 4.479e-06 4.479e-06 4.479e-06 0.00% MLMG::buildFineMask() 6 3.481e-06 3.481e-06 3.481e-06 0.00% Castro::computeNewDt() 5 3.369e-06 3.369e-06 3.369e-06 0.00% MLMG::computeMLResidual() 6 3.119e-06 3.119e-06 3.119e-06 0.00% MLMG::getGradSolution() 6 3.106e-06 3.106e-06 3.106e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3e-06 3e-06 3e-06 0.00% Gravity::set_mass_offset() 6 2.692e-06 2.692e-06 2.692e-06 0.00% MLMG::MLResNormInf() 6 2.573e-06 2.573e-06 2.573e-06 0.00% Castro::retry_advance_ctu() 5 1.951e-06 1.951e-06 1.951e-06 0.00% Castro::FluxRegFineAdd() 5 1.498e-06 1.498e-06 1.498e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.19e-06 1.19e-06 1.19e-06 0.00% Amr::init() 1 1.059e-06 1.059e-06 1.059e-06 0.00% AmrLevel::AmrLevel() 1 9.32e-07 9.32e-07 9.32e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4135 0.4135 0.4135 100.00% Amr::coarseTimeStep() 5 0.3341 0.3341 0.3341 80.80% Amr::timeStep() 5 0.3328 0.3328 0.3328 80.49% Castro::advance() 5 0.3294 0.3294 0.3294 79.66% Castro::subcycle_advance_ctu() 5 0.3177 0.3177 0.3177 76.82% Castro::do_advance_ctu() 5 0.317 0.317 0.317 76.66% Castro::construct_new_gravity() 5 0.1451 0.1451 0.1451 35.09% Gravity::solve_phi_with_mlmg() 6 0.1412 0.1412 0.1412 34.15% Gravity::solve_for_phi() 5 0.1356 0.1356 0.1356 32.79% Gravity::actual_solve_with_mlmg() 6 0.1348 0.1348 0.1348 32.59% Castro::construct_ctu_hydro_source() 5 0.1257 0.1257 0.1257 30.40% MLMG::solve() 6 0.1222 0.1222 0.1222 29.55% MLMG::oneIter() 36 0.1143 0.1143 0.1143 27.65% MLMG::mgVcycle() 36 0.1136 0.1136 0.1136 27.47% MLCellLinOp::smooth() 720 0.05951 0.05951 0.05951 14.39% Amr::init() 1 0.0553 0.0553 0.0553 13.37% Amr::restart() 1 0.0553 0.0553 0.0553 13.37% MLCellLinOp::applyBC() 1946 0.04526 0.04526 0.04526 10.95% AmrLevel::restart() 1 0.04451 0.04451 0.04451 10.76% StateData::restartDoit() 4 0.04443 0.04443 0.04443 10.74% VisMF::Read() 3 0.04358 0.04358 0.04358 10.54% MLMG::mgVcycle_bottom 36 0.03273 0.03273 0.03273 7.91% MLMG::actualBottomSolve() 36 0.03271 0.03271 0.03271 7.91% MLCGSolver::bicgstab 36 0.03239 0.03239 0.03239 7.83% MLPoisson::Fsmooth() 1440 0.02544 0.02544 0.02544 6.15% FillPatchIterator::Initialize 20 0.02526 0.02526 0.02526 6.11% Amr::writePlotFile() 1 0.02333 0.02333 0.02333 5.64% VisMF::Write(FabArray) 1 0.0219 0.0219 0.0219 5.30% FillPatchSingleLevel 20 0.02101 0.02101 0.02101 5.08% Castro::clean_state() 30 0.01909 0.01909 0.01909 4.62% StateDataPhysBCFunct::() 20 0.0189 0.0189 0.0189 4.57% FabArray::setVal() 537 0.01669 0.01669 0.01669 4.04% MLCellLinOp::apply() 500 0.01632 0.01632 0.01632 3.95% MLMG::mgVcycle_down::0 36 0.01587 0.01587 0.01587 3.84% Castro::initialize_do_advance() 5 0.01581 0.01581 0.01581 3.82% MLMG::mgVcycle_up::0 36 0.01348 0.01348 0.01348 3.26% Castro::expand_state() 5 0.01241 0.01241 0.01241 3.00% Castro::initialize_advance() 5 0.01165 0.01165 0.01165 2.82% StateData::FillBoundary(geom) 160 0.0112 0.0112 0.0112 2.71% Castro::computeTemp() 30 0.009599 0.009599 0.009599 2.32% MLPoisson::define() 6 0.009392 0.009392 0.009392 2.27% Gravity::get_new_grav_vector() 5 0.009323 0.009323 0.009323 2.25% MLCellLinOp::correctionResidual() 216 0.009241 0.009241 0.009241 2.23% MultiFab::Dot() 484 0.009072 0.009072 0.009072 2.19% Castro::construct_old_gravity() 5 0.008017 0.008017 0.008017 1.94% MLMG:computeResOfCorrection() 180 0.008012 0.008012 0.008012 1.94% Gravity::get_old_grav_vector() 5 0.00801 0.00801 0.00801 1.94% FabArray::FillBoundary() 1766 0.007965 0.007965 0.007965 1.93% MLMG::mgVcycle_down::1 36 0.007742 0.007742 0.007742 1.87% FabArray::ParallelCopy() 380 0.007613 0.007613 0.007613 1.84% FillBoundary_nowait() 1766 0.007545 0.007545 0.007545 1.82% FabArray::ParallelCopy_nowait() 380 0.007466 0.007466 0.007466 1.81% MLMG::mgVcycle_down::2 36 0.007421 0.007421 0.007421 1.79% MLMG::mgVcycle_down::3 36 0.007075 0.007075 0.007075 1.71% Castro::do_new_sources() 5 0.007016 0.007016 0.007016 1.70% MLMG::mgVcycle_down::4 36 0.006762 0.006762 0.006762 1.64% Castro::post_restart() 1 0.006483 0.006483 0.006483 1.57% MLCellLinOp::defineAuxData() 6 0.006433 0.006433 0.006433 1.56% Gravity::fill_multipole_BCs() 6 0.006317 0.006317 0.006317 1.53% Castro::enforce_min_density() 30 0.006191 0.006191 0.006191 1.50% Gravity::multilevel_solve_for_new_phi() 1 0.006093 0.006093 0.006093 1.47% Gravity::actual_multilevel_solve() 1 0.006077 0.006077 0.006077 1.47% MLMG::mgVcycle_up::2 36 0.005768 0.005768 0.005768 1.40% MLMG::addInterpCorrection() 180 0.005708 0.005708 0.005708 1.38% MLMG::mgVcycle_up::1 36 0.005704 0.005704 0.005704 1.38% amrex::average_down 180 0.005536 0.005536 0.005536 1.34% MLCGSolver::ParallelAllReduce 659 0.005507 0.005507 0.005507 1.33% MLMG::mgVcycle_up::3 36 0.005466 0.005466 0.005466 1.32% CGSolver::sxay() 690 0.005449 0.005449 0.005449 1.32% MLMG::mgVcycle_up::4 36 0.005404 0.005404 0.005404 1.31% MultiFab::LinComb() 690 0.005228 0.005228 0.005228 1.26% Castro::do_old_sources() 5 0.004634 0.004634 0.004634 1.12% MLPoisson::Fapply() 500 0.004556 0.004556 0.004556 1.10% FabArray::setDomainBndry() 20 0.004129 0.004129 0.004129 1.00% MLMG::prepareForSolve() 6 0.003454 0.003454 0.003454 0.84% MLCellLinOp::solutionResidual() 42 0.003362 0.003362 0.003362 0.81% Castro::post_timestep() 5 0.00331 0.00331 0.00331 0.80% MLMG::computeResidual() 36 0.002786 0.002786 0.002786 0.67% MLCellLinOp::defineBC() 6 0.002784 0.002784 0.002784 0.67% BndryData::define() 6 0.002683 0.002683 0.002683 0.65% MultiFab::Xpay() 258 0.002629 0.002629 0.002629 0.64% Castro::reset_internal_energy() 30 0.002454 0.002454 0.002454 0.59% Castro::estTimeStep() 10 0.002309 0.002309 0.002309 0.56% Castro::normalize_species() 30 0.002194 0.002194 0.002194 0.53% Castro::construct_new_source() 25 0.001736 0.001736 0.001736 0.42% MLMG::getGradSolution() 6 0.001655 0.001655 0.001655 0.40% MLCellLinOp::compGrad() 6 0.001652 0.001652 0.001652 0.40% Castro::construct_new_gravity_source() 5 0.001554 0.001554 0.001554 0.38% Castro::construct_old_source() 25 0.001358 0.001358 0.001358 0.33% Castro::construct_old_gravity_source() 5 0.001345 0.001345 0.001345 0.33% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001199 0.001199 0.001199 0.29% Castro::computeNewDt() 5 0.001197 0.001197 0.001197 0.29% Castro::enforce_speed_limit() 30 0.001065 0.001065 0.001065 0.26% Castro::apply_source_to_state() 10 0.0009241 0.0009241 0.0009241 0.22% MultiFab::Saxpy() 10 0.0009186 0.0009186 0.0009186 0.22% Gravity::swapTimeLevels() 5 0.0008889 0.0008889 0.0008889 0.21% FabArrayBase::getCPC() 632 0.0008637 0.0008637 0.0008637 0.21% MLMG::ResNormInf() 42 0.0008587 0.0008587 0.0008587 0.21% MLCellLinOp::setLevelBC() 6 0.0008173 0.0008173 0.0008173 0.20% Gravity::update_max_rhs() 6 0.0007357 0.0007357 0.0007357 0.18% MLPoisson::prepareForSolve() 6 0.0006963 0.0006963 0.0006963 0.17% MLCellLinOp::prepareForSolve() 6 0.000691 0.000691 0.000691 0.17% FabArray::mult() 22 0.0006571 0.0006571 0.0006571 0.16% MLMG::computeMLResidual() 6 0.0005948 0.0005948 0.0005948 0.14% FabArrayBase::CPC::define() 244 0.0004256 0.0004256 0.0004256 0.10% Amr::InitAmr() 1 0.0003975 0.0003975 0.0003975 0.10% FabArrayBase::getFB() 1766 0.0003839 0.0003839 0.0003839 0.09% MultiFab::Copy() 6 0.00037 0.00037 0.00037 0.09% MultiFab::min() 5 0.0001934 0.0001934 0.0001934 0.05% MLLinOp::define() 6 0.0001589 0.0001589 0.0001589 0.04% MLMG::MLResNormInf() 6 0.0001386 0.0001386 0.0001386 0.03% MultiFab::max() 6 0.0001336 0.0001336 0.0001336 0.03% MLLinOp::defineGrids() 6 0.0001326 0.0001326 0.0001326 0.03% MLMG::MLRhsNormInf() 6 0.0001054 0.0001054 0.0001054 0.03% FabArrayBase::FB::FB() 26 5.997e-05 5.997e-05 5.997e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.013e-05 3.013e-05 3.013e-05 0.01% Amr::writeSmallPlotFile() 1 2.834e-05 2.834e-05 2.834e-05 0.01% Castro::finalize_advance() 5 2.825e-05 2.825e-05 2.825e-05 0.01% Castro::swap_state_time_levels() 5 2.699e-05 2.699e-05 2.699e-05 0.01% makeSFC 30 2.208e-05 2.208e-05 2.208e-05 0.01% Castro::finalize_do_advance() 5 2.155e-05 2.155e-05 2.155e-05 0.01% Castro::create_source_corrector() 5 2.145e-05 2.145e-05 2.145e-05 0.01% Amr::initSubcycle() 1 1.014e-05 1.014e-05 1.014e-05 0.00% DistributionMapping::Distribute() 31 8.999e-06 8.999e-06 8.999e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.472e-06 4.472e-06 4.472e-06 0.00% MLMG::buildFineMask() 6 3.481e-06 3.481e-06 3.481e-06 0.00% Gravity::set_mass_offset() 6 2.692e-06 2.692e-06 2.692e-06 0.00% Castro::retry_advance_ctu() 5 1.951e-06 1.951e-06 1.951e-06 0.00% Castro::FluxRegFineAdd() 5 1.498e-06 1.498e-06 1.498e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.19e-06 1.19e-06 1.19e-06 0.00% AmrLevel::AmrLevel() 1 9.32e-07 9.32e-07 9.32e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-14-gd70f70e2f56e) finalized