Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.01-22-ge34f5c67fec9) initialized Starting run at 09:06:03 UTC on 2022-01-13. Successfully read inputs file ... Castro git describe: 21.12-23-g727e39bb6 AMReX git describe: 22.01-22-ge34f5c67f Microphysics git describe: 22.01-14-geeef682d reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.042021729 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.024886802 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.137456623 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.075833037 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.074328072 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.086727014 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.089486529 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.0398868 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.084983838 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.071881057 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.072025142 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.093358967 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.093692218 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.041813769 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024962386 seconds Ending run at 09:06:04 UTC on 2022-01-13. Run time = 1.154586533 Run time without initialization = 0.987136372 Average number of zones advanced per microsecond: 2.656 Average number of zones advanced per microsecond per rank: 2.656 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489334272 TinyProfiler total time across processes [min...avg...max]: 1.155 ... 1.155 ... 1.155 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2381 0.2381 0.2381 20.62% VisMF::Write(FabArray) 11 0.1654 0.1654 0.1654 14.32% MLCellLinOp::applyBC() 4433 0.1197 0.1197 0.1197 10.37% MLPoisson::Fsmooth() 3280 0.07536 0.07536 0.07536 6.53% FillBoundary_nowait() 4023 0.03999 0.03999 0.03999 3.46% MLCGSolver::bicgstab 82 0.0382 0.0382 0.0382 3.31% FabArray::setVal() 1140 0.03138 0.03138 0.03138 2.72% StateData::FillBoundary(geom) 328 0.02962 0.02962 0.02962 2.57% MultiFab::Dot() 1114 0.02821 0.02821 0.02821 2.44% MLCellLinOp::smooth() 1640 0.02374 0.02374 0.02374 2.06% FabArray::FillBoundary() 4023 0.02285 0.02285 0.02285 1.98% StateDataPhysBCFunct::() 41 0.02272 0.02272 0.02272 1.97% FabArray::ParallelCopy_nowait() 861 0.02207 0.02207 0.02207 1.91% MultiFab::LinComb() 1586 0.0202 0.0202 0.0202 1.75% Castro::computeTemp() 63 0.01619 0.01619 0.01619 1.40% MLPoisson::Fapply() 1142 0.01613 0.01613 0.01613 1.40% MLCellLinOp::defineAuxData() 11 0.01432 0.01432 0.01432 1.24% Castro::enforce_min_density() 62 0.01363 0.01363 0.01363 1.18% Gravity::fill_multipole_BCs() 11 0.01335 0.01335 0.01335 1.16% FabArrayBase::getFB() 4023 0.01192 0.01192 0.01192 1.03% MLMG::addInterpCorrection() 410 0.01003 0.01003 0.01003 0.87% MLCellLinOp::apply() 1142 0.009997 0.009997 0.009997 0.87% amrex::average_down 410 0.009381 0.009381 0.009381 0.81% CGSolver::sxay() 1586 0.009088 0.009088 0.009088 0.79% MultiFab::Xpay() 585 0.008963 0.008963 0.008963 0.78% Castro::reset_internal_energy() 63 0.00859 0.00859 0.00859 0.74% Castro::enforce_speed_limit() 62 0.006225 0.006225 0.006225 0.54% MLCGSolver::ParallelAllReduce 1514 0.006179 0.006179 0.006179 0.54% Castro::estTimeStep() 21 0.006115 0.006115 0.006115 0.53% FabArray::setDomainBndry() 41 0.005963 0.005963 0.005963 0.52% FabArrayBase::getCPC() 1323 0.005715 0.005715 0.005715 0.50% FabArray::ParallelCopy() 861 0.005068 0.005068 0.005068 0.44% Amr::checkPoint() 3 0.004842 0.004842 0.004842 0.42% BndryData::define() 11 0.004507 0.004507 0.004507 0.39% Castro::expand_state() 10 0.004505 0.004505 0.004505 0.39% MLCellLinOp::correctionResidual() 492 0.004226 0.004226 0.004226 0.37% Castro::do_advance_ctu() 10 0.003922 0.003922 0.003922 0.34% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.003681 0.003681 0.003681 0.32% Castro::construct_new_gravity_source() 10 0.003292 0.003292 0.003292 0.29% Amr::writePlotFile() 2 0.003251 0.003251 0.003251 0.28% MLMG::mgVcycle() 82 0.00312 0.00312 0.00312 0.27% Castro::normalize_species() 62 0.003082 0.003082 0.003082 0.27% Gravity::get_new_grav_vector() 11 0.003014 0.003014 0.003014 0.26% Castro::construct_old_gravity_source() 10 0.003007 0.003007 0.003007 0.26% MLMG:computeResOfCorrection() 410 0.002373 0.002373 0.002373 0.21% MLMG::ResNormInf() 93 0.002347 0.002347 0.002347 0.20% MLMG::oneIter() 82 0.002328 0.002328 0.002328 0.20% Amr::defBaseLevel() 1 0.002221 0.002221 0.002221 0.19% MLCellLinOp::compGrad() 11 0.002169 0.002169 0.002169 0.19% Gravity::get_old_grav_vector() 10 0.002146 0.002146 0.002146 0.19% FabArrayBase::CPC::define() 454 0.001956 0.001956 0.001956 0.17% MultiFab::Saxpy() 20 0.001941 0.001941 0.001941 0.17% MLCellLinOp::setLevelBC() 11 0.00171 0.00171 0.00171 0.15% MLMG::prepareForSolve() 11 0.00163 0.00163 0.00163 0.14% FabArray::mult() 43 0.001536 0.001536 0.001536 0.13% Gravity::actual_solve_with_mlmg() 11 0.001514 0.001514 0.001514 0.13% MLCellLinOp::prepareForSolve() 11 0.001443 0.001443 0.001443 0.13% MLMG::mgVcycle_down::1 82 0.001441 0.001441 0.001441 0.12% MLMG::mgVcycle_down::3 82 0.001417 0.001417 0.001417 0.12% MLMG::mgVcycle_down::2 82 0.001417 0.001417 0.001417 0.12% MLMG::mgVcycle_down::0 82 0.001417 0.001417 0.001417 0.12% MLMG::mgVcycle_down::4 82 0.001401 0.001401 0.001401 0.12% Castro::initData() 1 0.001387 0.001387 0.001387 0.12% MLMG::actualBottomSolve() 82 0.00126 0.00126 0.00126 0.11% MLMG::solve() 11 0.001002 0.001002 0.001002 0.09% MLMG::mgVcycle_up::4 82 0.0009613 0.0009613 0.0009613 0.08% MLMG::mgVcycle_up::0 82 0.000953 0.000953 0.000953 0.08% Castro::clean_state() 62 0.0009462 0.0009462 0.0009462 0.08% MLMG::mgVcycle_up::2 82 0.0009376 0.0009376 0.0009376 0.08% MLMG::mgVcycle_up::1 82 0.0009366 0.0009366 0.0009366 0.08% MLMG::mgVcycle_up::3 82 0.000936 0.000936 0.000936 0.08% MLCellLinOp::solutionResidual() 93 0.0008192 0.0008192 0.0008192 0.07% Gravity::update_max_rhs() 11 0.0007149 0.0007149 0.0007149 0.06% Gravity::solve_for_phi() 10 0.0006398 0.0006398 0.0006398 0.06% MultiFab::Copy() 11 0.00059 0.00059 0.00059 0.05% FillPatchIterator::Initialize 41 0.0005802 0.0005802 0.0005802 0.05% Amr::InitAmr() 1 0.0005705 0.0005705 0.0005705 0.05% Castro::subcycle_advance_ctu() 10 0.0005215 0.0005215 0.0005215 0.05% Amr::coarseTimeStep() 10 0.0005001 0.0005001 0.0005001 0.04% MLMG::computeResidual() 82 0.0004753 0.0004753 0.0004753 0.04% MLMG::mgVcycle_bottom 82 0.000475 0.000475 0.000475 0.04% Castro::initialize_advance() 10 0.0003991 0.0003991 0.0003991 0.03% FillPatchSingleLevel 41 0.0003641 0.0003641 0.0003641 0.03% MultiFab::min() 10 0.0003603 0.0003603 0.0003603 0.03% Castro::construct_new_gravity() 10 0.0003602 0.0003602 0.0003602 0.03% main() 1 0.0003441 0.0003441 0.0003441 0.03% Castro::advance() 10 0.0003304 0.0003304 0.0003304 0.03% makeSFC 55 0.0003281 0.0003281 0.0003281 0.03% MultiFab::max() 11 0.0003074 0.0003074 0.0003074 0.03% Castro::do_new_sources() 10 0.0002905 0.0002905 0.0002905 0.03% Castro::do_old_sources() 10 0.0002891 0.0002891 0.0002891 0.03% Amr::timeStep() 10 0.0002856 0.0002856 0.0002856 0.02% MLLinOp::defineGrids() 11 0.0002742 0.0002742 0.0002742 0.02% FabArrayBase::FB::FB() 56 0.00026 0.00026 0.00026 0.02% MLCellLinOp::defineBC() 11 0.0002563 0.0002563 0.0002563 0.02% MLMG::MLRhsNormInf() 11 0.0002445 0.0002445 0.0002445 0.02% Castro::construct_new_source() 50 0.0002398 0.0002398 0.0002398 0.02% Gravity::actual_multilevel_solve() 1 0.0002354 0.0002354 0.0002354 0.02% Castro::construct_old_source() 50 0.0002121 0.0002121 0.0002121 0.02% MLLinOp::makeAgglomeratedDMap 11 0.0002088 0.0002088 0.0002088 0.02% StateData::checkPoint() 12 0.0002053 0.0002053 0.0002053 0.02% DistributionMapping::Distribute() 56 0.0001684 0.0001684 0.0001684 0.01% Castro::initialize_do_advance() 10 0.0001541 0.0001541 0.0001541 0.01% Gravity::solve_phi_with_mlmg() 11 0.0001479 0.0001479 0.0001479 0.01% MLPoisson::define() 11 0.000144 0.000144 0.000144 0.01% Castro::finalize_advance() 10 0.0001348 0.0001348 0.0001348 0.01% Castro::apply_source_to_state() 20 0.0001232 0.0001232 0.0001232 0.01% Gravity::swapTimeLevels() 10 0.0001171 0.0001171 0.0001171 0.01% AmrLevel::checkPoint() 3 0.0001145 0.0001145 0.0001145 0.01% MLLinOp::define() 11 8.808e-05 8.808e-05 8.808e-05 0.01% MLMG::computeMLResidual() 11 8.229e-05 8.229e-05 8.229e-05 0.01% MLPoisson::prepareForSolve() 11 6.746e-05 6.746e-05 6.746e-05 0.01% Castro::finalize_do_advance() 10 6.72e-05 6.72e-05 6.72e-05 0.01% Castro::construct_old_gravity() 10 6.658e-05 6.658e-05 6.658e-05 0.01% MLMG::getGradSolution() 11 6.629e-05 6.629e-05 6.629e-05 0.01% Castro::swap_state_time_levels() 10 6.587e-05 6.587e-05 6.587e-05 0.01% MLMG::MLResNormInf() 11 6.483e-05 6.483e-05 6.483e-05 0.01% Castro::post_timestep() 10 6.296e-05 6.296e-05 6.296e-05 0.01% StateData::define() 4 5.854e-05 5.854e-05 5.854e-05 0.01% Castro::computeNewDt() 9 5.458e-05 5.458e-05 5.458e-05 0.00% Castro::enforce_consistent_e() 1 4.1e-05 4.1e-05 4.1e-05 0.00% Gravity::set_mass_offset() 11 3.683e-05 3.683e-05 3.683e-05 0.00% MLMG::buildFineMask() 11 3.633e-05 3.633e-05 3.633e-05 0.00% Castro::create_source_corrector() 10 3.246e-05 3.246e-05 3.246e-05 0.00% MLLinOp::makeSubCommunicator() 11 3.238e-05 3.238e-05 3.238e-05 0.00% Castro::retry_advance_ctu() 10 3.15e-05 3.15e-05 3.15e-05 0.00% Amr::FinalizeInit() 1 3.025e-05 3.025e-05 3.025e-05 0.00% Castro::FluxRegFineAdd() 10 3.008e-05 3.008e-05 3.008e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.567e-05 2.567e-05 2.567e-05 0.00% Amr::writeSmallPlotFile() 1 2.5e-05 2.5e-05 2.5e-05 0.00% AmrLevel::AmrLevel(dm) 1 2.271e-05 2.271e-05 2.271e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.171e-05 2.171e-05 2.171e-05 0.00% Castro::post_init() 1 1.833e-05 1.833e-05 1.833e-05 0.00% Amr::init() 1 1.425e-05 1.425e-05 1.425e-05 0.00% AmrLevel::checkPointPost() 3 1.363e-05 1.363e-05 1.363e-05 0.00% Amr::initSubcycle() 1 1.2e-05 1.2e-05 1.2e-05 0.00% Castro::computeInitialDt() 2 1.167e-05 1.167e-05 1.167e-05 0.00% Amr::InitializeInit() 1 1.15e-05 1.15e-05 1.15e-05 0.00% AmrLevel::checkPointPre() 3 1.046e-05 1.046e-05 1.046e-05 0.00% Amr::initialInit() 1 9.25e-06 9.25e-06 9.25e-06 0.00% Castro::post_regrid() 1 6.5e-06 6.5e-06 6.5e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 1.155 1.155 1.155 100.00% Amr::coarseTimeStep() 10 0.9619 0.9619 0.9619 83.31% Amr::timeStep() 10 0.876 0.876 0.876 75.87% Castro::advance() 10 0.8651 0.8651 0.8651 74.92% Castro::subcycle_advance_ctu() 10 0.848 0.848 0.848 73.45% Castro::do_advance_ctu() 10 0.8475 0.8475 0.8475 73.40% Gravity::solve_phi_with_mlmg() 11 0.5722 0.5722 0.5722 49.56% Gravity::actual_solve_with_mlmg() 11 0.5585 0.5585 0.5585 48.37% MLMG::solve() 11 0.5245 0.5245 0.5245 45.42% Castro::construct_new_gravity() 10 0.5096 0.5096 0.5096 44.13% MLMG::oneIter() 82 0.4997 0.4997 0.4997 43.28% MLMG::mgVcycle() 82 0.4974 0.4974 0.4974 43.08% Gravity::solve_for_phi() 10 0.488 0.488 0.488 42.27% MLCellLinOp::smooth() 1640 0.2417 0.2417 0.2417 20.94% Castro::construct_ctu_hydro_source() 10 0.2381 0.2381 0.2381 20.62% MLCellLinOp::applyBC() 4433 0.1948 0.1948 0.1948 16.87% Amr::init() 1 0.1667 0.1667 0.1667 14.44% VisMF::Write(FabArray) 11 0.1654 0.1654 0.1654 14.32% MLMG::mgVcycle_bottom 82 0.1491 0.1491 0.1491 12.91% MLMG::actualBottomSolve() 82 0.1486 0.1486 0.1486 12.87% MLCGSolver::bicgstab 82 0.1463 0.1463 0.1463 12.67% Amr::checkPoint() 3 0.1238 0.1238 0.1238 10.73% AmrLevel::checkPoint() 3 0.119 0.119 0.119 10.30% StateData::checkPoint() 12 0.1189 0.1189 0.1189 10.29% Amr::initialInit() 1 0.09968 0.09968 0.09968 8.63% Amr::FinalizeInit() 1 0.08899 0.08899 0.08899 7.71% Castro::post_init() 1 0.08819 0.08819 0.08819 7.64% Gravity::multilevel_solve_for_new_phi() 1 0.08511 0.08511 0.08511 7.37% Gravity::actual_multilevel_solve() 1 0.08508 0.08508 0.08508 7.37% MLCellLinOp::apply() 1142 0.0775 0.0775 0.0775 6.71% MLPoisson::Fsmooth() 3280 0.07536 0.07536 0.07536 6.53% FabArray::FillBoundary() 4023 0.07502 0.07502 0.07502 6.50% FillPatchIterator::Initialize 41 0.06404 0.06404 0.06404 5.55% FillPatchSingleLevel 41 0.0575 0.0575 0.0575 4.98% MLMG::mgVcycle_down::0 82 0.05508 0.05508 0.05508 4.77% StateDataPhysBCFunct::() 41 0.05234 0.05234 0.05234 4.53% FillBoundary_nowait() 4023 0.05217 0.05217 0.05217 4.52% Amr::writePlotFile() 2 0.04997 0.04997 0.04997 4.33% Castro::clean_state() 62 0.04783 0.04783 0.04783 4.14% MLMG::mgVcycle_up::0 82 0.04597 0.04597 0.04597 3.98% MLCellLinOp::correctionResidual() 492 0.04495 0.04495 0.04495 3.89% MLMG:computeResOfCorrection() 410 0.04033 0.04033 0.04033 3.49% MLMG::mgVcycle_down::1 82 0.03518 0.03518 0.03518 3.05% MLMG::mgVcycle_down::2 82 0.03481 0.03481 0.03481 3.01% MLMG::mgVcycle_down::3 82 0.03408 0.03408 0.03408 2.95% MLMG::mgVcycle_down::4 82 0.03331 0.03331 0.03331 2.89% FabArray::setVal() 1140 0.03138 0.03138 0.03138 2.72% FabArray::ParallelCopy() 861 0.03104 0.03104 0.03104 2.69% StateData::FillBoundary(geom) 328 0.02962 0.02962 0.02962 2.57% CGSolver::sxay() 1586 0.02929 0.02929 0.02929 2.54% MultiFab::Dot() 1114 0.02821 0.02821 0.02821 2.44% Castro::initialize_do_advance() 10 0.0279 0.0279 0.0279 2.42% MLPoisson::define() 11 0.02778 0.02778 0.02778 2.41% MLMG::mgVcycle_up::1 82 0.02704 0.02704 0.02704 2.34% MLMG::mgVcycle_up::2 82 0.02704 0.02704 0.02704 2.34% MLMG::mgVcycle_up::3 82 0.02635 0.02635 0.02635 2.28% MLMG::mgVcycle_up::4 82 0.0263 0.0263 0.0263 2.28% FabArray::ParallelCopy_nowait() 861 0.02597 0.02597 0.02597 2.25% Castro::computeTemp() 63 0.02478 0.02478 0.02478 2.15% Castro::construct_old_gravity() 10 0.02406 0.02406 0.02406 2.08% Gravity::get_new_grav_vector() 11 0.02403 0.02403 0.02403 2.08% Gravity::get_old_grav_vector() 10 0.024 0.024 0.024 2.08% MLMG::addInterpCorrection() 410 0.02313 0.02313 0.02313 2.00% MLCGSolver::ParallelAllReduce 1514 0.02271 0.02271 0.02271 1.97% amrex::average_down 410 0.02252 0.02252 0.02252 1.95% MultiFab::LinComb() 1586 0.0202 0.0202 0.0202 1.75% MLCellLinOp::defineAuxData() 11 0.02006 0.02006 0.02006 1.74% Castro::expand_state() 10 0.02002 0.02002 0.02002 1.73% Castro::initialize_advance() 10 0.01655 0.01655 0.01655 1.43% MLPoisson::Fapply() 1142 0.01613 0.01613 0.01613 1.40% Castro::enforce_min_density() 62 0.01363 0.01363 0.01363 1.18% Gravity::fill_multipole_BCs() 11 0.01335 0.01335 0.01335 1.16% Castro::do_new_sources() 10 0.0131 0.0131 0.0131 1.13% Castro::do_old_sources() 10 0.01255 0.01255 0.01255 1.09% MLCellLinOp::solutionResidual() 93 0.01218 0.01218 0.01218 1.05% FabArrayBase::getFB() 4023 0.01218 0.01218 0.01218 1.05% MLMG::computeResidual() 82 0.01104 0.01104 0.01104 0.96% Amr::InitializeInit() 1 0.01068 0.01068 0.01068 0.92% Castro::post_timestep() 10 0.01067 0.01067 0.01067 0.92% Amr::defBaseLevel() 1 0.01067 0.01067 0.01067 0.92% MultiFab::Xpay() 585 0.008963 0.008963 0.008963 0.78% Castro::reset_internal_energy() 63 0.00859 0.00859 0.00859 0.74% MLMG::prepareForSolve() 11 0.008367 0.008367 0.008367 0.72% FabArrayBase::getCPC() 1323 0.007672 0.007672 0.007672 0.66% MLCellLinOp::defineBC() 11 0.006488 0.006488 0.006488 0.56% BndryData::define() 11 0.006231 0.006231 0.006231 0.54% Castro::enforce_speed_limit() 62 0.006225 0.006225 0.006225 0.54% Castro::estTimeStep() 21 0.006115 0.006115 0.006115 0.53% FabArray::setDomainBndry() 41 0.005963 0.005963 0.005963 0.52% Castro::initData() 1 0.003731 0.003731 0.003731 0.32% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.003681 0.003681 0.003681 0.32% Castro::construct_new_source() 50 0.003532 0.003532 0.003532 0.31% Castro::construct_new_gravity_source() 10 0.003292 0.003292 0.003292 0.29% Castro::construct_old_source() 50 0.003219 0.003219 0.003219 0.28% Castro::computeNewDt() 9 0.003105 0.003105 0.003105 0.27% Castro::normalize_species() 62 0.003082 0.003082 0.003082 0.27% Castro::construct_old_gravity_source() 10 0.003007 0.003007 0.003007 0.26% MLMG::getGradSolution() 11 0.002988 0.002988 0.002988 0.26% MLCellLinOp::compGrad() 11 0.002922 0.002922 0.002922 0.25% MLMG::ResNormInf() 93 0.002347 0.002347 0.002347 0.20% Castro::apply_source_to_state() 20 0.002064 0.002064 0.002064 0.18% FabArrayBase::CPC::define() 454 0.001956 0.001956 0.001956 0.17% MultiFab::Saxpy() 20 0.001941 0.001941 0.001941 0.17% MLCellLinOp::setLevelBC() 11 0.00171 0.00171 0.00171 0.15% MLMG::computeMLResidual() 11 0.001695 0.001695 0.001695 0.15% FabArray::mult() 43 0.001536 0.001536 0.001536 0.13% MLPoisson::prepareForSolve() 11 0.001511 0.001511 0.001511 0.13% MLCellLinOp::prepareForSolve() 11 0.001443 0.001443 0.001443 0.13% Gravity::swapTimeLevels() 10 0.001404 0.001404 0.001404 0.12% Gravity::update_max_rhs() 11 0.001213 0.001213 0.001213 0.11% MLLinOp::define() 11 0.001096 0.001096 0.001096 0.09% MLLinOp::defineGrids() 11 0.001008 0.001008 0.001008 0.09% Castro::computeInitialDt() 2 0.0007041 0.0007041 0.0007041 0.06% MLLinOp::makeAgglomeratedDMap 11 0.0007014 0.0007014 0.0007014 0.06% MultiFab::Copy() 11 0.00059 0.00059 0.00059 0.05% Amr::InitAmr() 1 0.0005825 0.0005825 0.0005825 0.05% Castro::post_regrid() 1 0.0005425 0.0005425 0.0005425 0.05% makeSFC 55 0.0004926 0.0004926 0.0004926 0.04% MLMG::MLResNormInf() 11 0.0003703 0.0003703 0.0003703 0.03% MultiFab::min() 10 0.0003603 0.0003603 0.0003603 0.03% MultiFab::max() 11 0.0003074 0.0003074 0.0003074 0.03% FabArrayBase::FB::FB() 56 0.00026 0.00026 0.00026 0.02% MLMG::MLRhsNormInf() 11 0.0002445 0.0002445 0.0002445 0.02% DistributionMapping::Distribute() 56 0.0001684 0.0001684 0.0001684 0.01% Castro::finalize_advance() 10 0.0001648 0.0001648 0.0001648 0.01% AmrLevel::AmrLevel(dm) 1 8.125e-05 8.125e-05 8.125e-05 0.01% Castro::finalize_do_advance() 10 6.72e-05 6.72e-05 6.72e-05 0.01% Castro::swap_state_time_levels() 10 6.587e-05 6.587e-05 6.587e-05 0.01% StateData::define() 4 5.854e-05 5.854e-05 5.854e-05 0.01% Castro::enforce_consistent_e() 1 4.1e-05 4.1e-05 4.1e-05 0.00% Gravity::set_mass_offset() 11 3.683e-05 3.683e-05 3.683e-05 0.00% MLMG::buildFineMask() 11 3.633e-05 3.633e-05 3.633e-05 0.00% Castro::create_source_corrector() 10 3.246e-05 3.246e-05 3.246e-05 0.00% MLLinOp::makeSubCommunicator() 11 3.238e-05 3.238e-05 3.238e-05 0.00% Castro::retry_advance_ctu() 10 3.15e-05 3.15e-05 3.15e-05 0.00% Castro::FluxRegFineAdd() 10 3.008e-05 3.008e-05 3.008e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.95e-05 2.95e-05 2.95e-05 0.00% Amr::writeSmallPlotFile() 1 2.5e-05 2.5e-05 2.5e-05 0.00% AmrLevel::checkPointPost() 3 1.363e-05 1.363e-05 1.363e-05 0.00% Amr::initSubcycle() 1 1.2e-05 1.2e-05 1.2e-05 0.00% AmrLevel::checkPointPre() 3 1.046e-05 1.046e-05 1.046e-05 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10622 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.01-22-ge34f5c67fec9) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.01-22-ge34f5c67fec9) initialized Starting run at 09:06:05 UTC on 2022-01-13. Successfully read inputs file ... Castro git describe: 21.12-23-g727e39bb6 AMReX git describe: 22.01-22-ge34f5c67f Microphysics git describe: 22.01-14-geeef682d reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.671145027 Restart time = 0.058960654 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.146232129 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.073505066 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.086113761 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.088598565 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.089828363 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.049211351 seconds Ending run at 09:06:05 UTC on 2022-01-13. Run time = 0.593611856 Run time without initialization = 0.533877989 Average number of zones advanced per microsecond: 2.455 Average number of zones advanced per microsecond per rank: 2.455 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489334272 TinyProfiler total time across processes [min...avg...max]: 0.5937 ... 0.5937 ... 0.5937 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1510 0.1510 0.1510 25.44% MLCellLinOp::applyBC() 1946 0.05455 0.05455 0.05455 9.19% VisMF::Read() 3 0.04484 0.04484 0.04484 7.55% MLPoisson::Fsmooth() 1440 0.03261 0.03261 0.03261 5.49% VisMF::Write(FabArray) 1 0.02584 0.02584 0.02584 4.35% Amr::writePlotFile() 1 0.02346 0.02346 0.02346 3.95% FillBoundary_nowait() 1766 0.01782 0.01782 0.01782 3.00% FabArray::setVal() 533 0.01763 0.01763 0.01763 2.97% MLCGSolver::bicgstab 36 0.01664 0.01664 0.01664 2.80% StateData::FillBoundary(geom) 160 0.01226 0.01226 0.01226 2.06% MultiFab::Dot() 484 0.01223 0.01223 0.01223 2.06% MLCellLinOp::smooth() 720 0.01041 0.01041 0.01041 1.75% FabArray::ParallelCopy_nowait() 380 0.01022 0.01022 0.01022 1.72% StateDataPhysBCFunct::() 20 0.01019 0.01019 0.01019 1.72% FabArray::FillBoundary() 1766 0.0101 0.0101 0.0101 1.70% MultiFab::LinComb() 690 0.008767 0.008767 0.008767 1.48% Castro::computeTemp() 30 0.008469 0.008469 0.008469 1.43% MLCellLinOp::defineAuxData() 6 0.007512 0.007512 0.007512 1.27% Gravity::fill_multipole_BCs() 6 0.007434 0.007434 0.007434 1.25% MLPoisson::Fapply() 500 0.007008 0.007008 0.007008 1.18% Castro::enforce_min_density() 30 0.005548 0.005548 0.005548 0.93% FabArrayBase::getFB() 1766 0.005313 0.005313 0.005313 0.90% Amr::restart() 1 0.00472 0.00472 0.00472 0.80% MLMG::addInterpCorrection() 180 0.004491 0.004491 0.004491 0.76% MLCellLinOp::apply() 500 0.00434 0.00434 0.00434 0.73% Castro::expand_state() 5 0.004215 0.004215 0.004215 0.71% amrex::average_down 180 0.004144 0.004144 0.004144 0.70% FabArray::setDomainBndry() 20 0.004071 0.004071 0.004071 0.69% CGSolver::sxay() 690 0.004022 0.004022 0.004022 0.68% MultiFab::Xpay() 258 0.00398 0.00398 0.00398 0.67% Castro::estTimeStep() 10 0.00351 0.00351 0.00351 0.59% Castro::reset_internal_energy() 30 0.003366 0.003366 0.003366 0.57% FabArrayBase::getCPC() 632 0.002859 0.002859 0.002859 0.48% Castro::do_advance_ctu() 5 0.002821 0.002821 0.002821 0.48% MLCGSolver::ParallelAllReduce 659 0.002777 0.002777 0.002777 0.47% BndryData::define() 6 0.002463 0.002463 0.002463 0.41% FabArray::ParallelCopy() 380 0.002232 0.002232 0.002232 0.38% Gravity::get_new_grav_vector() 5 0.002111 0.002111 0.002111 0.36% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001958 0.001958 0.001958 0.33% MLCellLinOp::correctionResidual() 216 0.001895 0.001895 0.001895 0.32% MLCellLinOp::compGrad() 6 0.001487 0.001487 0.001487 0.25% Castro::construct_new_gravity_source() 5 0.001481 0.001481 0.001481 0.25% Castro::normalize_species() 30 0.001453 0.001453 0.001453 0.24% MLMG::mgVcycle() 36 0.001385 0.001385 0.001385 0.23% Castro::construct_old_gravity_source() 5 0.001238 0.001238 0.001238 0.21% FabArrayBase::CPC::define() 244 0.001114 0.001114 0.001114 0.19% Gravity::get_old_grav_vector() 5 0.001084 0.001084 0.001084 0.18% MLMG:computeResOfCorrection() 180 0.001068 0.001068 0.001068 0.18% MLMG::ResNormInf() 42 0.001057 0.001057 0.001057 0.18% MLMG::oneIter() 36 0.001013 0.001013 0.001013 0.17% MultiFab::Saxpy() 10 0.0009579 0.0009579 0.0009579 0.16% MLMG::prepareForSolve() 6 0.0009381 0.0009381 0.0009381 0.16% MLCellLinOp::setLevelBC() 6 0.000919 0.000919 0.000919 0.15% Castro::enforce_speed_limit() 30 0.0008848 0.0008848 0.0008848 0.15% Gravity::actual_solve_with_mlmg() 6 0.0008175 0.0008175 0.0008175 0.14% MLCellLinOp::prepareForSolve() 6 0.0007694 0.0007694 0.0007694 0.13% FabArray::mult() 22 0.0007572 0.0007572 0.0007572 0.13% Gravity::update_max_rhs() 6 0.0006657 0.0006657 0.0006657 0.11% MLMG::mgVcycle_down::0 36 0.0006395 0.0006395 0.0006395 0.11% MLMG::mgVcycle_down::1 36 0.0006385 0.0006385 0.0006385 0.11% MLMG::mgVcycle_down::3 36 0.0006344 0.0006344 0.0006344 0.11% MLMG::mgVcycle_down::2 36 0.0006221 0.0006221 0.0006221 0.10% MLMG::mgVcycle_down::4 36 0.0006201 0.0006201 0.0006201 0.10% MLMG::actualBottomSolve() 36 0.000552 0.000552 0.000552 0.09% Amr::InitAmr() 1 0.0005409 0.0005409 0.0005409 0.09% MLMG::solve() 6 0.0004818 0.0004818 0.0004818 0.08% Castro::clean_state() 30 0.0004425 0.0004425 0.0004425 0.07% MLMG::mgVcycle_up::4 36 0.0004343 0.0004343 0.0004343 0.07% MLMG::mgVcycle_up::1 36 0.0004268 0.0004268 0.0004268 0.07% MultiFab::Copy() 6 0.000416 0.000416 0.000416 0.07% MLMG::mgVcycle_up::3 36 0.0004147 0.0004147 0.0004147 0.07% MLMG::mgVcycle_up::0 36 0.0004128 0.0004128 0.0004128 0.07% MLMG::mgVcycle_up::2 36 0.0004127 0.0004127 0.0004127 0.07% MLCellLinOp::solutionResidual() 42 0.0003682 0.0003682 0.0003682 0.06% main() 1 0.0003212 0.0003212 0.0003212 0.05% Castro::construct_new_source() 25 0.0003115 0.0003115 0.0003115 0.05% FillPatchIterator::Initialize 20 0.000295 0.000295 0.000295 0.05% Castro::initialize_advance() 5 0.0002814 0.0002814 0.0002814 0.05% Gravity::solve_for_phi() 5 0.0002751 0.0002751 0.0002751 0.05% Castro::post_timestep() 5 0.0002279 0.0002279 0.0002279 0.04% Gravity::actual_multilevel_solve() 1 0.0002277 0.0002277 0.0002277 0.04% Castro::subcycle_advance_ctu() 5 0.000219 0.000219 0.000219 0.04% Castro::construct_new_gravity() 5 0.0002138 0.0002138 0.0002138 0.04% MLMG::mgVcycle_bottom 36 0.00021 0.00021 0.00021 0.04% MLMG::computeResidual() 36 0.0002084 0.0002084 0.0002084 0.04% Amr::coarseTimeStep() 5 0.0001939 0.0001939 0.0001939 0.03% FillPatchSingleLevel 20 0.0001923 0.0001923 0.0001923 0.03% MultiFab::min() 5 0.0001803 0.0001803 0.0001803 0.03% makeSFC 30 0.000179 0.000179 0.000179 0.03% MultiFab::max() 6 0.0001643 0.0001643 0.0001643 0.03% Castro::advance() 5 0.0001617 0.0001617 0.0001617 0.03% Castro::construct_old_source() 25 0.000155 0.000155 0.000155 0.03% Castro::initialize_do_advance() 5 0.0001544 0.0001544 0.0001544 0.03% Castro::do_new_sources() 5 0.0001452 0.0001452 0.0001452 0.02% Castro::do_old_sources() 5 0.0001443 0.0001443 0.0001443 0.02% MLLinOp::defineGrids() 6 0.0001435 0.0001435 0.0001435 0.02% MLMG::MLRhsNormInf() 6 0.0001429 0.0001429 0.0001429 0.02% FabArrayBase::FB::FB() 26 0.0001375 0.0001375 0.0001375 0.02% MLCellLinOp::defineBC() 6 0.0001365 0.0001365 0.0001365 0.02% Amr::timeStep() 5 0.0001295 0.0001295 0.0001295 0.02% Castro::create_source_corrector() 5 0.0001281 0.0001281 0.0001281 0.02% MLLinOp::makeAgglomeratedDMap 6 0.0001068 0.0001068 0.0001068 0.02% AmrLevel::restart() 1 0.0001025 0.0001025 0.0001025 0.02% StateData::restartDoit() 4 9.917e-05 9.917e-05 9.917e-05 0.02% DistributionMapping::Distribute() 31 9.4e-05 9.4e-05 9.4e-05 0.02% Castro::finalize_advance() 5 8.296e-05 8.296e-05 8.296e-05 0.01% MLPoisson::define() 6 8.2e-05 8.2e-05 8.2e-05 0.01% Gravity::solve_phi_with_mlmg() 6 7.996e-05 7.996e-05 7.996e-05 0.01% Castro::construct_old_gravity() 5 6.967e-05 6.967e-05 6.967e-05 0.01% Castro::apply_source_to_state() 10 6.079e-05 6.079e-05 6.079e-05 0.01% MLLinOp::define() 6 5.975e-05 5.975e-05 5.975e-05 0.01% Gravity::swapTimeLevels() 5 5.825e-05 5.825e-05 5.825e-05 0.01% Castro::swap_state_time_levels() 5 5.254e-05 5.254e-05 5.254e-05 0.01% MLMG::MLResNormInf() 6 5.071e-05 5.071e-05 5.071e-05 0.01% MLPoisson::prepareForSolve() 6 3.7e-05 3.7e-05 3.7e-05 0.01% MLMG::computeMLResidual() 6 3.692e-05 3.692e-05 3.692e-05 0.01% Castro::post_restart() 1 3.546e-05 3.546e-05 3.546e-05 0.01% MLMG::getGradSolution() 6 3.546e-05 3.546e-05 3.546e-05 0.01% Castro::finalize_do_advance() 5 3.25e-05 3.25e-05 3.25e-05 0.01% Castro::computeNewDt() 5 3.104e-05 3.104e-05 3.104e-05 0.01% Amr::writeSmallPlotFile() 1 2.729e-05 2.729e-05 2.729e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.142e-05 2.142e-05 2.142e-05 0.00% MLMG::buildFineMask() 6 1.929e-05 1.929e-05 1.929e-05 0.00% Gravity::set_mass_offset() 6 1.887e-05 1.887e-05 1.887e-05 0.00% MLLinOp::makeSubCommunicator() 6 1.742e-05 1.742e-05 1.742e-05 0.00% Castro::FluxRegFineAdd() 5 1.525e-05 1.525e-05 1.525e-05 0.00% Castro::retry_advance_ctu() 5 1.513e-05 1.513e-05 1.513e-05 0.00% Amr::initSubcycle() 1 1.292e-05 1.292e-05 1.292e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 9.834e-06 9.834e-06 9.834e-06 0.00% Amr::init() 1 6.874e-06 6.874e-06 6.874e-06 0.00% AmrLevel::AmrLevel() 1 3.375e-06 3.375e-06 3.375e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.5936 0.5936 0.5936 100.00% Amr::coarseTimeStep() 5 0.4844 0.4844 0.4844 81.60% Amr::timeStep() 5 0.4818 0.4818 0.4818 81.15% Castro::advance() 5 0.4786 0.4786 0.4786 80.63% Castro::subcycle_advance_ctu() 5 0.4663 0.4663 0.4663 78.55% Castro::do_advance_ctu() 5 0.4661 0.4661 0.4661 78.51% Castro::construct_new_gravity() 5 0.2637 0.2637 0.2637 44.43% Gravity::solve_phi_with_mlmg() 6 0.2599 0.2599 0.2599 43.78% Gravity::solve_for_phi() 5 0.2525 0.2525 0.2525 42.53% Gravity::actual_solve_with_mlmg() 6 0.2522 0.2522 0.2522 42.49% MLMG::solve() 6 0.2336 0.2336 0.2336 39.35% MLMG::oneIter() 36 0.2212 0.2212 0.2212 37.25% MLMG::mgVcycle() 36 0.2201 0.2201 0.2201 37.08% Castro::construct_ctu_hydro_source() 5 0.151 0.151 0.151 25.44% MLCellLinOp::smooth() 720 0.1073 0.1073 0.1073 18.07% MLCellLinOp::applyBC() 1946 0.08792 0.08792 0.08792 14.81% MLMG::mgVcycle_bottom 36 0.06536 0.06536 0.06536 11.01% MLMG::actualBottomSolve() 36 0.06515 0.06515 0.06515 10.97% MLCGSolver::bicgstab 36 0.06413 0.06413 0.06413 10.80% Amr::init() 1 0.05902 0.05902 0.05902 9.94% Amr::restart() 1 0.05901 0.05901 0.05901 9.94% Amr::writePlotFile() 1 0.0493 0.0493 0.0493 8.30% AmrLevel::restart() 1 0.04588 0.04588 0.04588 7.73% StateData::restartDoit() 4 0.04576 0.04576 0.04576 7.71% VisMF::Read() 3 0.04484 0.04484 0.04484 7.55% MLCellLinOp::apply() 500 0.03456 0.03456 0.03456 5.82% FabArray::FillBoundary() 1766 0.03337 0.03337 0.03337 5.62% MLPoisson::Fsmooth() 1440 0.03261 0.03261 0.03261 5.49% FillPatchIterator::Initialize 20 0.02941 0.02941 0.02941 4.95% VisMF::Write(FabArray) 1 0.02584 0.02584 0.02584 4.35% FillPatchSingleLevel 20 0.02504 0.02504 0.02504 4.22% MLMG::mgVcycle_down::0 36 0.02445 0.02445 0.02445 4.12% FillBoundary_nowait() 1766 0.02327 0.02327 0.02327 3.92% StateDataPhysBCFunct::() 20 0.02244 0.02244 0.02244 3.78% MLMG::mgVcycle_up::0 36 0.02065 0.02065 0.02065 3.48% Castro::clean_state() 30 0.02016 0.02016 0.02016 3.40% MLCellLinOp::correctionResidual() 216 0.02 0.02 0.02 3.37% MLMG:computeResOfCorrection() 180 0.01791 0.01791 0.01791 3.02% FabArray::setVal() 533 0.01763 0.01763 0.01763 2.97% Castro::initialize_do_advance() 5 0.01645 0.01645 0.01645 2.77% MLMG::mgVcycle_down::2 36 0.01557 0.01557 0.01557 2.62% MLMG::mgVcycle_down::1 36 0.01551 0.01551 0.01551 2.61% MLMG::mgVcycle_down::3 36 0.01498 0.01498 0.01498 2.52% MLPoisson::define() 6 0.01492 0.01492 0.01492 2.51% MLMG::mgVcycle_down::4 36 0.0149 0.0149 0.0149 2.51% FabArray::ParallelCopy() 380 0.01426 0.01426 0.01426 2.40% Castro::expand_state() 5 0.01314 0.01314 0.01314 2.21% CGSolver::sxay() 690 0.01279 0.01279 0.01279 2.15% StateData::FillBoundary(geom) 160 0.01226 0.01226 0.01226 2.06% MultiFab::Dot() 484 0.01223 0.01223 0.01223 2.06% Castro::initialize_advance() 5 0.01209 0.01209 0.01209 2.04% MLMG::mgVcycle_up::2 36 0.01209 0.01209 0.01209 2.04% FabArray::ParallelCopy_nowait() 380 0.01203 0.01203 0.01203 2.03% MLMG::mgVcycle_up::1 36 0.01184 0.01184 0.01184 1.99% Castro::computeTemp() 30 0.01184 0.01184 0.01184 1.99% MLMG::mgVcycle_up::4 36 0.01178 0.01178 0.01178 1.99% MLMG::mgVcycle_up::3 36 0.01163 0.01163 0.01163 1.96% Gravity::get_new_grav_vector() 5 0.01104 0.01104 0.01104 1.86% MLCellLinOp::defineAuxData() 6 0.01065 0.01065 0.01065 1.79% MLMG::addInterpCorrection() 180 0.01045 0.01045 0.01045 1.76% amrex::average_down 180 0.01003 0.01003 0.01003 1.69% Castro::construct_old_gravity() 5 0.01001 0.01001 0.01001 1.69% MLCGSolver::ParallelAllReduce 659 0.009999 0.009999 0.009999 1.68% Gravity::get_old_grav_vector() 5 0.009945 0.009945 0.009945 1.68% MultiFab::LinComb() 690 0.008767 0.008767 0.008767 1.48% Castro::post_restart() 1 0.008414 0.008414 0.008414 1.42% Castro::do_new_sources() 5 0.008239 0.008239 0.008239 1.39% Gravity::multilevel_solve_for_new_phi() 1 0.007921 0.007921 0.007921 1.33% Gravity::actual_multilevel_solve() 1 0.007899 0.007899 0.007899 1.33% Gravity::fill_multipole_BCs() 6 0.007434 0.007434 0.007434 1.25% MLPoisson::Fapply() 500 0.007008 0.007008 0.007008 1.18% Castro::enforce_min_density() 30 0.005548 0.005548 0.005548 0.93% MLCellLinOp::solutionResidual() 42 0.005538 0.005538 0.005538 0.93% FabArrayBase::getFB() 1766 0.005451 0.005451 0.005451 0.92% Castro::do_old_sources() 5 0.004961 0.004961 0.004961 0.84% MLMG::prepareForSolve() 6 0.004927 0.004927 0.004927 0.83% MLMG::computeResidual() 36 0.004822 0.004822 0.004822 0.81% FabArray::setDomainBndry() 20 0.004071 0.004071 0.004071 0.69% MultiFab::Xpay() 258 0.00398 0.00398 0.00398 0.67% FabArrayBase::getCPC() 632 0.003973 0.003973 0.003973 0.67% MLCellLinOp::defineBC() 6 0.003593 0.003593 0.003593 0.61% Castro::estTimeStep() 10 0.00351 0.00351 0.00351 0.59% BndryData::define() 6 0.003457 0.003457 0.003457 0.58% Castro::reset_internal_energy() 30 0.003366 0.003366 0.003366 0.57% Castro::post_timestep() 5 0.002976 0.002976 0.002976 0.50% Castro::computeNewDt() 5 0.002448 0.002448 0.002448 0.41% MLMG::getGradSolution() 6 0.001965 0.001965 0.001965 0.33% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001958 0.001958 0.001958 0.33% MLCellLinOp::compGrad() 6 0.00193 0.00193 0.00193 0.33% Castro::construct_new_source() 25 0.001793 0.001793 0.001793 0.30% Castro::construct_new_gravity_source() 5 0.001481 0.001481 0.001481 0.25% Castro::normalize_species() 30 0.001453 0.001453 0.001453 0.24% Castro::construct_old_source() 25 0.001393 0.001393 0.001393 0.23% Castro::construct_old_gravity_source() 5 0.001238 0.001238 0.001238 0.21% FabArrayBase::CPC::define() 244 0.001114 0.001114 0.001114 0.19% MLMG::ResNormInf() 42 0.001057 0.001057 0.001057 0.18% Gravity::swapTimeLevels() 5 0.001048 0.001048 0.001048 0.18% Castro::apply_source_to_state() 10 0.001019 0.001019 0.001019 0.17% MLMG::computeMLResidual() 6 0.0009614 0.0009614 0.0009614 0.16% MultiFab::Saxpy() 10 0.0009579 0.0009579 0.0009579 0.16% Gravity::update_max_rhs() 6 0.000936 0.000936 0.000936 0.16% MLCellLinOp::setLevelBC() 6 0.000919 0.000919 0.000919 0.15% Castro::enforce_speed_limit() 30 0.0008848 0.0008848 0.0008848 0.15% MLPoisson::prepareForSolve() 6 0.0008064 0.0008064 0.0008064 0.14% MLCellLinOp::prepareForSolve() 6 0.0007694 0.0007694 0.0007694 0.13% FabArray::mult() 22 0.0007572 0.0007572 0.0007572 0.13% MLLinOp::define() 6 0.0005965 0.0005965 0.0005965 0.10% Amr::InitAmr() 1 0.0005538 0.0005538 0.0005538 0.09% MLLinOp::defineGrids() 6 0.0005367 0.0005367 0.0005367 0.09% MultiFab::Copy() 6 0.000416 0.000416 0.000416 0.07% MLLinOp::makeAgglomeratedDMap 6 0.0003758 0.0003758 0.0003758 0.06% makeSFC 30 0.000269 0.000269 0.000269 0.05% MLMG::MLResNormInf() 6 0.0002119 0.0002119 0.0002119 0.04% MultiFab::min() 5 0.0001803 0.0001803 0.0001803 0.03% MultiFab::max() 6 0.0001643 0.0001643 0.0001643 0.03% MLMG::MLRhsNormInf() 6 0.0001429 0.0001429 0.0001429 0.02% FabArrayBase::FB::FB() 26 0.0001375 0.0001375 0.0001375 0.02% Castro::create_source_corrector() 5 0.0001281 0.0001281 0.0001281 0.02% Castro::finalize_advance() 5 9.821e-05 9.821e-05 9.821e-05 0.02% DistributionMapping::Distribute() 31 9.4e-05 9.4e-05 9.4e-05 0.02% Castro::swap_state_time_levels() 5 5.254e-05 5.254e-05 5.254e-05 0.01% Castro::finalize_do_advance() 5 3.25e-05 3.25e-05 3.25e-05 0.01% Amr::writeSmallPlotFile() 1 2.729e-05 2.729e-05 2.729e-05 0.00% MLMG::buildFineMask() 6 1.929e-05 1.929e-05 1.929e-05 0.00% Gravity::set_mass_offset() 6 1.887e-05 1.887e-05 1.887e-05 0.00% MLLinOp::makeSubCommunicator() 6 1.742e-05 1.742e-05 1.742e-05 0.00% Castro::FluxRegFineAdd() 5 1.525e-05 1.525e-05 1.525e-05 0.00% Castro::retry_advance_ctu() 5 1.513e-05 1.513e-05 1.513e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 1.383e-05 1.383e-05 1.383e-05 0.00% Amr::initSubcycle() 1 1.292e-05 1.292e-05 1.292e-05 0.00% AmrLevel::AmrLevel() 1 3.375e-06 3.375e-06 3.375e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10622 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.01-22-ge34f5c67fec9) finalized