Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-1-g4a53367b19e1) initialized Starting run at 13:58:22 UTC on 2022-12-02. Successfully read inputs file ... Castro git describe: 22.12 AMReX git describe: 22.12-1-g4a53367b1 Microphysics git describe: 22.12 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.054286317 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.03110834 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.050876996 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.053723471 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.06545323 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.06097281 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.080542603 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.090737606 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051512303 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.05036042 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.05017135 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.049577676 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062411474 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.050610275 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.030997861 seconds Ending run at 13:58:23 UTC on 2022-12-02. Run time = 0.887935298 Run time without initialization = 0.748594272 Average number of zones advanced per microsecond: 3.502 Average number of zones advanced per microsecond per rank: 3.502 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.888 ... 0.888 ... 0.888 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2095 0.2095 0.2095 23.59% Castro::construct_ctu_hydro_source() 10 0.1871 0.1871 0.1871 21.08% MLCellLinOp::applyBC() 4433 0.08249 0.08249 0.08249 9.29% MLPoisson::Fsmooth() 3280 0.06484 0.06484 0.06484 7.30% Amr::checkPoint() 3 0.04535 0.04535 0.04535 5.11% StateData::FillBoundary(geom) 328 0.02455 0.02455 0.02455 2.76% amrex::Dot() 1114 0.02306 0.02306 0.02306 2.60% amrex::Copy() 1029 0.01615 0.01615 0.01615 1.82% FabArray::LinComb() 1586 0.01481 0.01481 0.01481 1.67% FabArray::setVal() 1144 0.01458 0.01458 0.01458 1.64% FillBoundary_nowait() 4023 0.01446 0.01446 0.01446 1.63% Castro::normalize_species() 62 0.01422 0.01422 0.01422 1.60% FabArray::ParallelCopy_nowait() 861 0.01361 0.01361 0.01361 1.53% FabArray::norminf() 639 0.01337 0.01337 0.01337 1.51% Castro::computeTemp() 63 0.01311 0.01311 0.01311 1.48% StateDataPhysBCFunct::() 41 0.01255 0.01255 0.01255 1.41% MLCellLinOp::defineAuxData() 11 0.01212 0.01212 0.01212 1.36% MLPoisson::Fapply() 1142 0.01204 0.01204 0.01204 1.36% Gravity::fill_multipole_BCs() 11 0.008407 0.008407 0.008407 0.95% MLMG::addInterpCorrection() 410 0.007889 0.007889 0.007889 0.89% amrex::average_down 410 0.007044 0.007044 0.007044 0.79% FabArray::Xpay() 585 0.006756 0.006756 0.006756 0.76% Castro::enforce_min_density() 62 0.006281 0.006281 0.006281 0.71% Castro::estTimeStep() 21 0.005084 0.005084 0.005084 0.57% Castro::reset_internal_energy(MultiFab) 63 0.004562 0.004562 0.004562 0.51% BndryData::define() 11 0.004033 0.004033 0.004033 0.45% Castro::construct_new_gravity_source() 10 0.00328 0.00328 0.00328 0.37% MLCGSolver::bicgstab 82 0.003273 0.003273 0.003273 0.37% Castro::do_advance_ctu() 10 0.002736 0.002736 0.002736 0.31% amrex::Add() 164 0.002571 0.002571 0.002571 0.29% Castro::construct_old_gravity_source() 10 0.002561 0.002561 0.002561 0.29% Amr::writePlotFile() 2 0.002374 0.002374 0.002374 0.27% MLMG::ResNormInf() 93 0.002111 0.002111 0.002111 0.24% FabArray::Saxpy() 20 0.001819 0.001819 0.001819 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001717 0.001717 0.001717 0.19% Castro::reset_internal_energy(Fab) 504 0.001647 0.001647 0.001647 0.19% MLCellLinOp::setLevelBC() 11 0.001585 0.001585 0.001585 0.18% Castro::enforce_speed_limit() 62 0.001396 0.001396 0.001396 0.16% Gravity::actual_solve_with_mlmg() 11 0.001365 0.001365 0.001365 0.15% FabArray::setDomainBndry() 41 0.001345 0.001345 0.001345 0.15% FabArray::mult() 43 0.001333 0.001333 0.001333 0.15% MLMG::prepareForSolve() 11 0.001225 0.001225 0.001225 0.14% Castro::initData() 1 0.001214 0.001214 0.001214 0.14% MLCellLinOp::prepareForSolve() 11 0.001212 0.001212 0.001212 0.14% MultiFab::contains_nan() 20 0.001201 0.001201 0.001201 0.14% MLCellLinOp::smooth() 1640 0.00103 0.00103 0.00103 0.12% MLCellLinOp::compGrad() 11 0.0009337 0.0009337 0.0009337 0.11% FabArray::FillBoundary() 4023 0.0007759 0.0007759 0.0007759 0.09% FabArrayBase::getCPC() 1323 0.0007591 0.0007591 0.0007591 0.09% FabArrayBase::CPC::define() 454 0.0006876 0.0006876 0.0006876 0.08% Gravity::get_new_grav_vector() 11 0.0006354 0.0006354 0.0006354 0.07% FabArrayBase::getFB() 4023 0.0006028 0.0006028 0.0006028 0.07% Gravity::get_old_grav_vector() 10 0.0005557 0.0005557 0.0005557 0.06% Amr::InitAmr() 1 0.0004963 0.0004963 0.0004963 0.06% MLCellLinOp::apply() 1142 0.0004408 0.0004408 0.0004408 0.05% MLMG::mgVcycle() 82 0.0003792 0.0003792 0.0003792 0.04% MLLinOp::defineGrids() 11 0.0003566 0.0003566 0.0003566 0.04% CGSolver::sxay() 1586 0.0003511 0.0003511 0.0003511 0.04% Amr::coarseTimeStep() 10 0.0003208 0.0003208 0.0003208 0.04% main() 1 0.000309 0.000309 0.000309 0.03% FillPatchIterator::Initialize 41 0.0002936 0.0002936 0.0002936 0.03% MLCGSolver::ParallelAllReduce 1514 0.000291 0.000291 0.000291 0.03% MLCellLinOp::defineBC() 11 0.0002806 0.0002806 0.0002806 0.03% MultiFab::max() 11 0.0002583 0.0002583 0.0002583 0.03% FabArray::ParallelCopy() 861 0.000254 0.000254 0.000254 0.03% MLMG::MLRhsNormInf() 11 0.000219 0.000219 0.000219 0.02% MLCellLinOp::correctionResidual() 492 0.0002099 0.0002099 0.0002099 0.02% Castro::subcycle_advance_ctu() 10 0.0001651 0.0001651 0.0001651 0.02% Amr::timeStep() 10 0.0001503 0.0001503 0.0001503 0.02% StateData::checkPoint() 12 0.0001487 0.0001487 0.0001487 0.02% MLMG:computeResOfCorrection() 410 0.000142 0.000142 0.000142 0.02% Gravity::update_max_rhs() 11 0.0001408 0.0001408 0.0001408 0.02% Gravity::solve_for_phi() 10 0.0001211 0.0001211 0.0001211 0.01% MLMG::mgVcycle_down::0 82 0.0001109 0.0001109 0.0001109 0.01% AmrLevel::checkPoint() 3 0.0001056 0.0001056 0.0001056 0.01% Castro::finalize_advance() 10 0.0001022 0.0001022 0.0001022 0.01% MLMG::mgVcycle_down::1 82 9.433e-05 9.433e-05 9.433e-05 0.01% Castro::advance() 10 9.411e-05 9.411e-05 9.411e-05 0.01% MLMG::mgVcycle_down::2 82 8.636e-05 8.636e-05 8.636e-05 0.01% FabArrayBase::FB::FB() 56 8.576e-05 8.576e-05 8.576e-05 0.01% Castro::initialize_advance() 10 8.544e-05 8.544e-05 8.544e-05 0.01% Castro::Castro() 1 8.383e-05 8.383e-05 8.383e-05 0.01% MLMG::mgVcycle_down::3 82 8.118e-05 8.118e-05 8.118e-05 0.01% Castro::expand_state() 10 7.974e-05 7.974e-05 7.974e-05 0.01% MLMG::mgVcycle_down::4 82 7.904e-05 7.904e-05 7.904e-05 0.01% Castro::clean_state() 62 7.731e-05 7.731e-05 7.731e-05 0.01% MLMG::actualBottomSolve() 82 7.369e-05 7.369e-05 7.369e-05 0.01% MLMG::solve() 11 6.847e-05 6.847e-05 6.847e-05 0.01% MLMG::mgVcycle_up::4 82 6.822e-05 6.822e-05 6.822e-05 0.01% Castro::initialize_do_advance() 10 6.714e-05 6.714e-05 6.714e-05 0.01% MLMG::oneIter() 82 5.89e-05 5.89e-05 5.89e-05 0.01% MLMG::mgVcycle_up::0 82 5.809e-05 5.809e-05 5.809e-05 0.01% MLMG::mgVcycle_up::3 82 5.508e-05 5.508e-05 5.508e-05 0.01% MLMG::mgVcycle_up::1 82 5.347e-05 5.347e-05 5.347e-05 0.01% MLMG::mgVcycle_up::2 82 5.152e-05 5.152e-05 5.152e-05 0.01% MLCellLinOp::solutionResidual() 93 4.718e-05 4.718e-05 4.718e-05 0.01% StateData::define() 4 4.704e-05 4.704e-05 4.704e-05 0.01% Castro::swap_state_time_levels() 10 4.302e-05 4.302e-05 4.302e-05 0.00% MLMG::computeResidual() 82 4.035e-05 4.035e-05 4.035e-05 0.00% Castro::construct_new_source() 50 3.725e-05 3.725e-05 3.725e-05 0.00% Castro::enforce_consistent_e() 1 3.613e-05 3.613e-05 3.613e-05 0.00% MLMG::mgVcycle_bottom 82 3.446e-05 3.446e-05 3.446e-05 0.00% Castro::finalize_do_advance() 10 3.39e-05 3.39e-05 3.39e-05 0.00% MLPoisson::define() 11 3.311e-05 3.311e-05 3.311e-05 0.00% Castro::post_timestep() 10 3.241e-05 3.241e-05 3.241e-05 0.00% MultiFab::Add() 82 3.157e-05 3.157e-05 3.157e-05 0.00% FillPatchSingleLevel 41 3.086e-05 3.086e-05 3.086e-05 0.00% Castro::initMFs() 1 3.053e-05 3.053e-05 3.053e-05 0.00% Amr::writeSmallPlotFile() 1 2.622e-05 2.622e-05 2.622e-05 0.00% makeSFC 55 2.602e-05 2.602e-05 2.602e-05 0.00% Castro::construct_new_gravity() 10 2.534e-05 2.534e-05 2.534e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.511e-05 2.511e-05 2.511e-05 0.00% MLLinOp::define() 11 2.51e-05 2.51e-05 2.51e-05 0.00% Castro::buildMetrics() 1 2.356e-05 2.356e-05 2.356e-05 0.00% Castro::create_source_corrector() 10 2.207e-05 2.207e-05 2.207e-05 0.00% Amr::defBaseLevel() 1 2.028e-05 2.028e-05 2.028e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.005e-05 2.005e-05 2.005e-05 0.00% Amr::FinalizeInit() 1 1.999e-05 1.999e-05 1.999e-05 0.00% Castro::construct_old_source() 50 1.88e-05 1.88e-05 1.88e-05 0.00% Castro::do_new_sources() 10 1.61e-05 1.61e-05 1.61e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.57e-05 1.57e-05 1.57e-05 0.00% DistributionMapping::Distribute() 56 1.532e-05 1.532e-05 1.532e-05 0.00% Castro::do_old_sources() 10 1.519e-05 1.519e-05 1.519e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.506e-05 1.506e-05 1.506e-05 0.00% Castro::check_for_nan() 20 1.122e-05 1.122e-05 1.122e-05 0.00% Castro::construct_old_gravity() 10 1.036e-05 1.036e-05 1.036e-05 0.00% Castro::apply_source_to_state() 20 1.017e-05 1.017e-05 1.017e-05 0.00% Gravity::swapTimeLevels() 10 9.693e-06 9.693e-06 9.693e-06 0.00% Castro::post_init() 1 9.163e-06 9.163e-06 9.163e-06 0.00% Gravity::actual_multilevel_solve() 1 9.079e-06 9.079e-06 9.079e-06 0.00% MLPoisson::prepareForSolve() 11 8.849e-06 8.849e-06 8.849e-06 0.00% Amr::initSubcycle() 1 8.813e-06 8.813e-06 8.813e-06 0.00% MLMG::computeMLResidual() 11 8.741e-06 8.741e-06 8.741e-06 0.00% MLMG::getGradSolution() 11 7.131e-06 7.131e-06 7.131e-06 0.00% Castro::computeNewDt() 9 6.596e-06 6.596e-06 6.596e-06 0.00% Amr::InitializeInit() 1 5.953e-06 5.953e-06 5.953e-06 0.00% MultiFab::Copy() 11 5.365e-06 5.365e-06 5.365e-06 0.00% Castro::retry_advance_ctu() 10 4.172e-06 4.172e-06 4.172e-06 0.00% AmrLevel::checkPointPost() 3 3.983e-06 3.983e-06 3.983e-06 0.00% Gravity::set_mass_offset() 11 3.956e-06 3.956e-06 3.956e-06 0.00% AmrLevel::checkPointPre() 3 3.879e-06 3.879e-06 3.879e-06 0.00% MLMG::MLResNormInf() 11 3.587e-06 3.587e-06 3.587e-06 0.00% Castro::computeInitialDt() 2 3.362e-06 3.362e-06 3.362e-06 0.00% Castro::FluxRegCrseInit 10 2.71e-06 2.71e-06 2.71e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.672e-06 2.672e-06 2.672e-06 0.00% Amr::init() 1 2.368e-06 2.368e-06 2.368e-06 0.00% Castro::FluxRegFineAdd() 10 2.338e-06 2.338e-06 2.338e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.018e-06 2.018e-06 2.018e-06 0.00% Castro::post_regrid() 1 1.262e-06 1.262e-06 1.262e-06 0.00% Amr::initialInit() 1 9.66e-07 9.66e-07 9.66e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.888 0.888 0.888 100.00% Amr::coarseTimeStep() 10 0.7174 0.7174 0.7174 80.79% Amr::timeStep() 10 0.5724 0.5724 0.5724 64.46% Castro::advance() 10 0.5661 0.5661 0.5661 63.75% Castro::subcycle_advance_ctu() 10 0.5548 0.5548 0.5548 62.48% Castro::do_advance_ctu() 10 0.5546 0.5546 0.5546 62.46% Gravity::solve_phi_with_mlmg() 11 0.3231 0.3231 0.3231 36.39% Gravity::actual_solve_with_mlmg() 11 0.3143 0.3143 0.3143 35.39% Castro::construct_new_gravity() 10 0.2935 0.2935 0.2935 33.05% MLMG::solve() 11 0.2906 0.2906 0.2906 32.73% Gravity::solve_for_phi() 10 0.2779 0.2779 0.2779 31.30% MLMG::oneIter() 82 0.2753 0.2753 0.2753 31.01% MLMG::mgVcycle() 82 0.2735 0.2735 0.2735 30.80% VisMF::Write(FabArray) 11 0.2095 0.2095 0.2095 23.59% Amr::checkPoint() 3 0.1958 0.1958 0.1958 22.05% Castro::construct_ctu_hydro_source() 10 0.1871 0.1871 0.1871 21.08% AmrLevel::checkPoint() 3 0.1504 0.1504 0.1504 16.94% StateData::checkPoint() 12 0.1503 0.1503 0.1503 16.93% MLCellLinOp::smooth() 1640 0.1389 0.1389 0.1389 15.65% Amr::init() 1 0.1387 0.1387 0.1387 15.62% MLCellLinOp::applyBC() 4433 0.09841 0.09841 0.09841 11.08% MLMG::mgVcycle_bottom 82 0.08486 0.08486 0.08486 9.56% MLMG::actualBottomSolve() 82 0.08482 0.08482 0.08482 9.55% MLCGSolver::bicgstab 82 0.084 0.084 0.084 9.46% MLPoisson::Fsmooth() 3280 0.06484 0.06484 0.06484 7.30% Amr::writePlotFile() 2 0.06225 0.06225 0.06225 7.01% Amr::initialInit() 1 0.05317 0.05317 0.05317 5.99% Amr::FinalizeInit() 1 0.04882 0.04882 0.04882 5.50% Castro::post_init() 1 0.04753 0.04753 0.04753 5.35% Gravity::multilevel_solve_for_new_phi() 1 0.04565 0.04565 0.04565 5.14% Gravity::actual_multilevel_solve() 1 0.04563 0.04563 0.04563 5.14% FillPatchIterator::Initialize 41 0.04281 0.04281 0.04281 4.82% FillPatchSingleLevel 41 0.04117 0.04117 0.04117 4.64% Castro::clean_state() 62 0.04041 0.04041 0.04041 4.55% MLCellLinOp::apply() 1142 0.03732 0.03732 0.03732 4.20% StateDataPhysBCFunct::() 41 0.0371 0.0371 0.0371 4.18% MLMG::mgVcycle_down::0 82 0.03585 0.03585 0.03585 4.04% MLMG::mgVcycle_up::0 82 0.03085 0.03085 0.03085 3.47% StateData::FillBoundary(geom) 328 0.02455 0.02455 0.02455 2.76% amrex::Dot() 1114 0.02306 0.02306 0.02306 2.60% MLCellLinOp::correctionResidual() 492 0.0218 0.0218 0.0218 2.45% Castro::computeTemp() 63 0.01932 0.01932 0.01932 2.18% MLPoisson::define() 11 0.01923 0.01923 0.01923 2.17% MLMG:computeResOfCorrection() 410 0.01882 0.01882 0.01882 2.12% Castro::initialize_do_advance() 10 0.0183 0.0183 0.0183 2.06% MLMG::mgVcycle_down::1 82 0.0182 0.0182 0.0182 2.05% MLMG::mgVcycle_down::2 82 0.0177 0.0177 0.0177 1.99% Gravity::get_new_grav_vector() 11 0.01712 0.01712 0.01712 1.93% MLMG::mgVcycle_down::3 82 0.01685 0.01685 0.01685 1.90% amrex::Copy() 1029 0.01615 0.01615 0.01615 1.82% MLMG::mgVcycle_down::4 82 0.01606 0.01606 0.01606 1.81% FabArray::FillBoundary() 4023 0.01592 0.01592 0.01592 1.79% Castro::construct_old_gravity() 10 0.01541 0.01541 0.01541 1.74% Gravity::get_old_grav_vector() 10 0.0154 0.0154 0.0154 1.73% CGSolver::sxay() 1586 0.01516 0.01516 0.01516 1.71% FillBoundary_nowait() 4023 0.01515 0.01515 0.01515 1.71% FabArray::LinComb() 1586 0.01481 0.01481 0.01481 1.67% FabArray::ParallelCopy() 861 0.01469 0.01469 0.01469 1.65% FabArray::setVal() 1144 0.01458 0.01458 0.01458 1.64% FabArray::ParallelCopy_nowait() 861 0.01444 0.01444 0.01444 1.63% Castro::normalize_species() 62 0.01422 0.01422 0.01422 1.60% MLCGSolver::ParallelAllReduce 1514 0.01374 0.01374 0.01374 1.55% MLMG::mgVcycle_up::2 82 0.01361 0.01361 0.01361 1.53% MLCellLinOp::defineAuxData() 11 0.01347 0.01347 0.01347 1.52% MLMG::mgVcycle_up::1 82 0.01343 0.01343 0.01343 1.51% FabArray::norminf() 639 0.01337 0.01337 0.01337 1.51% MLMG::addInterpCorrection() 410 0.01322 0.01322 0.01322 1.49% MLMG::mgVcycle_up::3 82 0.01293 0.01293 0.01293 1.46% MLMG::mgVcycle_up::4 82 0.01282 0.01282 0.01282 1.44% amrex::average_down 410 0.01236 0.01236 0.01236 1.39% MLPoisson::Fapply() 1142 0.01204 0.01204 0.01204 1.36% Castro::expand_state() 10 0.01165 0.01165 0.01165 1.31% Castro::do_new_sources() 10 0.0115 0.0115 0.0115 1.30% Castro::initialize_advance() 10 0.01051 0.01051 0.01051 1.18% Castro::do_old_sources() 10 0.01042 0.01042 0.01042 1.17% Gravity::fill_multipole_BCs() 11 0.008646 0.008646 0.008646 0.97% MLCellLinOp::solutionResidual() 93 0.007319 0.007319 0.007319 0.82% FabArray::Xpay() 585 0.006756 0.006756 0.006756 0.76% MLMG::computeResidual() 82 0.006328 0.006328 0.006328 0.71% Castro::enforce_min_density() 62 0.006281 0.006281 0.006281 0.71% Castro::post_timestep() 10 0.006221 0.006221 0.006221 0.70% Castro::reset_internal_energy(MultiFab) 63 0.00621 0.00621 0.00621 0.70% MLMG::prepareForSolve() 11 0.005511 0.005511 0.005511 0.62% MLCellLinOp::defineBC() 11 0.005296 0.005296 0.005296 0.60% Castro::estTimeStep() 21 0.005084 0.005084 0.005084 0.57% BndryData::define() 11 0.005016 0.005016 0.005016 0.56% Amr::InitializeInit() 1 0.004347 0.004347 0.004347 0.49% Amr::defBaseLevel() 1 0.004341 0.004341 0.004341 0.49% Castro::initData() 1 0.003809 0.003809 0.003809 0.43% Castro::construct_new_source() 50 0.003317 0.003317 0.003317 0.37% Castro::construct_new_gravity_source() 10 0.00328 0.00328 0.00328 0.37% Castro::construct_old_source() 50 0.00258 0.00258 0.00258 0.29% amrex::Add() 164 0.002571 0.002571 0.002571 0.29% Castro::construct_old_gravity_source() 10 0.002561 0.002561 0.002561 0.29% Castro::computeNewDt() 9 0.002422 0.002422 0.002422 0.27% MLMG::ResNormInf() 93 0.002111 0.002111 0.002111 0.24% Castro::apply_source_to_state() 20 0.001829 0.001829 0.001829 0.21% FabArray::Saxpy() 20 0.001819 0.001819 0.001819 0.20% MultiFab::Add() 82 0.001734 0.001734 0.001734 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001717 0.001717 0.001717 0.19% Castro::reset_internal_energy(Fab) 504 0.001647 0.001647 0.001647 0.19% MLCellLinOp::setLevelBC() 11 0.001585 0.001585 0.001585 0.18% MLMG::getGradSolution() 11 0.00146 0.00146 0.00146 0.16% MLCellLinOp::compGrad() 11 0.001453 0.001453 0.001453 0.16% FabArrayBase::getCPC() 1323 0.001447 0.001447 0.001447 0.16% Castro::enforce_speed_limit() 62 0.001396 0.001396 0.001396 0.16% FabArray::setDomainBndry() 41 0.001345 0.001345 0.001345 0.15% FabArray::mult() 43 0.001333 0.001333 0.001333 0.15% MLPoisson::prepareForSolve() 11 0.001221 0.001221 0.001221 0.14% Castro::check_for_nan() 20 0.001213 0.001213 0.001213 0.14% MLCellLinOp::prepareForSolve() 11 0.001212 0.001212 0.001212 0.14% MultiFab::contains_nan() 20 0.001201 0.001201 0.001201 0.14% Castro::post_regrid() 1 0.001121 0.001121 0.001121 0.13% MLMG::computeMLResidual() 11 0.001041 0.001041 0.001041 0.12% Castro::computeInitialDt() 2 0.0008938 0.0008938 0.0008938 0.10% Gravity::update_max_rhs() 11 0.0008512 0.0008512 0.0008512 0.10% FabArrayBase::getFB() 4023 0.0006886 0.0006886 0.0006886 0.08% FabArrayBase::CPC::define() 454 0.0006876 0.0006876 0.0006876 0.08% Castro::finalize_advance() 10 0.0006279 0.0006279 0.0006279 0.07% Amr::InitAmr() 1 0.0005051 0.0005051 0.0005051 0.06% Gravity::swapTimeLevels() 10 0.000447 0.000447 0.000447 0.05% Castro::Castro() 1 0.0004457 0.0004457 0.0004457 0.05% MLLinOp::define() 11 0.0004388 0.0004388 0.0004388 0.05% MLLinOp::defineGrids() 11 0.0004137 0.0004137 0.0004137 0.05% MultiFab::Copy() 11 0.0003398 0.0003398 0.0003398 0.04% MLMG::MLResNormInf() 11 0.0002836 0.0002836 0.0002836 0.03% MultiFab::max() 11 0.0002583 0.0002583 0.0002583 0.03% MLMG::MLRhsNormInf() 11 0.000219 0.000219 0.000219 0.02% Castro::buildMetrics() 1 0.0001666 0.0001666 0.0001666 0.02% FabArrayBase::FB::FB() 56 8.576e-05 8.576e-05 8.576e-05 0.01% AmrLevel::AmrLevel(dm) 1 6.274e-05 6.274e-05 6.274e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.508e-05 5.508e-05 5.508e-05 0.01% StateData::define() 4 4.704e-05 4.704e-05 4.704e-05 0.01% Castro::swap_state_time_levels() 10 4.302e-05 4.302e-05 4.302e-05 0.00% makeSFC 55 4.002e-05 4.002e-05 4.002e-05 0.00% Castro::enforce_consistent_e() 1 3.613e-05 3.613e-05 3.613e-05 0.00% Castro::finalize_do_advance() 10 3.39e-05 3.39e-05 3.39e-05 0.00% Castro::initMFs() 1 3.053e-05 3.053e-05 3.053e-05 0.00% Amr::writeSmallPlotFile() 1 2.622e-05 2.622e-05 2.622e-05 0.00% Castro::create_source_corrector() 10 2.207e-05 2.207e-05 2.207e-05 0.00% DistributionMapping::Distribute() 56 1.532e-05 1.532e-05 1.532e-05 0.00% Amr::initSubcycle() 1 8.813e-06 8.813e-06 8.813e-06 0.00% Castro::retry_advance_ctu() 10 4.172e-06 4.172e-06 4.172e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4e-06 4e-06 4e-06 0.00% AmrLevel::checkPointPost() 3 3.983e-06 3.983e-06 3.983e-06 0.00% Gravity::set_mass_offset() 11 3.956e-06 3.956e-06 3.956e-06 0.00% AmrLevel::checkPointPre() 3 3.879e-06 3.879e-06 3.879e-06 0.00% Castro::FluxRegCrseInit 10 2.71e-06 2.71e-06 2.71e-06 0.00% Castro::FluxRegFineAdd() 10 2.338e-06 2.338e-06 2.338e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.018e-06 2.018e-06 2.018e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2464 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-1-g4a53367b19e1) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-1-g4a53367b19e1) initialized Starting run at 13:58:23 UTC on 2022-12-02. Successfully read inputs file ... Castro git describe: 22.12 AMReX git describe: 22.12-1-g4a53367b1 Microphysics git describe: 22.12 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.541632843 Restart time = 0.048398869 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.053909312 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050929292 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.052990146 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060716891 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063659237 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032857773 seconds Ending run at 13:58:24 UTC on 2022-12-02. Run time = 0.364439215 Run time without initialization = 0.315459265 Average number of zones advanced per microsecond: 4.155 Average number of zones advanced per microsecond per rank: 4.155 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3645 ... 0.3645 ... 0.3645 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0867 0.0867 0.0867 23.80% VisMF::Read() 3 0.04044 0.04044 0.04044 11.09% MLCellLinOp::applyBC() 1946 0.03556 0.03556 0.03556 9.76% VisMF::Write(FabArray) 1 0.03128 0.03128 0.03128 8.58% MLPoisson::Fsmooth() 1440 0.02774 0.02774 0.02774 7.61% StateData::FillBoundary(geom) 160 0.01199 0.01199 0.01199 3.29% amrex::Dot() 484 0.009814 0.009814 0.009814 2.69% amrex::Copy() 463 0.007489 0.007489 0.007489 2.05% Castro::computeTemp() 30 0.007267 0.007267 0.007267 1.99% FabArray::setVal() 537 0.006891 0.006891 0.006891 1.89% Castro::normalize_species() 30 0.006755 0.006755 0.006755 1.85% MLCellLinOp::defineAuxData() 6 0.006364 0.006364 0.006364 1.75% FillBoundary_nowait() 1766 0.006314 0.006314 0.006314 1.73% FabArray::LinComb() 690 0.006278 0.006278 0.006278 1.72% FabArray::ParallelCopy_nowait() 380 0.006136 0.006136 0.006136 1.68% StateDataPhysBCFunct::() 20 0.006035 0.006035 0.006035 1.66% FabArray::norminf() 278 0.005679 0.005679 0.005679 1.56% MLPoisson::Fapply() 500 0.005171 0.005171 0.005171 1.42% Gravity::fill_multipole_BCs() 6 0.004589 0.004589 0.004589 1.26% Amr::restart() 1 0.003607 0.003607 0.003607 0.99% MLMG::addInterpCorrection() 180 0.003377 0.003377 0.003377 0.93% amrex::average_down 180 0.003024 0.003024 0.003024 0.83% FabArray::Xpay() 258 0.002913 0.002913 0.002913 0.80% Castro::reset_internal_energy(MultiFab) 30 0.002253 0.002253 0.002253 0.62% Castro::enforce_min_density() 30 0.002196 0.002196 0.002196 0.60% BndryData::define() 6 0.002159 0.002159 0.002159 0.59% Castro::estTimeStep() 10 0.002111 0.002111 0.002111 0.58% Castro::construct_new_gravity_source() 5 0.001656 0.001656 0.001656 0.45% Amr::writePlotFile() 1 0.001406 0.001406 0.001406 0.39% MLCGSolver::bicgstab 36 0.001389 0.001389 0.001389 0.38% Castro::do_advance_ctu() 5 0.001312 0.001312 0.001312 0.36% Castro::construct_old_gravity_source() 5 0.001296 0.001296 0.001296 0.36% amrex::Add() 72 0.001101 0.001101 0.001101 0.30% MLMG::ResNormInf() 42 0.0009388 0.0009388 0.0009388 0.26% FabArray::Saxpy() 10 0.0009204 0.0009204 0.0009204 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000912 0.000912 0.000912 0.25% MLCellLinOp::setLevelBC() 6 0.000838 0.000838 0.000838 0.23% Castro::reset_internal_energy(Fab) 240 0.0007853 0.0007853 0.0007853 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007224 0.0007224 0.0007224 0.20% Castro::enforce_speed_limit() 30 0.000693 0.000693 0.000693 0.19% FabArray::setDomainBndry() 20 0.0006733 0.0006733 0.0006733 0.18% MLCellLinOp::prepareForSolve() 6 0.0006546 0.0006546 0.0006546 0.18% MLMG::prepareForSolve() 6 0.0006513 0.0006513 0.0006513 0.18% FabArray::mult() 22 0.000648 0.000648 0.000648 0.18% MultiFab::contains_nan() 10 0.0005965 0.0005965 0.0005965 0.16% MLCellLinOp::compGrad() 6 0.0004976 0.0004976 0.0004976 0.14% MLCellLinOp::smooth() 720 0.0004273 0.0004273 0.0004273 0.12% Amr::InitAmr() 1 0.0004059 0.0004059 0.0004059 0.11% FabArrayBase::CPC::define() 244 0.0003984 0.0003984 0.0003984 0.11% FabArrayBase::getCPC() 632 0.0003498 0.0003498 0.0003498 0.10% FabArray::FillBoundary() 1766 0.0003319 0.0003319 0.0003319 0.09% Gravity::get_old_grav_vector() 5 0.0003044 0.0003044 0.0003044 0.08% Gravity::get_new_grav_vector() 5 0.0002796 0.0002796 0.0002796 0.08% main() 1 0.0002633 0.0002633 0.0002633 0.07% FabArrayBase::getFB() 1766 0.0002525 0.0002525 0.0002525 0.07% MLCellLinOp::apply() 500 0.0002009 0.0002009 0.0002009 0.06% Amr::coarseTimeStep() 5 0.0001655 0.0001655 0.0001655 0.05% MLLinOp::defineGrids() 6 0.0001641 0.0001641 0.0001641 0.05% MLMG::mgVcycle() 36 0.0001638 0.0001638 0.0001638 0.04% CGSolver::sxay() 690 0.0001556 0.0001556 0.0001556 0.04% MLCellLinOp::defineBC() 6 0.0001443 0.0001443 0.0001443 0.04% FillPatchIterator::Initialize 20 0.0001381 0.0001381 0.0001381 0.04% MultiFab::max() 6 0.0001349 0.0001349 0.0001349 0.04% MLCGSolver::ParallelAllReduce 659 0.0001282 0.0001282 0.0001282 0.04% Castro::subcycle_advance_ctu() 5 0.0001174 0.0001174 0.0001174 0.03% MLMG::MLRhsNormInf() 6 0.0001162 0.0001162 0.0001162 0.03% FabArray::ParallelCopy() 380 0.0001149 0.0001149 0.0001149 0.03% MLCellLinOp::correctionResidual() 216 8.46e-05 8.46e-05 8.46e-05 0.02% AmrLevel::restart() 1 8.378e-05 8.378e-05 8.378e-05 0.02% Gravity::update_max_rhs() 6 7.642e-05 7.642e-05 7.642e-05 0.02% Amr::timeStep() 5 7.545e-05 7.545e-05 7.545e-05 0.02% StateData::restartDoit() 4 7.213e-05 7.213e-05 7.213e-05 0.02% Castro::advance() 5 6.788e-05 6.788e-05 6.788e-05 0.02% MLMG:computeResOfCorrection() 180 6.168e-05 6.168e-05 6.168e-05 0.02% Gravity::solve_for_phi() 5 6.083e-05 6.083e-05 6.083e-05 0.02% FabArrayBase::FB::FB() 26 5.837e-05 5.837e-05 5.837e-05 0.02% Castro::initialize_do_advance() 5 4.919e-05 4.919e-05 4.919e-05 0.01% Castro::expand_state() 5 4.605e-05 4.605e-05 4.605e-05 0.01% Castro::finalize_advance() 5 4.428e-05 4.428e-05 4.428e-05 0.01% MLMG::mgVcycle_down::0 36 4.394e-05 4.394e-05 4.394e-05 0.01% MLMG::mgVcycle_down::1 36 3.984e-05 3.984e-05 3.984e-05 0.01% Castro::initialize_advance() 5 3.952e-05 3.952e-05 3.952e-05 0.01% Castro::clean_state() 30 3.838e-05 3.838e-05 3.838e-05 0.01% Castro::construct_old_source() 25 3.609e-05 3.609e-05 3.609e-05 0.01% MLMG::mgVcycle_down::2 36 3.495e-05 3.495e-05 3.495e-05 0.01% MLMG::mgVcycle_down::4 36 3.406e-05 3.406e-05 3.406e-05 0.01% Castro::create_source_corrector() 5 3.373e-05 3.373e-05 3.373e-05 0.01% MLMG::mgVcycle_down::3 36 3.297e-05 3.297e-05 3.297e-05 0.01% Castro::buildMetrics() 1 3.28e-05 3.28e-05 3.28e-05 0.01% Castro::construct_new_source() 25 3.212e-05 3.212e-05 3.212e-05 0.01% MLMG::actualBottomSolve() 36 3.151e-05 3.151e-05 3.151e-05 0.01% MLMG::mgVcycle_up::4 36 3.149e-05 3.149e-05 3.149e-05 0.01% Castro::post_restart() 1 3.13e-05 3.13e-05 3.13e-05 0.01% MLMG::solve() 6 3.086e-05 3.086e-05 3.086e-05 0.01% MLMG::mgVcycle_up::0 36 2.944e-05 2.944e-05 2.944e-05 0.01% Castro::initMFs() 1 2.939e-05 2.939e-05 2.939e-05 0.01% Castro::swap_state_time_levels() 5 2.913e-05 2.913e-05 2.913e-05 0.01% Amr::writeSmallPlotFile() 1 2.795e-05 2.795e-05 2.795e-05 0.01% MLMG::oneIter() 36 2.578e-05 2.578e-05 2.578e-05 0.01% MLPoisson::define() 6 2.449e-05 2.449e-05 2.449e-05 0.01% MLMG::mgVcycle_up::3 36 2.313e-05 2.313e-05 2.313e-05 0.01% MLMG::mgVcycle_up::2 36 2.301e-05 2.301e-05 2.301e-05 0.01% MLMG::mgVcycle_up::1 36 2.174e-05 2.174e-05 2.174e-05 0.01% MLCellLinOp::solutionResidual() 42 2.089e-05 2.089e-05 2.089e-05 0.01% MLLinOp::define() 6 2.057e-05 2.057e-05 2.057e-05 0.01% Castro::construct_old_gravity() 5 2.041e-05 2.041e-05 2.041e-05 0.01% Castro::finalize_do_advance() 5 2.034e-05 2.034e-05 2.034e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.778e-05 1.778e-05 1.778e-05 0.00% MLMG::computeResidual() 36 1.684e-05 1.684e-05 1.684e-05 0.00% MLMG::mgVcycle_bottom 36 1.631e-05 1.631e-05 1.631e-05 0.00% FillPatchSingleLevel 20 1.501e-05 1.501e-05 1.501e-05 0.00% Castro::construct_new_gravity() 5 1.474e-05 1.474e-05 1.474e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.332e-05 1.332e-05 1.332e-05 0.00% makeSFC 30 1.3e-05 1.3e-05 1.3e-05 0.00% MultiFab::Add() 36 1.169e-05 1.169e-05 1.169e-05 0.00% DistributionMapping::Distribute() 31 8.718e-06 8.718e-06 8.718e-06 0.00% Amr::initSubcycle() 1 8.701e-06 8.701e-06 8.701e-06 0.00% Castro::do_new_sources() 5 8.531e-06 8.531e-06 8.531e-06 0.00% Gravity::actual_multilevel_solve() 1 8.288e-06 8.288e-06 8.288e-06 0.00% Castro::do_old_sources() 5 7.912e-06 7.912e-06 7.912e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.419e-06 7.419e-06 7.419e-06 0.00% Castro::check_for_nan() 10 5.515e-06 5.515e-06 5.515e-06 0.00% Castro::post_timestep() 5 5.402e-06 5.402e-06 5.402e-06 0.00% Castro::apply_source_to_state() 10 5.04e-06 5.04e-06 5.04e-06 0.00% MLPoisson::prepareForSolve() 6 4.735e-06 4.735e-06 4.735e-06 0.00% MLMG::computeMLResidual() 6 4.515e-06 4.515e-06 4.515e-06 0.00% Gravity::swapTimeLevels() 5 4.364e-06 4.364e-06 4.364e-06 0.00% Castro::computeNewDt() 5 3.629e-06 3.629e-06 3.629e-06 0.00% MLMG::getGradSolution() 6 3.49e-06 3.49e-06 3.49e-06 0.00% MultiFab::Copy() 6 2.909e-06 2.909e-06 2.909e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.768e-06 2.768e-06 2.768e-06 0.00% Gravity::set_mass_offset() 6 2.269e-06 2.269e-06 2.269e-06 0.00% MLMG::MLResNormInf() 6 2.139e-06 2.139e-06 2.139e-06 0.00% Castro::FluxRegCrseInit 5 1.572e-06 1.572e-06 1.572e-06 0.00% Castro::retry_advance_ctu() 5 1.554e-06 1.554e-06 1.554e-06 0.00% Castro::FluxRegFineAdd() 5 1.193e-06 1.193e-06 1.193e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.09e-06 1.09e-06 1.09e-06 0.00% Amr::init() 1 1.073e-06 1.073e-06 1.073e-06 0.00% AmrLevel::AmrLevel() 1 7.72e-07 7.72e-07 7.72e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3645 0.3645 0.3645 100.00% Amr::coarseTimeStep() 5 0.2824 0.2824 0.2824 77.47% Amr::timeStep() 5 0.2809 0.2809 0.2809 77.06% Castro::advance() 5 0.2777 0.2777 0.2777 76.19% Castro::subcycle_advance_ctu() 5 0.2715 0.2715 0.2715 74.48% Castro::do_advance_ctu() 5 0.2713 0.2713 0.2713 74.45% Castro::construct_new_gravity() 5 0.1469 0.1469 0.1469 40.30% Gravity::solve_phi_with_mlmg() 6 0.1427 0.1427 0.1427 39.14% Gravity::solve_for_phi() 5 0.1391 0.1391 0.1391 38.15% Gravity::actual_solve_with_mlmg() 6 0.1378 0.1378 0.1378 37.82% MLMG::solve() 6 0.1253 0.1253 0.1253 34.38% MLMG::oneIter() 36 0.118 0.118 0.118 32.36% MLMG::mgVcycle() 36 0.1172 0.1172 0.1172 32.15% Castro::construct_ctu_hydro_source() 5 0.08674 0.08674 0.08674 23.80% MLCellLinOp::smooth() 720 0.05957 0.05957 0.05957 16.35% Amr::init() 1 0.04844 0.04844 0.04844 13.29% Amr::restart() 1 0.04844 0.04844 0.04844 13.29% MLCellLinOp::applyBC() 1946 0.04251 0.04251 0.04251 11.66% AmrLevel::restart() 1 0.04065 0.04065 0.04065 11.15% StateData::restartDoit() 4 0.04056 0.04056 0.04056 11.13% VisMF::Read() 3 0.04044 0.04044 0.04044 11.09% MLMG::mgVcycle_bottom 36 0.03616 0.03616 0.03616 9.92% MLMG::actualBottomSolve() 36 0.03614 0.03614 0.03614 9.92% MLCGSolver::bicgstab 36 0.03579 0.03579 0.03579 9.82% Amr::writePlotFile() 1 0.03294 0.03294 0.03294 9.04% VisMF::Write(FabArray) 1 0.03128 0.03128 0.03128 8.58% MLPoisson::Fsmooth() 1440 0.02774 0.02774 0.02774 7.61% FillPatchIterator::Initialize 20 0.02084 0.02084 0.02084 5.72% FillPatchSingleLevel 20 0.02003 0.02003 0.02003 5.50% Castro::clean_state() 30 0.01999 0.01999 0.01999 5.48% StateDataPhysBCFunct::() 20 0.01803 0.01803 0.01803 4.95% MLCellLinOp::apply() 500 0.0162 0.0162 0.0162 4.44% MLMG::mgVcycle_down::0 36 0.01554 0.01554 0.01554 4.26% MLMG::mgVcycle_up::0 36 0.01336 0.01336 0.01336 3.67% StateData::FillBoundary(geom) 160 0.01199 0.01199 0.01199 3.29% Castro::computeTemp() 30 0.01031 0.01031 0.01031 2.83% MLPoisson::define() 6 0.01019 0.01019 0.01019 2.80% Castro::initialize_do_advance() 5 0.009892 0.009892 0.009892 2.71% amrex::Dot() 484 0.009814 0.009814 0.009814 2.69% MLCellLinOp::correctionResidual() 216 0.009409 0.009409 0.009409 2.58% MLMG:computeResOfCorrection() 180 0.008119 0.008119 0.008119 2.23% MLMG::mgVcycle_down::1 36 0.007785 0.007785 0.007785 2.14% Gravity::get_new_grav_vector() 5 0.007692 0.007692 0.007692 2.11% Castro::construct_old_gravity() 5 0.007601 0.007601 0.007601 2.09% Gravity::get_old_grav_vector() 5 0.007581 0.007581 0.007581 2.08% MLMG::mgVcycle_down::2 36 0.007568 0.007568 0.007568 2.08% amrex::Copy() 463 0.007489 0.007489 0.007489 2.05% MLMG::mgVcycle_down::3 36 0.007169 0.007169 0.007169 1.97% MLCellLinOp::defineAuxData() 6 0.007105 0.007105 0.007105 1.95% FabArray::FillBoundary() 1766 0.006957 0.006957 0.006957 1.91% MLMG::mgVcycle_down::4 36 0.006892 0.006892 0.006892 1.89% FabArray::setVal() 537 0.006891 0.006891 0.006891 1.89% Castro::normalize_species() 30 0.006755 0.006755 0.006755 1.85% FillBoundary_nowait() 1766 0.006625 0.006625 0.006625 1.82% FabArray::ParallelCopy() 380 0.006624 0.006624 0.006624 1.82% FabArray::ParallelCopy_nowait() 380 0.006509 0.006509 0.006509 1.79% CGSolver::sxay() 690 0.006433 0.006433 0.006433 1.77% Castro::do_new_sources() 5 0.006389 0.006389 0.006389 1.75% FabArray::LinComb() 690 0.006278 0.006278 0.006278 1.72% Castro::expand_state() 5 0.006063 0.006063 0.006063 1.66% MLCGSolver::ParallelAllReduce 659 0.005888 0.005888 0.005888 1.62% Castro::initialize_advance() 5 0.005839 0.005839 0.005839 1.60% MLMG::mgVcycle_up::2 36 0.005827 0.005827 0.005827 1.60% MLMG::mgVcycle_up::1 36 0.005748 0.005748 0.005748 1.58% MLMG::addInterpCorrection() 180 0.005699 0.005699 0.005699 1.56% FabArray::norminf() 278 0.005679 0.005679 0.005679 1.56% MLMG::mgVcycle_up::4 36 0.005493 0.005493 0.005493 1.51% MLMG::mgVcycle_up::3 36 0.005488 0.005488 0.005488 1.51% amrex::average_down 180 0.005335 0.005335 0.005335 1.46% MLPoisson::Fapply() 500 0.005171 0.005171 0.005171 1.42% Castro::do_old_sources() 5 0.004987 0.004987 0.004987 1.37% Gravity::fill_multipole_BCs() 6 0.004716 0.004716 0.004716 1.29% Castro::post_restart() 1 0.003991 0.003991 0.003991 1.10% Gravity::multilevel_solve_for_new_phi() 1 0.003864 0.003864 0.003864 1.06% Gravity::actual_multilevel_solve() 1 0.003846 0.003846 0.003846 1.06% MLCellLinOp::solutionResidual() 42 0.003284 0.003284 0.003284 0.90% Castro::post_timestep() 5 0.003103 0.003103 0.003103 0.85% Castro::reset_internal_energy(MultiFab) 30 0.003038 0.003038 0.003038 0.83% MLMG::prepareForSolve() 6 0.002955 0.002955 0.002955 0.81% FabArray::Xpay() 258 0.002913 0.002913 0.002913 0.80% MLCellLinOp::defineBC() 6 0.002849 0.002849 0.002849 0.78% MLMG::computeResidual() 36 0.002735 0.002735 0.002735 0.75% BndryData::define() 6 0.002704 0.002704 0.002704 0.74% Castro::enforce_min_density() 30 0.002196 0.002196 0.002196 0.60% Castro::estTimeStep() 10 0.002111 0.002111 0.002111 0.58% Castro::construct_new_source() 25 0.001688 0.001688 0.001688 0.46% Castro::construct_new_gravity_source() 5 0.001656 0.001656 0.001656 0.45% Castro::computeNewDt() 5 0.001341 0.001341 0.001341 0.37% Castro::construct_old_source() 25 0.001332 0.001332 0.001332 0.37% Castro::construct_old_gravity_source() 5 0.001296 0.001296 0.001296 0.36% amrex::Add() 72 0.001101 0.001101 0.001101 0.30% MLMG::ResNormInf() 42 0.0009388 0.0009388 0.0009388 0.26% Castro::apply_source_to_state() 10 0.0009255 0.0009255 0.0009255 0.25% FabArray::Saxpy() 10 0.0009204 0.0009204 0.0009204 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000912 0.000912 0.000912 0.25% MLCellLinOp::setLevelBC() 6 0.000838 0.000838 0.000838 0.23% Castro::reset_internal_energy(Fab) 240 0.0007853 0.0007853 0.0007853 0.22% MLMG::getGradSolution() 6 0.0007798 0.0007798 0.0007798 0.21% MLCellLinOp::compGrad() 6 0.0007763 0.0007763 0.0007763 0.21% FabArrayBase::getCPC() 632 0.0007483 0.0007483 0.0007483 0.21% MultiFab::Add() 36 0.0007381 0.0007381 0.0007381 0.20% Castro::enforce_speed_limit() 30 0.000693 0.000693 0.000693 0.19% FabArray::setDomainBndry() 20 0.0006733 0.0006733 0.0006733 0.18% MLPoisson::prepareForSolve() 6 0.0006593 0.0006593 0.0006593 0.18% MLCellLinOp::prepareForSolve() 6 0.0006546 0.0006546 0.0006546 0.18% FabArray::mult() 22 0.000648 0.000648 0.000648 0.18% Castro::check_for_nan() 10 0.000602 0.000602 0.000602 0.17% MultiFab::contains_nan() 10 0.0005965 0.0005965 0.0005965 0.16% MLMG::computeMLResidual() 6 0.0005706 0.0005706 0.0005706 0.16% Gravity::update_max_rhs() 6 0.0004596 0.0004596 0.0004596 0.13% Amr::InitAmr() 1 0.0004146 0.0004146 0.0004146 0.11% FabArrayBase::CPC::define() 244 0.0003984 0.0003984 0.0003984 0.11% FabArrayBase::getFB() 1766 0.0003109 0.0003109 0.0003109 0.09% Castro::finalize_advance() 5 0.0003079 0.0003079 0.0003079 0.08% Gravity::swapTimeLevels() 5 0.0002285 0.0002285 0.0002285 0.06% MLLinOp::define() 6 0.0002134 0.0002134 0.0002134 0.06% MLLinOp::defineGrids() 6 0.0001928 0.0001928 0.0001928 0.05% MultiFab::Copy() 6 0.0001847 0.0001847 0.0001847 0.05% Castro::buildMetrics() 1 0.0001574 0.0001574 0.0001574 0.04% MLMG::MLResNormInf() 6 0.0001503 0.0001503 0.0001503 0.04% MultiFab::max() 6 0.0001349 0.0001349 0.0001349 0.04% MLMG::MLRhsNormInf() 6 0.0001162 0.0001162 0.0001162 0.03% FabArrayBase::FB::FB() 26 5.837e-05 5.837e-05 5.837e-05 0.02% Castro::create_source_corrector() 5 3.373e-05 3.373e-05 3.373e-05 0.01% Castro::initMFs() 1 2.939e-05 2.939e-05 2.939e-05 0.01% Castro::swap_state_time_levels() 5 2.913e-05 2.913e-05 2.913e-05 0.01% Amr::writeSmallPlotFile() 1 2.795e-05 2.795e-05 2.795e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.761e-05 2.761e-05 2.761e-05 0.01% Castro::finalize_do_advance() 5 2.034e-05 2.034e-05 2.034e-05 0.01% makeSFC 30 2.019e-05 2.019e-05 2.019e-05 0.01% DistributionMapping::Distribute() 31 8.718e-06 8.718e-06 8.718e-06 0.00% Amr::initSubcycle() 1 8.701e-06 8.701e-06 8.701e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.295e-06 4.295e-06 4.295e-06 0.00% Gravity::set_mass_offset() 6 2.269e-06 2.269e-06 2.269e-06 0.00% Castro::FluxRegCrseInit 5 1.572e-06 1.572e-06 1.572e-06 0.00% Castro::retry_advance_ctu() 5 1.554e-06 1.554e-06 1.554e-06 0.00% Castro::FluxRegFineAdd() 5 1.193e-06 1.193e-06 1.193e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.09e-06 1.09e-06 1.09e-06 0.00% AmrLevel::AmrLevel() 1 7.72e-07 7.72e-07 7.72e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2464 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-1-g4a53367b19e1) finalized