Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-29-g329f81b889a6) initialized Starting run at 08:22:30 UTC on 2022-04-26. Successfully read inputs file ... Castro git describe: 22.04-28-g7e86dbe56 AMReX git describe: 22.04-29-g329f81b88 Microphysics git describe: 22.04-3-g3c498521 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.038151331 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022914912 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.110388546 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049485502 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.049659241 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.049623135 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.059047339 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.035382627 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.061819395 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.057515173 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.056889142 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.049079882 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.052955174 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.036084069 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.021941951 seconds Ending run at 08:22:31 UTC on 2022-04-26. Run time = 0.809469023 Run time without initialization = 0.690359061 Average number of zones advanced per microsecond: 3.797 Average number of zones advanced per microsecond per rank: 3.797 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8095 ... 0.8095 ... 0.8095 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2008 0.2008 0.2008 24.81% VisMF::Write(FabArray) 11 0.1473 0.1473 0.1473 18.20% MLCellLinOp::applyBC() 4379 0.09217 0.09217 0.09217 11.39% MLPoisson::Fsmooth() 3240 0.0587 0.0587 0.0587 7.25% FabArray::setVal() 1135 0.02589 0.02589 0.02589 3.20% StateData::FillBoundary(geom) 328 0.02287 0.02287 0.02287 2.82% MLCGSolver::bicgstab 81 0.02113 0.02113 0.02113 2.61% MultiFab::Dot() 1100 0.02074 0.02074 0.02074 2.56% FillBoundary_nowait() 3974 0.01749 0.01749 0.01749 2.16% StateDataPhysBCFunct::() 41 0.01646 0.01646 0.01646 2.03% FabArray::ParallelCopy_nowait() 851 0.01568 0.01568 0.01568 1.94% Castro::computeTemp() 63 0.0129 0.0129 0.0129 1.59% Gravity::fill_multipole_BCs() 11 0.01196 0.01196 0.01196 1.48% MultiFab::LinComb() 1566 0.01196 0.01196 0.01196 1.48% MLPoisson::Fapply() 1128 0.01039 0.01039 0.01039 1.28% MLCellLinOp::defineAuxData() 11 0.009955 0.009955 0.009955 1.23% Castro::enforce_min_density() 62 0.006962 0.006962 0.006962 0.86% MLMG::addInterpCorrection() 405 0.00678 0.00678 0.00678 0.84% amrex::average_down 405 0.006344 0.006344 0.006344 0.78% MultiFab::Xpay() 578 0.005969 0.005969 0.005969 0.74% FabArray::setDomainBndry() 41 0.005416 0.005416 0.005416 0.67% Castro::reset_internal_energy() 63 0.005406 0.005406 0.005406 0.67% Castro::do_advance_ctu() 10 0.005398 0.005398 0.005398 0.67% Castro::estTimeStep() 21 0.005317 0.005317 0.005317 0.66% Castro::expand_state() 10 0.004862 0.004862 0.004862 0.60% Castro::normalize_species() 62 0.004374 0.004374 0.004374 0.54% Amr::checkPoint() 3 0.003993 0.003993 0.003993 0.49% Castro::enforce_speed_limit() 62 0.003705 0.003705 0.003705 0.46% BndryData::define() 11 0.003569 0.003569 0.003569 0.44% Amr::writePlotFile() 2 0.003172 0.003172 0.003172 0.39% Castro::construct_new_gravity_source() 10 0.003099 0.003099 0.003099 0.38% Gravity::get_new_grav_vector() 11 0.002665 0.002665 0.002665 0.33% Castro::construct_old_gravity_source() 10 0.002555 0.002555 0.002555 0.32% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002266 0.002266 0.002266 0.28% MLCellLinOp::compGrad() 11 0.001922 0.001922 0.001922 0.24% MLMG::ResNormInf() 92 0.001868 0.001868 0.001868 0.23% MultiFab::Saxpy() 20 0.001807 0.001807 0.001807 0.22% Gravity::get_old_grav_vector() 10 0.00175 0.00175 0.00175 0.22% MLMG::oneIter() 81 0.001679 0.001679 0.001679 0.21% MLCellLinOp::setLevelBC() 11 0.001524 0.001524 0.001524 0.19% Gravity::actual_solve_with_mlmg() 11 0.001386 0.001386 0.001386 0.17% MLCellLinOp::prepareForSolve() 11 0.001348 0.001348 0.001348 0.17% FabArray::mult() 43 0.001334 0.001334 0.001334 0.16% Castro::initData() 1 0.001272 0.001272 0.001272 0.16% MultiFab::contains_nan() 20 0.001173 0.001173 0.001173 0.14% MLCellLinOp::smooth() 1620 0.001116 0.001116 0.001116 0.14% FabArray::FillBoundary() 3974 0.0008443 0.0008443 0.0008443 0.10% FabArrayBase::getCPC() 1313 0.0008411 0.0008411 0.0008411 0.10% MLMG::prepareForSolve() 11 0.0007697 0.0007697 0.0007697 0.10% FabArrayBase::CPC::define() 454 0.0007433 0.0007433 0.0007433 0.09% FabArrayBase::getFB() 3974 0.0007011 0.0007011 0.0007011 0.09% Gravity::update_max_rhs() 11 0.0005506 0.0005506 0.0005506 0.07% MultiFab::Copy() 11 0.0005136 0.0005136 0.0005136 0.06% MLCellLinOp::apply() 1128 0.000491 0.000491 0.000491 0.06% CGSolver::sxay() 1566 0.0004759 0.0004759 0.0004759 0.06% Gravity::solve_for_phi() 10 0.0004328 0.0004328 0.0004328 0.05% Amr::InitAmr() 1 0.0004217 0.0004217 0.0004217 0.05% MLMG::mgVcycle() 81 0.000401 0.000401 0.000401 0.05% MLCGSolver::ParallelAllReduce 1495 0.0003294 0.0003294 0.0003294 0.04% FabArray::ParallelCopy() 851 0.0002913 0.0002913 0.0002913 0.04% main() 1 0.0002779 0.0002779 0.0002779 0.03% MultiFab::max() 11 0.0002555 0.0002555 0.0002555 0.03% FillPatchIterator::Initialize 41 0.0002459 0.0002459 0.0002459 0.03% Castro::subcycle_advance_ctu() 10 0.0002391 0.0002391 0.0002391 0.03% Gravity::actual_multilevel_solve() 1 0.0002358 0.0002358 0.0002358 0.03% MLCellLinOp::correctionResidual() 486 0.0002323 0.0002323 0.0002323 0.03% Amr::coarseTimeStep() 10 0.0002101 0.0002101 0.0002101 0.03% Castro::construct_new_gravity() 10 0.0002079 0.0002079 0.0002079 0.03% Amr::timeStep() 10 0.0002033 0.0002033 0.0002033 0.03% MLMG::MLRhsNormInf() 11 0.0001986 0.0001986 0.0001986 0.02% MLCellLinOp::defineBC() 11 0.0001844 0.0001844 0.0001844 0.02% MLLinOp::defineGrids() 11 0.000174 0.000174 0.000174 0.02% Amr::defBaseLevel() 1 0.0001434 0.0001434 0.0001434 0.02% MLMG:computeResOfCorrection() 405 0.0001411 0.0001411 0.0001411 0.02% StateData::checkPoint() 12 0.0001349 0.0001349 0.0001349 0.02% MLMG::actualBottomSolve() 81 0.0001052 0.0001052 0.0001052 0.01% MLMG::mgVcycle_down::0 81 9.724e-05 9.724e-05 9.724e-05 0.01% FabArrayBase::FB::FB() 56 8.922e-05 8.922e-05 8.922e-05 0.01% MLMG::mgVcycle_down::1 81 8.639e-05 8.639e-05 8.639e-05 0.01% MLMG::solve() 11 8.555e-05 8.555e-05 8.555e-05 0.01% MLMG::mgVcycle_down::2 81 8.154e-05 8.154e-05 8.154e-05 0.01% MLMG::mgVcycle_down::4 81 7.798e-05 7.798e-05 7.798e-05 0.01% MLMG::mgVcycle_down::3 81 7.796e-05 7.796e-05 7.796e-05 0.01% Castro::initialize_advance() 10 7.735e-05 7.735e-05 7.735e-05 0.01% Castro::clean_state() 62 7.383e-05 7.383e-05 7.383e-05 0.01% AmrLevel::checkPoint() 3 7.083e-05 7.083e-05 7.083e-05 0.01% Castro::advance() 10 6.462e-05 6.462e-05 6.462e-05 0.01% MLMG::mgVcycle_up::4 81 6.217e-05 6.217e-05 6.217e-05 0.01% Castro::finalize_advance() 10 6.128e-05 6.128e-05 6.128e-05 0.01% Castro::initialize_do_advance() 10 5.626e-05 5.626e-05 5.626e-05 0.01% MLMG::mgVcycle_up::0 81 5.399e-05 5.399e-05 5.399e-05 0.01% MLMG::mgVcycle_up::1 81 5.032e-05 5.032e-05 5.032e-05 0.01% MLMG::mgVcycle_up::3 81 4.983e-05 4.983e-05 4.983e-05 0.01% MLCellLinOp::solutionResidual() 92 4.894e-05 4.894e-05 4.894e-05 0.01% MLMG::mgVcycle_up::2 81 4.752e-05 4.752e-05 4.752e-05 0.01% StateData::define() 4 4.285e-05 4.285e-05 4.285e-05 0.01% Castro::finalize_do_advance() 10 4.024e-05 4.024e-05 4.024e-05 0.00% Castro::swap_state_time_levels() 10 3.796e-05 3.796e-05 3.796e-05 0.00% MLMG::mgVcycle_bottom 81 3.59e-05 3.59e-05 3.59e-05 0.00% MLMG::computeResidual() 81 3.397e-05 3.397e-05 3.397e-05 0.00% Castro::enforce_consistent_e() 1 3.324e-05 3.324e-05 3.324e-05 0.00% FillPatchSingleLevel 41 3.083e-05 3.083e-05 3.083e-05 0.00% makeSFC 55 2.866e-05 2.866e-05 2.866e-05 0.00% MLLinOp::define() 11 2.862e-05 2.862e-05 2.862e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.711e-05 2.711e-05 2.711e-05 0.00% Amr::writeSmallPlotFile() 1 2.532e-05 2.532e-05 2.532e-05 0.00% MLPoisson::define() 11 2.227e-05 2.227e-05 2.227e-05 0.00% Castro::construct_old_source() 50 2.191e-05 2.191e-05 2.191e-05 0.00% Amr::FinalizeInit() 1 2.082e-05 2.082e-05 2.082e-05 0.00% Castro::construct_new_source() 50 1.863e-05 1.863e-05 1.863e-05 0.00% Castro::do_new_sources() 10 1.83e-05 1.83e-05 1.83e-05 0.00% DistributionMapping::Distribute() 56 1.734e-05 1.734e-05 1.734e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.659e-05 1.659e-05 1.659e-05 0.00% Castro::do_old_sources() 10 1.608e-05 1.608e-05 1.608e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.597e-05 1.597e-05 1.597e-05 0.00% Castro::check_for_nan() 20 1.226e-05 1.226e-05 1.226e-05 0.00% Castro::apply_source_to_state() 20 1.117e-05 1.117e-05 1.117e-05 0.00% Castro::construct_old_gravity() 10 1.056e-05 1.056e-05 1.056e-05 0.00% Gravity::swapTimeLevels() 10 1.025e-05 1.025e-05 1.025e-05 0.00% AmrLevel::AmrLevel(dm) 1 9.615e-06 9.615e-06 9.615e-06 0.00% Castro::post_timestep() 10 9.238e-06 9.238e-06 9.238e-06 0.00% Amr::initSubcycle() 1 8.972e-06 8.972e-06 8.972e-06 0.00% MLPoisson::prepareForSolve() 11 8.604e-06 8.604e-06 8.604e-06 0.00% MLMG::computeMLResidual() 11 7.793e-06 7.793e-06 7.793e-06 0.00% Castro::computeNewDt() 9 6.991e-06 6.991e-06 6.991e-06 0.00% Amr::InitializeInit() 1 6.794e-06 6.794e-06 6.794e-06 0.00% MLMG::getGradSolution() 11 6.005e-06 6.005e-06 6.005e-06 0.00% MLMG::buildFineMask() 11 5.792e-06 5.792e-06 5.792e-06 0.00% Castro::create_source_corrector() 10 5.78e-06 5.78e-06 5.78e-06 0.00% AmrLevel::checkPointPost() 3 5.597e-06 5.597e-06 5.597e-06 0.00% MLMG::MLResNormInf() 11 4.853e-06 4.853e-06 4.853e-06 0.00% Gravity::set_mass_offset() 11 4.325e-06 4.325e-06 4.325e-06 0.00% Castro::retry_advance_ctu() 10 3.826e-06 3.826e-06 3.826e-06 0.00% Castro::post_init() 1 3.335e-06 3.335e-06 3.335e-06 0.00% Castro::FluxRegFineAdd() 10 3.056e-06 3.056e-06 3.056e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.894e-06 2.894e-06 2.894e-06 0.00% Castro::computeInitialDt() 2 2.651e-06 2.651e-06 2.651e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.359e-06 2.359e-06 2.359e-06 0.00% Amr::init() 1 2.294e-06 2.294e-06 2.294e-06 0.00% AmrLevel::checkPointPre() 3 2.027e-06 2.027e-06 2.027e-06 0.00% Amr::initialInit() 1 1.076e-06 1.076e-06 1.076e-06 0.00% Castro::post_regrid() 1 9.7e-07 9.7e-07 9.7e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8095 0.8095 0.8095 100.00% Amr::coarseTimeStep() 10 0.6682 0.6682 0.6682 82.55% Amr::timeStep() 10 0.5936 0.5936 0.5936 73.33% Castro::advance() 10 0.5854 0.5854 0.5854 72.32% Castro::subcycle_advance_ctu() 10 0.5724 0.5724 0.5724 70.72% Castro::do_advance_ctu() 10 0.5722 0.5722 0.5722 70.69% Gravity::solve_phi_with_mlmg() 11 0.3195 0.3195 0.3195 39.47% Gravity::actual_solve_with_mlmg() 11 0.3073 0.3073 0.3073 37.96% Castro::construct_new_gravity() 10 0.2934 0.2934 0.2934 36.24% MLMG::solve() 11 0.2849 0.2849 0.2849 35.20% Gravity::solve_for_phi() 10 0.2754 0.2754 0.2754 34.02% MLMG::oneIter() 81 0.2692 0.2692 0.2692 33.26% MLMG::mgVcycle() 81 0.2675 0.2675 0.2675 33.05% Castro::construct_ctu_hydro_source() 10 0.2008 0.2008 0.2008 24.81% VisMF::Write(FabArray) 11 0.1473 0.1473 0.1473 18.20% MLCellLinOp::smooth() 1620 0.1424 0.1424 0.1424 17.59% Amr::init() 1 0.1186 0.1186 0.1186 14.65% MLCellLinOp::applyBC() 4379 0.1113 0.1113 0.1113 13.75% Amr::checkPoint() 3 0.1097 0.1097 0.1097 13.56% AmrLevel::checkPoint() 3 0.1057 0.1057 0.1057 13.06% StateData::checkPoint() 12 0.1057 0.1057 0.1057 13.05% MLMG::mgVcycle_bottom 81 0.07568 0.07568 0.07568 9.35% MLMG::actualBottomSolve() 81 0.07564 0.07564 0.07564 9.34% MLCGSolver::bicgstab 81 0.07493 0.07493 0.07493 9.26% MLPoisson::Fsmooth() 3240 0.0587 0.0587 0.0587 7.25% Amr::initialInit() 1 0.05738 0.05738 0.05738 7.09% FillPatchIterator::Initialize 41 0.04924 0.04924 0.04924 6.08% Amr::FinalizeInit() 1 0.048 0.048 0.048 5.93% Castro::post_init() 1 0.04728 0.04728 0.04728 5.84% Amr::writePlotFile() 2 0.04497 0.04497 0.04497 5.56% Gravity::multilevel_solve_for_new_phi() 1 0.04481 0.04481 0.04481 5.54% Gravity::actual_multilevel_solve() 1 0.0448 0.0448 0.0448 5.53% FillPatchSingleLevel 41 0.04357 0.04357 0.04357 5.38% StateDataPhysBCFunct::() 41 0.03933 0.03933 0.03933 4.86% MLCellLinOp::apply() 1128 0.03902 0.03902 0.03902 4.82% MLMG::mgVcycle_down::0 81 0.03805 0.03805 0.03805 4.70% Castro::clean_state() 62 0.03265 0.03265 0.03265 4.03% MLMG::mgVcycle_up::0 81 0.03253 0.03253 0.03253 4.02% FabArray::setVal() 1135 0.02589 0.02589 0.02589 3.20% Castro::initialize_do_advance() 10 0.02443 0.02443 0.02443 3.02% StateData::FillBoundary(geom) 328 0.02287 0.02287 0.02287 2.82% MLCellLinOp::correctionResidual() 486 0.02194 0.02194 0.02194 2.71% MultiFab::Dot() 1100 0.02074 0.02074 0.02074 2.56% Gravity::get_new_grav_vector() 11 0.02008 0.02008 0.02008 2.48% FabArray::FillBoundary() 3974 0.01912 0.01912 0.01912 2.36% MLMG:computeResOfCorrection() 405 0.01901 0.01901 0.01901 2.35% Castro::expand_state() 10 0.01873 0.01873 0.01873 2.31% Castro::computeTemp() 63 0.01831 0.01831 0.01831 2.26% FillBoundary_nowait() 3974 0.01828 0.01828 0.01828 2.26% MLMG::mgVcycle_down::1 81 0.01811 0.01811 0.01811 2.24% MLMG::mgVcycle_down::2 81 0.01751 0.01751 0.01751 2.16% MLPoisson::define() 11 0.01698 0.01698 0.01698 2.10% FabArray::ParallelCopy() 851 0.01685 0.01685 0.01685 2.08% MLMG::mgVcycle_down::3 81 0.01669 0.01669 0.01669 2.06% FabArray::ParallelCopy_nowait() 851 0.01655 0.01655 0.01655 2.04% MLMG::mgVcycle_down::4 81 0.01588 0.01588 0.01588 1.96% Castro::construct_old_gravity() 10 0.01562 0.01562 0.01562 1.93% Gravity::get_old_grav_vector() 10 0.01561 0.01561 0.01561 1.93% MLMG::mgVcycle_up::2 81 0.01362 0.01362 0.01362 1.68% MLMG::mgVcycle_up::1 81 0.01341 0.01341 0.01341 1.66% MLMG::addInterpCorrection() 405 0.01309 0.01309 0.01309 1.62% MLMG::mgVcycle_up::3 81 0.01291 0.01291 0.01291 1.60% Castro::initialize_advance() 10 0.01286 0.01286 0.01286 1.59% MLMG::mgVcycle_up::4 81 0.01273 0.01273 0.01273 1.57% amrex::average_down 405 0.01267 0.01267 0.01267 1.56% MLCGSolver::ParallelAllReduce 1495 0.01247 0.01247 0.01247 1.54% CGSolver::sxay() 1566 0.01244 0.01244 0.01244 1.54% Gravity::fill_multipole_BCs() 11 0.01196 0.01196 0.01196 1.48% MultiFab::LinComb() 1566 0.01196 0.01196 0.01196 1.48% MLCellLinOp::defineAuxData() 11 0.01167 0.01167 0.01167 1.44% Castro::do_new_sources() 10 0.01051 0.01051 0.01051 1.30% MLPoisson::Fapply() 1128 0.01039 0.01039 0.01039 1.28% Amr::InitializeInit() 1 0.009381 0.009381 0.009381 1.16% Amr::defBaseLevel() 1 0.009374 0.009374 0.009374 1.16% Castro::do_old_sources() 10 0.008322 0.008322 0.008322 1.03% Castro::post_timestep() 10 0.007995 0.007995 0.007995 0.99% MLCellLinOp::solutionResidual() 92 0.00776 0.00776 0.00776 0.96% Castro::enforce_min_density() 62 0.006962 0.006962 0.006962 0.86% MLMG::computeResidual() 81 0.006707 0.006707 0.006707 0.83% MultiFab::Xpay() 578 0.005969 0.005969 0.005969 0.74% MLMG::prepareForSolve() 11 0.005768 0.005768 0.005768 0.71% FabArray::setDomainBndry() 41 0.005416 0.005416 0.005416 0.67% Castro::reset_internal_energy() 63 0.005406 0.005406 0.005406 0.67% Castro::estTimeStep() 21 0.005317 0.005317 0.005317 0.66% MLCellLinOp::defineBC() 11 0.005016 0.005016 0.005016 0.62% BndryData::define() 11 0.004831 0.004831 0.004831 0.60% Castro::normalize_species() 62 0.004374 0.004374 0.004374 0.54% Castro::enforce_speed_limit() 62 0.003705 0.003705 0.003705 0.46% Castro::initData() 1 0.003576 0.003576 0.003576 0.44% Castro::construct_new_source() 50 0.003117 0.003117 0.003117 0.39% Castro::construct_new_gravity_source() 10 0.003099 0.003099 0.003099 0.38% Castro::construct_old_source() 50 0.002577 0.002577 0.002577 0.32% Castro::construct_old_gravity_source() 10 0.002555 0.002555 0.002555 0.32% MLMG::getGradSolution() 11 0.002491 0.002491 0.002491 0.31% MLCellLinOp::compGrad() 11 0.002485 0.002485 0.002485 0.31% Castro::computeNewDt() 9 0.002355 0.002355 0.002355 0.29% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002266 0.002266 0.002266 0.28% MLMG::ResNormInf() 92 0.001868 0.001868 0.001868 0.23% Castro::apply_source_to_state() 20 0.001818 0.001818 0.001818 0.22% MultiFab::Saxpy() 20 0.001807 0.001807 0.001807 0.22% FabArrayBase::getCPC() 1313 0.001584 0.001584 0.001584 0.20% MLCellLinOp::setLevelBC() 11 0.001524 0.001524 0.001524 0.19% MLPoisson::prepareForSolve() 11 0.001356 0.001356 0.001356 0.17% MLCellLinOp::prepareForSolve() 11 0.001348 0.001348 0.001348 0.17% FabArray::mult() 43 0.001334 0.001334 0.001334 0.16% Castro::check_for_nan() 20 0.001185 0.001185 0.001185 0.15% MultiFab::contains_nan() 20 0.001173 0.001173 0.001173 0.14% MLMG::computeMLResidual() 11 0.001095 0.001095 0.001095 0.14% Gravity::swapTimeLevels() 10 0.001052 0.001052 0.001052 0.13% Gravity::update_max_rhs() 11 0.0009462 0.0009462 0.0009462 0.12% FabArrayBase::getFB() 3974 0.0007904 0.0007904 0.0007904 0.10% FabArrayBase::CPC::define() 454 0.0007433 0.0007433 0.0007433 0.09% Castro::computeInitialDt() 2 0.0006805 0.0006805 0.0006805 0.08% MultiFab::Copy() 11 0.0005136 0.0005136 0.0005136 0.06% Castro::post_regrid() 1 0.0004821 0.0004821 0.0004821 0.06% Amr::InitAmr() 1 0.0004307 0.0004307 0.0004307 0.05% MLLinOp::define() 11 0.0002663 0.0002663 0.0002663 0.03% MultiFab::max() 11 0.0002555 0.0002555 0.0002555 0.03% MLMG::MLResNormInf() 11 0.0002525 0.0002525 0.0002525 0.03% MLLinOp::defineGrids() 11 0.0002377 0.0002377 0.0002377 0.03% MLMG::MLRhsNormInf() 11 0.0001986 0.0001986 0.0001986 0.02% FabArrayBase::FB::FB() 56 8.922e-05 8.922e-05 8.922e-05 0.01% Castro::finalize_advance() 10 6.434e-05 6.434e-05 6.434e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 6.132e-05 6.132e-05 6.132e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.247e-05 5.247e-05 5.247e-05 0.01% makeSFC 55 4.473e-05 4.473e-05 4.473e-05 0.01% StateData::define() 4 4.285e-05 4.285e-05 4.285e-05 0.01% Castro::finalize_do_advance() 10 4.024e-05 4.024e-05 4.024e-05 0.00% Castro::swap_state_time_levels() 10 3.796e-05 3.796e-05 3.796e-05 0.00% Castro::enforce_consistent_e() 1 3.324e-05 3.324e-05 3.324e-05 0.00% Amr::writeSmallPlotFile() 1 2.532e-05 2.532e-05 2.532e-05 0.00% DistributionMapping::Distribute() 56 1.734e-05 1.734e-05 1.734e-05 0.00% Amr::initSubcycle() 1 8.972e-06 8.972e-06 8.972e-06 0.00% MLMG::buildFineMask() 11 5.792e-06 5.792e-06 5.792e-06 0.00% Castro::create_source_corrector() 10 5.78e-06 5.78e-06 5.78e-06 0.00% AmrLevel::checkPointPost() 3 5.597e-06 5.597e-06 5.597e-06 0.00% Gravity::set_mass_offset() 11 4.325e-06 4.325e-06 4.325e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.167e-06 4.167e-06 4.167e-06 0.00% Castro::retry_advance_ctu() 10 3.826e-06 3.826e-06 3.826e-06 0.00% Castro::FluxRegFineAdd() 10 3.056e-06 3.056e-06 3.056e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.359e-06 2.359e-06 2.359e-06 0.00% AmrLevel::checkPointPre() 3 2.027e-06 2.027e-06 2.027e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-29-g329f81b889a6) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-29-g329f81b889a6) initialized Starting run at 08:22:32 UTC on 2022-04-26. Successfully read inputs file ... Castro git describe: 22.04-28-g7e86dbe56 AMReX git describe: 22.04-29-g329f81b88 Microphysics git describe: 22.04-3-g3c498521 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.472639257 Restart time = 0.054373674 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.112933469 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051753508 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.054065737 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.056684729 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062083156 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.047576061 seconds Ending run at 08:22:32 UTC on 2022-04-26. Run time = 0.440368689 Run time without initialization = 0.385441571 Average number of zones advanced per microsecond: 3.401 Average number of zones advanced per microsecond per rank: 3.401 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4404 ... 0.4404 ... 0.4404 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1316 0.1316 0.1316 29.88% VisMF::Read() 3 0.04271 0.04271 0.04271 9.70% MLCellLinOp::applyBC() 1946 0.03712 0.03712 0.03712 8.43% Amr::writePlotFile() 1 0.02566 0.02566 0.02566 5.83% MLPoisson::Fsmooth() 1440 0.02545 0.02545 0.02545 5.78% VisMF::Write(FabArray) 1 0.022 0.022 0.022 5.00% FabArray::setVal() 537 0.01658 0.01658 0.01658 3.77% StateData::FillBoundary(geom) 160 0.011 0.011 0.011 2.50% MLCGSolver::bicgstab 36 0.009159 0.009159 0.009159 2.08% MultiFab::Dot() 484 0.008992 0.008992 0.008992 2.04% StateDataPhysBCFunct::() 20 0.007675 0.007675 0.007675 1.74% FillBoundary_nowait() 1766 0.00702 0.00702 0.00702 1.59% Castro::computeTemp() 30 0.006979 0.006979 0.006979 1.58% FabArray::ParallelCopy_nowait() 380 0.006965 0.006965 0.006965 1.58% Gravity::fill_multipole_BCs() 6 0.006046 0.006046 0.006046 1.37% MLCellLinOp::defineAuxData() 6 0.005345 0.005345 0.005345 1.21% MultiFab::LinComb() 690 0.00518 0.00518 0.00518 1.18% MLPoisson::Fapply() 500 0.00455 0.00455 0.00455 1.03% Castro::expand_state() 5 0.00409 0.00409 0.00409 0.93% FabArray::setDomainBndry() 20 0.003955 0.003955 0.003955 0.90% Castro::do_advance_ctu() 5 0.003584 0.003584 0.003584 0.81% Castro::reset_internal_energy() 30 0.003222 0.003222 0.003222 0.73% Castro::estTimeStep() 10 0.003028 0.003028 0.003028 0.69% Amr::restart() 1 0.002944 0.002944 0.002944 0.67% MLMG::addInterpCorrection() 180 0.002921 0.002921 0.002921 0.66% Castro::enforce_min_density() 30 0.002914 0.002914 0.002914 0.66% amrex::average_down 180 0.002719 0.002719 0.002719 0.62% MultiFab::Xpay() 258 0.002624 0.002624 0.002624 0.60% Castro::normalize_species() 30 0.002189 0.002189 0.002189 0.50% BndryData::define() 6 0.001936 0.001936 0.001936 0.44% Gravity::get_new_grav_vector() 5 0.001735 0.001735 0.001735 0.39% Castro::construct_new_gravity_source() 5 0.001566 0.001566 0.001566 0.36% MLCellLinOp::compGrad() 6 0.001457 0.001457 0.001457 0.33% Castro::construct_old_gravity_source() 5 0.001358 0.001358 0.001358 0.31% Castro::enforce_speed_limit() 30 0.001239 0.001239 0.001239 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001162 0.001162 0.001162 0.26% Castro::subcycle_advance_ctu() 5 0.0009329 0.0009329 0.0009329 0.21% MultiFab::Saxpy() 10 0.000915 0.000915 0.000915 0.21% Gravity::get_old_grav_vector() 5 0.0009059 0.0009059 0.0009059 0.21% MLMG::ResNormInf() 42 0.0008215 0.0008215 0.0008215 0.19% MLCellLinOp::setLevelBC() 6 0.0008025 0.0008025 0.0008025 0.18% MLMG::oneIter() 36 0.000748 0.000748 0.000748 0.17% Gravity::actual_solve_with_mlmg() 6 0.0007109 0.0007109 0.0007109 0.16% MLCellLinOp::prepareForSolve() 6 0.0006837 0.0006837 0.0006837 0.16% FabArray::mult() 22 0.0006556 0.0006556 0.0006556 0.15% MultiFab::contains_nan() 10 0.0005895 0.0005895 0.0005895 0.13% Gravity::update_max_rhs() 6 0.0005328 0.0005328 0.0005328 0.12% MLCellLinOp::smooth() 720 0.0005245 0.0005245 0.0005245 0.12% FabArrayBase::CPC::define() 244 0.0004302 0.0004302 0.0004302 0.10% MLMG::prepareForSolve() 6 0.0004267 0.0004267 0.0004267 0.10% FabArray::FillBoundary() 1766 0.000396 0.000396 0.000396 0.09% FabArrayBase::getCPC() 632 0.0003945 0.0003945 0.0003945 0.09% MultiFab::Copy() 6 0.000372 0.000372 0.000372 0.08% Amr::InitAmr() 1 0.0003705 0.0003705 0.0003705 0.08% FabArrayBase::getFB() 1766 0.0002864 0.0002864 0.0002864 0.07% main() 1 0.0002634 0.0002634 0.0002634 0.06% Gravity::solve_for_phi() 5 0.0002624 0.0002624 0.0002624 0.06% MLCellLinOp::apply() 500 0.0002337 0.0002337 0.0002337 0.05% Gravity::actual_multilevel_solve() 1 0.000218 0.000218 0.000218 0.05% CGSolver::sxay() 690 0.0001945 0.0001945 0.0001945 0.04% MLCGSolver::ParallelAllReduce 659 0.0001589 0.0001589 0.0001589 0.04% MLMG::mgVcycle() 36 0.0001574 0.0001574 0.0001574 0.04% MultiFab::max() 6 0.0001335 0.0001335 0.0001335 0.03% FabArray::ParallelCopy() 380 0.0001333 0.0001333 0.0001333 0.03% Castro::initialize_advance() 5 0.000126 0.000126 0.000126 0.03% FillPatchIterator::Initialize 20 0.0001219 0.0001219 0.0001219 0.03% Castro::create_source_corrector() 5 0.000116 0.000116 0.000116 0.03% Castro::construct_new_gravity() 5 0.0001123 0.0001123 0.0001123 0.03% Amr::coarseTimeStep() 5 0.0001099 0.0001099 0.0001099 0.02% MLMG::MLRhsNormInf() 6 0.0001053 0.0001053 0.0001053 0.02% Amr::timeStep() 5 9.999e-05 9.999e-05 9.999e-05 0.02% MLCellLinOp::defineBC() 6 9.781e-05 9.781e-05 9.781e-05 0.02% MLCellLinOp::correctionResidual() 216 9.468e-05 9.468e-05 9.468e-05 0.02% MLLinOp::defineGrids() 6 9.25e-05 9.25e-05 9.25e-05 0.02% Castro::advance() 5 8.769e-05 8.769e-05 8.769e-05 0.02% StateData::restartDoit() 4 7.309e-05 7.309e-05 7.309e-05 0.02% AmrLevel::restart() 1 7.028e-05 7.028e-05 7.028e-05 0.02% Castro::construct_new_source() 25 6.832e-05 6.832e-05 6.832e-05 0.02% Castro::post_timestep() 5 6.395e-05 6.395e-05 6.395e-05 0.01% Castro::finalize_advance() 5 6.045e-05 6.045e-05 6.045e-05 0.01% FabArrayBase::FB::FB() 26 5.931e-05 5.931e-05 5.931e-05 0.01% MLMG:computeResOfCorrection() 180 5.533e-05 5.533e-05 5.533e-05 0.01% MLMG::actualBottomSolve() 36 4.663e-05 4.663e-05 4.663e-05 0.01% MLMG::solve() 6 4.101e-05 4.101e-05 4.101e-05 0.01% Castro::do_new_sources() 5 3.86e-05 3.86e-05 3.86e-05 0.01% MLMG::mgVcycle_down::0 36 3.762e-05 3.762e-05 3.762e-05 0.01% Castro::clean_state() 30 3.706e-05 3.706e-05 3.706e-05 0.01% Castro::initialize_do_advance() 5 3.574e-05 3.574e-05 3.574e-05 0.01% MLMG::mgVcycle_down::1 36 3.467e-05 3.467e-05 3.467e-05 0.01% MLMG::mgVcycle_down::2 36 3.199e-05 3.199e-05 3.199e-05 0.01% MLMG::mgVcycle_down::4 36 3.082e-05 3.082e-05 3.082e-05 0.01% MLMG::mgVcycle_down::3 36 2.98e-05 2.98e-05 2.98e-05 0.01% MLLinOp::define() 6 2.907e-05 2.907e-05 2.907e-05 0.01% Castro::swap_state_time_levels() 5 2.719e-05 2.719e-05 2.719e-05 0.01% MLMG::mgVcycle_up::4 36 2.718e-05 2.718e-05 2.718e-05 0.01% Amr::writeSmallPlotFile() 1 2.717e-05 2.717e-05 2.717e-05 0.01% MLMG::mgVcycle_up::0 36 2.376e-05 2.376e-05 2.376e-05 0.01% MLCellLinOp::solutionResidual() 42 2.233e-05 2.233e-05 2.233e-05 0.01% Castro::post_restart() 1 2.209e-05 2.209e-05 2.209e-05 0.01% MLMG::mgVcycle_up::3 36 2.191e-05 2.191e-05 2.191e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.168e-05 2.168e-05 2.168e-05 0.00% MLMG::mgVcycle_up::2 36 2.163e-05 2.163e-05 2.163e-05 0.00% MLMG::mgVcycle_up::1 36 2.071e-05 2.071e-05 2.071e-05 0.00% Castro::finalize_do_advance() 5 1.917e-05 1.917e-05 1.917e-05 0.00% MLMG::mgVcycle_bottom 36 1.714e-05 1.714e-05 1.714e-05 0.00% MLMG::computeResidual() 36 1.668e-05 1.668e-05 1.668e-05 0.00% FillPatchSingleLevel 20 1.58e-05 1.58e-05 1.58e-05 0.00% MLPoisson::define() 6 1.578e-05 1.578e-05 1.578e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.564e-05 1.564e-05 1.564e-05 0.00% makeSFC 30 1.505e-05 1.505e-05 1.505e-05 0.00% Castro::construct_old_source() 25 1.245e-05 1.245e-05 1.245e-05 0.00% Amr::initSubcycle() 1 9.401e-06 9.401e-06 9.401e-06 0.00% DistributionMapping::Distribute() 31 9.236e-06 9.236e-06 9.236e-06 0.00% Castro::do_old_sources() 5 8.481e-06 8.481e-06 8.481e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.258e-06 8.258e-06 8.258e-06 0.00% Castro::apply_source_to_state() 10 5.878e-06 5.878e-06 5.878e-06 0.00% Castro::check_for_nan() 10 5.816e-06 5.816e-06 5.816e-06 0.00% Castro::construct_old_gravity() 5 5.607e-06 5.607e-06 5.607e-06 0.00% Gravity::swapTimeLevels() 5 5.185e-06 5.185e-06 5.185e-06 0.00% MLPoisson::prepareForSolve() 6 4.308e-06 4.308e-06 4.308e-06 0.00% MLMG::computeMLResidual() 6 3.712e-06 3.712e-06 3.712e-06 0.00% MLMG::buildFineMask() 6 3.406e-06 3.406e-06 3.406e-06 0.00% MLMG::getGradSolution() 6 3.246e-06 3.246e-06 3.246e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.976e-06 2.976e-06 2.976e-06 0.00% Castro::computeNewDt() 5 2.827e-06 2.827e-06 2.827e-06 0.00% MLMG::MLResNormInf() 6 2.571e-06 2.571e-06 2.571e-06 0.00% Gravity::set_mass_offset() 6 2.039e-06 2.039e-06 2.039e-06 0.00% Castro::retry_advance_ctu() 5 1.893e-06 1.893e-06 1.893e-06 0.00% Castro::FluxRegFineAdd() 5 1.798e-06 1.798e-06 1.798e-06 0.00% Amr::init() 1 1.323e-06 1.323e-06 1.323e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.303e-06 1.303e-06 1.303e-06 0.00% AmrLevel::AmrLevel() 1 8.27e-07 8.27e-07 8.27e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4404 0.4404 0.4404 100.00% Amr::coarseTimeStep() 5 0.3376 0.3376 0.3376 76.66% Amr::timeStep() 5 0.3365 0.3365 0.3365 76.41% Castro::advance() 5 0.3328 0.3328 0.3328 75.58% Castro::subcycle_advance_ctu() 5 0.3208 0.3208 0.3208 72.84% Castro::do_advance_ctu() 5 0.3199 0.3199 0.3199 72.63% Castro::construct_new_gravity() 5 0.1435 0.1435 0.1435 32.59% Gravity::solve_phi_with_mlmg() 6 0.1397 0.1397 0.1397 31.72% Gravity::solve_for_phi() 5 0.134 0.134 0.134 30.42% Gravity::actual_solve_with_mlmg() 6 0.1335 0.1335 0.1335 30.31% Castro::construct_ctu_hydro_source() 5 0.1316 0.1316 0.1316 29.88% MLMG::solve() 6 0.1211 0.1211 0.1211 27.50% MLMG::oneIter() 36 0.1133 0.1133 0.1133 25.74% MLMG::mgVcycle() 36 0.1126 0.1126 0.1126 25.57% MLCellLinOp::smooth() 720 0.05913 0.05913 0.05913 13.43% Amr::init() 1 0.05443 0.05443 0.05443 12.36% Amr::restart() 1 0.05443 0.05443 0.05443 12.36% Amr::writePlotFile() 1 0.04766 0.04766 0.04766 10.82% MLCellLinOp::applyBC() 1946 0.04488 0.04488 0.04488 10.19% AmrLevel::restart() 1 0.04365 0.04365 0.04365 9.91% StateData::restartDoit() 4 0.04358 0.04358 0.04358 9.90% VisMF::Read() 3 0.04271 0.04271 0.04271 9.70% MLMG::mgVcycle_bottom 36 0.03247 0.03247 0.03247 7.37% MLMG::actualBottomSolve() 36 0.03245 0.03245 0.03245 7.37% MLCGSolver::bicgstab 36 0.03213 0.03213 0.03213 7.30% MLPoisson::Fsmooth() 1440 0.02545 0.02545 0.02545 5.78% FillPatchIterator::Initialize 20 0.02486 0.02486 0.02486 5.64% VisMF::Write(FabArray) 1 0.022 0.022 0.022 5.00% FillPatchSingleLevel 20 0.02078 0.02078 0.02078 4.72% StateDataPhysBCFunct::() 20 0.01867 0.01867 0.01867 4.24% FabArray::setVal() 537 0.01658 0.01658 0.01658 3.77% Castro::clean_state() 30 0.01658 0.01658 0.01658 3.76% MLCellLinOp::apply() 500 0.01623 0.01623 0.01623 3.68% MLMG::mgVcycle_down::0 36 0.01578 0.01578 0.01578 3.58% Castro::initialize_do_advance() 5 0.01475 0.01475 0.01475 3.35% MLMG::mgVcycle_up::0 36 0.01344 0.01344 0.01344 3.05% Castro::expand_state() 5 0.01228 0.01228 0.01228 2.79% Castro::initialize_advance() 5 0.01191 0.01191 0.01191 2.70% StateData::FillBoundary(geom) 160 0.011 0.011 0.011 2.50% Castro::computeTemp() 30 0.0102 0.0102 0.0102 2.32% Gravity::get_new_grav_vector() 5 0.009462 0.009462 0.009462 2.15% MLCellLinOp::correctionResidual() 216 0.009175 0.009175 0.009175 2.08% MLPoisson::define() 6 0.009133 0.009133 0.009133 2.07% MultiFab::Dot() 484 0.008992 0.008992 0.008992 2.04% MLMG:computeResOfCorrection() 180 0.007949 0.007949 0.007949 1.80% Castro::construct_old_gravity() 5 0.007831 0.007831 0.007831 1.78% Gravity::get_old_grav_vector() 5 0.007825 0.007825 0.007825 1.78% FabArray::FillBoundary() 1766 0.007761 0.007761 0.007761 1.76% MLMG::mgVcycle_down::1 36 0.007634 0.007634 0.007634 1.73% FabArray::ParallelCopy() 380 0.007501 0.007501 0.007501 1.70% FabArray::ParallelCopy_nowait() 380 0.007367 0.007367 0.007367 1.67% FillBoundary_nowait() 1766 0.007365 0.007365 0.007365 1.67% MLMG::mgVcycle_down::2 36 0.007335 0.007335 0.007335 1.67% MLMG::mgVcycle_down::3 36 0.006971 0.006971 0.006971 1.58% MLMG::mgVcycle_down::4 36 0.006669 0.006669 0.006669 1.51% Castro::post_restart() 1 0.006596 0.006596 0.006596 1.50% Castro::do_new_sources() 5 0.00638 0.00638 0.00638 1.45% MLCellLinOp::defineAuxData() 6 0.006237 0.006237 0.006237 1.42% Gravity::multilevel_solve_for_new_phi() 1 0.006205 0.006205 0.006205 1.41% Gravity::actual_multilevel_solve() 1 0.006183 0.006183 0.006183 1.40% Gravity::fill_multipole_BCs() 6 0.006046 0.006046 0.006046 1.37% MLMG::mgVcycle_up::2 36 0.005729 0.005729 0.005729 1.30% MLMG::mgVcycle_up::1 36 0.005659 0.005659 0.005659 1.29% MLMG::addInterpCorrection() 180 0.00562 0.00562 0.00562 1.28% MLCGSolver::ParallelAllReduce 659 0.00546 0.00546 0.00546 1.24% amrex::average_down 180 0.005425 0.005425 0.005425 1.23% MLMG::mgVcycle_up::3 36 0.005412 0.005412 0.005412 1.23% CGSolver::sxay() 690 0.005375 0.005375 0.005375 1.22% MLMG::mgVcycle_up::4 36 0.005347 0.005347 0.005347 1.21% MultiFab::LinComb() 690 0.00518 0.00518 0.00518 1.18% MLPoisson::Fapply() 500 0.00455 0.00455 0.00455 1.03% Castro::do_old_sources() 5 0.004454 0.004454 0.004454 1.01% FabArray::setDomainBndry() 20 0.003955 0.003955 0.003955 0.90% Castro::post_timestep() 5 0.003553 0.003553 0.003553 0.81% MLMG::prepareForSolve() 6 0.003422 0.003422 0.003422 0.78% MLCellLinOp::solutionResidual() 42 0.003353 0.003353 0.003353 0.76% Castro::reset_internal_energy() 30 0.003222 0.003222 0.003222 0.73% Castro::estTimeStep() 10 0.003028 0.003028 0.003028 0.69% Castro::enforce_min_density() 30 0.002914 0.002914 0.002914 0.66% MLMG::computeResidual() 36 0.002788 0.002788 0.002788 0.63% MLCellLinOp::defineBC() 6 0.002726 0.002726 0.002726 0.62% BndryData::define() 6 0.002629 0.002629 0.002629 0.60% MultiFab::Xpay() 258 0.002624 0.002624 0.002624 0.60% Castro::normalize_species() 30 0.002189 0.002189 0.002189 0.50% MLMG::getGradSolution() 6 0.001739 0.001739 0.001739 0.39% MLCellLinOp::compGrad() 6 0.001736 0.001736 0.001736 0.39% Castro::construct_new_source() 25 0.001634 0.001634 0.001634 0.37% Castro::construct_new_gravity_source() 5 0.001566 0.001566 0.001566 0.36% Castro::construct_old_source() 25 0.001371 0.001371 0.001371 0.31% Castro::construct_old_gravity_source() 5 0.001358 0.001358 0.001358 0.31% Castro::enforce_speed_limit() 30 0.001239 0.001239 0.001239 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001162 0.001162 0.001162 0.26% Castro::computeNewDt() 5 0.001017 0.001017 0.001017 0.23% Castro::apply_source_to_state() 10 0.0009209 0.0009209 0.0009209 0.21% MultiFab::Saxpy() 10 0.000915 0.000915 0.000915 0.21% Gravity::swapTimeLevels() 5 0.0008915 0.0008915 0.0008915 0.20% FabArrayBase::getCPC() 632 0.0008247 0.0008247 0.0008247 0.19% MLMG::ResNormInf() 42 0.0008215 0.0008215 0.0008215 0.19% MLCellLinOp::setLevelBC() 6 0.0008025 0.0008025 0.0008025 0.18% Gravity::update_max_rhs() 6 0.0007417 0.0007417 0.0007417 0.17% MLPoisson::prepareForSolve() 6 0.000688 0.000688 0.000688 0.16% MLCellLinOp::prepareForSolve() 6 0.0006837 0.0006837 0.0006837 0.16% FabArray::mult() 22 0.0006556 0.0006556 0.0006556 0.15% Castro::check_for_nan() 10 0.0005953 0.0005953 0.0005953 0.14% MultiFab::contains_nan() 10 0.0005895 0.0005895 0.0005895 0.13% MLMG::computeMLResidual() 6 0.000585 0.000585 0.000585 0.13% FabArrayBase::CPC::define() 244 0.0004302 0.0004302 0.0004302 0.10% Amr::InitAmr() 1 0.0003799 0.0003799 0.0003799 0.09% MultiFab::Copy() 6 0.000372 0.000372 0.000372 0.08% FabArrayBase::getFB() 1766 0.0003457 0.0003457 0.0003457 0.08% MLLinOp::define() 6 0.0001541 0.0001541 0.0001541 0.03% MultiFab::max() 6 0.0001335 0.0001335 0.0001335 0.03% MLMG::MLResNormInf() 6 0.000133 0.000133 0.000133 0.03% MLLinOp::defineGrids() 6 0.000125 0.000125 0.000125 0.03% Castro::create_source_corrector() 5 0.000116 0.000116 0.000116 0.03% MLMG::MLRhsNormInf() 6 0.0001053 0.0001053 0.0001053 0.02% Castro::finalize_advance() 5 6.225e-05 6.225e-05 6.225e-05 0.01% FabArrayBase::FB::FB() 26 5.931e-05 5.931e-05 5.931e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.124e-05 3.124e-05 3.124e-05 0.01% Castro::swap_state_time_levels() 5 2.719e-05 2.719e-05 2.719e-05 0.01% Amr::writeSmallPlotFile() 1 2.717e-05 2.717e-05 2.717e-05 0.01% makeSFC 30 2.298e-05 2.298e-05 2.298e-05 0.01% Castro::finalize_do_advance() 5 1.917e-05 1.917e-05 1.917e-05 0.00% Amr::initSubcycle() 1 9.401e-06 9.401e-06 9.401e-06 0.00% DistributionMapping::Distribute() 31 9.236e-06 9.236e-06 9.236e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.282e-06 4.282e-06 4.282e-06 0.00% MLMG::buildFineMask() 6 3.406e-06 3.406e-06 3.406e-06 0.00% Gravity::set_mass_offset() 6 2.039e-06 2.039e-06 2.039e-06 0.00% Castro::retry_advance_ctu() 5 1.893e-06 1.893e-06 1.893e-06 0.00% Castro::FluxRegFineAdd() 5 1.798e-06 1.798e-06 1.798e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.303e-06 1.303e-06 1.303e-06 0.00% AmrLevel::AmrLevel() 1 8.27e-07 8.27e-07 8.27e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-29-g329f81b889a6) finalized