Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-28-g56661522b4cd) initialized Starting run at 08:22:09 UTC on 2022-04-25. Successfully read inputs file ... Castro git describe: 22.04-28-g7e86dbe56 AMReX git describe: 22.04-28-g56661522b Microphysics git describe: 22.04-3-g3c498521 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.038459224 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.023039098 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.121590201 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049011365 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.048804679 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.047763325 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.067340999 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.103798718 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.049233843 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048985008 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058331299 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.061404007 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.066472402 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.03546303 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.022167491 seconds Ending run at 08:22:10 UTC on 2022-04-25. Run time = 0.900571008 Run time without initialization = 0.780856013 Average number of zones advanced per microsecond: 3.357 Average number of zones advanced per microsecond per rank: 3.357 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.9006 ... 0.9006 ... 0.9006 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2127 0.2127 0.2127 23.61% VisMF::Write(FabArray) 11 0.1481 0.1481 0.1481 16.45% MLCellLinOp::applyBC() 4379 0.09132 0.09132 0.09132 10.14% Amr::checkPoint() 3 0.07162 0.07162 0.07162 7.95% MLPoisson::Fsmooth() 3240 0.05856 0.05856 0.05856 6.50% StateDataPhysBCFunct::() 41 0.03179 0.03179 0.03179 3.53% FabArray::setVal() 1135 0.02653 0.02653 0.02653 2.95% StateData::FillBoundary(geom) 328 0.02305 0.02305 0.02305 2.56% MLCGSolver::bicgstab 81 0.02113 0.02113 0.02113 2.35% MultiFab::Dot() 1100 0.0207 0.0207 0.0207 2.30% FillBoundary_nowait() 3974 0.01658 0.01658 0.01658 1.84% FabArray::ParallelCopy_nowait() 851 0.01571 0.01571 0.01571 1.74% MultiFab::LinComb() 1566 0.01181 0.01181 0.01181 1.31% Gravity::fill_multipole_BCs() 11 0.0115 0.0115 0.0115 1.28% MLPoisson::Fapply() 1128 0.01026 0.01026 0.01026 1.14% Castro::computeTemp() 63 0.01026 0.01026 0.01026 1.14% MLCellLinOp::defineAuxData() 11 0.01005 0.01005 0.01005 1.12% Castro::reset_internal_energy() 63 0.006877 0.006877 0.006877 0.76% MLMG::addInterpCorrection() 405 0.006861 0.006861 0.006861 0.76% amrex::average_down 405 0.006468 0.006468 0.006468 0.72% Castro::enforce_min_density() 62 0.006298 0.006298 0.006298 0.70% MultiFab::Xpay() 578 0.005922 0.005922 0.005922 0.66% FabArray::setDomainBndry() 41 0.005693 0.005693 0.005693 0.63% Castro::do_advance_ctu() 10 0.005132 0.005132 0.005132 0.57% Castro::expand_state() 10 0.00485 0.00485 0.00485 0.54% Castro::normalize_species() 62 0.004359 0.004359 0.004359 0.48% Castro::estTimeStep() 21 0.004158 0.004158 0.004158 0.46% Castro::enforce_speed_limit() 62 0.003611 0.003611 0.003611 0.40% BndryData::define() 11 0.003583 0.003583 0.003583 0.40% Amr::writePlotFile() 2 0.003216 0.003216 0.003216 0.36% Castro::construct_new_gravity_source() 10 0.003153 0.003153 0.003153 0.35% Gravity::get_new_grav_vector() 11 0.002689 0.002689 0.002689 0.30% Castro::construct_old_gravity_source() 10 0.002498 0.002498 0.002498 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002252 0.002252 0.002252 0.25% MLCellLinOp::compGrad() 11 0.001995 0.001995 0.001995 0.22% MLMG::ResNormInf() 92 0.00186 0.00186 0.00186 0.21% MultiFab::Saxpy() 20 0.001809 0.001809 0.001809 0.20% Gravity::get_old_grav_vector() 10 0.001748 0.001748 0.001748 0.19% MLMG::oneIter() 81 0.001677 0.001677 0.001677 0.19% MLCellLinOp::setLevelBC() 11 0.001513 0.001513 0.001513 0.17% Gravity::actual_solve_with_mlmg() 11 0.001424 0.001424 0.001424 0.16% MLCellLinOp::prepareForSolve() 11 0.001355 0.001355 0.001355 0.15% FabArray::mult() 43 0.001333 0.001333 0.001333 0.15% Castro::initData() 1 0.00125 0.00125 0.00125 0.14% MultiFab::contains_nan() 20 0.001172 0.001172 0.001172 0.13% MLCellLinOp::smooth() 1620 0.001146 0.001146 0.001146 0.13% FabArrayBase::getCPC() 1313 0.000844 0.000844 0.000844 0.09% FabArray::FillBoundary() 3974 0.0008418 0.0008418 0.0008418 0.09% MLMG::prepareForSolve() 11 0.0007888 0.0007888 0.0007888 0.09% FabArrayBase::CPC::define() 454 0.0007225 0.0007225 0.0007225 0.08% FabArrayBase::getFB() 3974 0.0007096 0.0007096 0.0007096 0.08% Gravity::update_max_rhs() 11 0.0005755 0.0005755 0.0005755 0.06% MultiFab::Copy() 11 0.000519 0.000519 0.000519 0.06% MLCellLinOp::apply() 1128 0.0005063 0.0005063 0.0005063 0.06% Gravity::solve_for_phi() 10 0.0004572 0.0004572 0.0004572 0.05% CGSolver::sxay() 1566 0.0004544 0.0004544 0.0004544 0.05% Amr::InitAmr() 1 0.0004359 0.0004359 0.0004359 0.05% MLMG::mgVcycle() 81 0.0003926 0.0003926 0.0003926 0.04% MLCGSolver::ParallelAllReduce 1495 0.0003287 0.0003287 0.0003287 0.04% main() 1 0.0002844 0.0002844 0.0002844 0.03% FabArray::ParallelCopy() 851 0.0002832 0.0002832 0.0002832 0.03% FillPatchIterator::Initialize 41 0.0002676 0.0002676 0.0002676 0.03% MultiFab::max() 11 0.0002507 0.0002507 0.0002507 0.03% MLCellLinOp::correctionResidual() 486 0.0002304 0.0002304 0.0002304 0.03% Gravity::actual_multilevel_solve() 1 0.0002222 0.0002222 0.0002222 0.02% Amr::coarseTimeStep() 10 0.0002143 0.0002143 0.0002143 0.02% Castro::construct_new_gravity() 10 0.0002114 0.0002114 0.0002114 0.02% Amr::timeStep() 10 0.0002049 0.0002049 0.0002049 0.02% MLMG::MLRhsNormInf() 11 0.0001958 0.0001958 0.0001958 0.02% MLCellLinOp::defineBC() 11 0.0001895 0.0001895 0.0001895 0.02% MLLinOp::defineGrids() 11 0.0001736 0.0001736 0.0001736 0.02% Castro::subcycle_advance_ctu() 10 0.0001694 0.0001694 0.0001694 0.02% MLMG:computeResOfCorrection() 405 0.0001499 0.0001499 0.0001499 0.02% Amr::defBaseLevel() 1 0.0001402 0.0001402 0.0001402 0.02% StateData::checkPoint() 12 0.0001321 0.0001321 0.0001321 0.01% MLMG::actualBottomSolve() 81 0.0001091 0.0001091 0.0001091 0.01% MLMG::mgVcycle_down::0 81 9.181e-05 9.181e-05 9.181e-05 0.01% FabArrayBase::FB::FB() 56 9.06e-05 9.06e-05 9.06e-05 0.01% Castro::construct_new_source() 50 8.617e-05 8.617e-05 8.617e-05 0.01% MLMG::solve() 11 8.133e-05 8.133e-05 8.133e-05 0.01% Castro::initialize_advance() 10 7.958e-05 7.958e-05 7.958e-05 0.01% MLMG::mgVcycle_down::2 81 7.93e-05 7.93e-05 7.93e-05 0.01% MLMG::mgVcycle_down::1 81 7.747e-05 7.747e-05 7.747e-05 0.01% Castro::clean_state() 62 7.422e-05 7.422e-05 7.422e-05 0.01% AmrLevel::checkPoint() 3 7.314e-05 7.314e-05 7.314e-05 0.01% MLMG::mgVcycle_down::3 81 6.984e-05 6.984e-05 6.984e-05 0.01% MLMG::mgVcycle_down::4 81 6.972e-05 6.972e-05 6.972e-05 0.01% Castro::finalize_advance() 10 6.056e-05 6.056e-05 6.056e-05 0.01% Castro::initialize_do_advance() 10 5.86e-05 5.86e-05 5.86e-05 0.01% MLMG::mgVcycle_up::4 81 5.785e-05 5.785e-05 5.785e-05 0.01% MLCellLinOp::solutionResidual() 92 5.011e-05 5.011e-05 5.011e-05 0.01% MLMG::mgVcycle_up::0 81 4.782e-05 4.782e-05 4.782e-05 0.01% Castro::post_timestep() 10 4.74e-05 4.74e-05 4.74e-05 0.01% MLMG::mgVcycle_up::3 81 4.683e-05 4.683e-05 4.683e-05 0.01% MLMG::mgVcycle_up::1 81 4.558e-05 4.558e-05 4.558e-05 0.01% Castro::advance() 10 4.499e-05 4.499e-05 4.499e-05 0.00% StateData::define() 4 4.456e-05 4.456e-05 4.456e-05 0.00% MLMG::mgVcycle_up::2 81 4.411e-05 4.411e-05 4.411e-05 0.00% Castro::finalize_do_advance() 10 4.004e-05 4.004e-05 4.004e-05 0.00% Castro::swap_state_time_levels() 10 3.771e-05 3.771e-05 3.771e-05 0.00% MLMG::mgVcycle_bottom 81 3.479e-05 3.479e-05 3.479e-05 0.00% Castro::enforce_consistent_e() 1 3.359e-05 3.359e-05 3.359e-05 0.00% MLMG::computeResidual() 81 3.357e-05 3.357e-05 3.357e-05 0.00% FillPatchSingleLevel 41 3.064e-05 3.064e-05 3.064e-05 0.00% MLLinOp::define() 11 2.854e-05 2.854e-05 2.854e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.775e-05 2.775e-05 2.775e-05 0.00% Amr::FinalizeInit() 1 2.762e-05 2.762e-05 2.762e-05 0.00% makeSFC 55 2.691e-05 2.691e-05 2.691e-05 0.00% Amr::writeSmallPlotFile() 1 2.613e-05 2.613e-05 2.613e-05 0.00% MLPoisson::define() 11 2.526e-05 2.526e-05 2.526e-05 0.00% Castro::construct_old_source() 50 1.791e-05 1.791e-05 1.791e-05 0.00% Castro::do_new_sources() 10 1.773e-05 1.773e-05 1.773e-05 0.00% Castro::do_old_sources() 10 1.673e-05 1.673e-05 1.673e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.626e-05 1.626e-05 1.626e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.61e-05 1.61e-05 1.61e-05 0.00% DistributionMapping::Distribute() 56 1.61e-05 1.61e-05 1.61e-05 0.00% Castro::check_for_nan() 20 1.302e-05 1.302e-05 1.302e-05 0.00% Castro::apply_source_to_state() 20 1.227e-05 1.227e-05 1.227e-05 0.00% Gravity::swapTimeLevels() 10 1.049e-05 1.049e-05 1.049e-05 0.00% Castro::construct_old_gravity() 10 1.006e-05 1.006e-05 1.006e-05 0.00% AmrLevel::AmrLevel(dm) 1 9.614e-06 9.614e-06 9.614e-06 0.00% Amr::initSubcycle() 1 9.478e-06 9.478e-06 9.478e-06 0.00% MLPoisson::prepareForSolve() 11 9.102e-06 9.102e-06 9.102e-06 0.00% MLMG::computeMLResidual() 11 8.284e-06 8.284e-06 8.284e-06 0.00% Amr::InitializeInit() 1 6.409e-06 6.409e-06 6.409e-06 0.00% Castro::computeNewDt() 9 6.208e-06 6.208e-06 6.208e-06 0.00% MLMG::buildFineMask() 11 6.139e-06 6.139e-06 6.139e-06 0.00% MLMG::getGradSolution() 11 5.912e-06 5.912e-06 5.912e-06 0.00% Castro::create_source_corrector() 10 5.314e-06 5.314e-06 5.314e-06 0.00% MLMG::MLResNormInf() 11 5.31e-06 5.31e-06 5.31e-06 0.00% AmrLevel::checkPointPost() 3 4.813e-06 4.813e-06 4.813e-06 0.00% Castro::FluxRegFineAdd() 10 4.765e-06 4.765e-06 4.765e-06 0.00% Castro::retry_advance_ctu() 10 3.843e-06 3.843e-06 3.843e-06 0.00% Gravity::set_mass_offset() 11 3.809e-06 3.809e-06 3.809e-06 0.00% Castro::post_init() 1 3.775e-06 3.775e-06 3.775e-06 0.00% AmrLevel::checkPointPre() 3 2.794e-06 2.794e-06 2.794e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.706e-06 2.706e-06 2.706e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.373e-06 2.373e-06 2.373e-06 0.00% Amr::init() 1 2.327e-06 2.327e-06 2.327e-06 0.00% Castro::computeInitialDt() 2 2.206e-06 2.206e-06 2.206e-06 0.00% Castro::post_regrid() 1 1.059e-06 1.059e-06 1.059e-06 0.00% Amr::initialInit() 1 8.38e-07 8.38e-07 8.38e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9006 0.9006 0.9006 100.00% Amr::coarseTimeStep() 10 0.7585 0.7585 0.7585 84.22% Amr::timeStep() 10 0.6168 0.6168 0.6168 68.49% Castro::advance() 10 0.6087 0.6087 0.6087 67.59% Castro::subcycle_advance_ctu() 10 0.596 0.596 0.596 66.18% Castro::do_advance_ctu() 10 0.5958 0.5958 0.5958 66.16% Gravity::solve_phi_with_mlmg() 11 0.3172 0.3172 0.3172 35.22% Gravity::actual_solve_with_mlmg() 11 0.3054 0.3054 0.3054 33.91% Castro::construct_new_gravity() 10 0.3002 0.3002 0.3002 33.33% MLMG::solve() 11 0.2829 0.2829 0.2829 31.41% Gravity::solve_for_phi() 10 0.2732 0.2732 0.2732 30.34% MLMG::oneIter() 81 0.2674 0.2674 0.2674 29.69% MLMG::mgVcycle() 81 0.2657 0.2657 0.2657 29.50% Castro::construct_ctu_hydro_source() 10 0.2127 0.2127 0.2127 23.61% Amr::checkPoint() 3 0.1778 0.1778 0.1778 19.75% VisMF::Write(FabArray) 11 0.1481 0.1481 0.1481 16.45% MLCellLinOp::smooth() 1620 0.1409 0.1409 0.1409 15.64% Amr::init() 1 0.1191 0.1191 0.1191 13.23% MLCellLinOp::applyBC() 4379 0.1095 0.1095 0.1095 12.16% AmrLevel::checkPoint() 3 0.1062 0.1062 0.1062 11.79% StateData::checkPoint() 12 0.1061 0.1061 0.1061 11.79% MLMG::mgVcycle_bottom 81 0.07551 0.07551 0.07551 8.38% MLMG::actualBottomSolve() 81 0.07548 0.07548 0.07548 8.38% MLCGSolver::bicgstab 81 0.07477 0.07477 0.07477 8.30% FillPatchIterator::Initialize 41 0.06502 0.06502 0.06502 7.22% FillPatchSingleLevel 41 0.05906 0.05906 0.05906 6.56% MLPoisson::Fsmooth() 3240 0.05856 0.05856 0.05856 6.50% Amr::initialInit() 1 0.05753 0.05753 0.05753 6.39% StateDataPhysBCFunct::() 41 0.05484 0.05484 0.05484 6.09% Amr::FinalizeInit() 1 0.04784 0.04784 0.04784 5.31% Castro::post_init() 1 0.04707 0.04707 0.04707 5.23% Amr::writePlotFile() 2 0.04532 0.04532 0.04532 5.03% Gravity::multilevel_solve_for_new_phi() 1 0.04466 0.04466 0.04466 4.96% Gravity::actual_multilevel_solve() 1 0.04464 0.04464 0.04464 4.96% MLCellLinOp::apply() 1128 0.03862 0.03862 0.03862 4.29% MLMG::mgVcycle_down::0 81 0.03714 0.03714 0.03714 4.12% MLMG::mgVcycle_up::0 81 0.03181 0.03181 0.03181 3.53% Castro::clean_state() 62 0.03064 0.03064 0.03064 3.40% Gravity::get_new_grav_vector() 11 0.02899 0.02899 0.02899 3.22% FabArray::setVal() 1135 0.02653 0.02653 0.02653 2.95% Castro::initialize_do_advance() 10 0.02364 0.02364 0.02364 2.62% StateData::FillBoundary(geom) 328 0.02305 0.02305 0.02305 2.56% MLCellLinOp::correctionResidual() 486 0.02166 0.02166 0.02166 2.41% MultiFab::Dot() 1100 0.0207 0.0207 0.0207 2.30% MLMG:computeResOfCorrection() 405 0.01874 0.01874 0.01874 2.08% Castro::expand_state() 10 0.01851 0.01851 0.01851 2.06% FabArray::FillBoundary() 3974 0.01822 0.01822 0.01822 2.02% MLMG::mgVcycle_down::1 81 0.01808 0.01808 0.01808 2.01% MLMG::mgVcycle_down::2 81 0.01748 0.01748 0.01748 1.94% FillBoundary_nowait() 3974 0.01738 0.01738 0.01738 1.93% Castro::computeTemp() 63 0.01714 0.01714 0.01714 1.90% MLPoisson::define() 11 0.01705 0.01705 0.01705 1.89% FabArray::ParallelCopy() 851 0.01688 0.01688 0.01688 1.87% MLMG::mgVcycle_down::3 81 0.01669 0.01669 0.01669 1.85% FabArray::ParallelCopy_nowait() 851 0.01659 0.01659 0.01659 1.84% Castro::construct_old_gravity() 10 0.01593 0.01593 0.01593 1.77% Gravity::get_old_grav_vector() 10 0.01592 0.01592 0.01592 1.77% MLMG::mgVcycle_down::4 81 0.01588 0.01588 0.01588 1.76% MLMG::mgVcycle_up::2 81 0.01364 0.01364 0.01364 1.51% MLMG::mgVcycle_up::1 81 0.0134 0.0134 0.0134 1.49% MLMG::addInterpCorrection() 405 0.01322 0.01322 0.01322 1.47% MLMG::mgVcycle_up::3 81 0.01294 0.01294 0.01294 1.44% amrex::average_down 405 0.01279 0.01279 0.01279 1.42% MLMG::mgVcycle_up::4 81 0.01273 0.01273 0.01273 1.41% Castro::initialize_advance() 10 0.01255 0.01255 0.01255 1.39% MLCGSolver::ParallelAllReduce 1495 0.01246 0.01246 0.01246 1.38% CGSolver::sxay() 1566 0.01226 0.01226 0.01226 1.36% MultiFab::LinComb() 1566 0.01181 0.01181 0.01181 1.31% MLCellLinOp::defineAuxData() 11 0.01172 0.01172 0.01172 1.30% Gravity::fill_multipole_BCs() 11 0.0115 0.0115 0.0115 1.28% Castro::do_new_sources() 10 0.01097 0.01097 0.01097 1.22% MLPoisson::Fapply() 1128 0.01026 0.01026 0.01026 1.14% Amr::InitializeInit() 1 0.009685 0.009685 0.009685 1.08% Amr::defBaseLevel() 1 0.009679 0.009679 0.009679 1.07% Castro::post_timestep() 10 0.007908 0.007908 0.007908 0.88% Castro::do_old_sources() 10 0.007753 0.007753 0.007753 0.86% MLCellLinOp::solutionResidual() 92 0.007529 0.007529 0.007529 0.84% Castro::reset_internal_energy() 63 0.006877 0.006877 0.006877 0.76% MLMG::computeResidual() 81 0.006496 0.006496 0.006496 0.72% Castro::enforce_min_density() 62 0.006298 0.006298 0.006298 0.70% MultiFab::Xpay() 578 0.005922 0.005922 0.005922 0.66% MLMG::prepareForSolve() 11 0.005818 0.005818 0.005818 0.65% FabArray::setDomainBndry() 41 0.005693 0.005693 0.005693 0.63% MLCellLinOp::defineBC() 11 0.005036 0.005036 0.005036 0.56% BndryData::define() 11 0.004846 0.004846 0.004846 0.54% Castro::normalize_species() 62 0.004359 0.004359 0.004359 0.48% Castro::estTimeStep() 21 0.004158 0.004158 0.004158 0.46% Castro::initData() 1 0.003616 0.003616 0.003616 0.40% Castro::enforce_speed_limit() 62 0.003611 0.003611 0.003611 0.40% Castro::construct_new_source() 50 0.003239 0.003239 0.003239 0.36% Castro::construct_new_gravity_source() 10 0.003153 0.003153 0.003153 0.35% MLMG::getGradSolution() 11 0.002535 0.002535 0.002535 0.28% MLCellLinOp::compGrad() 11 0.002529 0.002529 0.002529 0.28% Castro::construct_old_source() 50 0.002516 0.002516 0.002516 0.28% Castro::construct_old_gravity_source() 10 0.002498 0.002498 0.002498 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002252 0.002252 0.002252 0.25% MLMG::ResNormInf() 92 0.00186 0.00186 0.00186 0.21% Castro::apply_source_to_state() 20 0.001821 0.001821 0.001821 0.20% MultiFab::Saxpy() 20 0.001809 0.001809 0.001809 0.20% Castro::computeNewDt() 9 0.001652 0.001652 0.001652 0.18% FabArrayBase::getCPC() 1313 0.001566 0.001566 0.001566 0.17% MLCellLinOp::setLevelBC() 11 0.001513 0.001513 0.001513 0.17% MLPoisson::prepareForSolve() 11 0.001364 0.001364 0.001364 0.15% MLCellLinOp::prepareForSolve() 11 0.001355 0.001355 0.001355 0.15% FabArray::mult() 43 0.001333 0.001333 0.001333 0.15% Castro::check_for_nan() 20 0.001185 0.001185 0.001185 0.13% MultiFab::contains_nan() 20 0.001172 0.001172 0.001172 0.13% Gravity::swapTimeLevels() 10 0.001084 0.001084 0.001084 0.12% MLMG::computeMLResidual() 11 0.001074 0.001074 0.001074 0.12% Gravity::update_max_rhs() 11 0.0009677 0.0009677 0.0009677 0.11% FabArrayBase::getFB() 3974 0.0008002 0.0008002 0.0008002 0.09% FabArrayBase::CPC::define() 454 0.0007225 0.0007225 0.0007225 0.08% Castro::computeInitialDt() 2 0.0006999 0.0006999 0.0006999 0.08% Castro::post_regrid() 1 0.000527 0.000527 0.000527 0.06% MultiFab::Copy() 11 0.000519 0.000519 0.000519 0.06% Amr::InitAmr() 1 0.0004454 0.0004454 0.0004454 0.05% MLLinOp::define() 11 0.0002623 0.0002623 0.0002623 0.03% MLMG::MLResNormInf() 11 0.0002512 0.0002512 0.0002512 0.03% MultiFab::max() 11 0.0002507 0.0002507 0.0002507 0.03% MLLinOp::defineGrids() 11 0.0002337 0.0002337 0.0002337 0.03% MLMG::MLRhsNormInf() 11 0.0001958 0.0001958 0.0001958 0.02% FabArrayBase::FB::FB() 56 9.06e-05 9.06e-05 9.06e-05 0.01% Castro::finalize_advance() 10 6.532e-05 6.532e-05 6.532e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.772e-05 5.772e-05 5.772e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.418e-05 5.418e-05 5.418e-05 0.01% StateData::define() 4 4.456e-05 4.456e-05 4.456e-05 0.00% makeSFC 55 4.162e-05 4.162e-05 4.162e-05 0.00% Castro::finalize_do_advance() 10 4.004e-05 4.004e-05 4.004e-05 0.00% Castro::swap_state_time_levels() 10 3.771e-05 3.771e-05 3.771e-05 0.00% Castro::enforce_consistent_e() 1 3.359e-05 3.359e-05 3.359e-05 0.00% Amr::writeSmallPlotFile() 1 2.613e-05 2.613e-05 2.613e-05 0.00% DistributionMapping::Distribute() 56 1.61e-05 1.61e-05 1.61e-05 0.00% Amr::initSubcycle() 1 9.478e-06 9.478e-06 9.478e-06 0.00% MLMG::buildFineMask() 11 6.139e-06 6.139e-06 6.139e-06 0.00% Castro::create_source_corrector() 10 5.314e-06 5.314e-06 5.314e-06 0.00% AmrLevel::checkPointPost() 3 4.813e-06 4.813e-06 4.813e-06 0.00% Castro::FluxRegFineAdd() 10 4.765e-06 4.765e-06 4.765e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.093e-06 4.093e-06 4.093e-06 0.00% Castro::retry_advance_ctu() 10 3.843e-06 3.843e-06 3.843e-06 0.00% Gravity::set_mass_offset() 11 3.809e-06 3.809e-06 3.809e-06 0.00% AmrLevel::checkPointPre() 3 2.794e-06 2.794e-06 2.794e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.373e-06 2.373e-06 2.373e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-28-g56661522b4cd) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-28-g56661522b4cd) initialized Starting run at 08:22:11 UTC on 2022-04-25. Successfully read inputs file ... Castro git describe: 22.04-28-g7e86dbe56 AMReX git describe: 22.04-28-g56661522b Microphysics git describe: 22.04-3-g3c498521 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.557945699 Restart time = 0.05531616 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.114814679 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050150966 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.059029368 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.066136982 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.069011207 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.023602331 seconds Ending run at 08:22:11 UTC on 2022-04-25. Run time = 0.438947294 Run time without initialization = 0.383084352 Average number of zones advanced per microsecond: 3.421 Average number of zones advanced per microsecond per rank: 3.421 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.439 ... 0.439 ... 0.439 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1525 0.1525 0.1525 34.75% VisMF::Read() 3 0.04358 0.04358 0.04358 9.93% MLCellLinOp::applyBC() 1946 0.03737 0.03737 0.03737 8.51% MLPoisson::Fsmooth() 1440 0.02558 0.02558 0.02558 5.83% VisMF::Write(FabArray) 1 0.02227 0.02227 0.02227 5.07% FabArray::setVal() 537 0.01658 0.01658 0.01658 3.78% StateData::FillBoundary(geom) 160 0.01141 0.01141 0.01141 2.60% MLCGSolver::bicgstab 36 0.009271 0.009271 0.009271 2.11% MultiFab::Dot() 484 0.009034 0.009034 0.009034 2.06% FillBoundary_nowait() 1766 0.007106 0.007106 0.007106 1.62% FabArray::ParallelCopy_nowait() 380 0.006976 0.006976 0.006976 1.59% Castro::computeTemp() 30 0.006839 0.006839 0.006839 1.56% StateDataPhysBCFunct::() 20 0.006387 0.006387 0.006387 1.45% Gravity::fill_multipole_BCs() 6 0.006316 0.006316 0.006316 1.44% MLCellLinOp::defineAuxData() 6 0.005498 0.005498 0.005498 1.25% MultiFab::LinComb() 690 0.005187 0.005187 0.005187 1.18% Castro::enforce_min_density() 30 0.0046 0.0046 0.0046 1.05% MLPoisson::Fapply() 500 0.004565 0.004565 0.004565 1.04% Castro::expand_state() 5 0.004127 0.004127 0.004127 0.94% FabArray::setDomainBndry() 20 0.003872 0.003872 0.003872 0.88% Castro::do_advance_ctu() 5 0.003769 0.003769 0.003769 0.86% Amr::restart() 1 0.002969 0.002969 0.002969 0.68% MLMG::addInterpCorrection() 180 0.002961 0.002961 0.002961 0.67% amrex::average_down 180 0.002802 0.002802 0.002802 0.64% MultiFab::Xpay() 258 0.002652 0.002652 0.002652 0.60% Castro::estTimeStep() 10 0.002562 0.002562 0.002562 0.58% Castro::reset_internal_energy() 30 0.002449 0.002449 0.002449 0.56% Castro::normalize_species() 30 0.002169 0.002169 0.002169 0.49% BndryData::define() 6 0.001965 0.001965 0.001965 0.45% Gravity::get_new_grav_vector() 5 0.001808 0.001808 0.001808 0.41% Castro::construct_new_gravity_source() 5 0.001636 0.001636 0.001636 0.37% Castro::enforce_speed_limit() 30 0.001588 0.001588 0.001588 0.36% MLCellLinOp::compGrad() 6 0.001462 0.001462 0.001462 0.33% Amr::writePlotFile() 1 0.001419 0.001419 0.001419 0.32% Castro::construct_old_gravity_source() 5 0.001377 0.001377 0.001377 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001197 0.001197 0.001197 0.27% MultiFab::Saxpy() 10 0.0009222 0.0009222 0.0009222 0.21% Gravity::get_old_grav_vector() 5 0.0009081 0.0009081 0.0009081 0.21% MLCellLinOp::setLevelBC() 6 0.0008305 0.0008305 0.0008305 0.19% MLMG::ResNormInf() 42 0.0008296 0.0008296 0.0008296 0.19% MLMG::oneIter() 36 0.0007442 0.0007442 0.0007442 0.17% Gravity::actual_solve_with_mlmg() 6 0.0007226 0.0007226 0.0007226 0.16% MLCellLinOp::prepareForSolve() 6 0.0006908 0.0006908 0.0006908 0.16% FabArray::mult() 22 0.0006584 0.0006584 0.0006584 0.15% MultiFab::contains_nan() 10 0.0005854 0.0005854 0.0005854 0.13% MLCellLinOp::smooth() 720 0.0005386 0.0005386 0.0005386 0.12% Gravity::update_max_rhs() 6 0.0005085 0.0005085 0.0005085 0.12% FabArrayBase::CPC::define() 244 0.0004377 0.0004377 0.0004377 0.10% MLMG::prepareForSolve() 6 0.0004375 0.0004375 0.0004375 0.10% FabArrayBase::getCPC() 632 0.0004314 0.0004314 0.0004314 0.10% FabArray::FillBoundary() 1766 0.0004079 0.0004079 0.0004079 0.09% Amr::InitAmr() 1 0.0003761 0.0003761 0.0003761 0.09% MultiFab::Copy() 6 0.0003719 0.0003719 0.0003719 0.08% FabArrayBase::getFB() 1766 0.0003167 0.0003167 0.0003167 0.07% Gravity::solve_for_phi() 5 0.0002804 0.0002804 0.0002804 0.06% main() 1 0.0002682 0.0002682 0.0002682 0.06% MLCellLinOp::apply() 500 0.000233 0.000233 0.000233 0.05% Gravity::actual_multilevel_solve() 1 0.0002308 0.0002308 0.0002308 0.05% CGSolver::sxay() 690 0.0001929 0.0001929 0.0001929 0.04% MLMG::mgVcycle() 36 0.0001925 0.0001925 0.0001925 0.04% MLCGSolver::ParallelAllReduce 659 0.0001554 0.0001554 0.0001554 0.04% Castro::construct_new_source() 25 0.0001552 0.0001552 0.0001552 0.04% FabArray::ParallelCopy() 380 0.0001435 0.0001435 0.0001435 0.03% MultiFab::max() 6 0.0001327 0.0001327 0.0001327 0.03% Castro::subcycle_advance_ctu() 5 0.0001287 0.0001287 0.0001287 0.03% FillPatchIterator::Initialize 20 0.000122 0.000122 0.000122 0.03% Castro::construct_new_gravity() 5 0.0001143 0.0001143 0.0001143 0.03% MLCellLinOp::correctionResidual() 216 0.0001099 0.0001099 0.0001099 0.03% Amr::coarseTimeStep() 5 0.0001082 0.0001082 0.0001082 0.02% MLMG::MLRhsNormInf() 6 0.0001056 0.0001056 0.0001056 0.02% Amr::timeStep() 5 0.0001039 0.0001039 0.0001039 0.02% MLCellLinOp::defineBC() 6 0.0001018 0.0001018 0.0001018 0.02% MLLinOp::defineGrids() 6 9.461e-05 9.461e-05 9.461e-05 0.02% AmrLevel::restart() 1 7.063e-05 7.063e-05 7.063e-05 0.02% StateData::restartDoit() 4 6.991e-05 6.991e-05 6.991e-05 0.02% MLMG:computeResOfCorrection() 180 6.421e-05 6.421e-05 6.421e-05 0.01% FabArrayBase::FB::FB() 26 5.991e-05 5.991e-05 5.991e-05 0.01% Castro::create_source_corrector() 5 5.852e-05 5.852e-05 5.852e-05 0.01% Castro::construct_old_source() 25 5.447e-05 5.447e-05 5.447e-05 0.01% Castro::advance() 5 5.003e-05 5.003e-05 5.003e-05 0.01% MLMG::actualBottomSolve() 36 4.87e-05 4.87e-05 4.87e-05 0.01% MLMG::solve() 6 4.295e-05 4.295e-05 4.295e-05 0.01% MLMG::mgVcycle_down::0 36 4.074e-05 4.074e-05 4.074e-05 0.01% MLMG::mgVcycle_down::1 36 4.048e-05 4.048e-05 4.048e-05 0.01% MLMG::mgVcycle_down::2 36 3.784e-05 3.784e-05 3.784e-05 0.01% Castro::do_new_sources() 5 3.77e-05 3.77e-05 3.77e-05 0.01% MLMG::mgVcycle_down::4 36 3.668e-05 3.668e-05 3.668e-05 0.01% Castro::clean_state() 30 3.592e-05 3.592e-05 3.592e-05 0.01% MLMG::mgVcycle_down::3 36 3.569e-05 3.569e-05 3.569e-05 0.01% Castro::initialize_advance() 5 3.529e-05 3.529e-05 3.529e-05 0.01% Castro::initialize_do_advance() 5 2.87e-05 2.87e-05 2.87e-05 0.01% MLMG::mgVcycle_up::4 36 2.796e-05 2.796e-05 2.796e-05 0.01% Castro::swap_state_time_levels() 5 2.67e-05 2.67e-05 2.67e-05 0.01% MLLinOp::define() 6 2.67e-05 2.67e-05 2.67e-05 0.01% Castro::finalize_advance() 5 2.581e-05 2.581e-05 2.581e-05 0.01% Amr::writeSmallPlotFile() 1 2.505e-05 2.505e-05 2.505e-05 0.01% MLMG::mgVcycle_up::0 36 2.491e-05 2.491e-05 2.491e-05 0.01% MLMG::mgVcycle_up::3 36 2.44e-05 2.44e-05 2.44e-05 0.01% MLMG::mgVcycle_up::2 36 2.354e-05 2.354e-05 2.354e-05 0.01% MLCellLinOp::solutionResidual() 42 2.341e-05 2.341e-05 2.341e-05 0.01% Castro::post_restart() 1 2.314e-05 2.314e-05 2.314e-05 0.01% Castro::post_timestep() 5 2.294e-05 2.294e-05 2.294e-05 0.01% MLMG::mgVcycle_up::1 36 2.279e-05 2.279e-05 2.279e-05 0.01% Castro::computeNewDt() 5 2.253e-05 2.253e-05 2.253e-05 0.01% Castro::finalize_do_advance() 5 2.1e-05 2.1e-05 2.1e-05 0.00% MLMG::mgVcycle_bottom 36 1.659e-05 1.659e-05 1.659e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.571e-05 1.571e-05 1.571e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.56e-05 1.56e-05 1.56e-05 0.00% MLPoisson::define() 6 1.55e-05 1.55e-05 1.55e-05 0.00% MLMG::computeResidual() 36 1.535e-05 1.535e-05 1.535e-05 0.00% makeSFC 30 1.518e-05 1.518e-05 1.518e-05 0.00% FillPatchSingleLevel 20 1.484e-05 1.484e-05 1.484e-05 0.00% DistributionMapping::Distribute() 31 9.839e-06 9.839e-06 9.839e-06 0.00% Amr::initSubcycle() 1 9.834e-06 9.834e-06 9.834e-06 0.00% Castro::do_old_sources() 5 8.444e-06 8.444e-06 8.444e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.369e-06 8.369e-06 8.369e-06 0.00% Castro::apply_source_to_state() 10 6.098e-06 6.098e-06 6.098e-06 0.00% Castro::check_for_nan() 10 5.902e-06 5.902e-06 5.902e-06 0.00% Castro::construct_old_gravity() 5 5.459e-06 5.459e-06 5.459e-06 0.00% MLPoisson::prepareForSolve() 6 4.966e-06 4.966e-06 4.966e-06 0.00% Gravity::swapTimeLevels() 5 4.069e-06 4.069e-06 4.069e-06 0.00% MLMG::computeMLResidual() 6 3.481e-06 3.481e-06 3.481e-06 0.00% MLMG::buildFineMask() 6 3.457e-06 3.457e-06 3.457e-06 0.00% MLMG::getGradSolution() 6 3.264e-06 3.264e-06 3.264e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.985e-06 2.985e-06 2.985e-06 0.00% Gravity::set_mass_offset() 6 2.848e-06 2.848e-06 2.848e-06 0.00% MLMG::MLResNormInf() 6 2.615e-06 2.615e-06 2.615e-06 0.00% Castro::retry_advance_ctu() 5 2.005e-06 2.005e-06 2.005e-06 0.00% Castro::FluxRegFineAdd() 5 1.876e-06 1.876e-06 1.876e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.272e-06 1.272e-06 1.272e-06 0.00% Amr::init() 1 9.92e-07 9.92e-07 9.92e-07 0.00% AmrLevel::AmrLevel() 1 9e-07 9e-07 9e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.439 0.439 0.439 100.00% Amr::coarseTimeStep() 5 0.3592 0.3592 0.3592 81.84% Amr::timeStep() 5 0.358 0.358 0.358 81.56% Castro::advance() 5 0.3546 0.3546 0.3546 80.77% Castro::subcycle_advance_ctu() 5 0.3434 0.3434 0.3434 78.22% Castro::do_advance_ctu() 5 0.3432 0.3432 0.3432 78.19% Castro::construct_ctu_hydro_source() 5 0.1525 0.1525 0.1525 34.75% Castro::construct_new_gravity() 5 0.145 0.145 0.145 33.04% Gravity::solve_phi_with_mlmg() 6 0.1413 0.1413 0.1413 32.20% Gravity::solve_for_phi() 5 0.1356 0.1356 0.1356 30.89% Gravity::actual_solve_with_mlmg() 6 0.1349 0.1349 0.1349 30.73% MLMG::solve() 6 0.1222 0.1222 0.1222 27.84% MLMG::oneIter() 36 0.1144 0.1144 0.1144 26.05% MLMG::mgVcycle() 36 0.1136 0.1136 0.1136 25.88% MLCellLinOp::smooth() 720 0.05956 0.05956 0.05956 13.57% Amr::init() 1 0.05535 0.05535 0.05535 12.61% Amr::restart() 1 0.05535 0.05535 0.05535 12.61% MLCellLinOp::applyBC() 1946 0.04526 0.04526 0.04526 10.31% AmrLevel::restart() 1 0.04451 0.04451 0.04451 10.14% StateData::restartDoit() 4 0.04444 0.04444 0.04444 10.12% VisMF::Read() 3 0.04358 0.04358 0.04358 9.93% MLMG::mgVcycle_bottom 36 0.03268 0.03268 0.03268 7.44% MLMG::actualBottomSolve() 36 0.03266 0.03266 0.03266 7.44% MLCGSolver::bicgstab 36 0.03235 0.03235 0.03235 7.37% MLPoisson::Fsmooth() 1440 0.02558 0.02558 0.02558 5.83% FillPatchIterator::Initialize 20 0.02388 0.02388 0.02388 5.44% Amr::writePlotFile() 1 0.02369 0.02369 0.02369 5.40% VisMF::Write(FabArray) 1 0.02227 0.02227 0.02227 5.07% FillPatchSingleLevel 20 0.01988 0.01988 0.01988 4.53% StateDataPhysBCFunct::() 20 0.0178 0.0178 0.0178 4.05% Castro::clean_state() 30 0.01768 0.01768 0.01768 4.03% FabArray::setVal() 537 0.01658 0.01658 0.01658 3.78% MLCellLinOp::apply() 500 0.01634 0.01634 0.01634 3.72% MLMG::mgVcycle_down::0 36 0.01595 0.01595 0.01595 3.63% Castro::initialize_do_advance() 5 0.01408 0.01408 0.01408 3.21% MLMG::mgVcycle_up::0 36 0.01355 0.01355 0.01355 3.09% StateData::FillBoundary(geom) 160 0.01141 0.01141 0.01141 2.60% Castro::expand_state() 5 0.01131 0.01131 0.01131 2.58% Castro::initialize_advance() 5 0.0111 0.0111 0.0111 2.53% MLPoisson::define() 6 0.009374 0.009374 0.009374 2.14% Gravity::get_new_grav_vector() 5 0.009321 0.009321 0.009321 2.12% Castro::computeTemp() 30 0.009288 0.009288 0.009288 2.12% MLCellLinOp::correctionResidual() 216 0.009259 0.009259 0.009259 2.11% MultiFab::Dot() 484 0.009034 0.009034 0.009034 2.06% MLMG:computeResOfCorrection() 180 0.008034 0.008034 0.008034 1.83% Castro::construct_old_gravity() 5 0.007907 0.007907 0.007907 1.80% Gravity::get_old_grav_vector() 5 0.007901 0.007901 0.007901 1.80% FabArray::FillBoundary() 1766 0.007891 0.007891 0.007891 1.80% MLMG::mgVcycle_down::1 36 0.007725 0.007725 0.007725 1.76% Castro::do_new_sources() 5 0.007668 0.007668 0.007668 1.75% FabArray::ParallelCopy() 380 0.007547 0.007547 0.007547 1.72% FillBoundary_nowait() 1766 0.007483 0.007483 0.007483 1.70% MLMG::mgVcycle_down::2 36 0.007416 0.007416 0.007416 1.69% FabArray::ParallelCopy_nowait() 380 0.007403 0.007403 0.007403 1.69% MLMG::mgVcycle_down::3 36 0.007057 0.007057 0.007057 1.61% MLMG::mgVcycle_down::4 36 0.006753 0.006753 0.006753 1.54% Castro::post_restart() 1 0.006634 0.006634 0.006634 1.51% MLCellLinOp::defineAuxData() 6 0.006433 0.006433 0.006433 1.47% Gravity::fill_multipole_BCs() 6 0.006316 0.006316 0.006316 1.44% Gravity::multilevel_solve_for_new_phi() 1 0.00626 0.00626 0.00626 1.43% Gravity::actual_multilevel_solve() 1 0.006245 0.006245 0.006245 1.42% MLMG::mgVcycle_up::2 36 0.005772 0.005772 0.005772 1.31% MLMG::addInterpCorrection() 180 0.005692 0.005692 0.005692 1.30% MLMG::mgVcycle_up::1 36 0.005676 0.005676 0.005676 1.29% amrex::average_down 180 0.005551 0.005551 0.005551 1.26% MLCGSolver::ParallelAllReduce 659 0.005488 0.005488 0.005488 1.25% MLMG::mgVcycle_up::3 36 0.005461 0.005461 0.005461 1.24% CGSolver::sxay() 690 0.00538 0.00538 0.00538 1.23% MLMG::mgVcycle_up::4 36 0.005378 0.005378 0.005378 1.23% MultiFab::LinComb() 690 0.005187 0.005187 0.005187 1.18% Castro::enforce_min_density() 30 0.0046 0.0046 0.0046 1.05% MLPoisson::Fapply() 500 0.004565 0.004565 0.004565 1.04% Castro::do_old_sources() 5 0.004343 0.004343 0.004343 0.99% FabArray::setDomainBndry() 20 0.003872 0.003872 0.003872 0.88% MLMG::prepareForSolve() 6 0.003489 0.003489 0.003489 0.79% MLCellLinOp::solutionResidual() 42 0.003369 0.003369 0.003369 0.77% Castro::post_timestep() 5 0.003362 0.003362 0.003362 0.77% MLMG::computeResidual() 36 0.002794 0.002794 0.002794 0.64% MLCellLinOp::defineBC() 6 0.002771 0.002771 0.002771 0.63% BndryData::define() 6 0.002669 0.002669 0.002669 0.61% MultiFab::Xpay() 258 0.002652 0.002652 0.002652 0.60% Castro::estTimeStep() 10 0.002562 0.002562 0.002562 0.58% Castro::reset_internal_energy() 30 0.002449 0.002449 0.002449 0.56% Castro::normalize_species() 30 0.002169 0.002169 0.002169 0.49% Castro::construct_new_source() 25 0.001791 0.001791 0.001791 0.41% MLMG::getGradSolution() 6 0.001749 0.001749 0.001749 0.40% MLCellLinOp::compGrad() 6 0.001745 0.001745 0.001745 0.40% Castro::construct_new_gravity_source() 5 0.001636 0.001636 0.001636 0.37% Castro::enforce_speed_limit() 30 0.001588 0.001588 0.001588 0.36% Castro::construct_old_source() 25 0.001431 0.001431 0.001431 0.33% Castro::construct_old_gravity_source() 5 0.001377 0.001377 0.001377 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001197 0.001197 0.001197 0.27% Castro::computeNewDt() 5 0.001116 0.001116 0.001116 0.25% Castro::apply_source_to_state() 10 0.0009283 0.0009283 0.0009283 0.21% MultiFab::Saxpy() 10 0.0009222 0.0009222 0.0009222 0.21% Gravity::swapTimeLevels() 5 0.0008908 0.0008908 0.0008908 0.20% FabArrayBase::getCPC() 632 0.0008691 0.0008691 0.0008691 0.20% MLCellLinOp::setLevelBC() 6 0.0008305 0.0008305 0.0008305 0.19% MLMG::ResNormInf() 42 0.0008296 0.0008296 0.0008296 0.19% Gravity::update_max_rhs() 6 0.0007179 0.0007179 0.0007179 0.16% MLPoisson::prepareForSolve() 6 0.0006958 0.0006958 0.0006958 0.16% MLCellLinOp::prepareForSolve() 6 0.0006908 0.0006908 0.0006908 0.16% FabArray::mult() 22 0.0006584 0.0006584 0.0006584 0.15% MLMG::computeMLResidual() 6 0.0005939 0.0005939 0.0005939 0.14% Castro::check_for_nan() 10 0.0005913 0.0005913 0.0005913 0.13% MultiFab::contains_nan() 10 0.0005854 0.0005854 0.0005854 0.13% FabArrayBase::CPC::define() 244 0.0004377 0.0004377 0.0004377 0.10% Amr::InitAmr() 1 0.000386 0.000386 0.000386 0.09% FabArrayBase::getFB() 1766 0.0003766 0.0003766 0.0003766 0.09% MultiFab::Copy() 6 0.0003719 0.0003719 0.0003719 0.08% MLLinOp::define() 6 0.0001545 0.0001545 0.0001545 0.04% MLMG::MLResNormInf() 6 0.0001336 0.0001336 0.0001336 0.03% MultiFab::max() 6 0.0001327 0.0001327 0.0001327 0.03% MLLinOp::defineGrids() 6 0.0001278 0.0001278 0.0001278 0.03% MLMG::MLRhsNormInf() 6 0.0001056 0.0001056 0.0001056 0.02% FabArrayBase::FB::FB() 26 5.991e-05 5.991e-05 5.991e-05 0.01% Castro::create_source_corrector() 5 5.852e-05 5.852e-05 5.852e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.193e-05 3.193e-05 3.193e-05 0.01% Castro::finalize_advance() 5 2.769e-05 2.769e-05 2.769e-05 0.01% Castro::swap_state_time_levels() 5 2.67e-05 2.67e-05 2.67e-05 0.01% Amr::writeSmallPlotFile() 1 2.505e-05 2.505e-05 2.505e-05 0.01% makeSFC 30 2.356e-05 2.356e-05 2.356e-05 0.01% Castro::finalize_do_advance() 5 2.1e-05 2.1e-05 2.1e-05 0.00% DistributionMapping::Distribute() 31 9.839e-06 9.839e-06 9.839e-06 0.00% Amr::initSubcycle() 1 9.834e-06 9.834e-06 9.834e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.442e-06 4.442e-06 4.442e-06 0.00% MLMG::buildFineMask() 6 3.457e-06 3.457e-06 3.457e-06 0.00% Gravity::set_mass_offset() 6 2.848e-06 2.848e-06 2.848e-06 0.00% Castro::retry_advance_ctu() 5 2.005e-06 2.005e-06 2.005e-06 0.00% Castro::FluxRegFineAdd() 5 1.876e-06 1.876e-06 1.876e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.272e-06 1.272e-06 1.272e-06 0.00% AmrLevel::AmrLevel() 1 9e-07 9e-07 9e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-28-g56661522b4cd) finalized