Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-5-g3ca867b8cce8) initialized Starting run at 08:29:37 UTC on 2022-05-10. Successfully read inputs file ... Castro git describe: 22.05-12-gb79cf4624 AMReX git describe: 22.05-5-g3ca867b8c Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.039324419 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022581498 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.046721694 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.049591482 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.065358075 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.058536994 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.05721706 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.036006853 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.0491942 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047449033 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058275066 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.058739809 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.05905393 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.048744779 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.022489103 seconds Ending run at 08:29:37 UTC on 2022-05-10. Run time = 0.769656836 Run time without initialization = 0.657898613 Average number of zones advanced per microsecond: 3.985 Average number of zones advanced per microsecond per rank: 3.985 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.7697 ... 0.7697 ... 0.7697 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1726 0.1726 0.1726 22.43% VisMF::Write(FabArray) 11 0.1509 0.1509 0.1509 19.60% MLCellLinOp::applyBC() 4433 0.07252 0.07252 0.07252 9.42% MLPoisson::Fsmooth() 3280 0.06041 0.06041 0.06041 7.85% StateData::FillBoundary(geom) 328 0.02337 0.02337 0.02337 3.04% MLCGSolver::bicgstab 82 0.02117 0.02117 0.02117 2.75% MultiFab::Dot() 1114 0.02068 0.02068 0.02068 2.69% Castro::normalize_species() 62 0.01846 0.01846 0.01846 2.40% StateDataPhysBCFunct::() 41 0.01729 0.01729 0.01729 2.25% Amr::checkPoint() 3 0.01589 0.01589 0.01589 2.06% Castro::computeTemp() 63 0.01572 0.01572 0.01572 2.04% FillBoundary_nowait() 4023 0.01423 0.01423 0.01423 1.85% Castro::enforce_min_density() 62 0.01304 0.01304 0.01304 1.69% FabArray::setVal() 1144 0.01302 0.01302 0.01302 1.69% FabArray::ParallelCopy_nowait() 861 0.0129 0.0129 0.0129 1.68% MultiFab::LinComb() 1586 0.01246 0.01246 0.01246 1.62% MLPoisson::Fapply() 1142 0.01058 0.01058 0.01058 1.37% MLCellLinOp::defineAuxData() 11 0.009645 0.009645 0.009645 1.25% Gravity::fill_multipole_BCs() 11 0.008487 0.008487 0.008487 1.10% Castro::estTimeStep() 21 0.007363 0.007363 0.007363 0.96% MLMG::addInterpCorrection() 410 0.006574 0.006574 0.006574 0.85% amrex::average_down 410 0.006134 0.006134 0.006134 0.80% MultiFab::Xpay() 585 0.00611 0.00611 0.00611 0.79% Castro::do_advance_ctu() 10 0.004941 0.004941 0.004941 0.64% Castro::reset_internal_energy(MultiFab) 63 0.00488 0.00488 0.00488 0.63% BndryData::define() 11 0.003575 0.003575 0.003575 0.46% Castro::construct_new_gravity_source() 10 0.002998 0.002998 0.002998 0.39% Castro::construct_old_gravity_source() 10 0.00249 0.00249 0.00249 0.32% Amr::writePlotFile() 2 0.002404 0.002404 0.002404 0.31% MLMG::ResNormInf() 93 0.001973 0.001973 0.001973 0.26% Gravity::get_new_grav_vector() 11 0.001915 0.001915 0.001915 0.25% MultiFab::Saxpy() 20 0.00181 0.00181 0.00181 0.24% Gravity::get_old_grav_vector() 10 0.001726 0.001726 0.001726 0.22% Castro::expand_state() 10 0.001726 0.001726 0.001726 0.22% MLMG::oneIter() 82 0.001685 0.001685 0.001685 0.22% Castro::enforce_speed_limit() 62 0.001659 0.001659 0.001659 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001629 0.001629 0.001629 0.21% Castro::reset_internal_energy(Fab) 504 0.00152 0.00152 0.00152 0.20% Gravity::actual_solve_with_mlmg() 11 0.00142 0.00142 0.00142 0.18% FabArray::mult() 43 0.001329 0.001329 0.001329 0.17% MLCellLinOp::setLevelBC() 11 0.001297 0.001297 0.001297 0.17% FabArray::setDomainBndry() 41 0.001286 0.001286 0.001286 0.17% MultiFab::contains_nan() 20 0.00118 0.00118 0.00118 0.15% Castro::initData() 1 0.001148 0.001148 0.001148 0.15% MLCellLinOp::smooth() 1640 0.001057 0.001057 0.001057 0.14% MLCellLinOp::prepareForSolve() 11 0.001056 0.001056 0.001056 0.14% MLCellLinOp::compGrad() 11 0.0008872 0.0008872 0.0008872 0.12% FabArrayBase::getCPC() 1323 0.0008573 0.0008573 0.0008573 0.11% FabArray::FillBoundary() 4023 0.0007968 0.0007968 0.0007968 0.10% MLMG::prepareForSolve() 11 0.0007726 0.0007726 0.0007726 0.10% FabArrayBase::CPC::define() 454 0.0006912 0.0006912 0.0006912 0.09% FabArrayBase::getFB() 4023 0.0006824 0.0006824 0.0006824 0.09% MLCellLinOp::apply() 1142 0.0005181 0.0005181 0.0005181 0.07% CGSolver::sxay() 1586 0.000459 0.000459 0.000459 0.06% Amr::InitAmr() 1 0.0004358 0.0004358 0.0004358 0.06% MLLinOp::defineGrids() 11 0.0004084 0.0004084 0.0004084 0.05% Gravity::update_max_rhs() 11 0.0004074 0.0004074 0.0004074 0.05% Gravity::solve_for_phi() 10 0.0003977 0.0003977 0.0003977 0.05% MLMG::mgVcycle() 82 0.0003952 0.0003952 0.0003952 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003434 0.0003434 0.0003434 0.04% main() 1 0.0002841 0.0002841 0.0002841 0.04% FabArray::ParallelCopy() 861 0.0002756 0.0002756 0.0002756 0.04% MultiFab::max() 11 0.0002569 0.0002569 0.0002569 0.03% MultiFab::Copy() 11 0.0002567 0.0002567 0.0002567 0.03% FillPatchIterator::Initialize 41 0.000251 0.000251 0.000251 0.03% MLCellLinOp::correctionResidual() 492 0.0002321 0.0002321 0.0002321 0.03% Amr::coarseTimeStep() 10 0.0002152 0.0002152 0.0002152 0.03% Castro::construct_new_gravity() 10 0.0002036 0.0002036 0.0002036 0.03% MLMG::MLRhsNormInf() 11 0.000198 0.000198 0.000198 0.03% Amr::timeStep() 10 0.0001956 0.0001956 0.0001956 0.03% Castro::subcycle_advance_ctu() 10 0.0001938 0.0001938 0.0001938 0.03% MLCellLinOp::defineBC() 11 0.0001811 0.0001811 0.0001811 0.02% MLMG:computeResOfCorrection() 410 0.000137 0.000137 0.000137 0.02% StateData::checkPoint() 12 0.0001335 0.0001335 0.0001335 0.02% AmrLevel::checkPoint() 3 0.0001209 0.0001209 0.0001209 0.02% MLMG::actualBottomSolve() 82 0.0001149 0.0001149 0.0001149 0.01% MLMG::mgVcycle_down::0 82 8.764e-05 8.764e-05 8.764e-05 0.01% FabArrayBase::FB::FB() 56 8.599e-05 8.599e-05 8.599e-05 0.01% MLMG::solve() 11 8.03e-05 8.03e-05 8.03e-05 0.01% Castro::Castro() 1 7.996e-05 7.996e-05 7.996e-05 0.01% Castro::advance() 10 7.961e-05 7.961e-05 7.961e-05 0.01% Castro::initialize_advance() 10 7.834e-05 7.834e-05 7.834e-05 0.01% MLMG::mgVcycle_down::1 82 7.72e-05 7.72e-05 7.72e-05 0.01% Castro::clean_state() 62 7.431e-05 7.431e-05 7.431e-05 0.01% MLMG::mgVcycle_down::2 82 7.303e-05 7.303e-05 7.303e-05 0.01% Castro::finalize_advance() 10 7.285e-05 7.285e-05 7.285e-05 0.01% MLMG::mgVcycle_down::4 82 6.99e-05 6.99e-05 6.99e-05 0.01% MLMG::mgVcycle_down::3 82 6.962e-05 6.962e-05 6.962e-05 0.01% MLMG::mgVcycle_up::4 82 6.039e-05 6.039e-05 6.039e-05 0.01% Castro::initialize_do_advance() 10 5.542e-05 5.542e-05 5.542e-05 0.01% Castro::post_timestep() 10 5.528e-05 5.528e-05 5.528e-05 0.01% MLCellLinOp::solutionResidual() 93 5.392e-05 5.392e-05 5.392e-05 0.01% MLMG::mgVcycle_up::0 82 5.046e-05 5.046e-05 5.046e-05 0.01% MLMG::mgVcycle_up::1 82 4.881e-05 4.881e-05 4.881e-05 0.01% MLMG::mgVcycle_up::3 82 4.858e-05 4.858e-05 4.858e-05 0.01% MLMG::mgVcycle_up::2 82 4.818e-05 4.818e-05 4.818e-05 0.01% Castro::construct_new_source() 50 4.211e-05 4.211e-05 4.211e-05 0.01% StateData::define() 4 4.091e-05 4.091e-05 4.091e-05 0.01% Castro::swap_state_time_levels() 10 3.843e-05 3.843e-05 3.843e-05 0.00% MLMG::mgVcycle_bottom 82 3.64e-05 3.64e-05 3.64e-05 0.00% Castro::finalize_do_advance() 10 3.58e-05 3.58e-05 3.58e-05 0.00% MLMG::computeResidual() 82 3.478e-05 3.478e-05 3.478e-05 0.00% Castro::enforce_consistent_e() 1 3.421e-05 3.421e-05 3.421e-05 0.00% Gravity::actual_multilevel_solve() 1 3.169e-05 3.169e-05 3.169e-05 0.00% FillPatchSingleLevel 41 3.05e-05 3.05e-05 3.05e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.594e-05 2.594e-05 2.594e-05 0.00% Castro::initMFs() 1 2.531e-05 2.531e-05 2.531e-05 0.00% makeSFC 55 2.497e-05 2.497e-05 2.497e-05 0.00% Amr::writeSmallPlotFile() 1 2.42e-05 2.42e-05 2.42e-05 0.00% MLLinOp::define() 11 2.381e-05 2.381e-05 2.381e-05 0.00% MLPoisson::define() 11 2.213e-05 2.213e-05 2.213e-05 0.00% Amr::FinalizeInit() 1 2.134e-05 2.134e-05 2.134e-05 0.00% Castro::construct_old_source() 50 1.963e-05 1.963e-05 1.963e-05 0.00% Castro::buildMetrics() 1 1.96e-05 1.96e-05 1.96e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.652e-05 1.652e-05 1.652e-05 0.00% Castro::do_new_sources() 10 1.651e-05 1.651e-05 1.651e-05 0.00% Amr::defBaseLevel() 1 1.545e-05 1.545e-05 1.545e-05 0.00% DistributionMapping::Distribute() 56 1.544e-05 1.544e-05 1.544e-05 0.00% Castro::do_old_sources() 10 1.526e-05 1.526e-05 1.526e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.366e-05 1.366e-05 1.366e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.346e-05 1.346e-05 1.346e-05 0.00% Castro::apply_source_to_state() 20 1.205e-05 1.205e-05 1.205e-05 0.00% Castro::construct_old_gravity() 10 1.039e-05 1.039e-05 1.039e-05 0.00% Castro::check_for_nan() 20 1.039e-05 1.039e-05 1.039e-05 0.00% Gravity::swapTimeLevels() 10 9.51e-06 9.51e-06 9.51e-06 0.00% Amr::initSubcycle() 1 9.158e-06 9.158e-06 9.158e-06 0.00% MLPoisson::prepareForSolve() 11 7.963e-06 7.963e-06 7.963e-06 0.00% MLMG::computeMLResidual() 11 6.62e-06 6.62e-06 6.62e-06 0.00% Amr::InitializeInit() 1 6.307e-06 6.307e-06 6.307e-06 0.00% MLMG::buildFineMask() 11 6.156e-06 6.156e-06 6.156e-06 0.00% Castro::computeNewDt() 9 6.078e-06 6.078e-06 6.078e-06 0.00% AmrLevel::checkPointPost() 3 5.971e-06 5.971e-06 5.971e-06 0.00% MLMG::getGradSolution() 11 5.943e-06 5.943e-06 5.943e-06 0.00% MLMG::MLResNormInf() 11 4.941e-06 4.941e-06 4.941e-06 0.00% Castro::create_source_corrector() 10 4.72e-06 4.72e-06 4.72e-06 0.00% Gravity::set_mass_offset() 11 4.6e-06 4.6e-06 4.6e-06 0.00% AmrLevel::checkPointPre() 3 4.26e-06 4.26e-06 4.26e-06 0.00% Castro::retry_advance_ctu() 10 4.149e-06 4.149e-06 4.149e-06 0.00% Castro::post_init() 1 3.871e-06 3.871e-06 3.871e-06 0.00% Castro::FluxRegCrseInit 10 3.025e-06 3.025e-06 3.025e-06 0.00% Amr::init() 1 2.918e-06 2.918e-06 2.918e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.803e-06 2.803e-06 2.803e-06 0.00% Castro::FluxRegFineAdd() 10 2.484e-06 2.484e-06 2.484e-06 0.00% Castro::computeInitialDt() 2 2.318e-06 2.318e-06 2.318e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.987e-06 1.987e-06 1.987e-06 0.00% Castro::post_regrid() 1 1.191e-06 1.191e-06 1.191e-06 0.00% Amr::initialInit() 1 1.064e-06 1.064e-06 1.064e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.7697 0.7697 0.7697 100.00% Amr::coarseTimeStep() 10 0.6352 0.6352 0.6352 82.53% Amr::timeStep() 10 0.5463 0.5463 0.5463 70.97% Castro::advance() 10 0.5365 0.5365 0.5365 69.71% Castro::subcycle_advance_ctu() 10 0.5242 0.5242 0.5242 68.11% Castro::do_advance_ctu() 10 0.524 0.524 0.524 68.09% Gravity::solve_phi_with_mlmg() 11 0.2895 0.2895 0.2895 37.61% Gravity::actual_solve_with_mlmg() 11 0.2807 0.2807 0.2807 36.47% Castro::construct_new_gravity() 10 0.2683 0.2683 0.2683 34.86% MLMG::solve() 11 0.2605 0.2605 0.2605 33.84% Gravity::solve_for_phi() 10 0.2478 0.2478 0.2478 32.19% MLMG::oneIter() 82 0.2467 0.2467 0.2467 32.05% MLMG::mgVcycle() 82 0.245 0.245 0.245 31.84% Castro::construct_ctu_hydro_source() 10 0.1726 0.1726 0.1726 22.43% VisMF::Write(FabArray) 11 0.1509 0.1509 0.1509 19.60% MLCellLinOp::smooth() 1640 0.1269 0.1269 0.1269 16.49% Amr::checkPoint() 3 0.1242 0.1242 0.1242 16.14% Amr::init() 1 0.1112 0.1112 0.1112 14.45% AmrLevel::checkPoint() 3 0.1083 0.1083 0.1083 14.07% StateData::checkPoint() 12 0.1082 0.1082 0.1082 14.06% MLCellLinOp::applyBC() 4433 0.08832 0.08832 0.08832 11.47% MLMG::mgVcycle_bottom 82 0.07359 0.07359 0.07359 9.56% MLMG::actualBottomSolve() 82 0.07355 0.07355 0.07355 9.56% MLCGSolver::bicgstab 82 0.07282 0.07282 0.07282 9.46% MLPoisson::Fsmooth() 3280 0.06041 0.06041 0.06041 7.85% Castro::clean_state() 62 0.05458 0.05458 0.05458 7.09% Amr::initialInit() 1 0.04916 0.04916 0.04916 6.39% FillPatchIterator::Initialize 41 0.04616 0.04616 0.04616 6.00% Amr::writePlotFile() 2 0.04519 0.04519 0.04519 5.87% Amr::FinalizeInit() 1 0.04516 0.04516 0.04516 5.87% FillPatchSingleLevel 41 0.04462 0.04462 0.04462 5.80% Castro::post_init() 1 0.04381 0.04381 0.04381 5.69% Gravity::multilevel_solve_for_new_phi() 1 0.0421 0.0421 0.0421 5.47% Gravity::actual_multilevel_solve() 1 0.04209 0.04209 0.04209 5.47% StateDataPhysBCFunct::() 41 0.04066 0.04066 0.04066 5.28% MLMG::mgVcycle_down::0 82 0.03485 0.03485 0.03485 4.53% MLCellLinOp::apply() 1142 0.03351 0.03351 0.03351 4.35% MLMG::mgVcycle_up::0 82 0.02962 0.02962 0.02962 3.85% StateData::FillBoundary(geom) 328 0.02337 0.02337 0.02337 3.04% Castro::computeTemp() 63 0.02212 0.02212 0.02212 2.87% Gravity::get_new_grav_vector() 11 0.02197 0.02197 0.02197 2.85% Castro::initialize_do_advance() 10 0.02084 0.02084 0.02084 2.71% MultiFab::Dot() 1114 0.02068 0.02068 0.02068 2.69% MLCellLinOp::correctionResidual() 492 0.01956 0.01956 0.01956 2.54% Castro::normalize_species() 62 0.01846 0.01846 0.01846 2.40% MLMG:computeResOfCorrection() 410 0.01699 0.01699 0.01699 2.21% MLMG::mgVcycle_down::1 82 0.01623 0.01623 0.01623 2.11% MLPoisson::define() 11 0.01618 0.01618 0.01618 2.10% FabArray::FillBoundary() 4023 0.0158 0.0158 0.0158 2.05% MLMG::mgVcycle_down::2 82 0.01554 0.01554 0.01554 2.02% FillBoundary_nowait() 4023 0.015 0.015 0.015 1.95% MLMG::mgVcycle_down::3 82 0.01469 0.01469 0.01469 1.91% Castro::construct_old_gravity() 10 0.01426 0.01426 0.01426 1.85% Gravity::get_old_grav_vector() 10 0.01425 0.01425 0.01425 1.85% FabArray::ParallelCopy() 861 0.01408 0.01408 0.01408 1.83% MLMG::mgVcycle_down::4 82 0.01387 0.01387 0.01387 1.80% FabArray::ParallelCopy_nowait() 861 0.0138 0.0138 0.0138 1.79% Castro::enforce_min_density() 62 0.01304 0.01304 0.01304 1.69% FabArray::setVal() 1144 0.01302 0.01302 0.01302 1.69% CGSolver::sxay() 1586 0.01292 0.01292 0.01292 1.68% Castro::do_new_sources() 10 0.01282 0.01282 0.01282 1.67% MultiFab::LinComb() 1586 0.01246 0.01246 0.01246 1.62% MLCGSolver::ParallelAllReduce 1514 0.0124 0.0124 0.0124 1.61% Castro::initialize_advance() 10 0.01214 0.01214 0.01214 1.58% Castro::do_old_sources() 10 0.01214 0.01214 0.01214 1.58% MLMG::mgVcycle_up::2 82 0.01205 0.01205 0.01205 1.57% MLMG::mgVcycle_up::1 82 0.01184 0.01184 0.01184 1.54% MLMG::addInterpCorrection() 410 0.01164 0.01164 0.01164 1.51% Castro::expand_state() 10 0.01128 0.01128 0.01128 1.47% MLMG::mgVcycle_up::3 82 0.01126 0.01126 0.01126 1.46% amrex::average_down 410 0.01122 0.01122 0.01122 1.46% MLMG::mgVcycle_up::4 82 0.0111 0.0111 0.0111 1.44% MLCellLinOp::defineAuxData() 11 0.011 0.011 0.011 1.43% MLPoisson::Fapply() 1142 0.01058 0.01058 0.01058 1.37% Castro::post_timestep() 10 0.009523 0.009523 0.009523 1.24% Gravity::fill_multipole_BCs() 11 0.008487 0.008487 0.008487 1.10% Castro::estTimeStep() 21 0.007363 0.007363 0.007363 0.96% MLCellLinOp::solutionResidual() 93 0.007041 0.007041 0.007041 0.91% Castro::reset_internal_energy(MultiFab) 63 0.006401 0.006401 0.006401 0.83% MultiFab::Xpay() 585 0.00611 0.00611 0.00611 0.79% MLMG::computeResidual() 82 0.006079 0.006079 0.006079 0.79% MLCellLinOp::defineBC() 11 0.004676 0.004676 0.004676 0.61% BndryData::define() 11 0.004495 0.004495 0.004495 0.58% MLMG::prepareForSolve() 11 0.004424 0.004424 0.004424 0.57% Amr::InitializeInit() 1 0.003999 0.003999 0.003999 0.52% Amr::defBaseLevel() 1 0.003993 0.003993 0.003993 0.52% Castro::initData() 1 0.003497 0.003497 0.003497 0.45% Castro::computeNewDt() 9 0.003426 0.003426 0.003426 0.45% Castro::construct_new_source() 50 0.00304 0.00304 0.00304 0.39% Castro::construct_new_gravity_source() 10 0.002998 0.002998 0.002998 0.39% Castro::construct_old_source() 50 0.002509 0.002509 0.002509 0.33% Castro::construct_old_gravity_source() 10 0.00249 0.00249 0.00249 0.32% MLMG::ResNormInf() 93 0.001973 0.001973 0.001973 0.26% Castro::apply_source_to_state() 20 0.001822 0.001822 0.001822 0.24% MultiFab::Saxpy() 20 0.00181 0.00181 0.00181 0.24% Castro::enforce_speed_limit() 62 0.001659 0.001659 0.001659 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001629 0.001629 0.001629 0.21% FabArrayBase::getCPC() 1323 0.001549 0.001549 0.001549 0.20% Castro::reset_internal_energy(Fab) 504 0.00152 0.00152 0.00152 0.20% MLMG::getGradSolution() 11 0.001348 0.001348 0.001348 0.18% MLCellLinOp::compGrad() 11 0.001342 0.001342 0.001342 0.17% FabArray::mult() 43 0.001329 0.001329 0.001329 0.17% MLCellLinOp::setLevelBC() 11 0.001297 0.001297 0.001297 0.17% FabArray::setDomainBndry() 41 0.001286 0.001286 0.001286 0.17% Castro::check_for_nan() 20 0.001191 0.001191 0.001191 0.15% MultiFab::contains_nan() 20 0.00118 0.00118 0.00118 0.15% Castro::post_regrid() 1 0.001115 0.001115 0.001115 0.14% MLPoisson::prepareForSolve() 11 0.001064 0.001064 0.001064 0.14% MLCellLinOp::prepareForSolve() 11 0.001056 0.001056 0.001056 0.14% MLMG::computeMLResidual() 11 0.001003 0.001003 0.001003 0.13% Gravity::update_max_rhs() 11 0.0008127 0.0008127 0.0008127 0.11% FabArrayBase::getFB() 4023 0.0007684 0.0007684 0.0007684 0.10% FabArrayBase::CPC::define() 454 0.0006912 0.0006912 0.0006912 0.09% Castro::computeInitialDt() 2 0.000648 0.000648 0.000648 0.08% MLLinOp::define() 11 0.0004871 0.0004871 0.0004871 0.06% MLLinOp::defineGrids() 11 0.0004633 0.0004633 0.0004633 0.06% Amr::InitAmr() 1 0.0004449 0.0004449 0.0004449 0.06% Gravity::swapTimeLevels() 10 0.0004369 0.0004369 0.0004369 0.06% Castro::Castro() 1 0.000422 0.000422 0.000422 0.05% MultiFab::max() 11 0.0002569 0.0002569 0.0002569 0.03% MultiFab::Copy() 11 0.0002567 0.0002567 0.0002567 0.03% MLMG::MLResNormInf() 11 0.000254 0.000254 0.000254 0.03% MLMG::MLRhsNormInf() 11 0.000198 0.000198 0.000198 0.03% Castro::buildMetrics() 1 0.0001556 0.0001556 0.0001556 0.02% FabArrayBase::FB::FB() 56 8.599e-05 8.599e-05 8.599e-05 0.01% Castro::finalize_advance() 10 7.836e-05 7.836e-05 7.836e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.437e-05 5.437e-05 5.437e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.289e-05 5.289e-05 5.289e-05 0.01% StateData::define() 4 4.091e-05 4.091e-05 4.091e-05 0.01% makeSFC 55 3.923e-05 3.923e-05 3.923e-05 0.01% Castro::swap_state_time_levels() 10 3.843e-05 3.843e-05 3.843e-05 0.00% Castro::finalize_do_advance() 10 3.58e-05 3.58e-05 3.58e-05 0.00% Castro::enforce_consistent_e() 1 3.421e-05 3.421e-05 3.421e-05 0.00% Castro::initMFs() 1 2.531e-05 2.531e-05 2.531e-05 0.00% Amr::writeSmallPlotFile() 1 2.42e-05 2.42e-05 2.42e-05 0.00% DistributionMapping::Distribute() 56 1.544e-05 1.544e-05 1.544e-05 0.00% Amr::initSubcycle() 1 9.158e-06 9.158e-06 9.158e-06 0.00% MLMG::buildFineMask() 11 6.156e-06 6.156e-06 6.156e-06 0.00% AmrLevel::checkPointPost() 3 5.971e-06 5.971e-06 5.971e-06 0.00% Castro::create_source_corrector() 10 4.72e-06 4.72e-06 4.72e-06 0.00% Gravity::set_mass_offset() 11 4.6e-06 4.6e-06 4.6e-06 0.00% AmrLevel::checkPointPre() 3 4.26e-06 4.26e-06 4.26e-06 0.00% Castro::retry_advance_ctu() 10 4.149e-06 4.149e-06 4.149e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.974e-06 3.974e-06 3.974e-06 0.00% Castro::FluxRegCrseInit 10 3.025e-06 3.025e-06 3.025e-06 0.00% Castro::FluxRegFineAdd() 10 2.484e-06 2.484e-06 2.484e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.987e-06 1.987e-06 1.987e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-5-g3ca867b8cce8) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.05-5-g3ca867b8cce8) initialized Starting run at 08:29:38 UTC on 2022-05-10. Successfully read inputs file ... Castro git describe: 22.05-12-gb79cf4624 AMReX git describe: 22.05-5-g3ca867b8c Microphysics git describe: 22.05 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.425130147 Restart time = 0.046766424 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.048894939 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.046460495 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.054159056 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.058848706 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.06024988 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024179466 seconds Ending run at 08:29:38 UTC on 2022-05-10. Run time = 0.340461667 Run time without initialization = 0.293143159 Average number of zones advanced per microsecond: 4.471 Average number of zones advanced per microsecond per rank: 4.471 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3405 ... 0.3405 ... 0.3405 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0805 0.0805 0.0805 23.64% VisMF::Read() 3 0.03975 0.03975 0.03975 11.67% MLCellLinOp::applyBC() 1946 0.03165 0.03165 0.03165 9.30% MLPoisson::Fsmooth() 1440 0.02566 0.02566 0.02566 7.54% VisMF::Write(FabArray) 1 0.02285 0.02285 0.02285 6.71% StateData::FillBoundary(geom) 160 0.01125 0.01125 0.01125 3.30% MLCGSolver::bicgstab 36 0.009101 0.009101 0.009101 2.67% MultiFab::Dot() 484 0.008793 0.008793 0.008793 2.58% Castro::normalize_species() 30 0.008766 0.008766 0.008766 2.57% Castro::computeTemp() 30 0.007623 0.007623 0.007623 2.24% FillBoundary_nowait() 1766 0.006256 0.006256 0.006256 1.84% FabArray::setVal() 537 0.006247 0.006247 0.006247 1.83% FabArray::ParallelCopy_nowait() 380 0.00589 0.00589 0.00589 1.73% StateDataPhysBCFunct::() 20 0.005842 0.005842 0.005842 1.72% Castro::enforce_min_density() 30 0.005777 0.005777 0.005777 1.70% MultiFab::LinComb() 690 0.0053 0.0053 0.0053 1.56% MLCellLinOp::defineAuxData() 6 0.005296 0.005296 0.005296 1.56% Gravity::fill_multipole_BCs() 6 0.004807 0.004807 0.004807 1.41% MLPoisson::Fapply() 500 0.004572 0.004572 0.004572 1.34% Castro::estTimeStep() 10 0.003763 0.003763 0.003763 1.11% Amr::restart() 1 0.002963 0.002963 0.002963 0.87% MLMG::addInterpCorrection() 180 0.002812 0.002812 0.002812 0.83% amrex::average_down 180 0.002659 0.002659 0.002659 0.78% MultiFab::Xpay() 258 0.00264 0.00264 0.00264 0.78% Castro::do_advance_ctu() 5 0.00241 0.00241 0.00241 0.71% Castro::reset_internal_energy(MultiFab) 30 0.002063 0.002063 0.002063 0.61% BndryData::define() 6 0.001982 0.001982 0.001982 0.58% Castro::construct_new_gravity_source() 5 0.001558 0.001558 0.001558 0.46% Amr::writePlotFile() 1 0.001409 0.001409 0.001409 0.41% Castro::construct_old_gravity_source() 5 0.001342 0.001342 0.001342 0.39% MultiFab::Saxpy() 10 0.0009196 0.0009196 0.0009196 0.27% Gravity::get_old_grav_vector() 5 0.0008867 0.0008867 0.0008867 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008841 0.0008841 0.0008841 0.26% MLMG::ResNormInf() 42 0.0008736 0.0008736 0.0008736 0.26% Castro::expand_state() 5 0.0008728 0.0008728 0.0008728 0.26% Gravity::get_new_grav_vector() 5 0.0008542 0.0008542 0.0008542 0.25% MLMG::oneIter() 36 0.0007455 0.0007455 0.0007455 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007407 0.0007407 0.0007407 0.22% Castro::reset_internal_energy(Fab) 240 0.0007318 0.0007318 0.0007318 0.21% MLCellLinOp::setLevelBC() 6 0.0007096 0.0007096 0.0007096 0.21% Castro::enforce_speed_limit() 30 0.0006647 0.0006647 0.0006647 0.20% FabArray::mult() 22 0.0006492 0.0006492 0.0006492 0.19% FabArray::setDomainBndry() 20 0.000637 0.000637 0.000637 0.19% MultiFab::contains_nan() 10 0.0005883 0.0005883 0.0005883 0.17% MLCellLinOp::prepareForSolve() 6 0.0005789 0.0005789 0.0005789 0.17% MLCellLinOp::compGrad() 6 0.000485 0.000485 0.000485 0.14% MLCellLinOp::smooth() 720 0.0004664 0.0004664 0.0004664 0.14% MLMG::prepareForSolve() 6 0.0004208 0.0004208 0.0004208 0.12% FabArrayBase::CPC::define() 244 0.0004082 0.0004082 0.0004082 0.12% FabArrayBase::getCPC() 632 0.0004074 0.0004074 0.0004074 0.12% Amr::InitAmr() 1 0.0003794 0.0003794 0.0003794 0.11% FabArray::FillBoundary() 1766 0.0003756 0.0003756 0.0003756 0.11% FabArrayBase::getFB() 1766 0.0002955 0.0002955 0.0002955 0.09% main() 1 0.0002799 0.0002799 0.0002799 0.08% MLCellLinOp::apply() 500 0.0002245 0.0002245 0.0002245 0.07% Gravity::update_max_rhs() 6 0.0002211 0.0002211 0.0002211 0.06% Gravity::solve_for_phi() 5 0.0002056 0.0002056 0.0002056 0.06% MLLinOp::defineGrids() 6 0.0001992 0.0001992 0.0001992 0.06% CGSolver::sxay() 690 0.0001951 0.0001951 0.0001951 0.06% MLMG::mgVcycle() 36 0.0001678 0.0001678 0.0001678 0.05% MLCGSolver::ParallelAllReduce 659 0.000156 0.000156 0.000156 0.05% Castro::subcycle_advance_ctu() 5 0.0001559 0.0001559 0.0001559 0.05% MultiFab::Copy() 6 0.0001426 0.0001426 0.0001426 0.04% FabArray::ParallelCopy() 380 0.000136 0.000136 0.000136 0.04% MultiFab::max() 6 0.0001335 0.0001335 0.0001335 0.04% FillPatchIterator::Initialize 20 0.0001196 0.0001196 0.0001196 0.04% Amr::coarseTimeStep() 5 0.0001106 0.0001106 0.0001106 0.03% MLMG::MLRhsNormInf() 6 0.0001051 0.0001051 0.0001051 0.03% Castro::construct_new_gravity() 5 0.0001051 0.0001051 0.0001051 0.03% Amr::timeStep() 5 0.0001019 0.0001019 0.0001019 0.03% MLCellLinOp::correctionResidual() 216 9.957e-05 9.957e-05 9.957e-05 0.03% MLCellLinOp::defineBC() 6 9.899e-05 9.899e-05 9.899e-05 0.03% AmrLevel::restart() 1 7.956e-05 7.956e-05 7.956e-05 0.02% StateData::restartDoit() 4 7.225e-05 7.225e-05 7.225e-05 0.02% Castro::advance() 5 7.144e-05 7.144e-05 7.144e-05 0.02% FabArrayBase::FB::FB() 26 5.961e-05 5.961e-05 5.961e-05 0.02% MLMG:computeResOfCorrection() 180 5.539e-05 5.539e-05 5.539e-05 0.02% Castro::create_source_corrector() 5 4.84e-05 4.84e-05 4.84e-05 0.01% MLMG::actualBottomSolve() 36 4.53e-05 4.53e-05 4.53e-05 0.01% Castro::initialize_do_advance() 5 4.084e-05 4.084e-05 4.084e-05 0.01% Castro::finalize_advance() 5 4.007e-05 4.007e-05 4.007e-05 0.01% MLMG::mgVcycle_down::0 36 3.903e-05 3.903e-05 3.903e-05 0.01% Castro::clean_state() 30 3.856e-05 3.856e-05 3.856e-05 0.01% Castro::initialize_advance() 5 3.665e-05 3.665e-05 3.665e-05 0.01% MLMG::solve() 6 3.663e-05 3.663e-05 3.663e-05 0.01% MLMG::mgVcycle_down::1 36 3.62e-05 3.62e-05 3.62e-05 0.01% Castro::construct_old_source() 25 3.586e-05 3.586e-05 3.586e-05 0.01% Castro::construct_new_source() 25 3.292e-05 3.292e-05 3.292e-05 0.01% MLMG::mgVcycle_down::2 36 3.27e-05 3.27e-05 3.27e-05 0.01% MLMG::mgVcycle_down::4 36 3.214e-05 3.214e-05 3.214e-05 0.01% Castro::buildMetrics() 1 3.128e-05 3.128e-05 3.128e-05 0.01% MLMG::mgVcycle_down::3 36 3.064e-05 3.064e-05 3.064e-05 0.01% Gravity::actual_multilevel_solve() 1 2.859e-05 2.859e-05 2.859e-05 0.01% MLMG::mgVcycle_up::4 36 2.766e-05 2.766e-05 2.766e-05 0.01% Castro::initMFs() 1 2.608e-05 2.608e-05 2.608e-05 0.01% Castro::swap_state_time_levels() 5 2.588e-05 2.588e-05 2.588e-05 0.01% Amr::writeSmallPlotFile() 1 2.493e-05 2.493e-05 2.493e-05 0.01% MLCellLinOp::solutionResidual() 42 2.456e-05 2.456e-05 2.456e-05 0.01% MLMG::mgVcycle_up::0 36 2.19e-05 2.19e-05 2.19e-05 0.01% MLMG::mgVcycle_up::3 36 2.181e-05 2.181e-05 2.181e-05 0.01% Castro::post_restart() 1 2.181e-05 2.181e-05 2.181e-05 0.01% MLLinOp::define() 6 2.089e-05 2.089e-05 2.089e-05 0.01% MLMG::mgVcycle_up::2 36 2.048e-05 2.048e-05 2.048e-05 0.01% MLMG::mgVcycle_up::1 36 1.98e-05 1.98e-05 1.98e-05 0.01% Castro::finalize_do_advance() 5 1.856e-05 1.856e-05 1.856e-05 0.01% MLMG::mgVcycle_bottom 36 1.709e-05 1.709e-05 1.709e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.615e-05 1.615e-05 1.615e-05 0.00% FillPatchSingleLevel 20 1.535e-05 1.535e-05 1.535e-05 0.00% MLMG::computeResidual() 36 1.511e-05 1.511e-05 1.511e-05 0.00% MLPoisson::define() 6 1.369e-05 1.369e-05 1.369e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.345e-05 1.345e-05 1.345e-05 0.00% makeSFC 30 1.336e-05 1.336e-05 1.336e-05 0.00% DistributionMapping::Distribute() 31 1.004e-05 1.004e-05 1.004e-05 0.00% Amr::initSubcycle() 1 8.89e-06 8.89e-06 8.89e-06 0.00% Castro::do_new_sources() 5 8.766e-06 8.766e-06 8.766e-06 0.00% Castro::do_old_sources() 5 8.198e-06 8.198e-06 8.198e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.708e-06 7.708e-06 7.708e-06 0.00% Castro::apply_source_to_state() 10 6.313e-06 6.313e-06 6.313e-06 0.00% Castro::check_for_nan() 10 5.74e-06 5.74e-06 5.74e-06 0.00% Castro::construct_old_gravity() 5 5.435e-06 5.435e-06 5.435e-06 0.00% Castro::post_timestep() 5 4.773e-06 4.773e-06 4.773e-06 0.00% MLPoisson::prepareForSolve() 6 4.696e-06 4.696e-06 4.696e-06 0.00% Gravity::swapTimeLevels() 5 4.104e-06 4.104e-06 4.104e-06 0.00% MLMG::computeMLResidual() 6 3.223e-06 3.223e-06 3.223e-06 0.00% MLMG::buildFineMask() 6 3.142e-06 3.142e-06 3.142e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.141e-06 3.141e-06 3.141e-06 0.00% Castro::computeNewDt() 5 2.901e-06 2.901e-06 2.901e-06 0.00% MLMG::getGradSolution() 6 2.883e-06 2.883e-06 2.883e-06 0.00% MLMG::MLResNormInf() 6 2.728e-06 2.728e-06 2.728e-06 0.00% Gravity::set_mass_offset() 6 2.288e-06 2.288e-06 2.288e-06 0.00% Castro::retry_advance_ctu() 5 2.041e-06 2.041e-06 2.041e-06 0.00% Castro::FluxRegCrseInit 5 1.939e-06 1.939e-06 1.939e-06 0.00% Castro::FluxRegFineAdd() 5 1.449e-06 1.449e-06 1.449e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.197e-06 1.197e-06 1.197e-06 0.00% AmrLevel::AmrLevel() 1 1.126e-06 1.126e-06 1.126e-06 0.00% Amr::init() 1 1.006e-06 1.006e-06 1.006e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3405 0.3405 0.3405 100.00% Amr::coarseTimeStep() 5 0.2687 0.2687 0.2687 78.92% Amr::timeStep() 5 0.2667 0.2667 0.2667 78.33% Castro::advance() 5 0.2625 0.2625 0.2625 77.09% Castro::subcycle_advance_ctu() 5 0.256 0.256 0.256 75.19% Castro::do_advance_ctu() 5 0.2558 0.2558 0.2558 75.14% Castro::construct_new_gravity() 5 0.1331 0.1331 0.1331 39.08% Gravity::solve_phi_with_mlmg() 6 0.129 0.129 0.129 37.87% Gravity::solve_for_phi() 5 0.1256 0.1256 0.1256 36.90% Gravity::actual_solve_with_mlmg() 6 0.124 0.124 0.124 36.43% MLMG::solve() 6 0.1129 0.1129 0.1129 33.16% MLMG::oneIter() 36 0.1063 0.1063 0.1063 31.21% MLMG::mgVcycle() 36 0.1055 0.1055 0.1055 30.99% Castro::construct_ctu_hydro_source() 5 0.0805 0.0805 0.0805 23.64% MLCellLinOp::smooth() 720 0.05466 0.05466 0.05466 16.05% Amr::init() 1 0.0468 0.0468 0.0468 13.75% Amr::restart() 1 0.0468 0.0468 0.0468 13.74% AmrLevel::restart() 1 0.03996 0.03996 0.03996 11.74% StateData::restartDoit() 4 0.03987 0.03987 0.03987 11.71% VisMF::Read() 3 0.03975 0.03975 0.03975 11.67% MLCellLinOp::applyBC() 1946 0.03864 0.03864 0.03864 11.35% MLMG::mgVcycle_bottom 36 0.03152 0.03152 0.03152 9.26% MLMG::actualBottomSolve() 36 0.0315 0.0315 0.0315 9.25% MLCGSolver::bicgstab 36 0.03118 0.03118 0.03118 9.16% Castro::clean_state() 30 0.02566 0.02566 0.02566 7.54% MLPoisson::Fsmooth() 1440 0.02566 0.02566 0.02566 7.54% Amr::writePlotFile() 1 0.02426 0.02426 0.02426 7.13% VisMF::Write(FabArray) 1 0.02285 0.02285 0.02285 6.71% FillPatchIterator::Initialize 20 0.01985 0.01985 0.01985 5.83% FillPatchSingleLevel 20 0.0191 0.0191 0.0191 5.61% StateDataPhysBCFunct::() 20 0.01709 0.01709 0.01709 5.02% MLMG::mgVcycle_down::0 36 0.01498 0.01498 0.01498 4.40% MLCellLinOp::apply() 500 0.01464 0.01464 0.01464 4.30% MLMG::mgVcycle_up::0 36 0.01267 0.01267 0.01267 3.72% StateData::FillBoundary(geom) 160 0.01125 0.01125 0.01125 3.30% Castro::initialize_do_advance() 5 0.01092 0.01092 0.01092 3.21% Castro::computeTemp() 30 0.01042 0.01042 0.01042 3.06% MLPoisson::define() 6 0.008921 0.008921 0.008921 2.62% MultiFab::Dot() 484 0.008793 0.008793 0.008793 2.58% Castro::normalize_species() 30 0.008766 0.008766 0.008766 2.57% MLCellLinOp::correctionResidual() 216 0.008508 0.008508 0.008508 2.50% MLMG:computeResOfCorrection() 180 0.007377 0.007377 0.007377 2.17% Gravity::get_new_grav_vector() 5 0.007339 0.007339 0.007339 2.16% Castro::construct_old_gravity() 5 0.007281 0.007281 0.007281 2.14% Gravity::get_old_grav_vector() 5 0.007275 0.007275 0.007275 2.14% MLMG::mgVcycle_down::1 36 0.007025 0.007025 0.007025 2.06% FabArray::FillBoundary() 1766 0.006987 0.006987 0.006987 2.05% MLMG::mgVcycle_down::2 36 0.006715 0.006715 0.006715 1.97% FillBoundary_nowait() 1766 0.006612 0.006612 0.006612 1.94% Castro::do_new_sources() 5 0.006578 0.006578 0.006578 1.93% FabArray::ParallelCopy() 380 0.006445 0.006445 0.006445 1.89% Castro::initialize_advance() 5 0.006365 0.006365 0.006365 1.87% MLMG::mgVcycle_down::3 36 0.006346 0.006346 0.006346 1.86% FabArray::ParallelCopy_nowait() 380 0.006309 0.006309 0.006309 1.85% FabArray::setVal() 537 0.006247 0.006247 0.006247 1.83% MLMG::mgVcycle_down::4 36 0.006066 0.006066 0.006066 1.78% MLCellLinOp::defineAuxData() 6 0.00604 0.00604 0.00604 1.77% Castro::expand_state() 5 0.005992 0.005992 0.005992 1.76% Castro::enforce_min_density() 30 0.005777 0.005777 0.005777 1.70% CGSolver::sxay() 690 0.005495 0.005495 0.005495 1.61% Castro::do_old_sources() 5 0.005386 0.005386 0.005386 1.58% MLCGSolver::ParallelAllReduce 659 0.005327 0.005327 0.005327 1.56% MultiFab::LinComb() 690 0.0053 0.0053 0.0053 1.56% MLMG::mgVcycle_up::2 36 0.0052 0.0052 0.0052 1.53% MLMG::mgVcycle_up::1 36 0.00512 0.00512 0.00512 1.50% MLMG::addInterpCorrection() 180 0.005031 0.005031 0.005031 1.48% amrex::average_down 180 0.004899 0.004899 0.004899 1.44% MLMG::mgVcycle_up::3 36 0.004867 0.004867 0.004867 1.43% MLMG::mgVcycle_up::4 36 0.004848 0.004848 0.004848 1.42% Gravity::fill_multipole_BCs() 6 0.004807 0.004807 0.004807 1.41% MLPoisson::Fapply() 500 0.004572 0.004572 0.004572 1.34% Castro::post_timestep() 5 0.004129 0.004129 0.004129 1.21% Castro::estTimeStep() 10 0.003763 0.003763 0.003763 1.11% Castro::post_restart() 1 0.003687 0.003687 0.003687 1.08% Gravity::multilevel_solve_for_new_phi() 1 0.003576 0.003576 0.003576 1.05% Gravity::actual_multilevel_solve() 1 0.00356 0.00356 0.00356 1.05% MLCellLinOp::solutionResidual() 42 0.003186 0.003186 0.003186 0.94% Castro::reset_internal_energy(MultiFab) 30 0.002795 0.002795 0.002795 0.82% MLMG::computeResidual() 36 0.002645 0.002645 0.002645 0.78% MultiFab::Xpay() 258 0.00264 0.00264 0.00264 0.78% MLCellLinOp::defineBC() 6 0.002617 0.002617 0.002617 0.77% BndryData::define() 6 0.002518 0.002518 0.002518 0.74% MLMG::prepareForSolve() 6 0.002418 0.002418 0.002418 0.71% Castro::computeNewDt() 5 0.001886 0.001886 0.001886 0.55% Castro::construct_new_source() 25 0.001591 0.001591 0.001591 0.47% Castro::construct_new_gravity_source() 5 0.001558 0.001558 0.001558 0.46% Castro::construct_old_source() 25 0.001378 0.001378 0.001378 0.40% Castro::construct_old_gravity_source() 5 0.001342 0.001342 0.001342 0.39% Castro::apply_source_to_state() 10 0.0009259 0.0009259 0.0009259 0.27% MultiFab::Saxpy() 10 0.0009196 0.0009196 0.0009196 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008841 0.0008841 0.0008841 0.26% MLMG::ResNormInf() 42 0.0008736 0.0008736 0.0008736 0.26% FabArrayBase::getCPC() 632 0.0008156 0.0008156 0.0008156 0.24% MLMG::getGradSolution() 6 0.0007478 0.0007478 0.0007478 0.22% MLCellLinOp::compGrad() 6 0.0007449 0.0007449 0.0007449 0.22% Castro::reset_internal_energy(Fab) 240 0.0007318 0.0007318 0.0007318 0.21% MLCellLinOp::setLevelBC() 6 0.0007096 0.0007096 0.0007096 0.21% Castro::enforce_speed_limit() 30 0.0006647 0.0006647 0.0006647 0.20% FabArray::mult() 22 0.0006492 0.0006492 0.0006492 0.19% FabArray::setDomainBndry() 20 0.000637 0.000637 0.000637 0.19% Castro::check_for_nan() 10 0.000594 0.000594 0.000594 0.17% MultiFab::contains_nan() 10 0.0005883 0.0005883 0.0005883 0.17% MLPoisson::prepareForSolve() 6 0.0005836 0.0005836 0.0005836 0.17% MLCellLinOp::prepareForSolve() 6 0.0005789 0.0005789 0.0005789 0.17% MLMG::computeMLResidual() 6 0.0005591 0.0005591 0.0005591 0.16% Gravity::update_max_rhs() 6 0.0004328 0.0004328 0.0004328 0.13% FabArrayBase::CPC::define() 244 0.0004082 0.0004082 0.0004082 0.12% Amr::InitAmr() 1 0.0003883 0.0003883 0.0003883 0.11% FabArrayBase::getFB() 1766 0.0003551 0.0003551 0.0003551 0.10% MLLinOp::define() 6 0.0002504 0.0002504 0.0002504 0.07% MLLinOp::defineGrids() 6 0.0002295 0.0002295 0.0002295 0.07% Gravity::swapTimeLevels() 5 0.0002295 0.0002295 0.0002295 0.07% Castro::buildMetrics() 1 0.000164 0.000164 0.000164 0.05% MultiFab::Copy() 6 0.0001426 0.0001426 0.0001426 0.04% MLMG::MLResNormInf() 6 0.0001337 0.0001337 0.0001337 0.04% MultiFab::max() 6 0.0001335 0.0001335 0.0001335 0.04% MLMG::MLRhsNormInf() 6 0.0001051 0.0001051 0.0001051 0.03% FabArrayBase::FB::FB() 26 5.961e-05 5.961e-05 5.961e-05 0.02% Castro::create_source_corrector() 5 4.84e-05 4.84e-05 4.84e-05 0.01% Castro::finalize_advance() 5 4.346e-05 4.346e-05 4.346e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.918e-05 2.918e-05 2.918e-05 0.01% Castro::initMFs() 1 2.608e-05 2.608e-05 2.608e-05 0.01% Castro::swap_state_time_levels() 5 2.588e-05 2.588e-05 2.588e-05 0.01% Amr::writeSmallPlotFile() 1 2.493e-05 2.493e-05 2.493e-05 0.01% makeSFC 30 2.147e-05 2.147e-05 2.147e-05 0.01% Castro::finalize_do_advance() 5 1.856e-05 1.856e-05 1.856e-05 0.01% DistributionMapping::Distribute() 31 1.004e-05 1.004e-05 1.004e-05 0.00% Amr::initSubcycle() 1 8.89e-06 8.89e-06 8.89e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.066e-06 5.066e-06 5.066e-06 0.00% MLMG::buildFineMask() 6 3.142e-06 3.142e-06 3.142e-06 0.00% Gravity::set_mass_offset() 6 2.288e-06 2.288e-06 2.288e-06 0.00% Castro::retry_advance_ctu() 5 2.041e-06 2.041e-06 2.041e-06 0.00% Castro::FluxRegCrseInit 5 1.939e-06 1.939e-06 1.939e-06 0.00% Castro::FluxRegFineAdd() 5 1.449e-06 1.449e-06 1.449e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.197e-06 1.197e-06 1.197e-06 0.00% AmrLevel::AmrLevel() 1 1.126e-06 1.126e-06 1.126e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.05-5-g3ca867b8cce8) finalized