Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-16-g826cd378f8ba) initialized Starting run at 08:32:30 UTC on 2022-09-20. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09-16-g826cd378f Microphysics git describe: 22.08-13-g1adf1bdb reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.053030703 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030278953 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.049714107 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052187346 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.061348057 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.063987996 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.064911139 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.049056989 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.072849149 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.075287543 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.055935176 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.064169733 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065727013 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.068125888 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031748777 seconds Ending run at 08:32:31 UTC on 2022-09-20. Run time = 0.912079696 Run time without initialization = 0.775651084 Average number of zones advanced per microsecond: 3.380 Average number of zones advanced per microsecond per rank: 3.380 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.9121 ... 0.9121 ... 0.9121 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2247 0.2247 0.2247 24.64% VisMF::Write(FabArray) 11 0.2066 0.2066 0.2066 22.65% MLCellLinOp::applyBC() 4433 0.08082 0.08082 0.08082 8.86% MLPoisson::Fsmooth() 3280 0.06477 0.06477 0.06477 7.10% MLCGSolver::bicgstab 82 0.02415 0.02415 0.02415 2.65% StateData::FillBoundary(geom) 328 0.02333 0.02333 0.02333 2.56% Amr::checkPoint() 3 0.02309 0.02309 0.02309 2.53% MultiFab::Dot() 1114 0.02282 0.02282 0.02282 2.50% StateDataPhysBCFunct::() 41 0.01843 0.01843 0.01843 2.02% Castro::normalize_species() 62 0.01671 0.01671 0.01671 1.83% Castro::computeTemp() 63 0.01586 0.01586 0.01586 1.74% MultiFab::LinComb() 1586 0.01455 0.01455 0.01455 1.60% FabArray::setVal() 1144 0.01437 0.01437 0.01437 1.58% FillBoundary_nowait() 4023 0.01434 0.01434 0.01434 1.57% FabArray::ParallelCopy_nowait() 861 0.01336 0.01336 0.01336 1.47% Castro::enforce_min_density() 62 0.01293 0.01293 0.01293 1.42% MLPoisson::Fapply() 1142 0.01187 0.01187 0.01187 1.30% MLCellLinOp::defineAuxData() 11 0.01172 0.01172 0.01172 1.28% Gravity::fill_multipole_BCs() 11 0.008682 0.008682 0.008682 0.95% MLMG::addInterpCorrection() 410 0.007863 0.007863 0.007863 0.86% amrex::average_down 410 0.006914 0.006914 0.006914 0.76% MultiFab::Xpay() 585 0.006658 0.006658 0.006658 0.73% Castro::estTimeStep() 21 0.006079 0.006079 0.006079 0.67% Castro::do_advance_ctu() 10 0.005663 0.005663 0.005663 0.62% Castro::reset_internal_energy(MultiFab) 63 0.004406 0.004406 0.004406 0.48% BndryData::define() 11 0.00397 0.00397 0.00397 0.44% Castro::construct_new_gravity_source() 10 0.003212 0.003212 0.003212 0.35% Amr::writePlotFile() 2 0.002556 0.002556 0.002556 0.28% Castro::construct_old_gravity_source() 10 0.002485 0.002485 0.002485 0.27% MLMG::ResNormInf() 93 0.002105 0.002105 0.002105 0.23% Gravity::get_new_grav_vector() 11 0.001942 0.001942 0.001942 0.21% MultiFab::Saxpy() 20 0.001817 0.001817 0.001817 0.20% Gravity::get_old_grav_vector() 10 0.001758 0.001758 0.001758 0.19% Castro::expand_state() 10 0.001743 0.001743 0.001743 0.19% MultiFab::Add() 82 0.001691 0.001691 0.001691 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001686 0.001686 0.001686 0.18% Castro::reset_internal_energy(Fab) 504 0.001587 0.001587 0.001587 0.17% MLCellLinOp::setLevelBC() 11 0.001554 0.001554 0.001554 0.17% Gravity::actual_solve_with_mlmg() 11 0.001434 0.001434 0.001434 0.16% FabArray::mult() 43 0.001327 0.001327 0.001327 0.15% FabArray::setDomainBndry() 41 0.00132 0.00132 0.00132 0.14% Castro::initData() 1 0.001242 0.001242 0.001242 0.14% MLMG::prepareForSolve() 11 0.001224 0.001224 0.001224 0.13% MLCellLinOp::prepareForSolve() 11 0.001182 0.001182 0.001182 0.13% MultiFab::contains_nan() 20 0.001177 0.001177 0.001177 0.13% Castro::enforce_speed_limit() 62 0.001101 0.001101 0.001101 0.12% MLCellLinOp::smooth() 1640 0.001024 0.001024 0.001024 0.11% MLCellLinOp::compGrad() 11 0.0009327 0.0009327 0.0009327 0.10% FabArray::FillBoundary() 4023 0.0008157 0.0008157 0.0008157 0.09% FabArrayBase::getCPC() 1323 0.0007662 0.0007662 0.0007662 0.08% FabArrayBase::CPC::define() 454 0.0006721 0.0006721 0.0006721 0.07% FabArrayBase::getFB() 4023 0.0005871 0.0005871 0.0005871 0.06% Amr::InitAmr() 1 0.0004868 0.0004868 0.0004868 0.05% MLCellLinOp::apply() 1142 0.0004547 0.0004547 0.0004547 0.05% Gravity::solve_for_phi() 10 0.0004358 0.0004358 0.0004358 0.05% Gravity::update_max_rhs() 11 0.000405 0.000405 0.000405 0.04% CGSolver::sxay() 1586 0.0003417 0.0003417 0.0003417 0.04% MultiFab::Copy() 11 0.0003334 0.0003334 0.0003334 0.04% Amr::coarseTimeStep() 10 0.0003163 0.0003163 0.0003163 0.03% MLCGSolver::ParallelAllReduce 1514 0.0003077 0.0003077 0.0003077 0.03% FillPatchIterator::Initialize 41 0.000287 0.000287 0.000287 0.03% MLCellLinOp::defineBC() 11 0.0002832 0.0002832 0.0002832 0.03% main() 1 0.0002661 0.0002661 0.0002661 0.03% Castro::subcycle_advance_ctu() 10 0.0002658 0.0002658 0.0002658 0.03% FabArray::ParallelCopy() 861 0.0002616 0.0002616 0.0002616 0.03% MultiFab::max() 11 0.000259 0.000259 0.000259 0.03% MLCellLinOp::correctionResidual() 492 0.0002242 0.0002242 0.0002242 0.02% MLMG::MLRhsNormInf() 11 0.0002175 0.0002175 0.0002175 0.02% Castro::construct_new_gravity() 10 0.000206 0.000206 0.000206 0.02% MLMG::mgVcycle() 82 0.000205 0.000205 0.000205 0.02% Amr::timeStep() 10 0.0001572 0.0001572 0.0001572 0.02% StateData::checkPoint() 12 0.0001496 0.0001496 0.0001496 0.02% MLMG:computeResOfCorrection() 410 0.0001429 0.0001429 0.0001429 0.02% MLLinOp::defineGrids() 11 0.0001316 0.0001316 0.0001316 0.01% AmrLevel::checkPoint() 3 0.0001201 0.0001201 0.0001201 0.01% MLMG::mgVcycle_down::0 82 0.0001049 0.0001049 0.0001049 0.01% MLMG::mgVcycle_down::1 82 8.967e-05 8.967e-05 8.967e-05 0.01% Castro::Castro() 1 8.517e-05 8.517e-05 8.517e-05 0.01% Castro::initialize_advance() 10 8.413e-05 8.413e-05 8.413e-05 0.01% MLMG::mgVcycle_down::2 82 8.4e-05 8.4e-05 8.4e-05 0.01% FabArrayBase::FB::FB() 56 8.299e-05 8.299e-05 8.299e-05 0.01% Castro::clean_state() 62 7.747e-05 7.747e-05 7.747e-05 0.01% MLMG::mgVcycle_down::3 82 7.649e-05 7.649e-05 7.649e-05 0.01% MLMG::mgVcycle_down::4 82 7.482e-05 7.482e-05 7.482e-05 0.01% MLMG::mgVcycle_up::4 82 7.453e-05 7.453e-05 7.453e-05 0.01% MLMG::actualBottomSolve() 82 7.442e-05 7.442e-05 7.442e-05 0.01% MLMG::solve() 11 6.87e-05 6.87e-05 6.87e-05 0.01% Castro::advance() 10 6.278e-05 6.278e-05 6.278e-05 0.01% Castro::initialize_do_advance() 10 6.221e-05 6.221e-05 6.221e-05 0.01% MLMG::mgVcycle_up::0 82 5.652e-05 5.652e-05 5.652e-05 0.01% MLMG::oneIter() 82 5.619e-05 5.619e-05 5.619e-05 0.01% Castro::finalize_advance() 10 5.552e-05 5.552e-05 5.552e-05 0.01% MLMG::mgVcycle_up::3 82 5.35e-05 5.35e-05 5.35e-05 0.01% MLMG::mgVcycle_up::1 82 5.251e-05 5.251e-05 5.251e-05 0.01% MLMG::mgVcycle_up::2 82 5.236e-05 5.236e-05 5.236e-05 0.01% MLCellLinOp::solutionResidual() 93 5.056e-05 5.056e-05 5.056e-05 0.01% StateData::define() 4 4.214e-05 4.214e-05 4.214e-05 0.00% Castro::construct_new_source() 50 3.962e-05 3.962e-05 3.962e-05 0.00% MLMG::computeResidual() 82 3.735e-05 3.735e-05 3.735e-05 0.00% Castro::swap_state_time_levels() 10 3.599e-05 3.599e-05 3.599e-05 0.00% Castro::enforce_consistent_e() 1 3.508e-05 3.508e-05 3.508e-05 0.00% FillPatchSingleLevel 41 3.407e-05 3.407e-05 3.407e-05 0.00% Gravity::actual_multilevel_solve() 1 3.186e-05 3.186e-05 3.186e-05 0.00% Castro::finalize_do_advance() 10 3.139e-05 3.139e-05 3.139e-05 0.00% MLMG::mgVcycle_bottom 82 3.026e-05 3.026e-05 3.026e-05 0.00% MLPoisson::define() 11 2.831e-05 2.831e-05 2.831e-05 0.00% Castro::initMFs() 1 2.78e-05 2.78e-05 2.78e-05 0.00% Amr::writeSmallPlotFile() 1 2.666e-05 2.666e-05 2.666e-05 0.00% makeSFC 55 2.62e-05 2.62e-05 2.62e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.416e-05 2.416e-05 2.416e-05 0.00% MLLinOp::define() 11 2.351e-05 2.351e-05 2.351e-05 0.00% Castro::buildMetrics() 1 2.202e-05 2.202e-05 2.202e-05 0.00% Amr::FinalizeInit() 1 2.001e-05 2.001e-05 2.001e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.903e-05 1.903e-05 1.903e-05 0.00% Amr::defBaseLevel() 1 1.881e-05 1.881e-05 1.881e-05 0.00% Castro::construct_old_source() 50 1.83e-05 1.83e-05 1.83e-05 0.00% Castro::do_new_sources() 10 1.634e-05 1.634e-05 1.634e-05 0.00% DistributionMapping::Distribute() 56 1.576e-05 1.576e-05 1.576e-05 0.00% Castro::do_old_sources() 10 1.538e-05 1.538e-05 1.538e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.277e-05 1.277e-05 1.277e-05 0.00% Castro::apply_source_to_state() 20 1.153e-05 1.153e-05 1.153e-05 0.00% Castro::check_for_nan() 20 1.136e-05 1.136e-05 1.136e-05 0.00% Castro::construct_old_gravity() 10 9.968e-06 9.968e-06 9.968e-06 0.00% Castro::post_timestep() 10 9.965e-06 9.965e-06 9.965e-06 0.00% MLMG::computeMLResidual() 11 9.616e-06 9.616e-06 9.616e-06 0.00% MLPoisson::prepareForSolve() 11 8.861e-06 8.861e-06 8.861e-06 0.00% Amr::initSubcycle() 1 8.752e-06 8.752e-06 8.752e-06 0.00% Gravity::swapTimeLevels() 10 8.748e-06 8.748e-06 8.748e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.17e-06 7.17e-06 7.17e-06 0.00% MLMG::getGradSolution() 11 6.425e-06 6.425e-06 6.425e-06 0.00% Castro::computeNewDt() 9 6.119e-06 6.119e-06 6.119e-06 0.00% Castro::post_init() 1 4.783e-06 4.783e-06 4.783e-06 0.00% Amr::InitializeInit() 1 4.764e-06 4.764e-06 4.764e-06 0.00% Castro::create_source_corrector() 10 4.595e-06 4.595e-06 4.595e-06 0.00% AmrLevel::checkPointPost() 3 4.388e-06 4.388e-06 4.388e-06 0.00% Gravity::set_mass_offset() 11 4.117e-06 4.117e-06 4.117e-06 0.00% AmrLevel::checkPointPre() 3 4.116e-06 4.116e-06 4.116e-06 0.00% Castro::retry_advance_ctu() 10 3.912e-06 3.912e-06 3.912e-06 0.00% MLMG::MLResNormInf() 11 3.591e-06 3.591e-06 3.591e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.632e-06 2.632e-06 2.632e-06 0.00% Castro::FluxRegCrseInit 10 2.56e-06 2.56e-06 2.56e-06 0.00% Amr::init() 1 2.529e-06 2.529e-06 2.529e-06 0.00% Castro::computeInitialDt() 2 2.476e-06 2.476e-06 2.476e-06 0.00% Castro::FluxRegFineAdd() 10 2.132e-06 2.132e-06 2.132e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.994e-06 1.994e-06 1.994e-06 0.00% Castro::post_regrid() 1 1.045e-06 1.045e-06 1.045e-06 0.00% Amr::initialInit() 1 9.23e-07 9.23e-07 9.23e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9121 0.9121 0.9121 100.00% Amr::coarseTimeStep() 10 0.7437 0.7437 0.7437 81.54% Amr::timeStep() 10 0.6229 0.6229 0.6229 68.29% Castro::advance() 10 0.6144 0.6144 0.6144 67.37% Castro::subcycle_advance_ctu() 10 0.6016 0.6016 0.6016 65.95% Castro::do_advance_ctu() 10 0.6013 0.6013 0.6013 65.92% Gravity::solve_phi_with_mlmg() 11 0.3182 0.3182 0.3182 34.88% Gravity::actual_solve_with_mlmg() 11 0.3093 0.3093 0.3093 33.91% Castro::construct_new_gravity() 10 0.2894 0.2894 0.2894 31.73% MLMG::solve() 11 0.2863 0.2863 0.2863 31.39% Gravity::solve_for_phi() 10 0.274 0.274 0.274 30.03% MLMG::oneIter() 82 0.2713 0.2713 0.2713 29.74% MLMG::mgVcycle() 82 0.2696 0.2696 0.2696 29.55% Castro::construct_ctu_hydro_source() 10 0.2247 0.2247 0.2247 24.64% VisMF::Write(FabArray) 11 0.2066 0.2066 0.2066 22.65% Amr::checkPoint() 3 0.1703 0.1703 0.1703 18.68% AmrLevel::checkPoint() 3 0.1472 0.1472 0.1472 16.14% StateData::checkPoint() 12 0.1471 0.1471 0.1471 16.13% MLCellLinOp::smooth() 1640 0.1377 0.1377 0.1377 15.09% Amr::init() 1 0.1358 0.1358 0.1358 14.89% MLCellLinOp::applyBC() 4433 0.09665 0.09665 0.09665 10.60% MLMG::mgVcycle_bottom 82 0.08304 0.08304 0.08304 9.10% MLMG::actualBottomSolve() 82 0.08301 0.08301 0.08301 9.10% MLCGSolver::bicgstab 82 0.0822 0.0822 0.0822 9.01% MLPoisson::Fsmooth() 3280 0.06477 0.06477 0.06477 7.10% Amr::writePlotFile() 2 0.06215 0.06215 0.06215 6.81% Amr::initialInit() 1 0.0524 0.0524 0.0524 5.74% Castro::clean_state() 62 0.05176 0.05176 0.05176 5.68% Amr::FinalizeInit() 1 0.0481 0.0481 0.0481 5.27% FillPatchIterator::Initialize 41 0.04737 0.04737 0.04737 5.19% Castro::post_init() 1 0.04667 0.04667 0.04667 5.12% FillPatchSingleLevel 41 0.04576 0.04576 0.04576 5.02% Gravity::multilevel_solve_for_new_phi() 1 0.0447 0.0447 0.0447 4.90% Gravity::actual_multilevel_solve() 1 0.04468 0.04468 0.04468 4.90% StateDataPhysBCFunct::() 41 0.04176 0.04176 0.04176 4.58% MLCellLinOp::apply() 1142 0.03661 0.03661 0.03661 4.01% MLMG::mgVcycle_down::0 82 0.0357 0.0357 0.0357 3.91% MLMG::mgVcycle_up::0 82 0.03077 0.03077 0.03077 3.37% Castro::initialize_do_advance() 10 0.02503 0.02503 0.02503 2.74% StateData::FillBoundary(geom) 328 0.02333 0.02333 0.02333 2.56% MultiFab::Dot() 1114 0.02282 0.02282 0.02282 2.50% Castro::computeTemp() 63 0.02185 0.02185 0.02185 2.40% MLCellLinOp::correctionResidual() 492 0.02147 0.02147 0.02147 2.35% MLMG:computeResOfCorrection() 410 0.01855 0.01855 0.01855 2.03% MLPoisson::define() 11 0.01851 0.01851 0.01851 2.03% MLMG::mgVcycle_down::1 82 0.01794 0.01794 0.01794 1.97% MLMG::mgVcycle_down::2 82 0.01742 0.01742 0.01742 1.91% Gravity::get_new_grav_vector() 11 0.0171 0.0171 0.0171 1.87% Castro::normalize_species() 62 0.01671 0.01671 0.01671 1.83% MLMG::mgVcycle_down::3 82 0.01656 0.01656 0.01656 1.82% Castro::expand_state() 10 0.01624 0.01624 0.01624 1.78% FabArray::FillBoundary() 4023 0.01582 0.01582 0.01582 1.73% MLMG::mgVcycle_down::4 82 0.01582 0.01582 0.01582 1.73% FillBoundary_nowait() 4023 0.01501 0.01501 0.01501 1.65% Castro::construct_old_gravity() 10 0.01497 0.01497 0.01497 1.64% Gravity::get_old_grav_vector() 10 0.01496 0.01496 0.01496 1.64% CGSolver::sxay() 1586 0.01489 0.01489 0.01489 1.63% MultiFab::LinComb() 1586 0.01455 0.01455 0.01455 1.60% FabArray::ParallelCopy() 861 0.01444 0.01444 0.01444 1.58% FabArray::setVal() 1144 0.01437 0.01437 0.01437 1.58% FabArray::ParallelCopy_nowait() 861 0.01418 0.01418 0.01418 1.55% MLCGSolver::ParallelAllReduce 1514 0.01361 0.01361 0.01361 1.49% MLMG::mgVcycle_up::2 82 0.01343 0.01343 0.01343 1.47% MLMG::mgVcycle_up::1 82 0.01319 0.01319 0.01319 1.45% MLMG::addInterpCorrection() 410 0.01309 0.01309 0.01309 1.44% MLCellLinOp::defineAuxData() 11 0.01306 0.01306 0.01306 1.43% Castro::enforce_min_density() 62 0.01293 0.01293 0.01293 1.42% Castro::initialize_advance() 10 0.01277 0.01277 0.01277 1.40% MLMG::mgVcycle_up::3 82 0.01276 0.01276 0.01276 1.40% MLMG::mgVcycle_up::4 82 0.01271 0.01271 0.01271 1.39% Castro::do_new_sources() 10 0.01259 0.01259 0.01259 1.38% amrex::average_down 410 0.01216 0.01216 0.01216 1.33% MLPoisson::Fapply() 1142 0.01187 0.01187 0.01187 1.30% Castro::do_old_sources() 10 0.01154 0.01154 0.01154 1.27% Gravity::fill_multipole_BCs() 11 0.008682 0.008682 0.008682 0.95% Castro::post_timestep() 10 0.008262 0.008262 0.008262 0.91% MLCellLinOp::solutionResidual() 93 0.007162 0.007162 0.007162 0.79% MultiFab::Xpay() 585 0.006658 0.006658 0.006658 0.73% MLMG::computeResidual() 82 0.00618 0.00618 0.00618 0.68% Castro::estTimeStep() 21 0.006079 0.006079 0.006079 0.67% Castro::reset_internal_energy(MultiFab) 63 0.005993 0.005993 0.005993 0.66% MLMG::prepareForSolve() 11 0.005416 0.005416 0.005416 0.59% MLCellLinOp::defineBC() 11 0.005214 0.005214 0.005214 0.57% BndryData::define() 11 0.004931 0.004931 0.004931 0.54% Amr::InitializeInit() 1 0.004297 0.004297 0.004297 0.47% Amr::defBaseLevel() 1 0.004292 0.004292 0.004292 0.47% Castro::initData() 1 0.003777 0.003777 0.003777 0.41% Castro::construct_new_source() 50 0.003252 0.003252 0.003252 0.36% Castro::construct_new_gravity_source() 10 0.003212 0.003212 0.003212 0.35% Castro::computeNewDt() 9 0.002751 0.002751 0.002751 0.30% Castro::construct_old_source() 50 0.002504 0.002504 0.002504 0.27% Castro::construct_old_gravity_source() 10 0.002485 0.002485 0.002485 0.27% MLMG::ResNormInf() 93 0.002105 0.002105 0.002105 0.23% Castro::apply_source_to_state() 20 0.001829 0.001829 0.001829 0.20% MultiFab::Saxpy() 20 0.001817 0.001817 0.001817 0.20% MultiFab::Add() 82 0.001691 0.001691 0.001691 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001686 0.001686 0.001686 0.18% Castro::reset_internal_energy(Fab) 504 0.001587 0.001587 0.001587 0.17% MLCellLinOp::setLevelBC() 11 0.001554 0.001554 0.001554 0.17% MLMG::getGradSolution() 11 0.00144 0.00144 0.00144 0.16% FabArrayBase::getCPC() 1323 0.001438 0.001438 0.001438 0.16% MLCellLinOp::compGrad() 11 0.001433 0.001433 0.001433 0.16% FabArray::mult() 43 0.001327 0.001327 0.001327 0.15% FabArray::setDomainBndry() 41 0.00132 0.00132 0.00132 0.14% MLPoisson::prepareForSolve() 11 0.001191 0.001191 0.001191 0.13% Castro::check_for_nan() 20 0.001188 0.001188 0.001188 0.13% MLCellLinOp::prepareForSolve() 11 0.001182 0.001182 0.001182 0.13% MultiFab::contains_nan() 20 0.001177 0.001177 0.001177 0.13% Castro::post_regrid() 1 0.001154 0.001154 0.001154 0.13% Castro::enforce_speed_limit() 62 0.001101 0.001101 0.001101 0.12% MLMG::computeMLResidual() 11 0.001028 0.001028 0.001028 0.11% Gravity::update_max_rhs() 11 0.0008132 0.0008132 0.0008132 0.09% Castro::computeInitialDt() 2 0.000738 0.000738 0.000738 0.08% FabArrayBase::CPC::define() 454 0.0006721 0.0006721 0.0006721 0.07% FabArrayBase::getFB() 4023 0.0006701 0.0006701 0.0006701 0.07% Amr::InitAmr() 1 0.0004955 0.0004955 0.0004955 0.05% Gravity::swapTimeLevels() 10 0.0004427 0.0004427 0.0004427 0.05% Castro::Castro() 1 0.0004426 0.0004426 0.0004426 0.05% MultiFab::Copy() 11 0.0003334 0.0003334 0.0003334 0.04% MLMG::MLResNormInf() 11 0.0002804 0.0002804 0.0002804 0.03% MultiFab::max() 11 0.000259 0.000259 0.000259 0.03% MLMG::MLRhsNormInf() 11 0.0002175 0.0002175 0.0002175 0.02% MLLinOp::define() 11 0.0002101 0.0002101 0.0002101 0.02% MLLinOp::defineGrids() 11 0.0001866 0.0001866 0.0001866 0.02% Castro::buildMetrics() 1 0.0001639 0.0001639 0.0001639 0.02% FabArrayBase::FB::FB() 56 8.299e-05 8.299e-05 8.299e-05 0.01% Castro::finalize_advance() 10 6.022e-05 6.022e-05 6.022e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.303e-05 5.303e-05 5.303e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.931e-05 4.931e-05 4.931e-05 0.01% StateData::define() 4 4.214e-05 4.214e-05 4.214e-05 0.00% makeSFC 55 4.027e-05 4.027e-05 4.027e-05 0.00% Castro::swap_state_time_levels() 10 3.599e-05 3.599e-05 3.599e-05 0.00% Castro::enforce_consistent_e() 1 3.508e-05 3.508e-05 3.508e-05 0.00% Castro::finalize_do_advance() 10 3.139e-05 3.139e-05 3.139e-05 0.00% Castro::initMFs() 1 2.78e-05 2.78e-05 2.78e-05 0.00% Amr::writeSmallPlotFile() 1 2.666e-05 2.666e-05 2.666e-05 0.00% DistributionMapping::Distribute() 56 1.576e-05 1.576e-05 1.576e-05 0.00% Amr::initSubcycle() 1 8.752e-06 8.752e-06 8.752e-06 0.00% Castro::create_source_corrector() 10 4.595e-06 4.595e-06 4.595e-06 0.00% AmrLevel::checkPointPost() 3 4.388e-06 4.388e-06 4.388e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.324e-06 4.324e-06 4.324e-06 0.00% Gravity::set_mass_offset() 11 4.117e-06 4.117e-06 4.117e-06 0.00% AmrLevel::checkPointPre() 3 4.116e-06 4.116e-06 4.116e-06 0.00% Castro::retry_advance_ctu() 10 3.912e-06 3.912e-06 3.912e-06 0.00% Castro::FluxRegCrseInit 10 2.56e-06 2.56e-06 2.56e-06 0.00% Castro::FluxRegFineAdd() 10 2.132e-06 2.132e-06 2.132e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.994e-06 1.994e-06 1.994e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-16-g826cd378f8ba) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09-16-g826cd378f8ba) initialized Starting run at 08:32:31 UTC on 2022-09-20. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09-16-g826cd378f Microphysics git describe: 22.08-13-g1adf1bdb reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.477655657 Restart time = 0.048247001 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052224236 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051226613 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.051559892 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.063306844 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.082270145 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032259507 seconds Ending run at 08:32:32 UTC on 2022-09-20. Run time = 0.382170815 Run time without initialization = 0.333296455 Average number of zones advanced per microsecond: 3.933 Average number of zones advanced per microsecond per rank: 3.933 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3822 ... 0.3822 ... 0.3822 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0961 0.0961 0.0961 25.14% VisMF::Read() 3 0.04032 0.04032 0.04032 10.55% MLCellLinOp::applyBC() 1946 0.03521 0.03521 0.03521 9.21% VisMF::Write(FabArray) 1 0.03077 0.03077 0.03077 8.05% MLPoisson::Fsmooth() 1440 0.02763 0.02763 0.02763 7.23% StateData::FillBoundary(geom) 160 0.01208 0.01208 0.01208 3.16% MLCGSolver::bicgstab 36 0.01034 0.01034 0.01034 2.71% Castro::computeTemp() 30 0.01025 0.01025 0.01025 2.68% MultiFab::Dot() 484 0.009749 0.009749 0.009749 2.55% Castro::normalize_species() 30 0.00883 0.00883 0.00883 2.31% FabArray::setVal() 537 0.006807 0.006807 0.006807 1.78% MultiFab::LinComb() 690 0.006288 0.006288 0.006288 1.65% FillBoundary_nowait() 1766 0.006268 0.006268 0.006268 1.64% MLCellLinOp::defineAuxData() 6 0.006263 0.006263 0.006263 1.64% FabArray::ParallelCopy_nowait() 380 0.006069 0.006069 0.006069 1.59% StateDataPhysBCFunct::() 20 0.00525 0.00525 0.00525 1.37% MLPoisson::Fapply() 500 0.005135 0.005135 0.005135 1.34% Castro::enforce_min_density() 30 0.004922 0.004922 0.004922 1.29% Gravity::fill_multipole_BCs() 6 0.004859 0.004859 0.004859 1.27% Amr::restart() 1 0.003615 0.003615 0.003615 0.95% Castro::estTimeStep() 10 0.003486 0.003486 0.003486 0.91% MLMG::addInterpCorrection() 180 0.003393 0.003393 0.003393 0.89% amrex::average_down 180 0.002985 0.002985 0.002985 0.78% MultiFab::Xpay() 258 0.002904 0.002904 0.002904 0.76% Castro::do_advance_ctu() 5 0.002307 0.002307 0.002307 0.60% BndryData::define() 6 0.00219 0.00219 0.00219 0.57% Castro::enforce_speed_limit() 30 0.001764 0.001764 0.001764 0.46% Castro::reset_internal_energy(MultiFab) 30 0.001684 0.001684 0.001684 0.44% Castro::construct_new_gravity_source() 5 0.001579 0.001579 0.001579 0.41% Amr::writePlotFile() 1 0.001573 0.001573 0.001573 0.41% Castro::construct_old_gravity_source() 5 0.001409 0.001409 0.001409 0.37% Castro::reset_internal_energy(Fab) 240 0.001164 0.001164 0.001164 0.30% Gravity::get_old_grav_vector() 5 0.0009646 0.0009646 0.0009646 0.25% MLMG::ResNormInf() 42 0.0009288 0.0009288 0.0009288 0.24% MultiFab::Saxpy() 10 0.0009241 0.0009241 0.0009241 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009008 0.0009008 0.0009008 0.24% Castro::expand_state() 5 0.0008754 0.0008754 0.0008754 0.23% Gravity::get_new_grav_vector() 5 0.0008737 0.0008737 0.0008737 0.23% MLCellLinOp::setLevelBC() 6 0.0008342 0.0008342 0.0008342 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007916 0.0007916 0.0007916 0.21% MultiFab::Add() 36 0.0007233 0.0007233 0.0007233 0.19% MLMG::prepareForSolve() 6 0.0006704 0.0006704 0.0006704 0.18% FabArray::setDomainBndry() 20 0.000652 0.000652 0.000652 0.17% FabArray::mult() 22 0.0006513 0.0006513 0.0006513 0.17% MLCellLinOp::prepareForSolve() 6 0.0006452 0.0006452 0.0006452 0.17% MultiFab::contains_nan() 10 0.0005831 0.0005831 0.0005831 0.15% MLCellLinOp::smooth() 720 0.0005018 0.0005018 0.0005018 0.13% MLCellLinOp::compGrad() 6 0.0004943 0.0004943 0.0004943 0.13% Amr::InitAmr() 1 0.0004475 0.0004475 0.0004475 0.12% FabArrayBase::CPC::define() 244 0.0003831 0.0003831 0.0003831 0.10% FabArray::FillBoundary() 1766 0.0003821 0.0003821 0.0003821 0.10% FabArrayBase::getCPC() 632 0.0003648 0.0003648 0.0003648 0.10% main() 1 0.0002905 0.0002905 0.0002905 0.08% FabArrayBase::getFB() 1766 0.0002519 0.0002519 0.0002519 0.07% Castro::subcycle_advance_ctu() 5 0.000234 0.000234 0.000234 0.06% Gravity::update_max_rhs() 6 0.0002291 0.0002291 0.0002291 0.06% Gravity::solve_for_phi() 5 0.0002136 0.0002136 0.0002136 0.06% MLCellLinOp::apply() 500 0.0002 0.0002 0.0002 0.05% Amr::coarseTimeStep() 5 0.0001869 0.0001869 0.0001869 0.05% Castro::construct_new_gravity() 5 0.0001833 0.0001833 0.0001833 0.05% MultiFab::Copy() 6 0.0001791 0.0001791 0.0001791 0.05% Castro::construct_new_source() 25 0.000175 0.000175 0.000175 0.05% Castro::create_source_corrector() 5 0.000168 0.000168 0.000168 0.04% CGSolver::sxay() 690 0.0001652 0.0001652 0.0001652 0.04% MLCellLinOp::defineBC() 6 0.0001467 0.0001467 0.0001467 0.04% FillPatchIterator::Initialize 20 0.0001369 0.0001369 0.0001369 0.04% MultiFab::max() 6 0.0001354 0.0001354 0.0001354 0.04% FabArray::ParallelCopy() 380 0.0001253 0.0001253 0.0001253 0.03% MLCGSolver::ParallelAllReduce 659 0.0001248 0.0001248 0.0001248 0.03% Castro::construct_old_source() 25 0.0001203 0.0001203 0.0001203 0.03% MLMG::MLRhsNormInf() 6 0.0001142 0.0001142 0.0001142 0.03% MLMG::mgVcycle() 36 0.0001015 0.0001015 0.0001015 0.03% MLCellLinOp::correctionResidual() 216 9.403e-05 9.403e-05 9.403e-05 0.02% Castro::initialize_do_advance() 5 9.014e-05 9.014e-05 9.014e-05 0.02% Castro::advance() 5 8.464e-05 8.464e-05 8.464e-05 0.02% MLLinOp::defineGrids() 6 8.21e-05 8.21e-05 8.21e-05 0.02% Castro::computeNewDt() 5 8.028e-05 8.028e-05 8.028e-05 0.02% Amr::timeStep() 5 7.903e-05 7.903e-05 7.903e-05 0.02% Castro::post_timestep() 5 7.622e-05 7.622e-05 7.622e-05 0.02% Castro::construct_old_gravity() 5 7.336e-05 7.336e-05 7.336e-05 0.02% AmrLevel::restart() 1 7.32e-05 7.32e-05 7.32e-05 0.02% StateData::restartDoit() 4 7.179e-05 7.179e-05 7.179e-05 0.02% MLMG:computeResOfCorrection() 180 6.693e-05 6.693e-05 6.693e-05 0.02% Castro::finalize_advance() 5 6.606e-05 6.606e-05 6.606e-05 0.02% FabArrayBase::FB::FB() 26 5.694e-05 5.694e-05 5.694e-05 0.01% Castro::initialize_advance() 5 5.351e-05 5.351e-05 5.351e-05 0.01% MLMG::mgVcycle_down::0 36 4.509e-05 4.509e-05 4.509e-05 0.01% MLMG::mgVcycle_down::1 36 4.156e-05 4.156e-05 4.156e-05 0.01% Castro::clean_state() 30 3.878e-05 3.878e-05 3.878e-05 0.01% MLMG::mgVcycle_down::2 36 3.867e-05 3.867e-05 3.867e-05 0.01% MLMG::mgVcycle_down::4 36 3.586e-05 3.586e-05 3.586e-05 0.01% MLMG::mgVcycle_down::3 36 3.479e-05 3.479e-05 3.479e-05 0.01% MLMG::actualBottomSolve() 36 3.446e-05 3.446e-05 3.446e-05 0.01% Amr::writeSmallPlotFile() 1 3.296e-05 3.296e-05 3.296e-05 0.01% MLMG::solve() 6 3.179e-05 3.179e-05 3.179e-05 0.01% MLMG::mgVcycle_up::4 36 3.156e-05 3.156e-05 3.156e-05 0.01% Castro::post_restart() 1 3.123e-05 3.123e-05 3.123e-05 0.01% Castro::buildMetrics() 1 3.081e-05 3.081e-05 3.081e-05 0.01% Gravity::actual_multilevel_solve() 1 3.027e-05 3.027e-05 3.027e-05 0.01% Castro::swap_state_time_levels() 5 2.685e-05 2.685e-05 2.685e-05 0.01% MLMG::mgVcycle_up::0 36 2.627e-05 2.627e-05 2.627e-05 0.01% Castro::initMFs() 1 2.609e-05 2.609e-05 2.609e-05 0.01% MLMG::oneIter() 36 2.53e-05 2.53e-05 2.53e-05 0.01% MLMG::mgVcycle_up::3 36 2.42e-05 2.42e-05 2.42e-05 0.01% MLCellLinOp::solutionResidual() 42 2.388e-05 2.388e-05 2.388e-05 0.01% MLMG::mgVcycle_up::2 36 2.326e-05 2.326e-05 2.326e-05 0.01% MLMG::mgVcycle_up::1 36 2.18e-05 2.18e-05 2.18e-05 0.01% MLPoisson::define() 6 2.092e-05 2.092e-05 2.092e-05 0.01% MLLinOp::define() 6 2.074e-05 2.074e-05 2.074e-05 0.01% Castro::finalize_do_advance() 5 1.906e-05 1.906e-05 1.906e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.838e-05 1.838e-05 1.838e-05 0.00% MLMG::computeResidual() 36 1.677e-05 1.677e-05 1.677e-05 0.00% MLMG::mgVcycle_bottom 36 1.437e-05 1.437e-05 1.437e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.379e-05 1.379e-05 1.379e-05 0.00% makeSFC 30 1.362e-05 1.362e-05 1.362e-05 0.00% FillPatchSingleLevel 20 1.361e-05 1.361e-05 1.361e-05 0.00% Amr::initSubcycle() 1 9.512e-06 9.512e-06 9.512e-06 0.00% Castro::do_new_sources() 5 9.105e-06 9.105e-06 9.105e-06 0.00% DistributionMapping::Distribute() 31 8.462e-06 8.462e-06 8.462e-06 0.00% Castro::do_old_sources() 5 8.301e-06 8.301e-06 8.301e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.004e-06 7.004e-06 7.004e-06 0.00% Castro::check_for_nan() 10 5.911e-06 5.911e-06 5.911e-06 0.00% Castro::apply_source_to_state() 10 5.415e-06 5.415e-06 5.415e-06 0.00% MLMG::computeMLResidual() 6 4.828e-06 4.828e-06 4.828e-06 0.00% MLPoisson::prepareForSolve() 6 4.608e-06 4.608e-06 4.608e-06 0.00% Gravity::swapTimeLevels() 5 4.12e-06 4.12e-06 4.12e-06 0.00% MLMG::getGradSolution() 6 3.2e-06 3.2e-06 3.2e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.893e-06 2.893e-06 2.893e-06 0.00% Gravity::set_mass_offset() 6 2.281e-06 2.281e-06 2.281e-06 0.00% MLMG::MLResNormInf() 6 2.275e-06 2.275e-06 2.275e-06 0.00% Castro::retry_advance_ctu() 5 1.789e-06 1.789e-06 1.789e-06 0.00% Castro::FluxRegCrseInit 5 1.515e-06 1.515e-06 1.515e-06 0.00% Castro::FluxRegFineAdd() 5 1.084e-06 1.084e-06 1.084e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.032e-06 1.032e-06 1.032e-06 0.00% AmrLevel::AmrLevel() 1 1.001e-06 1.001e-06 1.001e-06 0.00% Amr::init() 1 1.001e-06 1.001e-06 1.001e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3822 0.3822 0.3822 100.00% Amr::coarseTimeStep() 5 0.3008 0.3008 0.3008 78.69% Amr::timeStep() 5 0.2984 0.2984 0.2984 78.09% Castro::advance() 5 0.2942 0.2942 0.2942 76.98% Castro::subcycle_advance_ctu() 5 0.287 0.287 0.287 75.10% Castro::do_advance_ctu() 5 0.2868 0.2868 0.2868 75.04% Castro::construct_new_gravity() 5 0.146 0.146 0.146 38.20% Gravity::solve_phi_with_mlmg() 6 0.1417 0.1417 0.1417 37.09% Gravity::solve_for_phi() 5 0.1381 0.1381 0.1381 36.14% Gravity::actual_solve_with_mlmg() 6 0.1368 0.1368 0.1368 35.78% MLMG::solve() 6 0.1243 0.1243 0.1243 32.53% MLMG::oneIter() 36 0.1171 0.1171 0.1171 30.63% MLMG::mgVcycle() 36 0.1163 0.1163 0.1163 30.43% Castro::construct_ctu_hydro_source() 5 0.09608 0.09608 0.09608 25.14% MLCellLinOp::smooth() 720 0.05942 0.05942 0.05942 15.55% Amr::init() 1 0.04829 0.04829 0.04829 12.63% Amr::restart() 1 0.04829 0.04829 0.04829 12.63% MLCellLinOp::applyBC() 1946 0.04217 0.04217 0.04217 11.03% AmrLevel::restart() 1 0.04052 0.04052 0.04052 10.60% StateData::restartDoit() 4 0.04044 0.04044 0.04044 10.58% VisMF::Read() 3 0.04032 0.04032 0.04032 10.55% MLMG::mgVcycle_bottom 36 0.03572 0.03572 0.03572 9.35% MLMG::actualBottomSolve() 36 0.0357 0.0357 0.0357 9.34% MLCGSolver::bicgstab 36 0.03535 0.03535 0.03535 9.25% Amr::writePlotFile() 1 0.03234 0.03234 0.03234 8.46% VisMF::Write(FabArray) 1 0.03077 0.03077 0.03077 8.05% Castro::clean_state() 30 0.02865 0.02865 0.02865 7.50% MLPoisson::Fsmooth() 1440 0.02763 0.02763 0.02763 7.23% FillPatchIterator::Initialize 20 0.02013 0.02013 0.02013 5.27% FillPatchSingleLevel 20 0.01934 0.01934 0.01934 5.06% StateDataPhysBCFunct::() 20 0.01732 0.01732 0.01732 4.53% MLCellLinOp::apply() 500 0.01594 0.01594 0.01594 4.17% MLMG::mgVcycle_down::0 36 0.01549 0.01549 0.01549 4.05% MLMG::mgVcycle_up::0 36 0.01334 0.01334 0.01334 3.49% Castro::computeTemp() 30 0.0131 0.0131 0.0131 3.43% StateData::FillBoundary(geom) 160 0.01208 0.01208 0.01208 3.16% Castro::initialize_do_advance() 5 0.01138 0.01138 0.01138 2.98% MLPoisson::define() 6 0.01003 0.01003 0.01003 2.63% MultiFab::Dot() 484 0.009749 0.009749 0.009749 2.55% MLCellLinOp::correctionResidual() 216 0.009308 0.009308 0.009308 2.44% Castro::normalize_species() 30 0.00883 0.00883 0.00883 2.31% Castro::do_new_sources() 5 0.008124 0.008124 0.008124 2.13% MLMG:computeResOfCorrection() 180 0.00802 0.00802 0.00802 2.10% MLMG::mgVcycle_down::1 36 0.007722 0.007722 0.007722 2.02% Gravity::get_new_grav_vector() 5 0.007686 0.007686 0.007686 2.01% Castro::construct_old_gravity() 5 0.007562 0.007562 0.007562 1.98% MLMG::mgVcycle_down::2 36 0.007498 0.007498 0.007498 1.96% Gravity::get_old_grav_vector() 5 0.007488 0.007488 0.007488 1.96% MLMG::mgVcycle_down::3 36 0.007115 0.007115 0.007115 1.86% Castro::initialize_advance() 5 0.00705 0.00705 0.00705 1.84% MLCellLinOp::defineAuxData() 6 0.007002 0.007002 0.007002 1.83% FabArray::FillBoundary() 1766 0.006959 0.006959 0.006959 1.82% MLMG::mgVcycle_down::4 36 0.006834 0.006834 0.006834 1.79% FabArray::setVal() 537 0.006807 0.006807 0.006807 1.78% FillBoundary_nowait() 1766 0.006577 0.006577 0.006577 1.72% FabArray::ParallelCopy() 380 0.006562 0.006562 0.006562 1.72% CGSolver::sxay() 690 0.006453 0.006453 0.006453 1.69% FabArray::ParallelCopy_nowait() 380 0.006437 0.006437 0.006437 1.68% MultiFab::LinComb() 690 0.006288 0.006288 0.006288 1.65% Castro::do_old_sources() 5 0.006036 0.006036 0.006036 1.58% MLCGSolver::ParallelAllReduce 659 0.005855 0.005855 0.005855 1.53% MLMG::mgVcycle_up::2 36 0.005778 0.005778 0.005778 1.51% MLMG::mgVcycle_up::1 36 0.005712 0.005712 0.005712 1.49% MLMG::addInterpCorrection() 180 0.00568 0.00568 0.00568 1.49% Castro::expand_state() 5 0.005603 0.005603 0.005603 1.47% MLMG::mgVcycle_up::4 36 0.005513 0.005513 0.005513 1.44% MLMG::mgVcycle_up::3 36 0.005498 0.005498 0.005498 1.44% amrex::average_down 180 0.00526 0.00526 0.00526 1.38% MLPoisson::Fapply() 500 0.005135 0.005135 0.005135 1.34% Castro::enforce_min_density() 30 0.004922 0.004922 0.004922 1.29% Gravity::fill_multipole_BCs() 6 0.004859 0.004859 0.004859 1.27% Castro::post_timestep() 5 0.00414 0.00414 0.00414 1.08% Castro::post_restart() 1 0.003982 0.003982 0.003982 1.04% Gravity::multilevel_solve_for_new_phi() 1 0.003856 0.003856 0.003856 1.01% Gravity::actual_multilevel_solve() 1 0.003838 0.003838 0.003838 1.00% Castro::estTimeStep() 10 0.003486 0.003486 0.003486 0.91% MLCellLinOp::solutionResidual() 42 0.003226 0.003226 0.003226 0.84% MLMG::prepareForSolve() 6 0.002939 0.002939 0.002939 0.77% MultiFab::Xpay() 258 0.002904 0.002904 0.002904 0.76% MLCellLinOp::defineBC() 6 0.002878 0.002878 0.002878 0.75% Castro::reset_internal_energy(MultiFab) 30 0.002848 0.002848 0.002848 0.75% BndryData::define() 6 0.002731 0.002731 0.002731 0.71% MLMG::computeResidual() 36 0.002682 0.002682 0.002682 0.70% Castro::computeNewDt() 5 0.002136 0.002136 0.002136 0.56% Castro::enforce_speed_limit() 30 0.001764 0.001764 0.001764 0.46% Castro::construct_new_source() 25 0.001754 0.001754 0.001754 0.46% Castro::construct_new_gravity_source() 5 0.001579 0.001579 0.001579 0.41% Castro::construct_old_source() 25 0.00153 0.00153 0.00153 0.40% Castro::construct_old_gravity_source() 5 0.001409 0.001409 0.001409 0.37% Castro::reset_internal_energy(Fab) 240 0.001164 0.001164 0.001164 0.30% Castro::apply_source_to_state() 10 0.0009295 0.0009295 0.0009295 0.24% MLMG::ResNormInf() 42 0.0009288 0.0009288 0.0009288 0.24% MultiFab::Saxpy() 10 0.0009241 0.0009241 0.0009241 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009008 0.0009008 0.0009008 0.24% MLCellLinOp::setLevelBC() 6 0.0008342 0.0008342 0.0008342 0.22% MLMG::getGradSolution() 6 0.0007706 0.0007706 0.0007706 0.20% MLCellLinOp::compGrad() 6 0.0007674 0.0007674 0.0007674 0.20% FabArrayBase::getCPC() 632 0.0007479 0.0007479 0.0007479 0.20% MultiFab::Add() 36 0.0007233 0.0007233 0.0007233 0.19% FabArray::setDomainBndry() 20 0.000652 0.000652 0.000652 0.17% FabArray::mult() 22 0.0006513 0.0006513 0.0006513 0.17% MLPoisson::prepareForSolve() 6 0.0006498 0.0006498 0.0006498 0.17% MLCellLinOp::prepareForSolve() 6 0.0006452 0.0006452 0.0006452 0.17% Castro::check_for_nan() 10 0.0005891 0.0005891 0.0005891 0.15% MultiFab::contains_nan() 10 0.0005831 0.0005831 0.0005831 0.15% MLMG::computeMLResidual() 6 0.0005655 0.0005655 0.0005655 0.15% Amr::InitAmr() 1 0.000457 0.000457 0.000457 0.12% Gravity::update_max_rhs() 6 0.0004429 0.0004429 0.0004429 0.12% FabArrayBase::CPC::define() 244 0.0003831 0.0003831 0.0003831 0.10% FabArrayBase::getFB() 1766 0.0003089 0.0003089 0.0003089 0.08% Gravity::swapTimeLevels() 5 0.0002298 0.0002298 0.0002298 0.06% MultiFab::Copy() 6 0.0001791 0.0001791 0.0001791 0.05% Castro::create_source_corrector() 5 0.000168 0.000168 0.000168 0.04% MLMG::MLResNormInf() 6 0.0001508 0.0001508 0.0001508 0.04% Castro::buildMetrics() 1 0.0001464 0.0001464 0.0001464 0.04% MultiFab::max() 6 0.0001354 0.0001354 0.0001354 0.04% MLLinOp::define() 6 0.0001317 0.0001317 0.0001317 0.03% MLMG::MLRhsNormInf() 6 0.0001142 0.0001142 0.0001142 0.03% MLLinOp::defineGrids() 6 0.000111 0.000111 0.000111 0.03% Castro::finalize_advance() 5 6.865e-05 6.865e-05 6.865e-05 0.02% FabArrayBase::FB::FB() 26 5.694e-05 5.694e-05 5.694e-05 0.01% Amr::writeSmallPlotFile() 1 3.296e-05 3.296e-05 3.296e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.782e-05 2.782e-05 2.782e-05 0.01% Castro::swap_state_time_levels() 5 2.685e-05 2.685e-05 2.685e-05 0.01% Castro::initMFs() 1 2.609e-05 2.609e-05 2.609e-05 0.01% makeSFC 30 2.082e-05 2.082e-05 2.082e-05 0.01% Castro::finalize_do_advance() 5 1.906e-05 1.906e-05 1.906e-05 0.00% Amr::initSubcycle() 1 9.512e-06 9.512e-06 9.512e-06 0.00% DistributionMapping::Distribute() 31 8.462e-06 8.462e-06 8.462e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.158e-06 4.158e-06 4.158e-06 0.00% Gravity::set_mass_offset() 6 2.281e-06 2.281e-06 2.281e-06 0.00% Castro::retry_advance_ctu() 5 1.789e-06 1.789e-06 1.789e-06 0.00% Castro::FluxRegCrseInit 5 1.515e-06 1.515e-06 1.515e-06 0.00% Castro::FluxRegFineAdd() 5 1.084e-06 1.084e-06 1.084e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.032e-06 1.032e-06 1.032e-06 0.00% AmrLevel::AmrLevel() 1 1.001e-06 1.001e-06 1.001e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09-16-g826cd378f8ba) finalized