Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-9-g84a1f7d0715b) initialized Starting run at 08:25:40 UTC on 2022-04-07. Successfully read inputs file ... Castro git describe: 22.04-2-gb74783602 AMReX git describe: 22.04-9-g84a1f7d07 Microphysics git describe: 22.04-1-gef526ccd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.037837934 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022647218 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.110437454 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.055805023 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.048498672 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.048369855 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.070572021 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.035562157 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.060846814 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.055177724 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.056850581 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.049677636 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.060690859 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.035317328 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.022175831 seconds Ending run at 08:25:41 UTC on 2022-04-07. Run time = 0.833149219 Run time without initialization = 0.710500403 Average number of zones advanced per microsecond: 3.690 Average number of zones advanced per microsecond per rank: 3.690 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8332 ... 0.8332 ... 0.8332 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2150 0.2150 0.2150 25.80% VisMF::Write(FabArray) 11 0.1461 0.1461 0.1461 17.54% MLCellLinOp::applyBC() 4433 0.09326 0.09326 0.09326 11.19% MLPoisson::Fsmooth() 3280 0.05953 0.05953 0.05953 7.15% FabArray::setVal() 1144 0.02662 0.02662 0.02662 3.20% StateDataPhysBCFunct::() 41 0.02359 0.02359 0.02359 2.83% StateData::FillBoundary(geom) 328 0.02308 0.02308 0.02308 2.77% MLCGSolver::bicgstab 82 0.02162 0.02162 0.02162 2.60% MultiFab::Dot() 1114 0.02109 0.02109 0.02109 2.53% FillBoundary_nowait() 4023 0.01787 0.01787 0.01787 2.14% FabArray::ParallelCopy_nowait() 861 0.01599 0.01599 0.01599 1.92% Castro::computeTemp() 63 0.01274 0.01274 0.01274 1.53% MultiFab::LinComb() 1586 0.01207 0.01207 0.01207 1.45% Gravity::fill_multipole_BCs() 11 0.01196 0.01196 0.01196 1.44% MLPoisson::Fapply() 1142 0.01038 0.01038 0.01038 1.25% MLCellLinOp::defineAuxData() 11 0.009999 0.009999 0.009999 1.20% Castro::enforce_min_density() 62 0.008434 0.008434 0.008434 1.01% MLMG::addInterpCorrection() 410 0.006963 0.006963 0.006963 0.84% amrex::average_down 410 0.00654 0.00654 0.00654 0.78% MultiFab::Xpay() 585 0.005982 0.005982 0.005982 0.72% FabArray::setDomainBndry() 41 0.005931 0.005931 0.005931 0.71% Castro::reset_internal_energy() 63 0.005543 0.005543 0.005543 0.67% Castro::estTimeStep() 21 0.005006 0.005006 0.005006 0.60% Castro::expand_state() 10 0.004811 0.004811 0.004811 0.58% Castro::normalize_species() 62 0.004409 0.004409 0.004409 0.53% Amr::checkPoint() 3 0.004162 0.004162 0.004162 0.50% BndryData::define() 11 0.003595 0.003595 0.003595 0.43% Amr::writePlotFile() 2 0.003275 0.003275 0.003275 0.39% Castro::do_advance_ctu() 10 0.003182 0.003182 0.003182 0.38% Castro::construct_new_gravity_source() 10 0.003125 0.003125 0.003125 0.38% Castro::enforce_speed_limit() 62 0.003089 0.003089 0.003089 0.37% Gravity::get_new_grav_vector() 11 0.002659 0.002659 0.002659 0.32% Castro::construct_old_gravity_source() 10 0.002479 0.002479 0.002479 0.30% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002236 0.002236 0.002236 0.27% MLCellLinOp::compGrad() 11 0.00196 0.00196 0.00196 0.24% MLMG::ResNormInf() 93 0.001932 0.001932 0.001932 0.23% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.22% Gravity::get_old_grav_vector() 10 0.001755 0.001755 0.001755 0.21% MLMG::oneIter() 82 0.00168 0.00168 0.00168 0.20% MLCellLinOp::setLevelBC() 11 0.001524 0.001524 0.001524 0.18% Gravity::actual_solve_with_mlmg() 11 0.001365 0.001365 0.001365 0.16% MLCellLinOp::prepareForSolve() 11 0.001329 0.001329 0.001329 0.16% MLCellLinOp::smooth() 1640 0.001329 0.001329 0.001329 0.16% FabArray::mult() 43 0.001313 0.001313 0.001313 0.16% Castro::initData() 1 0.001245 0.001245 0.001245 0.15% FabArray::FillBoundary() 4023 0.0008564 0.0008564 0.0008564 0.10% FabArrayBase::getCPC() 1323 0.0008257 0.0008257 0.0008257 0.10% MLMG::prepareForSolve() 11 0.0007877 0.0007877 0.0007877 0.09% FabArrayBase::CPC::define() 454 0.0007203 0.0007203 0.0007203 0.09% FabArrayBase::getFB() 4023 0.0006629 0.0006629 0.0006629 0.08% MLCellLinOp::apply() 1142 0.0005629 0.0005629 0.0005629 0.07% Gravity::update_max_rhs() 11 0.0005597 0.0005597 0.0005597 0.07% MultiFab::Copy() 11 0.0005209 0.0005209 0.0005209 0.06% CGSolver::sxay() 1586 0.0004725 0.0004725 0.0004725 0.06% Gravity::solve_for_phi() 10 0.0004396 0.0004396 0.0004396 0.05% Amr::InitAmr() 1 0.0004316 0.0004316 0.0004316 0.05% MLMG::mgVcycle() 82 0.0003931 0.0003931 0.0003931 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003232 0.0003232 0.0003232 0.04% MultiFab::min() 10 0.0003207 0.0003207 0.0003207 0.04% main() 1 0.0003172 0.0003172 0.0003172 0.04% FabArray::ParallelCopy() 861 0.00031 0.00031 0.00031 0.04% FillPatchIterator::Initialize 41 0.0002587 0.0002587 0.0002587 0.03% MultiFab::max() 11 0.0002555 0.0002555 0.0002555 0.03% MLCellLinOp::correctionResidual() 492 0.0002443 0.0002443 0.0002443 0.03% Castro::construct_new_gravity() 10 0.0002401 0.0002401 0.0002401 0.03% Gravity::actual_multilevel_solve() 1 0.0002153 0.0002153 0.0002153 0.03% Amr::timeStep() 10 0.0002066 0.0002066 0.0002066 0.02% Amr::coarseTimeStep() 10 0.0002059 0.0002059 0.0002059 0.02% MLMG::MLRhsNormInf() 11 0.0001972 0.0001972 0.0001972 0.02% MLCellLinOp::defineBC() 11 0.0001864 0.0001864 0.0001864 0.02% Castro::subcycle_advance_ctu() 10 0.0001842 0.0001842 0.0001842 0.02% MLLinOp::defineGrids() 11 0.0001729 0.0001729 0.0001729 0.02% Amr::defBaseLevel() 1 0.0001477 0.0001477 0.0001477 0.02% Castro::advance() 10 0.0001358 0.0001358 0.0001358 0.02% MLMG:computeResOfCorrection() 410 0.0001345 0.0001345 0.0001345 0.02% StateData::checkPoint() 12 0.0001304 0.0001304 0.0001304 0.02% MLMG::actualBottomSolve() 82 0.0001201 0.0001201 0.0001201 0.01% MLMG::mgVcycle_down::0 82 0.0001009 0.0001009 0.0001009 0.01% MLMG::mgVcycle_down::1 82 9.302e-05 9.302e-05 9.302e-05 0.01% MLMG::mgVcycle_down::2 82 9.086e-05 9.086e-05 9.086e-05 0.01% FabArrayBase::FB::FB() 56 8.93e-05 8.93e-05 8.93e-05 0.01% MLMG::mgVcycle_down::4 82 8.772e-05 8.772e-05 8.772e-05 0.01% MLMG::mgVcycle_down::3 82 8.621e-05 8.621e-05 8.621e-05 0.01% MLMG::solve() 11 8.282e-05 8.282e-05 8.282e-05 0.01% Castro::initialize_advance() 10 7.985e-05 7.985e-05 7.985e-05 0.01% Castro::clean_state() 62 7.797e-05 7.797e-05 7.797e-05 0.01% AmrLevel::checkPoint() 3 7.111e-05 7.111e-05 7.111e-05 0.01% MLMG::mgVcycle_up::4 82 6.708e-05 6.708e-05 6.708e-05 0.01% Castro::initialize_do_advance() 10 6.248e-05 6.248e-05 6.248e-05 0.01% MLMG::mgVcycle_up::0 82 5.974e-05 5.974e-05 5.974e-05 0.01% Castro::finalize_advance() 10 5.916e-05 5.916e-05 5.916e-05 0.01% MLMG::mgVcycle_up::1 82 5.861e-05 5.861e-05 5.861e-05 0.01% MLMG::mgVcycle_up::3 82 5.748e-05 5.748e-05 5.748e-05 0.01% MLMG::mgVcycle_up::2 82 5.688e-05 5.688e-05 5.688e-05 0.01% MLCellLinOp::solutionResidual() 93 5.329e-05 5.329e-05 5.329e-05 0.01% StateData::define() 4 4.15e-05 4.15e-05 4.15e-05 0.00% Castro::construct_new_source() 50 3.987e-05 3.987e-05 3.987e-05 0.00% Castro::swap_state_time_levels() 10 3.84e-05 3.84e-05 3.84e-05 0.00% Castro::finalize_do_advance() 10 3.796e-05 3.796e-05 3.796e-05 0.00% MLMG::mgVcycle_bottom 82 3.612e-05 3.612e-05 3.612e-05 0.00% Castro::enforce_consistent_e() 1 3.46e-05 3.46e-05 3.46e-05 0.00% MLMG::computeResidual() 82 3.355e-05 3.355e-05 3.355e-05 0.00% FillPatchSingleLevel 41 2.883e-05 2.883e-05 2.883e-05 0.00% MLLinOp::define() 11 2.75e-05 2.75e-05 2.75e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.721e-05 2.721e-05 2.721e-05 0.00% Amr::writeSmallPlotFile() 1 2.637e-05 2.637e-05 2.637e-05 0.00% MLPoisson::define() 11 2.52e-05 2.52e-05 2.52e-05 0.00% makeSFC 55 2.481e-05 2.481e-05 2.481e-05 0.00% Amr::FinalizeInit() 1 2.208e-05 2.208e-05 2.208e-05 0.00% Castro::construct_old_source() 50 2.033e-05 2.033e-05 2.033e-05 0.00% Castro::do_new_sources() 10 1.716e-05 1.716e-05 1.716e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.65e-05 1.65e-05 1.65e-05 0.00% Castro::do_old_sources() 10 1.632e-05 1.632e-05 1.632e-05 0.00% DistributionMapping::Distribute() 56 1.584e-05 1.584e-05 1.584e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.572e-05 1.572e-05 1.572e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.504e-05 1.504e-05 1.504e-05 0.00% Castro::apply_source_to_state() 20 1.117e-05 1.117e-05 1.117e-05 0.00% Castro::construct_old_gravity() 10 1.088e-05 1.088e-05 1.088e-05 0.00% Gravity::swapTimeLevels() 10 9.687e-06 9.687e-06 9.687e-06 0.00% Amr::initSubcycle() 1 9.357e-06 9.357e-06 9.357e-06 0.00% MLPoisson::prepareForSolve() 11 7.836e-06 7.836e-06 7.836e-06 0.00% Castro::post_timestep() 10 7.564e-06 7.564e-06 7.564e-06 0.00% MLMG::computeMLResidual() 11 6.85e-06 6.85e-06 6.85e-06 0.00% MLMG::getGradSolution() 11 5.897e-06 5.897e-06 5.897e-06 0.00% Castro::create_source_corrector() 10 5.839e-06 5.839e-06 5.839e-06 0.00% Castro::computeNewDt() 9 5.821e-06 5.821e-06 5.821e-06 0.00% MLMG::buildFineMask() 11 5.77e-06 5.77e-06 5.77e-06 0.00% Amr::InitializeInit() 1 5.639e-06 5.639e-06 5.639e-06 0.00% AmrLevel::checkPointPost() 3 5.404e-06 5.404e-06 5.404e-06 0.00% MLMG::MLResNormInf() 11 5.007e-06 5.007e-06 5.007e-06 0.00% Castro::post_init() 1 4.73e-06 4.73e-06 4.73e-06 0.00% Castro::retry_advance_ctu() 10 4.41e-06 4.41e-06 4.41e-06 0.00% Gravity::set_mass_offset() 11 4.122e-06 4.122e-06 4.122e-06 0.00% Castro::FluxRegFineAdd() 10 3.52e-06 3.52e-06 3.52e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.875e-06 2.875e-06 2.875e-06 0.00% Amr::init() 1 2.364e-06 2.364e-06 2.364e-06 0.00% AmrLevel::checkPointPre() 3 2.183e-06 2.183e-06 2.183e-06 0.00% Castro::computeInitialDt() 2 2.129e-06 2.129e-06 2.129e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.988e-06 1.988e-06 1.988e-06 0.00% Castro::post_regrid() 1 1.273e-06 1.273e-06 1.273e-06 0.00% Amr::initialInit() 1 1.029e-06 1.029e-06 1.029e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8332 0.8332 0.8332 100.00% Amr::coarseTimeStep() 10 0.6881 0.6881 0.6881 82.58% Amr::timeStep() 10 0.6142 0.6142 0.6142 73.72% Castro::advance() 10 0.6083 0.6083 0.6083 73.01% Castro::subcycle_advance_ctu() 10 0.5941 0.5941 0.5941 71.30% Castro::do_advance_ctu() 10 0.5939 0.5939 0.5939 71.28% Gravity::solve_phi_with_mlmg() 11 0.3239 0.3239 0.3239 38.88% Gravity::actual_solve_with_mlmg() 11 0.3117 0.3117 0.3117 37.41% Castro::construct_new_gravity() 10 0.2943 0.2943 0.2943 35.32% MLMG::solve() 11 0.2893 0.2893 0.2893 34.73% Gravity::solve_for_phi() 10 0.2761 0.2761 0.2761 33.14% MLMG::oneIter() 82 0.2735 0.2735 0.2735 32.82% MLMG::mgVcycle() 82 0.2718 0.2718 0.2718 32.62% Castro::construct_ctu_hydro_source() 10 0.215 0.215 0.215 25.80% VisMF::Write(FabArray) 11 0.1461 0.1461 0.1461 17.54% MLCellLinOp::smooth() 1640 0.1446 0.1446 0.1446 17.35% Amr::init() 1 0.1221 0.1221 0.1221 14.65% MLCellLinOp::applyBC() 4433 0.1127 0.1127 0.1127 13.53% Amr::checkPoint() 3 0.1088 0.1088 0.1088 13.06% AmrLevel::checkPoint() 3 0.1047 0.1047 0.1047 12.56% StateData::checkPoint() 12 0.1046 0.1046 0.1046 12.55% MLMG::mgVcycle_bottom 82 0.07679 0.07679 0.07679 9.22% MLMG::actualBottomSolve() 82 0.07675 0.07675 0.07675 9.21% MLCGSolver::bicgstab 82 0.07602 0.07602 0.07602 9.12% Amr::initialInit() 1 0.06148 0.06148 0.06148 7.38% MLPoisson::Fsmooth() 3280 0.05953 0.05953 0.05953 7.15% FillPatchIterator::Initialize 41 0.05716 0.05716 0.05716 6.86% Amr::FinalizeInit() 1 0.05181 0.05181 0.05181 6.22% Castro::post_init() 1 0.05106 0.05106 0.05106 6.13% FillPatchSingleLevel 41 0.05097 0.05097 0.05097 6.12% Gravity::multilevel_solve_for_new_phi() 1 0.04849 0.04849 0.04849 5.82% Gravity::actual_multilevel_solve() 1 0.04847 0.04847 0.04847 5.82% StateDataPhysBCFunct::() 41 0.04667 0.04667 0.04667 5.60% Amr::writePlotFile() 2 0.04495 0.04495 0.04495 5.40% MLCellLinOp::apply() 1142 0.03939 0.03939 0.03939 4.73% MLMG::mgVcycle_down::0 82 0.03848 0.03848 0.03848 4.62% Castro::clean_state() 62 0.03347 0.03347 0.03347 4.02% MLMG::mgVcycle_up::0 82 0.03299 0.03299 0.03299 3.96% FabArray::setVal() 1144 0.02662 0.02662 0.02662 3.20% Castro::initialize_do_advance() 10 0.02498 0.02498 0.02498 3.00% StateData::FillBoundary(geom) 328 0.02308 0.02308 0.02308 2.77% MLCellLinOp::correctionResidual() 492 0.02216 0.02216 0.02216 2.66% MultiFab::Dot() 1114 0.02109 0.02109 0.02109 2.53% Gravity::get_new_grav_vector() 11 0.02032 0.02032 0.02032 2.44% FabArray::FillBoundary() 4023 0.01947 0.01947 0.01947 2.34% Castro::expand_state() 10 0.01921 0.01921 0.01921 2.31% MLMG:computeResOfCorrection() 410 0.0192 0.0192 0.0192 2.30% FillBoundary_nowait() 4023 0.01862 0.01862 0.01862 2.23% MLMG::mgVcycle_down::1 82 0.01842 0.01842 0.01842 2.21% Castro::computeTemp() 63 0.01828 0.01828 0.01828 2.19% MLMG::mgVcycle_down::2 82 0.0178 0.0178 0.0178 2.14% FabArray::ParallelCopy() 861 0.01718 0.01718 0.01718 2.06% MLMG::mgVcycle_down::3 82 0.01699 0.01699 0.01699 2.04% MLPoisson::define() 11 0.01697 0.01697 0.01697 2.04% FabArray::ParallelCopy_nowait() 861 0.01687 0.01687 0.01687 2.02% MLMG::mgVcycle_down::4 82 0.01618 0.01618 0.01618 1.94% Castro::construct_old_gravity() 10 0.0161 0.0161 0.0161 1.93% Gravity::get_old_grav_vector() 10 0.01609 0.01609 0.01609 1.93% Castro::initialize_advance() 10 0.01401 0.01401 0.01401 1.68% MLMG::mgVcycle_up::2 82 0.01388 0.01388 0.01388 1.67% MLMG::mgVcycle_up::1 82 0.01365 0.01365 0.01365 1.64% MLMG::addInterpCorrection() 410 0.01345 0.01345 0.01345 1.61% MLMG::mgVcycle_up::3 82 0.01318 0.01318 0.01318 1.58% MLMG::mgVcycle_up::4 82 0.01302 0.01302 0.01302 1.56% amrex::average_down 410 0.01296 0.01296 0.01296 1.56% MLCGSolver::ParallelAllReduce 1514 0.01271 0.01271 0.01271 1.53% CGSolver::sxay() 1586 0.01254 0.01254 0.01254 1.51% MultiFab::LinComb() 1586 0.01207 0.01207 0.01207 1.45% Gravity::fill_multipole_BCs() 11 0.01196 0.01196 0.01196 1.44% MLCellLinOp::defineAuxData() 11 0.01165 0.01165 0.01165 1.40% Castro::do_new_sources() 10 0.0112 0.0112 0.0112 1.34% MLPoisson::Fapply() 1142 0.01038 0.01038 0.01038 1.25% Amr::InitializeInit() 1 0.009663 0.009663 0.009663 1.16% Amr::defBaseLevel() 1 0.009658 0.009658 0.009658 1.16% Castro::enforce_min_density() 62 0.008434 0.008434 0.008434 1.01% Castro::do_old_sources() 10 0.008235 0.008235 0.008235 0.99% MLCellLinOp::solutionResidual() 93 0.007841 0.007841 0.007841 0.94% MLMG::computeResidual() 82 0.006784 0.006784 0.006784 0.81% MultiFab::Xpay() 585 0.005982 0.005982 0.005982 0.72% FabArray::setDomainBndry() 41 0.005931 0.005931 0.005931 0.71% MLMG::prepareForSolve() 11 0.005774 0.005774 0.005774 0.69% Castro::post_timestep() 10 0.005716 0.005716 0.005716 0.69% Castro::reset_internal_energy() 63 0.005543 0.005543 0.005543 0.67% MLCellLinOp::defineBC() 11 0.005034 0.005034 0.005034 0.60% Castro::estTimeStep() 21 0.005006 0.005006 0.005006 0.60% BndryData::define() 11 0.004847 0.004847 0.004847 0.58% Castro::normalize_species() 62 0.004409 0.004409 0.004409 0.53% Castro::initData() 1 0.003615 0.003615 0.003615 0.43% Castro::construct_new_source() 50 0.003165 0.003165 0.003165 0.38% Castro::construct_new_gravity_source() 10 0.003125 0.003125 0.003125 0.38% Castro::enforce_speed_limit() 62 0.003089 0.003089 0.003089 0.37% MLMG::getGradSolution() 11 0.002523 0.002523 0.002523 0.30% MLCellLinOp::compGrad() 11 0.002517 0.002517 0.002517 0.30% Castro::construct_old_source() 50 0.002499 0.002499 0.002499 0.30% Castro::construct_old_gravity_source() 10 0.002479 0.002479 0.002479 0.30% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002236 0.002236 0.002236 0.27% Castro::computeNewDt() 9 0.002224 0.002224 0.002224 0.27% MLMG::ResNormInf() 93 0.001932 0.001932 0.001932 0.23% Castro::apply_source_to_state() 20 0.001822 0.001822 0.001822 0.22% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.22% FabArrayBase::getCPC() 1323 0.001546 0.001546 0.001546 0.19% MLCellLinOp::setLevelBC() 11 0.001524 0.001524 0.001524 0.18% MLPoisson::prepareForSolve() 11 0.001337 0.001337 0.001337 0.16% MLCellLinOp::prepareForSolve() 11 0.001329 0.001329 0.001329 0.16% FabArray::mult() 43 0.001313 0.001313 0.001313 0.16% MLMG::computeMLResidual() 11 0.001097 0.001097 0.001097 0.13% Gravity::swapTimeLevels() 10 0.001049 0.001049 0.001049 0.13% Gravity::update_max_rhs() 11 0.0009569 0.0009569 0.0009569 0.11% FabArrayBase::getFB() 4023 0.0007522 0.0007522 0.0007522 0.09% FabArrayBase::CPC::define() 454 0.0007203 0.0007203 0.0007203 0.09% Castro::computeInitialDt() 2 0.0006719 0.0006719 0.0006719 0.08% MultiFab::Copy() 11 0.0005209 0.0005209 0.0005209 0.06% Castro::post_regrid() 1 0.0005134 0.0005134 0.0005134 0.06% Amr::InitAmr() 1 0.000441 0.000441 0.000441 0.05% MultiFab::min() 10 0.0003207 0.0003207 0.0003207 0.04% MLLinOp::define() 11 0.0002573 0.0002573 0.0002573 0.03% MultiFab::max() 11 0.0002555 0.0002555 0.0002555 0.03% MLMG::MLResNormInf() 11 0.0002508 0.0002508 0.0002508 0.03% MLLinOp::defineGrids() 11 0.0002298 0.0002298 0.0002298 0.03% MLMG::MLRhsNormInf() 11 0.0001972 0.0001972 0.0001972 0.02% FabArrayBase::FB::FB() 56 8.93e-05 8.93e-05 8.93e-05 0.01% Castro::finalize_advance() 10 6.268e-05 6.268e-05 6.268e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.655e-05 5.655e-05 5.655e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.491e-05 5.491e-05 5.491e-05 0.01% StateData::define() 4 4.15e-05 4.15e-05 4.15e-05 0.00% makeSFC 55 3.92e-05 3.92e-05 3.92e-05 0.00% Castro::swap_state_time_levels() 10 3.84e-05 3.84e-05 3.84e-05 0.00% Castro::finalize_do_advance() 10 3.796e-05 3.796e-05 3.796e-05 0.00% Castro::enforce_consistent_e() 1 3.46e-05 3.46e-05 3.46e-05 0.00% Amr::writeSmallPlotFile() 1 2.637e-05 2.637e-05 2.637e-05 0.00% DistributionMapping::Distribute() 56 1.584e-05 1.584e-05 1.584e-05 0.00% Amr::initSubcycle() 1 9.357e-06 9.357e-06 9.357e-06 0.00% Castro::create_source_corrector() 10 5.839e-06 5.839e-06 5.839e-06 0.00% MLMG::buildFineMask() 11 5.77e-06 5.77e-06 5.77e-06 0.00% AmrLevel::checkPointPost() 3 5.404e-06 5.404e-06 5.404e-06 0.00% Castro::retry_advance_ctu() 10 4.41e-06 4.41e-06 4.41e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.325e-06 4.325e-06 4.325e-06 0.00% Gravity::set_mass_offset() 11 4.122e-06 4.122e-06 4.122e-06 0.00% Castro::FluxRegFineAdd() 10 3.52e-06 3.52e-06 3.52e-06 0.00% AmrLevel::checkPointPre() 3 2.183e-06 2.183e-06 2.183e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.988e-06 1.988e-06 1.988e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-9-g84a1f7d0715b) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-9-g84a1f7d0715b) initialized Starting run at 08:25:42 UTC on 2022-04-07. Successfully read inputs file ... Castro git describe: 22.04-2-gb74783602 AMReX git describe: 22.04-9-g84a1f7d07 Microphysics git describe: 22.04-1-gef526ccd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.491655463 Restart time = 0.05516671 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.11238009 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050494108 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.05671532 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057685262 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.064193443 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.023336615 seconds Ending run at 08:25:42 UTC on 2022-04-07. Run time = 0.420892876 Run time without initialization = 0.365153345 Average number of zones advanced per microsecond: 3.590 Average number of zones advanced per microsecond per rank: 3.590 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4209 ... 0.4209 ... 0.4209 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1368 0.1368 0.1368 32.50% VisMF::Read() 3 0.04346 0.04346 0.04346 10.33% MLCellLinOp::applyBC() 1946 0.03759 0.03759 0.03759 8.93% MLPoisson::Fsmooth() 1440 0.02537 0.02537 0.02537 6.03% VisMF::Write(FabArray) 1 0.02199 0.02199 0.02199 5.22% FabArray::setVal() 537 0.01621 0.01621 0.01621 3.85% StateData::FillBoundary(geom) 160 0.01147 0.01147 0.01147 2.73% MLCGSolver::bicgstab 36 0.009291 0.009291 0.009291 2.21% MultiFab::Dot() 484 0.0091 0.0091 0.0091 2.16% FillBoundary_nowait() 1766 0.007129 0.007129 0.007129 1.69% Castro::computeTemp() 30 0.007074 0.007074 0.007074 1.68% FabArray::ParallelCopy_nowait() 380 0.007073 0.007073 0.007073 1.68% StateDataPhysBCFunct::() 20 0.006891 0.006891 0.006891 1.64% Gravity::fill_multipole_BCs() 6 0.006308 0.006308 0.006308 1.50% MLCellLinOp::defineAuxData() 6 0.00549 0.00549 0.00549 1.30% MultiFab::LinComb() 690 0.005243 0.005243 0.005243 1.25% Castro::enforce_min_density() 30 0.004658 0.004658 0.004658 1.11% MLPoisson::Fapply() 500 0.004558 0.004558 0.004558 1.08% Castro::expand_state() 5 0.003937 0.003937 0.003937 0.94% FabArray::setDomainBndry() 20 0.003658 0.003658 0.003658 0.87% MLMG::addInterpCorrection() 180 0.00297 0.00297 0.00297 0.71% Amr::restart() 1 0.002945 0.002945 0.002945 0.70% amrex::average_down 180 0.002772 0.002772 0.002772 0.66% MultiFab::Xpay() 258 0.002639 0.002639 0.002639 0.63% Castro::estTimeStep() 10 0.002619 0.002619 0.002619 0.62% Castro::do_advance_ctu() 5 0.002575 0.002575 0.002575 0.61% Castro::reset_internal_energy() 30 0.002419 0.002419 0.002419 0.57% Castro::normalize_species() 30 0.002197 0.002197 0.002197 0.52% BndryData::define() 6 0.001984 0.001984 0.001984 0.47% Gravity::get_new_grav_vector() 5 0.00172 0.00172 0.00172 0.41% Castro::construct_new_gravity_source() 5 0.001574 0.001574 0.001574 0.37% MLCellLinOp::compGrad() 6 0.001469 0.001469 0.001469 0.35% Amr::writePlotFile() 1 0.001434 0.001434 0.001434 0.34% Castro::construct_old_gravity_source() 5 0.001341 0.001341 0.001341 0.32% Castro::enforce_speed_limit() 30 0.001227 0.001227 0.001227 0.29% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001211 0.001211 0.001211 0.29% MultiFab::Saxpy() 10 0.0009185 0.0009185 0.0009185 0.22% Gravity::get_old_grav_vector() 5 0.0009019 0.0009019 0.0009019 0.21% MLMG::ResNormInf() 42 0.0008545 0.0008545 0.0008545 0.20% MLCellLinOp::setLevelBC() 6 0.0008176 0.0008176 0.0008176 0.19% MLMG::oneIter() 36 0.0007464 0.0007464 0.0007464 0.18% Gravity::actual_solve_with_mlmg() 6 0.0007091 0.0007091 0.0007091 0.17% MLCellLinOp::prepareForSolve() 6 0.0006848 0.0006848 0.0006848 0.16% FabArray::mult() 22 0.0006592 0.0006592 0.0006592 0.16% MLCellLinOp::smooth() 720 0.0005604 0.0005604 0.0005604 0.13% Gravity::update_max_rhs() 6 0.0005156 0.0005156 0.0005156 0.12% MLMG::prepareForSolve() 6 0.0004337 0.0004337 0.0004337 0.10% FabArrayBase::CPC::define() 244 0.0004276 0.0004276 0.0004276 0.10% FabArrayBase::getCPC() 632 0.0004235 0.0004235 0.0004235 0.10% FabArray::FillBoundary() 1766 0.000403 0.000403 0.000403 0.10% Amr::InitAmr() 1 0.0003766 0.0003766 0.0003766 0.09% MultiFab::Copy() 6 0.0003718 0.0003718 0.0003718 0.09% FabArrayBase::getFB() 1766 0.0003134 0.0003134 0.0003134 0.07% main() 1 0.0002773 0.0002773 0.0002773 0.07% Gravity::actual_multilevel_solve() 1 0.0002323 0.0002323 0.0002323 0.06% MLCellLinOp::apply() 500 0.0002211 0.0002211 0.0002211 0.05% Gravity::solve_for_phi() 5 0.0002156 0.0002156 0.0002156 0.05% CGSolver::sxay() 690 0.0002127 0.0002127 0.0002127 0.05% MultiFab::min() 5 0.0001815 0.0001815 0.0001815 0.04% Castro::construct_new_gravity() 5 0.0001764 0.0001764 0.0001764 0.04% MLMG::mgVcycle() 36 0.0001754 0.0001754 0.0001754 0.04% Castro::construct_new_source() 25 0.0001645 0.0001645 0.0001645 0.04% FabArray::ParallelCopy() 380 0.0001488 0.0001488 0.0001488 0.04% MLCGSolver::ParallelAllReduce 659 0.0001424 0.0001424 0.0001424 0.03% MultiFab::max() 6 0.0001333 0.0001333 0.0001333 0.03% FillPatchIterator::Initialize 20 0.0001215 0.0001215 0.0001215 0.03% Amr::coarseTimeStep() 5 0.000118 0.000118 0.000118 0.03% Amr::timeStep() 5 0.0001074 0.0001074 0.0001074 0.03% MLMG::MLRhsNormInf() 6 0.0001054 0.0001054 0.0001054 0.03% MLCellLinOp::defineBC() 6 0.0001 0.0001 0.0001 0.02% MLCellLinOp::correctionResidual() 216 9.335e-05 9.335e-05 9.335e-05 0.02% MLLinOp::defineGrids() 6 9.267e-05 9.267e-05 9.267e-05 0.02% AmrLevel::restart() 1 7.555e-05 7.555e-05 7.555e-05 0.02% StateData::restartDoit() 4 7.121e-05 7.121e-05 7.121e-05 0.02% Castro::subcycle_advance_ctu() 5 6.844e-05 6.844e-05 6.844e-05 0.02% Castro::post_timestep() 5 6.326e-05 6.326e-05 6.326e-05 0.02% FabArrayBase::FB::FB() 26 6.03e-05 6.03e-05 6.03e-05 0.01% MLMG:computeResOfCorrection() 180 5.359e-05 5.359e-05 5.359e-05 0.01% MLMG::actualBottomSolve() 36 4.938e-05 4.938e-05 4.938e-05 0.01% Castro::create_source_corrector() 5 4.152e-05 4.152e-05 4.152e-05 0.01% MLMG::mgVcycle_down::0 36 3.88e-05 3.88e-05 3.88e-05 0.01% Castro::do_new_sources() 5 3.787e-05 3.787e-05 3.787e-05 0.01% MLMG::mgVcycle_down::1 36 3.701e-05 3.701e-05 3.701e-05 0.01% Castro::clean_state() 30 3.6e-05 3.6e-05 3.6e-05 0.01% Castro::construct_old_source() 25 3.552e-05 3.552e-05 3.552e-05 0.01% Castro::initialize_advance() 5 3.532e-05 3.532e-05 3.532e-05 0.01% MLMG::solve() 6 3.53e-05 3.53e-05 3.53e-05 0.01% MLMG::mgVcycle_down::2 36 3.474e-05 3.474e-05 3.474e-05 0.01% MLMG::mgVcycle_down::4 36 3.337e-05 3.337e-05 3.337e-05 0.01% Castro::advance() 5 3.303e-05 3.303e-05 3.303e-05 0.01% MLMG::mgVcycle_down::3 36 3.252e-05 3.252e-05 3.252e-05 0.01% Castro::initialize_do_advance() 5 2.769e-05 2.769e-05 2.769e-05 0.01% MLLinOp::define() 6 2.754e-05 2.754e-05 2.754e-05 0.01% MLMG::mgVcycle_up::4 36 2.668e-05 2.668e-05 2.668e-05 0.01% Castro::finalize_advance() 5 2.576e-05 2.576e-05 2.576e-05 0.01% Amr::writeSmallPlotFile() 1 2.571e-05 2.571e-05 2.571e-05 0.01% Castro::swap_state_time_levels() 5 2.568e-05 2.568e-05 2.568e-05 0.01% Castro::post_restart() 1 2.529e-05 2.529e-05 2.529e-05 0.01% MLCellLinOp::solutionResidual() 42 2.407e-05 2.407e-05 2.407e-05 0.01% MLMG::mgVcycle_up::3 36 2.239e-05 2.239e-05 2.239e-05 0.01% MLMG::mgVcycle_up::0 36 2.146e-05 2.146e-05 2.146e-05 0.01% MLMG::mgVcycle_up::2 36 2.143e-05 2.143e-05 2.143e-05 0.01% MLMG::mgVcycle_up::1 36 2.066e-05 2.066e-05 2.066e-05 0.00% Castro::finalize_do_advance() 5 1.935e-05 1.935e-05 1.935e-05 0.00% MLMG::mgVcycle_bottom 36 1.626e-05 1.626e-05 1.626e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.548e-05 1.548e-05 1.548e-05 0.00% MLPoisson::define() 6 1.525e-05 1.525e-05 1.525e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.486e-05 1.486e-05 1.486e-05 0.00% MLMG::computeResidual() 36 1.478e-05 1.478e-05 1.478e-05 0.00% FillPatchSingleLevel 20 1.45e-05 1.45e-05 1.45e-05 0.00% makeSFC 30 1.401e-05 1.401e-05 1.401e-05 0.00% DistributionMapping::Distribute() 31 9.859e-06 9.859e-06 9.859e-06 0.00% Amr::initSubcycle() 1 9.546e-06 9.546e-06 9.546e-06 0.00% Castro::do_old_sources() 5 8.262e-06 8.262e-06 8.262e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.968e-06 7.968e-06 7.968e-06 0.00% Castro::apply_source_to_state() 10 6.168e-06 6.168e-06 6.168e-06 0.00% Castro::construct_old_gravity() 5 5.527e-06 5.527e-06 5.527e-06 0.00% MLPoisson::prepareForSolve() 6 5.078e-06 5.078e-06 5.078e-06 0.00% Gravity::swapTimeLevels() 5 4.712e-06 4.712e-06 4.712e-06 0.00% MLMG::buildFineMask() 6 3.368e-06 3.368e-06 3.368e-06 0.00% Castro::computeNewDt() 5 3.262e-06 3.262e-06 3.262e-06 0.00% MLMG::getGradSolution() 6 3.177e-06 3.177e-06 3.177e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.172e-06 3.172e-06 3.172e-06 0.00% MLMG::computeMLResidual() 6 3.154e-06 3.154e-06 3.154e-06 0.00% MLMG::MLResNormInf() 6 2.636e-06 2.636e-06 2.636e-06 0.00% Gravity::set_mass_offset() 6 2.451e-06 2.451e-06 2.451e-06 0.00% Castro::retry_advance_ctu() 5 1.806e-06 1.806e-06 1.806e-06 0.00% Castro::FluxRegFineAdd() 5 1.488e-06 1.488e-06 1.488e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.13e-06 1.13e-06 1.13e-06 0.00% Amr::init() 1 1.072e-06 1.072e-06 1.072e-06 0.00% AmrLevel::AmrLevel() 1 7.62e-07 7.62e-07 7.62e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4209 0.4209 0.4209 100.00% Amr::coarseTimeStep() 5 0.3416 0.3416 0.3416 81.15% Amr::timeStep() 5 0.3402 0.3402 0.3402 80.82% Castro::advance() 5 0.3366 0.3366 0.3366 79.97% Castro::subcycle_advance_ctu() 5 0.3256 0.3256 0.3256 77.36% Castro::do_advance_ctu() 5 0.3255 0.3255 0.3255 77.34% Castro::construct_new_gravity() 5 0.1452 0.1452 0.1452 34.49% Gravity::solve_phi_with_mlmg() 6 0.1415 0.1415 0.1415 33.62% Castro::construct_ctu_hydro_source() 5 0.1368 0.1368 0.1368 32.50% Gravity::solve_for_phi() 5 0.1357 0.1357 0.1357 32.25% Gravity::actual_solve_with_mlmg() 6 0.1351 0.1351 0.1351 32.09% MLMG::solve() 6 0.1224 0.1224 0.1224 29.08% MLMG::oneIter() 36 0.1146 0.1146 0.1146 27.22% MLMG::mgVcycle() 36 0.1138 0.1138 0.1138 27.04% MLCellLinOp::smooth() 720 0.05953 0.05953 0.05953 14.14% Amr::init() 1 0.05522 0.05522 0.05522 13.12% Amr::restart() 1 0.05522 0.05522 0.05522 13.12% MLCellLinOp::applyBC() 1946 0.0455 0.0455 0.0455 10.81% AmrLevel::restart() 1 0.04437 0.04437 0.04437 10.54% StateData::restartDoit() 4 0.04429 0.04429 0.04429 10.52% VisMF::Read() 3 0.04346 0.04346 0.04346 10.33% MLMG::mgVcycle_bottom 36 0.03289 0.03289 0.03289 7.81% MLMG::actualBottomSolve() 36 0.03288 0.03288 0.03288 7.81% MLCGSolver::bicgstab 36 0.03256 0.03256 0.03256 7.74% MLPoisson::Fsmooth() 1440 0.02537 0.02537 0.02537 6.03% FillPatchIterator::Initialize 20 0.02422 0.02422 0.02422 5.75% Amr::writePlotFile() 1 0.02342 0.02342 0.02342 5.56% VisMF::Write(FabArray) 1 0.02199 0.02199 0.02199 5.22% FillPatchSingleLevel 20 0.02044 0.02044 0.02044 4.86% StateDataPhysBCFunct::() 20 0.01837 0.01837 0.01837 4.36% Castro::clean_state() 30 0.01761 0.01761 0.01761 4.18% MLCellLinOp::apply() 500 0.01639 0.01639 0.01639 3.89% FabArray::setVal() 537 0.01621 0.01621 0.01621 3.85% MLMG::mgVcycle_down::0 36 0.01585 0.01585 0.01585 3.77% Castro::initialize_do_advance() 5 0.01463 0.01463 0.01463 3.47% MLMG::mgVcycle_up::0 36 0.01345 0.01345 0.01345 3.19% Castro::expand_state() 5 0.01168 0.01168 0.01168 2.77% StateData::FillBoundary(geom) 160 0.01147 0.01147 0.01147 2.73% Castro::initialize_advance() 5 0.01094 0.01094 0.01094 2.60% Castro::computeTemp() 30 0.009493 0.009493 0.009493 2.26% MLPoisson::define() 6 0.00938 0.00938 0.00938 2.23% Gravity::get_new_grav_vector() 5 0.009274 0.009274 0.009274 2.20% MLCellLinOp::correctionResidual() 216 0.009251 0.009251 0.009251 2.20% MultiFab::Dot() 484 0.0091 0.0091 0.0091 2.16% MLMG:computeResOfCorrection() 180 0.008004 0.008004 0.008004 1.90% FabArray::FillBoundary() 1766 0.007906 0.007906 0.007906 1.88% MLMG::mgVcycle_down::1 36 0.00774 0.00774 0.00774 1.84% Castro::construct_old_gravity() 5 0.007667 0.007667 0.007667 1.82% Gravity::get_old_grav_vector() 5 0.007661 0.007661 0.007661 1.82% FabArray::ParallelCopy() 380 0.007644 0.007644 0.007644 1.82% FillBoundary_nowait() 1766 0.007503 0.007503 0.007503 1.78% FabArray::ParallelCopy_nowait() 380 0.007495 0.007495 0.007495 1.78% MLMG::mgVcycle_down::2 36 0.007432 0.007432 0.007432 1.77% MLMG::mgVcycle_down::3 36 0.007076 0.007076 0.007076 1.68% MLMG::mgVcycle_down::4 36 0.006787 0.006787 0.006787 1.61% Castro::do_new_sources() 5 0.00672 0.00672 0.00672 1.60% Castro::post_restart() 1 0.006614 0.006614 0.006614 1.57% MLCellLinOp::defineAuxData() 6 0.006418 0.006418 0.006418 1.52% Gravity::fill_multipole_BCs() 6 0.006308 0.006308 0.006308 1.50% Gravity::multilevel_solve_for_new_phi() 1 0.006232 0.006232 0.006232 1.48% Gravity::actual_multilevel_solve() 1 0.006217 0.006217 0.006217 1.48% MLMG::mgVcycle_up::2 36 0.005772 0.005772 0.005772 1.37% MLMG::addInterpCorrection() 180 0.005768 0.005768 0.005768 1.37% MLMG::mgVcycle_up::1 36 0.005705 0.005705 0.005705 1.36% amrex::average_down 180 0.005558 0.005558 0.005558 1.32% MLCGSolver::ParallelAllReduce 659 0.00552 0.00552 0.00552 1.31% MLMG::mgVcycle_up::3 36 0.005505 0.005505 0.005505 1.31% CGSolver::sxay() 690 0.005456 0.005456 0.005456 1.30% MLMG::mgVcycle_up::4 36 0.005439 0.005439 0.005439 1.29% MultiFab::LinComb() 690 0.005243 0.005243 0.005243 1.25% Castro::enforce_min_density() 30 0.004658 0.004658 0.004658 1.11% MLPoisson::Fapply() 500 0.004558 0.004558 0.004558 1.08% Castro::do_old_sources() 5 0.004293 0.004293 0.004293 1.02% FabArray::setDomainBndry() 20 0.003658 0.003658 0.003658 0.87% Castro::post_timestep() 5 0.003464 0.003464 0.003464 0.82% MLMG::prepareForSolve() 6 0.003434 0.003434 0.003434 0.82% MLCellLinOp::solutionResidual() 42 0.003368 0.003368 0.003368 0.80% MLCellLinOp::defineBC() 6 0.002795 0.002795 0.002795 0.66% MLMG::computeResidual() 36 0.002795 0.002795 0.002795 0.66% BndryData::define() 6 0.002695 0.002695 0.002695 0.64% MultiFab::Xpay() 258 0.002639 0.002639 0.002639 0.63% Castro::estTimeStep() 10 0.002619 0.002619 0.002619 0.62% Castro::reset_internal_energy() 30 0.002419 0.002419 0.002419 0.57% Castro::normalize_species() 30 0.002197 0.002197 0.002197 0.52% MLMG::getGradSolution() 6 0.001761 0.001761 0.001761 0.42% MLCellLinOp::compGrad() 6 0.001758 0.001758 0.001758 0.42% Castro::construct_new_source() 25 0.001739 0.001739 0.001739 0.41% Castro::construct_new_gravity_source() 5 0.001574 0.001574 0.001574 0.37% Castro::construct_old_source() 25 0.001376 0.001376 0.001376 0.33% Castro::construct_old_gravity_source() 5 0.001341 0.001341 0.001341 0.32% Castro::computeNewDt() 5 0.001277 0.001277 0.001277 0.30% Castro::enforce_speed_limit() 30 0.001227 0.001227 0.001227 0.29% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001211 0.001211 0.001211 0.29% Castro::apply_source_to_state() 10 0.0009247 0.0009247 0.0009247 0.22% MultiFab::Saxpy() 10 0.0009185 0.0009185 0.0009185 0.22% Gravity::swapTimeLevels() 5 0.0008827 0.0008827 0.0008827 0.21% MLMG::ResNormInf() 42 0.0008545 0.0008545 0.0008545 0.20% FabArrayBase::getCPC() 632 0.000851 0.000851 0.000851 0.20% MLCellLinOp::setLevelBC() 6 0.0008176 0.0008176 0.0008176 0.19% Gravity::update_max_rhs() 6 0.0007262 0.0007262 0.0007262 0.17% MLPoisson::prepareForSolve() 6 0.0006899 0.0006899 0.0006899 0.16% MLCellLinOp::prepareForSolve() 6 0.0006848 0.0006848 0.0006848 0.16% FabArray::mult() 22 0.0006592 0.0006592 0.0006592 0.16% MLMG::computeMLResidual() 6 0.000591 0.000591 0.000591 0.14% FabArrayBase::CPC::define() 244 0.0004276 0.0004276 0.0004276 0.10% Amr::InitAmr() 1 0.0003861 0.0003861 0.0003861 0.09% FabArrayBase::getFB() 1766 0.0003737 0.0003737 0.0003737 0.09% MultiFab::Copy() 6 0.0003718 0.0003718 0.0003718 0.09% MultiFab::min() 5 0.0001815 0.0001815 0.0001815 0.04% MLLinOp::define() 6 0.0001516 0.0001516 0.0001516 0.04% MLMG::MLResNormInf() 6 0.0001335 0.0001335 0.0001335 0.03% MultiFab::max() 6 0.0001333 0.0001333 0.0001333 0.03% MLLinOp::defineGrids() 6 0.000124 0.000124 0.000124 0.03% MLMG::MLRhsNormInf() 6 0.0001054 0.0001054 0.0001054 0.03% FabArrayBase::FB::FB() 26 6.03e-05 6.03e-05 6.03e-05 0.01% Castro::create_source_corrector() 5 4.152e-05 4.152e-05 4.152e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.024e-05 3.024e-05 3.024e-05 0.01% Castro::finalize_advance() 5 2.725e-05 2.725e-05 2.725e-05 0.01% Amr::writeSmallPlotFile() 1 2.571e-05 2.571e-05 2.571e-05 0.01% Castro::swap_state_time_levels() 5 2.568e-05 2.568e-05 2.568e-05 0.01% makeSFC 30 2.227e-05 2.227e-05 2.227e-05 0.01% Castro::finalize_do_advance() 5 1.935e-05 1.935e-05 1.935e-05 0.00% DistributionMapping::Distribute() 31 9.859e-06 9.859e-06 9.859e-06 0.00% Amr::initSubcycle() 1 9.546e-06 9.546e-06 9.546e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.767e-06 4.767e-06 4.767e-06 0.00% MLMG::buildFineMask() 6 3.368e-06 3.368e-06 3.368e-06 0.00% Gravity::set_mass_offset() 6 2.451e-06 2.451e-06 2.451e-06 0.00% Castro::retry_advance_ctu() 5 1.806e-06 1.806e-06 1.806e-06 0.00% Castro::FluxRegFineAdd() 5 1.488e-06 1.488e-06 1.488e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.13e-06 1.13e-06 1.13e-06 0.00% AmrLevel::AmrLevel() 1 7.62e-07 7.62e-07 7.62e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-9-g84a1f7d0715b) finalized