Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-19-gacc223f99182) initialized Starting run at 08:29:47 UTC on 2022-08-31. Successfully read inputs file ... Castro git describe: 22.08-11-ga978fcf88 AMReX git describe: 22.08-19-gacc223f99 Microphysics git describe: 22.08-6-ge3a6cb11 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.053812784 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030905398 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.049330202 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051513918 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.050290286 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.054129093 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.064046818 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.049600661 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.077372786 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.059049623 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.050636613 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.050172965 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.057446859 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.049849392 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031101787 seconds Ending run at 08:29:48 UTC on 2022-08-31. Run time = 0.833342111 Run time without initialization = 0.695169972 Average number of zones advanced per microsecond: 3.771 Average number of zones advanced per microsecond per rank: 3.771 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8334 ... 0.8334 ... 0.8334 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2070 0.2070 0.2070 24.84% Castro::construct_ctu_hydro_source() 10 0.1752 0.1752 0.1752 21.02% MLCellLinOp::applyBC() 4433 0.08225 0.08225 0.08225 9.87% MLPoisson::Fsmooth() 3280 0.06502 0.06502 0.06502 7.80% StateData::FillBoundary(geom) 328 0.02515 0.02515 0.02515 3.02% MLCGSolver::bicgstab 82 0.0246 0.0246 0.0246 2.95% MultiFab::Dot() 1114 0.023 0.023 0.023 2.76% MultiFab::LinComb() 1586 0.01473 0.01473 0.01473 1.77% FabArray::setVal() 1144 0.01456 0.01456 0.01456 1.75% FillBoundary_nowait() 4023 0.01446 0.01446 0.01446 1.73% Castro::computeTemp() 63 0.01432 0.01432 0.01432 1.72% Castro::normalize_species() 62 0.01349 0.01349 0.01349 1.62% FabArray::ParallelCopy_nowait() 861 0.01341 0.01341 0.01341 1.61% StateDataPhysBCFunct::() 41 0.01223 0.01223 0.01223 1.47% MLPoisson::Fapply() 1142 0.012 0.012 0.012 1.44% MLCellLinOp::defineAuxData() 11 0.01187 0.01187 0.01187 1.42% Castro::enforce_min_density() 62 0.008862 0.008862 0.008862 1.06% Gravity::fill_multipole_BCs() 11 0.008366 0.008366 0.008366 1.00% MLMG::addInterpCorrection() 410 0.007826 0.007826 0.007826 0.94% amrex::average_down 410 0.007076 0.007076 0.007076 0.85% MultiFab::Xpay() 585 0.006749 0.006749 0.006749 0.81% Amr::checkPoint() 3 0.00542 0.00542 0.00542 0.65% Castro::estTimeStep() 21 0.005271 0.005271 0.005271 0.63% Castro::do_advance_ctu() 10 0.004301 0.004301 0.004301 0.52% Castro::reset_internal_energy(MultiFab) 63 0.00408 0.00408 0.00408 0.49% BndryData::define() 11 0.003995 0.003995 0.003995 0.48% Amr::writePlotFile() 2 0.002858 0.002858 0.002858 0.34% Castro::construct_new_gravity_source() 10 0.002853 0.002853 0.002853 0.34% MLMG::ResNormInf() 93 0.002205 0.002205 0.002205 0.26% Gravity::get_new_grav_vector() 11 0.001973 0.001973 0.001973 0.24% Castro::construct_old_gravity_source() 10 0.001936 0.001936 0.001936 0.23% MultiFab::Saxpy() 20 0.001815 0.001815 0.001815 0.22% Gravity::get_old_grav_vector() 10 0.001755 0.001755 0.001755 0.21% Castro::expand_state() 10 0.001746 0.001746 0.001746 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001711 0.001711 0.001711 0.21% MultiFab::Add() 82 0.001706 0.001706 0.001706 0.20% MLCellLinOp::setLevelBC() 11 0.001575 0.001575 0.001575 0.19% Castro::enforce_speed_limit() 62 0.001552 0.001552 0.001552 0.19% Castro::reset_internal_energy(Fab) 504 0.00155 0.00155 0.00155 0.19% Gravity::actual_solve_with_mlmg() 11 0.001394 0.001394 0.001394 0.17% FabArray::mult() 43 0.00134 0.00134 0.00134 0.16% FabArray::setDomainBndry() 41 0.001316 0.001316 0.001316 0.16% Castro::initData() 1 0.001274 0.001274 0.001274 0.15% MLMG::prepareForSolve() 11 0.001255 0.001255 0.001255 0.15% MLCellLinOp::smooth() 1640 0.001238 0.001238 0.001238 0.15% MLCellLinOp::prepareForSolve() 11 0.00121 0.00121 0.00121 0.15% MultiFab::contains_nan() 20 0.00119 0.00119 0.00119 0.14% MLCellLinOp::compGrad() 11 0.000937 0.000937 0.000937 0.11% FabArray::FillBoundary() 4023 0.000859 0.000859 0.000859 0.10% FabArrayBase::getCPC() 1323 0.0007726 0.0007726 0.0007726 0.09% FabArrayBase::CPC::define() 454 0.0006677 0.0006677 0.0006677 0.08% FabArrayBase::getFB() 4023 0.0005912 0.0005912 0.0005912 0.07% MLCellLinOp::apply() 1142 0.0005209 0.0005209 0.0005209 0.06% Amr::InitAmr() 1 0.0004924 0.0004924 0.0004924 0.06% Gravity::solve_for_phi() 10 0.0004658 0.0004658 0.0004658 0.06% Gravity::update_max_rhs() 11 0.0004201 0.0004201 0.0004201 0.05% CGSolver::sxay() 1586 0.0003948 0.0003948 0.0003948 0.05% Amr::coarseTimeStep() 10 0.000344 0.000344 0.000344 0.04% MultiFab::Copy() 11 0.0003385 0.0003385 0.0003385 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002938 0.0002938 0.0002938 0.04% FillPatchIterator::Initialize 41 0.0002913 0.0002913 0.0002913 0.03% FabArray::ParallelCopy() 861 0.0002768 0.0002768 0.0002768 0.03% MLCellLinOp::defineBC() 11 0.0002752 0.0002752 0.0002752 0.03% main() 1 0.0002605 0.0002605 0.0002605 0.03% MultiFab::max() 11 0.0002588 0.0002588 0.0002588 0.03% MLCellLinOp::correctionResidual() 492 0.0002345 0.0002345 0.0002345 0.03% Castro::subcycle_advance_ctu() 10 0.0002253 0.0002253 0.0002253 0.03% MLMG::MLRhsNormInf() 11 0.0002223 0.0002223 0.0002223 0.03% Castro::construct_new_gravity() 10 0.000215 0.000215 0.000215 0.03% Amr::timeStep() 10 0.0002029 0.0002029 0.0002029 0.02% MLMG::mgVcycle() 82 0.0001997 0.0001997 0.0001997 0.02% MLMG:computeResOfCorrection() 410 0.0001618 0.0001618 0.0001618 0.02% StateData::checkPoint() 12 0.0001465 0.0001465 0.0001465 0.02% MLLinOp::defineGrids() 11 0.00013 0.00013 0.00013 0.02% MLMG::mgVcycle_down::0 82 0.0001131 0.0001131 0.0001131 0.01% MLMG::mgVcycle_down::1 82 9.645e-05 9.645e-05 9.645e-05 0.01% MLMG::mgVcycle_down::2 82 9.219e-05 9.219e-05 9.219e-05 0.01% Castro::Castro() 1 8.934e-05 8.934e-05 8.934e-05 0.01% MLMG::mgVcycle_down::3 82 8.658e-05 8.658e-05 8.658e-05 0.01% Castro::initialize_advance() 10 8.534e-05 8.534e-05 8.534e-05 0.01% FabArrayBase::FB::FB() 56 8.522e-05 8.522e-05 8.522e-05 0.01% MLMG::mgVcycle_down::4 82 8.482e-05 8.482e-05 8.482e-05 0.01% Castro::clean_state() 62 8.009e-05 8.009e-05 8.009e-05 0.01% MLMG::actualBottomSolve() 82 7.987e-05 7.987e-05 7.987e-05 0.01% AmrLevel::checkPoint() 3 7.346e-05 7.346e-05 7.346e-05 0.01% MLMG::solve() 11 7.313e-05 7.313e-05 7.313e-05 0.01% MLMG::mgVcycle_up::4 82 7.096e-05 7.096e-05 7.096e-05 0.01% Castro::finalize_advance() 10 7.046e-05 7.046e-05 7.046e-05 0.01% Castro::advance() 10 6.739e-05 6.739e-05 6.739e-05 0.01% Castro::initialize_do_advance() 10 6.535e-05 6.535e-05 6.535e-05 0.01% MLMG::mgVcycle_up::0 82 6.08e-05 6.08e-05 6.08e-05 0.01% MLMG::oneIter() 82 5.836e-05 5.836e-05 5.836e-05 0.01% MLMG::mgVcycle_up::1 82 5.757e-05 5.757e-05 5.757e-05 0.01% MLMG::mgVcycle_up::3 82 5.713e-05 5.713e-05 5.713e-05 0.01% MLMG::mgVcycle_up::2 82 5.608e-05 5.608e-05 5.608e-05 0.01% MLCellLinOp::solutionResidual() 93 5.341e-05 5.341e-05 5.341e-05 0.01% StateData::define() 4 5.079e-05 5.079e-05 5.079e-05 0.01% Amr::FinalizeInit() 1 4.367e-05 4.367e-05 4.367e-05 0.01% MLMG::computeResidual() 82 4.117e-05 4.117e-05 4.117e-05 0.00% Castro::swap_state_time_levels() 10 3.971e-05 3.971e-05 3.971e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.661e-05 3.661e-05 3.661e-05 0.00% Castro::finalize_do_advance() 10 3.601e-05 3.601e-05 3.601e-05 0.00% Castro::enforce_consistent_e() 1 3.432e-05 3.432e-05 3.432e-05 0.00% MLMG::mgVcycle_bottom 82 3.29e-05 3.29e-05 3.29e-05 0.00% Gravity::actual_multilevel_solve() 1 3.243e-05 3.243e-05 3.243e-05 0.00% MLPoisson::define() 11 2.834e-05 2.834e-05 2.834e-05 0.00% FillPatchSingleLevel 41 2.822e-05 2.822e-05 2.822e-05 0.00% Amr::defBaseLevel() 1 2.665e-05 2.665e-05 2.665e-05 0.00% Amr::writeSmallPlotFile() 1 2.541e-05 2.541e-05 2.541e-05 0.00% makeSFC 55 2.501e-05 2.501e-05 2.501e-05 0.00% Castro::initMFs() 1 2.495e-05 2.495e-05 2.495e-05 0.00% MLLinOp::define() 11 2.387e-05 2.387e-05 2.387e-05 0.00% Castro::buildMetrics() 1 2.21e-05 2.21e-05 2.21e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.928e-05 1.928e-05 1.928e-05 0.00% Castro::construct_new_source() 50 1.847e-05 1.847e-05 1.847e-05 0.00% Castro::do_new_sources() 10 1.751e-05 1.751e-05 1.751e-05 0.00% Castro::construct_old_source() 50 1.739e-05 1.739e-05 1.739e-05 0.00% Castro::do_old_sources() 10 1.598e-05 1.598e-05 1.598e-05 0.00% DistributionMapping::Distribute() 56 1.504e-05 1.504e-05 1.504e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.359e-05 1.359e-05 1.359e-05 0.00% Castro::check_for_nan() 20 1.102e-05 1.102e-05 1.102e-05 0.00% Castro::apply_source_to_state() 20 1.093e-05 1.093e-05 1.093e-05 0.00% Gravity::swapTimeLevels() 10 9.912e-06 9.912e-06 9.912e-06 0.00% MLMG::computeMLResidual() 11 9.556e-06 9.556e-06 9.556e-06 0.00% Castro::construct_old_gravity() 10 9.515e-06 9.515e-06 9.515e-06 0.00% Castro::post_timestep() 10 9.435e-06 9.435e-06 9.435e-06 0.00% Amr::initSubcycle() 1 8.866e-06 8.866e-06 8.866e-06 0.00% MLPoisson::prepareForSolve() 11 8.581e-06 8.581e-06 8.581e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.591e-06 7.591e-06 7.591e-06 0.00% Amr::InitializeInit() 1 7.123e-06 7.123e-06 7.123e-06 0.00% MLMG::getGradSolution() 11 6.774e-06 6.774e-06 6.774e-06 0.00% Castro::computeNewDt() 9 6.406e-06 6.406e-06 6.406e-06 0.00% Castro::create_source_corrector() 10 4.45e-06 4.45e-06 4.45e-06 0.00% AmrLevel::checkPointPost() 3 4.299e-06 4.299e-06 4.299e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.233e-06 4.233e-06 4.233e-06 0.00% Castro::post_init() 1 4.066e-06 4.066e-06 4.066e-06 0.00% Gravity::set_mass_offset() 11 3.958e-06 3.958e-06 3.958e-06 0.00% Castro::retry_advance_ctu() 10 3.738e-06 3.738e-06 3.738e-06 0.00% Castro::FluxRegCrseInit 10 3.695e-06 3.695e-06 3.695e-06 0.00% MLMG::MLResNormInf() 11 3.653e-06 3.653e-06 3.653e-06 0.00% Amr::init() 1 2.747e-06 2.747e-06 2.747e-06 0.00% AmrLevel::checkPointPre() 3 2.674e-06 2.674e-06 2.674e-06 0.00% Castro::computeInitialDt() 2 2.58e-06 2.58e-06 2.58e-06 0.00% Castro::FluxRegFineAdd() 10 2.527e-06 2.527e-06 2.527e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.203e-06 2.203e-06 2.203e-06 0.00% Castro::post_regrid() 1 1.819e-06 1.819e-06 1.819e-06 0.00% Amr::initialInit() 1 1.54e-06 1.54e-06 1.54e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8334 0.8334 0.8334 100.00% Amr::coarseTimeStep() 10 0.6639 0.6639 0.6639 79.66% Amr::timeStep() 10 0.5611 0.5611 0.5611 67.33% Castro::advance() 10 0.5534 0.5534 0.5534 66.40% Castro::subcycle_advance_ctu() 10 0.5413 0.5413 0.5413 64.95% Castro::do_advance_ctu() 10 0.5411 0.5411 0.5411 64.93% Gravity::solve_phi_with_mlmg() 11 0.3219 0.3219 0.3219 38.62% Gravity::actual_solve_with_mlmg() 11 0.3132 0.3132 0.3132 37.59% Castro::construct_new_gravity() 10 0.293 0.293 0.293 35.16% MLMG::solve() 11 0.2901 0.2901 0.2901 34.81% Gravity::solve_for_phi() 10 0.2771 0.2771 0.2771 33.25% MLMG::oneIter() 82 0.2747 0.2747 0.2747 32.97% MLMG::mgVcycle() 82 0.273 0.273 0.273 32.75% VisMF::Write(FabArray) 11 0.207 0.207 0.207 24.84% Castro::construct_ctu_hydro_source() 10 0.1752 0.1752 0.1752 21.02% Amr::checkPoint() 3 0.1534 0.1534 0.1534 18.41% AmrLevel::checkPoint() 3 0.148 0.148 0.148 17.76% StateData::checkPoint() 12 0.1479 0.1479 0.1479 17.75% MLCellLinOp::smooth() 1640 0.1394 0.1394 0.1394 16.72% Amr::init() 1 0.1376 0.1376 0.1376 16.51% MLCellLinOp::applyBC() 4433 0.09824 0.09824 0.09824 11.79% MLMG::mgVcycle_bottom 82 0.08418 0.08418 0.08418 10.10% MLMG::actualBottomSolve() 82 0.08414 0.08414 0.08414 10.10% MLCGSolver::bicgstab 82 0.08332 0.08332 0.08332 10.00% MLPoisson::Fsmooth() 3280 0.06502 0.06502 0.06502 7.80% Amr::writePlotFile() 2 0.06213 0.06213 0.06213 7.46% Amr::initialInit() 1 0.05272 0.05272 0.05272 6.33% Amr::FinalizeInit() 1 0.04867 0.04867 0.04867 5.84% Castro::post_init() 1 0.04728 0.04728 0.04728 5.67% Gravity::multilevel_solve_for_new_phi() 1 0.04529 0.04529 0.04529 5.44% Gravity::actual_multilevel_solve() 1 0.04527 0.04527 0.04527 5.43% Castro::clean_state() 62 0.04328 0.04328 0.04328 5.19% FillPatchIterator::Initialize 41 0.04303 0.04303 0.04303 5.16% FillPatchSingleLevel 41 0.04142 0.04142 0.04142 4.97% StateDataPhysBCFunct::() 41 0.03738 0.03738 0.03738 4.48% MLCellLinOp::apply() 1142 0.03714 0.03714 0.03714 4.46% MLMG::mgVcycle_down::0 82 0.03611 0.03611 0.03611 4.33% MLMG::mgVcycle_up::0 82 0.03089 0.03089 0.03089 3.71% StateData::FillBoundary(geom) 328 0.02515 0.02515 0.02515 3.02% MultiFab::Dot() 1114 0.023 0.023 0.023 2.76% MLCellLinOp::correctionResidual() 492 0.02181 0.02181 0.02181 2.62% Castro::computeTemp() 63 0.01995 0.01995 0.01995 2.39% Castro::initialize_do_advance() 10 0.01918 0.01918 0.01918 2.30% MLMG:computeResOfCorrection() 410 0.01883 0.01883 0.01883 2.26% MLPoisson::define() 11 0.01871 0.01871 0.01871 2.25% MLMG::mgVcycle_down::1 82 0.01819 0.01819 0.01819 2.18% MLMG::mgVcycle_down::2 82 0.01772 0.01772 0.01772 2.13% Gravity::get_new_grav_vector() 11 0.01761 0.01761 0.01761 2.11% MLMG::mgVcycle_down::3 82 0.01691 0.01691 0.01691 2.03% MLMG::mgVcycle_down::4 82 0.01604 0.01604 0.01604 1.93% FabArray::FillBoundary() 4023 0.01599 0.01599 0.01599 1.92% Castro::construct_old_gravity() 10 0.01537 0.01537 0.01537 1.84% Gravity::get_old_grav_vector() 10 0.01536 0.01536 0.01536 1.84% FillBoundary_nowait() 4023 0.01513 0.01513 0.01513 1.82% CGSolver::sxay() 1586 0.01512 0.01512 0.01512 1.81% MultiFab::LinComb() 1586 0.01473 0.01473 0.01473 1.77% FabArray::setVal() 1144 0.01456 0.01456 0.01456 1.75% FabArray::ParallelCopy() 861 0.01451 0.01451 0.01451 1.74% FabArray::ParallelCopy_nowait() 861 0.01423 0.01423 0.01423 1.71% MLCGSolver::ParallelAllReduce 1514 0.01368 0.01368 0.01368 1.64% MLMG::mgVcycle_up::2 82 0.01362 0.01362 0.01362 1.63% Castro::normalize_species() 62 0.01349 0.01349 0.01349 1.62% MLMG::mgVcycle_up::1 82 0.01339 0.01339 0.01339 1.61% MLCellLinOp::defineAuxData() 11 0.01323 0.01323 0.01323 1.59% MLMG::addInterpCorrection() 410 0.01302 0.01302 0.01302 1.56% MLMG::mgVcycle_up::3 82 0.01293 0.01293 0.01293 1.55% MLMG::mgVcycle_up::4 82 0.01279 0.01279 0.01279 1.53% amrex::average_down 410 0.01237 0.01237 0.01237 1.48% MLPoisson::Fapply() 1142 0.012 0.012 0.012 1.44% Castro::initialize_advance() 10 0.01192 0.01192 0.01192 1.43% Castro::expand_state() 10 0.01152 0.01152 0.01152 1.38% Castro::do_new_sources() 10 0.01011 0.01011 0.01011 1.21% Castro::do_old_sources() 10 0.009563 0.009563 0.009563 1.15% Castro::enforce_min_density() 62 0.008862 0.008862 0.008862 1.06% Gravity::fill_multipole_BCs() 11 0.008366 0.008366 0.008366 1.00% Castro::post_timestep() 10 0.007531 0.007531 0.007531 0.90% MLCellLinOp::solutionResidual() 93 0.00729 0.00729 0.00729 0.87% MultiFab::Xpay() 585 0.006749 0.006749 0.006749 0.81% MLMG::computeResidual() 82 0.006299 0.006299 0.006299 0.76% Castro::reset_internal_energy(MultiFab) 63 0.00563 0.00563 0.00563 0.68% MLMG::prepareForSolve() 11 0.005535 0.005535 0.005535 0.66% Castro::estTimeStep() 21 0.005271 0.005271 0.005271 0.63% MLCellLinOp::defineBC() 11 0.005242 0.005242 0.005242 0.63% BndryData::define() 11 0.004966 0.004966 0.004966 0.60% Amr::InitializeInit() 1 0.004051 0.004051 0.004051 0.49% Amr::defBaseLevel() 1 0.004044 0.004044 0.004044 0.49% Castro::initData() 1 0.003521 0.003521 0.003521 0.42% Castro::construct_new_source() 50 0.002872 0.002872 0.002872 0.34% Castro::construct_new_gravity_source() 10 0.002853 0.002853 0.002853 0.34% Castro::computeNewDt() 9 0.002376 0.002376 0.002376 0.29% MLMG::ResNormInf() 93 0.002205 0.002205 0.002205 0.26% Castro::construct_old_source() 50 0.001954 0.001954 0.001954 0.23% Castro::construct_old_gravity_source() 10 0.001936 0.001936 0.001936 0.23% Castro::apply_source_to_state() 20 0.001826 0.001826 0.001826 0.22% MultiFab::Saxpy() 20 0.001815 0.001815 0.001815 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001711 0.001711 0.001711 0.21% MultiFab::Add() 82 0.001706 0.001706 0.001706 0.20% MLCellLinOp::setLevelBC() 11 0.001575 0.001575 0.001575 0.19% Castro::enforce_speed_limit() 62 0.001552 0.001552 0.001552 0.19% Castro::reset_internal_energy(Fab) 504 0.00155 0.00155 0.00155 0.19% MLMG::getGradSolution() 11 0.001454 0.001454 0.001454 0.17% MLCellLinOp::compGrad() 11 0.001447 0.001447 0.001447 0.17% FabArrayBase::getCPC() 1323 0.00144 0.00144 0.00144 0.17% FabArray::mult() 43 0.00134 0.00134 0.00134 0.16% FabArray::setDomainBndry() 41 0.001316 0.001316 0.001316 0.16% MLPoisson::prepareForSolve() 11 0.001219 0.001219 0.001219 0.15% MLCellLinOp::prepareForSolve() 11 0.00121 0.00121 0.00121 0.15% Castro::check_for_nan() 20 0.001201 0.001201 0.001201 0.14% MultiFab::contains_nan() 20 0.00119 0.00119 0.00119 0.14% Castro::post_regrid() 1 0.001107 0.001107 0.001107 0.13% MLMG::computeMLResidual() 11 0.001041 0.001041 0.001041 0.12% Gravity::update_max_rhs() 11 0.0008272 0.0008272 0.0008272 0.10% Castro::computeInitialDt() 2 0.0007384 0.0007384 0.0007384 0.09% FabArrayBase::getFB() 4023 0.0006764 0.0006764 0.0006764 0.08% FabArrayBase::CPC::define() 454 0.0006677 0.0006677 0.0006677 0.08% Amr::InitAmr() 1 0.0005012 0.0005012 0.0005012 0.06% Gravity::swapTimeLevels() 10 0.0004467 0.0004467 0.0004467 0.05% Castro::Castro() 1 0.000432 0.000432 0.000432 0.05% MultiFab::Copy() 11 0.0003385 0.0003385 0.0003385 0.04% MLMG::MLResNormInf() 11 0.0002855 0.0002855 0.0002855 0.03% MultiFab::max() 11 0.0002588 0.0002588 0.0002588 0.03% MLMG::MLRhsNormInf() 11 0.0002223 0.0002223 0.0002223 0.03% MLLinOp::define() 11 0.0002087 0.0002087 0.0002087 0.03% MLLinOp::defineGrids() 11 0.0001848 0.0001848 0.0001848 0.02% Castro::buildMetrics() 1 0.0001523 0.0001523 0.0001523 0.02% FabArrayBase::FB::FB() 56 8.522e-05 8.522e-05 8.522e-05 0.01% Castro::finalize_advance() 10 7.668e-05 7.668e-05 7.668e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.838e-05 5.838e-05 5.838e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.262e-05 5.262e-05 5.262e-05 0.01% StateData::define() 4 5.079e-05 5.079e-05 5.079e-05 0.01% Castro::swap_state_time_levels() 10 3.971e-05 3.971e-05 3.971e-05 0.00% makeSFC 55 3.903e-05 3.903e-05 3.903e-05 0.00% Castro::finalize_do_advance() 10 3.601e-05 3.601e-05 3.601e-05 0.00% Castro::enforce_consistent_e() 1 3.432e-05 3.432e-05 3.432e-05 0.00% Amr::writeSmallPlotFile() 1 2.541e-05 2.541e-05 2.541e-05 0.00% Castro::initMFs() 1 2.495e-05 2.495e-05 2.495e-05 0.00% DistributionMapping::Distribute() 56 1.504e-05 1.504e-05 1.504e-05 0.00% Amr::initSubcycle() 1 8.866e-06 8.866e-06 8.866e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.259e-06 5.259e-06 5.259e-06 0.00% Castro::create_source_corrector() 10 4.45e-06 4.45e-06 4.45e-06 0.00% AmrLevel::checkPointPost() 3 4.299e-06 4.299e-06 4.299e-06 0.00% Gravity::set_mass_offset() 11 3.958e-06 3.958e-06 3.958e-06 0.00% Castro::retry_advance_ctu() 10 3.738e-06 3.738e-06 3.738e-06 0.00% Castro::FluxRegCrseInit 10 3.695e-06 3.695e-06 3.695e-06 0.00% AmrLevel::checkPointPre() 3 2.674e-06 2.674e-06 2.674e-06 0.00% Castro::FluxRegFineAdd() 10 2.527e-06 2.527e-06 2.527e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.203e-06 2.203e-06 2.203e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-19-gacc223f99182) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-19-gacc223f99182) initialized Starting run at 08:29:49 UTC on 2022-08-31. Successfully read inputs file ... Castro git describe: 22.08-11-ga978fcf88 AMReX git describe: 22.08-19-gacc223f99 Microphysics git describe: 22.08-6-ge3a6cb11 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.457116047 Restart time = 0.048284891 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051790937 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.05176174 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.062629404 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.079821724 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.076880664 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.060967788 seconds Ending run at 08:29:49 UTC on 2022-08-31. Run time = 0.433216122 Run time without initialization = 0.384325268 Average number of zones advanced per microsecond: 3.410 Average number of zones advanced per microsecond per rank: 3.410 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4332 ... 0.4332 ... 0.4332 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1136 0.1136 0.1136 26.23% VisMF::Read() 3 0.04024 0.04024 0.04024 9.29% MLCellLinOp::applyBC() 1946 0.03509 0.03509 0.03509 8.10% VisMF::Write(FabArray) 1 0.03137 0.03137 0.03137 7.24% Amr::writePlotFile() 1 0.02968 0.02968 0.02968 6.85% MLPoisson::Fsmooth() 1440 0.02747 0.02747 0.02747 6.34% StateData::FillBoundary(geom) 160 0.01205 0.01205 0.01205 2.78% MLCGSolver::bicgstab 36 0.01038 0.01038 0.01038 2.40% Castro::normalize_species() 30 0.01032 0.01032 0.01032 2.38% Castro::computeTemp() 30 0.01026 0.01026 0.01026 2.37% MultiFab::Dot() 484 0.009643 0.009643 0.009643 2.23% FabArray::setVal() 537 0.006817 0.006817 0.006817 1.57% MLCellLinOp::defineAuxData() 6 0.006393 0.006393 0.006393 1.48% FillBoundary_nowait() 1766 0.006354 0.006354 0.006354 1.47% MultiFab::LinComb() 690 0.006176 0.006176 0.006176 1.43% FabArray::ParallelCopy_nowait() 380 0.006064 0.006064 0.006064 1.40% Castro::enforce_min_density() 30 0.006018 0.006018 0.006018 1.39% StateDataPhysBCFunct::() 20 0.005787 0.005787 0.005787 1.34% MLPoisson::Fapply() 500 0.005126 0.005126 0.005126 1.18% Gravity::fill_multipole_BCs() 6 0.004538 0.004538 0.004538 1.05% Castro::estTimeStep() 10 0.004268 0.004268 0.004268 0.99% Amr::restart() 1 0.003727 0.003727 0.003727 0.86% MLMG::addInterpCorrection() 180 0.00333 0.00333 0.00333 0.77% amrex::average_down 180 0.003037 0.003037 0.003037 0.70% MultiFab::Xpay() 258 0.002914 0.002914 0.002914 0.67% Castro::do_advance_ctu() 5 0.002265 0.002265 0.002265 0.52% BndryData::define() 6 0.002158 0.002158 0.002158 0.50% Castro::enforce_speed_limit() 30 0.001927 0.001927 0.001927 0.44% Castro::construct_new_gravity_source() 5 0.001738 0.001738 0.001738 0.40% Castro::reset_internal_energy(MultiFab) 30 0.001673 0.001673 0.001673 0.39% Castro::construct_old_gravity_source() 5 0.00153 0.00153 0.00153 0.35% Castro::reset_internal_energy(Fab) 240 0.001109 0.001109 0.001109 0.26% Gravity::get_old_grav_vector() 5 0.001003 0.001003 0.001003 0.23% MLMG::ResNormInf() 42 0.0009476 0.0009476 0.0009476 0.22% Gravity::get_new_grav_vector() 5 0.0009402 0.0009402 0.0009402 0.22% MultiFab::Saxpy() 10 0.0009254 0.0009254 0.0009254 0.21% Castro::construct_new_source() 25 0.0009179 0.0009179 0.0009179 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009108 0.0009108 0.0009108 0.21% Castro::expand_state() 5 0.0008754 0.0008754 0.0008754 0.20% MLCellLinOp::setLevelBC() 6 0.00083 0.00083 0.00083 0.19% Gravity::actual_solve_with_mlmg() 6 0.0008165 0.0008165 0.0008165 0.19% MultiFab::Add() 36 0.000722 0.000722 0.000722 0.17% MLMG::prepareForSolve() 6 0.0006563 0.0006563 0.0006563 0.15% FabArray::mult() 22 0.0006511 0.0006511 0.0006511 0.15% FabArray::setDomainBndry() 20 0.0006475 0.0006475 0.0006475 0.15% MLCellLinOp::prepareForSolve() 6 0.0006395 0.0006395 0.0006395 0.15% MultiFab::contains_nan() 10 0.0005895 0.0005895 0.0005895 0.14% MLCellLinOp::smooth() 720 0.0005744 0.0005744 0.0005744 0.13% MLCellLinOp::compGrad() 6 0.0004924 0.0004924 0.0004924 0.11% Amr::InitAmr() 1 0.0004334 0.0004334 0.0004334 0.10% FabArray::FillBoundary() 1766 0.0003931 0.0003931 0.0003931 0.09% FabArrayBase::CPC::define() 244 0.0003859 0.0003859 0.0003859 0.09% FabArrayBase::getCPC() 632 0.0003636 0.0003636 0.0003636 0.08% main() 1 0.0002857 0.0002857 0.0002857 0.07% FabArrayBase::getFB() 1766 0.0002601 0.0002601 0.0002601 0.06% MLCellLinOp::apply() 500 0.000242 0.000242 0.000242 0.06% Castro::subcycle_advance_ctu() 5 0.0002358 0.0002358 0.0002358 0.05% Gravity::update_max_rhs() 6 0.0002326 0.0002326 0.0002326 0.05% Gravity::solve_for_phi() 5 0.0002277 0.0002277 0.0002277 0.05% Amr::coarseTimeStep() 5 0.0002106 0.0002106 0.0002106 0.05% CGSolver::sxay() 690 0.0001915 0.0001915 0.0001915 0.04% MultiFab::Copy() 6 0.0001802 0.0001802 0.0001802 0.04% Castro::construct_new_gravity() 5 0.0001678 0.0001678 0.0001678 0.04% MLCellLinOp::defineBC() 6 0.0001505 0.0001505 0.0001505 0.03% Castro::advance() 5 0.0001457 0.0001457 0.0001457 0.03% Castro::create_source_corrector() 5 0.0001386 0.0001386 0.0001386 0.03% FillPatchIterator::Initialize 20 0.000137 0.000137 0.000137 0.03% MultiFab::max() 6 0.0001353 0.0001353 0.0001353 0.03% MLCGSolver::ParallelAllReduce 659 0.0001334 0.0001334 0.0001334 0.03% FabArray::ParallelCopy() 380 0.0001326 0.0001326 0.0001326 0.03% MLMG::MLRhsNormInf() 6 0.0001155 0.0001155 0.0001155 0.03% MLCellLinOp::correctionResidual() 216 0.000107 0.000107 0.000107 0.02% Amr::timeStep() 5 0.0001065 0.0001065 0.0001065 0.02% MLMG::mgVcycle() 36 9.052e-05 9.052e-05 9.052e-05 0.02% Castro::construct_old_source() 25 9.005e-05 9.005e-05 9.005e-05 0.02% AmrLevel::restart() 1 7.951e-05 7.951e-05 7.951e-05 0.02% Castro::initialize_do_advance() 5 7.935e-05 7.935e-05 7.935e-05 0.02% StateData::restartDoit() 4 7.549e-05 7.549e-05 7.549e-05 0.02% MLLinOp::defineGrids() 6 7.178e-05 7.178e-05 7.178e-05 0.02% Castro::finalize_advance() 5 7.092e-05 7.092e-05 7.092e-05 0.02% Castro::post_timestep() 5 7.06e-05 7.06e-05 7.06e-05 0.02% MLMG:computeResOfCorrection() 180 7.036e-05 7.036e-05 7.036e-05 0.02% Castro::initialize_advance() 5 5.68e-05 5.68e-05 5.68e-05 0.01% Castro::computeNewDt() 5 5.665e-05 5.665e-05 5.665e-05 0.01% FabArrayBase::FB::FB() 26 5.612e-05 5.612e-05 5.612e-05 0.01% Castro::construct_old_gravity() 5 5.421e-05 5.421e-05 5.421e-05 0.01% MLMG::mgVcycle_down::0 36 4.863e-05 4.863e-05 4.863e-05 0.01% MLMG::mgVcycle_down::1 36 4.512e-05 4.512e-05 4.512e-05 0.01% MLMG::mgVcycle_down::2 36 4.244e-05 4.244e-05 4.244e-05 0.01% Castro::clean_state() 30 4.243e-05 4.243e-05 4.243e-05 0.01% MLMG::mgVcycle_down::4 36 4.119e-05 4.119e-05 4.119e-05 0.01% MLMG::mgVcycle_down::3 36 3.975e-05 3.975e-05 3.975e-05 0.01% MLMG::actualBottomSolve() 36 3.735e-05 3.735e-05 3.735e-05 0.01% MLMG::mgVcycle_up::4 36 3.526e-05 3.526e-05 3.526e-05 0.01% MLMG::solve() 6 3.484e-05 3.484e-05 3.484e-05 0.01% Castro::buildMetrics() 1 3.299e-05 3.299e-05 3.299e-05 0.01% Gravity::actual_multilevel_solve() 1 3.062e-05 3.062e-05 3.062e-05 0.01% MLMG::mgVcycle_up::0 36 2.891e-05 2.891e-05 2.891e-05 0.01% Castro::post_restart() 1 2.849e-05 2.849e-05 2.849e-05 0.01% Castro::swap_state_time_levels() 5 2.805e-05 2.805e-05 2.805e-05 0.01% MLMG::mgVcycle_up::3 36 2.789e-05 2.789e-05 2.789e-05 0.01% Amr::writeSmallPlotFile() 1 2.683e-05 2.683e-05 2.683e-05 0.01% MLMG::oneIter() 36 2.655e-05 2.655e-05 2.655e-05 0.01% Castro::initMFs() 1 2.637e-05 2.637e-05 2.637e-05 0.01% MLMG::mgVcycle_up::2 36 2.616e-05 2.616e-05 2.616e-05 0.01% MLMG::mgVcycle_up::1 36 2.565e-05 2.565e-05 2.565e-05 0.01% MLCellLinOp::solutionResidual() 42 2.475e-05 2.475e-05 2.475e-05 0.01% MLPoisson::define() 6 2.203e-05 2.203e-05 2.203e-05 0.01% Castro::finalize_do_advance() 5 2.02e-05 2.02e-05 2.02e-05 0.00% MLLinOp::define() 6 2.014e-05 2.014e-05 2.014e-05 0.00% MLMG::computeResidual() 36 1.858e-05 1.858e-05 1.858e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.706e-05 1.706e-05 1.706e-05 0.00% MLMG::mgVcycle_bottom 36 1.569e-05 1.569e-05 1.569e-05 0.00% makeSFC 30 1.449e-05 1.449e-05 1.449e-05 0.00% FillPatchSingleLevel 20 1.444e-05 1.444e-05 1.444e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.422e-05 1.422e-05 1.422e-05 0.00% Castro::do_new_sources() 5 1.032e-05 1.032e-05 1.032e-05 0.00% Amr::initSubcycle() 1 9.338e-06 9.338e-06 9.338e-06 0.00% DistributionMapping::Distribute() 31 9.134e-06 9.134e-06 9.134e-06 0.00% Castro::do_old_sources() 5 8.425e-06 8.425e-06 8.425e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.294e-06 7.294e-06 7.294e-06 0.00% Castro::check_for_nan() 10 6.448e-06 6.448e-06 6.448e-06 0.00% Castro::apply_source_to_state() 10 5.779e-06 5.779e-06 5.779e-06 0.00% MLPoisson::prepareForSolve() 6 4.842e-06 4.842e-06 4.842e-06 0.00% Gravity::swapTimeLevels() 5 4.552e-06 4.552e-06 4.552e-06 0.00% MLMG::computeMLResidual() 6 4.471e-06 4.471e-06 4.471e-06 0.00% MLMG::getGradSolution() 6 3.196e-06 3.196e-06 3.196e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.107e-06 3.107e-06 3.107e-06 0.00% MLMG::MLResNormInf() 6 2.278e-06 2.278e-06 2.278e-06 0.00% Gravity::set_mass_offset() 6 2.26e-06 2.26e-06 2.26e-06 0.00% Castro::retry_advance_ctu() 5 2.062e-06 2.062e-06 2.062e-06 0.00% Castro::FluxRegCrseInit 5 1.707e-06 1.707e-06 1.707e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.394e-06 1.394e-06 1.394e-06 0.00% Amr::init() 1 1.157e-06 1.157e-06 1.157e-06 0.00% Castro::FluxRegFineAdd() 5 1.123e-06 1.123e-06 1.123e-06 0.00% AmrLevel::AmrLevel() 1 8.5e-07 8.5e-07 8.5e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4332 0.4332 0.4332 100.00% Amr::coarseTimeStep() 5 0.3231 0.3231 0.3231 74.57% Amr::timeStep() 5 0.3209 0.3209 0.3209 74.08% Castro::advance() 5 0.3159 0.3159 0.3159 72.91% Castro::subcycle_advance_ctu() 5 0.3077 0.3077 0.3077 71.03% Castro::do_advance_ctu() 5 0.3075 0.3075 0.3075 70.98% Castro::construct_new_gravity() 5 0.1458 0.1458 0.1458 33.66% Gravity::solve_phi_with_mlmg() 6 0.1414 0.1414 0.1414 32.64% Gravity::solve_for_phi() 5 0.1378 0.1378 0.1378 31.81% Gravity::actual_solve_with_mlmg() 6 0.1367 0.1367 0.1367 31.56% MLMG::solve() 6 0.1242 0.1242 0.1242 28.67% MLMG::oneIter() 36 0.1169 0.1169 0.1169 26.98% MLMG::mgVcycle() 36 0.1162 0.1162 0.1162 26.81% Castro::construct_ctu_hydro_source() 5 0.1136 0.1136 0.1136 26.23% Amr::writePlotFile() 1 0.06106 0.06106 0.06106 14.09% MLCellLinOp::smooth() 720 0.05931 0.05931 0.05931 13.69% Amr::init() 1 0.04833 0.04833 0.04833 11.16% Amr::restart() 1 0.04833 0.04833 0.04833 11.16% MLCellLinOp::applyBC() 1946 0.04215 0.04215 0.04215 9.73% AmrLevel::restart() 1 0.04045 0.04045 0.04045 9.34% StateData::restartDoit() 4 0.04037 0.04037 0.04037 9.32% VisMF::Read() 3 0.04024 0.04024 0.04024 9.29% MLMG::mgVcycle_bottom 36 0.03553 0.03553 0.03553 8.20% MLMG::actualBottomSolve() 36 0.03552 0.03552 0.03552 8.20% MLCGSolver::bicgstab 36 0.03516 0.03516 0.03516 8.12% VisMF::Write(FabArray) 1 0.03137 0.03137 0.03137 7.24% Castro::clean_state() 30 0.03134 0.03134 0.03134 7.23% MLPoisson::Fsmooth() 1440 0.02747 0.02747 0.02747 6.34% FillPatchIterator::Initialize 20 0.02062 0.02062 0.02062 4.76% FillPatchSingleLevel 20 0.01984 0.01984 0.01984 4.58% StateDataPhysBCFunct::() 20 0.01784 0.01784 0.01784 4.12% MLCellLinOp::apply() 500 0.01599 0.01599 0.01599 3.69% MLMG::mgVcycle_down::0 36 0.01558 0.01558 0.01558 3.60% MLMG::mgVcycle_up::0 36 0.01331 0.01331 0.01331 3.07% Castro::computeTemp() 30 0.01304 0.01304 0.01304 3.01% StateData::FillBoundary(geom) 160 0.01205 0.01205 0.01205 2.78% Castro::initialize_do_advance() 5 0.01174 0.01174 0.01174 2.71% Castro::normalize_species() 30 0.01032 0.01032 0.01032 2.38% MLPoisson::define() 6 0.01013 0.01013 0.01013 2.34% MultiFab::Dot() 484 0.009643 0.009643 0.009643 2.23% MLCellLinOp::correctionResidual() 216 0.009356 0.009356 0.009356 2.16% Castro::do_new_sources() 5 0.008136 0.008136 0.008136 1.88% Castro::do_old_sources() 5 0.008134 0.008134 0.008134 1.88% MLMG:computeResOfCorrection() 180 0.008086 0.008086 0.008086 1.87% Castro::construct_old_gravity() 5 0.007934 0.007934 0.007934 1.83% Castro::initialize_advance() 5 0.007905 0.007905 0.007905 1.82% Gravity::get_old_grav_vector() 5 0.00788 0.00788 0.00788 1.82% Gravity::get_new_grav_vector() 5 0.007814 0.007814 0.007814 1.80% MLMG::mgVcycle_down::1 36 0.007784 0.007784 0.007784 1.80% MLMG::mgVcycle_down::2 36 0.007523 0.007523 0.007523 1.74% MLMG::mgVcycle_down::3 36 0.007159 0.007159 0.007159 1.65% MLCellLinOp::defineAuxData() 6 0.007134 0.007134 0.007134 1.65% FabArray::FillBoundary() 1766 0.007063 0.007063 0.007063 1.63% MLMG::mgVcycle_down::4 36 0.006836 0.006836 0.006836 1.58% FabArray::setVal() 537 0.006817 0.006817 0.006817 1.57% FillBoundary_nowait() 1766 0.00667 0.00667 0.00667 1.54% FabArray::ParallelCopy() 380 0.006574 0.006574 0.006574 1.52% FabArray::ParallelCopy_nowait() 380 0.006441 0.006441 0.006441 1.49% CGSolver::sxay() 690 0.006367 0.006367 0.006367 1.47% MultiFab::LinComb() 690 0.006176 0.006176 0.006176 1.43% Castro::enforce_min_density() 30 0.006018 0.006018 0.006018 1.39% MLMG::mgVcycle_up::2 36 0.005765 0.005765 0.005765 1.33% MLCGSolver::ParallelAllReduce 659 0.005764 0.005764 0.005764 1.33% Castro::expand_state() 5 0.005739 0.005739 0.005739 1.32% MLMG::mgVcycle_up::1 36 0.005678 0.005678 0.005678 1.31% MLMG::addInterpCorrection() 180 0.005602 0.005602 0.005602 1.29% MLMG::mgVcycle_up::3 36 0.005463 0.005463 0.005463 1.26% MLMG::mgVcycle_up::4 36 0.005431 0.005431 0.005431 1.25% amrex::average_down 180 0.005353 0.005353 0.005353 1.24% MLPoisson::Fapply() 500 0.005126 0.005126 0.005126 1.18% Castro::post_timestep() 5 0.00496 0.00496 0.00496 1.14% Gravity::fill_multipole_BCs() 6 0.004538 0.004538 0.004538 1.05% Castro::estTimeStep() 10 0.004268 0.004268 0.004268 0.99% Castro::post_restart() 1 0.003967 0.003967 0.003967 0.92% Gravity::multilevel_solve_for_new_phi() 1 0.003845 0.003845 0.003845 0.89% Gravity::actual_multilevel_solve() 1 0.003828 0.003828 0.003828 0.88% MLCellLinOp::solutionResidual() 42 0.003264 0.003264 0.003264 0.75% MLMG::prepareForSolve() 6 0.002917 0.002917 0.002917 0.67% MultiFab::Xpay() 258 0.002914 0.002914 0.002914 0.67% MLCellLinOp::defineBC() 6 0.002851 0.002851 0.002851 0.66% Castro::reset_internal_energy(MultiFab) 30 0.002782 0.002782 0.002782 0.64% MLMG::computeResidual() 36 0.002714 0.002714 0.002714 0.63% BndryData::define() 6 0.0027 0.0027 0.0027 0.62% Castro::construct_new_source() 25 0.002655 0.002655 0.002655 0.61% Castro::computeNewDt() 5 0.001946 0.001946 0.001946 0.45% Castro::enforce_speed_limit() 30 0.001927 0.001927 0.001927 0.44% Castro::construct_new_gravity_source() 5 0.001738 0.001738 0.001738 0.40% Castro::construct_old_source() 25 0.00162 0.00162 0.00162 0.37% Castro::construct_old_gravity_source() 5 0.00153 0.00153 0.00153 0.35% Castro::reset_internal_energy(Fab) 240 0.001109 0.001109 0.001109 0.26% MLMG::ResNormInf() 42 0.0009476 0.0009476 0.0009476 0.22% Castro::apply_source_to_state() 10 0.0009312 0.0009312 0.0009312 0.21% MultiFab::Saxpy() 10 0.0009254 0.0009254 0.0009254 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009108 0.0009108 0.0009108 0.21% MLCellLinOp::setLevelBC() 6 0.00083 0.00083 0.00083 0.19% MLMG::getGradSolution() 6 0.0007674 0.0007674 0.0007674 0.18% MLCellLinOp::compGrad() 6 0.0007642 0.0007642 0.0007642 0.18% FabArrayBase::getCPC() 632 0.0007495 0.0007495 0.0007495 0.17% MultiFab::Add() 36 0.000722 0.000722 0.000722 0.17% FabArray::mult() 22 0.0006511 0.0006511 0.0006511 0.15% FabArray::setDomainBndry() 20 0.0006475 0.0006475 0.0006475 0.15% MLPoisson::prepareForSolve() 6 0.0006443 0.0006443 0.0006443 0.15% MLCellLinOp::prepareForSolve() 6 0.0006395 0.0006395 0.0006395 0.15% Castro::check_for_nan() 10 0.000596 0.000596 0.000596 0.14% MultiFab::contains_nan() 10 0.0005895 0.0005895 0.0005895 0.14% MLMG::computeMLResidual() 6 0.0005724 0.0005724 0.0005724 0.13% Gravity::update_max_rhs() 6 0.0004455 0.0004455 0.0004455 0.10% Amr::InitAmr() 1 0.0004427 0.0004427 0.0004427 0.10% FabArrayBase::CPC::define() 244 0.0003859 0.0003859 0.0003859 0.09% FabArrayBase::getFB() 1766 0.0003162 0.0003162 0.0003162 0.07% Gravity::swapTimeLevels() 5 0.0002309 0.0002309 0.0002309 0.05% MultiFab::Copy() 6 0.0001802 0.0001802 0.0001802 0.04% Castro::buildMetrics() 1 0.0001556 0.0001556 0.0001556 0.04% MLMG::MLResNormInf() 6 0.0001496 0.0001496 0.0001496 0.03% Castro::create_source_corrector() 5 0.0001386 0.0001386 0.0001386 0.03% MultiFab::max() 6 0.0001353 0.0001353 0.0001353 0.03% MLLinOp::define() 6 0.000123 0.000123 0.000123 0.03% MLMG::MLRhsNormInf() 6 0.0001155 0.0001155 0.0001155 0.03% MLLinOp::defineGrids() 6 0.0001029 0.0001029 0.0001029 0.02% Castro::finalize_advance() 5 7.375e-05 7.375e-05 7.375e-05 0.02% FabArrayBase::FB::FB() 26 5.612e-05 5.612e-05 5.612e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.972e-05 2.972e-05 2.972e-05 0.01% Castro::swap_state_time_levels() 5 2.805e-05 2.805e-05 2.805e-05 0.01% Amr::writeSmallPlotFile() 1 2.683e-05 2.683e-05 2.683e-05 0.01% Castro::initMFs() 1 2.637e-05 2.637e-05 2.637e-05 0.01% makeSFC 30 2.242e-05 2.242e-05 2.242e-05 0.01% Castro::finalize_do_advance() 5 2.02e-05 2.02e-05 2.02e-05 0.00% Amr::initSubcycle() 1 9.338e-06 9.338e-06 9.338e-06 0.00% DistributionMapping::Distribute() 31 9.134e-06 9.134e-06 9.134e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.304e-06 4.304e-06 4.304e-06 0.00% Gravity::set_mass_offset() 6 2.26e-06 2.26e-06 2.26e-06 0.00% Castro::retry_advance_ctu() 5 2.062e-06 2.062e-06 2.062e-06 0.00% Castro::FluxRegCrseInit 5 1.707e-06 1.707e-06 1.707e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.394e-06 1.394e-06 1.394e-06 0.00% Castro::FluxRegFineAdd() 5 1.123e-06 1.123e-06 1.123e-06 0.00% AmrLevel::AmrLevel() 1 8.5e-07 8.5e-07 8.5e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-19-gacc223f99182) finalized