Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.07-19-g48702b48836d) initialized Starting run at 08:51:45 UTC on 2022-08-01. Successfully read inputs file ... Castro git describe: 22.07-14-ga2a554e52 AMReX git describe: 22.07-19-g48702b488 Microphysics git describe: 22.07-16-g593f02b0 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.051566472 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030026382 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.050092371 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050538427 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.065091742 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.064663174 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.06285801 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.047564543 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.0718986 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.063253063 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.04793925 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.047798747 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.051451587 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.047476627 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029307757 seconds Ending run at 08:51:46 UTC on 2022-08-01. Run time = 0.830864112 Run time without initialization = 0.700586092 Average number of zones advanced per microsecond: 3.742 Average number of zones advanced per microsecond per rank: 3.742 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8309 ... 0.8309 ... 0.8309 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1983 0.1983 0.1983 23.87% VisMF::Write(FabArray) 11 0.198 0.198 0.198 23.82% MLCellLinOp::applyBC() 4379 0.07765 0.07765 0.07765 9.35% MLPoisson::Fsmooth() 3240 0.06189 0.06189 0.06189 7.45% StateData::FillBoundary(geom) 328 0.02396 0.02396 0.02396 2.88% MLCGSolver::bicgstab 81 0.02307 0.02307 0.02307 2.78% MultiFab::Dot() 1100 0.02146 0.02146 0.02146 2.58% Castro::computeTemp() 63 0.01446 0.01446 0.01446 1.74% Castro::normalize_species() 62 0.01438 0.01438 0.01438 1.73% FabArray::setVal() 1135 0.01385 0.01385 0.01385 1.67% FillBoundary_nowait() 3974 0.01379 0.01379 0.01379 1.66% MultiFab::LinComb() 1566 0.01379 0.01379 0.01379 1.66% StateDataPhysBCFunct::() 41 0.01303 0.01303 0.01303 1.57% FabArray::ParallelCopy_nowait() 851 0.01259 0.01259 0.01259 1.51% MLPoisson::Fapply() 1128 0.0113 0.0113 0.0113 1.36% MLCellLinOp::defineAuxData() 11 0.01111 0.01111 0.01111 1.34% Castro::enforce_min_density() 62 0.009554 0.009554 0.009554 1.15% Gravity::fill_multipole_BCs() 11 0.008239 0.008239 0.008239 0.99% MLMG::addInterpCorrection() 405 0.007172 0.007172 0.007172 0.86% amrex::average_down 405 0.006614 0.006614 0.006614 0.80% MultiFab::Xpay() 578 0.00639 0.00639 0.00639 0.77% Castro::estTimeStep() 21 0.005338 0.005338 0.005338 0.64% Amr::checkPoint() 3 0.005198 0.005198 0.005198 0.63% Castro::do_advance_ctu() 10 0.004565 0.004565 0.004565 0.55% Castro::reset_internal_energy(MultiFab) 63 0.004491 0.004491 0.004491 0.54% BndryData::define() 11 0.003697 0.003697 0.003697 0.44% Castro::construct_new_gravity_source() 10 0.003272 0.003272 0.003272 0.39% Amr::writePlotFile() 2 0.002846 0.002846 0.002846 0.34% Castro::construct_old_gravity_source() 10 0.002643 0.002643 0.002643 0.32% Gravity::get_new_grav_vector() 11 0.00191 0.00191 0.00191 0.23% MLMG::ResNormInf() 92 0.001904 0.001904 0.001904 0.23% MultiFab::Saxpy() 20 0.001803 0.001803 0.001803 0.22% Castro::expand_state() 10 0.00173 0.00173 0.00173 0.21% Gravity::get_old_grav_vector() 10 0.001711 0.001711 0.001711 0.21% MLMG::oneIter() 81 0.001666 0.001666 0.001666 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00159 0.00159 0.00159 0.19% MLCellLinOp::setLevelBC() 11 0.001513 0.001513 0.001513 0.18% Castro::enforce_speed_limit() 62 0.001503 0.001503 0.001503 0.18% Castro::reset_internal_energy(Fab) 504 0.001406 0.001406 0.001406 0.17% Gravity::actual_solve_with_mlmg() 11 0.001343 0.001343 0.001343 0.16% FabArray::mult() 43 0.001301 0.001301 0.001301 0.16% FabArray::setDomainBndry() 41 0.001284 0.001284 0.001284 0.15% Castro::initData() 1 0.001239 0.001239 0.001239 0.15% MultiFab::contains_nan() 20 0.00118 0.00118 0.00118 0.14% MLCellLinOp::prepareForSolve() 11 0.001132 0.001132 0.001132 0.14% MLCellLinOp::smooth() 1620 0.001084 0.001084 0.001084 0.13% MLMG::prepareForSolve() 11 0.001005 0.001005 0.001005 0.12% MLCellLinOp::compGrad() 11 0.0009151 0.0009151 0.0009151 0.11% FabArray::FillBoundary() 3974 0.0008289 0.0008289 0.0008289 0.10% FabArrayBase::getCPC() 1313 0.000773 0.000773 0.000773 0.09% FabArrayBase::CPC::define() 454 0.0006594 0.0006594 0.0006594 0.08% FabArrayBase::getFB() 3974 0.0005947 0.0005947 0.0005947 0.07% Amr::InitAmr() 1 0.0004822 0.0004822 0.0004822 0.06% Gravity::solve_for_phi() 10 0.0004434 0.0004434 0.0004434 0.05% MLCellLinOp::apply() 1128 0.0004251 0.0004251 0.0004251 0.05% Gravity::update_max_rhs() 11 0.000398 0.000398 0.000398 0.05% CGSolver::sxay() 1566 0.0003754 0.0003754 0.0003754 0.05% Amr::coarseTimeStep() 10 0.0003513 0.0003513 0.0003513 0.04% FillPatchIterator::Initialize 41 0.0002929 0.0002929 0.0002929 0.04% MLCGSolver::ParallelAllReduce 1495 0.0002879 0.0002879 0.0002879 0.03% MLCellLinOp::defineBC() 11 0.0002773 0.0002773 0.0002773 0.03% FabArray::ParallelCopy() 851 0.0002713 0.0002713 0.0002713 0.03% main() 1 0.0002656 0.0002656 0.0002656 0.03% MultiFab::max() 11 0.000252 0.000252 0.000252 0.03% MultiFab::Copy() 11 0.0002504 0.0002504 0.0002504 0.03% Castro::subcycle_advance_ctu() 10 0.0002366 0.0002366 0.0002366 0.03% MLCellLinOp::correctionResidual() 486 0.0002319 0.0002319 0.0002319 0.03% Amr::timeStep() 10 0.0002089 0.0002089 0.0002089 0.03% Castro::construct_new_gravity() 10 0.0002031 0.0002031 0.0002031 0.02% MLMG::MLRhsNormInf() 11 0.0002019 0.0002019 0.0002019 0.02% MLMG::mgVcycle() 81 0.0001864 0.0001864 0.0001864 0.02% MLLinOp::defineGrids() 11 0.0001571 0.0001571 0.0001571 0.02% StateData::checkPoint() 12 0.0001311 0.0001311 0.0001311 0.02% MLMG:computeResOfCorrection() 405 0.0001264 0.0001264 0.0001264 0.02% MLMG::actualBottomSolve() 81 9.833e-05 9.833e-05 9.833e-05 0.01% MLMG::mgVcycle_down::0 81 8.885e-05 8.885e-05 8.885e-05 0.01% Castro::Castro() 1 8.589e-05 8.589e-05 8.589e-05 0.01% FabArrayBase::FB::FB() 56 8.166e-05 8.166e-05 8.166e-05 0.01% Castro::initialize_advance() 10 8.09e-05 8.09e-05 8.09e-05 0.01% Castro::clean_state() 62 7.767e-05 7.767e-05 7.767e-05 0.01% AmrLevel::checkPoint() 3 7.308e-05 7.308e-05 7.308e-05 0.01% MLMG::mgVcycle_down::1 81 7.168e-05 7.168e-05 7.168e-05 0.01% MLMG::solve() 11 6.999e-05 6.999e-05 6.999e-05 0.01% MLMG::mgVcycle_down::2 81 6.939e-05 6.939e-05 6.939e-05 0.01% MLMG::mgVcycle_down::3 81 6.463e-05 6.463e-05 6.463e-05 0.01% MLMG::mgVcycle_down::4 81 6.442e-05 6.442e-05 6.442e-05 0.01% Castro::initialize_do_advance() 10 6.111e-05 6.111e-05 6.111e-05 0.01% Castro::advance() 10 6.103e-05 6.103e-05 6.103e-05 0.01% MLMG::mgVcycle_up::4 81 5.777e-05 5.777e-05 5.777e-05 0.01% Castro::finalize_advance() 10 5.295e-05 5.295e-05 5.295e-05 0.01% MLMG::mgVcycle_up::0 81 5.061e-05 5.061e-05 5.061e-05 0.01% MLCellLinOp::solutionResidual() 92 4.985e-05 4.985e-05 4.985e-05 0.01% MLMG::mgVcycle_up::1 81 4.853e-05 4.853e-05 4.853e-05 0.01% MLMG::mgVcycle_up::3 81 4.833e-05 4.833e-05 4.833e-05 0.01% MLMG::mgVcycle_up::2 81 4.697e-05 4.697e-05 4.697e-05 0.01% Castro::finalize_do_advance() 10 4.04e-05 4.04e-05 4.04e-05 0.00% Castro::swap_state_time_levels() 10 3.97e-05 3.97e-05 3.97e-05 0.00% StateData::define() 4 3.816e-05 3.816e-05 3.816e-05 0.00% Castro::construct_new_source() 50 3.385e-05 3.385e-05 3.385e-05 0.00% MLMG::computeResidual() 81 3.278e-05 3.278e-05 3.278e-05 0.00% Castro::enforce_consistent_e() 1 3.209e-05 3.209e-05 3.209e-05 0.00% MLMG::mgVcycle_bottom 81 3.085e-05 3.085e-05 3.085e-05 0.00% Gravity::actual_multilevel_solve() 1 3.052e-05 3.052e-05 3.052e-05 0.00% FillPatchSingleLevel 41 2.753e-05 2.753e-05 2.753e-05 0.00% Amr::defBaseLevel() 1 2.631e-05 2.631e-05 2.631e-05 0.00% Castro::initMFs() 1 2.629e-05 2.629e-05 2.629e-05 0.00% makeSFC 55 2.561e-05 2.561e-05 2.561e-05 0.00% MLPoisson::define() 11 2.505e-05 2.505e-05 2.505e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.47e-05 2.47e-05 2.47e-05 0.00% Amr::writeSmallPlotFile() 1 2.465e-05 2.465e-05 2.465e-05 0.00% MLLinOp::define() 11 2.364e-05 2.364e-05 2.364e-05 0.00% Amr::FinalizeInit() 1 2.221e-05 2.221e-05 2.221e-05 0.00% Castro::buildMetrics() 1 2.21e-05 2.21e-05 2.21e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.806e-05 1.806e-05 1.806e-05 0.00% Castro::construct_old_source() 50 1.79e-05 1.79e-05 1.79e-05 0.00% Castro::do_new_sources() 10 1.71e-05 1.71e-05 1.71e-05 0.00% Castro::do_old_sources() 10 1.489e-05 1.489e-05 1.489e-05 0.00% DistributionMapping::Distribute() 56 1.43e-05 1.43e-05 1.43e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.331e-05 1.331e-05 1.331e-05 0.00% Castro::apply_source_to_state() 20 1.142e-05 1.142e-05 1.142e-05 0.00% Castro::check_for_nan() 20 1.098e-05 1.098e-05 1.098e-05 0.00% Castro::construct_old_gravity() 10 9.852e-06 9.852e-06 9.852e-06 0.00% Amr::initSubcycle() 1 8.905e-06 8.905e-06 8.905e-06 0.00% Castro::post_timestep() 10 8.785e-06 8.785e-06 8.785e-06 0.00% Gravity::swapTimeLevels() 10 8.436e-06 8.436e-06 8.436e-06 0.00% MLPoisson::prepareForSolve() 11 7.229e-06 7.229e-06 7.229e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.053e-06 7.053e-06 7.053e-06 0.00% Amr::InitializeInit() 1 6.612e-06 6.612e-06 6.612e-06 0.00% MLMG::computeMLResidual() 11 6.105e-06 6.105e-06 6.105e-06 0.00% Castro::computeNewDt() 9 6.102e-06 6.102e-06 6.102e-06 0.00% MLMG::getGradSolution() 11 5.846e-06 5.846e-06 5.846e-06 0.00% MLMG::buildFineMask() 11 5.226e-06 5.226e-06 5.226e-06 0.00% MLMG::MLResNormInf() 11 4.736e-06 4.736e-06 4.736e-06 0.00% Castro::create_source_corrector() 10 4.642e-06 4.642e-06 4.642e-06 0.00% AmrLevel::checkPointPost() 3 4.346e-06 4.346e-06 4.346e-06 0.00% Gravity::set_mass_offset() 11 4.041e-06 4.041e-06 4.041e-06 0.00% Castro::post_init() 1 3.651e-06 3.651e-06 3.651e-06 0.00% Castro::retry_advance_ctu() 10 3.629e-06 3.629e-06 3.629e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.925e-06 2.925e-06 2.925e-06 0.00% Castro::FluxRegCrseInit 10 2.576e-06 2.576e-06 2.576e-06 0.00% Castro::FluxRegFineAdd() 10 2.545e-06 2.545e-06 2.545e-06 0.00% Amr::init() 1 2.492e-06 2.492e-06 2.492e-06 0.00% AmrLevel::checkPointPre() 3 2.374e-06 2.374e-06 2.374e-06 0.00% Castro::computeInitialDt() 2 2.298e-06 2.298e-06 2.298e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.708e-06 1.708e-06 1.708e-06 0.00% Castro::post_regrid() 1 1.172e-06 1.172e-06 1.172e-06 0.00% Amr::initialInit() 1 9.79e-07 9.79e-07 9.79e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8309 0.8309 0.8309 100.00% Amr::coarseTimeStep() 10 0.6711 0.6711 0.6711 80.76% Amr::timeStep() 10 0.5726 0.5726 0.5726 68.92% Castro::advance() 10 0.5655 0.5655 0.5655 68.06% Castro::subcycle_advance_ctu() 10 0.554 0.554 0.554 66.67% Castro::do_advance_ctu() 10 0.5537 0.5537 0.5537 66.64% Gravity::solve_phi_with_mlmg() 11 0.3031 0.3031 0.3031 36.48% Gravity::actual_solve_with_mlmg() 11 0.2946 0.2946 0.2946 35.46% Castro::construct_new_gravity() 10 0.2787 0.2787 0.2787 33.54% MLMG::solve() 11 0.2728 0.2728 0.2728 32.84% Gravity::solve_for_phi() 10 0.2632 0.2632 0.2632 31.68% MLMG::oneIter() 81 0.2586 0.2586 0.2586 31.12% MLMG::mgVcycle() 81 0.2569 0.2569 0.2569 30.92% Castro::construct_ctu_hydro_source() 10 0.1983 0.1983 0.1983 23.87% VisMF::Write(FabArray) 11 0.198 0.198 0.198 23.82% Amr::checkPoint() 3 0.1468 0.1468 0.1468 17.66% AmrLevel::checkPoint() 3 0.1415 0.1415 0.1415 17.04% StateData::checkPoint() 12 0.1415 0.1415 0.1415 17.03% MLCellLinOp::smooth() 1620 0.1322 0.1322 0.1322 15.91% Amr::init() 1 0.1297 0.1297 0.1297 15.61% MLCellLinOp::applyBC() 4379 0.09294 0.09294 0.09294 11.19% MLMG::mgVcycle_bottom 81 0.07878 0.07878 0.07878 9.48% MLMG::actualBottomSolve() 81 0.07875 0.07875 0.07875 9.48% MLCGSolver::bicgstab 81 0.07795 0.07795 0.07795 9.38% MLPoisson::Fsmooth() 3240 0.06189 0.06189 0.06189 7.45% Amr::writePlotFile() 2 0.05946 0.05946 0.05946 7.16% Amr::initialInit() 1 0.04796 0.04796 0.04796 5.77% Castro::clean_state() 62 0.04502 0.04502 0.04502 5.42% Amr::FinalizeInit() 1 0.04376 0.04376 0.04376 5.27% FillPatchIterator::Initialize 41 0.04254 0.04254 0.04254 5.12% Castro::post_init() 1 0.04234 0.04234 0.04234 5.10% FillPatchSingleLevel 41 0.04096 0.04096 0.04096 4.93% Gravity::multilevel_solve_for_new_phi() 1 0.04038 0.04038 0.04038 4.86% Gravity::actual_multilevel_solve() 1 0.04036 0.04036 0.04036 4.86% StateDataPhysBCFunct::() 41 0.03699 0.03699 0.03699 4.45% MLCellLinOp::apply() 1128 0.035 0.035 0.035 4.21% MLMG::mgVcycle_down::0 81 0.03456 0.03456 0.03456 4.16% MLMG::mgVcycle_up::0 81 0.02952 0.02952 0.02952 3.55% StateData::FillBoundary(geom) 328 0.02396 0.02396 0.02396 2.88% MultiFab::Dot() 1100 0.02146 0.02146 0.02146 2.58% Castro::initialize_do_advance() 10 0.02108 0.02108 0.02108 2.54% MLCellLinOp::correctionResidual() 486 0.02059 0.02059 0.02059 2.48% Castro::computeTemp() 63 0.02035 0.02035 0.02035 2.45% MLMG:computeResOfCorrection() 405 0.01776 0.01776 0.01776 2.14% MLPoisson::define() 11 0.01755 0.01755 0.01755 2.11% Gravity::get_new_grav_vector() 11 0.01718 0.01718 0.01718 2.07% MLMG::mgVcycle_down::1 81 0.01707 0.01707 0.01707 2.05% MLMG::mgVcycle_down::2 81 0.01664 0.01664 0.01664 2.00% MLMG::mgVcycle_down::3 81 0.01579 0.01579 0.01579 1.90% FabArray::FillBoundary() 3974 0.0153 0.0153 0.0153 1.84% MLMG::mgVcycle_down::4 81 0.01504 0.01504 0.01504 1.81% Castro::construct_old_gravity() 10 0.01463 0.01463 0.01463 1.76% Gravity::get_old_grav_vector() 10 0.01462 0.01462 0.01462 1.76% FillBoundary_nowait() 3974 0.01447 0.01447 0.01447 1.74% Castro::normalize_species() 62 0.01438 0.01438 0.01438 1.73% CGSolver::sxay() 1566 0.01416 0.01416 0.01416 1.70% FabArray::setVal() 1135 0.01385 0.01385 0.01385 1.67% MultiFab::LinComb() 1566 0.01379 0.01379 0.01379 1.66% FabArray::ParallelCopy() 851 0.01367 0.01367 0.01367 1.65% FabArray::ParallelCopy_nowait() 851 0.0134 0.0134 0.0134 1.61% MLCGSolver::ParallelAllReduce 1495 0.01281 0.01281 0.01281 1.54% MLMG::mgVcycle_up::2 81 0.01278 0.01278 0.01278 1.54% MLMG::mgVcycle_up::1 81 0.01255 0.01255 0.01255 1.51% MLCellLinOp::defineAuxData() 11 0.01241 0.01241 0.01241 1.49% Castro::expand_state() 10 0.01219 0.01219 0.01219 1.47% MLMG::mgVcycle_up::3 81 0.01209 0.01209 0.01209 1.46% MLMG::addInterpCorrection() 405 0.01201 0.01201 0.01201 1.45% MLMG::mgVcycle_up::4 81 0.01195 0.01195 0.01195 1.44% Castro::do_new_sources() 10 0.01182 0.01182 0.01182 1.42% amrex::average_down 405 0.0115 0.0115 0.0115 1.38% Castro::initialize_advance() 10 0.01143 0.01143 0.01143 1.38% MLPoisson::Fapply() 1128 0.0113 0.0113 0.0113 1.36% Castro::do_old_sources() 10 0.01051 0.01051 0.01051 1.26% Castro::enforce_min_density() 62 0.009554 0.009554 0.009554 1.15% Gravity::fill_multipole_BCs() 11 0.008239 0.008239 0.008239 0.99% MLCellLinOp::solutionResidual() 92 0.007002 0.007002 0.007002 0.84% Castro::post_timestep() 10 0.006869 0.006869 0.006869 0.83% MultiFab::Xpay() 578 0.00639 0.00639 0.00639 0.77% MLMG::computeResidual() 81 0.006026 0.006026 0.006026 0.73% Castro::reset_internal_energy(MultiFab) 63 0.005896 0.005896 0.005896 0.71% Castro::estTimeStep() 21 0.005338 0.005338 0.005338 0.64% MLMG::prepareForSolve() 11 0.004989 0.004989 0.004989 0.60% MLCellLinOp::defineBC() 11 0.004879 0.004879 0.004879 0.59% BndryData::define() 11 0.004602 0.004602 0.004602 0.55% Amr::InitializeInit() 1 0.004195 0.004195 0.004195 0.50% Amr::defBaseLevel() 1 0.004189 0.004189 0.004189 0.50% Castro::initData() 1 0.003691 0.003691 0.003691 0.44% Castro::construct_new_source() 50 0.003306 0.003306 0.003306 0.40% Castro::construct_new_gravity_source() 10 0.003272 0.003272 0.003272 0.39% Castro::construct_old_source() 50 0.002661 0.002661 0.002661 0.32% Castro::construct_old_gravity_source() 10 0.002643 0.002643 0.002643 0.32% Castro::computeNewDt() 9 0.00248 0.00248 0.00248 0.30% MLMG::ResNormInf() 92 0.001904 0.001904 0.001904 0.23% Castro::apply_source_to_state() 20 0.001815 0.001815 0.001815 0.22% MultiFab::Saxpy() 20 0.001803 0.001803 0.001803 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00159 0.00159 0.00159 0.19% MLCellLinOp::setLevelBC() 11 0.001513 0.001513 0.001513 0.18% Castro::enforce_speed_limit() 62 0.001503 0.001503 0.001503 0.18% FabArrayBase::getCPC() 1313 0.001432 0.001432 0.001432 0.17% Castro::reset_internal_energy(Fab) 504 0.001406 0.001406 0.001406 0.17% MLMG::getGradSolution() 11 0.001395 0.001395 0.001395 0.17% MLCellLinOp::compGrad() 11 0.001389 0.001389 0.001389 0.17% FabArray::mult() 43 0.001301 0.001301 0.001301 0.16% FabArray::setDomainBndry() 41 0.001284 0.001284 0.001284 0.15% Castro::post_regrid() 1 0.001201 0.001201 0.001201 0.14% Castro::check_for_nan() 20 0.001191 0.001191 0.001191 0.14% MultiFab::contains_nan() 20 0.00118 0.00118 0.00118 0.14% MLPoisson::prepareForSolve() 11 0.001139 0.001139 0.001139 0.14% MLCellLinOp::prepareForSolve() 11 0.001132 0.001132 0.001132 0.14% MLMG::computeMLResidual() 11 0.001015 0.001015 0.001015 0.12% Gravity::update_max_rhs() 11 0.0007917 0.0007917 0.0007917 0.10% Castro::computeInitialDt() 2 0.0006779 0.0006779 0.0006779 0.08% FabArrayBase::getFB() 3974 0.0006763 0.0006763 0.0006763 0.08% FabArrayBase::CPC::define() 454 0.0006594 0.0006594 0.0006594 0.08% Amr::InitAmr() 1 0.0004911 0.0004911 0.0004911 0.06% Gravity::swapTimeLevels() 10 0.0004362 0.0004362 0.0004362 0.05% Castro::Castro() 1 0.000422 0.000422 0.000422 0.05% MLMG::MLResNormInf() 11 0.0002544 0.0002544 0.0002544 0.03% MultiFab::max() 11 0.000252 0.000252 0.000252 0.03% MultiFab::Copy() 11 0.0002504 0.0002504 0.0002504 0.03% MLLinOp::define() 11 0.0002346 0.0002346 0.0002346 0.03% MLLinOp::defineGrids() 11 0.000211 0.000211 0.000211 0.03% MLMG::MLRhsNormInf() 11 0.0002019 0.0002019 0.0002019 0.02% Castro::buildMetrics() 1 0.0001462 0.0001462 0.0001462 0.02% FabArrayBase::FB::FB() 56 8.166e-05 8.166e-05 8.166e-05 0.01% Castro::finalize_advance() 10 5.807e-05 5.807e-05 5.807e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.22e-05 5.22e-05 5.22e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.521e-05 4.521e-05 4.521e-05 0.01% Castro::finalize_do_advance() 10 4.04e-05 4.04e-05 4.04e-05 0.00% Castro::swap_state_time_levels() 10 3.97e-05 3.97e-05 3.97e-05 0.00% makeSFC 55 3.889e-05 3.889e-05 3.889e-05 0.00% StateData::define() 4 3.816e-05 3.816e-05 3.816e-05 0.00% Castro::enforce_consistent_e() 1 3.209e-05 3.209e-05 3.209e-05 0.00% Castro::initMFs() 1 2.629e-05 2.629e-05 2.629e-05 0.00% Amr::writeSmallPlotFile() 1 2.465e-05 2.465e-05 2.465e-05 0.00% DistributionMapping::Distribute() 56 1.43e-05 1.43e-05 1.43e-05 0.00% Amr::initSubcycle() 1 8.905e-06 8.905e-06 8.905e-06 0.00% MLMG::buildFineMask() 11 5.226e-06 5.226e-06 5.226e-06 0.00% Castro::create_source_corrector() 10 4.642e-06 4.642e-06 4.642e-06 0.00% AmrLevel::checkPointPost() 3 4.346e-06 4.346e-06 4.346e-06 0.00% Gravity::set_mass_offset() 11 4.041e-06 4.041e-06 4.041e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.94e-06 3.94e-06 3.94e-06 0.00% Castro::retry_advance_ctu() 10 3.629e-06 3.629e-06 3.629e-06 0.00% Castro::FluxRegCrseInit 10 2.576e-06 2.576e-06 2.576e-06 0.00% Castro::FluxRegFineAdd() 10 2.545e-06 2.545e-06 2.545e-06 0.00% AmrLevel::checkPointPre() 3 2.374e-06 2.374e-06 2.374e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.708e-06 1.708e-06 1.708e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.07-19-g48702b48836d) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.07-19-g48702b48836d) initialized Starting run at 08:51:47 UTC on 2022-08-01. Successfully read inputs file ... Castro git describe: 22.07-14-ga2a554e52 AMReX git describe: 22.07-19-g48702b488 Microphysics git describe: 22.07-16-g593f02b0 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.471125739 Restart time = 0.095908121 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051623118 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049205607 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.049328322 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.048239968 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065981096 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.035136076 seconds Ending run at 08:51:47 UTC on 2022-08-01. Run time = 0.396495383 Run time without initialization = 0.30001812 Average number of zones advanced per microsecond: 4.369 Average number of zones advanced per microsecond per rank: 4.369 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3965 ... 0.3965 ... 0.3965 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0766 0.0766 0.0766 19.32% Amr::restart() 1 0.05067 0.05067 0.05067 12.78% VisMF::Read() 3 0.04106 0.04106 0.04106 10.36% MLCellLinOp::applyBC() 1946 0.03404 0.03404 0.03404 8.58% VisMF::Write(FabArray) 1 0.03355 0.03355 0.03355 8.46% MLPoisson::Fsmooth() 1440 0.02645 0.02645 0.02645 6.67% StateData::FillBoundary(geom) 160 0.01157 0.01157 0.01157 2.92% MLCGSolver::bicgstab 36 0.009993 0.009993 0.009993 2.52% MultiFab::Dot() 484 0.009221 0.009221 0.009221 2.33% FabArray::setVal() 537 0.006594 0.006594 0.006594 1.66% Castro::computeTemp() 30 0.006533 0.006533 0.006533 1.65% FillBoundary_nowait() 1766 0.006129 0.006129 0.006129 1.55% MLCellLinOp::defineAuxData() 6 0.005981 0.005981 0.005981 1.51% StateDataPhysBCFunct::() 20 0.005964 0.005964 0.005964 1.50% MultiFab::LinComb() 690 0.005935 0.005935 0.005935 1.50% FabArray::ParallelCopy_nowait() 380 0.005818 0.005818 0.005818 1.47% Castro::normalize_species() 30 0.005392 0.005392 0.005392 1.36% MLPoisson::Fapply() 500 0.004927 0.004927 0.004927 1.24% Gravity::fill_multipole_BCs() 6 0.004636 0.004636 0.004636 1.17% Castro::enforce_min_density() 30 0.004259 0.004259 0.004259 1.07% MLMG::addInterpCorrection() 180 0.003097 0.003097 0.003097 0.78% amrex::average_down 180 0.002881 0.002881 0.002881 0.73% MultiFab::Xpay() 258 0.002816 0.002816 0.002816 0.71% Castro::estTimeStep() 10 0.002245 0.002245 0.002245 0.57% Castro::reset_internal_energy(MultiFab) 30 0.002194 0.002194 0.002194 0.55% Castro::do_advance_ctu() 5 0.00207 0.00207 0.00207 0.52% BndryData::define() 6 0.002024 0.002024 0.002024 0.51% Amr::writePlotFile() 1 0.001687 0.001687 0.001687 0.43% Castro::construct_new_gravity_source() 5 0.001669 0.001669 0.001669 0.42% Castro::construct_old_gravity_source() 5 0.001351 0.001351 0.001351 0.34% MultiFab::Saxpy() 10 0.0009199 0.0009199 0.0009199 0.23% Gravity::get_old_grav_vector() 5 0.0009007 0.0009007 0.0009007 0.23% Castro::enforce_speed_limit() 30 0.0008834 0.0008834 0.0008834 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008659 0.0008659 0.0008659 0.22% Castro::expand_state() 5 0.000864 0.000864 0.000864 0.22% Gravity::get_new_grav_vector() 5 0.0008571 0.0008571 0.0008571 0.22% MLMG::ResNormInf() 42 0.0008463 0.0008463 0.0008463 0.21% MLCellLinOp::setLevelBC() 6 0.0007962 0.0007962 0.0007962 0.20% MLMG::oneIter() 36 0.0007336 0.0007336 0.0007336 0.19% Gravity::actual_solve_with_mlmg() 6 0.0007026 0.0007026 0.0007026 0.18% Castro::reset_internal_energy(Fab) 240 0.0006521 0.0006521 0.0006521 0.16% FabArray::mult() 22 0.0006457 0.0006457 0.0006457 0.16% FabArray::setDomainBndry() 20 0.0006457 0.0006457 0.0006457 0.16% MLCellLinOp::prepareForSolve() 6 0.000611 0.000611 0.000611 0.15% MultiFab::contains_nan() 10 0.0005985 0.0005985 0.0005985 0.15% MLMG::prepareForSolve() 6 0.0005461 0.0005461 0.0005461 0.14% MLCellLinOp::compGrad() 6 0.0004864 0.0004864 0.0004864 0.12% MLCellLinOp::smooth() 720 0.0004673 0.0004673 0.0004673 0.12% Amr::InitAmr() 1 0.0004072 0.0004072 0.0004072 0.10% FabArrayBase::CPC::define() 244 0.0003932 0.0003932 0.0003932 0.10% FabArrayBase::getCPC() 632 0.0003828 0.0003828 0.0003828 0.10% FabArray::FillBoundary() 1766 0.0003822 0.0003822 0.0003822 0.10% main() 1 0.0003307 0.0003307 0.0003307 0.08% FabArrayBase::getFB() 1766 0.0002551 0.0002551 0.0002551 0.06% Gravity::update_max_rhs() 6 0.0002258 0.0002258 0.0002258 0.06% Gravity::solve_for_phi() 5 0.0002161 0.0002161 0.0002161 0.05% MLCellLinOp::apply() 500 0.0001886 0.0001886 0.0001886 0.05% CGSolver::sxay() 690 0.0001779 0.0001779 0.0001779 0.04% Amr::coarseTimeStep() 5 0.0001739 0.0001739 0.0001739 0.04% Castro::subcycle_advance_ctu() 5 0.0001471 0.0001471 0.0001471 0.04% MLCellLinOp::defineBC() 6 0.0001461 0.0001461 0.0001461 0.04% FillPatchIterator::Initialize 20 0.0001371 0.0001371 0.0001371 0.03% MultiFab::Copy() 6 0.0001356 0.0001356 0.0001356 0.03% MultiFab::max() 6 0.0001347 0.0001347 0.0001347 0.03% FabArray::ParallelCopy() 380 0.0001297 0.0001297 0.0001297 0.03% MLCGSolver::ParallelAllReduce 659 0.0001242 0.0001242 0.0001242 0.03% Amr::timeStep() 5 0.0001084 0.0001084 0.0001084 0.03% Castro::construct_new_gravity() 5 0.0001043 0.0001043 0.0001043 0.03% MLMG::MLRhsNormInf() 6 0.0001038 0.0001038 0.0001038 0.03% MLMG::mgVcycle() 36 9.811e-05 9.811e-05 9.811e-05 0.02% Castro::create_source_corrector() 5 9.35e-05 9.35e-05 9.35e-05 0.02% MLCellLinOp::correctionResidual() 216 9.33e-05 9.33e-05 9.33e-05 0.02% MLLinOp::defineGrids() 6 9.321e-05 9.321e-05 9.321e-05 0.02% AmrLevel::restart() 1 9.067e-05 9.067e-05 9.067e-05 0.02% StateData::restartDoit() 4 6.788e-05 6.788e-05 6.788e-05 0.02% MLMG:computeResOfCorrection() 180 5.594e-05 5.594e-05 5.594e-05 0.01% FabArrayBase::FB::FB() 26 5.466e-05 5.466e-05 5.466e-05 0.01% MLMG::actualBottomSolve() 36 4.522e-05 4.522e-05 4.522e-05 0.01% Castro::initialize_do_advance() 5 4.395e-05 4.395e-05 4.395e-05 0.01% Castro::clean_state() 30 3.949e-05 3.949e-05 3.949e-05 0.01% Castro::initialize_advance() 5 3.921e-05 3.921e-05 3.921e-05 0.01% Castro::construct_old_source() 25 3.887e-05 3.887e-05 3.887e-05 0.01% MLMG::mgVcycle_down::0 36 3.796e-05 3.796e-05 3.796e-05 0.01% Castro::construct_new_source() 25 3.638e-05 3.638e-05 3.638e-05 0.01% MLMG::solve() 6 3.493e-05 3.493e-05 3.493e-05 0.01% MLMG::mgVcycle_down::1 36 3.426e-05 3.426e-05 3.426e-05 0.01% MLMG::mgVcycle_down::2 36 3.284e-05 3.284e-05 3.284e-05 0.01% Castro::finalize_advance() 5 3.217e-05 3.217e-05 3.217e-05 0.01% MLMG::mgVcycle_down::4 36 3.163e-05 3.163e-05 3.163e-05 0.01% Castro::post_restart() 1 3.106e-05 3.106e-05 3.106e-05 0.01% Castro::buildMetrics() 1 3.1e-05 3.1e-05 3.1e-05 0.01% MLMG::mgVcycle_down::3 36 3.034e-05 3.034e-05 3.034e-05 0.01% Gravity::actual_multilevel_solve() 1 2.912e-05 2.912e-05 2.912e-05 0.01% Castro::advance() 5 2.901e-05 2.901e-05 2.901e-05 0.01% Amr::writeSmallPlotFile() 1 2.777e-05 2.777e-05 2.777e-05 0.01% MLMG::mgVcycle_up::4 36 2.723e-05 2.723e-05 2.723e-05 0.01% Castro::swap_state_time_levels() 5 2.661e-05 2.661e-05 2.661e-05 0.01% Castro::initMFs() 1 2.548e-05 2.548e-05 2.548e-05 0.01% MLCellLinOp::solutionResidual() 42 2.43e-05 2.43e-05 2.43e-05 0.01% MLMG::mgVcycle_up::0 36 2.4e-05 2.4e-05 2.4e-05 0.01% MLMG::mgVcycle_up::2 36 2.273e-05 2.273e-05 2.273e-05 0.01% MLMG::mgVcycle_up::3 36 2.25e-05 2.25e-05 2.25e-05 0.01% MLLinOp::define() 6 2.066e-05 2.066e-05 2.066e-05 0.01% MLMG::mgVcycle_up::1 36 2.036e-05 2.036e-05 2.036e-05 0.01% Castro::finalize_do_advance() 5 1.867e-05 1.867e-05 1.867e-05 0.00% MLPoisson::define() 6 1.696e-05 1.696e-05 1.696e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.69e-05 1.69e-05 1.69e-05 0.00% MLMG::computeResidual() 36 1.545e-05 1.545e-05 1.545e-05 0.00% makeSFC 30 1.424e-05 1.424e-05 1.424e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.407e-05 1.407e-05 1.407e-05 0.00% MLMG::mgVcycle_bottom 36 1.394e-05 1.394e-05 1.394e-05 0.00% FillPatchSingleLevel 20 1.3e-05 1.3e-05 1.3e-05 0.00% DistributionMapping::Distribute() 31 9.589e-06 9.589e-06 9.589e-06 0.00% Castro::do_new_sources() 5 9.263e-06 9.263e-06 9.263e-06 0.00% Castro::do_old_sources() 5 8.304e-06 8.304e-06 8.304e-06 0.00% Castro::check_for_nan() 10 8.158e-06 8.158e-06 8.158e-06 0.00% Amr::initSubcycle() 1 8.145e-06 8.145e-06 8.145e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.558e-06 7.558e-06 7.558e-06 0.00% Castro::construct_old_gravity() 5 5.418e-06 5.418e-06 5.418e-06 0.00% Castro::apply_source_to_state() 10 5.392e-06 5.392e-06 5.392e-06 0.00% Castro::post_timestep() 5 4.914e-06 4.914e-06 4.914e-06 0.00% MLPoisson::prepareForSolve() 6 4.25e-06 4.25e-06 4.25e-06 0.00% Gravity::swapTimeLevels() 5 3.993e-06 3.993e-06 3.993e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.502e-06 3.502e-06 3.502e-06 0.00% MLMG::buildFineMask() 6 3.285e-06 3.285e-06 3.285e-06 0.00% Castro::computeNewDt() 5 3.157e-06 3.157e-06 3.157e-06 0.00% MLMG::computeMLResidual() 6 3.006e-06 3.006e-06 3.006e-06 0.00% MLMG::getGradSolution() 6 2.9e-06 2.9e-06 2.9e-06 0.00% MLMG::MLResNormInf() 6 2.39e-06 2.39e-06 2.39e-06 0.00% Gravity::set_mass_offset() 6 2.031e-06 2.031e-06 2.031e-06 0.00% Castro::retry_advance_ctu() 5 2.026e-06 2.026e-06 2.026e-06 0.00% Castro::FluxRegCrseInit 5 1.747e-06 1.747e-06 1.747e-06 0.00% AmrLevel::AmrLevel() 1 1.596e-06 1.596e-06 1.596e-06 0.00% Amr::init() 1 1.129e-06 1.129e-06 1.129e-06 0.00% Castro::FluxRegFineAdd() 5 1.082e-06 1.082e-06 1.082e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.012e-06 1.012e-06 1.012e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3965 0.3965 0.3965 100.00% Amr::coarseTimeStep() 5 0.2645 0.2645 0.2645 66.72% Amr::timeStep() 5 0.2631 0.2631 0.2631 66.35% Castro::advance() 5 0.2596 0.2596 0.2596 65.46% Castro::subcycle_advance_ctu() 5 0.2534 0.2534 0.2534 63.90% Castro::do_advance_ctu() 5 0.2532 0.2532 0.2532 63.86% Castro::construct_new_gravity() 5 0.1399 0.1399 0.1399 35.29% Gravity::solve_phi_with_mlmg() 6 0.1357 0.1357 0.1357 34.22% Gravity::solve_for_phi() 5 0.1322 0.1322 0.1322 33.35% Gravity::actual_solve_with_mlmg() 6 0.1309 0.1309 0.1309 33.02% MLMG::solve() 6 0.1191 0.1191 0.1191 30.04% MLMG::oneIter() 36 0.1123 0.1123 0.1123 28.31% MLMG::mgVcycle() 36 0.1115 0.1115 0.1115 28.13% Amr::init() 1 0.09595 0.09595 0.09595 24.20% Amr::restart() 1 0.09595 0.09595 0.09595 24.20% Castro::construct_ctu_hydro_source() 5 0.07661 0.07661 0.07661 19.32% MLCellLinOp::smooth() 720 0.0572 0.0572 0.0572 14.43% AmrLevel::restart() 1 0.04128 0.04128 0.04128 10.41% StateData::restartDoit() 4 0.04118 0.04118 0.04118 10.38% VisMF::Read() 3 0.04106 0.04106 0.04106 10.36% MLCellLinOp::applyBC() 1946 0.04086 0.04086 0.04086 10.30% Amr::writePlotFile() 1 0.03524 0.03524 0.03524 8.89% MLMG::mgVcycle_bottom 36 0.03412 0.03412 0.03412 8.60% MLMG::actualBottomSolve() 36 0.03411 0.03411 0.03411 8.60% MLCGSolver::bicgstab 36 0.03376 0.03376 0.03376 8.51% VisMF::Write(FabArray) 1 0.03355 0.03355 0.03355 8.46% MLPoisson::Fsmooth() 1440 0.02645 0.02645 0.02645 6.67% FillPatchIterator::Initialize 20 0.02033 0.02033 0.02033 5.13% Castro::clean_state() 30 0.01995 0.01995 0.01995 5.03% FillPatchSingleLevel 20 0.01954 0.01954 0.01954 4.93% StateDataPhysBCFunct::() 20 0.01754 0.01754 0.01754 4.42% MLCellLinOp::apply() 500 0.01543 0.01543 0.01543 3.89% MLMG::mgVcycle_down::0 36 0.01504 0.01504 0.01504 3.79% MLMG::mgVcycle_up::0 36 0.01284 0.01284 0.01284 3.24% StateData::FillBoundary(geom) 160 0.01157 0.01157 0.01157 2.92% Castro::initialize_do_advance() 5 0.01005 0.01005 0.01005 2.53% MLPoisson::define() 6 0.009565 0.009565 0.009565 2.41% Castro::computeTemp() 30 0.00938 0.00938 0.00938 2.37% MultiFab::Dot() 484 0.009221 0.009221 0.009221 2.33% MLCellLinOp::correctionResidual() 216 0.009002 0.009002 0.009002 2.27% MLMG:computeResOfCorrection() 180 0.007772 0.007772 0.007772 1.96% Gravity::get_new_grav_vector() 5 0.007579 0.007579 0.007579 1.91% MLMG::mgVcycle_down::1 36 0.007445 0.007445 0.007445 1.88% Castro::construct_old_gravity() 5 0.007409 0.007409 0.007409 1.87% Gravity::get_old_grav_vector() 5 0.007403 0.007403 0.007403 1.87% MLMG::mgVcycle_down::2 36 0.007214 0.007214 0.007214 1.82% MLMG::mgVcycle_down::3 36 0.006833 0.006833 0.006833 1.72% FabArray::FillBoundary() 1766 0.006821 0.006821 0.006821 1.72% MLCellLinOp::defineAuxData() 6 0.006706 0.006706 0.006706 1.69% FabArray::setVal() 537 0.006594 0.006594 0.006594 1.66% MLMG::mgVcycle_down::4 36 0.00656 0.00656 0.00656 1.65% FillBoundary_nowait() 1766 0.006439 0.006439 0.006439 1.62% FabArray::ParallelCopy() 380 0.006337 0.006337 0.006337 1.60% FabArray::ParallelCopy_nowait() 380 0.006208 0.006208 0.006208 1.57% CGSolver::sxay() 690 0.006113 0.006113 0.006113 1.54% Castro::initialize_advance() 5 0.006107 0.006107 0.006107 1.54% Castro::expand_state() 5 0.006059 0.006059 0.006059 1.53% MultiFab::LinComb() 690 0.005935 0.005935 0.005935 1.50% MLCGSolver::ParallelAllReduce 659 0.00555 0.00555 0.00555 1.40% MLMG::mgVcycle_up::2 36 0.005529 0.005529 0.005529 1.39% MLMG::mgVcycle_up::1 36 0.005435 0.005435 0.005435 1.37% Castro::normalize_species() 30 0.005392 0.005392 0.005392 1.36% MLMG::addInterpCorrection() 180 0.005245 0.005245 0.005245 1.32% MLMG::mgVcycle_up::3 36 0.005236 0.005236 0.005236 1.32% Castro::do_old_sources() 5 0.005229 0.005229 0.005229 1.32% MLMG::mgVcycle_up::4 36 0.005185 0.005185 0.005185 1.31% Castro::do_new_sources() 5 0.00508 0.00508 0.00508 1.28% amrex::average_down 180 0.005077 0.005077 0.005077 1.28% MLPoisson::Fapply() 500 0.004927 0.004927 0.004927 1.24% Gravity::fill_multipole_BCs() 6 0.004636 0.004636 0.004636 1.17% Castro::enforce_min_density() 30 0.004259 0.004259 0.004259 1.07% Castro::post_restart() 1 0.003835 0.003835 0.003835 0.97% Gravity::multilevel_solve_for_new_phi() 1 0.003711 0.003711 0.003711 0.94% Gravity::actual_multilevel_solve() 1 0.003694 0.003694 0.003694 0.93% Castro::post_timestep() 5 0.003422 0.003422 0.003422 0.86% MLCellLinOp::solutionResidual() 42 0.003185 0.003185 0.003185 0.80% Castro::reset_internal_energy(MultiFab) 30 0.002846 0.002846 0.002846 0.72% MultiFab::Xpay() 258 0.002816 0.002816 0.002816 0.71% MLCellLinOp::defineBC() 6 0.002698 0.002698 0.002698 0.68% MLMG::prepareForSolve() 6 0.002671 0.002671 0.002671 0.67% MLMG::computeResidual() 36 0.002646 0.002646 0.002646 0.67% BndryData::define() 6 0.002552 0.002552 0.002552 0.64% Castro::estTimeStep() 10 0.002245 0.002245 0.002245 0.57% Castro::construct_new_source() 25 0.001706 0.001706 0.001706 0.43% Castro::construct_new_gravity_source() 5 0.001669 0.001669 0.001669 0.42% Castro::construct_old_source() 25 0.00139 0.00139 0.00139 0.35% Castro::construct_old_gravity_source() 5 0.001351 0.001351 0.001351 0.34% Castro::computeNewDt() 5 0.001281 0.001281 0.001281 0.32% Castro::apply_source_to_state() 10 0.0009253 0.0009253 0.0009253 0.23% MultiFab::Saxpy() 10 0.0009199 0.0009199 0.0009199 0.23% Castro::enforce_speed_limit() 30 0.0008834 0.0008834 0.0008834 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008659 0.0008659 0.0008659 0.22% MLMG::ResNormInf() 42 0.0008463 0.0008463 0.0008463 0.21% MLCellLinOp::setLevelBC() 6 0.0007962 0.0007962 0.0007962 0.20% FabArrayBase::getCPC() 632 0.000776 0.000776 0.000776 0.20% MLMG::getGradSolution() 6 0.0007497 0.0007497 0.0007497 0.19% MLCellLinOp::compGrad() 6 0.0007468 0.0007468 0.0007468 0.19% Castro::reset_internal_energy(Fab) 240 0.0006521 0.0006521 0.0006521 0.16% FabArray::mult() 22 0.0006457 0.0006457 0.0006457 0.16% FabArray::setDomainBndry() 20 0.0006457 0.0006457 0.0006457 0.16% MLPoisson::prepareForSolve() 6 0.0006152 0.0006152 0.0006152 0.16% MLCellLinOp::prepareForSolve() 6 0.000611 0.000611 0.000611 0.15% Castro::check_for_nan() 10 0.0006067 0.0006067 0.0006067 0.15% MultiFab::contains_nan() 10 0.0005985 0.0005985 0.0005985 0.15% MLMG::computeMLResidual() 6 0.0005569 0.0005569 0.0005569 0.14% Gravity::update_max_rhs() 6 0.0004394 0.0004394 0.0004394 0.11% Amr::InitAmr() 1 0.0004154 0.0004154 0.0004154 0.10% FabArrayBase::CPC::define() 244 0.0003932 0.0003932 0.0003932 0.10% FabArrayBase::getFB() 1766 0.0003097 0.0003097 0.0003097 0.08% Gravity::swapTimeLevels() 5 0.0002221 0.0002221 0.0002221 0.06% Castro::buildMetrics() 1 0.0001465 0.0001465 0.0001465 0.04% MLLinOp::define() 6 0.0001445 0.0001445 0.0001445 0.04% MultiFab::Copy() 6 0.0001356 0.0001356 0.0001356 0.03% MultiFab::max() 6 0.0001347 0.0001347 0.0001347 0.03% MLMG::MLResNormInf() 6 0.0001337 0.0001337 0.0001337 0.03% MLLinOp::defineGrids() 6 0.0001238 0.0001238 0.0001238 0.03% MLMG::MLRhsNormInf() 6 0.0001038 0.0001038 0.0001038 0.03% Castro::create_source_corrector() 5 9.35e-05 9.35e-05 9.35e-05 0.02% FabArrayBase::FB::FB() 26 5.466e-05 5.466e-05 5.466e-05 0.01% Castro::finalize_advance() 5 3.5e-05 3.5e-05 3.5e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.962e-05 2.962e-05 2.962e-05 0.01% Amr::writeSmallPlotFile() 1 2.777e-05 2.777e-05 2.777e-05 0.01% Castro::swap_state_time_levels() 5 2.661e-05 2.661e-05 2.661e-05 0.01% Castro::initMFs() 1 2.548e-05 2.548e-05 2.548e-05 0.01% makeSFC 30 2.207e-05 2.207e-05 2.207e-05 0.01% Castro::finalize_do_advance() 5 1.867e-05 1.867e-05 1.867e-05 0.00% DistributionMapping::Distribute() 31 9.589e-06 9.589e-06 9.589e-06 0.00% Amr::initSubcycle() 1 8.145e-06 8.145e-06 8.145e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.266e-06 5.266e-06 5.266e-06 0.00% MLMG::buildFineMask() 6 3.285e-06 3.285e-06 3.285e-06 0.00% Gravity::set_mass_offset() 6 2.031e-06 2.031e-06 2.031e-06 0.00% Castro::retry_advance_ctu() 5 2.026e-06 2.026e-06 2.026e-06 0.00% Castro::FluxRegCrseInit 5 1.747e-06 1.747e-06 1.747e-06 0.00% AmrLevel::AmrLevel() 1 1.596e-06 1.596e-06 1.596e-06 0.00% Castro::FluxRegFineAdd() 5 1.082e-06 1.082e-06 1.082e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.012e-06 1.012e-06 1.012e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.07-19-g48702b48836d) finalized