Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-13-gbd5f6a9f6a1a) initialized Starting run at 08:37:41 UTC on 2022-08-18. Successfully read inputs file ... Castro git describe: 22.08-11-ga978fcf88 AMReX git describe: 22.08-13-gbd5f6a9f6 Microphysics git describe: 22.08-3-gfe93b5a1 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.052979566 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030492184 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048542803 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052204431 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.051572341 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.062043554 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.067148752 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.049388763 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.06803837 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.069544675 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.056082821 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.04943277 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.064386451 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.049333594 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.030399901 seconds Ending run at 08:37:42 UTC on 2022-08-18. Run time = 0.855290874 Run time without initialization = 0.718759443 Average number of zones advanced per microsecond: 3.647 Average number of zones advanced per microsecond per rank: 3.647 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8553 ... 0.8553 ... 0.8553 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2048 0.2048 0.2048 23.94% Castro::construct_ctu_hydro_source() 10 0.1925 0.1925 0.1925 22.51% MLCellLinOp::applyBC() 4433 0.08157 0.08157 0.08157 9.54% MLPoisson::Fsmooth() 3280 0.06496 0.06496 0.06496 7.59% MLCGSolver::bicgstab 82 0.0245 0.0245 0.0245 2.86% StateData::FillBoundary(geom) 328 0.0239 0.0239 0.0239 2.79% MultiFab::Dot() 1114 0.02285 0.02285 0.02285 2.67% Castro::normalize_species() 62 0.0161 0.0161 0.0161 1.88% MultiFab::LinComb() 1586 0.01473 0.01473 0.01473 1.72% FillBoundary_nowait() 4023 0.01456 0.01456 0.01456 1.70% FabArray::setVal() 1144 0.01455 0.01455 0.01455 1.70% Castro::computeTemp() 63 0.01439 0.01439 0.01439 1.68% FabArray::ParallelCopy_nowait() 861 0.01335 0.01335 0.01335 1.56% StateDataPhysBCFunct::() 41 0.01304 0.01304 0.01304 1.52% MLPoisson::Fapply() 1142 0.01193 0.01193 0.01193 1.39% MLCellLinOp::defineAuxData() 11 0.01176 0.01176 0.01176 1.38% Castro::enforce_min_density() 62 0.01032 0.01032 0.01032 1.21% Gravity::fill_multipole_BCs() 11 0.01017 0.01017 0.01017 1.19% MLMG::addInterpCorrection() 410 0.007873 0.007873 0.007873 0.92% amrex::average_down 410 0.006989 0.006989 0.006989 0.82% MultiFab::Xpay() 585 0.006739 0.006739 0.006739 0.79% Castro::estTimeStep() 21 0.006457 0.006457 0.006457 0.75% Castro::do_advance_ctu() 10 0.005349 0.005349 0.005349 0.63% Amr::checkPoint() 3 0.005094 0.005094 0.005094 0.60% Castro::reset_internal_energy(MultiFab) 63 0.004257 0.004257 0.004257 0.50% BndryData::define() 11 0.004013 0.004013 0.004013 0.47% Castro::construct_new_gravity_source() 10 0.003281 0.003281 0.003281 0.38% Castro::construct_old_gravity_source() 10 0.002833 0.002833 0.002833 0.33% Amr::writePlotFile() 2 0.002791 0.002791 0.002791 0.33% MLMG::ResNormInf() 93 0.002132 0.002132 0.002132 0.25% Gravity::get_new_grav_vector() 11 0.001937 0.001937 0.001937 0.23% MultiFab::Saxpy() 20 0.001812 0.001812 0.001812 0.21% Gravity::get_old_grav_vector() 10 0.001738 0.001738 0.001738 0.20% Castro::expand_state() 10 0.001735 0.001735 0.001735 0.20% MultiFab::Add() 82 0.00169 0.00169 0.00169 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001687 0.001687 0.001687 0.20% Castro::reset_internal_energy(Fab) 504 0.00158 0.00158 0.00158 0.18% MLCellLinOp::setLevelBC() 11 0.001571 0.001571 0.001571 0.18% Gravity::actual_solve_with_mlmg() 11 0.001431 0.001431 0.001431 0.17% FabArray::mult() 43 0.001329 0.001329 0.001329 0.16% FabArray::setDomainBndry() 41 0.001321 0.001321 0.001321 0.15% Castro::initData() 1 0.001264 0.001264 0.001264 0.15% MLMG::prepareForSolve() 11 0.001249 0.001249 0.001249 0.15% MLCellLinOp::prepareForSolve() 11 0.001199 0.001199 0.001199 0.14% MultiFab::contains_nan() 20 0.001181 0.001181 0.001181 0.14% MLCellLinOp::smooth() 1640 0.001099 0.001099 0.001099 0.13% Castro::enforce_speed_limit() 62 0.001083 0.001083 0.001083 0.13% MLCellLinOp::compGrad() 11 0.0009373 0.0009373 0.0009373 0.11% FabArray::FillBoundary() 4023 0.0008776 0.0008776 0.0008776 0.10% FabArrayBase::getCPC() 1323 0.0007479 0.0007479 0.0007479 0.09% FabArrayBase::CPC::define() 454 0.0006761 0.0006761 0.0006761 0.08% FabArrayBase::getFB() 4023 0.0005935 0.0005935 0.0005935 0.07% Amr::InitAmr() 1 0.000494 0.000494 0.000494 0.06% MLCellLinOp::apply() 1142 0.0004792 0.0004792 0.0004792 0.06% Gravity::solve_for_phi() 10 0.0004423 0.0004423 0.0004423 0.05% Gravity::update_max_rhs() 11 0.0004108 0.0004108 0.0004108 0.05% CGSolver::sxay() 1586 0.000351 0.000351 0.000351 0.04% Amr::coarseTimeStep() 10 0.000348 0.000348 0.000348 0.04% MultiFab::Copy() 11 0.0003417 0.0003417 0.0003417 0.04% FillPatchIterator::Initialize 41 0.0002968 0.0002968 0.0002968 0.03% MLCGSolver::ParallelAllReduce 1514 0.0002957 0.0002957 0.0002957 0.03% MLCellLinOp::defineBC() 11 0.0002801 0.0002801 0.0002801 0.03% FabArray::ParallelCopy() 861 0.0002652 0.0002652 0.0002652 0.03% MultiFab::max() 11 0.0002592 0.0002592 0.0002592 0.03% main() 1 0.000258 0.000258 0.000258 0.03% MLMG::MLRhsNormInf() 11 0.0002201 0.0002201 0.0002201 0.03% Amr::timeStep() 10 0.0002123 0.0002123 0.0002123 0.02% MLCellLinOp::correctionResidual() 492 0.0002112 0.0002112 0.0002112 0.02% Castro::construct_new_gravity() 10 0.0002049 0.0002049 0.0002049 0.02% MLMG::mgVcycle() 82 0.0002006 0.0002006 0.0002006 0.02% Castro::construct_new_source() 50 0.0001958 0.0001958 0.0001958 0.02% Castro::subcycle_advance_ctu() 10 0.000177 0.000177 0.000177 0.02% MLMG:computeResOfCorrection() 410 0.0001481 0.0001481 0.0001481 0.02% MLLinOp::defineGrids() 11 0.000134 0.000134 0.000134 0.02% StateData::checkPoint() 12 0.0001279 0.0001279 0.0001279 0.01% MLMG::mgVcycle_down::0 82 0.0001079 0.0001079 0.0001079 0.01% Castro::advance() 10 9.611e-05 9.611e-05 9.611e-05 0.01% MLMG::mgVcycle_down::1 82 9.422e-05 9.422e-05 9.422e-05 0.01% MLMG::mgVcycle_down::2 82 9.049e-05 9.049e-05 9.049e-05 0.01% FabArrayBase::FB::FB() 56 9.016e-05 9.016e-05 9.016e-05 0.01% Castro::Castro() 1 8.808e-05 8.808e-05 8.808e-05 0.01% MLMG::mgVcycle_down::3 82 8.571e-05 8.571e-05 8.571e-05 0.01% MLMG::mgVcycle_down::4 82 8.435e-05 8.435e-05 8.435e-05 0.01% Castro::initialize_advance() 10 8.067e-05 8.067e-05 8.067e-05 0.01% MLMG::actualBottomSolve() 82 7.873e-05 7.873e-05 7.873e-05 0.01% Castro::clean_state() 62 7.557e-05 7.557e-05 7.557e-05 0.01% MLMG::solve() 11 7.172e-05 7.172e-05 7.172e-05 0.01% AmrLevel::checkPoint() 3 7.144e-05 7.144e-05 7.144e-05 0.01% MLMG::mgVcycle_up::4 82 7.105e-05 7.105e-05 7.105e-05 0.01% MLMG::mgVcycle_up::0 82 6.424e-05 6.424e-05 6.424e-05 0.01% Castro::initialize_do_advance() 10 6.341e-05 6.341e-05 6.341e-05 0.01% MLMG::mgVcycle_up::3 82 6.034e-05 6.034e-05 6.034e-05 0.01% MLMG::mgVcycle_up::1 82 5.915e-05 5.915e-05 5.915e-05 0.01% MLMG::oneIter() 82 5.849e-05 5.849e-05 5.849e-05 0.01% Castro::finalize_advance() 10 5.811e-05 5.811e-05 5.811e-05 0.01% MLMG::mgVcycle_up::2 82 5.631e-05 5.631e-05 5.631e-05 0.01% MLCellLinOp::solutionResidual() 93 4.953e-05 4.953e-05 4.953e-05 0.01% StateData::define() 4 4.358e-05 4.358e-05 4.358e-05 0.01% MLMG::computeResidual() 82 4.04e-05 4.04e-05 4.04e-05 0.00% Castro::swap_state_time_levels() 10 3.823e-05 3.823e-05 3.823e-05 0.00% MLMG::mgVcycle_bottom 82 3.577e-05 3.577e-05 3.577e-05 0.00% Castro::enforce_consistent_e() 1 3.499e-05 3.499e-05 3.499e-05 0.00% Castro::finalize_do_advance() 10 3.384e-05 3.384e-05 3.384e-05 0.00% Gravity::actual_multilevel_solve() 1 3.076e-05 3.076e-05 3.076e-05 0.00% MLPoisson::define() 11 3.027e-05 3.027e-05 3.027e-05 0.00% Amr::defBaseLevel() 1 2.848e-05 2.848e-05 2.848e-05 0.00% FillPatchSingleLevel 41 2.835e-05 2.835e-05 2.835e-05 0.00% Castro::initMFs() 1 2.703e-05 2.703e-05 2.703e-05 0.00% makeSFC 55 2.626e-05 2.626e-05 2.626e-05 0.00% MLLinOp::define() 11 2.576e-05 2.576e-05 2.576e-05 0.00% Amr::writeSmallPlotFile() 1 2.556e-05 2.556e-05 2.556e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.438e-05 2.438e-05 2.438e-05 0.00% Amr::FinalizeInit() 1 2.31e-05 2.31e-05 2.31e-05 0.00% Castro::buildMetrics() 1 2.235e-05 2.235e-05 2.235e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.81e-05 1.81e-05 1.81e-05 0.00% Castro::construct_old_source() 50 1.634e-05 1.634e-05 1.634e-05 0.00% Castro::do_new_sources() 10 1.61e-05 1.61e-05 1.61e-05 0.00% Castro::do_old_sources() 10 1.487e-05 1.487e-05 1.487e-05 0.00% DistributionMapping::Distribute() 56 1.477e-05 1.477e-05 1.477e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.28e-05 1.28e-05 1.28e-05 0.00% Castro::check_for_nan() 20 1.178e-05 1.178e-05 1.178e-05 0.00% MLPoisson::prepareForSolve() 11 1.022e-05 1.022e-05 1.022e-05 0.00% Castro::construct_old_gravity() 10 1.018e-05 1.018e-05 1.018e-05 0.00% Castro::apply_source_to_state() 20 9.777e-06 9.777e-06 9.777e-06 0.00% MLMG::computeMLResidual() 11 9.228e-06 9.228e-06 9.228e-06 0.00% Amr::initSubcycle() 1 8.949e-06 8.949e-06 8.949e-06 0.00% Gravity::swapTimeLevels() 10 8.407e-06 8.407e-06 8.407e-06 0.00% Castro::post_timestep() 10 7.678e-06 7.678e-06 7.678e-06 0.00% Amr::InitializeInit() 1 7.38e-06 7.38e-06 7.38e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.987e-06 6.987e-06 6.987e-06 0.00% MLMG::getGradSolution() 11 6.541e-06 6.541e-06 6.541e-06 0.00% Castro::computeNewDt() 9 5.986e-06 5.986e-06 5.986e-06 0.00% AmrLevel::checkPointPost() 3 4.889e-06 4.889e-06 4.889e-06 0.00% Castro::create_source_corrector() 10 4.825e-06 4.825e-06 4.825e-06 0.00% Castro::post_init() 1 4.189e-06 4.189e-06 4.189e-06 0.00% Gravity::set_mass_offset() 11 4.112e-06 4.112e-06 4.112e-06 0.00% Castro::retry_advance_ctu() 10 3.895e-06 3.895e-06 3.895e-06 0.00% MLMG::MLResNormInf() 11 3.579e-06 3.579e-06 3.579e-06 0.00% Castro::computeInitialDt() 2 2.9e-06 2.9e-06 2.9e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.672e-06 2.672e-06 2.672e-06 0.00% Castro::FluxRegCrseInit 10 2.535e-06 2.535e-06 2.535e-06 0.00% Amr::init() 1 2.43e-06 2.43e-06 2.43e-06 0.00% AmrLevel::checkPointPre() 3 2.057e-06 2.057e-06 2.057e-06 0.00% Castro::FluxRegFineAdd() 10 2.032e-06 2.032e-06 2.032e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.937e-06 1.937e-06 1.937e-06 0.00% Castro::post_regrid() 1 1.3e-06 1.3e-06 1.3e-06 0.00% Amr::initialInit() 1 1e-06 1e-06 1e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8553 0.8553 0.8553 100.00% Amr::coarseTimeStep() 10 0.6881 0.6881 0.6881 80.45% Amr::timeStep() 10 0.5856 0.5856 0.5856 68.46% Castro::advance() 10 0.5783 0.5783 0.5783 67.61% Castro::subcycle_advance_ctu() 10 0.5661 0.5661 0.5661 66.19% Castro::do_advance_ctu() 10 0.566 0.566 0.566 66.17% Gravity::solve_phi_with_mlmg() 11 0.3222 0.3222 0.3222 37.67% Gravity::actual_solve_with_mlmg() 11 0.3118 0.3118 0.3118 36.45% Castro::construct_new_gravity() 10 0.2932 0.2932 0.2932 34.28% MLMG::solve() 11 0.2887 0.2887 0.2887 33.76% Gravity::solve_for_phi() 10 0.2777 0.2777 0.2777 32.47% MLMG::oneIter() 82 0.2735 0.2735 0.2735 31.97% MLMG::mgVcycle() 82 0.2717 0.2717 0.2717 31.77% VisMF::Write(FabArray) 11 0.2048 0.2048 0.2048 23.94% Castro::construct_ctu_hydro_source() 10 0.1925 0.1925 0.1925 22.51% Amr::checkPoint() 3 0.1518 0.1518 0.1518 17.75% AmrLevel::checkPoint() 3 0.1467 0.1467 0.1467 17.16% StateData::checkPoint() 12 0.1467 0.1467 0.1467 17.15% MLCellLinOp::smooth() 1640 0.1388 0.1388 0.1388 16.23% Amr::init() 1 0.1359 0.1359 0.1359 15.89% MLCellLinOp::applyBC() 4433 0.09769 0.09769 0.09769 11.42% MLMG::mgVcycle_bottom 82 0.08376 0.08376 0.08376 9.79% MLMG::actualBottomSolve() 82 0.08372 0.08372 0.08372 9.79% MLCGSolver::bicgstab 82 0.08289 0.08289 0.08289 9.69% MLPoisson::Fsmooth() 3280 0.06496 0.06496 0.06496 7.59% Amr::writePlotFile() 2 0.06102 0.06102 0.06102 7.13% Amr::initialInit() 1 0.05233 0.05233 0.05233 6.12% Amr::FinalizeInit() 1 0.0482 0.0482 0.0482 5.64% Castro::clean_state() 62 0.04697 0.04697 0.04697 5.49% Castro::post_init() 1 0.04686 0.04686 0.04686 5.48% Gravity::multilevel_solve_for_new_phi() 1 0.04499 0.04499 0.04499 5.26% Gravity::actual_multilevel_solve() 1 0.04497 0.04497 0.04497 5.26% FillPatchIterator::Initialize 41 0.04256 0.04256 0.04256 4.98% FillPatchSingleLevel 41 0.04094 0.04094 0.04094 4.79% StateDataPhysBCFunct::() 41 0.03694 0.03694 0.03694 4.32% MLCellLinOp::apply() 1142 0.03686 0.03686 0.03686 4.31% MLMG::mgVcycle_down::0 82 0.03604 0.03604 0.03604 4.21% MLMG::mgVcycle_up::0 82 0.03087 0.03087 0.03087 3.61% StateData::FillBoundary(geom) 328 0.0239 0.0239 0.0239 2.79% MultiFab::Dot() 1114 0.02285 0.02285 0.02285 2.67% MLCellLinOp::correctionResidual() 492 0.02162 0.02162 0.02162 2.53% Castro::initialize_do_advance() 10 0.02056 0.02056 0.02056 2.40% Castro::computeTemp() 63 0.02023 0.02023 0.02023 2.36% MLMG:computeResOfCorrection() 410 0.01867 0.01867 0.01867 2.18% MLPoisson::define() 11 0.01861 0.01861 0.01861 2.18% MLMG::mgVcycle_down::1 82 0.01802 0.01802 0.01802 2.11% MLMG::mgVcycle_down::2 82 0.01762 0.01762 0.01762 2.06% Gravity::get_new_grav_vector() 11 0.01707 0.01707 0.01707 2.00% MLMG::mgVcycle_down::3 82 0.01674 0.01674 0.01674 1.96% FabArray::FillBoundary() 4023 0.01612 0.01612 0.01612 1.89% Castro::normalize_species() 62 0.0161 0.0161 0.0161 1.88% MLMG::mgVcycle_down::4 82 0.01595 0.01595 0.01595 1.86% FillBoundary_nowait() 4023 0.01525 0.01525 0.01525 1.78% CGSolver::sxay() 1586 0.01508 0.01508 0.01508 1.76% Castro::construct_old_gravity() 10 0.01485 0.01485 0.01485 1.74% Gravity::get_old_grav_vector() 10 0.01484 0.01484 0.01484 1.73% MultiFab::LinComb() 1586 0.01473 0.01473 0.01473 1.72% FabArray::setVal() 1144 0.01455 0.01455 0.01455 1.70% FabArray::ParallelCopy() 861 0.01442 0.01442 0.01442 1.69% FabArray::ParallelCopy_nowait() 861 0.01415 0.01415 0.01415 1.65% MLCGSolver::ParallelAllReduce 1514 0.01361 0.01361 0.01361 1.59% MLMG::mgVcycle_up::2 82 0.01356 0.01356 0.01356 1.59% MLMG::mgVcycle_up::1 82 0.01334 0.01334 0.01334 1.56% MLCellLinOp::defineAuxData() 11 0.01311 0.01311 0.01311 1.53% MLMG::addInterpCorrection() 410 0.01306 0.01306 0.01306 1.53% MLMG::mgVcycle_up::3 82 0.01288 0.01288 0.01288 1.51% MLMG::mgVcycle_up::4 82 0.01276 0.01276 0.01276 1.49% Castro::do_new_sources() 10 0.01233 0.01233 0.01233 1.44% amrex::average_down 410 0.01225 0.01225 0.01225 1.43% Castro::initialize_advance() 10 0.01199 0.01199 0.01199 1.40% MLPoisson::Fapply() 1142 0.01193 0.01193 0.01193 1.39% Castro::expand_state() 10 0.01187 0.01187 0.01187 1.39% Castro::do_old_sources() 10 0.01114 0.01114 0.01114 1.30% Castro::enforce_min_density() 62 0.01032 0.01032 0.01032 1.21% Gravity::fill_multipole_BCs() 11 0.01017 0.01017 0.01017 1.19% MLCellLinOp::solutionResidual() 93 0.007237 0.007237 0.007237 0.85% Castro::post_timestep() 10 0.007085 0.007085 0.007085 0.83% MultiFab::Xpay() 585 0.006739 0.006739 0.006739 0.79% Castro::estTimeStep() 21 0.006457 0.006457 0.006457 0.75% MLMG::computeResidual() 82 0.006254 0.006254 0.006254 0.73% Castro::reset_internal_energy(MultiFab) 63 0.005837 0.005837 0.005837 0.68% MLMG::prepareForSolve() 11 0.005547 0.005547 0.005547 0.65% MLCellLinOp::defineBC() 11 0.005257 0.005257 0.005257 0.61% BndryData::define() 11 0.004977 0.004977 0.004977 0.58% Amr::InitializeInit() 1 0.004129 0.004129 0.004129 0.48% Amr::defBaseLevel() 1 0.004122 0.004122 0.004122 0.48% Castro::initData() 1 0.003603 0.003603 0.003603 0.42% Castro::construct_new_source() 50 0.003476 0.003476 0.003476 0.41% Castro::construct_new_gravity_source() 10 0.003281 0.003281 0.003281 0.38% Castro::computeNewDt() 9 0.002922 0.002922 0.002922 0.34% Castro::construct_old_source() 50 0.00285 0.00285 0.00285 0.33% Castro::construct_old_gravity_source() 10 0.002833 0.002833 0.002833 0.33% MLMG::ResNormInf() 93 0.002132 0.002132 0.002132 0.25% Castro::apply_source_to_state() 20 0.001822 0.001822 0.001822 0.21% MultiFab::Saxpy() 20 0.001812 0.001812 0.001812 0.21% MultiFab::Add() 82 0.00169 0.00169 0.00169 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001687 0.001687 0.001687 0.20% Castro::reset_internal_energy(Fab) 504 0.00158 0.00158 0.00158 0.18% MLCellLinOp::setLevelBC() 11 0.001571 0.001571 0.001571 0.18% MLMG::getGradSolution() 11 0.001448 0.001448 0.001448 0.17% MLCellLinOp::compGrad() 11 0.001441 0.001441 0.001441 0.17% FabArrayBase::getCPC() 1323 0.001424 0.001424 0.001424 0.17% FabArray::mult() 43 0.001329 0.001329 0.001329 0.16% FabArray::setDomainBndry() 41 0.001321 0.001321 0.001321 0.15% MLPoisson::prepareForSolve() 11 0.001209 0.001209 0.001209 0.14% MLCellLinOp::prepareForSolve() 11 0.001199 0.001199 0.001199 0.14% Castro::check_for_nan() 20 0.001193 0.001193 0.001193 0.14% MultiFab::contains_nan() 20 0.001181 0.001181 0.001181 0.14% Castro::post_regrid() 1 0.00114 0.00114 0.00114 0.13% Castro::enforce_speed_limit() 62 0.001083 0.001083 0.001083 0.13% MLMG::computeMLResidual() 11 0.001033 0.001033 0.001033 0.12% Gravity::update_max_rhs() 11 0.0008174 0.0008174 0.0008174 0.10% FabArrayBase::getFB() 4023 0.0006836 0.0006836 0.0006836 0.08% FabArrayBase::CPC::define() 454 0.0006761 0.0006761 0.0006761 0.08% Castro::computeInitialDt() 2 0.0006506 0.0006506 0.0006506 0.08% Amr::InitAmr() 1 0.000503 0.000503 0.000503 0.06% Gravity::swapTimeLevels() 10 0.0004488 0.0004488 0.0004488 0.05% Castro::Castro() 1 0.0004356 0.0004356 0.0004356 0.05% MultiFab::Copy() 11 0.0003417 0.0003417 0.0003417 0.04% MLMG::MLResNormInf() 11 0.0002804 0.0002804 0.0002804 0.03% MultiFab::max() 11 0.0002592 0.0002592 0.0002592 0.03% MLMG::MLRhsNormInf() 11 0.0002201 0.0002201 0.0002201 0.03% MLLinOp::define() 11 0.0002143 0.0002143 0.0002143 0.03% MLLinOp::defineGrids() 11 0.0001886 0.0001886 0.0001886 0.02% Castro::buildMetrics() 1 0.0001535 0.0001535 0.0001535 0.02% FabArrayBase::FB::FB() 56 9.016e-05 9.016e-05 9.016e-05 0.01% Castro::finalize_advance() 10 6.268e-05 6.268e-05 6.268e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.268e-05 5.268e-05 5.268e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.056e-05 5.056e-05 5.056e-05 0.01% StateData::define() 4 4.358e-05 4.358e-05 4.358e-05 0.01% makeSFC 55 3.988e-05 3.988e-05 3.988e-05 0.00% Castro::swap_state_time_levels() 10 3.823e-05 3.823e-05 3.823e-05 0.00% Castro::enforce_consistent_e() 1 3.499e-05 3.499e-05 3.499e-05 0.00% Castro::finalize_do_advance() 10 3.384e-05 3.384e-05 3.384e-05 0.00% Castro::initMFs() 1 2.703e-05 2.703e-05 2.703e-05 0.00% Amr::writeSmallPlotFile() 1 2.556e-05 2.556e-05 2.556e-05 0.00% DistributionMapping::Distribute() 56 1.477e-05 1.477e-05 1.477e-05 0.00% Amr::initSubcycle() 1 8.949e-06 8.949e-06 8.949e-06 0.00% AmrLevel::checkPointPost() 3 4.889e-06 4.889e-06 4.889e-06 0.00% Castro::create_source_corrector() 10 4.825e-06 4.825e-06 4.825e-06 0.00% Gravity::set_mass_offset() 11 4.112e-06 4.112e-06 4.112e-06 0.00% Castro::retry_advance_ctu() 10 3.895e-06 3.895e-06 3.895e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.816e-06 3.816e-06 3.816e-06 0.00% Castro::FluxRegCrseInit 10 2.535e-06 2.535e-06 2.535e-06 0.00% AmrLevel::checkPointPre() 3 2.057e-06 2.057e-06 2.057e-06 0.00% Castro::FluxRegFineAdd() 10 2.032e-06 2.032e-06 2.032e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.937e-06 1.937e-06 1.937e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-13-gbd5f6a9f6a1a) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-13-gbd5f6a9f6a1a) initialized Starting run at 08:37:43 UTC on 2022-08-18. Successfully read inputs file ... Castro git describe: 22.08-11-ga978fcf88 AMReX git describe: 22.08-13-gbd5f6a9f6 Microphysics git describe: 22.08-3-gfe93b5a1 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.467472035 Restart time = 0.048100318 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.05473053 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051055164 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.06054801 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.071725321 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.072631482 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032332086 seconds Ending run at 08:37:43 UTC on 2022-08-18. Run time = 0.392202552 Run time without initialization = 0.34346702 Average number of zones advanced per microsecond: 3.816 Average number of zones advanced per microsecond per rank: 3.816 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3922 ... 0.3922 ... 0.3922 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1120 0.1120 0.1120 28.55% VisMF::Read() 3 0.04012 0.04012 0.04012 10.23% MLCellLinOp::applyBC() 1946 0.03443 0.03443 0.03443 8.78% VisMF::Write(FabArray) 1 0.03078 0.03078 0.03078 7.85% MLPoisson::Fsmooth() 1440 0.02723 0.02723 0.02723 6.94% StateData::FillBoundary(geom) 160 0.01192 0.01192 0.01192 3.04% MLCGSolver::bicgstab 36 0.01027 0.01027 0.01027 2.62% MultiFab::Dot() 484 0.009561 0.009561 0.009561 2.44% Castro::normalize_species() 30 0.007183 0.007183 0.007183 1.83% Castro::enforce_min_density() 30 0.00713 0.00713 0.00713 1.82% Castro::computeTemp() 30 0.006823 0.006823 0.006823 1.74% FabArray::setVal() 537 0.006749 0.006749 0.006749 1.72% FillBoundary_nowait() 1766 0.006296 0.006296 0.006296 1.61% MLCellLinOp::defineAuxData() 6 0.006209 0.006209 0.006209 1.58% MultiFab::LinComb() 690 0.006146 0.006146 0.006146 1.57% StateDataPhysBCFunct::() 20 0.00602 0.00602 0.00602 1.53% FabArray::ParallelCopy_nowait() 380 0.005914 0.005914 0.005914 1.51% Gravity::fill_multipole_BCs() 6 0.005906 0.005906 0.005906 1.51% MLPoisson::Fapply() 500 0.005067 0.005067 0.005067 1.29% Amr::restart() 1 0.003661 0.003661 0.003661 0.93% MLMG::addInterpCorrection() 180 0.003327 0.003327 0.003327 0.85% amrex::average_down 180 0.002953 0.002953 0.002953 0.75% MultiFab::Xpay() 258 0.002881 0.002881 0.002881 0.73% Castro::estTimeStep() 10 0.002639 0.002639 0.002639 0.67% BndryData::define() 6 0.002158 0.002158 0.002158 0.55% Castro::do_advance_ctu() 5 0.002124 0.002124 0.002124 0.54% Castro::reset_internal_energy(MultiFab) 30 0.001692 0.001692 0.001692 0.43% Amr::writePlotFile() 1 0.00166 0.00166 0.00166 0.42% Castro::construct_new_gravity_source() 5 0.00162 0.00162 0.00162 0.41% Castro::construct_old_gravity_source() 5 0.001427 0.001427 0.001427 0.36% MLMG::ResNormInf() 42 0.0009351 0.0009351 0.0009351 0.24% MultiFab::Saxpy() 10 0.000918 0.000918 0.000918 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009037 0.0009037 0.0009037 0.23% Gravity::get_new_grav_vector() 5 0.0008895 0.0008895 0.0008895 0.23% Gravity::get_old_grav_vector() 5 0.0008797 0.0008797 0.0008797 0.22% Castro::expand_state() 5 0.0008717 0.0008717 0.0008717 0.22% MLCellLinOp::setLevelBC() 6 0.0008238 0.0008238 0.0008238 0.21% Castro::create_source_corrector() 5 0.0007915 0.0007915 0.0007915 0.20% Castro::reset_internal_energy(Fab) 240 0.0007637 0.0007637 0.0007637 0.19% Gravity::actual_solve_with_mlmg() 6 0.0007479 0.0007479 0.0007479 0.19% Castro::enforce_speed_limit() 30 0.0007324 0.0007324 0.0007324 0.19% MultiFab::Add() 36 0.0007138 0.0007138 0.0007138 0.18% MLMG::prepareForSolve() 6 0.000658 0.000658 0.000658 0.17% FabArray::mult() 22 0.0006509 0.0006509 0.0006509 0.17% FabArray::setDomainBndry() 20 0.0006492 0.0006492 0.0006492 0.17% MLCellLinOp::prepareForSolve() 6 0.0006287 0.0006287 0.0006287 0.16% MultiFab::contains_nan() 10 0.0005858 0.0005858 0.0005858 0.15% MLCellLinOp::compGrad() 6 0.0005027 0.0005027 0.0005027 0.13% MLCellLinOp::smooth() 720 0.0004972 0.0004972 0.0004972 0.13% Amr::InitAmr() 1 0.0004497 0.0004497 0.0004497 0.11% FabArrayBase::CPC::define() 244 0.0003895 0.0003895 0.0003895 0.10% FabArray::FillBoundary() 1766 0.0003861 0.0003861 0.0003861 0.10% FabArrayBase::getCPC() 632 0.0003653 0.0003653 0.0003653 0.09% main() 1 0.0002686 0.0002686 0.0002686 0.07% FabArrayBase::getFB() 1766 0.0002631 0.0002631 0.0002631 0.07% Gravity::update_max_rhs() 6 0.0002264 0.0002264 0.0002264 0.06% Gravity::solve_for_phi() 5 0.0002152 0.0002152 0.0002152 0.05% MLCellLinOp::apply() 500 0.0002105 0.0002105 0.0002105 0.05% MultiFab::Copy() 6 0.0001785 0.0001785 0.0001785 0.05% Amr::coarseTimeStep() 5 0.0001693 0.0001693 0.0001693 0.04% CGSolver::sxay() 690 0.000165 0.000165 0.000165 0.04% FillPatchIterator::Initialize 20 0.0001546 0.0001546 0.0001546 0.04% MLCellLinOp::defineBC() 6 0.0001489 0.0001489 0.0001489 0.04% MultiFab::max() 6 0.0001361 0.0001361 0.0001361 0.03% Castro::subcycle_advance_ctu() 5 0.0001277 0.0001277 0.0001277 0.03% MLCGSolver::ParallelAllReduce 659 0.0001225 0.0001225 0.0001225 0.03% FabArray::ParallelCopy() 380 0.0001175 0.0001175 0.0001175 0.03% MLMG::MLRhsNormInf() 6 0.0001125 0.0001125 0.0001125 0.03% Castro::construct_new_gravity() 5 0.0001121 0.0001121 0.0001121 0.03% Amr::timeStep() 5 0.0001003 0.0001003 0.0001003 0.03% MLMG::mgVcycle() 36 9e-05 9e-05 9e-05 0.02% MLCellLinOp::correctionResidual() 216 8.873e-05 8.873e-05 8.873e-05 0.02% Castro::construct_new_source() 25 8.205e-05 8.205e-05 8.205e-05 0.02% AmrLevel::restart() 1 7.639e-05 7.639e-05 7.639e-05 0.02% StateData::restartDoit() 4 7.332e-05 7.332e-05 7.332e-05 0.02% MLLinOp::defineGrids() 6 7.322e-05 7.322e-05 7.322e-05 0.02% MLMG:computeResOfCorrection() 180 6.519e-05 6.519e-05 6.519e-05 0.02% FabArrayBase::FB::FB() 26 5.656e-05 5.656e-05 5.656e-05 0.01% Castro::finalize_advance() 5 5.379e-05 5.379e-05 5.379e-05 0.01% Castro::advance() 5 5.051e-05 5.051e-05 5.051e-05 0.01% Castro::construct_old_source() 25 4.726e-05 4.726e-05 4.726e-05 0.01% MLMG::mgVcycle_down::0 36 4.555e-05 4.555e-05 4.555e-05 0.01% MLMG::mgVcycle_down::1 36 4.18e-05 4.18e-05 4.18e-05 0.01% MLMG::mgVcycle_down::2 36 3.989e-05 3.989e-05 3.989e-05 0.01% Castro::initialize_advance() 5 3.826e-05 3.826e-05 3.826e-05 0.01% Castro::clean_state() 30 3.811e-05 3.811e-05 3.811e-05 0.01% MLMG::mgVcycle_down::4 36 3.803e-05 3.803e-05 3.803e-05 0.01% MLMG::mgVcycle_down::3 36 3.745e-05 3.745e-05 3.745e-05 0.01% MLMG::solve() 6 3.514e-05 3.514e-05 3.514e-05 0.01% MLMG::actualBottomSolve() 36 3.463e-05 3.463e-05 3.463e-05 0.01% MLMG::mgVcycle_up::4 36 3.339e-05 3.339e-05 3.339e-05 0.01% Castro::buildMetrics() 1 3.225e-05 3.225e-05 3.225e-05 0.01% Castro::initialize_do_advance() 5 3.15e-05 3.15e-05 3.15e-05 0.01% Gravity::actual_multilevel_solve() 1 3.021e-05 3.021e-05 3.021e-05 0.01% Castro::post_restart() 1 2.932e-05 2.932e-05 2.932e-05 0.01% MLMG::mgVcycle_up::0 36 2.901e-05 2.901e-05 2.901e-05 0.01% Amr::writeSmallPlotFile() 1 2.898e-05 2.898e-05 2.898e-05 0.01% Castro::initMFs() 1 2.887e-05 2.887e-05 2.887e-05 0.01% Castro::swap_state_time_levels() 5 2.727e-05 2.727e-05 2.727e-05 0.01% MLMG::mgVcycle_up::3 36 2.663e-05 2.663e-05 2.663e-05 0.01% MLMG::oneIter() 36 2.604e-05 2.604e-05 2.604e-05 0.01% MLMG::mgVcycle_up::2 36 2.476e-05 2.476e-05 2.476e-05 0.01% MLMG::mgVcycle_up::1 36 2.393e-05 2.393e-05 2.393e-05 0.01% MLCellLinOp::solutionResidual() 42 2.242e-05 2.242e-05 2.242e-05 0.01% MLPoisson::define() 6 2.206e-05 2.206e-05 2.206e-05 0.01% MLLinOp::define() 6 2.005e-05 2.005e-05 2.005e-05 0.01% Castro::finalize_do_advance() 5 1.748e-05 1.748e-05 1.748e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.739e-05 1.739e-05 1.739e-05 0.00% MLMG::computeResidual() 36 1.739e-05 1.739e-05 1.739e-05 0.00% DistributionMapping::Distribute() 31 1.618e-05 1.618e-05 1.618e-05 0.00% MLMG::mgVcycle_bottom 36 1.429e-05 1.429e-05 1.429e-05 0.00% makeSFC 30 1.426e-05 1.426e-05 1.426e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.332e-05 1.332e-05 1.332e-05 0.00% FillPatchSingleLevel 20 1.305e-05 1.305e-05 1.305e-05 0.00% Amr::initSubcycle() 1 9.33e-06 9.33e-06 9.33e-06 0.00% Castro::do_new_sources() 5 9.29e-06 9.29e-06 9.29e-06 0.00% Castro::do_old_sources() 5 8.755e-06 8.755e-06 8.755e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.466e-06 7.466e-06 7.466e-06 0.00% Castro::check_for_nan() 10 5.839e-06 5.839e-06 5.839e-06 0.00% Castro::construct_old_gravity() 5 5.669e-06 5.669e-06 5.669e-06 0.00% MLPoisson::prepareForSolve() 6 4.985e-06 4.985e-06 4.985e-06 0.00% Castro::post_timestep() 5 4.965e-06 4.965e-06 4.965e-06 0.00% Castro::apply_source_to_state() 10 4.96e-06 4.96e-06 4.96e-06 0.00% MLMG::computeMLResidual() 6 4.544e-06 4.544e-06 4.544e-06 0.00% Gravity::swapTimeLevels() 5 4.263e-06 4.263e-06 4.263e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.107e-06 3.107e-06 3.107e-06 0.00% MLMG::getGradSolution() 6 3.072e-06 3.072e-06 3.072e-06 0.00% Castro::computeNewDt() 5 2.799e-06 2.799e-06 2.799e-06 0.00% Gravity::set_mass_offset() 6 2.068e-06 2.068e-06 2.068e-06 0.00% MLMG::MLResNormInf() 6 2.029e-06 2.029e-06 2.029e-06 0.00% Castro::retry_advance_ctu() 5 1.813e-06 1.813e-06 1.813e-06 0.00% Castro::FluxRegCrseInit 5 1.616e-06 1.616e-06 1.616e-06 0.00% Amr::init() 1 1.151e-06 1.151e-06 1.151e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.058e-06 1.058e-06 1.058e-06 0.00% Castro::FluxRegFineAdd() 5 1.031e-06 1.031e-06 1.031e-06 0.00% AmrLevel::AmrLevel() 1 9.25e-07 9.25e-07 9.25e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3922 0.3922 0.3922 100.00% Amr::coarseTimeStep() 5 0.3109 0.3109 0.3109 79.25% Amr::timeStep() 5 0.3091 0.3091 0.3091 78.82% Castro::advance() 5 0.3053 0.3053 0.3053 77.83% Castro::subcycle_advance_ctu() 5 0.2989 0.2989 0.2989 76.20% Castro::do_advance_ctu() 5 0.2987 0.2987 0.2987 76.16% Castro::construct_new_gravity() 5 0.1451 0.1451 0.1451 36.98% Gravity::solve_phi_with_mlmg() 6 0.1407 0.1407 0.1407 35.87% Gravity::solve_for_phi() 5 0.1371 0.1371 0.1371 34.96% Gravity::actual_solve_with_mlmg() 6 0.1347 0.1347 0.1347 34.34% MLMG::solve() 6 0.1224 0.1224 0.1224 31.20% MLMG::oneIter() 36 0.1152 0.1152 0.1152 29.36% MLMG::mgVcycle() 36 0.1144 0.1144 0.1144 29.17% Castro::construct_ctu_hydro_source() 5 0.112 0.112 0.112 28.55% MLCellLinOp::smooth() 720 0.05846 0.05846 0.05846 14.91% Amr::init() 1 0.04816 0.04816 0.04816 12.28% Amr::restart() 1 0.04816 0.04816 0.04816 12.28% MLCellLinOp::applyBC() 1946 0.04143 0.04143 0.04143 10.56% AmrLevel::restart() 1 0.04033 0.04033 0.04033 10.28% StateData::restartDoit() 4 0.04025 0.04025 0.04025 10.26% VisMF::Read() 3 0.04012 0.04012 0.04012 10.23% MLMG::mgVcycle_bottom 36 0.03517 0.03517 0.03517 8.97% MLMG::actualBottomSolve() 36 0.03515 0.03515 0.03515 8.96% MLCGSolver::bicgstab 36 0.0348 0.0348 0.0348 8.87% Amr::writePlotFile() 1 0.03244 0.03244 0.03244 8.27% VisMF::Write(FabArray) 1 0.03078 0.03078 0.03078 7.85% MLPoisson::Fsmooth() 1440 0.02723 0.02723 0.02723 6.94% Castro::clean_state() 30 0.02436 0.02436 0.02436 6.21% FillPatchIterator::Initialize 20 0.02075 0.02075 0.02075 5.29% FillPatchSingleLevel 20 0.01995 0.01995 0.01995 5.09% StateDataPhysBCFunct::() 20 0.01794 0.01794 0.01794 4.57% MLCellLinOp::apply() 500 0.01571 0.01571 0.01571 4.01% MLMG::mgVcycle_down::0 36 0.01533 0.01533 0.01533 3.91% MLMG::mgVcycle_up::0 36 0.0131 0.0131 0.0131 3.34% StateData::FillBoundary(geom) 160 0.01192 0.01192 0.01192 3.04% Castro::initialize_do_advance() 5 0.01097 0.01097 0.01097 2.80% MLPoisson::define() 6 0.009945 0.009945 0.009945 2.54% MultiFab::Dot() 484 0.009561 0.009561 0.009561 2.44% Castro::computeTemp() 30 0.009279 0.009279 0.009279 2.37% MLCellLinOp::correctionResidual() 216 0.009152 0.009152 0.009152 2.33% MLMG:computeResOfCorrection() 180 0.007894 0.007894 0.007894 2.01% Gravity::get_new_grav_vector() 5 0.007842 0.007842 0.007842 2.00% MLMG::mgVcycle_down::1 36 0.007589 0.007589 0.007589 1.93% Castro::construct_old_gravity() 5 0.007574 0.007574 0.007574 1.93% Gravity::get_old_grav_vector() 5 0.007569 0.007569 0.007569 1.93% MLMG::mgVcycle_down::2 36 0.007356 0.007356 0.007356 1.88% Castro::normalize_species() 30 0.007183 0.007183 0.007183 1.83% Castro::enforce_min_density() 30 0.00713 0.00713 0.00713 1.82% MLMG::mgVcycle_down::3 36 0.007023 0.007023 0.007023 1.79% FabArray::FillBoundary() 1766 0.007002 0.007002 0.007002 1.79% Castro::do_new_sources() 5 0.006984 0.006984 0.006984 1.78% MLCellLinOp::defineAuxData() 6 0.006956 0.006956 0.006956 1.77% FabArray::setVal() 537 0.006749 0.006749 0.006749 1.72% MLMG::mgVcycle_down::4 36 0.006724 0.006724 0.006724 1.71% FillBoundary_nowait() 1766 0.006616 0.006616 0.006616 1.69% FabArray::ParallelCopy() 380 0.006405 0.006405 0.006405 1.63% CGSolver::sxay() 690 0.006311 0.006311 0.006311 1.61% Castro::initialize_advance() 5 0.006302 0.006302 0.006302 1.61% FabArray::ParallelCopy_nowait() 380 0.006288 0.006288 0.006288 1.60% MultiFab::LinComb() 690 0.006146 0.006146 0.006146 1.57% Castro::expand_state() 5 0.00603 0.00603 0.00603 1.54% Gravity::fill_multipole_BCs() 6 0.005906 0.005906 0.005906 1.51% MLCGSolver::ParallelAllReduce 659 0.005735 0.005735 0.005735 1.46% Castro::do_old_sources() 5 0.005672 0.005672 0.005672 1.45% MLMG::mgVcycle_up::2 36 0.005667 0.005667 0.005667 1.44% MLMG::mgVcycle_up::1 36 0.005597 0.005597 0.005597 1.43% MLMG::addInterpCorrection() 180 0.005519 0.005519 0.005519 1.41% MLMG::mgVcycle_up::3 36 0.005399 0.005399 0.005399 1.38% MLMG::mgVcycle_up::4 36 0.005386 0.005386 0.005386 1.37% amrex::average_down 180 0.005173 0.005173 0.005173 1.32% MLPoisson::Fapply() 500 0.005067 0.005067 0.005067 1.29% Castro::post_restart() 1 0.003984 0.003984 0.003984 1.02% Gravity::multilevel_solve_for_new_phi() 1 0.003859 0.003859 0.003859 0.98% Gravity::actual_multilevel_solve() 1 0.003842 0.003842 0.003842 0.98% Castro::post_timestep() 5 0.003768 0.003768 0.003768 0.96% MLCellLinOp::solutionResidual() 42 0.003218 0.003218 0.003218 0.82% MLMG::prepareForSolve() 6 0.002892 0.002892 0.002892 0.74% MultiFab::Xpay() 258 0.002881 0.002881 0.002881 0.73% MLCellLinOp::defineBC() 6 0.002844 0.002844 0.002844 0.73% BndryData::define() 6 0.002695 0.002695 0.002695 0.69% MLMG::computeResidual() 36 0.002673 0.002673 0.002673 0.68% Castro::estTimeStep() 10 0.002639 0.002639 0.002639 0.67% Castro::reset_internal_energy(MultiFab) 30 0.002456 0.002456 0.002456 0.63% Castro::construct_new_source() 25 0.001702 0.001702 0.001702 0.43% Castro::construct_new_gravity_source() 5 0.00162 0.00162 0.00162 0.41% Castro::computeNewDt() 5 0.001543 0.001543 0.001543 0.39% Castro::construct_old_source() 25 0.001475 0.001475 0.001475 0.38% Castro::construct_old_gravity_source() 5 0.001427 0.001427 0.001427 0.36% MLMG::ResNormInf() 42 0.0009351 0.0009351 0.0009351 0.24% Castro::apply_source_to_state() 10 0.0009229 0.0009229 0.0009229 0.24% MultiFab::Saxpy() 10 0.000918 0.000918 0.000918 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009037 0.0009037 0.0009037 0.23% MLCellLinOp::setLevelBC() 6 0.0008238 0.0008238 0.0008238 0.21% Castro::create_source_corrector() 5 0.0007915 0.0007915 0.0007915 0.20% MLMG::getGradSolution() 6 0.0007774 0.0007774 0.0007774 0.20% MLCellLinOp::compGrad() 6 0.0007743 0.0007743 0.0007743 0.20% Castro::reset_internal_energy(Fab) 240 0.0007637 0.0007637 0.0007637 0.19% FabArrayBase::getCPC() 632 0.0007548 0.0007548 0.0007548 0.19% Castro::enforce_speed_limit() 30 0.0007324 0.0007324 0.0007324 0.19% MultiFab::Add() 36 0.0007138 0.0007138 0.0007138 0.18% FabArray::mult() 22 0.0006509 0.0006509 0.0006509 0.17% FabArray::setDomainBndry() 20 0.0006492 0.0006492 0.0006492 0.17% MLPoisson::prepareForSolve() 6 0.0006337 0.0006337 0.0006337 0.16% MLCellLinOp::prepareForSolve() 6 0.0006287 0.0006287 0.0006287 0.16% Castro::check_for_nan() 10 0.0005916 0.0005916 0.0005916 0.15% MultiFab::contains_nan() 10 0.0005858 0.0005858 0.0005858 0.15% MLMG::computeMLResidual() 6 0.0005669 0.0005669 0.0005669 0.14% Amr::InitAmr() 1 0.0004591 0.0004591 0.0004591 0.12% Gravity::update_max_rhs() 6 0.0004398 0.0004398 0.0004398 0.11% FabArrayBase::CPC::define() 244 0.0003895 0.0003895 0.0003895 0.10% FabArrayBase::getFB() 1766 0.0003197 0.0003197 0.0003197 0.08% Gravity::swapTimeLevels() 5 0.0002271 0.0002271 0.0002271 0.06% MultiFab::Copy() 6 0.0001785 0.0001785 0.0001785 0.05% Castro::buildMetrics() 1 0.0001543 0.0001543 0.0001543 0.04% MLMG::MLResNormInf() 6 0.000147 0.000147 0.000147 0.04% MultiFab::max() 6 0.0001361 0.0001361 0.0001361 0.03% MLLinOp::define() 6 0.0001235 0.0001235 0.0001235 0.03% MLMG::MLRhsNormInf() 6 0.0001125 0.0001125 0.0001125 0.03% MLLinOp::defineGrids() 6 0.0001035 0.0001035 0.0001035 0.03% FabArrayBase::FB::FB() 26 5.656e-05 5.656e-05 5.656e-05 0.01% Castro::finalize_advance() 5 5.644e-05 5.644e-05 5.644e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.921e-05 2.921e-05 2.921e-05 0.01% Amr::writeSmallPlotFile() 1 2.898e-05 2.898e-05 2.898e-05 0.01% Castro::initMFs() 1 2.887e-05 2.887e-05 2.887e-05 0.01% Castro::swap_state_time_levels() 5 2.727e-05 2.727e-05 2.727e-05 0.01% makeSFC 30 2.175e-05 2.175e-05 2.175e-05 0.01% Castro::finalize_do_advance() 5 1.748e-05 1.748e-05 1.748e-05 0.00% DistributionMapping::Distribute() 31 1.618e-05 1.618e-05 1.618e-05 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 1.179e-05 1.179e-05 1.179e-05 0.00% Amr::initSubcycle() 1 9.33e-06 9.33e-06 9.33e-06 0.00% Gravity::set_mass_offset() 6 2.068e-06 2.068e-06 2.068e-06 0.00% Castro::retry_advance_ctu() 5 1.813e-06 1.813e-06 1.813e-06 0.00% Castro::FluxRegCrseInit 5 1.616e-06 1.616e-06 1.616e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.058e-06 1.058e-06 1.058e-06 0.00% Castro::FluxRegFineAdd() 5 1.031e-06 1.031e-06 1.031e-06 0.00% AmrLevel::AmrLevel() 1 9.25e-07 9.25e-07 9.25e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-13-gbd5f6a9f6a1a) finalized