Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-16-g96d8c53fbf09) initialized Starting run at 09:59:53 UTC on 2022-12-19. Successfully read inputs file ... Castro git describe: 22.12-5-g3ad2ef0c0 AMReX git describe: 22.12-16-g96d8c53fb Microphysics git describe: 22.12-23-gacd17b4d reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.056667866 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.032491107 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.047337926 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.048065866 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.048901498 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.056224654 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.063574021 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.055935099 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.062237947 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.067223021 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.065191136 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.054939152 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.060325011 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.056135313 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032519406 seconds Ending run at 09:59:54 UTC on 2022-12-19. Run time = 0.856634189 Run time without initialization = 0.71921768 Average number of zones advanced per microsecond: 3.645 Average number of zones advanced per microsecond per rank: 3.645 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8567 ... 0.8567 ... 0.8567 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2270 0.2270 0.2270 26.49% Castro::construct_ctu_hydro_source() 10 0.2104 0.2104 0.2104 24.56% MLCellLinOp::applyBC() 4433 0.07341 0.07341 0.07341 8.57% MLPoisson::Fsmooth() 3280 0.0591 0.0591 0.0591 6.90% StateData::FillBoundary(geom) 328 0.02285 0.02285 0.02285 2.67% amrex::Dot() 1114 0.01973 0.01973 0.01973 2.30% StateDataPhysBCFunct::() 41 0.01889 0.01889 0.01889 2.21% Castro::normalize_species() 62 0.01603 0.01603 0.01603 1.87% amrex::Copy() 1029 0.01448 0.01448 0.01448 1.69% FabArray::norminf() 743 0.01397 0.01397 0.01397 1.63% FillBoundary_nowait() 4023 0.01392 0.01392 0.01392 1.62% Castro::computeTemp() 63 0.01387 0.01387 0.01387 1.62% FabArray::setVal() 1144 0.01285 0.01285 0.01285 1.50% FabArray::ParallelCopy_nowait() 861 0.01275 0.01275 0.01275 1.49% MLPoisson::Fapply() 1142 0.01014 0.01014 0.01014 1.18% MLCellLinOp::defineAuxData() 11 0.009229 0.009229 0.009229 1.08% Castro::enforce_min_density() 62 0.009218 0.009218 0.009218 1.08% FabArray::Saxpy() 813 0.008011 0.008011 0.008011 0.94% FabArray::Xpay() 821 0.007977 0.007977 0.007977 0.93% Gravity::fill_multipole_BCs() 11 0.006434 0.006434 0.006434 0.75% MLMG::addInterpCorrection() 410 0.00639 0.00639 0.00639 0.75% amrex::average_down 410 0.005722 0.005722 0.005722 0.67% Castro::estTimeStep() 21 0.005474 0.005474 0.005474 0.64% Castro::reset_internal_energy(MultiFab) 63 0.00454 0.00454 0.00454 0.53% FabArray::LinComb() 557 0.004352 0.004352 0.004352 0.51% Amr::checkPoint() 3 0.003995 0.003995 0.003995 0.47% BndryData::define() 11 0.003489 0.003489 0.003489 0.41% Castro::construct_new_gravity_source() 10 0.003245 0.003245 0.003245 0.38% Castro::do_advance_ctu() 10 0.002886 0.002886 0.002886 0.34% Castro::construct_old_gravity_source() 10 0.0027 0.0027 0.0027 0.32% amrex::Add() 164 0.002348 0.002348 0.002348 0.27% Amr::writePlotFile() 2 0.002311 0.002311 0.002311 0.27% MLCGSolver::bicgstab 82 0.00209 0.00209 0.00209 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001587 0.001587 0.001587 0.19% Castro::reset_internal_energy(Fab) 504 0.001542 0.001542 0.001542 0.18% MLCellLinOp::setLevelBC() 11 0.001348 0.001348 0.001348 0.16% FabArray::mult() 43 0.001315 0.001315 0.001315 0.15% FabArray::setDomainBndry() 41 0.001314 0.001314 0.001314 0.15% Gravity::actual_solve_with_mlmg() 11 0.001299 0.001299 0.001299 0.15% Castro::enforce_speed_limit() 62 0.001282 0.001282 0.001282 0.15% Castro::initData() 1 0.001274 0.001274 0.001274 0.15% MultiFab::contains_nan() 20 0.001175 0.001175 0.001175 0.14% MLCellLinOp::smooth() 1640 0.001111 0.001111 0.001111 0.13% MLCellLinOp::prepareForSolve() 11 0.001085 0.001085 0.001085 0.13% MLCellLinOp::compGrad() 11 0.0009031 0.0009031 0.0009031 0.11% MLMG::prepareForSolve() 11 0.0008231 0.0008231 0.0008231 0.10% FabArray::FillBoundary() 4023 0.0008165 0.0008165 0.0008165 0.10% FabArrayBase::getCPC() 1323 0.0007229 0.0007229 0.0007229 0.08% FabArrayBase::CPC::define() 454 0.0006746 0.0006746 0.0006746 0.08% Gravity::get_new_grav_vector() 11 0.0006136 0.0006136 0.0006136 0.07% FabArrayBase::getFB() 4023 0.0006058 0.0006058 0.0006058 0.07% Gravity::get_old_grav_vector() 10 0.0005275 0.0005275 0.0005275 0.06% Amr::InitAmr() 1 0.0004776 0.0004776 0.0004776 0.06% MLCellLinOp::apply() 1142 0.0004719 0.0004719 0.0004719 0.06% MLMG::mgVcycle() 82 0.0003833 0.0003833 0.0003833 0.04% Amr::coarseTimeStep() 10 0.0003352 0.0003352 0.0003352 0.04% MLLinOp::defineGrids() 11 0.0003039 0.0003039 0.0003039 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002908 0.0002908 0.0002908 0.03% main() 1 0.0002871 0.0002871 0.0002871 0.03% MultiFab::max() 11 0.0002562 0.0002562 0.0002562 0.03% FabArray::ParallelCopy() 861 0.0002409 0.0002409 0.0002409 0.03% MLCellLinOp::correctionResidual() 492 0.0002344 0.0002344 0.0002344 0.03% FillPatchIterator::Initialize 41 0.0001979 0.0001979 0.0001979 0.02% MLCellLinOp::defineBC() 11 0.0001964 0.0001964 0.0001964 0.02% Castro::subcycle_advance_ctu() 10 0.0001583 0.0001583 0.0001583 0.02% Amr::timeStep() 10 0.000155 0.000155 0.000155 0.02% StateData::checkPoint() 12 0.0001445 0.0001445 0.0001445 0.02% Gravity::update_max_rhs() 11 0.0001146 0.0001146 0.0001146 0.01% MLMG:computeResOfCorrection() 410 0.0001139 0.0001139 0.0001139 0.01% Gravity::solve_for_phi() 10 0.0001107 0.0001107 0.0001107 0.01% Castro::advance() 10 0.0001039 0.0001039 0.0001039 0.01% MLMG::actualBottomSolve() 82 9.356e-05 9.356e-05 9.356e-05 0.01% MLMG::mgVcycle_down::0 82 9.189e-05 9.189e-05 9.189e-05 0.01% Castro::finalize_advance() 10 8.576e-05 8.576e-05 8.576e-05 0.01% FabArrayBase::FB::FB() 56 8.254e-05 8.254e-05 8.254e-05 0.01% MLMG::mgVcycle_down::1 82 8.198e-05 8.198e-05 8.198e-05 0.01% Castro::clean_state() 62 8.124e-05 8.124e-05 8.124e-05 0.01% MLMG::mgVcycle_down::2 82 8.049e-05 8.049e-05 8.049e-05 0.01% AmrLevel::checkPoint() 3 7.714e-05 7.714e-05 7.714e-05 0.01% Castro::Castro() 1 7.632e-05 7.632e-05 7.632e-05 0.01% MLMG::mgVcycle_down::3 82 7.581e-05 7.581e-05 7.581e-05 0.01% MLMG::mgVcycle_down::4 82 7.538e-05 7.538e-05 7.538e-05 0.01% Castro::expand_state() 10 7.522e-05 7.522e-05 7.522e-05 0.01% MLMG::solve() 11 7.454e-05 7.454e-05 7.454e-05 0.01% Castro::initialize_advance() 10 6.875e-05 6.875e-05 6.875e-05 0.01% MLMG::mgVcycle_up::4 82 6.387e-05 6.387e-05 6.387e-05 0.01% MLMG::oneIter() 82 5.646e-05 5.646e-05 5.646e-05 0.01% MLMG::mgVcycle_up::0 82 5.381e-05 5.381e-05 5.381e-05 0.01% MLCellLinOp::solutionResidual() 93 5.244e-05 5.244e-05 5.244e-05 0.01% MLMG::mgVcycle_up::1 82 5.219e-05 5.219e-05 5.219e-05 0.01% MLMG::mgVcycle_up::3 82 5.197e-05 5.197e-05 5.197e-05 0.01% MLMG::mgVcycle_up::2 82 5.047e-05 5.047e-05 5.047e-05 0.01% Castro::initialize_do_advance() 10 4.344e-05 4.344e-05 4.344e-05 0.01% Castro::swap_state_time_levels() 10 3.566e-05 3.566e-05 3.566e-05 0.00% MLMG::ResNormInf() 93 3.346e-05 3.346e-05 3.346e-05 0.00% Castro::enforce_consistent_e() 1 3.337e-05 3.337e-05 3.337e-05 0.00% StateData::define() 4 3.275e-05 3.275e-05 3.275e-05 0.00% MLMG::computeResidual() 82 3.25e-05 3.25e-05 3.25e-05 0.00% MLMG::mgVcycle_bottom 82 3.242e-05 3.242e-05 3.242e-05 0.00% Castro::finalize_do_advance() 10 3.154e-05 3.154e-05 3.154e-05 0.00% FillPatchSingleLevel 41 3.039e-05 3.039e-05 3.039e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.854e-05 2.854e-05 2.854e-05 0.00% makeSFC 55 2.626e-05 2.626e-05 2.626e-05 0.00% Castro::construct_new_gravity() 10 2.453e-05 2.453e-05 2.453e-05 0.00% Castro::create_source_corrector() 10 2.446e-05 2.446e-05 2.446e-05 0.00% Amr::writeSmallPlotFile() 1 2.335e-05 2.335e-05 2.335e-05 0.00% MLPoisson::define() 11 2.144e-05 2.144e-05 2.144e-05 0.00% Amr::FinalizeInit() 1 2.05e-05 2.05e-05 2.05e-05 0.00% Castro::construct_old_source() 50 1.996e-05 1.996e-05 1.996e-05 0.00% Castro::initMFs() 1 1.903e-05 1.903e-05 1.903e-05 0.00% Castro::do_new_sources() 10 1.821e-05 1.821e-05 1.821e-05 0.00% Castro::buildMetrics() 1 1.668e-05 1.668e-05 1.668e-05 0.00% DistributionMapping::Distribute() 56 1.605e-05 1.605e-05 1.605e-05 0.00% Castro::do_old_sources() 10 1.576e-05 1.576e-05 1.576e-05 0.00% Castro::construct_new_source() 50 1.529e-05 1.529e-05 1.529e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.466e-05 1.466e-05 1.466e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.447e-05 1.447e-05 1.447e-05 0.00% Amr::defBaseLevel() 1 1.346e-05 1.346e-05 1.346e-05 0.00% Castro::check_for_nan() 20 1.207e-05 1.207e-05 1.207e-05 0.00% MLLinOp::define() 11 1.139e-05 1.139e-05 1.139e-05 0.00% Castro::apply_source_to_state() 20 1.012e-05 1.012e-05 1.012e-05 0.00% Gravity::swapTimeLevels() 10 9.628e-06 9.628e-06 9.628e-06 0.00% Castro::post_init() 1 9.535e-06 9.535e-06 9.535e-06 0.00% Castro::construct_old_gravity() 10 8.994e-06 8.994e-06 8.994e-06 0.00% Amr::initSubcycle() 1 8.788e-06 8.788e-06 8.788e-06 0.00% AmrLevel::AmrLevel(dm) 1 8.722e-06 8.722e-06 8.722e-06 0.00% Castro::post_timestep() 10 8.714e-06 8.714e-06 8.714e-06 0.00% MLPoisson::prepareForSolve() 11 8.696e-06 8.696e-06 8.696e-06 0.00% MLMG::computeMLResidual() 11 7.456e-06 7.456e-06 7.456e-06 0.00% Gravity::actual_multilevel_solve() 1 7.208e-06 7.208e-06 7.208e-06 0.00% Castro::computeNewDt() 9 6.393e-06 6.393e-06 6.393e-06 0.00% MLMG::getGradSolution() 11 6.192e-06 6.192e-06 6.192e-06 0.00% AmrLevel::checkPointPost() 3 5.572e-06 5.572e-06 5.572e-06 0.00% Amr::InitializeInit() 1 4.829e-06 4.829e-06 4.829e-06 0.00% Castro::retry_advance_ctu() 10 4.355e-06 4.355e-06 4.355e-06 0.00% Gravity::set_mass_offset() 11 4.085e-06 4.085e-06 4.085e-06 0.00% MLMG::MLRhsNormInf() 11 3.864e-06 3.864e-06 3.864e-06 0.00% MLMG::MLResNormInf() 11 3.63e-06 3.63e-06 3.63e-06 0.00% Castro::FluxRegCrseInit 10 3.225e-06 3.225e-06 3.225e-06 0.00% Castro::computeInitialDt() 2 3.035e-06 3.035e-06 3.035e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.61e-06 2.61e-06 2.61e-06 0.00% Amr::init() 1 2.421e-06 2.421e-06 2.421e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.119e-06 2.119e-06 2.119e-06 0.00% Castro::FluxRegFineAdd() 10 1.944e-06 1.944e-06 1.944e-06 0.00% AmrLevel::checkPointPre() 3 1.898e-06 1.898e-06 1.898e-06 0.00% Amr::initialInit() 1 1.114e-06 1.114e-06 1.114e-06 0.00% Castro::post_regrid() 1 1.029e-06 1.029e-06 1.029e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8566 0.8566 0.8566 100.00% Amr::coarseTimeStep() 10 0.6865 0.6865 0.6865 80.14% Amr::timeStep() 10 0.5703 0.5703 0.5703 66.58% Castro::advance() 10 0.5635 0.5635 0.5635 65.77% Castro::subcycle_advance_ctu() 10 0.552 0.552 0.552 64.43% Castro::do_advance_ctu() 10 0.5518 0.5518 0.5518 64.42% Gravity::solve_phi_with_mlmg() 11 0.2828 0.2828 0.2828 33.01% Gravity::actual_solve_with_mlmg() 11 0.2759 0.2759 0.2759 32.21% Castro::construct_new_gravity() 10 0.2634 0.2634 0.2634 30.75% MLMG::solve() 11 0.2564 0.2564 0.2564 29.93% Gravity::solve_for_phi() 10 0.2431 0.2431 0.2431 28.38% MLMG::oneIter() 82 0.2424 0.2424 0.2424 28.30% MLMG::mgVcycle() 82 0.2408 0.2408 0.2408 28.11% VisMF::Write(FabArray) 11 0.227 0.227 0.227 26.49% Castro::construct_ctu_hydro_source() 10 0.2104 0.2104 0.2104 24.56% Amr::checkPoint() 3 0.1689 0.1689 0.1689 19.71% AmrLevel::checkPoint() 3 0.1649 0.1649 0.1649 19.25% StateData::checkPoint() 12 0.1648 0.1648 0.1648 19.24% Amr::init() 1 0.1368 0.1368 0.1368 15.97% MLCellLinOp::smooth() 1640 0.1264 0.1264 0.1264 14.75% MLCellLinOp::applyBC() 4433 0.08883 0.08883 0.08883 10.37% MLMG::mgVcycle_bottom 82 0.07117 0.07117 0.07117 8.31% MLMG::actualBottomSolve() 82 0.07114 0.07114 0.07114 8.30% MLCGSolver::bicgstab 82 0.07046 0.07046 0.07046 8.22% Amr::writePlotFile() 2 0.06513 0.06513 0.06513 7.60% MLPoisson::Fsmooth() 3280 0.0591 0.0591 0.0591 6.90% Amr::initialInit() 1 0.0475 0.0475 0.0475 5.55% FillPatchIterator::Initialize 41 0.04729 0.04729 0.04729 5.52% FillPatchSingleLevel 41 0.04577 0.04577 0.04577 5.34% Castro::clean_state() 62 0.04564 0.04564 0.04564 5.33% Amr::FinalizeInit() 1 0.04323 0.04323 0.04323 5.05% Castro::post_init() 1 0.04192 0.04192 0.04192 4.89% StateDataPhysBCFunct::() 41 0.04174 0.04174 0.04174 4.87% Gravity::multilevel_solve_for_new_phi() 1 0.04015 0.04015 0.04015 4.69% Gravity::actual_multilevel_solve() 1 0.04014 0.04014 0.04014 4.69% MLMG::mgVcycle_down::0 82 0.03437 0.03437 0.03437 4.01% MLCellLinOp::apply() 1142 0.03285 0.03285 0.03285 3.84% MLMG::mgVcycle_up::0 82 0.02934 0.02934 0.02934 3.42% StateData::FillBoundary(geom) 328 0.02285 0.02285 0.02285 2.67% Gravity::get_new_grav_vector() 11 0.02179 0.02179 0.02179 2.54% Castro::initialize_do_advance() 10 0.02082 0.02082 0.02082 2.43% Castro::computeTemp() 63 0.01995 0.01995 0.01995 2.33% amrex::Dot() 1114 0.01973 0.01973 0.01973 2.30% MLCellLinOp::correctionResidual() 492 0.0193 0.0193 0.0193 2.25% MLMG:computeResOfCorrection() 410 0.01676 0.01676 0.01676 1.96% Castro::normalize_species() 62 0.01603 0.01603 0.01603 1.87% MLMG::mgVcycle_down::1 82 0.01587 0.01587 0.01587 1.85% MLPoisson::define() 11 0.01549 0.01549 0.01549 1.81% FabArray::FillBoundary() 4023 0.01542 0.01542 0.01542 1.80% MLMG::mgVcycle_down::2 82 0.01532 0.01532 0.01532 1.79% FillBoundary_nowait() 4023 0.01461 0.01461 0.01461 1.70% Castro::construct_old_gravity() 10 0.01458 0.01458 0.01458 1.70% Gravity::get_old_grav_vector() 10 0.01457 0.01457 0.01457 1.70% MLMG::mgVcycle_down::3 82 0.01452 0.01452 0.01452 1.70% amrex::Copy() 1029 0.01448 0.01448 0.01448 1.69% FabArray::norminf() 743 0.01397 0.01397 0.01397 1.63% FabArray::ParallelCopy() 861 0.0138 0.0138 0.0138 1.61% MLMG::mgVcycle_down::4 82 0.01373 0.01373 0.01373 1.60% FabArray::ParallelCopy_nowait() 861 0.01355 0.01355 0.01355 1.58% FabArray::setVal() 1144 0.01285 0.01285 0.01285 1.50% Castro::do_new_sources() 10 0.01253 0.01253 0.01253 1.46% Castro::expand_state() 10 0.01207 0.01207 0.01207 1.41% MLMG::mgVcycle_up::2 82 0.0119 0.0119 0.0119 1.39% MLCGSolver::ParallelAllReduce 1514 0.01184 0.01184 0.01184 1.38% MLMG::mgVcycle_up::1 82 0.01171 0.01171 0.01171 1.37% MLMG::addInterpCorrection() 410 0.01131 0.01131 0.01131 1.32% MLMG::mgVcycle_up::3 82 0.01124 0.01124 0.01124 1.31% MLMG::mgVcycle_up::4 82 0.01121 0.01121 0.01121 1.31% Castro::do_old_sources() 10 0.01093 0.01093 0.01093 1.28% Castro::initialize_advance() 10 0.01077 0.01077 0.01077 1.26% amrex::average_down 410 0.01059 0.01059 0.01059 1.24% MLCellLinOp::defineAuxData() 11 0.0105 0.0105 0.0105 1.23% MLPoisson::Fapply() 1142 0.01014 0.01014 0.01014 1.18% Castro::enforce_min_density() 62 0.009218 0.009218 0.009218 1.08% FabArray::Saxpy() 813 0.008011 0.008011 0.008011 0.94% FabArray::Xpay() 821 0.007977 0.007977 0.007977 0.93% MLCellLinOp::solutionResidual() 93 0.007051 0.007051 0.007051 0.82% Castro::post_timestep() 10 0.00673 0.00673 0.00673 0.79% Gravity::fill_multipole_BCs() 11 0.006678 0.006678 0.006678 0.78% MLMG::computeResidual() 82 0.006092 0.006092 0.006092 0.71% Castro::reset_internal_energy(MultiFab) 63 0.006082 0.006082 0.006082 0.71% Castro::estTimeStep() 21 0.005474 0.005474 0.005474 0.64% MLCellLinOp::defineBC() 11 0.004591 0.004591 0.004591 0.54% MLMG::prepareForSolve() 11 0.004505 0.004505 0.004505 0.53% BndryData::define() 11 0.004394 0.004394 0.004394 0.51% FabArray::LinComb() 557 0.004352 0.004352 0.004352 0.51% Amr::InitializeInit() 1 0.004275 0.004275 0.004275 0.50% Amr::defBaseLevel() 1 0.00427 0.00427 0.00427 0.50% Castro::initData() 1 0.003796 0.003796 0.003796 0.44% Castro::construct_new_source() 50 0.00326 0.00326 0.00326 0.38% Castro::construct_new_gravity_source() 10 0.003245 0.003245 0.003245 0.38% Castro::computeNewDt() 9 0.002918 0.002918 0.002918 0.34% Castro::construct_old_source() 50 0.00272 0.00272 0.00272 0.32% Castro::construct_old_gravity_source() 10 0.0027 0.0027 0.0027 0.32% amrex::Add() 164 0.002348 0.002348 0.002348 0.27% MLMG::ResNormInf() 93 0.002113 0.002113 0.002113 0.25% Castro::apply_source_to_state() 20 0.001811 0.001811 0.001811 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001587 0.001587 0.001587 0.19% Castro::reset_internal_energy(Fab) 504 0.001542 0.001542 0.001542 0.18% FabArrayBase::getCPC() 1323 0.001398 0.001398 0.001398 0.16% MLMG::getGradSolution() 11 0.001365 0.001365 0.001365 0.16% MLCellLinOp::compGrad() 11 0.001359 0.001359 0.001359 0.16% MLCellLinOp::setLevelBC() 11 0.001348 0.001348 0.001348 0.16% FabArray::mult() 43 0.001315 0.001315 0.001315 0.15% FabArray::setDomainBndry() 41 0.001314 0.001314 0.001314 0.15% Castro::enforce_speed_limit() 62 0.001282 0.001282 0.001282 0.15% Castro::check_for_nan() 20 0.001187 0.001187 0.001187 0.14% MultiFab::contains_nan() 20 0.001175 0.001175 0.001175 0.14% Castro::post_regrid() 1 0.001129 0.001129 0.001129 0.13% MLPoisson::prepareForSolve() 11 0.001094 0.001094 0.001094 0.13% MLCellLinOp::prepareForSolve() 11 0.001085 0.001085 0.001085 0.13% MLMG::computeMLResidual() 11 0.0009993 0.0009993 0.0009993 0.12% Castro::computeInitialDt() 2 0.0009005 0.0009005 0.0009005 0.11% Gravity::update_max_rhs() 11 0.0007956 0.0007956 0.0007956 0.09% FabArrayBase::getFB() 4023 0.0006883 0.0006883 0.0006883 0.08% FabArrayBase::CPC::define() 454 0.0006746 0.0006746 0.0006746 0.08% Castro::finalize_advance() 10 0.0005963 0.0005963 0.0005963 0.07% Amr::InitAmr() 1 0.0004864 0.0004864 0.0004864 0.06% Gravity::swapTimeLevels() 10 0.0004297 0.0004297 0.0004297 0.05% Castro::Castro() 1 0.000415 0.000415 0.000415 0.05% MLLinOp::define() 11 0.0003732 0.0003732 0.0003732 0.04% MLLinOp::defineGrids() 11 0.0003618 0.0003618 0.0003618 0.04% MLMG::MLResNormInf() 11 0.0002814 0.0002814 0.0002814 0.03% MultiFab::max() 11 0.0002562 0.0002562 0.0002562 0.03% MLMG::MLRhsNormInf() 11 0.0002168 0.0002168 0.0002168 0.03% Castro::buildMetrics() 1 0.0001615 0.0001615 0.0001615 0.02% FabArrayBase::FB::FB() 56 8.254e-05 8.254e-05 8.254e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.575e-05 5.575e-05 5.575e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.147e-05 4.147e-05 4.147e-05 0.00% makeSFC 55 4.109e-05 4.109e-05 4.109e-05 0.00% Castro::swap_state_time_levels() 10 3.566e-05 3.566e-05 3.566e-05 0.00% Castro::enforce_consistent_e() 1 3.337e-05 3.337e-05 3.337e-05 0.00% StateData::define() 4 3.275e-05 3.275e-05 3.275e-05 0.00% Castro::finalize_do_advance() 10 3.154e-05 3.154e-05 3.154e-05 0.00% Castro::create_source_corrector() 10 2.446e-05 2.446e-05 2.446e-05 0.00% Amr::writeSmallPlotFile() 1 2.335e-05 2.335e-05 2.335e-05 0.00% Castro::initMFs() 1 1.903e-05 1.903e-05 1.903e-05 0.00% DistributionMapping::Distribute() 56 1.605e-05 1.605e-05 1.605e-05 0.00% Amr::initSubcycle() 1 8.788e-06 8.788e-06 8.788e-06 0.00% AmrLevel::checkPointPost() 3 5.572e-06 5.572e-06 5.572e-06 0.00% Castro::retry_advance_ctu() 10 4.355e-06 4.355e-06 4.355e-06 0.00% Gravity::set_mass_offset() 11 4.085e-06 4.085e-06 4.085e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.824e-06 3.824e-06 3.824e-06 0.00% Castro::FluxRegCrseInit 10 3.225e-06 3.225e-06 3.225e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.119e-06 2.119e-06 2.119e-06 0.00% Castro::FluxRegFineAdd() 10 1.944e-06 1.944e-06 1.944e-06 0.00% AmrLevel::checkPointPre() 3 1.898e-06 1.898e-06 1.898e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-16-g96d8c53fbf09) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-16-g96d8c53fbf09) initialized Starting run at 09:59:54 UTC on 2022-12-19. Successfully read inputs file ... Castro git describe: 22.12-5-g3ad2ef0c0 AMReX git describe: 22.12-16-g96d8c53fb Microphysics git describe: 22.12-23-gacd17b4d reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.45745749 Restart time = 0.047797561 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.049521708 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049420236 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.057118791 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057012587 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.071637938 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032437589 seconds Ending run at 09:59:55 UTC on 2022-12-19. Run time = 0.365973056 Run time without initialization = 0.317593115 Average number of zones advanced per microsecond: 4.127 Average number of zones advanced per microsecond per rank: 4.127 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.366 ... 0.366 ... 0.366 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1015 0.1015 0.1015 27.74% VisMF::Read() 3 0.04172 0.04172 0.04172 11.40% MLCellLinOp::applyBC() 1946 0.03176 0.03176 0.03176 8.68% VisMF::Write(FabArray) 1 0.03088 0.03088 0.03088 8.44% MLPoisson::Fsmooth() 1440 0.02511 0.02511 0.02511 6.86% StateData::FillBoundary(geom) 160 0.01072 0.01072 0.01072 2.93% Castro::normalize_species() 30 0.01004 0.01004 0.01004 2.74% amrex::Dot() 484 0.008422 0.008422 0.008422 2.30% Castro::computeTemp() 30 0.00761 0.00761 0.00761 2.08% amrex::Copy() 463 0.006832 0.006832 0.006832 1.87% FillBoundary_nowait() 1766 0.006095 0.006095 0.006095 1.67% FabArray::setVal() 537 0.006068 0.006068 0.006068 1.66% FabArray::norminf() 326 0.006002 0.006002 0.006002 1.64% FabArray::ParallelCopy_nowait() 380 0.005789 0.005789 0.005789 1.58% StateDataPhysBCFunct::() 20 0.005201 0.005201 0.005201 1.42% MLCellLinOp::defineAuxData() 6 0.005009 0.005009 0.005009 1.37% MLPoisson::Fapply() 500 0.004392 0.004392 0.004392 1.20% FabArray::Saxpy() 355 0.003588 0.003588 0.003588 0.98% FabArray::Xpay() 361 0.003463 0.003463 0.003463 0.95% Castro::enforce_min_density() 30 0.003151 0.003151 0.003151 0.86% MLMG::addInterpCorrection() 180 0.002738 0.002738 0.002738 0.75% Gravity::fill_multipole_BCs() 6 0.002603 0.002603 0.002603 0.71% amrex::average_down 180 0.002478 0.002478 0.002478 0.68% Amr::restart() 1 0.002467 0.002467 0.002467 0.67% Castro::estTimeStep() 10 0.002285 0.002285 0.002285 0.62% BndryData::define() 6 0.001909 0.001909 0.001909 0.52% Castro::reset_internal_energy(MultiFab) 30 0.001872 0.001872 0.001872 0.51% Castro::do_advance_ctu() 5 0.001871 0.001871 0.001871 0.51% FabArray::LinComb() 242 0.001863 0.001863 0.001863 0.51% Castro::construct_new_gravity_source() 5 0.001771 0.001771 0.001771 0.48% Castro::construct_old_gravity_source() 5 0.001468 0.001468 0.001468 0.40% Amr::writePlotFile() 1 0.001387 0.001387 0.001387 0.38% amrex::Add() 72 0.001026 0.001026 0.001026 0.28% Castro::enforce_speed_limit() 30 0.001024 0.001024 0.001024 0.28% MLCGSolver::bicgstab 36 0.0009899 0.0009899 0.0009899 0.27% Castro::reset_internal_energy(Fab) 240 0.0009277 0.0009277 0.0009277 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000862 0.000862 0.000862 0.24% Castro::post_timestep() 5 0.0008291 0.0008291 0.0008291 0.23% MLCellLinOp::setLevelBC() 6 0.0007181 0.0007181 0.0007181 0.20% Castro::create_source_corrector() 5 0.0007147 0.0007147 0.0007147 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007022 0.0007022 0.0007022 0.19% FabArray::mult() 22 0.0006549 0.0006549 0.0006549 0.18% FabArray::setDomainBndry() 20 0.000621 0.000621 0.000621 0.17% MultiFab::contains_nan() 10 0.0005972 0.0005972 0.0005972 0.16% MLCellLinOp::prepareForSolve() 6 0.0005843 0.0005843 0.0005843 0.16% MLCellLinOp::smooth() 720 0.0005708 0.0005708 0.0005708 0.16% MLCellLinOp::compGrad() 6 0.0004818 0.0004818 0.0004818 0.13% MLMG::prepareForSolve() 6 0.0004506 0.0004506 0.0004506 0.12% FabArrayBase::CPC::define() 244 0.0004036 0.0004036 0.0004036 0.11% FabArray::FillBoundary() 1766 0.0004012 0.0004012 0.0004012 0.11% Amr::InitAmr() 1 0.0003959 0.0003959 0.0003959 0.11% FabArrayBase::getCPC() 632 0.0003577 0.0003577 0.0003577 0.10% Gravity::get_old_grav_vector() 5 0.000315 0.000315 0.000315 0.09% FabArrayBase::getFB() 1766 0.0002898 0.0002898 0.0002898 0.08% main() 1 0.0002696 0.0002696 0.0002696 0.07% Gravity::get_new_grav_vector() 5 0.0002632 0.0002632 0.0002632 0.07% MLCellLinOp::apply() 500 0.0002306 0.0002306 0.0002306 0.06% Amr::coarseTimeStep() 5 0.0002193 0.0002193 0.0002193 0.06% MLMG::mgVcycle() 36 0.0001733 0.0001733 0.0001733 0.05% MLCGSolver::ParallelAllReduce 659 0.000139 0.000139 0.000139 0.04% MultiFab::max() 6 0.000135 0.000135 0.000135 0.04% Castro::subcycle_advance_ctu() 5 0.0001305 0.0001305 0.0001305 0.04% Castro::construct_new_source() 25 0.0001283 0.0001283 0.0001283 0.04% MLLinOp::defineGrids() 6 0.0001187 0.0001187 0.0001187 0.03% FabArray::ParallelCopy() 380 0.0001177 0.0001177 0.0001177 0.03% MLCellLinOp::correctionResidual() 216 0.0001089 0.0001089 0.0001089 0.03% MLCellLinOp::defineBC() 6 0.0001064 0.0001064 0.0001064 0.03% Castro::finalize_advance() 5 0.0001021 0.0001021 0.0001021 0.03% FillPatchIterator::Initialize 20 9.877e-05 9.877e-05 9.877e-05 0.03% Amr::timeStep() 5 8.008e-05 8.008e-05 8.008e-05 0.02% AmrLevel::restart() 1 7.438e-05 7.438e-05 7.438e-05 0.02% Gravity::solve_for_phi() 5 6.92e-05 6.92e-05 6.92e-05 0.02% Gravity::update_max_rhs() 6 6.139e-05 6.139e-05 6.139e-05 0.02% StateData::restartDoit() 4 5.987e-05 5.987e-05 5.987e-05 0.02% FabArrayBase::FB::FB() 26 5.859e-05 5.859e-05 5.859e-05 0.02% MLMG:computeResOfCorrection() 180 5.736e-05 5.736e-05 5.736e-05 0.02% Castro::initialize_do_advance() 5 5.328e-05 5.328e-05 5.328e-05 0.01% Castro::construct_new_gravity() 5 5.155e-05 5.155e-05 5.155e-05 0.01% MLMG::mgVcycle_down::0 36 4.53e-05 4.53e-05 4.53e-05 0.01% MLMG::mgVcycle_down::1 36 4.325e-05 4.325e-05 4.325e-05 0.01% MLMG::actualBottomSolve() 36 4.208e-05 4.208e-05 4.208e-05 0.01% Castro::clean_state() 30 4.108e-05 4.108e-05 4.108e-05 0.01% MLMG::mgVcycle_down::4 36 3.949e-05 3.949e-05 3.949e-05 0.01% MLMG::mgVcycle_down::2 36 3.927e-05 3.927e-05 3.927e-05 0.01% MLMG::mgVcycle_down::3 36 3.844e-05 3.844e-05 3.844e-05 0.01% MLMG::solve() 6 3.821e-05 3.821e-05 3.821e-05 0.01% Castro::computeNewDt() 5 3.779e-05 3.779e-05 3.779e-05 0.01% Castro::expand_state() 5 3.726e-05 3.726e-05 3.726e-05 0.01% Castro::advance() 5 3.274e-05 3.274e-05 3.274e-05 0.01% Castro::initialize_advance() 5 3.231e-05 3.231e-05 3.231e-05 0.01% Castro::finalize_do_advance() 5 3.028e-05 3.028e-05 3.028e-05 0.01% MLMG::mgVcycle_up::4 36 3.003e-05 3.003e-05 3.003e-05 0.01% Castro::construct_old_source() 25 2.973e-05 2.973e-05 2.973e-05 0.01% MLMG::mgVcycle_up::0 36 2.725e-05 2.725e-05 2.725e-05 0.01% Castro::buildMetrics() 1 2.703e-05 2.703e-05 2.703e-05 0.01% MLMG::mgVcycle_up::3 36 2.631e-05 2.631e-05 2.631e-05 0.01% MLMG::mgVcycle_up::2 36 2.598e-05 2.598e-05 2.598e-05 0.01% MLMG::oneIter() 36 2.54e-05 2.54e-05 2.54e-05 0.01% MLMG::mgVcycle_up::1 36 2.483e-05 2.483e-05 2.483e-05 0.01% Amr::writeSmallPlotFile() 1 2.477e-05 2.477e-05 2.477e-05 0.01% MLCellLinOp::solutionResidual() 42 2.453e-05 2.453e-05 2.453e-05 0.01% Castro::swap_state_time_levels() 5 2.191e-05 2.191e-05 2.191e-05 0.01% Castro::post_restart() 1 2.178e-05 2.178e-05 2.178e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 2.062e-05 2.062e-05 2.062e-05 0.01% Castro::initMFs() 1 1.893e-05 1.893e-05 1.893e-05 0.01% MLMG::ResNormInf() 42 1.836e-05 1.836e-05 1.836e-05 0.01% MLMG::mgVcycle_bottom 36 1.614e-05 1.614e-05 1.614e-05 0.00% FillPatchSingleLevel 20 1.526e-05 1.526e-05 1.526e-05 0.00% makeSFC 30 1.514e-05 1.514e-05 1.514e-05 0.00% MLMG::computeResidual() 36 1.491e-05 1.491e-05 1.491e-05 0.00% MLPoisson::define() 6 1.414e-05 1.414e-05 1.414e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.237e-05 1.237e-05 1.237e-05 0.00% Castro::do_new_sources() 5 1.014e-05 1.014e-05 1.014e-05 0.00% DistributionMapping::Distribute() 31 9.826e-06 9.826e-06 9.826e-06 0.00% Castro::do_old_sources() 5 8.974e-06 8.974e-06 8.974e-06 0.00% Amr::initSubcycle() 1 8.45e-06 8.45e-06 8.45e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.721e-06 7.721e-06 7.721e-06 0.00% Castro::check_for_nan() 10 6.783e-06 6.783e-06 6.783e-06 0.00% Gravity::actual_multilevel_solve() 1 6.62e-06 6.62e-06 6.62e-06 0.00% Castro::apply_source_to_state() 10 5.867e-06 5.867e-06 5.867e-06 0.00% MLLinOp::define() 6 5.689e-06 5.689e-06 5.689e-06 0.00% Castro::construct_old_gravity() 5 5.037e-06 5.037e-06 5.037e-06 0.00% Gravity::swapTimeLevels() 5 4.344e-06 4.344e-06 4.344e-06 0.00% MLPoisson::prepareForSolve() 6 4.34e-06 4.34e-06 4.34e-06 0.00% MLMG::computeMLResidual() 6 3.673e-06 3.673e-06 3.673e-06 0.00% MLMG::getGradSolution() 6 3.267e-06 3.267e-06 3.267e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.844e-06 2.844e-06 2.844e-06 0.00% Gravity::set_mass_offset() 6 2.473e-06 2.473e-06 2.473e-06 0.00% MLMG::MLResNormInf() 6 2.462e-06 2.462e-06 2.462e-06 0.00% MLMG::MLRhsNormInf() 6 2.381e-06 2.381e-06 2.381e-06 0.00% Castro::retry_advance_ctu() 5 2.367e-06 2.367e-06 2.367e-06 0.00% Castro::FluxRegCrseInit 5 2.307e-06 2.307e-06 2.307e-06 0.00% Castro::FluxRegFineAdd() 5 1.737e-06 1.737e-06 1.737e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.235e-06 1.235e-06 1.235e-06 0.00% Amr::init() 1 8.7e-07 8.7e-07 8.7e-07 0.00% AmrLevel::AmrLevel() 1 8.67e-07 8.67e-07 8.67e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.366 0.366 0.366 99.99% Amr::coarseTimeStep() 5 0.2849 0.2849 0.2849 77.85% Amr::timeStep() 5 0.2835 0.2835 0.2835 77.45% Castro::advance() 5 0.2777 0.2777 0.2777 75.87% Castro::subcycle_advance_ctu() 5 0.2716 0.2716 0.2716 74.21% Castro::do_advance_ctu() 5 0.2715 0.2715 0.2715 74.17% Castro::construct_new_gravity() 5 0.1288 0.1288 0.1288 35.19% Gravity::solve_phi_with_mlmg() 6 0.1247 0.1247 0.1247 34.07% Gravity::actual_solve_with_mlmg() 6 0.1218 0.1218 0.1218 33.29% Gravity::solve_for_phi() 5 0.1218 0.1218 0.1218 33.27% MLMG::solve() 6 0.1112 0.1112 0.1112 30.39% MLMG::oneIter() 36 0.1046 0.1046 0.1046 28.57% MLMG::mgVcycle() 36 0.1038 0.1038 0.1038 28.37% Castro::construct_ctu_hydro_source() 5 0.1015 0.1015 0.1015 27.74% MLCellLinOp::smooth() 720 0.0543 0.0543 0.0543 14.84% Amr::init() 1 0.04784 0.04784 0.04784 13.07% Amr::restart() 1 0.04784 0.04784 0.04784 13.07% AmrLevel::restart() 1 0.04191 0.04191 0.04191 11.45% StateData::restartDoit() 4 0.04183 0.04183 0.04183 11.43% VisMF::Read() 3 0.04172 0.04172 0.04172 11.40% MLCellLinOp::applyBC() 1946 0.0386 0.0386 0.0386 10.55% Amr::writePlotFile() 1 0.03252 0.03252 0.03252 8.89% VisMF::Write(FabArray) 1 0.03088 0.03088 0.03088 8.44% MLMG::mgVcycle_bottom 36 0.03068 0.03068 0.03068 8.38% MLMG::actualBottomSolve() 36 0.03066 0.03066 0.03066 8.38% MLCGSolver::bicgstab 36 0.03036 0.03036 0.03036 8.30% MLPoisson::Fsmooth() 1440 0.02511 0.02511 0.02511 6.86% Castro::clean_state() 30 0.02467 0.02467 0.02467 6.74% FillPatchIterator::Initialize 20 0.01862 0.01862 0.01862 5.09% FillPatchSingleLevel 20 0.0179 0.0179 0.0179 4.89% StateDataPhysBCFunct::() 20 0.01592 0.01592 0.01592 4.35% MLMG::mgVcycle_down::0 36 0.0148 0.0148 0.0148 4.04% MLCellLinOp::apply() 500 0.01435 0.01435 0.01435 3.92% MLMG::mgVcycle_up::0 36 0.01263 0.01263 0.01263 3.45% StateData::FillBoundary(geom) 160 0.01072 0.01072 0.01072 2.93% Castro::computeTemp() 30 0.01041 0.01041 0.01041 2.84% Castro::initialize_do_advance() 5 0.01017 0.01017 0.01017 2.78% Castro::normalize_species() 30 0.01004 0.01004 0.01004 2.74% MLPoisson::define() 6 0.008444 0.008444 0.008444 2.31% amrex::Dot() 484 0.008422 0.008422 0.008422 2.30% MLCellLinOp::correctionResidual() 216 0.008395 0.008395 0.008395 2.29% Castro::do_new_sources() 5 0.007874 0.007874 0.007874 2.15% MLMG:computeResOfCorrection() 180 0.007283 0.007283 0.007283 1.99% Castro::construct_old_gravity() 5 0.006872 0.006872 0.006872 1.88% Gravity::get_old_grav_vector() 5 0.006867 0.006867 0.006867 1.88% Gravity::get_new_grav_vector() 5 0.006865 0.006865 0.006865 1.88% FabArray::FillBoundary() 1766 0.006845 0.006845 0.006845 1.87% MLMG::mgVcycle_down::1 36 0.006844 0.006844 0.006844 1.87% amrex::Copy() 463 0.006832 0.006832 0.006832 1.87% MLMG::mgVcycle_down::2 36 0.006638 0.006638 0.006638 1.81% FillBoundary_nowait() 1766 0.006444 0.006444 0.006444 1.76% FabArray::ParallelCopy() 380 0.006281 0.006281 0.006281 1.72% MLMG::mgVcycle_down::3 36 0.006275 0.006275 0.006275 1.71% FabArray::ParallelCopy_nowait() 380 0.006163 0.006163 0.006163 1.68% FabArray::setVal() 537 0.006068 0.006068 0.006068 1.66% FabArray::norminf() 326 0.006002 0.006002 0.006002 1.64% MLMG::mgVcycle_down::4 36 0.005986 0.005986 0.005986 1.64% MLCellLinOp::defineAuxData() 6 0.005731 0.005731 0.005731 1.57% Castro::post_timestep() 5 0.005717 0.005717 0.005717 1.56% Castro::initialize_advance() 5 0.005654 0.005654 0.005654 1.54% Castro::expand_state() 5 0.005635 0.005635 0.005635 1.54% Castro::do_old_sources() 5 0.005431 0.005431 0.005431 1.48% MLMG::mgVcycle_up::2 36 0.00511 0.00511 0.00511 1.40% MLCGSolver::ParallelAllReduce 659 0.005104 0.005104 0.005104 1.39% MLMG::mgVcycle_up::1 36 0.005024 0.005024 0.005024 1.37% MLMG::addInterpCorrection() 180 0.00489 0.00489 0.00489 1.34% MLMG::mgVcycle_up::4 36 0.004844 0.004844 0.004844 1.32% MLMG::mgVcycle_up::3 36 0.004844 0.004844 0.004844 1.32% amrex::average_down 180 0.004648 0.004648 0.004648 1.27% MLPoisson::Fapply() 500 0.004392 0.004392 0.004392 1.20% FabArray::Saxpy() 355 0.003588 0.003588 0.003588 0.98% FabArray::Xpay() 361 0.003463 0.003463 0.003463 0.95% Castro::post_restart() 1 0.003298 0.003298 0.003298 0.90% Gravity::multilevel_solve_for_new_phi() 1 0.003186 0.003186 0.003186 0.87% MLCellLinOp::solutionResidual() 42 0.003167 0.003167 0.003167 0.87% Gravity::actual_multilevel_solve() 1 0.003166 0.003166 0.003166 0.86% Castro::enforce_min_density() 30 0.003151 0.003151 0.003151 0.86% Castro::reset_internal_energy(MultiFab) 30 0.002799 0.002799 0.002799 0.76% Gravity::fill_multipole_BCs() 6 0.002742 0.002742 0.002742 0.75% MLMG::computeResidual() 36 0.002631 0.002631 0.002631 0.72% MLCellLinOp::defineBC() 6 0.002542 0.002542 0.002542 0.69% BndryData::define() 6 0.002436 0.002436 0.002436 0.67% MLMG::prepareForSolve() 6 0.002406 0.002406 0.002406 0.66% Castro::estTimeStep() 10 0.002285 0.002285 0.002285 0.62% Castro::construct_new_source() 25 0.001899 0.001899 0.001899 0.52% FabArray::LinComb() 242 0.001863 0.001863 0.001863 0.51% Castro::construct_new_gravity_source() 5 0.001771 0.001771 0.001771 0.48% Castro::construct_old_source() 25 0.001498 0.001498 0.001498 0.41% Castro::construct_old_gravity_source() 5 0.001468 0.001468 0.001468 0.40% Castro::computeNewDt() 5 0.001234 0.001234 0.001234 0.34% amrex::Add() 72 0.001026 0.001026 0.001026 0.28% Castro::enforce_speed_limit() 30 0.001024 0.001024 0.001024 0.28% MLMG::ResNormInf() 42 0.0009329 0.0009329 0.0009329 0.25% Castro::reset_internal_energy(Fab) 240 0.0009277 0.0009277 0.0009277 0.25% Castro::apply_source_to_state() 10 0.0009236 0.0009236 0.0009236 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000862 0.000862 0.000862 0.24% FabArrayBase::getCPC() 632 0.0007613 0.0007613 0.0007613 0.21% MLMG::getGradSolution() 6 0.0007405 0.0007405 0.0007405 0.20% MLCellLinOp::compGrad() 6 0.0007372 0.0007372 0.0007372 0.20% MLCellLinOp::setLevelBC() 6 0.0007181 0.0007181 0.0007181 0.20% Castro::create_source_corrector() 5 0.0007147 0.0007147 0.0007147 0.20% FabArray::mult() 22 0.0006549 0.0006549 0.0006549 0.18% FabArray::setDomainBndry() 20 0.000621 0.000621 0.000621 0.17% Castro::check_for_nan() 10 0.000604 0.000604 0.000604 0.17% MultiFab::contains_nan() 10 0.0005972 0.0005972 0.0005972 0.16% MLPoisson::prepareForSolve() 6 0.0005886 0.0005886 0.0005886 0.16% MLCellLinOp::prepareForSolve() 6 0.0005843 0.0005843 0.0005843 0.16% MLMG::computeMLResidual() 6 0.0005551 0.0005551 0.0005551 0.15% Gravity::update_max_rhs() 6 0.0004393 0.0004393 0.0004393 0.12% Amr::InitAmr() 1 0.0004043 0.0004043 0.0004043 0.11% FabArrayBase::CPC::define() 244 0.0004036 0.0004036 0.0004036 0.11% Castro::finalize_advance() 5 0.000376 0.000376 0.000376 0.10% FabArrayBase::getFB() 1766 0.0003484 0.0003484 0.0003484 0.10% Gravity::swapTimeLevels() 5 0.0002185 0.0002185 0.0002185 0.06% MLLinOp::define() 6 0.0001568 0.0001568 0.0001568 0.04% MLLinOp::defineGrids() 6 0.0001511 0.0001511 0.0001511 0.04% MLMG::MLResNormInf() 6 0.0001492 0.0001492 0.0001492 0.04% Castro::buildMetrics() 1 0.000143 0.000143 0.000143 0.04% MultiFab::max() 6 0.000135 0.000135 0.000135 0.04% MLMG::MLRhsNormInf() 6 0.0001144 0.0001144 0.0001144 0.03% FabArrayBase::FB::FB() 26 5.859e-05 5.859e-05 5.859e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 3.116e-05 3.116e-05 3.116e-05 0.01% Castro::finalize_do_advance() 5 3.028e-05 3.028e-05 3.028e-05 0.01% Amr::writeSmallPlotFile() 1 2.477e-05 2.477e-05 2.477e-05 0.01% makeSFC 30 2.344e-05 2.344e-05 2.344e-05 0.01% Castro::swap_state_time_levels() 5 2.191e-05 2.191e-05 2.191e-05 0.01% Castro::initMFs() 1 1.893e-05 1.893e-05 1.893e-05 0.01% DistributionMapping::Distribute() 31 9.826e-06 9.826e-06 9.826e-06 0.00% Amr::initSubcycle() 1 8.45e-06 8.45e-06 8.45e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.371e-06 4.371e-06 4.371e-06 0.00% Gravity::set_mass_offset() 6 2.473e-06 2.473e-06 2.473e-06 0.00% Castro::retry_advance_ctu() 5 2.367e-06 2.367e-06 2.367e-06 0.00% Castro::FluxRegCrseInit 5 2.307e-06 2.307e-06 2.307e-06 0.00% Castro::FluxRegFineAdd() 5 1.737e-06 1.737e-06 1.737e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.235e-06 1.235e-06 1.235e-06 0.00% AmrLevel::AmrLevel() 1 8.67e-07 8.67e-07 8.67e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-16-g96d8c53fbf09) finalized