Initializing CUDA... CUDA initialized with 1 device. AMReX (23.01-26-ga232be4e275b) initialized Starting run at 10:11:31 UTC on 2023-01-26. Successfully read inputs file ... Castro git describe: 23.01-20-g08651568d AMReX git describe: 23.01-26-ga232be4e2 Microphysics git describe: 23.01-5-g4a082ffc reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.059025294 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.034066258 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.04774644 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.047972033 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.049998236 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.054969377 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.073983595 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.058162035 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.063076036 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.057884482 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.056605261 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059142435 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.060554897 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.057837907 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.03364178 seconds Ending run at 10:11:31 UTC on 2023-01-26. Run time = 0.860852218 Run time without initialization = 0.722194089 Average number of zones advanced per microsecond: 3.630 Average number of zones advanced per microsecond per rank: 3.630 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.8609 ... 0.8609 ... 0.8609 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2366 0.2366 0.2366 27.49% Castro::construct_ctu_hydro_source() 10 0.214 0.214 0.214 24.86% MLCellLinOp::applyBC() 4379 0.0749 0.0749 0.0749 8.70% MLPoisson::Fsmooth() 3240 0.0324 0.0324 0.0324 3.76% FillBoundary_nowait() 3974 0.03076 0.03076 0.03076 3.57% StateData::FillBoundary(geom) 328 0.02352 0.02352 0.02352 2.73% amrex::Dot() 1100 0.02042 0.02042 0.02042 2.37% amrex::Copy() 1019 0.01486 0.01486 0.01486 1.73% Castro::computeTemp() 63 0.01454 0.01454 0.01454 1.69% FabArray::norminf() 734 0.01433 0.01433 0.01433 1.66% Castro::normalize_species() 62 0.01426 0.01426 0.01426 1.66% StateDataPhysBCFunct::() 41 0.01367 0.01367 0.01367 1.59% FabArray::setVal() 1135 0.01323 0.01323 0.01323 1.54% FabArray::ParallelCopy_nowait() 851 0.01313 0.01313 0.01313 1.53% MLPoisson::Fapply() 1128 0.01042 0.01042 0.01042 1.21% MLCellLinOp::defineAuxData() 11 0.009659 0.009659 0.009659 1.12% Castro::enforce_min_density() 62 0.00944 0.00944 0.00944 1.10% FabArray::Saxpy() 803 0.008136 0.008136 0.008136 0.95% FabArray::Xpay() 811 0.008097 0.008097 0.008097 0.94% MLMG::addInterpCorrection() 405 0.006528 0.006528 0.006528 0.76% Gravity::fill_multipole_BCs() 11 0.00642 0.00642 0.00642 0.75% amrex::average_down 405 0.005814 0.005814 0.005814 0.68% FabArray::LinComb() 550 0.004509 0.004509 0.004509 0.52% amrex::Add() 162 0.004286 0.004286 0.004286 0.50% Castro::estTimeStep() 21 0.004264 0.004264 0.004264 0.50% Castro::reset_internal_energy(MultiFab) 63 0.004084 0.004084 0.004084 0.47% BndryData::define() 11 0.003662 0.003662 0.003662 0.43% Amr::checkPoint() 3 0.003579 0.003579 0.003579 0.42% Castro::do_advance_ctu() 10 0.003312 0.003312 0.003312 0.38% Castro::construct_new_gravity_source() 10 0.003271 0.003271 0.003271 0.38% Castro::construct_old_gravity_source() 10 0.002626 0.002626 0.002626 0.30% Castro::reset_internal_energy(Fab) 504 0.002356 0.002356 0.002356 0.27% Amr::writePlotFile() 2 0.002119 0.002119 0.002119 0.25% MLCGSolver::bicgstab 81 0.002002 0.002002 0.002002 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001658 0.001658 0.001658 0.19% Castro::enforce_speed_limit() 62 0.001403 0.001403 0.001403 0.16% MLCellLinOp::setLevelBC() 11 0.00138 0.00138 0.00138 0.16% Gravity::actual_solve_with_mlmg() 11 0.001357 0.001357 0.001357 0.16% FabArray::mult() 43 0.001342 0.001342 0.001342 0.16% FabArray::setDomainBndry() 41 0.001322 0.001322 0.001322 0.15% Castro::initData() 1 0.001263 0.001263 0.001263 0.15% MultiFab::contains_nan() 20 0.001192 0.001192 0.001192 0.14% MLCellLinOp::smooth() 1620 0.001125 0.001125 0.001125 0.13% MLCellLinOp::prepareForSolve() 11 0.00111 0.00111 0.00111 0.13% MLCellLinOp::compGrad() 11 0.0009188 0.0009188 0.0009188 0.11% MLMG::prepareForSolve() 11 0.0008188 0.0008188 0.0008188 0.10% FabArray::FillBoundary() 3974 0.0007816 0.0007816 0.0007816 0.09% FabArrayBase::getCPC() 1313 0.0007365 0.0007365 0.0007365 0.09% FabArrayBase::CPC::define() 454 0.0006865 0.0006865 0.0006865 0.08% Gravity::get_new_grav_vector() 11 0.0006097 0.0006097 0.0006097 0.07% FabArrayBase::getFB() 3974 0.0005655 0.0005655 0.0005655 0.07% Gravity::get_old_grav_vector() 10 0.000532 0.000532 0.000532 0.06% Amr::InitAmr() 1 0.0005149 0.0005149 0.0005149 0.06% MLCellLinOp::apply() 1128 0.0004775 0.0004775 0.0004775 0.06% MLMG::mgVcycle() 81 0.000383 0.000383 0.000383 0.04% Amr::coarseTimeStep() 10 0.0003245 0.0003245 0.0003245 0.04% main() 1 0.0002919 0.0002919 0.0002919 0.03% MLCGSolver::ParallelAllReduce 1495 0.000278 0.000278 0.000278 0.03% MultiFab::max() 11 0.0002595 0.0002595 0.0002595 0.03% FabArray::ParallelCopy() 851 0.000237 0.000237 0.000237 0.03% MLCellLinOp::correctionResidual() 486 0.0002164 0.0002164 0.0002164 0.03% MLLinOp::defineGrids() 11 0.0002144 0.0002144 0.0002144 0.02% FillPatchIterator::Initialize 41 0.0002108 0.0002108 0.0002108 0.02% MLCellLinOp::defineBC() 11 0.0001988 0.0001988 0.0001988 0.02% Castro::subcycle_advance_ctu() 10 0.0001536 0.0001536 0.0001536 0.02% Amr::timeStep() 10 0.0001526 0.0001526 0.0001526 0.02% StateData::checkPoint() 12 0.0001339 0.0001339 0.0001339 0.02% Gravity::solve_for_phi() 10 0.0001314 0.0001314 0.0001314 0.02% MLMG:computeResOfCorrection() 405 0.0001153 0.0001153 0.0001153 0.01% Gravity::update_max_rhs() 11 0.000107 0.000107 0.000107 0.01% Castro::finalize_advance() 10 0.000106 0.000106 0.000106 0.01% MLMG::mgVcycle_down::0 81 9.65e-05 9.65e-05 9.65e-05 0.01% MLMG::mgVcycle_down::2 81 9.089e-05 9.089e-05 9.089e-05 0.01% MLMG::actualBottomSolve() 81 9.043e-05 9.043e-05 9.043e-05 0.01% Castro::advance() 10 8.783e-05 8.783e-05 8.783e-05 0.01% AmrLevel::checkPoint() 3 8.636e-05 8.636e-05 8.636e-05 0.01% Castro::clean_state() 62 8.48e-05 8.48e-05 8.48e-05 0.01% FabArrayBase::FB::FB() 56 8.222e-05 8.222e-05 8.222e-05 0.01% MLMG::solve() 11 8.211e-05 8.211e-05 8.211e-05 0.01% MLMG::mgVcycle_down::1 81 7.863e-05 7.863e-05 7.863e-05 0.01% Castro::Castro() 1 7.453e-05 7.453e-05 7.453e-05 0.01% Castro::expand_state() 10 7.434e-05 7.434e-05 7.434e-05 0.01% MLMG::mgVcycle_down::4 81 7.08e-05 7.08e-05 7.08e-05 0.01% MLMG::mgVcycle_down::3 81 7.079e-05 7.079e-05 7.079e-05 0.01% Castro::initialize_advance() 10 6.553e-05 6.553e-05 6.553e-05 0.01% MLMG::mgVcycle_up::4 81 6.16e-05 6.16e-05 6.16e-05 0.01% MLMG::mgVcycle_up::0 81 5.66e-05 5.66e-05 5.66e-05 0.01% MLMG::oneIter() 81 5.228e-05 5.228e-05 5.228e-05 0.01% MLMG::mgVcycle_up::3 81 5.2e-05 5.2e-05 5.2e-05 0.01% MLMG::mgVcycle_up::1 81 5.165e-05 5.165e-05 5.165e-05 0.01% MLMG::mgVcycle_up::2 81 5.011e-05 5.011e-05 5.011e-05 0.01% MLCellLinOp::solutionResidual() 92 4.997e-05 4.997e-05 4.997e-05 0.01% Castro::initialize_do_advance() 10 4.287e-05 4.287e-05 4.287e-05 0.00% Castro::enforce_consistent_e() 1 3.644e-05 3.644e-05 3.644e-05 0.00% Castro::finalize_do_advance() 10 3.525e-05 3.525e-05 3.525e-05 0.00% MLMG::ResNormInf() 92 3.429e-05 3.429e-05 3.429e-05 0.00% Castro::swap_state_time_levels() 10 3.356e-05 3.356e-05 3.356e-05 0.00% MLMG::mgVcycle_bottom 81 3.159e-05 3.159e-05 3.159e-05 0.00% MLMG::computeResidual() 81 3.142e-05 3.142e-05 3.142e-05 0.00% FillPatchSingleLevel 41 2.98e-05 2.98e-05 2.98e-05 0.00% StateData::define() 4 2.565e-05 2.565e-05 2.565e-05 0.00% makeSFC 55 2.499e-05 2.499e-05 2.499e-05 0.00% Amr::writeSmallPlotFile() 1 2.456e-05 2.456e-05 2.456e-05 0.00% Castro::construct_new_gravity() 10 2.439e-05 2.439e-05 2.439e-05 0.00% Castro::create_source_corrector() 10 2.343e-05 2.343e-05 2.343e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.328e-05 2.328e-05 2.328e-05 0.00% MLPoisson::define() 11 2.081e-05 2.081e-05 2.081e-05 0.00% Amr::FinalizeInit() 1 2.07e-05 2.07e-05 2.07e-05 0.00% Amr::defBaseLevel() 1 1.904e-05 1.904e-05 1.904e-05 0.00% Castro::initMFs() 1 1.826e-05 1.826e-05 1.826e-05 0.00% Castro::construct_new_source() 50 1.762e-05 1.762e-05 1.762e-05 0.00% Castro::do_new_sources() 10 1.684e-05 1.684e-05 1.684e-05 0.00% Castro::construct_old_source() 50 1.672e-05 1.672e-05 1.672e-05 0.00% Castro::buildMetrics() 1 1.629e-05 1.629e-05 1.629e-05 0.00% Castro::do_old_sources() 10 1.589e-05 1.589e-05 1.589e-05 0.00% DistributionMapping::Distribute() 56 1.487e-05 1.487e-05 1.487e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.415e-05 1.415e-05 1.415e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.409e-05 1.409e-05 1.409e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.301e-05 1.301e-05 1.301e-05 0.00% Castro::check_for_nan() 20 1.294e-05 1.294e-05 1.294e-05 0.00% MLLinOp::define() 11 1.017e-05 1.017e-05 1.017e-05 0.00% Amr::InitializeInit() 1 9.888e-06 9.888e-06 9.888e-06 0.00% Castro::construct_old_gravity() 10 9.768e-06 9.768e-06 9.768e-06 0.00% Castro::apply_source_to_state() 20 9.731e-06 9.731e-06 9.731e-06 0.00% Castro::post_timestep() 10 9.468e-06 9.468e-06 9.468e-06 0.00% Gravity::swapTimeLevels() 10 9.284e-06 9.284e-06 9.284e-06 0.00% Castro::post_init() 1 9.191e-06 9.191e-06 9.191e-06 0.00% Amr::initSubcycle() 1 8.538e-06 8.538e-06 8.538e-06 0.00% MLMG::computeMLResidual() 11 8.057e-06 8.057e-06 8.057e-06 0.00% MLPoisson::prepareForSolve() 11 7.438e-06 7.438e-06 7.438e-06 0.00% Castro::computeNewDt() 9 6.994e-06 6.994e-06 6.994e-06 0.00% Gravity::actual_multilevel_solve() 1 6.928e-06 6.928e-06 6.928e-06 0.00% MLMG::getGradSolution() 11 5.707e-06 5.707e-06 5.707e-06 0.00% AmrLevel::checkPointPost() 3 5.098e-06 5.098e-06 5.098e-06 0.00% Gravity::set_mass_offset() 11 3.905e-06 3.905e-06 3.905e-06 0.00% MLMG::MLRhsNormInf() 11 3.703e-06 3.703e-06 3.703e-06 0.00% Castro::retry_advance_ctu() 10 3.585e-06 3.585e-06 3.585e-06 0.00% MLMG::MLResNormInf() 11 3.396e-06 3.396e-06 3.396e-06 0.00% Castro::computeInitialDt() 2 3.326e-06 3.326e-06 3.326e-06 0.00% Castro::FluxRegCrseInit 10 2.928e-06 2.928e-06 2.928e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.857e-06 2.857e-06 2.857e-06 0.00% Amr::init() 1 2.459e-06 2.459e-06 2.459e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.217e-06 2.217e-06 2.217e-06 0.00% Castro::FluxRegFineAdd() 10 2.152e-06 2.152e-06 2.152e-06 0.00% AmrLevel::checkPointPre() 3 1.453e-06 1.453e-06 1.453e-06 0.00% Castro::post_regrid() 1 1.217e-06 1.217e-06 1.217e-06 0.00% Amr::initialInit() 1 1.008e-06 1.008e-06 1.008e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8609 0.8609 0.8609 100.00% Amr::coarseTimeStep() 10 0.6883 0.6883 0.6883 79.96% Amr::timeStep() 10 0.5695 0.5695 0.5695 66.15% Castro::advance() 10 0.5625 0.5625 0.5625 65.34% Castro::subcycle_advance_ctu() 10 0.5497 0.5497 0.5497 63.85% Castro::do_advance_ctu() 10 0.5495 0.5495 0.5495 63.83% Gravity::solve_phi_with_mlmg() 11 0.2798 0.2798 0.2798 32.50% Gravity::actual_solve_with_mlmg() 11 0.2729 0.2729 0.2729 31.70% Castro::construct_new_gravity() 10 0.2582 0.2582 0.2582 30.00% MLMG::solve() 11 0.2527 0.2527 0.2527 29.35% Gravity::solve_for_phi() 10 0.2427 0.2427 0.2427 28.19% MLMG::oneIter() 81 0.2385 0.2385 0.2385 27.70% VisMF::Write(FabArray) 11 0.2366 0.2366 0.2366 27.49% MLMG::mgVcycle() 81 0.2349 0.2349 0.2349 27.29% Castro::construct_ctu_hydro_source() 10 0.214 0.214 0.214 24.86% Amr::checkPoint() 3 0.1752 0.1752 0.1752 20.35% AmrLevel::checkPoint() 3 0.1716 0.1716 0.1716 19.93% StateData::checkPoint() 12 0.1715 0.1715 0.1715 19.92% Amr::init() 1 0.138 0.138 0.138 16.03% MLCellLinOp::smooth() 1620 0.1153 0.1153 0.1153 13.39% MLCellLinOp::applyBC() 4379 0.1071 0.1071 0.1071 12.44% MLMG::mgVcycle_bottom 81 0.07316 0.07316 0.07316 8.50% MLMG::actualBottomSolve() 81 0.07313 0.07313 0.07313 8.49% MLCGSolver::bicgstab 81 0.07242 0.07242 0.07242 8.41% Amr::writePlotFile() 2 0.06787 0.06787 0.06787 7.88% Castro::clean_state() 62 0.04542 0.04542 0.04542 5.28% Amr::initialInit() 1 0.04471 0.04471 0.04471 5.19% FillPatchIterator::Initialize 41 0.04278 0.04278 0.04278 4.97% FillPatchSingleLevel 41 0.04125 0.04125 0.04125 4.79% Amr::FinalizeInit() 1 0.04067 0.04067 0.04067 4.72% Castro::post_init() 1 0.0394 0.0394 0.0394 4.58% Gravity::multilevel_solve_for_new_phi() 1 0.03756 0.03756 0.03756 4.36% Gravity::actual_multilevel_solve() 1 0.03755 0.03755 0.03755 4.36% StateDataPhysBCFunct::() 41 0.03719 0.03719 0.03719 4.32% MLCellLinOp::apply() 1128 0.03577 0.03577 0.03577 4.16% MLMG::mgVcycle_down::0 81 0.03331 0.03331 0.03331 3.87% MLPoisson::Fsmooth() 3240 0.0324 0.0324 0.0324 3.76% FabArray::FillBoundary() 3974 0.03219 0.03219 0.03219 3.74% FillBoundary_nowait() 3974 0.03141 0.03141 0.03141 3.65% MLMG::mgVcycle_up::0 81 0.0252 0.0252 0.0252 2.93% StateData::FillBoundary(geom) 328 0.02352 0.02352 0.02352 2.73% MLCellLinOp::correctionResidual() 486 0.02193 0.02193 0.02193 2.55% Castro::computeTemp() 63 0.02098 0.02098 0.02098 2.44% amrex::Dot() 1100 0.02042 0.02042 0.02042 2.37% Castro::initialize_do_advance() 10 0.01982 0.01982 0.01982 2.30% MLMG:computeResOfCorrection() 405 0.01931 0.01931 0.01931 2.24% Gravity::get_new_grav_vector() 11 0.0171 0.0171 0.0171 1.99% MLPoisson::define() 11 0.0161 0.0161 0.0161 1.87% MLMG::mgVcycle_down::1 81 0.01561 0.01561 0.01561 1.81% amrex::Copy() 1019 0.01486 0.01486 0.01486 1.73% Castro::construct_old_gravity() 10 0.01459 0.01459 0.01459 1.69% Gravity::get_old_grav_vector() 10 0.01458 0.01458 0.01458 1.69% MLMG::mgVcycle_down::2 81 0.01452 0.01452 0.01452 1.69% FabArray::norminf() 734 0.01433 0.01433 0.01433 1.66% Castro::normalize_species() 62 0.01426 0.01426 0.01426 1.66% MLMG::mgVcycle_down::3 81 0.0142 0.0142 0.0142 1.65% FabArray::ParallelCopy() 851 0.01417 0.01417 0.01417 1.65% MLMG::mgVcycle_down::4 81 0.01399 0.01399 0.01399 1.62% FabArray::ParallelCopy_nowait() 851 0.01394 0.01394 0.01394 1.62% FabArray::setVal() 1135 0.01323 0.01323 0.01323 1.54% Castro::expand_state() 10 0.01254 0.01254 0.01254 1.46% MLCGSolver::ParallelAllReduce 1495 0.01227 0.01227 0.01227 1.43% Castro::initialize_advance() 10 0.0121 0.0121 0.0121 1.41% Castro::do_new_sources() 10 0.01184 0.01184 0.01184 1.38% MLMG::addInterpCorrection() 405 0.01161 0.01161 0.01161 1.35% Castro::do_old_sources() 10 0.01137 0.01137 0.01137 1.32% MLMG::mgVcycle_up::4 81 0.01136 0.01136 0.01136 1.32% MLMG::mgVcycle_up::1 81 0.0113 0.0113 0.0113 1.31% MLMG::mgVcycle_up::2 81 0.01101 0.01101 0.01101 1.28% MLCellLinOp::defineAuxData() 11 0.01099 0.01099 0.01099 1.28% MLMG::mgVcycle_up::3 81 0.01089 0.01089 0.01089 1.26% amrex::average_down 405 0.01088 0.01088 0.01088 1.26% MLPoisson::Fapply() 1128 0.01042 0.01042 0.01042 1.21% Castro::enforce_min_density() 62 0.00944 0.00944 0.00944 1.10% FabArray::Saxpy() 803 0.008136 0.008136 0.008136 0.95% FabArray::Xpay() 811 0.008097 0.008097 0.008097 0.94% MLCellLinOp::solutionResidual() 92 0.007124 0.007124 0.007124 0.83% Castro::post_timestep() 10 0.006805 0.006805 0.006805 0.79% Gravity::fill_multipole_BCs() 11 0.006661 0.006661 0.006661 0.77% Castro::reset_internal_energy(MultiFab) 63 0.00644 0.00644 0.00644 0.75% MLMG::computeResidual() 81 0.006143 0.006143 0.006143 0.71% MLCellLinOp::defineBC() 11 0.004811 0.004811 0.004811 0.56% MLMG::prepareForSolve() 11 0.004619 0.004619 0.004619 0.54% BndryData::define() 11 0.004612 0.004612 0.004612 0.54% FabArray::LinComb() 550 0.004509 0.004509 0.004509 0.52% amrex::Add() 162 0.004286 0.004286 0.004286 0.50% Castro::estTimeStep() 21 0.004264 0.004264 0.004264 0.50% Amr::InitializeInit() 1 0.00404 0.00404 0.00404 0.47% Amr::defBaseLevel() 1 0.00403 0.00403 0.00403 0.47% Castro::initData() 1 0.003562 0.003562 0.003562 0.41% Castro::construct_new_source() 50 0.003288 0.003288 0.003288 0.38% Castro::construct_new_gravity_source() 10 0.003271 0.003271 0.003271 0.38% Castro::construct_old_source() 50 0.002642 0.002642 0.002642 0.31% Castro::construct_old_gravity_source() 10 0.002626 0.002626 0.002626 0.30% Castro::reset_internal_energy(Fab) 504 0.002356 0.002356 0.002356 0.27% MLMG::ResNormInf() 92 0.002078 0.002078 0.002078 0.24% Castro::apply_source_to_state() 20 0.001824 0.001824 0.001824 0.21% Castro::computeNewDt() 9 0.001762 0.001762 0.001762 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001658 0.001658 0.001658 0.19% FabArrayBase::getCPC() 1313 0.001423 0.001423 0.001423 0.17% Castro::enforce_speed_limit() 62 0.001403 0.001403 0.001403 0.16% MLMG::getGradSolution() 11 0.001395 0.001395 0.001395 0.16% MLCellLinOp::compGrad() 11 0.001389 0.001389 0.001389 0.16% MLCellLinOp::setLevelBC() 11 0.00138 0.00138 0.00138 0.16% FabArray::mult() 43 0.001342 0.001342 0.001342 0.16% FabArray::setDomainBndry() 41 0.001322 0.001322 0.001322 0.15% Castro::check_for_nan() 20 0.001205 0.001205 0.001205 0.14% MultiFab::contains_nan() 20 0.001192 0.001192 0.001192 0.14% MLPoisson::prepareForSolve() 11 0.001117 0.001117 0.001117 0.13% MLCellLinOp::prepareForSolve() 11 0.00111 0.00111 0.00111 0.13% Castro::post_regrid() 1 0.0011 0.0011 0.0011 0.13% MLMG::computeMLResidual() 11 0.00102 0.00102 0.00102 0.12% Castro::computeInitialDt() 2 0.0008551 0.0008551 0.0008551 0.10% Gravity::update_max_rhs() 11 0.00082 0.00082 0.00082 0.10% FabArrayBase::CPC::define() 454 0.0006865 0.0006865 0.0006865 0.08% FabArrayBase::getFB() 3974 0.0006477 0.0006477 0.0006477 0.08% Castro::finalize_advance() 10 0.0006326 0.0006326 0.0006326 0.07% Amr::InitAmr() 1 0.0005234 0.0005234 0.0005234 0.06% Gravity::swapTimeLevels() 10 0.0004401 0.0004401 0.0004401 0.05% Castro::Castro() 1 0.0004056 0.0004056 0.0004056 0.05% MLMG::MLResNormInf() 11 0.0002819 0.0002819 0.0002819 0.03% MLLinOp::define() 11 0.0002785 0.0002785 0.0002785 0.03% MLLinOp::defineGrids() 11 0.0002683 0.0002683 0.0002683 0.03% MultiFab::max() 11 0.0002595 0.0002595 0.0002595 0.03% MLMG::MLRhsNormInf() 11 0.0002218 0.0002218 0.0002218 0.03% Castro::buildMetrics() 1 0.0001521 0.0001521 0.0001521 0.02% FabArrayBase::FB::FB() 56 8.222e-05 8.222e-05 8.222e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.168e-05 5.168e-05 5.168e-05 0.01% AmrLevel::AmrLevel(dm) 1 3.98e-05 3.98e-05 3.98e-05 0.00% makeSFC 55 3.868e-05 3.868e-05 3.868e-05 0.00% Castro::enforce_consistent_e() 1 3.644e-05 3.644e-05 3.644e-05 0.00% Castro::finalize_do_advance() 10 3.525e-05 3.525e-05 3.525e-05 0.00% Castro::swap_state_time_levels() 10 3.356e-05 3.356e-05 3.356e-05 0.00% StateData::define() 4 2.565e-05 2.565e-05 2.565e-05 0.00% Amr::writeSmallPlotFile() 1 2.456e-05 2.456e-05 2.456e-05 0.00% Castro::create_source_corrector() 10 2.343e-05 2.343e-05 2.343e-05 0.00% Castro::initMFs() 1 1.826e-05 1.826e-05 1.826e-05 0.00% DistributionMapping::Distribute() 56 1.487e-05 1.487e-05 1.487e-05 0.00% Amr::initSubcycle() 1 8.538e-06 8.538e-06 8.538e-06 0.00% AmrLevel::checkPointPost() 3 5.098e-06 5.098e-06 5.098e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.039e-06 4.039e-06 4.039e-06 0.00% Gravity::set_mass_offset() 11 3.905e-06 3.905e-06 3.905e-06 0.00% Castro::retry_advance_ctu() 10 3.585e-06 3.585e-06 3.585e-06 0.00% Castro::FluxRegCrseInit 10 2.928e-06 2.928e-06 2.928e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.217e-06 2.217e-06 2.217e-06 0.00% Castro::FluxRegFineAdd() 10 2.152e-06 2.152e-06 2.152e-06 0.00% AmrLevel::checkPointPre() 3 1.453e-06 1.453e-06 1.453e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-26-ga232be4e275b) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.01-26-ga232be4e275b) initialized Starting run at 10:11:32 UTC on 2023-01-26. Successfully read inputs file ... Castro git describe: 23.01-20-g08651568d AMReX git describe: 23.01-26-ga232be4e2 Microphysics git describe: 23.01-5-g4a082ffc reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.471479515 Restart time = 0.046699815 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.047627384 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.046043159 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.053611862 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057147324 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062197404 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.049197626 seconds Ending run at 10:11:33 UTC on 2023-01-26. Run time = 0.363500731 Run time without initialization = 0.316211995 Average number of zones advanced per microsecond: 4.145 Average number of zones advanced per microsecond per rank: 4.145 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3635 ... 0.3635 ... 0.3635 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0923 0.0923 0.0923 25.40% VisMF::Read() 3 0.04054 0.04054 0.04054 11.15% MLCellLinOp::applyBC() 1946 0.0325 0.0325 0.0325 8.94% VisMF::Write(FabArray) 1 0.0325 0.0325 0.0325 8.94% Amr::writePlotFile() 1 0.01653 0.01653 0.01653 4.55% MLPoisson::Fsmooth() 1440 0.014 0.014 0.014 3.85% FillBoundary_nowait() 1766 0.01265 0.01265 0.01265 3.48% StateData::FillBoundary(geom) 160 0.01184 0.01184 0.01184 3.26% amrex::Dot() 484 0.008712 0.008712 0.008712 2.40% amrex::Copy() 463 0.006968 0.006968 0.006968 1.92% FabArray::setVal() 537 0.006249 0.006249 0.006249 1.72% FabArray::norminf() 326 0.006214 0.006214 0.006214 1.71% Castro::normalize_species() 30 0.006099 0.006099 0.006099 1.68% FabArray::ParallelCopy_nowait() 380 0.005911 0.005911 0.005911 1.63% Castro::computeTemp() 30 0.005891 0.005891 0.005891 1.62% StateDataPhysBCFunct::() 20 0.005543 0.005543 0.005543 1.52% MLCellLinOp::defineAuxData() 6 0.005299 0.005299 0.005299 1.46% MLPoisson::Fapply() 500 0.004495 0.004495 0.004495 1.24% FabArray::Saxpy() 355 0.003626 0.003626 0.003626 1.00% FabArray::Xpay() 361 0.003526 0.003526 0.003526 0.97% Gravity::fill_multipole_BCs() 6 0.003056 0.003056 0.003056 0.84% Castro::estTimeStep() 10 0.00288 0.00288 0.00288 0.79% MLMG::addInterpCorrection() 180 0.002859 0.002859 0.002859 0.79% amrex::average_down 180 0.002532 0.002532 0.002532 0.70% Castro::enforce_min_density() 30 0.002461 0.002461 0.002461 0.68% Amr::restart() 1 0.002453 0.002453 0.002453 0.67% BndryData::define() 6 0.002068 0.002068 0.002068 0.57% FabArray::LinComb() 242 0.001942 0.001942 0.001942 0.53% Castro::reset_internal_energy(MultiFab) 30 0.001853 0.001853 0.001853 0.51% amrex::Add() 72 0.001836 0.001836 0.001836 0.51% Castro::construct_new_gravity_source() 5 0.001647 0.001647 0.001647 0.45% Castro::construct_old_gravity_source() 5 0.001284 0.001284 0.001284 0.35% Castro::do_advance_ctu() 5 0.001218 0.001218 0.001218 0.34% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009119 0.0009119 0.0009119 0.25% MLCGSolver::bicgstab 36 0.0008768 0.0008768 0.0008768 0.24% MLCellLinOp::setLevelBC() 6 0.0007437 0.0007437 0.0007437 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007256 0.0007256 0.0007256 0.20% Castro::reset_internal_energy(Fab) 240 0.0006768 0.0006768 0.0006768 0.19% Castro::enforce_speed_limit() 30 0.0006645 0.0006645 0.0006645 0.18% FabArray::mult() 22 0.000654 0.000654 0.000654 0.18% FabArray::setDomainBndry() 20 0.0006419 0.0006419 0.0006419 0.18% MLCellLinOp::prepareForSolve() 6 0.0005999 0.0005999 0.0005999 0.17% MultiFab::contains_nan() 10 0.0005883 0.0005883 0.0005883 0.16% MLCellLinOp::smooth() 720 0.0005021 0.0005021 0.0005021 0.14% MLCellLinOp::compGrad() 6 0.0004881 0.0004881 0.0004881 0.13% MLMG::prepareForSolve() 6 0.0004374 0.0004374 0.0004374 0.12% FabArrayBase::CPC::define() 244 0.0004016 0.0004016 0.0004016 0.11% Amr::InitAmr() 1 0.0003935 0.0003935 0.0003935 0.11% FabArrayBase::getCPC() 632 0.0003507 0.0003507 0.0003507 0.10% FabArray::FillBoundary() 1766 0.0003431 0.0003431 0.0003431 0.09% Gravity::get_old_grav_vector() 5 0.0002946 0.0002946 0.0002946 0.08% Gravity::get_new_grav_vector() 5 0.0002796 0.0002796 0.0002796 0.08% main() 1 0.0002768 0.0002768 0.0002768 0.08% FabArrayBase::getFB() 1766 0.0002408 0.0002408 0.0002408 0.07% MLCellLinOp::apply() 500 0.0001993 0.0001993 0.0001993 0.05% MLMG::mgVcycle() 36 0.0001646 0.0001646 0.0001646 0.05% Amr::coarseTimeStep() 5 0.0001601 0.0001601 0.0001601 0.04% MLLinOp::defineGrids() 6 0.0001529 0.0001529 0.0001529 0.04% MultiFab::max() 6 0.0001382 0.0001382 0.0001382 0.04% MLCGSolver::ParallelAllReduce 659 0.0001229 0.0001229 0.0001229 0.03% FabArray::ParallelCopy() 380 0.0001081 0.0001081 0.0001081 0.03% MLCellLinOp::defineBC() 6 0.0001078 0.0001078 0.0001078 0.03% FillPatchIterator::Initialize 20 0.0001009 0.0001009 0.0001009 0.03% MLCellLinOp::correctionResidual() 216 9.716e-05 9.716e-05 9.716e-05 0.03% Amr::timeStep() 5 7.887e-05 7.887e-05 7.887e-05 0.02% Castro::subcycle_advance_ctu() 5 7.758e-05 7.758e-05 7.758e-05 0.02% Gravity::solve_for_phi() 5 6.853e-05 6.853e-05 6.853e-05 0.02% AmrLevel::restart() 1 6.784e-05 6.784e-05 6.784e-05 0.02% StateData::restartDoit() 4 6.059e-05 6.059e-05 6.059e-05 0.02% FabArrayBase::FB::FB() 26 5.68e-05 5.68e-05 5.68e-05 0.02% Gravity::update_max_rhs() 6 5.362e-05 5.362e-05 5.362e-05 0.01% Castro::advance() 5 5.265e-05 5.265e-05 5.265e-05 0.01% MLMG:computeResOfCorrection() 180 5.131e-05 5.131e-05 5.131e-05 0.01% Castro::create_source_corrector() 5 4.285e-05 4.285e-05 4.285e-05 0.01% MLMG::mgVcycle_down::0 36 4.075e-05 4.075e-05 4.075e-05 0.01% MLMG::actualBottomSolve() 36 4.007e-05 4.007e-05 4.007e-05 0.01% MLMG::mgVcycle_down::1 36 3.963e-05 3.963e-05 3.963e-05 0.01% Castro::clean_state() 30 3.888e-05 3.888e-05 3.888e-05 0.01% Castro::expand_state() 5 3.785e-05 3.785e-05 3.785e-05 0.01% MLMG::solve() 6 3.544e-05 3.544e-05 3.544e-05 0.01% MLMG::mgVcycle_down::2 36 3.243e-05 3.243e-05 3.243e-05 0.01% MLMG::mgVcycle_down::4 36 3.232e-05 3.232e-05 3.232e-05 0.01% MLMG::mgVcycle_down::3 36 3.162e-05 3.162e-05 3.162e-05 0.01% Castro::initialize_advance() 5 3.01e-05 3.01e-05 3.01e-05 0.01% MLMG::mgVcycle_up::4 36 3.001e-05 3.001e-05 3.001e-05 0.01% Castro::finalize_advance() 5 2.781e-05 2.781e-05 2.781e-05 0.01% Castro::buildMetrics() 1 2.555e-05 2.555e-05 2.555e-05 0.01% Amr::writeSmallPlotFile() 1 2.451e-05 2.451e-05 2.451e-05 0.01% MLMG::mgVcycle_up::0 36 2.444e-05 2.444e-05 2.444e-05 0.01% MLMG::oneIter() 36 2.374e-05 2.374e-05 2.374e-05 0.01% Castro::initialize_do_advance() 5 2.268e-05 2.268e-05 2.268e-05 0.01% MLMG::mgVcycle_up::3 36 2.213e-05 2.213e-05 2.213e-05 0.01% MLMG::mgVcycle_up::2 36 2.159e-05 2.159e-05 2.159e-05 0.01% MLCellLinOp::solutionResidual() 42 2.132e-05 2.132e-05 2.132e-05 0.01% Castro::post_restart() 1 2.098e-05 2.098e-05 2.098e-05 0.01% MLMG::mgVcycle_up::1 36 2.065e-05 2.065e-05 2.065e-05 0.01% Castro::finalize_do_advance() 5 2.045e-05 2.045e-05 2.045e-05 0.01% Castro::swap_state_time_levels() 5 2.025e-05 2.025e-05 2.025e-05 0.01% Castro::initMFs() 1 2.004e-05 2.004e-05 2.004e-05 0.01% MLMG::ResNormInf() 42 1.682e-05 1.682e-05 1.682e-05 0.00% MLMG::mgVcycle_bottom 36 1.499e-05 1.499e-05 1.499e-05 0.00% FillPatchSingleLevel 20 1.45e-05 1.45e-05 1.45e-05 0.00% MLMG::computeResidual() 36 1.388e-05 1.388e-05 1.388e-05 0.00% MLPoisson::define() 6 1.37e-05 1.37e-05 1.37e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.361e-05 1.361e-05 1.361e-05 0.00% makeSFC 30 1.353e-05 1.353e-05 1.353e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.318e-05 1.318e-05 1.318e-05 0.00% Castro::construct_new_gravity() 5 1.187e-05 1.187e-05 1.187e-05 0.00% Castro::construct_old_source() 25 9.544e-06 9.544e-06 9.544e-06 0.00% Castro::construct_new_source() 25 8.947e-06 8.947e-06 8.947e-06 0.00% DistributionMapping::Distribute() 31 8.599e-06 8.599e-06 8.599e-06 0.00% Amr::initSubcycle() 1 8.487e-06 8.487e-06 8.487e-06 0.00% Castro::do_new_sources() 5 8.383e-06 8.383e-06 8.383e-06 0.00% Castro::do_old_sources() 5 8.361e-06 8.361e-06 8.361e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.266e-06 7.266e-06 7.266e-06 0.00% Castro::check_for_nan() 10 7.119e-06 7.119e-06 7.119e-06 0.00% Castro::construct_old_gravity() 5 6.905e-06 6.905e-06 6.905e-06 0.00% Gravity::actual_multilevel_solve() 1 6.575e-06 6.575e-06 6.575e-06 0.00% MLLinOp::define() 6 5.454e-06 5.454e-06 5.454e-06 0.00% Castro::post_timestep() 5 5.253e-06 5.253e-06 5.253e-06 0.00% Castro::apply_source_to_state() 10 5.001e-06 5.001e-06 5.001e-06 0.00% MLPoisson::prepareForSolve() 6 4.385e-06 4.385e-06 4.385e-06 0.00% Gravity::swapTimeLevels() 5 3.985e-06 3.985e-06 3.985e-06 0.00% Castro::computeNewDt() 5 3.932e-06 3.932e-06 3.932e-06 0.00% MLMG::computeMLResidual() 6 3.725e-06 3.725e-06 3.725e-06 0.00% MLMG::getGradSolution() 6 2.856e-06 2.856e-06 2.856e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.855e-06 2.855e-06 2.855e-06 0.00% MLMG::MLResNormInf() 6 2.2e-06 2.2e-06 2.2e-06 0.00% MLMG::MLRhsNormInf() 6 2.133e-06 2.133e-06 2.133e-06 0.00% Castro::retry_advance_ctu() 5 2.03e-06 2.03e-06 2.03e-06 0.00% Gravity::set_mass_offset() 6 1.871e-06 1.871e-06 1.871e-06 0.00% Castro::FluxRegCrseInit 5 1.465e-06 1.465e-06 1.465e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.178e-06 1.178e-06 1.178e-06 0.00% Amr::init() 1 1.089e-06 1.089e-06 1.089e-06 0.00% Castro::FluxRegFineAdd() 5 1.067e-06 1.067e-06 1.067e-06 0.00% AmrLevel::AmrLevel() 1 7.75e-07 7.75e-07 7.75e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3635 0.3635 0.3635 100.00% Amr::coarseTimeStep() 5 0.2668 0.2668 0.2668 73.39% Amr::timeStep() 5 0.2644 0.2644 0.2644 72.74% Castro::advance() 5 0.2619 0.2619 0.2619 72.04% Castro::subcycle_advance_ctu() 5 0.2563 0.2563 0.2563 70.50% Castro::do_advance_ctu() 5 0.2562 0.2562 0.2562 70.48% Castro::construct_new_gravity() 5 0.1283 0.1283 0.1283 35.29% Gravity::solve_phi_with_mlmg() 6 0.1236 0.1236 0.1236 34.00% Gravity::solve_for_phi() 5 0.1206 0.1206 0.1206 33.17% Gravity::actual_solve_with_mlmg() 6 0.1203 0.1203 0.1203 33.10% MLMG::solve() 6 0.1091 0.1091 0.1091 30.02% MLMG::oneIter() 36 0.1023 0.1023 0.1023 28.15% MLMG::mgVcycle() 36 0.1008 0.1008 0.1008 27.73% Castro::construct_ctu_hydro_source() 5 0.09232 0.09232 0.09232 25.40% Amr::writePlotFile() 1 0.04929 0.04929 0.04929 13.56% MLCellLinOp::smooth() 720 0.04924 0.04924 0.04924 13.54% Amr::init() 1 0.04674 0.04674 0.04674 12.86% Amr::restart() 1 0.04674 0.04674 0.04674 12.86% MLCellLinOp::applyBC() 1946 0.04579 0.04579 0.04579 12.60% AmrLevel::restart() 1 0.04072 0.04072 0.04072 11.20% StateData::restartDoit() 4 0.04065 0.04065 0.04065 11.18% VisMF::Read() 3 0.04054 0.04054 0.04054 11.15% VisMF::Write(FabArray) 1 0.0325 0.0325 0.0325 8.94% MLMG::mgVcycle_bottom 36 0.03147 0.03147 0.03147 8.66% MLMG::actualBottomSolve() 36 0.03146 0.03146 0.03146 8.65% MLCGSolver::bicgstab 36 0.03115 0.03115 0.03115 8.57% FillPatchIterator::Initialize 20 0.02011 0.02011 0.02011 5.53% FillPatchSingleLevel 20 0.01937 0.01937 0.01937 5.33% Castro::clean_state() 30 0.01768 0.01768 0.01768 4.86% StateDataPhysBCFunct::() 20 0.01738 0.01738 0.01738 4.78% MLCellLinOp::apply() 500 0.01549 0.01549 0.01549 4.26% MLMG::mgVcycle_down::0 36 0.01409 0.01409 0.01409 3.88% MLPoisson::Fsmooth() 1440 0.014 0.014 0.014 3.85% FabArray::FillBoundary() 1766 0.01329 0.01329 0.01329 3.65% FillBoundary_nowait() 1766 0.01294 0.01294 0.01294 3.56% StateData::FillBoundary(geom) 160 0.01184 0.01184 0.01184 3.26% MLMG::mgVcycle_up::0 36 0.01057 0.01057 0.01057 2.91% MLCellLinOp::correctionResidual() 216 0.00942 0.00942 0.00942 2.59% Castro::initialize_do_advance() 5 0.009014 0.009014 0.009014 2.48% MLPoisson::define() 6 0.008967 0.008967 0.008967 2.47% amrex::Dot() 484 0.008712 0.008712 0.008712 2.40% Castro::computeTemp() 30 0.008421 0.008421 0.008421 2.32% MLMG:computeResOfCorrection() 180 0.008272 0.008272 0.008272 2.28% Gravity::get_new_grav_vector() 5 0.007608 0.007608 0.007608 2.09% Castro::construct_old_gravity() 5 0.007274 0.007274 0.007274 2.00% Gravity::get_old_grav_vector() 5 0.007267 0.007267 0.007267 2.00% amrex::Copy() 463 0.006968 0.006968 0.006968 1.92% Castro::do_new_sources() 5 0.006763 0.006763 0.006763 1.86% MLMG::mgVcycle_down::1 36 0.006759 0.006759 0.006759 1.86% FabArray::ParallelCopy() 380 0.006393 0.006393 0.006393 1.76% FabArray::ParallelCopy_nowait() 380 0.006285 0.006285 0.006285 1.73% MLMG::mgVcycle_down::2 36 0.006259 0.006259 0.006259 1.72% FabArray::setVal() 537 0.006249 0.006249 0.006249 1.72% FabArray::norminf() 326 0.006214 0.006214 0.006214 1.71% MLMG::mgVcycle_down::3 36 0.00614 0.00614 0.00614 1.69% Castro::normalize_species() 30 0.006099 0.006099 0.006099 1.68% MLMG::mgVcycle_down::4 36 0.006067 0.006067 0.006067 1.67% MLCellLinOp::defineAuxData() 6 0.006041 0.006041 0.006041 1.66% Castro::expand_state() 5 0.005702 0.005702 0.005702 1.57% MLCGSolver::ParallelAllReduce 659 0.005253 0.005253 0.005253 1.45% Castro::initialize_advance() 5 0.00524 0.00524 0.00524 1.44% MLMG::addInterpCorrection() 180 0.005077 0.005077 0.005077 1.40% MLMG::mgVcycle_up::4 36 0.004936 0.004936 0.004936 1.36% MLMG::mgVcycle_up::1 36 0.004882 0.004882 0.004882 1.34% MLMG::mgVcycle_up::2 36 0.004786 0.004786 0.004786 1.32% amrex::average_down 180 0.004733 0.004733 0.004733 1.30% MLMG::mgVcycle_up::3 36 0.004679 0.004679 0.004679 1.29% MLPoisson::Fapply() 500 0.004495 0.004495 0.004495 1.24% Castro::do_old_sources() 5 0.00406 0.00406 0.00406 1.12% FabArray::Saxpy() 355 0.003626 0.003626 0.003626 1.00% FabArray::Xpay() 361 0.003526 0.003526 0.003526 0.97% Castro::post_restart() 1 0.003399 0.003399 0.003399 0.94% Gravity::multilevel_solve_for_new_phi() 1 0.003286 0.003286 0.003286 0.90% Gravity::actual_multilevel_solve() 1 0.003273 0.003273 0.003273 0.90% MLCellLinOp::solutionResidual() 42 0.00321 0.00321 0.00321 0.88% Gravity::fill_multipole_BCs() 6 0.00318 0.00318 0.00318 0.87% Castro::estTimeStep() 10 0.00288 0.00288 0.00288 0.79% MLCellLinOp::defineBC() 6 0.002725 0.002725 0.002725 0.75% MLMG::computeResidual() 36 0.002671 0.002671 0.002671 0.73% BndryData::define() 6 0.002617 0.002617 0.002617 0.72% Castro::reset_internal_energy(MultiFab) 30 0.00253 0.00253 0.00253 0.70% MLMG::prepareForSolve() 6 0.002479 0.002479 0.002479 0.68% Castro::post_timestep() 5 0.002476 0.002476 0.002476 0.68% Castro::enforce_min_density() 30 0.002461 0.002461 0.002461 0.68% Castro::computeNewDt() 5 0.00219 0.00219 0.00219 0.60% FabArray::LinComb() 242 0.001942 0.001942 0.001942 0.53% amrex::Add() 72 0.001836 0.001836 0.001836 0.51% Castro::construct_new_source() 25 0.001655 0.001655 0.001655 0.46% Castro::construct_new_gravity_source() 5 0.001647 0.001647 0.001647 0.45% Castro::construct_old_source() 25 0.001294 0.001294 0.001294 0.36% Castro::construct_old_gravity_source() 5 0.001284 0.001284 0.001284 0.35% MLMG::ResNormInf() 42 0.0009303 0.0009303 0.0009303 0.26% Castro::apply_source_to_state() 10 0.0009178 0.0009178 0.0009178 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009119 0.0009119 0.0009119 0.25% FabArrayBase::getCPC() 632 0.0007523 0.0007523 0.0007523 0.21% MLMG::getGradSolution() 6 0.0007488 0.0007488 0.0007488 0.21% MLCellLinOp::compGrad() 6 0.000746 0.000746 0.000746 0.21% MLCellLinOp::setLevelBC() 6 0.0007437 0.0007437 0.0007437 0.20% Castro::reset_internal_energy(Fab) 240 0.0006768 0.0006768 0.0006768 0.19% Castro::enforce_speed_limit() 30 0.0006645 0.0006645 0.0006645 0.18% FabArray::mult() 22 0.000654 0.000654 0.000654 0.18% FabArray::setDomainBndry() 20 0.0006419 0.0006419 0.0006419 0.18% MLPoisson::prepareForSolve() 6 0.0006043 0.0006043 0.0006043 0.17% MLCellLinOp::prepareForSolve() 6 0.0005999 0.0005999 0.0005999 0.17% Castro::check_for_nan() 10 0.0005955 0.0005955 0.0005955 0.16% MultiFab::contains_nan() 10 0.0005883 0.0005883 0.0005883 0.16% MLMG::computeMLResidual() 6 0.0005566 0.0005566 0.0005566 0.15% Gravity::update_max_rhs() 6 0.0004474 0.0004474 0.0004474 0.12% Amr::InitAmr() 1 0.000402 0.000402 0.000402 0.11% FabArrayBase::CPC::define() 244 0.0004016 0.0004016 0.0004016 0.11% FabArrayBase::getFB() 1766 0.0002976 0.0002976 0.0002976 0.08% Castro::finalize_advance() 5 0.0002869 0.0002869 0.0002869 0.08% Gravity::swapTimeLevels() 5 0.0002239 0.0002239 0.0002239 0.06% MLLinOp::define() 6 0.0001874 0.0001874 0.0001874 0.05% MLLinOp::defineGrids() 6 0.000182 0.000182 0.000182 0.05% Castro::buildMetrics() 1 0.0001504 0.0001504 0.0001504 0.04% MLMG::MLResNormInf() 6 0.0001502 0.0001502 0.0001502 0.04% MultiFab::max() 6 0.0001382 0.0001382 0.0001382 0.04% MLMG::MLRhsNormInf() 6 0.0001167 0.0001167 0.0001167 0.03% FabArrayBase::FB::FB() 26 5.68e-05 5.68e-05 5.68e-05 0.02% Castro::create_source_corrector() 5 4.285e-05 4.285e-05 4.285e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.793e-05 2.793e-05 2.793e-05 0.01% Amr::writeSmallPlotFile() 1 2.451e-05 2.451e-05 2.451e-05 0.01% makeSFC 30 2.066e-05 2.066e-05 2.066e-05 0.01% Castro::finalize_do_advance() 5 2.045e-05 2.045e-05 2.045e-05 0.01% Castro::swap_state_time_levels() 5 2.025e-05 2.025e-05 2.025e-05 0.01% Castro::initMFs() 1 2.004e-05 2.004e-05 2.004e-05 0.01% DistributionMapping::Distribute() 31 8.599e-06 8.599e-06 8.599e-06 0.00% Amr::initSubcycle() 1 8.487e-06 8.487e-06 8.487e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.32e-06 4.32e-06 4.32e-06 0.00% Castro::retry_advance_ctu() 5 2.03e-06 2.03e-06 2.03e-06 0.00% Gravity::set_mass_offset() 6 1.871e-06 1.871e-06 1.871e-06 0.00% Castro::FluxRegCrseInit 5 1.465e-06 1.465e-06 1.465e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.178e-06 1.178e-06 1.178e-06 0.00% Castro::FluxRegFineAdd() 5 1.067e-06 1.067e-06 1.067e-06 0.00% AmrLevel::AmrLevel() 1 7.75e-07 7.75e-07 7.75e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-26-ga232be4e275b) finalized