Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.06-8-g8a1c76c0af89) initialized Starting run at 08:29:41 UTC on 2022-06-07. Successfully read inputs file ... Castro git describe: 22.06-4-gfbc287aee AMReX git describe: 22.06-8-g8a1c76c0a Microphysics git describe: 22.06-1-gb8e427c1 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.043934968 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025464865 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.05012568 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.052319143 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.06611097 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.077767745 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.056421496 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.041111401 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.058714047 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.05325211 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.06312415 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.068631635 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.07073332 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.041077323 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.025237776 seconds Ending run at 08:29:42 UTC on 2022-06-07. Run time = 0.847455465 Run time without initialization = 0.725222309 Average number of zones advanced per microsecond: 3.615 Average number of zones advanced per microsecond per rank: 3.615 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8475 ... 0.8475 ... 0.8475 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2125 0.2125 0.2125 25.07% VisMF::Write(FabArray) 11 0.1705 0.1705 0.1705 20.12% MLCellLinOp::applyBC() 4433 0.08093 0.08093 0.08093 9.55% MLPoisson::Fsmooth() 3280 0.06411 0.06411 0.06411 7.56% MLCGSolver::bicgstab 82 0.02416 0.02416 0.02416 2.85% StateData::FillBoundary(geom) 328 0.02397 0.02397 0.02397 2.83% MultiFab::Dot() 1114 0.02254 0.02254 0.02254 2.66% StateDataPhysBCFunct::() 41 0.02235 0.02235 0.02235 2.64% Castro::normalize_species() 62 0.01724 0.01724 0.01724 2.03% MultiFab::LinComb() 1586 0.01449 0.01449 0.01449 1.71% Castro::computeTemp() 63 0.01447 0.01447 0.01447 1.71% FabArray::setVal() 1144 0.01431 0.01431 0.01431 1.69% FillBoundary_nowait() 4023 0.01425 0.01425 0.01425 1.68% Castro::enforce_min_density() 62 0.01389 0.01389 0.01389 1.64% FabArray::ParallelCopy_nowait() 861 0.01311 0.01311 0.01311 1.55% MLPoisson::Fapply() 1142 0.01178 0.01178 0.01178 1.39% MLCellLinOp::defineAuxData() 11 0.01148 0.01148 0.01148 1.35% Gravity::fill_multipole_BCs() 11 0.009732 0.009732 0.009732 1.15% MLMG::addInterpCorrection() 410 0.007391 0.007391 0.007391 0.87% amrex::average_down 410 0.006858 0.006858 0.006858 0.81% MultiFab::Xpay() 585 0.006599 0.006599 0.006599 0.78% Castro::estTimeStep() 21 0.006488 0.006488 0.006488 0.77% Castro::do_advance_ctu() 10 0.005434 0.005434 0.005434 0.64% Castro::reset_internal_energy(MultiFab) 63 0.004132 0.004132 0.004132 0.49% Amr::checkPoint() 3 0.004033 0.004033 0.004033 0.48% BndryData::define() 11 0.003873 0.003873 0.003873 0.46% Castro::construct_new_gravity_source() 10 0.00309 0.00309 0.00309 0.36% Castro::construct_old_gravity_source() 10 0.00287 0.00287 0.00287 0.34% Amr::writePlotFile() 2 0.002351 0.002351 0.002351 0.28% Gravity::get_new_grav_vector() 11 0.001938 0.001938 0.001938 0.23% MLMG::ResNormInf() 93 0.001936 0.001936 0.001936 0.23% MultiFab::Saxpy() 20 0.001812 0.001812 0.001812 0.21% Gravity::get_old_grav_vector() 10 0.001733 0.001733 0.001733 0.20% Castro::expand_state() 10 0.00173 0.00173 0.00173 0.20% MLMG::oneIter() 82 0.001712 0.001712 0.001712 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00163 0.00163 0.00163 0.19% MLCellLinOp::setLevelBC() 11 0.001547 0.001547 0.001547 0.18% Castro::reset_internal_energy(Fab) 504 0.001547 0.001547 0.001547 0.18% Gravity::actual_solve_with_mlmg() 11 0.001336 0.001336 0.001336 0.16% FabArray::setDomainBndry() 41 0.001327 0.001327 0.001327 0.16% FabArray::mult() 43 0.001324 0.001324 0.001324 0.16% MultiFab::contains_nan() 20 0.001183 0.001183 0.001183 0.14% MLCellLinOp::smooth() 1640 0.001174 0.001174 0.001174 0.14% MLCellLinOp::prepareForSolve() 11 0.001164 0.001164 0.001164 0.14% Castro::initData() 1 0.001155 0.001155 0.001155 0.14% MLMG::prepareForSolve() 11 0.001038 0.001038 0.001038 0.12% Castro::enforce_speed_limit() 62 0.0009937 0.0009937 0.0009937 0.12% MLCellLinOp::compGrad() 11 0.0009333 0.0009333 0.0009333 0.11% FabArray::FillBoundary() 4023 0.0008272 0.0008272 0.0008272 0.10% FabArrayBase::getCPC() 1323 0.0007703 0.0007703 0.0007703 0.09% FabArrayBase::CPC::define() 454 0.000666 0.000666 0.000666 0.08% FabArrayBase::getFB() 4023 0.0006057 0.0006057 0.0006057 0.07% Amr::InitAmr() 1 0.0004849 0.0004849 0.0004849 0.06% MLCellLinOp::apply() 1142 0.0004772 0.0004772 0.0004772 0.06% Gravity::solve_for_phi() 10 0.0004308 0.0004308 0.0004308 0.05% Gravity::update_max_rhs() 11 0.0004049 0.0004049 0.0004049 0.05% CGSolver::sxay() 1586 0.0003688 0.0003688 0.0003688 0.04% Amr::coarseTimeStep() 10 0.0003118 0.0003118 0.0003118 0.04% FillPatchIterator::Initialize 41 0.0003013 0.0003013 0.0003013 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002932 0.0002932 0.0002932 0.03% FabArray::ParallelCopy() 861 0.0002872 0.0002872 0.0002872 0.03% MLCellLinOp::defineBC() 11 0.0002798 0.0002798 0.0002798 0.03% main() 1 0.0002627 0.0002627 0.0002627 0.03% MultiFab::max() 11 0.0002561 0.0002561 0.0002561 0.03% MultiFab::Copy() 11 0.0002558 0.0002558 0.0002558 0.03% MLCellLinOp::correctionResidual() 492 0.0002185 0.0002185 0.0002185 0.03% MLMG::mgVcycle() 82 0.0002146 0.0002146 0.0002146 0.03% Castro::construct_new_gravity() 10 0.0002037 0.0002037 0.0002037 0.02% MLMG::MLRhsNormInf() 11 0.0002033 0.0002033 0.0002033 0.02% Castro::create_source_corrector() 10 0.0001916 0.0001916 0.0001916 0.02% Amr::timeStep() 10 0.000191 0.000191 0.000191 0.02% MLLinOp::defineGrids() 11 0.0001701 0.0001701 0.0001701 0.02% StateData::checkPoint() 12 0.0001472 0.0001472 0.0001472 0.02% Castro::subcycle_advance_ctu() 10 0.0001401 0.0001401 0.0001401 0.02% MLMG:computeResOfCorrection() 410 0.000128 0.000128 0.000128 0.02% MLMG::actualBottomSolve() 82 0.0001062 0.0001062 0.0001062 0.01% MLMG::mgVcycle_down::0 82 8.914e-05 8.914e-05 8.914e-05 0.01% Castro::Castro() 1 8.589e-05 8.589e-05 8.589e-05 0.01% MLMG::mgVcycle_down::3 82 8.498e-05 8.498e-05 8.498e-05 0.01% FabArrayBase::FB::FB() 56 8.472e-05 8.472e-05 8.472e-05 0.01% Castro::initialize_advance() 10 8.32e-05 8.32e-05 8.32e-05 0.01% MLMG::mgVcycle_down::1 82 8.206e-05 8.206e-05 8.206e-05 0.01% MLMG::mgVcycle_down::2 82 8.203e-05 8.203e-05 8.203e-05 0.01% Castro::clean_state() 62 7.403e-05 7.403e-05 7.403e-05 0.01% MLMG::mgVcycle_down::4 82 7.356e-05 7.356e-05 7.356e-05 0.01% AmrLevel::checkPoint() 3 7.229e-05 7.229e-05 7.229e-05 0.01% MLMG::solve() 11 7.186e-05 7.186e-05 7.186e-05 0.01% Castro::initialize_do_advance() 10 6.425e-05 6.425e-05 6.425e-05 0.01% MLMG::mgVcycle_up::4 82 6.122e-05 6.122e-05 6.122e-05 0.01% Castro::advance() 10 6.021e-05 6.021e-05 6.021e-05 0.01% Castro::finalize_advance() 10 5.462e-05 5.462e-05 5.462e-05 0.01% MLMG::mgVcycle_up::1 82 5.19e-05 5.19e-05 5.19e-05 0.01% MLMG::mgVcycle_up::0 82 5.174e-05 5.174e-05 5.174e-05 0.01% MLMG::mgVcycle_up::2 82 5.12e-05 5.12e-05 5.12e-05 0.01% MLMG::mgVcycle_up::3 82 5.054e-05 5.054e-05 5.054e-05 0.01% MLCellLinOp::solutionResidual() 93 4.982e-05 4.982e-05 4.982e-05 0.01% StateData::define() 4 4.863e-05 4.863e-05 4.863e-05 0.01% Castro::construct_new_source() 50 4.174e-05 4.174e-05 4.174e-05 0.00% FillPatchSingleLevel 41 4.103e-05 4.103e-05 4.103e-05 0.00% Castro::swap_state_time_levels() 10 4.063e-05 4.063e-05 4.063e-05 0.00% Castro::finalize_do_advance() 10 3.631e-05 3.631e-05 3.631e-05 0.00% MLMG::computeResidual() 82 3.445e-05 3.445e-05 3.445e-05 0.00% Castro::enforce_consistent_e() 1 3.44e-05 3.44e-05 3.44e-05 0.00% MLMG::mgVcycle_bottom 82 3.274e-05 3.274e-05 3.274e-05 0.00% Gravity::actual_multilevel_solve() 1 3.137e-05 3.137e-05 3.137e-05 0.00% Castro::initMFs() 1 2.848e-05 2.848e-05 2.848e-05 0.00% makeSFC 55 2.619e-05 2.619e-05 2.619e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.523e-05 2.523e-05 2.523e-05 0.00% MLLinOp::define() 11 2.449e-05 2.449e-05 2.449e-05 0.00% Amr::writeSmallPlotFile() 1 2.412e-05 2.412e-05 2.412e-05 0.00% Castro::buildMetrics() 1 2.371e-05 2.371e-05 2.371e-05 0.00% Amr::FinalizeInit() 1 2.275e-05 2.275e-05 2.275e-05 0.00% MLPoisson::define() 11 2.254e-05 2.254e-05 2.254e-05 0.00% Amr::defBaseLevel() 1 2.084e-05 2.084e-05 2.084e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.871e-05 1.871e-05 1.871e-05 0.00% Castro::do_new_sources() 10 1.705e-05 1.705e-05 1.705e-05 0.00% Castro::construct_old_source() 50 1.696e-05 1.696e-05 1.696e-05 0.00% Castro::do_old_sources() 10 1.52e-05 1.52e-05 1.52e-05 0.00% DistributionMapping::Distribute() 56 1.514e-05 1.514e-05 1.514e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.275e-05 1.275e-05 1.275e-05 0.00% Castro::check_for_nan() 20 1.273e-05 1.273e-05 1.273e-05 0.00% Castro::construct_old_gravity() 10 1.107e-05 1.107e-05 1.107e-05 0.00% Castro::apply_source_to_state() 20 1.098e-05 1.098e-05 1.098e-05 0.00% Gravity::swapTimeLevels() 10 8.773e-06 8.773e-06 8.773e-06 0.00% Amr::initSubcycle() 1 8.685e-06 8.685e-06 8.685e-06 0.00% Castro::post_init() 1 8.331e-06 8.331e-06 8.331e-06 0.00% MLPoisson::prepareForSolve() 11 8.244e-06 8.244e-06 8.244e-06 0.00% Castro::post_timestep() 10 7.538e-06 7.538e-06 7.538e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.158e-06 7.158e-06 7.158e-06 0.00% MLMG::computeMLResidual() 11 7.102e-06 7.102e-06 7.102e-06 0.00% Castro::computeNewDt() 9 6.36e-06 6.36e-06 6.36e-06 0.00% MLMG::getGradSolution() 11 6.278e-06 6.278e-06 6.278e-06 0.00% MLMG::buildFineMask() 11 5.801e-06 5.801e-06 5.801e-06 0.00% Amr::InitializeInit() 1 5.616e-06 5.616e-06 5.616e-06 0.00% AmrLevel::checkPointPost() 3 4.85e-06 4.85e-06 4.85e-06 0.00% MLMG::MLResNormInf() 11 4.559e-06 4.559e-06 4.559e-06 0.00% Gravity::set_mass_offset() 11 4.27e-06 4.27e-06 4.27e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.077e-06 4.077e-06 4.077e-06 0.00% Castro::retry_advance_ctu() 10 3.789e-06 3.789e-06 3.789e-06 0.00% Castro::FluxRegCrseInit 10 2.843e-06 2.843e-06 2.843e-06 0.00% Amr::init() 1 2.816e-06 2.816e-06 2.816e-06 0.00% Castro::FluxRegFineAdd() 10 2.33e-06 2.33e-06 2.33e-06 0.00% Castro::computeInitialDt() 2 2.011e-06 2.011e-06 2.011e-06 0.00% AmrLevel::checkPointPre() 3 1.858e-06 1.858e-06 1.858e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.838e-06 1.838e-06 1.838e-06 0.00% Amr::initialInit() 1 1.548e-06 1.548e-06 1.548e-06 0.00% Castro::post_regrid() 1 1.236e-06 1.236e-06 1.236e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8475 0.8475 0.8475 100.00% Amr::coarseTimeStep() 10 0.6998 0.6998 0.6998 82.57% Amr::timeStep() 10 0.6137 0.6137 0.6137 72.42% Castro::advance() 10 0.6062 0.6062 0.6062 71.53% Castro::subcycle_advance_ctu() 10 0.595 0.595 0.595 70.21% Castro::do_advance_ctu() 10 0.5948 0.5948 0.5948 70.19% Gravity::solve_phi_with_mlmg() 11 0.3165 0.3165 0.3165 37.35% Gravity::actual_solve_with_mlmg() 11 0.3066 0.3066 0.3066 36.18% Castro::construct_new_gravity() 10 0.2873 0.2873 0.2873 33.90% MLMG::solve() 11 0.2841 0.2841 0.2841 33.53% Gravity::solve_for_phi() 10 0.2721 0.2721 0.2721 32.10% MLMG::oneIter() 82 0.2696 0.2696 0.2696 31.81% MLMG::mgVcycle() 82 0.2678 0.2678 0.2678 31.60% Castro::construct_ctu_hydro_source() 10 0.2125 0.2125 0.2125 25.07% VisMF::Write(FabArray) 11 0.1705 0.1705 0.1705 20.12% MLCellLinOp::smooth() 1640 0.1373 0.1373 0.1373 16.20% Amr::checkPoint() 3 0.1263 0.1263 0.1263 14.90% AmrLevel::checkPoint() 3 0.1222 0.1222 0.1222 14.42% StateData::checkPoint() 12 0.1221 0.1221 0.1221 14.41% Amr::init() 1 0.1216 0.1216 0.1216 14.35% MLCellLinOp::applyBC() 4433 0.0967 0.0967 0.0967 11.41% MLMG::mgVcycle_bottom 82 0.08257 0.08257 0.08257 9.74% MLMG::actualBottomSolve() 82 0.08254 0.08254 0.08254 9.74% MLCGSolver::bicgstab 82 0.0817 0.0817 0.0817 9.64% MLPoisson::Fsmooth() 3280 0.06411 0.06411 0.06411 7.56% Amr::initialInit() 1 0.05209 0.05209 0.05209 6.15% FillPatchIterator::Initialize 41 0.05194 0.05194 0.05194 6.13% Castro::clean_state() 62 0.0515 0.0515 0.0515 6.08% Amr::writePlotFile() 2 0.05084 0.05084 0.05084 6.00% FillPatchSingleLevel 41 0.05032 0.05032 0.05032 5.94% Amr::FinalizeInit() 1 0.04811 0.04811 0.04811 5.68% Castro::post_init() 1 0.04682 0.04682 0.04682 5.52% StateDataPhysBCFunct::() 41 0.04632 0.04632 0.04632 5.47% Gravity::multilevel_solve_for_new_phi() 1 0.04495 0.04495 0.04495 5.30% Gravity::actual_multilevel_solve() 1 0.04493 0.04493 0.04493 5.30% MLCellLinOp::apply() 1142 0.03643 0.03643 0.03643 4.30% MLMG::mgVcycle_down::0 82 0.03584 0.03584 0.03584 4.23% MLMG::mgVcycle_up::0 82 0.03056 0.03056 0.03056 3.61% Castro::initialize_do_advance() 10 0.02576 0.02576 0.02576 3.04% StateData::FillBoundary(geom) 328 0.02397 0.02397 0.02397 2.83% MultiFab::Dot() 1114 0.02254 0.02254 0.02254 2.66% MLCellLinOp::correctionResidual() 492 0.02137 0.02137 0.02137 2.52% Castro::computeTemp() 63 0.02015 0.02015 0.02015 2.38% MLMG:computeResOfCorrection() 410 0.01844 0.01844 0.01844 2.18% MLPoisson::define() 11 0.01817 0.01817 0.01817 2.14% MLMG::mgVcycle_down::1 82 0.01782 0.01782 0.01782 2.10% MLMG::mgVcycle_down::2 82 0.01735 0.01735 0.01735 2.05% Castro::normalize_species() 62 0.01724 0.01724 0.01724 2.03% Gravity::get_new_grav_vector() 11 0.0168 0.0168 0.0168 1.98% MLMG::mgVcycle_down::3 82 0.01644 0.01644 0.01644 1.94% FabArray::FillBoundary() 4023 0.01577 0.01577 0.01577 1.86% MLMG::mgVcycle_down::4 82 0.01572 0.01572 0.01572 1.85% FillBoundary_nowait() 4023 0.01494 0.01494 0.01494 1.76% CGSolver::sxay() 1586 0.01486 0.01486 0.01486 1.75% Castro::expand_state() 10 0.01482 0.01482 0.01482 1.75% Castro::construct_old_gravity() 10 0.01455 0.01455 0.01455 1.72% Gravity::get_old_grav_vector() 10 0.01454 0.01454 0.01454 1.72% MultiFab::LinComb() 1586 0.01449 0.01449 0.01449 1.71% Castro::do_new_sources() 10 0.01433 0.01433 0.01433 1.69% FabArray::setVal() 1144 0.01431 0.01431 0.01431 1.69% FabArray::ParallelCopy() 861 0.0142 0.0142 0.0142 1.68% FabArray::ParallelCopy_nowait() 861 0.01391 0.01391 0.01391 1.64% Castro::enforce_min_density() 62 0.01389 0.01389 0.01389 1.64% MLCGSolver::ParallelAllReduce 1514 0.01344 0.01344 0.01344 1.59% MLMG::mgVcycle_up::2 82 0.01327 0.01327 0.01327 1.57% MLMG::mgVcycle_up::1 82 0.01308 0.01308 0.01308 1.54% MLCellLinOp::defineAuxData() 11 0.01282 0.01282 0.01282 1.51% MLMG::mgVcycle_up::3 82 0.01255 0.01255 0.01255 1.48% MLMG::addInterpCorrection() 410 0.01248 0.01248 0.01248 1.47% MLMG::mgVcycle_up::4 82 0.01244 0.01244 0.01244 1.47% amrex::average_down 410 0.01201 0.01201 0.01201 1.42% MLPoisson::Fapply() 1142 0.01178 0.01178 0.01178 1.39% Castro::do_old_sources() 10 0.01112 0.01112 0.01112 1.31% Castro::initialize_advance() 10 0.01111 0.01111 0.01111 1.31% Gravity::fill_multipole_BCs() 11 0.009732 0.009732 0.009732 1.15% Castro::post_timestep() 10 0.007317 0.007317 0.007317 0.86% MLCellLinOp::solutionResidual() 93 0.007184 0.007184 0.007184 0.85% MultiFab::Xpay() 585 0.006599 0.006599 0.006599 0.78% Castro::estTimeStep() 21 0.006488 0.006488 0.006488 0.77% MLMG::computeResidual() 82 0.006188 0.006188 0.006188 0.73% Castro::reset_internal_energy(MultiFab) 63 0.005678 0.005678 0.005678 0.67% MLMG::prepareForSolve() 11 0.005121 0.005121 0.005121 0.60% MLCellLinOp::defineBC() 11 0.005081 0.005081 0.005081 0.60% BndryData::define() 11 0.004801 0.004801 0.004801 0.57% Amr::InitializeInit() 1 0.003985 0.003985 0.003985 0.47% Amr::defBaseLevel() 1 0.003979 0.003979 0.003979 0.47% Castro::initData() 1 0.003458 0.003458 0.003458 0.41% Castro::construct_new_source() 50 0.003131 0.003131 0.003131 0.37% Castro::construct_new_gravity_source() 10 0.00309 0.00309 0.00309 0.36% Castro::computeNewDt() 9 0.00299 0.00299 0.00299 0.35% Castro::construct_old_source() 50 0.002887 0.002887 0.002887 0.34% Castro::construct_old_gravity_source() 10 0.00287 0.00287 0.00287 0.34% MLMG::ResNormInf() 93 0.001936 0.001936 0.001936 0.23% Castro::apply_source_to_state() 20 0.001823 0.001823 0.001823 0.22% MultiFab::Saxpy() 20 0.001812 0.001812 0.001812 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.00163 0.00163 0.00163 0.19% MLCellLinOp::setLevelBC() 11 0.001547 0.001547 0.001547 0.18% Castro::reset_internal_energy(Fab) 504 0.001547 0.001547 0.001547 0.18% FabArrayBase::getCPC() 1323 0.001436 0.001436 0.001436 0.17% MLMG::getGradSolution() 11 0.001421 0.001421 0.001421 0.17% MLCellLinOp::compGrad() 11 0.001415 0.001415 0.001415 0.17% FabArray::setDomainBndry() 41 0.001327 0.001327 0.001327 0.16% FabArray::mult() 43 0.001324 0.001324 0.001324 0.16% Castro::check_for_nan() 20 0.001196 0.001196 0.001196 0.14% MultiFab::contains_nan() 20 0.001183 0.001183 0.001183 0.14% MLPoisson::prepareForSolve() 11 0.001173 0.001173 0.001173 0.14% MLCellLinOp::prepareForSolve() 11 0.001164 0.001164 0.001164 0.14% Castro::post_regrid() 1 0.001082 0.001082 0.001082 0.13% MLMG::computeMLResidual() 11 0.001037 0.001037 0.001037 0.12% Castro::enforce_speed_limit() 62 0.0009937 0.0009937 0.0009937 0.12% Gravity::update_max_rhs() 11 0.0008101 0.0008101 0.0008101 0.10% FabArrayBase::getFB() 4023 0.0006904 0.0006904 0.0006904 0.08% FabArrayBase::CPC::define() 454 0.000666 0.000666 0.000666 0.08% Castro::computeInitialDt() 2 0.000658 0.000658 0.000658 0.08% Amr::InitAmr() 1 0.0004936 0.0004936 0.0004936 0.06% Gravity::swapTimeLevels() 10 0.000443 0.000443 0.000443 0.05% Castro::Castro() 1 0.0004396 0.0004396 0.0004396 0.05% MultiFab::max() 11 0.0002561 0.0002561 0.0002561 0.03% MultiFab::Copy() 11 0.0002558 0.0002558 0.0002558 0.03% MLMG::MLResNormInf() 11 0.0002534 0.0002534 0.0002534 0.03% MLLinOp::define() 11 0.0002493 0.0002493 0.0002493 0.03% MLLinOp::defineGrids() 11 0.0002248 0.0002248 0.0002248 0.03% MLMG::MLRhsNormInf() 11 0.0002033 0.0002033 0.0002033 0.02% Castro::create_source_corrector() 10 0.0001916 0.0001916 0.0001916 0.02% Castro::buildMetrics() 1 0.0001578 0.0001578 0.0001578 0.02% FabArrayBase::FB::FB() 56 8.472e-05 8.472e-05 8.472e-05 0.01% Castro::finalize_advance() 10 5.979e-05 5.979e-05 5.979e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.579e-05 5.579e-05 5.579e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.294e-05 5.294e-05 5.294e-05 0.01% StateData::define() 4 4.863e-05 4.863e-05 4.863e-05 0.01% Castro::swap_state_time_levels() 10 4.063e-05 4.063e-05 4.063e-05 0.00% makeSFC 55 4.019e-05 4.019e-05 4.019e-05 0.00% Castro::finalize_do_advance() 10 3.631e-05 3.631e-05 3.631e-05 0.00% Castro::enforce_consistent_e() 1 3.44e-05 3.44e-05 3.44e-05 0.00% Castro::initMFs() 1 2.848e-05 2.848e-05 2.848e-05 0.00% Amr::writeSmallPlotFile() 1 2.412e-05 2.412e-05 2.412e-05 0.00% DistributionMapping::Distribute() 56 1.514e-05 1.514e-05 1.514e-05 0.00% Amr::initSubcycle() 1 8.685e-06 8.685e-06 8.685e-06 0.00% MLMG::buildFineMask() 11 5.801e-06 5.801e-06 5.801e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.216e-06 5.216e-06 5.216e-06 0.00% AmrLevel::checkPointPost() 3 4.85e-06 4.85e-06 4.85e-06 0.00% Gravity::set_mass_offset() 11 4.27e-06 4.27e-06 4.27e-06 0.00% Castro::retry_advance_ctu() 10 3.789e-06 3.789e-06 3.789e-06 0.00% Castro::FluxRegCrseInit 10 2.843e-06 2.843e-06 2.843e-06 0.00% Castro::FluxRegFineAdd() 10 2.33e-06 2.33e-06 2.33e-06 0.00% AmrLevel::checkPointPre() 3 1.858e-06 1.858e-06 1.858e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.838e-06 1.838e-06 1.838e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.06-8-g8a1c76c0af89) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.06-8-g8a1c76c0af89) initialized Starting run at 08:29:43 UTC on 2022-06-07. Successfully read inputs file ... Castro git describe: 22.06-4-gfbc287aee AMReX git describe: 22.06-8-g8a1c76c0a Microphysics git describe: 22.06-1-gb8e427c1 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.466110825 Restart time = 0.04671113 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.049478934 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050315402 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.05034626 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.068985295 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.070250627 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026766432 seconds Ending run at 08:29:43 UTC on 2022-06-07. Run time = 0.363837382 Run time without initialization = 0.316536439 Average number of zones advanced per microsecond: 4.141 Average number of zones advanced per microsecond per rank: 4.141 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3639 ... 0.3639 ... 0.3639 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0947 0.0947 0.0947 26.01% VisMF::Read() 3 0.03937 0.03937 0.03937 10.82% MLCellLinOp::applyBC() 1946 0.03466 0.03466 0.03466 9.52% MLPoisson::Fsmooth() 1440 0.02716 0.02716 0.02716 7.46% VisMF::Write(FabArray) 1 0.0255 0.0255 0.0255 7.01% StateData::FillBoundary(geom) 160 0.01162 0.01162 0.01162 3.19% MLCGSolver::bicgstab 36 0.01027 0.01027 0.01027 2.82% MultiFab::Dot() 484 0.009521 0.009521 0.009521 2.62% Castro::normalize_species() 30 0.007891 0.007891 0.007891 2.17% FabArray::setVal() 537 0.006746 0.006746 0.006746 1.85% MLCellLinOp::defineAuxData() 6 0.006243 0.006243 0.006243 1.72% FillBoundary_nowait() 1766 0.006217 0.006217 0.006217 1.71% MultiFab::LinComb() 690 0.006135 0.006135 0.006135 1.69% FabArray::ParallelCopy_nowait() 380 0.005963 0.005963 0.005963 1.64% Castro::computeTemp() 30 0.005751 0.005751 0.005751 1.58% Castro::enforce_min_density() 30 0.005412 0.005412 0.005412 1.49% StateDataPhysBCFunct::() 20 0.005342 0.005342 0.005342 1.47% Gravity::fill_multipole_BCs() 6 0.005296 0.005296 0.005296 1.46% MLPoisson::Fapply() 500 0.005096 0.005096 0.005096 1.40% Castro::estTimeStep() 10 0.003174 0.003174 0.003174 0.87% MLMG::addInterpCorrection() 180 0.003161 0.003161 0.003161 0.87% Amr::restart() 1 0.003063 0.003063 0.003063 0.84% amrex::average_down 180 0.00297 0.00297 0.00297 0.82% MultiFab::Xpay() 258 0.002834 0.002834 0.002834 0.78% Castro::do_advance_ctu() 5 0.002201 0.002201 0.002201 0.60% BndryData::define() 6 0.002147 0.002147 0.002147 0.59% Castro::reset_internal_energy(MultiFab) 30 0.00209 0.00209 0.00209 0.57% Castro::construct_new_gravity_source() 5 0.001598 0.001598 0.001598 0.44% Castro::construct_old_gravity_source() 5 0.001416 0.001416 0.001416 0.39% Amr::writePlotFile() 1 0.001355 0.001355 0.001355 0.37% MultiFab::Saxpy() 10 0.0009196 0.0009196 0.0009196 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000882 0.000882 0.000882 0.24% Gravity::get_old_grav_vector() 5 0.0008763 0.0008763 0.0008763 0.24% Castro::expand_state() 5 0.0008696 0.0008696 0.0008696 0.24% Gravity::get_new_grav_vector() 5 0.0008639 0.0008639 0.0008639 0.24% MLMG::ResNormInf() 42 0.0008425 0.0008425 0.0008425 0.23% MLCellLinOp::setLevelBC() 6 0.0008163 0.0008163 0.0008163 0.22% MLMG::oneIter() 36 0.0007396 0.0007396 0.0007396 0.20% Castro::reset_internal_energy(Fab) 240 0.0007317 0.0007317 0.0007317 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007115 0.0007115 0.0007115 0.20% FabArray::mult() 22 0.0006456 0.0006456 0.0006456 0.18% FabArray::setDomainBndry() 20 0.0006445 0.0006445 0.0006445 0.18% MLCellLinOp::prepareForSolve() 6 0.0006308 0.0006308 0.0006308 0.17% MultiFab::contains_nan() 10 0.0005887 0.0005887 0.0005887 0.16% MLMG::prepareForSolve() 6 0.0005791 0.0005791 0.0005791 0.16% Castro::enforce_speed_limit() 30 0.000574 0.000574 0.000574 0.16% MLCellLinOp::compGrad() 6 0.0004943 0.0004943 0.0004943 0.14% MLCellLinOp::smooth() 720 0.0004681 0.0004681 0.0004681 0.13% FabArrayBase::CPC::define() 244 0.0003939 0.0003939 0.0003939 0.11% FabArray::FillBoundary() 1766 0.0003867 0.0003867 0.0003867 0.11% Amr::InitAmr() 1 0.0003754 0.0003754 0.0003754 0.10% FabArrayBase::getCPC() 632 0.0003613 0.0003613 0.0003613 0.10% main() 1 0.0002757 0.0002757 0.0002757 0.08% FabArrayBase::getFB() 1766 0.0002549 0.0002549 0.0002549 0.07% Gravity::update_max_rhs() 6 0.0002345 0.0002345 0.0002345 0.06% Gravity::solve_for_phi() 5 0.0002083 0.0002083 0.0002083 0.06% MLCellLinOp::apply() 500 0.0001991 0.0001991 0.0001991 0.05% CGSolver::sxay() 690 0.0001738 0.0001738 0.0001738 0.05% Amr::coarseTimeStep() 5 0.0001563 0.0001563 0.0001563 0.04% MLCellLinOp::defineBC() 6 0.0001515 0.0001515 0.0001515 0.04% FillPatchIterator::Initialize 20 0.0001384 0.0001384 0.0001384 0.04% MultiFab::Copy() 6 0.000138 0.000138 0.000138 0.04% MultiFab::max() 6 0.0001358 0.0001358 0.0001358 0.04% MLLinOp::defineGrids() 6 0.0001344 0.0001344 0.0001344 0.04% FabArray::ParallelCopy() 380 0.0001286 0.0001286 0.0001286 0.04% Castro::subcycle_advance_ctu() 5 0.0001195 0.0001195 0.0001195 0.03% MLCGSolver::ParallelAllReduce 659 0.0001174 0.0001174 0.0001174 0.03% Castro::create_source_corrector() 5 0.0001145 0.0001145 0.0001145 0.03% Castro::construct_new_gravity() 5 0.0001073 0.0001073 0.0001073 0.03% MLMG::MLRhsNormInf() 6 0.0001061 0.0001061 0.0001061 0.03% Amr::timeStep() 5 0.0001001 0.0001001 0.0001001 0.03% Castro::finalize_advance() 5 9.995e-05 9.995e-05 9.995e-05 0.03% MLCellLinOp::correctionResidual() 216 9.456e-05 9.456e-05 9.456e-05 0.03% MLMG::mgVcycle() 36 9.199e-05 9.199e-05 9.199e-05 0.03% AmrLevel::restart() 1 7.409e-05 7.409e-05 7.409e-05 0.02% StateData::restartDoit() 4 6.872e-05 6.872e-05 6.872e-05 0.02% Castro::initialize_advance() 5 5.918e-05 5.918e-05 5.918e-05 0.02% FabArrayBase::FB::FB() 26 5.779e-05 5.779e-05 5.779e-05 0.02% MLMG:computeResOfCorrection() 180 5.324e-05 5.324e-05 5.324e-05 0.01% Castro::construct_new_source() 25 4.836e-05 4.836e-05 4.836e-05 0.01% Castro::construct_old_source() 25 4.573e-05 4.573e-05 4.573e-05 0.01% MLMG::actualBottomSolve() 36 4.464e-05 4.464e-05 4.464e-05 0.01% Castro::clean_state() 30 3.939e-05 3.939e-05 3.939e-05 0.01% MLMG::mgVcycle_down::0 36 3.819e-05 3.819e-05 3.819e-05 0.01% MLMG::mgVcycle_down::1 36 3.594e-05 3.594e-05 3.594e-05 0.01% MLMG::mgVcycle_down::2 36 3.475e-05 3.475e-05 3.475e-05 0.01% Castro::post_restart() 1 3.416e-05 3.416e-05 3.416e-05 0.01% MLMG::solve() 6 3.384e-05 3.384e-05 3.384e-05 0.01% MLMG::mgVcycle_down::4 36 3.294e-05 3.294e-05 3.294e-05 0.01% Castro::buildMetrics() 1 3.21e-05 3.21e-05 3.21e-05 0.01% MLMG::mgVcycle_down::3 36 3.192e-05 3.192e-05 3.192e-05 0.01% Castro::initialize_do_advance() 5 3.182e-05 3.182e-05 3.182e-05 0.01% Gravity::actual_multilevel_solve() 1 3.046e-05 3.046e-05 3.046e-05 0.01% Castro::advance() 5 2.975e-05 2.975e-05 2.975e-05 0.01% Castro::swap_state_time_levels() 5 2.715e-05 2.715e-05 2.715e-05 0.01% MLMG::mgVcycle_up::4 36 2.675e-05 2.675e-05 2.675e-05 0.01% Castro::initMFs() 1 2.632e-05 2.632e-05 2.632e-05 0.01% Amr::writeSmallPlotFile() 1 2.523e-05 2.523e-05 2.523e-05 0.01% MLMG::mgVcycle_up::0 36 2.443e-05 2.443e-05 2.443e-05 0.01% MLCellLinOp::solutionResidual() 42 2.317e-05 2.317e-05 2.317e-05 0.01% MLMG::mgVcycle_up::2 36 2.309e-05 2.309e-05 2.309e-05 0.01% MLMG::mgVcycle_up::3 36 2.303e-05 2.303e-05 2.303e-05 0.01% MLMG::mgVcycle_up::1 36 2.113e-05 2.113e-05 2.113e-05 0.01% MLLinOp::define() 6 2.05e-05 2.05e-05 2.05e-05 0.01% Castro::finalize_do_advance() 5 1.854e-05 1.854e-05 1.854e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.775e-05 1.775e-05 1.775e-05 0.00% MLMG::computeResidual() 36 1.554e-05 1.554e-05 1.554e-05 0.00% MLMG::mgVcycle_bottom 36 1.442e-05 1.442e-05 1.442e-05 0.00% FillPatchSingleLevel 20 1.413e-05 1.413e-05 1.413e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.34e-05 1.34e-05 1.34e-05 0.00% makeSFC 30 1.339e-05 1.339e-05 1.339e-05 0.00% MLPoisson::define() 6 1.337e-05 1.337e-05 1.337e-05 0.00% Castro::do_new_sources() 5 9.076e-06 9.076e-06 9.076e-06 0.00% DistributionMapping::Distribute() 31 8.681e-06 8.681e-06 8.681e-06 0.00% Castro::do_old_sources() 5 8.48e-06 8.48e-06 8.48e-06 0.00% Amr::initSubcycle() 1 8.209e-06 8.209e-06 8.209e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.092e-06 7.092e-06 7.092e-06 0.00% Castro::check_for_nan() 10 7.074e-06 7.074e-06 7.074e-06 0.00% Castro::apply_source_to_state() 10 6.04e-06 6.04e-06 6.04e-06 0.00% Castro::construct_old_gravity() 5 5.506e-06 5.506e-06 5.506e-06 0.00% Gravity::swapTimeLevels() 5 4.832e-06 4.832e-06 4.832e-06 0.00% Castro::post_timestep() 5 4.669e-06 4.669e-06 4.669e-06 0.00% MLPoisson::prepareForSolve() 6 4.57e-06 4.57e-06 4.57e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.098e-06 4.098e-06 4.098e-06 0.00% MLMG::buildFineMask() 6 3.387e-06 3.387e-06 3.387e-06 0.00% MLMG::computeMLResidual() 6 3.239e-06 3.239e-06 3.239e-06 0.00% Castro::computeNewDt() 5 3.139e-06 3.139e-06 3.139e-06 0.00% MLMG::getGradSolution() 6 2.935e-06 2.935e-06 2.935e-06 0.00% MLMG::MLResNormInf() 6 2.32e-06 2.32e-06 2.32e-06 0.00% Gravity::set_mass_offset() 6 2.08e-06 2.08e-06 2.08e-06 0.00% Castro::retry_advance_ctu() 5 1.867e-06 1.867e-06 1.867e-06 0.00% Castro::FluxRegCrseInit 5 1.808e-06 1.808e-06 1.808e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.215e-06 1.215e-06 1.215e-06 0.00% Castro::FluxRegFineAdd() 5 1.179e-06 1.179e-06 1.179e-06 0.00% Amr::init() 1 1.082e-06 1.082e-06 1.082e-06 0.00% AmrLevel::AmrLevel() 1 8.76e-07 8.76e-07 8.76e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3639 0.3639 0.3639 100.00% Amr::coarseTimeStep() 5 0.2895 0.2895 0.2895 79.57% Amr::timeStep() 5 0.2873 0.2873 0.2873 78.95% Castro::advance() 5 0.2837 0.2837 0.2837 77.97% Castro::subcycle_advance_ctu() 5 0.2783 0.2783 0.2783 76.47% Castro::do_advance_ctu() 5 0.2781 0.2781 0.2781 76.44% Castro::construct_new_gravity() 5 0.1435 0.1435 0.1435 39.44% Gravity::solve_phi_with_mlmg() 6 0.1397 0.1397 0.1397 38.39% Gravity::solve_for_phi() 5 0.1361 0.1361 0.1361 37.41% Gravity::actual_solve_with_mlmg() 6 0.1343 0.1343 0.1343 36.91% MLMG::solve() 6 0.122 0.122 0.122 33.52% MLMG::oneIter() 36 0.115 0.115 0.115 31.60% MLMG::mgVcycle() 36 0.1143 0.1143 0.1143 31.40% Castro::construct_ctu_hydro_source() 5 0.09466 0.09466 0.09466 26.01% MLCellLinOp::smooth() 720 0.05843 0.05843 0.05843 16.06% Amr::init() 1 0.04679 0.04679 0.04679 12.86% Amr::restart() 1 0.04678 0.04678 0.04678 12.86% MLCellLinOp::applyBC() 1946 0.04157 0.04157 0.04157 11.43% AmrLevel::restart() 1 0.03957 0.03957 0.03957 10.88% StateData::restartDoit() 4 0.03949 0.03949 0.03949 10.85% VisMF::Read() 3 0.03937 0.03937 0.03937 10.82% MLMG::mgVcycle_bottom 36 0.03512 0.03512 0.03512 9.65% MLMG::actualBottomSolve() 36 0.03511 0.03511 0.03511 9.65% MLCGSolver::bicgstab 36 0.03475 0.03475 0.03475 9.55% MLPoisson::Fsmooth() 1440 0.02716 0.02716 0.02716 7.46% Amr::writePlotFile() 1 0.02685 0.02685 0.02685 7.38% VisMF::Write(FabArray) 1 0.0255 0.0255 0.0255 7.01% Castro::clean_state() 30 0.02249 0.02249 0.02249 6.18% FillPatchIterator::Initialize 20 0.01974 0.01974 0.01974 5.43% FillPatchSingleLevel 20 0.01896 0.01896 0.01896 5.21% StateDataPhysBCFunct::() 20 0.01696 0.01696 0.01696 4.66% MLCellLinOp::apply() 500 0.0158 0.0158 0.0158 4.34% MLMG::mgVcycle_down::0 36 0.01532 0.01532 0.01532 4.21% MLMG::mgVcycle_up::0 36 0.01304 0.01304 0.01304 3.58% StateData::FillBoundary(geom) 160 0.01162 0.01162 0.01162 3.19% MLPoisson::define() 6 0.01001 0.01001 0.01001 2.75% Castro::initialize_do_advance() 5 0.009564 0.009564 0.009564 2.63% MultiFab::Dot() 484 0.009521 0.009521 0.009521 2.62% MLCellLinOp::correctionResidual() 216 0.009205 0.009205 0.009205 2.53% Castro::computeTemp() 30 0.008572 0.008572 0.008572 2.36% Castro::do_new_sources() 5 0.008462 0.008462 0.008462 2.33% MLMG:computeResOfCorrection() 180 0.007932 0.007932 0.007932 2.18% Castro::normalize_species() 30 0.007891 0.007891 0.007891 2.17% MLMG::mgVcycle_down::1 36 0.007614 0.007614 0.007614 2.09% Castro::construct_old_gravity() 5 0.007441 0.007441 0.007441 2.05% Gravity::get_old_grav_vector() 5 0.007436 0.007436 0.007436 2.04% MLMG::mgVcycle_down::2 36 0.007415 0.007415 0.007415 2.04% Gravity::get_new_grav_vector() 5 0.007301 0.007301 0.007301 2.01% MLMG::mgVcycle_down::3 36 0.006999 0.006999 0.006999 1.92% MLCellLinOp::defineAuxData() 6 0.00697 0.00697 0.00697 1.92% FabArray::FillBoundary() 1766 0.006917 0.006917 0.006917 1.90% FabArray::setVal() 537 0.006746 0.006746 0.006746 1.85% MLMG::mgVcycle_down::4 36 0.006707 0.006707 0.006707 1.84% FillBoundary_nowait() 1766 0.00653 0.00653 0.00653 1.79% FabArray::ParallelCopy() 380 0.006461 0.006461 0.006461 1.78% FabArray::ParallelCopy_nowait() 380 0.006332 0.006332 0.006332 1.74% CGSolver::sxay() 690 0.006308 0.006308 0.006308 1.73% MultiFab::LinComb() 690 0.006135 0.006135 0.006135 1.69% MLCGSolver::ParallelAllReduce 659 0.005708 0.005708 0.005708 1.57% MLMG::mgVcycle_up::2 36 0.005663 0.005663 0.005663 1.56% Castro::expand_state() 5 0.005604 0.005604 0.005604 1.54% MLMG::mgVcycle_up::1 36 0.005597 0.005597 0.005597 1.54% Castro::enforce_min_density() 30 0.005412 0.005412 0.005412 1.49% MLMG::addInterpCorrection() 180 0.005392 0.005392 0.005392 1.48% MLMG::mgVcycle_up::3 36 0.005356 0.005356 0.005356 1.47% MLMG::mgVcycle_up::4 36 0.005324 0.005324 0.005324 1.46% Castro::initialize_advance() 5 0.005321 0.005321 0.005321 1.46% Gravity::fill_multipole_BCs() 6 0.005296 0.005296 0.005296 1.46% amrex::average_down 180 0.005218 0.005218 0.005218 1.43% MLPoisson::Fapply() 500 0.005096 0.005096 0.005096 1.40% Castro::do_old_sources() 5 0.004737 0.004737 0.004737 1.30% Castro::post_restart() 1 0.003967 0.003967 0.003967 1.09% Gravity::multilevel_solve_for_new_phi() 1 0.003835 0.003835 0.003835 1.05% Gravity::actual_multilevel_solve() 1 0.003817 0.003817 0.003817 1.05% Castro::post_timestep() 5 0.003445 0.003445 0.003445 0.95% MLCellLinOp::solutionResidual() 42 0.003211 0.003211 0.003211 0.88% Castro::estTimeStep() 10 0.003174 0.003174 0.003174 0.87% MLCellLinOp::defineBC() 6 0.002839 0.002839 0.002839 0.78% MultiFab::Xpay() 258 0.002834 0.002834 0.002834 0.78% Castro::reset_internal_energy(MultiFab) 30 0.002822 0.002822 0.002822 0.78% MLMG::prepareForSolve() 6 0.002773 0.002773 0.002773 0.76% BndryData::define() 6 0.002688 0.002688 0.002688 0.74% MLMG::computeResidual() 36 0.002663 0.002663 0.002663 0.73% Castro::computeNewDt() 5 0.002111 0.002111 0.002111 0.58% Castro::construct_new_source() 25 0.001646 0.001646 0.001646 0.45% Castro::construct_new_gravity_source() 5 0.001598 0.001598 0.001598 0.44% Castro::construct_old_source() 25 0.001462 0.001462 0.001462 0.40% Castro::construct_old_gravity_source() 5 0.001416 0.001416 0.001416 0.39% Castro::apply_source_to_state() 10 0.0009256 0.0009256 0.0009256 0.25% MultiFab::Saxpy() 10 0.0009196 0.0009196 0.0009196 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000882 0.000882 0.000882 0.24% MLMG::ResNormInf() 42 0.0008425 0.0008425 0.0008425 0.23% MLCellLinOp::setLevelBC() 6 0.0008163 0.0008163 0.0008163 0.22% MLMG::getGradSolution() 6 0.0007657 0.0007657 0.0007657 0.21% MLCellLinOp::compGrad() 6 0.0007627 0.0007627 0.0007627 0.21% FabArrayBase::getCPC() 632 0.0007553 0.0007553 0.0007553 0.21% Castro::reset_internal_energy(Fab) 240 0.0007317 0.0007317 0.0007317 0.20% FabArray::mult() 22 0.0006456 0.0006456 0.0006456 0.18% FabArray::setDomainBndry() 20 0.0006445 0.0006445 0.0006445 0.18% MLPoisson::prepareForSolve() 6 0.0006354 0.0006354 0.0006354 0.17% MLCellLinOp::prepareForSolve() 6 0.0006308 0.0006308 0.0006308 0.17% Castro::check_for_nan() 10 0.0005958 0.0005958 0.0005958 0.16% MultiFab::contains_nan() 10 0.0005887 0.0005887 0.0005887 0.16% Castro::enforce_speed_limit() 30 0.000574 0.000574 0.000574 0.16% MLMG::computeMLResidual() 6 0.0005669 0.0005669 0.0005669 0.16% Gravity::update_max_rhs() 6 0.0004491 0.0004491 0.0004491 0.12% FabArrayBase::CPC::define() 244 0.0003939 0.0003939 0.0003939 0.11% Amr::InitAmr() 1 0.0003837 0.0003837 0.0003837 0.11% FabArrayBase::getFB() 1766 0.0003127 0.0003127 0.0003127 0.09% Gravity::swapTimeLevels() 5 0.0002285 0.0002285 0.0002285 0.06% MLLinOp::define() 6 0.0001837 0.0001837 0.0001837 0.05% MLLinOp::defineGrids() 6 0.0001632 0.0001632 0.0001632 0.04% Castro::buildMetrics() 1 0.0001543 0.0001543 0.0001543 0.04% MultiFab::Copy() 6 0.000138 0.000138 0.000138 0.04% MultiFab::max() 6 0.0001358 0.0001358 0.0001358 0.04% MLMG::MLResNormInf() 6 0.0001336 0.0001336 0.0001336 0.04% Castro::create_source_corrector() 5 0.0001145 0.0001145 0.0001145 0.03% MLMG::MLRhsNormInf() 6 0.0001061 0.0001061 0.0001061 0.03% Castro::finalize_advance() 5 0.0001029 0.0001029 0.0001029 0.03% FabArrayBase::FB::FB() 26 5.779e-05 5.779e-05 5.779e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 2.761e-05 2.761e-05 2.761e-05 0.01% Castro::swap_state_time_levels() 5 2.715e-05 2.715e-05 2.715e-05 0.01% Castro::initMFs() 1 2.632e-05 2.632e-05 2.632e-05 0.01% Amr::writeSmallPlotFile() 1 2.523e-05 2.523e-05 2.523e-05 0.01% makeSFC 30 2.052e-05 2.052e-05 2.052e-05 0.01% Castro::finalize_do_advance() 5 1.854e-05 1.854e-05 1.854e-05 0.01% DistributionMapping::Distribute() 31 8.681e-06 8.681e-06 8.681e-06 0.00% Amr::initSubcycle() 1 8.209e-06 8.209e-06 8.209e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.646e-06 5.646e-06 5.646e-06 0.00% MLMG::buildFineMask() 6 3.387e-06 3.387e-06 3.387e-06 0.00% Gravity::set_mass_offset() 6 2.08e-06 2.08e-06 2.08e-06 0.00% Castro::retry_advance_ctu() 5 1.867e-06 1.867e-06 1.867e-06 0.00% Castro::FluxRegCrseInit 5 1.808e-06 1.808e-06 1.808e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.215e-06 1.215e-06 1.215e-06 0.00% Castro::FluxRegFineAdd() 5 1.179e-06 1.179e-06 1.179e-06 0.00% AmrLevel::AmrLevel() 1 8.76e-07 8.76e-07 8.76e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.06-8-g8a1c76c0af89) finalized