Initializing CUDA... CUDA initialized with 1 device. AMReX (23.01-10-g2e0b2b463306) initialized Starting run at 10:09:55 UTC on 2023-01-06. Successfully read inputs file ... Castro git describe: 23.01-12-g63195e409 AMReX git describe: 23.01-10-g2e0b2b463 Microphysics git describe: 23.01 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.056693072 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.032475061 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.051820834 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.056116253 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.063326867 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.067271068 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.061918743 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.056939873 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.075482432 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.074616228 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.059582613 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.062523985 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065413609 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.056230273 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032459895 seconds Ending run at 10:09:56 UTC on 2023-01-06. Run time = 0.924690555 Run time without initialization = 0.784332327 Average number of zones advanced per microsecond: 3.342 Average number of zones advanced per microsecond per rank: 3.342 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.9247 ... 0.9247 ... 0.9247 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2379 0.2379 0.2379 25.73% VisMF::Write(FabArray) 11 0.2271 0.2271 0.2271 24.55% MLCellLinOp::applyBC() 4433 0.08081 0.08081 0.08081 8.74% StateData::FillBoundary(geom) 328 0.05762 0.05762 0.05762 6.23% MLPoisson::Fsmooth() 3280 0.03401 0.03401 0.03401 3.68% FillBoundary_nowait() 4023 0.03066 0.03066 0.03066 3.32% amrex::Dot() 1114 0.02081 0.02081 0.02081 2.25% amrex::Copy() 1029 0.01522 0.01522 0.01522 1.65% FabArray::norminf() 743 0.01479 0.01479 0.01479 1.60% FabArray::ParallelCopy_nowait() 861 0.01404 0.01404 0.01404 1.52% Castro::computeTemp() 63 0.01386 0.01386 0.01386 1.50% FabArray::setVal() 1144 0.01362 0.01362 0.01362 1.47% Castro::normalize_species() 62 0.01344 0.01344 0.01344 1.45% Castro::enforce_min_density() 62 0.01156 0.01156 0.01156 1.25% MLPoisson::Fapply() 1142 0.01078 0.01078 0.01078 1.17% MLCellLinOp::defineAuxData() 11 0.01027 0.01027 0.01027 1.11% StateDataPhysBCFunct::() 41 0.009054 0.009054 0.009054 0.98% FabArray::Saxpy() 813 0.008368 0.008368 0.008368 0.90% FabArray::Xpay() 821 0.008205 0.008205 0.008205 0.89% MLCGSolver::bicgstab 82 0.007771 0.007771 0.007771 0.84% MLMG::addInterpCorrection() 410 0.007117 0.007117 0.007117 0.77% amrex::average_down 410 0.006541 0.006541 0.006541 0.71% Gravity::fill_multipole_BCs() 11 0.005873 0.005873 0.005873 0.64% Amr::checkPoint() 3 0.004892 0.004892 0.004892 0.53% FabArray::LinComb() 557 0.004583 0.004583 0.004583 0.50% amrex::Add() 164 0.004297 0.004297 0.004297 0.46% Castro::reset_internal_energy(MultiFab) 63 0.004105 0.004105 0.004105 0.44% Castro::estTimeStep() 21 0.004077 0.004077 0.004077 0.44% BndryData::define() 11 0.003766 0.003766 0.003766 0.41% Castro::construct_new_gravity_source() 10 0.00268 0.00268 0.00268 0.29% Castro::do_advance_ctu() 10 0.002669 0.002669 0.002669 0.29% Amr::writePlotFile() 2 0.002327 0.002327 0.002327 0.25% Castro::construct_old_gravity_source() 10 0.002074 0.002074 0.002074 0.22% Castro::reset_internal_energy(Fab) 504 0.001775 0.001775 0.001775 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001772 0.001772 0.001772 0.19% MLCellLinOp::smooth() 1640 0.00164 0.00164 0.00164 0.18% MLCellLinOp::setLevelBC() 11 0.001397 0.001397 0.001397 0.15% Gravity::actual_solve_with_mlmg() 11 0.001393 0.001393 0.001393 0.15% FabArray::mult() 43 0.001349 0.001349 0.001349 0.15% FabArray::setDomainBndry() 41 0.001328 0.001328 0.001328 0.14% MultiFab::contains_nan() 20 0.001213 0.001213 0.001213 0.13% MLCellLinOp::prepareForSolve() 11 0.001187 0.001187 0.001187 0.13% Castro::initData() 1 0.001182 0.001182 0.001182 0.13% Castro::enforce_speed_limit() 62 0.001066 0.001066 0.001066 0.12% MLMG::prepareForSolve() 11 0.0009654 0.0009654 0.0009654 0.10% FabArray::FillBoundary() 4023 0.0009546 0.0009546 0.0009546 0.10% MLCellLinOp::compGrad() 11 0.00094 0.00094 0.00094 0.10% FabArrayBase::getCPC() 1323 0.000761 0.000761 0.000761 0.08% FabArrayBase::CPC::define() 454 0.0007295 0.0007295 0.0007295 0.08% Castro::subcycle_advance_ctu() 10 0.0007228 0.0007228 0.0007228 0.08% MLCellLinOp::apply() 1142 0.0006362 0.0006362 0.0006362 0.07% Gravity::get_new_grav_vector() 11 0.0006318 0.0006318 0.0006318 0.07% FabArrayBase::getFB() 4023 0.0006212 0.0006212 0.0006212 0.07% Gravity::get_old_grav_vector() 10 0.0005522 0.0005522 0.0005522 0.06% MLMG::mgVcycle() 82 0.0005368 0.0005368 0.0005368 0.06% Amr::InitAmr() 1 0.0004803 0.0004803 0.0004803 0.05% Amr::coarseTimeStep() 10 0.0003443 0.0003443 0.0003443 0.04% MLCGSolver::ParallelAllReduce 1514 0.000324 0.000324 0.000324 0.04% MLCellLinOp::correctionResidual() 492 0.0003059 0.0003059 0.0003059 0.03% main() 1 0.000295 0.000295 0.000295 0.03% FabArray::ParallelCopy() 861 0.0002741 0.0002741 0.0002741 0.03% MultiFab::max() 11 0.0002705 0.0002705 0.0002705 0.03% FillPatchIterator::Initialize 41 0.0002412 0.0002412 0.0002412 0.03% MLCellLinOp::defineBC() 11 0.0002022 0.0002022 0.0002022 0.02% MLLinOp::defineGrids() 11 0.0001905 0.0001905 0.0001905 0.02% Amr::timeStep() 10 0.0001777 0.0001777 0.0001777 0.02% Gravity::solve_for_phi() 10 0.0001683 0.0001683 0.0001683 0.02% StateData::checkPoint() 12 0.0001409 0.0001409 0.0001409 0.02% Gravity::update_max_rhs() 11 0.0001338 0.0001338 0.0001338 0.01% MLMG::mgVcycle_down::0 82 0.0001229 0.0001229 0.0001229 0.01% MLMG:computeResOfCorrection() 410 0.000122 0.000122 0.000122 0.01% Castro::finalize_advance() 10 0.0001188 0.0001188 0.0001188 0.01% MLMG::mgVcycle_down::1 82 0.0001163 0.0001163 0.0001163 0.01% MLMG::mgVcycle_down::2 82 0.0001139 0.0001139 0.0001139 0.01% MLMG::actualBottomSolve() 82 0.0001125 0.0001125 0.0001125 0.01% MLMG::mgVcycle_down::3 82 0.0001102 0.0001102 0.0001102 0.01% MLMG::mgVcycle_down::4 82 0.0001095 0.0001095 0.0001095 0.01% Castro::clean_state() 62 0.0001002 0.0001002 0.0001002 0.01% FabArrayBase::FB::FB() 56 9.582e-05 9.582e-05 9.582e-05 0.01% MLMG::solve() 11 9.262e-05 9.262e-05 9.262e-05 0.01% Castro::expand_state() 10 8.806e-05 8.806e-05 8.806e-05 0.01% Castro::Castro() 1 8.052e-05 8.052e-05 8.052e-05 0.01% Castro::initialize_advance() 10 8.023e-05 8.023e-05 8.023e-05 0.01% AmrLevel::checkPoint() 3 7.757e-05 7.757e-05 7.757e-05 0.01% MLMG::mgVcycle_up::0 82 7.198e-05 7.198e-05 7.198e-05 0.01% MLMG::mgVcycle_up::4 82 6.893e-05 6.893e-05 6.893e-05 0.01% MLMG::mgVcycle_up::1 82 6.369e-05 6.369e-05 6.369e-05 0.01% MLMG::mgVcycle_up::3 82 6.268e-05 6.268e-05 6.268e-05 0.01% MLMG::mgVcycle_up::2 82 6.147e-05 6.147e-05 6.147e-05 0.01% MLCellLinOp::solutionResidual() 93 5.923e-05 5.923e-05 5.923e-05 0.01% MLMG::oneIter() 82 5.779e-05 5.779e-05 5.779e-05 0.01% Castro::initialize_do_advance() 10 5.109e-05 5.109e-05 5.109e-05 0.01% Amr::InitializeInit() 1 4.55e-05 4.55e-05 4.55e-05 0.00% Castro::finalize_do_advance() 10 3.922e-05 3.922e-05 3.922e-05 0.00% Castro::swap_state_time_levels() 10 3.788e-05 3.788e-05 3.788e-05 0.00% FillPatchSingleLevel 41 3.756e-05 3.756e-05 3.756e-05 0.00% MLMG::ResNormInf() 93 3.664e-05 3.664e-05 3.664e-05 0.00% Castro::enforce_consistent_e() 1 3.515e-05 3.515e-05 3.515e-05 0.00% MLMG::mgVcycle_bottom 82 3.38e-05 3.38e-05 3.38e-05 0.00% MLMG::computeResidual() 82 3.318e-05 3.318e-05 3.318e-05 0.00% Castro::construct_new_gravity() 10 3.167e-05 3.167e-05 3.167e-05 0.00% StateData::define() 4 2.726e-05 2.726e-05 2.726e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.589e-05 2.589e-05 2.589e-05 0.00% makeSFC 55 2.506e-05 2.506e-05 2.506e-05 0.00% Amr::writeSmallPlotFile() 1 2.292e-05 2.292e-05 2.292e-05 0.00% MLPoisson::define() 11 2.276e-05 2.276e-05 2.276e-05 0.00% Castro::advance() 10 2.232e-05 2.232e-05 2.232e-05 0.00% Amr::FinalizeInit() 1 2.193e-05 2.193e-05 2.193e-05 0.00% Castro::do_new_sources() 10 2.107e-05 2.107e-05 2.107e-05 0.00% Castro::do_old_sources() 10 1.961e-05 1.961e-05 1.961e-05 0.00% Castro::initMFs() 1 1.927e-05 1.927e-05 1.927e-05 0.00% Castro::construct_old_source() 50 1.923e-05 1.923e-05 1.923e-05 0.00% Amr::defBaseLevel() 1 1.882e-05 1.882e-05 1.882e-05 0.00% Castro::buildMetrics() 1 1.802e-05 1.802e-05 1.802e-05 0.00% Castro::construct_new_source() 50 1.791e-05 1.791e-05 1.791e-05 0.00% DistributionMapping::Distribute() 56 1.694e-05 1.694e-05 1.694e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.505e-05 1.505e-05 1.505e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.473e-05 1.473e-05 1.473e-05 0.00% Castro::check_for_nan() 20 1.317e-05 1.317e-05 1.317e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.311e-05 1.311e-05 1.311e-05 0.00% Gravity::swapTimeLevels() 10 1.264e-05 1.264e-05 1.264e-05 0.00% Castro::apply_source_to_state() 20 1.195e-05 1.195e-05 1.195e-05 0.00% Castro::construct_old_gravity() 10 1.158e-05 1.158e-05 1.158e-05 0.00% Castro::post_timestep() 10 1.151e-05 1.151e-05 1.151e-05 0.00% Castro::post_init() 1 1.043e-05 1.043e-05 1.043e-05 0.00% MLLinOp::define() 11 9.284e-06 9.284e-06 9.284e-06 0.00% Amr::initSubcycle() 1 8.641e-06 8.641e-06 8.641e-06 0.00% MLPoisson::prepareForSolve() 11 8.199e-06 8.199e-06 8.199e-06 0.00% MLMG::computeMLResidual() 11 8.154e-06 8.154e-06 8.154e-06 0.00% Castro::computeNewDt() 9 7.91e-06 7.91e-06 7.91e-06 0.00% Gravity::actual_multilevel_solve() 1 6.986e-06 6.986e-06 6.986e-06 0.00% MLMG::getGradSolution() 11 6.333e-06 6.333e-06 6.333e-06 0.00% AmrLevel::checkPointPost() 3 6.28e-06 6.28e-06 6.28e-06 0.00% Gravity::set_mass_offset() 11 5.064e-06 5.064e-06 5.064e-06 0.00% Castro::retry_advance_ctu() 10 4.486e-06 4.486e-06 4.486e-06 0.00% Castro::create_source_corrector() 10 4.474e-06 4.474e-06 4.474e-06 0.00% MLMG::MLRhsNormInf() 11 3.915e-06 3.915e-06 3.915e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.718e-06 3.718e-06 3.718e-06 0.00% MLMG::MLResNormInf() 11 3.42e-06 3.42e-06 3.42e-06 0.00% Castro::FluxRegCrseInit 10 3.222e-06 3.222e-06 3.222e-06 0.00% Castro::computeInitialDt() 2 2.635e-06 2.635e-06 2.635e-06 0.00% Castro::FluxRegFineAdd() 10 2.462e-06 2.462e-06 2.462e-06 0.00% Amr::init() 1 2.448e-06 2.448e-06 2.448e-06 0.00% AmrLevel::checkPointPre() 3 2.148e-06 2.148e-06 2.148e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.962e-06 1.962e-06 1.962e-06 0.00% Castro::post_regrid() 1 1.364e-06 1.364e-06 1.364e-06 0.00% Amr::initialInit() 1 1.116e-06 1.116e-06 1.116e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9247 0.9247 0.9247 100.00% Amr::coarseTimeStep() 10 0.7517 0.7517 0.7517 81.28% Amr::timeStep() 10 0.6357 0.6357 0.6357 68.75% Castro::advance() 10 0.6292 0.6292 0.6292 68.05% Castro::subcycle_advance_ctu() 10 0.6164 0.6164 0.6164 66.66% Castro::do_advance_ctu() 10 0.6157 0.6157 0.6157 66.58% Gravity::solve_phi_with_mlmg() 11 0.2996 0.2996 0.2996 32.39% Gravity::actual_solve_with_mlmg() 11 0.2932 0.2932 0.2932 31.71% Castro::construct_new_gravity() 10 0.2841 0.2841 0.2841 30.72% MLMG::solve() 11 0.272 0.272 0.272 29.42% Gravity::solve_for_phi() 10 0.2575 0.2575 0.2575 27.85% MLMG::oneIter() 82 0.2573 0.2573 0.2573 27.82% MLMG::mgVcycle() 82 0.2537 0.2537 0.2537 27.44% Castro::construct_ctu_hydro_source() 10 0.2379 0.2379 0.2379 25.73% VisMF::Write(FabArray) 11 0.2271 0.2271 0.2271 24.55% Amr::checkPoint() 3 0.17 0.17 0.17 18.38% AmrLevel::checkPoint() 3 0.1651 0.1651 0.1651 17.85% StateData::checkPoint() 12 0.165 0.165 0.165 17.84% Amr::init() 1 0.1397 0.1397 0.1397 15.11% MLCellLinOp::smooth() 1640 0.1218 0.1218 0.1218 13.17% MLCellLinOp::applyBC() 4433 0.1131 0.1131 0.1131 12.24% MLMG::mgVcycle_bottom 82 0.08178 0.08178 0.08178 8.84% MLMG::actualBottomSolve() 82 0.08174 0.08174 0.08174 8.84% MLCGSolver::bicgstab 82 0.08099 0.08099 0.08099 8.76% FillPatchIterator::Initialize 41 0.07228 0.07228 0.07228 7.82% FillPatchSingleLevel 41 0.07071 0.07071 0.07071 7.65% StateDataPhysBCFunct::() 41 0.06668 0.06668 0.06668 7.21% Amr::writePlotFile() 2 0.06505 0.06505 0.06505 7.03% StateData::FillBoundary(geom) 328 0.05762 0.05762 0.05762 6.23% Amr::initialInit() 1 0.05045 0.05045 0.05045 5.46% Amr::FinalizeInit() 1 0.04671 0.04671 0.04671 5.05% Castro::post_init() 1 0.04543 0.04543 0.04543 4.91% Castro::clean_state() 62 0.04527 0.04527 0.04527 4.90% Gravity::multilevel_solve_for_new_phi() 1 0.04254 0.04254 0.04254 4.60% Gravity::actual_multilevel_solve() 1 0.04253 0.04253 0.04253 4.60% MLCellLinOp::apply() 1142 0.03797 0.03797 0.03797 4.11% MLMG::mgVcycle_down::0 82 0.03412 0.03412 0.03412 3.69% MLPoisson::Fsmooth() 3280 0.03401 0.03401 0.03401 3.68% FabArray::FillBoundary() 4023 0.03233 0.03233 0.03233 3.50% FillBoundary_nowait() 4023 0.03138 0.03138 0.03138 3.39% Gravity::get_new_grav_vector() 11 0.0291 0.0291 0.0291 3.15% Castro::construct_old_gravity() 10 0.02613 0.02613 0.02613 2.83% Gravity::get_old_grav_vector() 10 0.02612 0.02612 0.02612 2.82% MLMG::mgVcycle_up::0 82 0.02607 0.02607 0.02607 2.82% Castro::initialize_do_advance() 10 0.02309 0.02309 0.02309 2.50% MLCellLinOp::correctionResidual() 492 0.02294 0.02294 0.02294 2.48% amrex::Dot() 1114 0.02081 0.02081 0.02081 2.25% MLMG:computeResOfCorrection() 410 0.02013 0.02013 0.02013 2.18% Castro::computeTemp() 63 0.01974 0.01974 0.01974 2.13% MLPoisson::define() 11 0.01695 0.01695 0.01695 1.83% MLMG::mgVcycle_down::1 82 0.01667 0.01667 0.01667 1.80% MLMG::mgVcycle_down::2 82 0.0156 0.0156 0.0156 1.69% MLMG::mgVcycle_down::3 82 0.01557 0.01557 0.01557 1.68% Castro::expand_state() 10 0.01551 0.01551 0.01551 1.68% amrex::Copy() 1029 0.01522 0.01522 0.01522 1.65% MLMG::mgVcycle_down::4 82 0.01516 0.01516 0.01516 1.64% FabArray::ParallelCopy() 861 0.01515 0.01515 0.01515 1.64% FabArray::ParallelCopy_nowait() 861 0.01487 0.01487 0.01487 1.61% FabArray::norminf() 743 0.01479 0.01479 0.01479 1.60% FabArray::setVal() 1144 0.01362 0.01362 0.01362 1.47% Castro::normalize_species() 62 0.01344 0.01344 0.01344 1.45% MLMG::addInterpCorrection() 410 0.01275 0.01275 0.01275 1.38% MLCGSolver::ParallelAllReduce 1514 0.01252 0.01252 0.01252 1.35% MLMG::mgVcycle_up::4 82 0.0123 0.0123 0.0123 1.33% MLMG::mgVcycle_up::1 82 0.01221 0.01221 0.01221 1.32% Castro::initialize_advance() 10 0.01214 0.01214 0.01214 1.31% amrex::average_down 410 0.01206 0.01206 0.01206 1.30% MLMG::mgVcycle_up::2 82 0.01196 0.01196 0.01196 1.29% Castro::do_new_sources() 10 0.01188 0.01188 0.01188 1.29% MLMG::mgVcycle_up::3 82 0.01175 0.01175 0.01175 1.27% MLCellLinOp::defineAuxData() 11 0.0117 0.0117 0.0117 1.26% Castro::enforce_min_density() 62 0.01156 0.01156 0.01156 1.25% MLPoisson::Fapply() 1142 0.01078 0.01078 0.01078 1.17% Castro::do_old_sources() 10 0.01013 0.01013 0.01013 1.10% FabArray::Saxpy() 813 0.008368 0.008368 0.008368 0.90% FabArray::Xpay() 821 0.008205 0.008205 0.008205 0.89% MLCellLinOp::solutionResidual() 93 0.007342 0.007342 0.007342 0.79% MLMG::computeResidual() 82 0.006353 0.006353 0.006353 0.69% Castro::post_timestep() 10 0.006302 0.006302 0.006302 0.68% Gravity::fill_multipole_BCs() 11 0.006115 0.006115 0.006115 0.66% Castro::reset_internal_energy(MultiFab) 63 0.005881 0.005881 0.005881 0.64% MLCellLinOp::defineBC() 11 0.004975 0.004975 0.004975 0.54% MLMG::prepareForSolve() 11 0.004915 0.004915 0.004915 0.53% BndryData::define() 11 0.004773 0.004773 0.004773 0.52% FabArray::LinComb() 557 0.004583 0.004583 0.004583 0.50% amrex::Add() 164 0.004297 0.004297 0.004297 0.46% Castro::estTimeStep() 21 0.004077 0.004077 0.004077 0.44% Amr::InitializeInit() 1 0.00374 0.00374 0.00374 0.40% Amr::defBaseLevel() 1 0.003694 0.003694 0.003694 0.40% Castro::initData() 1 0.003186 0.003186 0.003186 0.34% Castro::construct_new_source() 50 0.002698 0.002698 0.002698 0.29% Castro::construct_new_gravity_source() 10 0.00268 0.00268 0.00268 0.29% MLMG::ResNormInf() 93 0.00214 0.00214 0.00214 0.23% Castro::construct_old_source() 50 0.002093 0.002093 0.002093 0.23% Castro::construct_old_gravity_source() 10 0.002074 0.002074 0.002074 0.22% Castro::apply_source_to_state() 20 0.001834 0.001834 0.001834 0.20% Castro::reset_internal_energy(Fab) 504 0.001775 0.001775 0.001775 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001772 0.001772 0.001772 0.19% Castro::computeNewDt() 9 0.001638 0.001638 0.001638 0.18% FabArrayBase::getCPC() 1323 0.001491 0.001491 0.001491 0.16% MLMG::getGradSolution() 11 0.001437 0.001437 0.001437 0.16% MLCellLinOp::compGrad() 11 0.00143 0.00143 0.00143 0.15% MLCellLinOp::setLevelBC() 11 0.001397 0.001397 0.001397 0.15% FabArray::mult() 43 0.001349 0.001349 0.001349 0.15% FabArray::setDomainBndry() 41 0.001328 0.001328 0.001328 0.14% Castro::check_for_nan() 20 0.001226 0.001226 0.001226 0.13% MultiFab::contains_nan() 20 0.001213 0.001213 0.001213 0.13% MLPoisson::prepareForSolve() 11 0.001195 0.001195 0.001195 0.13% MLCellLinOp::prepareForSolve() 11 0.001187 0.001187 0.001187 0.13% Castro::post_regrid() 1 0.001112 0.001112 0.001112 0.12% Castro::enforce_speed_limit() 62 0.001066 0.001066 0.001066 0.12% MLMG::computeMLResidual() 11 0.00103 0.00103 0.00103 0.11% Gravity::update_max_rhs() 11 0.0008743 0.0008743 0.0008743 0.09% Castro::computeInitialDt() 2 0.0008502 0.0008502 0.0008502 0.09% FabArrayBase::CPC::define() 454 0.0007295 0.0007295 0.0007295 0.08% FabArrayBase::getFB() 4023 0.000717 0.000717 0.000717 0.08% Castro::finalize_advance() 10 0.0006572 0.0006572 0.0006572 0.07% Amr::InitAmr() 1 0.0004889 0.0004889 0.0004889 0.05% Gravity::swapTimeLevels() 10 0.000458 0.000458 0.000458 0.05% Castro::Castro() 1 0.0004417 0.0004417 0.0004417 0.05% MLMG::MLResNormInf() 11 0.0002848 0.0002848 0.0002848 0.03% MultiFab::max() 11 0.0002705 0.0002705 0.0002705 0.03% MLLinOp::define() 11 0.0002556 0.0002556 0.0002556 0.03% MLLinOp::defineGrids() 11 0.0002463 0.0002463 0.0002463 0.03% MLMG::MLRhsNormInf() 11 0.0002207 0.0002207 0.0002207 0.02% Castro::buildMetrics() 1 0.0001763 0.0001763 0.0001763 0.02% FabArrayBase::FB::FB() 56 9.582e-05 9.582e-05 9.582e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.382e-05 5.382e-05 5.382e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.231e-05 4.231e-05 4.231e-05 0.00% makeSFC 55 4.072e-05 4.072e-05 4.072e-05 0.00% Castro::finalize_do_advance() 10 3.922e-05 3.922e-05 3.922e-05 0.00% Castro::swap_state_time_levels() 10 3.788e-05 3.788e-05 3.788e-05 0.00% Castro::enforce_consistent_e() 1 3.515e-05 3.515e-05 3.515e-05 0.00% StateData::define() 4 2.726e-05 2.726e-05 2.726e-05 0.00% Amr::writeSmallPlotFile() 1 2.292e-05 2.292e-05 2.292e-05 0.00% Castro::initMFs() 1 1.927e-05 1.927e-05 1.927e-05 0.00% DistributionMapping::Distribute() 56 1.694e-05 1.694e-05 1.694e-05 0.00% Amr::initSubcycle() 1 8.641e-06 8.641e-06 8.641e-06 0.00% AmrLevel::checkPointPost() 3 6.28e-06 6.28e-06 6.28e-06 0.00% Gravity::set_mass_offset() 11 5.064e-06 5.064e-06 5.064e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.997e-06 4.997e-06 4.997e-06 0.00% Castro::retry_advance_ctu() 10 4.486e-06 4.486e-06 4.486e-06 0.00% Castro::create_source_corrector() 10 4.474e-06 4.474e-06 4.474e-06 0.00% Castro::FluxRegCrseInit 10 3.222e-06 3.222e-06 3.222e-06 0.00% Castro::FluxRegFineAdd() 10 2.462e-06 2.462e-06 2.462e-06 0.00% AmrLevel::checkPointPre() 3 2.148e-06 2.148e-06 2.148e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.962e-06 1.962e-06 1.962e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-10-g2e0b2b463306) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.01-10-g2e0b2b463306) initialized Starting run at 10:09:57 UTC on 2023-01-06. Successfully read inputs file ... Castro git describe: 23.01-12-g63195e409 AMReX git describe: 23.01-10-g2e0b2b463 Microphysics git describe: 23.01 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.497761824 Restart time = 0.046712593 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.047019773 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.046819265 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.055163632 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.055173983 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.069889045 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032168953 seconds Ending run at 10:09:58 UTC on 2023-01-06. Run time = 0.35396522 Run time without initialization = 0.306683937 Average number of zones advanced per microsecond: 4.274 Average number of zones advanced per microsecond per rank: 4.274 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.354 ... 0.354 ... 0.354 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1000 0.1000 0.1000 28.25% VisMF::Read() 3 0.0406 0.0406 0.0406 11.47% MLCellLinOp::applyBC() 1946 0.03159 0.03159 0.03159 8.92% VisMF::Write(FabArray) 1 0.0306 0.0306 0.0306 8.64% MLPoisson::Fsmooth() 1440 0.01355 0.01355 0.01355 3.83% FillBoundary_nowait() 1766 0.01308 0.01308 0.01308 3.70% StateData::FillBoundary(geom) 160 0.0111 0.0111 0.0111 3.14% Castro::normalize_species() 30 0.008824 0.008824 0.008824 2.49% amrex::Dot() 484 0.00833 0.00833 0.00833 2.35% amrex::Copy() 463 0.006819 0.006819 0.006819 1.93% Castro::computeTemp() 30 0.006035 0.006035 0.006035 1.70% FabArray::setVal() 537 0.006009 0.006009 0.006009 1.70% FabArray::norminf() 326 0.005934 0.005934 0.005934 1.68% FabArray::ParallelCopy_nowait() 380 0.005768 0.005768 0.005768 1.63% StateDataPhysBCFunct::() 20 0.005696 0.005696 0.005696 1.61% MLCellLinOp::defineAuxData() 6 0.005089 0.005089 0.005089 1.44% MLPoisson::Fapply() 500 0.004332 0.004332 0.004332 1.22% FabArray::Saxpy() 355 0.003504 0.003504 0.003504 0.99% FabArray::Xpay() 361 0.003394 0.003394 0.003394 0.96% Gravity::fill_multipole_BCs() 6 0.002821 0.002821 0.002821 0.80% MLMG::addInterpCorrection() 180 0.002738 0.002738 0.002738 0.77% Amr::restart() 1 0.002472 0.002472 0.002472 0.70% amrex::average_down 180 0.002427 0.002427 0.002427 0.69% Castro::enforce_min_density() 30 0.00236 0.00236 0.00236 0.67% Castro::estTimeStep() 10 0.002242 0.002242 0.002242 0.63% BndryData::define() 6 0.001893 0.001893 0.001893 0.53% Castro::reset_internal_energy(MultiFab) 30 0.001868 0.001868 0.001868 0.53% FabArray::LinComb() 242 0.001866 0.001866 0.001866 0.53% amrex::Add() 72 0.001823 0.001823 0.001823 0.51% Castro::construct_new_gravity_source() 5 0.001501 0.001501 0.001501 0.42% Amr::writePlotFile() 1 0.001414 0.001414 0.001414 0.40% Castro::do_advance_ctu() 5 0.001237 0.001237 0.001237 0.35% Castro::enforce_speed_limit() 30 0.001207 0.001207 0.001207 0.34% Castro::construct_old_gravity_source() 5 0.001206 0.001206 0.001206 0.34% Castro::reset_internal_energy(Fab) 240 0.0009014 0.0009014 0.0009014 0.25% MLCGSolver::bicgstab 36 0.0008964 0.0008964 0.0008964 0.25% Castro::advance() 5 0.0008757 0.0008757 0.0008757 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008667 0.0008667 0.0008667 0.24% MLCellLinOp::setLevelBC() 6 0.000705 0.000705 0.000705 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007024 0.0007024 0.0007024 0.20% FabArray::mult() 22 0.0006503 0.0006503 0.0006503 0.18% FabArray::setDomainBndry() 20 0.0006444 0.0006444 0.0006444 0.18% MultiFab::contains_nan() 10 0.0005912 0.0005912 0.0005912 0.17% MLCellLinOp::prepareForSolve() 6 0.0005717 0.0005717 0.0005717 0.16% MLCellLinOp::compGrad() 6 0.000481 0.000481 0.000481 0.14% MLCellLinOp::smooth() 720 0.00047 0.00047 0.00047 0.13% MLMG::prepareForSolve() 6 0.0004476 0.0004476 0.0004476 0.13% FabArrayBase::CPC::define() 244 0.0004183 0.0004183 0.0004183 0.12% Amr::InitAmr() 1 0.0003888 0.0003888 0.0003888 0.11% FabArrayBase::getCPC() 632 0.0003403 0.0003403 0.0003403 0.10% FabArray::FillBoundary() 1766 0.0003391 0.0003391 0.0003391 0.10% Gravity::get_old_grav_vector() 5 0.0003343 0.0003343 0.0003343 0.09% Gravity::get_new_grav_vector() 5 0.0002665 0.0002665 0.0002665 0.08% main() 1 0.0002608 0.0002608 0.0002608 0.07% FabArrayBase::getFB() 1766 0.0002538 0.0002538 0.0002538 0.07% Amr::coarseTimeStep() 5 0.0002203 0.0002203 0.0002203 0.06% MLCellLinOp::apply() 500 0.0001902 0.0001902 0.0001902 0.05% MLMG::mgVcycle() 36 0.0001689 0.0001689 0.0001689 0.05% MultiFab::max() 6 0.0001333 0.0001333 0.0001333 0.04% MLCGSolver::ParallelAllReduce 659 0.0001291 0.0001291 0.0001291 0.04% Castro::construct_new_source() 25 0.0001258 0.0001258 0.0001258 0.04% Castro::subcycle_advance_ctu() 5 0.0001168 0.0001168 0.0001168 0.03% FillPatchIterator::Initialize 20 0.0001089 0.0001089 0.0001089 0.03% FabArray::ParallelCopy() 380 0.0001083 0.0001083 0.0001083 0.03% MLCellLinOp::defineBC() 6 0.0001048 0.0001048 0.0001048 0.03% MLCellLinOp::correctionResidual() 216 9.91e-05 9.91e-05 9.91e-05 0.03% MLLinOp::defineGrids() 6 8.997e-05 8.997e-05 8.997e-05 0.03% Castro::finalize_advance() 5 8.935e-05 8.935e-05 8.935e-05 0.03% Amr::timeStep() 5 8.026e-05 8.026e-05 8.026e-05 0.02% Gravity::solve_for_phi() 5 6.855e-05 6.855e-05 6.855e-05 0.02% AmrLevel::restart() 1 6.846e-05 6.846e-05 6.846e-05 0.02% Castro::post_timestep() 5 6.725e-05 6.725e-05 6.725e-05 0.02% FabArrayBase::FB::FB() 26 5.927e-05 5.927e-05 5.927e-05 0.02% Castro::initialize_do_advance() 5 5.749e-05 5.749e-05 5.749e-05 0.02% StateData::restartDoit() 4 5.694e-05 5.694e-05 5.694e-05 0.02% Gravity::update_max_rhs() 6 5.363e-05 5.363e-05 5.363e-05 0.02% Castro::construct_new_gravity() 5 4.792e-05 4.792e-05 4.792e-05 0.01% MLMG:computeResOfCorrection() 180 4.545e-05 4.545e-05 4.545e-05 0.01% Castro::clean_state() 30 4.336e-05 4.336e-05 4.336e-05 0.01% Castro::expand_state() 5 4.106e-05 4.106e-05 4.106e-05 0.01% Castro::create_source_corrector() 5 4.014e-05 4.014e-05 4.014e-05 0.01% MLMG::mgVcycle_down::0 36 3.929e-05 3.929e-05 3.929e-05 0.01% MLMG::actualBottomSolve() 36 3.546e-05 3.546e-05 3.546e-05 0.01% Castro::computeNewDt() 5 3.506e-05 3.506e-05 3.506e-05 0.01% MLMG::mgVcycle_down::1 36 3.457e-05 3.457e-05 3.457e-05 0.01% MLMG::solve() 6 3.24e-05 3.24e-05 3.24e-05 0.01% MLMG::mgVcycle_down::2 36 3.115e-05 3.115e-05 3.115e-05 0.01% MLMG::mgVcycle_down::4 36 3.075e-05 3.075e-05 3.075e-05 0.01% Castro::initialize_advance() 5 2.99e-05 2.99e-05 2.99e-05 0.01% MLMG::mgVcycle_down::3 36 2.94e-05 2.94e-05 2.94e-05 0.01% MLMG::mgVcycle_up::4 36 2.695e-05 2.695e-05 2.695e-05 0.01% Castro::finalize_do_advance() 5 2.565e-05 2.565e-05 2.565e-05 0.01% Castro::buildMetrics() 1 2.521e-05 2.521e-05 2.521e-05 0.01% Amr::writeSmallPlotFile() 1 2.443e-05 2.443e-05 2.443e-05 0.01% MLMG::mgVcycle_up::0 36 2.337e-05 2.337e-05 2.337e-05 0.01% MLMG::oneIter() 36 2.336e-05 2.336e-05 2.336e-05 0.01% MLCellLinOp::solutionResidual() 42 2.173e-05 2.173e-05 2.173e-05 0.01% Castro::post_restart() 1 2.162e-05 2.162e-05 2.162e-05 0.01% Castro::swap_state_time_levels() 5 2.155e-05 2.155e-05 2.155e-05 0.01% MLMG::mgVcycle_up::3 36 2.051e-05 2.051e-05 2.051e-05 0.01% MLMG::mgVcycle_up::2 36 2.013e-05 2.013e-05 2.013e-05 0.01% MLMG::mgVcycle_up::1 36 1.936e-05 1.936e-05 1.936e-05 0.01% Castro::initMFs() 1 1.923e-05 1.923e-05 1.923e-05 0.01% Gravity::solve_phi_with_mlmg() 6 1.79e-05 1.79e-05 1.79e-05 0.01% Castro::check_for_nan() 10 1.77e-05 1.77e-05 1.77e-05 0.01% MLMG::ResNormInf() 42 1.679e-05 1.679e-05 1.679e-05 0.00% MLMG::mgVcycle_bottom 36 1.48e-05 1.48e-05 1.48e-05 0.00% FillPatchSingleLevel 20 1.471e-05 1.471e-05 1.471e-05 0.00% MLPoisson::define() 6 1.469e-05 1.469e-05 1.469e-05 0.00% MLMG::computeResidual() 36 1.32e-05 1.32e-05 1.32e-05 0.00% makeSFC 30 1.318e-05 1.318e-05 1.318e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.29e-05 1.29e-05 1.29e-05 0.00% Castro::do_new_sources() 5 1.058e-05 1.058e-05 1.058e-05 0.00% Castro::construct_old_source() 25 1.024e-05 1.024e-05 1.024e-05 0.00% Castro::do_old_sources() 5 8.755e-06 8.755e-06 8.755e-06 0.00% DistributionMapping::Distribute() 31 8.484e-06 8.484e-06 8.484e-06 0.00% Amr::initSubcycle() 1 8.024e-06 8.024e-06 8.024e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.348e-06 7.348e-06 7.348e-06 0.00% Gravity::actual_multilevel_solve() 1 6.487e-06 6.487e-06 6.487e-06 0.00% Castro::apply_source_to_state() 10 5.8e-06 5.8e-06 5.8e-06 0.00% MLLinOp::define() 6 5.192e-06 5.192e-06 5.192e-06 0.00% Castro::construct_old_gravity() 5 4.816e-06 4.816e-06 4.816e-06 0.00% MLPoisson::prepareForSolve() 6 4.312e-06 4.312e-06 4.312e-06 0.00% Gravity::swapTimeLevels() 5 4.038e-06 4.038e-06 4.038e-06 0.00% MLMG::computeMLResidual() 6 3.349e-06 3.349e-06 3.349e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.115e-06 3.115e-06 3.115e-06 0.00% MLMG::getGradSolution() 6 3.027e-06 3.027e-06 3.027e-06 0.00% Gravity::set_mass_offset() 6 2.582e-06 2.582e-06 2.582e-06 0.00% Castro::retry_advance_ctu() 5 2.411e-06 2.411e-06 2.411e-06 0.00% MLMG::MLResNormInf() 6 2.167e-06 2.167e-06 2.167e-06 0.00% MLMG::MLRhsNormInf() 6 2.036e-06 2.036e-06 2.036e-06 0.00% Castro::FluxRegCrseInit 5 1.563e-06 1.563e-06 1.563e-06 0.00% Amr::init() 1 1.234e-06 1.234e-06 1.234e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.172e-06 1.172e-06 1.172e-06 0.00% Castro::FluxRegFineAdd() 5 1.131e-06 1.131e-06 1.131e-06 0.00% AmrLevel::AmrLevel() 1 7.42e-07 7.42e-07 7.42e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.354 0.354 0.354 100.00% Amr::coarseTimeStep() 5 0.2743 0.2743 0.2743 77.48% Amr::timeStep() 5 0.2728 0.2728 0.2728 77.06% Castro::advance() 5 0.2686 0.2686 0.2686 75.89% Castro::subcycle_advance_ctu() 5 0.2613 0.2613 0.2613 73.83% Castro::do_advance_ctu() 5 0.2612 0.2612 0.2612 73.79% Castro::construct_new_gravity() 5 0.1244 0.1244 0.1244 35.14% Gravity::solve_phi_with_mlmg() 6 0.1199 0.1199 0.1199 33.88% Gravity::solve_for_phi() 5 0.1169 0.1169 0.1169 33.03% Gravity::actual_solve_with_mlmg() 6 0.1169 0.1169 0.1169 33.01% MLMG::solve() 6 0.1062 0.1062 0.1062 30.01% Castro::construct_ctu_hydro_source() 5 0.1 0.1 0.1 28.25% MLMG::oneIter() 36 0.09961 0.09961 0.09961 28.14% MLMG::mgVcycle() 36 0.09808 0.09808 0.09808 27.71% MLCellLinOp::smooth() 720 0.04853 0.04853 0.04853 13.71% Amr::init() 1 0.04676 0.04676 0.04676 13.21% Amr::restart() 1 0.04675 0.04675 0.04675 13.21% MLCellLinOp::applyBC() 1946 0.04532 0.04532 0.04532 12.80% AmrLevel::restart() 1 0.04078 0.04078 0.04078 11.52% StateData::restartDoit() 4 0.04071 0.04071 0.04071 11.50% VisMF::Read() 3 0.0406 0.0406 0.0406 11.47% Amr::writePlotFile() 1 0.03226 0.03226 0.03226 9.11% VisMF::Write(FabArray) 1 0.0306 0.0306 0.0306 8.64% MLMG::mgVcycle_bottom 36 0.03012 0.03012 0.03012 8.51% MLMG::actualBottomSolve() 36 0.03011 0.03011 0.03011 8.51% MLCGSolver::bicgstab 36 0.02982 0.02982 0.02982 8.42% Castro::clean_state() 30 0.02124 0.02124 0.02124 6.00% FillPatchIterator::Initialize 20 0.01958 0.01958 0.01958 5.53% FillPatchSingleLevel 20 0.01883 0.01883 0.01883 5.32% StateDataPhysBCFunct::() 20 0.01679 0.01679 0.01679 4.74% MLCellLinOp::apply() 500 0.01508 0.01508 0.01508 4.26% MLMG::mgVcycle_down::0 36 0.01411 0.01411 0.01411 3.99% FabArray::FillBoundary() 1766 0.01373 0.01373 0.01373 3.88% MLPoisson::Fsmooth() 1440 0.01355 0.01355 0.01355 3.83% FillBoundary_nowait() 1766 0.01339 0.01339 0.01339 3.78% StateData::FillBoundary(geom) 160 0.0111 0.0111 0.0111 3.14% MLMG::mgVcycle_up::0 36 0.01067 0.01067 0.01067 3.01% MLCellLinOp::correctionResidual() 216 0.009208 0.009208 0.009208 2.60% Castro::initialize_do_advance() 5 0.009033 0.009033 0.009033 2.55% Castro::normalize_species() 30 0.008824 0.008824 0.008824 2.49% Castro::computeTemp() 30 0.008805 0.008805 0.008805 2.49% MLPoisson::define() 6 0.008483 0.008483 0.008483 2.40% amrex::Dot() 484 0.00833 0.00833 0.00833 2.35% MLMG:computeResOfCorrection() 180 0.008095 0.008095 0.008095 2.29% Gravity::get_new_grav_vector() 5 0.007307 0.007307 0.007307 2.06% Castro::construct_old_gravity() 5 0.007134 0.007134 0.007134 2.02% Gravity::get_old_grav_vector() 5 0.00713 0.00713 0.00713 2.01% amrex::Copy() 463 0.006819 0.006819 0.006819 1.93% Castro::do_new_sources() 5 0.006637 0.006637 0.006637 1.87% MLMG::mgVcycle_down::1 36 0.006515 0.006515 0.006515 1.84% FabArray::ParallelCopy() 380 0.006243 0.006243 0.006243 1.76% FabArray::ParallelCopy_nowait() 380 0.006135 0.006135 0.006135 1.73% Castro::initialize_advance() 5 0.006075 0.006075 0.006075 1.72% MLMG::mgVcycle_down::2 36 0.006061 0.006061 0.006061 1.71% FabArray::setVal() 537 0.006009 0.006009 0.006009 1.70% FabArray::norminf() 326 0.005934 0.005934 0.005934 1.68% MLMG::mgVcycle_down::3 36 0.005909 0.005909 0.005909 1.67% MLMG::mgVcycle_down::4 36 0.005883 0.005883 0.005883 1.66% MLCellLinOp::defineAuxData() 6 0.005824 0.005824 0.005824 1.65% Castro::expand_state() 5 0.005726 0.005726 0.005726 1.62% MLCGSolver::ParallelAllReduce 659 0.005036 0.005036 0.005036 1.42% Castro::do_old_sources() 5 0.004881 0.004881 0.004881 1.38% MLMG::addInterpCorrection() 180 0.004865 0.004865 0.004865 1.37% MLMG::mgVcycle_up::4 36 0.004772 0.004772 0.004772 1.35% MLMG::mgVcycle_up::1 36 0.004731 0.004731 0.004731 1.34% MLMG::mgVcycle_up::2 36 0.004621 0.004621 0.004621 1.31% amrex::average_down 180 0.004527 0.004527 0.004527 1.28% MLMG::mgVcycle_up::3 36 0.004516 0.004516 0.004516 1.28% MLPoisson::Fapply() 500 0.004332 0.004332 0.004332 1.22% Castro::post_timestep() 5 0.004083 0.004083 0.004083 1.15% FabArray::Saxpy() 355 0.003504 0.003504 0.003504 0.99% FabArray::Xpay() 361 0.003394 0.003394 0.003394 0.96% Castro::post_restart() 1 0.00334 0.00334 0.00334 0.94% Gravity::multilevel_solve_for_new_phi() 1 0.003225 0.003225 0.003225 0.91% Gravity::actual_multilevel_solve() 1 0.003212 0.003212 0.003212 0.91% MLCellLinOp::solutionResidual() 42 0.003141 0.003141 0.003141 0.89% Gravity::fill_multipole_BCs() 6 0.002952 0.002952 0.002952 0.83% Castro::reset_internal_energy(MultiFab) 30 0.00277 0.00277 0.00277 0.78% MLMG::computeResidual() 36 0.002607 0.002607 0.002607 0.74% MLCellLinOp::defineBC() 6 0.00252 0.00252 0.00252 0.71% BndryData::define() 6 0.002415 0.002415 0.002415 0.68% MLMG::prepareForSolve() 6 0.002381 0.002381 0.002381 0.67% Castro::enforce_min_density() 30 0.00236 0.00236 0.00236 0.67% Castro::estTimeStep() 10 0.002242 0.002242 0.002242 0.63% FabArray::LinComb() 242 0.001866 0.001866 0.001866 0.53% amrex::Add() 72 0.001823 0.001823 0.001823 0.51% Castro::construct_new_source() 25 0.001626 0.001626 0.001626 0.46% Castro::construct_new_gravity_source() 5 0.001501 0.001501 0.001501 0.42% Castro::computeNewDt() 5 0.001258 0.001258 0.001258 0.36% Castro::construct_old_source() 25 0.001216 0.001216 0.001216 0.34% Castro::enforce_speed_limit() 30 0.001207 0.001207 0.001207 0.34% Castro::construct_old_gravity_source() 5 0.001206 0.001206 0.001206 0.34% MLMG::ResNormInf() 42 0.0009257 0.0009257 0.0009257 0.26% Castro::apply_source_to_state() 10 0.0009174 0.0009174 0.0009174 0.26% Castro::reset_internal_energy(Fab) 240 0.0009014 0.0009014 0.0009014 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008667 0.0008667 0.0008667 0.24% FabArrayBase::getCPC() 632 0.0007586 0.0007586 0.0007586 0.21% MLMG::getGradSolution() 6 0.0007389 0.0007389 0.0007389 0.21% MLCellLinOp::compGrad() 6 0.0007359 0.0007359 0.0007359 0.21% MLCellLinOp::setLevelBC() 6 0.000705 0.000705 0.000705 0.20% FabArray::mult() 22 0.0006503 0.0006503 0.0006503 0.18% FabArray::setDomainBndry() 20 0.0006444 0.0006444 0.0006444 0.18% Castro::check_for_nan() 10 0.0006089 0.0006089 0.0006089 0.17% MultiFab::contains_nan() 10 0.0005912 0.0005912 0.0005912 0.17% MLPoisson::prepareForSolve() 6 0.000576 0.000576 0.000576 0.16% MLCellLinOp::prepareForSolve() 6 0.0005717 0.0005717 0.0005717 0.16% MLMG::computeMLResidual() 6 0.0005509 0.0005509 0.0005509 0.16% Gravity::update_max_rhs() 6 0.000445 0.000445 0.000445 0.13% FabArrayBase::CPC::define() 244 0.0004183 0.0004183 0.0004183 0.12% Amr::InitAmr() 1 0.0003968 0.0003968 0.0003968 0.11% Castro::finalize_advance() 5 0.0003466 0.0003466 0.0003466 0.10% FabArrayBase::getFB() 1766 0.000313 0.000313 0.000313 0.09% Gravity::swapTimeLevels() 5 0.0002191 0.0002191 0.0002191 0.06% MLMG::MLResNormInf() 6 0.000148 0.000148 0.000148 0.04% Castro::buildMetrics() 1 0.0001448 0.0001448 0.0001448 0.04% MultiFab::max() 6 0.0001333 0.0001333 0.0001333 0.04% MLLinOp::define() 6 0.0001241 0.0001241 0.0001241 0.04% MLLinOp::defineGrids() 6 0.0001189 0.0001189 0.0001189 0.03% MLMG::MLRhsNormInf() 6 0.0001137 0.0001137 0.0001137 0.03% FabArrayBase::FB::FB() 26 5.927e-05 5.927e-05 5.927e-05 0.02% Castro::create_source_corrector() 5 4.014e-05 4.014e-05 4.014e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.78e-05 2.78e-05 2.78e-05 0.01% Castro::finalize_do_advance() 5 2.565e-05 2.565e-05 2.565e-05 0.01% Amr::writeSmallPlotFile() 1 2.443e-05 2.443e-05 2.443e-05 0.01% Castro::swap_state_time_levels() 5 2.155e-05 2.155e-05 2.155e-05 0.01% makeSFC 30 2.045e-05 2.045e-05 2.045e-05 0.01% Castro::initMFs() 1 1.923e-05 1.923e-05 1.923e-05 0.01% DistributionMapping::Distribute() 31 8.484e-06 8.484e-06 8.484e-06 0.00% Amr::initSubcycle() 1 8.024e-06 8.024e-06 8.024e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.322e-06 4.322e-06 4.322e-06 0.00% Gravity::set_mass_offset() 6 2.582e-06 2.582e-06 2.582e-06 0.00% Castro::retry_advance_ctu() 5 2.411e-06 2.411e-06 2.411e-06 0.00% Castro::FluxRegCrseInit 5 1.563e-06 1.563e-06 1.563e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.172e-06 1.172e-06 1.172e-06 0.00% Castro::FluxRegFineAdd() 5 1.131e-06 1.131e-06 1.131e-06 0.00% AmrLevel::AmrLevel() 1 7.42e-07 7.42e-07 7.42e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-10-g2e0b2b463306) finalized