Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.06-44-g027f2ff77fed) initialized Starting run at 08:27:48 UTC on 2022-06-27. Successfully read inputs file ... Castro git describe: 22.06-12-g556652b03 AMReX git describe: 22.06-44-g027f2ff77 Microphysics git describe: 22.06-6-g5146750a reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.043536468 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025051167 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.047409674 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050074794 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.05000394 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.048770675 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.049023704 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.040377517 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.088723227 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.080555306 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.053570922 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.054829921 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.066706516 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.040330398 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024704417 seconds Ending run at 08:27:49 UTC on 2022-06-27. Run time = 0.813498252 Run time without initialization = 0.695692756 Average number of zones advanced per microsecond: 3.768 Average number of zones advanced per microsecond per rank: 3.768 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8135 ... 0.8135 ... 0.8135 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1974 0.1974 0.1974 24.26% VisMF::Write(FabArray) 11 0.1676 0.1676 0.1676 20.60% MLCellLinOp::applyBC() 4379 0.08034 0.08034 0.08034 9.88% MLPoisson::Fsmooth() 3240 0.06342 0.06342 0.06342 7.80% StateData::FillBoundary(geom) 328 0.0241 0.0241 0.0241 2.96% MLCGSolver::bicgstab 81 0.02402 0.02402 0.02402 2.95% MultiFab::Dot() 1100 0.02251 0.02251 0.02251 2.77% StateDataPhysBCFunct::() 41 0.02101 0.02101 0.02101 2.58% MultiFab::LinComb() 1566 0.01449 0.01449 0.01449 1.78% FabArray::setVal() 1135 0.01435 0.01435 0.01435 1.76% FillBoundary_nowait() 3974 0.01404 0.01404 0.01404 1.73% Castro::computeTemp() 63 0.01359 0.01359 0.01359 1.67% Castro::normalize_species() 62 0.01326 0.01326 0.01326 1.63% FabArray::ParallelCopy_nowait() 851 0.01308 0.01308 0.01308 1.61% MLPoisson::Fapply() 1128 0.01176 0.01176 0.01176 1.44% MLCellLinOp::defineAuxData() 11 0.01171 0.01171 0.01171 1.44% Castro::enforce_min_density() 62 0.009077 0.009077 0.009077 1.12% Gravity::fill_multipole_BCs() 11 0.00816 0.00816 0.00816 1.00% MLMG::addInterpCorrection() 405 0.007457 0.007457 0.007457 0.92% amrex::average_down 405 0.006839 0.006839 0.006839 0.84% MultiFab::Xpay() 578 0.006607 0.006607 0.006607 0.81% Castro::estTimeStep() 21 0.005247 0.005247 0.005247 0.64% Castro::do_advance_ctu() 10 0.004297 0.004297 0.004297 0.53% Amr::checkPoint() 3 0.004115 0.004115 0.004115 0.51% BndryData::define() 11 0.003985 0.003985 0.003985 0.49% Castro::reset_internal_energy(MultiFab) 63 0.003875 0.003875 0.003875 0.48% Castro::construct_new_gravity_source() 10 0.00279 0.00279 0.00279 0.34% Amr::writePlotFile() 2 0.002319 0.002319 0.002319 0.29% Gravity::get_new_grav_vector() 11 0.001934 0.001934 0.001934 0.24% MLMG::ResNormInf() 92 0.00192 0.00192 0.00192 0.24% Castro::construct_old_gravity_source() 10 0.001863 0.001863 0.001863 0.23% MultiFab::Saxpy() 20 0.001813 0.001813 0.001813 0.22% Gravity::get_old_grav_vector() 10 0.001753 0.001753 0.001753 0.22% Castro::expand_state() 10 0.001733 0.001733 0.001733 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001673 0.001673 0.001673 0.21% MLMG::oneIter() 81 0.001659 0.001659 0.001659 0.20% MLCellLinOp::setLevelBC() 11 0.001576 0.001576 0.001576 0.19% Castro::reset_internal_energy(Fab) 504 0.001568 0.001568 0.001568 0.19% Castro::subcycle_advance_ctu() 10 0.001557 0.001557 0.001557 0.19% Gravity::actual_solve_with_mlmg() 11 0.001364 0.001364 0.001364 0.17% FabArray::setDomainBndry() 41 0.00133 0.00133 0.00133 0.16% FabArray::mult() 43 0.001327 0.001327 0.001327 0.16% MLCellLinOp::prepareForSolve() 11 0.001192 0.001192 0.001192 0.15% Castro::initData() 1 0.001183 0.001183 0.001183 0.15% MultiFab::contains_nan() 20 0.001173 0.001173 0.001173 0.14% MLCellLinOp::smooth() 1620 0.00116 0.00116 0.00116 0.14% Castro::enforce_speed_limit() 62 0.001086 0.001086 0.001086 0.13% MLMG::prepareForSolve() 11 0.001066 0.001066 0.001066 0.13% MLCellLinOp::compGrad() 11 0.0009263 0.0009263 0.0009263 0.11% FabArray::FillBoundary() 3974 0.000801 0.000801 0.000801 0.10% FabArrayBase::getCPC() 1313 0.0007761 0.0007761 0.0007761 0.10% FabArrayBase::CPC::define() 454 0.0006802 0.0006802 0.0006802 0.08% FabArrayBase::getFB() 3974 0.0006614 0.0006614 0.0006614 0.08% Amr::InitAmr() 1 0.0004866 0.0004866 0.0004866 0.06% MLCellLinOp::apply() 1128 0.000476 0.000476 0.000476 0.06% Gravity::solve_for_phi() 10 0.0004615 0.0004615 0.0004615 0.06% CGSolver::sxay() 1566 0.0004287 0.0004287 0.0004287 0.05% Gravity::update_max_rhs() 11 0.0004281 0.0004281 0.0004281 0.05% MLCGSolver::ParallelAllReduce 1495 0.000325 0.000325 0.000325 0.04% FillPatchIterator::Initialize 41 0.000313 0.000313 0.000313 0.04% Amr::coarseTimeStep() 10 0.0003099 0.0003099 0.0003099 0.04% main() 1 0.0003058 0.0003058 0.0003058 0.04% FabArray::ParallelCopy() 851 0.0002843 0.0002843 0.0002843 0.03% MLCellLinOp::defineBC() 11 0.0002812 0.0002812 0.0002812 0.03% MultiFab::Copy() 11 0.0002562 0.0002562 0.0002562 0.03% MultiFab::max() 11 0.0002516 0.0002516 0.0002516 0.03% MLCellLinOp::correctionResidual() 486 0.0002263 0.0002263 0.0002263 0.03% Castro::construct_new_gravity() 10 0.0002048 0.0002048 0.0002048 0.03% MLMG::MLRhsNormInf() 11 0.0002035 0.0002035 0.0002035 0.03% MLMG::mgVcycle() 81 0.0001972 0.0001972 0.0001972 0.02% Amr::timeStep() 10 0.0001903 0.0001903 0.0001903 0.02% MLMG:computeResOfCorrection() 405 0.0001365 0.0001365 0.0001365 0.02% MLLinOp::defineGrids() 11 0.000133 0.000133 0.000133 0.02% StateData::checkPoint() 12 0.0001291 0.0001291 0.0001291 0.02% MLMG::actualBottomSolve() 81 0.0001035 0.0001035 0.0001035 0.01% MLMG::mgVcycle_down::0 81 9.025e-05 9.025e-05 9.025e-05 0.01% Castro::Castro() 1 9.012e-05 9.012e-05 9.012e-05 0.01% FabArrayBase::FB::FB() 56 8.926e-05 8.926e-05 8.926e-05 0.01% Castro::initialize_advance() 10 8.6e-05 8.6e-05 8.6e-05 0.01% MLMG::solve() 11 8.431e-05 8.431e-05 8.431e-05 0.01% MLMG::mgVcycle_down::1 81 8.12e-05 8.12e-05 8.12e-05 0.01% Castro::finalize_advance() 10 8.067e-05 8.067e-05 8.067e-05 0.01% MLMG::mgVcycle_down::2 81 7.906e-05 7.906e-05 7.906e-05 0.01% Castro::clean_state() 62 7.652e-05 7.652e-05 7.652e-05 0.01% Castro::advance() 10 7.311e-05 7.311e-05 7.311e-05 0.01% AmrLevel::checkPoint() 3 7.175e-05 7.175e-05 7.175e-05 0.01% MLMG::mgVcycle_down::3 81 7.17e-05 7.17e-05 7.17e-05 0.01% MLMG::mgVcycle_down::4 81 7.092e-05 7.092e-05 7.092e-05 0.01% Castro::initialize_do_advance() 10 6.693e-05 6.693e-05 6.693e-05 0.01% MLMG::mgVcycle_up::4 81 6.135e-05 6.135e-05 6.135e-05 0.01% MLCellLinOp::solutionResidual() 92 5.353e-05 5.353e-05 5.353e-05 0.01% MLMG::mgVcycle_up::0 81 5.243e-05 5.243e-05 5.243e-05 0.01% MLMG::mgVcycle_up::2 81 5.079e-05 5.079e-05 5.079e-05 0.01% MLMG::mgVcycle_up::1 81 5.077e-05 5.077e-05 5.077e-05 0.01% MLMG::mgVcycle_up::3 81 5.031e-05 5.031e-05 5.031e-05 0.01% Castro::swap_state_time_levels() 10 4.002e-05 4.002e-05 4.002e-05 0.00% StateData::define() 4 3.976e-05 3.976e-05 3.976e-05 0.00% MLMG::computeResidual() 81 3.615e-05 3.615e-05 3.615e-05 0.00% Castro::finalize_do_advance() 10 3.567e-05 3.567e-05 3.567e-05 0.00% Castro::enforce_consistent_e() 1 3.435e-05 3.435e-05 3.435e-05 0.00% MLMG::mgVcycle_bottom 81 3.378e-05 3.378e-05 3.378e-05 0.00% Gravity::actual_multilevel_solve() 1 3.1e-05 3.1e-05 3.1e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.037e-05 3.037e-05 3.037e-05 0.00% FillPatchSingleLevel 41 2.891e-05 2.891e-05 2.891e-05 0.00% Castro::initMFs() 1 2.799e-05 2.799e-05 2.799e-05 0.00% makeSFC 55 2.633e-05 2.633e-05 2.633e-05 0.00% Castro::buildMetrics() 1 2.564e-05 2.564e-05 2.564e-05 0.00% Amr::writeSmallPlotFile() 1 2.525e-05 2.525e-05 2.525e-05 0.00% Castro::create_source_corrector() 10 2.286e-05 2.286e-05 2.286e-05 0.00% MLPoisson::define() 11 2.218e-05 2.218e-05 2.218e-05 0.00% Amr::defBaseLevel() 1 2.199e-05 2.199e-05 2.199e-05 0.00% MLLinOp::define() 11 2.168e-05 2.168e-05 2.168e-05 0.00% Amr::FinalizeInit() 1 2.009e-05 2.009e-05 2.009e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.913e-05 1.913e-05 1.913e-05 0.00% Castro::construct_old_source() 50 1.86e-05 1.86e-05 1.86e-05 0.00% Castro::construct_new_source() 50 1.813e-05 1.813e-05 1.813e-05 0.00% Castro::do_new_sources() 10 1.678e-05 1.678e-05 1.678e-05 0.00% Castro::do_old_sources() 10 1.568e-05 1.568e-05 1.568e-05 0.00% DistributionMapping::Distribute() 56 1.405e-05 1.405e-05 1.405e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.281e-05 1.281e-05 1.281e-05 0.00% Castro::check_for_nan() 20 1.211e-05 1.211e-05 1.211e-05 0.00% Castro::apply_source_to_state() 20 1.136e-05 1.136e-05 1.136e-05 0.00% Castro::construct_old_gravity() 10 9.995e-06 9.995e-06 9.995e-06 0.00% Gravity::swapTimeLevels() 10 9.315e-06 9.315e-06 9.315e-06 0.00% Amr::initSubcycle() 1 9.016e-06 9.016e-06 9.016e-06 0.00% Castro::post_timestep() 10 7.815e-06 7.815e-06 7.815e-06 0.00% MLPoisson::prepareForSolve() 11 7.481e-06 7.481e-06 7.481e-06 0.00% AmrLevel::AmrLevel(dm) 1 7.479e-06 7.479e-06 7.479e-06 0.00% MLMG::computeMLResidual() 11 7.038e-06 7.038e-06 7.038e-06 0.00% Amr::InitializeInit() 1 6.3e-06 6.3e-06 6.3e-06 0.00% Castro::computeNewDt() 9 6.196e-06 6.196e-06 6.196e-06 0.00% MLMG::getGradSolution() 11 5.98e-06 5.98e-06 5.98e-06 0.00% MLMG::buildFineMask() 11 5.429e-06 5.429e-06 5.429e-06 0.00% MLMG::MLResNormInf() 11 4.681e-06 4.681e-06 4.681e-06 0.00% AmrLevel::checkPointPost() 3 4.528e-06 4.528e-06 4.528e-06 0.00% Gravity::set_mass_offset() 11 4.254e-06 4.254e-06 4.254e-06 0.00% Castro::retry_advance_ctu() 10 4.087e-06 4.087e-06 4.087e-06 0.00% Castro::post_init() 1 3.83e-06 3.83e-06 3.83e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.38e-06 3.38e-06 3.38e-06 0.00% Castro::computeInitialDt() 2 3.129e-06 3.129e-06 3.129e-06 0.00% Castro::FluxRegCrseInit 10 2.746e-06 2.746e-06 2.746e-06 0.00% Amr::init() 1 2.459e-06 2.459e-06 2.459e-06 0.00% Castro::FluxRegFineAdd() 10 2.06e-06 2.06e-06 2.06e-06 0.00% AmrLevel::checkPointPre() 3 1.77e-06 1.77e-06 1.77e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.751e-06 1.751e-06 1.751e-06 0.00% Amr::initialInit() 1 1.393e-06 1.393e-06 1.393e-06 0.00% Castro::post_regrid() 1 1.135e-06 1.135e-06 1.135e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8135 0.8135 0.8135 100.00% Amr::coarseTimeStep() 10 0.6708 0.6708 0.6708 82.45% Amr::timeStep() 10 0.5866 0.5866 0.5866 72.10% Castro::advance() 10 0.5805 0.5805 0.5805 71.35% Castro::subcycle_advance_ctu() 10 0.5693 0.5693 0.5693 69.98% Castro::do_advance_ctu() 10 0.5678 0.5678 0.5678 69.79% Gravity::solve_phi_with_mlmg() 11 0.3139 0.3139 0.3139 38.58% Gravity::actual_solve_with_mlmg() 11 0.3055 0.3055 0.3055 37.55% Castro::construct_new_gravity() 10 0.2881 0.2881 0.2881 35.42% MLMG::solve() 11 0.2826 0.2826 0.2826 34.74% Gravity::solve_for_phi() 10 0.2729 0.2729 0.2729 33.54% MLMG::oneIter() 81 0.268 0.268 0.268 32.94% MLMG::mgVcycle() 81 0.2663 0.2663 0.2663 32.74% Castro::construct_ctu_hydro_source() 10 0.1974 0.1974 0.1974 24.26% VisMF::Write(FabArray) 11 0.1676 0.1676 0.1676 20.60% MLCellLinOp::smooth() 1620 0.136 0.136 0.136 16.72% Amr::checkPoint() 3 0.1244 0.1244 0.1244 15.29% AmrLevel::checkPoint() 3 0.1202 0.1202 0.1202 14.78% StateData::checkPoint() 12 0.1202 0.1202 0.1202 14.77% Amr::init() 1 0.1172 0.1172 0.1172 14.41% MLCellLinOp::applyBC() 4379 0.09593 0.09593 0.09593 11.79% MLMG::mgVcycle_bottom 81 0.08245 0.08245 0.08245 10.13% MLMG::actualBottomSolve() 81 0.08241 0.08241 0.08241 10.13% MLCGSolver::bicgstab 81 0.08158 0.08158 0.08158 10.03% MLPoisson::Fsmooth() 3240 0.06342 0.06342 0.06342 7.80% FillPatchIterator::Initialize 41 0.05074 0.05074 0.05074 6.24% Amr::writePlotFile() 2 0.04987 0.04987 0.04987 6.13% FillPatchSingleLevel 41 0.0491 0.0491 0.0491 6.04% Amr::initialInit() 1 0.04849 0.04849 0.04849 5.96% StateDataPhysBCFunct::() 41 0.04511 0.04511 0.04511 5.54% Amr::FinalizeInit() 1 0.04456 0.04456 0.04456 5.48% Castro::post_init() 1 0.04323 0.04323 0.04323 5.31% Castro::clean_state() 62 0.0418 0.0418 0.0418 5.14% Gravity::multilevel_solve_for_new_phi() 1 0.0415 0.0415 0.0415 5.10% Gravity::actual_multilevel_solve() 1 0.04148 0.04148 0.04148 5.10% MLCellLinOp::apply() 1128 0.03625 0.03625 0.03625 4.46% MLMG::mgVcycle_down::0 81 0.03532 0.03532 0.03532 4.34% MLMG::mgVcycle_up::0 81 0.03025 0.03025 0.03025 3.72% Castro::construct_old_gravity() 10 0.02452 0.02452 0.02452 3.01% Gravity::get_old_grav_vector() 10 0.02451 0.02451 0.02451 3.01% StateData::FillBoundary(geom) 328 0.0241 0.0241 0.0241 2.96% MultiFab::Dot() 1100 0.02251 0.02251 0.02251 2.77% MLCellLinOp::correctionResidual() 486 0.02127 0.02127 0.02127 2.61% Castro::computeTemp() 63 0.01903 0.01903 0.01903 2.34% MLPoisson::define() 11 0.01852 0.01852 0.01852 2.28% MLMG:computeResOfCorrection() 405 0.01837 0.01837 0.01837 2.26% Castro::initialize_do_advance() 10 0.01791 0.01791 0.01791 2.20% MLMG::mgVcycle_down::1 81 0.01772 0.01772 0.01772 2.18% MLMG::mgVcycle_down::2 81 0.01727 0.01727 0.01727 2.12% Gravity::get_new_grav_vector() 11 0.0167 0.0167 0.0167 2.05% MLMG::mgVcycle_down::3 81 0.01639 0.01639 0.01639 2.01% FabArray::FillBoundary() 3974 0.01559 0.01559 0.01559 1.92% MLMG::mgVcycle_down::4 81 0.01558 0.01558 0.01558 1.92% CGSolver::sxay() 1566 0.01492 0.01492 0.01492 1.83% FillBoundary_nowait() 3974 0.01479 0.01479 0.01479 1.82% MultiFab::LinComb() 1566 0.01449 0.01449 0.01449 1.78% FabArray::setVal() 1135 0.01435 0.01435 0.01435 1.76% FabArray::ParallelCopy() 851 0.01419 0.01419 0.01419 1.74% FabArray::ParallelCopy_nowait() 851 0.0139 0.0139 0.0139 1.71% MLCGSolver::ParallelAllReduce 1495 0.01349 0.01349 0.01349 1.66% Castro::normalize_species() 62 0.01326 0.01326 0.01326 1.63% MLMG::mgVcycle_up::2 81 0.01325 0.01325 0.01325 1.63% MLCellLinOp::defineAuxData() 11 0.01306 0.01306 0.01306 1.60% MLMG::mgVcycle_up::1 81 0.01299 0.01299 0.01299 1.60% MLMG::mgVcycle_up::3 81 0.01253 0.01253 0.01253 1.54% MLMG::addInterpCorrection() 405 0.01252 0.01252 0.01252 1.54% MLMG::mgVcycle_up::4 81 0.0124 0.0124 0.0124 1.52% amrex::average_down 405 0.012 0.012 0.012 1.47% MLPoisson::Fapply() 1128 0.01176 0.01176 0.01176 1.44% Castro::do_new_sources() 10 0.0115 0.0115 0.0115 1.41% Castro::initialize_advance() 10 0.01098 0.01098 0.01098 1.35% Castro::expand_state() 10 0.01094 0.01094 0.01094 1.34% Castro::do_old_sources() 10 0.009365 0.009365 0.009365 1.15% Castro::enforce_min_density() 62 0.009077 0.009077 0.009077 1.12% Gravity::fill_multipole_BCs() 11 0.00816 0.00816 0.00816 1.00% MLCellLinOp::solutionResidual() 92 0.007136 0.007136 0.007136 0.88% MultiFab::Xpay() 578 0.006607 0.006607 0.006607 0.81% MLMG::computeResidual() 81 0.00613 0.00613 0.00613 0.75% Castro::post_timestep() 10 0.005946 0.005946 0.005946 0.73% Castro::reset_internal_energy(MultiFab) 63 0.005443 0.005443 0.005443 0.67% Castro::estTimeStep() 21 0.005247 0.005247 0.005247 0.64% MLCellLinOp::defineBC() 11 0.005231 0.005231 0.005231 0.64% MLMG::prepareForSolve() 11 0.005221 0.005221 0.005221 0.64% BndryData::define() 11 0.00495 0.00495 0.00495 0.61% Amr::InitializeInit() 1 0.003932 0.003932 0.003932 0.48% Amr::defBaseLevel() 1 0.003926 0.003926 0.003926 0.48% Castro::initData() 1 0.003401 0.003401 0.003401 0.42% Castro::construct_new_source() 50 0.002808 0.002808 0.002808 0.35% Castro::construct_new_gravity_source() 10 0.00279 0.00279 0.00279 0.34% Castro::computeNewDt() 9 0.002583 0.002583 0.002583 0.32% MLMG::ResNormInf() 92 0.00192 0.00192 0.00192 0.24% Castro::construct_old_source() 50 0.001881 0.001881 0.001881 0.23% Castro::construct_old_gravity_source() 10 0.001863 0.001863 0.001863 0.23% Castro::apply_source_to_state() 20 0.001825 0.001825 0.001825 0.22% MultiFab::Saxpy() 20 0.001813 0.001813 0.001813 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001673 0.001673 0.001673 0.21% MLCellLinOp::setLevelBC() 11 0.001576 0.001576 0.001576 0.19% Castro::reset_internal_energy(Fab) 504 0.001568 0.001568 0.001568 0.19% FabArrayBase::getCPC() 1313 0.001456 0.001456 0.001456 0.18% MLMG::getGradSolution() 11 0.001415 0.001415 0.001415 0.17% MLCellLinOp::compGrad() 11 0.001409 0.001409 0.001409 0.17% FabArray::setDomainBndry() 41 0.00133 0.00133 0.00133 0.16% FabArray::mult() 43 0.001327 0.001327 0.001327 0.16% MLPoisson::prepareForSolve() 11 0.0012 0.0012 0.0012 0.15% MLCellLinOp::prepareForSolve() 11 0.001192 0.001192 0.001192 0.15% Castro::check_for_nan() 20 0.001185 0.001185 0.001185 0.15% MultiFab::contains_nan() 20 0.001173 0.001173 0.001173 0.14% Castro::post_regrid() 1 0.001088 0.001088 0.001088 0.13% Castro::enforce_speed_limit() 62 0.001086 0.001086 0.001086 0.13% MLMG::computeMLResidual() 11 0.001048 0.001048 0.001048 0.13% Gravity::update_max_rhs() 11 0.0008294 0.0008294 0.0008294 0.10% FabArrayBase::getFB() 3974 0.0007506 0.0007506 0.0007506 0.09% Castro::computeInitialDt() 2 0.0006996 0.0006996 0.0006996 0.09% FabArrayBase::CPC::define() 454 0.0006802 0.0006802 0.0006802 0.08% Amr::InitAmr() 1 0.0004956 0.0004956 0.0004956 0.06% Castro::Castro() 1 0.0004506 0.0004506 0.0004506 0.06% Gravity::swapTimeLevels() 10 0.00044 0.00044 0.00044 0.05% MLMG::MLResNormInf() 11 0.0002563 0.0002563 0.0002563 0.03% MultiFab::Copy() 11 0.0002562 0.0002562 0.0002562 0.03% MultiFab::max() 11 0.0002516 0.0002516 0.0002516 0.03% MLLinOp::define() 11 0.0002084 0.0002084 0.0002084 0.03% MLMG::MLRhsNormInf() 11 0.0002035 0.0002035 0.0002035 0.03% MLLinOp::defineGrids() 11 0.0001867 0.0001867 0.0001867 0.02% Castro::buildMetrics() 1 0.0001661 0.0001661 0.0001661 0.02% FabArrayBase::FB::FB() 56 8.926e-05 8.926e-05 8.926e-05 0.01% Castro::finalize_advance() 10 8.547e-05 8.547e-05 8.547e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.196e-05 5.196e-05 5.196e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.724e-05 4.724e-05 4.724e-05 0.01% Castro::swap_state_time_levels() 10 4.002e-05 4.002e-05 4.002e-05 0.00% StateData::define() 4 3.976e-05 3.976e-05 3.976e-05 0.00% makeSFC 55 3.915e-05 3.915e-05 3.915e-05 0.00% Castro::finalize_do_advance() 10 3.567e-05 3.567e-05 3.567e-05 0.00% Castro::enforce_consistent_e() 1 3.435e-05 3.435e-05 3.435e-05 0.00% Castro::initMFs() 1 2.799e-05 2.799e-05 2.799e-05 0.00% Amr::writeSmallPlotFile() 1 2.525e-05 2.525e-05 2.525e-05 0.00% Castro::create_source_corrector() 10 2.286e-05 2.286e-05 2.286e-05 0.00% DistributionMapping::Distribute() 56 1.405e-05 1.405e-05 1.405e-05 0.00% Amr::initSubcycle() 1 9.016e-06 9.016e-06 9.016e-06 0.00% MLMG::buildFineMask() 11 5.429e-06 5.429e-06 5.429e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.613e-06 4.613e-06 4.613e-06 0.00% AmrLevel::checkPointPost() 3 4.528e-06 4.528e-06 4.528e-06 0.00% Gravity::set_mass_offset() 11 4.254e-06 4.254e-06 4.254e-06 0.00% Castro::retry_advance_ctu() 10 4.087e-06 4.087e-06 4.087e-06 0.00% Castro::FluxRegCrseInit 10 2.746e-06 2.746e-06 2.746e-06 0.00% Castro::FluxRegFineAdd() 10 2.06e-06 2.06e-06 2.06e-06 0.00% AmrLevel::checkPointPre() 3 1.77e-06 1.77e-06 1.77e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.751e-06 1.751e-06 1.751e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.06-44-g027f2ff77fed) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.06-44-g027f2ff77fed) initialized Starting run at 08:27:49 UTC on 2022-06-27. Successfully read inputs file ... Castro git describe: 22.06-12-g556652b03 AMReX git describe: 22.06-44-g027f2ff77 Microphysics git describe: 22.06-6-g5146750a reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.403480019 Restart time = 0.04692476 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.049931428 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049412049 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.062566741 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.065398291 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.071082706 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026510809 seconds Ending run at 08:27:50 UTC on 2022-06-27. Run time = 0.372782652 Run time without initialization = 0.325311533 Average number of zones advanced per microsecond: 4.029 Average number of zones advanced per microsecond per rank: 4.029 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3728 ... 0.3728 ... 0.3728 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0992 0.0992 0.0992 26.62% VisMF::Read() 3 0.03946 0.03946 0.03946 10.58% MLCellLinOp::applyBC() 1946 0.03535 0.03535 0.03535 9.48% MLPoisson::Fsmooth() 1440 0.02756 0.02756 0.02756 7.39% VisMF::Write(FabArray) 1 0.02511 0.02511 0.02511 6.74% StateData::FillBoundary(geom) 160 0.01187 0.01187 0.01187 3.18% MLCGSolver::bicgstab 36 0.01034 0.01034 0.01034 2.77% MultiFab::Dot() 484 0.009675 0.009675 0.009675 2.60% Castro::normalize_species() 30 0.00799 0.00799 0.00799 2.14% Castro::computeTemp() 30 0.007576 0.007576 0.007576 2.03% FabArray::setVal() 537 0.00683 0.00683 0.00683 1.83% MLCellLinOp::defineAuxData() 6 0.006528 0.006528 0.006528 1.75% Castro::enforce_min_density() 30 0.006349 0.006349 0.006349 1.70% FillBoundary_nowait() 1766 0.006226 0.006226 0.006226 1.67% MultiFab::LinComb() 690 0.006164 0.006164 0.006164 1.65% FabArray::ParallelCopy_nowait() 380 0.005983 0.005983 0.005983 1.60% MLPoisson::Fapply() 500 0.00508 0.00508 0.00508 1.36% Gravity::fill_multipole_BCs() 6 0.004914 0.004914 0.004914 1.32% StateDataPhysBCFunct::() 20 0.004712 0.004712 0.004712 1.26% MLMG::addInterpCorrection() 180 0.003229 0.003229 0.003229 0.87% Amr::restart() 1 0.003207 0.003207 0.003207 0.86% amrex::average_down 180 0.003001 0.003001 0.003001 0.81% MultiFab::Xpay() 258 0.002913 0.002913 0.002913 0.78% Castro::estTimeStep() 10 0.002551 0.002551 0.002551 0.68% Castro::do_advance_ctu() 5 0.002196 0.002196 0.002196 0.59% BndryData::define() 6 0.00218 0.00218 0.00218 0.58% Castro::reset_internal_energy(MultiFab) 30 0.001565 0.001565 0.001565 0.42% Castro::construct_new_gravity_source() 5 0.001556 0.001556 0.001556 0.42% Amr::writePlotFile() 1 0.001501 0.001501 0.001501 0.40% Castro::enforce_speed_limit() 30 0.001233 0.001233 0.001233 0.33% Castro::construct_old_gravity_source() 5 0.001148 0.001148 0.001148 0.31% Gravity::get_old_grav_vector() 5 0.00103 0.00103 0.00103 0.28% Castro::reset_internal_energy(Fab) 240 0.0009621 0.0009621 0.0009621 0.26% Gravity::get_new_grav_vector() 5 0.0009587 0.0009587 0.0009587 0.26% MultiFab::Saxpy() 10 0.0009247 0.0009247 0.0009247 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008998 0.0008998 0.0008998 0.24% Castro::expand_state() 5 0.0008809 0.0008809 0.0008809 0.24% MLMG::ResNormInf() 42 0.0008461 0.0008461 0.0008461 0.23% MLCellLinOp::setLevelBC() 6 0.0008415 0.0008415 0.0008415 0.23% Gravity::actual_solve_with_mlmg() 6 0.0007807 0.0007807 0.0007807 0.21% MLMG::oneIter() 36 0.0007374 0.0007374 0.0007374 0.20% FabArray::mult() 22 0.0006554 0.0006554 0.0006554 0.18% FabArray::setDomainBndry() 20 0.0006486 0.0006486 0.0006486 0.17% MLCellLinOp::prepareForSolve() 6 0.0006405 0.0006405 0.0006405 0.17% MultiFab::contains_nan() 10 0.0005945 0.0005945 0.0005945 0.16% MLMG::prepareForSolve() 6 0.0005667 0.0005667 0.0005667 0.15% MLCellLinOp::compGrad() 6 0.0004964 0.0004964 0.0004964 0.13% MLCellLinOp::smooth() 720 0.0004807 0.0004807 0.0004807 0.13% FabArrayBase::CPC::define() 244 0.0004131 0.0004131 0.0004131 0.11% Amr::InitAmr() 1 0.0003786 0.0003786 0.0003786 0.10% FabArray::FillBoundary() 1766 0.0003771 0.0003771 0.0003771 0.10% FabArrayBase::getCPC() 632 0.0003667 0.0003667 0.0003667 0.10% FabArrayBase::getFB() 1766 0.0002614 0.0002614 0.0002614 0.07% main() 1 0.0002505 0.0002505 0.0002505 0.07% Castro::subcycle_advance_ctu() 5 0.0002417 0.0002417 0.0002417 0.06% Gravity::update_max_rhs() 6 0.0002319 0.0002319 0.0002319 0.06% Gravity::solve_for_phi() 5 0.0002023 0.0002023 0.0002023 0.05% MLCellLinOp::apply() 500 0.0002022 0.0002022 0.0002022 0.05% CGSolver::sxay() 690 0.0001772 0.0001772 0.0001772 0.05% Castro::construct_new_gravity() 5 0.0001677 0.0001677 0.0001677 0.04% Castro::create_source_corrector() 5 0.000159 0.000159 0.000159 0.04% Amr::coarseTimeStep() 5 0.0001542 0.0001542 0.0001542 0.04% MLCellLinOp::defineBC() 6 0.0001505 0.0001505 0.0001505 0.04% Castro::construct_new_source() 25 0.0001464 0.0001464 0.0001464 0.04% FillPatchIterator::Initialize 20 0.0001433 0.0001433 0.0001433 0.04% Castro::advance() 5 0.0001424 0.0001424 0.0001424 0.04% MultiFab::Copy() 6 0.0001389 0.0001389 0.0001389 0.04% MultiFab::max() 6 0.0001364 0.0001364 0.0001364 0.04% MLCGSolver::ParallelAllReduce 659 0.000131 0.000131 0.000131 0.04% FabArray::ParallelCopy() 380 0.0001286 0.0001286 0.0001286 0.03% MLLinOp::defineGrids() 6 0.0001254 0.0001254 0.0001254 0.03% MLMG::MLRhsNormInf() 6 0.000107 0.000107 0.000107 0.03% MLCellLinOp::correctionResidual() 216 9.994e-05 9.994e-05 9.994e-05 0.03% Amr::timeStep() 5 9.805e-05 9.805e-05 9.805e-05 0.03% MLMG::mgVcycle() 36 8.435e-05 8.435e-05 8.435e-05 0.02% AmrLevel::restart() 1 7.635e-05 7.635e-05 7.635e-05 0.02% StateData::restartDoit() 4 7.008e-05 7.008e-05 7.008e-05 0.02% Castro::finalize_advance() 5 6.935e-05 6.935e-05 6.935e-05 0.02% Castro::computeNewDt() 5 6.618e-05 6.618e-05 6.618e-05 0.02% MLMG:computeResOfCorrection() 180 5.818e-05 5.818e-05 5.818e-05 0.02% FabArrayBase::FB::FB() 26 5.627e-05 5.627e-05 5.627e-05 0.02% Castro::initialize_do_advance() 5 4.782e-05 4.782e-05 4.782e-05 0.01% Castro::construct_old_source() 25 4.684e-05 4.684e-05 4.684e-05 0.01% MLMG::actualBottomSolve() 36 4.546e-05 4.546e-05 4.546e-05 0.01% Castro::clean_state() 30 4.12e-05 4.12e-05 4.12e-05 0.01% Castro::initialize_advance() 5 4.111e-05 4.111e-05 4.111e-05 0.01% MLMG::mgVcycle_down::0 36 3.812e-05 3.812e-05 3.812e-05 0.01% MLMG::solve() 6 3.761e-05 3.761e-05 3.761e-05 0.01% Castro::post_restart() 1 3.74e-05 3.74e-05 3.74e-05 0.01% MLMG::mgVcycle_down::1 36 3.562e-05 3.562e-05 3.562e-05 0.01% MLMG::mgVcycle_down::2 36 3.413e-05 3.413e-05 3.413e-05 0.01% MLMG::mgVcycle_down::4 36 3.192e-05 3.192e-05 3.192e-05 0.01% Castro::buildMetrics() 1 3.146e-05 3.146e-05 3.146e-05 0.01% Gravity::actual_multilevel_solve() 1 3.067e-05 3.067e-05 3.067e-05 0.01% MLMG::mgVcycle_down::3 36 3.017e-05 3.017e-05 3.017e-05 0.01% MLMG::mgVcycle_up::4 36 2.831e-05 2.831e-05 2.831e-05 0.01% Amr::writeSmallPlotFile() 1 2.809e-05 2.809e-05 2.809e-05 0.01% Castro::initMFs() 1 2.688e-05 2.688e-05 2.688e-05 0.01% Castro::swap_state_time_levels() 5 2.686e-05 2.686e-05 2.686e-05 0.01% MLCellLinOp::solutionResidual() 42 2.374e-05 2.374e-05 2.374e-05 0.01% MLMG::mgVcycle_up::0 36 2.3e-05 2.3e-05 2.3e-05 0.01% MLMG::mgVcycle_up::3 36 2.29e-05 2.29e-05 2.29e-05 0.01% MLMG::mgVcycle_up::2 36 2.238e-05 2.238e-05 2.238e-05 0.01% MLLinOp::define() 6 2.222e-05 2.222e-05 2.222e-05 0.01% MLMG::mgVcycle_up::1 36 2.045e-05 2.045e-05 2.045e-05 0.01% Castro::post_timestep() 5 2.027e-05 2.027e-05 2.027e-05 0.01% Castro::finalize_do_advance() 5 1.802e-05 1.802e-05 1.802e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.767e-05 1.767e-05 1.767e-05 0.00% makeSFC 30 1.748e-05 1.748e-05 1.748e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.559e-05 1.559e-05 1.559e-05 0.00% MLPoisson::define() 6 1.516e-05 1.516e-05 1.516e-05 0.00% MLMG::computeResidual() 36 1.51e-05 1.51e-05 1.51e-05 0.00% MLMG::mgVcycle_bottom 36 1.485e-05 1.485e-05 1.485e-05 0.00% FillPatchSingleLevel 20 1.357e-05 1.357e-05 1.357e-05 0.00% DistributionMapping::Distribute() 31 1.105e-05 1.105e-05 1.105e-05 0.00% Castro::do_new_sources() 5 1.013e-05 1.013e-05 1.013e-05 0.00% MLLinOp::makeAgglomeratedDMap 6 9.379e-06 9.379e-06 9.379e-06 0.00% Castro::do_old_sources() 5 9.068e-06 9.068e-06 9.068e-06 0.00% Amr::initSubcycle() 1 8.317e-06 8.317e-06 8.317e-06 0.00% Castro::apply_source_to_state() 10 6.031e-06 6.031e-06 6.031e-06 0.00% Castro::check_for_nan() 10 6.019e-06 6.019e-06 6.019e-06 0.00% Castro::construct_old_gravity() 5 5.77e-06 5.77e-06 5.77e-06 0.00% MLPoisson::prepareForSolve() 6 4.525e-06 4.525e-06 4.525e-06 0.00% Gravity::swapTimeLevels() 5 4.271e-06 4.271e-06 4.271e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.376e-06 3.376e-06 3.376e-06 0.00% MLMG::computeMLResidual() 6 3.006e-06 3.006e-06 3.006e-06 0.00% MLMG::getGradSolution() 6 2.923e-06 2.923e-06 2.923e-06 0.00% MLMG::buildFineMask() 6 2.867e-06 2.867e-06 2.867e-06 0.00% MLMG::MLResNormInf() 6 2.365e-06 2.365e-06 2.365e-06 0.00% Gravity::set_mass_offset() 6 2.198e-06 2.198e-06 2.198e-06 0.00% Castro::retry_advance_ctu() 5 2.001e-06 2.001e-06 2.001e-06 0.00% Castro::FluxRegCrseInit 5 1.872e-06 1.872e-06 1.872e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.424e-06 1.424e-06 1.424e-06 0.00% Castro::FluxRegFineAdd() 5 1.127e-06 1.127e-06 1.127e-06 0.00% AmrLevel::AmrLevel() 1 1.047e-06 1.047e-06 1.047e-06 0.00% Amr::init() 1 8.73e-07 8.73e-07 8.73e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3728 0.3728 0.3728 100.00% Amr::coarseTimeStep() 5 0.2985 0.2985 0.2985 80.08% Amr::timeStep() 5 0.2973 0.2973 0.2973 79.75% Castro::advance() 5 0.2933 0.2933 0.2933 78.67% Castro::subcycle_advance_ctu() 5 0.2866 0.2866 0.2866 76.87% Castro::do_advance_ctu() 5 0.2864 0.2864 0.2864 76.81% Castro::construct_new_gravity() 5 0.1457 0.1457 0.1457 39.07% Gravity::solve_phi_with_mlmg() 6 0.1414 0.1414 0.1414 37.93% Gravity::solve_for_phi() 5 0.1379 0.1379 0.1379 36.98% Gravity::actual_solve_with_mlmg() 6 0.1364 0.1364 0.1364 36.57% MLMG::solve() 6 0.1236 0.1236 0.1236 33.15% MLMG::oneIter() 36 0.1166 0.1166 0.1166 31.27% MLMG::mgVcycle() 36 0.1158 0.1158 0.1158 31.07% Castro::construct_ctu_hydro_source() 5 0.09923 0.09923 0.09923 26.62% MLCellLinOp::smooth() 720 0.05939 0.05939 0.05939 15.93% Amr::init() 1 0.04698 0.04698 0.04698 12.60% Amr::restart() 1 0.04698 0.04698 0.04698 12.60% MLCellLinOp::applyBC() 1946 0.04227 0.04227 0.04227 11.34% AmrLevel::restart() 1 0.03967 0.03967 0.03967 10.64% StateData::restartDoit() 4 0.03959 0.03959 0.03959 10.62% VisMF::Read() 3 0.03946 0.03946 0.03946 10.58% MLMG::mgVcycle_bottom 36 0.03547 0.03547 0.03547 9.51% MLMG::actualBottomSolve() 36 0.03546 0.03546 0.03546 9.51% MLCGSolver::bicgstab 36 0.03509 0.03509 0.03509 9.41% MLPoisson::Fsmooth() 1440 0.02756 0.02756 0.02756 7.39% Amr::writePlotFile() 1 0.02661 0.02661 0.02661 7.14% Castro::clean_state() 30 0.02572 0.02572 0.02572 6.90% VisMF::Write(FabArray) 1 0.02511 0.02511 0.02511 6.74% FillPatchIterator::Initialize 20 0.01937 0.01937 0.01937 5.19% FillPatchSingleLevel 20 0.01857 0.01857 0.01857 4.98% StateDataPhysBCFunct::() 20 0.01658 0.01658 0.01658 4.45% MLCellLinOp::apply() 500 0.01593 0.01593 0.01593 4.27% MLMG::mgVcycle_down::0 36 0.01553 0.01553 0.01553 4.17% MLMG::mgVcycle_up::0 36 0.01331 0.01331 0.01331 3.57% StateData::FillBoundary(geom) 160 0.01187 0.01187 0.01187 3.18% MLPoisson::define() 6 0.01036 0.01036 0.01036 2.78% Castro::computeTemp() 30 0.0101 0.0101 0.0101 2.71% Castro::initialize_do_advance() 5 0.01004 0.01004 0.01004 2.69% MultiFab::Dot() 484 0.009675 0.009675 0.009675 2.60% MLCellLinOp::correctionResidual() 216 0.009354 0.009354 0.009354 2.51% MLMG:computeResOfCorrection() 180 0.00806 0.00806 0.00806 2.16% Castro::normalize_species() 30 0.00799 0.00799 0.00799 2.14% MLMG::mgVcycle_down::1 36 0.007744 0.007744 0.007744 2.08% Gravity::get_new_grav_vector() 5 0.007637 0.007637 0.007637 2.05% MLMG::mgVcycle_down::2 36 0.007549 0.007549 0.007549 2.02% Castro::construct_old_gravity() 5 0.007535 0.007535 0.007535 2.02% Gravity::get_old_grav_vector() 5 0.00753 0.00753 0.00753 2.02% MLCellLinOp::defineAuxData() 6 0.007284 0.007284 0.007284 1.95% MLMG::mgVcycle_down::3 36 0.007102 0.007102 0.007102 1.91% FabArray::FillBoundary() 1766 0.006921 0.006921 0.006921 1.86% FabArray::setVal() 537 0.00683 0.00683 0.00683 1.83% MLMG::mgVcycle_down::4 36 0.006811 0.006811 0.006811 1.83% Castro::do_new_sources() 5 0.006811 0.006811 0.006811 1.83% FillBoundary_nowait() 1766 0.006544 0.006544 0.006544 1.76% Castro::initialize_advance() 5 0.006499 0.006499 0.006499 1.74% FabArray::ParallelCopy() 380 0.006489 0.006489 0.006489 1.74% FabArray::ParallelCopy_nowait() 380 0.00636 0.00636 0.00636 1.71% Castro::enforce_min_density() 30 0.006349 0.006349 0.006349 1.70% CGSolver::sxay() 690 0.006341 0.006341 0.006341 1.70% MultiFab::LinComb() 690 0.006164 0.006164 0.006164 1.65% MLCGSolver::ParallelAllReduce 659 0.005808 0.005808 0.005808 1.56% MLMG::mgVcycle_up::2 36 0.005763 0.005763 0.005763 1.55% MLMG::mgVcycle_up::1 36 0.005656 0.005656 0.005656 1.52% Castro::do_old_sources() 5 0.00564 0.00564 0.00564 1.51% MLMG::addInterpCorrection() 180 0.005475 0.005475 0.005475 1.47% MLMG::mgVcycle_up::3 36 0.005432 0.005432 0.005432 1.46% MLMG::mgVcycle_up::4 36 0.005376 0.005376 0.005376 1.44% amrex::average_down 180 0.005262 0.005262 0.005262 1.41% MLPoisson::Fapply() 500 0.00508 0.00508 0.00508 1.36% Castro::expand_state() 5 0.00506 0.00506 0.00506 1.36% Gravity::fill_multipole_BCs() 6 0.004914 0.004914 0.004914 1.32% Castro::post_restart() 1 0.003928 0.003928 0.003928 1.05% Castro::post_timestep() 5 0.00392 0.00392 0.00392 1.05% Gravity::multilevel_solve_for_new_phi() 1 0.003798 0.003798 0.003798 1.02% Gravity::actual_multilevel_solve() 1 0.00378 0.00378 0.00378 1.01% MLCellLinOp::solutionResidual() 42 0.003232 0.003232 0.003232 0.87% MultiFab::Xpay() 258 0.002913 0.002913 0.002913 0.78% MLCellLinOp::defineBC() 6 0.002877 0.002877 0.002877 0.77% MLMG::prepareForSolve() 6 0.002786 0.002786 0.002786 0.75% BndryData::define() 6 0.002726 0.002726 0.002726 0.73% MLMG::computeResidual() 36 0.002687 0.002687 0.002687 0.72% Castro::estTimeStep() 10 0.002551 0.002551 0.002551 0.68% Castro::reset_internal_energy(MultiFab) 30 0.002527 0.002527 0.002527 0.68% Castro::construct_new_source() 25 0.001702 0.001702 0.001702 0.46% Castro::construct_new_gravity_source() 5 0.001556 0.001556 0.001556 0.42% Castro::enforce_speed_limit() 30 0.001233 0.001233 0.001233 0.33% Castro::construct_old_source() 25 0.001194 0.001194 0.001194 0.32% Castro::construct_old_gravity_source() 5 0.001148 0.001148 0.001148 0.31% Castro::computeNewDt() 5 0.001056 0.001056 0.001056 0.28% Castro::reset_internal_energy(Fab) 240 0.0009621 0.0009621 0.0009621 0.26% Castro::apply_source_to_state() 10 0.0009307 0.0009307 0.0009307 0.25% MultiFab::Saxpy() 10 0.0009247 0.0009247 0.0009247 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008998 0.0008998 0.0008998 0.24% MLMG::ResNormInf() 42 0.0008461 0.0008461 0.0008461 0.23% MLCellLinOp::setLevelBC() 6 0.0008415 0.0008415 0.0008415 0.23% FabArrayBase::getCPC() 632 0.0007798 0.0007798 0.0007798 0.21% MLMG::getGradSolution() 6 0.0007721 0.0007721 0.0007721 0.21% MLCellLinOp::compGrad() 6 0.0007692 0.0007692 0.0007692 0.21% FabArray::mult() 22 0.0006554 0.0006554 0.0006554 0.18% FabArray::setDomainBndry() 20 0.0006486 0.0006486 0.0006486 0.17% MLPoisson::prepareForSolve() 6 0.0006451 0.0006451 0.0006451 0.17% MLCellLinOp::prepareForSolve() 6 0.0006405 0.0006405 0.0006405 0.17% Castro::check_for_nan() 10 0.0006005 0.0006005 0.0006005 0.16% MultiFab::contains_nan() 10 0.0005945 0.0005945 0.0005945 0.16% MLMG::computeMLResidual() 6 0.0005625 0.0005625 0.0005625 0.15% Gravity::update_max_rhs() 6 0.0004471 0.0004471 0.0004471 0.12% FabArrayBase::CPC::define() 244 0.0004131 0.0004131 0.0004131 0.11% Amr::InitAmr() 1 0.000387 0.000387 0.000387 0.10% FabArrayBase::getFB() 1766 0.0003176 0.0003176 0.0003176 0.09% Gravity::swapTimeLevels() 5 0.0002503 0.0002503 0.0002503 0.07% MLLinOp::define() 6 0.0001853 0.0001853 0.0001853 0.05% MLLinOp::defineGrids() 6 0.0001631 0.0001631 0.0001631 0.04% Castro::create_source_corrector() 5 0.000159 0.000159 0.000159 0.04% Castro::buildMetrics() 1 0.0001521 0.0001521 0.0001521 0.04% MultiFab::Copy() 6 0.0001389 0.0001389 0.0001389 0.04% MultiFab::max() 6 0.0001364 0.0001364 0.0001364 0.04% MLMG::MLResNormInf() 6 0.0001352 0.0001352 0.0001352 0.04% MLMG::MLRhsNormInf() 6 0.000107 0.000107 0.000107 0.03% Castro::finalize_advance() 5 7.235e-05 7.235e-05 7.235e-05 0.02% FabArrayBase::FB::FB() 26 5.627e-05 5.627e-05 5.627e-05 0.02% MLLinOp::makeAgglomeratedDMap 6 3.625e-05 3.625e-05 3.625e-05 0.01% Amr::writeSmallPlotFile() 1 2.809e-05 2.809e-05 2.809e-05 0.01% Castro::initMFs() 1 2.688e-05 2.688e-05 2.688e-05 0.01% makeSFC 30 2.687e-05 2.687e-05 2.687e-05 0.01% Castro::swap_state_time_levels() 5 2.686e-05 2.686e-05 2.686e-05 0.01% Castro::finalize_do_advance() 5 1.802e-05 1.802e-05 1.802e-05 0.00% DistributionMapping::Distribute() 31 1.105e-05 1.105e-05 1.105e-05 0.00% Amr::initSubcycle() 1 8.317e-06 8.317e-06 8.317e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.029e-06 5.029e-06 5.029e-06 0.00% MLMG::buildFineMask() 6 2.867e-06 2.867e-06 2.867e-06 0.00% Gravity::set_mass_offset() 6 2.198e-06 2.198e-06 2.198e-06 0.00% Castro::retry_advance_ctu() 5 2.001e-06 2.001e-06 2.001e-06 0.00% Castro::FluxRegCrseInit 5 1.872e-06 1.872e-06 1.872e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.424e-06 1.424e-06 1.424e-06 0.00% Castro::FluxRegFineAdd() 5 1.127e-06 1.127e-06 1.127e-06 0.00% AmrLevel::AmrLevel() 1 1.047e-06 1.047e-06 1.047e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.06-44-g027f2ff77fed) finalized