Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09) initialized Starting run at 08:30:01 UTC on 2022-09-02. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09 Microphysics git describe: 22.08-7-g8d22e048 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.052016046 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.029728025 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048667351 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050772432 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.05880508 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.060044244 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.058658402 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.047748777 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.063531871 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.064845249 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.06300406 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.056931721 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.071038572 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.049832164 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029572616 seconds Ending run at 08:30:01 UTC on 2022-09-02. Run time = 0.856926319 Run time without initialization = 0.724102314 Average number of zones advanced per microsecond: 3.620 Average number of zones advanced per microsecond per rank: 3.620 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.857 ... 0.857 ... 0.857 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2147 0.2147 0.2147 25.06% VisMF::Write(FabArray) 11 0.2009 0.2009 0.2009 23.44% MLCellLinOp::applyBC() 4433 0.078 0.078 0.078 9.10% MLPoisson::Fsmooth() 3280 0.06216 0.06216 0.06216 7.25% StateData::FillBoundary(geom) 328 0.0243 0.0243 0.0243 2.84% MLCGSolver::bicgstab 82 0.02333 0.02333 0.02333 2.72% MultiFab::Dot() 1114 0.02158 0.02158 0.02158 2.52% Castro::computeTemp() 63 0.01412 0.01412 0.01412 1.65% FillBoundary_nowait() 4023 0.01411 0.01411 0.01411 1.65% MultiFab::LinComb() 1586 0.01395 0.01395 0.01395 1.63% FabArray::setVal() 1144 0.01388 0.01388 0.01388 1.62% Castro::normalize_species() 62 0.01385 0.01385 0.01385 1.62% FabArray::ParallelCopy_nowait() 861 0.01279 0.01279 0.01279 1.49% StateDataPhysBCFunct::() 41 0.01238 0.01238 0.01238 1.44% Castro::enforce_min_density() 62 0.01178 0.01178 0.01178 1.37% MLPoisson::Fapply() 1142 0.01144 0.01144 0.01144 1.33% MLCellLinOp::defineAuxData() 11 0.01139 0.01139 0.01139 1.33% Gravity::fill_multipole_BCs() 11 0.009798 0.009798 0.009798 1.14% MLMG::addInterpCorrection() 410 0.007538 0.007538 0.007538 0.88% amrex::average_down 410 0.006751 0.006751 0.006751 0.79% MultiFab::Xpay() 585 0.006409 0.006409 0.006409 0.75% Castro::estTimeStep() 21 0.006128 0.006128 0.006128 0.72% Amr::checkPoint() 3 0.005239 0.005239 0.005239 0.61% Castro::do_advance_ctu() 10 0.005003 0.005003 0.005003 0.58% Castro::reset_internal_energy(MultiFab) 63 0.004069 0.004069 0.004069 0.47% BndryData::define() 11 0.0037 0.0037 0.0037 0.43% Castro::construct_new_gravity_source() 10 0.003288 0.003288 0.003288 0.38% Castro::construct_old_gravity_source() 10 0.002898 0.002898 0.002898 0.34% Amr::writePlotFile() 2 0.002824 0.002824 0.002824 0.33% MLMG::ResNormInf() 93 0.002084 0.002084 0.002084 0.24% Gravity::get_new_grav_vector() 11 0.001902 0.001902 0.001902 0.22% MultiFab::Saxpy() 20 0.001815 0.001815 0.001815 0.21% Castro::expand_state() 10 0.001727 0.001727 0.001727 0.20% Gravity::get_old_grav_vector() 10 0.001721 0.001721 0.001721 0.20% MultiFab::Add() 82 0.001656 0.001656 0.001656 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001584 0.001584 0.001584 0.18% MLCellLinOp::setLevelBC() 11 0.001484 0.001484 0.001484 0.17% Gravity::actual_solve_with_mlmg() 11 0.001453 0.001453 0.001453 0.17% Castro::reset_internal_energy(Fab) 504 0.001417 0.001417 0.001417 0.17% FabArray::mult() 43 0.001304 0.001304 0.001304 0.15% FabArray::setDomainBndry() 41 0.001282 0.001282 0.001282 0.15% MLMG::prepareForSolve() 11 0.001189 0.001189 0.001189 0.14% MultiFab::contains_nan() 20 0.001184 0.001184 0.001184 0.14% MLCellLinOp::prepareForSolve() 11 0.001131 0.001131 0.001131 0.13% Castro::initData() 1 0.00109 0.00109 0.00109 0.13% MLCellLinOp::smooth() 1640 0.001089 0.001089 0.001089 0.13% Castro::enforce_speed_limit() 62 0.0009841 0.0009841 0.0009841 0.11% MLCellLinOp::compGrad() 11 0.0009067 0.0009067 0.0009067 0.11% FabArray::FillBoundary() 4023 0.00081 0.00081 0.00081 0.09% FabArrayBase::getCPC() 1323 0.0007599 0.0007599 0.0007599 0.09% FabArrayBase::CPC::define() 454 0.0006882 0.0006882 0.0006882 0.08% FabArrayBase::getFB() 4023 0.0005928 0.0005928 0.0005928 0.07% Amr::InitAmr() 1 0.0004639 0.0004639 0.0004639 0.05% MLCellLinOp::apply() 1142 0.0004613 0.0004613 0.0004613 0.05% Gravity::solve_for_phi() 10 0.0004326 0.0004326 0.0004326 0.05% Gravity::update_max_rhs() 11 0.0004069 0.0004069 0.0004069 0.05% CGSolver::sxay() 1586 0.0003622 0.0003622 0.0003622 0.04% Amr::coarseTimeStep() 10 0.0003522 0.0003522 0.0003522 0.04% MultiFab::Copy() 11 0.000316 0.000316 0.000316 0.04% FillPatchIterator::Initialize 41 0.0002932 0.0002932 0.0002932 0.03% MLCGSolver::ParallelAllReduce 1514 0.0002855 0.0002855 0.0002855 0.03% MLCellLinOp::defineBC() 11 0.0002785 0.0002785 0.0002785 0.03% main() 1 0.0002706 0.0002706 0.0002706 0.03% Castro::construct_new_source() 50 0.0002554 0.0002554 0.0002554 0.03% FabArray::ParallelCopy() 861 0.0002553 0.0002553 0.0002553 0.03% MultiFab::max() 11 0.0002538 0.0002538 0.0002538 0.03% MLCellLinOp::correctionResidual() 492 0.0002241 0.0002241 0.0002241 0.03% Amr::timeStep() 10 0.0002158 0.0002158 0.0002158 0.03% Castro::construct_new_gravity() 10 0.0002125 0.0002125 0.0002125 0.02% MLMG::MLRhsNormInf() 11 0.0002108 0.0002108 0.0002108 0.02% MLMG::mgVcycle() 82 0.0002025 0.0002025 0.0002025 0.02% MLLinOp::defineGrids() 11 0.0001832 0.0001832 0.0001832 0.02% MLMG:computeResOfCorrection() 410 0.0001494 0.0001494 0.0001494 0.02% Castro::subcycle_advance_ctu() 10 0.0001466 0.0001466 0.0001466 0.02% StateData::checkPoint() 12 0.0001364 0.0001364 0.0001364 0.02% MLMG::mgVcycle_down::0 82 0.000116 0.000116 0.000116 0.01% MLMG::mgVcycle_down::1 82 9.491e-05 9.491e-05 9.491e-05 0.01% MLMG::mgVcycle_down::2 82 8.901e-05 8.901e-05 8.901e-05 0.01% MLMG::mgVcycle_down::3 82 8.836e-05 8.836e-05 8.836e-05 0.01% FabArrayBase::FB::FB() 56 8.77e-05 8.77e-05 8.77e-05 0.01% Castro::Castro() 1 8.768e-05 8.768e-05 8.768e-05 0.01% MLMG::actualBottomSolve() 82 8.324e-05 8.324e-05 8.324e-05 0.01% Castro::initialize_advance() 10 8.306e-05 8.306e-05 8.306e-05 0.01% MLMG::mgVcycle_down::4 82 8.195e-05 8.195e-05 8.195e-05 0.01% MLMG::solve() 11 7.812e-05 7.812e-05 7.812e-05 0.01% Castro::clean_state() 62 7.797e-05 7.797e-05 7.797e-05 0.01% AmrLevel::checkPoint() 3 7.291e-05 7.291e-05 7.291e-05 0.01% MLMG::mgVcycle_up::4 82 6.906e-05 6.906e-05 6.906e-05 0.01% Castro::initialize_do_advance() 10 6.43e-05 6.43e-05 6.43e-05 0.01% MLMG::mgVcycle_up::3 82 6.08e-05 6.08e-05 6.08e-05 0.01% MLMG::mgVcycle_up::0 82 5.905e-05 5.905e-05 5.905e-05 0.01% MLMG::oneIter() 82 5.897e-05 5.897e-05 5.897e-05 0.01% MLMG::mgVcycle_up::2 82 5.676e-05 5.676e-05 5.676e-05 0.01% MLMG::mgVcycle_up::1 82 5.653e-05 5.653e-05 5.653e-05 0.01% Castro::advance() 10 5.501e-05 5.501e-05 5.501e-05 0.01% Castro::finalize_advance() 10 5.28e-05 5.28e-05 5.28e-05 0.01% MLCellLinOp::solutionResidual() 93 5.058e-05 5.058e-05 5.058e-05 0.01% StateData::define() 4 4.225e-05 4.225e-05 4.225e-05 0.00% Castro::swap_state_time_levels() 10 4.002e-05 4.002e-05 4.002e-05 0.00% MLMG::computeResidual() 82 3.667e-05 3.667e-05 3.667e-05 0.00% Castro::finalize_do_advance() 10 3.61e-05 3.61e-05 3.61e-05 0.00% Castro::enforce_consistent_e() 1 3.38e-05 3.38e-05 3.38e-05 0.00% MLMG::mgVcycle_bottom 82 3.318e-05 3.318e-05 3.318e-05 0.00% Gravity::actual_multilevel_solve() 1 3.184e-05 3.184e-05 3.184e-05 0.00% MLPoisson::define() 11 2.843e-05 2.843e-05 2.843e-05 0.00% FillPatchSingleLevel 41 2.71e-05 2.71e-05 2.71e-05 0.00% makeSFC 55 2.666e-05 2.666e-05 2.666e-05 0.00% Castro::initMFs() 1 2.642e-05 2.642e-05 2.642e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.641e-05 2.641e-05 2.641e-05 0.00% Amr::defBaseLevel() 1 2.598e-05 2.598e-05 2.598e-05 0.00% Amr::writeSmallPlotFile() 1 2.586e-05 2.586e-05 2.586e-05 0.00% MLLinOp::define() 11 2.503e-05 2.503e-05 2.503e-05 0.00% Amr::FinalizeInit() 1 2.36e-05 2.36e-05 2.36e-05 0.00% Castro::buildMetrics() 1 2.319e-05 2.319e-05 2.319e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.908e-05 1.908e-05 1.908e-05 0.00% Castro::do_new_sources() 10 1.759e-05 1.759e-05 1.759e-05 0.00% Castro::construct_old_source() 50 1.639e-05 1.639e-05 1.639e-05 0.00% Castro::do_old_sources() 10 1.557e-05 1.557e-05 1.557e-05 0.00% DistributionMapping::Distribute() 56 1.513e-05 1.513e-05 1.513e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.364e-05 1.364e-05 1.364e-05 0.00% Castro::check_for_nan() 20 1.32e-05 1.32e-05 1.32e-05 0.00% Castro::apply_source_to_state() 20 1.098e-05 1.098e-05 1.098e-05 0.00% Castro::construct_old_gravity() 10 1.052e-05 1.052e-05 1.052e-05 0.00% MLMG::computeMLResidual() 11 9.408e-06 9.408e-06 9.408e-06 0.00% Amr::initSubcycle() 1 8.745e-06 8.745e-06 8.745e-06 0.00% Gravity::swapTimeLevels() 10 8.663e-06 8.663e-06 8.663e-06 0.00% MLPoisson::prepareForSolve() 11 8.609e-06 8.609e-06 8.609e-06 0.00% Castro::post_timestep() 10 8.214e-06 8.214e-06 8.214e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.805e-06 6.805e-06 6.805e-06 0.00% MLMG::getGradSolution() 11 6.559e-06 6.559e-06 6.559e-06 0.00% Castro::computeNewDt() 9 6.353e-06 6.353e-06 6.353e-06 0.00% Amr::InitializeInit() 1 6.309e-06 6.309e-06 6.309e-06 0.00% AmrLevel::checkPointPost() 3 5.183e-06 5.183e-06 5.183e-06 0.00% Castro::post_init() 1 4.323e-06 4.323e-06 4.323e-06 0.00% Castro::retry_advance_ctu() 10 4.202e-06 4.202e-06 4.202e-06 0.00% Gravity::set_mass_offset() 11 3.691e-06 3.691e-06 3.691e-06 0.00% MLMG::MLResNormInf() 11 3.399e-06 3.399e-06 3.399e-06 0.00% Castro::create_source_corrector() 10 3.392e-06 3.392e-06 3.392e-06 0.00% Castro::FluxRegCrseInit 10 3.303e-06 3.303e-06 3.303e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.68e-06 2.68e-06 2.68e-06 0.00% Amr::init() 1 2.601e-06 2.601e-06 2.601e-06 0.00% Castro::computeInitialDt() 2 2.413e-06 2.413e-06 2.413e-06 0.00% AmrLevel::checkPointPre() 3 2.25e-06 2.25e-06 2.25e-06 0.00% Castro::FluxRegFineAdd() 10 2.229e-06 2.229e-06 2.229e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.988e-06 1.988e-06 1.988e-06 0.00% Castro::post_regrid() 1 1.279e-06 1.279e-06 1.279e-06 0.00% Amr::initialInit() 1 1.122e-06 1.122e-06 1.122e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8569 0.8569 0.8569 100.00% Amr::coarseTimeStep() 10 0.6943 0.6943 0.6943 81.02% Amr::timeStep() 10 0.5928 0.5928 0.5928 69.18% Castro::advance() 10 0.5854 0.5854 0.5854 68.32% Castro::subcycle_advance_ctu() 10 0.5738 0.5738 0.5738 66.96% Castro::do_advance_ctu() 10 0.5737 0.5737 0.5737 66.94% Gravity::solve_phi_with_mlmg() 11 0.3081 0.3081 0.3081 35.95% Gravity::actual_solve_with_mlmg() 11 0.2981 0.2981 0.2981 34.78% Castro::construct_new_gravity() 10 0.2801 0.2801 0.2801 32.69% MLMG::solve() 11 0.2759 0.2759 0.2759 32.19% Gravity::solve_for_phi() 10 0.2648 0.2648 0.2648 30.90% MLMG::oneIter() 82 0.2612 0.2612 0.2612 30.48% MLMG::mgVcycle() 82 0.2595 0.2595 0.2595 30.28% Castro::construct_ctu_hydro_source() 10 0.2147 0.2147 0.2147 25.06% VisMF::Write(FabArray) 11 0.2009 0.2009 0.2009 23.44% Amr::checkPoint() 3 0.1498 0.1498 0.1498 17.48% AmrLevel::checkPoint() 3 0.1445 0.1445 0.1445 16.86% StateData::checkPoint() 12 0.1444 0.1444 0.1444 16.86% MLCellLinOp::smooth() 1640 0.1329 0.1329 0.1329 15.51% Amr::init() 1 0.1322 0.1322 0.1322 15.43% MLCellLinOp::applyBC() 4433 0.0936 0.0936 0.0936 10.92% MLMG::mgVcycle_bottom 82 0.07957 0.07957 0.07957 9.29% MLMG::actualBottomSolve() 82 0.07954 0.07954 0.07954 9.28% MLCGSolver::bicgstab 82 0.07875 0.07875 0.07875 9.19% MLPoisson::Fsmooth() 3280 0.06216 0.06216 0.06216 7.25% Amr::writePlotFile() 2 0.05943 0.05943 0.05943 6.94% Amr::initialInit() 1 0.05034 0.05034 0.05034 5.87% Amr::FinalizeInit() 1 0.04683 0.04683 0.04683 5.46% Castro::clean_state() 62 0.04581 0.04581 0.04581 5.35% Castro::post_init() 1 0.04573 0.04573 0.04573 5.34% Gravity::multilevel_solve_for_new_phi() 1 0.0438 0.0438 0.0438 5.11% Gravity::actual_multilevel_solve() 1 0.04378 0.04378 0.04378 5.11% FillPatchIterator::Initialize 41 0.04227 0.04227 0.04227 4.93% FillPatchSingleLevel 41 0.0407 0.0407 0.0407 4.75% StateDataPhysBCFunct::() 41 0.03667 0.03667 0.03667 4.28% MLCellLinOp::apply() 1142 0.03537 0.03537 0.03537 4.13% MLMG::mgVcycle_down::0 82 0.03467 0.03467 0.03467 4.05% MLMG::mgVcycle_up::0 82 0.02971 0.02971 0.02971 3.47% StateData::FillBoundary(geom) 328 0.0243 0.0243 0.0243 2.84% MultiFab::Dot() 1114 0.02158 0.02158 0.02158 2.52% MLCellLinOp::correctionResidual() 492 0.02071 0.02071 0.02071 2.42% Castro::initialize_do_advance() 10 0.01975 0.01975 0.01975 2.30% Castro::computeTemp() 63 0.01961 0.01961 0.01961 2.29% MLMG:computeResOfCorrection() 410 0.0179 0.0179 0.0179 2.09% MLPoisson::define() 11 0.01787 0.01787 0.01787 2.09% MLMG::mgVcycle_down::1 82 0.01732 0.01732 0.01732 2.02% Gravity::get_new_grav_vector() 11 0.01695 0.01695 0.01695 1.98% MLMG::mgVcycle_down::2 82 0.01674 0.01674 0.01674 1.95% MLMG::mgVcycle_down::3 82 0.01606 0.01606 0.01606 1.87% FabArray::FillBoundary() 4023 0.0156 0.0156 0.0156 1.82% MLMG::mgVcycle_down::4 82 0.01522 0.01522 0.01522 1.78% Castro::construct_old_gravity() 10 0.01498 0.01498 0.01498 1.75% Gravity::get_old_grav_vector() 10 0.01496 0.01496 0.01496 1.75% FillBoundary_nowait() 4023 0.01479 0.01479 0.01479 1.73% CGSolver::sxay() 1586 0.01431 0.01431 0.01431 1.67% MultiFab::LinComb() 1586 0.01395 0.01395 0.01395 1.63% FabArray::setVal() 1144 0.01388 0.01388 0.01388 1.62% FabArray::ParallelCopy() 861 0.01387 0.01387 0.01387 1.62% Castro::normalize_species() 62 0.01385 0.01385 0.01385 1.62% FabArray::ParallelCopy_nowait() 861 0.01362 0.01362 0.01362 1.59% MLMG::mgVcycle_up::2 82 0.01295 0.01295 0.01295 1.51% MLCGSolver::ParallelAllReduce 1514 0.01286 0.01286 0.01286 1.50% MLMG::mgVcycle_up::1 82 0.01272 0.01272 0.01272 1.48% Castro::do_new_sources() 10 0.01272 0.01272 0.01272 1.48% MLCellLinOp::defineAuxData() 11 0.0127 0.0127 0.0127 1.48% MLMG::addInterpCorrection() 410 0.01242 0.01242 0.01242 1.45% MLMG::mgVcycle_up::3 82 0.01226 0.01226 0.01226 1.43% MLMG::mgVcycle_up::4 82 0.01211 0.01211 0.01211 1.41% Castro::enforce_min_density() 62 0.01178 0.01178 0.01178 1.37% amrex::average_down 410 0.01174 0.01174 0.01174 1.37% Castro::expand_state() 10 0.01169 0.01169 0.01169 1.36% Castro::initialize_advance() 10 0.01151 0.01151 0.01151 1.34% MLPoisson::Fapply() 1142 0.01144 0.01144 0.01144 1.33% Castro::do_old_sources() 10 0.0109 0.0109 0.0109 1.27% Gravity::fill_multipole_BCs() 11 0.009798 0.009798 0.009798 1.14% Castro::post_timestep() 10 0.007184 0.007184 0.007184 0.84% MLCellLinOp::solutionResidual() 93 0.00702 0.00702 0.00702 0.82% MultiFab::Xpay() 585 0.006409 0.006409 0.006409 0.75% Castro::estTimeStep() 21 0.006128 0.006128 0.006128 0.72% MLMG::computeResidual() 82 0.006072 0.006072 0.006072 0.71% Castro::reset_internal_energy(MultiFab) 63 0.005487 0.005487 0.005487 0.64% MLMG::prepareForSolve() 11 0.00519 0.00519 0.00519 0.61% MLCellLinOp::defineBC() 11 0.004882 0.004882 0.004882 0.57% BndryData::define() 11 0.004604 0.004604 0.004604 0.54% Castro::construct_new_source() 50 0.003544 0.003544 0.003544 0.41% Amr::InitializeInit() 1 0.00351 0.00351 0.00351 0.41% Amr::defBaseLevel() 1 0.003504 0.003504 0.003504 0.41% Castro::construct_new_gravity_source() 10 0.003288 0.003288 0.003288 0.38% Castro::initData() 1 0.002998 0.002998 0.002998 0.35% Castro::computeNewDt() 9 0.002925 0.002925 0.002925 0.34% Castro::construct_old_source() 50 0.002915 0.002915 0.002915 0.34% Castro::construct_old_gravity_source() 10 0.002898 0.002898 0.002898 0.34% MLMG::ResNormInf() 93 0.002084 0.002084 0.002084 0.24% Castro::apply_source_to_state() 20 0.001826 0.001826 0.001826 0.21% MultiFab::Saxpy() 20 0.001815 0.001815 0.001815 0.21% MultiFab::Add() 82 0.001656 0.001656 0.001656 0.19% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001584 0.001584 0.001584 0.18% MLCellLinOp::setLevelBC() 11 0.001484 0.001484 0.001484 0.17% FabArrayBase::getCPC() 1323 0.001448 0.001448 0.001448 0.17% Castro::reset_internal_energy(Fab) 504 0.001417 0.001417 0.001417 0.17% MLMG::getGradSolution() 11 0.001408 0.001408 0.001408 0.16% MLCellLinOp::compGrad() 11 0.001401 0.001401 0.001401 0.16% FabArray::mult() 43 0.001304 0.001304 0.001304 0.15% FabArray::setDomainBndry() 41 0.001282 0.001282 0.001282 0.15% Castro::check_for_nan() 20 0.001197 0.001197 0.001197 0.14% MultiFab::contains_nan() 20 0.001184 0.001184 0.001184 0.14% MLPoisson::prepareForSolve() 11 0.00114 0.00114 0.00114 0.13% MLCellLinOp::prepareForSolve() 11 0.001131 0.001131 0.001131 0.13% MLMG::computeMLResidual() 11 0.0009943 0.0009943 0.0009943 0.12% Castro::enforce_speed_limit() 62 0.0009841 0.0009841 0.0009841 0.11% Castro::post_regrid() 1 0.0009387 0.0009387 0.0009387 0.11% Gravity::update_max_rhs() 11 0.000803 0.000803 0.000803 0.09% FabArrayBase::CPC::define() 454 0.0006882 0.0006882 0.0006882 0.08% FabArrayBase::getFB() 4023 0.0006805 0.0006805 0.0006805 0.08% Castro::computeInitialDt() 2 0.0006596 0.0006596 0.0006596 0.08% Amr::InitAmr() 1 0.0004726 0.0004726 0.0004726 0.06% Gravity::swapTimeLevels() 10 0.0004283 0.0004283 0.0004283 0.05% Castro::Castro() 1 0.0004272 0.0004272 0.0004272 0.05% MultiFab::Copy() 11 0.000316 0.000316 0.000316 0.04% MLMG::MLResNormInf() 11 0.0002709 0.0002709 0.0002709 0.03% MLLinOp::define() 11 0.0002641 0.0002641 0.0002641 0.03% MultiFab::max() 11 0.0002538 0.0002538 0.0002538 0.03% MLLinOp::defineGrids() 11 0.0002391 0.0002391 0.0002391 0.03% MLMG::MLRhsNormInf() 11 0.0002108 0.0002108 0.0002108 0.02% Castro::buildMetrics() 1 0.0001519 0.0001519 0.0001519 0.02% FabArrayBase::FB::FB() 56 8.77e-05 8.77e-05 8.77e-05 0.01% Castro::finalize_advance() 10 5.834e-05 5.834e-05 5.834e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.391e-05 5.391e-05 5.391e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.905e-05 4.905e-05 4.905e-05 0.01% StateData::define() 4 4.225e-05 4.225e-05 4.225e-05 0.00% makeSFC 55 4.027e-05 4.027e-05 4.027e-05 0.00% Castro::swap_state_time_levels() 10 4.002e-05 4.002e-05 4.002e-05 0.00% Castro::finalize_do_advance() 10 3.61e-05 3.61e-05 3.61e-05 0.00% Castro::enforce_consistent_e() 1 3.38e-05 3.38e-05 3.38e-05 0.00% Castro::initMFs() 1 2.642e-05 2.642e-05 2.642e-05 0.00% Amr::writeSmallPlotFile() 1 2.586e-05 2.586e-05 2.586e-05 0.00% DistributionMapping::Distribute() 56 1.513e-05 1.513e-05 1.513e-05 0.00% Amr::initSubcycle() 1 8.745e-06 8.745e-06 8.745e-06 0.00% AmrLevel::checkPointPost() 3 5.183e-06 5.183e-06 5.183e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.207e-06 4.207e-06 4.207e-06 0.00% Castro::retry_advance_ctu() 10 4.202e-06 4.202e-06 4.202e-06 0.00% Gravity::set_mass_offset() 11 3.691e-06 3.691e-06 3.691e-06 0.00% Castro::create_source_corrector() 10 3.392e-06 3.392e-06 3.392e-06 0.00% Castro::FluxRegCrseInit 10 3.303e-06 3.303e-06 3.303e-06 0.00% AmrLevel::checkPointPre() 3 2.25e-06 2.25e-06 2.25e-06 0.00% Castro::FluxRegFineAdd() 10 2.229e-06 2.229e-06 2.229e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.988e-06 1.988e-06 1.988e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.09) initialized Starting run at 08:30:02 UTC on 2022-09-02. Successfully read inputs file ... Castro git describe: 22.09 AMReX git describe: 22.09 Microphysics git describe: 22.08-7-g8d22e048 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.457557672 Restart time = 0.048570154 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052394993 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049704845 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.049327146 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057369581 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062206663 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031553137 seconds Ending run at 08:30:03 UTC on 2022-09-02. Run time = 0.352103207 Run time without initialization = 0.302974262 Average number of zones advanced per microsecond: 4.326 Average number of zones advanced per microsecond per rank: 4.326 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3521 ... 0.3521 ... 0.3521 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0798 0.0798 0.0798 22.65% VisMF::Read() 3 0.04076 0.04076 0.04076 11.58% MLCellLinOp::applyBC() 1946 0.03415 0.03415 0.03415 9.70% VisMF::Write(FabArray) 1 0.02998 0.02998 0.02998 8.52% MLPoisson::Fsmooth() 1440 0.02658 0.02658 0.02658 7.55% StateData::FillBoundary(geom) 160 0.01184 0.01184 0.01184 3.36% MLCGSolver::bicgstab 36 0.01006 0.01006 0.01006 2.86% MultiFab::Dot() 484 0.009279 0.009279 0.009279 2.64% FabArray::setVal() 537 0.006643 0.006643 0.006643 1.89% Castro::computeTemp() 30 0.006621 0.006621 0.006621 1.88% FillBoundary_nowait() 1766 0.006248 0.006248 0.006248 1.77% MLCellLinOp::defineAuxData() 6 0.00618 0.00618 0.00618 1.75% MultiFab::LinComb() 690 0.005967 0.005967 0.005967 1.69% FabArray::ParallelCopy_nowait() 380 0.005857 0.005857 0.005857 1.66% Gravity::fill_multipole_BCs() 6 0.005403 0.005403 0.005403 1.53% Castro::normalize_species() 30 0.005237 0.005237 0.005237 1.49% Castro::enforce_min_density() 30 0.00518 0.00518 0.00518 1.47% StateDataPhysBCFunct::() 20 0.005177 0.005177 0.005177 1.47% MLPoisson::Fapply() 500 0.004966 0.004966 0.004966 1.41% Amr::restart() 1 0.003621 0.003621 0.003621 1.03% MLMG::addInterpCorrection() 180 0.003256 0.003256 0.003256 0.92% Castro::estTimeStep() 10 0.003166 0.003166 0.003166 0.90% amrex::average_down 180 0.00296 0.00296 0.00296 0.84% MultiFab::Xpay() 258 0.002819 0.002819 0.002819 0.80% BndryData::define() 6 0.002062 0.002062 0.002062 0.59% Castro::do_advance_ctu() 5 0.002003 0.002003 0.002003 0.57% Castro::reset_internal_energy(MultiFab) 30 0.001988 0.001988 0.001988 0.56% Amr::writePlotFile() 1 0.001661 0.001661 0.001661 0.47% Castro::construct_new_gravity_source() 5 0.001374 0.001374 0.001374 0.39% Castro::construct_old_gravity_source() 5 0.00125 0.00125 0.00125 0.35% MLMG::ResNormInf() 42 0.0009308 0.0009308 0.0009308 0.26% MultiFab::Saxpy() 10 0.0009183 0.0009183 0.0009183 0.26% Gravity::get_old_grav_vector() 5 0.0008842 0.0008842 0.0008842 0.25% Castro::expand_state() 5 0.0008703 0.0008703 0.0008703 0.25% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008694 0.0008694 0.0008694 0.25% Gravity::get_new_grav_vector() 5 0.0008624 0.0008624 0.0008624 0.24% Castro::subcycle_advance_ctu() 5 0.0008498 0.0008498 0.0008498 0.24% MLCellLinOp::setLevelBC() 6 0.000816 0.000816 0.000816 0.23% Gravity::actual_solve_with_mlmg() 6 0.0007599 0.0007599 0.0007599 0.22% Castro::reset_internal_energy(Fab) 240 0.000739 0.000739 0.000739 0.21% MultiFab::Add() 36 0.0007171 0.0007171 0.0007171 0.20% Castro::enforce_speed_limit() 30 0.0006778 0.0006778 0.0006778 0.19% MLMG::prepareForSolve() 6 0.0006552 0.0006552 0.0006552 0.19% FabArray::mult() 22 0.0006483 0.0006483 0.0006483 0.18% FabArray::setDomainBndry() 20 0.0006376 0.0006376 0.0006376 0.18% MLCellLinOp::prepareForSolve() 6 0.0006236 0.0006236 0.0006236 0.18% MultiFab::contains_nan() 10 0.0005938 0.0005938 0.0005938 0.17% MLCellLinOp::compGrad() 6 0.0004864 0.0004864 0.0004864 0.14% MLCellLinOp::smooth() 720 0.0004744 0.0004744 0.0004744 0.13% Amr::InitAmr() 1 0.000399 0.000399 0.000399 0.11% FabArray::FillBoundary() 1766 0.0003799 0.0003799 0.0003799 0.11% FabArrayBase::CPC::define() 244 0.0003798 0.0003798 0.0003798 0.11% FabArrayBase::getCPC() 632 0.0003613 0.0003613 0.0003613 0.10% FabArrayBase::getFB() 1766 0.0002539 0.0002539 0.0002539 0.07% main() 1 0.0002459 0.0002459 0.0002459 0.07% Gravity::update_max_rhs() 6 0.0002261 0.0002261 0.0002261 0.06% Gravity::solve_for_phi() 5 0.0002221 0.0002221 0.0002221 0.06% MLCellLinOp::apply() 500 0.0002035 0.0002035 0.0002035 0.06% MultiFab::Copy() 6 0.0001724 0.0001724 0.0001724 0.05% Amr::coarseTimeStep() 5 0.0001708 0.0001708 0.0001708 0.05% CGSolver::sxay() 690 0.0001655 0.0001655 0.0001655 0.05% MLCellLinOp::defineBC() 6 0.0001485 0.0001485 0.0001485 0.04% FillPatchIterator::Initialize 20 0.0001436 0.0001436 0.0001436 0.04% Castro::construct_new_source() 25 0.0001389 0.0001389 0.0001389 0.04% MultiFab::max() 6 0.0001346 0.0001346 0.0001346 0.04% MLCGSolver::ParallelAllReduce 659 0.000127 0.000127 0.000127 0.04% Castro::construct_new_gravity() 5 0.0001246 0.0001246 0.0001246 0.04% FabArray::ParallelCopy() 380 0.0001192 0.0001192 0.0001192 0.03% MLMG::MLRhsNormInf() 6 0.0001117 0.0001117 0.0001117 0.03% Amr::timeStep() 5 0.0001098 0.0001098 0.0001098 0.03% MLCellLinOp::correctionResidual() 216 9.263e-05 9.263e-05 9.263e-05 0.03% MLMG::mgVcycle() 36 8.885e-05 8.885e-05 8.885e-05 0.03% StateData::restartDoit() 4 7.903e-05 7.903e-05 7.903e-05 0.02% AmrLevel::restart() 1 7.504e-05 7.504e-05 7.504e-05 0.02% MLLinOp::defineGrids() 6 7.235e-05 7.235e-05 7.235e-05 0.02% MLMG:computeResOfCorrection() 180 7.037e-05 7.037e-05 7.037e-05 0.02% FabArrayBase::FB::FB() 26 5.582e-05 5.582e-05 5.582e-05 0.02% MLMG::mgVcycle_down::0 36 4.864e-05 4.864e-05 4.864e-05 0.01% Castro::advance() 5 4.364e-05 4.364e-05 4.364e-05 0.01% MLMG::mgVcycle_down::1 36 4.327e-05 4.327e-05 4.327e-05 0.01% Castro::construct_old_source() 25 4.297e-05 4.297e-05 4.297e-05 0.01% Castro::clean_state() 30 4.08e-05 4.08e-05 4.08e-05 0.01% MLMG::mgVcycle_down::2 36 3.958e-05 3.958e-05 3.958e-05 0.01% Castro::initialize_advance() 5 3.912e-05 3.912e-05 3.912e-05 0.01% Castro::create_source_corrector() 5 3.882e-05 3.882e-05 3.882e-05 0.01% MLMG::mgVcycle_down::4 36 3.853e-05 3.853e-05 3.853e-05 0.01% Castro::finalize_advance() 5 3.803e-05 3.803e-05 3.803e-05 0.01% MLMG::mgVcycle_down::3 36 3.718e-05 3.718e-05 3.718e-05 0.01% MLMG::actualBottomSolve() 36 3.365e-05 3.365e-05 3.365e-05 0.01% MLMG::mgVcycle_up::4 36 3.32e-05 3.32e-05 3.32e-05 0.01% Castro::initialize_do_advance() 5 3.228e-05 3.228e-05 3.228e-05 0.01% Amr::writeSmallPlotFile() 1 3.209e-05 3.209e-05 3.209e-05 0.01% MLMG::solve() 6 3.12e-05 3.12e-05 3.12e-05 0.01% Castro::buildMetrics() 1 3.078e-05 3.078e-05 3.078e-05 0.01% Castro::post_restart() 1 3.032e-05 3.032e-05 3.032e-05 0.01% Gravity::actual_multilevel_solve() 1 2.916e-05 2.916e-05 2.916e-05 0.01% Castro::initMFs() 1 2.802e-05 2.802e-05 2.802e-05 0.01% Castro::swap_state_time_levels() 5 2.7e-05 2.7e-05 2.7e-05 0.01% MLMG::oneIter() 36 2.664e-05 2.664e-05 2.664e-05 0.01% MLMG::mgVcycle_up::3 36 2.64e-05 2.64e-05 2.64e-05 0.01% MLMG::mgVcycle_up::0 36 2.601e-05 2.601e-05 2.601e-05 0.01% MLMG::mgVcycle_up::2 36 2.548e-05 2.548e-05 2.548e-05 0.01% MLMG::mgVcycle_up::1 36 2.414e-05 2.414e-05 2.414e-05 0.01% MLCellLinOp::solutionResidual() 42 2.282e-05 2.282e-05 2.282e-05 0.01% MLLinOp::define() 6 2.203e-05 2.203e-05 2.203e-05 0.01% MLPoisson::define() 6 2.163e-05 2.163e-05 2.163e-05 0.01% Castro::finalize_do_advance() 5 1.78e-05 1.78e-05 1.78e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.726e-05 1.726e-05 1.726e-05 0.00% MLMG::computeResidual() 36 1.652e-05 1.652e-05 1.652e-05 0.00% Castro::construct_old_gravity() 5 1.632e-05 1.632e-05 1.632e-05 0.00% makeSFC 30 1.517e-05 1.517e-05 1.517e-05 0.00% MLMG::mgVcycle_bottom 36 1.492e-05 1.492e-05 1.492e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.485e-05 1.485e-05 1.485e-05 0.00% FillPatchSingleLevel 20 1.387e-05 1.387e-05 1.387e-05 0.00% Amr::initSubcycle() 1 1.025e-05 1.025e-05 1.025e-05 0.00% Castro::do_new_sources() 5 9.188e-06 9.188e-06 9.188e-06 0.00% Castro::do_old_sources() 5 8.404e-06 8.404e-06 8.404e-06 0.00% DistributionMapping::Distribute() 31 8.34e-06 8.34e-06 8.34e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.842e-06 7.842e-06 7.842e-06 0.00% Castro::check_for_nan() 10 6.629e-06 6.629e-06 6.629e-06 0.00% Castro::apply_source_to_state() 10 5.681e-06 5.681e-06 5.681e-06 0.00% MLPoisson::prepareForSolve() 6 5.233e-06 5.233e-06 5.233e-06 0.00% MLMG::computeMLResidual() 6 4.719e-06 4.719e-06 4.719e-06 0.00% Castro::post_timestep() 5 4.708e-06 4.708e-06 4.708e-06 0.00% Gravity::swapTimeLevels() 5 4.112e-06 4.112e-06 4.112e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.215e-06 3.215e-06 3.215e-06 0.00% MLMG::getGradSolution() 6 3.111e-06 3.111e-06 3.111e-06 0.00% Castro::computeNewDt() 5 2.931e-06 2.931e-06 2.931e-06 0.00% Gravity::set_mass_offset() 6 2.404e-06 2.404e-06 2.404e-06 0.00% MLMG::MLResNormInf() 6 2.008e-06 2.008e-06 2.008e-06 0.00% Castro::FluxRegCrseInit 5 1.928e-06 1.928e-06 1.928e-06 0.00% Castro::retry_advance_ctu() 5 1.813e-06 1.813e-06 1.813e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.14e-06 1.14e-06 1.14e-06 0.00% Castro::FluxRegFineAdd() 5 1.084e-06 1.084e-06 1.084e-06 0.00% Amr::init() 1 1.042e-06 1.042e-06 1.042e-06 0.00% AmrLevel::AmrLevel() 1 9.63e-07 9.63e-07 9.63e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3521 0.3521 0.3521 100.00% Amr::coarseTimeStep() 5 0.2712 0.2712 0.2712 77.01% Amr::timeStep() 5 0.2689 0.2689 0.2689 76.36% Castro::advance() 5 0.2659 0.2659 0.2659 75.50% Castro::subcycle_advance_ctu() 5 0.2608 0.2608 0.2608 74.06% Castro::do_advance_ctu() 5 0.2599 0.2599 0.2599 73.82% Castro::construct_new_gravity() 5 0.1422 0.1422 0.1422 40.39% Gravity::solve_phi_with_mlmg() 6 0.1379 0.1379 0.1379 39.17% Gravity::solve_for_phi() 5 0.1345 0.1345 0.1345 38.20% Gravity::actual_solve_with_mlmg() 6 0.1324 0.1324 0.1324 37.61% MLMG::solve() 6 0.1203 0.1203 0.1203 34.17% MLMG::oneIter() 36 0.1132 0.1132 0.1132 32.14% MLMG::mgVcycle() 36 0.1124 0.1124 0.1124 31.93% Castro::construct_ctu_hydro_source() 5 0.07977 0.07977 0.07977 22.65% MLCellLinOp::smooth() 720 0.0575 0.0575 0.0575 16.33% Amr::init() 1 0.04862 0.04862 0.04862 13.81% Amr::restart() 1 0.04861 0.04861 0.04861 13.81% MLCellLinOp::applyBC() 1946 0.04109 0.04109 0.04109 11.67% AmrLevel::restart() 1 0.04097 0.04097 0.04097 11.64% StateData::restartDoit() 4 0.04089 0.04089 0.04089 11.61% VisMF::Read() 3 0.04076 0.04076 0.04076 11.58% MLMG::mgVcycle_bottom 36 0.03434 0.03434 0.03434 9.75% MLMG::actualBottomSolve() 36 0.03432 0.03432 0.03432 9.75% MLCGSolver::bicgstab 36 0.03398 0.03398 0.03398 9.65% Amr::writePlotFile() 1 0.03165 0.03165 0.03165 8.99% VisMF::Write(FabArray) 1 0.02998 0.02998 0.02998 8.52% MLPoisson::Fsmooth() 1440 0.02658 0.02658 0.02658 7.55% Castro::clean_state() 30 0.02048 0.02048 0.02048 5.82% FillPatchIterator::Initialize 20 0.01979 0.01979 0.01979 5.62% FillPatchSingleLevel 20 0.01901 0.01901 0.01901 5.40% StateDataPhysBCFunct::() 20 0.01701 0.01701 0.01701 4.83% MLCellLinOp::apply() 500 0.01554 0.01554 0.01554 4.41% MLMG::mgVcycle_down::0 36 0.01513 0.01513 0.01513 4.30% MLMG::mgVcycle_up::0 36 0.01289 0.01289 0.01289 3.66% StateData::FillBoundary(geom) 160 0.01184 0.01184 0.01184 3.36% Castro::initialize_do_advance() 5 0.01014 0.01014 0.01014 2.88% MLPoisson::define() 6 0.009776 0.009776 0.009776 2.78% Castro::computeTemp() 30 0.009348 0.009348 0.009348 2.65% MultiFab::Dot() 484 0.009279 0.009279 0.009279 2.64% MLCellLinOp::correctionResidual() 216 0.009048 0.009048 0.009048 2.57% MLMG:computeResOfCorrection() 180 0.007812 0.007812 0.007812 2.22% Gravity::get_new_grav_vector() 5 0.007585 0.007585 0.007585 2.15% MLMG::mgVcycle_down::1 36 0.007514 0.007514 0.007514 2.13% MLMG::mgVcycle_down::2 36 0.007284 0.007284 0.007284 2.07% Castro::construct_old_gravity() 5 0.007149 0.007149 0.007149 2.03% Gravity::get_old_grav_vector() 5 0.007133 0.007133 0.007133 2.03% MLMG::mgVcycle_down::3 36 0.006944 0.006944 0.006944 1.97% FabArray::FillBoundary() 1766 0.006938 0.006938 0.006938 1.97% MLCellLinOp::defineAuxData() 6 0.006892 0.006892 0.006892 1.96% FabArray::setVal() 537 0.006643 0.006643 0.006643 1.89% MLMG::mgVcycle_down::4 36 0.006623 0.006623 0.006623 1.88% FillBoundary_nowait() 1766 0.006558 0.006558 0.006558 1.86% FabArray::ParallelCopy() 380 0.006348 0.006348 0.006348 1.80% FabArray::ParallelCopy_nowait() 380 0.006229 0.006229 0.006229 1.77% CGSolver::sxay() 690 0.006132 0.006132 0.006132 1.74% Castro::do_new_sources() 5 0.006094 0.006094 0.006094 1.73% MultiFab::LinComb() 690 0.005967 0.005967 0.005967 1.69% Castro::expand_state() 5 0.005701 0.005701 0.005701 1.62% MLMG::mgVcycle_up::2 36 0.005581 0.005581 0.005581 1.58% MLCGSolver::ParallelAllReduce 659 0.005561 0.005561 0.005561 1.58% MLMG::mgVcycle_up::1 36 0.005509 0.005509 0.005509 1.56% MLMG::addInterpCorrection() 180 0.005409 0.005409 0.005409 1.54% Gravity::fill_multipole_BCs() 6 0.005403 0.005403 0.005403 1.53% Castro::do_old_sources() 5 0.00532 0.00532 0.00532 1.51% MLMG::mgVcycle_up::3 36 0.005273 0.005273 0.005273 1.50% MLMG::mgVcycle_up::4 36 0.00525 0.00525 0.00525 1.49% Castro::normalize_species() 30 0.005237 0.005237 0.005237 1.49% Castro::enforce_min_density() 30 0.00518 0.00518 0.00518 1.47% amrex::average_down 180 0.005176 0.005176 0.005176 1.47% Castro::initialize_advance() 5 0.004983 0.004983 0.004983 1.42% MLPoisson::Fapply() 500 0.004966 0.004966 0.004966 1.41% Castro::post_restart() 1 0.003839 0.003839 0.003839 1.09% Gravity::multilevel_solve_for_new_phi() 1 0.003714 0.003714 0.003714 1.05% Gravity::actual_multilevel_solve() 1 0.003697 0.003697 0.003697 1.05% MLCellLinOp::solutionResidual() 42 0.003204 0.003204 0.003204 0.91% Castro::estTimeStep() 10 0.003166 0.003166 0.003166 0.90% Castro::post_timestep() 5 0.0029 0.0029 0.0029 0.82% MLMG::prepareForSolve() 6 0.002842 0.002842 0.002842 0.81% MultiFab::Xpay() 258 0.002819 0.002819 0.002819 0.80% MLCellLinOp::defineBC() 6 0.002737 0.002737 0.002737 0.78% Castro::reset_internal_energy(MultiFab) 30 0.002727 0.002727 0.002727 0.77% MLMG::computeResidual() 36 0.002661 0.002661 0.002661 0.76% BndryData::define() 6 0.002588 0.002588 0.002588 0.74% Castro::computeNewDt() 5 0.002122 0.002122 0.002122 0.60% Castro::construct_new_source() 25 0.001513 0.001513 0.001513 0.43% Castro::construct_new_gravity_source() 5 0.001374 0.001374 0.001374 0.39% Castro::construct_old_source() 25 0.001293 0.001293 0.001293 0.37% Castro::construct_old_gravity_source() 5 0.00125 0.00125 0.00125 0.35% MLMG::ResNormInf() 42 0.0009308 0.0009308 0.0009308 0.26% Castro::apply_source_to_state() 10 0.000924 0.000924 0.000924 0.26% MultiFab::Saxpy() 10 0.0009183 0.0009183 0.0009183 0.26% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008694 0.0008694 0.0008694 0.25% MLCellLinOp::setLevelBC() 6 0.000816 0.000816 0.000816 0.23% MLMG::getGradSolution() 6 0.0007604 0.0007604 0.0007604 0.22% MLCellLinOp::compGrad() 6 0.0007573 0.0007573 0.0007573 0.22% FabArrayBase::getCPC() 632 0.0007411 0.0007411 0.0007411 0.21% Castro::reset_internal_energy(Fab) 240 0.000739 0.000739 0.000739 0.21% MultiFab::Add() 36 0.0007171 0.0007171 0.0007171 0.20% Castro::enforce_speed_limit() 30 0.0006778 0.0006778 0.0006778 0.19% FabArray::mult() 22 0.0006483 0.0006483 0.0006483 0.18% FabArray::setDomainBndry() 20 0.0006376 0.0006376 0.0006376 0.18% MLPoisson::prepareForSolve() 6 0.0006288 0.0006288 0.0006288 0.18% MLCellLinOp::prepareForSolve() 6 0.0006236 0.0006236 0.0006236 0.18% Castro::check_for_nan() 10 0.0006004 0.0006004 0.0006004 0.17% MultiFab::contains_nan() 10 0.0005938 0.0005938 0.0005938 0.17% MLMG::computeMLResidual() 6 0.000564 0.000564 0.000564 0.16% Gravity::update_max_rhs() 6 0.0004374 0.0004374 0.0004374 0.12% Amr::InitAmr() 1 0.0004093 0.0004093 0.0004093 0.12% FabArrayBase::CPC::define() 244 0.0003798 0.0003798 0.0003798 0.11% FabArrayBase::getFB() 1766 0.0003098 0.0003098 0.0003098 0.09% Gravity::swapTimeLevels() 5 0.0002204 0.0002204 0.0002204 0.06% MultiFab::Copy() 6 0.0001724 0.0001724 0.0001724 0.05% Castro::buildMetrics() 1 0.0001535 0.0001535 0.0001535 0.04% MLMG::MLResNormInf() 6 0.0001465 0.0001465 0.0001465 0.04% MultiFab::max() 6 0.0001346 0.0001346 0.0001346 0.04% MLLinOp::define() 6 0.0001257 0.0001257 0.0001257 0.04% MLMG::MLRhsNormInf() 6 0.0001117 0.0001117 0.0001117 0.03% MLLinOp::defineGrids() 6 0.0001036 0.0001036 0.0001036 0.03% FabArrayBase::FB::FB() 26 5.582e-05 5.582e-05 5.582e-05 0.02% Castro::finalize_advance() 5 4.105e-05 4.105e-05 4.105e-05 0.01% Castro::create_source_corrector() 5 3.882e-05 3.882e-05 3.882e-05 0.01% Amr::writeSmallPlotFile() 1 3.209e-05 3.209e-05 3.209e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.014e-05 3.014e-05 3.014e-05 0.01% Castro::initMFs() 1 2.802e-05 2.802e-05 2.802e-05 0.01% Castro::swap_state_time_levels() 5 2.7e-05 2.7e-05 2.7e-05 0.01% makeSFC 30 2.229e-05 2.229e-05 2.229e-05 0.01% Castro::finalize_do_advance() 5 1.78e-05 1.78e-05 1.78e-05 0.01% Amr::initSubcycle() 1 1.025e-05 1.025e-05 1.025e-05 0.00% DistributionMapping::Distribute() 31 8.34e-06 8.34e-06 8.34e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.435e-06 4.435e-06 4.435e-06 0.00% Gravity::set_mass_offset() 6 2.404e-06 2.404e-06 2.404e-06 0.00% Castro::FluxRegCrseInit 5 1.928e-06 1.928e-06 1.928e-06 0.00% Castro::retry_advance_ctu() 5 1.813e-06 1.813e-06 1.813e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.14e-06 1.14e-06 1.14e-06 0.00% Castro::FluxRegFineAdd() 5 1.084e-06 1.084e-06 1.084e-06 0.00% AmrLevel::AmrLevel() 1 9.63e-07 9.63e-07 9.63e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.09) finalized