Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.02-1-gbf267b095426) initialized Starting run at 08:50:51 UTC on 2022-02-03. Successfully read inputs file ... Castro git describe: 22.02-1-g342411b81 AMReX git describe: 22.02-1-gbf267b095 Microphysics git describe: 22.02 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.037524289 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.02240562 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.111053394 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.05030884 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.049241523 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.058415489 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.05889667 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.034538387 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051756275 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.049030956 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.048947461 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.057767042 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.062611545 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.034465847 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.02167124 seconds Ending run at 08:50:52 UTC on 2022-02-03. Run time = 0.809941105 Run time without initialization = 0.689179207 Average number of zones advanced per microsecond: 3.804 Average number of zones advanced per microsecond per rank: 3.804 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.81 ... 0.81 ... 0.81 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2055 0.2055 0.2055 25.37% VisMF::Write(FabArray) 11 0.1437 0.1437 0.1437 17.75% MLCellLinOp::applyBC() 4433 0.09149 0.09149 0.09149 11.30% MLPoisson::Fsmooth() 3280 0.05861 0.05861 0.05861 7.24% FabArray::setVal() 1144 0.02571 0.02571 0.02571 3.17% StateData::FillBoundary(geom) 328 0.02324 0.02324 0.02324 2.87% MLCGSolver::bicgstab 82 0.02113 0.02113 0.02113 2.61% MultiFab::Dot() 1114 0.02107 0.02107 0.02107 2.60% FillBoundary_nowait() 4023 0.01687 0.01687 0.01687 2.08% FabArray::ParallelCopy_nowait() 861 0.01582 0.01582 0.01582 1.95% StateDataPhysBCFunct::() 41 0.01572 0.01572 0.01572 1.94% Castro::computeTemp() 63 0.01396 0.01396 0.01396 1.72% Gravity::fill_multipole_BCs() 11 0.01234 0.01234 0.01234 1.52% MultiFab::LinComb() 1586 0.01211 0.01211 0.01211 1.49% Castro::enforce_min_density() 62 0.01077 0.01077 0.01077 1.33% MLPoisson::Fapply() 1142 0.01029 0.01029 0.01029 1.27% MLCellLinOp::defineAuxData() 11 0.01003 0.01003 0.01003 1.24% MLMG::addInterpCorrection() 410 0.006952 0.006952 0.006952 0.86% amrex::average_down 410 0.006434 0.006434 0.006434 0.79% MultiFab::Xpay() 585 0.00591 0.00591 0.00591 0.73% FabArray::setDomainBndry() 41 0.005576 0.005576 0.005576 0.69% Castro::reset_internal_energy() 63 0.005277 0.005277 0.005277 0.65% Castro::estTimeStep() 21 0.005013 0.005013 0.005013 0.62% Castro::enforce_speed_limit() 62 0.004649 0.004649 0.004649 0.57% Castro::expand_state() 10 0.004589 0.004589 0.004589 0.57% Amr::checkPoint() 3 0.003843 0.003843 0.003843 0.47% BndryData::define() 11 0.003614 0.003614 0.003614 0.45% Castro::do_advance_ctu() 10 0.00318 0.00318 0.00318 0.39% Amr::writePlotFile() 2 0.003052 0.003052 0.003052 0.38% Castro::construct_new_gravity_source() 10 0.003034 0.003034 0.003034 0.37% Gravity::get_new_grav_vector() 11 0.002736 0.002736 0.002736 0.34% Castro::normalize_species() 62 0.002591 0.002591 0.002591 0.32% Castro::construct_old_gravity_source() 10 0.002588 0.002588 0.002588 0.32% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002207 0.002207 0.002207 0.27% MLCellLinOp::compGrad() 11 0.001989 0.001989 0.001989 0.25% MLMG::ResNormInf() 93 0.001868 0.001868 0.001868 0.23% MultiFab::Saxpy() 20 0.001806 0.001806 0.001806 0.22% Gravity::get_old_grav_vector() 10 0.001745 0.001745 0.001745 0.22% MLMG::oneIter() 82 0.001635 0.001635 0.001635 0.20% MLCellLinOp::setLevelBC() 11 0.001511 0.001511 0.001511 0.19% Gravity::actual_solve_with_mlmg() 11 0.001378 0.001378 0.001378 0.17% FabArray::mult() 43 0.001323 0.001323 0.001323 0.16% MLCellLinOp::prepareForSolve() 11 0.001308 0.001308 0.001308 0.16% Castro::initData() 1 0.001229 0.001229 0.001229 0.15% MLCellLinOp::smooth() 1640 0.001175 0.001175 0.001175 0.15% FabArrayBase::getCPC() 1323 0.0008846 0.0008846 0.0008846 0.11% FabArray::FillBoundary() 4023 0.0007988 0.0007988 0.0007988 0.10% MLMG::prepareForSolve() 11 0.0007688 0.0007688 0.0007688 0.09% FabArrayBase::CPC::define() 454 0.0007396 0.0007396 0.0007396 0.09% FabArrayBase::getFB() 4023 0.0007108 0.0007108 0.0007108 0.09% Gravity::update_max_rhs() 11 0.0005481 0.0005481 0.0005481 0.07% MultiFab::Copy() 11 0.0005121 0.0005121 0.0005121 0.06% MLCellLinOp::apply() 1142 0.000509 0.000509 0.000509 0.06% Gravity::solve_for_phi() 10 0.000462 0.000462 0.000462 0.06% Amr::InitAmr() 1 0.0004353 0.0004353 0.0004353 0.05% CGSolver::sxay() 1586 0.0004336 0.0004336 0.0004336 0.05% MLMG::mgVcycle() 82 0.0003815 0.0003815 0.0003815 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003474 0.0003474 0.0003474 0.04% MultiFab::min() 10 0.0003157 0.0003157 0.0003157 0.04% FabArray::ParallelCopy() 861 0.0002984 0.0002984 0.0002984 0.04% main() 1 0.0002908 0.0002908 0.0002908 0.04% Castro::construct_new_gravity() 10 0.0002644 0.0002644 0.0002644 0.03% MultiFab::max() 11 0.000252 0.000252 0.000252 0.03% FillPatchIterator::Initialize 41 0.0002467 0.0002467 0.0002467 0.03% Gravity::actual_multilevel_solve() 1 0.000224 0.000224 0.000224 0.03% MLCellLinOp::correctionResidual() 492 0.0002104 0.0002104 0.0002104 0.03% MLLinOp::defineGrids() 11 0.0002024 0.0002024 0.0002024 0.02% Amr::coarseTimeStep() 10 0.000201 0.000201 0.000201 0.02% MLCellLinOp::defineBC() 11 0.0001983 0.0001983 0.0001983 0.02% MLMG::MLRhsNormInf() 11 0.0001946 0.0001946 0.0001946 0.02% Amr::timeStep() 10 0.0001873 0.0001873 0.0001873 0.02% Amr::defBaseLevel() 1 0.0001556 0.0001556 0.0001556 0.02% Castro::subcycle_advance_ctu() 10 0.000143 0.000143 0.000143 0.02% StateData::checkPoint() 12 0.0001391 0.0001391 0.0001391 0.02% MLMG:computeResOfCorrection() 410 0.0001309 0.0001309 0.0001309 0.02% MLMG::actualBottomSolve() 82 0.0001076 0.0001076 0.0001076 0.01% MLMG::mgVcycle_down::0 82 9.617e-05 9.617e-05 9.617e-05 0.01% Castro::post_timestep() 10 9.548e-05 9.548e-05 9.548e-05 0.01% FabArrayBase::FB::FB() 56 9.449e-05 9.449e-05 9.449e-05 0.01% MLMG::mgVcycle_down::1 82 8.083e-05 8.083e-05 8.083e-05 0.01% Castro::initialize_advance() 10 8.013e-05 8.013e-05 8.013e-05 0.01% AmrLevel::checkPoint() 3 7.775e-05 7.775e-05 7.775e-05 0.01% MLMG::mgVcycle_down::4 82 7.478e-05 7.478e-05 7.478e-05 0.01% MLMG::mgVcycle_down::2 82 7.387e-05 7.387e-05 7.387e-05 0.01% Castro::clean_state() 62 7.288e-05 7.288e-05 7.288e-05 0.01% MLMG::solve() 11 7.171e-05 7.171e-05 7.171e-05 0.01% MLMG::mgVcycle_down::3 82 7.157e-05 7.157e-05 7.157e-05 0.01% MLMG::mgVcycle_up::4 82 6.359e-05 6.359e-05 6.359e-05 0.01% Castro::advance() 10 5.924e-05 5.924e-05 5.924e-05 0.01% Castro::finalize_advance() 10 5.679e-05 5.679e-05 5.679e-05 0.01% Castro::initialize_do_advance() 10 5.47e-05 5.47e-05 5.47e-05 0.01% MLMG::mgVcycle_up::0 82 5.201e-05 5.201e-05 5.201e-05 0.01% MLCellLinOp::solutionResidual() 93 5.094e-05 5.094e-05 5.094e-05 0.01% MLMG::mgVcycle_up::3 82 4.952e-05 4.952e-05 4.952e-05 0.01% MLMG::mgVcycle_up::1 82 4.692e-05 4.692e-05 4.692e-05 0.01% MLMG::mgVcycle_up::2 82 4.597e-05 4.597e-05 4.597e-05 0.01% StateData::define() 4 4.262e-05 4.262e-05 4.262e-05 0.01% Castro::swap_state_time_levels() 10 3.813e-05 3.813e-05 3.813e-05 0.00% Castro::finalize_do_advance() 10 3.788e-05 3.788e-05 3.788e-05 0.00% Castro::construct_new_source() 50 3.497e-05 3.497e-05 3.497e-05 0.00% MLMG::computeResidual() 82 3.49e-05 3.49e-05 3.49e-05 0.00% Castro::enforce_consistent_e() 1 3.448e-05 3.448e-05 3.448e-05 0.00% MLMG::mgVcycle_bottom 82 3.406e-05 3.406e-05 3.406e-05 0.00% makeSFC 55 3.176e-05 3.176e-05 3.176e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.971e-05 2.971e-05 2.971e-05 0.00% MLLinOp::define() 11 2.932e-05 2.932e-05 2.932e-05 0.00% FillPatchSingleLevel 41 2.818e-05 2.818e-05 2.818e-05 0.00% MLPoisson::define() 11 2.642e-05 2.642e-05 2.642e-05 0.00% Amr::writeSmallPlotFile() 1 2.19e-05 2.19e-05 2.19e-05 0.00% Amr::FinalizeInit() 1 2.115e-05 2.115e-05 2.115e-05 0.00% Castro::construct_old_source() 50 1.977e-05 1.977e-05 1.977e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.811e-05 1.811e-05 1.811e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.644e-05 1.644e-05 1.644e-05 0.00% Castro::do_new_sources() 10 1.625e-05 1.625e-05 1.625e-05 0.00% Castro::do_old_sources() 10 1.509e-05 1.509e-05 1.509e-05 0.00% DistributionMapping::Distribute() 56 1.391e-05 1.391e-05 1.391e-05 0.00% Castro::apply_source_to_state() 20 1.215e-05 1.215e-05 1.215e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.015e-05 1.015e-05 1.015e-05 0.00% Castro::construct_old_gravity() 10 9.937e-06 9.937e-06 9.937e-06 0.00% Gravity::swapTimeLevels() 10 9.531e-06 9.531e-06 9.531e-06 0.00% MLPoisson::prepareForSolve() 11 8.617e-06 8.617e-06 8.617e-06 0.00% Amr::initSubcycle() 1 8.6e-06 8.6e-06 8.6e-06 0.00% MLMG::computeMLResidual() 11 7.277e-06 7.277e-06 7.277e-06 0.00% Castro::computeNewDt() 9 6.672e-06 6.672e-06 6.672e-06 0.00% MLMG::getGradSolution() 11 6.17e-06 6.17e-06 6.17e-06 0.00% MLMG::buildFineMask() 11 5.98e-06 5.98e-06 5.98e-06 0.00% Amr::InitializeInit() 1 5.547e-06 5.547e-06 5.547e-06 0.00% MLMG::MLResNormInf() 11 5.175e-06 5.175e-06 5.175e-06 0.00% AmrLevel::checkPointPost() 3 4.842e-06 4.842e-06 4.842e-06 0.00% Castro::post_init() 1 4.786e-06 4.786e-06 4.786e-06 0.00% Gravity::set_mass_offset() 11 4.6e-06 4.6e-06 4.6e-06 0.00% Castro::create_source_corrector() 10 4.151e-06 4.151e-06 4.151e-06 0.00% Castro::retry_advance_ctu() 10 4.094e-06 4.094e-06 4.094e-06 0.00% Amr::init() 1 3.748e-06 3.748e-06 3.748e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.626e-06 3.626e-06 3.626e-06 0.00% Castro::FluxRegFineAdd() 10 3.292e-06 3.292e-06 3.292e-06 0.00% Castro::computeInitialDt() 2 2.888e-06 2.888e-06 2.888e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.233e-06 2.233e-06 2.233e-06 0.00% AmrLevel::checkPointPre() 3 1.723e-06 1.723e-06 1.723e-06 0.00% Castro::post_regrid() 1 1.324e-06 1.324e-06 1.324e-06 0.00% Amr::initialInit() 1 1.222e-06 1.222e-06 1.222e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.81 0.81 0.81 100.00% Amr::coarseTimeStep() 10 0.6673 0.6673 0.6673 82.39% Amr::timeStep() 10 0.5954 0.5954 0.5954 73.51% Castro::advance() 10 0.5875 0.5875 0.5875 72.53% Castro::subcycle_advance_ctu() 10 0.574 0.574 0.574 70.87% Castro::do_advance_ctu() 10 0.5739 0.5739 0.5739 70.85% Gravity::solve_phi_with_mlmg() 11 0.3192 0.3192 0.3192 39.40% Gravity::actual_solve_with_mlmg() 11 0.3066 0.3066 0.3066 37.85% Castro::construct_new_gravity() 10 0.29 0.29 0.29 35.81% MLMG::solve() 11 0.2841 0.2841 0.2841 35.07% Gravity::solve_for_phi() 10 0.2719 0.2719 0.2719 33.56% MLMG::oneIter() 82 0.2687 0.2687 0.2687 33.17% MLMG::mgVcycle() 82 0.267 0.267 0.267 32.97% Castro::construct_ctu_hydro_source() 10 0.2055 0.2055 0.2055 25.37% VisMF::Write(FabArray) 11 0.1437 0.1437 0.1437 17.75% MLCellLinOp::smooth() 1640 0.1414 0.1414 0.1414 17.46% Amr::init() 1 0.1202 0.1202 0.1202 14.84% MLCellLinOp::applyBC() 4433 0.11 0.11 0.11 13.58% Amr::checkPoint() 3 0.1067 0.1067 0.1067 13.17% AmrLevel::checkPoint() 3 0.1028 0.1028 0.1028 12.69% StateData::checkPoint() 12 0.1027 0.1027 0.1027 12.68% MLMG::mgVcycle_bottom 82 0.0761 0.0761 0.0761 9.40% MLMG::actualBottomSolve() 82 0.07607 0.07607 0.07607 9.39% MLCGSolver::bicgstab 82 0.07535 0.07535 0.07535 9.30% Amr::initialInit() 1 0.06013 0.06013 0.06013 7.42% MLPoisson::Fsmooth() 3280 0.05861 0.05861 0.05861 7.24% Amr::FinalizeInit() 1 0.05121 0.05121 0.05121 6.32% Castro::post_init() 1 0.05048 0.05048 0.05048 6.23% FillPatchIterator::Initialize 41 0.04905 0.04905 0.04905 6.06% Gravity::multilevel_solve_for_new_phi() 1 0.04801 0.04801 0.04801 5.93% Gravity::actual_multilevel_solve() 1 0.04799 0.04799 0.04799 5.93% Amr::writePlotFile() 2 0.04419 0.04419 0.04419 5.46% FillPatchSingleLevel 41 0.04322 0.04322 0.04322 5.34% StateDataPhysBCFunct::() 41 0.03896 0.03896 0.03896 4.81% MLCellLinOp::apply() 1142 0.03863 0.03863 0.03863 4.77% MLMG::mgVcycle_down::0 82 0.03717 0.03717 0.03717 4.59% Castro::clean_state() 62 0.03665 0.03665 0.03665 4.53% MLMG::mgVcycle_up::0 82 0.0319 0.0319 0.0319 3.94% FabArray::setVal() 1144 0.02571 0.02571 0.02571 3.17% Castro::initialize_do_advance() 10 0.02414 0.02414 0.02414 2.98% StateData::FillBoundary(geom) 328 0.02324 0.02324 0.02324 2.87% MLCellLinOp::correctionResidual() 492 0.02169 0.02169 0.02169 2.68% MultiFab::Dot() 1114 0.02107 0.02107 0.02107 2.60% Gravity::get_new_grav_vector() 11 0.02022 0.02022 0.02022 2.50% Castro::computeTemp() 63 0.01924 0.01924 0.01924 2.38% MLMG:computeResOfCorrection() 410 0.01876 0.01876 0.01876 2.32% FabArray::FillBoundary() 4023 0.01848 0.01848 0.01848 2.28% Castro::expand_state() 10 0.01829 0.01829 0.01829 2.26% MLMG::mgVcycle_down::1 82 0.01811 0.01811 0.01811 2.24% FillBoundary_nowait() 4023 0.01768 0.01768 0.01768 2.18% MLMG::mgVcycle_down::2 82 0.01754 0.01754 0.01754 2.17% MLPoisson::define() 11 0.01709 0.01709 0.01709 2.11% FabArray::ParallelCopy() 861 0.01703 0.01703 0.01703 2.10% MLMG::mgVcycle_down::3 82 0.01678 0.01678 0.01678 2.07% FabArray::ParallelCopy_nowait() 861 0.01673 0.01673 0.01673 2.07% MLMG::mgVcycle_down::4 82 0.01598 0.01598 0.01598 1.97% Castro::construct_old_gravity() 10 0.01542 0.01542 0.01542 1.90% Gravity::get_old_grav_vector() 10 0.01541 0.01541 0.01541 1.90% MLMG::mgVcycle_up::2 82 0.01369 0.01369 0.01369 1.69% MLMG::mgVcycle_up::1 82 0.01347 0.01347 0.01347 1.66% MLMG::addInterpCorrection() 410 0.01334 0.01334 0.01334 1.65% Castro::initialize_advance() 10 0.01332 0.01332 0.01332 1.64% MLMG::mgVcycle_up::3 82 0.01304 0.01304 0.01304 1.61% MLMG::mgVcycle_up::4 82 0.01286 0.01286 0.01286 1.59% amrex::average_down 410 0.01283 0.01283 0.01283 1.58% MLCGSolver::ParallelAllReduce 1514 0.01273 0.01273 0.01273 1.57% CGSolver::sxay() 1586 0.01254 0.01254 0.01254 1.55% Gravity::fill_multipole_BCs() 11 0.01234 0.01234 0.01234 1.52% MultiFab::LinComb() 1586 0.01211 0.01211 0.01211 1.49% MLCellLinOp::defineAuxData() 11 0.0117 0.0117 0.0117 1.44% Castro::do_new_sources() 10 0.01128 0.01128 0.01128 1.39% Castro::enforce_min_density() 62 0.01077 0.01077 0.01077 1.33% MLPoisson::Fapply() 1142 0.01029 0.01029 0.01029 1.27% Amr::InitializeInit() 1 0.008918 0.008918 0.008918 1.10% Amr::defBaseLevel() 1 0.008912 0.008912 0.008912 1.10% Castro::do_old_sources() 10 0.008323 0.008323 0.008323 1.03% Castro::post_timestep() 10 0.007747 0.007747 0.007747 0.96% MLCellLinOp::solutionResidual() 93 0.007555 0.007555 0.007555 0.93% MLMG::computeResidual() 82 0.006532 0.006532 0.006532 0.81% MultiFab::Xpay() 585 0.00591 0.00591 0.00591 0.73% MLMG::prepareForSolve() 11 0.005675 0.005675 0.005675 0.70% FabArray::setDomainBndry() 41 0.005576 0.005576 0.005576 0.69% Castro::reset_internal_energy() 63 0.005277 0.005277 0.005277 0.65% MLCellLinOp::defineBC() 11 0.005061 0.005061 0.005061 0.62% Castro::estTimeStep() 21 0.005013 0.005013 0.005013 0.62% BndryData::define() 11 0.004863 0.004863 0.004863 0.60% Castro::enforce_speed_limit() 62 0.004649 0.004649 0.004649 0.57% Castro::initData() 1 0.003382 0.003382 0.003382 0.42% Castro::construct_new_source() 50 0.003068 0.003068 0.003068 0.38% Castro::construct_new_gravity_source() 10 0.003034 0.003034 0.003034 0.37% Castro::construct_old_source() 50 0.002608 0.002608 0.002608 0.32% Castro::normalize_species() 62 0.002591 0.002591 0.002591 0.32% Castro::construct_old_gravity_source() 10 0.002588 0.002588 0.002588 0.32% MLMG::getGradSolution() 11 0.002523 0.002523 0.002523 0.31% MLCellLinOp::compGrad() 11 0.002517 0.002517 0.002517 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002207 0.002207 0.002207 0.27% Castro::computeNewDt() 9 0.002169 0.002169 0.002169 0.27% MLMG::ResNormInf() 93 0.001868 0.001868 0.001868 0.23% Castro::apply_source_to_state() 20 0.001818 0.001818 0.001818 0.22% MultiFab::Saxpy() 20 0.001806 0.001806 0.001806 0.22% FabArrayBase::getCPC() 1323 0.001624 0.001624 0.001624 0.20% MLCellLinOp::setLevelBC() 11 0.001511 0.001511 0.001511 0.19% FabArray::mult() 43 0.001323 0.001323 0.001323 0.16% MLPoisson::prepareForSolve() 11 0.001317 0.001317 0.001317 0.16% MLCellLinOp::prepareForSolve() 11 0.001308 0.001308 0.001308 0.16% MLMG::computeMLResidual() 11 0.001066 0.001066 0.001066 0.13% Gravity::swapTimeLevels() 10 0.001003 0.001003 0.001003 0.12% Gravity::update_max_rhs() 11 0.0009414 0.0009414 0.0009414 0.12% FabArrayBase::getFB() 4023 0.0008053 0.0008053 0.0008053 0.10% FabArrayBase::CPC::define() 454 0.0007396 0.0007396 0.0007396 0.09% Castro::computeInitialDt() 2 0.0006449 0.0006449 0.0006449 0.08% MultiFab::Copy() 11 0.0005121 0.0005121 0.0005121 0.06% Castro::post_regrid() 1 0.0004876 0.0004876 0.0004876 0.06% Amr::InitAmr() 1 0.0004439 0.0004439 0.0004439 0.05% MultiFab::min() 10 0.0003157 0.0003157 0.0003157 0.04% MLLinOp::define() 11 0.0002968 0.0002968 0.0002968 0.04% MLLinOp::defineGrids() 11 0.0002675 0.0002675 0.0002675 0.03% MultiFab::max() 11 0.000252 0.000252 0.000252 0.03% MLMG::MLResNormInf() 11 0.0002501 0.0002501 0.0002501 0.03% MLMG::MLRhsNormInf() 11 0.0001946 0.0001946 0.0001946 0.02% FabArrayBase::FB::FB() 56 9.449e-05 9.449e-05 9.449e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 6.288e-05 6.288e-05 6.288e-05 0.01% Castro::finalize_advance() 10 6.008e-05 6.008e-05 6.008e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.277e-05 5.277e-05 5.277e-05 0.01% makeSFC 55 4.477e-05 4.477e-05 4.477e-05 0.01% StateData::define() 4 4.262e-05 4.262e-05 4.262e-05 0.01% Castro::swap_state_time_levels() 10 3.813e-05 3.813e-05 3.813e-05 0.00% Castro::finalize_do_advance() 10 3.788e-05 3.788e-05 3.788e-05 0.00% Castro::enforce_consistent_e() 1 3.448e-05 3.448e-05 3.448e-05 0.00% Amr::writeSmallPlotFile() 1 2.19e-05 2.19e-05 2.19e-05 0.00% DistributionMapping::Distribute() 56 1.391e-05 1.391e-05 1.391e-05 0.00% Amr::initSubcycle() 1 8.6e-06 8.6e-06 8.6e-06 0.00% MLMG::buildFineMask() 11 5.98e-06 5.98e-06 5.98e-06 0.00% AmrLevel::checkPointPost() 3 4.842e-06 4.842e-06 4.842e-06 0.00% Gravity::set_mass_offset() 11 4.6e-06 4.6e-06 4.6e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.521e-06 4.521e-06 4.521e-06 0.00% Castro::create_source_corrector() 10 4.151e-06 4.151e-06 4.151e-06 0.00% Castro::retry_advance_ctu() 10 4.094e-06 4.094e-06 4.094e-06 0.00% Castro::FluxRegFineAdd() 10 3.292e-06 3.292e-06 3.292e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.233e-06 2.233e-06 2.233e-06 0.00% AmrLevel::checkPointPre() 3 1.723e-06 1.723e-06 1.723e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.02-1-gbf267b095426) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.02-1-gbf267b095426) initialized Starting run at 08:50:53 UTC on 2022-02-03. Successfully read inputs file ... Castro git describe: 22.02-1-g342411b81 AMReX git describe: 22.02-1-gbf267b095 Microphysics git describe: 22.02 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.483148948 Restart time = 0.054553418 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.115049077 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.04769097 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.047632057 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059515905 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.061914099 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.074721572 seconds Ending run at 08:50:53 UTC on 2022-02-03. Run time = 0.461981418 Run time without initialization = 0.406883667 Average number of zones advanced per microsecond: 3.221 Average number of zones advanced per microsecond per rank: 3.221 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.462 ... 0.462 ... 0.462 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1250 0.1250 0.1250 27.07% Amr::writePlotFile() 1 0.05274 0.05274 0.05274 11.42% VisMF::Read() 3 0.04294 0.04294 0.04294 9.30% MLCellLinOp::applyBC() 1946 0.03758 0.03758 0.03758 8.13% MLPoisson::Fsmooth() 1440 0.02542 0.02542 0.02542 5.50% VisMF::Write(FabArray) 1 0.02208 0.02208 0.02208 4.78% FabArray::setVal() 537 0.0167 0.0167 0.0167 3.61% StateData::FillBoundary(geom) 160 0.01089 0.01089 0.01089 2.36% MLCGSolver::bicgstab 36 0.009203 0.009203 0.009203 1.99% MultiFab::Dot() 484 0.009106 0.009106 0.009106 1.97% Castro::computeTemp() 30 0.008271 0.008271 0.008271 1.79% StateDataPhysBCFunct::() 20 0.007857 0.007857 0.007857 1.70% FillBoundary_nowait() 1766 0.007154 0.007154 0.007154 1.55% FabArray::ParallelCopy_nowait() 380 0.007062 0.007062 0.007062 1.53% Gravity::fill_multipole_BCs() 6 0.00635 0.00635 0.00635 1.37% Castro::enforce_min_density() 30 0.005757 0.005757 0.005757 1.25% MLCellLinOp::defineAuxData() 6 0.005429 0.005429 0.005429 1.18% MultiFab::LinComb() 690 0.0052 0.0052 0.0052 1.13% MLPoisson::Fapply() 500 0.004521 0.004521 0.004521 0.98% Castro::expand_state() 5 0.003945 0.003945 0.003945 0.85% FabArray::setDomainBndry() 20 0.003771 0.003771 0.003771 0.82% Amr::restart() 1 0.002999 0.002999 0.002999 0.65% MLMG::addInterpCorrection() 180 0.002991 0.002991 0.002991 0.65% amrex::average_down 180 0.00276 0.00276 0.00276 0.60% MultiFab::Xpay() 258 0.002617 0.002617 0.002617 0.57% Castro::do_advance_ctu() 5 0.002581 0.002581 0.002581 0.56% Castro::estTimeStep() 10 0.002507 0.002507 0.002507 0.54% Castro::reset_internal_energy() 30 0.002467 0.002467 0.002467 0.53% BndryData::define() 6 0.00192 0.00192 0.00192 0.42% Gravity::get_new_grav_vector() 5 0.001792 0.001792 0.001792 0.39% Castro::construct_new_gravity_source() 5 0.001569 0.001569 0.001569 0.34% MLCellLinOp::compGrad() 6 0.001429 0.001429 0.001429 0.31% Castro::construct_old_gravity_source() 5 0.001304 0.001304 0.001304 0.28% Castro::normalize_species() 30 0.001278 0.001278 0.001278 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001204 0.001204 0.001204 0.26% Castro::enforce_speed_limit() 30 0.00109 0.00109 0.00109 0.24% MultiFab::Saxpy() 10 0.0009164 0.0009164 0.0009164 0.20% Gravity::get_old_grav_vector() 5 0.0009024 0.0009024 0.0009024 0.20% MLMG::ResNormInf() 42 0.0008356 0.0008356 0.0008356 0.18% MLCellLinOp::setLevelBC() 6 0.0008218 0.0008218 0.0008218 0.18% Gravity::actual_solve_with_mlmg() 6 0.0007433 0.0007433 0.0007433 0.16% MLMG::oneIter() 36 0.000728 0.000728 0.000728 0.16% MLCellLinOp::prepareForSolve() 6 0.000687 0.000687 0.000687 0.15% FabArray::mult() 22 0.0006511 0.0006511 0.0006511 0.14% Gravity::update_max_rhs() 6 0.0005231 0.0005231 0.0005231 0.11% MLCellLinOp::smooth() 720 0.0005003 0.0005003 0.0005003 0.11% FabArrayBase::getCPC() 632 0.0004552 0.0004552 0.0004552 0.10% MLMG::prepareForSolve() 6 0.0004347 0.0004347 0.0004347 0.09% FabArrayBase::CPC::define() 244 0.0004256 0.0004256 0.0004256 0.09% FabArray::FillBoundary() 1766 0.0003856 0.0003856 0.0003856 0.08% MultiFab::Copy() 6 0.0003712 0.0003712 0.0003712 0.08% Amr::InitAmr() 1 0.00037 0.00037 0.00037 0.08% FabArrayBase::getFB() 1766 0.0003447 0.0003447 0.0003447 0.07% main() 1 0.000278 0.000278 0.000278 0.06% CGSolver::sxay() 690 0.0002158 0.0002158 0.0002158 0.05% Gravity::solve_for_phi() 5 0.0002157 0.0002157 0.0002157 0.05% MLCellLinOp::apply() 500 0.0002114 0.0002114 0.0002114 0.05% Gravity::actual_multilevel_solve() 1 0.0002077 0.0002077 0.0002077 0.04% Castro::construct_new_gravity() 5 0.0002011 0.0002011 0.0002011 0.04% MLMG::mgVcycle() 36 0.0001856 0.0001856 0.0001856 0.04% MultiFab::min() 5 0.0001619 0.0001619 0.0001619 0.04% FabArray::ParallelCopy() 380 0.000141 0.000141 0.000141 0.03% MLCGSolver::ParallelAllReduce 659 0.0001407 0.0001407 0.0001407 0.03% MultiFab::max() 6 0.0001361 0.0001361 0.0001361 0.03% FillPatchIterator::Initialize 20 0.0001198 0.0001198 0.0001198 0.03% Castro::construct_new_source() 25 0.0001137 0.0001137 0.0001137 0.02% MLCellLinOp::defineBC() 6 0.0001065 0.0001065 0.0001065 0.02% MLMG::MLRhsNormInf() 6 0.0001061 0.0001061 0.0001061 0.02% MLLinOp::defineGrids() 6 0.000106 0.000106 0.000106 0.02% Amr::coarseTimeStep() 5 0.0001052 0.0001052 0.0001052 0.02% MLCellLinOp::correctionResidual() 216 9.613e-05 9.613e-05 9.613e-05 0.02% Amr::timeStep() 5 9.566e-05 9.566e-05 9.566e-05 0.02% Castro::subcycle_advance_ctu() 5 8.725e-05 8.725e-05 8.725e-05 0.02% AmrLevel::restart() 1 7.934e-05 7.934e-05 7.934e-05 0.02% StateData::restartDoit() 4 7.621e-05 7.621e-05 7.621e-05 0.02% Castro::advance() 5 6.807e-05 6.807e-05 6.807e-05 0.01% FabArrayBase::FB::FB() 26 6.48e-05 6.48e-05 6.48e-05 0.01% MLMG:computeResOfCorrection() 180 6.023e-05 6.023e-05 6.023e-05 0.01% Castro::initialize_advance() 5 5.756e-05 5.756e-05 5.756e-05 0.01% MLMG::actualBottomSolve() 36 5.045e-05 5.045e-05 5.045e-05 0.01% Castro::post_timestep() 5 4.138e-05 4.138e-05 4.138e-05 0.01% MLMG::mgVcycle_down::0 36 3.861e-05 3.861e-05 3.861e-05 0.01% Castro::do_new_sources() 5 3.648e-05 3.648e-05 3.648e-05 0.01% MLMG::mgVcycle_down::1 36 3.443e-05 3.443e-05 3.443e-05 0.01% Castro::clean_state() 30 3.355e-05 3.355e-05 3.355e-05 0.01% Castro::post_restart() 1 3.325e-05 3.325e-05 3.325e-05 0.01% MLMG::mgVcycle_down::2 36 3.288e-05 3.288e-05 3.288e-05 0.01% MLMG::solve() 6 3.126e-05 3.126e-05 3.126e-05 0.01% MLMG::mgVcycle_down::4 36 3.11e-05 3.11e-05 3.11e-05 0.01% MLMG::mgVcycle_down::3 36 3.068e-05 3.068e-05 3.068e-05 0.01% Castro::construct_old_source() 25 3.029e-05 3.029e-05 3.029e-05 0.01% Castro::initialize_do_advance() 5 2.905e-05 2.905e-05 2.905e-05 0.01% MLLinOp::define() 6 2.851e-05 2.851e-05 2.851e-05 0.01% MLMG::mgVcycle_up::4 36 2.582e-05 2.582e-05 2.582e-05 0.01% Castro::swap_state_time_levels() 5 2.548e-05 2.548e-05 2.548e-05 0.01% Amr::writeSmallPlotFile() 1 2.534e-05 2.534e-05 2.534e-05 0.01% Castro::finalize_advance() 5 2.445e-05 2.445e-05 2.445e-05 0.01% MLCellLinOp::solutionResidual() 42 2.427e-05 2.427e-05 2.427e-05 0.01% Castro::create_source_corrector() 5 2.405e-05 2.405e-05 2.405e-05 0.01% MLMG::mgVcycle_up::3 36 2.233e-05 2.233e-05 2.233e-05 0.00% MLMG::mgVcycle_up::0 36 2.211e-05 2.211e-05 2.211e-05 0.00% MLMG::mgVcycle_up::2 36 2.107e-05 2.107e-05 2.107e-05 0.00% Castro::finalize_do_advance() 5 2.104e-05 2.104e-05 2.104e-05 0.00% MLMG::mgVcycle_up::1 36 2.03e-05 2.03e-05 2.03e-05 0.00% MLPoisson::define() 6 1.787e-05 1.787e-05 1.787e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.684e-05 1.684e-05 1.684e-05 0.00% MLMG::mgVcycle_bottom 36 1.618e-05 1.618e-05 1.618e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.583e-05 1.583e-05 1.583e-05 0.00% MLMG::computeResidual() 36 1.582e-05 1.582e-05 1.582e-05 0.00% makeSFC 30 1.444e-05 1.444e-05 1.444e-05 0.00% FillPatchSingleLevel 20 1.436e-05 1.436e-05 1.436e-05 0.00% Amr::initSubcycle() 1 8.767e-06 8.767e-06 8.767e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.547e-06 8.547e-06 8.547e-06 0.00% DistributionMapping::Distribute() 31 8.205e-06 8.205e-06 8.205e-06 0.00% Castro::do_old_sources() 5 7.939e-06 7.939e-06 7.939e-06 0.00% Castro::apply_source_to_state() 10 6.077e-06 6.077e-06 6.077e-06 0.00% Castro::construct_old_gravity() 5 5.401e-06 5.401e-06 5.401e-06 0.00% MLPoisson::prepareForSolve() 6 4.832e-06 4.832e-06 4.832e-06 0.00% Gravity::swapTimeLevels() 5 4.177e-06 4.177e-06 4.177e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.445e-06 3.445e-06 3.445e-06 0.00% MLMG::buildFineMask() 6 3.386e-06 3.386e-06 3.386e-06 0.00% Castro::computeNewDt() 5 3.287e-06 3.287e-06 3.287e-06 0.00% MLMG::computeMLResidual() 6 3.268e-06 3.268e-06 3.268e-06 0.00% MLMG::getGradSolution() 6 2.815e-06 2.815e-06 2.815e-06 0.00% MLMG::MLResNormInf() 6 2.664e-06 2.664e-06 2.664e-06 0.00% Gravity::set_mass_offset() 6 2.426e-06 2.426e-06 2.426e-06 0.00% Castro::retry_advance_ctu() 5 1.717e-06 1.717e-06 1.717e-06 0.00% Castro::FluxRegFineAdd() 5 1.632e-06 1.632e-06 1.632e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.23e-06 1.23e-06 1.23e-06 0.00% Amr::init() 1 1.199e-06 1.199e-06 1.199e-06 0.00% AmrLevel::AmrLevel() 1 8.6e-07 8.6e-07 8.6e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.462 0.462 0.462 100.00% Amr::coarseTimeStep() 5 0.3319 0.3319 0.3319 71.84% Amr::timeStep() 5 0.3305 0.3305 0.3305 71.54% Castro::advance() 5 0.3266 0.3266 0.3266 70.69% Castro::subcycle_advance_ctu() 5 0.316 0.316 0.316 68.39% Castro::do_advance_ctu() 5 0.3159 0.3159 0.3159 68.37% Castro::construct_new_gravity() 5 0.1452 0.1452 0.1452 31.44% Gravity::solve_phi_with_mlmg() 6 0.1412 0.1412 0.1412 30.56% Gravity::solve_for_phi() 5 0.1355 0.1355 0.1355 29.33% Gravity::actual_solve_with_mlmg() 6 0.1347 0.1347 0.1347 29.16% Castro::construct_ctu_hydro_source() 5 0.125 0.125 0.125 27.07% MLMG::solve() 6 0.1222 0.1222 0.1222 26.44% MLMG::oneIter() 36 0.1144 0.1144 0.1144 24.76% MLMG::mgVcycle() 36 0.1136 0.1136 0.1136 24.60% Amr::writePlotFile() 1 0.07482 0.07482 0.07482 16.20% MLCellLinOp::smooth() 720 0.05955 0.05955 0.05955 12.89% Amr::init() 1 0.05459 0.05459 0.05459 11.82% Amr::restart() 1 0.05459 0.05459 0.05459 11.82% MLCellLinOp::applyBC() 1946 0.04553 0.04553 0.04553 9.86% AmrLevel::restart() 1 0.04386 0.04386 0.04386 9.49% StateData::restartDoit() 4 0.04377 0.04377 0.04377 9.47% VisMF::Read() 3 0.04294 0.04294 0.04294 9.30% MLMG::mgVcycle_bottom 36 0.03273 0.03273 0.03273 7.08% MLMG::actualBottomSolve() 36 0.03271 0.03271 0.03271 7.08% MLCGSolver::bicgstab 36 0.0324 0.0324 0.0324 7.01% MLPoisson::Fsmooth() 1440 0.02542 0.02542 0.02542 5.50% FillPatchIterator::Initialize 20 0.02472 0.02472 0.02472 5.35% VisMF::Write(FabArray) 1 0.02208 0.02208 0.02208 4.78% FillPatchSingleLevel 20 0.02083 0.02083 0.02083 4.51% Castro::clean_state() 30 0.0189 0.0189 0.0189 4.09% StateDataPhysBCFunct::() 20 0.01874 0.01874 0.01874 4.06% FabArray::setVal() 537 0.0167 0.0167 0.0167 3.61% MLCellLinOp::apply() 500 0.01635 0.01635 0.01635 3.54% MLMG::mgVcycle_down::0 36 0.01586 0.01586 0.01586 3.43% Castro::initialize_do_advance() 5 0.01515 0.01515 0.01515 3.28% MLMG::mgVcycle_up::0 36 0.01351 0.01351 0.01351 2.92% Castro::expand_state() 5 0.01173 0.01173 0.01173 2.54% StateData::FillBoundary(geom) 160 0.01089 0.01089 0.01089 2.36% Castro::computeTemp() 30 0.01074 0.01074 0.01074 2.32% Castro::initialize_advance() 5 0.01053 0.01053 0.01053 2.28% Gravity::get_new_grav_vector() 5 0.009515 0.009515 0.009515 2.06% MLPoisson::define() 6 0.009285 0.009285 0.009285 2.01% MLCellLinOp::correctionResidual() 216 0.009224 0.009224 0.009224 2.00% MultiFab::Dot() 484 0.009106 0.009106 0.009106 1.97% MLMG:computeResOfCorrection() 180 0.007988 0.007988 0.007988 1.73% FabArray::FillBoundary() 1766 0.007949 0.007949 0.007949 1.72% Castro::construct_old_gravity() 5 0.007933 0.007933 0.007933 1.72% Gravity::get_old_grav_vector() 5 0.007927 0.007927 0.007927 1.72% MLMG::mgVcycle_down::1 36 0.007723 0.007723 0.007723 1.67% FabArray::ParallelCopy() 380 0.007642 0.007642 0.007642 1.65% FillBoundary_nowait() 1766 0.007563 0.007563 0.007563 1.64% Castro::do_new_sources() 5 0.007517 0.007517 0.007517 1.63% FabArray::ParallelCopy_nowait() 380 0.007501 0.007501 0.007501 1.62% MLMG::mgVcycle_down::2 36 0.007416 0.007416 0.007416 1.61% MLMG::mgVcycle_down::3 36 0.007067 0.007067 0.007067 1.53% MLMG::mgVcycle_down::4 36 0.006769 0.006769 0.006769 1.47% Castro::post_restart() 1 0.00652 0.00652 0.00652 1.41% MLCellLinOp::defineAuxData() 6 0.006363 0.006363 0.006363 1.38% Gravity::fill_multipole_BCs() 6 0.00635 0.00635 0.00635 1.37% Gravity::multilevel_solve_for_new_phi() 1 0.006123 0.006123 0.006123 1.33% Gravity::actual_multilevel_solve() 1 0.006106 0.006106 0.006106 1.32% MLMG::mgVcycle_up::2 36 0.005766 0.005766 0.005766 1.25% MLMG::addInterpCorrection() 180 0.005764 0.005764 0.005764 1.25% Castro::enforce_min_density() 30 0.005757 0.005757 0.005757 1.25% MLMG::mgVcycle_up::1 36 0.005713 0.005713 0.005713 1.24% amrex::average_down 180 0.005562 0.005562 0.005562 1.20% MLCGSolver::ParallelAllReduce 659 0.005508 0.005508 0.005508 1.19% MLMG::mgVcycle_up::3 36 0.005485 0.005485 0.005485 1.19% MLMG::mgVcycle_up::4 36 0.005416 0.005416 0.005416 1.17% CGSolver::sxay() 690 0.005416 0.005416 0.005416 1.17% MultiFab::LinComb() 690 0.0052 0.0052 0.0052 1.13% Castro::do_old_sources() 5 0.004971 0.004971 0.004971 1.08% MLPoisson::Fapply() 500 0.004521 0.004521 0.004521 0.98% Castro::post_timestep() 5 0.003815 0.003815 0.003815 0.83% FabArray::setDomainBndry() 20 0.003771 0.003771 0.003771 0.82% MLMG::prepareForSolve() 6 0.003446 0.003446 0.003446 0.75% MLCellLinOp::solutionResidual() 42 0.003349 0.003349 0.003349 0.72% MLMG::computeResidual() 36 0.002775 0.002775 0.002775 0.60% MLCellLinOp::defineBC() 6 0.002738 0.002738 0.002738 0.59% BndryData::define() 6 0.002632 0.002632 0.002632 0.57% MultiFab::Xpay() 258 0.002617 0.002617 0.002617 0.57% Castro::estTimeStep() 10 0.002507 0.002507 0.002507 0.54% Castro::reset_internal_energy() 30 0.002467 0.002467 0.002467 0.53% MLMG::getGradSolution() 6 0.001713 0.001713 0.001713 0.37% MLCellLinOp::compGrad() 6 0.001711 0.001711 0.001711 0.37% Castro::construct_new_source() 25 0.001683 0.001683 0.001683 0.36% Castro::construct_new_gravity_source() 5 0.001569 0.001569 0.001569 0.34% Castro::construct_old_source() 25 0.001335 0.001335 0.001335 0.29% Castro::construct_old_gravity_source() 5 0.001304 0.001304 0.001304 0.28% Castro::computeNewDt() 5 0.001282 0.001282 0.001282 0.28% Castro::normalize_species() 30 0.001278 0.001278 0.001278 0.28% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001204 0.001204 0.001204 0.26% Castro::enforce_speed_limit() 30 0.00109 0.00109 0.00109 0.24% Castro::apply_source_to_state() 10 0.0009225 0.0009225 0.0009225 0.20% MultiFab::Saxpy() 10 0.0009164 0.0009164 0.0009164 0.20% FabArrayBase::getCPC() 632 0.0008809 0.0008809 0.0008809 0.19% Gravity::swapTimeLevels() 5 0.0008769 0.0008769 0.0008769 0.19% MLMG::ResNormInf() 42 0.0008356 0.0008356 0.0008356 0.18% MLCellLinOp::setLevelBC() 6 0.0008218 0.0008218 0.0008218 0.18% Gravity::update_max_rhs() 6 0.0007365 0.0007365 0.0007365 0.16% MLPoisson::prepareForSolve() 6 0.0006918 0.0006918 0.0006918 0.15% MLCellLinOp::prepareForSolve() 6 0.000687 0.000687 0.000687 0.15% FabArray::mult() 22 0.0006511 0.0006511 0.0006511 0.14% MLMG::computeMLResidual() 6 0.0005925 0.0005925 0.0005925 0.13% FabArrayBase::CPC::define() 244 0.0004256 0.0004256 0.0004256 0.09% FabArrayBase::getFB() 1766 0.0004095 0.0004095 0.0004095 0.09% Amr::InitAmr() 1 0.0003788 0.0003788 0.0003788 0.08% MultiFab::Copy() 6 0.0003712 0.0003712 0.0003712 0.08% MLLinOp::define() 6 0.0001658 0.0001658 0.0001658 0.04% MultiFab::min() 5 0.0001619 0.0001619 0.0001619 0.04% MLLinOp::defineGrids() 6 0.0001373 0.0001373 0.0001373 0.03% MultiFab::max() 6 0.0001361 0.0001361 0.0001361 0.03% MLMG::MLResNormInf() 6 0.0001352 0.0001352 0.0001352 0.03% MLMG::MLRhsNormInf() 6 0.0001061 0.0001061 0.0001061 0.02% FabArrayBase::FB::FB() 26 6.48e-05 6.48e-05 6.48e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.002e-05 3.002e-05 3.002e-05 0.01% Castro::finalize_advance() 5 2.609e-05 2.609e-05 2.609e-05 0.01% Castro::swap_state_time_levels() 5 2.548e-05 2.548e-05 2.548e-05 0.01% Amr::writeSmallPlotFile() 1 2.534e-05 2.534e-05 2.534e-05 0.01% Castro::create_source_corrector() 5 2.405e-05 2.405e-05 2.405e-05 0.01% makeSFC 30 2.147e-05 2.147e-05 2.147e-05 0.00% Castro::finalize_do_advance() 5 2.104e-05 2.104e-05 2.104e-05 0.00% Amr::initSubcycle() 1 8.767e-06 8.767e-06 8.767e-06 0.00% DistributionMapping::Distribute() 31 8.205e-06 8.205e-06 8.205e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.616e-06 4.616e-06 4.616e-06 0.00% MLMG::buildFineMask() 6 3.386e-06 3.386e-06 3.386e-06 0.00% Gravity::set_mass_offset() 6 2.426e-06 2.426e-06 2.426e-06 0.00% Castro::retry_advance_ctu() 5 1.717e-06 1.717e-06 1.717e-06 0.00% Castro::FluxRegFineAdd() 5 1.632e-06 1.632e-06 1.632e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.23e-06 1.23e-06 1.23e-06 0.00% AmrLevel::AmrLevel() 1 8.6e-07 8.6e-07 8.6e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.02-1-gbf267b095426) finalized