Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.03-26-ge213d767baab) initialized Starting run at 08:23:26 UTC on 2022-03-23. Successfully read inputs file ... Castro git describe: 22.02-14-g23fc313a7 AMReX git describe: 22.03-26-ge213d767b Microphysics git describe: 22.03-9-g65902e18 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.039465506 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.023313809 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.109589903 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.047793606 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.049130586 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.06642593 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.069720426 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.048774943 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.053826791 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.047890774 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.055378358 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.056757856 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.061488301 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.035714071 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.021775611 seconds Ending run at 08:23:27 UTC on 2022-03-23. Run time = 0.849167948 Run time without initialization = 0.724818302 Average number of zones advanced per microsecond: 3.617 Average number of zones advanced per microsecond per rank: 3.617 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8492 ... 0.8492 ... 0.8492 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2174 0.2174 0.2174 25.60% VisMF::Write(FabArray) 11 0.1614 0.1614 0.1614 19.01% MLCellLinOp::applyBC() 4433 0.09312 0.09312 0.09312 10.97% MLPoisson::Fsmooth() 3280 0.05998 0.05998 0.05998 7.06% FabArray::setVal() 1144 0.0263 0.0263 0.0263 3.10% StateData::FillBoundary(geom) 328 0.02568 0.02568 0.02568 3.02% MLCGSolver::bicgstab 82 0.02171 0.02171 0.02171 2.56% MultiFab::Dot() 1114 0.02144 0.02144 0.02144 2.52% FillBoundary_nowait() 4023 0.01733 0.01733 0.01733 2.04% FabArray::ParallelCopy_nowait() 861 0.01608 0.01608 0.01608 1.89% StateDataPhysBCFunct::() 41 0.01572 0.01572 0.01572 1.85% Castro::computeTemp() 63 0.01379 0.01379 0.01379 1.62% MultiFab::LinComb() 1586 0.01239 0.01239 0.01239 1.46% Gravity::fill_multipole_BCs() 11 0.01203 0.01203 0.01203 1.42% MLPoisson::Fapply() 1142 0.01059 0.01059 0.01059 1.25% MLCellLinOp::defineAuxData() 11 0.01027 0.01027 0.01027 1.21% Castro::enforce_min_density() 62 0.008365 0.008365 0.008365 0.99% MLMG::addInterpCorrection() 410 0.007156 0.007156 0.007156 0.84% amrex::average_down 410 0.006732 0.006732 0.006732 0.79% FabArray::setDomainBndry() 41 0.006184 0.006184 0.006184 0.73% Castro::estTimeStep() 21 0.006143 0.006143 0.006143 0.72% MultiFab::Xpay() 585 0.006045 0.006045 0.006045 0.71% Castro::reset_internal_energy() 63 0.005979 0.005979 0.005979 0.70% Castro::expand_state() 10 0.004864 0.004864 0.004864 0.57% Amr::checkPoint() 3 0.004536 0.004536 0.004536 0.53% BndryData::define() 11 0.003686 0.003686 0.003686 0.43% Castro::do_advance_ctu() 10 0.003304 0.003304 0.003304 0.39% Castro::construct_new_gravity_source() 10 0.003135 0.003135 0.003135 0.37% Amr::writePlotFile() 2 0.003065 0.003065 0.003065 0.36% Castro::enforce_speed_limit() 62 0.002971 0.002971 0.002971 0.35% Gravity::get_new_grav_vector() 11 0.002776 0.002776 0.002776 0.33% Castro::normalize_species() 62 0.002657 0.002657 0.002657 0.31% Castro::construct_old_gravity_source() 10 0.00265 0.00265 0.00265 0.31% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002252 0.002252 0.002252 0.27% MLCellLinOp::compGrad() 11 0.001924 0.001924 0.001924 0.23% MLMG::ResNormInf() 93 0.001893 0.001893 0.001893 0.22% MultiFab::Saxpy() 20 0.001836 0.001836 0.001836 0.22% Gravity::get_old_grav_vector() 10 0.001812 0.001812 0.001812 0.21% MLMG::oneIter() 82 0.001646 0.001646 0.001646 0.19% MLCellLinOp::setLevelBC() 11 0.001539 0.001539 0.001539 0.18% Gravity::actual_solve_with_mlmg() 11 0.001497 0.001497 0.001497 0.18% FabArray::mult() 43 0.001341 0.001341 0.001341 0.16% MLCellLinOp::prepareForSolve() 11 0.001322 0.001322 0.001322 0.16% Castro::initData() 1 0.001269 0.001269 0.001269 0.15% MLCellLinOp::smooth() 1640 0.001117 0.001117 0.001117 0.13% FabArrayBase::getCPC() 1323 0.0009186 0.0009186 0.0009186 0.11% FabArray::FillBoundary() 4023 0.000865 0.000865 0.000865 0.10% MLMG::prepareForSolve() 11 0.0008179 0.0008179 0.0008179 0.10% FabArrayBase::CPC::define() 454 0.0007811 0.0007811 0.0007811 0.09% FabArrayBase::getFB() 4023 0.0006984 0.0006984 0.0006984 0.08% Gravity::update_max_rhs() 11 0.0006073 0.0006073 0.0006073 0.07% MLCellLinOp::apply() 1142 0.000527 0.000527 0.000527 0.06% MultiFab::Copy() 11 0.0005163 0.0005163 0.0005163 0.06% CGSolver::sxay() 1586 0.0004801 0.0004801 0.0004801 0.06% Amr::InitAmr() 1 0.0004493 0.0004493 0.0004493 0.05% Gravity::solve_for_phi() 10 0.0004453 0.0004453 0.0004453 0.05% MLMG::mgVcycle() 82 0.0004159 0.0004159 0.0004159 0.05% MultiFab::min() 10 0.0003419 0.0003419 0.0003419 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003396 0.0003396 0.0003396 0.04% main() 1 0.0003194 0.0003194 0.0003194 0.04% FabArray::ParallelCopy() 861 0.000313 0.000313 0.000313 0.04% FillPatchIterator::Initialize 41 0.0002964 0.0002964 0.0002964 0.03% MultiFab::max() 11 0.0002633 0.0002633 0.0002633 0.03% Castro::construct_new_gravity() 10 0.0002596 0.0002596 0.0002596 0.03% Amr::coarseTimeStep() 10 0.0002496 0.0002496 0.0002496 0.03% Amr::timeStep() 10 0.0002459 0.0002459 0.0002459 0.03% MLCellLinOp::correctionResidual() 492 0.0002366 0.0002366 0.0002366 0.03% Gravity::actual_multilevel_solve() 1 0.0002261 0.0002261 0.0002261 0.03% MLMG::MLRhsNormInf() 11 0.0001949 0.0001949 0.0001949 0.02% MLCellLinOp::defineBC() 11 0.0001944 0.0001944 0.0001944 0.02% MLLinOp::defineGrids() 11 0.0001819 0.0001819 0.0001819 0.02% StateData::checkPoint() 12 0.000161 0.000161 0.000161 0.02% Castro::subcycle_advance_ctu() 10 0.0001546 0.0001546 0.0001546 0.02% Amr::defBaseLevel() 1 0.0001487 0.0001487 0.0001487 0.02% MLMG:computeResOfCorrection() 410 0.0001401 0.0001401 0.0001401 0.02% MLMG::actualBottomSolve() 82 0.0001206 0.0001206 0.0001206 0.01% MLMG::mgVcycle_down::0 82 9.955e-05 9.955e-05 9.955e-05 0.01% Castro::initialize_advance() 10 9.55e-05 9.55e-05 9.55e-05 0.01% FabArrayBase::FB::FB() 56 9.545e-05 9.545e-05 9.545e-05 0.01% Castro::construct_new_source() 50 9.215e-05 9.215e-05 9.215e-05 0.01% AmrLevel::checkPoint() 3 8.82e-05 8.82e-05 8.82e-05 0.01% MLMG::mgVcycle_down::1 82 8.641e-05 8.641e-05 8.641e-05 0.01% MLMG::mgVcycle_down::4 82 8.563e-05 8.563e-05 8.563e-05 0.01% MLMG::mgVcycle_down::2 82 8.502e-05 8.502e-05 8.502e-05 0.01% MLMG::mgVcycle_down::3 82 8.31e-05 8.31e-05 8.31e-05 0.01% MLMG::solve() 11 8.025e-05 8.025e-05 8.025e-05 0.01% Castro::clean_state() 62 7.829e-05 7.829e-05 7.829e-05 0.01% Castro::post_timestep() 10 7.76e-05 7.76e-05 7.76e-05 0.01% Castro::advance() 10 7.713e-05 7.713e-05 7.713e-05 0.01% MLMG::mgVcycle_up::4 82 6.91e-05 6.91e-05 6.91e-05 0.01% Castro::finalize_advance() 10 6.669e-05 6.669e-05 6.669e-05 0.01% Castro::initialize_do_advance() 10 5.836e-05 5.836e-05 5.836e-05 0.01% MLMG::mgVcycle_up::0 82 5.722e-05 5.722e-05 5.722e-05 0.01% MLMG::mgVcycle_up::1 82 5.536e-05 5.536e-05 5.536e-05 0.01% MLMG::mgVcycle_up::3 82 5.191e-05 5.191e-05 5.191e-05 0.01% MLCellLinOp::solutionResidual() 93 5.146e-05 5.146e-05 5.146e-05 0.01% MLMG::mgVcycle_up::2 82 4.962e-05 4.962e-05 4.962e-05 0.01% Castro::construct_old_source() 50 4.735e-05 4.735e-05 4.735e-05 0.01% StateData::define() 4 4.364e-05 4.364e-05 4.364e-05 0.01% Castro::finalize_do_advance() 10 4.319e-05 4.319e-05 4.319e-05 0.01% Castro::swap_state_time_levels() 10 4.195e-05 4.195e-05 4.195e-05 0.00% MLMG::mgVcycle_bottom 82 4.034e-05 4.034e-05 4.034e-05 0.00% MLMG::computeResidual() 82 3.842e-05 3.842e-05 3.842e-05 0.00% Castro::enforce_consistent_e() 1 3.428e-05 3.428e-05 3.428e-05 0.00% MLLinOp::define() 11 3.304e-05 3.304e-05 3.304e-05 0.00% FillPatchSingleLevel 41 3.3e-05 3.3e-05 3.3e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.029e-05 3.029e-05 3.029e-05 0.00% makeSFC 55 2.932e-05 2.932e-05 2.932e-05 0.00% MLPoisson::define() 11 2.569e-05 2.569e-05 2.569e-05 0.00% Amr::writeSmallPlotFile() 1 2.473e-05 2.473e-05 2.473e-05 0.00% Amr::FinalizeInit() 1 2.168e-05 2.168e-05 2.168e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.823e-05 1.823e-05 1.823e-05 0.00% Castro::do_old_sources() 10 1.8e-05 1.8e-05 1.8e-05 0.00% Castro::do_new_sources() 10 1.789e-05 1.789e-05 1.789e-05 0.00% DistributionMapping::Distribute() 56 1.733e-05 1.733e-05 1.733e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.65e-05 1.65e-05 1.65e-05 0.00% Castro::apply_source_to_state() 20 1.394e-05 1.394e-05 1.394e-05 0.00% Castro::construct_old_gravity() 10 1.37e-05 1.37e-05 1.37e-05 0.00% Gravity::swapTimeLevels() 10 1.017e-05 1.017e-05 1.017e-05 0.00% MLPoisson::prepareForSolve() 11 1.004e-05 1.004e-05 1.004e-05 0.00% AmrLevel::AmrLevel(dm) 1 9.868e-06 9.868e-06 9.868e-06 0.00% Amr::initSubcycle() 1 8.855e-06 8.855e-06 8.855e-06 0.00% Castro::computeNewDt() 9 8.15e-06 8.15e-06 8.15e-06 0.00% MLMG::computeMLResidual() 11 7.316e-06 7.316e-06 7.316e-06 0.00% Castro::create_source_corrector() 10 6.95e-06 6.95e-06 6.95e-06 0.00% Amr::InitializeInit() 1 6.677e-06 6.677e-06 6.677e-06 0.00% MLMG::getGradSolution() 11 6.064e-06 6.064e-06 6.064e-06 0.00% AmrLevel::checkPointPost() 3 5.875e-06 5.875e-06 5.875e-06 0.00% MLMG::buildFineMask() 11 5.651e-06 5.651e-06 5.651e-06 0.00% MLMG::MLResNormInf() 11 5.185e-06 5.185e-06 5.185e-06 0.00% Castro::post_init() 1 4.316e-06 4.316e-06 4.316e-06 0.00% Gravity::set_mass_offset() 11 4.165e-06 4.165e-06 4.165e-06 0.00% Castro::retry_advance_ctu() 10 4.128e-06 4.128e-06 4.128e-06 0.00% Castro::FluxRegFineAdd() 10 3.299e-06 3.299e-06 3.299e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.059e-06 3.059e-06 3.059e-06 0.00% Castro::computeInitialDt() 2 3.035e-06 3.035e-06 3.035e-06 0.00% Amr::init() 1 3.025e-06 3.025e-06 3.025e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.052e-06 2.052e-06 2.052e-06 0.00% AmrLevel::checkPointPre() 3 1.881e-06 1.881e-06 1.881e-06 0.00% Castro::post_regrid() 1 1.386e-06 1.386e-06 1.386e-06 0.00% Amr::initialInit() 1 1.119e-06 1.119e-06 1.119e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8492 0.8492 0.8492 100.00% Amr::coarseTimeStep() 10 0.7028 0.7028 0.7028 82.76% Amr::timeStep() 10 0.6148 0.6148 0.6148 72.39% Castro::advance() 10 0.6079 0.6079 0.6079 71.58% Castro::subcycle_advance_ctu() 10 0.5945 0.5945 0.5945 70.00% Castro::do_advance_ctu() 10 0.5943 0.5943 0.5943 69.98% Gravity::solve_phi_with_mlmg() 11 0.3257 0.3257 0.3257 38.36% Gravity::actual_solve_with_mlmg() 11 0.3135 0.3135 0.3135 36.91% Castro::construct_new_gravity() 10 0.2969 0.2969 0.2969 34.97% MLMG::solve() 11 0.2905 0.2905 0.2905 34.21% Gravity::solve_for_phi() 10 0.2784 0.2784 0.2784 32.78% MLMG::oneIter() 82 0.2748 0.2748 0.2748 32.35% MLMG::mgVcycle() 82 0.2731 0.2731 0.2731 32.16% Castro::construct_ctu_hydro_source() 10 0.2174 0.2174 0.2174 25.60% VisMF::Write(FabArray) 11 0.1614 0.1614 0.1614 19.01% MLCellLinOp::smooth() 1640 0.1442 0.1442 0.1442 16.98% Amr::checkPoint() 3 0.1241 0.1241 0.1241 14.61% Amr::init() 1 0.1237 0.1237 0.1237 14.57% AmrLevel::checkPoint() 3 0.1196 0.1196 0.1196 14.08% StateData::checkPoint() 12 0.1195 0.1195 0.1195 14.07% MLCellLinOp::applyBC() 4433 0.1121 0.1121 0.1121 13.20% MLMG::mgVcycle_bottom 82 0.07798 0.07798 0.07798 9.18% MLMG::actualBottomSolve() 82 0.07794 0.07794 0.07794 9.18% MLCGSolver::bicgstab 82 0.07718 0.07718 0.07718 9.09% Amr::initialInit() 1 0.06085 0.06085 0.06085 7.17% MLPoisson::Fsmooth() 3280 0.05998 0.05998 0.05998 7.06% FillPatchIterator::Initialize 41 0.05224 0.05224 0.05224 6.15% Amr::FinalizeInit() 1 0.05143 0.05143 0.05143 6.06% Castro::post_init() 1 0.05071 0.05071 0.05071 5.97% Gravity::multilevel_solve_for_new_phi() 1 0.04805 0.04805 0.04805 5.66% Gravity::actual_multilevel_solve() 1 0.04803 0.04803 0.04803 5.66% FillPatchSingleLevel 41 0.04576 0.04576 0.04576 5.39% Amr::writePlotFile() 2 0.04521 0.04521 0.04521 5.32% StateDataPhysBCFunct::() 41 0.0414 0.0414 0.0414 4.88% MLCellLinOp::apply() 1142 0.03958 0.03958 0.03958 4.66% MLMG::mgVcycle_down::0 82 0.03803 0.03803 0.03803 4.48% Castro::clean_state() 62 0.03301 0.03301 0.03301 3.89% MLMG::mgVcycle_up::0 82 0.03274 0.03274 0.03274 3.86% FabArray::setVal() 1144 0.0263 0.0263 0.0263 3.10% StateData::FillBoundary(geom) 328 0.02568 0.02568 0.02568 3.02% Castro::initialize_do_advance() 10 0.02479 0.02479 0.02479 2.92% MLCellLinOp::correctionResidual() 492 0.02218 0.02218 0.02218 2.61% MultiFab::Dot() 1114 0.02144 0.02144 0.02144 2.52% Gravity::get_new_grav_vector() 11 0.02077 0.02077 0.02077 2.45% Castro::computeTemp() 63 0.01977 0.01977 0.01977 2.33% Castro::expand_state() 10 0.01932 0.01932 0.01932 2.27% MLMG:computeResOfCorrection() 410 0.01921 0.01921 0.01921 2.26% FabArray::FillBoundary() 4023 0.01899 0.01899 0.01899 2.24% MLMG::mgVcycle_down::1 82 0.0184 0.0184 0.0184 2.17% FillBoundary_nowait() 4023 0.01813 0.01813 0.01813 2.13% MLMG::mgVcycle_down::2 82 0.01789 0.01789 0.01789 2.11% MLPoisson::define() 11 0.01744 0.01744 0.01744 2.05% FabArray::ParallelCopy() 861 0.01736 0.01736 0.01736 2.04% MLMG::mgVcycle_down::3 82 0.01715 0.01715 0.01715 2.02% FabArray::ParallelCopy_nowait() 861 0.01705 0.01705 0.01705 2.01% Castro::construct_old_gravity() 10 0.01698 0.01698 0.01698 2.00% Gravity::get_old_grav_vector() 10 0.01696 0.01696 0.01696 2.00% MLMG::mgVcycle_down::4 82 0.01642 0.01642 0.01642 1.93% MLMG::mgVcycle_up::2 82 0.01394 0.01394 0.01394 1.64% MLMG::mgVcycle_up::1 82 0.01373 0.01373 0.01373 1.62% MLMG::addInterpCorrection() 410 0.0137 0.0137 0.0137 1.61% MLMG::mgVcycle_up::3 82 0.01331 0.01331 0.01331 1.57% Castro::initialize_advance() 10 0.01324 0.01324 0.01324 1.56% amrex::average_down 410 0.01323 0.01323 0.01323 1.56% MLMG::mgVcycle_up::4 82 0.01313 0.01313 0.01313 1.55% MLCGSolver::ParallelAllReduce 1514 0.01293 0.01293 0.01293 1.52% CGSolver::sxay() 1586 0.01287 0.01287 0.01287 1.52% MultiFab::LinComb() 1586 0.01239 0.01239 0.01239 1.46% Gravity::fill_multipole_BCs() 11 0.01203 0.01203 0.01203 1.42% MLCellLinOp::defineAuxData() 11 0.01198 0.01198 0.01198 1.41% Castro::do_new_sources() 10 0.01165 0.01165 0.01165 1.37% MLPoisson::Fapply() 1142 0.01059 0.01059 0.01059 1.25% Amr::InitializeInit() 1 0.009425 0.009425 0.009425 1.11% Amr::defBaseLevel() 1 0.009418 0.009418 0.009418 1.11% Castro::enforce_min_density() 62 0.008365 0.008365 0.008365 0.99% Castro::do_old_sources() 10 0.008052 0.008052 0.008052 0.95% MLCellLinOp::solutionResidual() 93 0.007722 0.007722 0.007722 0.91% MLMG::computeResidual() 82 0.00669 0.00669 0.00669 0.79% Castro::post_timestep() 10 0.00668 0.00668 0.00668 0.79% FabArray::setDomainBndry() 41 0.006184 0.006184 0.006184 0.73% Castro::estTimeStep() 21 0.006143 0.006143 0.006143 0.72% MultiFab::Xpay() 585 0.006045 0.006045 0.006045 0.71% Castro::reset_internal_energy() 63 0.005979 0.005979 0.005979 0.70% MLMG::prepareForSolve() 11 0.005815 0.005815 0.005815 0.68% MLCellLinOp::defineBC() 11 0.005154 0.005154 0.005154 0.61% BndryData::define() 11 0.004959 0.004959 0.004959 0.58% Castro::initData() 1 0.003567 0.003567 0.003567 0.42% Castro::construct_new_source() 50 0.003227 0.003227 0.003227 0.38% Castro::construct_new_gravity_source() 10 0.003135 0.003135 0.003135 0.37% Castro::enforce_speed_limit() 62 0.002971 0.002971 0.002971 0.35% Castro::computeNewDt() 9 0.002731 0.002731 0.002731 0.32% Castro::construct_old_source() 50 0.002698 0.002698 0.002698 0.32% Castro::normalize_species() 62 0.002657 0.002657 0.002657 0.31% Castro::construct_old_gravity_source() 10 0.00265 0.00265 0.00265 0.31% MLMG::getGradSolution() 11 0.00247 0.00247 0.00247 0.29% MLCellLinOp::compGrad() 11 0.002464 0.002464 0.002464 0.29% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002252 0.002252 0.002252 0.27% MLMG::ResNormInf() 93 0.001893 0.001893 0.001893 0.22% Castro::apply_source_to_state() 20 0.00185 0.00185 0.00185 0.22% MultiFab::Saxpy() 20 0.001836 0.001836 0.001836 0.22% FabArrayBase::getCPC() 1323 0.0017 0.0017 0.0017 0.20% MLCellLinOp::setLevelBC() 11 0.001539 0.001539 0.001539 0.18% FabArray::mult() 43 0.001341 0.001341 0.001341 0.16% MLPoisson::prepareForSolve() 11 0.001332 0.001332 0.001332 0.16% MLCellLinOp::prepareForSolve() 11 0.001322 0.001322 0.001322 0.16% MLMG::computeMLResidual() 11 0.001078 0.001078 0.001078 0.13% Gravity::swapTimeLevels() 10 0.00107 0.00107 0.00107 0.13% Gravity::update_max_rhs() 11 0.00102 0.00102 0.00102 0.12% FabArrayBase::getFB() 4023 0.0007939 0.0007939 0.0007939 0.09% FabArrayBase::CPC::define() 454 0.0007811 0.0007811 0.0007811 0.09% Castro::computeInitialDt() 2 0.0006821 0.0006821 0.0006821 0.08% MultiFab::Copy() 11 0.0005163 0.0005163 0.0005163 0.06% Castro::post_regrid() 1 0.0004843 0.0004843 0.0004843 0.06% Amr::InitAmr() 1 0.0004582 0.0004582 0.0004582 0.05% MultiFab::min() 10 0.0003419 0.0003419 0.0003419 0.04% MLLinOp::define() 11 0.0002807 0.0002807 0.0002807 0.03% MultiFab::max() 11 0.0002633 0.0002633 0.0002633 0.03% MLMG::MLResNormInf() 11 0.0002553 0.0002553 0.0002553 0.03% MLLinOp::defineGrids() 11 0.0002477 0.0002477 0.0002477 0.03% MLMG::MLRhsNormInf() 11 0.0001949 0.0001949 0.0001949 0.02% FabArrayBase::FB::FB() 56 9.545e-05 9.545e-05 9.545e-05 0.01% Castro::finalize_advance() 10 6.999e-05 6.999e-05 6.999e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 6.365e-05 6.365e-05 6.365e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.351e-05 5.351e-05 5.351e-05 0.01% makeSFC 55 4.543e-05 4.543e-05 4.543e-05 0.01% StateData::define() 4 4.364e-05 4.364e-05 4.364e-05 0.01% Castro::finalize_do_advance() 10 4.319e-05 4.319e-05 4.319e-05 0.01% Castro::swap_state_time_levels() 10 4.195e-05 4.195e-05 4.195e-05 0.00% Castro::enforce_consistent_e() 1 3.428e-05 3.428e-05 3.428e-05 0.00% Amr::writeSmallPlotFile() 1 2.473e-05 2.473e-05 2.473e-05 0.00% DistributionMapping::Distribute() 56 1.733e-05 1.733e-05 1.733e-05 0.00% Amr::initSubcycle() 1 8.855e-06 8.855e-06 8.855e-06 0.00% Castro::create_source_corrector() 10 6.95e-06 6.95e-06 6.95e-06 0.00% AmrLevel::checkPointPost() 3 5.875e-06 5.875e-06 5.875e-06 0.00% MLMG::buildFineMask() 11 5.651e-06 5.651e-06 5.651e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.283e-06 4.283e-06 4.283e-06 0.00% Gravity::set_mass_offset() 11 4.165e-06 4.165e-06 4.165e-06 0.00% Castro::retry_advance_ctu() 10 4.128e-06 4.128e-06 4.128e-06 0.00% Castro::FluxRegFineAdd() 10 3.299e-06 3.299e-06 3.299e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.052e-06 2.052e-06 2.052e-06 0.00% AmrLevel::checkPointPre() 3 1.881e-06 1.881e-06 1.881e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.03-26-ge213d767baab) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.03-26-ge213d767baab) initialized Starting run at 08:23:27 UTC on 2022-03-23. Successfully read inputs file ... Castro git describe: 22.02-14-g23fc313a7 AMReX git describe: 22.03-26-ge213d767b Microphysics git describe: 22.03-9-g65902e18 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.515722715 Restart time = 0.055221518 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.115390524 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048894954 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.048397495 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059081792 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.063505711 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.023946561 seconds Ending run at 08:23:28 UTC on 2022-03-23. Run time = 0.41533995 Run time without initialization = 0.359560353 Average number of zones advanced per microsecond: 3.645 Average number of zones advanced per microsecond per rank: 3.645 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4154 ... 0.4154 ... 0.4154 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1293 0.1293 0.1293 31.13% VisMF::Read() 3 0.04348 0.04348 0.04348 10.47% MLCellLinOp::applyBC() 1946 0.03727 0.03727 0.03727 8.97% MLPoisson::Fsmooth() 1440 0.02528 0.02528 0.02528 6.09% VisMF::Write(FabArray) 1 0.02263 0.02263 0.02263 5.45% FabArray::setVal() 537 0.0168 0.0168 0.0168 4.04% StateData::FillBoundary(geom) 160 0.01093 0.01093 0.01093 2.63% MLCGSolver::bicgstab 36 0.009098 0.009098 0.009098 2.19% MultiFab::Dot() 484 0.009069 0.009069 0.009069 2.18% Castro::computeTemp() 30 0.007628 0.007628 0.007628 1.84% StateDataPhysBCFunct::() 20 0.00747 0.00747 0.00747 1.80% FillBoundary_nowait() 1766 0.007068 0.007068 0.007068 1.70% FabArray::ParallelCopy_nowait() 380 0.006957 0.006957 0.006957 1.68% Gravity::fill_multipole_BCs() 6 0.006289 0.006289 0.006289 1.51% Castro::enforce_min_density() 30 0.005787 0.005787 0.005787 1.39% MLCellLinOp::defineAuxData() 6 0.005443 0.005443 0.005443 1.31% MultiFab::LinComb() 690 0.005249 0.005249 0.005249 1.26% MLPoisson::Fapply() 500 0.004506 0.004506 0.004506 1.08% Castro::expand_state() 5 0.004169 0.004169 0.004169 1.00% FabArray::setDomainBndry() 20 0.003961 0.003961 0.003961 0.95% Castro::estTimeStep() 10 0.00314 0.00314 0.00314 0.76% Amr::restart() 1 0.002996 0.002996 0.002996 0.72% MLMG::addInterpCorrection() 180 0.002985 0.002985 0.002985 0.72% amrex::average_down 180 0.002762 0.002762 0.002762 0.67% MultiFab::Xpay() 258 0.002601 0.002601 0.002601 0.63% Castro::do_advance_ctu() 5 0.00253 0.00253 0.00253 0.61% Castro::reset_internal_energy() 30 0.002388 0.002388 0.002388 0.57% BndryData::define() 6 0.001957 0.001957 0.001957 0.47% Gravity::get_new_grav_vector() 5 0.001889 0.001889 0.001889 0.45% Castro::construct_new_gravity_source() 5 0.001576 0.001576 0.001576 0.38% Castro::normalize_species() 30 0.001431 0.001431 0.001431 0.34% MLCellLinOp::compGrad() 6 0.001425 0.001425 0.001425 0.34% Amr::writePlotFile() 1 0.001402 0.001402 0.001402 0.34% Castro::construct_old_gravity_source() 5 0.001342 0.001342 0.001342 0.32% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001206 0.001206 0.001206 0.29% Gravity::get_old_grav_vector() 5 0.001005 0.001005 0.001005 0.24% MultiFab::Saxpy() 10 0.0009236 0.0009236 0.0009236 0.22% MLMG::ResNormInf() 42 0.0008367 0.0008367 0.0008367 0.20% MLCellLinOp::setLevelBC() 6 0.0008106 0.0008106 0.0008106 0.20% Gravity::actual_solve_with_mlmg() 6 0.0007916 0.0007916 0.0007916 0.19% MLMG::oneIter() 36 0.0007248 0.0007248 0.0007248 0.17% Castro::enforce_speed_limit() 30 0.0007129 0.0007129 0.0007129 0.17% MLCellLinOp::prepareForSolve() 6 0.0006833 0.0006833 0.0006833 0.16% FabArray::mult() 22 0.0006543 0.0006543 0.0006543 0.16% Gravity::update_max_rhs() 6 0.0005438 0.0005438 0.0005438 0.13% MLCellLinOp::smooth() 720 0.0004869 0.0004869 0.0004869 0.12% FabArrayBase::CPC::define() 244 0.0004263 0.0004263 0.0004263 0.10% MLMG::prepareForSolve() 6 0.0004238 0.0004238 0.0004238 0.10% FabArrayBase::getCPC() 632 0.0004104 0.0004104 0.0004104 0.10% FabArray::FillBoundary() 1766 0.0004082 0.0004082 0.0004082 0.10% Amr::InitAmr() 1 0.000374 0.000374 0.000374 0.09% MultiFab::Copy() 6 0.0003631 0.0003631 0.0003631 0.09% FabArrayBase::getFB() 1766 0.0002992 0.0002992 0.0002992 0.07% main() 1 0.0002662 0.0002662 0.0002662 0.06% Castro::construct_new_gravity() 5 0.0002475 0.0002475 0.0002475 0.06% Gravity::actual_multilevel_solve() 1 0.0002223 0.0002223 0.0002223 0.05% Gravity::solve_for_phi() 5 0.0002168 0.0002168 0.0002168 0.05% MLCellLinOp::apply() 500 0.0002134 0.0002134 0.0002134 0.05% CGSolver::sxay() 690 0.0002028 0.0002028 0.0002028 0.05% MultiFab::min() 5 0.0001741 0.0001741 0.0001741 0.04% MLMG::mgVcycle() 36 0.0001721 0.0001721 0.0001721 0.04% FabArray::ParallelCopy() 380 0.0001424 0.0001424 0.0001424 0.03% MLCGSolver::ParallelAllReduce 659 0.0001341 0.0001341 0.0001341 0.03% MultiFab::max() 6 0.0001331 0.0001331 0.0001331 0.03% FillPatchIterator::Initialize 20 0.0001265 0.0001265 0.0001265 0.03% Castro::advance() 5 0.0001116 0.0001116 0.0001116 0.03% Amr::coarseTimeStep() 5 0.0001102 0.0001102 0.0001102 0.03% MLMG::MLRhsNormInf() 6 0.0001049 0.0001049 0.0001049 0.03% Amr::timeStep() 5 0.0001021 0.0001021 0.0001021 0.02% MLCellLinOp::defineBC() 6 9.656e-05 9.656e-05 9.656e-05 0.02% MLCellLinOp::correctionResidual() 216 9.478e-05 9.478e-05 9.478e-05 0.02% MLLinOp::defineGrids() 6 9.141e-05 9.141e-05 9.141e-05 0.02% Castro::construct_new_source() 25 8.273e-05 8.273e-05 8.273e-05 0.02% Castro::post_timestep() 5 8.169e-05 8.169e-05 8.169e-05 0.02% StateData::restartDoit() 4 7.569e-05 7.569e-05 7.569e-05 0.02% AmrLevel::restart() 1 6.994e-05 6.994e-05 6.994e-05 0.02% FabArrayBase::FB::FB() 26 6.262e-05 6.262e-05 6.262e-05 0.02% MLMG:computeResOfCorrection() 180 6.17e-05 6.17e-05 6.17e-05 0.01% Castro::subcycle_advance_ctu() 5 6.015e-05 6.015e-05 6.015e-05 0.01% Castro::create_source_corrector() 5 5.773e-05 5.773e-05 5.773e-05 0.01% Castro::initialize_advance() 5 5.696e-05 5.696e-05 5.696e-05 0.01% Castro::construct_old_source() 25 5.327e-05 5.327e-05 5.327e-05 0.01% MLMG::actualBottomSolve() 36 5.068e-05 5.068e-05 5.068e-05 0.01% MLMG::mgVcycle_down::0 36 4.124e-05 4.124e-05 4.124e-05 0.01% MLMG::mgVcycle_down::1 36 3.911e-05 3.911e-05 3.911e-05 0.01% MLMG::mgVcycle_down::2 36 3.619e-05 3.619e-05 3.619e-05 0.01% MLMG::mgVcycle_down::4 36 3.419e-05 3.419e-05 3.419e-05 0.01% Castro::clean_state() 30 3.405e-05 3.405e-05 3.405e-05 0.01% MLMG::mgVcycle_down::3 36 3.381e-05 3.381e-05 3.381e-05 0.01% MLMG::solve() 6 3.289e-05 3.289e-05 3.289e-05 0.01% Castro::initialize_do_advance() 5 2.861e-05 2.861e-05 2.861e-05 0.01% MLMG::mgVcycle_up::4 36 2.806e-05 2.806e-05 2.806e-05 0.01% MLLinOp::define() 6 2.722e-05 2.722e-05 2.722e-05 0.01% Castro::swap_state_time_levels() 5 2.67e-05 2.67e-05 2.67e-05 0.01% Amr::writeSmallPlotFile() 1 2.581e-05 2.581e-05 2.581e-05 0.01% Castro::finalize_advance() 5 2.519e-05 2.519e-05 2.519e-05 0.01% MLMG::mgVcycle_up::0 36 2.512e-05 2.512e-05 2.512e-05 0.01% MLMG::mgVcycle_up::3 36 2.315e-05 2.315e-05 2.315e-05 0.01% Castro::post_restart() 1 2.306e-05 2.306e-05 2.306e-05 0.01% MLMG::mgVcycle_up::2 36 2.256e-05 2.256e-05 2.256e-05 0.01% MLCellLinOp::solutionResidual() 42 2.213e-05 2.213e-05 2.213e-05 0.01% MLMG::mgVcycle_up::1 36 2.21e-05 2.21e-05 2.21e-05 0.01% Castro::finalize_do_advance() 5 2.117e-05 2.117e-05 2.117e-05 0.01% MLMG::mgVcycle_bottom 36 1.882e-05 1.882e-05 1.882e-05 0.00% MLMG::computeResidual() 36 1.682e-05 1.682e-05 1.682e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.568e-05 1.568e-05 1.568e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.553e-05 1.553e-05 1.553e-05 0.00% MLPoisson::define() 6 1.528e-05 1.528e-05 1.528e-05 0.00% makeSFC 30 1.456e-05 1.456e-05 1.456e-05 0.00% FillPatchSingleLevel 20 1.393e-05 1.393e-05 1.393e-05 0.00% Amr::initSubcycle() 1 9.438e-06 9.438e-06 9.438e-06 0.00% DistributionMapping::Distribute() 31 9.168e-06 9.168e-06 9.168e-06 0.00% Castro::do_new_sources() 5 9.108e-06 9.108e-06 9.108e-06 0.00% Castro::do_old_sources() 5 8.213e-06 8.213e-06 8.213e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.013e-06 8.013e-06 8.013e-06 0.00% Castro::construct_old_gravity() 5 6.865e-06 6.865e-06 6.865e-06 0.00% Castro::apply_source_to_state() 10 6.414e-06 6.414e-06 6.414e-06 0.00% MLPoisson::prepareForSolve() 6 4.539e-06 4.539e-06 4.539e-06 0.00% Gravity::swapTimeLevels() 5 4.04e-06 4.04e-06 4.04e-06 0.00% Castro::computeNewDt() 5 3.247e-06 3.247e-06 3.247e-06 0.00% MLMG::buildFineMask() 6 3.234e-06 3.234e-06 3.234e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.114e-06 3.114e-06 3.114e-06 0.00% MLMG::computeMLResidual() 6 3.105e-06 3.105e-06 3.105e-06 0.00% Castro::FluxRegFineAdd() 5 2.982e-06 2.982e-06 2.982e-06 0.00% MLMG::getGradSolution() 6 2.951e-06 2.951e-06 2.951e-06 0.00% MLMG::MLResNormInf() 6 2.57e-06 2.57e-06 2.57e-06 0.00% Gravity::set_mass_offset() 6 2.258e-06 2.258e-06 2.258e-06 0.00% Castro::retry_advance_ctu() 5 1.626e-06 1.626e-06 1.626e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.05e-06 1.05e-06 1.05e-06 0.00% Amr::init() 1 9.51e-07 9.51e-07 9.51e-07 0.00% AmrLevel::AmrLevel() 1 8.64e-07 8.64e-07 8.64e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4154 0.4154 0.4154 100.00% Amr::coarseTimeStep() 5 0.3354 0.3354 0.3354 80.74% Amr::timeStep() 5 0.3332 0.3332 0.3332 80.22% Castro::advance() 5 0.3301 0.3301 0.3301 79.47% Castro::subcycle_advance_ctu() 5 0.3189 0.3189 0.3189 76.78% Castro::do_advance_ctu() 5 0.3189 0.3189 0.3189 76.77% Castro::construct_new_gravity() 5 0.1443 0.1443 0.1443 34.73% Gravity::solve_phi_with_mlmg() 6 0.1403 0.1403 0.1403 33.78% Gravity::solve_for_phi() 5 0.1345 0.1345 0.1345 32.39% Gravity::actual_solve_with_mlmg() 6 0.1339 0.1339 0.1339 32.23% Castro::construct_ctu_hydro_source() 5 0.1293 0.1293 0.1293 31.13% MLMG::solve() 6 0.1213 0.1213 0.1213 29.20% MLMG::oneIter() 36 0.1135 0.1135 0.1135 27.33% MLMG::mgVcycle() 36 0.1128 0.1128 0.1128 27.16% MLCellLinOp::smooth() 720 0.05907 0.05907 0.05907 14.22% Amr::init() 1 0.05527 0.05527 0.05527 13.31% Amr::restart() 1 0.05527 0.05527 0.05527 13.31% MLCellLinOp::applyBC() 1946 0.0451 0.0451 0.0451 10.86% AmrLevel::restart() 1 0.04439 0.04439 0.04439 10.69% StateData::restartDoit() 4 0.04432 0.04432 0.04432 10.67% VisMF::Read() 3 0.04348 0.04348 0.04348 10.47% MLMG::mgVcycle_bottom 36 0.03257 0.03257 0.03257 7.84% MLMG::actualBottomSolve() 36 0.03255 0.03255 0.03255 7.84% MLCGSolver::bicgstab 36 0.03223 0.03223 0.03223 7.76% MLPoisson::Fsmooth() 1440 0.02528 0.02528 0.02528 6.09% FillPatchIterator::Initialize 20 0.02456 0.02456 0.02456 5.91% Amr::writePlotFile() 1 0.02403 0.02403 0.02403 5.79% VisMF::Write(FabArray) 1 0.02263 0.02263 0.02263 5.45% FillPatchSingleLevel 20 0.02047 0.02047 0.02047 4.93% StateDataPhysBCFunct::() 20 0.0184 0.0184 0.0184 4.43% Castro::clean_state() 30 0.01798 0.01798 0.01798 4.33% FabArray::setVal() 537 0.0168 0.0168 0.0168 4.04% MLCellLinOp::apply() 500 0.01623 0.01623 0.01623 3.91% MLMG::mgVcycle_down::0 36 0.01576 0.01576 0.01576 3.79% Castro::initialize_do_advance() 5 0.01498 0.01498 0.01498 3.61% MLMG::mgVcycle_up::0 36 0.01345 0.01345 0.01345 3.24% Castro::expand_state() 5 0.01205 0.01205 0.01205 2.90% Castro::initialize_advance() 5 0.01102 0.01102 0.01102 2.65% StateData::FillBoundary(geom) 160 0.01093 0.01093 0.01093 2.63% Castro::computeTemp() 30 0.01002 0.01002 0.01002 2.41% Gravity::get_new_grav_vector() 5 0.009476 0.009476 0.009476 2.28% MLPoisson::define() 6 0.00929 0.00929 0.00929 2.24% MLCellLinOp::correctionResidual() 216 0.009133 0.009133 0.009133 2.20% MultiFab::Dot() 484 0.009069 0.009069 0.009069 2.18% Castro::do_new_sources() 5 0.008402 0.008402 0.008402 2.02% Castro::construct_old_gravity() 5 0.008084 0.008084 0.008084 1.95% Gravity::get_old_grav_vector() 5 0.008078 0.008078 0.008078 1.94% MLMG:computeResOfCorrection() 180 0.007911 0.007911 0.007911 1.90% FabArray::FillBoundary() 1766 0.007838 0.007838 0.007838 1.89% MLMG::mgVcycle_down::1 36 0.007628 0.007628 0.007628 1.84% FabArray::ParallelCopy() 380 0.007514 0.007514 0.007514 1.81% FillBoundary_nowait() 1766 0.007429 0.007429 0.007429 1.79% FabArray::ParallelCopy_nowait() 380 0.007372 0.007372 0.007372 1.77% MLMG::mgVcycle_down::2 36 0.007335 0.007335 0.007335 1.77% MLMG::mgVcycle_down::3 36 0.006999 0.006999 0.006999 1.68% MLMG::mgVcycle_down::4 36 0.006708 0.006708 0.006708 1.62% Castro::post_restart() 1 0.006623 0.006623 0.006623 1.59% MLCellLinOp::defineAuxData() 6 0.006361 0.006361 0.006361 1.53% Gravity::fill_multipole_BCs() 6 0.006289 0.006289 0.006289 1.51% Gravity::multilevel_solve_for_new_phi() 1 0.006218 0.006218 0.006218 1.50% Gravity::actual_multilevel_solve() 1 0.006202 0.006202 0.006202 1.49% Castro::enforce_min_density() 30 0.005787 0.005787 0.005787 1.39% MLMG::mgVcycle_up::2 36 0.005721 0.005721 0.005721 1.38% MLMG::addInterpCorrection() 180 0.005721 0.005721 0.005721 1.38% MLMG::mgVcycle_up::1 36 0.005653 0.005653 0.005653 1.36% amrex::average_down 180 0.005484 0.005484 0.005484 1.32% MLCGSolver::ParallelAllReduce 659 0.005477 0.005477 0.005477 1.32% CGSolver::sxay() 690 0.005452 0.005452 0.005452 1.31% MLMG::mgVcycle_up::3 36 0.005434 0.005434 0.005434 1.31% MLMG::mgVcycle_up::4 36 0.005374 0.005374 0.005374 1.29% MultiFab::LinComb() 690 0.005249 0.005249 0.005249 1.26% MLPoisson::Fapply() 500 0.004506 0.004506 0.004506 1.08% Castro::do_old_sources() 5 0.004222 0.004222 0.004222 1.02% FabArray::setDomainBndry() 20 0.003961 0.003961 0.003961 0.95% MLMG::prepareForSolve() 6 0.003399 0.003399 0.003399 0.82% MLCellLinOp::solutionResidual() 42 0.003353 0.003353 0.003353 0.81% Castro::estTimeStep() 10 0.00314 0.00314 0.00314 0.76% Castro::post_timestep() 5 0.003024 0.003024 0.003024 0.73% MLMG::computeResidual() 36 0.002784 0.002784 0.002784 0.67% MLCellLinOp::defineBC() 6 0.002763 0.002763 0.002763 0.67% BndryData::define() 6 0.002667 0.002667 0.002667 0.64% MultiFab::Xpay() 258 0.002601 0.002601 0.002601 0.63% Castro::reset_internal_energy() 30 0.002388 0.002388 0.002388 0.57% Castro::computeNewDt() 5 0.002046 0.002046 0.002046 0.49% MLMG::getGradSolution() 6 0.001717 0.001717 0.001717 0.41% MLCellLinOp::compGrad() 6 0.001714 0.001714 0.001714 0.41% Castro::construct_new_source() 25 0.001659 0.001659 0.001659 0.40% Castro::construct_new_gravity_source() 5 0.001576 0.001576 0.001576 0.38% Castro::normalize_species() 30 0.001431 0.001431 0.001431 0.34% Castro::construct_old_source() 25 0.001396 0.001396 0.001396 0.34% Castro::construct_old_gravity_source() 5 0.001342 0.001342 0.001342 0.32% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001206 0.001206 0.001206 0.29% Castro::apply_source_to_state() 10 0.00093 0.00093 0.00093 0.22% MultiFab::Saxpy() 10 0.0009236 0.0009236 0.0009236 0.22% Gravity::swapTimeLevels() 5 0.0008852 0.0008852 0.0008852 0.21% FabArrayBase::getCPC() 632 0.0008367 0.0008367 0.0008367 0.20% MLMG::ResNormInf() 42 0.0008367 0.0008367 0.0008367 0.20% MLCellLinOp::setLevelBC() 6 0.0008106 0.0008106 0.0008106 0.20% Gravity::update_max_rhs() 6 0.0007543 0.0007543 0.0007543 0.18% Castro::enforce_speed_limit() 30 0.0007129 0.0007129 0.0007129 0.17% MLPoisson::prepareForSolve() 6 0.0006879 0.0006879 0.0006879 0.17% MLCellLinOp::prepareForSolve() 6 0.0006833 0.0006833 0.0006833 0.16% FabArray::mult() 22 0.0006543 0.0006543 0.0006543 0.16% MLMG::computeMLResidual() 6 0.0005881 0.0005881 0.0005881 0.14% FabArrayBase::CPC::define() 244 0.0004263 0.0004263 0.0004263 0.10% Amr::InitAmr() 1 0.0003834 0.0003834 0.0003834 0.09% MultiFab::Copy() 6 0.0003631 0.0003631 0.0003631 0.09% FabArrayBase::getFB() 1766 0.0003618 0.0003618 0.0003618 0.09% MultiFab::min() 5 0.0001741 0.0001741 0.0001741 0.04% MLLinOp::define() 6 0.0001501 0.0001501 0.0001501 0.04% MLMG::MLResNormInf() 6 0.0001392 0.0001392 0.0001392 0.03% MultiFab::max() 6 0.0001331 0.0001331 0.0001331 0.03% MLLinOp::defineGrids() 6 0.0001229 0.0001229 0.0001229 0.03% MLMG::MLRhsNormInf() 6 0.0001049 0.0001049 0.0001049 0.03% FabArrayBase::FB::FB() 26 6.262e-05 6.262e-05 6.262e-05 0.02% Castro::create_source_corrector() 5 5.773e-05 5.773e-05 5.773e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.042e-05 3.042e-05 3.042e-05 0.01% Castro::finalize_advance() 5 2.817e-05 2.817e-05 2.817e-05 0.01% Castro::swap_state_time_levels() 5 2.67e-05 2.67e-05 2.67e-05 0.01% Amr::writeSmallPlotFile() 1 2.581e-05 2.581e-05 2.581e-05 0.01% makeSFC 30 2.241e-05 2.241e-05 2.241e-05 0.01% Castro::finalize_do_advance() 5 2.117e-05 2.117e-05 2.117e-05 0.01% Amr::initSubcycle() 1 9.438e-06 9.438e-06 9.438e-06 0.00% DistributionMapping::Distribute() 31 9.168e-06 9.168e-06 9.168e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.429e-06 4.429e-06 4.429e-06 0.00% MLMG::buildFineMask() 6 3.234e-06 3.234e-06 3.234e-06 0.00% Castro::FluxRegFineAdd() 5 2.982e-06 2.982e-06 2.982e-06 0.00% Gravity::set_mass_offset() 6 2.258e-06 2.258e-06 2.258e-06 0.00% Castro::retry_advance_ctu() 5 1.626e-06 1.626e-06 1.626e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.05e-06 1.05e-06 1.05e-06 0.00% AmrLevel::AmrLevel() 1 8.64e-07 8.64e-07 8.64e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.03-26-ge213d767baab) finalized