Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-20-gfc55c67f72d2) initialized Starting run at 08:24:43 UTC on 2022-04-15. Successfully read inputs file ... Castro git describe: 22.04-2-gb74783602 AMReX git describe: 22.04-20-gfc55c67f7 Microphysics git describe: 22.04-1-gef526ccd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.038418626 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.022702896 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.107080516 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.048737237 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.049379714 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.07328585 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.057121428 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.036137333 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052298586 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048773026 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.056864303 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.068723511 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.068817829 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.036764087 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.022476404 seconds Ending run at 08:24:44 UTC on 2022-04-15. Run time = 0.849202896 Run time without initialization = 0.726959417 Average number of zones advanced per microsecond: 3.606 Average number of zones advanced per microsecond per rank: 3.606 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8492 ... 0.8492 ... 0.8492 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2224 0.2224 0.2224 26.19% VisMF::Write(FabArray) 11 0.1496 0.1496 0.1496 17.61% MLCellLinOp::applyBC() 4433 0.09328 0.09328 0.09328 10.98% MLPoisson::Fsmooth() 3280 0.05918 0.05918 0.05918 6.97% StateDataPhysBCFunct::() 41 0.02919 0.02919 0.02919 3.44% FabArray::setVal() 1144 0.0259 0.0259 0.0259 3.05% StateData::FillBoundary(geom) 328 0.02327 0.02327 0.02327 2.74% MLCGSolver::bicgstab 82 0.02151 0.02151 0.02151 2.53% MultiFab::Dot() 1114 0.02112 0.02112 0.02112 2.49% FillBoundary_nowait() 4023 0.01764 0.01764 0.01764 2.08% FabArray::ParallelCopy_nowait() 861 0.01593 0.01593 0.01593 1.88% Castro::computeTemp() 63 0.01394 0.01394 0.01394 1.64% MultiFab::LinComb() 1586 0.01212 0.01212 0.01212 1.43% Castro::enforce_min_density() 62 0.01127 0.01127 0.01127 1.33% Gravity::fill_multipole_BCs() 11 0.0112 0.0112 0.0112 1.32% MLPoisson::Fapply() 1142 0.01046 0.01046 0.01046 1.23% MLCellLinOp::defineAuxData() 11 0.01023 0.01023 0.01023 1.20% MLMG::addInterpCorrection() 410 0.006907 0.006907 0.006907 0.81% amrex::average_down 410 0.006502 0.006502 0.006502 0.77% MultiFab::Xpay() 585 0.006011 0.006011 0.006011 0.71% FabArray::setDomainBndry() 41 0.005676 0.005676 0.005676 0.67% Castro::reset_internal_energy() 63 0.00547 0.00547 0.00547 0.64% Castro::estTimeStep() 21 0.004882 0.004882 0.004882 0.57% Castro::expand_state() 10 0.004714 0.004714 0.004714 0.56% Castro::normalize_species() 62 0.004428 0.004428 0.004428 0.52% Amr::checkPoint() 3 0.004125 0.004125 0.004125 0.49% BndryData::define() 11 0.003621 0.003621 0.003621 0.43% Castro::enforce_speed_limit() 62 0.003251 0.003251 0.003251 0.38% Castro::do_advance_ctu() 10 0.003133 0.003133 0.003133 0.37% Amr::writePlotFile() 2 0.002837 0.002837 0.002837 0.33% Castro::construct_new_gravity_source() 10 0.00255 0.00255 0.00255 0.30% Gravity::get_new_grav_vector() 11 0.002288 0.002288 0.002288 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002266 0.002266 0.002266 0.27% MLCellLinOp::compGrad() 11 0.002013 0.002013 0.002013 0.24% Castro::construct_old_gravity_source() 10 0.001993 0.001993 0.001993 0.23% MLMG::ResNormInf() 93 0.001922 0.001922 0.001922 0.23% MultiFab::Saxpy() 20 0.001812 0.001812 0.001812 0.21% Gravity::get_old_grav_vector() 10 0.00176 0.00176 0.00176 0.21% MLMG::oneIter() 82 0.001687 0.001687 0.001687 0.20% MLCellLinOp::setLevelBC() 11 0.001511 0.001511 0.001511 0.18% Gravity::actual_solve_with_mlmg() 11 0.00137 0.00137 0.00137 0.16% FabArray::mult() 43 0.001326 0.001326 0.001326 0.16% MLCellLinOp::prepareForSolve() 11 0.00132 0.00132 0.00132 0.16% Castro::initData() 1 0.001221 0.001221 0.001221 0.14% MLCellLinOp::smooth() 1640 0.00107 0.00107 0.00107 0.13% FabArrayBase::getCPC() 1323 0.0008497 0.0008497 0.0008497 0.10% FabArray::FillBoundary() 4023 0.0008438 0.0008438 0.0008438 0.10% MLMG::prepareForSolve() 11 0.000767 0.000767 0.000767 0.09% FabArrayBase::CPC::define() 454 0.0007064 0.0007064 0.0007064 0.08% FabArrayBase::getFB() 4023 0.0006671 0.0006671 0.0006671 0.08% Gravity::update_max_rhs() 11 0.0005441 0.0005441 0.0005441 0.06% MLCellLinOp::apply() 1142 0.0004757 0.0004757 0.0004757 0.06% Gravity::solve_for_phi() 10 0.0004706 0.0004706 0.0004706 0.06% MultiFab::Copy() 11 0.0004556 0.0004556 0.0004556 0.05% Amr::InitAmr() 1 0.0004514 0.0004514 0.0004514 0.05% CGSolver::sxay() 1586 0.0004417 0.0004417 0.0004417 0.05% MLMG::mgVcycle() 82 0.0004193 0.0004193 0.0004193 0.05% MLCGSolver::ParallelAllReduce 1514 0.0003362 0.0003362 0.0003362 0.04% MultiFab::min() 10 0.0003212 0.0003212 0.0003212 0.04% main() 1 0.0003095 0.0003095 0.0003095 0.04% FabArray::ParallelCopy() 861 0.0002935 0.0002935 0.0002935 0.03% MultiFab::max() 11 0.0002521 0.0002521 0.0002521 0.03% FillPatchIterator::Initialize 41 0.000248 0.000248 0.000248 0.03% Castro::construct_new_gravity() 10 0.0002411 0.0002411 0.0002411 0.03% MLCellLinOp::correctionResidual() 492 0.0002202 0.0002202 0.0002202 0.03% Amr::coarseTimeStep() 10 0.0002194 0.0002194 0.0002194 0.03% Gravity::actual_multilevel_solve() 1 0.000217 0.000217 0.000217 0.03% Amr::timeStep() 10 0.0002075 0.0002075 0.0002075 0.02% MLMG::MLRhsNormInf() 11 0.0001961 0.0001961 0.0001961 0.02% MLCellLinOp::defineBC() 11 0.0001827 0.0001827 0.0001827 0.02% MLLinOp::defineGrids() 11 0.0001826 0.0001826 0.0001826 0.02% Amr::defBaseLevel() 1 0.0001553 0.0001553 0.0001553 0.02% MLMG:computeResOfCorrection() 410 0.0001378 0.0001378 0.0001378 0.02% StateData::checkPoint() 12 0.0001331 0.0001331 0.0001331 0.02% Castro::subcycle_advance_ctu() 10 0.0001279 0.0001279 0.0001279 0.02% MLMG::actualBottomSolve() 82 0.0001105 0.0001105 0.0001105 0.01% MLMG::mgVcycle_down::0 82 9.915e-05 9.915e-05 9.915e-05 0.01% FabArrayBase::FB::FB() 56 9.179e-05 9.179e-05 9.179e-05 0.01% MLMG::mgVcycle_down::1 82 8.652e-05 8.652e-05 8.652e-05 0.01% Castro::initialize_advance() 10 8.289e-05 8.289e-05 8.289e-05 0.01% MLMG::mgVcycle_down::2 82 8.273e-05 8.273e-05 8.273e-05 0.01% MLMG::solve() 11 8.024e-05 8.024e-05 8.024e-05 0.01% MLMG::mgVcycle_down::3 82 7.973e-05 7.973e-05 7.973e-05 0.01% MLMG::mgVcycle_down::4 82 7.945e-05 7.945e-05 7.945e-05 0.01% Castro::construct_new_source() 50 7.632e-05 7.632e-05 7.632e-05 0.01% Castro::clean_state() 62 7.513e-05 7.513e-05 7.513e-05 0.01% AmrLevel::checkPoint() 3 7.411e-05 7.411e-05 7.411e-05 0.01% Castro::post_timestep() 10 6.854e-05 6.854e-05 6.854e-05 0.01% MLMG::mgVcycle_up::4 82 6.322e-05 6.322e-05 6.322e-05 0.01% Castro::finalize_advance() 10 6.235e-05 6.235e-05 6.235e-05 0.01% Castro::advance() 10 5.629e-05 5.629e-05 5.629e-05 0.01% Castro::initialize_do_advance() 10 5.556e-05 5.556e-05 5.556e-05 0.01% MLMG::mgVcycle_up::0 82 5.513e-05 5.513e-05 5.513e-05 0.01% MLMG::mgVcycle_up::3 82 5.112e-05 5.112e-05 5.112e-05 0.01% MLMG::mgVcycle_up::1 82 5.107e-05 5.107e-05 5.107e-05 0.01% MLCellLinOp::solutionResidual() 93 4.918e-05 4.918e-05 4.918e-05 0.01% MLMG::mgVcycle_up::2 82 4.896e-05 4.896e-05 4.896e-05 0.01% StateData::define() 4 4.26e-05 4.26e-05 4.26e-05 0.01% MLMG::mgVcycle_bottom 82 4.006e-05 4.006e-05 4.006e-05 0.00% Castro::finalize_do_advance() 10 3.727e-05 3.727e-05 3.727e-05 0.00% Castro::swap_state_time_levels() 10 3.683e-05 3.683e-05 3.683e-05 0.00% Castro::enforce_consistent_e() 1 3.678e-05 3.678e-05 3.678e-05 0.00% MLMG::computeResidual() 82 3.601e-05 3.601e-05 3.601e-05 0.00% MLLinOp::define() 11 2.96e-05 2.96e-05 2.96e-05 0.00% makeSFC 55 2.846e-05 2.846e-05 2.846e-05 0.00% FillPatchSingleLevel 41 2.841e-05 2.841e-05 2.841e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.733e-05 2.733e-05 2.733e-05 0.00% MLPoisson::define() 11 2.601e-05 2.601e-05 2.601e-05 0.00% Amr::writeSmallPlotFile() 1 2.466e-05 2.466e-05 2.466e-05 0.00% Castro::construct_old_source() 50 2.097e-05 2.097e-05 2.097e-05 0.00% Amr::FinalizeInit() 1 2.062e-05 2.062e-05 2.062e-05 0.00% Castro::do_new_sources() 10 1.76e-05 1.76e-05 1.76e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.666e-05 1.666e-05 1.666e-05 0.00% DistributionMapping::Distribute() 56 1.64e-05 1.64e-05 1.64e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.615e-05 1.615e-05 1.615e-05 0.00% Castro::do_old_sources() 10 1.561e-05 1.561e-05 1.561e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.459e-05 1.459e-05 1.459e-05 0.00% Castro::apply_source_to_state() 20 1.235e-05 1.235e-05 1.235e-05 0.00% Castro::construct_old_gravity() 10 1.061e-05 1.061e-05 1.061e-05 0.00% Gravity::swapTimeLevels() 10 1.016e-05 1.016e-05 1.016e-05 0.00% MLPoisson::prepareForSolve() 11 9.347e-06 9.347e-06 9.347e-06 0.00% Amr::initSubcycle() 1 9.055e-06 9.055e-06 9.055e-06 0.00% MLMG::computeMLResidual() 11 7.342e-06 7.342e-06 7.342e-06 0.00% AmrLevel::checkPointPost() 3 7.084e-06 7.084e-06 7.084e-06 0.00% MLMG::buildFineMask() 11 6.613e-06 6.613e-06 6.613e-06 0.00% Amr::InitializeInit() 1 6.456e-06 6.456e-06 6.456e-06 0.00% Castro::computeNewDt() 9 6.223e-06 6.223e-06 6.223e-06 0.00% MLMG::getGradSolution() 11 5.857e-06 5.857e-06 5.857e-06 0.00% Castro::create_source_corrector() 10 4.905e-06 4.905e-06 4.905e-06 0.00% Gravity::set_mass_offset() 11 4.86e-06 4.86e-06 4.86e-06 0.00% Castro::retry_advance_ctu() 10 4.839e-06 4.839e-06 4.839e-06 0.00% MLMG::MLResNormInf() 11 4.64e-06 4.64e-06 4.64e-06 0.00% Castro::post_init() 1 3.778e-06 3.778e-06 3.778e-06 0.00% Castro::FluxRegFineAdd() 10 3e-06 3e-06 3e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.782e-06 2.782e-06 2.782e-06 0.00% Amr::init() 1 2.773e-06 2.773e-06 2.773e-06 0.00% Castro::computeInitialDt() 2 2.684e-06 2.684e-06 2.684e-06 0.00% AmrLevel::checkPointPre() 3 2.541e-06 2.541e-06 2.541e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.198e-06 2.198e-06 2.198e-06 0.00% Castro::post_regrid() 1 1.084e-06 1.084e-06 1.084e-06 0.00% Amr::initialInit() 1 8.53e-07 8.53e-07 8.53e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8492 0.8492 0.8492 100.00% Amr::coarseTimeStep() 10 0.7043 0.7043 0.7043 82.93% Amr::timeStep() 10 0.6286 0.6286 0.6286 74.02% Castro::advance() 10 0.6209 0.6209 0.6209 73.11% Castro::subcycle_advance_ctu() 10 0.6064 0.6064 0.6064 71.40% Castro::do_advance_ctu() 10 0.6062 0.6062 0.6062 71.38% Gravity::solve_phi_with_mlmg() 11 0.3223 0.3223 0.3223 37.95% Gravity::actual_solve_with_mlmg() 11 0.3108 0.3108 0.3108 36.60% Castro::construct_new_gravity() 10 0.3063 0.3063 0.3063 36.07% MLMG::solve() 11 0.2881 0.2881 0.2881 33.93% Gravity::solve_for_phi() 10 0.275 0.275 0.275 32.38% MLMG::oneIter() 82 0.2724 0.2724 0.2724 32.07% MLMG::mgVcycle() 82 0.2707 0.2707 0.2707 31.87% Castro::construct_ctu_hydro_source() 10 0.2224 0.2224 0.2224 26.19% VisMF::Write(FabArray) 11 0.1496 0.1496 0.1496 17.61% MLCellLinOp::smooth() 1640 0.1438 0.1438 0.1438 16.94% Amr::init() 1 0.1216 0.1216 0.1216 14.32% MLCellLinOp::applyBC() 4433 0.1125 0.1125 0.1125 13.25% Amr::checkPoint() 3 0.1114 0.1114 0.1114 13.12% AmrLevel::checkPoint() 3 0.1073 0.1073 0.1073 12.64% StateData::checkPoint() 12 0.1072 0.1072 0.1072 12.63% MLMG::mgVcycle_bottom 82 0.0767 0.0767 0.0767 9.03% MLMG::actualBottomSolve() 82 0.07666 0.07666 0.07666 9.03% MLCGSolver::bicgstab 82 0.07594 0.07594 0.07594 8.94% FillPatchIterator::Initialize 41 0.06268 0.06268 0.06268 7.38% Amr::initialInit() 1 0.06041 0.06041 0.06041 7.11% MLPoisson::Fsmooth() 3280 0.05918 0.05918 0.05918 6.97% FillPatchSingleLevel 41 0.05676 0.05676 0.05676 6.68% StateDataPhysBCFunct::() 41 0.05246 0.05246 0.05246 6.18% Amr::FinalizeInit() 1 0.05134 0.05134 0.05134 6.04% Castro::post_init() 1 0.0506 0.0506 0.0506 5.96% Gravity::multilevel_solve_for_new_phi() 1 0.04802 0.04802 0.04802 5.65% Gravity::actual_multilevel_solve() 1 0.048 0.048 0.048 5.65% Amr::writePlotFile() 2 0.0453 0.0453 0.0453 5.33% MLCellLinOp::apply() 1142 0.03933 0.03933 0.03933 4.63% MLMG::mgVcycle_down::0 82 0.03836 0.03836 0.03836 4.52% Castro::clean_state() 62 0.03777 0.03777 0.03777 4.45% Gravity::get_new_grav_vector() 11 0.03353 0.03353 0.03353 3.95% MLMG::mgVcycle_up::0 82 0.03277 0.03277 0.03277 3.86% FabArray::setVal() 1144 0.0259 0.0259 0.0259 3.05% Castro::initialize_do_advance() 10 0.02432 0.02432 0.02432 2.86% StateData::FillBoundary(geom) 328 0.02327 0.02327 0.02327 2.74% MLCellLinOp::correctionResidual() 492 0.02211 0.02211 0.02211 2.60% MultiFab::Dot() 1114 0.02112 0.02112 0.02112 2.49% Castro::computeTemp() 63 0.01941 0.01941 0.01941 2.29% FabArray::FillBoundary() 4023 0.01924 0.01924 0.01924 2.27% MLMG:computeResOfCorrection() 410 0.01915 0.01915 0.01915 2.26% FillBoundary_nowait() 4023 0.0184 0.0184 0.0184 2.17% MLMG::mgVcycle_down::1 82 0.01833 0.01833 0.01833 2.16% Castro::expand_state() 10 0.01797 0.01797 0.01797 2.12% MLMG::mgVcycle_down::2 82 0.01773 0.01773 0.01773 2.09% MLPoisson::define() 11 0.01727 0.01727 0.01727 2.03% FabArray::ParallelCopy() 861 0.01711 0.01711 0.01711 2.01% MLMG::mgVcycle_down::3 82 0.0169 0.0169 0.0169 1.99% FabArray::ParallelCopy_nowait() 861 0.01681 0.01681 0.01681 1.98% MLMG::mgVcycle_down::4 82 0.01608 0.01608 0.01608 1.89% Castro::construct_old_gravity() 10 0.01578 0.01578 0.01578 1.86% Gravity::get_old_grav_vector() 10 0.01577 0.01577 0.01577 1.86% Castro::initialize_advance() 10 0.01438 0.01438 0.01438 1.69% MLMG::mgVcycle_up::2 82 0.01379 0.01379 0.01379 1.62% MLMG::mgVcycle_up::1 82 0.01354 0.01354 0.01354 1.59% MLMG::addInterpCorrection() 410 0.01335 0.01335 0.01335 1.57% MLMG::mgVcycle_up::3 82 0.0131 0.0131 0.0131 1.54% MLMG::mgVcycle_up::4 82 0.01295 0.01295 0.01295 1.52% amrex::average_down 410 0.0129 0.0129 0.0129 1.52% MLCGSolver::ParallelAllReduce 1514 0.01274 0.01274 0.01274 1.50% CGSolver::sxay() 1586 0.01256 0.01256 0.01256 1.48% MultiFab::LinComb() 1586 0.01212 0.01212 0.01212 1.43% MLCellLinOp::defineAuxData() 11 0.0119 0.0119 0.0119 1.40% Castro::enforce_min_density() 62 0.01127 0.01127 0.01127 1.33% Castro::do_new_sources() 10 0.01124 0.01124 0.01124 1.32% Gravity::fill_multipole_BCs() 11 0.0112 0.0112 0.0112 1.32% MLPoisson::Fapply() 1142 0.01046 0.01046 0.01046 1.23% Amr::InitializeInit() 1 0.00907 0.00907 0.00907 1.07% Amr::defBaseLevel() 1 0.009063 0.009063 0.009063 1.07% Castro::do_old_sources() 10 0.0082 0.0082 0.0082 0.97% MLCellLinOp::solutionResidual() 93 0.00786 0.00786 0.00786 0.93% Castro::post_timestep() 10 0.007502 0.007502 0.007502 0.88% MLMG::computeResidual() 82 0.006787 0.006787 0.006787 0.80% MultiFab::Xpay() 585 0.006011 0.006011 0.006011 0.71% FabArray::setDomainBndry() 41 0.005676 0.005676 0.005676 0.67% MLMG::prepareForSolve() 11 0.005643 0.005643 0.005643 0.66% Castro::reset_internal_energy() 63 0.00547 0.00547 0.00547 0.64% MLCellLinOp::defineBC() 11 0.005068 0.005068 0.005068 0.60% BndryData::define() 11 0.004885 0.004885 0.004885 0.58% Castro::estTimeStep() 21 0.004882 0.004882 0.004882 0.57% Castro::normalize_species() 62 0.004428 0.004428 0.004428 0.52% Castro::initData() 1 0.003372 0.003372 0.003372 0.40% Castro::enforce_speed_limit() 62 0.003251 0.003251 0.003251 0.38% Castro::construct_new_source() 50 0.002627 0.002627 0.002627 0.31% MLMG::getGradSolution() 11 0.002577 0.002577 0.002577 0.30% MLCellLinOp::compGrad() 11 0.002571 0.002571 0.002571 0.30% Castro::construct_new_gravity_source() 10 0.00255 0.00255 0.00255 0.30% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.002266 0.002266 0.002266 0.27% Castro::computeNewDt() 9 0.002173 0.002173 0.002173 0.26% Castro::construct_old_source() 50 0.002014 0.002014 0.002014 0.24% Castro::construct_old_gravity_source() 10 0.001993 0.001993 0.001993 0.23% MLMG::ResNormInf() 93 0.001922 0.001922 0.001922 0.23% Castro::apply_source_to_state() 20 0.001825 0.001825 0.001825 0.21% MultiFab::Saxpy() 20 0.001812 0.001812 0.001812 0.21% FabArrayBase::getCPC() 1323 0.001556 0.001556 0.001556 0.18% MLCellLinOp::setLevelBC() 11 0.001511 0.001511 0.001511 0.18% MLPoisson::prepareForSolve() 11 0.001329 0.001329 0.001329 0.16% FabArray::mult() 43 0.001326 0.001326 0.001326 0.16% MLCellLinOp::prepareForSolve() 11 0.00132 0.00132 0.00132 0.16% MLMG::computeMLResidual() 11 0.001117 0.001117 0.001117 0.13% Gravity::swapTimeLevels() 10 0.001036 0.001036 0.001036 0.12% Gravity::update_max_rhs() 11 0.0009382 0.0009382 0.0009382 0.11% FabArrayBase::getFB() 4023 0.0007589 0.0007589 0.0007589 0.09% FabArrayBase::CPC::define() 454 0.0007064 0.0007064 0.0007064 0.08% Castro::computeInitialDt() 2 0.0005242 0.0005242 0.0005242 0.06% Castro::post_regrid() 1 0.0005087 0.0005087 0.0005087 0.06% Amr::InitAmr() 1 0.0004605 0.0004605 0.0004605 0.05% MultiFab::Copy() 11 0.0004556 0.0004556 0.0004556 0.05% MultiFab::min() 10 0.0003212 0.0003212 0.0003212 0.04% MLLinOp::define() 11 0.0002741 0.0002741 0.0002741 0.03% MultiFab::max() 11 0.0002521 0.0002521 0.0002521 0.03% MLMG::MLResNormInf() 11 0.0002519 0.0002519 0.0002519 0.03% MLLinOp::defineGrids() 11 0.0002445 0.0002445 0.0002445 0.03% MLMG::MLRhsNormInf() 11 0.0001961 0.0001961 0.0001961 0.02% FabArrayBase::FB::FB() 56 9.179e-05 9.179e-05 9.179e-05 0.01% Castro::finalize_advance() 10 6.535e-05 6.535e-05 6.535e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.975e-05 5.975e-05 5.975e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.719e-05 5.719e-05 5.719e-05 0.01% makeSFC 55 4.36e-05 4.36e-05 4.36e-05 0.01% StateData::define() 4 4.26e-05 4.26e-05 4.26e-05 0.01% Castro::finalize_do_advance() 10 3.727e-05 3.727e-05 3.727e-05 0.00% Castro::swap_state_time_levels() 10 3.683e-05 3.683e-05 3.683e-05 0.00% Castro::enforce_consistent_e() 1 3.678e-05 3.678e-05 3.678e-05 0.00% Amr::writeSmallPlotFile() 1 2.466e-05 2.466e-05 2.466e-05 0.00% DistributionMapping::Distribute() 56 1.64e-05 1.64e-05 1.64e-05 0.00% Amr::initSubcycle() 1 9.055e-06 9.055e-06 9.055e-06 0.00% AmrLevel::checkPointPost() 3 7.084e-06 7.084e-06 7.084e-06 0.00% MLMG::buildFineMask() 11 6.613e-06 6.613e-06 6.613e-06 0.00% Castro::create_source_corrector() 10 4.905e-06 4.905e-06 4.905e-06 0.00% Gravity::set_mass_offset() 11 4.86e-06 4.86e-06 4.86e-06 0.00% Castro::retry_advance_ctu() 10 4.839e-06 4.839e-06 4.839e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.039e-06 4.039e-06 4.039e-06 0.00% Castro::FluxRegFineAdd() 10 3e-06 3e-06 3e-06 0.00% AmrLevel::checkPointPre() 3 2.541e-06 2.541e-06 2.541e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.198e-06 2.198e-06 2.198e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-20-gfc55c67f72d2) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.04-20-gfc55c67f72d2) initialized Starting run at 08:24:45 UTC on 2022-04-15. Successfully read inputs file ... Castro git describe: 22.04-2-gb74783602 AMReX git describe: 22.04-20-gfc55c67f7 Microphysics git describe: 22.04-1-gef526ccd reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.493915922 Restart time = 0.054173301 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.110883913 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048866239 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.049283041 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.052590303 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.060475815 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.047448414 seconds Ending run at 08:24:45 UTC on 2022-04-15. Run time = 0.424659065 Run time without initialization = 0.369926914 Average number of zones advanced per microsecond: 3.543 Average number of zones advanced per microsecond per rank: 3.543 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4247 ... 0.4247 ... 0.4247 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1213 0.1213 0.1213 28.56% VisMF::Read() 3 0.04275 0.04275 0.04275 10.07% MLCellLinOp::applyBC() 1946 0.03744 0.03744 0.03744 8.82% MLPoisson::Fsmooth() 1440 0.02534 0.02534 0.02534 5.97% Amr::writePlotFile() 1 0.0249 0.0249 0.0249 5.86% VisMF::Write(FabArray) 1 0.02265 0.02265 0.02265 5.33% FabArray::setVal() 537 0.01599 0.01599 0.01599 3.77% StateData::FillBoundary(geom) 160 0.01165 0.01165 0.01165 2.74% MLCGSolver::bicgstab 36 0.009228 0.009228 0.009228 2.17% MultiFab::Dot() 484 0.009052 0.009052 0.009052 2.13% FillBoundary_nowait() 1766 0.007132 0.007132 0.007132 1.68% FabArray::ParallelCopy_nowait() 380 0.007057 0.007057 0.007057 1.66% StateDataPhysBCFunct::() 20 0.00647 0.00647 0.00647 1.52% Gravity::fill_multipole_BCs() 6 0.005825 0.005825 0.005825 1.37% Castro::computeTemp() 30 0.005765 0.005765 0.005765 1.36% MLCellLinOp::defineAuxData() 6 0.005527 0.005527 0.005527 1.30% MultiFab::LinComb() 690 0.005212 0.005212 0.005212 1.23% MLPoisson::Fapply() 500 0.004552 0.004552 0.004552 1.07% Castro::expand_state() 5 0.00396 0.00396 0.00396 0.93% FabArray::setDomainBndry() 20 0.003644 0.003644 0.003644 0.86% Castro::reset_internal_energy() 30 0.003133 0.003133 0.003133 0.74% Amr::restart() 1 0.00302 0.00302 0.00302 0.71% MLMG::addInterpCorrection() 180 0.002979 0.002979 0.002979 0.70% amrex::average_down 180 0.002781 0.002781 0.002781 0.65% Castro::enforce_min_density() 30 0.002685 0.002685 0.002685 0.63% MultiFab::Xpay() 258 0.002656 0.002656 0.002656 0.63% Castro::do_advance_ctu() 5 0.00253 0.00253 0.00253 0.60% Castro::estTimeStep() 10 0.002491 0.002491 0.002491 0.59% Castro::normalize_species() 30 0.002195 0.002195 0.002195 0.52% BndryData::define() 6 0.002006 0.002006 0.002006 0.47% Castro::enforce_speed_limit() 30 0.001981 0.001981 0.001981 0.47% MLCellLinOp::compGrad() 6 0.001469 0.001469 0.001469 0.35% Castro::construct_new_gravity_source() 5 0.001434 0.001434 0.001434 0.34% Gravity::get_new_grav_vector() 5 0.001221 0.001221 0.001221 0.29% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001204 0.001204 0.001204 0.28% Castro::construct_old_gravity_source() 5 0.001012 0.001012 0.001012 0.24% MultiFab::Saxpy() 10 0.0009184 0.0009184 0.0009184 0.22% Gravity::get_old_grav_vector() 5 0.0009097 0.0009097 0.0009097 0.21% MLMG::ResNormInf() 42 0.0008621 0.0008621 0.0008621 0.20% MLCellLinOp::setLevelBC() 6 0.0008262 0.0008262 0.0008262 0.19% MLMG::oneIter() 36 0.0007504 0.0007504 0.0007504 0.18% Gravity::actual_solve_with_mlmg() 6 0.0007284 0.0007284 0.0007284 0.17% MLCellLinOp::prepareForSolve() 6 0.0006875 0.0006875 0.0006875 0.16% FabArray::mult() 22 0.0006541 0.0006541 0.0006541 0.15% Gravity::update_max_rhs() 6 0.0005208 0.0005208 0.0005208 0.12% MLCellLinOp::smooth() 720 0.0004756 0.0004756 0.0004756 0.11% FabArrayBase::CPC::define() 244 0.00046 0.00046 0.00046 0.11% FabArrayBase::getCPC() 632 0.0004385 0.0004385 0.0004385 0.10% MLMG::prepareForSolve() 6 0.0004201 0.0004201 0.0004201 0.10% FabArray::FillBoundary() 1766 0.0003951 0.0003951 0.0003951 0.09% Amr::InitAmr() 1 0.0003844 0.0003844 0.0003844 0.09% MultiFab::Copy() 6 0.000314 0.000314 0.000314 0.07% FabArrayBase::getFB() 1766 0.0003047 0.0003047 0.0003047 0.07% main() 1 0.0002817 0.0002817 0.0002817 0.07% Gravity::solve_for_phi() 5 0.0002352 0.0002352 0.0002352 0.06% Gravity::actual_multilevel_solve() 1 0.0002183 0.0002183 0.0002183 0.05% MLCellLinOp::apply() 500 0.0002043 0.0002043 0.0002043 0.05% CGSolver::sxay() 690 0.0001958 0.0001958 0.0001958 0.05% MLMG::mgVcycle() 36 0.0001846 0.0001846 0.0001846 0.04% Castro::construct_new_gravity() 5 0.0001811 0.0001811 0.0001811 0.04% MultiFab::min() 5 0.0001614 0.0001614 0.0001614 0.04% FabArray::ParallelCopy() 380 0.0001436 0.0001436 0.0001436 0.03% MLCGSolver::ParallelAllReduce 659 0.0001401 0.0001401 0.0001401 0.03% Castro::construct_new_source() 25 0.0001385 0.0001385 0.0001385 0.03% MultiFab::max() 6 0.0001331 0.0001331 0.0001331 0.03% FillPatchIterator::Initialize 20 0.0001306 0.0001306 0.0001306 0.03% Amr::coarseTimeStep() 5 0.0001201 0.0001201 0.0001201 0.03% MLCellLinOp::defineBC() 6 0.000111 0.000111 0.000111 0.03% Amr::timeStep() 5 0.0001073 0.0001073 0.0001073 0.03% MLMG::MLRhsNormInf() 6 0.0001063 0.0001063 0.0001063 0.03% MLLinOp::defineGrids() 6 9.787e-05 9.787e-05 9.787e-05 0.02% MLCellLinOp::correctionResidual() 216 9.403e-05 9.403e-05 9.403e-05 0.02% StateData::restartDoit() 4 7.935e-05 7.935e-05 7.935e-05 0.02% AmrLevel::restart() 1 7.915e-05 7.915e-05 7.915e-05 0.02% Castro::advance() 5 6.394e-05 6.394e-05 6.394e-05 0.02% FabArrayBase::FB::FB() 26 6.073e-05 6.073e-05 6.073e-05 0.01% MLMG:computeResOfCorrection() 180 6.046e-05 6.046e-05 6.046e-05 0.01% Castro::subcycle_advance_ctu() 5 5.866e-05 5.866e-05 5.866e-05 0.01% Castro::create_source_corrector() 5 5.414e-05 5.414e-05 5.414e-05 0.01% MLMG::actualBottomSolve() 36 4.976e-05 4.976e-05 4.976e-05 0.01% MLMG::mgVcycle_down::0 36 4.096e-05 4.096e-05 4.096e-05 0.01% MLMG::mgVcycle_down::1 36 3.947e-05 3.947e-05 3.947e-05 0.01% Castro::post_timestep() 5 3.778e-05 3.778e-05 3.778e-05 0.01% MLMG::solve() 6 3.756e-05 3.756e-05 3.756e-05 0.01% Castro::initialize_advance() 5 3.695e-05 3.695e-05 3.695e-05 0.01% Castro::clean_state() 30 3.663e-05 3.663e-05 3.663e-05 0.01% MLMG::mgVcycle_down::2 36 3.593e-05 3.593e-05 3.593e-05 0.01% MLMG::mgVcycle_down::3 36 3.489e-05 3.489e-05 3.489e-05 0.01% MLMG::mgVcycle_down::4 36 3.414e-05 3.414e-05 3.414e-05 0.01% Castro::do_new_sources() 5 3.413e-05 3.413e-05 3.413e-05 0.01% Castro::initialize_do_advance() 5 2.92e-05 2.92e-05 2.92e-05 0.01% MLLinOp::define() 6 2.91e-05 2.91e-05 2.91e-05 0.01% Castro::construct_old_source() 25 2.885e-05 2.885e-05 2.885e-05 0.01% Castro::finalize_advance() 5 2.788e-05 2.788e-05 2.788e-05 0.01% Amr::writeSmallPlotFile() 1 2.781e-05 2.781e-05 2.781e-05 0.01% MLMG::mgVcycle_up::4 36 2.716e-05 2.716e-05 2.716e-05 0.01% Castro::swap_state_time_levels() 5 2.661e-05 2.661e-05 2.661e-05 0.01% Castro::post_restart() 1 2.469e-05 2.469e-05 2.469e-05 0.01% MLMG::mgVcycle_up::0 36 2.4e-05 2.4e-05 2.4e-05 0.01% MLCellLinOp::solutionResidual() 42 2.351e-05 2.351e-05 2.351e-05 0.01% MLMG::mgVcycle_up::3 36 2.275e-05 2.275e-05 2.275e-05 0.01% MLMG::mgVcycle_up::2 36 2.17e-05 2.17e-05 2.17e-05 0.01% MLMG::mgVcycle_up::1 36 2.132e-05 2.132e-05 2.132e-05 0.01% Castro::finalize_do_advance() 5 2.126e-05 2.126e-05 2.126e-05 0.01% MLMG::mgVcycle_bottom 36 1.792e-05 1.792e-05 1.792e-05 0.00% MLPoisson::define() 6 1.667e-05 1.667e-05 1.667e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.637e-05 1.637e-05 1.637e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.612e-05 1.612e-05 1.612e-05 0.00% MLMG::computeResidual() 36 1.608e-05 1.608e-05 1.608e-05 0.00% FillPatchSingleLevel 20 1.551e-05 1.551e-05 1.551e-05 0.00% makeSFC 30 1.465e-05 1.465e-05 1.465e-05 0.00% DistributionMapping::Distribute() 31 1.024e-05 1.024e-05 1.024e-05 0.00% Amr::initSubcycle() 1 9.603e-06 9.603e-06 9.603e-06 0.00% Castro::do_old_sources() 5 8.51e-06 8.51e-06 8.51e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 8.171e-06 8.171e-06 8.171e-06 0.00% Castro::apply_source_to_state() 10 6.208e-06 6.208e-06 6.208e-06 0.00% Castro::construct_old_gravity() 5 5.249e-06 5.249e-06 5.249e-06 0.00% MLPoisson::prepareForSolve() 6 4.875e-06 4.875e-06 4.875e-06 0.00% Gravity::swapTimeLevels() 5 4.5e-06 4.5e-06 4.5e-06 0.00% MLMG::buildFineMask() 6 3.52e-06 3.52e-06 3.52e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.315e-06 3.315e-06 3.315e-06 0.00% MLMG::computeMLResidual() 6 3.306e-06 3.306e-06 3.306e-06 0.00% Castro::computeNewDt() 5 3.279e-06 3.279e-06 3.279e-06 0.00% MLMG::getGradSolution() 6 2.952e-06 2.952e-06 2.952e-06 0.00% Gravity::set_mass_offset() 6 2.663e-06 2.663e-06 2.663e-06 0.00% MLMG::MLResNormInf() 6 2.433e-06 2.433e-06 2.433e-06 0.00% Castro::retry_advance_ctu() 5 1.657e-06 1.657e-06 1.657e-06 0.00% Castro::FluxRegFineAdd() 5 1.432e-06 1.432e-06 1.432e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.261e-06 1.261e-06 1.261e-06 0.00% Amr::init() 1 9.09e-07 9.09e-07 9.09e-07 0.00% AmrLevel::AmrLevel() 1 7.61e-07 7.61e-07 7.61e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4247 0.4247 0.4247 100.00% Amr::coarseTimeStep() 5 0.3222 0.3222 0.3222 75.87% Amr::timeStep() 5 0.3206 0.3206 0.3206 75.49% Castro::advance() 5 0.3176 0.3176 0.3176 74.78% Castro::subcycle_advance_ctu() 5 0.3066 0.3066 0.3066 72.20% Castro::do_advance_ctu() 5 0.3066 0.3066 0.3066 72.18% Castro::construct_new_gravity() 5 0.144 0.144 0.144 33.90% Gravity::solve_phi_with_mlmg() 6 0.1406 0.1406 0.1406 33.12% Gravity::solve_for_phi() 5 0.1351 0.1351 0.1351 31.81% Gravity::actual_solve_with_mlmg() 6 0.1347 0.1347 0.1347 31.71% MLMG::solve() 6 0.1219 0.1219 0.1219 28.70% Castro::construct_ctu_hydro_source() 5 0.1213 0.1213 0.1213 28.56% MLMG::oneIter() 36 0.1141 0.1141 0.1141 26.86% MLMG::mgVcycle() 36 0.1133 0.1133 0.1133 26.69% MLCellLinOp::smooth() 720 0.05929 0.05929 0.05929 13.96% Amr::init() 1 0.05421 0.05421 0.05421 12.76% Amr::restart() 1 0.05421 0.05421 0.05421 12.76% Amr::writePlotFile() 1 0.04755 0.04755 0.04755 11.20% MLCellLinOp::applyBC() 1946 0.04533 0.04533 0.04533 10.67% AmrLevel::restart() 1 0.04358 0.04358 0.04358 10.26% StateData::restartDoit() 4 0.0435 0.0435 0.0435 10.24% VisMF::Read() 3 0.04275 0.04275 0.04275 10.07% MLMG::mgVcycle_bottom 36 0.03264 0.03264 0.03264 7.69% MLMG::actualBottomSolve() 36 0.03262 0.03262 0.03262 7.68% MLCGSolver::bicgstab 36 0.03231 0.03231 0.03231 7.61% MLPoisson::Fsmooth() 1440 0.02534 0.02534 0.02534 5.97% FillPatchIterator::Initialize 20 0.02401 0.02401 0.02401 5.65% VisMF::Write(FabArray) 1 0.02265 0.02265 0.02265 5.33% FillPatchSingleLevel 20 0.02023 0.02023 0.02023 4.76% StateDataPhysBCFunct::() 20 0.01812 0.01812 0.01812 4.27% MLCellLinOp::apply() 500 0.01633 0.01633 0.01633 3.85% FabArray::setVal() 537 0.01599 0.01599 0.01599 3.77% MLMG::mgVcycle_down::0 36 0.01588 0.01588 0.01588 3.74% Castro::clean_state() 30 0.0158 0.0158 0.0158 3.72% Castro::initialize_do_advance() 5 0.01365 0.01365 0.01365 3.21% MLMG::mgVcycle_up::0 36 0.01346 0.01346 0.01346 3.17% StateData::FillBoundary(geom) 160 0.01165 0.01165 0.01165 2.74% Castro::expand_state() 5 0.01148 0.01148 0.01148 2.70% Castro::initialize_advance() 5 0.01086 0.01086 0.01086 2.56% MLPoisson::define() 6 0.009499 0.009499 0.009499 2.24% MLCellLinOp::correctionResidual() 216 0.009271 0.009271 0.009271 2.18% MultiFab::Dot() 484 0.009052 0.009052 0.009052 2.13% Castro::computeTemp() 30 0.008899 0.008899 0.008899 2.10% Gravity::get_new_grav_vector() 5 0.008703 0.008703 0.008703 2.05% MLMG:computeResOfCorrection() 180 0.008036 0.008036 0.008036 1.89% FabArray::FillBoundary() 1766 0.007893 0.007893 0.007893 1.86% Castro::construct_old_gravity() 5 0.00773 0.00773 0.00773 1.82% Gravity::get_old_grav_vector() 5 0.007724 0.007724 0.007724 1.82% MLMG::mgVcycle_down::1 36 0.007709 0.007709 0.007709 1.82% FabArray::ParallelCopy() 380 0.007623 0.007623 0.007623 1.79% Castro::do_new_sources() 5 0.00755 0.00755 0.00755 1.78% FillBoundary_nowait() 1766 0.007498 0.007498 0.007498 1.77% FabArray::ParallelCopy_nowait() 380 0.007479 0.007479 0.007479 1.76% MLMG::mgVcycle_down::2 36 0.007393 0.007393 0.007393 1.74% MLMG::mgVcycle_down::3 36 0.007061 0.007061 0.007061 1.66% MLMG::mgVcycle_down::4 36 0.006719 0.006719 0.006719 1.58% MLCellLinOp::defineAuxData() 6 0.006489 0.006489 0.006489 1.53% Castro::post_restart() 1 0.006401 0.006401 0.006401 1.51% Gravity::multilevel_solve_for_new_phi() 1 0.006023 0.006023 0.006023 1.42% Gravity::actual_multilevel_solve() 1 0.006007 0.006007 0.006007 1.41% Gravity::fill_multipole_BCs() 6 0.005825 0.005825 0.005825 1.37% MLMG::mgVcycle_up::2 36 0.005745 0.005745 0.005745 1.35% MLMG::addInterpCorrection() 180 0.005737 0.005737 0.005737 1.35% MLMG::mgVcycle_up::1 36 0.005673 0.005673 0.005673 1.34% amrex::average_down 180 0.005547 0.005547 0.005547 1.31% MLCGSolver::ParallelAllReduce 659 0.005491 0.005491 0.005491 1.29% MLMG::mgVcycle_up::3 36 0.005445 0.005445 0.005445 1.28% MLMG::mgVcycle_up::4 36 0.005423 0.005423 0.005423 1.28% CGSolver::sxay() 690 0.005408 0.005408 0.005408 1.27% MultiFab::LinComb() 690 0.005212 0.005212 0.005212 1.23% MLPoisson::Fapply() 500 0.004552 0.004552 0.004552 1.07% FabArray::setDomainBndry() 20 0.003644 0.003644 0.003644 0.86% MLMG::prepareForSolve() 6 0.003381 0.003381 0.003381 0.80% MLCellLinOp::solutionResidual() 42 0.003379 0.003379 0.003379 0.80% Castro::do_old_sources() 5 0.003196 0.003196 0.003196 0.75% Castro::reset_internal_energy() 30 0.003133 0.003133 0.003133 0.74% Castro::post_timestep() 5 0.002905 0.002905 0.002905 0.68% MLCellLinOp::defineBC() 6 0.002834 0.002834 0.002834 0.67% MLMG::computeResidual() 36 0.002798 0.002798 0.002798 0.66% BndryData::define() 6 0.002723 0.002723 0.002723 0.64% Castro::enforce_min_density() 30 0.002685 0.002685 0.002685 0.63% MultiFab::Xpay() 258 0.002656 0.002656 0.002656 0.63% Castro::estTimeStep() 10 0.002491 0.002491 0.002491 0.59% Castro::normalize_species() 30 0.002195 0.002195 0.002195 0.52% Castro::enforce_speed_limit() 30 0.001981 0.001981 0.001981 0.47% MLMG::getGradSolution() 6 0.001759 0.001759 0.001759 0.41% MLCellLinOp::compGrad() 6 0.001756 0.001756 0.001756 0.41% Castro::construct_new_source() 25 0.001572 0.001572 0.001572 0.37% Castro::computeNewDt() 5 0.001507 0.001507 0.001507 0.35% Castro::construct_new_gravity_source() 5 0.001434 0.001434 0.001434 0.34% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001204 0.001204 0.001204 0.28% Castro::construct_old_source() 25 0.001041 0.001041 0.001041 0.25% Castro::construct_old_gravity_source() 5 0.001012 0.001012 0.001012 0.24% Castro::apply_source_to_state() 10 0.0009246 0.0009246 0.0009246 0.22% MultiFab::Saxpy() 10 0.0009184 0.0009184 0.0009184 0.22% FabArrayBase::getCPC() 632 0.0008985 0.0008985 0.0008985 0.21% Gravity::swapTimeLevels() 5 0.0008891 0.0008891 0.0008891 0.21% MLMG::ResNormInf() 42 0.0008621 0.0008621 0.0008621 0.20% MLCellLinOp::setLevelBC() 6 0.0008262 0.0008262 0.0008262 0.19% Gravity::update_max_rhs() 6 0.0007316 0.0007316 0.0007316 0.17% MLPoisson::prepareForSolve() 6 0.0006924 0.0006924 0.0006924 0.16% MLCellLinOp::prepareForSolve() 6 0.0006875 0.0006875 0.0006875 0.16% FabArray::mult() 22 0.0006541 0.0006541 0.0006541 0.15% MLMG::computeMLResidual() 6 0.0006 0.0006 0.0006 0.14% FabArrayBase::CPC::define() 244 0.00046 0.00046 0.00046 0.11% Amr::InitAmr() 1 0.000394 0.000394 0.000394 0.09% FabArrayBase::getFB() 1766 0.0003655 0.0003655 0.0003655 0.09% MultiFab::Copy() 6 0.000314 0.000314 0.000314 0.07% MultiFab::min() 5 0.0001614 0.0001614 0.0001614 0.04% MLLinOp::define() 6 0.0001597 0.0001597 0.0001597 0.04% MLMG::MLResNormInf() 6 0.0001347 0.0001347 0.0001347 0.03% MultiFab::max() 6 0.0001331 0.0001331 0.0001331 0.03% MLLinOp::defineGrids() 6 0.0001306 0.0001306 0.0001306 0.03% MLMG::MLRhsNormInf() 6 0.0001063 0.0001063 0.0001063 0.03% FabArrayBase::FB::FB() 26 6.073e-05 6.073e-05 6.073e-05 0.01% Castro::create_source_corrector() 5 5.414e-05 5.414e-05 5.414e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 3.144e-05 3.144e-05 3.144e-05 0.01% Castro::finalize_advance() 5 2.931e-05 2.931e-05 2.931e-05 0.01% Amr::writeSmallPlotFile() 1 2.781e-05 2.781e-05 2.781e-05 0.01% Castro::swap_state_time_levels() 5 2.661e-05 2.661e-05 2.661e-05 0.01% makeSFC 30 2.326e-05 2.326e-05 2.326e-05 0.01% Castro::finalize_do_advance() 5 2.126e-05 2.126e-05 2.126e-05 0.01% DistributionMapping::Distribute() 31 1.024e-05 1.024e-05 1.024e-05 0.00% Amr::initSubcycle() 1 9.603e-06 9.603e-06 9.603e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.933e-06 4.933e-06 4.933e-06 0.00% MLMG::buildFineMask() 6 3.52e-06 3.52e-06 3.52e-06 0.00% Gravity::set_mass_offset() 6 2.663e-06 2.663e-06 2.663e-06 0.00% Castro::retry_advance_ctu() 5 1.657e-06 1.657e-06 1.657e-06 0.00% Castro::FluxRegFineAdd() 5 1.432e-06 1.432e-06 1.432e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.261e-06 1.261e-06 1.261e-06 0.00% AmrLevel::AmrLevel() 1 7.61e-07 7.61e-07 7.61e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 10619 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.04-20-gfc55c67f72d2) finalized