Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.06-51-g8fb23ec17a58) initialized Starting run at 08:26:12 UTC on 2022-07-01. Successfully read inputs file ... Castro git describe: 22.06-14-gda0f2312b AMReX git describe: 22.06-51-g8fb23ec17 Microphysics git describe: 22.06-7-ga6a75714 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.043486119 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025280957 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.047882841 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051475641 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.058561883 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.0513761 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.066243311 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.040640853 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.08050151 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.062277398 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.05308948 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.05244981 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.065584646 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.040992281 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.025246955 seconds Ending run at 08:26:13 UTC on 2022-07-01. Run time = 0.818409664 Run time without initialization = 0.696960239 Average number of zones advanced per microsecond: 3.761 Average number of zones advanced per microsecond per rank: 3.761 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8184 ... 0.8184 ... 0.8184 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1991 0.1991 0.1991 24.33% VisMF::Write(FabArray) 11 0.1688 0.1688 0.1688 20.63% MLCellLinOp::applyBC() 4433 0.08143 0.08143 0.08143 9.95% MLPoisson::Fsmooth() 3280 0.06426 0.06426 0.06426 7.85% StateData::FillBoundary(geom) 328 0.02456 0.02456 0.02456 3.00% MLCGSolver::bicgstab 82 0.02444 0.02444 0.02444 2.99% MultiFab::Dot() 1114 0.02274 0.02274 0.02274 2.78% StateDataPhysBCFunct::() 41 0.01627 0.01627 0.01627 1.99% Castro::computeTemp() 63 0.01464 0.01464 0.01464 1.79% MultiFab::LinComb() 1586 0.01459 0.01459 0.01459 1.78% FabArray::setVal() 1144 0.01446 0.01446 0.01446 1.77% FillBoundary_nowait() 4023 0.01435 0.01435 0.01435 1.75% FabArray::ParallelCopy_nowait() 861 0.01346 0.01346 0.01346 1.64% Castro::normalize_species() 62 0.01314 0.01314 0.01314 1.61% MLCellLinOp::defineAuxData() 11 0.01191 0.01191 0.01191 1.46% MLPoisson::Fapply() 1142 0.01183 0.01183 0.01183 1.45% Gravity::fill_multipole_BCs() 11 0.00828 0.00828 0.00828 1.01% Castro::enforce_min_density() 62 0.007845 0.007845 0.007845 0.96% MLMG::addInterpCorrection() 410 0.007625 0.007625 0.007625 0.93% amrex::average_down 410 0.007047 0.007047 0.007047 0.86% MultiFab::Xpay() 585 0.006664 0.006664 0.006664 0.81% Castro::estTimeStep() 21 0.006441 0.006441 0.006441 0.79% Castro::do_advance_ctu() 10 0.004843 0.004843 0.004843 0.59% Amr::checkPoint() 3 0.004324 0.004324 0.004324 0.53% BndryData::define() 11 0.003982 0.003982 0.003982 0.49% Castro::reset_internal_energy(MultiFab) 63 0.003849 0.003849 0.003849 0.47% Castro::construct_new_gravity_source() 10 0.003145 0.003145 0.003145 0.38% Amr::writePlotFile() 2 0.002553 0.002553 0.002553 0.31% Castro::construct_old_gravity_source() 10 0.00251 0.00251 0.00251 0.31% Gravity::get_new_grav_vector() 11 0.001941 0.001941 0.001941 0.24% MLMG::ResNormInf() 93 0.001926 0.001926 0.001926 0.24% MultiFab::Saxpy() 20 0.001807 0.001807 0.001807 0.22% Gravity::get_old_grav_vector() 10 0.001749 0.001749 0.001749 0.21% Castro::expand_state() 10 0.00173 0.00173 0.00173 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001713 0.001713 0.001713 0.21% MLMG::oneIter() 82 0.001674 0.001674 0.001674 0.20% MLCellLinOp::setLevelBC() 11 0.00155 0.00155 0.00155 0.19% Castro::reset_internal_energy(Fab) 504 0.001548 0.001548 0.001548 0.19% Gravity::actual_solve_with_mlmg() 11 0.001353 0.001353 0.001353 0.17% FabArray::mult() 43 0.001332 0.001332 0.001332 0.16% FabArray::setDomainBndry() 41 0.001324 0.001324 0.001324 0.16% MLCellLinOp::prepareForSolve() 11 0.001203 0.001203 0.001203 0.15% Castro::initData() 1 0.001202 0.001202 0.001202 0.15% MultiFab::contains_nan() 20 0.00118 0.00118 0.00118 0.14% MLCellLinOp::smooth() 1640 0.001111 0.001111 0.001111 0.14% Castro::enforce_speed_limit() 62 0.001092 0.001092 0.001092 0.13% MLMG::prepareForSolve() 11 0.001057 0.001057 0.001057 0.13% MLCellLinOp::compGrad() 11 0.0009173 0.0009173 0.0009173 0.11% FabArray::FillBoundary() 4023 0.0008018 0.0008018 0.0008018 0.10% FabArrayBase::getCPC() 1323 0.0007893 0.0007893 0.0007893 0.10% FabArrayBase::CPC::define() 454 0.0006664 0.0006664 0.0006664 0.08% Castro::finalize_advance() 10 0.0006645 0.0006645 0.0006645 0.08% FabArrayBase::getFB() 4023 0.000617 0.000617 0.000617 0.08% Gravity::solve_for_phi() 10 0.0004593 0.0004593 0.0004593 0.06% Amr::InitAmr() 1 0.0004583 0.0004583 0.0004583 0.06% MLCellLinOp::apply() 1142 0.0004466 0.0004466 0.0004466 0.05% Gravity::update_max_rhs() 11 0.0004154 0.0004154 0.0004154 0.05% CGSolver::sxay() 1586 0.0004033 0.0004033 0.0004033 0.05% Amr::coarseTimeStep() 10 0.0003194 0.0003194 0.0003194 0.04% FillPatchIterator::Initialize 41 0.0003192 0.0003192 0.0003192 0.04% MLCGSolver::ParallelAllReduce 1514 0.0003041 0.0003041 0.0003041 0.04% MLCellLinOp::defineBC() 11 0.0002819 0.0002819 0.0002819 0.03% FabArray::ParallelCopy() 861 0.0002759 0.0002759 0.0002759 0.03% main() 1 0.000263 0.000263 0.000263 0.03% MultiFab::Copy() 11 0.0002594 0.0002594 0.0002594 0.03% MultiFab::max() 11 0.0002549 0.0002549 0.0002549 0.03% MLCellLinOp::correctionResidual() 492 0.0002194 0.0002194 0.0002194 0.03% Castro::construct_new_gravity() 10 0.0002055 0.0002055 0.0002055 0.03% MLMG::MLRhsNormInf() 11 0.0002022 0.0002022 0.0002022 0.02% Castro::subcycle_advance_ctu() 10 0.0001958 0.0001958 0.0001958 0.02% MLMG::mgVcycle() 82 0.0001924 0.0001924 0.0001924 0.02% Amr::timeStep() 10 0.000191 0.000191 0.000191 0.02% MLMG:computeResOfCorrection() 410 0.0001384 0.0001384 0.0001384 0.02% StateData::checkPoint() 12 0.000128 0.000128 0.000128 0.02% MLLinOp::defineGrids() 11 0.0001273 0.0001273 0.0001273 0.02% MLMG::actualBottomSolve() 82 0.0001084 0.0001084 0.0001084 0.01% MLMG::mgVcycle_down::0 82 0.0001009 0.0001009 0.0001009 0.01% Castro::Castro() 1 9.131e-05 9.131e-05 9.131e-05 0.01% MLMG::mgVcycle_down::1 82 8.835e-05 8.835e-05 8.835e-05 0.01% FabArrayBase::FB::FB() 56 8.652e-05 8.652e-05 8.652e-05 0.01% Castro::initialize_advance() 10 8.365e-05 8.365e-05 8.365e-05 0.01% MLMG::mgVcycle_down::2 82 8.049e-05 8.049e-05 8.049e-05 0.01% Castro::clean_state() 62 7.585e-05 7.585e-05 7.585e-05 0.01% MLMG::solve() 11 7.575e-05 7.575e-05 7.575e-05 0.01% MLMG::mgVcycle_down::3 82 7.503e-05 7.503e-05 7.503e-05 0.01% MLMG::mgVcycle_down::4 82 7.488e-05 7.488e-05 7.488e-05 0.01% AmrLevel::checkPoint() 3 7.341e-05 7.341e-05 7.341e-05 0.01% Castro::initialize_do_advance() 10 6.564e-05 6.564e-05 6.564e-05 0.01% Castro::advance() 10 6.234e-05 6.234e-05 6.234e-05 0.01% MLMG::mgVcycle_up::4 82 6.069e-05 6.069e-05 6.069e-05 0.01% MLMG::mgVcycle_up::0 82 5.512e-05 5.512e-05 5.512e-05 0.01% MLMG::mgVcycle_up::1 82 5.365e-05 5.365e-05 5.365e-05 0.01% MLMG::mgVcycle_up::3 82 5.342e-05 5.342e-05 5.342e-05 0.01% MLMG::mgVcycle_up::2 82 5.342e-05 5.342e-05 5.342e-05 0.01% MLCellLinOp::solutionResidual() 93 5.168e-05 5.168e-05 5.168e-05 0.01% MLMG::mgVcycle_bottom 82 4.559e-05 4.559e-05 4.559e-05 0.01% Castro::swap_state_time_levels() 10 3.959e-05 3.959e-05 3.959e-05 0.00% StateData::define() 4 3.693e-05 3.693e-05 3.693e-05 0.00% Castro::finalize_do_advance() 10 3.667e-05 3.667e-05 3.667e-05 0.00% Castro::construct_new_source() 50 3.52e-05 3.52e-05 3.52e-05 0.00% Castro::enforce_consistent_e() 1 3.407e-05 3.407e-05 3.407e-05 0.00% Gravity::actual_multilevel_solve() 1 3.198e-05 3.198e-05 3.198e-05 0.00% MLMG::computeResidual() 82 3.097e-05 3.097e-05 3.097e-05 0.00% FillPatchSingleLevel 41 2.86e-05 2.86e-05 2.86e-05 0.00% Amr::writeSmallPlotFile() 1 2.682e-05 2.682e-05 2.682e-05 0.00% makeSFC 55 2.617e-05 2.617e-05 2.617e-05 0.00% Castro::initMFs() 1 2.606e-05 2.606e-05 2.606e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.604e-05 2.604e-05 2.604e-05 0.00% MLLinOp::define() 11 2.551e-05 2.551e-05 2.551e-05 0.00% Amr::defBaseLevel() 1 2.444e-05 2.444e-05 2.444e-05 0.00% Castro::buildMetrics() 1 2.312e-05 2.312e-05 2.312e-05 0.00% MLPoisson::define() 11 2.219e-05 2.219e-05 2.219e-05 0.00% Amr::FinalizeInit() 1 2.025e-05 2.025e-05 2.025e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.959e-05 1.959e-05 1.959e-05 0.00% Castro::do_new_sources() 10 1.754e-05 1.754e-05 1.754e-05 0.00% Castro::construct_old_source() 50 1.617e-05 1.617e-05 1.617e-05 0.00% Castro::do_old_sources() 10 1.617e-05 1.617e-05 1.617e-05 0.00% DistributionMapping::Distribute() 56 1.392e-05 1.392e-05 1.392e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.335e-05 1.335e-05 1.335e-05 0.00% Castro::check_for_nan() 20 1.28e-05 1.28e-05 1.28e-05 0.00% Castro::apply_source_to_state() 20 1.133e-05 1.133e-05 1.133e-05 0.00% Castro::construct_old_gravity() 10 1.025e-05 1.025e-05 1.025e-05 0.00% Gravity::swapTimeLevels() 10 9.218e-06 9.218e-06 9.218e-06 0.00% Amr::initSubcycle() 1 8.655e-06 8.655e-06 8.655e-06 0.00% MLPoisson::prepareForSolve() 11 8.277e-06 8.277e-06 8.277e-06 0.00% Castro::post_timestep() 10 8.075e-06 8.075e-06 8.075e-06 0.00% Castro::computeNewDt() 9 7.28e-06 7.28e-06 7.28e-06 0.00% MLMG::computeMLResidual() 11 6.689e-06 6.689e-06 6.689e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.526e-06 6.526e-06 6.526e-06 0.00% Amr::InitializeInit() 1 6.502e-06 6.502e-06 6.502e-06 0.00% MLMG::getGradSolution() 11 6.209e-06 6.209e-06 6.209e-06 0.00% Castro::create_source_corrector() 10 5.17e-06 5.17e-06 5.17e-06 0.00% MLMG::MLResNormInf() 11 4.825e-06 4.825e-06 4.825e-06 0.00% MLMG::buildFineMask() 11 4.671e-06 4.671e-06 4.671e-06 0.00% AmrLevel::checkPointPost() 3 4.307e-06 4.307e-06 4.307e-06 0.00% Castro::post_init() 1 4.006e-06 4.006e-06 4.006e-06 0.00% Gravity::set_mass_offset() 11 3.771e-06 3.771e-06 3.771e-06 0.00% Castro::retry_advance_ctu() 10 3.771e-06 3.771e-06 3.771e-06 0.00% Castro::FluxRegCrseInit 10 3.074e-06 3.074e-06 3.074e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.05e-06 3.05e-06 3.05e-06 0.00% Castro::computeInitialDt() 2 2.694e-06 2.694e-06 2.694e-06 0.00% Amr::init() 1 2.408e-06 2.408e-06 2.408e-06 0.00% Castro::FluxRegFineAdd() 10 2.374e-06 2.374e-06 2.374e-06 0.00% AmrLevel::checkPointPre() 3 2.179e-06 2.179e-06 2.179e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.946e-06 1.946e-06 1.946e-06 0.00% Amr::initialInit() 1 1.469e-06 1.469e-06 1.469e-06 0.00% Castro::post_regrid() 1 1.023e-06 1.023e-06 1.023e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8184 0.8184 0.8184 100.00% Amr::coarseTimeStep() 10 0.6715 0.6715 0.6715 82.04% Amr::timeStep() 10 0.5852 0.5852 0.5852 71.51% Castro::advance() 10 0.5785 0.5785 0.5785 70.68% Castro::subcycle_advance_ctu() 10 0.5671 0.5671 0.5671 69.29% Castro::do_advance_ctu() 10 0.5669 0.5669 0.5669 69.26% Gravity::solve_phi_with_mlmg() 11 0.318 0.318 0.318 38.85% Gravity::actual_solve_with_mlmg() 11 0.3095 0.3095 0.3095 37.81% Castro::construct_new_gravity() 10 0.2941 0.2941 0.2941 35.93% MLMG::solve() 11 0.2864 0.2864 0.2864 35.00% Gravity::solve_for_phi() 10 0.2736 0.2736 0.2736 33.43% MLMG::oneIter() 82 0.2717 0.2717 0.2717 33.20% MLMG::mgVcycle() 82 0.27 0.27 0.27 32.99% Castro::construct_ctu_hydro_source() 10 0.1991 0.1991 0.1991 24.33% VisMF::Write(FabArray) 11 0.1688 0.1688 0.1688 20.63% MLCellLinOp::smooth() 1640 0.1378 0.1378 0.1378 16.83% Amr::checkPoint() 3 0.1253 0.1253 0.1253 15.30% AmrLevel::checkPoint() 3 0.1209 0.1209 0.1209 14.78% Amr::init() 1 0.1209 0.1209 0.1209 14.77% StateData::checkPoint() 12 0.1209 0.1209 0.1209 14.77% MLCellLinOp::applyBC() 4433 0.09729 0.09729 0.09729 11.89% MLMG::mgVcycle_bottom 82 0.08338 0.08338 0.08338 10.19% MLMG::actualBottomSolve() 82 0.08334 0.08334 0.08334 10.18% MLCGSolver::bicgstab 82 0.08249 0.08249 0.08249 10.08% MLPoisson::Fsmooth() 3280 0.06426 0.06426 0.06426 7.85% Amr::initialInit() 1 0.05199 0.05199 0.05199 6.35% Amr::writePlotFile() 2 0.05066 0.05066 0.05066 6.19% Amr::FinalizeInit() 1 0.04813 0.04813 0.04813 5.88% Castro::post_init() 1 0.04682 0.04682 0.04682 5.72% FillPatchIterator::Initialize 41 0.0465 0.0465 0.0465 5.68% Gravity::multilevel_solve_for_new_phi() 1 0.04489 0.04489 0.04489 5.48% Gravity::actual_multilevel_solve() 1 0.04487 0.04487 0.04487 5.48% FillPatchSingleLevel 41 0.04485 0.04485 0.04485 5.48% Castro::clean_state() 62 0.04155 0.04155 0.04155 5.08% StateDataPhysBCFunct::() 41 0.04084 0.04084 0.04084 4.99% MLCellLinOp::apply() 1142 0.03668 0.03668 0.03668 4.48% MLMG::mgVcycle_down::0 82 0.03584 0.03584 0.03584 4.38% MLMG::mgVcycle_up::0 82 0.03075 0.03075 0.03075 3.76% StateData::FillBoundary(geom) 328 0.02456 0.02456 0.02456 3.00% MultiFab::Dot() 1114 0.02274 0.02274 0.02274 2.78% Gravity::get_new_grav_vector() 11 0.0221 0.0221 0.0221 2.70% MLCellLinOp::correctionResidual() 492 0.02152 0.02152 0.02152 2.63% Castro::computeTemp() 63 0.02004 0.02004 0.02004 2.45% Castro::initialize_do_advance() 10 0.01879 0.01879 0.01879 2.30% MLPoisson::define() 11 0.01874 0.01874 0.01874 2.29% MLMG:computeResOfCorrection() 410 0.01857 0.01857 0.01857 2.27% MLMG::mgVcycle_down::1 82 0.01796 0.01796 0.01796 2.19% MLMG::mgVcycle_down::2 82 0.01753 0.01753 0.01753 2.14% MLMG::mgVcycle_down::3 82 0.01661 0.01661 0.01661 2.03% FabArray::FillBoundary() 4023 0.01586 0.01586 0.01586 1.94% MLMG::mgVcycle_down::4 82 0.01584 0.01584 0.01584 1.94% FillBoundary_nowait() 4023 0.01505 0.01505 0.01505 1.84% CGSolver::sxay() 1586 0.01499 0.01499 0.01499 1.83% Castro::construct_old_gravity() 10 0.01469 0.01469 0.01469 1.80% Gravity::get_old_grav_vector() 10 0.01468 0.01468 0.01468 1.79% MultiFab::LinComb() 1586 0.01459 0.01459 0.01459 1.78% FabArray::ParallelCopy() 861 0.01456 0.01456 0.01456 1.78% FabArray::setVal() 1144 0.01446 0.01446 0.01446 1.77% FabArray::ParallelCopy_nowait() 861 0.01429 0.01429 0.01429 1.75% MLCGSolver::ParallelAllReduce 1514 0.01355 0.01355 0.01355 1.66% MLMG::mgVcycle_up::2 82 0.01344 0.01344 0.01344 1.64% MLCellLinOp::defineAuxData() 11 0.01327 0.01327 0.01327 1.62% MLMG::mgVcycle_up::1 82 0.01324 0.01324 0.01324 1.62% Castro::normalize_species() 62 0.01314 0.01314 0.01314 1.61% MLMG::addInterpCorrection() 410 0.01291 0.01291 0.01291 1.58% MLMG::mgVcycle_up::3 82 0.01268 0.01268 0.01268 1.55% MLMG::mgVcycle_up::4 82 0.01257 0.01257 0.01257 1.54% amrex::average_down 410 0.01233 0.01233 0.01233 1.51% MLPoisson::Fapply() 1142 0.01183 0.01183 0.01183 1.45% Castro::do_new_sources() 10 0.01136 0.01136 0.01136 1.39% Castro::expand_state() 10 0.01109 0.01109 0.01109 1.36% Castro::initialize_advance() 10 0.01068 0.01068 0.01068 1.30% Castro::do_old_sources() 10 0.009635 0.009635 0.009635 1.18% Gravity::fill_multipole_BCs() 11 0.00828 0.00828 0.00828 1.01% Castro::enforce_min_density() 62 0.007845 0.007845 0.007845 0.96% MLCellLinOp::solutionResidual() 93 0.00724 0.00724 0.00724 0.88% MultiFab::Xpay() 585 0.006664 0.006664 0.006664 0.81% Castro::post_timestep() 10 0.006547 0.006547 0.006547 0.80% Castro::estTimeStep() 21 0.006441 0.006441 0.006441 0.79% MLMG::computeResidual() 82 0.006236 0.006236 0.006236 0.76% Castro::reset_internal_energy(MultiFab) 63 0.005397 0.005397 0.005397 0.66% MLCellLinOp::defineBC() 11 0.00524 0.00524 0.00524 0.64% MLMG::prepareForSolve() 11 0.005224 0.005224 0.005224 0.64% BndryData::define() 11 0.004958 0.004958 0.004958 0.61% Amr::InitializeInit() 1 0.003865 0.003865 0.003865 0.47% Amr::defBaseLevel() 1 0.003858 0.003858 0.003858 0.47% Castro::computeNewDt() 9 0.003725 0.003725 0.003725 0.46% Castro::initData() 1 0.003336 0.003336 0.003336 0.41% Castro::construct_new_source() 50 0.00318 0.00318 0.00318 0.39% Castro::construct_new_gravity_source() 10 0.003145 0.003145 0.003145 0.38% Castro::construct_old_source() 50 0.002526 0.002526 0.002526 0.31% Castro::construct_old_gravity_source() 10 0.00251 0.00251 0.00251 0.31% MLMG::ResNormInf() 93 0.001926 0.001926 0.001926 0.24% Castro::apply_source_to_state() 20 0.001818 0.001818 0.001818 0.22% MultiFab::Saxpy() 20 0.001807 0.001807 0.001807 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001713 0.001713 0.001713 0.21% MLCellLinOp::setLevelBC() 11 0.00155 0.00155 0.00155 0.19% Castro::reset_internal_energy(Fab) 504 0.001548 0.001548 0.001548 0.19% FabArrayBase::getCPC() 1323 0.001456 0.001456 0.001456 0.18% MLMG::getGradSolution() 11 0.001411 0.001411 0.001411 0.17% MLCellLinOp::compGrad() 11 0.001404 0.001404 0.001404 0.17% FabArray::mult() 43 0.001332 0.001332 0.001332 0.16% FabArray::setDomainBndry() 41 0.001324 0.001324 0.001324 0.16% MLPoisson::prepareForSolve() 11 0.001211 0.001211 0.001211 0.15% MLCellLinOp::prepareForSolve() 11 0.001203 0.001203 0.001203 0.15% Castro::check_for_nan() 20 0.001193 0.001193 0.001193 0.15% MultiFab::contains_nan() 20 0.00118 0.00118 0.00118 0.14% Castro::enforce_speed_limit() 62 0.001092 0.001092 0.001092 0.13% Castro::post_regrid() 1 0.001069 0.001069 0.001069 0.13% MLMG::computeMLResidual() 11 0.001042 0.001042 0.001042 0.13% Gravity::update_max_rhs() 11 0.0008281 0.0008281 0.0008281 0.10% FabArrayBase::getFB() 4023 0.0007035 0.0007035 0.0007035 0.09% Castro::computeInitialDt() 2 0.000686 0.000686 0.000686 0.08% Castro::finalize_advance() 10 0.00067 0.00067 0.00067 0.08% FabArrayBase::CPC::define() 454 0.0006664 0.0006664 0.0006664 0.08% Amr::InitAmr() 1 0.0004669 0.0004669 0.0004669 0.06% Gravity::swapTimeLevels() 10 0.0004523 0.0004523 0.0004523 0.06% Castro::Castro() 1 0.0004504 0.0004504 0.0004504 0.06% MultiFab::Copy() 11 0.0002594 0.0002594 0.0002594 0.03% MLMG::MLResNormInf() 11 0.0002557 0.0002557 0.0002557 0.03% MultiFab::max() 11 0.0002549 0.0002549 0.0002549 0.03% MLLinOp::define() 11 0.0002071 0.0002071 0.0002071 0.03% MLMG::MLRhsNormInf() 11 0.0002022 0.0002022 0.0002022 0.02% MLLinOp::defineGrids() 11 0.0001816 0.0001816 0.0001816 0.02% Castro::buildMetrics() 1 0.0001683 0.0001683 0.0001683 0.02% FabArrayBase::FB::FB() 56 8.652e-05 8.652e-05 8.652e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.237e-05 5.237e-05 5.237e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.345e-05 4.345e-05 4.345e-05 0.01% Castro::swap_state_time_levels() 10 3.959e-05 3.959e-05 3.959e-05 0.00% makeSFC 55 3.902e-05 3.902e-05 3.902e-05 0.00% StateData::define() 4 3.693e-05 3.693e-05 3.693e-05 0.00% Castro::finalize_do_advance() 10 3.667e-05 3.667e-05 3.667e-05 0.00% Castro::enforce_consistent_e() 1 3.407e-05 3.407e-05 3.407e-05 0.00% Amr::writeSmallPlotFile() 1 2.682e-05 2.682e-05 2.682e-05 0.00% Castro::initMFs() 1 2.606e-05 2.606e-05 2.606e-05 0.00% DistributionMapping::Distribute() 56 1.392e-05 1.392e-05 1.392e-05 0.00% Amr::initSubcycle() 1 8.655e-06 8.655e-06 8.655e-06 0.00% Castro::create_source_corrector() 10 5.17e-06 5.17e-06 5.17e-06 0.00% MLMG::buildFineMask() 11 4.671e-06 4.671e-06 4.671e-06 0.00% AmrLevel::checkPointPost() 3 4.307e-06 4.307e-06 4.307e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.117e-06 4.117e-06 4.117e-06 0.00% Gravity::set_mass_offset() 11 3.771e-06 3.771e-06 3.771e-06 0.00% Castro::retry_advance_ctu() 10 3.771e-06 3.771e-06 3.771e-06 0.00% Castro::FluxRegCrseInit 10 3.074e-06 3.074e-06 3.074e-06 0.00% Castro::FluxRegFineAdd() 10 2.374e-06 2.374e-06 2.374e-06 0.00% AmrLevel::checkPointPre() 3 2.179e-06 2.179e-06 2.179e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.946e-06 1.946e-06 1.946e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.06-51-g8fb23ec17a58) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.06-51-g8fb23ec17a58) initialized Starting run at 08:26:14 UTC on 2022-07-01. Successfully read inputs file ... Castro git describe: 22.06-14-gda0f2312b AMReX git describe: 22.06-51-g8fb23ec17 Microphysics git describe: 22.06-7-ga6a75714 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.437662394 Restart time = 0.086102991 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051674344 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.056399807 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.065847656 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.066625094 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.073227729 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026426056 seconds Ending run at 08:26:14 UTC on 2022-07-01. Run time = 0.427286556 Run time without initialization = 0.340636655 Average number of zones advanced per microsecond: 3.848 Average number of zones advanced per microsecond per rank: 3.848 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.4273 ... 0.4273 ... 0.4273 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1120 0.1120 0.1120 26.22% Amr::restart() 1 0.04177 0.04177 0.04177 9.78% VisMF::Read() 3 0.04004 0.04004 0.04004 9.37% MLCellLinOp::applyBC() 1946 0.035 0.035 0.035 8.19% MLPoisson::Fsmooth() 1440 0.0272 0.0272 0.0272 6.37% VisMF::Write(FabArray) 1 0.02493 0.02493 0.02493 5.83% StateData::FillBoundary(geom) 160 0.01158 0.01158 0.01158 2.71% MLCGSolver::bicgstab 36 0.01035 0.01035 0.01035 2.42% Castro::normalize_species() 30 0.009725 0.009725 0.009725 2.28% MultiFab::Dot() 484 0.009623 0.009623 0.009623 2.25% Castro::computeTemp() 30 0.007803 0.007803 0.007803 1.83% FabArray::setVal() 537 0.00679 0.00679 0.00679 1.59% Castro::enforce_min_density() 30 0.006676 0.006676 0.006676 1.56% MLCellLinOp::defineAuxData() 6 0.006332 0.006332 0.006332 1.48% FillBoundary_nowait() 1766 0.006286 0.006286 0.006286 1.47% MultiFab::LinComb() 690 0.006174 0.006174 0.006174 1.44% FabArray::ParallelCopy_nowait() 380 0.006023 0.006023 0.006023 1.41% StateDataPhysBCFunct::() 20 0.005343 0.005343 0.005343 1.25% MLPoisson::Fapply() 500 0.005039 0.005039 0.005039 1.18% Gravity::fill_multipole_BCs() 6 0.004464 0.004464 0.004464 1.04% MLMG::addInterpCorrection() 180 0.003234 0.003234 0.003234 0.76% amrex::average_down 180 0.002984 0.002984 0.002984 0.70% Castro::do_advance_ctu() 5 0.002917 0.002917 0.002917 0.68% MultiFab::Xpay() 258 0.002865 0.002865 0.002865 0.67% BndryData::define() 6 0.002183 0.002183 0.002183 0.51% Castro::estTimeStep() 10 0.00206 0.00206 0.00206 0.48% Castro::reset_internal_energy(MultiFab) 30 0.001764 0.001764 0.001764 0.41% Castro::construct_new_gravity_source() 5 0.001738 0.001738 0.001738 0.41% Amr::writePlotFile() 1 0.001578 0.001578 0.001578 0.37% Castro::subcycle_advance_ctu() 5 0.001516 0.001516 0.001516 0.35% Castro::construct_old_gravity_source() 5 0.001428 0.001428 0.001428 0.33% Castro::enforce_speed_limit() 30 0.001265 0.001265 0.001265 0.30% MultiFab::Saxpy() 10 0.0009148 0.0009148 0.0009148 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009017 0.0009017 0.0009017 0.21% Gravity::get_old_grav_vector() 5 0.0008849 0.0008849 0.0008849 0.21% Castro::expand_state() 5 0.0008742 0.0008742 0.0008742 0.20% Gravity::get_new_grav_vector() 5 0.000868 0.000868 0.000868 0.20% Castro::reset_internal_energy(Fab) 240 0.0008664 0.0008664 0.0008664 0.20% MLMG::ResNormInf() 42 0.0008436 0.0008436 0.0008436 0.20% MLCellLinOp::setLevelBC() 6 0.0008343 0.0008343 0.0008343 0.20% MLMG::oneIter() 36 0.0007415 0.0007415 0.0007415 0.17% Gravity::actual_solve_with_mlmg() 6 0.000699 0.000699 0.000699 0.16% FabArray::mult() 22 0.0006644 0.0006644 0.0006644 0.16% MLCellLinOp::prepareForSolve() 6 0.0006477 0.0006477 0.0006477 0.15% FabArray::setDomainBndry() 20 0.0006431 0.0006431 0.0006431 0.15% MultiFab::contains_nan() 10 0.0005852 0.0005852 0.0005852 0.14% MLMG::prepareForSolve() 6 0.0005708 0.0005708 0.0005708 0.13% MLCellLinOp::compGrad() 6 0.0005058 0.0005058 0.0005058 0.12% MLCellLinOp::smooth() 720 0.0004962 0.0004962 0.0004962 0.12% FabArrayBase::CPC::define() 244 0.0004142 0.0004142 0.0004142 0.10% Amr::InitAmr() 1 0.0003867 0.0003867 0.0003867 0.09% FabArrayBase::getCPC() 632 0.0003694 0.0003694 0.0003694 0.09% FabArray::FillBoundary() 1766 0.0003653 0.0003653 0.0003653 0.09% FabArrayBase::getFB() 1766 0.0002581 0.0002581 0.0002581 0.06% main() 1 0.0002454 0.0002454 0.0002454 0.06% Gravity::update_max_rhs() 6 0.0002357 0.0002357 0.0002357 0.06% Gravity::solve_for_phi() 5 0.0002088 0.0002088 0.0002088 0.05% MLCellLinOp::apply() 500 0.0002054 0.0002054 0.0002054 0.05% Amr::coarseTimeStep() 5 0.0002042 0.0002042 0.0002042 0.05% CGSolver::sxay() 690 0.00018 0.00018 0.00018 0.04% FillPatchIterator::Initialize 20 0.0001648 0.0001648 0.0001648 0.04% MLCellLinOp::defineBC() 6 0.0001515 0.0001515 0.0001515 0.04% MultiFab::Copy() 6 0.0001395 0.0001395 0.0001395 0.03% MultiFab::max() 6 0.0001347 0.0001347 0.0001347 0.03% FabArray::ParallelCopy() 380 0.0001307 0.0001307 0.0001307 0.03% MLCGSolver::ParallelAllReduce 659 0.0001287 0.0001287 0.0001287 0.03% Castro::construct_new_gravity() 5 0.0001105 0.0001105 0.0001105 0.03% Castro::construct_new_source() 25 0.0001068 0.0001068 0.0001068 0.02% MLMG::MLRhsNormInf() 6 0.0001056 0.0001056 0.0001056 0.02% Amr::timeStep() 5 0.0001012 0.0001012 0.0001012 0.02% MLCellLinOp::correctionResidual() 216 9.826e-05 9.826e-05 9.826e-05 0.02% MLMG::mgVcycle() 36 9.23e-05 9.23e-05 9.23e-05 0.02% MLLinOp::defineGrids() 6 8.542e-05 8.542e-05 8.542e-05 0.02% AmrLevel::restart() 1 7.723e-05 7.723e-05 7.723e-05 0.02% StateData::restartDoit() 4 7.083e-05 7.083e-05 7.083e-05 0.02% MLMG:computeResOfCorrection() 180 5.814e-05 5.814e-05 5.814e-05 0.01% FabArrayBase::FB::FB() 26 5.673e-05 5.673e-05 5.673e-05 0.01% Castro::advance() 5 4.926e-05 4.926e-05 4.926e-05 0.01% MLMG::actualBottomSolve() 36 4.581e-05 4.581e-05 4.581e-05 0.01% Castro::initialize_do_advance() 5 4.575e-05 4.575e-05 4.575e-05 0.01% MLMG::mgVcycle_down::0 36 4.291e-05 4.291e-05 4.291e-05 0.01% Castro::create_source_corrector() 5 4.112e-05 4.112e-05 4.112e-05 0.01% Castro::finalize_advance() 5 4.093e-05 4.093e-05 4.093e-05 0.01% Castro::initialize_advance() 5 3.972e-05 3.972e-05 3.972e-05 0.01% Castro::post_restart() 1 3.915e-05 3.915e-05 3.915e-05 0.01% MLMG::mgVcycle_down::1 36 3.892e-05 3.892e-05 3.892e-05 0.01% MLMG::mgVcycle_down::2 36 3.86e-05 3.86e-05 3.86e-05 0.01% Castro::clean_state() 30 3.746e-05 3.746e-05 3.746e-05 0.01% MLMG::solve() 6 3.658e-05 3.658e-05 3.658e-05 0.01% MLMG::mgVcycle_down::4 36 3.489e-05 3.489e-05 3.489e-05 0.01% MLMG::mgVcycle_down::3 36 3.377e-05 3.377e-05 3.377e-05 0.01% Castro::buildMetrics() 1 3.197e-05 3.197e-05 3.197e-05 0.01% Gravity::actual_multilevel_solve() 1 3.077e-05 3.077e-05 3.077e-05 0.01% MLMG::mgVcycle_up::4 36 3.06e-05 3.06e-05 3.06e-05 0.01% Castro::initMFs() 1 2.802e-05 2.802e-05 2.802e-05 0.01% Amr::writeSmallPlotFile() 1 2.709e-05 2.709e-05 2.709e-05 0.01% Castro::swap_state_time_levels() 5 2.705e-05 2.705e-05 2.705e-05 0.01% MLMG::mgVcycle_up::3 36 2.502e-05 2.502e-05 2.502e-05 0.01% MLMG::mgVcycle_up::0 36 2.494e-05 2.494e-05 2.494e-05 0.01% MLMG::mgVcycle_up::2 36 2.439e-05 2.439e-05 2.439e-05 0.01% Castro::computeNewDt() 5 2.424e-05 2.424e-05 2.424e-05 0.01% MLCellLinOp::solutionResidual() 42 2.343e-05 2.343e-05 2.343e-05 0.01% MLLinOp::define() 6 2.289e-05 2.289e-05 2.289e-05 0.01% MLMG::mgVcycle_up::1 36 2.283e-05 2.283e-05 2.283e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.946e-05 1.946e-05 1.946e-05 0.00% Castro::finalize_do_advance() 5 1.883e-05 1.883e-05 1.883e-05 0.00% MLMG::mgVcycle_bottom 36 1.596e-05 1.596e-05 1.596e-05 0.00% MLMG::computeResidual() 36 1.538e-05 1.538e-05 1.538e-05 0.00% MLPoisson::define() 6 1.463e-05 1.463e-05 1.463e-05 0.00% makeSFC 30 1.448e-05 1.448e-05 1.448e-05 0.00% FillPatchSingleLevel 20 1.403e-05 1.403e-05 1.403e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.391e-05 1.391e-05 1.391e-05 0.00% Castro::construct_old_source() 25 1.043e-05 1.043e-05 1.043e-05 0.00% Castro::do_new_sources() 5 1.022e-05 1.022e-05 1.022e-05 0.00% DistributionMapping::Distribute() 31 8.945e-06 8.945e-06 8.945e-06 0.00% Castro::do_old_sources() 5 8.674e-06 8.674e-06 8.674e-06 0.00% Amr::initSubcycle() 1 8.397e-06 8.397e-06 8.397e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.13e-06 7.13e-06 7.13e-06 0.00% Castro::check_for_nan() 10 7.123e-06 7.123e-06 7.123e-06 0.00% Castro::apply_source_to_state() 10 5.894e-06 5.894e-06 5.894e-06 0.00% Castro::post_timestep() 5 5.723e-06 5.723e-06 5.723e-06 0.00% Castro::construct_old_gravity() 5 5.22e-06 5.22e-06 5.22e-06 0.00% MLPoisson::prepareForSolve() 6 4.311e-06 4.311e-06 4.311e-06 0.00% Gravity::swapTimeLevels() 5 4.165e-06 4.165e-06 4.165e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.716e-06 3.716e-06 3.716e-06 0.00% MLMG::computeMLResidual() 6 2.98e-06 2.98e-06 2.98e-06 0.00% MLMG::buildFineMask() 6 2.937e-06 2.937e-06 2.937e-06 0.00% MLMG::getGradSolution() 6 2.849e-06 2.849e-06 2.849e-06 0.00% Gravity::set_mass_offset() 6 2.786e-06 2.786e-06 2.786e-06 0.00% MLMG::MLResNormInf() 6 2.317e-06 2.317e-06 2.317e-06 0.00% Castro::retry_advance_ctu() 5 2.019e-06 2.019e-06 2.019e-06 0.00% Castro::FluxRegCrseInit 5 1.647e-06 1.647e-06 1.647e-06 0.00% AmrLevel::AmrLevel() 1 1.466e-06 1.466e-06 1.466e-06 0.00% Castro::FluxRegFineAdd() 5 1.195e-06 1.195e-06 1.195e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.105e-06 1.105e-06 1.105e-06 0.00% Amr::init() 1 1.024e-06 1.024e-06 1.024e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4273 0.4273 0.4273 100.00% Amr::coarseTimeStep() 5 0.314 0.314 0.314 73.48% Amr::timeStep() 5 0.3126 0.3126 0.3126 73.16% Castro::advance() 5 0.3076 0.3076 0.3076 71.99% Castro::subcycle_advance_ctu() 5 0.3021 0.3021 0.3021 70.70% Castro::do_advance_ctu() 5 0.3006 0.3006 0.3006 70.34% Castro::construct_new_gravity() 5 0.144 0.144 0.144 33.69% Gravity::solve_phi_with_mlmg() 6 0.1399 0.1399 0.1399 32.74% Gravity::solve_for_phi() 5 0.1364 0.1364 0.1364 31.92% Gravity::actual_solve_with_mlmg() 6 0.1353 0.1353 0.1353 31.67% MLMG::solve() 6 0.1229 0.1229 0.1229 28.76% MLMG::oneIter() 36 0.1158 0.1158 0.1158 27.11% MLMG::mgVcycle() 36 0.1151 0.1151 0.1151 26.94% Castro::construct_ctu_hydro_source() 5 0.112 0.112 0.112 26.22% Amr::init() 1 0.08615 0.08615 0.08615 20.16% Amr::restart() 1 0.08615 0.08615 0.08615 20.16% MLCellLinOp::smooth() 720 0.05885 0.05885 0.05885 13.77% MLCellLinOp::applyBC() 1946 0.04197 0.04197 0.04197 9.82% AmrLevel::restart() 1 0.04025 0.04025 0.04025 9.42% StateData::restartDoit() 4 0.04017 0.04017 0.04017 9.40% VisMF::Read() 3 0.04004 0.04004 0.04004 9.37% MLMG::mgVcycle_bottom 36 0.03536 0.03536 0.03536 8.27% MLMG::actualBottomSolve() 36 0.03534 0.03534 0.03534 8.27% MLCGSolver::bicgstab 36 0.03498 0.03498 0.03498 8.19% Castro::clean_state() 30 0.02814 0.02814 0.02814 6.58% MLPoisson::Fsmooth() 1440 0.0272 0.0272 0.0272 6.37% Amr::writePlotFile() 1 0.02651 0.02651 0.02651 6.20% VisMF::Write(FabArray) 1 0.02493 0.02493 0.02493 5.83% FillPatchIterator::Initialize 20 0.01975 0.01975 0.01975 4.62% FillPatchSingleLevel 20 0.01894 0.01894 0.01894 4.43% StateDataPhysBCFunct::() 20 0.01692 0.01692 0.01692 3.96% MLCellLinOp::apply() 500 0.0158 0.0158 0.0158 3.70% MLMG::mgVcycle_down::0 36 0.01548 0.01548 0.01548 3.62% MLMG::mgVcycle_up::0 36 0.01322 0.01322 0.01322 3.09% StateData::FillBoundary(geom) 160 0.01158 0.01158 0.01158 2.71% Castro::computeTemp() 30 0.01043 0.01043 0.01043 2.44% MLPoisson::define() 6 0.01012 0.01012 0.01012 2.37% Castro::initialize_do_advance() 5 0.01007 0.01007 0.01007 2.36% Castro::normalize_species() 30 0.009725 0.009725 0.009725 2.28% MultiFab::Dot() 484 0.009623 0.009623 0.009623 2.25% MLCellLinOp::correctionResidual() 216 0.009213 0.009213 0.009213 2.16% Castro::do_new_sources() 5 0.00874 0.00874 0.00874 2.05% MLMG:computeResOfCorrection() 180 0.007945 0.007945 0.007945 1.86% MLMG::mgVcycle_down::1 36 0.007658 0.007658 0.007658 1.79% Gravity::get_new_grav_vector() 5 0.00747 0.00747 0.00747 1.75% MLMG::mgVcycle_down::2 36 0.007425 0.007425 0.007425 1.74% Castro::construct_old_gravity() 5 0.007352 0.007352 0.007352 1.72% Gravity::get_old_grav_vector() 5 0.007346 0.007346 0.007346 1.72% Castro::do_old_sources() 5 0.007297 0.007297 0.007297 1.71% MLCellLinOp::defineAuxData() 6 0.007085 0.007085 0.007085 1.66% MLMG::mgVcycle_down::3 36 0.007053 0.007053 0.007053 1.65% FabArray::FillBoundary() 1766 0.006966 0.006966 0.006966 1.63% FabArray::setVal() 537 0.00679 0.00679 0.00679 1.59% MLMG::mgVcycle_down::4 36 0.006731 0.006731 0.006731 1.58% Castro::enforce_min_density() 30 0.006676 0.006676 0.006676 1.56% FillBoundary_nowait() 1766 0.006601 0.006601 0.006601 1.54% FabArray::ParallelCopy() 380 0.006541 0.006541 0.006541 1.53% FabArray::ParallelCopy_nowait() 380 0.00641 0.00641 0.00641 1.50% CGSolver::sxay() 690 0.006354 0.006354 0.006354 1.49% MultiFab::LinComb() 690 0.006174 0.006174 0.006174 1.44% MLCGSolver::ParallelAllReduce 659 0.00579 0.00579 0.00579 1.35% MLMG::mgVcycle_up::2 36 0.005701 0.005701 0.005701 1.33% MLMG::mgVcycle_up::1 36 0.00561 0.00561 0.00561 1.31% Castro::expand_state() 5 0.005529 0.005529 0.005529 1.29% MLMG::addInterpCorrection() 180 0.005506 0.005506 0.005506 1.29% Castro::initialize_advance() 5 0.005432 0.005432 0.005432 1.27% MLMG::mgVcycle_up::3 36 0.00543 0.00543 0.00543 1.27% MLMG::mgVcycle_up::4 36 0.005349 0.005349 0.005349 1.25% amrex::average_down 180 0.005255 0.005255 0.005255 1.23% MLPoisson::Fapply() 500 0.005039 0.005039 0.005039 1.18% Castro::post_timestep() 5 0.00491 0.00491 0.00491 1.15% Gravity::fill_multipole_BCs() 6 0.004464 0.004464 0.004464 1.04% Castro::post_restart() 1 0.003939 0.003939 0.003939 0.92% Gravity::multilevel_solve_for_new_phi() 1 0.003782 0.003782 0.003782 0.89% Gravity::actual_multilevel_solve() 1 0.003762 0.003762 0.003762 0.88% MLCellLinOp::solutionResidual() 42 0.003248 0.003248 0.003248 0.76% MLCellLinOp::defineBC() 6 0.00288 0.00288 0.00288 0.67% MultiFab::Xpay() 258 0.002865 0.002865 0.002865 0.67% MLMG::prepareForSolve() 6 0.002798 0.002798 0.002798 0.65% BndryData::define() 6 0.002728 0.002728 0.002728 0.64% MLMG::computeResidual() 36 0.002696 0.002696 0.002696 0.63% Castro::reset_internal_energy(MultiFab) 30 0.00263 0.00263 0.00263 0.62% Castro::estTimeStep() 10 0.00206 0.00206 0.00206 0.48% Castro::construct_new_source() 25 0.001845 0.001845 0.001845 0.43% Castro::construct_new_gravity_source() 5 0.001738 0.001738 0.001738 0.41% Castro::construct_old_source() 25 0.001439 0.001439 0.001439 0.34% Castro::construct_old_gravity_source() 5 0.001428 0.001428 0.001428 0.33% Castro::enforce_speed_limit() 30 0.001265 0.001265 0.001265 0.30% Castro::computeNewDt() 5 0.001132 0.001132 0.001132 0.26% Castro::apply_source_to_state() 10 0.0009207 0.0009207 0.0009207 0.22% MultiFab::Saxpy() 10 0.0009148 0.0009148 0.0009148 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0009017 0.0009017 0.0009017 0.21% Castro::reset_internal_energy(Fab) 240 0.0008664 0.0008664 0.0008664 0.20% MLMG::ResNormInf() 42 0.0008436 0.0008436 0.0008436 0.20% MLCellLinOp::setLevelBC() 6 0.0008343 0.0008343 0.0008343 0.20% FabArrayBase::getCPC() 632 0.0007836 0.0007836 0.0007836 0.18% MLMG::getGradSolution() 6 0.0007779 0.0007779 0.0007779 0.18% MLCellLinOp::compGrad() 6 0.0007751 0.0007751 0.0007751 0.18% FabArray::mult() 22 0.0006644 0.0006644 0.0006644 0.16% MLPoisson::prepareForSolve() 6 0.000652 0.000652 0.000652 0.15% MLCellLinOp::prepareForSolve() 6 0.0006477 0.0006477 0.0006477 0.15% FabArray::setDomainBndry() 20 0.0006431 0.0006431 0.0006431 0.15% Castro::check_for_nan() 10 0.0005923 0.0005923 0.0005923 0.14% MultiFab::contains_nan() 10 0.0005852 0.0005852 0.0005852 0.14% MLMG::computeMLResidual() 6 0.0005705 0.0005705 0.0005705 0.13% Gravity::update_max_rhs() 6 0.0004647 0.0004647 0.0004647 0.11% FabArrayBase::CPC::define() 244 0.0004142 0.0004142 0.0004142 0.10% Amr::InitAmr() 1 0.0003951 0.0003951 0.0003951 0.09% FabArrayBase::getFB() 1766 0.0003148 0.0003148 0.0003148 0.07% Gravity::swapTimeLevels() 5 0.0002346 0.0002346 0.0002346 0.05% Castro::buildMetrics() 1 0.000154 0.000154 0.000154 0.04% MultiFab::Copy() 6 0.0001395 0.0001395 0.0001395 0.03% MLLinOp::define() 6 0.0001382 0.0001382 0.0001382 0.03% MLMG::MLResNormInf() 6 0.0001356 0.0001356 0.0001356 0.03% MultiFab::max() 6 0.0001347 0.0001347 0.0001347 0.03% MLLinOp::defineGrids() 6 0.0001154 0.0001154 0.0001154 0.03% MLMG::MLRhsNormInf() 6 0.0001056 0.0001056 0.0001056 0.02% FabArrayBase::FB::FB() 26 5.673e-05 5.673e-05 5.673e-05 0.01% Castro::finalize_advance() 5 4.377e-05 4.377e-05 4.377e-05 0.01% Castro::create_source_corrector() 5 4.112e-05 4.112e-05 4.112e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.883e-05 2.883e-05 2.883e-05 0.01% Castro::initMFs() 1 2.802e-05 2.802e-05 2.802e-05 0.01% Amr::writeSmallPlotFile() 1 2.709e-05 2.709e-05 2.709e-05 0.01% Castro::swap_state_time_levels() 5 2.705e-05 2.705e-05 2.705e-05 0.01% makeSFC 30 2.17e-05 2.17e-05 2.17e-05 0.01% Castro::finalize_do_advance() 5 1.883e-05 1.883e-05 1.883e-05 0.00% DistributionMapping::Distribute() 31 8.945e-06 8.945e-06 8.945e-06 0.00% Amr::initSubcycle() 1 8.397e-06 8.397e-06 8.397e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.442e-06 5.442e-06 5.442e-06 0.00% MLMG::buildFineMask() 6 2.937e-06 2.937e-06 2.937e-06 0.00% Gravity::set_mass_offset() 6 2.786e-06 2.786e-06 2.786e-06 0.00% Castro::retry_advance_ctu() 5 2.019e-06 2.019e-06 2.019e-06 0.00% Castro::FluxRegCrseInit 5 1.647e-06 1.647e-06 1.647e-06 0.00% AmrLevel::AmrLevel() 1 1.466e-06 1.466e-06 1.466e-06 0.00% Castro::FluxRegFineAdd() 5 1.195e-06 1.195e-06 1.195e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.105e-06 1.105e-06 1.105e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.06-51-g8fb23ec17a58) finalized