Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.06-13-g7b335c5f9617) initialized Starting run at 08:26:40 UTC on 2022-06-09. Successfully read inputs file ... Castro git describe: 22.06-9-g26873b1cc AMReX git describe: 22.06-13-g7b335c5f9 Microphysics git describe: 22.06-1-gb8e427c1 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.043477727 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025142152 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.049165292 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.050680328 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.049708969 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.055239544 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.081190602 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.040434324 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.073702787 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.05026607 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.048354018 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.052930548 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.056418332 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.040689663 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024963118 seconds Ending run at 08:26:41 UTC on 2022-06-09. Run time = 0.796002283 Run time without initialization = 0.674353338 Average number of zones advanced per microsecond: 3.887 Average number of zones advanced per microsecond per rank: 3.887 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.796 ... 0.796 ... 0.796 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.1776 0.1776 0.1776 22.31% VisMF::Write(FabArray) 11 0.1684 0.1684 0.1684 21.16% MLCellLinOp::applyBC() 4433 0.08207 0.08207 0.08207 10.31% MLPoisson::Fsmooth() 3280 0.06438 0.06438 0.06438 8.09% StateData::FillBoundary(geom) 328 0.02498 0.02498 0.02498 3.14% MLCGSolver::bicgstab 82 0.02424 0.02424 0.02424 3.05% MultiFab::Dot() 1114 0.02272 0.02272 0.02272 2.85% StateDataPhysBCFunct::() 41 0.01844 0.01844 0.01844 2.32% MultiFab::LinComb() 1586 0.01459 0.01459 0.01459 1.83% FabArray::setVal() 1144 0.01441 0.01441 0.01441 1.81% FillBoundary_nowait() 4023 0.01435 0.01435 0.01435 1.80% Castro::computeTemp() 63 0.0142 0.0142 0.0142 1.78% FabArray::ParallelCopy_nowait() 861 0.01334 0.01334 0.01334 1.68% Castro::normalize_species() 62 0.01316 0.01316 0.01316 1.65% MLPoisson::Fapply() 1142 0.01188 0.01188 0.01188 1.49% MLCellLinOp::defineAuxData() 11 0.01163 0.01163 0.01163 1.46% Gravity::fill_multipole_BCs() 11 0.008391 0.008391 0.008391 1.05% MLMG::addInterpCorrection() 410 0.007585 0.007585 0.007585 0.95% Castro::enforce_min_density() 62 0.007344 0.007344 0.007344 0.92% amrex::average_down 410 0.006825 0.006825 0.006825 0.86% MultiFab::Xpay() 585 0.006627 0.006627 0.006627 0.83% Castro::estTimeStep() 21 0.004735 0.004735 0.004735 0.59% Castro::do_advance_ctu() 10 0.00457 0.00457 0.00457 0.57% Castro::reset_internal_energy(MultiFab) 63 0.004169 0.004169 0.004169 0.52% Amr::checkPoint() 3 0.004003 0.004003 0.004003 0.50% BndryData::define() 11 0.003959 0.003959 0.003959 0.50% Castro::construct_new_gravity_source() 10 0.003169 0.003169 0.003169 0.40% Castro::construct_old_gravity_source() 10 0.00246 0.00246 0.00246 0.31% Amr::writePlotFile() 2 0.002348 0.002348 0.002348 0.30% MLMG::ResNormInf() 93 0.00195 0.00195 0.00195 0.25% Gravity::get_new_grav_vector() 11 0.001941 0.001941 0.001941 0.24% Castro::enforce_speed_limit() 62 0.001875 0.001875 0.001875 0.24% MultiFab::Saxpy() 20 0.001808 0.001808 0.001808 0.23% Castro::expand_state() 10 0.001739 0.001739 0.001739 0.22% Gravity::get_old_grav_vector() 10 0.001738 0.001738 0.001738 0.22% MLMG::oneIter() 82 0.001716 0.001716 0.001716 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001664 0.001664 0.001664 0.21% MLCellLinOp::setLevelBC() 11 0.00157 0.00157 0.00157 0.20% Castro::reset_internal_energy(Fab) 504 0.00154 0.00154 0.00154 0.19% Gravity::actual_solve_with_mlmg() 11 0.001371 0.001371 0.001371 0.17% FabArray::mult() 43 0.001342 0.001342 0.001342 0.17% FabArray::setDomainBndry() 41 0.001313 0.001313 0.001313 0.16% Castro::initData() 1 0.001206 0.001206 0.001206 0.15% MLCellLinOp::prepareForSolve() 11 0.001185 0.001185 0.001185 0.15% MultiFab::contains_nan() 20 0.001175 0.001175 0.001175 0.15% MLCellLinOp::smooth() 1640 0.00111 0.00111 0.00111 0.14% MLMG::prepareForSolve() 11 0.001045 0.001045 0.001045 0.13% MLCellLinOp::compGrad() 11 0.0009255 0.0009255 0.0009255 0.12% FabArrayBase::getCPC() 1323 0.0007761 0.0007761 0.0007761 0.10% FabArray::FillBoundary() 4023 0.0007731 0.0007731 0.0007731 0.10% FabArrayBase::CPC::define() 454 0.0006972 0.0006972 0.0006972 0.09% FabArrayBase::getFB() 4023 0.0006481 0.0006481 0.0006481 0.08% Amr::InitAmr() 1 0.0004908 0.0004908 0.0004908 0.06% MLCellLinOp::apply() 1142 0.0004639 0.0004639 0.0004639 0.06% Gravity::solve_for_phi() 10 0.0004452 0.0004452 0.0004452 0.06% Gravity::update_max_rhs() 11 0.0004213 0.0004213 0.0004213 0.05% CGSolver::sxay() 1586 0.0003663 0.0003663 0.0003663 0.05% Amr::coarseTimeStep() 10 0.0003264 0.0003264 0.0003264 0.04% FillPatchIterator::Initialize 41 0.0002956 0.0002956 0.0002956 0.04% MLCellLinOp::defineBC() 11 0.0002827 0.0002827 0.0002827 0.04% main() 1 0.0002815 0.0002815 0.0002815 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002786 0.0002786 0.0002786 0.04% FabArray::ParallelCopy() 861 0.0002745 0.0002745 0.0002745 0.03% MultiFab::max() 11 0.0002564 0.0002564 0.0002564 0.03% MultiFab::Copy() 11 0.0002563 0.0002563 0.0002563 0.03% Castro::subcycle_advance_ctu() 10 0.0002368 0.0002368 0.0002368 0.03% MLCellLinOp::correctionResidual() 492 0.0002206 0.0002206 0.0002206 0.03% Castro::construct_new_gravity() 10 0.0002073 0.0002073 0.0002073 0.03% MLMG::MLRhsNormInf() 11 0.000206 0.000206 0.000206 0.03% MLLinOp::defineGrids() 11 0.0002035 0.0002035 0.0002035 0.03% MLMG::mgVcycle() 82 0.0002016 0.0002016 0.0002016 0.03% Amr::timeStep() 10 0.0001984 0.0001984 0.0001984 0.02% StateData::checkPoint() 12 0.0001265 0.0001265 0.0001265 0.02% MLMG:computeResOfCorrection() 410 0.00012 0.00012 0.00012 0.02% MLMG::actualBottomSolve() 82 9.89e-05 9.89e-05 9.89e-05 0.01% MLMG::mgVcycle_down::0 82 9.692e-05 9.692e-05 9.692e-05 0.01% Castro::Castro() 1 8.764e-05 8.764e-05 8.764e-05 0.01% FabArrayBase::FB::FB() 56 8.73e-05 8.73e-05 8.73e-05 0.01% Castro::initialize_advance() 10 8.456e-05 8.456e-05 8.456e-05 0.01% MLMG::mgVcycle_down::1 82 8.122e-05 8.122e-05 8.122e-05 0.01% MLMG::mgVcycle_down::2 82 7.606e-05 7.606e-05 7.606e-05 0.01% Castro::clean_state() 62 7.431e-05 7.431e-05 7.431e-05 0.01% MLMG::solve() 11 7.362e-05 7.362e-05 7.362e-05 0.01% AmrLevel::checkPoint() 3 7.086e-05 7.086e-05 7.086e-05 0.01% MLMG::mgVcycle_down::3 82 7.07e-05 7.07e-05 7.07e-05 0.01% MLMG::mgVcycle_down::4 82 7.068e-05 7.068e-05 7.068e-05 0.01% Castro::initialize_do_advance() 10 6.914e-05 6.914e-05 6.914e-05 0.01% Castro::finalize_advance() 10 5.625e-05 5.625e-05 5.625e-05 0.01% MLMG::mgVcycle_up::4 82 5.298e-05 5.298e-05 5.298e-05 0.01% MLMG::mgVcycle_up::0 82 4.966e-05 4.966e-05 4.966e-05 0.01% MLCellLinOp::solutionResidual() 93 4.878e-05 4.878e-05 4.878e-05 0.01% MLMG::mgVcycle_up::1 82 4.765e-05 4.765e-05 4.765e-05 0.01% MLMG::mgVcycle_up::3 82 4.552e-05 4.552e-05 4.552e-05 0.01% MLMG::mgVcycle_up::2 82 4.521e-05 4.521e-05 4.521e-05 0.01% Castro::advance() 10 4.228e-05 4.228e-05 4.228e-05 0.01% StateData::define() 4 4.033e-05 4.033e-05 4.033e-05 0.01% Castro::swap_state_time_levels() 10 3.958e-05 3.958e-05 3.958e-05 0.00% Castro::finalize_do_advance() 10 3.631e-05 3.631e-05 3.631e-05 0.00% Castro::enforce_consistent_e() 1 3.331e-05 3.331e-05 3.331e-05 0.00% MLMG::mgVcycle_bottom 82 3.268e-05 3.268e-05 3.268e-05 0.00% Gravity::actual_multilevel_solve() 1 3.158e-05 3.158e-05 3.158e-05 0.00% MLMG::computeResidual() 82 3.114e-05 3.114e-05 3.114e-05 0.00% Castro::initMFs() 1 2.94e-05 2.94e-05 2.94e-05 0.00% FillPatchSingleLevel 41 2.702e-05 2.702e-05 2.702e-05 0.00% makeSFC 55 2.656e-05 2.656e-05 2.656e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.645e-05 2.645e-05 2.645e-05 0.00% Amr::writeSmallPlotFile() 1 2.531e-05 2.531e-05 2.531e-05 0.00% Amr::defBaseLevel() 1 2.516e-05 2.516e-05 2.516e-05 0.00% Castro::buildMetrics() 1 2.324e-05 2.324e-05 2.324e-05 0.00% MLPoisson::define() 11 2.244e-05 2.244e-05 2.244e-05 0.00% MLLinOp::define() 11 2.194e-05 2.194e-05 2.194e-05 0.00% Amr::FinalizeInit() 1 1.99e-05 1.99e-05 1.99e-05 0.00% Castro::construct_new_source() 50 1.8e-05 1.8e-05 1.8e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.788e-05 1.788e-05 1.788e-05 0.00% Castro::construct_old_source() 50 1.732e-05 1.732e-05 1.732e-05 0.00% Castro::do_new_sources() 10 1.699e-05 1.699e-05 1.699e-05 0.00% Castro::do_old_sources() 10 1.605e-05 1.605e-05 1.605e-05 0.00% DistributionMapping::Distribute() 56 1.47e-05 1.47e-05 1.47e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.281e-05 1.281e-05 1.281e-05 0.00% Castro::apply_source_to_state() 20 1.073e-05 1.073e-05 1.073e-05 0.00% Castro::check_for_nan() 20 1.067e-05 1.067e-05 1.067e-05 0.00% Castro::construct_old_gravity() 10 1.056e-05 1.056e-05 1.056e-05 0.00% Castro::post_timestep() 10 8.861e-06 8.861e-06 8.861e-06 0.00% Amr::initSubcycle() 1 8.794e-06 8.794e-06 8.794e-06 0.00% Gravity::swapTimeLevels() 10 8.698e-06 8.698e-06 8.698e-06 0.00% MLPoisson::prepareForSolve() 11 8.637e-06 8.637e-06 8.637e-06 0.00% MLMG::computeMLResidual() 11 6.764e-06 6.764e-06 6.764e-06 0.00% MLMG::getGradSolution() 11 6.521e-06 6.521e-06 6.521e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.463e-06 6.463e-06 6.463e-06 0.00% Castro::computeNewDt() 9 5.944e-06 5.944e-06 5.944e-06 0.00% Amr::InitializeInit() 1 5.676e-06 5.676e-06 5.676e-06 0.00% MLMG::buildFineMask() 11 5.126e-06 5.126e-06 5.126e-06 0.00% AmrLevel::checkPointPost() 3 4.719e-06 4.719e-06 4.719e-06 0.00% Castro::post_init() 1 4.444e-06 4.444e-06 4.444e-06 0.00% MLMG::MLResNormInf() 11 4.396e-06 4.396e-06 4.396e-06 0.00% Castro::retry_advance_ctu() 10 4.233e-06 4.233e-06 4.233e-06 0.00% Castro::create_source_corrector() 10 3.941e-06 3.941e-06 3.941e-06 0.00% Gravity::set_mass_offset() 11 3.683e-06 3.683e-06 3.683e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.336e-06 3.336e-06 3.336e-06 0.00% Castro::FluxRegCrseInit 10 2.886e-06 2.886e-06 2.886e-06 0.00% Amr::init() 1 2.309e-06 2.309e-06 2.309e-06 0.00% Castro::FluxRegFineAdd() 10 2.188e-06 2.188e-06 2.188e-06 0.00% Castro::computeInitialDt() 2 1.942e-06 1.942e-06 1.942e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.871e-06 1.871e-06 1.871e-06 0.00% AmrLevel::checkPointPre() 3 1.801e-06 1.801e-06 1.801e-06 0.00% Amr::initialInit() 1 1.314e-06 1.314e-06 1.314e-06 0.00% Castro::post_regrid() 1 1.041e-06 1.041e-06 1.041e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.796 0.796 0.796 100.00% Amr::coarseTimeStep() 10 0.6492 0.6492 0.6492 81.55% Amr::timeStep() 10 0.5652 0.5652 0.5652 71.00% Castro::advance() 10 0.5584 0.5584 0.5584 70.15% Castro::subcycle_advance_ctu() 10 0.5476 0.5476 0.5476 68.79% Castro::do_advance_ctu() 10 0.5473 0.5473 0.5473 68.76% Gravity::solve_phi_with_mlmg() 11 0.3179 0.3179 0.3179 39.93% Gravity::actual_solve_with_mlmg() 11 0.3093 0.3093 0.3093 38.85% Castro::construct_new_gravity() 10 0.289 0.289 0.289 36.30% MLMG::solve() 11 0.2864 0.2864 0.2864 35.98% Gravity::solve_for_phi() 10 0.2736 0.2736 0.2736 34.36% MLMG::oneIter() 82 0.2717 0.2717 0.2717 34.14% MLMG::mgVcycle() 82 0.27 0.27 0.27 33.92% Castro::construct_ctu_hydro_source() 10 0.1776 0.1776 0.1776 22.31% VisMF::Write(FabArray) 11 0.1684 0.1684 0.1684 21.16% MLCellLinOp::smooth() 1640 0.1384 0.1384 0.1384 17.38% Amr::checkPoint() 3 0.1247 0.1247 0.1247 15.67% Amr::init() 1 0.121 0.121 0.121 15.20% AmrLevel::checkPoint() 3 0.1207 0.1207 0.1207 15.17% StateData::checkPoint() 12 0.1206 0.1206 0.1206 15.16% MLCellLinOp::applyBC() 4433 0.09793 0.09793 0.09793 12.30% MLMG::mgVcycle_bottom 82 0.08317 0.08317 0.08317 10.45% MLMG::actualBottomSolve() 82 0.08314 0.08314 0.08314 10.44% MLCGSolver::bicgstab 82 0.0823 0.0823 0.0823 10.34% MLPoisson::Fsmooth() 3280 0.06438 0.06438 0.06438 8.09% Amr::initialInit() 1 0.05227 0.05227 0.05227 6.57% Amr::writePlotFile() 2 0.05024 0.05024 0.05024 6.31% FillPatchIterator::Initialize 41 0.04907 0.04907 0.04907 6.16% Amr::FinalizeInit() 1 0.04819 0.04819 0.04819 6.05% FillPatchSingleLevel 41 0.04746 0.04746 0.04746 5.96% Castro::post_init() 1 0.04673 0.04673 0.04673 5.87% Gravity::multilevel_solve_for_new_phi() 1 0.04483 0.04483 0.04483 5.63% Gravity::actual_multilevel_solve() 1 0.04482 0.04482 0.04482 5.63% StateDataPhysBCFunct::() 41 0.04342 0.04342 0.04342 5.45% Castro::clean_state() 62 0.04157 0.04157 0.04157 5.22% MLCellLinOp::apply() 1142 0.03692 0.03692 0.03692 4.64% MLMG::mgVcycle_down::0 82 0.03599 0.03599 0.03599 4.52% MLMG::mgVcycle_up::0 82 0.03088 0.03088 0.03088 3.88% StateData::FillBoundary(geom) 328 0.02498 0.02498 0.02498 3.14% MultiFab::Dot() 1114 0.02272 0.02272 0.02272 2.85% MLCellLinOp::correctionResidual() 492 0.02158 0.02158 0.02158 2.71% Castro::initialize_do_advance() 10 0.02112 0.02112 0.02112 2.65% Castro::construct_old_gravity() 10 0.02101 0.02101 0.02101 2.64% Gravity::get_old_grav_vector() 10 0.021 0.021 0.021 2.64% Castro::computeTemp() 63 0.01991 0.01991 0.01991 2.50% MLMG:computeResOfCorrection() 410 0.01862 0.01862 0.01862 2.34% MLPoisson::define() 11 0.01848 0.01848 0.01848 2.32% MLMG::mgVcycle_down::1 82 0.01799 0.01799 0.01799 2.26% MLMG::mgVcycle_down::2 82 0.01753 0.01753 0.01753 2.20% Gravity::get_new_grav_vector() 11 0.01703 0.01703 0.01703 2.14% MLMG::mgVcycle_down::3 82 0.01656 0.01656 0.01656 2.08% FabArray::FillBoundary() 4023 0.01586 0.01586 0.01586 1.99% MLMG::mgVcycle_down::4 82 0.01584 0.01584 0.01584 1.99% FillBoundary_nowait() 4023 0.01509 0.01509 0.01509 1.90% CGSolver::sxay() 1586 0.01495 0.01495 0.01495 1.88% MultiFab::LinComb() 1586 0.01459 0.01459 0.01459 1.83% FabArray::ParallelCopy() 861 0.01446 0.01446 0.01446 1.82% FabArray::setVal() 1144 0.01441 0.01441 0.01441 1.81% FabArray::ParallelCopy_nowait() 861 0.01419 0.01419 0.01419 1.78% MLCGSolver::ParallelAllReduce 1514 0.01352 0.01352 0.01352 1.70% MLMG::mgVcycle_up::2 82 0.01342 0.01342 0.01342 1.69% MLMG::mgVcycle_up::1 82 0.01318 0.01318 0.01318 1.66% Castro::normalize_species() 62 0.01316 0.01316 0.01316 1.65% MLCellLinOp::defineAuxData() 11 0.01299 0.01299 0.01299 1.63% MLMG::addInterpCorrection() 410 0.01278 0.01278 0.01278 1.61% MLMG::mgVcycle_up::3 82 0.01272 0.01272 0.01272 1.60% Castro::expand_state() 10 0.01266 0.01266 0.01266 1.59% MLMG::mgVcycle_up::4 82 0.01254 0.01254 0.01254 1.58% amrex::average_down 410 0.01208 0.01208 0.01208 1.52% MLPoisson::Fapply() 1142 0.01188 0.01188 0.01188 1.49% Castro::do_new_sources() 10 0.01113 0.01113 0.01113 1.40% Castro::initialize_advance() 10 0.01071 0.01071 0.01071 1.35% Castro::do_old_sources() 10 0.009625 0.009625 0.009625 1.21% Gravity::fill_multipole_BCs() 11 0.008391 0.008391 0.008391 1.05% Castro::enforce_min_density() 62 0.007344 0.007344 0.007344 0.92% MLCellLinOp::solutionResidual() 93 0.007255 0.007255 0.007255 0.91% MultiFab::Xpay() 585 0.006627 0.006627 0.006627 0.83% Castro::post_timestep() 10 0.00658 0.00658 0.00658 0.83% MLMG::computeResidual() 82 0.00625 0.00625 0.00625 0.79% Castro::reset_internal_energy(MultiFab) 63 0.005708 0.005708 0.005708 0.72% MLCellLinOp::defineBC() 11 0.005188 0.005188 0.005188 0.65% MLMG::prepareForSolve() 11 0.005181 0.005181 0.005181 0.65% BndryData::define() 11 0.004905 0.004905 0.004905 0.62% Castro::estTimeStep() 21 0.004735 0.004735 0.004735 0.59% Amr::InitializeInit() 1 0.004071 0.004071 0.004071 0.51% Amr::defBaseLevel() 1 0.004065 0.004065 0.004065 0.51% Castro::initData() 1 0.003542 0.003542 0.003542 0.44% Castro::construct_new_source() 50 0.003187 0.003187 0.003187 0.40% Castro::construct_new_gravity_source() 10 0.003169 0.003169 0.003169 0.40% Castro::construct_old_source() 50 0.002478 0.002478 0.002478 0.31% Castro::construct_old_gravity_source() 10 0.00246 0.00246 0.00246 0.31% Castro::computeNewDt() 9 0.002015 0.002015 0.002015 0.25% MLMG::ResNormInf() 93 0.00195 0.00195 0.00195 0.25% Castro::enforce_speed_limit() 62 0.001875 0.001875 0.001875 0.24% Castro::apply_source_to_state() 20 0.001819 0.001819 0.001819 0.23% MultiFab::Saxpy() 20 0.001808 0.001808 0.001808 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001664 0.001664 0.001664 0.21% MLCellLinOp::setLevelBC() 11 0.00157 0.00157 0.00157 0.20% Castro::reset_internal_energy(Fab) 504 0.00154 0.00154 0.00154 0.19% FabArrayBase::getCPC() 1323 0.001473 0.001473 0.001473 0.19% MLMG::getGradSolution() 11 0.001419 0.001419 0.001419 0.18% MLCellLinOp::compGrad() 11 0.001412 0.001412 0.001412 0.18% FabArray::mult() 43 0.001342 0.001342 0.001342 0.17% FabArray::setDomainBndry() 41 0.001313 0.001313 0.001313 0.16% Castro::post_regrid() 1 0.001245 0.001245 0.001245 0.16% MLPoisson::prepareForSolve() 11 0.001194 0.001194 0.001194 0.15% Castro::check_for_nan() 20 0.001186 0.001186 0.001186 0.15% MLCellLinOp::prepareForSolve() 11 0.001185 0.001185 0.001185 0.15% MultiFab::contains_nan() 20 0.001175 0.001175 0.001175 0.15% MLMG::computeMLResidual() 11 0.001043 0.001043 0.001043 0.13% Gravity::update_max_rhs() 11 0.0008257 0.0008257 0.0008257 0.10% FabArrayBase::getFB() 4023 0.0007354 0.0007354 0.0007354 0.09% FabArrayBase::CPC::define() 454 0.0006972 0.0006972 0.0006972 0.09% Castro::computeInitialDt() 2 0.0006325 0.0006325 0.0006325 0.08% Amr::InitAmr() 1 0.0004996 0.0004996 0.0004996 0.06% Gravity::swapTimeLevels() 10 0.0004487 0.0004487 0.0004487 0.06% Castro::Castro() 1 0.0004471 0.0004471 0.0004471 0.06% MLLinOp::define() 11 0.0002802 0.0002802 0.0002802 0.04% MLLinOp::defineGrids() 11 0.0002582 0.0002582 0.0002582 0.03% MultiFab::max() 11 0.0002564 0.0002564 0.0002564 0.03% MLMG::MLResNormInf() 11 0.0002563 0.0002563 0.0002563 0.03% MultiFab::Copy() 11 0.0002563 0.0002563 0.0002563 0.03% MLMG::MLRhsNormInf() 11 0.000206 0.000206 0.000206 0.03% Castro::buildMetrics() 1 0.000164 0.000164 0.000164 0.02% FabArrayBase::FB::FB() 56 8.73e-05 8.73e-05 8.73e-05 0.01% Castro::finalize_advance() 10 6.132e-05 6.132e-05 6.132e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.289e-05 5.289e-05 5.289e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.68e-05 4.68e-05 4.68e-05 0.01% StateData::define() 4 4.033e-05 4.033e-05 4.033e-05 0.01% makeSFC 55 4.008e-05 4.008e-05 4.008e-05 0.01% Castro::swap_state_time_levels() 10 3.958e-05 3.958e-05 3.958e-05 0.00% Castro::finalize_do_advance() 10 3.631e-05 3.631e-05 3.631e-05 0.00% Castro::enforce_consistent_e() 1 3.331e-05 3.331e-05 3.331e-05 0.00% Castro::initMFs() 1 2.94e-05 2.94e-05 2.94e-05 0.00% Amr::writeSmallPlotFile() 1 2.531e-05 2.531e-05 2.531e-05 0.00% DistributionMapping::Distribute() 56 1.47e-05 1.47e-05 1.47e-05 0.00% Amr::initSubcycle() 1 8.794e-06 8.794e-06 8.794e-06 0.00% MLMG::buildFineMask() 11 5.126e-06 5.126e-06 5.126e-06 0.00% AmrLevel::checkPointPost() 3 4.719e-06 4.719e-06 4.719e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.517e-06 4.517e-06 4.517e-06 0.00% Castro::retry_advance_ctu() 10 4.233e-06 4.233e-06 4.233e-06 0.00% Castro::create_source_corrector() 10 3.941e-06 3.941e-06 3.941e-06 0.00% Gravity::set_mass_offset() 11 3.683e-06 3.683e-06 3.683e-06 0.00% Castro::FluxRegCrseInit 10 2.886e-06 2.886e-06 2.886e-06 0.00% Castro::FluxRegFineAdd() 10 2.188e-06 2.188e-06 2.188e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.871e-06 1.871e-06 1.871e-06 0.00% AmrLevel::checkPointPre() 3 1.801e-06 1.801e-06 1.801e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.06-13-g7b335c5f9617) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.06-13-g7b335c5f9617) initialized Starting run at 08:26:41 UTC on 2022-06-09. Successfully read inputs file ... Castro git describe: 22.06-9-g26873b1cc AMReX git describe: 22.06-13-g7b335c5f9 Microphysics git describe: 22.06-1-gb8e427c1 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.448071571 Restart time = 0.047053637 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052864077 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.0500635 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.059652486 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.067800786 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.071198272 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.02637973 seconds Ending run at 08:26:42 UTC on 2022-06-09. Run time = 0.375991711 Run time without initialization = 0.328380882 Average number of zones advanced per microsecond: 3.991 Average number of zones advanced per microsecond per rank: 3.991 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.376 ... 0.376 ... 0.376 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1028 0.1028 0.1028 27.33% VisMF::Read() 3 0.03974 0.03974 0.03974 10.57% MLCellLinOp::applyBC() 1946 0.03506 0.03506 0.03506 9.32% MLPoisson::Fsmooth() 1440 0.02743 0.02743 0.02743 7.29% VisMF::Write(FabArray) 1 0.025 0.025 0.025 6.65% StateData::FillBoundary(geom) 160 0.01165 0.01165 0.01165 3.10% MLCGSolver::bicgstab 36 0.01034 0.01034 0.01034 2.75% MultiFab::Dot() 484 0.009564 0.009564 0.009564 2.54% Castro::normalize_species() 30 0.008173 0.008173 0.008173 2.17% Castro::enforce_min_density() 30 0.007444 0.007444 0.007444 1.98% FabArray::setVal() 537 0.006784 0.006784 0.006784 1.80% MLCellLinOp::defineAuxData() 6 0.006266 0.006266 0.006266 1.67% FillBoundary_nowait() 1766 0.006235 0.006235 0.006235 1.66% MultiFab::LinComb() 690 0.006128 0.006128 0.006128 1.63% FabArray::ParallelCopy_nowait() 380 0.005962 0.005962 0.005962 1.59% Castro::computeTemp() 30 0.005669 0.005669 0.005669 1.51% StateDataPhysBCFunct::() 20 0.005207 0.005207 0.005207 1.38% MLPoisson::Fapply() 500 0.005076 0.005076 0.005076 1.35% Gravity::fill_multipole_BCs() 6 0.004196 0.004196 0.004196 1.12% MLMG::addInterpCorrection() 180 0.003244 0.003244 0.003244 0.86% Amr::restart() 1 0.003037 0.003037 0.003037 0.81% amrex::average_down 180 0.002957 0.002957 0.002957 0.79% MultiFab::Xpay() 258 0.002826 0.002826 0.002826 0.75% Castro::do_advance_ctu() 5 0.002612 0.002612 0.002612 0.69% Castro::estTimeStep() 10 0.002229 0.002229 0.002229 0.59% BndryData::define() 6 0.002143 0.002143 0.002143 0.57% Castro::reset_internal_energy(MultiFab) 30 0.001805 0.001805 0.001805 0.48% Castro::construct_new_gravity_source() 5 0.001607 0.001607 0.001607 0.43% Amr::writePlotFile() 1 0.001482 0.001482 0.001482 0.39% Castro::subcycle_advance_ctu() 5 0.001473 0.001473 0.001473 0.39% Castro::advance() 5 0.001371 0.001371 0.001371 0.36% Castro::construct_old_gravity_source() 5 0.001292 0.001292 0.001292 0.34% MultiFab::Saxpy() 10 0.0009164 0.0009164 0.0009164 0.24% Gravity::get_old_grav_vector() 5 0.0009126 0.0009126 0.0009126 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008895 0.0008895 0.0008895 0.24% Castro::expand_state() 5 0.0008696 0.0008696 0.0008696 0.23% Gravity::get_new_grav_vector() 5 0.0008654 0.0008654 0.0008654 0.23% MLMG::ResNormInf() 42 0.000851 0.000851 0.000851 0.23% MLCellLinOp::setLevelBC() 6 0.0008205 0.0008205 0.0008205 0.22% Castro::reset_internal_energy(Fab) 240 0.0007968 0.0007968 0.0007968 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007839 0.0007839 0.0007839 0.21% MLMG::oneIter() 36 0.0007393 0.0007393 0.0007393 0.20% FabArray::setDomainBndry() 20 0.0006618 0.0006618 0.0006618 0.18% FabArray::mult() 22 0.0006584 0.0006584 0.0006584 0.18% MLCellLinOp::prepareForSolve() 6 0.000644 0.000644 0.000644 0.17% MultiFab::contains_nan() 10 0.0005859 0.0005859 0.0005859 0.16% MLMG::prepareForSolve() 6 0.0005627 0.0005627 0.0005627 0.15% MLCellLinOp::smooth() 720 0.0005337 0.0005337 0.0005337 0.14% Castro::enforce_speed_limit() 30 0.0005094 0.0005094 0.0005094 0.14% MLCellLinOp::compGrad() 6 0.000496 0.000496 0.000496 0.13% Amr::InitAmr() 1 0.0004037 0.0004037 0.0004037 0.11% FabArrayBase::CPC::define() 244 0.0003991 0.0003991 0.0003991 0.11% FabArrayBase::getCPC() 632 0.0003803 0.0003803 0.0003803 0.10% FabArray::FillBoundary() 1766 0.0003663 0.0003663 0.0003663 0.10% FabArrayBase::getFB() 1766 0.0002861 0.0002861 0.0002861 0.08% main() 1 0.0002517 0.0002517 0.0002517 0.07% Gravity::update_max_rhs() 6 0.0002287 0.0002287 0.0002287 0.06% Gravity::solve_for_phi() 5 0.0002175 0.0002175 0.0002175 0.06% MLCellLinOp::apply() 500 0.0002171 0.0002171 0.0002171 0.06% CGSolver::sxay() 690 0.0001701 0.0001701 0.0001701 0.05% Amr::coarseTimeStep() 5 0.0001588 0.0001588 0.0001588 0.04% Castro::post_timestep() 5 0.0001558 0.0001558 0.0001558 0.04% MLCellLinOp::defineBC() 6 0.0001513 0.0001513 0.0001513 0.04% FillPatchIterator::Initialize 20 0.0001467 0.0001467 0.0001467 0.04% MLLinOp::defineGrids() 6 0.0001433 0.0001433 0.0001433 0.04% MultiFab::Copy() 6 0.0001376 0.0001376 0.0001376 0.04% Castro::construct_new_gravity() 5 0.0001356 0.0001356 0.0001356 0.04% MultiFab::max() 6 0.0001337 0.0001337 0.0001337 0.04% FabArray::ParallelCopy() 380 0.0001308 0.0001308 0.0001308 0.03% MLCGSolver::ParallelAllReduce 659 0.000121 0.000121 0.000121 0.03% MLMG::MLRhsNormInf() 6 0.0001067 0.0001067 0.0001067 0.03% MLCellLinOp::correctionResidual() 216 9.857e-05 9.857e-05 9.857e-05 0.03% Amr::timeStep() 5 9.686e-05 9.686e-05 9.686e-05 0.03% MLMG::mgVcycle() 36 9.572e-05 9.572e-05 9.572e-05 0.03% AmrLevel::restart() 1 7.675e-05 7.675e-05 7.675e-05 0.02% StateData::restartDoit() 4 6.852e-05 6.852e-05 6.852e-05 0.02% Castro::construct_new_source() 25 5.989e-05 5.989e-05 5.989e-05 0.02% MLMG:computeResOfCorrection() 180 5.598e-05 5.598e-05 5.598e-05 0.01% FabArrayBase::FB::FB() 26 5.594e-05 5.594e-05 5.594e-05 0.01% Castro::create_source_corrector() 5 5.499e-05 5.499e-05 5.499e-05 0.01% Castro::initialize_do_advance() 5 4.45e-05 4.45e-05 4.45e-05 0.01% MLMG::actualBottomSolve() 36 4.442e-05 4.442e-05 4.442e-05 0.01% Castro::post_restart() 1 4.021e-05 4.021e-05 4.021e-05 0.01% MLMG::mgVcycle_down::0 36 3.965e-05 3.965e-05 3.965e-05 0.01% Castro::initialize_advance() 5 3.954e-05 3.954e-05 3.954e-05 0.01% Castro::finalize_advance() 5 3.83e-05 3.83e-05 3.83e-05 0.01% Castro::clean_state() 30 3.816e-05 3.816e-05 3.816e-05 0.01% Castro::do_new_sources() 5 3.73e-05 3.73e-05 3.73e-05 0.01% MLMG::mgVcycle_down::1 36 3.727e-05 3.727e-05 3.727e-05 0.01% MLMG::mgVcycle_down::2 36 3.671e-05 3.671e-05 3.671e-05 0.01% MLMG::solve() 6 3.6e-05 3.6e-05 3.6e-05 0.01% MLMG::mgVcycle_down::4 36 3.507e-05 3.507e-05 3.507e-05 0.01% MLMG::mgVcycle_down::3 36 3.353e-05 3.353e-05 3.353e-05 0.01% Castro::buildMetrics() 1 3.232e-05 3.232e-05 3.232e-05 0.01% Gravity::actual_multilevel_solve() 1 3.018e-05 3.018e-05 3.018e-05 0.01% Amr::writeSmallPlotFile() 1 2.851e-05 2.851e-05 2.851e-05 0.01% MLMG::mgVcycle_up::4 36 2.798e-05 2.798e-05 2.798e-05 0.01% Castro::initMFs() 1 2.759e-05 2.759e-05 2.759e-05 0.01% MLMG::mgVcycle_up::2 36 2.705e-05 2.705e-05 2.705e-05 0.01% Castro::swap_state_time_levels() 5 2.701e-05 2.701e-05 2.701e-05 0.01% MLMG::mgVcycle_up::3 36 2.395e-05 2.395e-05 2.395e-05 0.01% MLMG::mgVcycle_up::0 36 2.376e-05 2.376e-05 2.376e-05 0.01% MLCellLinOp::solutionResidual() 42 2.368e-05 2.368e-05 2.368e-05 0.01% Gravity::solve_phi_with_mlmg() 6 2.19e-05 2.19e-05 2.19e-05 0.01% MLMG::mgVcycle_up::1 36 2.16e-05 2.16e-05 2.16e-05 0.01% MLLinOp::define() 6 2.142e-05 2.142e-05 2.142e-05 0.01% Castro::finalize_do_advance() 5 1.823e-05 1.823e-05 1.823e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.785e-05 1.785e-05 1.785e-05 0.00% MLMG::computeResidual() 36 1.62e-05 1.62e-05 1.62e-05 0.00% MLMG::mgVcycle_bottom 36 1.549e-05 1.549e-05 1.549e-05 0.00% MLPoisson::define() 6 1.508e-05 1.508e-05 1.508e-05 0.00% makeSFC 30 1.411e-05 1.411e-05 1.411e-05 0.00% FillPatchSingleLevel 20 1.397e-05 1.397e-05 1.397e-05 0.00% Castro::construct_old_source() 25 1.025e-05 1.025e-05 1.025e-05 0.00% DistributionMapping::Distribute() 31 9.414e-06 9.414e-06 9.414e-06 0.00% Castro::do_old_sources() 5 8.16e-06 8.16e-06 8.16e-06 0.00% Amr::initSubcycle() 1 8.037e-06 8.037e-06 8.037e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.361e-06 7.361e-06 7.361e-06 0.00% Castro::check_for_nan() 10 7.311e-06 7.311e-06 7.311e-06 0.00% Castro::construct_old_gravity() 5 5.991e-06 5.991e-06 5.991e-06 0.00% Castro::apply_source_to_state() 10 5.108e-06 5.108e-06 5.108e-06 0.00% Gravity::swapTimeLevels() 5 4.669e-06 4.669e-06 4.669e-06 0.00% MLPoisson::prepareForSolve() 6 4.419e-06 4.419e-06 4.419e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.189e-06 3.189e-06 3.189e-06 0.00% Castro::computeNewDt() 5 3.08e-06 3.08e-06 3.08e-06 0.00% MLMG::computeMLResidual() 6 2.986e-06 2.986e-06 2.986e-06 0.00% MLMG::getGradSolution() 6 2.94e-06 2.94e-06 2.94e-06 0.00% MLMG::buildFineMask() 6 2.839e-06 2.839e-06 2.839e-06 0.00% Gravity::set_mass_offset() 6 2.391e-06 2.391e-06 2.391e-06 0.00% MLMG::MLResNormInf() 6 2.378e-06 2.378e-06 2.378e-06 0.00% Castro::retry_advance_ctu() 5 2.155e-06 2.155e-06 2.155e-06 0.00% Castro::FluxRegCrseInit 5 1.633e-06 1.633e-06 1.633e-06 0.00% Amr::init() 1 1.159e-06 1.159e-06 1.159e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.139e-06 1.139e-06 1.139e-06 0.00% Castro::FluxRegFineAdd() 5 1.04e-06 1.04e-06 1.04e-06 0.00% AmrLevel::AmrLevel() 1 7.71e-07 7.71e-07 7.71e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.376 0.376 0.376 100.00% Amr::coarseTimeStep() 5 0.3017 0.3017 0.3017 80.24% Amr::timeStep() 5 0.3 0.3 0.3 79.78% Castro::advance() 5 0.2957 0.2957 0.2957 78.64% Castro::subcycle_advance_ctu() 5 0.2889 0.2889 0.2889 76.84% Castro::do_advance_ctu() 5 0.2875 0.2875 0.2875 76.45% Castro::construct_new_gravity() 5 0.1437 0.1437 0.1437 38.22% Gravity::solve_phi_with_mlmg() 6 0.1397 0.1397 0.1397 37.15% Gravity::solve_for_phi() 5 0.1362 0.1362 0.1362 36.22% Gravity::actual_solve_with_mlmg() 6 0.1354 0.1354 0.1354 36.00% MLMG::solve() 6 0.123 0.123 0.123 32.70% MLMG::oneIter() 36 0.116 0.116 0.116 30.84% MLMG::mgVcycle() 36 0.1152 0.1152 0.1152 30.64% Castro::construct_ctu_hydro_source() 5 0.1028 0.1028 0.1028 27.33% MLCellLinOp::smooth() 720 0.05912 0.05912 0.05912 15.72% Amr::init() 1 0.0471 0.0471 0.0471 12.53% Amr::restart() 1 0.0471 0.0471 0.0471 12.52% MLCellLinOp::applyBC() 1946 0.042 0.042 0.042 11.17% AmrLevel::restart() 1 0.03995 0.03995 0.03995 10.62% StateData::restartDoit() 4 0.03987 0.03987 0.03987 10.60% VisMF::Read() 3 0.03974 0.03974 0.03974 10.57% MLMG::mgVcycle_bottom 36 0.03526 0.03526 0.03526 9.38% MLMG::actualBottomSolve() 36 0.03524 0.03524 0.03524 9.37% MLCGSolver::bicgstab 36 0.03488 0.03488 0.03488 9.28% MLPoisson::Fsmooth() 1440 0.02743 0.02743 0.02743 7.29% Amr::writePlotFile() 1 0.02648 0.02648 0.02648 7.04% VisMF::Write(FabArray) 1 0.025 0.025 0.025 6.65% Castro::clean_state() 30 0.02443 0.02443 0.02443 6.50% FillPatchIterator::Initialize 20 0.01969 0.01969 0.01969 5.24% FillPatchSingleLevel 20 0.01888 0.01888 0.01888 5.02% StateDataPhysBCFunct::() 20 0.01686 0.01686 0.01686 4.48% MLCellLinOp::apply() 500 0.01587 0.01587 0.01587 4.22% MLMG::mgVcycle_down::0 36 0.01549 0.01549 0.01549 4.12% MLMG::mgVcycle_up::0 36 0.01325 0.01325 0.01325 3.52% StateData::FillBoundary(geom) 160 0.01165 0.01165 0.01165 3.10% Castro::initialize_do_advance() 5 0.01009 0.01009 0.01009 2.68% MLPoisson::define() 6 0.01005 0.01005 0.01005 2.67% MultiFab::Dot() 484 0.009564 0.009564 0.009564 2.54% MLCellLinOp::correctionResidual() 216 0.009234 0.009234 0.009234 2.46% Castro::computeTemp() 30 0.00827 0.00827 0.00827 2.20% Castro::normalize_species() 30 0.008173 0.008173 0.008173 2.17% MLMG:computeResOfCorrection() 180 0.007961 0.007961 0.007961 2.12% MLMG::mgVcycle_down::1 36 0.00768 0.00768 0.00768 2.04% Castro::enforce_min_density() 30 0.007444 0.007444 0.007444 1.98% MLMG::mgVcycle_down::2 36 0.007432 0.007432 0.007432 1.98% Castro::do_new_sources() 5 0.007405 0.007405 0.007405 1.97% Gravity::get_new_grav_vector() 5 0.007392 0.007392 0.007392 1.97% Castro::construct_old_gravity() 5 0.007303 0.007303 0.007303 1.94% Gravity::get_old_grav_vector() 5 0.007297 0.007297 0.007297 1.94% MLMG::mgVcycle_down::3 36 0.007072 0.007072 0.007072 1.88% MLCellLinOp::defineAuxData() 6 0.007013 0.007013 0.007013 1.87% FabArray::FillBoundary() 1766 0.006944 0.006944 0.006944 1.85% FabArray::setVal() 537 0.006784 0.006784 0.006784 1.80% MLMG::mgVcycle_down::4 36 0.006769 0.006769 0.006769 1.80% FillBoundary_nowait() 1766 0.006577 0.006577 0.006577 1.75% FabArray::ParallelCopy() 380 0.006481 0.006481 0.006481 1.72% FabArray::ParallelCopy_nowait() 380 0.00635 0.00635 0.00635 1.69% CGSolver::sxay() 690 0.006298 0.006298 0.006298 1.67% Castro::do_old_sources() 5 0.006162 0.006162 0.006162 1.64% MultiFab::LinComb() 690 0.006128 0.006128 0.006128 1.63% Castro::expand_state() 5 0.005773 0.005773 0.005773 1.54% MLCGSolver::ParallelAllReduce 659 0.005724 0.005724 0.005724 1.52% MLMG::mgVcycle_up::2 36 0.005711 0.005711 0.005711 1.52% MLMG::mgVcycle_up::1 36 0.005629 0.005629 0.005629 1.50% MLMG::addInterpCorrection() 180 0.005473 0.005473 0.005473 1.46% MLMG::mgVcycle_up::3 36 0.005427 0.005427 0.005427 1.44% MLMG::mgVcycle_up::4 36 0.005405 0.005405 0.005405 1.44% Castro::initialize_advance() 5 0.005348 0.005348 0.005348 1.42% amrex::average_down 180 0.005203 0.005203 0.005203 1.38% MLPoisson::Fapply() 500 0.005076 0.005076 0.005076 1.35% Castro::post_timestep() 5 0.004217 0.004217 0.004217 1.12% Gravity::fill_multipole_BCs() 6 0.004196 0.004196 0.004196 1.12% Castro::post_restart() 1 0.003923 0.003923 0.003923 1.04% Gravity::multilevel_solve_for_new_phi() 1 0.003788 0.003788 0.003788 1.01% Gravity::actual_multilevel_solve() 1 0.00377 0.00377 0.00377 1.00% MLCellLinOp::solutionResidual() 42 0.003233 0.003233 0.003233 0.86% MLCellLinOp::defineBC() 6 0.002827 0.002827 0.002827 0.75% MultiFab::Xpay() 258 0.002826 0.002826 0.002826 0.75% MLMG::prepareForSolve() 6 0.002757 0.002757 0.002757 0.73% MLMG::computeResidual() 36 0.002684 0.002684 0.002684 0.71% BndryData::define() 6 0.002676 0.002676 0.002676 0.71% Castro::reset_internal_energy(MultiFab) 30 0.002601 0.002601 0.002601 0.69% Castro::estTimeStep() 10 0.002229 0.002229 0.002229 0.59% Castro::construct_new_source() 25 0.001667 0.001667 0.001667 0.44% Castro::construct_new_gravity_source() 5 0.001607 0.001607 0.001607 0.43% Castro::computeNewDt() 5 0.00157 0.00157 0.00157 0.42% Castro::construct_old_source() 25 0.001302 0.001302 0.001302 0.35% Castro::construct_old_gravity_source() 5 0.001292 0.001292 0.001292 0.34% Castro::apply_source_to_state() 10 0.0009215 0.0009215 0.0009215 0.25% MultiFab::Saxpy() 10 0.0009164 0.0009164 0.0009164 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008895 0.0008895 0.0008895 0.24% MLMG::ResNormInf() 42 0.000851 0.000851 0.000851 0.23% MLCellLinOp::setLevelBC() 6 0.0008205 0.0008205 0.0008205 0.22% Castro::reset_internal_energy(Fab) 240 0.0007968 0.0007968 0.0007968 0.21% FabArrayBase::getCPC() 632 0.0007793 0.0007793 0.0007793 0.21% MLMG::getGradSolution() 6 0.0007659 0.0007659 0.0007659 0.20% MLCellLinOp::compGrad() 6 0.000763 0.000763 0.000763 0.20% FabArray::setDomainBndry() 20 0.0006618 0.0006618 0.0006618 0.18% FabArray::mult() 22 0.0006584 0.0006584 0.0006584 0.18% MLPoisson::prepareForSolve() 6 0.0006484 0.0006484 0.0006484 0.17% MLCellLinOp::prepareForSolve() 6 0.000644 0.000644 0.000644 0.17% Castro::check_for_nan() 10 0.0005932 0.0005932 0.0005932 0.16% MultiFab::contains_nan() 10 0.0005859 0.0005859 0.0005859 0.16% MLMG::computeMLResidual() 6 0.0005682 0.0005682 0.0005682 0.15% Castro::enforce_speed_limit() 30 0.0005094 0.0005094 0.0005094 0.14% Gravity::update_max_rhs() 6 0.0004407 0.0004407 0.0004407 0.12% Amr::InitAmr() 1 0.0004117 0.0004117 0.0004117 0.11% FabArrayBase::CPC::define() 244 0.0003991 0.0003991 0.0003991 0.11% FabArrayBase::getFB() 1766 0.000342 0.000342 0.000342 0.09% Gravity::swapTimeLevels() 5 0.0002258 0.0002258 0.0002258 0.06% MLLinOp::define() 6 0.0001947 0.0001947 0.0001947 0.05% MLLinOp::defineGrids() 6 0.0001733 0.0001733 0.0001733 0.05% Castro::buildMetrics() 1 0.0001581 0.0001581 0.0001581 0.04% MultiFab::Copy() 6 0.0001376 0.0001376 0.0001376 0.04% MLMG::MLResNormInf() 6 0.0001343 0.0001343 0.0001343 0.04% MultiFab::max() 6 0.0001337 0.0001337 0.0001337 0.04% MLMG::MLRhsNormInf() 6 0.0001067 0.0001067 0.0001067 0.03% FabArrayBase::FB::FB() 26 5.594e-05 5.594e-05 5.594e-05 0.01% Castro::create_source_corrector() 5 5.499e-05 5.499e-05 5.499e-05 0.01% Castro::finalize_advance() 5 4.098e-05 4.098e-05 4.098e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.882e-05 2.882e-05 2.882e-05 0.01% Amr::writeSmallPlotFile() 1 2.851e-05 2.851e-05 2.851e-05 0.01% Castro::initMFs() 1 2.759e-05 2.759e-05 2.759e-05 0.01% Castro::swap_state_time_levels() 5 2.701e-05 2.701e-05 2.701e-05 0.01% makeSFC 30 2.146e-05 2.146e-05 2.146e-05 0.01% Castro::finalize_do_advance() 5 1.823e-05 1.823e-05 1.823e-05 0.00% DistributionMapping::Distribute() 31 9.414e-06 9.414e-06 9.414e-06 0.00% Amr::initSubcycle() 1 8.037e-06 8.037e-06 8.037e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.256e-06 5.256e-06 5.256e-06 0.00% MLMG::buildFineMask() 6 2.839e-06 2.839e-06 2.839e-06 0.00% Gravity::set_mass_offset() 6 2.391e-06 2.391e-06 2.391e-06 0.00% Castro::retry_advance_ctu() 5 2.155e-06 2.155e-06 2.155e-06 0.00% Castro::FluxRegCrseInit 5 1.633e-06 1.633e-06 1.633e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.139e-06 1.139e-06 1.139e-06 0.00% Castro::FluxRegFineAdd() 5 1.04e-06 1.04e-06 1.04e-06 0.00% AmrLevel::AmrLevel() 1 7.71e-07 7.71e-07 7.71e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.06-13-g7b335c5f9617) finalized