Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.07-9-g40b3d2176b17) initialized Starting run at 08:44:08 UTC on 2022-07-18. Successfully read inputs file ... Castro git describe: 22.07-6-g8dff7e76c AMReX git describe: 22.07-9-g40b3d2176 Microphysics git describe: 22.07-13-g29a6e0f1 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.043811738 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.025125527 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.048785155 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.05224911 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.050432336 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.049598774 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.077890739 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.040947274 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.067356392 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.062252077 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.064343361 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.065475122 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.067720365 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.040514036 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.024768958 seconds Ending run at 08:44:09 UTC on 2022-07-18. Run time = 0.834779349 Run time without initialization = 0.712921354 Average number of zones advanced per microsecond: 3.677 Average number of zones advanced per microsecond per rank: 3.677 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8348 ... 0.8348 ... 0.8348 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2116 0.2116 0.2116 25.35% VisMF::Write(FabArray) 11 0.1684 0.1684 0.1684 20.18% MLCellLinOp::applyBC() 4433 0.08185 0.08185 0.08185 9.80% MLPoisson::Fsmooth() 3280 0.06421 0.06421 0.06421 7.69% MLCGSolver::bicgstab 82 0.02439 0.02439 0.02439 2.92% StateData::FillBoundary(geom) 328 0.0234 0.0234 0.0234 2.80% MultiFab::Dot() 1114 0.02276 0.02276 0.02276 2.73% Castro::normalize_species() 62 0.01728 0.01728 0.01728 2.07% MultiFab::LinComb() 1586 0.01465 0.01465 0.01465 1.76% Castro::computeTemp() 63 0.01459 0.01459 0.01459 1.75% FabArray::setVal() 1144 0.01444 0.01444 0.01444 1.73% FillBoundary_nowait() 4023 0.01422 0.01422 0.01422 1.70% FabArray::ParallelCopy_nowait() 861 0.01327 0.01327 0.01327 1.59% StateDataPhysBCFunct::() 41 0.01221 0.01221 0.01221 1.46% Castro::enforce_min_density() 62 0.01219 0.01219 0.01219 1.46% MLPoisson::Fapply() 1142 0.01188 0.01188 0.01188 1.42% MLCellLinOp::defineAuxData() 11 0.01157 0.01157 0.01157 1.39% Gravity::fill_multipole_BCs() 11 0.009473 0.009473 0.009473 1.13% MLMG::addInterpCorrection() 410 0.007518 0.007518 0.007518 0.90% amrex::average_down 410 0.006908 0.006908 0.006908 0.83% Castro::estTimeStep() 21 0.006894 0.006894 0.006894 0.83% MultiFab::Xpay() 585 0.006741 0.006741 0.006741 0.81% Castro::do_advance_ctu() 10 0.004942 0.004942 0.004942 0.59% Amr::checkPoint() 3 0.004314 0.004314 0.004314 0.52% BndryData::define() 11 0.003895 0.003895 0.003895 0.47% Castro::reset_internal_energy(MultiFab) 63 0.003769 0.003769 0.003769 0.45% Castro::construct_new_gravity_source() 10 0.003057 0.003057 0.003057 0.37% Castro::construct_old_gravity_source() 10 0.002759 0.002759 0.002759 0.33% Amr::writePlotFile() 2 0.002477 0.002477 0.002477 0.30% MLMG::ResNormInf() 93 0.001959 0.001959 0.001959 0.23% Gravity::get_new_grav_vector() 11 0.001936 0.001936 0.001936 0.23% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.22% Gravity::get_old_grav_vector() 10 0.001738 0.001738 0.001738 0.21% Castro::expand_state() 10 0.001734 0.001734 0.001734 0.21% MLMG::oneIter() 82 0.001711 0.001711 0.001711 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001659 0.001659 0.001659 0.20% MLCellLinOp::setLevelBC() 11 0.001567 0.001567 0.001567 0.19% Castro::reset_internal_energy(Fab) 504 0.001545 0.001545 0.001545 0.19% Gravity::actual_solve_with_mlmg() 11 0.001331 0.001331 0.001331 0.16% FabArray::mult() 43 0.001318 0.001318 0.001318 0.16% FabArray::setDomainBndry() 41 0.001306 0.001306 0.001306 0.16% Castro::initData() 1 0.00121 0.00121 0.00121 0.14% MLCellLinOp::prepareForSolve() 11 0.001203 0.001203 0.001203 0.14% MLCellLinOp::smooth() 1640 0.00118 0.00118 0.00118 0.14% MultiFab::contains_nan() 20 0.001175 0.001175 0.001175 0.14% MLMG::prepareForSolve() 11 0.001036 0.001036 0.001036 0.12% Castro::enforce_speed_limit() 62 0.001029 0.001029 0.001029 0.12% MLCellLinOp::compGrad() 11 0.0009285 0.0009285 0.0009285 0.11% FabArray::FillBoundary() 4023 0.000793 0.000793 0.000793 0.09% FabArrayBase::getCPC() 1323 0.0007603 0.0007603 0.0007603 0.09% Castro::subcycle_advance_ctu() 10 0.0006908 0.0006908 0.0006908 0.08% FabArrayBase::CPC::define() 454 0.0006651 0.0006651 0.0006651 0.08% FabArrayBase::getFB() 4023 0.0006076 0.0006076 0.0006076 0.07% Amr::InitAmr() 1 0.0005068 0.0005068 0.0005068 0.06% MLCellLinOp::apply() 1142 0.0004676 0.0004676 0.0004676 0.06% Gravity::solve_for_phi() 10 0.0004391 0.0004391 0.0004391 0.05% Gravity::update_max_rhs() 11 0.0004112 0.0004112 0.0004112 0.05% CGSolver::sxay() 1586 0.0003675 0.0003675 0.0003675 0.04% Amr::coarseTimeStep() 10 0.0003096 0.0003096 0.0003096 0.04% MLCGSolver::ParallelAllReduce 1514 0.0002969 0.0002969 0.0002969 0.04% FillPatchIterator::Initialize 41 0.0002927 0.0002927 0.0002927 0.04% FabArray::ParallelCopy() 861 0.0002885 0.0002885 0.0002885 0.03% MLCellLinOp::defineBC() 11 0.0002779 0.0002779 0.0002779 0.03% MultiFab::Copy() 11 0.0002612 0.0002612 0.0002612 0.03% main() 1 0.0002564 0.0002564 0.0002564 0.03% MultiFab::max() 11 0.0002546 0.0002546 0.0002546 0.03% MLCellLinOp::correctionResidual() 492 0.0002233 0.0002233 0.0002233 0.03% Castro::construct_new_gravity() 10 0.0002072 0.0002072 0.0002072 0.02% MLMG::MLRhsNormInf() 11 0.0002033 0.0002033 0.0002033 0.02% Amr::timeStep() 10 0.0001968 0.0001968 0.0001968 0.02% MLMG::mgVcycle() 82 0.0001868 0.0001868 0.0001868 0.02% MLLinOp::defineGrids() 11 0.0001534 0.0001534 0.0001534 0.02% MLMG:computeResOfCorrection() 410 0.0001279 0.0001279 0.0001279 0.02% StateData::checkPoint() 12 0.0001265 0.0001265 0.0001265 0.02% MLMG::actualBottomSolve() 82 9.907e-05 9.907e-05 9.907e-05 0.01% Castro::Castro() 1 9.831e-05 9.831e-05 9.831e-05 0.01% MLMG::mgVcycle_down::0 82 9.502e-05 9.502e-05 9.502e-05 0.01% Castro::create_source_corrector() 10 9.495e-05 9.495e-05 9.495e-05 0.01% Castro::clean_state() 62 8.797e-05 8.797e-05 8.797e-05 0.01% Castro::initialize_advance() 10 8.46e-05 8.46e-05 8.46e-05 0.01% FabArrayBase::FB::FB() 56 8.352e-05 8.352e-05 8.352e-05 0.01% MLMG::mgVcycle_down::1 82 8.317e-05 8.317e-05 8.317e-05 0.01% MLMG::mgVcycle_down::2 82 8.171e-05 8.171e-05 8.171e-05 0.01% MLMG::solve() 11 7.811e-05 7.811e-05 7.811e-05 0.01% MLMG::mgVcycle_down::3 82 7.608e-05 7.608e-05 7.608e-05 0.01% MLMG::mgVcycle_down::4 82 7.534e-05 7.534e-05 7.534e-05 0.01% AmrLevel::checkPoint() 3 7.04e-05 7.04e-05 7.04e-05 0.01% Castro::initialize_do_advance() 10 6.333e-05 6.333e-05 6.333e-05 0.01% MLMG::mgVcycle_up::4 82 5.912e-05 5.912e-05 5.912e-05 0.01% Castro::finalize_advance() 10 5.268e-05 5.268e-05 5.268e-05 0.01% MLMG::mgVcycle_up::2 82 5.192e-05 5.192e-05 5.192e-05 0.01% MLMG::mgVcycle_up::3 82 5.155e-05 5.155e-05 5.155e-05 0.01% MLMG::mgVcycle_up::0 82 5.154e-05 5.154e-05 5.154e-05 0.01% MLCellLinOp::solutionResidual() 93 5.154e-05 5.154e-05 5.154e-05 0.01% MLMG::mgVcycle_up::1 82 5.136e-05 5.136e-05 5.136e-05 0.01% Castro::swap_state_time_levels() 10 3.86e-05 3.86e-05 3.86e-05 0.00% StateData::define() 4 3.86e-05 3.86e-05 3.86e-05 0.00% Castro::post_timestep() 10 3.858e-05 3.858e-05 3.858e-05 0.00% Castro::enforce_consistent_e() 1 3.598e-05 3.598e-05 3.598e-05 0.00% Castro::finalize_do_advance() 10 3.549e-05 3.549e-05 3.549e-05 0.00% MLMG::computeResidual() 82 3.24e-05 3.24e-05 3.24e-05 0.00% MLMG::mgVcycle_bottom 82 3.226e-05 3.226e-05 3.226e-05 0.00% Gravity::actual_multilevel_solve() 1 3.117e-05 3.117e-05 3.117e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.966e-05 2.966e-05 2.966e-05 0.00% FillPatchSingleLevel 41 2.933e-05 2.933e-05 2.933e-05 0.00% Castro::initMFs() 1 2.898e-05 2.898e-05 2.898e-05 0.00% Amr::defBaseLevel() 1 2.648e-05 2.648e-05 2.648e-05 0.00% makeSFC 55 2.557e-05 2.557e-05 2.557e-05 0.00% Amr::writeSmallPlotFile() 1 2.511e-05 2.511e-05 2.511e-05 0.00% MLPoisson::define() 11 2.455e-05 2.455e-05 2.455e-05 0.00% Castro::buildMetrics() 1 2.43e-05 2.43e-05 2.43e-05 0.00% MLLinOp::define() 11 2.395e-05 2.395e-05 2.395e-05 0.00% Amr::FinalizeInit() 1 2.137e-05 2.137e-05 2.137e-05 0.00% Castro::construct_old_source() 50 1.881e-05 1.881e-05 1.881e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.822e-05 1.822e-05 1.822e-05 0.00% Castro::do_new_sources() 10 1.781e-05 1.781e-05 1.781e-05 0.00% Castro::advance() 10 1.755e-05 1.755e-05 1.755e-05 0.00% Castro::construct_new_source() 50 1.748e-05 1.748e-05 1.748e-05 0.00% Castro::do_old_sources() 10 1.667e-05 1.667e-05 1.667e-05 0.00% DistributionMapping::Distribute() 56 1.478e-05 1.478e-05 1.478e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.413e-05 1.413e-05 1.413e-05 0.00% Castro::check_for_nan() 20 1.259e-05 1.259e-05 1.259e-05 0.00% Castro::apply_source_to_state() 20 1.09e-05 1.09e-05 1.09e-05 0.00% Castro::construct_old_gravity() 10 1.019e-05 1.019e-05 1.019e-05 0.00% Gravity::swapTimeLevels() 10 9.816e-06 9.816e-06 9.816e-06 0.00% Amr::initSubcycle() 1 8.621e-06 8.621e-06 8.621e-06 0.00% MLPoisson::prepareForSolve() 11 8.105e-06 8.105e-06 8.105e-06 0.00% MLMG::computeMLResidual() 11 6.889e-06 6.889e-06 6.889e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.87e-06 6.87e-06 6.87e-06 0.00% Castro::computeNewDt() 9 6.756e-06 6.756e-06 6.756e-06 0.00% Amr::InitializeInit() 1 6.436e-06 6.436e-06 6.436e-06 0.00% MLMG::getGradSolution() 11 6.084e-06 6.084e-06 6.084e-06 0.00% MLMG::buildFineMask() 11 5.631e-06 5.631e-06 5.631e-06 0.00% AmrLevel::checkPointPost() 3 4.63e-06 4.63e-06 4.63e-06 0.00% MLMG::MLResNormInf() 11 4.488e-06 4.488e-06 4.488e-06 0.00% Castro::FluxRegCrseInit 10 3.857e-06 3.857e-06 3.857e-06 0.00% Castro::retry_advance_ctu() 10 3.852e-06 3.852e-06 3.852e-06 0.00% Castro::post_init() 1 3.713e-06 3.713e-06 3.713e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.586e-06 3.586e-06 3.586e-06 0.00% Gravity::set_mass_offset() 11 3.448e-06 3.448e-06 3.448e-06 0.00% Castro::computeInitialDt() 2 2.928e-06 2.928e-06 2.928e-06 0.00% Amr::init() 1 2.832e-06 2.832e-06 2.832e-06 0.00% Castro::FluxRegFineAdd() 10 2.152e-06 2.152e-06 2.152e-06 0.00% AmrLevel::checkPointPre() 3 2.059e-06 2.059e-06 2.059e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.876e-06 1.876e-06 1.876e-06 0.00% Castro::post_regrid() 1 1.265e-06 1.265e-06 1.265e-06 0.00% Amr::initialInit() 1 1.14e-06 1.14e-06 1.14e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8348 0.8348 0.8348 100.00% Amr::coarseTimeStep() 10 0.6879 0.6879 0.6879 82.41% Amr::timeStep() 10 0.6018 0.6018 0.6018 72.09% Castro::advance() 10 0.5946 0.5946 0.5946 71.23% Castro::subcycle_advance_ctu() 10 0.5839 0.5839 0.5839 69.95% Castro::do_advance_ctu() 10 0.5832 0.5832 0.5832 69.86% Gravity::solve_phi_with_mlmg() 11 0.3187 0.3187 0.3187 38.18% Gravity::actual_solve_with_mlmg() 11 0.309 0.309 0.309 37.02% Castro::construct_new_gravity() 10 0.2893 0.2893 0.2893 34.65% MLMG::solve() 11 0.2864 0.2864 0.2864 34.31% Gravity::solve_for_phi() 10 0.2742 0.2742 0.2742 32.85% MLMG::oneIter() 82 0.2717 0.2717 0.2717 32.55% MLMG::mgVcycle() 82 0.27 0.27 0.27 32.34% Castro::construct_ctu_hydro_source() 10 0.2116 0.2116 0.2116 25.35% VisMF::Write(FabArray) 11 0.1684 0.1684 0.1684 20.18% MLCellLinOp::smooth() 1640 0.138 0.138 0.138 16.54% Amr::checkPoint() 3 0.1254 0.1254 0.1254 15.02% Amr::init() 1 0.1212 0.1212 0.1212 14.52% AmrLevel::checkPoint() 3 0.1211 0.1211 0.1211 14.51% StateData::checkPoint() 12 0.121 0.121 0.121 14.50% MLCellLinOp::applyBC() 4433 0.09755 0.09755 0.09755 11.69% MLMG::mgVcycle_bottom 82 0.08344 0.08344 0.08344 9.99% MLMG::actualBottomSolve() 82 0.0834 0.0834 0.0834 9.99% MLCGSolver::bicgstab 82 0.08257 0.08257 0.08257 9.89% MLPoisson::Fsmooth() 3280 0.06421 0.06421 0.06421 7.69% Amr::initialInit() 1 0.05216 0.05216 0.05216 6.25% Amr::writePlotFile() 2 0.05003 0.05003 0.05003 5.99% Castro::clean_state() 62 0.0498 0.0498 0.0498 5.97% Amr::FinalizeInit() 1 0.04823 0.04823 0.04823 5.78% Castro::post_init() 1 0.04695 0.04695 0.04695 5.62% Gravity::multilevel_solve_for_new_phi() 1 0.04499 0.04499 0.04499 5.39% Gravity::actual_multilevel_solve() 1 0.04497 0.04497 0.04497 5.39% FillPatchIterator::Initialize 41 0.0412 0.0412 0.0412 4.93% FillPatchSingleLevel 41 0.0396 0.0396 0.0396 4.74% MLCellLinOp::apply() 1142 0.03676 0.03676 0.03676 4.40% MLMG::mgVcycle_down::0 82 0.03587 0.03587 0.03587 4.30% StateDataPhysBCFunct::() 41 0.0356 0.0356 0.0356 4.26% MLMG::mgVcycle_up::0 82 0.03061 0.03061 0.03061 3.67% StateData::FillBoundary(geom) 328 0.0234 0.0234 0.0234 2.80% Castro::initialize_do_advance() 10 0.02276 0.02276 0.02276 2.73% MultiFab::Dot() 1114 0.02276 0.02276 0.02276 2.73% MLCellLinOp::correctionResidual() 492 0.02167 0.02167 0.02167 2.60% Castro::computeTemp() 63 0.01991 0.01991 0.01991 2.38% MLMG:computeResOfCorrection() 410 0.01866 0.01866 0.01866 2.24% MLPoisson::define() 11 0.01826 0.01826 0.01826 2.19% MLMG::mgVcycle_down::1 82 0.0179 0.0179 0.0179 2.14% MLMG::mgVcycle_down::2 82 0.01751 0.01751 0.01751 2.10% Castro::normalize_species() 62 0.01728 0.01728 0.01728 2.07% Gravity::get_new_grav_vector() 11 0.01675 0.01675 0.01675 2.01% MLMG::mgVcycle_down::3 82 0.01661 0.01661 0.01661 1.99% MLMG::mgVcycle_down::4 82 0.01583 0.01583 0.01583 1.90% FabArray::FillBoundary() 4023 0.0157 0.0157 0.0157 1.88% CGSolver::sxay() 1586 0.01502 0.01502 0.01502 1.80% FillBoundary_nowait() 4023 0.01491 0.01491 0.01491 1.79% MultiFab::LinComb() 1586 0.01465 0.01465 0.01465 1.76% Castro::construct_old_gravity() 10 0.01462 0.01462 0.01462 1.75% Gravity::get_old_grav_vector() 10 0.01461 0.01461 0.01461 1.75% FabArray::setVal() 1144 0.01444 0.01444 0.01444 1.73% FabArray::ParallelCopy() 861 0.01437 0.01437 0.01437 1.72% FabArray::ParallelCopy_nowait() 861 0.01408 0.01408 0.01408 1.69% MLCGSolver::ParallelAllReduce 1514 0.01356 0.01356 0.01356 1.62% MLMG::mgVcycle_up::2 82 0.01347 0.01347 0.01347 1.61% MLMG::mgVcycle_up::1 82 0.01325 0.01325 0.01325 1.59% Castro::do_new_sources() 10 0.01303 0.01303 0.01303 1.56% MLCellLinOp::defineAuxData() 11 0.01289 0.01289 0.01289 1.54% MLMG::mgVcycle_up::3 82 0.01271 0.01271 0.01271 1.52% MLMG::addInterpCorrection() 410 0.01271 0.01271 0.01271 1.52% MLMG::mgVcycle_up::4 82 0.01259 0.01259 0.01259 1.51% Castro::enforce_min_density() 62 0.01219 0.01219 0.01219 1.46% amrex::average_down 410 0.01212 0.01212 0.01212 1.45% MLPoisson::Fapply() 1142 0.01188 0.01188 0.01188 1.42% Castro::do_old_sources() 10 0.01181 0.01181 0.01181 1.41% Castro::expand_state() 10 0.01126 0.01126 0.01126 1.35% Castro::initialize_advance() 10 0.01066 0.01066 0.01066 1.28% Gravity::fill_multipole_BCs() 11 0.009473 0.009473 0.009473 1.13% MLCellLinOp::solutionResidual() 93 0.007224 0.007224 0.007224 0.87% Castro::post_timestep() 10 0.006941 0.006941 0.006941 0.83% Castro::estTimeStep() 21 0.006894 0.006894 0.006894 0.83% MultiFab::Xpay() 585 0.006741 0.006741 0.006741 0.81% MLMG::computeResidual() 82 0.006222 0.006222 0.006222 0.75% Castro::reset_internal_energy(MultiFab) 63 0.005314 0.005314 0.005314 0.64% MLMG::prepareForSolve() 11 0.005227 0.005227 0.005227 0.63% MLCellLinOp::defineBC() 11 0.005116 0.005116 0.005116 0.61% BndryData::define() 11 0.004838 0.004838 0.004838 0.58% Amr::InitializeInit() 1 0.003924 0.003924 0.003924 0.47% Amr::defBaseLevel() 1 0.003917 0.003917 0.003917 0.47% Castro::computeNewDt() 9 0.003826 0.003826 0.003826 0.46% Castro::initData() 1 0.00338 0.00338 0.00338 0.40% Castro::construct_new_source() 50 0.003075 0.003075 0.003075 0.37% Castro::construct_new_gravity_source() 10 0.003057 0.003057 0.003057 0.37% Castro::construct_old_source() 50 0.002778 0.002778 0.002778 0.33% Castro::construct_old_gravity_source() 10 0.002759 0.002759 0.002759 0.33% MLMG::ResNormInf() 93 0.001959 0.001959 0.001959 0.23% Castro::apply_source_to_state() 20 0.001822 0.001822 0.001822 0.22% MultiFab::Saxpy() 20 0.001811 0.001811 0.001811 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001659 0.001659 0.001659 0.20% MLCellLinOp::setLevelBC() 11 0.001567 0.001567 0.001567 0.19% Castro::reset_internal_energy(Fab) 504 0.001545 0.001545 0.001545 0.19% FabArrayBase::getCPC() 1323 0.001425 0.001425 0.001425 0.17% MLMG::getGradSolution() 11 0.00142 0.00142 0.00142 0.17% MLCellLinOp::compGrad() 11 0.001414 0.001414 0.001414 0.17% FabArray::mult() 43 0.001318 0.001318 0.001318 0.16% FabArray::setDomainBndry() 41 0.001306 0.001306 0.001306 0.16% MLPoisson::prepareForSolve() 11 0.001211 0.001211 0.001211 0.15% MLCellLinOp::prepareForSolve() 11 0.001203 0.001203 0.001203 0.14% Castro::check_for_nan() 20 0.001187 0.001187 0.001187 0.14% MultiFab::contains_nan() 20 0.001175 0.001175 0.001175 0.14% Castro::post_regrid() 1 0.001079 0.001079 0.001079 0.13% MLMG::computeMLResidual() 11 0.001042 0.001042 0.001042 0.12% Castro::enforce_speed_limit() 62 0.001029 0.001029 0.001029 0.12% Gravity::update_max_rhs() 11 0.0008128 0.0008128 0.0008128 0.10% FabArrayBase::getFB() 4023 0.0006912 0.0006912 0.0006912 0.08% FabArrayBase::CPC::define() 454 0.0006651 0.0006651 0.0006651 0.08% Castro::computeInitialDt() 2 0.0006594 0.0006594 0.0006594 0.08% Amr::InitAmr() 1 0.0005154 0.0005154 0.0005154 0.06% Castro::Castro() 1 0.0004607 0.0004607 0.0004607 0.06% Gravity::swapTimeLevels() 10 0.000444 0.000444 0.000444 0.05% MultiFab::Copy() 11 0.0002612 0.0002612 0.0002612 0.03% MLMG::MLResNormInf() 11 0.0002582 0.0002582 0.0002582 0.03% MultiFab::max() 11 0.0002546 0.0002546 0.0002546 0.03% MLLinOp::define() 11 0.0002325 0.0002325 0.0002325 0.03% MLLinOp::defineGrids() 11 0.0002086 0.0002086 0.0002086 0.02% MLMG::MLRhsNormInf() 11 0.0002033 0.0002033 0.0002033 0.02% Castro::buildMetrics() 1 0.0001653 0.0001653 0.0001653 0.02% Castro::create_source_corrector() 10 9.495e-05 9.495e-05 9.495e-05 0.01% FabArrayBase::FB::FB() 56 8.352e-05 8.352e-05 8.352e-05 0.01% Castro::finalize_advance() 10 5.869e-05 5.869e-05 5.869e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.325e-05 5.325e-05 5.325e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.547e-05 4.547e-05 4.547e-05 0.01% makeSFC 55 3.912e-05 3.912e-05 3.912e-05 0.00% Castro::swap_state_time_levels() 10 3.86e-05 3.86e-05 3.86e-05 0.00% StateData::define() 4 3.86e-05 3.86e-05 3.86e-05 0.00% Castro::enforce_consistent_e() 1 3.598e-05 3.598e-05 3.598e-05 0.00% Castro::finalize_do_advance() 10 3.549e-05 3.549e-05 3.549e-05 0.00% Castro::initMFs() 1 2.898e-05 2.898e-05 2.898e-05 0.00% Amr::writeSmallPlotFile() 1 2.511e-05 2.511e-05 2.511e-05 0.00% DistributionMapping::Distribute() 56 1.478e-05 1.478e-05 1.478e-05 0.00% Amr::initSubcycle() 1 8.621e-06 8.621e-06 8.621e-06 0.00% MLMG::buildFineMask() 11 5.631e-06 5.631e-06 5.631e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.817e-06 4.817e-06 4.817e-06 0.00% AmrLevel::checkPointPost() 3 4.63e-06 4.63e-06 4.63e-06 0.00% Castro::FluxRegCrseInit 10 3.857e-06 3.857e-06 3.857e-06 0.00% Castro::retry_advance_ctu() 10 3.852e-06 3.852e-06 3.852e-06 0.00% Gravity::set_mass_offset() 11 3.448e-06 3.448e-06 3.448e-06 0.00% Castro::FluxRegFineAdd() 10 2.152e-06 2.152e-06 2.152e-06 0.00% AmrLevel::checkPointPre() 3 2.059e-06 2.059e-06 2.059e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.876e-06 1.876e-06 1.876e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.07-9-g40b3d2176b17) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.07-9-g40b3d2176b17) initialized Starting run at 08:44:10 UTC on 2022-07-18. Successfully read inputs file ... Castro git describe: 22.07-6-g8dff7e76c AMReX git describe: 22.07-9-g40b3d2176 Microphysics git describe: 22.07-13-g29a6e0f1 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.441785347 Restart time = 0.046063434 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.05109013 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048328728 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.050671361 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.048398749 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.068765895 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026084741 seconds Ending run at 08:44:10 UTC on 2022-07-18. Run time = 0.340355353 Run time without initialization = 0.293722326 Average number of zones advanced per microsecond: 4.462 Average number of zones advanced per microsecond per rank: 4.462 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.3404 ... 0.3404 ... 0.3404 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0738 0.0738 0.0738 21.68% VisMF::Read() 3 0.0388 0.0388 0.0388 11.40% MLCellLinOp::applyBC() 1946 0.03511 0.03511 0.03511 10.31% MLPoisson::Fsmooth() 1440 0.02697 0.02697 0.02697 7.92% VisMF::Write(FabArray) 1 0.02478 0.02478 0.02478 7.28% StateData::FillBoundary(geom) 160 0.01156 0.01156 0.01156 3.40% MLCGSolver::bicgstab 36 0.01025 0.01025 0.01025 3.01% MultiFab::Dot() 484 0.00952 0.00952 0.00952 2.80% Castro::normalize_species() 30 0.007554 0.007554 0.007554 2.22% FabArray::setVal() 537 0.006733 0.006733 0.006733 1.98% FillBoundary_nowait() 1766 0.006225 0.006225 0.006225 1.83% MLCellLinOp::defineAuxData() 6 0.006156 0.006156 0.006156 1.81% MultiFab::LinComb() 690 0.006076 0.006076 0.006076 1.79% FabArray::ParallelCopy_nowait() 380 0.005973 0.005973 0.005973 1.75% StateDataPhysBCFunct::() 20 0.005401 0.005401 0.005401 1.59% Castro::computeTemp() 30 0.005328 0.005328 0.005328 1.57% Castro::enforce_min_density() 30 0.005204 0.005204 0.005204 1.53% Gravity::fill_multipole_BCs() 6 0.005157 0.005157 0.005157 1.51% MLPoisson::Fapply() 500 0.005043 0.005043 0.005043 1.48% MLMG::addInterpCorrection() 180 0.00319 0.00319 0.00319 0.94% Amr::restart() 1 0.003052 0.003052 0.003052 0.90% amrex::average_down 180 0.002967 0.002967 0.002967 0.87% Castro::estTimeStep() 10 0.002936 0.002936 0.002936 0.86% MultiFab::Xpay() 258 0.002869 0.002869 0.002869 0.84% Castro::reset_internal_energy(MultiFab) 30 0.002246 0.002246 0.002246 0.66% Castro::do_advance_ctu() 5 0.002181 0.002181 0.002181 0.64% BndryData::define() 6 0.002106 0.002106 0.002106 0.62% Castro::construct_new_gravity_source() 5 0.001575 0.001575 0.001575 0.46% Castro::construct_old_gravity_source() 5 0.001408 0.001408 0.001408 0.41% Amr::writePlotFile() 1 0.00139 0.00139 0.00139 0.41% MultiFab::Saxpy() 10 0.0009206 0.0009206 0.0009206 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000898 0.000898 0.000898 0.26% Gravity::get_old_grav_vector() 5 0.0008811 0.0008811 0.0008811 0.26% Castro::expand_state() 5 0.0008735 0.0008735 0.0008735 0.26% Gravity::get_new_grav_vector() 5 0.0008648 0.0008648 0.0008648 0.25% MLMG::ResNormInf() 42 0.0008561 0.0008561 0.0008561 0.25% MLCellLinOp::setLevelBC() 6 0.0008174 0.0008174 0.0008174 0.24% MLMG::oneIter() 36 0.0007383 0.0007383 0.0007383 0.22% Gravity::actual_solve_with_mlmg() 6 0.0006929 0.0006929 0.0006929 0.20% Castro::reset_internal_energy(Fab) 240 0.0006749 0.0006749 0.0006749 0.20% FabArray::mult() 22 0.0006563 0.0006563 0.0006563 0.19% FabArray::setDomainBndry() 20 0.0006392 0.0006392 0.0006392 0.19% MLCellLinOp::prepareForSolve() 6 0.0006329 0.0006329 0.0006329 0.19% MultiFab::contains_nan() 10 0.0005862 0.0005862 0.0005862 0.17% Castro::enforce_speed_limit() 30 0.0005737 0.0005737 0.0005737 0.17% MLMG::prepareForSolve() 6 0.0005597 0.0005597 0.0005597 0.16% MLCellLinOp::smooth() 720 0.0005235 0.0005235 0.0005235 0.15% MLCellLinOp::compGrad() 6 0.0004898 0.0004898 0.0004898 0.14% Amr::InitAmr() 1 0.0003969 0.0003969 0.0003969 0.12% FabArrayBase::CPC::define() 244 0.0003936 0.0003936 0.0003936 0.12% FabArrayBase::getCPC() 632 0.0003603 0.0003603 0.0003603 0.11% FabArray::FillBoundary() 1766 0.0003589 0.0003589 0.0003589 0.11% FabArrayBase::getFB() 1766 0.0002597 0.0002597 0.0002597 0.08% main() 1 0.0002415 0.0002415 0.0002415 0.07% Gravity::update_max_rhs() 6 0.0002376 0.0002376 0.0002376 0.07% Gravity::solve_for_phi() 5 0.000222 0.000222 0.000222 0.07% MLCellLinOp::apply() 500 0.0001993 0.0001993 0.0001993 0.06% CGSolver::sxay() 690 0.0001766 0.0001766 0.0001766 0.05% Amr::coarseTimeStep() 5 0.0001517 0.0001517 0.0001517 0.04% MLCellLinOp::defineBC() 6 0.000146 0.000146 0.000146 0.04% FillPatchIterator::Initialize 20 0.0001421 0.0001421 0.0001421 0.04% MultiFab::Copy() 6 0.0001398 0.0001398 0.0001398 0.04% MultiFab::max() 6 0.0001359 0.0001359 0.0001359 0.04% FabArray::ParallelCopy() 380 0.0001324 0.0001324 0.0001324 0.04% Castro::subcycle_advance_ctu() 5 0.0001246 0.0001246 0.0001246 0.04% MLCGSolver::ParallelAllReduce 659 0.0001242 0.0001242 0.0001242 0.04% MLMG::MLRhsNormInf() 6 0.0001063 0.0001063 0.0001063 0.03% Castro::create_source_corrector() 5 0.0001056 0.0001056 0.0001056 0.03% Castro::construct_new_gravity() 5 0.0001038 0.0001038 0.0001038 0.03% Amr::timeStep() 5 9.898e-05 9.898e-05 9.898e-05 0.03% MLCellLinOp::correctionResidual() 216 9.576e-05 9.576e-05 9.576e-05 0.03% MLMG::mgVcycle() 36 8.536e-05 8.536e-05 8.536e-05 0.03% StateData::restartDoit() 4 7.654e-05 7.654e-05 7.654e-05 0.02% AmrLevel::restart() 1 7.414e-05 7.414e-05 7.414e-05 0.02% Castro::construct_new_source() 25 7.306e-05 7.306e-05 7.306e-05 0.02% MLLinOp::defineGrids() 6 6.743e-05 6.743e-05 6.743e-05 0.02% MLMG:computeResOfCorrection() 180 5.547e-05 5.547e-05 5.547e-05 0.02% FabArrayBase::FB::FB() 26 5.533e-05 5.533e-05 5.533e-05 0.02% Castro::initialize_advance() 5 5.01e-05 5.01e-05 5.01e-05 0.01% Castro::clean_state() 30 4.493e-05 4.493e-05 4.493e-05 0.01% MLMG::actualBottomSolve() 36 4.18e-05 4.18e-05 4.18e-05 0.01% MLMG::solve() 6 3.85e-05 3.85e-05 3.85e-05 0.01% MLMG::mgVcycle_down::0 36 3.8e-05 3.8e-05 3.8e-05 0.01% MLMG::mgVcycle_down::1 36 3.757e-05 3.757e-05 3.757e-05 0.01% MLMG::mgVcycle_down::2 36 3.693e-05 3.693e-05 3.693e-05 0.01% MLMG::mgVcycle_down::4 36 3.473e-05 3.473e-05 3.473e-05 0.01% MLMG::mgVcycle_down::3 36 3.361e-05 3.361e-05 3.361e-05 0.01% Castro::buildMetrics() 1 3.334e-05 3.334e-05 3.334e-05 0.01% Gravity::actual_multilevel_solve() 1 3.241e-05 3.241e-05 3.241e-05 0.01% Castro::post_restart() 1 3.174e-05 3.174e-05 3.174e-05 0.01% Castro::initialize_do_advance() 5 3.084e-05 3.084e-05 3.084e-05 0.01% MLMG::mgVcycle_up::4 36 2.832e-05 2.832e-05 2.832e-05 0.01% Castro::initMFs() 1 2.79e-05 2.79e-05 2.79e-05 0.01% Castro::swap_state_time_levels() 5 2.762e-05 2.762e-05 2.762e-05 0.01% Castro::finalize_advance() 5 2.599e-05 2.599e-05 2.599e-05 0.01% Amr::writeSmallPlotFile() 1 2.543e-05 2.543e-05 2.543e-05 0.01% MLMG::mgVcycle_up::2 36 2.499e-05 2.499e-05 2.499e-05 0.01% MLMG::mgVcycle_up::3 36 2.476e-05 2.476e-05 2.476e-05 0.01% MLMG::mgVcycle_up::0 36 2.437e-05 2.437e-05 2.437e-05 0.01% MLCellLinOp::solutionResidual() 42 2.396e-05 2.396e-05 2.396e-05 0.01% MLMG::mgVcycle_up::1 36 2.31e-05 2.31e-05 2.31e-05 0.01% MLLinOp::define() 6 1.904e-05 1.904e-05 1.904e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 1.834e-05 1.834e-05 1.834e-05 0.01% Castro::finalize_do_advance() 5 1.832e-05 1.832e-05 1.832e-05 0.01% MLMG::computeResidual() 36 1.481e-05 1.481e-05 1.481e-05 0.00% MLMG::mgVcycle_bottom 36 1.478e-05 1.478e-05 1.478e-05 0.00% MLPoisson::define() 6 1.473e-05 1.473e-05 1.473e-05 0.00% FillPatchSingleLevel 20 1.392e-05 1.392e-05 1.392e-05 0.00% makeSFC 30 1.359e-05 1.359e-05 1.359e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.309e-05 1.309e-05 1.309e-05 0.00% Castro::advance() 5 1.132e-05 1.132e-05 1.132e-05 0.00% Castro::construct_old_source() 25 1.051e-05 1.051e-05 1.051e-05 0.00% DistributionMapping::Distribute() 31 9.256e-06 9.256e-06 9.256e-06 0.00% Castro::do_new_sources() 5 8.791e-06 8.791e-06 8.791e-06 0.00% Amr::initSubcycle() 1 8.589e-06 8.589e-06 8.589e-06 0.00% Castro::do_old_sources() 5 8.362e-06 8.362e-06 8.362e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.253e-06 7.253e-06 7.253e-06 0.00% Castro::construct_old_gravity() 5 5.857e-06 5.857e-06 5.857e-06 0.00% Castro::check_for_nan() 10 5.44e-06 5.44e-06 5.44e-06 0.00% Castro::apply_source_to_state() 10 5.297e-06 5.297e-06 5.297e-06 0.00% Castro::post_timestep() 5 4.948e-06 4.948e-06 4.948e-06 0.00% Gravity::swapTimeLevels() 5 4.502e-06 4.502e-06 4.502e-06 0.00% MLPoisson::prepareForSolve() 6 4.367e-06 4.367e-06 4.367e-06 0.00% MLMG::buildFineMask() 6 3.239e-06 3.239e-06 3.239e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.196e-06 3.196e-06 3.196e-06 0.00% Castro::computeNewDt() 5 3.147e-06 3.147e-06 3.147e-06 0.00% MLMG::computeMLResidual() 6 3.032e-06 3.032e-06 3.032e-06 0.00% MLMG::getGradSolution() 6 2.755e-06 2.755e-06 2.755e-06 0.00% MLMG::MLResNormInf() 6 2.337e-06 2.337e-06 2.337e-06 0.00% Gravity::set_mass_offset() 6 2.069e-06 2.069e-06 2.069e-06 0.00% Castro::retry_advance_ctu() 5 1.977e-06 1.977e-06 1.977e-06 0.00% Castro::FluxRegCrseInit 5 1.835e-06 1.835e-06 1.835e-06 0.00% AmrLevel::AmrLevel() 1 1.446e-06 1.446e-06 1.446e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.377e-06 1.377e-06 1.377e-06 0.00% Castro::FluxRegFineAdd() 5 1.138e-06 1.138e-06 1.138e-06 0.00% Amr::init() 1 1.037e-06 1.037e-06 1.037e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3404 0.3404 0.3404 100.00% Amr::coarseTimeStep() 5 0.2674 0.2674 0.2674 78.56% Amr::timeStep() 5 0.2652 0.2652 0.2652 77.92% Castro::advance() 5 0.2622 0.2622 0.2622 77.03% Castro::subcycle_advance_ctu() 5 0.2571 0.2571 0.2571 75.52% Castro::do_advance_ctu() 5 0.2569 0.2569 0.2569 75.49% Castro::construct_new_gravity() 5 0.1439 0.1439 0.1439 42.26% Gravity::solve_phi_with_mlmg() 6 0.1396 0.1396 0.1396 41.01% Gravity::solve_for_phi() 5 0.1361 0.1361 0.1361 39.99% Gravity::actual_solve_with_mlmg() 6 0.1343 0.1343 0.1343 39.46% MLMG::solve() 6 0.1222 0.1222 0.1222 35.91% MLMG::oneIter() 36 0.1152 0.1152 0.1152 33.85% MLMG::mgVcycle() 36 0.1145 0.1145 0.1145 33.63% Castro::construct_ctu_hydro_source() 5 0.07379 0.07379 0.07379 21.68% MLCellLinOp::smooth() 720 0.05866 0.05866 0.05866 17.23% Amr::init() 1 0.04613 0.04613 0.04613 13.55% Amr::restart() 1 0.04613 0.04613 0.04613 13.55% MLCellLinOp::applyBC() 1946 0.04201 0.04201 0.04201 12.34% AmrLevel::restart() 1 0.039 0.039 0.039 11.46% StateData::restartDoit() 4 0.03892 0.03892 0.03892 11.44% VisMF::Read() 3 0.0388 0.0388 0.0388 11.40% MLMG::mgVcycle_bottom 36 0.03506 0.03506 0.03506 10.30% MLMG::actualBottomSolve() 36 0.03505 0.03505 0.03505 10.30% MLCGSolver::bicgstab 36 0.03469 0.03469 0.03469 10.19% MLPoisson::Fsmooth() 1440 0.02697 0.02697 0.02697 7.92% Amr::writePlotFile() 1 0.02617 0.02617 0.02617 7.69% VisMF::Write(FabArray) 1 0.02478 0.02478 0.02478 7.28% Castro::clean_state() 30 0.02163 0.02163 0.02163 6.35% FillPatchIterator::Initialize 20 0.01975 0.01975 0.01975 5.80% FillPatchSingleLevel 20 0.01897 0.01897 0.01897 5.57% StateDataPhysBCFunct::() 20 0.01696 0.01696 0.01696 4.98% MLCellLinOp::apply() 500 0.01582 0.01582 0.01582 4.65% MLMG::mgVcycle_down::0 36 0.01534 0.01534 0.01534 4.51% MLMG::mgVcycle_up::0 36 0.01311 0.01311 0.01311 3.85% StateData::FillBoundary(geom) 160 0.01156 0.01156 0.01156 3.40% Castro::initialize_do_advance() 5 0.01099 0.01099 0.01099 3.23% MLPoisson::define() 6 0.009819 0.009819 0.009819 2.88% MultiFab::Dot() 484 0.00952 0.00952 0.00952 2.80% MLCellLinOp::correctionResidual() 216 0.009218 0.009218 0.009218 2.71% Castro::computeTemp() 30 0.008248 0.008248 0.008248 2.42% MLMG:computeResOfCorrection() 180 0.007947 0.007947 0.007947 2.33% MLMG::mgVcycle_down::1 36 0.007652 0.007652 0.007652 2.25% Gravity::get_new_grav_vector() 5 0.007638 0.007638 0.007638 2.24% Castro::normalize_species() 30 0.007554 0.007554 0.007554 2.22% MLMG::mgVcycle_down::2 36 0.00742 0.00742 0.00742 2.18% Castro::construct_old_gravity() 5 0.007239 0.007239 0.007239 2.13% Gravity::get_old_grav_vector() 5 0.007233 0.007233 0.007233 2.12% Castro::do_new_sources() 5 0.007225 0.007225 0.007225 2.12% MLMG::mgVcycle_down::3 36 0.007048 0.007048 0.007048 2.07% FabArray::FillBoundary() 1766 0.006899 0.006899 0.006899 2.03% MLCellLinOp::defineAuxData() 6 0.006893 0.006893 0.006893 2.03% MLMG::mgVcycle_down::4 36 0.00675 0.00675 0.00675 1.98% FabArray::setVal() 537 0.006733 0.006733 0.006733 1.98% FillBoundary_nowait() 1766 0.00654 0.00654 0.00654 1.92% FabArray::ParallelCopy() 380 0.006477 0.006477 0.006477 1.90% FabArray::ParallelCopy_nowait() 380 0.006345 0.006345 0.006345 1.86% CGSolver::sxay() 690 0.006253 0.006253 0.006253 1.84% MultiFab::LinComb() 690 0.006076 0.006076 0.006076 1.79% MLCGSolver::ParallelAllReduce 659 0.005709 0.005709 0.005709 1.68% MLMG::mgVcycle_up::2 36 0.005675 0.005675 0.005675 1.67% MLMG::mgVcycle_up::1 36 0.005604 0.005604 0.005604 1.65% Castro::expand_state() 5 0.005594 0.005594 0.005594 1.64% MLMG::addInterpCorrection() 180 0.00542 0.00542 0.00542 1.59% MLMG::mgVcycle_up::3 36 0.005382 0.005382 0.005382 1.58% MLMG::mgVcycle_up::4 36 0.005351 0.005351 0.005351 1.57% amrex::average_down 180 0.005227 0.005227 0.005227 1.54% Castro::enforce_min_density() 30 0.005204 0.005204 0.005204 1.53% Gravity::fill_multipole_BCs() 6 0.005157 0.005157 0.005157 1.51% Castro::initialize_advance() 5 0.005084 0.005084 0.005084 1.49% MLPoisson::Fapply() 500 0.005043 0.005043 0.005043 1.48% Castro::do_old_sources() 5 0.00485 0.00485 0.00485 1.42% Castro::post_restart() 1 0.003887 0.003887 0.003887 1.14% Gravity::multilevel_solve_for_new_phi() 1 0.003758 0.003758 0.003758 1.10% Gravity::actual_multilevel_solve() 1 0.00374 0.00374 0.00374 1.10% MLCellLinOp::solutionResidual() 42 0.003239 0.003239 0.003239 0.95% Castro::post_timestep() 5 0.002943 0.002943 0.002943 0.86% Castro::estTimeStep() 10 0.002936 0.002936 0.002936 0.86% Castro::reset_internal_energy(MultiFab) 30 0.002921 0.002921 0.002921 0.86% MultiFab::Xpay() 258 0.002869 0.002869 0.002869 0.84% MLCellLinOp::defineBC() 6 0.002795 0.002795 0.002795 0.82% MLMG::prepareForSolve() 6 0.002751 0.002751 0.002751 0.81% MLMG::computeResidual() 36 0.00269 0.00269 0.00269 0.79% BndryData::define() 6 0.002649 0.002649 0.002649 0.78% Castro::computeNewDt() 5 0.002006 0.002006 0.002006 0.59% Castro::construct_new_source() 25 0.001648 0.001648 0.001648 0.48% Castro::construct_new_gravity_source() 5 0.001575 0.001575 0.001575 0.46% Castro::construct_old_source() 25 0.001418 0.001418 0.001418 0.42% Castro::construct_old_gravity_source() 5 0.001408 0.001408 0.001408 0.41% Castro::apply_source_to_state() 10 0.0009259 0.0009259 0.0009259 0.27% MultiFab::Saxpy() 10 0.0009206 0.0009206 0.0009206 0.27% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.000898 0.000898 0.000898 0.26% MLMG::ResNormInf() 42 0.0008561 0.0008561 0.0008561 0.25% MLCellLinOp::setLevelBC() 6 0.0008174 0.0008174 0.0008174 0.24% MLMG::getGradSolution() 6 0.0007595 0.0007595 0.0007595 0.22% MLCellLinOp::compGrad() 6 0.0007567 0.0007567 0.0007567 0.22% FabArrayBase::getCPC() 632 0.0007539 0.0007539 0.0007539 0.22% Castro::reset_internal_energy(Fab) 240 0.0006749 0.0006749 0.0006749 0.20% FabArray::mult() 22 0.0006563 0.0006563 0.0006563 0.19% FabArray::setDomainBndry() 20 0.0006392 0.0006392 0.0006392 0.19% MLPoisson::prepareForSolve() 6 0.0006372 0.0006372 0.0006372 0.19% MLCellLinOp::prepareForSolve() 6 0.0006329 0.0006329 0.0006329 0.19% Castro::check_for_nan() 10 0.0005917 0.0005917 0.0005917 0.17% MultiFab::contains_nan() 10 0.0005862 0.0005862 0.0005862 0.17% Castro::enforce_speed_limit() 30 0.0005737 0.0005737 0.0005737 0.17% MLMG::computeMLResidual() 6 0.0005666 0.0005666 0.0005666 0.17% Gravity::update_max_rhs() 6 0.0004623 0.0004623 0.0004623 0.14% Amr::InitAmr() 1 0.0004055 0.0004055 0.0004055 0.12% FabArrayBase::CPC::define() 244 0.0003936 0.0003936 0.0003936 0.12% FabArrayBase::getFB() 1766 0.000315 0.000315 0.000315 0.09% Gravity::swapTimeLevels() 5 0.0002297 0.0002297 0.0002297 0.07% Castro::buildMetrics() 1 0.0001543 0.0001543 0.0001543 0.05% MultiFab::Copy() 6 0.0001398 0.0001398 0.0001398 0.04% MultiFab::max() 6 0.0001359 0.0001359 0.0001359 0.04% MLMG::MLResNormInf() 6 0.0001355 0.0001355 0.0001355 0.04% MLLinOp::define() 6 0.0001161 0.0001161 0.0001161 0.03% MLMG::MLRhsNormInf() 6 0.0001063 0.0001063 0.0001063 0.03% Castro::create_source_corrector() 5 0.0001056 0.0001056 0.0001056 0.03% MLLinOp::defineGrids() 6 9.703e-05 9.703e-05 9.703e-05 0.03% FabArrayBase::FB::FB() 26 5.533e-05 5.533e-05 5.533e-05 0.02% Castro::finalize_advance() 5 2.897e-05 2.897e-05 2.897e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.822e-05 2.822e-05 2.822e-05 0.01% Castro::initMFs() 1 2.79e-05 2.79e-05 2.79e-05 0.01% Castro::swap_state_time_levels() 5 2.762e-05 2.762e-05 2.762e-05 0.01% Amr::writeSmallPlotFile() 1 2.543e-05 2.543e-05 2.543e-05 0.01% makeSFC 30 2.097e-05 2.097e-05 2.097e-05 0.01% Castro::finalize_do_advance() 5 1.832e-05 1.832e-05 1.832e-05 0.01% DistributionMapping::Distribute() 31 9.256e-06 9.256e-06 9.256e-06 0.00% Amr::initSubcycle() 1 8.589e-06 8.589e-06 8.589e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.072e-06 5.072e-06 5.072e-06 0.00% MLMG::buildFineMask() 6 3.239e-06 3.239e-06 3.239e-06 0.00% Gravity::set_mass_offset() 6 2.069e-06 2.069e-06 2.069e-06 0.00% Castro::retry_advance_ctu() 5 1.977e-06 1.977e-06 1.977e-06 0.00% Castro::FluxRegCrseInit 5 1.835e-06 1.835e-06 1.835e-06 0.00% AmrLevel::AmrLevel() 1 1.446e-06 1.446e-06 1.446e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.377e-06 1.377e-06 1.377e-06 0.00% Castro::FluxRegFineAdd() 5 1.138e-06 1.138e-06 1.138e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.07-9-g40b3d2176b17) finalized