Initializing CUDA... CUDA initialized with 1 device. AMReX (23.01-12-ga05384c0b7c4) initialized Starting run at 10:11:07 UTC on 2023-01-10. Successfully read inputs file ... Castro git describe: 23.01-12-g63195e409 AMReX git describe: 23.01-12-ga05384c0b Microphysics git describe: 23.01 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.058062295 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.033684124 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.047797015 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.056657892 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.070650007 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.077265625 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.072964777 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.058567052 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.059679782 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.058188207 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.068248045 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.069773223 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.059723421 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.05727639 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.03347865 seconds Ending run at 10:11:08 UTC on 2023-01-10. Run time = 0.931493144 Run time without initialization = 0.790994645 Average number of zones advanced per microsecond: 3.314 Average number of zones advanced per microsecond per rank: 3.314 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.9315 ... 0.9315 ... 0.9315 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2898 0.2898 0.2898 31.11% VisMF::Write(FabArray) 11 0.2333 0.2333 0.2333 25.04% MLCellLinOp::applyBC() 4433 0.07505 0.07505 0.07505 8.06% MLPoisson::Fsmooth() 3280 0.03231 0.03231 0.03231 3.47% FillBoundary_nowait() 4023 0.03037 0.03037 0.03037 3.26% StateData::FillBoundary(geom) 328 0.02878 0.02878 0.02878 3.09% amrex::Dot() 1114 0.02037 0.02037 0.02037 2.19% amrex::Copy() 1029 0.01489 0.01489 0.01489 1.60% FabArray::norminf() 743 0.01435 0.01435 0.01435 1.54% Castro::normalize_species() 62 0.01352 0.01352 0.01352 1.45% FabArray::setVal() 1144 0.01317 0.01317 0.01317 1.41% FabArray::ParallelCopy_nowait() 861 0.01312 0.01312 0.01312 1.41% Castro::computeTemp() 63 0.01306 0.01306 0.01306 1.40% StateDataPhysBCFunct::() 41 0.01181 0.01181 0.01181 1.27% Castro::enforce_min_density() 62 0.01112 0.01112 0.01112 1.19% MLPoisson::Fapply() 1142 0.01039 0.01039 0.01039 1.12% MLCellLinOp::defineAuxData() 11 0.009689 0.009689 0.009689 1.04% FabArray::Saxpy() 813 0.008108 0.008108 0.008108 0.87% FabArray::Xpay() 821 0.008038 0.008038 0.008038 0.86% MLMG::addInterpCorrection() 410 0.00656 0.00656 0.00656 0.70% amrex::average_down 410 0.005765 0.005765 0.005765 0.62% Amr::checkPoint() 3 0.004965 0.004965 0.004965 0.53% Gravity::fill_multipole_BCs() 11 0.004571 0.004571 0.004571 0.49% FabArray::LinComb() 557 0.004483 0.004483 0.004483 0.48% amrex::Add() 164 0.004292 0.004292 0.004292 0.46% Castro::reset_internal_energy(MultiFab) 63 0.003993 0.003993 0.003993 0.43% BndryData::define() 11 0.003669 0.003669 0.003669 0.39% Castro::estTimeStep() 21 0.003613 0.003613 0.003613 0.39% Amr::writePlotFile() 2 0.002391 0.002391 0.002391 0.26% Castro::construct_new_gravity_source() 10 0.002305 0.002305 0.002305 0.25% Castro::do_advance_ctu() 10 0.002286 0.002286 0.002286 0.25% MLCGSolver::bicgstab 82 0.002057 0.002057 0.002057 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001658 0.001658 0.001658 0.18% Castro::reset_internal_energy(Fab) 504 0.001649 0.001649 0.001649 0.18% Castro::construct_old_gravity_source() 10 0.001614 0.001614 0.001614 0.17% Castro::enforce_speed_limit() 62 0.001509 0.001509 0.001509 0.16% Gravity::actual_solve_with_mlmg() 11 0.001415 0.001415 0.001415 0.15% MLCellLinOp::setLevelBC() 11 0.001371 0.001371 0.001371 0.15% FabArray::mult() 43 0.001339 0.001339 0.001339 0.14% Castro::initData() 1 0.001337 0.001337 0.001337 0.14% FabArray::setDomainBndry() 41 0.001324 0.001324 0.001324 0.14% MultiFab::contains_nan() 20 0.001205 0.001205 0.001205 0.13% MLCellLinOp::smooth() 1640 0.001128 0.001128 0.001128 0.12% MLCellLinOp::prepareForSolve() 11 0.001106 0.001106 0.001106 0.12% MLCellLinOp::compGrad() 11 0.0009254 0.0009254 0.0009254 0.10% MLMG::prepareForSolve() 11 0.0008363 0.0008363 0.0008363 0.09% FabArray::FillBoundary() 4023 0.0008067 0.0008067 0.0008067 0.09% FabArrayBase::getCPC() 1323 0.0007325 0.0007325 0.0007325 0.08% FabArrayBase::CPC::define() 454 0.0006853 0.0006853 0.0006853 0.07% Gravity::get_new_grav_vector() 11 0.0005996 0.0005996 0.0005996 0.06% FabArrayBase::getFB() 4023 0.0005901 0.0005901 0.0005901 0.06% Gravity::get_old_grav_vector() 10 0.0005539 0.0005539 0.0005539 0.06% Amr::InitAmr() 1 0.0005218 0.0005218 0.0005218 0.06% MLCellLinOp::apply() 1142 0.0004439 0.0004439 0.0004439 0.05% MLMG::mgVcycle() 82 0.0004002 0.0004002 0.0004002 0.04% Amr::coarseTimeStep() 10 0.0003866 0.0003866 0.0003866 0.04% main() 1 0.0003182 0.0003182 0.0003182 0.03% MLCGSolver::ParallelAllReduce 1514 0.0002872 0.0002872 0.0002872 0.03% MultiFab::max() 11 0.0002653 0.0002653 0.0002653 0.03% MLCellLinOp::correctionResidual() 492 0.0002463 0.0002463 0.0002463 0.03% FabArray::ParallelCopy() 861 0.0002405 0.0002405 0.0002405 0.03% FillPatchIterator::Initialize 41 0.000231 0.000231 0.000231 0.02% MLCellLinOp::defineBC() 11 0.0002149 0.0002149 0.0002149 0.02% MLLinOp::defineGrids() 11 0.0002132 0.0002132 0.0002132 0.02% Amr::timeStep() 10 0.0001759 0.0001759 0.0001759 0.02% Gravity::solve_for_phi() 10 0.0001491 0.0001491 0.0001491 0.02% StateData::checkPoint() 12 0.0001313 0.0001313 0.0001313 0.01% Castro::subcycle_advance_ctu() 10 0.0001206 0.0001206 0.0001206 0.01% Gravity::update_max_rhs() 11 0.0001177 0.0001177 0.0001177 0.01% MLMG:computeResOfCorrection() 410 0.00011 0.00011 0.00011 0.01% Castro::clean_state() 62 9.558e-05 9.558e-05 9.558e-05 0.01% MLMG::mgVcycle_down::0 82 9.476e-05 9.476e-05 9.476e-05 0.01% Castro::expand_state() 10 9.171e-05 9.171e-05 9.171e-05 0.01% FabArrayBase::FB::FB() 56 8.888e-05 8.888e-05 8.888e-05 0.01% MLMG::actualBottomSolve() 82 8.582e-05 8.582e-05 8.582e-05 0.01% Castro::Castro() 1 8.072e-05 8.072e-05 8.072e-05 0.01% MLMG::mgVcycle_down::1 82 7.805e-05 7.805e-05 7.805e-05 0.01% MLMG::solve() 11 7.541e-05 7.541e-05 7.541e-05 0.01% AmrLevel::checkPoint() 3 7.433e-05 7.433e-05 7.433e-05 0.01% MLMG::mgVcycle_down::2 82 7.32e-05 7.32e-05 7.32e-05 0.01% Castro::initialize_advance() 10 7.021e-05 7.021e-05 7.021e-05 0.01% MLMG::mgVcycle_down::3 82 6.917e-05 6.917e-05 6.917e-05 0.01% Castro::finalize_advance() 10 6.861e-05 6.861e-05 6.861e-05 0.01% MLMG::mgVcycle_down::4 82 6.783e-05 6.783e-05 6.783e-05 0.01% Castro::advance() 10 6.059e-05 6.059e-05 6.059e-05 0.01% MLMG::oneIter() 82 5.757e-05 5.757e-05 5.757e-05 0.01% MLMG::mgVcycle_up::4 82 5.45e-05 5.45e-05 5.45e-05 0.01% MLMG::mgVcycle_up::0 82 5.243e-05 5.243e-05 5.243e-05 0.01% Castro::initialize_do_advance() 10 5.215e-05 5.215e-05 5.215e-05 0.01% MLCellLinOp::solutionResidual() 93 4.928e-05 4.928e-05 4.928e-05 0.01% MLMG::mgVcycle_up::1 82 4.729e-05 4.729e-05 4.729e-05 0.01% MLMG::mgVcycle_up::3 82 4.653e-05 4.653e-05 4.653e-05 0.00% MLMG::mgVcycle_up::2 82 4.502e-05 4.502e-05 4.502e-05 0.00% Castro::finalize_do_advance() 10 4.309e-05 4.309e-05 4.309e-05 0.00% MLMG::mgVcycle_bottom 82 3.557e-05 3.557e-05 3.557e-05 0.00% Castro::swap_state_time_levels() 10 3.541e-05 3.541e-05 3.541e-05 0.00% FillPatchSingleLevel 41 3.507e-05 3.507e-05 3.507e-05 0.00% Castro::enforce_consistent_e() 1 3.484e-05 3.484e-05 3.484e-05 0.00% MLMG::ResNormInf() 93 3.151e-05 3.151e-05 3.151e-05 0.00% Castro::construct_new_gravity() 10 3.07e-05 3.07e-05 3.07e-05 0.00% StateData::define() 4 3.032e-05 3.032e-05 3.032e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 2.96e-05 2.96e-05 2.96e-05 0.00% makeSFC 55 2.843e-05 2.843e-05 2.843e-05 0.00% MLMG::computeResidual() 82 2.797e-05 2.797e-05 2.797e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.691e-05 2.691e-05 2.691e-05 0.00% Amr::writeSmallPlotFile() 1 2.451e-05 2.451e-05 2.451e-05 0.00% MLPoisson::define() 11 2.413e-05 2.413e-05 2.413e-05 0.00% Amr::defBaseLevel() 1 2.22e-05 2.22e-05 2.22e-05 0.00% Amr::FinalizeInit() 1 2.195e-05 2.195e-05 2.195e-05 0.00% Castro::initMFs() 1 2.035e-05 2.035e-05 2.035e-05 0.00% Castro::construct_old_source() 50 1.902e-05 1.902e-05 1.902e-05 0.00% Castro::construct_new_source() 50 1.855e-05 1.855e-05 1.855e-05 0.00% Castro::buildMetrics() 1 1.81e-05 1.81e-05 1.81e-05 0.00% Castro::do_new_sources() 10 1.775e-05 1.775e-05 1.775e-05 0.00% Gravity::swapTimeLevels() 10 1.74e-05 1.74e-05 1.74e-05 0.00% DistributionMapping::Distribute() 56 1.638e-05 1.638e-05 1.638e-05 0.00% Castro::do_old_sources() 10 1.635e-05 1.635e-05 1.635e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.595e-05 1.595e-05 1.595e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.577e-05 1.577e-05 1.577e-05 0.00% Amr::InitializeInit() 1 1.576e-05 1.576e-05 1.576e-05 0.00% Castro::check_for_nan() 20 1.398e-05 1.398e-05 1.398e-05 0.00% Castro::construct_old_gravity() 10 1.184e-05 1.184e-05 1.184e-05 0.00% Castro::post_timestep() 10 1.183e-05 1.183e-05 1.183e-05 0.00% Castro::post_init() 1 1.127e-05 1.127e-05 1.127e-05 0.00% MLLinOp::define() 11 9.975e-06 9.975e-06 9.975e-06 0.00% Castro::apply_source_to_state() 20 9.723e-06 9.723e-06 9.723e-06 0.00% Amr::initSubcycle() 1 8.61e-06 8.61e-06 8.61e-06 0.00% MLPoisson::prepareForSolve() 11 8.346e-06 8.346e-06 8.346e-06 0.00% MLMG::computeMLResidual() 11 8.091e-06 8.091e-06 8.091e-06 0.00% Castro::computeNewDt() 9 7.859e-06 7.859e-06 7.859e-06 0.00% Gravity::actual_multilevel_solve() 1 7.406e-06 7.406e-06 7.406e-06 0.00% MLMG::getGradSolution() 11 5.901e-06 5.901e-06 5.901e-06 0.00% Gravity::set_mass_offset() 11 5.179e-06 5.179e-06 5.179e-06 0.00% Castro::retry_advance_ctu() 10 4.696e-06 4.696e-06 4.696e-06 0.00% Castro::create_source_corrector() 10 4.401e-06 4.401e-06 4.401e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.365e-06 4.365e-06 4.365e-06 0.00% AmrLevel::checkPointPost() 3 4.021e-06 4.021e-06 4.021e-06 0.00% MLMG::MLRhsNormInf() 11 3.849e-06 3.849e-06 3.849e-06 0.00% Castro::computeInitialDt() 2 3.476e-06 3.476e-06 3.476e-06 0.00% MLMG::MLResNormInf() 11 3.445e-06 3.445e-06 3.445e-06 0.00% Castro::FluxRegCrseInit 10 3.133e-06 3.133e-06 3.133e-06 0.00% Amr::init() 1 2.858e-06 2.858e-06 2.858e-06 0.00% Castro::FluxRegFineAdd() 10 2.667e-06 2.667e-06 2.667e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.287e-06 2.287e-06 2.287e-06 0.00% AmrLevel::checkPointPre() 3 1.815e-06 1.815e-06 1.815e-06 0.00% Castro::post_regrid() 1 1.714e-06 1.714e-06 1.714e-06 0.00% Amr::initialInit() 1 1.075e-06 1.075e-06 1.075e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.9315 0.9315 0.9315 100.00% Amr::coarseTimeStep() 10 0.7573 0.7573 0.7573 81.29% Amr::timeStep() 10 0.6388 0.6388 0.6388 68.58% Castro::advance() 10 0.6315 0.6315 0.6315 67.80% Castro::subcycle_advance_ctu() 10 0.6191 0.6191 0.6191 66.46% Castro::do_advance_ctu() 10 0.619 0.619 0.619 66.45% Castro::construct_ctu_hydro_source() 10 0.2898 0.2898 0.2898 31.11% Gravity::solve_phi_with_mlmg() 11 0.2775 0.2775 0.2775 29.79% Gravity::actual_solve_with_mlmg() 11 0.2725 0.2725 0.2725 29.25% Castro::construct_new_gravity() 10 0.2527 0.2527 0.2527 27.13% MLMG::solve() 11 0.2521 0.2521 0.2521 27.07% MLMG::oneIter() 82 0.238 0.238 0.238 25.55% Gravity::solve_for_phi() 10 0.2375 0.2375 0.2375 25.50% MLMG::mgVcycle() 82 0.2344 0.2344 0.2344 25.16% VisMF::Write(FabArray) 11 0.2333 0.2333 0.2333 25.04% Amr::checkPoint() 3 0.1741 0.1741 0.1741 18.69% AmrLevel::checkPoint() 3 0.1691 0.1691 0.1691 18.15% StateData::checkPoint() 12 0.169 0.169 0.169 18.14% Amr::init() 1 0.1398 0.1398 0.1398 15.01% MLCellLinOp::smooth() 1640 0.115 0.115 0.115 12.35% MLCellLinOp::applyBC() 4433 0.1069 0.1069 0.1069 11.48% MLMG::mgVcycle_bottom 82 0.07307 0.07307 0.07307 7.84% MLMG::actualBottomSolve() 82 0.07304 0.07304 0.07304 7.84% MLCGSolver::bicgstab 82 0.07233 0.07233 0.07233 7.76% Amr::writePlotFile() 2 0.0673 0.0673 0.0673 7.23% Amr::initialInit() 1 0.04794 0.04794 0.04794 5.15% FillPatchIterator::Initialize 41 0.04622 0.04622 0.04622 4.96% FillPatchSingleLevel 41 0.04467 0.04467 0.04467 4.80% Castro::clean_state() 62 0.04412 0.04412 0.04412 4.74% Amr::FinalizeInit() 1 0.0437 0.0437 0.0437 4.69% Castro::post_init() 1 0.04236 0.04236 0.04236 4.55% StateDataPhysBCFunct::() 41 0.04059 0.04059 0.04059 4.36% Gravity::multilevel_solve_for_new_phi() 1 0.04045 0.04045 0.04045 4.34% Gravity::actual_multilevel_solve() 1 0.04043 0.04043 0.04043 4.34% MLCellLinOp::apply() 1142 0.03571 0.03571 0.03571 3.83% MLMG::mgVcycle_down::0 82 0.03318 0.03318 0.03318 3.56% MLPoisson::Fsmooth() 3280 0.03231 0.03231 0.03231 3.47% FabArray::FillBoundary() 4023 0.03186 0.03186 0.03186 3.42% FillBoundary_nowait() 4023 0.03105 0.03105 0.03105 3.33% StateData::FillBoundary(geom) 328 0.02878 0.02878 0.02878 3.09% MLMG::mgVcycle_up::0 82 0.02492 0.02492 0.02492 2.68% Castro::initialize_do_advance() 10 0.02225 0.02225 0.02225 2.39% MLCellLinOp::correctionResidual() 492 0.02185 0.02185 0.02185 2.35% amrex::Dot() 1114 0.02037 0.02037 0.02037 2.19% MLMG:computeResOfCorrection() 410 0.01921 0.01921 0.01921 2.06% Castro::computeTemp() 63 0.01871 0.01871 0.01871 2.01% Gravity::get_new_grav_vector() 11 0.01678 0.01678 0.01678 1.80% MLPoisson::define() 11 0.01616 0.01616 0.01616 1.74% Castro::expand_state() 10 0.01595 0.01595 0.01595 1.71% MLMG::mgVcycle_down::1 82 0.01555 0.01555 0.01555 1.67% amrex::Copy() 1029 0.01489 0.01489 0.01489 1.60% Castro::construct_old_gravity() 10 0.01475 0.01475 0.01475 1.58% Gravity::get_old_grav_vector() 10 0.01474 0.01474 0.01474 1.58% MLMG::mgVcycle_down::2 82 0.01442 0.01442 0.01442 1.55% FabArray::norminf() 743 0.01435 0.01435 0.01435 1.54% MLMG::mgVcycle_down::3 82 0.01425 0.01425 0.01425 1.53% FabArray::ParallelCopy() 861 0.01417 0.01417 0.01417 1.52% MLMG::mgVcycle_down::4 82 0.01398 0.01398 0.01398 1.50% FabArray::ParallelCopy_nowait() 861 0.01393 0.01393 0.01393 1.50% Castro::normalize_species() 62 0.01352 0.01352 0.01352 1.45% FabArray::setVal() 1144 0.01317 0.01317 0.01317 1.41% MLCGSolver::ParallelAllReduce 1514 0.01223 0.01223 0.01223 1.31% Castro::initialize_advance() 10 0.01177 0.01177 0.01177 1.26% MLMG::addInterpCorrection() 410 0.01164 0.01164 0.01164 1.25% MLMG::mgVcycle_up::4 82 0.01145 0.01145 0.01145 1.23% MLMG::mgVcycle_up::1 82 0.01126 0.01126 0.01126 1.21% Castro::do_new_sources() 10 0.01115 0.01115 0.01115 1.20% Castro::enforce_min_density() 62 0.01112 0.01112 0.01112 1.19% MLMG::mgVcycle_up::2 82 0.01104 0.01104 0.01104 1.19% MLCellLinOp::defineAuxData() 11 0.01102 0.01102 0.01102 1.18% MLMG::mgVcycle_up::3 82 0.01087 0.01087 0.01087 1.17% amrex::average_down 410 0.01081 0.01081 0.01081 1.16% MLPoisson::Fapply() 1142 0.01039 0.01039 0.01039 1.12% Castro::do_old_sources() 10 0.009706 0.009706 0.009706 1.04% FabArray::Saxpy() 813 0.008108 0.008108 0.008108 0.87% FabArray::Xpay() 821 0.008038 0.008038 0.008038 0.86% Castro::post_timestep() 10 0.007107 0.007107 0.007107 0.76% MLCellLinOp::solutionResidual() 93 0.007097 0.007097 0.007097 0.76% MLMG::computeResidual() 82 0.00611 0.00611 0.00611 0.66% Castro::reset_internal_energy(MultiFab) 63 0.005643 0.005643 0.005643 0.61% MLCellLinOp::defineBC() 11 0.004822 0.004822 0.004822 0.52% Gravity::fill_multipole_BCs() 11 0.004804 0.004804 0.004804 0.52% BndryData::define() 11 0.004607 0.004607 0.004607 0.49% MLMG::prepareForSolve() 11 0.004588 0.004588 0.004588 0.49% FabArray::LinComb() 557 0.004483 0.004483 0.004483 0.48% amrex::Add() 164 0.004292 0.004292 0.004292 0.46% Amr::InitializeInit() 1 0.004236 0.004236 0.004236 0.45% Amr::defBaseLevel() 1 0.00422 0.00422 0.00422 0.45% Castro::initData() 1 0.003725 0.003725 0.003725 0.40% Castro::estTimeStep() 21 0.003613 0.003613 0.003613 0.39% Castro::construct_new_source() 50 0.002323 0.002323 0.002323 0.25% Castro::construct_new_gravity_source() 10 0.002305 0.002305 0.002305 0.25% MLMG::ResNormInf() 93 0.002134 0.002134 0.002134 0.23% Castro::apply_source_to_state() 20 0.001834 0.001834 0.001834 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001658 0.001658 0.001658 0.18% Castro::reset_internal_energy(Fab) 504 0.001649 0.001649 0.001649 0.18% Castro::construct_old_source() 50 0.001633 0.001633 0.001633 0.18% Castro::construct_old_gravity_source() 10 0.001614 0.001614 0.001614 0.17% Castro::enforce_speed_limit() 62 0.001509 0.001509 0.001509 0.16% FabArrayBase::getCPC() 1323 0.001418 0.001418 0.001418 0.15% MLMG::getGradSolution() 11 0.001392 0.001392 0.001392 0.15% MLCellLinOp::compGrad() 11 0.001386 0.001386 0.001386 0.15% Castro::computeNewDt() 9 0.00138 0.00138 0.00138 0.15% MLCellLinOp::setLevelBC() 11 0.001371 0.001371 0.001371 0.15% FabArray::mult() 43 0.001339 0.001339 0.001339 0.14% FabArray::setDomainBndry() 41 0.001324 0.001324 0.001324 0.14% Castro::check_for_nan() 20 0.001219 0.001219 0.001219 0.13% MultiFab::contains_nan() 20 0.001205 0.001205 0.001205 0.13% Castro::post_regrid() 1 0.001158 0.001158 0.001158 0.12% MLPoisson::prepareForSolve() 11 0.001114 0.001114 0.001114 0.12% MLCellLinOp::prepareForSolve() 11 0.001106 0.001106 0.001106 0.12% MLMG::computeMLResidual() 11 0.001023 0.001023 0.001023 0.11% Castro::computeInitialDt() 2 0.0008849 0.0008849 0.0008849 0.09% Gravity::update_max_rhs() 11 0.0008418 0.0008418 0.0008418 0.09% FabArrayBase::CPC::define() 454 0.0006853 0.0006853 0.0006853 0.07% FabArrayBase::getFB() 4023 0.000679 0.000679 0.000679 0.07% Castro::finalize_advance() 10 0.0006055 0.0006055 0.0006055 0.07% Amr::InitAmr() 1 0.0005304 0.0005304 0.0005304 0.06% Gravity::swapTimeLevels() 10 0.0004497 0.0004497 0.0004497 0.05% Castro::Castro() 1 0.0004213 0.0004213 0.0004213 0.05% MLLinOp::define() 11 0.0002985 0.0002985 0.0002985 0.03% MLMG::MLResNormInf() 11 0.0002887 0.0002887 0.0002887 0.03% MLLinOp::defineGrids() 11 0.0002885 0.0002885 0.0002885 0.03% MultiFab::max() 11 0.0002653 0.0002653 0.0002653 0.03% MLMG::MLRhsNormInf() 11 0.0002164 0.0002164 0.0002164 0.02% Castro::buildMetrics() 1 0.0001595 0.0001595 0.0001595 0.02% FabArrayBase::FB::FB() 56 8.888e-05 8.888e-05 8.888e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 7.301e-05 7.301e-05 7.301e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.609e-05 4.609e-05 4.609e-05 0.00% makeSFC 55 4.341e-05 4.341e-05 4.341e-05 0.00% Castro::finalize_do_advance() 10 4.309e-05 4.309e-05 4.309e-05 0.00% Castro::swap_state_time_levels() 10 3.541e-05 3.541e-05 3.541e-05 0.00% Castro::enforce_consistent_e() 1 3.484e-05 3.484e-05 3.484e-05 0.00% StateData::define() 4 3.032e-05 3.032e-05 3.032e-05 0.00% Amr::writeSmallPlotFile() 1 2.451e-05 2.451e-05 2.451e-05 0.00% Castro::initMFs() 1 2.035e-05 2.035e-05 2.035e-05 0.00% DistributionMapping::Distribute() 56 1.638e-05 1.638e-05 1.638e-05 0.00% Amr::initSubcycle() 1 8.61e-06 8.61e-06 8.61e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.769e-06 5.769e-06 5.769e-06 0.00% Gravity::set_mass_offset() 11 5.179e-06 5.179e-06 5.179e-06 0.00% Castro::retry_advance_ctu() 10 4.696e-06 4.696e-06 4.696e-06 0.00% Castro::create_source_corrector() 10 4.401e-06 4.401e-06 4.401e-06 0.00% AmrLevel::checkPointPost() 3 4.021e-06 4.021e-06 4.021e-06 0.00% Castro::FluxRegCrseInit 10 3.133e-06 3.133e-06 3.133e-06 0.00% Castro::FluxRegFineAdd() 10 2.667e-06 2.667e-06 2.667e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.287e-06 2.287e-06 2.287e-06 0.00% AmrLevel::checkPointPre() 3 1.815e-06 1.815e-06 1.815e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-12-ga05384c0b7c4) finalized Initializing CUDA... CUDA initialized with 1 device. AMReX (23.01-12-ga05384c0b7c4) initialized Starting run at 10:11:09 UTC on 2023-01-10. Successfully read inputs file ... Castro git describe: 23.01-12-g63195e409 AMReX git describe: 23.01-12-ga05384c0b Microphysics git describe: 23.01 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.52444041 Restart time = 0.046960498 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.051550217 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.056009262 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.060501783 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059754658 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.075054983 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.03328149 seconds Ending run at 10:11:09 UTC on 2023-01-10. Run time = 0.384144173 Run time without initialization = 0.336572575 Average number of zones advanced per microsecond: 3.894 Average number of zones advanced per microsecond per rank: 3.894 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9481961472 TinyProfiler total time across processes [min...avg...max]: 0.3842 ... 0.3842 ... 0.3842 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1191 0.1191 0.1191 31.00% VisMF::Read() 3 0.04073 0.04073 0.04073 10.60% MLCellLinOp::applyBC() 1946 0.03277 0.03277 0.03277 8.53% VisMF::Write(FabArray) 1 0.03174 0.03174 0.03174 8.26% MLPoisson::Fsmooth() 1440 0.01399 0.01399 0.01399 3.64% FillBoundary_nowait() 1766 0.01275 0.01275 0.01275 3.32% StateData::FillBoundary(geom) 160 0.01175 0.01175 0.01175 3.06% Castro::normalize_species() 30 0.00959 0.00959 0.00959 2.50% amrex::Dot() 484 0.008661 0.008661 0.008661 2.25% Castro::computeTemp() 30 0.007567 0.007567 0.007567 1.97% amrex::Copy() 463 0.00693 0.00693 0.00693 1.80% FabArray::setVal() 537 0.006198 0.006198 0.006198 1.61% FabArray::norminf() 326 0.006173 0.006173 0.006173 1.61% FabArray::ParallelCopy_nowait() 380 0.00594 0.00594 0.00594 1.55% StateDataPhysBCFunct::() 20 0.005846 0.005846 0.005846 1.52% MLCellLinOp::defineAuxData() 6 0.005328 0.005328 0.005328 1.39% Castro::enforce_min_density() 30 0.005288 0.005288 0.005288 1.38% MLPoisson::Fapply() 500 0.004475 0.004475 0.004475 1.16% FabArray::Saxpy() 355 0.003614 0.003614 0.003614 0.94% FabArray::Xpay() 361 0.003515 0.003515 0.003515 0.91% Gravity::fill_multipole_BCs() 6 0.003006 0.003006 0.003006 0.78% MLMG::addInterpCorrection() 180 0.002812 0.002812 0.002812 0.73% amrex::average_down 180 0.002495 0.002495 0.002495 0.65% Castro::estTimeStep() 10 0.002488 0.002488 0.002488 0.65% Amr::restart() 1 0.002468 0.002468 0.002468 0.64% BndryData::define() 6 0.001998 0.001998 0.001998 0.52% FabArray::LinComb() 242 0.001937 0.001937 0.001937 0.50% amrex::Add() 72 0.001837 0.001837 0.001837 0.48% Castro::reset_internal_energy(MultiFab) 30 0.00177 0.00177 0.00177 0.46% Castro::do_advance_ctu() 5 0.001743 0.001743 0.001743 0.45% Castro::construct_new_gravity_source() 5 0.001688 0.001688 0.001688 0.44% Amr::writePlotFile() 1 0.001377 0.001377 0.001377 0.36% Castro::construct_old_gravity_source() 5 0.00135 0.00135 0.00135 0.35% Castro::enforce_speed_limit() 30 0.001245 0.001245 0.001245 0.32% Castro::reset_internal_energy(Fab) 240 0.0009859 0.0009859 0.0009859 0.26% MLCGSolver::bicgstab 36 0.0009016 0.0009016 0.0009016 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008933 0.0008933 0.0008933 0.23% MLCellLinOp::setLevelBC() 6 0.0007332 0.0007332 0.0007332 0.19% Gravity::actual_solve_with_mlmg() 6 0.000708 0.000708 0.000708 0.18% FabArray::setDomainBndry() 20 0.0006621 0.0006621 0.0006621 0.17% FabArray::mult() 22 0.0006578 0.0006578 0.0006578 0.17% MLCellLinOp::prepareForSolve() 6 0.0005952 0.0005952 0.0005952 0.15% MultiFab::contains_nan() 10 0.0005878 0.0005878 0.0005878 0.15% MLCellLinOp::smooth() 720 0.0005152 0.0005152 0.0005152 0.13% MLCellLinOp::compGrad() 6 0.0004885 0.0004885 0.0004885 0.13% MLMG::prepareForSolve() 6 0.0004427 0.0004427 0.0004427 0.12% Amr::InitAmr() 1 0.0004196 0.0004196 0.0004196 0.11% FabArrayBase::CPC::define() 244 0.000415 0.000415 0.000415 0.11% FabArray::FillBoundary() 1766 0.0003492 0.0003492 0.0003492 0.09% FabArrayBase::getCPC() 632 0.0003413 0.0003413 0.0003413 0.09% Gravity::get_old_grav_vector() 5 0.0003112 0.0003112 0.0003112 0.08% main() 1 0.0002694 0.0002694 0.0002694 0.07% Gravity::get_new_grav_vector() 5 0.0002682 0.0002682 0.0002682 0.07% FabArrayBase::getFB() 1766 0.0002587 0.0002587 0.0002587 0.07% Amr::coarseTimeStep() 5 0.0001991 0.0001991 0.0001991 0.05% MLCellLinOp::apply() 500 0.0001978 0.0001978 0.0001978 0.05% MLMG::mgVcycle() 36 0.0001917 0.0001917 0.0001917 0.05% MultiFab::max() 6 0.0001344 0.0001344 0.0001344 0.03% MLCGSolver::ParallelAllReduce 659 0.0001281 0.0001281 0.0001281 0.03% Castro::subcycle_advance_ctu() 5 0.0001124 0.0001124 0.0001124 0.03% MLCellLinOp::defineBC() 6 0.0001112 0.0001112 0.0001112 0.03% FabArray::ParallelCopy() 380 0.0001099 0.0001099 0.0001099 0.03% Castro::construct_new_source() 25 0.0001052 0.0001052 0.0001052 0.03% Castro::advance() 5 0.0001017 0.0001017 0.0001017 0.03% FillPatchIterator::Initialize 20 0.0001012 0.0001012 0.0001012 0.03% MLCellLinOp::correctionResidual() 216 9.653e-05 9.653e-05 9.653e-05 0.03% MLLinOp::defineGrids() 6 8.826e-05 8.826e-05 8.826e-05 0.02% Amr::timeStep() 5 7.869e-05 7.869e-05 7.869e-05 0.02% Gravity::solve_for_phi() 5 7.747e-05 7.747e-05 7.747e-05 0.02% AmrLevel::restart() 1 6.88e-05 6.88e-05 6.88e-05 0.02% Castro::finalize_advance() 5 5.99e-05 5.99e-05 5.99e-05 0.02% StateData::restartDoit() 4 5.85e-05 5.85e-05 5.85e-05 0.02% FabArrayBase::FB::FB() 26 5.727e-05 5.727e-05 5.727e-05 0.01% Gravity::update_max_rhs() 6 5.367e-05 5.367e-05 5.367e-05 0.01% Castro::construct_new_gravity() 5 4.984e-05 4.984e-05 4.984e-05 0.01% MLMG:computeResOfCorrection() 180 4.677e-05 4.677e-05 4.677e-05 0.01% Castro::clean_state() 30 4.47e-05 4.47e-05 4.47e-05 0.01% Castro::initialize_do_advance() 5 4.331e-05 4.331e-05 4.331e-05 0.01% MLMG::mgVcycle_down::0 36 4.017e-05 4.017e-05 4.017e-05 0.01% Castro::expand_state() 5 3.803e-05 3.803e-05 3.803e-05 0.01% MLMG::actualBottomSolve() 36 3.747e-05 3.747e-05 3.747e-05 0.01% Castro::post_timestep() 5 3.609e-05 3.609e-05 3.609e-05 0.01% MLMG::mgVcycle_down::1 36 3.554e-05 3.554e-05 3.554e-05 0.01% MLMG::solve() 6 3.463e-05 3.463e-05 3.463e-05 0.01% MLMG::mgVcycle_down::2 36 3.176e-05 3.176e-05 3.176e-05 0.01% MLMG::mgVcycle_down::4 36 3.11e-05 3.11e-05 3.11e-05 0.01% MLMG::mgVcycle_down::3 36 3.019e-05 3.019e-05 3.019e-05 0.01% Castro::initialize_advance() 5 2.917e-05 2.917e-05 2.917e-05 0.01% MLMG::mgVcycle_up::4 36 2.642e-05 2.642e-05 2.642e-05 0.01% Amr::writeSmallPlotFile() 1 2.485e-05 2.485e-05 2.485e-05 0.01% MLMG::mgVcycle_up::0 36 2.436e-05 2.436e-05 2.436e-05 0.01% Castro::buildMetrics() 1 2.431e-05 2.431e-05 2.431e-05 0.01% MLMG::oneIter() 36 2.415e-05 2.415e-05 2.415e-05 0.01% MLMG::mgVcycle_up::3 36 2.149e-05 2.149e-05 2.149e-05 0.01% MLMG::mgVcycle_up::2 36 2.144e-05 2.144e-05 2.144e-05 0.01% Castro::post_restart() 1 2.124e-05 2.124e-05 2.124e-05 0.01% MLCellLinOp::solutionResidual() 42 2.121e-05 2.121e-05 2.121e-05 0.01% Castro::swap_state_time_levels() 5 2.112e-05 2.112e-05 2.112e-05 0.01% Castro::computeNewDt() 5 2.09e-05 2.09e-05 2.09e-05 0.01% MLMG::mgVcycle_up::1 36 2.029e-05 2.029e-05 2.029e-05 0.01% Castro::initMFs() 1 1.984e-05 1.984e-05 1.984e-05 0.01% Castro::finalize_do_advance() 5 1.79e-05 1.79e-05 1.79e-05 0.00% MLMG::ResNormInf() 42 1.749e-05 1.749e-05 1.749e-05 0.00% MLPoisson::define() 6 1.538e-05 1.538e-05 1.538e-05 0.00% FillPatchSingleLevel 20 1.457e-05 1.457e-05 1.457e-05 0.00% makeSFC 30 1.43e-05 1.43e-05 1.43e-05 0.00% MLMG::mgVcycle_bottom 36 1.419e-05 1.419e-05 1.419e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.31e-05 1.31e-05 1.31e-05 0.00% MLMG::computeResidual() 36 1.272e-05 1.272e-05 1.272e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.268e-05 1.268e-05 1.268e-05 0.00% Castro::do_new_sources() 5 1.011e-05 1.011e-05 1.011e-05 0.00% DistributionMapping::Distribute() 31 9.286e-06 9.286e-06 9.286e-06 0.00% Castro::construct_old_source() 25 8.825e-06 8.825e-06 8.825e-06 0.00% Amr::initSubcycle() 1 8.202e-06 8.202e-06 8.202e-06 0.00% Castro::do_old_sources() 5 8.049e-06 8.049e-06 8.049e-06 0.00% Gravity::actual_multilevel_solve() 1 6.953e-06 6.953e-06 6.953e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 6.944e-06 6.944e-06 6.944e-06 0.00% Castro::check_for_nan() 10 6.081e-06 6.081e-06 6.081e-06 0.00% Castro::apply_source_to_state() 10 5.694e-06 5.694e-06 5.694e-06 0.00% Castro::construct_old_gravity() 5 4.938e-06 4.938e-06 4.938e-06 0.00% MLLinOp::define() 6 4.519e-06 4.519e-06 4.519e-06 0.00% MLPoisson::prepareForSolve() 6 4.33e-06 4.33e-06 4.33e-06 0.00% Gravity::swapTimeLevels() 5 3.963e-06 3.963e-06 3.963e-06 0.00% MLMG::computeMLResidual() 6 3.712e-06 3.712e-06 3.712e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.46e-06 3.46e-06 3.46e-06 0.00% MLMG::getGradSolution() 6 2.872e-06 2.872e-06 2.872e-06 0.00% Castro::create_source_corrector() 5 2.406e-06 2.406e-06 2.406e-06 0.00% Gravity::set_mass_offset() 6 2.135e-06 2.135e-06 2.135e-06 0.00% MLMG::MLResNormInf() 6 2.132e-06 2.132e-06 2.132e-06 0.00% MLMG::MLRhsNormInf() 6 2.023e-06 2.023e-06 2.023e-06 0.00% Castro::retry_advance_ctu() 5 1.862e-06 1.862e-06 1.862e-06 0.00% Castro::FluxRegCrseInit 5 1.551e-06 1.551e-06 1.551e-06 0.00% Amr::init() 1 1.46e-06 1.46e-06 1.46e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.212e-06 1.212e-06 1.212e-06 0.00% Castro::FluxRegFineAdd() 5 1.09e-06 1.09e-06 1.09e-06 0.00% AmrLevel::AmrLevel() 1 6.62e-07 6.62e-07 6.62e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.3842 0.3842 0.3842 100.00% Amr::coarseTimeStep() 5 0.3031 0.3031 0.3031 78.89% Amr::timeStep() 5 0.3014 0.3014 0.3014 78.46% Castro::advance() 5 0.2962 0.2962 0.2962 77.10% Castro::subcycle_advance_ctu() 5 0.2889 0.2889 0.2889 75.19% Castro::do_advance_ctu() 5 0.2888 0.2888 0.2888 75.17% Castro::construct_new_gravity() 5 0.1282 0.1282 0.1282 33.37% Gravity::solve_phi_with_mlmg() 6 0.1235 0.1235 0.1235 32.16% Gravity::solve_for_phi() 5 0.1205 0.1205 0.1205 31.36% Gravity::actual_solve_with_mlmg() 6 0.1203 0.1203 0.1203 31.31% Castro::construct_ctu_hydro_source() 5 0.1191 0.1191 0.1191 31.00% MLMG::solve() 6 0.1093 0.1093 0.1093 28.44% MLMG::oneIter() 36 0.1025 0.1025 0.1025 26.68% MLMG::mgVcycle() 36 0.101 0.101 0.101 26.28% MLCellLinOp::smooth() 720 0.04955 0.04955 0.04955 12.90% Amr::init() 1 0.047 0.047 0.047 12.24% Amr::restart() 1 0.047 0.047 0.047 12.23% MLCellLinOp::applyBC() 1946 0.04618 0.04618 0.04618 12.02% AmrLevel::restart() 1 0.04091 0.04091 0.04091 10.65% StateData::restartDoit() 4 0.04084 0.04084 0.04084 10.63% VisMF::Read() 3 0.04073 0.04073 0.04073 10.60% Amr::writePlotFile() 1 0.03337 0.03337 0.03337 8.69% VisMF::Write(FabArray) 1 0.03174 0.03174 0.03174 8.26% MLMG::mgVcycle_bottom 36 0.03133 0.03133 0.03133 8.15% MLMG::actualBottomSolve() 36 0.03131 0.03131 0.03131 8.15% MLCGSolver::bicgstab 36 0.03101 0.03101 0.03101 8.07% Castro::clean_state() 30 0.02649 0.02649 0.02649 6.90% FillPatchIterator::Initialize 20 0.02035 0.02035 0.02035 5.30% FillPatchSingleLevel 20 0.01959 0.01959 0.01959 5.10% StateDataPhysBCFunct::() 20 0.01759 0.01759 0.01759 4.58% MLCellLinOp::apply() 500 0.01555 0.01555 0.01555 4.05% MLMG::mgVcycle_down::0 36 0.01419 0.01419 0.01419 3.69% MLPoisson::Fsmooth() 1440 0.01399 0.01399 0.01399 3.64% FabArray::FillBoundary() 1766 0.01341 0.01341 0.01341 3.49% FillBoundary_nowait() 1766 0.01306 0.01306 0.01306 3.40% StateData::FillBoundary(geom) 160 0.01175 0.01175 0.01175 3.06% MLMG::mgVcycle_up::0 36 0.0106 0.0106 0.0106 2.76% Castro::initialize_do_advance() 5 0.01055 0.01055 0.01055 2.74% Castro::computeTemp() 30 0.01032 0.01032 0.01032 2.69% Castro::normalize_species() 30 0.00959 0.00959 0.00959 2.50% MLCellLinOp::correctionResidual() 216 0.009471 0.009471 0.009471 2.47% MLPoisson::define() 6 0.008859 0.008859 0.008859 2.31% amrex::Dot() 484 0.008661 0.008661 0.008661 2.25% MLMG:computeResOfCorrection() 180 0.00832 0.00832 0.00832 2.17% Gravity::get_new_grav_vector() 5 0.007591 0.007591 0.007591 1.98% Castro::construct_old_gravity() 5 0.007498 0.007498 0.007498 1.95% Gravity::get_old_grav_vector() 5 0.007493 0.007493 0.007493 1.95% Castro::do_new_sources() 5 0.007486 0.007486 0.007486 1.95% amrex::Copy() 463 0.00693 0.00693 0.00693 1.80% Castro::initialize_advance() 5 0.006888 0.006888 0.006888 1.79% MLMG::mgVcycle_down::1 36 0.006725 0.006725 0.006725 1.75% FabArray::ParallelCopy() 380 0.006416 0.006416 0.006416 1.67% MLMG::mgVcycle_down::2 36 0.006316 0.006316 0.006316 1.64% FabArray::ParallelCopy_nowait() 380 0.006306 0.006306 0.006306 1.64% FabArray::setVal() 537 0.006198 0.006198 0.006198 1.61% FabArray::norminf() 326 0.006173 0.006173 0.006173 1.61% MLMG::mgVcycle_down::3 36 0.006136 0.006136 0.006136 1.60% MLMG::mgVcycle_down::4 36 0.006097 0.006097 0.006097 1.59% MLCellLinOp::defineAuxData() 6 0.006081 0.006081 0.006081 1.58% Castro::expand_state() 5 0.005815 0.005815 0.005815 1.51% Castro::do_old_sources() 5 0.005321 0.005321 0.005321 1.39% Castro::enforce_min_density() 30 0.005288 0.005288 0.005288 1.38% MLCGSolver::ParallelAllReduce 659 0.005226 0.005226 0.005226 1.36% Castro::post_timestep() 5 0.005167 0.005167 0.005167 1.35% MLMG::addInterpCorrection() 180 0.005031 0.005031 0.005031 1.31% MLMG::mgVcycle_up::4 36 0.00496 0.00496 0.00496 1.29% MLMG::mgVcycle_up::1 36 0.004919 0.004919 0.004919 1.28% MLMG::mgVcycle_up::2 36 0.004804 0.004804 0.004804 1.25% amrex::average_down 180 0.004706 0.004706 0.004706 1.22% MLMG::mgVcycle_up::3 36 0.004697 0.004697 0.004697 1.22% MLPoisson::Fapply() 500 0.004475 0.004475 0.004475 1.16% FabArray::Saxpy() 355 0.003614 0.003614 0.003614 0.94% FabArray::Xpay() 361 0.003515 0.003515 0.003515 0.91% Castro::post_restart() 1 0.003457 0.003457 0.003457 0.90% Gravity::multilevel_solve_for_new_phi() 1 0.003343 0.003343 0.003343 0.87% Gravity::actual_multilevel_solve() 1 0.00333 0.00333 0.00333 0.87% MLCellLinOp::solutionResidual() 42 0.003202 0.003202 0.003202 0.83% Gravity::fill_multipole_BCs() 6 0.003127 0.003127 0.003127 0.81% Castro::reset_internal_energy(MultiFab) 30 0.002756 0.002756 0.002756 0.72% MLMG::computeResidual() 36 0.002661 0.002661 0.002661 0.69% MLCellLinOp::defineBC() 6 0.00264 0.00264 0.00264 0.69% BndryData::define() 6 0.002529 0.002529 0.002529 0.66% Castro::estTimeStep() 10 0.002488 0.002488 0.002488 0.65% MLMG::prepareForSolve() 6 0.002449 0.002449 0.002449 0.64% FabArray::LinComb() 242 0.001937 0.001937 0.001937 0.50% amrex::Add() 72 0.001837 0.001837 0.001837 0.48% Castro::construct_new_source() 25 0.001794 0.001794 0.001794 0.47% Castro::construct_new_gravity_source() 5 0.001688 0.001688 0.001688 0.44% Castro::computeNewDt() 5 0.001423 0.001423 0.001423 0.37% Castro::construct_old_source() 25 0.001359 0.001359 0.001359 0.35% Castro::construct_old_gravity_source() 5 0.00135 0.00135 0.00135 0.35% Castro::enforce_speed_limit() 30 0.001245 0.001245 0.001245 0.32% Castro::reset_internal_energy(Fab) 240 0.0009859 0.0009859 0.0009859 0.26% MLMG::ResNormInf() 42 0.0009331 0.0009331 0.0009331 0.24% Castro::apply_source_to_state() 10 0.0009258 0.0009258 0.0009258 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008933 0.0008933 0.0008933 0.23% FabArrayBase::getCPC() 632 0.0007563 0.0007563 0.0007563 0.20% MLMG::getGradSolution() 6 0.0007525 0.0007525 0.0007525 0.20% MLCellLinOp::compGrad() 6 0.0007496 0.0007496 0.0007496 0.20% MLCellLinOp::setLevelBC() 6 0.0007332 0.0007332 0.0007332 0.19% FabArray::setDomainBndry() 20 0.0006621 0.0006621 0.0006621 0.17% FabArray::mult() 22 0.0006578 0.0006578 0.0006578 0.17% MLPoisson::prepareForSolve() 6 0.0005995 0.0005995 0.0005995 0.16% MLCellLinOp::prepareForSolve() 6 0.0005952 0.0005952 0.0005952 0.15% Castro::check_for_nan() 10 0.0005939 0.0005939 0.0005939 0.15% MultiFab::contains_nan() 10 0.0005878 0.0005878 0.0005878 0.15% MLMG::computeMLResidual() 6 0.0005573 0.0005573 0.0005573 0.15% Gravity::update_max_rhs() 6 0.0004419 0.0004419 0.0004419 0.12% Amr::InitAmr() 1 0.0004278 0.0004278 0.0004278 0.11% FabArrayBase::CPC::define() 244 0.000415 0.000415 0.000415 0.11% Castro::finalize_advance() 5 0.0003254 0.0003254 0.0003254 0.08% FabArrayBase::getFB() 1766 0.0003159 0.0003159 0.0003159 0.08% Gravity::swapTimeLevels() 5 0.0002239 0.0002239 0.0002239 0.06% MLMG::MLResNormInf() 6 0.0001495 0.0001495 0.0001495 0.04% Castro::buildMetrics() 1 0.0001444 0.0001444 0.0001444 0.04% MultiFab::max() 6 0.0001344 0.0001344 0.0001344 0.03% MLLinOp::define() 6 0.000123 0.000123 0.000123 0.03% MLLinOp::defineGrids() 6 0.0001185 0.0001185 0.0001185 0.03% MLMG::MLRhsNormInf() 6 0.0001152 0.0001152 0.0001152 0.03% FabArrayBase::FB::FB() 26 5.727e-05 5.727e-05 5.727e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.898e-05 2.898e-05 2.898e-05 0.01% Amr::writeSmallPlotFile() 1 2.485e-05 2.485e-05 2.485e-05 0.01% makeSFC 30 2.204e-05 2.204e-05 2.204e-05 0.01% Castro::swap_state_time_levels() 5 2.112e-05 2.112e-05 2.112e-05 0.01% Castro::initMFs() 1 1.984e-05 1.984e-05 1.984e-05 0.01% Castro::finalize_do_advance() 5 1.79e-05 1.79e-05 1.79e-05 0.00% DistributionMapping::Distribute() 31 9.286e-06 9.286e-06 9.286e-06 0.00% Amr::initSubcycle() 1 8.202e-06 8.202e-06 8.202e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 5.013e-06 5.013e-06 5.013e-06 0.00% Castro::create_source_corrector() 5 2.406e-06 2.406e-06 2.406e-06 0.00% Gravity::set_mass_offset() 6 2.135e-06 2.135e-06 2.135e-06 0.00% Castro::retry_advance_ctu() 5 1.862e-06 1.862e-06 1.862e-06 0.00% Castro::FluxRegCrseInit 5 1.551e-06 1.551e-06 1.551e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.212e-06 1.212e-06 1.212e-06 0.00% Castro::FluxRegFineAdd() 5 1.09e-06 1.09e-06 1.09e-06 0.00% AmrLevel::AmrLevel() 1 6.62e-07 6.62e-07 6.62e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12056 Free GPU global memory (MB): 2459 [The Arena] space allocated (MB): 9042 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (23.01-12-ga05384c0b7c4) finalized