Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-8-g3f715d29c94b) initialized Starting run at 08:54:56 UTC on 2022-08-10. Successfully read inputs file ... Castro git describe: 22.08-8-g2409a13d7 AMReX git describe: 22.08-8-g3f715d29c Microphysics git describe: 22.08-1-gffefdd62 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.05192164 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.029810447 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.047962212 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.053534153 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.050410556 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.060645416 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.07381472 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.047509884 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.066616133 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.051082221 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.052100971 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.068822186 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.072598502 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.04736699 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.054418727 seconds Ending run at 08:54:57 UTC on 2022-08-10. Run time = 0.878252422 Run time without initialization = 0.747640388 Average number of zones advanced per microsecond: 3.506 Average number of zones advanced per microsecond per rank: 3.506 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8783 ... 0.8783 ... 0.8783 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 11 0.2230 0.2230 0.2230 25.39% Castro::construct_ctu_hydro_source() 10 0.2104 0.2104 0.2104 23.95% MLCellLinOp::applyBC() 4379 0.0776 0.0776 0.0776 8.84% MLPoisson::Fsmooth() 3240 0.06212 0.06212 0.06212 7.07% StateData::FillBoundary(geom) 328 0.02366 0.02366 0.02366 2.69% MLCGSolver::bicgstab 81 0.02313 0.02313 0.02313 2.63% MultiFab::Dot() 1100 0.0215 0.0215 0.0215 2.45% StateDataPhysBCFunct::() 41 0.01623 0.01623 0.01623 1.85% Castro::normalize_species() 62 0.01492 0.01492 0.01492 1.70% Castro::computeTemp() 63 0.01413 0.01413 0.01413 1.61% FillBoundary_nowait() 3974 0.01398 0.01398 0.01398 1.59% FabArray::setVal() 1135 0.0139 0.0139 0.0139 1.58% MultiFab::LinComb() 1566 0.0139 0.0139 0.0139 1.58% FabArray::ParallelCopy_nowait() 851 0.01266 0.01266 0.01266 1.44% Castro::enforce_min_density() 62 0.01182 0.01182 0.01182 1.35% MLPoisson::Fapply() 1128 0.0114 0.0114 0.0114 1.30% MLCellLinOp::defineAuxData() 11 0.0111 0.0111 0.0111 1.26% Gravity::fill_multipole_BCs() 11 0.009995 0.009995 0.009995 1.14% MLMG::addInterpCorrection() 405 0.007468 0.007468 0.007468 0.85% amrex::average_down 405 0.006683 0.006683 0.006683 0.76% MultiFab::Xpay() 578 0.006423 0.006423 0.006423 0.73% Castro::estTimeStep() 21 0.006087 0.006087 0.006087 0.69% Amr::checkPoint() 3 0.005257 0.005257 0.005257 0.60% Castro::do_advance_ctu() 10 0.004643 0.004643 0.004643 0.53% Castro::reset_internal_energy(MultiFab) 63 0.004387 0.004387 0.004387 0.50% BndryData::define() 11 0.003705 0.003705 0.003705 0.42% Castro::construct_new_gravity_source() 10 0.003424 0.003424 0.003424 0.39% Castro::construct_old_gravity_source() 10 0.002923 0.002923 0.002923 0.33% Amr::writePlotFile() 2 0.002889 0.002889 0.002889 0.33% MLMG::ResNormInf() 92 0.002019 0.002019 0.002019 0.23% Gravity::get_new_grav_vector() 11 0.001911 0.001911 0.001911 0.22% MultiFab::Saxpy() 20 0.001809 0.001809 0.001809 0.21% Castro::expand_state() 10 0.001739 0.001739 0.001739 0.20% Gravity::get_old_grav_vector() 10 0.001737 0.001737 0.001737 0.20% MultiFab::Add() 81 0.001619 0.001619 0.001619 0.18% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001593 0.001593 0.001593 0.18% MLCellLinOp::setLevelBC() 11 0.001503 0.001503 0.001503 0.17% Castro::reset_internal_energy(Fab) 504 0.00145 0.00145 0.00145 0.17% Gravity::actual_solve_with_mlmg() 11 0.001431 0.001431 0.001431 0.16% FabArray::mult() 43 0.001308 0.001308 0.001308 0.15% FabArray::setDomainBndry() 41 0.001281 0.001281 0.001281 0.15% Castro::initData() 1 0.001244 0.001244 0.001244 0.14% Castro::enforce_speed_limit() 62 0.001228 0.001228 0.001228 0.14% MLMG::prepareForSolve() 11 0.001215 0.001215 0.001215 0.14% MultiFab::contains_nan() 20 0.001174 0.001174 0.001174 0.13% MLCellLinOp::prepareForSolve() 11 0.001144 0.001144 0.001144 0.13% MLCellLinOp::smooth() 1620 0.001096 0.001096 0.001096 0.12% MLCellLinOp::compGrad() 11 0.0009141 0.0009141 0.0009141 0.10% FabArray::FillBoundary() 3974 0.000834 0.000834 0.000834 0.09% FabArrayBase::getCPC() 1313 0.0007759 0.0007759 0.0007759 0.09% FabArrayBase::CPC::define() 454 0.0006614 0.0006614 0.0006614 0.08% FabArrayBase::getFB() 3974 0.0006016 0.0006016 0.0006016 0.07% Amr::InitAmr() 1 0.0004977 0.0004977 0.0004977 0.06% Amr::coarseTimeStep() 10 0.0004332 0.0004332 0.0004332 0.05% MLCellLinOp::apply() 1128 0.0004329 0.0004329 0.0004329 0.05% Gravity::solve_for_phi() 10 0.0004192 0.0004192 0.0004192 0.05% Gravity::update_max_rhs() 11 0.0004075 0.0004075 0.0004075 0.05% CGSolver::sxay() 1566 0.000355 0.000355 0.000355 0.04% MultiFab::Copy() 11 0.0003239 0.0003239 0.0003239 0.04% MLCGSolver::ParallelAllReduce 1495 0.0002937 0.0002937 0.0002937 0.03% FillPatchIterator::Initialize 41 0.0002918 0.0002918 0.0002918 0.03% MLCellLinOp::defineBC() 11 0.0002851 0.0002851 0.0002851 0.03% FabArray::ParallelCopy() 851 0.0002689 0.0002689 0.0002689 0.03% main() 1 0.0002605 0.0002605 0.0002605 0.03% MultiFab::max() 11 0.0002515 0.0002515 0.0002515 0.03% MLCellLinOp::correctionResidual() 486 0.0002208 0.0002208 0.0002208 0.03% Amr::timeStep() 10 0.0002166 0.0002166 0.0002166 0.02% MLMG::MLRhsNormInf() 11 0.0002144 0.0002144 0.0002144 0.02% Castro::construct_new_gravity() 10 0.0002105 0.0002105 0.0002105 0.02% MLMG::mgVcycle() 81 0.0001932 0.0001932 0.0001932 0.02% MLLinOp::defineGrids() 11 0.000181 0.000181 0.000181 0.02% MLMG:computeResOfCorrection() 405 0.0001536 0.0001536 0.0001536 0.02% Castro::subcycle_advance_ctu() 10 0.0001476 0.0001476 0.0001476 0.02% Castro::create_source_corrector() 10 0.0001363 0.0001363 0.0001363 0.02% StateData::checkPoint() 12 0.0001343 0.0001343 0.0001343 0.02% MLMG::mgVcycle_down::0 81 0.0001075 0.0001075 0.0001075 0.01% Castro::Castro() 1 9.253e-05 9.253e-05 9.253e-05 0.01% MLMG::mgVcycle_down::1 81 9.114e-05 9.114e-05 9.114e-05 0.01% MLMG::mgVcycle_down::2 81 8.893e-05 8.893e-05 8.893e-05 0.01% FabArrayBase::FB::FB() 56 8.59e-05 8.59e-05 8.59e-05 0.01% MLMG::mgVcycle_down::3 81 8.382e-05 8.382e-05 8.382e-05 0.01% Castro::initialize_advance() 10 8.361e-05 8.361e-05 8.361e-05 0.01% MLMG::mgVcycle_down::4 81 8.296e-05 8.296e-05 8.296e-05 0.01% Castro::clean_state() 62 7.976e-05 7.976e-05 7.976e-05 0.01% MLMG::actualBottomSolve() 81 7.883e-05 7.883e-05 7.883e-05 0.01% AmrLevel::checkPoint() 3 7.345e-05 7.345e-05 7.345e-05 0.01% MLMG::mgVcycle_up::4 81 7.051e-05 7.051e-05 7.051e-05 0.01% MLMG::solve() 11 6.674e-05 6.674e-05 6.674e-05 0.01% Castro::initialize_do_advance() 10 6.617e-05 6.617e-05 6.617e-05 0.01% MLMG::oneIter() 81 6.085e-05 6.085e-05 6.085e-05 0.01% MLMG::mgVcycle_up::2 81 5.827e-05 5.827e-05 5.827e-05 0.01% MLMG::mgVcycle_up::1 81 5.82e-05 5.82e-05 5.82e-05 0.01% MLMG::mgVcycle_up::0 81 5.811e-05 5.811e-05 5.811e-05 0.01% MLMG::mgVcycle_up::3 81 5.79e-05 5.79e-05 5.79e-05 0.01% MLCellLinOp::solutionResidual() 92 5.341e-05 5.341e-05 5.341e-05 0.01% Castro::finalize_advance() 10 4.985e-05 4.985e-05 4.985e-05 0.01% Castro::advance() 10 4.498e-05 4.498e-05 4.498e-05 0.01% MLMG::computeResidual() 81 4.052e-05 4.052e-05 4.052e-05 0.00% StateData::define() 4 4.034e-05 4.034e-05 4.034e-05 0.00% Castro::swap_state_time_levels() 10 3.918e-05 3.918e-05 3.918e-05 0.00% Castro::finalize_do_advance() 10 3.473e-05 3.473e-05 3.473e-05 0.00% MLMG::mgVcycle_bottom 81 3.361e-05 3.361e-05 3.361e-05 0.00% Castro::enforce_consistent_e() 1 3.309e-05 3.309e-05 3.309e-05 0.00% Gravity::solve_phi_with_mlmg() 11 3.295e-05 3.295e-05 3.295e-05 0.00% Castro::post_timestep() 10 3.268e-05 3.268e-05 3.268e-05 0.00% Gravity::actual_multilevel_solve() 1 3.2e-05 3.2e-05 3.2e-05 0.00% Castro::initMFs() 1 2.922e-05 2.922e-05 2.922e-05 0.00% FillPatchSingleLevel 41 2.898e-05 2.898e-05 2.898e-05 0.00% Amr::defBaseLevel() 1 2.73e-05 2.73e-05 2.73e-05 0.00% MLLinOp::define() 11 2.6e-05 2.6e-05 2.6e-05 0.00% Amr::writeSmallPlotFile() 1 2.524e-05 2.524e-05 2.524e-05 0.00% makeSFC 55 2.499e-05 2.499e-05 2.499e-05 0.00% Castro::buildMetrics() 1 2.383e-05 2.383e-05 2.383e-05 0.00% Amr::FinalizeInit() 1 2.321e-05 2.321e-05 2.321e-05 0.00% MLPoisson::define() 11 2.269e-05 2.269e-05 2.269e-05 0.00% Castro::construct_old_source() 50 1.949e-05 1.949e-05 1.949e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.876e-05 1.876e-05 1.876e-05 0.00% Castro::do_new_sources() 10 1.75e-05 1.75e-05 1.75e-05 0.00% Castro::construct_new_source() 50 1.708e-05 1.708e-05 1.708e-05 0.00% Castro::do_old_sources() 10 1.67e-05 1.67e-05 1.67e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.467e-05 1.467e-05 1.467e-05 0.00% MLMG::computeMLResidual() 11 1.459e-05 1.459e-05 1.459e-05 0.00% DistributionMapping::Distribute() 56 1.418e-05 1.418e-05 1.418e-05 0.00% Castro::check_for_nan() 20 1.205e-05 1.205e-05 1.205e-05 0.00% Castro::apply_source_to_state() 20 1.202e-05 1.202e-05 1.202e-05 0.00% Castro::construct_old_gravity() 10 1.111e-05 1.111e-05 1.111e-05 0.00% Amr::initSubcycle() 1 1.029e-05 1.029e-05 1.029e-05 0.00% MLPoisson::prepareForSolve() 11 9.317e-06 9.317e-06 9.317e-06 0.00% Gravity::swapTimeLevels() 10 8.226e-06 8.226e-06 8.226e-06 0.00% Amr::InitializeInit() 1 6.869e-06 6.869e-06 6.869e-06 0.00% AmrLevel::AmrLevel(dm) 1 6.355e-06 6.355e-06 6.355e-06 0.00% MLMG::getGradSolution() 11 6.187e-06 6.187e-06 6.187e-06 0.00% Castro::computeNewDt() 9 5.872e-06 5.872e-06 5.872e-06 0.00% Gravity::set_mass_offset() 11 4.631e-06 4.631e-06 4.631e-06 0.00% AmrLevel::checkPointPost() 3 4.551e-06 4.551e-06 4.551e-06 0.00% Castro::retry_advance_ctu() 10 3.707e-06 3.707e-06 3.707e-06 0.00% MLMG::MLResNormInf() 11 3.593e-06 3.593e-06 3.593e-06 0.00% Castro::post_init() 1 3.556e-06 3.556e-06 3.556e-06 0.00% Castro::FluxRegCrseInit 10 2.753e-06 2.753e-06 2.753e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.736e-06 2.736e-06 2.736e-06 0.00% Castro::computeInitialDt() 2 2.687e-06 2.687e-06 2.687e-06 0.00% Amr::init() 1 2.353e-06 2.353e-06 2.353e-06 0.00% Castro::FluxRegFineAdd() 10 2.057e-06 2.057e-06 2.057e-06 0.00% AmrLevel::checkPointPre() 3 1.941e-06 1.941e-06 1.941e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.87e-06 1.87e-06 1.87e-06 0.00% Castro::post_regrid() 1 1.268e-06 1.268e-06 1.268e-06 0.00% Amr::initialInit() 1 1.171e-06 1.171e-06 1.171e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8783 0.8783 0.8783 100.00% Amr::coarseTimeStep() 10 0.693 0.693 0.693 78.90% Amr::timeStep() 10 0.5939 0.5939 0.5939 67.62% Castro::advance() 10 0.586 0.586 0.586 66.73% Castro::subcycle_advance_ctu() 10 0.5748 0.5748 0.5748 65.45% Castro::do_advance_ctu() 10 0.5746 0.5746 0.5746 65.43% Gravity::solve_phi_with_mlmg() 11 0.3067 0.3067 0.3067 34.92% Gravity::actual_solve_with_mlmg() 11 0.2965 0.2965 0.2965 33.76% Castro::construct_new_gravity() 10 0.2849 0.2849 0.2849 32.43% MLMG::solve() 11 0.2746 0.2746 0.2746 31.26% Gravity::solve_for_phi() 10 0.2665 0.2665 0.2665 30.34% MLMG::oneIter() 81 0.2599 0.2599 0.2599 29.60% MLMG::mgVcycle() 81 0.2582 0.2582 0.2582 29.40% VisMF::Write(FabArray) 11 0.223 0.223 0.223 25.39% Castro::construct_ctu_hydro_source() 10 0.2104 0.2104 0.2104 23.95% Amr::checkPoint() 3 0.147 0.147 0.147 16.73% AmrLevel::checkPoint() 3 0.1417 0.1417 0.1417 16.13% StateData::checkPoint() 12 0.1416 0.1416 0.1416 16.13% MLCellLinOp::smooth() 1620 0.1325 0.1325 0.1325 15.09% Amr::init() 1 0.13 0.13 0.13 14.80% MLCellLinOp::applyBC() 4379 0.0931 0.0931 0.0931 10.60% Amr::writePlotFile() 2 0.08437 0.08437 0.08437 9.61% MLMG::mgVcycle_bottom 81 0.07908 0.07908 0.07908 9.00% MLMG::actualBottomSolve() 81 0.07905 0.07905 0.07905 9.00% MLCGSolver::bicgstab 81 0.07826 0.07826 0.07826 8.91% MLPoisson::Fsmooth() 3240 0.06212 0.06212 0.06212 7.07% Amr::initialInit() 1 0.04812 0.04812 0.04812 5.48% Castro::clean_state() 62 0.04711 0.04711 0.04711 5.36% FillPatchIterator::Initialize 41 0.04547 0.04547 0.04547 5.18% FillPatchSingleLevel 41 0.04389 0.04389 0.04389 5.00% Amr::FinalizeInit() 1 0.04381 0.04381 0.04381 4.99% Castro::post_init() 1 0.04244 0.04244 0.04244 4.83% Gravity::multilevel_solve_for_new_phi() 1 0.04069 0.04069 0.04069 4.63% Gravity::actual_multilevel_solve() 1 0.04067 0.04067 0.04067 4.63% StateDataPhysBCFunct::() 41 0.03989 0.03989 0.03989 4.54% MLCellLinOp::apply() 1128 0.03514 0.03514 0.03514 4.00% MLMG::mgVcycle_down::0 81 0.03465 0.03465 0.03465 3.95% MLMG::mgVcycle_up::0 81 0.02962 0.02962 0.02962 3.37% StateData::FillBoundary(geom) 328 0.02366 0.02366 0.02366 2.69% MultiFab::Dot() 1100 0.0215 0.0215 0.0215 2.45% MLCellLinOp::correctionResidual() 486 0.02063 0.02063 0.02063 2.35% Castro::initialize_do_advance() 10 0.02033 0.02033 0.02033 2.31% Castro::computeTemp() 63 0.01996 0.01996 0.01996 2.27% Gravity::get_new_grav_vector() 11 0.01979 0.01979 0.01979 2.25% MLMG:computeResOfCorrection() 405 0.01782 0.01782 0.01782 2.03% MLPoisson::define() 11 0.01759 0.01759 0.01759 2.00% MLMG::mgVcycle_down::1 81 0.0172 0.0172 0.0172 1.96% MLMG::mgVcycle_down::2 81 0.01669 0.01669 0.01669 1.90% MLMG::mgVcycle_down::3 81 0.01586 0.01586 0.01586 1.81% FabArray::FillBoundary() 3974 0.0155 0.0155 0.0155 1.76% MLMG::mgVcycle_down::4 81 0.01509 0.01509 0.01509 1.72% Castro::normalize_species() 62 0.01492 0.01492 0.01492 1.70% Castro::construct_old_gravity() 10 0.01481 0.01481 0.01481 1.69% Gravity::get_old_grav_vector() 10 0.0148 0.0148 0.0148 1.69% FillBoundary_nowait() 3974 0.01466 0.01466 0.01466 1.67% CGSolver::sxay() 1566 0.01425 0.01425 0.01425 1.62% FabArray::setVal() 1135 0.0139 0.0139 0.0139 1.58% MultiFab::LinComb() 1566 0.0139 0.0139 0.0139 1.58% FabArray::ParallelCopy() 851 0.01375 0.01375 0.01375 1.57% FabArray::ParallelCopy_nowait() 851 0.01348 0.01348 0.01348 1.53% Castro::do_new_sources() 10 0.01308 0.01308 0.01308 1.49% MLCGSolver::ParallelAllReduce 1495 0.01284 0.01284 0.01284 1.46% MLMG::mgVcycle_up::2 81 0.01284 0.01284 0.01284 1.46% MLMG::mgVcycle_up::1 81 0.01267 0.01267 0.01267 1.44% MLCellLinOp::defineAuxData() 11 0.01239 0.01239 0.01239 1.41% MLMG::addInterpCorrection() 405 0.01233 0.01233 0.01233 1.40% MLMG::mgVcycle_up::3 81 0.01221 0.01221 0.01221 1.39% MLMG::mgVcycle_up::4 81 0.01215 0.01215 0.01215 1.38% Castro::expand_state() 10 0.01208 0.01208 0.01208 1.38% Castro::enforce_min_density() 62 0.01182 0.01182 0.01182 1.35% amrex::average_down 405 0.01159 0.01159 0.01159 1.32% Castro::do_old_sources() 10 0.0114 0.0114 0.0114 1.30% MLPoisson::Fapply() 1128 0.0114 0.0114 0.0114 1.30% Castro::initialize_advance() 10 0.01114 0.01114 0.01114 1.27% Gravity::fill_multipole_BCs() 11 0.009995 0.009995 0.009995 1.14% Castro::post_timestep() 10 0.007628 0.007628 0.007628 0.87% MLCellLinOp::solutionResidual() 92 0.007028 0.007028 0.007028 0.80% MultiFab::Xpay() 578 0.006423 0.006423 0.006423 0.73% Castro::estTimeStep() 21 0.006087 0.006087 0.006087 0.69% MLMG::computeResidual() 81 0.006056 0.006056 0.006056 0.69% Castro::reset_internal_energy(MultiFab) 63 0.005837 0.005837 0.005837 0.66% MLMG::prepareForSolve() 11 0.005256 0.005256 0.005256 0.60% MLCellLinOp::defineBC() 11 0.004909 0.004909 0.004909 0.56% BndryData::define() 11 0.004624 0.004624 0.004624 0.53% Amr::InitializeInit() 1 0.00431 0.00431 0.00431 0.49% Amr::defBaseLevel() 1 0.004303 0.004303 0.004303 0.49% Castro::initData() 1 0.003788 0.003788 0.003788 0.43% Castro::construct_new_source() 50 0.003441 0.003441 0.003441 0.39% Castro::construct_new_gravity_source() 10 0.003424 0.003424 0.003424 0.39% Castro::computeNewDt() 9 0.003194 0.003194 0.003194 0.36% Castro::construct_old_source() 50 0.002942 0.002942 0.002942 0.34% Castro::construct_old_gravity_source() 10 0.002923 0.002923 0.002923 0.33% MLMG::ResNormInf() 92 0.002019 0.002019 0.002019 0.23% Castro::apply_source_to_state() 20 0.001821 0.001821 0.001821 0.21% MultiFab::Saxpy() 20 0.001809 0.001809 0.001809 0.21% MultiFab::Add() 81 0.001619 0.001619 0.001619 0.18% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001593 0.001593 0.001593 0.18% MLCellLinOp::setLevelBC() 11 0.001503 0.001503 0.001503 0.17% Castro::reset_internal_energy(Fab) 504 0.00145 0.00145 0.00145 0.17% FabArrayBase::getCPC() 1313 0.001437 0.001437 0.001437 0.16% MLMG::getGradSolution() 11 0.001414 0.001414 0.001414 0.16% MLCellLinOp::compGrad() 11 0.001408 0.001408 0.001408 0.16% FabArray::mult() 43 0.001308 0.001308 0.001308 0.15% FabArray::setDomainBndry() 41 0.001281 0.001281 0.001281 0.15% Castro::enforce_speed_limit() 62 0.001228 0.001228 0.001228 0.14% Castro::check_for_nan() 20 0.001186 0.001186 0.001186 0.14% MultiFab::contains_nan() 20 0.001174 0.001174 0.001174 0.13% Castro::post_regrid() 1 0.001162 0.001162 0.001162 0.13% MLPoisson::prepareForSolve() 11 0.001153 0.001153 0.001153 0.13% MLCellLinOp::prepareForSolve() 11 0.001144 0.001144 0.001144 0.13% MLMG::computeMLResidual() 11 0.001027 0.001027 0.001027 0.12% Gravity::update_max_rhs() 11 0.0008013 0.0008013 0.0008013 0.09% FabArrayBase::getFB() 3974 0.0006875 0.0006875 0.0006875 0.08% Castro::computeInitialDt() 2 0.0006824 0.0006824 0.0006824 0.08% FabArrayBase::CPC::define() 454 0.0006614 0.0006614 0.0006614 0.08% Amr::InitAmr() 1 0.000508 0.000508 0.000508 0.06% Gravity::swapTimeLevels() 10 0.0004404 0.0004404 0.0004404 0.05% Castro::Castro() 1 0.000437 0.000437 0.000437 0.05% MultiFab::Copy() 11 0.0003239 0.0003239 0.0003239 0.04% MLMG::MLResNormInf() 11 0.0002794 0.0002794 0.0002794 0.03% MLLinOp::define() 11 0.0002614 0.0002614 0.0002614 0.03% MultiFab::max() 11 0.0002515 0.0002515 0.0002515 0.03% MLLinOp::defineGrids() 11 0.0002354 0.0002354 0.0002354 0.03% MLMG::MLRhsNormInf() 11 0.0002144 0.0002144 0.0002144 0.02% Castro::buildMetrics() 1 0.0001511 0.0001511 0.0001511 0.02% Castro::create_source_corrector() 10 0.0001363 0.0001363 0.0001363 0.02% FabArrayBase::FB::FB() 56 8.59e-05 8.59e-05 8.59e-05 0.01% Castro::finalize_advance() 10 5.466e-05 5.466e-05 5.466e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.257e-05 5.257e-05 5.257e-05 0.01% AmrLevel::AmrLevel(dm) 1 4.67e-05 4.67e-05 4.67e-05 0.01% StateData::define() 4 4.034e-05 4.034e-05 4.034e-05 0.00% Castro::swap_state_time_levels() 10 3.918e-05 3.918e-05 3.918e-05 0.00% makeSFC 55 3.791e-05 3.791e-05 3.791e-05 0.00% Castro::finalize_do_advance() 10 3.473e-05 3.473e-05 3.473e-05 0.00% Castro::enforce_consistent_e() 1 3.309e-05 3.309e-05 3.309e-05 0.00% Castro::initMFs() 1 2.922e-05 2.922e-05 2.922e-05 0.00% Amr::writeSmallPlotFile() 1 2.524e-05 2.524e-05 2.524e-05 0.00% DistributionMapping::Distribute() 56 1.418e-05 1.418e-05 1.418e-05 0.00% Amr::initSubcycle() 1 1.029e-05 1.029e-05 1.029e-05 0.00% Gravity::set_mass_offset() 11 4.631e-06 4.631e-06 4.631e-06 0.00% AmrLevel::checkPointPost() 3 4.551e-06 4.551e-06 4.551e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.999e-06 3.999e-06 3.999e-06 0.00% Castro::retry_advance_ctu() 10 3.707e-06 3.707e-06 3.707e-06 0.00% Castro::FluxRegCrseInit 10 2.753e-06 2.753e-06 2.753e-06 0.00% Castro::FluxRegFineAdd() 10 2.057e-06 2.057e-06 2.057e-06 0.00% AmrLevel::checkPointPre() 3 1.941e-06 1.941e-06 1.941e-06 0.00% MLLinOp::makeSubCommunicator() 11 1.87e-06 1.87e-06 1.87e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-8-g3f715d29c94b) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.08-8-g3f715d29c94b) initialized Starting run at 08:54:58 UTC on 2022-08-10. Successfully read inputs file ... Castro git describe: 22.08-8-g2409a13d7 AMReX git describe: 22.08-8-g3f715d29c Microphysics git describe: 22.08-1-gffefdd62 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.464540679 Restart time = 0.047833716 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.053214138 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.048726208 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.052301588 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.066148943 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.070393407 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.031343426 seconds Ending run at 08:54:58 UTC on 2022-08-10. Run time = 0.370957364 Run time without initialization = 0.322543661 Average number of zones advanced per microsecond: 4.064 Average number of zones advanced per microsecond per rank: 4.064 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.371 ... 0.371 ... 0.371 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0964 0.0964 0.0964 25.98% VisMF::Read() 3 0.04002 0.04002 0.04002 10.79% MLCellLinOp::applyBC() 1946 0.03365 0.03365 0.03365 9.07% VisMF::Write(FabArray) 1 0.02979 0.02979 0.02979 8.03% MLPoisson::Fsmooth() 1440 0.02679 0.02679 0.02679 7.22% StateData::FillBoundary(geom) 160 0.01113 0.01113 0.01113 3.00% MLCGSolver::bicgstab 36 0.009923 0.009923 0.009923 2.67% MultiFab::Dot() 484 0.009248 0.009248 0.009248 2.49% StateDataPhysBCFunct::() 20 0.007666 0.007666 0.007666 2.07% Castro::normalize_species() 30 0.007005 0.007005 0.007005 1.89% Castro::enforce_min_density() 30 0.006701 0.006701 0.006701 1.81% FabArray::setVal() 537 0.006548 0.006548 0.006548 1.76% FillBoundary_nowait() 1766 0.006234 0.006234 0.006234 1.68% MLCellLinOp::defineAuxData() 6 0.005921 0.005921 0.005921 1.60% MultiFab::LinComb() 690 0.005913 0.005913 0.005913 1.59% FabArray::ParallelCopy_nowait() 380 0.005826 0.005826 0.005826 1.57% Castro::computeTemp() 30 0.005686 0.005686 0.005686 1.53% Gravity::fill_multipole_BCs() 6 0.00554 0.00554 0.00554 1.49% MLPoisson::Fapply() 500 0.004922 0.004922 0.004922 1.33% Amr::restart() 1 0.003648 0.003648 0.003648 0.98% MLMG::addInterpCorrection() 180 0.003249 0.003249 0.003249 0.88% amrex::average_down 180 0.002906 0.002906 0.002906 0.78% MultiFab::Xpay() 258 0.002797 0.002797 0.002797 0.75% Castro::estTimeStep() 10 0.002765 0.002765 0.002765 0.75% Castro::do_advance_ctu() 5 0.002257 0.002257 0.002257 0.61% Castro::reset_internal_energy(MultiFab) 30 0.00221 0.00221 0.00221 0.60% BndryData::define() 6 0.002013 0.002013 0.002013 0.54% Castro::construct_new_gravity_source() 5 0.001667 0.001667 0.001667 0.45% Amr::writePlotFile() 1 0.001649 0.001649 0.001649 0.44% Castro::construct_old_gravity_source() 5 0.00147 0.00147 0.00147 0.40% MultiFab::Saxpy() 10 0.0009163 0.0009163 0.0009163 0.25% MLMG::ResNormInf() 42 0.0009003 0.0009003 0.0009003 0.24% Gravity::get_old_grav_vector() 5 0.0008995 0.0008995 0.0008995 0.24% Castro::enforce_speed_limit() 30 0.0008884 0.0008884 0.0008884 0.24% Castro::expand_state() 5 0.0008701 0.0008701 0.0008701 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008583 0.0008583 0.0008583 0.23% Gravity::get_new_grav_vector() 5 0.0008546 0.0008546 0.0008546 0.23% MLCellLinOp::setLevelBC() 6 0.0007997 0.0007997 0.0007997 0.22% Gravity::actual_solve_with_mlmg() 6 0.0007616 0.0007616 0.0007616 0.21% MultiFab::Add() 36 0.0007063 0.0007063 0.0007063 0.19% Castro::reset_internal_energy(Fab) 240 0.0006973 0.0006973 0.0006973 0.19% FabArray::mult() 22 0.0006455 0.0006455 0.0006455 0.17% MLMG::prepareForSolve() 6 0.0006411 0.0006411 0.0006411 0.17% FabArray::setDomainBndry() 20 0.0006332 0.0006332 0.0006332 0.17% MLCellLinOp::prepareForSolve() 6 0.0006096 0.0006096 0.0006096 0.16% MultiFab::contains_nan() 10 0.0005879 0.0005879 0.0005879 0.16% MLCellLinOp::compGrad() 6 0.0004871 0.0004871 0.0004871 0.13% MLCellLinOp::smooth() 720 0.0004852 0.0004852 0.0004852 0.13% Amr::InitAmr() 1 0.0004014 0.0004014 0.0004014 0.11% FabArrayBase::CPC::define() 244 0.0003924 0.0003924 0.0003924 0.11% FabArray::FillBoundary() 1766 0.0003847 0.0003847 0.0003847 0.10% FabArrayBase::getCPC() 632 0.0003734 0.0003734 0.0003734 0.10% main() 1 0.0002675 0.0002675 0.0002675 0.07% FabArrayBase::getFB() 1766 0.0002672 0.0002672 0.0002672 0.07% Gravity::update_max_rhs() 6 0.0002354 0.0002354 0.0002354 0.06% Gravity::solve_for_phi() 5 0.0002173 0.0002173 0.0002173 0.06% MLCellLinOp::apply() 500 0.0002018 0.0002018 0.0002018 0.05% CGSolver::sxay() 690 0.0001744 0.0001744 0.0001744 0.05% MultiFab::Copy() 6 0.0001735 0.0001735 0.0001735 0.05% Amr::coarseTimeStep() 5 0.000173 0.000173 0.000173 0.05% FillPatchIterator::Initialize 20 0.0001498 0.0001498 0.0001498 0.04% MLCellLinOp::defineBC() 6 0.0001436 0.0001436 0.0001436 0.04% MultiFab::max() 6 0.0001363 0.0001363 0.0001363 0.04% MLCGSolver::ParallelAllReduce 659 0.000123 0.000123 0.000123 0.03% FabArray::ParallelCopy() 380 0.0001204 0.0001204 0.0001204 0.03% Castro::construct_new_gravity() 5 0.0001181 0.0001181 0.0001181 0.03% MLMG::MLRhsNormInf() 6 0.0001122 0.0001122 0.0001122 0.03% Amr::timeStep() 5 0.000112 0.000112 0.000112 0.03% Castro::create_source_corrector() 5 0.0001106 0.0001106 0.0001106 0.03% Castro::subcycle_advance_ctu() 5 9.649e-05 9.649e-05 9.649e-05 0.03% MLCellLinOp::correctionResidual() 216 9.606e-05 9.606e-05 9.606e-05 0.03% MLMG::mgVcycle() 36 8.685e-05 8.685e-05 8.685e-05 0.02% AmrLevel::restart() 1 7.45e-05 7.45e-05 7.45e-05 0.02% MLLinOp::defineGrids() 6 7.181e-05 7.181e-05 7.181e-05 0.02% MLMG:computeResOfCorrection() 180 7.005e-05 7.005e-05 7.005e-05 0.02% StateData::restartDoit() 4 6.843e-05 6.843e-05 6.843e-05 0.02% FabArrayBase::FB::FB() 26 5.669e-05 5.669e-05 5.669e-05 0.02% MLMG::mgVcycle_down::0 36 5.07e-05 5.07e-05 5.07e-05 0.01% Castro::construct_new_source() 25 4.663e-05 4.663e-05 4.663e-05 0.01% Castro::advance() 5 4.606e-05 4.606e-05 4.606e-05 0.01% Castro::construct_old_source() 25 4.606e-05 4.606e-05 4.606e-05 0.01% MLMG::mgVcycle_down::1 36 4.181e-05 4.181e-05 4.181e-05 0.01% Castro::clean_state() 30 4.025e-05 4.025e-05 4.025e-05 0.01% MLMG::mgVcycle_down::2 36 3.947e-05 3.947e-05 3.947e-05 0.01% MLMG::mgVcycle_down::4 36 3.89e-05 3.89e-05 3.89e-05 0.01% Castro::initialize_advance() 5 3.872e-05 3.872e-05 3.872e-05 0.01% MLMG::mgVcycle_down::3 36 3.724e-05 3.724e-05 3.724e-05 0.01% Castro::finalize_advance() 5 3.598e-05 3.598e-05 3.598e-05 0.01% MLMG::actualBottomSolve() 36 3.564e-05 3.564e-05 3.564e-05 0.01% MLMG::mgVcycle_up::4 36 3.421e-05 3.421e-05 3.421e-05 0.01% Castro::buildMetrics() 1 3.285e-05 3.285e-05 3.285e-05 0.01% Castro::initialize_do_advance() 5 3.136e-05 3.136e-05 3.136e-05 0.01% MLMG::solve() 6 3.122e-05 3.122e-05 3.122e-05 0.01% Gravity::actual_multilevel_solve() 1 2.942e-05 2.942e-05 2.942e-05 0.01% Castro::post_restart() 1 2.932e-05 2.932e-05 2.932e-05 0.01% Castro::initMFs() 1 2.823e-05 2.823e-05 2.823e-05 0.01% Amr::writeSmallPlotFile() 1 2.746e-05 2.746e-05 2.746e-05 0.01% MLMG::mgVcycle_up::0 36 2.739e-05 2.739e-05 2.739e-05 0.01% Castro::swap_state_time_levels() 5 2.733e-05 2.733e-05 2.733e-05 0.01% MLMG::oneIter() 36 2.72e-05 2.72e-05 2.72e-05 0.01% MLMG::mgVcycle_up::3 36 2.709e-05 2.709e-05 2.709e-05 0.01% MLMG::mgVcycle_up::2 36 2.632e-05 2.632e-05 2.632e-05 0.01% MLMG::mgVcycle_up::1 36 2.538e-05 2.538e-05 2.538e-05 0.01% Castro::computeNewDt() 5 2.413e-05 2.413e-05 2.413e-05 0.01% MLCellLinOp::solutionResidual() 42 2.188e-05 2.188e-05 2.188e-05 0.01% MLLinOp::define() 6 2.115e-05 2.115e-05 2.115e-05 0.01% MLMG::computeResidual() 36 1.769e-05 1.769e-05 1.769e-05 0.00% Castro::finalize_do_advance() 5 1.744e-05 1.744e-05 1.744e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 1.692e-05 1.692e-05 1.692e-05 0.00% MLPoisson::define() 6 1.552e-05 1.552e-05 1.552e-05 0.00% MLMG::mgVcycle_bottom 36 1.535e-05 1.535e-05 1.535e-05 0.00% FillPatchSingleLevel 20 1.416e-05 1.416e-05 1.416e-05 0.00% makeSFC 30 1.353e-05 1.353e-05 1.353e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.346e-05 1.346e-05 1.346e-05 0.00% MLPoisson::prepareForSolve() 6 9.54e-06 9.54e-06 9.54e-06 0.00% Castro::do_new_sources() 5 9.202e-06 9.202e-06 9.202e-06 0.00% Amr::initSubcycle() 1 8.974e-06 8.974e-06 8.974e-06 0.00% DistributionMapping::Distribute() 31 8.745e-06 8.745e-06 8.745e-06 0.00% Castro::do_old_sources() 5 8.207e-06 8.207e-06 8.207e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.49e-06 7.49e-06 7.49e-06 0.00% Castro::construct_old_gravity() 5 6.062e-06 6.062e-06 6.062e-06 0.00% Castro::check_for_nan() 10 5.712e-06 5.712e-06 5.712e-06 0.00% Castro::apply_source_to_state() 10 5.571e-06 5.571e-06 5.571e-06 0.00% Castro::post_timestep() 5 4.714e-06 4.714e-06 4.714e-06 0.00% MLMG::computeMLResidual() 6 4.644e-06 4.644e-06 4.644e-06 0.00% Gravity::swapTimeLevels() 5 4.263e-06 4.263e-06 4.263e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.397e-06 3.397e-06 3.397e-06 0.00% MLMG::getGradSolution() 6 3.04e-06 3.04e-06 3.04e-06 0.00% Gravity::set_mass_offset() 6 2.174e-06 2.174e-06 2.174e-06 0.00% MLMG::MLResNormInf() 6 2.086e-06 2.086e-06 2.086e-06 0.00% Castro::retry_advance_ctu() 5 1.65e-06 1.65e-06 1.65e-06 0.00% Castro::FluxRegCrseInit 5 1.5e-06 1.5e-06 1.5e-06 0.00% Amr::init() 1 1.332e-06 1.332e-06 1.332e-06 0.00% Castro::FluxRegFineAdd() 5 1.144e-06 1.144e-06 1.144e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.082e-06 1.082e-06 1.082e-06 0.00% AmrLevel::AmrLevel() 1 8.9e-07 8.9e-07 8.9e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.371 0.371 0.371 100.00% Amr::coarseTimeStep() 5 0.291 0.291 0.291 78.42% Amr::timeStep() 5 0.2892 0.2892 0.2892 77.95% Castro::advance() 5 0.2856 0.2856 0.2856 76.97% Castro::subcycle_advance_ctu() 5 0.2791 0.2791 0.2791 75.22% Castro::do_advance_ctu() 5 0.279 0.279 0.279 75.19% Castro::construct_new_gravity() 5 0.1418 0.1418 0.1418 38.23% Gravity::solve_phi_with_mlmg() 6 0.1369 0.1369 0.1369 36.91% Gravity::solve_for_phi() 5 0.1335 0.1335 0.1335 35.98% Gravity::actual_solve_with_mlmg() 6 0.1313 0.1313 0.1313 35.39% MLMG::solve() 6 0.1195 0.1195 0.1195 32.21% MLMG::oneIter() 36 0.1124 0.1124 0.1124 30.31% MLMG::mgVcycle() 36 0.1117 0.1117 0.1117 30.11% Castro::construct_ctu_hydro_source() 5 0.09639 0.09639 0.09639 25.98% MLCellLinOp::smooth() 720 0.05735 0.05735 0.05735 15.46% Amr::init() 1 0.0479 0.0479 0.0479 12.91% Amr::restart() 1 0.0479 0.0479 0.0479 12.91% MLCellLinOp::applyBC() 1946 0.0406 0.0406 0.0406 10.94% AmrLevel::restart() 1 0.04022 0.04022 0.04022 10.84% StateData::restartDoit() 4 0.04014 0.04014 0.04014 10.82% VisMF::Read() 3 0.04002 0.04002 0.04002 10.79% MLMG::mgVcycle_bottom 36 0.03398 0.03398 0.03398 9.16% MLMG::actualBottomSolve() 36 0.03397 0.03397 0.03397 9.16% MLCGSolver::bicgstab 36 0.03363 0.03363 0.03363 9.06% Amr::writePlotFile() 1 0.03143 0.03143 0.03143 8.47% VisMF::Write(FabArray) 1 0.02979 0.02979 0.02979 8.03% MLPoisson::Fsmooth() 1440 0.02679 0.02679 0.02679 7.22% Castro::clean_state() 30 0.02323 0.02323 0.02323 6.26% FillPatchIterator::Initialize 20 0.02161 0.02161 0.02161 5.82% FillPatchSingleLevel 20 0.02082 0.02082 0.02082 5.61% StateDataPhysBCFunct::() 20 0.0188 0.0188 0.0188 5.07% MLCellLinOp::apply() 500 0.01538 0.01538 0.01538 4.14% MLMG::mgVcycle_down::0 36 0.01513 0.01513 0.01513 4.08% MLMG::mgVcycle_up::0 36 0.0129 0.0129 0.0129 3.48% StateData::FillBoundary(geom) 160 0.01113 0.01113 0.01113 3.00% Castro::initialize_do_advance() 5 0.01039 0.01039 0.01039 2.80% MLPoisson::define() 6 0.009464 0.009464 0.009464 2.55% MultiFab::Dot() 484 0.009248 0.009248 0.009248 2.49% MLCellLinOp::correctionResidual() 216 0.008955 0.008955 0.008955 2.41% Castro::computeTemp() 30 0.008594 0.008594 0.008594 2.32% Gravity::get_new_grav_vector() 5 0.008219 0.008219 0.008219 2.22% Castro::construct_old_gravity() 5 0.007867 0.007867 0.007867 2.12% Gravity::get_old_grav_vector() 5 0.007861 0.007861 0.007861 2.12% MLMG:computeResOfCorrection() 180 0.007746 0.007746 0.007746 2.09% MLMG::mgVcycle_down::1 36 0.007426 0.007426 0.007426 2.00% MLMG::mgVcycle_down::2 36 0.007219 0.007219 0.007219 1.95% Castro::do_new_sources() 5 0.007097 0.007097 0.007097 1.91% Castro::normalize_species() 30 0.007005 0.007005 0.007005 1.89% FabArray::FillBoundary() 1766 0.006942 0.006942 0.006942 1.87% MLMG::mgVcycle_down::3 36 0.006847 0.006847 0.006847 1.85% Castro::enforce_min_density() 30 0.006701 0.006701 0.006701 1.81% MLCellLinOp::defineAuxData() 6 0.006637 0.006637 0.006637 1.79% MLMG::mgVcycle_down::4 36 0.006561 0.006561 0.006561 1.77% FillBoundary_nowait() 1766 0.006557 0.006557 0.006557 1.77% FabArray::setVal() 537 0.006548 0.006548 0.006548 1.76% Castro::initialize_advance() 5 0.006418 0.006418 0.006418 1.73% FabArray::ParallelCopy() 380 0.006323 0.006323 0.006323 1.70% FabArray::ParallelCopy_nowait() 380 0.006202 0.006202 0.006202 1.67% CGSolver::sxay() 690 0.006088 0.006088 0.006088 1.64% Castro::expand_state() 5 0.005984 0.005984 0.005984 1.61% MultiFab::LinComb() 690 0.005913 0.005913 0.005913 1.59% MLCGSolver::ParallelAllReduce 659 0.005545 0.005545 0.005545 1.49% MLMG::mgVcycle_up::2 36 0.005542 0.005542 0.005542 1.49% Gravity::fill_multipole_BCs() 6 0.00554 0.00554 0.00554 1.49% MLMG::mgVcycle_up::1 36 0.005456 0.005456 0.005456 1.47% Castro::do_old_sources() 5 0.0054 0.0054 0.0054 1.46% MLMG::addInterpCorrection() 180 0.005395 0.005395 0.005395 1.45% MLMG::mgVcycle_up::4 36 0.005281 0.005281 0.005281 1.42% MLMG::mgVcycle_up::3 36 0.005275 0.005275 0.005275 1.42% amrex::average_down 180 0.005074 0.005074 0.005074 1.37% MLPoisson::Fapply() 500 0.004922 0.004922 0.004922 1.33% Castro::post_restart() 1 0.003849 0.003849 0.003849 1.04% Gravity::multilevel_solve_for_new_phi() 1 0.003726 0.003726 0.003726 1.00% Gravity::actual_multilevel_solve() 1 0.003709 0.003709 0.003709 1.00% Castro::post_timestep() 5 0.003503 0.003503 0.003503 0.94% MLCellLinOp::solutionResidual() 42 0.0032 0.0032 0.0032 0.86% Castro::reset_internal_energy(MultiFab) 30 0.002908 0.002908 0.002908 0.78% MultiFab::Xpay() 258 0.002797 0.002797 0.002797 0.75% MLMG::prepareForSolve() 6 0.00279 0.00279 0.00279 0.75% Castro::estTimeStep() 10 0.002765 0.002765 0.002765 0.75% MLCellLinOp::defineBC() 6 0.002688 0.002688 0.002688 0.72% MLMG::computeResidual() 36 0.002659 0.002659 0.002659 0.72% BndryData::define() 6 0.002545 0.002545 0.002545 0.69% Castro::construct_new_source() 25 0.001714 0.001714 0.001714 0.46% Castro::construct_new_gravity_source() 5 0.001667 0.001667 0.001667 0.45% Castro::computeNewDt() 5 0.001585 0.001585 0.001585 0.43% Castro::construct_old_source() 25 0.001516 0.001516 0.001516 0.41% Castro::construct_old_gravity_source() 5 0.00147 0.00147 0.00147 0.40% Castro::apply_source_to_state() 10 0.0009219 0.0009219 0.0009219 0.25% MultiFab::Saxpy() 10 0.0009163 0.0009163 0.0009163 0.25% MLMG::ResNormInf() 42 0.0009003 0.0009003 0.0009003 0.24% Castro::enforce_speed_limit() 30 0.0008884 0.0008884 0.0008884 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008583 0.0008583 0.0008583 0.23% MLCellLinOp::setLevelBC() 6 0.0007997 0.0007997 0.0007997 0.22% FabArrayBase::getCPC() 632 0.0007658 0.0007658 0.0007658 0.21% MLMG::getGradSolution() 6 0.0007586 0.0007586 0.0007586 0.20% MLCellLinOp::compGrad() 6 0.0007555 0.0007555 0.0007555 0.20% MultiFab::Add() 36 0.0007063 0.0007063 0.0007063 0.19% Castro::reset_internal_energy(Fab) 240 0.0006973 0.0006973 0.0006973 0.19% FabArray::mult() 22 0.0006455 0.0006455 0.0006455 0.17% FabArray::setDomainBndry() 20 0.0006332 0.0006332 0.0006332 0.17% MLPoisson::prepareForSolve() 6 0.0006192 0.0006192 0.0006192 0.17% MLCellLinOp::prepareForSolve() 6 0.0006096 0.0006096 0.0006096 0.16% Castro::check_for_nan() 10 0.0005937 0.0005937 0.0005937 0.16% MultiFab::contains_nan() 10 0.0005879 0.0005879 0.0005879 0.16% MLMG::computeMLResidual() 6 0.0005629 0.0005629 0.0005629 0.15% Gravity::update_max_rhs() 6 0.0004478 0.0004478 0.0004478 0.12% Amr::InitAmr() 1 0.0004104 0.0004104 0.0004104 0.11% FabArrayBase::CPC::define() 244 0.0003924 0.0003924 0.0003924 0.11% FabArrayBase::getFB() 1766 0.0003239 0.0003239 0.0003239 0.09% Gravity::swapTimeLevels() 5 0.0002235 0.0002235 0.0002235 0.06% MultiFab::Copy() 6 0.0001735 0.0001735 0.0001735 0.05% Castro::buildMetrics() 1 0.000148 0.000148 0.000148 0.04% MLMG::MLResNormInf() 6 0.0001461 0.0001461 0.0001461 0.04% MultiFab::max() 6 0.0001363 0.0001363 0.0001363 0.04% MLLinOp::define() 6 0.0001224 0.0001224 0.0001224 0.03% MLMG::MLRhsNormInf() 6 0.0001122 0.0001122 0.0001122 0.03% Castro::create_source_corrector() 5 0.0001106 0.0001106 0.0001106 0.03% MLLinOp::defineGrids() 6 0.0001013 0.0001013 0.0001013 0.03% FabArrayBase::FB::FB() 26 5.669e-05 5.669e-05 5.669e-05 0.02% Castro::finalize_advance() 5 3.862e-05 3.862e-05 3.862e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.841e-05 2.841e-05 2.841e-05 0.01% Castro::initMFs() 1 2.823e-05 2.823e-05 2.823e-05 0.01% Amr::writeSmallPlotFile() 1 2.746e-05 2.746e-05 2.746e-05 0.01% Castro::swap_state_time_levels() 5 2.733e-05 2.733e-05 2.733e-05 0.01% makeSFC 30 2.092e-05 2.092e-05 2.092e-05 0.01% Castro::finalize_do_advance() 5 1.744e-05 1.744e-05 1.744e-05 0.00% Amr::initSubcycle() 1 8.974e-06 8.974e-06 8.974e-06 0.00% DistributionMapping::Distribute() 31 8.745e-06 8.745e-06 8.745e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.755e-06 4.755e-06 4.755e-06 0.00% Gravity::set_mass_offset() 6 2.174e-06 2.174e-06 2.174e-06 0.00% Castro::retry_advance_ctu() 5 1.65e-06 1.65e-06 1.65e-06 0.00% Castro::FluxRegCrseInit 5 1.5e-06 1.5e-06 1.5e-06 0.00% Castro::FluxRegFineAdd() 5 1.144e-06 1.144e-06 1.144e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.082e-06 1.082e-06 1.082e-06 0.00% AmrLevel::AmrLevel() 1 8.9e-07 8.9e-07 8.9e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2465 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.08-8-g3f715d29c94b) finalized