Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-2-g1a5b308c8af7) initialized Starting run at 09:54:18 UTC on 2022-12-07. Successfully read inputs file ... Castro git describe: 22.12-3-gdebbe0b78 AMReX git describe: 22.12-2-g1a5b308c8 Microphysics git describe: 22.12-11-ge920a815 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.052329841 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.030292949 seconds [Level 0 step 1] ADVANCE with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.049379948 [STEP 1] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.05190312 [STEP 2] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.051080415 [STEP 3] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.05812341 [STEP 4] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.072481282 [STEP 5] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.048409297 secs. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.06715163 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.05061979 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.05966547 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.059573537 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.06184144 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.04821012 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029836043 seconds Ending run at 09:54:19 UTC on 2022-12-07. Run time = 0.843688872 Run time without initialization = 0.708905444 Average number of zones advanced per microsecond: 3.698 Average number of zones advanced per microsecond per rank: 3.698 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.8437 ... 0.8437 ... 0.8437 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2024 0.2024 0.2024 23.99% VisMF::Write(FabArray) 11 0.2009 0.2009 0.2009 23.81% MLCellLinOp::applyBC() 4433 0.08008 0.08008 0.08008 9.49% MLPoisson::Fsmooth() 3280 0.06338 0.06338 0.06338 7.51% StateData::FillBoundary(geom) 328 0.02313 0.02313 0.02313 2.74% amrex::Dot() 1114 0.02221 0.02221 0.02221 2.63% amrex::Copy() 1029 0.01578 0.01578 0.01578 1.87% FabArray::LinComb() 1586 0.01429 0.01429 0.01429 1.69% FabArray::setVal() 1144 0.01421 0.01421 0.01421 1.68% FillBoundary_nowait() 4023 0.01406 0.01406 0.01406 1.67% Castro::normalize_species() 62 0.0139 0.0139 0.0139 1.65% FabArray::ParallelCopy_nowait() 861 0.01316 0.01316 0.01316 1.56% StateDataPhysBCFunct::() 41 0.01288 0.01288 0.01288 1.53% FabArray::norminf() 639 0.01286 0.01286 0.01286 1.52% Castro::computeTemp() 63 0.01266 0.01266 0.01266 1.50% MLPoisson::Fapply() 1142 0.01167 0.01167 0.01167 1.38% MLCellLinOp::defineAuxData() 11 0.01159 0.01159 0.01159 1.37% Castro::enforce_min_density() 62 0.008895 0.008895 0.008895 1.05% Gravity::fill_multipole_BCs() 11 0.008046 0.008046 0.008046 0.95% MLMG::addInterpCorrection() 410 0.007734 0.007734 0.007734 0.92% amrex::average_down 410 0.00684 0.00684 0.00684 0.81% FabArray::Xpay() 585 0.006564 0.006564 0.006564 0.78% Amr::checkPoint() 3 0.005331 0.005331 0.005331 0.63% Castro::estTimeStep() 21 0.00431 0.00431 0.00431 0.51% Castro::reset_internal_energy(MultiFab) 63 0.004003 0.004003 0.004003 0.47% BndryData::define() 11 0.003804 0.003804 0.003804 0.45% Castro::construct_new_gravity_source() 10 0.00331 0.00331 0.00331 0.39% MLCGSolver::bicgstab 82 0.003185 0.003185 0.003185 0.38% Castro::do_advance_ctu() 10 0.002706 0.002706 0.002706 0.32% Castro::construct_old_gravity_source() 10 0.002547 0.002547 0.002547 0.30% amrex::Add() 164 0.00252 0.00252 0.00252 0.30% Amr::writePlotFile() 2 0.002371 0.002371 0.002371 0.28% MLMG::ResNormInf() 93 0.002072 0.002072 0.002072 0.25% FabArray::Saxpy() 20 0.001806 0.001806 0.001806 0.21% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001628 0.001628 0.001628 0.19% Castro::reset_internal_energy(Fab) 504 0.001579 0.001579 0.001579 0.19% MLCellLinOp::setLevelBC() 11 0.00151 0.00151 0.00151 0.18% Castro::enforce_speed_limit() 62 0.001414 0.001414 0.001414 0.17% Gravity::actual_solve_with_mlmg() 11 0.00135 0.00135 0.00135 0.16% FabArray::mult() 43 0.001313 0.001313 0.001313 0.16% FabArray::setDomainBndry() 41 0.001295 0.001295 0.001295 0.15% MLMG::prepareForSolve() 11 0.001215 0.001215 0.001215 0.14% Castro::initData() 1 0.001199 0.001199 0.001199 0.14% MultiFab::contains_nan() 20 0.001197 0.001197 0.001197 0.14% MLCellLinOp::prepareForSolve() 11 0.001168 0.001168 0.001168 0.14% MLCellLinOp::smooth() 1640 0.0009703 0.0009703 0.0009703 0.12% MLCellLinOp::compGrad() 11 0.0009077 0.0009077 0.0009077 0.11% FabArray::FillBoundary() 4023 0.0007683 0.0007683 0.0007683 0.09% FabArrayBase::getCPC() 1323 0.0007462 0.0007462 0.0007462 0.09% FabArrayBase::CPC::define() 454 0.0006799 0.0006799 0.0006799 0.08% Gravity::get_new_grav_vector() 11 0.000625 0.000625 0.000625 0.07% FabArrayBase::getFB() 4023 0.0006023 0.0006023 0.0006023 0.07% Gravity::get_old_grav_vector() 10 0.0005508 0.0005508 0.0005508 0.07% Amr::InitAmr() 1 0.0005061 0.0005061 0.0005061 0.06% MLCellLinOp::apply() 1142 0.0004335 0.0004335 0.0004335 0.05% MLMG::mgVcycle() 82 0.0003799 0.0003799 0.0003799 0.05% CGSolver::sxay() 1586 0.0003408 0.0003408 0.0003408 0.04% MLLinOp::defineGrids() 11 0.0003398 0.0003398 0.0003398 0.04% Amr::coarseTimeStep() 10 0.0003371 0.0003371 0.0003371 0.04% FillPatchIterator::Initialize 41 0.0002996 0.0002996 0.0002996 0.04% MLCellLinOp::defineBC() 11 0.0002879 0.0002879 0.0002879 0.03% MLCGSolver::ParallelAllReduce 1514 0.0002873 0.0002873 0.0002873 0.03% main() 1 0.0002699 0.0002699 0.0002699 0.03% MultiFab::max() 11 0.0002555 0.0002555 0.0002555 0.03% FabArray::ParallelCopy() 861 0.0002554 0.0002554 0.0002554 0.03% MLMG::MLRhsNormInf() 11 0.0002167 0.0002167 0.0002167 0.03% MLCellLinOp::correctionResidual() 492 0.0001932 0.0001932 0.0001932 0.02% Amr::timeStep() 10 0.0001553 0.0001553 0.0001553 0.02% Gravity::solve_for_phi() 10 0.0001458 0.0001458 0.0001458 0.02% MLMG:computeResOfCorrection() 410 0.000143 0.000143 0.000143 0.02% Gravity::update_max_rhs() 11 0.0001414 0.0001414 0.0001414 0.02% Castro::subcycle_advance_ctu() 10 0.0001367 0.0001367 0.0001367 0.02% StateData::checkPoint() 12 0.0001335 0.0001335 0.0001335 0.02% MLMG::mgVcycle_down::0 82 9.977e-05 9.977e-05 9.977e-05 0.01% Castro::finalize_advance() 10 9.022e-05 9.022e-05 9.022e-05 0.01% MLMG::mgVcycle_down::1 82 8.794e-05 8.794e-05 8.794e-05 0.01% FabArrayBase::FB::FB() 56 8.405e-05 8.405e-05 8.405e-05 0.01% Castro::initialize_advance() 10 8.318e-05 8.318e-05 8.318e-05 0.01% Castro::clean_state() 62 8.157e-05 8.157e-05 8.157e-05 0.01% Castro::Castro() 1 8.144e-05 8.144e-05 8.144e-05 0.01% MLMG::mgVcycle_down::2 82 8.134e-05 8.134e-05 8.134e-05 0.01% Castro::advance() 10 7.963e-05 7.963e-05 7.963e-05 0.01% Castro::expand_state() 10 7.705e-05 7.705e-05 7.705e-05 0.01% MLMG::mgVcycle_down::3 82 7.591e-05 7.591e-05 7.591e-05 0.01% MLMG::mgVcycle_down::4 82 7.527e-05 7.527e-05 7.527e-05 0.01% MLMG::actualBottomSolve() 82 7.431e-05 7.431e-05 7.431e-05 0.01% AmrLevel::checkPoint() 3 7.09e-05 7.09e-05 7.09e-05 0.01% MLMG::solve() 11 6.983e-05 6.983e-05 6.983e-05 0.01% MLMG::mgVcycle_up::4 82 6.658e-05 6.658e-05 6.658e-05 0.01% Castro::initialize_do_advance() 10 6.597e-05 6.597e-05 6.597e-05 0.01% MLMG::oneIter() 82 5.785e-05 5.785e-05 5.785e-05 0.01% MLMG::mgVcycle_up::0 82 5.573e-05 5.573e-05 5.573e-05 0.01% MLMG::mgVcycle_up::1 82 5.277e-05 5.277e-05 5.277e-05 0.01% MLMG::mgVcycle_up::3 82 5.227e-05 5.227e-05 5.227e-05 0.01% MLMG::mgVcycle_up::2 82 5.186e-05 5.186e-05 5.186e-05 0.01% StateData::define() 4 4.49e-05 4.49e-05 4.49e-05 0.01% MLCellLinOp::solutionResidual() 93 4.366e-05 4.366e-05 4.366e-05 0.01% Castro::swap_state_time_levels() 10 4.339e-05 4.339e-05 4.339e-05 0.01% MLMG::computeResidual() 82 4.15e-05 4.15e-05 4.15e-05 0.00% MLMG::mgVcycle_bottom 82 3.508e-05 3.508e-05 3.508e-05 0.00% MLPoisson::define() 11 3.376e-05 3.376e-05 3.376e-05 0.00% Castro::finalize_do_advance() 10 3.34e-05 3.34e-05 3.34e-05 0.00% Castro::enforce_consistent_e() 1 3.298e-05 3.298e-05 3.298e-05 0.00% FillPatchSingleLevel 41 3.041e-05 3.041e-05 3.041e-05 0.00% MultiFab::Add() 82 2.96e-05 2.96e-05 2.96e-05 0.00% Castro::initMFs() 1 2.854e-05 2.854e-05 2.854e-05 0.00% Castro::post_timestep() 10 2.801e-05 2.801e-05 2.801e-05 0.00% MLLinOp::define() 11 2.675e-05 2.675e-05 2.675e-05 0.00% makeSFC 55 2.591e-05 2.591e-05 2.591e-05 0.00% Amr::writeSmallPlotFile() 1 2.555e-05 2.555e-05 2.555e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.446e-05 2.446e-05 2.446e-05 0.00% Castro::construct_new_gravity() 10 2.421e-05 2.421e-05 2.421e-05 0.00% Castro::buildMetrics() 1 2.297e-05 2.297e-05 2.297e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.039e-05 2.039e-05 2.039e-05 0.00% Amr::FinalizeInit() 1 1.988e-05 1.988e-05 1.988e-05 0.00% Amr::defBaseLevel() 1 1.983e-05 1.983e-05 1.983e-05 0.00% Castro::construct_old_source() 50 1.706e-05 1.706e-05 1.706e-05 0.00% Castro::do_new_sources() 10 1.643e-05 1.643e-05 1.643e-05 0.00% Castro::construct_new_source() 50 1.643e-05 1.643e-05 1.643e-05 0.00% Castro::do_old_sources() 10 1.58e-05 1.58e-05 1.58e-05 0.00% MLLinOp::makeAgglomeratedDMap 11 1.512e-05 1.512e-05 1.512e-05 0.00% DistributionMapping::Distribute() 56 1.506e-05 1.506e-05 1.506e-05 0.00% AmrLevel::AmrLevel(dm) 1 1.496e-05 1.496e-05 1.496e-05 0.00% Castro::check_for_nan() 20 1.148e-05 1.148e-05 1.148e-05 0.00% Castro::apply_source_to_state() 20 1.013e-05 1.013e-05 1.013e-05 0.00% Castro::construct_old_gravity() 10 9.862e-06 9.862e-06 9.862e-06 0.00% Amr::initSubcycle() 1 9.505e-06 9.505e-06 9.505e-06 0.00% Gravity::swapTimeLevels() 10 9.214e-06 9.214e-06 9.214e-06 0.00% MLPoisson::prepareForSolve() 11 9.028e-06 9.028e-06 9.028e-06 0.00% Gravity::actual_multilevel_solve() 1 9.009e-06 9.009e-06 9.009e-06 0.00% Castro::post_init() 1 8.97e-06 8.97e-06 8.97e-06 0.00% MLMG::computeMLResidual() 11 8.562e-06 8.562e-06 8.562e-06 0.00% MLMG::getGradSolution() 11 6.976e-06 6.976e-06 6.976e-06 0.00% Castro::computeNewDt() 9 6.306e-06 6.306e-06 6.306e-06 0.00% MultiFab::Copy() 11 5.37e-06 5.37e-06 5.37e-06 0.00% Amr::InitializeInit() 1 5.203e-06 5.203e-06 5.203e-06 0.00% AmrLevel::checkPointPost() 3 4.798e-06 4.798e-06 4.798e-06 0.00% Gravity::set_mass_offset() 11 4.365e-06 4.365e-06 4.365e-06 0.00% Castro::retry_advance_ctu() 10 3.795e-06 3.795e-06 3.795e-06 0.00% Castro::create_source_corrector() 10 3.548e-06 3.548e-06 3.548e-06 0.00% MLMG::MLResNormInf() 11 3.527e-06 3.527e-06 3.527e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.406e-06 3.406e-06 3.406e-06 0.00% Castro::FluxRegCrseInit 10 2.621e-06 2.621e-06 2.621e-06 0.00% Castro::computeInitialDt() 2 2.444e-06 2.444e-06 2.444e-06 0.00% Amr::init() 1 2.369e-06 2.369e-06 2.369e-06 0.00% AmrLevel::checkPointPre() 3 2.135e-06 2.135e-06 2.135e-06 0.00% Castro::FluxRegFineAdd() 10 2.1e-06 2.1e-06 2.1e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.027e-06 2.027e-06 2.027e-06 0.00% Amr::initialInit() 1 1.209e-06 1.209e-06 1.209e-06 0.00% Castro::post_regrid() 1 1.074e-06 1.074e-06 1.074e-06 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8437 0.8437 0.8437 100.00% Amr::coarseTimeStep() 10 0.6789 0.6789 0.6789 80.46% Amr::timeStep() 10 0.5791 0.5791 0.5791 68.63% Castro::advance() 10 0.572 0.572 0.572 67.79% Castro::subcycle_advance_ctu() 10 0.5597 0.5597 0.5597 66.34% Castro::do_advance_ctu() 10 0.5595 0.5595 0.5595 66.32% Gravity::solve_phi_with_mlmg() 11 0.3133 0.3133 0.3133 37.14% Gravity::actual_solve_with_mlmg() 11 0.3048 0.3048 0.3048 36.13% Castro::construct_new_gravity() 10 0.2849 0.2849 0.2849 33.77% MLMG::solve() 11 0.2822 0.2822 0.2822 33.44% Gravity::solve_for_phi() 10 0.2697 0.2697 0.2697 31.96% MLMG::oneIter() 82 0.2673 0.2673 0.2673 31.68% MLMG::mgVcycle() 82 0.2655 0.2655 0.2655 31.47% Castro::construct_ctu_hydro_source() 10 0.2024 0.2024 0.2024 23.99% VisMF::Write(FabArray) 11 0.2009 0.2009 0.2009 23.81% Amr::checkPoint() 3 0.1491 0.1491 0.1491 17.67% AmrLevel::checkPoint() 3 0.1438 0.1438 0.1438 17.04% StateData::checkPoint() 12 0.1437 0.1437 0.1437 17.03% MLCellLinOp::smooth() 1640 0.1353 0.1353 0.1353 16.04% Amr::init() 1 0.1341 0.1341 0.1341 15.90% MLCellLinOp::applyBC() 4433 0.09559 0.09559 0.09559 11.33% MLMG::mgVcycle_bottom 82 0.08195 0.08195 0.08195 9.71% MLMG::actualBottomSolve() 82 0.08191 0.08191 0.08191 9.71% MLCGSolver::bicgstab 82 0.08112 0.08112 0.08112 9.61% MLPoisson::Fsmooth() 3280 0.06338 0.06338 0.06338 7.51% Amr::writePlotFile() 2 0.06026 0.06026 0.06026 7.14% Amr::initialInit() 1 0.05138 0.05138 0.05138 6.09% Amr::FinalizeInit() 1 0.04744 0.04744 0.04744 5.62% Castro::post_init() 1 0.04616 0.04616 0.04616 5.47% Gravity::multilevel_solve_for_new_phi() 1 0.04412 0.04412 0.04412 5.23% Gravity::actual_multilevel_solve() 1 0.0441 0.0441 0.0441 5.23% Castro::clean_state() 62 0.04178 0.04178 0.04178 4.95% FillPatchIterator::Initialize 41 0.04164 0.04164 0.04164 4.94% FillPatchSingleLevel 41 0.04005 0.04005 0.04005 4.75% MLCellLinOp::apply() 1142 0.03623 0.03623 0.03623 4.29% StateDataPhysBCFunct::() 41 0.03601 0.03601 0.03601 4.27% MLMG::mgVcycle_down::0 82 0.03506 0.03506 0.03506 4.16% MLMG::mgVcycle_up::0 82 0.03008 0.03008 0.03008 3.57% StateData::FillBoundary(geom) 328 0.02313 0.02313 0.02313 2.74% amrex::Dot() 1114 0.02221 0.02221 0.02221 2.63% MLCellLinOp::correctionResidual() 492 0.02115 0.02115 0.02115 2.51% MLPoisson::define() 11 0.01837 0.01837 0.01837 2.18% Castro::initialize_do_advance() 10 0.0183 0.0183 0.0183 2.17% MLMG:computeResOfCorrection() 410 0.01826 0.01826 0.01826 2.16% Castro::computeTemp() 63 0.01824 0.01824 0.01824 2.16% MLMG::mgVcycle_down::1 82 0.01764 0.01764 0.01764 2.09% MLMG::mgVcycle_down::2 82 0.0172 0.0172 0.0172 2.04% Gravity::get_new_grav_vector() 11 0.01699 0.01699 0.01699 2.01% MLMG::mgVcycle_down::3 82 0.01631 0.01631 0.01631 1.93% amrex::Copy() 1029 0.01578 0.01578 0.01578 1.87% FabArray::FillBoundary() 4023 0.01551 0.01551 0.01551 1.84% MLMG::mgVcycle_down::4 82 0.01549 0.01549 0.01549 1.84% FillBoundary_nowait() 4023 0.01475 0.01475 0.01475 1.75% Castro::construct_old_gravity() 10 0.01474 0.01474 0.01474 1.75% Gravity::get_old_grav_vector() 10 0.01473 0.01473 0.01473 1.75% CGSolver::sxay() 1586 0.01463 0.01463 0.01463 1.73% FabArray::LinComb() 1586 0.01429 0.01429 0.01429 1.69% FabArray::ParallelCopy() 861 0.01424 0.01424 0.01424 1.69% FabArray::setVal() 1144 0.01421 0.01421 0.01421 1.68% FabArray::ParallelCopy_nowait() 861 0.01398 0.01398 0.01398 1.66% Castro::normalize_species() 62 0.0139 0.0139 0.0139 1.65% MLMG::mgVcycle_up::2 82 0.01326 0.01326 0.01326 1.57% MLCGSolver::ParallelAllReduce 1514 0.01324 0.01324 0.01324 1.57% MLMG::mgVcycle_up::1 82 0.01307 0.01307 0.01307 1.55% MLCellLinOp::defineAuxData() 11 0.01289 0.01289 0.01289 1.53% MLMG::addInterpCorrection() 410 0.01288 0.01288 0.01288 1.53% FabArray::norminf() 639 0.01286 0.01286 0.01286 1.52% MLMG::mgVcycle_up::3 82 0.01256 0.01256 0.01256 1.49% MLMG::mgVcycle_up::4 82 0.01252 0.01252 0.01252 1.48% amrex::average_down 410 0.01192 0.01192 0.01192 1.41% MLPoisson::Fapply() 1142 0.01167 0.01167 0.01167 1.38% Castro::initialize_advance() 10 0.01159 0.01159 0.01159 1.37% Castro::expand_state() 10 0.01115 0.01115 0.01115 1.32% Castro::do_new_sources() 10 0.01106 0.01106 0.01106 1.31% Castro::do_old_sources() 10 0.009803 0.009803 0.009803 1.16% Castro::enforce_min_density() 62 0.008895 0.008895 0.008895 1.05% Gravity::fill_multipole_BCs() 11 0.008278 0.008278 0.008278 0.98% MLCellLinOp::solutionResidual() 93 0.007153 0.007153 0.007153 0.85% Castro::post_timestep() 10 0.006944 0.006944 0.006944 0.82% FabArray::Xpay() 585 0.006564 0.006564 0.006564 0.78% MLMG::computeResidual() 82 0.006185 0.006185 0.006185 0.73% Castro::reset_internal_energy(MultiFab) 63 0.005582 0.005582 0.005582 0.66% MLMG::prepareForSolve() 11 0.005343 0.005343 0.005343 0.63% MLCellLinOp::defineBC() 11 0.005019 0.005019 0.005019 0.59% BndryData::define() 11 0.004732 0.004732 0.004732 0.56% Castro::estTimeStep() 21 0.00431 0.00431 0.00431 0.51% Amr::InitializeInit() 1 0.003945 0.003945 0.003945 0.47% Amr::defBaseLevel() 1 0.00394 0.00394 0.00394 0.47% Castro::initData() 1 0.003426 0.003426 0.003426 0.41% Castro::construct_new_source() 50 0.003327 0.003327 0.003327 0.39% Castro::construct_new_gravity_source() 10 0.00331 0.00331 0.00331 0.39% Castro::construct_old_source() 50 0.002564 0.002564 0.002564 0.30% Castro::construct_old_gravity_source() 10 0.002547 0.002547 0.002547 0.30% amrex::Add() 164 0.00252 0.00252 0.00252 0.30% MLMG::ResNormInf() 93 0.002072 0.002072 0.002072 0.25% Castro::computeNewDt() 9 0.001996 0.001996 0.001996 0.24% Castro::apply_source_to_state() 20 0.001816 0.001816 0.001816 0.22% FabArray::Saxpy() 20 0.001806 0.001806 0.001806 0.21% MultiFab::Add() 82 0.001705 0.001705 0.001705 0.20% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001628 0.001628 0.001628 0.19% Castro::reset_internal_energy(Fab) 504 0.001579 0.001579 0.001579 0.19% MLCellLinOp::setLevelBC() 11 0.00151 0.00151 0.00151 0.18% FabArrayBase::getCPC() 1323 0.001426 0.001426 0.001426 0.17% Castro::enforce_speed_limit() 62 0.001414 0.001414 0.001414 0.17% MLMG::getGradSolution() 11 0.001407 0.001407 0.001407 0.17% MLCellLinOp::compGrad() 11 0.0014 0.0014 0.0014 0.17% FabArray::mult() 43 0.001313 0.001313 0.001313 0.16% FabArray::setDomainBndry() 41 0.001295 0.001295 0.001295 0.15% Castro::check_for_nan() 20 0.001208 0.001208 0.001208 0.14% MultiFab::contains_nan() 20 0.001197 0.001197 0.001197 0.14% MLPoisson::prepareForSolve() 11 0.001177 0.001177 0.001177 0.14% MLCellLinOp::prepareForSolve() 11 0.001168 0.001168 0.001168 0.14% Castro::post_regrid() 1 0.001099 0.001099 0.001099 0.13% MLMG::computeMLResidual() 11 0.001018 0.001018 0.001018 0.12% Castro::computeInitialDt() 2 0.0009061 0.0009061 0.0009061 0.11% Gravity::update_max_rhs() 11 0.0008609 0.0008609 0.0008609 0.10% FabArrayBase::getFB() 4023 0.0006864 0.0006864 0.0006864 0.08% FabArrayBase::CPC::define() 454 0.0006799 0.0006799 0.0006799 0.08% Castro::finalize_advance() 10 0.0006072 0.0006072 0.0006072 0.07% Amr::InitAmr() 1 0.0005156 0.0005156 0.0005156 0.06% Gravity::swapTimeLevels() 10 0.0004324 0.0004324 0.0004324 0.05% Castro::Castro() 1 0.0004294 0.0004294 0.0004294 0.05% MLLinOp::define() 11 0.0004235 0.0004235 0.0004235 0.05% MLLinOp::defineGrids() 11 0.0003967 0.0003967 0.0003967 0.05% MultiFab::Copy() 11 0.0003289 0.0003289 0.0003289 0.04% MLMG::MLResNormInf() 11 0.00028 0.00028 0.00028 0.03% MultiFab::max() 11 0.0002555 0.0002555 0.0002555 0.03% MLMG::MLRhsNormInf() 11 0.0002167 0.0002167 0.0002167 0.03% Castro::buildMetrics() 1 0.0001586 0.0001586 0.0001586 0.02% FabArrayBase::FB::FB() 56 8.405e-05 8.405e-05 8.405e-05 0.01% AmrLevel::AmrLevel(dm) 1 5.986e-05 5.986e-05 5.986e-05 0.01% MLLinOp::makeAgglomeratedDMap 11 5.485e-05 5.485e-05 5.485e-05 0.01% StateData::define() 4 4.49e-05 4.49e-05 4.49e-05 0.01% Castro::swap_state_time_levels() 10 4.339e-05 4.339e-05 4.339e-05 0.01% makeSFC 55 3.973e-05 3.973e-05 3.973e-05 0.00% Castro::finalize_do_advance() 10 3.34e-05 3.34e-05 3.34e-05 0.00% Castro::enforce_consistent_e() 1 3.298e-05 3.298e-05 3.298e-05 0.00% Castro::initMFs() 1 2.854e-05 2.854e-05 2.854e-05 0.00% Amr::writeSmallPlotFile() 1 2.555e-05 2.555e-05 2.555e-05 0.00% DistributionMapping::Distribute() 56 1.506e-05 1.506e-05 1.506e-05 0.00% Amr::initSubcycle() 1 9.505e-06 9.505e-06 9.505e-06 0.00% AmrLevel::checkPointPost() 3 4.798e-06 4.798e-06 4.798e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.641e-06 4.641e-06 4.641e-06 0.00% Gravity::set_mass_offset() 11 4.365e-06 4.365e-06 4.365e-06 0.00% Castro::retry_advance_ctu() 10 3.795e-06 3.795e-06 3.795e-06 0.00% Castro::create_source_corrector() 10 3.548e-06 3.548e-06 3.548e-06 0.00% Castro::FluxRegCrseInit 10 2.621e-06 2.621e-06 2.621e-06 0.00% AmrLevel::checkPointPre() 3 2.135e-06 2.135e-06 2.135e-06 0.00% Castro::FluxRegFineAdd() 10 2.1e-06 2.1e-06 2.1e-06 0.00% MLLinOp::makeSubCommunicator() 11 2.027e-06 2.027e-06 2.027e-06 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2464 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-2-g1a5b308c8af7) finalized Initializing CUDA... CUDA initialized with 1 GPU AMReX (22.12-2-g1a5b308c8af7) initialized Starting run at 09:54:20 UTC on 2022-12-07. Successfully read inputs file ... Castro git describe: 22.12-3-gdebbe0b78 AMReX git describe: 22.12-2-g1a5b308c8 Microphysics git describe: 22.12-11-ge920a815 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.466167478 Restart time = 0.049583707 seconds. [Level 0 step 6] ADVANCE with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.052815608 [STEP 6] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.053770349 [STEP 7] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.058676762 [STEP 8] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.060446049 [STEP 9] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.076514138 [STEP 10] FAB kilobyte spread across MPI nodes: [319638 ... 319638] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.032096604 seconds Ending run at 09:54:20 UTC on 2022-12-07. Run time = 0.385010454 Run time without initialization = 0.334766635 Average number of zones advanced per microsecond: 3.915 Average number of zones advanced per microsecond per rank: 3.915 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9489481728 TinyProfiler total time across processes [min...avg...max]: 0.385 ... 0.385 ... 0.385 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.1055 0.1055 0.1055 27.41% VisMF::Read() 3 0.04145 0.04145 0.04145 10.77% MLCellLinOp::applyBC() 1946 0.03467 0.03467 0.03467 9.00% VisMF::Write(FabArray) 1 0.03046 0.03046 0.03046 7.91% MLPoisson::Fsmooth() 1440 0.02721 0.02721 0.02721 7.07% StateData::FillBoundary(geom) 160 0.01158 0.01158 0.01158 3.01% Castro::normalize_species() 30 0.0103 0.0103 0.0103 2.68% amrex::Dot() 484 0.009528 0.009528 0.009528 2.47% Castro::computeTemp() 30 0.007606 0.007606 0.007606 1.98% amrex::Copy() 463 0.007363 0.007363 0.007363 1.91% FabArray::setVal() 537 0.006709 0.006709 0.006709 1.74% MLCellLinOp::defineAuxData() 6 0.00627 0.00627 0.00627 1.63% FillBoundary_nowait() 1766 0.006233 0.006233 0.006233 1.62% FabArray::LinComb() 690 0.006052 0.006052 0.006052 1.57% FabArray::ParallelCopy_nowait() 380 0.005966 0.005966 0.005966 1.55% StateDataPhysBCFunct::() 20 0.005862 0.005862 0.005862 1.52% FabArray::norminf() 278 0.005533 0.005533 0.005533 1.44% MLPoisson::Fapply() 500 0.005022 0.005022 0.005022 1.30% Gravity::fill_multipole_BCs() 6 0.004302 0.004302 0.004302 1.12% Amr::restart() 1 0.003794 0.003794 0.003794 0.99% MLMG::addInterpCorrection() 180 0.003343 0.003343 0.003343 0.87% amrex::average_down 180 0.002997 0.002997 0.002997 0.78% FabArray::Xpay() 258 0.002843 0.002843 0.002843 0.74% Castro::enforce_min_density() 30 0.002625 0.002625 0.002625 0.68% Castro::estTimeStep() 10 0.002405 0.002405 0.002405 0.62% BndryData::define() 6 0.002103 0.002103 0.002103 0.55% Castro::construct_new_gravity_source() 5 0.001754 0.001754 0.001754 0.46% Castro::reset_internal_energy(MultiFab) 30 0.001666 0.001666 0.001666 0.43% Amr::writePlotFile() 1 0.001462 0.001462 0.001462 0.38% Castro::construct_old_gravity_source() 5 0.001418 0.001418 0.001418 0.37% MLCGSolver::bicgstab 36 0.001406 0.001406 0.001406 0.37% Castro::do_advance_ctu() 5 0.001177 0.001177 0.001177 0.31% Castro::reset_internal_energy(Fab) 240 0.001126 0.001126 0.001126 0.29% amrex::Add() 72 0.001078 0.001078 0.001078 0.28% MLMG::ResNormInf() 42 0.0009236 0.0009236 0.0009236 0.24% FabArray::Saxpy() 10 0.0009169 0.0009169 0.0009169 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008796 0.0008796 0.0008796 0.23% MLCellLinOp::setLevelBC() 6 0.0007989 0.0007989 0.0007989 0.21% Gravity::actual_solve_with_mlmg() 6 0.0007753 0.0007753 0.0007753 0.20% Castro::enforce_speed_limit() 30 0.0006727 0.0006727 0.0006727 0.17% FabArray::mult() 22 0.000648 0.000648 0.000648 0.17% MLMG::prepareForSolve() 6 0.0006465 0.0006465 0.0006465 0.17% FabArray::setDomainBndry() 20 0.0006321 0.0006321 0.0006321 0.16% MLCellLinOp::prepareForSolve() 6 0.0006312 0.0006312 0.0006312 0.16% MultiFab::contains_nan() 10 0.0005932 0.0005932 0.0005932 0.15% MLCellLinOp::compGrad() 6 0.0004883 0.0004883 0.0004883 0.13% Amr::InitAmr() 1 0.0004418 0.0004418 0.0004418 0.11% Gravity::get_old_grav_vector() 5 0.0004392 0.0004392 0.0004392 0.11% MLCellLinOp::smooth() 720 0.0004376 0.0004376 0.0004376 0.11% FabArrayBase::CPC::define() 244 0.0004186 0.0004186 0.0004186 0.11% Gravity::get_new_grav_vector() 5 0.000368 0.000368 0.000368 0.10% FabArrayBase::getCPC() 632 0.000354 0.000354 0.000354 0.09% FabArray::FillBoundary() 1766 0.0003531 0.0003531 0.0003531 0.09% main() 1 0.0002769 0.0002769 0.0002769 0.07% FabArrayBase::getFB() 1766 0.0002562 0.0002562 0.0002562 0.07% Amr::coarseTimeStep() 5 0.000225 0.000225 0.000225 0.06% MLCellLinOp::apply() 500 0.0001993 0.0001993 0.0001993 0.05% Castro::subcycle_advance_ctu() 5 0.0001973 0.0001973 0.0001973 0.05% CGSolver::sxay() 690 0.0001633 0.0001633 0.0001633 0.04% MLMG::mgVcycle() 36 0.0001618 0.0001618 0.0001618 0.04% MLCellLinOp::defineBC() 6 0.0001475 0.0001475 0.0001475 0.04% FillPatchIterator::Initialize 20 0.0001437 0.0001437 0.0001437 0.04% MLLinOp::defineGrids() 6 0.000136 0.000136 0.000136 0.04% MultiFab::max() 6 0.0001344 0.0001344 0.0001344 0.03% MLCGSolver::ParallelAllReduce 659 0.0001317 0.0001317 0.0001317 0.03% FabArray::ParallelCopy() 380 0.0001175 0.0001175 0.0001175 0.03% Castro::post_timestep() 5 0.0001167 0.0001167 0.0001167 0.03% MLMG::MLRhsNormInf() 6 0.0001143 0.0001143 0.0001143 0.03% Castro::construct_new_source() 25 0.0001129 0.0001129 0.0001129 0.03% Castro::advance() 5 0.0001057 0.0001057 0.0001057 0.03% MLCellLinOp::correctionResidual() 216 9.875e-05 9.875e-05 9.875e-05 0.03% Castro::finalize_advance() 5 9.81e-05 9.81e-05 9.81e-05 0.03% StateData::restartDoit() 4 8.8e-05 8.8e-05 8.8e-05 0.02% AmrLevel::restart() 1 8.579e-05 8.579e-05 8.579e-05 0.02% Amr::timeStep() 5 8.292e-05 8.292e-05 8.292e-05 0.02% Gravity::update_max_rhs() 6 7.922e-05 7.922e-05 7.922e-05 0.02% Castro::create_source_corrector() 5 6.448e-05 6.448e-05 6.448e-05 0.02% Gravity::solve_for_phi() 5 6.403e-05 6.403e-05 6.403e-05 0.02% MLMG:computeResOfCorrection() 180 6.393e-05 6.393e-05 6.393e-05 0.02% FabArrayBase::FB::FB() 26 5.919e-05 5.919e-05 5.919e-05 0.02% Castro::construct_new_gravity() 5 5.89e-05 5.89e-05 5.89e-05 0.02% Castro::construct_old_source() 25 5.215e-05 5.215e-05 5.215e-05 0.01% Castro::initialize_do_advance() 5 4.863e-05 4.863e-05 4.863e-05 0.01% MLMG::mgVcycle_down::0 36 4.612e-05 4.612e-05 4.612e-05 0.01% Castro::post_restart() 1 4.224e-05 4.224e-05 4.224e-05 0.01% MLMG::mgVcycle_down::2 36 4.144e-05 4.144e-05 4.144e-05 0.01% MLMG::mgVcycle_down::1 36 4.111e-05 4.111e-05 4.111e-05 0.01% Castro::clean_state() 30 4.088e-05 4.088e-05 4.088e-05 0.01% Castro::expand_state() 5 4.047e-05 4.047e-05 4.047e-05 0.01% Castro::initialize_advance() 5 3.969e-05 3.969e-05 3.969e-05 0.01% Castro::computeNewDt() 5 3.599e-05 3.599e-05 3.599e-05 0.01% MLMG::mgVcycle_down::4 36 3.526e-05 3.526e-05 3.526e-05 0.01% MLMG::mgVcycle_down::3 36 3.432e-05 3.432e-05 3.432e-05 0.01% Castro::buildMetrics() 1 3.383e-05 3.383e-05 3.383e-05 0.01% MLMG::mgVcycle_up::4 36 3.194e-05 3.194e-05 3.194e-05 0.01% MLMG::actualBottomSolve() 36 3.191e-05 3.191e-05 3.191e-05 0.01% MLMG::solve() 6 3.173e-05 3.173e-05 3.173e-05 0.01% Castro::initMFs() 1 3.032e-05 3.032e-05 3.032e-05 0.01% Castro::swap_state_time_levels() 5 2.959e-05 2.959e-05 2.959e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 2.908e-05 2.908e-05 2.908e-05 0.01% MLPoisson::define() 6 2.713e-05 2.713e-05 2.713e-05 0.01% MLMG::mgVcycle_up::0 36 2.685e-05 2.685e-05 2.685e-05 0.01% MLMG::oneIter() 36 2.578e-05 2.578e-05 2.578e-05 0.01% Amr::writeSmallPlotFile() 1 2.562e-05 2.562e-05 2.562e-05 0.01% MLMG::mgVcycle_up::3 36 2.391e-05 2.391e-05 2.391e-05 0.01% MLMG::mgVcycle_up::2 36 2.326e-05 2.326e-05 2.326e-05 0.01% MLLinOp::define() 6 2.289e-05 2.289e-05 2.289e-05 0.01% MLMG::mgVcycle_up::1 36 2.207e-05 2.207e-05 2.207e-05 0.01% Castro::construct_old_gravity() 5 2.08e-05 2.08e-05 2.08e-05 0.01% MLCellLinOp::solutionResidual() 42 1.978e-05 1.978e-05 1.978e-05 0.01% Castro::finalize_do_advance() 5 1.881e-05 1.881e-05 1.881e-05 0.00% MLMG::computeResidual() 36 1.698e-05 1.698e-05 1.698e-05 0.00% Castro::check_for_nan() 10 1.684e-05 1.684e-05 1.684e-05 0.00% MLMG::mgVcycle_bottom 36 1.584e-05 1.584e-05 1.584e-05 0.00% FillPatchSingleLevel 20 1.504e-05 1.504e-05 1.504e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.341e-05 1.341e-05 1.341e-05 0.00% makeSFC 30 1.334e-05 1.334e-05 1.334e-05 0.00% MultiFab::Add() 36 1.208e-05 1.208e-05 1.208e-05 0.00% DistributionMapping::Distribute() 31 9.503e-06 9.503e-06 9.503e-06 0.00% Castro::do_new_sources() 5 9.218e-06 9.218e-06 9.218e-06 0.00% Gravity::actual_multilevel_solve() 1 9.125e-06 9.125e-06 9.125e-06 0.00% Amr::initSubcycle() 1 9.063e-06 9.063e-06 9.063e-06 0.00% Castro::do_old_sources() 5 8.3e-06 8.3e-06 8.3e-06 0.00% MLLinOp::makeAgglomeratedDMap 6 7.194e-06 7.194e-06 7.194e-06 0.00% Castro::apply_source_to_state() 10 5.369e-06 5.369e-06 5.369e-06 0.00% MLPoisson::prepareForSolve() 6 4.68e-06 4.68e-06 4.68e-06 0.00% Gravity::swapTimeLevels() 5 4.499e-06 4.499e-06 4.499e-06 0.00% MLMG::computeMLResidual() 6 4.141e-06 4.141e-06 4.141e-06 0.00% MLMG::getGradSolution() 6 3.42e-06 3.42e-06 3.42e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 3.358e-06 3.358e-06 3.358e-06 0.00% Gravity::set_mass_offset() 6 3.196e-06 3.196e-06 3.196e-06 0.00% MultiFab::Copy() 6 2.902e-06 2.902e-06 2.902e-06 0.00% MLMG::MLResNormInf() 6 2.179e-06 2.179e-06 2.179e-06 0.00% Castro::FluxRegCrseInit 5 1.725e-06 1.725e-06 1.725e-06 0.00% Castro::retry_advance_ctu() 5 1.718e-06 1.718e-06 1.718e-06 0.00% Castro::FluxRegFineAdd() 5 1.387e-06 1.387e-06 1.387e-06 0.00% Amr::init() 1 1.097e-06 1.097e-06 1.097e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.083e-06 1.083e-06 1.083e-06 0.00% AmrLevel::AmrLevel() 1 7.4e-07 7.4e-07 7.4e-07 0.00% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.385 0.385 0.385 100.00% Amr::coarseTimeStep() 5 0.3024 0.3024 0.3024 78.55% Amr::timeStep() 5 0.3009 0.3009 0.3009 78.14% Castro::advance() 5 0.2962 0.2962 0.2962 76.91% Castro::subcycle_advance_ctu() 5 0.2907 0.2907 0.2907 75.50% Castro::do_advance_ctu() 5 0.2905 0.2905 0.2905 75.45% Castro::construct_new_gravity() 5 0.1436 0.1436 0.1436 37.29% Gravity::solve_phi_with_mlmg() 6 0.1393 0.1393 0.1393 36.19% Gravity::solve_for_phi() 5 0.1358 0.1358 0.1358 35.26% Gravity::actual_solve_with_mlmg() 6 0.1348 0.1348 0.1348 35.01% MLMG::solve() 6 0.1225 0.1225 0.1225 31.80% MLMG::oneIter() 36 0.1153 0.1153 0.1153 29.94% MLMG::mgVcycle() 36 0.1145 0.1145 0.1145 29.74% Castro::construct_ctu_hydro_source() 5 0.1055 0.1055 0.1055 27.41% MLCellLinOp::smooth() 720 0.05838 0.05838 0.05838 15.16% Amr::init() 1 0.04965 0.04965 0.04965 12.89% Amr::restart() 1 0.04965 0.04965 0.04965 12.89% AmrLevel::restart() 1 0.04168 0.04168 0.04168 10.82% StateData::restartDoit() 4 0.04159 0.04159 0.04159 10.80% MLCellLinOp::applyBC() 1946 0.04157 0.04157 0.04157 10.80% VisMF::Read() 3 0.04145 0.04145 0.04145 10.77% MLMG::mgVcycle_bottom 36 0.03516 0.03516 0.03516 9.13% MLMG::actualBottomSolve() 36 0.03514 0.03514 0.03514 9.13% MLCGSolver::bicgstab 36 0.03479 0.03479 0.03479 9.04% Amr::writePlotFile() 1 0.03218 0.03218 0.03218 8.36% VisMF::Write(FabArray) 1 0.03046 0.03046 0.03046 7.91% MLPoisson::Fsmooth() 1440 0.02721 0.02721 0.02721 7.07% Castro::clean_state() 30 0.02404 0.02404 0.02404 6.24% FillPatchIterator::Initialize 20 0.02022 0.02022 0.02022 5.25% FillPatchSingleLevel 20 0.01945 0.01945 0.01945 5.05% StateDataPhysBCFunct::() 20 0.01744 0.01744 0.01744 4.53% MLCellLinOp::apply() 500 0.0158 0.0158 0.0158 4.10% MLMG::mgVcycle_down::0 36 0.01529 0.01529 0.01529 3.97% MLMG::mgVcycle_up::0 36 0.01312 0.01312 0.01312 3.41% StateData::FillBoundary(geom) 160 0.01158 0.01158 0.01158 3.01% Castro::computeTemp() 30 0.0104 0.0104 0.0104 2.70% Castro::normalize_species() 30 0.0103 0.0103 0.0103 2.68% MLPoisson::define() 6 0.01001 0.01001 0.01001 2.60% Castro::initialize_do_advance() 5 0.0099 0.0099 0.0099 2.57% amrex::Dot() 484 0.009528 0.009528 0.009528 2.47% MLCellLinOp::correctionResidual() 216 0.009166 0.009166 0.009166 2.38% Castro::do_new_sources() 5 0.008353 0.008353 0.008353 2.17% MLMG:computeResOfCorrection() 180 0.007914 0.007914 0.007914 2.06% Gravity::get_new_grav_vector() 5 0.007643 0.007643 0.007643 1.99% MLMG::mgVcycle_down::1 36 0.007616 0.007616 0.007616 1.98% Castro::construct_old_gravity() 5 0.007585 0.007585 0.007585 1.97% Gravity::get_old_grav_vector() 5 0.007565 0.007565 0.007565 1.96% MLMG::mgVcycle_down::2 36 0.007375 0.007375 0.007375 1.92% amrex::Copy() 463 0.007363 0.007363 0.007363 1.91% MLMG::mgVcycle_down::3 36 0.007031 0.007031 0.007031 1.83% MLCellLinOp::defineAuxData() 6 0.007008 0.007008 0.007008 1.82% FabArray::FillBoundary() 1766 0.006901 0.006901 0.006901 1.79% MLMG::mgVcycle_down::4 36 0.006722 0.006722 0.006722 1.75% FabArray::setVal() 537 0.006709 0.006709 0.006709 1.74% FillBoundary_nowait() 1766 0.006548 0.006548 0.006548 1.70% FabArray::ParallelCopy() 380 0.006463 0.006463 0.006463 1.68% FabArray::ParallelCopy_nowait() 380 0.006345 0.006345 0.006345 1.65% CGSolver::sxay() 690 0.006215 0.006215 0.006215 1.61% FabArray::LinComb() 690 0.006052 0.006052 0.006052 1.57% Castro::do_old_sources() 5 0.005867 0.005867 0.005867 1.52% Castro::expand_state() 5 0.005793 0.005793 0.005793 1.50% MLCGSolver::ParallelAllReduce 659 0.005716 0.005716 0.005716 1.48% MLMG::mgVcycle_up::2 36 0.005675 0.005675 0.005675 1.47% MLMG::mgVcycle_up::1 36 0.005625 0.005625 0.005625 1.46% MLMG::addInterpCorrection() 180 0.005573 0.005573 0.005573 1.45% FabArray::norminf() 278 0.005533 0.005533 0.005533 1.44% MLMG::mgVcycle_up::3 36 0.005386 0.005386 0.005386 1.40% MLMG::mgVcycle_up::4 36 0.005363 0.005363 0.005363 1.39% amrex::average_down 180 0.005237 0.005237 0.005237 1.36% MLPoisson::Fapply() 500 0.005022 0.005022 0.005022 1.30% Castro::initialize_advance() 5 0.004966 0.004966 0.004966 1.29% Castro::post_timestep() 5 0.00463 0.00463 0.00463 1.20% Gravity::fill_multipole_BCs() 6 0.004423 0.004423 0.004423 1.15% Castro::post_restart() 1 0.003991 0.003991 0.003991 1.04% Gravity::multilevel_solve_for_new_phi() 1 0.00385 0.00385 0.00385 1.00% Gravity::actual_multilevel_solve() 1 0.003821 0.003821 0.003821 0.99% MLCellLinOp::solutionResidual() 42 0.003234 0.003234 0.003234 0.84% MLMG::prepareForSolve() 6 0.002853 0.002853 0.002853 0.74% FabArray::Xpay() 258 0.002843 0.002843 0.002843 0.74% Castro::reset_internal_energy(MultiFab) 30 0.002792 0.002792 0.002792 0.73% MLCellLinOp::defineBC() 6 0.002786 0.002786 0.002786 0.72% MLMG::computeResidual() 36 0.002691 0.002691 0.002691 0.70% BndryData::define() 6 0.002638 0.002638 0.002638 0.69% Castro::enforce_min_density() 30 0.002625 0.002625 0.002625 0.68% Castro::estTimeStep() 10 0.002405 0.002405 0.002405 0.62% Castro::construct_new_source() 25 0.001867 0.001867 0.001867 0.48% Castro::construct_new_gravity_source() 5 0.001754 0.001754 0.001754 0.46% Castro::construct_old_source() 25 0.00147 0.00147 0.00147 0.38% Castro::construct_old_gravity_source() 5 0.001418 0.001418 0.001418 0.37% Castro::computeNewDt() 5 0.001348 0.001348 0.001348 0.35% Castro::reset_internal_energy(Fab) 240 0.001126 0.001126 0.001126 0.29% amrex::Add() 72 0.001078 0.001078 0.001078 0.28% MLMG::ResNormInf() 42 0.0009236 0.0009236 0.0009236 0.24% Castro::apply_source_to_state() 10 0.0009223 0.0009223 0.0009223 0.24% FabArray::Saxpy() 10 0.0009169 0.0009169 0.0009169 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.0008796 0.0008796 0.0008796 0.23% MLCellLinOp::setLevelBC() 6 0.0007989 0.0007989 0.0007989 0.21% FabArrayBase::getCPC() 632 0.0007726 0.0007726 0.0007726 0.20% MLMG::getGradSolution() 6 0.0007632 0.0007632 0.0007632 0.20% MLCellLinOp::compGrad() 6 0.0007598 0.0007598 0.0007598 0.20% MultiFab::Add() 36 0.0007263 0.0007263 0.0007263 0.19% Castro::enforce_speed_limit() 30 0.0006727 0.0006727 0.0006727 0.17% FabArray::mult() 22 0.000648 0.000648 0.000648 0.17% MLPoisson::prepareForSolve() 6 0.0006359 0.0006359 0.0006359 0.17% FabArray::setDomainBndry() 20 0.0006321 0.0006321 0.0006321 0.16% MLCellLinOp::prepareForSolve() 6 0.0006312 0.0006312 0.0006312 0.16% Castro::check_for_nan() 10 0.00061 0.00061 0.00061 0.16% MultiFab::contains_nan() 10 0.0005932 0.0005932 0.0005932 0.15% MLMG::computeMLResidual() 6 0.0005648 0.0005648 0.0005648 0.15% Gravity::update_max_rhs() 6 0.0004738 0.0004738 0.0004738 0.12% Amr::InitAmr() 1 0.0004509 0.0004509 0.0004509 0.12% FabArrayBase::CPC::define() 244 0.0004186 0.0004186 0.0004186 0.11% Castro::finalize_advance() 5 0.0003609 0.0003609 0.0003609 0.09% FabArrayBase::getFB() 1766 0.0003154 0.0003154 0.0003154 0.08% Gravity::swapTimeLevels() 5 0.0002242 0.0002242 0.0002242 0.06% MLLinOp::define() 6 0.0001886 0.0001886 0.0001886 0.05% MultiFab::Copy() 6 0.0001747 0.0001747 0.0001747 0.05% MLLinOp::defineGrids() 6 0.0001657 0.0001657 0.0001657 0.04% Castro::buildMetrics() 1 0.0001534 0.0001534 0.0001534 0.04% MLMG::MLResNormInf() 6 0.0001517 0.0001517 0.0001517 0.04% MultiFab::max() 6 0.0001344 0.0001344 0.0001344 0.03% MLMG::MLRhsNormInf() 6 0.0001143 0.0001143 0.0001143 0.03% Castro::create_source_corrector() 5 6.448e-05 6.448e-05 6.448e-05 0.02% FabArrayBase::FB::FB() 26 5.919e-05 5.919e-05 5.919e-05 0.02% Castro::initMFs() 1 3.032e-05 3.032e-05 3.032e-05 0.01% Castro::swap_state_time_levels() 5 2.959e-05 2.959e-05 2.959e-05 0.01% MLLinOp::makeAgglomeratedDMap 6 2.862e-05 2.862e-05 2.862e-05 0.01% Amr::writeSmallPlotFile() 1 2.562e-05 2.562e-05 2.562e-05 0.01% makeSFC 30 2.143e-05 2.143e-05 2.143e-05 0.01% Castro::finalize_do_advance() 5 1.881e-05 1.881e-05 1.881e-05 0.00% DistributionMapping::Distribute() 31 9.503e-06 9.503e-06 9.503e-06 0.00% Amr::initSubcycle() 1 9.063e-06 9.063e-06 9.063e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 4.774e-06 4.774e-06 4.774e-06 0.00% Gravity::set_mass_offset() 6 3.196e-06 3.196e-06 3.196e-06 0.00% Castro::FluxRegCrseInit 5 1.725e-06 1.725e-06 1.725e-06 0.00% Castro::retry_advance_ctu() 5 1.718e-06 1.718e-06 1.718e-06 0.00% Castro::FluxRegFineAdd() 5 1.387e-06 1.387e-06 1.387e-06 0.00% MLLinOp::makeSubCommunicator() 6 1.083e-06 1.083e-06 1.083e-06 0.00% AmrLevel::AmrLevel() 1 7.4e-07 7.4e-07 7.4e-07 0.00% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Total GPU global memory (MB): 12066 Free GPU global memory (MB): 2464 [The Arena] space allocated (MB): 9049 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (22.12-2-g1a5b308c8af7) finalized