Initializing AMReX (24.07-13-g361f4ab839f2)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.07-13-g361f4ab839f2) initialized Starting run at 09:24:44 UTC on 2024-07-09. Successfully read inputs file ... Castro git describe: 24.07-3-g79841a784 AMReX git describe: 24.07-13-g361f4ab83 Microphysics git describe: 24.07-9-ga3bc2465 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 Successfully read inputs file ... INITIAL GRIDS Level 0 8 grids 262144 cells 100 % of domain smallest grid: 32 x 32 x 32 biggest grid: 32 x 32 x 32 CHECKPOINT: file = dustcollapse-restart_chk00000 checkPoint() time = 0.050125874 secs. PLOTFILE: file = dustcollapse-restart_plt00000 Write plotfile time = 0.027088339 seconds [Level 0 step 1] ADVANCE at time 0 with dt = 4.541742215e-05 [Level 0 step 1] Advanced 262144 cells [STEP 1] Coarse TimeStep time: 0.059376178 [STEP 1] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 1 TIME = 4.541742215e-05 DT = 4.541742215e-05 [Level 0 step 2] ADVANCE at time 4.541742215e-05 with dt = 4.768829326e-05 [Level 0 step 2] Advanced 262144 cells [STEP 2] Coarse TimeStep time: 0.051154906 [STEP 2] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 2 TIME = 9.31057154e-05 DT = 4.768829326e-05 [Level 0 step 3] ADVANCE at time 9.31057154e-05 with dt = 5.007270792e-05 [Level 0 step 3] Advanced 262144 cells [STEP 3] Coarse TimeStep time: 0.051179869 [STEP 3] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 3 TIME = 0.0001431784233 DT = 5.007270792e-05 [Level 0 step 4] ADVANCE at time 0.0001431784233 with dt = 5.257634331e-05 [Level 0 step 4] Advanced 262144 cells [STEP 4] Coarse TimeStep time: 0.056914332 [STEP 4] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 4 TIME = 0.0001957547666 DT = 5.257634331e-05 [Level 0 step 5] ADVANCE at time 0.0001957547666 with dt = 5.520516048e-05 [Level 0 step 5] Advanced 262144 cells [STEP 5] Coarse TimeStep time: 0.052764369 [STEP 5] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 5 TIME = 0.0002509599271 DT = 5.520516048e-05 CHECKPOINT: file = dustcollapse-restart_chk00005 checkPoint() time = 0.048657506 secs. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.060320034 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.063781007 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.072581825 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.069330365 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.061136908 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 CHECKPOINT: file = dustcollapse-restart_chk00010 checkPoint() time = 0.047169688 secs. PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.026916018 seconds Ending run at 09:24:45 UTC on 2024-07-09. Run time = 0.854351348 Run time without initialization = 0.721953445 Average number of zones advanced per microsecond: 3.631 Average number of zones advanced per microsecond per rank: 3.631 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.8544 ... 0.8544 ... 0.8544 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 10 0.2029 0.2029 0.2029 23.75% VisMF::Write(FabArray) 11 0.1921 0.1921 0.1921 22.49% MLCellLinOp::applyBC() 4351 0.08418 0.08418 0.08418 9.85% MLPoisson::Fsmooth() 3280 0.03528 0.03528 0.03528 4.13% FillBoundary_nowait() 3941 0.0338 0.0338 0.0338 3.96% StateData::FillBoundary(geom) 328 0.02749 0.02749 0.02749 3.22% amrex::Dot() 1114 0.02249 0.02249 0.02249 2.63% FabArray::norminf() 1061 0.02091 0.02091 0.02091 2.45% Castro::reset_internal_energy(MultiFab) 63 0.01967 0.01967 0.01967 2.30% StateDataPhysBCFunct::() 41 0.01862 0.01862 0.01862 2.18% Castro::normalize_species() 62 0.01508 0.01508 0.01508 1.77% FabArray::ParallelCopy_nowait() 861 0.01411 0.01411 0.01411 1.65% FabArray::setVal() 1062 0.01395 0.01395 0.01395 1.63% FabArray::Saxpy() 1370 0.0137 0.0137 0.0137 1.60% amrex::Copy() 472 0.01134 0.01134 0.01134 1.33% MLPoisson::Fapply() 1060 0.01083 0.01083 0.01083 1.27% MLCellLinOp::defineAuxData() 11 0.01054 0.01054 0.01054 1.23% Gravity::fill_multipole_BCs() 11 0.01001 0.01001 0.01001 1.17% Castro::enforce_min_density() 62 0.009827 0.009827 0.009827 1.15% FabArray::Xpay() 739 0.008238 0.008238 0.008238 0.96% MLMG::addInterpCorrection() 410 0.007259 0.007259 0.007259 0.85% amrex::average_down 410 0.006506 0.006506 0.006506 0.76% Amr::checkPoint() 3 0.005599 0.005599 0.005599 0.66% Castro::estTimeStep() 21 0.005112 0.005112 0.005112 0.60% BndryData::define() 11 0.004121 0.004121 0.004121 0.48% amrex::Add() 82 0.003818 0.003818 0.003818 0.45% Castro::construct_new_gravity_source() 10 0.003716 0.003716 0.003716 0.43% Castro::computeTemp() 63 0.003191 0.003191 0.003191 0.37% Castro::construct_old_gravity_source() 10 0.002959 0.002959 0.002959 0.35% Castro::enforce_speed_limit() 62 0.00293 0.00293 0.00293 0.34% Castro::reset_internal_energy(Fab) 504 0.001966 0.001966 0.001966 0.23% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001827 0.001827 0.001827 0.21% Amr::writePlotFile() 2 0.001818 0.001818 0.001818 0.21% MLCGSolver::bicgstab 82 0.001679 0.001679 0.001679 0.20% MLCellLinOp::setLevelBC() 11 0.001636 0.001636 0.001636 0.19% Castro::initData() 1 0.001585 0.001585 0.001585 0.19% Gravity::actual_solve_with_mlmg() 11 0.001559 0.001559 0.001559 0.18% check_for_negative_density() 10 0.001509 0.001509 0.001509 0.18% FabArray::mult() 43 0.001421 0.001421 0.001421 0.17% FabArray::setDomainBndry() 41 0.001417 0.001417 0.001417 0.17% MLCellLinOp::prepareForSolve() 11 0.001359 0.001359 0.001359 0.16% MultiFab::contains_nan() 20 0.001284 0.001284 0.001284 0.15% MLCellLinOp::smooth() 1640 0.001233 0.001233 0.001233 0.14% MLCellLinOp::compGrad() 11 0.001123 0.001123 0.001123 0.13% MLMG::prepareForSolve() 11 0.0009877 0.0009877 0.0009877 0.12% FabArrayBase::getCPC() 1323 0.0007821 0.0007821 0.0007821 0.09% FabArray::FillBoundary() 3941 0.000751 0.000751 0.000751 0.09% Gravity::get_new_grav_vector() 11 0.0007138 0.0007138 0.0007138 0.08% Gravity::get_old_grav_vector() 10 0.0005386 0.0005386 0.0005386 0.06% MLCellLinOp::apply() 1060 0.0004432 0.0004432 0.0004432 0.05% AmrLevel::FillPatch() 41 0.0004369 0.0004369 0.0004369 0.05% MLCGSolver::ParallelAllReduce 1832 0.0003292 0.0003292 0.0003292 0.04% Amr::coarseTimeStep() 10 0.000321 0.000321 0.000321 0.04% main() 1 0.0003101 0.0003101 0.0003101 0.04% Castro::subcycle_advance_ctu() 10 0.0002661 0.0002661 0.0002661 0.03% MLCellLinOp::defineBC() 11 0.0002585 0.0002585 0.0002585 0.03% FabArray::ParallelCopy() 861 0.0002411 0.0002411 0.0002411 0.03% FillPatchIterator::Initialize 41 0.0002176 0.0002176 0.0002176 0.03% MLMG::mgVcycle() 82 0.0002093 0.0002093 0.0002093 0.02% MLCellLinOp::correctionResidual() 410 0.0001803 0.0001803 0.0001803 0.02% Castro::advance() 10 0.0001639 0.0001639 0.0001639 0.02% Amr::timeStep() 10 0.0001593 0.0001593 0.0001593 0.02% Gravity::update_max_rhs() 11 0.000137 0.000137 0.000137 0.02% StateData::checkPoint() 12 0.0001189 0.0001189 0.0001189 0.01% MLMG:computeResOfCorrection() 410 0.000118 0.000118 0.000118 0.01% Gravity::solve_for_phi() 10 0.0001081 0.0001081 0.0001081 0.01% MLMG::actualBottomSolve() 82 8.762e-05 8.762e-05 8.762e-05 0.01% MLMG::mgVcycle_down::0 82 8.608e-05 8.608e-05 8.608e-05 0.01% MLMG::solve() 11 7.931e-05 7.931e-05 7.931e-05 0.01% MLMG::mgVcycle_down::1 82 7.806e-05 7.806e-05 7.806e-05 0.01% Castro::initialize_advance() 10 7.666e-05 7.666e-05 7.666e-05 0.01% MLMG::mgVcycle_down::2 82 7.552e-05 7.552e-05 7.552e-05 0.01% MLMG::mgVcycle_down::4 82 7.253e-05 7.253e-05 7.253e-05 0.01% Castro::do_advance_ctu() 10 7.177e-05 7.177e-05 7.177e-05 0.01% Castro::clean_state() 62 7.174e-05 7.174e-05 7.174e-05 0.01% MLMG::mgVcycle_down::3 82 7.033e-05 7.033e-05 7.033e-05 0.01% FillPatchIterator::FillFromLevel0() 41 6.344e-05 6.344e-05 6.344e-05 0.01% AmrLevel::checkPoint() 3 6.24e-05 6.24e-05 6.24e-05 0.01% Castro::initialize_do_advance() 10 6.001e-05 6.001e-05 6.001e-05 0.01% MLMG::oneIter() 82 5.425e-05 5.425e-05 5.425e-05 0.01% MLMG::mgVcycle_up::4 82 5.416e-05 5.416e-05 5.416e-05 0.01% MLMG::mgVcycle_up::0 82 5.243e-05 5.243e-05 5.243e-05 0.01% Castro::finalize_do_advance() 10 4.957e-05 4.957e-05 4.957e-05 0.01% MLCellLinOp::solutionResidual() 93 4.737e-05 4.737e-05 4.737e-05 0.01% MLMG::mgVcycle_up::3 82 4.737e-05 4.737e-05 4.737e-05 0.01% MLMG::mgVcycle_up::1 82 4.63e-05 4.63e-05 4.63e-05 0.01% MLMG::mgVcycle_up::2 82 4.523e-05 4.523e-05 4.523e-05 0.01% FillPatchSingleLevel 41 4.048e-05 4.048e-05 4.048e-05 0.00% Amr::defBaseLevel() 1 3.84e-05 3.84e-05 3.84e-05 0.00% MLMG::ResNormInf() 93 3.554e-05 3.554e-05 3.554e-05 0.00% MLMG::mgVcycle_bottom 82 3.43e-05 3.43e-05 3.43e-05 0.00% MLMG::computeResidual() 82 3.393e-05 3.393e-05 3.393e-05 0.00% Castro::construct_new_source() 50 2.807e-05 2.807e-05 2.807e-05 0.00% Castro::construct_new_gravity() 10 2.721e-05 2.721e-05 2.721e-05 0.00% MLPoisson::define() 11 2.565e-05 2.565e-05 2.565e-05 0.00% Castro::do_new_sources() 10 2.379e-05 2.379e-05 2.379e-05 0.00% Gravity::multilevel_solve_for_new_phi() 1 2.32e-05 2.32e-05 2.32e-05 0.00% Gravity::solve_phi_with_mlmg() 11 2.24e-05 2.24e-05 2.24e-05 0.00% Castro::apply_source_to_state() 20 2.208e-05 2.208e-05 2.208e-05 0.00% Castro::do_old_sources() 10 2.11e-05 2.11e-05 2.11e-05 0.00% Amr::FinalizeInit() 1 1.857e-05 1.857e-05 1.857e-05 0.00% Castro::construct_old_source() 50 1.786e-05 1.786e-05 1.786e-05 0.00% MLPoisson::prepareForSolve() 11 1.453e-05 1.453e-05 1.453e-05 0.00% Castro::check_for_nan() 20 1.092e-05 1.092e-05 1.092e-05 0.00% MLMG::computeMLResidual() 11 1.082e-05 1.082e-05 1.082e-05 0.00% Castro::construct_old_gravity() 10 1.051e-05 1.051e-05 1.051e-05 0.00% Castro::post_timestep() 10 8.894e-06 8.894e-06 8.894e-06 0.00% Gravity::actual_multilevel_solve() 1 8.653e-06 8.653e-06 8.653e-06 0.00% Amr::InitializeInit() 1 7.212e-06 7.212e-06 7.212e-06 0.00% Castro::computeNewDt() 9 6.995e-06 6.995e-06 6.995e-06 0.00% Castro::expand_state() 10 6.331e-06 6.331e-06 6.331e-06 0.00% MLMG::getGradSolution() 11 6.007e-06 6.007e-06 6.007e-06 0.00% Castro::post_init() 1 4.01e-06 4.01e-06 4.01e-06 0.00% Amr::init() 1 2.773e-06 2.773e-06 2.773e-06 0.00% Amr::initialInit() 1 1.21e-06 1.21e-06 1.21e-06 0.00% Other 4742 0.003 0.003 0.003 0.35% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.8544 0.8544 0.8544 100.00% Amr::coarseTimeStep() 10 0.6948 0.6948 0.6948 81.32% Amr::timeStep() 10 0.5956 0.5956 0.5956 69.71% Castro::advance() 10 0.5868 0.5868 0.5868 68.68% Castro::subcycle_advance_ctu() 10 0.5747 0.5747 0.5747 67.26% Castro::do_advance_ctu() 10 0.5744 0.5744 0.5744 67.23% Gravity::solve_phi_with_mlmg() 11 0.3092 0.3092 0.3092 36.19% Gravity::actual_solve_with_mlmg() 11 0.2987 0.2987 0.2987 34.96% Castro::construct_new_gravity() 10 0.2877 0.2877 0.2877 33.67% MLMG::solve() 11 0.2762 0.2762 0.2762 32.32% Gravity::solve_for_phi() 10 0.2637 0.2637 0.2637 30.86% MLMG::oneIter() 82 0.26 0.26 0.26 30.43% MLMG::mgVcycle() 82 0.2562 0.2562 0.2562 29.98% Castro::construct_ctu_hydro_source() 10 0.2134 0.2134 0.2134 24.98% VisMF::Write(FabArray) 11 0.1921 0.1921 0.1921 22.49% Amr::checkPoint() 3 0.1461 0.1461 0.1461 17.10% AmrLevel::checkPoint() 3 0.1405 0.1405 0.1405 16.45% StateData::checkPoint() 12 0.1405 0.1405 0.1405 16.44% Amr::init() 1 0.1318 0.1318 0.1318 15.42% MLCellLinOp::smooth() 1640 0.1286 0.1286 0.1286 15.05% MLCellLinOp::applyBC() 4351 0.1195 0.1195 0.1195 13.98% MLMG::mgVcycle_bottom 82 0.07631 0.07631 0.07631 8.93% MLMG::actualBottomSolve() 82 0.07627 0.07627 0.07627 8.93% MLCGSolver::bicgstab 82 0.07542 0.07542 0.07542 8.83% AmrLevel::FillPatch() 41 0.0565 0.0565 0.0565 6.61% Amr::initialInit() 1 0.0544 0.0544 0.0544 6.37% Amr::writePlotFile() 2 0.05415 0.05415 0.05415 6.34% FillPatchIterator::Initialize 41 0.05201 0.05201 0.05201 6.09% Castro::clean_state() 62 0.05171 0.05171 0.05171 6.05% FillPatchIterator::FillFromLevel0() 41 0.05038 0.05038 0.05038 5.90% FillPatchSingleLevel 41 0.05031 0.05031 0.05031 5.89% Amr::FinalizeInit() 1 0.04916 0.04916 0.04916 5.75% Castro::post_init() 1 0.04829 0.04829 0.04829 5.65% StateDataPhysBCFunct::() 41 0.0461 0.0461 0.0461 5.40% Gravity::multilevel_solve_for_new_phi() 1 0.04595 0.04595 0.04595 5.38% Gravity::actual_multilevel_solve() 1 0.04593 0.04593 0.04593 5.38% MLCellLinOp::apply() 1060 0.03814 0.03814 0.03814 4.46% MLMG::mgVcycle_down::0 82 0.03687 0.03687 0.03687 4.32% FabArray::FillBoundary() 3941 0.0353 0.0353 0.0353 4.13% MLPoisson::Fsmooth() 3280 0.03528 0.03528 0.03528 4.13% FillBoundary_nowait() 3941 0.03455 0.03455 0.03455 4.04% MLMG::mgVcycle_up::0 82 0.02813 0.02813 0.02813 3.29% StateData::FillBoundary(geom) 328 0.02749 0.02749 0.02749 3.22% Gravity::get_new_grav_vector() 11 0.02591 0.02591 0.02591 3.03% Castro::computeTemp() 63 0.02483 0.02483 0.02483 2.91% amrex::Dot() 1114 0.02249 0.02249 0.02249 2.63% Castro::initialize_do_advance() 10 0.02244 0.02244 0.02244 2.63% Castro::reset_internal_energy(MultiFab) 63 0.02164 0.02164 0.02164 2.53% MLMG:computeResOfCorrection() 410 0.02157 0.02157 0.02157 2.52% MLCellLinOp::correctionResidual() 410 0.02145 0.02145 0.02145 2.51% FabArray::norminf() 1061 0.02091 0.02091 0.02091 2.45% Castro::do_old_sources() 10 0.01916 0.01916 0.01916 2.24% MLPoisson::define() 11 0.01766 0.01766 0.01766 2.07% MLMG::mgVcycle_down::1 82 0.01739 0.01739 0.01739 2.04% MLMG::mgVcycle_down::2 82 0.01625 0.01625 0.01625 1.90% Castro::construct_old_gravity() 10 0.01584 0.01584 0.01584 1.85% Gravity::get_old_grav_vector() 10 0.01583 0.01583 0.01583 1.85% MLMG::mgVcycle_down::3 82 0.01582 0.01582 0.01582 1.85% MLMG::mgVcycle_down::4 82 0.0157 0.0157 0.0157 1.84% FabArray::ParallelCopy() 861 0.01517 0.01517 0.01517 1.78% Castro::normalize_species() 62 0.01508 0.01508 0.01508 1.77% FabArray::ParallelCopy_nowait() 861 0.01493 0.01493 0.01493 1.75% FabArray::setVal() 1062 0.01395 0.01395 0.01395 1.63% FabArray::Saxpy() 1370 0.0137 0.0137 0.0137 1.60% MLCGSolver::ParallelAllReduce 1832 0.01345 0.01345 0.01345 1.57% Castro::do_new_sources() 10 0.01281 0.01281 0.01281 1.50% MLMG::addInterpCorrection() 410 0.01274 0.01274 0.01274 1.49% MLMG::mgVcycle_up::1 82 0.01258 0.01258 0.01258 1.47% MLMG::mgVcycle_up::4 82 0.01252 0.01252 0.01252 1.47% Castro::expand_state() 10 0.01238 0.01238 0.01238 1.45% MLMG::mgVcycle_up::2 82 0.0123 0.0123 0.0123 1.44% MLMG::mgVcycle_up::3 82 0.01207 0.01207 0.01207 1.41% amrex::average_down 410 0.01203 0.01203 0.01203 1.41% MLCellLinOp::defineAuxData() 11 0.01198 0.01198 0.01198 1.40% amrex::Copy() 472 0.01134 0.01134 0.01134 1.33% Castro::initialize_advance() 10 0.01131 0.01131 0.01131 1.32% MLPoisson::Fapply() 1060 0.01083 0.01083 0.01083 1.27% Gravity::fill_multipole_BCs() 11 0.01027 0.01027 0.01027 1.20% Castro::enforce_min_density() 62 0.009827 0.009827 0.009827 1.15% Castro::post_timestep() 10 0.008674 0.008674 0.008674 1.02% FabArray::Xpay() 739 0.008238 0.008238 0.008238 0.96% MLCellLinOp::solutionResidual() 93 0.008116 0.008116 0.008116 0.95% MLMG::computeResidual() 82 0.0068 0.0068 0.0068 0.80% MLCellLinOp::defineBC() 11 0.005421 0.005421 0.005421 0.63% MLMG::prepareForSolve() 11 0.005358 0.005358 0.005358 0.63% Amr::InitializeInit() 1 0.005235 0.005235 0.005235 0.61% Amr::defBaseLevel() 1 0.005228 0.005228 0.005228 0.61% BndryData::define() 11 0.005163 0.005163 0.005163 0.60% Castro::estTimeStep() 21 0.005112 0.005112 0.005112 0.60% Castro::initData() 1 0.004535 0.004535 0.004535 0.53% amrex::Add() 82 0.003818 0.003818 0.003818 0.45% Castro::construct_new_source() 50 0.003744 0.003744 0.003744 0.44% Castro::construct_new_gravity_source() 10 0.003716 0.003716 0.003716 0.43% Castro::construct_old_source() 50 0.002977 0.002977 0.002977 0.35% Castro::construct_old_gravity_source() 10 0.002959 0.002959 0.002959 0.35% Castro::enforce_speed_limit() 62 0.00293 0.00293 0.00293 0.34% MLMG::ResNormInf() 93 0.00228 0.00228 0.00228 0.27% Castro::computeNewDt() 9 0.002148 0.002148 0.002148 0.25% Castro::finalize_do_advance() 10 0.002031 0.002031 0.002031 0.24% Castro::reset_internal_energy(Fab) 504 0.001966 0.001966 0.001966 0.23% Castro::apply_source_to_state() 20 0.001902 0.001902 0.001902 0.22% FabArray::setVal(val, thecmd, scomp, ncomp) 462 0.001827 0.001827 0.001827 0.21% MLMG::getGradSolution() 11 0.001662 0.001662 0.001662 0.19% MLCellLinOp::compGrad() 11 0.001656 0.001656 0.001656 0.19% MLCellLinOp::setLevelBC() 11 0.001636 0.001636 0.001636 0.19% check_for_negative_density() 10 0.001509 0.001509 0.001509 0.18% FabArrayBase::getCPC() 1323 0.001472 0.001472 0.001472 0.17% FabArray::mult() 43 0.001421 0.001421 0.001421 0.17% FabArray::setDomainBndry() 41 0.001417 0.001417 0.001417 0.17% MLPoisson::prepareForSolve() 11 0.001373 0.001373 0.001373 0.16% MLMG::computeMLResidual() 11 0.001361 0.001361 0.001361 0.16% MLCellLinOp::prepareForSolve() 11 0.001359 0.001359 0.001359 0.16% Castro::check_for_nan() 20 0.001295 0.001295 0.001295 0.15% MultiFab::contains_nan() 20 0.001284 0.001284 0.001284 0.15% Gravity::update_max_rhs() 11 0.001041 0.001041 0.001041 0.12% Other 4742 0.007515 0.007515 0.007515 0.88% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 6420 KiB 9037 MiB Castro::initMFs() 48 48 67 MiB 68 MiB Castro::swap_state_time_levels() 32 32 47 MiB 55 MiB StateData::define() 32 32 55 MiB 55 MiB FillPatchIterator::Initialize 328 328 1209 KiB 39 MiB Castro::initialize_do_advance() 80 80 26 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 16 16 1781 KiB 28 MiB Castro::initialize_advance() 80 80 16 MiB 23 MiB Castro::buildMetrics() 32 32 15 MiB 15 MiB Castro::Castro() 48 48 7616 KiB 14 MiB MLMG::prepareForSolve() 660 660 3975 KiB 12 MiB Gravity::get_new_grav_vector() 91 91 313 KiB 10 MiB Gravity::get_old_grav_vector() 80 80 189 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 7516 KiB 7586 KiB Gravity::fill_multipole_BCs() 154 154 20 KiB 2053 KiB Gravity::update_max_rhs() 88 88 2362 B 2048 KiB Gravity::solve_for_phi() 80 80 631 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 109 KiB 2048 KiB BndryData::define() 1056 1056 363 KiB 1095 KiB MLCellLinOp::defineAuxData() 1716 1716 231 KiB 671 KiB Castro::estTimeStep() 21 21 2916 B 480 KiB VisMF::Write(FabArray) 656 656 3758 B 320 KiB Castro::normalize_species() 62 62 5736 B 320 KiB amrex::average_down 1067 1067 1711 B 257 KiB MLMG::addInterpCorrection() 1066 1066 1284 B 257 KiB amrex::Dot() 1360 1360 3894 B 160 KiB FabArray::norminf() 1143 1143 3793 B 160 KiB check_for_negative_density() 10 10 282 B 160 KiB Castro::initData() 1 1 55 B 160 KiB MultiFab::max() 11 11 62 B 160 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MultiFab::contains_nan() 20 20 30 B 20 KiB MLPoisson::Fsmooth() 132 132 3881 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 47 B 10 KiB FillBoundary_nowait() 760 760 346 B 9648 B MLCellLinOp::applyBC() 8702 8702 246 B 9344 B MLCellLinOp::prepareForSolve() 66 66 4 B 7792 B amrex::Copy() 100 100 3936 B 6144 B StateData::FillBoundary(geom) 1992 1992 42 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLCellLinOp::defineBC() 66 66 409 B 1248 B MLCGSolver::bicgstab 410 410 106 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- The_Managed_Arena::Initialize() 1 1 813 B 8192 KiB ----------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 43 KiB 8192 KiB VisMF::Write(FabArray) 744 744 501 KiB 3584 KiB FabArray::setVal() 106 106 25 KiB 30 KiB MLPoisson::Fsmooth() 132 132 3881 B 12 KiB FabArray::ParallelCopy_nowait() 861 861 47 B 10 KiB FillBoundary_nowait() 760 760 346 B 9648 B MLCellLinOp::applyBC() 4351 4351 244 B 9328 B MLCellLinOp::prepareForSolve() 66 66 4 B 7792 B amrex::Copy() 100 100 3937 B 6144 B Gravity::get_new_grav_vector() 3 3 2878 B 3072 B StateData::FillBoundary(geom) 1992 1992 43 B 3024 B Gravity::fill_multipole_BCs() 33 33 4 B 2832 B amrex::average_down 83 83 617 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 132 132 1 B 1616 B MLMG::addInterpCorrection() 82 82 3 B 1024 B MLMG::prepareForSolve() 11 11 331 B 1024 B MLCellLinOp::setLevelBC() 66 66 0 B 768 B amrex::Dot() 1360 1360 28 B 400 B FabArray::norminf() 1143 1143 11 B 144 B Castro::estTimeStep() 21 21 0 B 32 B check_for_negative_density() 10 10 0 B 16 B MultiFab::max() 11 11 0 B 16 B MultiFab::contains_nan() 20 20 0 B 16 B Castro::normalize_species() 62 62 0 B 16 B Castro::initData() 1 1 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2109 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.07-13-g361f4ab839f2) finalized Initializing AMReX (24.07-13-g361f4ab839f2)... Initializing CUDA... CUDA initialized with 1 device. AMReX (24.07-13-g361f4ab839f2) initialized Starting run at 09:24:45 UTC on 2024-07-09. Successfully read inputs file ... Castro git describe: 24.07-3-g79841a784 AMReX git describe: 24.07-13-g361f4ab83 Microphysics git describe: 24.07-9-ga3bc2465 reading extern runtime parameters ... 3 Species: C12 O16 Mg24 restarting calculation from file: dustcollapse-restart_chk00005 Successfully read inputs file ... read CPU time: 0.452446833 Restart time = 0.12352679 seconds. [Level 0 step 6] ADVANCE at time 0.0002509599271 with dt = 5.79654185e-05 [Level 0 step 6] Advanced 262144 cells [STEP 6] Coarse TimeStep time: 0.066475614 [STEP 6] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 6 TIME = 0.0003089253456 DT = 5.79654185e-05 [Level 0 step 7] ADVANCE at time 0.0003089253456 with dt = 6.086368943e-05 [Level 0 step 7] Advanced 262144 cells [STEP 7] Coarse TimeStep time: 0.050612866 [STEP 7] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 7 TIME = 0.000369789035 DT = 6.086368943e-05 [Level 0 step 8] ADVANCE at time 0.000369789035 with dt = 6.39068739e-05 [Level 0 step 8] Advanced 262144 cells [STEP 8] Coarse TimeStep time: 0.053154484 [STEP 8] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 8 TIME = 0.000433695909 DT = 6.39068739e-05 [Level 0 step 9] ADVANCE at time 0.000433695909 with dt = 6.71022176e-05 [Level 0 step 9] Advanced 262144 cells [STEP 9] Coarse TimeStep time: 0.054208223 [STEP 9] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 9 TIME = 0.0005007981265 DT = 6.71022176e-05 [Level 0 step 10] ADVANCE at time 0.0005007981265 with dt = 7.045732848e-05 [Level 0 step 10] Advanced 262144 cells [STEP 10] Coarse TimeStep time: 0.064946246 [STEP 10] FAB kilobyte spread across MPI nodes: [368280 ... 368280] STEP = 10 TIME = 0.000571255455 DT = 7.045732848e-05 PLOTFILE: file = dustcollapse-restart_plt00010 Write plotfile time = 0.029456465 seconds Ending run at 09:24:46 UTC on 2024-07-09. Run time = 0.443372778 Run time without initialization = 0.319260246 Average number of zones advanced per microsecond: 4.105 Average number of zones advanced per microsecond per rank: 4.105 CPU(0): Heap Space (bytes) used by Coalescing FAB Arena: 9476456448 TinyProfiler total time across processes [min...avg...max]: 0.4434 ... 0.4434 ... 0.4434 -------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------- Castro::construct_ctu_hydro_source() 5 0.0912 0.0912 0.0912 20.57% VisMF::Read() 3 0.06203 0.06203 0.06203 13.99% Amr::restart() 1 0.05489 0.05489 0.05489 12.38% MLCellLinOp::applyBC() 1910 0.03782 0.03782 0.03782 8.53% VisMF::Write(FabArray) 1 0.02691 0.02691 0.02691 6.07% MLPoisson::Fsmooth() 1440 0.01599 0.01599 0.01599 3.61% FillBoundary_nowait() 1730 0.01458 0.01458 0.01458 3.29% StateData::FillBoundary(geom) 160 0.01323 0.01323 0.01323 2.98% amrex::Dot() 484 0.009856 0.009856 0.009856 2.22% FabArray::norminf() 465 0.009297 0.009297 0.009297 2.10% Castro::reset_internal_energy(MultiFab) 30 0.009157 0.009157 0.009157 2.07% FabArray::setVal() 501 0.006987 0.006987 0.006987 1.58% Castro::normalize_species() 30 0.006879 0.006879 0.006879 1.55% FabArray::ParallelCopy_nowait() 380 0.006484 0.006484 0.006484 1.46% FabArray::Saxpy() 597 0.006266 0.006266 0.006266 1.41% StateDataPhysBCFunct::() 20 0.006176 0.006176 0.006176 1.39% MLCellLinOp::defineAuxData() 6 0.005867 0.005867 0.005867 1.32% amrex::Copy() 221 0.005676 0.005676 0.005676 1.28% Gravity::fill_multipole_BCs() 6 0.005363 0.005363 0.005363 1.21% MLPoisson::Fapply() 464 0.004888 0.004888 0.004888 1.10% FabArray::Xpay() 325 0.003725 0.003725 0.003725 0.84% MLMG::addInterpCorrection() 180 0.003336 0.003336 0.003336 0.75% amrex::average_down 180 0.002937 0.002937 0.002937 0.66% Castro::estTimeStep() 10 0.002786 0.002786 0.002786 0.63% Castro::enforce_min_density() 30 0.002748 0.002748 0.002748 0.62% Amr::writePlotFile() 1 0.002364 0.002364 0.002364 0.53% BndryData::define() 6 0.002312 0.002312 0.002312 0.52% Castro::construct_new_gravity_source() 5 0.00188 0.00188 0.00188 0.42% amrex::Add() 36 0.00165 0.00165 0.00165 0.37% Castro::computeTemp() 30 0.001643 0.001643 0.001643 0.37% Castro::construct_old_gravity_source() 5 0.001538 0.001538 0.001538 0.35% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001026 0.001026 0.001026 0.23% Castro::reset_internal_energy(Fab) 240 0.001021 0.001021 0.001021 0.23% MLCellLinOp::setLevelBC() 6 0.0009185 0.0009185 0.0009185 0.21% Gravity::actual_solve_with_mlmg() 6 0.0008536 0.0008536 0.0008536 0.19% Castro::enforce_speed_limit() 30 0.0008427 0.0008427 0.0008427 0.19% MLCellLinOp::prepareForSolve() 6 0.0007814 0.0007814 0.0007814 0.18% MLCGSolver::bicgstab 36 0.0007745 0.0007745 0.0007745 0.17% FabArray::setDomainBndry() 20 0.0007247 0.0007247 0.0007247 0.16% FabArray::mult() 22 0.0007234 0.0007234 0.0007234 0.16% MLCellLinOp::compGrad() 6 0.0006333 0.0006333 0.0006333 0.14% MLCellLinOp::smooth() 720 0.0005868 0.0005868 0.0005868 0.13% MLMG::prepareForSolve() 6 0.0005595 0.0005595 0.0005595 0.13% Gravity::get_old_grav_vector() 5 0.0004143 0.0004143 0.0004143 0.09% FabArrayBase::getCPC() 632 0.0003723 0.0003723 0.0003723 0.08% Gravity::get_new_grav_vector() 5 0.0003453 0.0003453 0.0003453 0.08% FabArray::FillBoundary() 1730 0.0003359 0.0003359 0.0003359 0.08% main() 1 0.0002716 0.0002716 0.0002716 0.06% Castro::subcycle_advance_ctu() 5 0.0002202 0.0002202 0.0002202 0.05% MLCellLinOp::apply() 464 0.0002103 0.0002103 0.0002103 0.05% AmrLevel::FillPatch() 20 0.0002006 0.0002006 0.0002006 0.05% Amr::coarseTimeStep() 5 0.0001637 0.0001637 0.0001637 0.04% MLCGSolver::ParallelAllReduce 798 0.0001456 0.0001456 0.0001456 0.03% MLCellLinOp::defineBC() 6 0.0001446 0.0001446 0.0001446 0.03% FabArray::ParallelCopy() 380 0.0001093 0.0001093 0.0001093 0.02% FillPatchIterator::Initialize 20 0.000106 0.000106 0.000106 0.02% MLMG::mgVcycle() 36 9.506e-05 9.506e-05 9.506e-05 0.02% Castro::advance() 5 9.262e-05 9.262e-05 9.262e-05 0.02% Amr::timeStep() 5 8.338e-05 8.338e-05 8.338e-05 0.02% MLCellLinOp::correctionResidual() 180 8.174e-05 8.174e-05 8.174e-05 0.02% AmrLevel::restart() 1 7.867e-05 7.867e-05 7.867e-05 0.02% Castro::finalize_do_advance() 5 7.455e-05 7.455e-05 7.455e-05 0.02% Gravity::update_max_rhs() 6 6.853e-05 6.853e-05 6.853e-05 0.02% StateData::restartDoit() 4 6.226e-05 6.226e-05 6.226e-05 0.01% Castro::initialize_advance() 5 6.055e-05 6.055e-05 6.055e-05 0.01% MLMG:computeResOfCorrection() 180 5.996e-05 5.996e-05 5.996e-05 0.01% Castro::initialize_do_advance() 5 5.306e-05 5.306e-05 5.306e-05 0.01% Gravity::solve_for_phi() 5 4.795e-05 4.795e-05 4.795e-05 0.01% MLMG::mgVcycle_down::0 36 4.308e-05 4.308e-05 4.308e-05 0.01% MLMG::actualBottomSolve() 36 3.98e-05 3.98e-05 3.98e-05 0.01% Castro::clean_state() 30 3.86e-05 3.86e-05 3.86e-05 0.01% MLMG::solve() 6 3.842e-05 3.842e-05 3.842e-05 0.01% MLMG::mgVcycle_down::1 36 3.814e-05 3.814e-05 3.814e-05 0.01% MLMG::mgVcycle_down::2 36 3.554e-05 3.554e-05 3.554e-05 0.01% MLMG::mgVcycle_down::4 36 3.411e-05 3.411e-05 3.411e-05 0.01% MLMG::mgVcycle_down::3 36 3.29e-05 3.29e-05 3.29e-05 0.01% Gravity::multilevel_solve_for_new_phi() 1 2.953e-05 2.953e-05 2.953e-05 0.01% MLMG::mgVcycle_up::4 36 2.763e-05 2.763e-05 2.763e-05 0.01% MLMG::oneIter() 36 2.695e-05 2.695e-05 2.695e-05 0.01% Castro::do_advance_ctu() 5 2.542e-05 2.542e-05 2.542e-05 0.01% MLMG::mgVcycle_up::3 36 2.285e-05 2.285e-05 2.285e-05 0.01% FillPatchIterator::FillFromLevel0() 20 2.222e-05 2.222e-05 2.222e-05 0.01% MLCellLinOp::solutionResidual() 42 2.214e-05 2.214e-05 2.214e-05 0.00% MLMG::mgVcycle_up::0 36 2.209e-05 2.209e-05 2.209e-05 0.00% Castro::post_restart() 1 2.176e-05 2.176e-05 2.176e-05 0.00% MLMG::ResNormInf() 42 2.129e-05 2.129e-05 2.129e-05 0.00% MLMG::mgVcycle_up::2 36 2.111e-05 2.111e-05 2.111e-05 0.00% MLMG::mgVcycle_up::1 36 2.044e-05 2.044e-05 2.044e-05 0.00% MLPoisson::define() 6 1.959e-05 1.959e-05 1.959e-05 0.00% FillPatchSingleLevel 20 1.639e-05 1.639e-05 1.639e-05 0.00% MLMG::computeResidual() 36 1.601e-05 1.601e-05 1.601e-05 0.00% MLMG::mgVcycle_bottom 36 1.485e-05 1.485e-05 1.485e-05 0.00% Gravity::solve_phi_with_mlmg() 6 1.331e-05 1.331e-05 1.331e-05 0.00% Castro::do_old_sources() 5 1.174e-05 1.174e-05 1.174e-05 0.00% Castro::construct_new_gravity() 5 1.165e-05 1.165e-05 1.165e-05 0.00% Castro::construct_new_source() 25 1.078e-05 1.078e-05 1.078e-05 0.00% MLPoisson::prepareForSolve() 6 1.026e-05 1.026e-05 1.026e-05 0.00% Castro::do_new_sources() 5 9.776e-06 9.776e-06 9.776e-06 0.00% Castro::construct_old_source() 25 9.503e-06 9.503e-06 9.503e-06 0.00% Castro::expand_state() 5 8.781e-06 8.781e-06 8.781e-06 0.00% Gravity::actual_multilevel_solve() 1 8.495e-06 8.495e-06 8.495e-06 0.00% Castro::apply_source_to_state() 10 6.11e-06 6.11e-06 6.11e-06 0.00% Castro::check_for_nan() 10 5.856e-06 5.856e-06 5.856e-06 0.00% Castro::construct_old_gravity() 5 5.685e-06 5.685e-06 5.685e-06 0.00% Castro::post_timestep() 5 5.396e-06 5.396e-06 5.396e-06 0.00% MLMG::computeMLResidual() 6 4.227e-06 4.227e-06 4.227e-06 0.00% Castro::computeNewDt() 5 3.729e-06 3.729e-06 3.729e-06 0.00% MLMG::getGradSolution() 6 3.214e-06 3.214e-06 3.214e-06 0.00% Amr::init() 1 7.55e-07 7.55e-07 7.55e-07 0.00% Other 2165 0.002822 0.002822 0.002822 0.64% -------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------- main() 1 0.4434 0.4434 0.4434 99.99% Amr::coarseTimeStep() 5 0.2896 0.2896 0.2896 65.30% Amr::timeStep() 5 0.2879 0.2879 0.2879 64.92% Castro::advance() 5 0.2846 0.2846 0.2846 64.18% Castro::subcycle_advance_ctu() 5 0.2787 0.2787 0.2787 62.85% Castro::do_advance_ctu() 5 0.2785 0.2785 0.2785 62.80% Castro::construct_new_gravity() 5 0.1453 0.1453 0.1453 32.77% Gravity::solve_phi_with_mlmg() 6 0.1422 0.1422 0.1422 32.07% Gravity::solve_for_phi() 5 0.1367 0.1367 0.1367 30.84% Gravity::actual_solve_with_mlmg() 6 0.1366 0.1366 0.1366 30.79% MLMG::solve() 6 0.1239 0.1239 0.1239 27.95% Amr::init() 1 0.1236 0.1236 0.1236 27.87% Amr::restart() 1 0.1236 0.1236 0.1236 27.87% MLMG::oneIter() 36 0.1157 0.1157 0.1157 26.10% MLMG::mgVcycle() 36 0.1141 0.1141 0.1141 25.72% Castro::construct_ctu_hydro_source() 5 0.09545 0.09545 0.09545 21.53% AmrLevel::restart() 1 0.0624 0.0624 0.0624 14.07% StateData::restartDoit() 4 0.06232 0.06232 0.06232 14.05% VisMF::Read() 3 0.06203 0.06203 0.06203 13.99% MLCellLinOp::smooth() 720 0.05718 0.05718 0.05718 12.89% MLCellLinOp::applyBC() 1910 0.05307 0.05307 0.05307 11.97% MLMG::mgVcycle_bottom 36 0.03384 0.03384 0.03384 7.63% MLMG::actualBottomSolve() 36 0.03382 0.03382 0.03382 7.63% MLCGSolver::bicgstab 36 0.03344 0.03344 0.03344 7.54% Amr::writePlotFile() 1 0.02954 0.02954 0.02954 6.66% VisMF::Write(FabArray) 1 0.02691 0.02691 0.02691 6.07% AmrLevel::FillPatch() 20 0.02449 0.02449 0.02449 5.52% FillPatchIterator::Initialize 20 0.02236 0.02236 0.02236 5.04% Castro::clean_state() 30 0.02233 0.02233 0.02233 5.04% FillPatchIterator::FillFromLevel0() 20 0.02153 0.02153 0.02153 4.85% FillPatchSingleLevel 20 0.0215 0.0215 0.0215 4.85% StateDataPhysBCFunct::() 20 0.01941 0.01941 0.01941 4.38% MLCellLinOp::apply() 464 0.01726 0.01726 0.01726 3.89% MLMG::mgVcycle_down::0 36 0.01647 0.01647 0.01647 3.71% MLPoisson::Fsmooth() 1440 0.01599 0.01599 0.01599 3.61% FabArray::FillBoundary() 1730 0.01525 0.01525 0.01525 3.44% FillBoundary_nowait() 1730 0.01492 0.01492 0.01492 3.36% StateData::FillBoundary(geom) 160 0.01323 0.01323 0.01323 2.98% MLMG::mgVcycle_up::0 36 0.01228 0.01228 0.01228 2.77% Castro::computeTemp() 30 0.01182 0.01182 0.01182 2.67% Castro::initialize_do_advance() 5 0.01143 0.01143 0.01143 2.58% Castro::reset_internal_energy(MultiFab) 30 0.01018 0.01018 0.01018 2.30% Castro::do_old_sources() 5 0.009976 0.009976 0.009976 2.25% MLPoisson::define() 6 0.009913 0.009913 0.009913 2.24% amrex::Dot() 484 0.009856 0.009856 0.009856 2.22% MLMG:computeResOfCorrection() 180 0.009635 0.009635 0.009635 2.17% MLCellLinOp::correctionResidual() 180 0.009575 0.009575 0.009575 2.16% FabArray::norminf() 465 0.009297 0.009297 0.009297 2.10% Gravity::get_new_grav_vector() 5 0.008457 0.008457 0.008457 1.91% Castro::construct_old_gravity() 5 0.007936 0.007936 0.007936 1.79% Gravity::get_old_grav_vector() 5 0.00793 0.00793 0.00793 1.79% MLMG::mgVcycle_down::1 36 0.007893 0.007893 0.007893 1.78% MLMG::mgVcycle_down::2 36 0.007248 0.007248 0.007248 1.63% MLMG::mgVcycle_down::3 36 0.007072 0.007072 0.007072 1.59% MLMG::mgVcycle_down::4 36 0.006998 0.006998 0.006998 1.58% FabArray::ParallelCopy() 380 0.006993 0.006993 0.006993 1.58% FabArray::setVal() 501 0.006987 0.006987 0.006987 1.58% Castro::expand_state() 5 0.00689 0.00689 0.00689 1.55% FabArray::ParallelCopy_nowait() 380 0.006884 0.006884 0.006884 1.55% Castro::normalize_species() 30 0.006879 0.006879 0.006879 1.55% MLCellLinOp::defineAuxData() 6 0.006686 0.006686 0.006686 1.51% Castro::do_new_sources() 5 0.006484 0.006484 0.006484 1.46% FabArray::Saxpy() 597 0.006266 0.006266 0.006266 1.41% Castro::post_restart() 1 0.006085 0.006085 0.006085 1.37% MLCGSolver::ParallelAllReduce 798 0.005927 0.005927 0.005927 1.34% MLMG::addInterpCorrection() 180 0.005775 0.005775 0.005775 1.30% Gravity::multilevel_solve_for_new_phi() 1 0.005693 0.005693 0.005693 1.28% amrex::Copy() 221 0.005676 0.005676 0.005676 1.28% Gravity::actual_multilevel_solve() 1 0.005663 0.005663 0.005663 1.28% MLMG::mgVcycle_up::4 36 0.005638 0.005638 0.005638 1.27% MLMG::mgVcycle_up::1 36 0.005618 0.005618 0.005618 1.27% Gravity::fill_multipole_BCs() 6 0.005505 0.005505 0.005505 1.24% Castro::initialize_advance() 5 0.005499 0.005499 0.005499 1.24% MLMG::mgVcycle_up::2 36 0.005486 0.005486 0.005486 1.24% MLMG::mgVcycle_up::3 36 0.005424 0.005424 0.005424 1.22% amrex::average_down 180 0.005408 0.005408 0.005408 1.22% MLPoisson::Fapply() 464 0.004888 0.004888 0.004888 1.10% MLCellLinOp::solutionResidual() 42 0.003895 0.003895 0.003895 0.88% FabArray::Xpay() 325 0.003725 0.003725 0.003725 0.84% Castro::post_timestep() 5 0.003203 0.003203 0.003203 0.72% MLCellLinOp::defineBC() 6 0.00307 0.00307 0.00307 0.69% MLMG::computeResidual() 36 0.003036 0.003036 0.003036 0.68% MLMG::prepareForSolve() 6 0.003021 0.003021 0.003021 0.68% BndryData::define() 6 0.002925 0.002925 0.002925 0.66% Castro::estTimeStep() 10 0.002786 0.002786 0.002786 0.63% Castro::enforce_min_density() 30 0.002748 0.002748 0.002748 0.62% Castro::construct_new_source() 25 0.00189 0.00189 0.00189 0.43% Castro::construct_new_gravity_source() 5 0.00188 0.00188 0.00188 0.42% amrex::Add() 36 0.00165 0.00165 0.00165 0.37% Castro::construct_old_source() 25 0.001547 0.001547 0.001547 0.35% Castro::construct_old_gravity_source() 5 0.001538 0.001538 0.001538 0.35% Castro::computeNewDt() 5 0.001496 0.001496 0.001496 0.34% Castro::finalize_do_advance() 5 0.001368 0.001368 0.001368 0.31% MLMG::ResNormInf() 42 0.001071 0.001071 0.001071 0.24% FabArray::setVal(val, thecmd, scomp, ncomp) 252 0.001026 0.001026 0.001026 0.23% Castro::reset_internal_energy(Fab) 240 0.001021 0.001021 0.001021 0.23% Castro::apply_source_to_state() 10 0.001002 0.001002 0.001002 0.23% MLMG::getGradSolution() 6 0.0009448 0.0009448 0.0009448 0.21% MLCellLinOp::compGrad() 6 0.0009416 0.0009416 0.0009416 0.21% MLCellLinOp::setLevelBC() 6 0.0009185 0.0009185 0.0009185 0.21% MLMG::computeMLResidual() 6 0.0008795 0.0008795 0.0008795 0.20% Castro::enforce_speed_limit() 30 0.0008427 0.0008427 0.0008427 0.19% FabArrayBase::getCPC() 632 0.0008059 0.0008059 0.0008059 0.18% MLPoisson::prepareForSolve() 6 0.0007916 0.0007916 0.0007916 0.18% MLCellLinOp::prepareForSolve() 6 0.0007814 0.0007814 0.0007814 0.18% Gravity::update_max_rhs() 6 0.0007627 0.0007627 0.0007627 0.17% FabArray::setDomainBndry() 20 0.0007247 0.0007247 0.0007247 0.16% FabArray::mult() 22 0.0007234 0.0007234 0.0007234 0.16% Castro::check_for_nan() 10 0.0006801 0.0006801 0.0006801 0.15% Other 2165 0.004118 0.004118 0.004118 0.93% -------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.ref_ratio(nvals = 4) :: [2, 2, 2, 2] [TOP]::amr.regrid_int(nvals = 4) :: [2, 2, 2, 2] Device Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Arena::Initialize() 1 1 12 MiB 9037 MiB Castro::initMFs() 48 48 49 MiB 68 MiB Castro::swap_state_time_levels() 32 32 40 MiB 55 MiB StateData::restartDoit() 32 32 46 MiB 55 MiB FillPatchIterator::Initialize 160 160 1127 KiB 39 MiB Castro::initialize_do_advance() 40 40 24 MiB 39 MiB ResizeRandomSeed 1 1 30 MiB 30 MiB Amr::writePlotFile() 8 8 1789 KiB 28 MiB Castro::initialize_advance() 40 40 15 MiB 23 MiB Castro::buildMetrics() 32 32 11 MiB 15 MiB Castro::post_restart() 48 48 5564 KiB 14 MiB MLMG::prepareForSolve() 361 361 3431 KiB 12 MiB Gravity::get_old_grav_vector() 43 43 185 KiB 10 MiB Gravity::get_new_grav_vector() 40 40 195 KiB 10 MiB Gravity::multilevel_solve_for_new_phi() 24 24 5550 KiB 7586 KiB Gravity::fill_multipole_BCs() 84 84 20 KiB 2053 KiB Gravity::update_max_rhs() 48 48 3432 B 2048 KiB Gravity::solve_for_phi() 40 40 630 KiB 2048 KiB Gravity::actual_multilevel_solve() 8 8 26 KiB 2048 KiB BndryData::define() 576 576 316 KiB 1095 KiB MLCellLinOp::defineAuxData() 936 936 202 KiB 671 KiB Castro::estTimeStep() 10 10 2998 B 480 KiB VisMF::Write(FabArray) 112 112 1344 B 320 KiB Castro::normalize_species() 30 30 5041 B 320 KiB amrex::average_down 469 469 1357 B 257 KiB MLMG::addInterpCorrection() 468 468 1111 B 257 KiB amrex::Dot() 592 592 3287 B 160 KiB FabArray::norminf() 501 501 3246 B 160 KiB check_for_negative_density() 5 5 154 B 160 KiB MultiFab::max() 6 6 76 B 160 KiB FabArray::setVal() 66 66 18 KiB 27 KiB MultiFab::contains_nan() 10 10 30 B 20 KiB MLPoisson::Fsmooth() 60 60 3326 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 45 B 10 KiB FillBoundary_nowait() 336 336 286 B 9648 B MLCellLinOp::applyBC() 3820 3820 215 B 9344 B amrex::Copy() 56 56 5149 B 8816 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B StateData::FillBoundary(geom) 960 960 44 B 3024 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLCellLinOp::defineBC() 36 36 355 B 1248 B MLCGSolver::bicgstab 180 180 90 B 1216 B ------------------------------------------------------------------------------ Managed Memory Usage: ------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------ The_Managed_Arena::Initialize() 1 1 1416 B 8192 KiB ------------------------------------------------------------------ Pinned Memory Usage: ------------------------------------------------------------------------------ Name Nalloc Nfree AvgMem MaxMem ------------------------------------------------------------------------------ The_Pinned_Arena::Initialize() 1 1 82 KiB 8192 KiB VisMF::Write(FabArray) 120 120 172 KiB 3584 KiB VisMF::Read() 24 24 211 KiB 3000 KiB FabArray::setVal() 66 66 18 KiB 27 KiB MLPoisson::Fsmooth() 60 60 3326 B 12 KiB FabArray::ParallelCopy_nowait() 380 380 45 B 10 KiB FillBoundary_nowait() 336 336 286 B 9648 B MLCellLinOp::applyBC() 1910 1910 213 B 9328 B amrex::Copy() 56 56 5150 B 8816 B MLCellLinOp::prepareForSolve() 36 36 4 B 7792 B Gravity::get_old_grav_vector() 3 3 2165 B 3072 B StateData::FillBoundary(geom) 960 960 45 B 3024 B Gravity::fill_multipole_BCs() 18 18 6 B 2832 B MLMG::prepareForSolve() 7 7 736 B 1648 B amrex::average_down 37 37 397 B 1648 B FabArray::setVal(val, thecmd, scomp, ncomp) 72 72 2 B 1616 B MLMG::addInterpCorrection() 36 36 2 B 1024 B MLCellLinOp::setLevelBC() 36 36 0 B 768 B amrex::Dot() 592 592 24 B 400 B FabArray::norminf() 501 501 9 B 144 B Castro::estTimeStep() 10 10 0 B 32 B check_for_negative_density() 5 5 0 B 16 B MultiFab::max() 6 6 0 B 16 B MultiFab::contains_nan() 10 10 0 B 16 B Castro::normalize_species() 30 30 0 B 16 B ------------------------------------------------------------------------------ Total GPU global memory (MB): 12049 Free GPU global memory (MB): 2109 [The Arena] space allocated (MB): 9037 [The Arena] space used (MB): 0 [The Managed Arena] space allocated (MB): 8 [The Managed Arena] space used (MB): 0 [The Pinned Arena] space allocated (MB): 8 [The Pinned Arena] space used (MB): 0 AMReX (24.07-13-g361f4ab839f2) finalized