commit c1e8e73b63dc8b6732c6cdfe51914f7ae144575b Author: Axel Huebl Date: Mon Apr 17 18:57:49 2023 -0700 Conduit: Update ParticleTile API (#3260) ## Summary Update to use the new `ParticleTile` template parameters. ## Additional background Follow-up to #2878. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint_ParticlesI.H commit 6b294a5cd3e5c55b36c7d0afe2993501775eb42f Author: thierry <70574092+Thierry992@users.noreply.github.com> Date: Mon Apr 17 11:40:34 2023 -0700 Support SoA-Only Particles (#2878) Add support for storing particle positions and ids SoA style. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [x] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Andrew Myers Co-authored-by: Axel Huebl Src/AmrCore/AMReX_AmrParticles.H Src/Base/AMReX_Array.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_TypeTraits.H Src/EB/AMReX_EB2.H Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_MakeParticle.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_WriteBinaryParticleData.H Src/Particle/CMakeLists.txt Src/Particle/Make.package Tests/Particles/ParticleTransformations/main.cpp Tests/Particles/RedistributeSOA/CMakeLists.txt Tests/Particles/RedistributeSOA/GNUmakefile Tests/Particles/RedistributeSOA/Make.package Tests/Particles/RedistributeSOA/inputs Tests/Particles/RedistributeSOA/inputs.rt Tests/Particles/RedistributeSOA/inputs.rt.cuda Tests/Particles/RedistributeSOA/inputs.rt.cuda.big Tests/Particles/RedistributeSOA/inputs.rt.cuda.mr Tests/Particles/RedistributeSOA/inputs.rt.cuda.nonperiodic Tests/Particles/RedistributeSOA/inputs.rt.cuda.sort Tests/Particles/RedistributeSOA/main.cpp Tests/Particles/SOAParticle/CMakeLists.txt Tests/Particles/SOAParticle/main.cpp commit 27ef4ea56945af7f250a6cec9634ebdcaf863b0b Author: Weiqun Zhang Date: Wed Apr 12 13:41:55 2023 -0700 Roundoff errors in computeRoundoffDomain (#3255) Making changes to let the Intel CI tests pass. This is not really an issue in practice. Src/Base/AMReX_Geometry.cpp commit 2f47d132d71c49abbeac5e081fe9ce886c621d54 Author: Luca Fedeli Date: Wed Apr 12 21:40:31 2023 +0200 Add powi function in AMReX_Math.H (#3251) ## Summary This PR adds a replacement of `std::pow` (`powi`) that is intended to be used when the exponent is integer and known at compile time. If fast math optimizations are not enabled, `powi` can be one order of magnitude faster than `std::pow`. ## Additional background Adding this function to `AMReX_Math.H` was discussed with @WeiqunZhang and @ax3l in https://github.com/ECP-WarpX/WarpX/pull/3824 ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [X] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_Math.H commit bdb1552d8c248f4fccd2e1bb642d4515eff7a31e Author: pramenku <7664080+pramenku@users.noreply.github.com> Date: Wed Apr 12 21:59:55 2023 +0530 update hiprand header path (#3252) Need to update hiprand header path to "" so it will work with future rocm release. It will work with 5.4 release too. So to be ready well ahead with future rocm changes, requesting the change, In file included from /home/master/amr-wind/submods/amrex/Src/Base/AMReX_RandomEngine.H:11: /opt/rocm-xxxx/include/hiprand.hpp:16:2: error: "This file is deprecated. Use the header file from /opt/rocm-xxx/include/hiprand/hiprand.hpp by using #include " #error "This file is deprecated. Use the header file from /opt/rocm-xxxx/include/hiprand/hiprand.hpp by using #include " ^ The proposed changes: It will help user to be ready to use future rocm release alongwith previous releases like 5.4. Src/Base/AMReX_RandomEngine.H commit 1cb5f01a7339b52e98b5d098e956456a6faf79ab Author: Luca Fedeli Date: Wed Apr 12 17:11:58 2023 +0200 Fix a typo in a comment in AMReX_Math.H (#3253) Src/Base/AMReX_Math.H commit 15038d466c05fc607a0dd6ad0c192f8ed025281a Author: Weiqun Zhang Date: Tue Apr 11 13:36:21 2023 -0700 Add Tests/RoundoffDomain (#3248) The new test exposes an issue with Intel compiler. So we have to rewrite the computeRoundofDomain function in a slightly different way to avoid roundoff difference between `auto r = f(x-y)` and `z = x-y; auto r = f(z)`. Also fix an assertion issue in #3247. Src/Base/AMReX_Geometry.cpp Src/Particle/AMReX_ParticleUtil.H Tests/CMakeLists.txt Tests/RoundoffDomain/CMakeLists.txt Tests/RoundoffDomain/GNUmakefile Tests/RoundoffDomain/Make.package Tests/RoundoffDomain/main.cpp commit a54fee8706c812925c4ec7c29cecbc3e068a09bc Author: Weiqun Zhang Date: Mon Apr 10 17:27:26 2023 -0700 Installation using configure: Forgot to install Parser headers (#3249) GNUmakefile.in commit bb47ec9a6e4fbaea36839d94cd2bf7bdba038168 Author: Weiqun Zhang Date: Mon Apr 10 15:50:07 2023 -0700 Fix Roundoff Domain (#3247) There was a flaw in amrex::Geometry::computeRoundoffDomain. If probhi is close to zero, std::nextafter towards negative infinity will be a very small number (e.g., 1.e-300). So the function will be able to find the lowest value of roundoff_hi that's outside the domain within a reasonable number of iterations. In the new implementation, we use bisection to find the lowest value of roundoff_lo that's inside the domain and the highest value fo roundoff_hi that's inside the domain, up to a tolerance. X-ref: - #3243 - #3199 Src/Base/AMReX_Geometry.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H commit 8cae8024915ff8d31fe655665d17783fd34cda8f Author: Weiqun Zhang Date: Sat Apr 8 09:02:40 2023 -0700 Adjust GitHub Cache Size Again (#3246) The sizes seemed to be too small, and we had a lot of cache misses. .github/workflows/clang.yml .github/workflows/cuda.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/hypre.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/sundials.yml .github/workflows/windows.yml commit d21ba63af5269c5ba69f0869473d61ddc3b406c3 Author: Weiqun Zhang Date: Fri Apr 7 17:23:06 2023 -0700 CMake Tests Setup (#3244) Previously most of the tests even if compiled with MPI were run on only one process because they did not specify the number of tasks explicitly in their setup. For the three tests that did specify, they were run twice during CI, first on a single process and then on two processes, due to the setup in Tests/CMakeLists.txt. The setup was similar for OpenMP. This has changed. For tests compiled with OMP, they are run with two OMP threads, otherwise they are run with two MPI processes if they are built with MPI. This should make the CI tests faster. .github/workflows/clang.yml .github/workflows/macos.yml Tests/CMakeLists.txt Tests/Particles/GhostsAndVirtuals/CMakeLists.txt Tests/Particles/InitFromAscii/CMakeLists.txt Tests/Particles/NeighborParticles/CMakeLists.txt Tests/Particles/ParallelContext/CMakeLists.txt Tests/Particles/ParticleReduce/CMakeLists.txt Tests/Particles/Redistribute/CMakeLists.txt Tests/Particles/TypeDescriptor/CMakeLists.txt commit cfc4c822e3c5747c3212b84fcb9eda33302bfff0 Author: Axel Huebl Date: Fri Apr 7 16:27:17 2023 -0700 ParticleTileData: No Restrict in Storage (#3245) ## Summary Remove the `restrict` qualifier from `GpuArray` members in `ParticleTileData`. We see compilation problems with, among others Clang 14.0.6 and AppleClang, for this construct. We expect that using `restrict` (`AMREX_RESTRICT`) on APIs as well as aliased pointers in these arrays before access should be sufficient to prevent that the compiler thinks that individual arrays alias each other in hot loops. ## Additional background - WarpX build regressions since 23.03, e.g., - https://github.com/conda-forge/warpx-feedstock/pull/63 - https://github.com/conda-forge/warpx-feedstock/pull/64 - https://github.com/ECP-WarpX/WarpX/pull/3817 - independently, about to be remove in https://github.com/AMReX-Codes/amrex/pull/2878 Src/Particle/AMReX_ParticleTile.H commit eab8beead3e9a8d7cb2d34e29429144d272a8ce2 Author: Andrew Myers Date: Fri Apr 7 13:12:58 2023 -0700 Fix periodic boundary bug in #3199 (#3243) PR #3199 broke the Nyx regression tests. The issue is we require the particle positions to be in [plo, phi). This won't happen if rhi is exactly equal to phi. The fix is to make sure rhi is always slightly inside the domain. EDIT - fixed this in a different way. Now, when applying periodic boundaries, the particle positions will be set to be strictly less than rhi. Src/Particle/AMReX_ParticleUtil.H commit 2f68f00d5f56d03dc4c9930f688c89c5621c33b6 Author: Axel Huebl Date: Thu Apr 6 19:30:39 2023 -0700 CI: Retry Apt Repos (#3242) Since Intel Repos are often out-of-sync with each others for apt packages provided (checksum errors), we try to pause and retry up to five times to get them installed. Also set apt retries to 3 for `apt update`. .github/workflows/dependencies/dependencies.sh .github/workflows/dependencies/dependencies_clang-tidy.sh .github/workflows/dependencies/dependencies_clang.sh .github/workflows/dependencies/dependencies_dpcpp.sh .github/workflows/dependencies/dependencies_gcc.sh .github/workflows/dependencies/dependencies_hip.sh .github/workflows/dependencies/dependencies_nofortran.sh .github/workflows/dependencies/dependencies_nvcc11.sh .github/workflows/dependencies/dependencies_nvcc12.sh .github/workflows/dependencies/dependencies_nvhpc.sh .github/workflows/dependencies/documentation.sh commit 7d6d87a24988c27a9f0f721917f9b815ab4bf267 Author: haymanpf <73078585+haymanpf@users.noreply.github.com> Date: Fri Apr 7 12:29:46 2023 +1200 Silence SWFFT (#3240) This PR adds a check for AMReX verbosity level to SWFFT output statements. ## Additional background SWFFT somewhat impolitely outputs a lot of information to stdout and stderr. Certainly it's less than ideal to modify external code, but for anyone running an FFT more than once per run, the excess output can be troublesome. Co-authored-by: Peter Hayman Src/Extern/SWFFT/Make.package Src/Extern/SWFFT/TimingStats.h Src/Extern/SWFFT/distribution.c Src/Extern/SWFFT/verbosity.cpp Src/Extern/SWFFT/verbosity.h commit d7052c6751e8c4f7a9712006d4302dc34056675b Author: aveksler1 <124003120+aveksler1@users.noreply.github.com> Date: Thu Apr 6 16:38:05 2023 -0700 Support complete elliptic integrals of 1st and 2nd type in Parser (#3225) We want to use complete elliptic integrals from `cmath`, `comp_ellint_1` and `comp_ellint_2` in the parser in order to make it easier to load in magnetic fields from coils in WarpX. I added these to the parser, and tested against the same functions in scipy - the results match. Since clang does not support special functions in C++17 (https://en.cppreference.com/w/cpp/compiler_support/17#C.2B.2B17_library_features) I implemented it for GCC compilers only, following the structure of how `jn` is implemented in the parser. Co-authored-by: Avigdor Veksler Docs/sphinx_documentation/source/Basics.rst Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_iparser.lex.cpp Src/Base/Parser/amrex_iparser.lex.h Src/Base/Parser/amrex_iparser.lex.nolint.H Src/Base/Parser/amrex_iparser.tab.h Src/Base/Parser/amrex_iparser.tab.nolint.H Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp Src/Base/Parser/amrex_parser.lex.h Src/Base/Parser/amrex_parser.lex.nolint.H Src/Base/Parser/amrex_parser.tab.h Src/Base/Parser/amrex_parser.tab.nolint.H commit ce3401fb0bbed39a879bf5f571c63a1824569613 Author: David Grote Date: Thu Apr 6 14:28:57 2023 -0700 Rework handling of roundoff domain extent (#3199) ## Summary This reworks previous changes that created the round off domain which was done to avoid round off errors when locating particles in the domain, specifically when handling periodic boundaries. This PR changes how this is done so that the roundoff domain makes as small a change as possible. ## Additional background Previous PRs #2839 and #2950 created a round off domain that has a slightly smaller extent that the actual domain. Any particles that ended up outside the round off domain but inside the actual domain were shifted to be on the boundary of the round off domain. The main purpose was to handle problems with periodic boundary with mixed precision (single precision particles and double precision grid). In that case, it is possible for a particle to have `z < lo` and `z+(hi-lo) > hi`. It is also possible for a particle to have `z < hi` but `(z-lo)*dzinv > ihi` putting it in a grid cell beyond the domain. Unfortunately, the roundoff domains was being set further inside that was needed and would affect valid particles that would not have those issues. While the problem is small, the effects can be noticeable when examining simulation properties, for example charge conservation in WarpX. This PR changes the roundoff domain so that only problematic particles are affected. The code now uses `std::nextafter` to find the exact place where the problems occurs. At the lower end, it ensures that the casted position will be greater than ProbLo. At the upper end, it ensures that `(z-lo)*dzinv <= ihi`. One comment is that this assumes that particle grid coordinates will always be calculated at the higher precision, with expressions like `iz = (int)(z - lo)`, without `lo` being cast to `ParticleReal`. Co-authored-by: atmyers Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp commit 1e73fa8ec7c31fbaf994fbb4f59792b90ae78433 Author: Axel Huebl Date: Tue Apr 4 21:43:15 2023 -0700 CI: oneAPI -Wmissing-braces (#3239) ## Summary The latest oneAPI release, 2023.1.10, ships with `-Wmissing-braces` warnings in the public MKL headers... ``` /opt/intel/oneapi/mkl/2023.1.0/include/oneapi/mkl/rng/device/detail/mrg32k3a_impl.hpp:163:33: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces] std::uint32_t temp[3][3] = {1, 0, 0, 0, 1, 0, 0, 0, 1}; ^~~~~~~ { } /opt/intel/oneapi/mkl/2023.1.0/include/oneapi/mkl/rng/device/detail/mrg32k3a_impl.hpp:163:42: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces] std::uint32_t temp[3][3] = {1, 0, 0, 0, 1, 0, 0, 0, 1}; ^~~~~~~ { } /opt/intel/oneapi/mkl/2023.1.0/include/oneapi/mkl/rng/device/detail/mrg32k3a_impl.hpp:163:51: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces] std::uint32_t temp[3][3] = {1, 0, 0, 0, 1, 0, 0, 0, 1}; ^~~~~~~ { } ``` ## Additional background Broke our CI. attn @rscohn2 FYI - let's report this? :) ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate .github/workflows/intel.yml commit bacaa10a76366ed40b90b422cf5c7da43565281a Author: Weiqun Zhang Date: Sun Apr 2 21:00:31 2023 -0700 Notes on Cache Cleanup (#3237) Add notes on cache cleanup workflows. This could be useful for debugging and reasoning about how things work. Fix a minor issue that only affects forks. To clean up cache left by push events for a feature branch on a fork, we should use github.event.workflow_run.head_branch (i.e., the feature branch) instead of github.ref (i.e., the default branch). This is because the cache is associated with the feature branch, whereas the cleanup workflow is run on the default branch. This is not a problem for the main repo because a direct push is not allowed. In the case of "indrect" push due to a PR merge, the head branch is the default branch for us because we have only one branch. .github/workflows/cache-cleanup-notes.md .github/workflows/cleanup-cache.yml commit 8325f8e2eeaa6cde20d30c4cbf12874f5cb1e548 Author: Weiqun Zhang Date: Sun Apr 2 20:56:58 2023 -0700 Adjust GitHub Cache Size (#3236) Testing using the current development branch shows that ccache will generate about 2 GB data, if we start from scratch. We have adjusted the cache size limit for individual jobs so that the whole set at its limit will use roughly 3 GB of space. This allows us to fit one default branch and 2 additional PR branches' data in the 10 GB available to us on GitHub. .github/workflows/clang.yml .github/workflows/cleanup-cache.yml .github/workflows/cuda.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/hypre.yml .github/workflows/intel.yml .github/workflows/petsc.yml .github/workflows/sundials.yml .github/workflows/windows.yml commit 604af7751cc5588c88101e647a54b468a4c3c742 Author: Weiqun Zhang Date: Fri Mar 31 19:13:10 2023 -0700 Ccache: Use depend mode for Intel SYCL compiler (#3234) The Intel SYCL compiler generates temporary files during compilations that result in cache misses for ccache. The ccache depend mode forgoes the preprocessed file information, and thus works around the issue. .github/workflows/intel.yml Tools/GNUMake/Make.rules Tools/GNUMake/comps/dpcpp.mak commit 615d0880406448bf124ed5a5af2b49152466e0f0 Author: Axel Huebl Date: Fri Mar 31 19:12:09 2023 -0700 Fix: queryAdd for resizing vectors (#3220) ## Summary I think there is a bug in `ParmParse::queryAdd` and `ParmParse::queryarr` for `std::vector` arguments. The passed array is only up-sized if needed, but not down-sized if the key exists. This leads to bugs for long default values and shorter user-provided arrays. Reproducer: ```C++ std::vector cos_coef = ez.default_cos_coef; // long default vector pp_element.queryAdd("cos_coefficients", cos_coef); // can be shorter user input amrex::Print() << "cos_coef.size()" << cos_coef.size() << "\n"; // size of the default vector ``` ## Additional background First seen in https://github.com/ECP-WarpX/impactx/pull/322 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Weiqun Zhang Src/Base/AMReX_ParmParse.H commit 03016933ab7c77fdd16f3de7920c0d1d8d7d14ad Author: Weiqun Zhang Date: Fri Mar 31 18:39:53 2023 -0700 GitHub Cache Management (#3229) The total size of all caches in a GitHub repo is limited to 10 GB. So we need to proactively manage the caches to avoid undesired cache eviction. GitHub cache has its scope associated with branch. When a new PR is opened, the last used cache of the default branch (i.e., development for AMReX) is restored for the PR's actions. After the actions finish, the caches will be saved under a unique name every time and only that PR has access to it. (Presumably this is for security reason to avoid cache poison.) If the actions are run many times, a large number of caches will be accumulated. This uses a lot of space, even though we only need to keep the most recent cache. In this PR, we implement the following approach to keep only the last used cache for each job and remove all caches associated with a PR when it's closed. We have added a workflow that is triggered by the events of completed workflows (e.g., LinuxClang workflow that has a number of jobs using cache.) This new workflow runs on the default branch and it's therefore safe to be granted write permission. Each triggering workflow saves the PR number as an artifact, if it's a pull request run. This is a way to pass the PR number to the cache cleanup workflow, which will clean up old caches associated with the jobs in the triggering workflow. When a PR is closed or merged, a post PR workflow is trigger, and it will in turn trigger another workflow that cleans up all the caches associated with the PR. After the merge, the actions will be run on the default branch. The newly added workflows will also clean up the old caches in the default branch. .github/workflows/clang.yml .github/workflows/cleanup-cache-postpr.yml .github/workflows/cleanup-cache.yml .github/workflows/cuda.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/hypre.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/petsc.yml .github/workflows/post-pr.yml .github/workflows/sundials.yml .github/workflows/windows.yml commit c0e81dc9d0ace42fb412632d7902a3ad98db5dee Author: Axel Huebl Date: Fri Mar 31 18:24:08 2023 -0700 CI: CMake-Easyinstall Unused (#3230) ## Summary Not used anymore. ## Additional background git.io service is discontinued and shows elevated internal error rates. .github/workflows/dependencies/dependencies_hip.sh commit fb022f2cfe3a9a4a25919e3045ad7024e52237f1 Author: Weiqun Zhang Date: Fri Mar 31 17:40:28 2023 -0700 NVHPC CI (#3235) Use the default version whatever that is. Right now, if we explicitly ask for 23.1, the latest version (i.e., 23.3) gets installed as well. Then we run out of space in GitHub CI. .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_nvhpc.sh commit 0ec0d0dd3c0bf68ff61f5abf3493d4eaa4c70d7b Author: Weiqun Zhang Date: Fri Mar 31 14:47:26 2023 -0700 CodeQL: Add query filters (#3233) This should reduce the number of warnings and fix the SARIF error in CI. .github/workflows/codeql.yml .github/workflows/codeql/codeql-config.yml commit 5631b6ee4abdeb3eefac842f889270ed47c17f38 Author: Weiqun Zhang Date: Fri Mar 31 14:38:20 2023 -0700 Update CHANGES for 23.04 (#3232) CHANGES commit d781598bc94b4601c1799e0c15f46595ef204f35 Author: Weiqun Zhang Date: Thu Mar 30 10:41:10 2023 -0700 Speedup clang tidy (#3219) We make ccache save a log file, from which we find out which files need a rebuild. Instead of running clang-tidy on every file, we now only run it on those files that had a miss in ccache. This significantly speeds up clang-tidy in CIs. Note that we treat clang-tidy warnings as errors in CIs. So if there is a hit in the ccache's cache, the file is free of clang-tidy warnings. Also note that we set CCACHE_EXTRAFILES to the clang-tidy configuration file. So any changes to the config file will invalidate the cache as it should. If a CI job successfully passes the compilation stage with ccache enabled, but fails at the clang-tidy stage, the ccache's cache has been updated. One might think this will cause issues because we now have files that have been cached by ccache and meanwhile have failed the clang-tidy checks. However, this is not an issue for Github CIs. This is because ccache's cache will not be saved as a Github cache, if a CI fails. Fix various clang-tidy warnings in Tests/. They were not reported before because clang-tidy was not used on the tests. .github/workflows/clang.yml .github/workflows/gcc.yml .github/workflows/hypre.yml .github/workflows/macos.yml .github/workflows/petsc.yml .github/workflows/sundials.yml Src/AmrCore/AMReX_AmrParticles.H Src/Base/AMReX_Any.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_Lazy.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/Parser/AMReX_Parser_Exe.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_IF_Complement.H Src/EB/AMReX_EB2_IF_Difference.H Src/EB/AMReX_EB2_IF_Extrusion.H Src/EB/AMReX_EB2_IF_Lathe.H Src/EB/AMReX_EB2_IF_Plane.H Src/EB/AMReX_EB2_IF_Polynomial.H Src/EB/AMReX_EB2_IF_Rotation.H Src/EB/AMReX_EB2_IF_Scale.H Src/EB/AMReX_EB2_IF_Translation.H Src/Particle/AMReX_BinIterator.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleArray.H Src/Particle/AMReX_ParticleInterpolators.H Tests/Amr/Advection_AmrCore/Source/AdvancePhiAllLevels.cpp Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tests/Amr/Advection_AmrCore/Source/Src_K/Adv_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/compute_flux_2D_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/compute_flux_3D_K.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d_K.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Amr/Advection_AmrLevel/Source/LevelBldAdv.cpp Tests/AsyncOut/multifab/main.cpp Tests/EB/CNS/Source/CNS.H Tests/EB/CNS/Source/CNS.cpp Tests/EB/CNS/Source/CNSBld.cpp Tests/EB/CNS/Source/CNS_advance.cpp Tests/EB/CNS/Source/CNS_init_eb2.cpp Tests/EB/CNS/Source/main.cpp Tests/LinearSolvers/ABecLap_SP/MyTestPlotfile.cpp Tests/LinearSolvers/NodalPoisson/MyTestPlotfile.cpp Tests/LinearSolvers/NodeTensorLap/CMakeLists.txt Tests/MultiBlock/Advection/main.cpp Tests/MultiBlock/IndexType/main.cpp Tests/Parser/main.cpp Tests/Particles/AssignDensity/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/AsyncIO/main.cpp Tests/Particles/CheckpointRestart/main.cpp Tests/Particles/DenseBins/main.cpp Tests/Particles/InitFromAscii/main.cpp Tests/Particles/Intersection/main.cpp Tests/Particles/NeighborList/main.cpp Tests/Particles/NeighborParticles/MDParticleContainer.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/main.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleIterator/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/main.cpp Tests/Particles/ParticleReduce/main.cpp Tests/Particles/ParticleTransformations/main.cpp Tests/Particles/Redistribute/main.cpp Tests/Particles/SparseBins/main.cpp Tools/C_scripts/mmclt.py Tools/GNUMake/Make.rules Tools/Plotfile/fboxinfo.cpp Tools/Plotfile/fcompare.cpp Tools/Plotfile/fextract.cpp Tools/Plotfile/fsnapshot.cpp commit b41c44166e12705cc08657d6bc33744f23bc8741 Author: Weiqun Zhang Date: Wed Mar 29 13:23:08 2023 -0700 Use ccache in Github Actions (#3218) This works very well for CUDA and HIP builds. However, the benefit is relatively small for GCC, Clang and Mac builds because most of their time are spent on clang-tidy and running tests. There are two remaining issues. Ccache works with Intel classic compiler icpc, but it does not work with the Intel SYCL compiler, icpx. On Windows, MSVC works, but I cannot get ClangCl to work. We can revisit these two issues in the future. In earlier versions of this PR, we compiled ccache from source because the package version on Ubuntu 20.04 is too old. Because of that, we had to move `env: {CXXFLAGS: ...}` down to AMReX's `Build & Install` section to avoid the flags being picked up when compiling ccache. In the current version, we download the binary version of ccache 4.8. However, we did not move the env setup back. It seems to make sense to keep it in the `Build & Install` section. To use github cache, we need to provide a primary key and optionally restore keys. If the primary key is found, it will be used, but the change to the cache during the CI will NOT be saved at the end of the job. That's how github cache works. If it is not found, partially matched restore keys will be considered and the latest one will be used. At the end, the cache (whether it's updated or not) will be saved under the name of the primary key. Below is an example of our setup. ``` with: path: ~/.cache key: ccache-${{ github.workflow }}-${{ github.job }}-git-${{ github.sha }} restore-keys: | ccache-clang.yml-${{ env.cache-name }}-git- ``` Here, we use the workflow string and job string as part of the names for the keys. We also include the git commit hash `${{github.sha}}` in the name of the primary key. In this setup, the CI will always fail to find the primary key. This is what we want because it guarantees the latest ccache data will be saved. Furthermore, the most recently saved cache from the last CI job will be used to restore the cache during initialization. .github/workflows/clang.yml .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_ccache.sh .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/hypre.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/petsc.yml .github/workflows/sundials.yml .github/workflows/windows.yml commit 831d1487fa19feb04b4330976a8dd90c604624c5 Author: asalmgren Date: Tue Mar 28 21:13:53 2023 -0700 fixes to make clang-tidy happy (#3227) Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EBDataCollection.cpp commit 4478d678c13ea00f1163385e05fd5cd09bf9d78e Author: asalmgren Date: Tue Mar 28 16:47:47 2023 -0700 fix for clang-tidy (#3224) Src/Base/AMReX_FabArray.H commit 6f10c63358d1e7d93dbce000dad1d09ae77c0931 Author: hsitaram Date: Tue Mar 28 15:49:20 2023 -0600 bug fix in creating EB from STL files (#3223) ## Summary Triangle data from an STL file needs to be broadcasted across MPI ranks ## Additional background The STL file is currently read only by ioprocessor and the tri_data on host is set only on ioprocessor. FillFab function would only fill the correct data for processor 0 Co-authored-by: Hariswaran Sitaraman Src/EB/AMReX_EB_STL_utils.cpp commit 27ebdfbbbb428b27d1608677f1afed712bb4bda6 Author: asalmgren Date: Tue Mar 28 13:49:46 2023 -0700 fixes suggested by clang-tidy (#3222) Src/Base/AMReX_BCRec.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_IntegratorBase.H commit 4fab19a02359d05f011b5a5ee4aacfda4b86f640 Author: Lucas Esclapez <13371051+esclapez@users.noreply.github.com> Date: Mon Mar 27 08:56:27 2023 -0700 Fix bug in MLEBABecLap getEBFluxes(). (#3217) Bug introduced when adding anisotropy to MLEBABecLap, bareascaling goes on the denominator when calculating the EBfluxes. Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H commit 7aad6ca7f64956c136b68d5fedfbc5e28ab1aab7 Author: Weiqun Zhang Date: Thu Mar 23 10:44:32 2023 -0700 Add comments to FluxRegister::OverwriteFlux (#3216) Src/AmrCore/AMReX_FluxRegister.H commit 82e38a8f5ebef4622805db256183cdf01970326d Author: Weiqun Zhang Date: Wed Mar 22 16:53:51 2023 -0700 Add build option for Intel SYCL AOT GRF mode (#3173) By default, the Intel SYCL compiler uses up to 128 registers. In the large register file mode, it's up to 256 registers. There is also the auto-large mode. For CMake, one can use `-DAMReX_SYCL_AOT_GRF_MODE=[Large|AutoLarge]` to choose the large register file or auto-large mode. AOT must also be enabled with `-DAMReX_SYCL_AOT=TRUE`. For GNU Make, one can use `SYCL_AOT_GRF_MODE=[Large|AutoLarge]`, and it depends on AOT being enabled by `SYCL_AOT=TRUE`. Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit ea670122d5956506aa66827193ad601c86b72d06 Author: Weiqun Zhang Date: Wed Mar 22 15:38:48 2023 -0700 SYCL sub-group size (#3180) Add option to specify SYCL sub-group size. The default is 32. It can be changed to 16 with `-DAMReX_SYCL_SUB_GROUP_SIZE=16` in CMake and `SYCL_SUB_GROUP_SIZE=16` in GNU Make. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_GpuDevice.H Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs commit f7d36cb62ca480c41d866e7db702064ec4912539 Author: Weiqun Zhang Date: Wed Mar 22 14:56:19 2023 -0700 Warning about CPU core oversubscription (#3211) If the user forgets to set OMP_NUM_THREADS, OMP might be oversubscribing CPU cores. If that's the case (to the best of our knowledge) and verbosity is on, we print out a warning. We use `std::thread::hardware_concurrency` to detect the number of cores available. Note that this is only a hint according to the C++ standard. To detect the number of MPI ranks per node, we use OMP_COMM_TYPE_NODE for Open MPI and MPI_COMM_TYPE_SHARED for others. Since MPI_COMM_TYPE_SHARED might group processes across nodes, the return value of `ParallelDescriptor::NProcsPerNode` might also be wrong. Nevertheless, both `std::thread::hardware_concurrency` and `ParallelDescriptor::NProcsPerNode` seem to work well. Src/Base/AMReX.cpp Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp commit 9d4ca1ed28c3833ca5564ab5e695c3aa5f40ad7f Author: Weiqun Zhang Date: Wed Mar 22 12:39:52 2023 -0700 Clang-Tidy VII (#3203) Fix warnings in the interfaces for Hypre, PETSc and SUNDIALS. .github/workflows/clang.yml .github/workflows/dependencies/dependencies_clang7.sh .github/workflows/hypre.yml .github/workflows/petsc.yml .github/workflows/sundials.yml Src/Base/AMReX_CuptiTrace.cpp Src/EB/AMReX_EB2_IF_Difference.H Src/EB/AMReX_EB2_IF_Lathe.H Src/EB/AMReX_EB2_IF_Scale.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_Hypre.cpp Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.H Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/AMReX_PETSc.cpp Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Src/Extern/SUNDIALS/AMReX_SUNMemory.H Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp Src/Extern/SUNDIALS/AMReX_Sundials_Core.H Src/Extern/SUNDIALS/AMReX_Sundials_Core.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleIO.H Tests/LinearSolvers/ABecLaplacian_C/MyTest.cpp Tests/LinearSolvers/ABecLaplacian_C/MyTestPlotfile.cpp Tests/LinearSolvers/CellEB/MyEB.H Tools/GNUMake/packages/Make.hypre Tools/GNUMake/packages/Make.petsc commit f22705a9a2c63e13e5abb0dd6d99dd3158c911ed Author: Weiqun Zhang Date: Tue Mar 21 18:29:07 2023 -0700 Clang-Tidy VI (#3202) * Update GNU Make. One can use `USE_CLANG_TIDY=TRUE` to enable clang-tidy check. For example, to enable clang-tidy check, specify the clang-tidy command (if it's the default clang-tidy), and treat warnings as errors, one can run make USE_CLANG_TIDY=TRUE CLANG_TIDY=clang-tidy-12 CLANG_TIDY_WARN_ERROR=TRUE By default amrex/.clang-tidy is used as the config file. One can change it with CLANG_TIDY_CONFIG_FILE=my_clang-tidy-conf. If you want to let clang-tidy fix errors for you, you can run make USE_CLANG_TIDY=TRUE CLANG_TIDY="clang-tidy --fix-errors" To avoid race conditions, the make job should not use `-j` (unless it's `-j1`) when fixing errors. * Add clang-tidy to GNU Make CIs. * Fix warnings in Fortran interface, particle and plotfile tools. .github/workflows/clang.yml .github/workflows/gcc.yml Src/AmrCore/AMReX_AmrParticles.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_Box.H Src/Base/AMReX_PODVector.H Src/F_Interfaces/AmrCore/AMReX_FAmrCore.H Src/F_Interfaces/AmrCore/AMReX_FAmrCore.cpp Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.H Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.cpp Src/F_Interfaces/AmrCore/AMReX_fillpatch_fi.cpp Src/F_Interfaces/Base/AMReX_FPhysBC.H Src/F_Interfaces/Base/AMReX_init_fi.cpp Src/F_Interfaces/LinearSolvers/AMReX_abeclaplacian_fi.cpp Src/F_Interfaces/LinearSolvers/AMReX_poisson_fi.cpp Src/F_Interfaces/Octree/AMReX_octree_fi.cpp Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_TracerParticles.H Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/sites/Make.unknown Tools/GNUMake/tools/Make.clang-tidy Tools/Plotfile/AMReX_PPMUtil.cpp Tools/Plotfile/fboxinfo.cpp Tools/Plotfile/fcompare.cpp Tools/Plotfile/fextrema.cpp Tools/Plotfile/fsnapshot.cpp Tools/Plotfile/fvolumesum.cpp commit d50cf55330b18a320754b5a85bbaca244bff62b7 Author: Weiqun Zhang Date: Tue Mar 21 11:36:58 2023 -0700 Clang-Tidy V (#3201) This is the fifth batch of changes fixing clang-tidy warnings. The changes are most related to single precision, 1D, 2D, and OMP. Enable clang-tidy in GCC and Clang CIs. Switch to Ubuntu 22.04 and GCC 12 in one test. .clang-tidy .github/workflows/clang.yml .github/workflows/dependencies/dependencies_clang-tidy.sh .github/workflows/dependencies/dependencies_clang.sh .github/workflows/dependencies/dependencies_gcc.sh .github/workflows/dependencies/dependencies_gcc8.sh .github/workflows/gcc.yml .github/workflows/sundials.yml Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_AuxBoundaryData.cpp Src/Amr/AMReX_Extrapolater.H Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/Amr/AMReX_extrapolater_2D_K.H Src/Amr/AMReX_extrapolater_3D_K.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_InterpFaceReg_2D_C.H Src/AmrCore/AMReX_InterpFaceReg_3D_C.H Src/AmrCore/AMReX_InterpFaceRegister.cpp Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/Base/AMReX_BaseFab.cpp Src/Base/AMReX_Box.H Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp Src/Base/AMReX_CoordSys.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FilCC_C.cpp Src/Base/AMReX_FileSystem.H Src/Base/AMReX_FileSystem.cpp Src/Base/AMReX_Geometry.H Src/Base/AMReX_Gpu.H Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuKernelInfo.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_LayoutData.H Src/Base/AMReX_MFParallelForC.H Src/Base/AMReX_MFParallelForG.H Src/Base/AMReX_MPMD.H Src/Base/AMReX_Math.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H Src/Base/AMReX_MultiFabUtil_nd_C.H Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_OpenMP.H Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_PhysBCFunct.cpp Src/Base/AMReX_RealBox.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_RungeKutta.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_ccse-mpi.H Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_YAFluxRegister.H Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Graph.H Src/EB/AMReX_EB2_IF_AllRegular.H Src/EB/AMReX_EB2_IF_Box.H Src/EB/AMReX_EB2_IF_Complement.H Src/EB/AMReX_EB2_IF_Cylinder.H Src/EB/AMReX_EB2_IF_DevicePtr.H Src/EB/AMReX_EB2_IF_Difference.H Src/EB/AMReX_EB2_IF_Ellipsoid.H Src/EB/AMReX_EB2_IF_Extrusion.H Src/EB/AMReX_EB2_IF_Intersection.H Src/EB/AMReX_EB2_IF_Lathe.H Src/EB/AMReX_EB2_IF_Parser.H Src/EB/AMReX_EB2_IF_Plane.H Src/EB/AMReX_EB2_IF_Polynomial.H Src/EB/AMReX_EB2_IF_Rotation.H Src/EB/AMReX_EB2_IF_Scale.H Src/EB/AMReX_EB2_IF_Sphere.H Src/EB/AMReX_EB2_IF_Spline.H Src/EB/AMReX_EB2_IF_Torus.H Src/EB/AMReX_EB2_IF_Translation.H Src/EB/AMReX_EB2_IF_Union.H Src/EB/AMReX_EB2_IndexSpace_STL.H Src/EB/AMReX_EB2_IndexSpace_chkpt_file.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level_STL.H Src/EB/AMReX_EB2_Level_chkpt_file.H Src/EB/AMReX_EB2_MultiGFab.H Src/EB/AMReX_EBFluxRegister.cpp Src/EB/AMReX_EBFluxRegister_2D_C.H Src/EB/AMReX_EBFluxRegister_3D_C.H Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_EB_LeastSquares_3D_K.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_chkpt_file.H Src/EB/AMReX_EB_chkpt_file.cpp Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_algoim.H Src/EB/AMReX_algoim_K.H Src/EB/AMReX_distFcnElement.cpp Src/Extern/HYPRE/AMReX_Habec_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMGBndry.H Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_eb.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp_grad.cpp Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp Src/LinearSolvers/OpenBC/AMReX_OpenBC_K.H Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_ParticleUtil.cpp Src/Particle/AMReX_Particle_mod_K.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.cpp commit 5f8030af0ddfe99139968bcf896b5662cccf94a4 Author: Michael Zingale Date: Sun Mar 19 18:34:07 2023 -0400 fix typo in debugging docs (#3210) ## Summary ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Docs/sphinx_documentation/source/Debugging.rst commit c4e2bbdf777e11bd6a99cd28cc00b409592c9002 Author: Weiqun Zhang Date: Fri Mar 17 20:28:10 2023 -0700 Add Jean's name to the development team (#3209) .zenodo.json commit fe0ae235cd39697b1d9d6bcc9df6bbac4e0ead86 Author: Weiqun Zhang Date: Fri Mar 17 14:31:57 2023 -0700 Fix issues in ParticleContainerBase (#3208) * The move constructor and assignment operator were not safe, because the m_gdb member could be the address of m_gdb_object. After the move that can no longer be true. This is fixed by making m_gdb_object a unique_ptr. Moving a unique_ptr does not change its pointer inside. * Fix SetParticleBoxArray, SetParticleDistributionMapping and SetParticleGeometry. The issue was these functions took a reference, which could be aliasing what was inside the m_gdb_object. ParGDB has a move assignment operator that could render the reference pointing to an invalid object. So now these functions take by value. This was not an issue before #3200, because the move assignment of ParGDB did copy instead of move. Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp commit 254d3faba017c9b13800d68499abcfe7d0405128 Author: Weiqun Zhang Date: Thu Mar 16 16:45:12 2023 -0700 Fix #3200 (#3206) It was a mistake to delete the move assignment operator of ParticleIDWrapper and ParticleCPUWrapper. Src/Particle/AMReX_Particle.H commit adcf4d4721b44bd47e4de614f62ddeffd6d71a47 Author: Weiqun Zhang Date: Thu Mar 16 08:18:22 2023 -0700 Clang-Tidy IV (#3200) This is the fourth batch of changes fixing clang-tidy warnings. Most of them are in particle code. Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrParGDB.H Src/Base/AMReX_GpuAtomic.H Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleBufferMap.H Src/Particle/AMReX_ParticleBufferMap.cpp Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleMPIUtil.cpp Src/Particle/AMReX_ParticleUtil.cpp Src/Particle/AMReX_StructOfArrays.H Src/Particle/AMReX_TracerParticles.H Src/Particle/AMReX_TracerParticles.cpp Src/Particle/AMReX_WriteBinaryParticleData.H commit 0f4f9877c81e827eb850ba5dc6f0980360552e04 Author: Weiqun Zhang Date: Wed Mar 15 13:59:10 2023 -0700 Clang-Tidy III (#3197) This is the third batch of changes fixing clang-tidy warnings. Most of the warnings are in EB code. Use Clang 14 in one of the Clang CI tests that used to use Clang 7. .clang-tidy .github/workflows/clang.yml .github/workflows/dependencies/dependencies_clang.sh Src/Amr/AMReX_AmrLevel.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_PlotFileUtil.cpp Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Graph.H Src/EB/AMReX_EB2_IF_Box.H Src/EB/AMReX_EB2_IF_Cylinder.H Src/EB/AMReX_EB2_IF_Ellipsoid.H Src/EB/AMReX_EB2_IF_Parser.H Src/EB/AMReX_EB2_IF_Sphere.H Src/EB/AMReX_EB2_IF_Torus.H Src/EB/AMReX_EB2_IndexSpace_STL.H Src/EB/AMReX_EB2_IndexSpace_STL.cpp Src/EB/AMReX_EB2_IndexSpace_chkpt_file.H Src/EB/AMReX_EB2_IndexSpace_chkpt_file.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EB2_Level_STL.cpp Src/EB/AMReX_EB2_Level_chkpt_file.cpp Src/EB/AMReX_EB2_MultiGFab.cpp Src/EB/AMReX_EB2_ND_C.cpp Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBCellFlag.cpp Src/EB/AMReX_EBFArrayBox.H Src/EB/AMReX_EBFArrayBox.cpp Src/EB/AMReX_EBFabFactory.H Src/EB/AMReX_EBFabFactory.cpp Src/EB/AMReX_EBFluxRegister.H Src/EB/AMReX_EBInterpolater.H Src/EB/AMReX_EBInterpolater.cpp Src/EB/AMReX_EBMFInterpolater.H Src/EB/AMReX_EBMFInterpolater.cpp Src/EB/AMReX_EBMultiFabUtil.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBToPVD.H Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_chkpt_file.H Src/EB/AMReX_EB_chkpt_file.cpp Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_MultiCutFab.H Src/EB/AMReX_MultiCutFab.cpp Src/EB/AMReX_WriteEBSurface.cpp Src/EB/AMReX_algoim.cpp Src/EB/AMReX_distFcnElement.H Src/EB/AMReX_distFcnElement.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_F.cpp Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp_bc.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp commit fdcbb7a0152baf96e81a500035fd139e9c11f40c Author: Andrew Myers Date: Wed Mar 15 13:58:18 2023 -0700 Add assert that we are not inside threaded region when InitRandom is called. (#3204) Src/Base/AMReX_Random.cpp commit d54067a4857042be858a36a346f5fe7a6fe7f99a Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Wed Mar 15 21:39:44 2023 +0100 Sorting for faster current deposition (#3198) ## Summary For current deposition in codes such as HiPACE++ and WarpX, it is mush better to sort by `x -> y -> z -> ppc` instead of `ppc -> x -> y -> z`. ## Additional background To use this, pass in {0,0,0} (sort by cell), {-1,-1,-1} (sort by node) or {-2, -2, -2} (sort by cell and node) as a bin_size into SortParticlesByBin. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H commit aaab804f1438b5eeef3f613ca527c62bcabb5425 Author: David Gardner Date: Mon Mar 13 19:29:44 2023 -0700 Update SUNDIALS NVector constructors (#3165) ## Summary Update MultiFab NVector wrappers to accept a sundials context object. ## Additional background Profiling (and in the future error handling) in sundials requires objects to be create with a non-NULL context. This PR updates the MultiFab NVector wrappers with an optional input for the sundials context that defaults to the context created at initialization for the corresponding OpenMP thread. Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Src/Extern/SUNDIALS/AMReX_SUNMemory.H Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp Src/Extern/SUNDIALS/AMReX_Sundials.H Src/Extern/SUNDIALS/AMReX_Sundials_Core.H Src/Extern/SUNDIALS/AMReX_Sundials_Core.cpp Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package commit 6d30e83c944e4e6167178f2d145df2a3e67d2b24 Author: Weiqun Zhang Date: Fri Mar 10 20:48:40 2023 -0800 Clang-Tidy II (#3190) This is the second batch of changes fixing clang-tidy warnings. Add CMake option AMReX_CLANG_TIDY_WERROR to update warnings to errors. Use clang-tidy in CI: GNU@8.4 C++17 Release [lib]. .clang-tidy .github/workflows/gcc.yml Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_AuxBoundaryData.H Src/Amr/AMReX_AuxBoundaryData.cpp Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Derive.cpp Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FillPatcher.H Src/AmrCore/AMReX_InterpBase.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_TagBox.H Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_Array.H Src/Base/AMReX_BackgroundThread.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_Box.cpp Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabFactory.H Src/Base/AMReX_ForkJoin.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_LayoutData.H Src/Base/AMReX_MFCopyDescriptor.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_MPMD.H Src/Base/AMReX_MPMD.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBC.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_PCI.H Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParallelReduce.H Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_Utility.cpp Src/Base/AMReX_VisMF.H Src/Base/AMReX_VisMF.cpp Src/Base/Parser/AMReX_IParser.H Src/Base/Parser/AMReX_Parser.H Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_Mask.H Src/Boundary/AMReX_Mask.cpp Src/Boundary/AMReX_YAFluxRegister.H Src/Extern/amrdata/AMReX_AmrData.cpp Src/Extern/amrdata/AMReX_DataServices.cpp Src/F_Interfaces/Base/AMReX_init_fi.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp_F.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/OpenBC/AMReX_OpenBC.H Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp Tools/CMake/AMReXClangTidy.cmake Tools/CMake/AMReXOptions.cmake commit 5b7fa9867d0260fc3407ee9404927c5c99fc691e Author: Weiqun Zhang Date: Tue Mar 7 18:27:30 2023 -0800 Fix an issue in #3176 (#3188) We need to add another version of SetTag now, otherwise it breaks WarpX. Src/Base/AMReX_FabArray.H commit 041a3ce16cd274bbe3eff012ab4d2d78db009b31 Author: Weiqun Zhang Date: Tue Mar 7 13:42:27 2023 -0800 ReduceToPlane (#3187) * Add a function template for reducing MultiFab data to a plane. * Add KeyValuePair as an alias to ValLocPair. * Add more versions of MPI reduce functions for KeyValuePair/ValLocPair. Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_ParallelReduce.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_ValLocPair.H commit 184d997fa1a7023e7a79ce6ef887ed755ab9489d Author: Weiqun Zhang Date: Tue Mar 7 13:05:48 2023 -0800 Clang-Tidy (#3176) Add a configuration file .clang-tidy for clang-tidy. Add support to it in CMake. It can be enabled by `-DAMReX_CLANG_TIDY=ON`. Fix many warnings raised by clang-tidy. Most of them are fixed by `clang-tidy --fix-errors`. To bypass clang-tidy, some unfixable codes have been moved to `*.nolint.H`. A number of issues have been found and fixed. * There was use-after-move in testing OpenMP thread safety for communication. * The MultiFab tags were not handled correctly during vector resize. * Parentheses were missing in the definition of AMREX_REAL_LOWEST. .clang-tidy .github/workflows/style/check_tabs.sh .github/workflows/style/check_trailing_whitespaces.sh Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_AuxBoundaryData.cpp Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Derive.cpp Src/Amr/AMReX_Extrapolater.cpp Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FluxRegister.H Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_InterpBase.H Src/AmrCore/AMReX_InterpBase.cpp Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_TagBox.H Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_Algorithm.H Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_Array.H Src/Base/AMReX_Array4.H Src/Base/AMReX_AsyncOut.cpp Src/Base/AMReX_BCRec.H Src/Base/AMReX_BC_TYPES.H Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseFab.cpp Src/Base/AMReX_Box.H Src/Base/AMReX_Box.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxDomain.cpp Src/Base/AMReX_BoxIterator.H Src/Base/AMReX_BoxList.H Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp Src/Base/AMReX_CoordSys.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_DataAllocator.H Src/Base/AMReX_Demangle.H Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabConv.H Src/Base/AMReX_FabConv.cpp Src/Base/AMReX_FabFactory.H Src/Base/AMReX_FileSystem.H Src/Base/AMReX_FileSystem.cpp Src/Base/AMReX_ForkJoin.H Src/Base/AMReX_ForkJoin.cpp Src/Base/AMReX_Geometry.H Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuControl.cpp Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuElixir.H Src/Base/AMReX_GpuElixir.cpp Src/Base/AMReX_GpuKernelInfo.H Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunch.nolint.H Src/Base/AMReX_GpuLaunchMacrosC.H Src/Base/AMReX_GpuLaunchMacrosC.nolint.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuLaunchMacrosG.nolint.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_GpuRange.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_GpuUtility.cpp Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_IndexType.H Src/Base/AMReX_IntConv.cpp Src/Base/AMReX_IntVect.cpp Src/Base/AMReX_Loop.H Src/Base/AMReX_Loop.nolint.H Src/Base/AMReX_MFCopyDescriptor.H Src/Base/AMReX_MFCopyDescriptor.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_MPMD.H Src/Base/AMReX_MPMD.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBC.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_Orientation.H Src/Base/AMReX_PArena.cpp Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParallelReduce.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_Periodicity.cpp Src/Base/AMReX_PlotFileDataImpl.H Src/Base/AMReX_PlotFileDataImpl.cpp Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_PlotFileUtil.cpp Src/Base/AMReX_REAL.H Src/Base/AMReX_Random.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_Utility.H Src/Base/AMReX_Utility.cpp Src/Base/AMReX_Vector.H Src/Base/AMReX_VectorIO.cpp Src/Base/AMReX_VisMF.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_iMultiFab.cpp Src/Base/AMReX_parmparse_fi.cpp Src/Base/CMakeLists.txt Src/Base/Make.package Src/Base/Parser/AMReX_IParser.H Src/Base/Parser/AMReX_IParser.cpp Src/Base/Parser/AMReX_IParser_Exe.H Src/Base/Parser/AMReX_IParser_Exe.cpp Src/Base/Parser/AMReX_IParser_Y.H Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Exe.H Src/Base/Parser/AMReX_Parser_Exe.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/GNUmakefile Src/Base/Parser/amrex_iparser.lex.cpp Src/Base/Parser/amrex_iparser.lex.nolint.H Src/Base/Parser/amrex_iparser.tab.cpp Src/Base/Parser/amrex_iparser.tab.nolint.H Src/Base/Parser/amrex_parser.lex.cpp Src/Base/Parser/amrex_parser.lex.nolint.H Src/Base/Parser/amrex_parser.tab.cpp Src/Base/Parser/amrex_parser.tab.nolint.H Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_Mask.H Src/Boundary/AMReX_Mask.cpp Src/Boundary/AMReX_MultiMask.cpp Src/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/OpenBC/AMReX_OpenBC.H Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp Tools/CMake/AMReXClangTidy.cmake Tools/CMake/AMReXOptions.cmake commit 1ffc6e427e3aea9302602190c8fc1cfbea0c9725 Author: Weiqun Zhang Date: Mon Mar 6 08:48:05 2023 -0800 ParticleContainer: Resize runtime components (#3186) Add two new functions ResizeRuntimeRealComp and ResizeRuntimeIntComp to ParticelContainer. Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerI.H commit 6a7da99093780cc9b93b20e284f478018014ec63 Author: Nuno Miguel Nobre Date: Sun Mar 5 18:54:32 2023 +0000 SYCL: Tweak #3164 and switch to multi_ptr for sincos (#3185) * Use std::isnan and std::isinf instead of ::isnan and ::isinf for HIP and CUDA. * Add sycl::isinf and sycl::isfinite back to amrex::Math because Intel compiler's aggressive fp mode makes std::isinf and std::isfinite always return false in release build. * Use sycl::isinf and sycl::isnan in amrex::isnan and amrex::isinf when SYCL is on for the same reason as above. * Fix call to sycl::sincos. According to the SYCL standard the pointer should be a multi_ptr instead of a raw pointer. Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_Math.H commit b96f6e5bd13d156998edead7d92b21235bbea99d Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Wed Mar 1 20:33:34 2023 +0100 Fix race condition with TinyProfiler for memory (#3169) Fix #3156 by giving every Arena its own map for profiling. Src/Base/AMReX.cpp Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit 3ffd848fca27b8ca8a4938e44a9a0dd64095328a Author: Weiqun Zhang Date: Wed Mar 1 07:59:55 2023 -0800 Update CHANGES for 23.03 (#3183) CHANGES commit d0d2fb461204d1c014007ec669f109c5ba441ffd Author: Weiqun Zhang Date: Wed Mar 1 05:35:36 2023 -0800 SYCL: Fix #3172 (#3181) This may not be our bug. But using sycl::reqd_work_group_size does not work for the 2d range cases. So we have to remove that. Src/Base/AMReX_GpuLaunchMacrosG.H commit 32a65ffa8aba81ca6fc8e527cca1cff7e6a5c2ba Author: Michael Zingale Date: Mon Feb 27 10:46:26 2023 -0500 fix recently introduced tmp_build_dir build bug (#3179) the make variable was renamed from TmpBuildDir to TMP_BUILD_DIR breaking application codes. This adds an alias to the old make variable name so code compile again. addresses bug in #3177 closes #3178 Tools/GNUMake/Make.defs commit ef5c8b765f7821cf10f1040f122474b0344b796c Author: Miren Radia <32646026+mirenradia@users.noreply.github.com> Date: Sun Feb 26 21:13:53 2023 +0000 GNU Make: Allow customizing tmp_build_dir (#3177) ## Summary This allows the user to change the default temporary build directory to something other than `tmp_build_dir`. ## Additional background This would allow for users to build multiple AMReX applications using the same build directory and thus reduce compile times without having to use `ccache`. Of course, there might be a small risk in terms of users potentially accidentally mixing build configurations hence I would suggest not advertising this feature or at least advising caution. Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules commit 8014c80f478256adb99ac4bf04ca79e0f8ce3b79 Author: Miren Radia <32646026+mirenradia@users.noreply.github.com> Date: Sun Feb 26 21:10:35 2023 +0000 GNU Make: Allow choice of Intel compiler variant (#3175) ## Summary This allows the user to specify `COMP = intel-classic` or `COMP = intel-llvm` if they wish to use a specific one. ## Additional background Note that the existing behaviour is maintained if `COMP = intel` (i.e. intel-llvm is chosen if the `icpx` command exists and intel-classic is chosen otherwise following #3126). However, the `AMREX_FCOMP`, `AMREX_CCOMP` and `lowercase_comp` makefile variables are changed to one of `intel-llvm` or `intel-classic` depending on which is used. This further allows the user to use information about the used variant in their `Make.local`. For example they may wish to set something like ``` CXX = mpiicpc -cxx=icpx ``` in the case `AMREX_CCOMP = intel-llvm`, `USE_MPI = TRUE` and they are using Intel MPI. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [x] include documentation in the code and/or rst files, if appropriate Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/GNUMake/Make.defs Tools/GNUMake/comps/intel-classic.mak Tools/GNUMake/comps/intel-llvm.mak Tools/GNUMake/tools/Make.vtune Tools/libamrex/configure.py Tools/libamrex/mkconfig.py commit e42172a2bcd041bfad3df41af0b9c87ba574f93b Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Sun Feb 26 14:54:26 2023 -0500 Hypre setup and solve profile (#3168) I've wrapped the Hypre setup and solve functions so they can be more easily profiled. Co-authored-by: Paul Mullowney Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit 3d9bacd961c43a88ed6592d58ec091abf5ee3b90 Author: Weiqun Zhang Date: Sun Feb 26 05:10:01 2023 -0800 CMake: Parallel HDF5 Detection (#3170) ## Summary It turns out the words in the settings summary of HDF5 came from the words used when HDF5 was configured with CMake. The issue is `h5pcc -showconfig` may show `Parallel HDF5: ON`, but the current versions of CMake uses `Parallel HDF5: yes` to detect Parallel HDF5. This commit adds a more reliable way of detecting Parallel HDF5. ## Additional background - https://github.com/spack/spack/issues/35546 - https://github.com/HDFGroup/hdf5/issues/2488 - https://gitlab.kitware.com/cmake/cmake/-/merge_requests/8234 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Tools/CMake/AMReXThirdPartyLibraries.cmake commit 42acecf9c7f7651bda3b172eba71dfefc7fe2be9 Author: Weiqun Zhang Date: Thu Feb 23 08:27:38 2023 -0800 TableData: Relax the static assertion (#3174) Relax the requirement of T to trivially copyable (so that htod memcpy will work) and trivially destructible. After this change, TableData> should work. Src/Base/AMReX_TableData.H commit a3c4c9adc20852ef812785ee8d1422c78a3fdf10 Author: Weiqun Zhang Date: Wed Feb 22 19:41:29 2023 -0800 SYCL: Group and subgroup size (#3172) * Use [[sycl::reqd_work_group_size]] when we know the group size at compile time. I have not seen perfromance difference, but it should not hurt. * Replace [[intel::reqd_sub_group_size]] with [[sycl::reqd_sub_group_size]] since the Intel extension is in the standard now. Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_MFParallelForG.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_TagParallelFor.H commit 606589d8ccc2ac43cb6f9f25a54b53709c23d230 Author: Miren Radia <32646026+mirenradia@users.noreply.github.com> Date: Wed Feb 22 17:49:45 2023 +0000 Add Make.local-pre to gitignore (#3171) Given that `Make.local` files are in the `.gitignore`, this should probably be there too. .gitignore commit d8e6401642ff61aa72f1169ec1f004af406600f1 Author: Weiqun Zhang Date: Sun Feb 19 09:38:00 2023 -0800 New functions for getting data in cell or line. (#3166) Add get_cell_data and get_line_data to get the data in a cell or a line, respectively. Note that we already had get_slice_data. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_MultiFabUtil.H commit b1e9669ffc3e7ee05fdbaf20f4dd6ce4b4522326 Author: Weiqun Zhang Date: Sat Feb 18 09:46:16 2023 -0800 SYCL: Optimization of small reduction (#3167) For small reductions, it's faster to do the second pass on CPU. The change is for SYCL only. Src/Base/AMReX_Reduce.H commit 3e72725a55d0be2d0f26079261ed6a724b8ade14 Author: Weiqun Zhang Date: Fri Feb 17 16:00:28 2023 -0800 amrex::Math: No longer need to use sycl math functions (#3164) We introduced `amrex::Math::floor`, etc. because their std versions were not supported in device code by the Intel compiler at that time. Otherwise, the user would have to call either sycl::floor or std::floor with the help of macros. This is no longer the case. Now we can switch to use the std versions. The amrex::Math versions are kept for backward compatibility. The particle code is not updated because it's undergoing some large changes. Docs/sphinx_documentation/source/GPU.rst Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_InterpFaceReg_2D_C.H Src/AmrCore/AMReX_InterpFaceReg_3D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/Base/AMReX_Algorithm.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_COORDSYS_1D_C.H Src/Base/AMReX_COORDSYS_2D_C.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_GpuComplex.H Src/Base/AMReX_IntVect.H Src/Base/AMReX_Math.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil_nd_C.H Src/Base/AMReX_RealVect.H Src/Base/Parser/AMReX_IParser_Y.H Src/Base/Parser/AMReX_Parser_Y.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_chkpt_file.cpp Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_algoim_K.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/slope_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/slope_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/tagging_K.H Tests/EB_CNS/Source/CNS_K.H Tests/EB_CNS/Source/CNS_tagging.H Tests/EB_CNS/Source/hydro/CNS_divop_K.H Tests/EB_CNS/Source/hydro/CNS_hydro_K.H Tests/EB_CNS/Source/hydro/CNS_hydro_eb_K.H Tests/GPU/CNS/Source/CNS_K.H Tests/GPU/CNS/Source/CNS_tagging.H Tests/GPU/CNS/Source/hydro/CNS_hydro_K.H Tests/Particles/GhostsAndVirtuals/main.cpp commit 0a8eeb864a13ce3ad3aaee591bde18fdc94fd4f7 Author: Axel Huebl Date: Fri Feb 17 09:19:46 2023 -0800 gitignore: Editors & IDEs (#3163) ## Summary Add a few typical editor and IDE project directories to `.gitignore`. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate .gitignore commit 5729dd1c7eb2461f971584cbe1b50216727dfbcc Author: Weiqun Zhang Date: Thu Feb 16 14:09:55 2023 -0800 SYCL Device Info (#3161) Print out more SYCL device information if available. Currently, free memory and max memory band width are not available yet. Src/Base/AMReX_GpuDevice.cpp commit cbe2b291d3edd114dad171eaa6786150fb6684f4 Author: Weiqun Zhang Date: Mon Feb 13 15:51:57 2023 -0800 SYCL AOT: Fix warning (#3159) Finally set the proper flags without generating warnings about unused compiler arguments. Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit 2f88f61297d0048d3266b41c27ecf453f67160c5 Author: Weiqun Zhang Date: Mon Feb 13 13:54:53 2023 -0800 GNU Make: make it an error if intel gpu arch for aot is unknown (#3157) This is a follow-up on #3155 to make it consistent with CMake, and also because `-device *` does not actually work on the testbed. Tools/GNUMake/comps/dpcpp.mak commit e45a6a624be02f4f5bd1fdf759df778d469678bf Author: Weiqun Zhang Date: Mon Feb 13 10:03:42 2023 -0800 SYCL: Fix AOT (#3155) This reverts part of #3123 because the AOT argument is a compile option. Also we must provide AMReX_INTEL_ARCH when SYCL AOT is on. Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit 774f5443a7703524a9a054a59d2efa955fdde3c9 Author: Weiqun Zhang Date: Sun Feb 12 11:29:49 2023 -0800 CI: Add sundials (#3153) Also fixes warnings. .github/workflows/dependencies/dependencies_nvcc12.sh .github/workflows/sundials.yml Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Src/Extern/SUNDIALS/AMReX_SUNMemory.H Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp commit dbe1ebd464103524a14a093ff4504f58aadede51 Author: Weiqun Zhang Date: Sat Feb 11 19:12:57 2023 -0800 Add static_assert to Mpi_typemap (#3154) Src/Base/AMReX_ParallelDescriptor.H commit bb39ad028a071e0265b55f68b3b9444abb574dd9 Author: Weiqun Zhang Date: Sat Feb 11 11:07:58 2023 -0800 SUNDIALS: Replace SUNDIALS math macros with std math functions (#3151) This fixes https://github.com/xsdk-project/xsdk-issues/issues/211. Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp commit e52fb91f805b2b076fb81b72b00fa30eb93e1a44 Author: asalmgren Date: Fri Feb 10 18:00:16 2023 -0800 Fix case where a norm is 1e-15 but not technically zero (#3149) Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H commit ccada32844229697d91ca4bdaf3a29cc7f5f87ed Author: Andrew Myers Date: Fri Feb 10 17:15:07 2023 -0800 Fix bug in async IO for particles (#3150) Without this fix the following will fail: > amrex/Tests/Particles/NeighborParticles(development)$ make -j8 TINY_PROFILE=TRUE BL_NO_FORT=TRUE USE_MPI=FALSE DEBUG=TRUE > $ ./main3d.gnu.DEBUG.TPROF.CUDA.ex inputs amrex.async_out=1 amrex.the_arena_is_managed=0 > Initializing CUDA... > CUDA initialized with 1 device. > AMReX (23.02-21-g858e72dcc651) initialized > 0::Assertion `index >= 0 && index < NReal + static_cast(m_runtime_rdata.size())' failed, file "../../..//Src/Particle/AMReX_StructOfArrays.H", line 43 !!! > SIGABRT The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_WriteBinaryParticleData.H commit d4822cc06b9afe1a6cd46027f936199959d9762c Author: Weiqun Zhang Date: Fri Feb 10 15:27:42 2023 -0800 DPCPP -> SYCL (#3140) The name dpcpp has been deprecated by Intel. This PR replaces DPCPP with SYCL. For backward compatibility, we have kept the AMREX_USE_DPCPP macro. However, build options like AMReX_DPCPP_AOT for cmake and DPCPP_AOT for GNU Make have been replaced by AMReX_SYCL_AOT and SYCL_AOT. Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Docs/sphinx_documentation/source/Introduction.rst Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.cpp Src/Base/AMReX_Arena.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseFabUtility.H Src/Base/AMReX_BlockMutex.H Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuAssert.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuAsyncArray.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuElixir.cpp Src/Base/AMReX_GpuError.H Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuRange.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_MFParallelForG.H Src/Base/AMReX_Math.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_Random.cpp Src/Base/AMReX_RandomEngine.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H Src/Base/AMReX_TagParallelFor.H Src/Base/Parser/AMReX_Parser_Y.H Src/EB/AMReX_algoim_K.H Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp_bc.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp Tests/GPU/AnyOf/GNUmakefile Tests/Parser/GNUmakefile Tests/Particles/DenseBins/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSYCL.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/sites/Make.unknown Tools/RegressionTesting/AMReX-dpcpp-tests.ini Tools/RegressionTesting/AMReX-sycl-tests.ini Tools/libamrex/configure.py Tools/libamrex/mkconfig.py commit 522f1bfb237c6967fe2ac228131e3b917242b7ea Author: Weiqun Zhang Date: Fri Feb 10 14:07:09 2023 -0800 Add MPI support to GpuComplex (#3148) This creates a specialization of Mpi_typemap for GpuComplex. Needed by https://github.com/ECP-WarpX/WarpX/pull/3618/ ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_ParallelDescriptor.H Src/EB/AMReX_distFcnElement.cpp commit 74a5398a941c9f9e864ed8a9d1706d4d895c7ca7 Author: Weiqun Zhang Date: Fri Feb 10 13:45:40 2023 -0800 GNU Make: Add support for Intel LLVM compiler in CPU builds (#3126) icpc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. Src/Base/AMReX_Extension.H Tools/GNUMake/comps/intel-classic.mak Tools/GNUMake/comps/intel-llvm.mak Tools/GNUMake/comps/intel.mak commit 087cf8be30e910414535f163d7d2e411328fe119 Author: Weiqun Zhang Date: Fri Feb 10 13:28:54 2023 -0800 SYCL: Fix the order of destruction (#3146) Although we have not observed any issues, we should destroy the SYCL context, device, and queues in the reverse order of their construction. Src/Base/AMReX_GpuDevice.cpp commit 9ecf6e47764a81d44e05b63eb49ec8e084e4944e Author: Weiqun Zhang Date: Fri Feb 10 13:28:01 2023 -0800 Adjust subgroup size for Intel GPUs (#3139) We can now use 32 as the subgroup size for Intel GPUs. This improves performance in most situations. Src/Base/AMReX_GpuDevice.H commit fc004c395e9ddb37389b04f37cdd165492a48fce Author: Weiqun Zhang Date: Fri Feb 10 13:27:03 2023 -0800 Runtime parameters for Signal handling controls (#3125) Add amrex.handle_sigsegv, handle_sigint, handle_sigabrt, and handle_sigfpe for more fine controls. Disable sigsegv handling for certain Intel GPUs because it's incompatible with the managed memory. Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_GpuDevice.cpp commit 489cadbe49759080911bd3b28f1423ca88ac2ba9 Author: Weiqun Zhang Date: Fri Feb 10 13:26:22 2023 -0800 FillPatcher: Update interpolation in time (#3106) The previous implementation was susceptible to roundoff errors. It was observed in 7 levels PeleC runs that the code could fail because the times in StateData were out of sync. The updated logic is closer to that in the FillPatch function and the StateData class. Src/AmrCore/AMReX_FillPatcher.H commit d322dc977efd49c1f2c293c7b4dae775bf78e1b1 Author: Weiqun Zhang Date: Fri Feb 10 12:58:46 2023 -0800 AmrLevel::FillPatch: Abort if grids are not properly nested for GPU or EB (#3098) Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp commit 858e72dcc65175e8e1ee5a8984fc611ba193182a Author: Weiqun Zhang Date: Fri Feb 10 11:54:48 2023 -0800 Fix FabArray::shift (#3147) ## Summary The cached metadata used by fused GPU kernel launches needs to be updated in FabArray::shift because the BoxArray has changed. ## Additional background Close #3145 Src/Base/AMReX_FabArray.H commit 285f5b5d10200127404d3b8b28642eec28798ef0 Author: Andrew Myers Date: Fri Feb 10 11:54:17 2023 -0800 Fix nl test (#3142) This fixes a bug introduced in PR #3060 related to building neighbor lists between particles of different types. I also made it so that CI tests would have caught this bug. Closes #3127 Src/Particle/AMReX_NeighborParticlesI.H Tests/Particles/NeighborList/CMakeLists.txt Tests/Particles/NeighborList/main.cpp commit 1f7a2f52b84dda058b2645a97f5a4135bf7a8b31 Author: Weiqun Zhang Date: Thu Feb 9 16:33:17 2023 -0800 Fix bug in Device::Finalize: Forgot to clear streams (#3143) ## Summary We forgot to clear the non-owning gpu_stream vector. They still held the old streams after we destroyed the streams in Device::Finalize. This bug affects codes that call amrex::Initialize and amrex::Finalize repeatedly. Note that we have the non-owning gpu stream vector for OMP CPU thread safety. The bug was introduced in #1064. ## Additional background #3141 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_GpuDevice.cpp commit 5cd5852f6276a5ac8f5f1bd99ef17531595d25df Author: Lucas Esclapez <13371051+esclapez@users.noreply.github.com> Date: Thu Feb 9 16:14:57 2023 -0800 Add a compute face-centered velocity gradient in MLEBTensor (#3133) Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H commit 23d6b40ef6102d8927a3d859cd7eaed36dbe25a9 Author: Axel Huebl Date: Thu Feb 9 14:35:39 2023 -0800 Test: Reinit AMReX (#3141) ## Summary Run a test that inits and finalizes AMReX multiple times. ```console cmake -S . -B build -DAMReX_GPU_BACKEND=CUDA -DAMReX_ENABLE_TESTS=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo cmake --build build -j 8 ctest --test-dir build --verbose -R Reinit ``` ```console build/Tests/Reinit cuda-gdb -ex r -ex bt --args ./Test_Reinit inputs ``` ## Additional background This is often done in pytest tests, where the same process is reused for multiple tests. Needed for pyAMReX, ImpactX, WarpX et al. This also demonstrates a bug for GPU testing. Tests/CMakeLists.txt Tests/Reinit/CMakeLists.txt Tests/Reinit/GNUmakefile Tests/Reinit/Make.package Tests/Reinit/inputs Tests/Reinit/main.cpp commit 0c444b898f8abc1f1f7dd81cb06e2b29faabb935 Author: Weiqun Zhang Date: Wed Feb 8 12:57:59 2023 -0800 Functional: Less -> Minimum, Greater -> Maximum (#3138) The functional classes for returning the lesser and greater of two arguments have been misnamed, because std::less and std::greater are for comparison and they return bool. To avoid confusion, they have been renamed. Src/Base/AMReX_Functional.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_ParallelReduce.H commit 188aaf6654bf27a1d4a0eec338ac044c60b58840 Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Wed Feb 8 19:57:17 2023 +0100 Initialize TinyProfiler earlier only for memory (#3137) Initialize TinyProfiler earlier only for memory and finalize it later only for memory as well. Src/Base/AMReX.cpp Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit 6dc1f61f3567614e67fb27bad3907eff398cc712 Author: Weiqun Zhang Date: Tue Feb 7 19:04:22 2023 -0800 Fix #3105: missing include (#3134) Src/Base/AMReX_TinyProfiler.H commit d6923bff490b80ebd807056ac49b52a1e1a22cd9 Author: Weiqun Zhang Date: Tue Feb 7 12:59:10 2023 -0800 PODVector: Add some stream synchronization (#3130) The correctness of some functions depends on the asynchronous behavior of memcpy (especially when pageable host memory is involved). It seems that that is not well defined in the SYCL standard. So for safety, stream synchronization is added to a few places. Src/Base/AMReX_PODVector.H commit 50d7cd1980c4c7657fc425e746918863020c110f Author: Weiqun Zhang Date: Tue Feb 7 12:55:25 2023 -0800 SENSIE CI: run on ubuntu-20.04. 18.04 is deprecated. (#3131) .github/workflows/sensei.yml commit 35dd973033f00e3ff1c4b86ef63037b8496699fe Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Tue Feb 7 19:50:02 2023 +0100 Profile memory allocations of the CArena using TinyProfiler (#3105) ## Summary The following per-section exclusive metrics of all CArenas are collected: * Nalloc Number of memory allocations inside a profiler section. * Nfree How many of the allocations counted in Nalloc where freed before `Finalize`. Works if the memory was freed in a different section compared to where it was allocated. * AvgMem The amount of memory allocated in a section averaged over execution time, taking into account how long it takes until the memory is freed again. * MaxMem The maximum amount of memory allocated by a section at any given time. In contrast to the other metrics, MaxMem is not additive between two sections, only satisfying the triangle inequality. If more the one rank is used, Nalloc and Nfree are summed between all ranks while for AvgMem and MaxMem the minimum, average and maximum value across all ranks is printed. Sample output from HiPACE++ (1 Rank) ``` Device Memory Usage: --------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem --------------------------------------------------------------- The_Device_Arena::Initialize() 1 1 4 B 8192 KiB --------------------------------------------------------------- Managed Memory Usage: --------------------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem --------------------------------------------------------------------------- The_Arena::Initialize() 1 1 43 KiB 29 GiB sortBeamParticlesByBox() 44 44 7377 MiB 11 GiB BeamParticleContainer::InitParticles 10 10 17 MiB 7390 MiB PlasmaParticleContainer::InitParticles 34 34 4554 MiB 4737 MiB Fields::AllocData() 3 3 2820 MiB 2820 MiB Hipace::ExplicitMGSolveBxBy() 45 45 1332 MiB 1366 MiB DenseBins::buildGPU 36 36 1182 MiB 1210 MiB FFTPoissonSolverDirichlet::define() 3 3 383 MiB 383 MiB AnyDST::CreatePlan() 2 2 255 MiB 255 MiB ResizeRandomSeed 1 1 40 MiB 40 MiB AdaptiveTimeStep::Calculate() 4 4 6 B 972 KiB hpmg::MultiGrid::solve1() 17320 17320 108 KiB 486 KiB Hipace::InitData() 7 7 273 B 243 KiB MultiPlasma::InitData() 2 2 0 B 243 KiB main() 3 3 79 B 80 B Fields::Copy() 1 1 62 B 64 B DepositCurrent_PlasmaParticleContainer() 10000 10000 1 B 16 B --------------------------------------------------------------------------- Pinned Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- Hipace::Evolve() 1 1 1682 MiB 1718 MiB The_Pinned_Arena::Initialize() 1 1 52 B 8192 KiB Hipace::ExplicitMGSolveBxBy() 1 1 998 B 1024 B sortBeamParticlesByBox() 24 24 127 B 256 B Hipace::InitData() 37 37 207 B 240 B main() 41 41 79 B 96 B Fields::Copy() 1 1 62 B 64 B AdaptiveTimeStep::Calculate() 4 4 0 B 32 B MultiPlasma::InitData() 2 2 0 B 16 B hpmg::MultiGrid::solve1() 17320 17320 6 B 16 B ----------------------------------------------------------------- Cpu (amrex_mempool) Memory Usage: ----------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------- amrex_mempool_init() 1 1 25 B 8192 KiB ----------------------------------------------------- ``` 4 Ranks: ``` Device Memory Usage: --------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max --------------------------------------------------------------------------------------------------------------------- The_Device_Arena::Initialize() 4 4 4 B 4 B 5 B 8192 KiB 8192 KiB 8192 KiB --------------------------------------------------------------------------------------------------------------------- Managed Memory Usage: -------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max -------------------------------------------------------------------------------------------------------------------------------- The_Arena::Initialize() 4 4 16 KiB 19 KiB 23 KiB 29 GiB 29 GiB 29 GiB sortBeamParticlesByBox() 657 657 1458 MiB 1915 MiB 2808 MiB 4221 MiB 5987 MiB 11 GiB MultiBeam::PackLocalGhostParticles() 10 10 0 B 500 MiB 2003 MiB 0 B 2078 MiB 8313 MiB BeamParticleContainer::InitParticles 10 10 0 B 1252 KiB 5010 KiB 0 B 1847 MiB 7390 MiB PlasmaParticleContainer::InitParticles 136 136 4556 MiB 4557 MiB 4557 MiB 4737 MiB 4737 MiB 4737 MiB Hipace::Wait() 301 301 1836 KiB 4227 KiB 5413 KiB 3695 MiB 3695 MiB 3695 MiB Fields::AllocData() 12 12 2820 MiB 2820 MiB 2820 MiB 2820 MiB 2820 MiB 2820 MiB Hipace::ExplicitMGSolveBxBy() 180 180 1011 MiB 1176 MiB 1363 MiB 1366 MiB 1366 MiB 1366 MiB DenseBins::buildGPU 744 744 240 MiB 283 MiB 325 MiB 605 MiB 605 MiB 605 MiB FFTPoissonSolverDirichlet::define() 12 12 383 MiB 383 MiB 383 MiB 383 MiB 383 MiB 383 MiB AnyDST::CreatePlan() 8 8 255 MiB 255 MiB 255 MiB 255 MiB 255 MiB 255 MiB ResizeRandomSeed 4 4 40 MiB 40 MiB 40 MiB 40 MiB 40 MiB 40 MiB AdaptiveTimeStep::Calculate() 66 66 2 B 3 B 5 B 972 KiB 972 KiB 972 KiB hpmg::MultiGrid::solve1() 154398 154398 86 KiB 99 KiB 111 KiB 486 KiB 486 KiB 486 KiB Hipace::InitData() 28 28 271 B 271 B 272 B 243 KiB 243 KiB 243 KiB MultiPlasma::InitData() 8 8 0 B 0 B 0 B 243 KiB 243 KiB 243 KiB Hipace::Notify() 196 196 0 B 0 B 2 B 128 B 128 B 128 B main() 12 12 79 B 79 B 79 B 80 B 80 B 80 B Fields::Copy() 4 4 47 B 54 B 63 B 64 B 64 B 64 B DepositCurrent_PlasmaParticleContainer() 80000 80000 1 B 1 B 1 B 16 B 16 B 16 B -------------------------------------------------------------------------------------------------------------------------------- Pinned Memory Usage: ---------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max ---------------------------------------------------------------------------------------------------------------------- Hipace::Notify() 245 245 554 MiB 932 MiB 1166 MiB 3051 MiB 3051 MiB 3051 MiB Hipace::Wait() 245 245 9 MiB 20 MiB 32 MiB 3051 MiB 3051 MiB 3051 MiB Hipace::Evolve() 4 4 317 MiB 369 MiB 428 MiB 429 MiB 429 MiB 429 MiB The_Pinned_Arena::Initialize() 4 4 21 B 24 B 27 B 8192 KiB 8192 KiB 8192 KiB Hipace::ExplicitMGSolveBxBy() 4 4 757 B 881 B 1022 B 1024 B 1024 B 1024 B sortBeamParticlesByBox() 576 576 287 B 287 B 287 B 416 B 416 B 416 B Hipace::InitData() 148 148 207 B 207 B 207 B 240 B 240 B 240 B main() 164 164 79 B 79 B 79 B 96 B 96 B 96 B Fields::Copy() 4 4 47 B 54 B 63 B 64 B 64 B 64 B AdaptiveTimeStep::Calculate() 66 66 0 B 0 B 0 B 32 B 32 B 32 B MultiPlasma::InitData() 8 8 0 B 0 B 0 B 16 B 16 B 16 B hpmg::MultiGrid::solve1() 154398 154398 5 B 5 B 7 B 16 B 16 B 16 B ---------------------------------------------------------------------------------------------------------------------- Cpu (amrex_mempool) Memory Usage: ----------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max ----------------------------------------------------------------------------------------------------------- amrex_mempool_init() 4 4 9 B 9 B 11 B 8192 KiB 8192 KiB 8192 KiB ----------------------------------------------------------------------------------------------------------- ``` ## Additional background A major difficulty in implementing this is that different omp threads could be in different functions. This means that each thread needs its own profiler function stack. Additionally multiple threads may share the same TinyProfiler object. To clean up the Profiler output, profiler sections were added to arena preallocate/initialize functions. Additionally `amrex::DeallocateRandomSeedDevArray()` was moved before `BL_TINY_PROFILE_FINALIZE()` so the deallocation is not missing in the profile. Src/Base/AMReX.cpp Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit a7322024263f62d779e7d7a905659de090267e20 Author: Weiqun Zhang Date: Mon Feb 6 10:30:07 2023 -0800 NeighborParticles: memcpy -> memcpy_async (#3129) There are no reasons to use memcpy, which contains a stream sync. Moreover, the scan function later has to do a stream sync internally because it has to allocate some temporary space. Src/Particle/AMReX_NeighborParticlesI.H commit e81a9c232c2608de90b3514ece0f211687b41cfe Author: Weiqun Zhang Date: Fri Feb 3 19:28:05 2023 -0800 Rename AMReX_CUDA_MAX_THREADS to AMReX_GPU_MAX_THREADS (#3115) Rename cmake option AMReX_CUDA_MAX_THREADS to AMReX_GPU_MAX_THREADS. This option can now be used for all GPU backends. GNU Make has also been updated for this. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_GpuControl.H Tools/CMake/AMReXCUDAOptions.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs commit 8595daac71aab852070f045856e1b28028e5f7b5 Author: Axel Huebl Date: Fri Feb 3 16:36:05 2023 -0800 NVHPC: CUDA (#3124) In my last commit #3122, I accidentially switched NVHPC to compile for OpenMP instead of the CUDA backend. .github/workflows/cuda.yml commit 48f5ffc629cfcded97f9214dabc551248ab11d26 Author: Weiqun Zhang Date: Fri Feb 3 14:50:33 2023 -0800 Update Build Systems for oneAPI (#3123) * For DEBUG builds, link with -fsycl-link-huge-device-code, otherwise it might crash due to device code larger than 2GB. * The AOT arguments are for the linker only. So they are removed from compile options. * Change the default of AMReX_INTEL_ARCH to * (i.e., all devices). This does not seem to increase link time or executable size that much. The movtivation for this is that the users do not have to set the device target to some magic numbers. * Add new CI to test EB. Also switch to use Intel MPI. * Fix a mistake in GNU Make. DPCPP_SPLIT_KERNEL can be used independent of AOT. .github/workflows/dependencies/dependencies_dpcpp.sh .github/workflows/intel.yml Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit 25fcdfc433ac747cf1022fa85f858b36cb771f45 Author: Axel Huebl Date: Thu Feb 2 17:01:13 2023 -0800 NVHPC: OpenMP Atomic Capture Fixed (#2378) Src/Base/AMReX_FabArrayUtility.H commit 2ad56772eac9095b9678de5ced1e5440c4e579c9 Author: Axel Huebl Date: Thu Feb 2 13:07:57 2023 -0800 CI: NVHPC 23.1 (#3122) Update CI to Test against the latest NVHPC release, v23.1 See #2378 .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_nvhpc23-1.sh Src/Particle/AMReX_ParticleArray.H commit e6e2fbc3cd753e82bcd95d213a4adf436f94796f Author: Weiqun Zhang Date: Wed Feb 1 17:36:48 2023 -0800 oneAPI: Work around compiler bug (#3121) This makes Tests/LinearSolvers/NodeEB/ 3D test work with oneAPI. Src/EB/AMReX_algoim_K.H commit 4e8ca9aa0667b81cdebd144c55f85e16e25f785d Author: Aaron M. Lattanzi <103702284+AMLattanzi@users.noreply.github.com> Date: Wed Feb 1 18:16:32 2023 -0700 Multi-grid NeighborList GPU Sync (#3120) Synchronizing at the end of building the neighbor list avoids memory mis-references on GPU with the multi-grid approach. Src/Particle/AMReX_NeighborList.H commit 78dd7a0585840c2920b4e86d3e355a8b156d8497 Author: Andrew Myers Date: Wed Feb 1 13:06:25 2023 -0800 Modify the neighbor particles test to also test selectActualNeighbors. (#3118) The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_NeighborParticlesI.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/inputs Tests/Particles/NeighborParticles/main.cpp commit c09ed7de228d9c20d2240f0b732ca3535df13691 Author: Weiqun Zhang Date: Wed Feb 1 13:05:21 2023 -0800 Remove Atomic::Inc and Dec (#3117) The implementation for Atomic::Inc and Dec in our SYCL backend use compare and exchange, because they do not exist in SYCL standard. This turns out to be very inefficient. Although both CUDA and HIP have efficient implementation of atomicInc and atomicDec, we are removing Atomic::Inc and Dec from amrex to avoid surprises. In all cases that we have used these functions, they can be replaced with Atomic::Add. Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_Scan.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleCommunication.H commit eefe2463884081a27025973d4a99dc909657b4f0 Author: Weiqun Zhang Date: Wed Feb 1 09:09:38 2023 -0800 Update CHANGES for 23.02 (#3119) CHANGES commit 225c605680e825e1f0793780746b4112d4262381 Author: Cyrus Harrison Date: Fri Jan 27 15:15:46 2023 -0800 update github actions to use ascent 0.9.0 release (#3114) Updates ascent in situ github actions CI test to use new ascent release 0.9.0. .github/workflows/ascent.yml commit b45ea138416ef76e1c3122dd08fe54da04645427 Author: Weiqun Zhang Date: Fri Jan 27 13:24:48 2023 -0800 Documentation: Masks (#3113) Add documentation on various masks. Docs/sphinx_documentation/source/Basics.rst commit 756468d421064b03605dafd262e8b707e751d0b6 Author: Axel Huebl Date: Thu Jan 26 08:48:55 2023 -0800 CMake 3.18+: Stay OLD CUDA_ARCHITECTURES Policy (#3112) ## Summary For now... We removed this guard in WarpX & ImpactX a while ago. Need to revisit how we want to add the logic, probably by jumping to CMake 3.20+ requirements and ditching legacy CUDA logic. ## Additional background Regression from switching to CMake 3.18+, which then enables new policies. Introduced in #3107 - https://cmake.org/cmake/help/latest/policy/CMP0104.html - https://cmake.org/cmake/help/v3.24/prop_tgt/CUDA_ARCHITECTURES.html#prop_tgt:CUDA_ARCHITECTURES ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate CMakeLists.txt Tools/CMake/AMReXUtils.cmake commit a232be4e275bec58ab26f302f6423e736c75a840 Author: Weiqun Zhang Date: Tue Jan 24 13:54:53 2023 -0800 OpenBCSolver: Fix reuse (#3111) When the open BC solver is reused, we must update the level BC. Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp commit a83c130b34ffca97e43e7451e2df6a948830fd54 Author: Axel Huebl Date: Tue Jan 24 12:53:29 2023 -0800 CMake: Fix AMD Flags for ROCm 5.5 and Bump cmake minimum version to 3.18 (#3107) ## First Commit: Shell-Escape Tokens - [x] The new ROCm 5.5 flag `-Xoffload-linker --whole-archive` is a flag with an argument. CMake will deduplicate the command line if it encounters another `-Xoffload-linker` on it, which will break compilations in the future. This marks the whole string as a token that can only be deduplicated as a whole. Ref.: https://cmake.org/cmake/help/v3.25/command/target_link_options.html#option-de-duplication ## Second Commit: Only ROCm 5.5+ - [x] Do not add the flags `-Xoffload-linker --whole-archive` in earlier ROCm releases, avoiding compile-time warnings. ## Third Commit: Clean Up CI - [x] Remove warning suppression introduced in #3097 ## Additional background Follow-up to @david-salinas's #3097 .github/workflows/hip.yml CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/BuildingAMReX_Chapter.rst Tests/CMakeTestInstall/CMakeLists.txt Tests/SpackSmokeTest/CMakeLists.txt Tools/CMake/AMReXParallelBackends.cmake commit 1c18d00a48a685441636f2e6e90715817d55ee97 Author: Marc T. Henry de Frahan Date: Tue Jan 24 09:34:36 2023 -0700 Check component and derive names for `canDerive` and `get` (#3109) `canDerive` and `get` in the `DeriveList` class only check the derive name but not the variable component names. This code adds a check on the variable names. @esclapez did this internally for PeleLMeX and we need it for PeleC as well so I thought this should just belong in AMReX. Src/Amr/AMReX_Derive.cpp commit e472c108e72174112d387e9828f272277a16f68e Author: Weiqun Zhang Date: Mon Jan 23 21:08:51 2023 -0800 CMake clean up (#3100) * Clean up oneAPI compiler setup. We no longer support early beta versions. * Move AMREX_USE_EB macro definition from AMReXSetDefines.cmake to Src/EB/CMakeLists.txt. The former is not supposed to set defines for optional components (as mentioned in its comments). * AMReXParallelBackends.cmake: unset temporary variable _genex. * Update documentation for CMAKE_CXX_STANDARD. C++14 is no longer supported. * Update oneAPI CI test. Use icx and ifx instead of clang and gfortran for C and Fortran, respectively. .github/workflows/dependencies/dependencies_dpcpp.sh .github/workflows/intel.yml Docs/sphinx_documentation/source/BuildingAMReX.rst Src/EB/CMakeLists.txt Tools/CMake/AMReXFlagsTargets.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSYCL.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXUtils.cmake commit d61ec5efdeb698edaece3986a332c24700d406c5 Author: Weiqun Zhang Date: Mon Jan 23 17:36:41 2023 -0800 Replace sprintf with snprintf to fix compiler warnings (#3108) Tests/HDF5Benchmark/main.cpp Tests/Particles/CheckpointRestart/main.cpp commit b487434b948fcdc848498062cb094ce53d5414ad Author: Weiqun Zhang Date: Thu Jan 19 10:26:21 2023 -0800 Fix comment for OwnerMask (#3102) Src/Base/AMReX_iMultiFab.H commit 12730dec3e9a64dba5f7585b360b470a5d722b41 Author: Weiqun Zhang Date: Thu Jan 19 08:43:34 2023 -0800 ParmParse: support nan and inf (#3101) When the ParamParse table is dumped, if a floating point number is a NaN, it will be printed as nan. If it's infinity, it will be printed as inf. However, if one tried to read it back, the code will crash because ParmParse did not recognize nan and inf as valid floating point numbers. This commit adds support for nan and inf in ParmParse. Src/Base/AMReX_ParmParse.cpp commit a5063a5d705b5192c772d5b969d13a36b97da0da Author: Axel Huebl Date: Tue Jan 17 16:22:15 2023 -0800 CI: actions/checkout@v3 (#3099) Update leftover actions that use the deprecated v2. .github/workflows/ascent.yml .github/workflows/style.yml commit d6f8b2cce0fba1967485902b100bbfc80cb07d35 Author: David Salinas Date: Fri Jan 13 17:30:21 2023 -0500 CMake: add offload linker option --whole-archive for ROCm hipcc (#3097) ## Summary Hipcc linking will be relying on clang to offload all code objects from archives, so we need to add -Xoffload-linker --whole-archive option. ## Additional background Existing hipcc would offload code objects in archive files. Now it will rely on clang to do this, but we need to tell the offload linker to include all symbols/functions in an archive file. Co-authored-by: David Salinas Co-authored-by: Weiqun Zhang .github/workflows/hip.yml Tools/CMake/AMReXParallelBackends.cmake commit d5ddf3b22e949e7cc346a2f8660d1c73edfa391a Author: Axel Huebl Date: Thu Jan 12 16:45:07 2023 -0800 SinCos Guard: Apple Define (#3096) ## Summary The issue also occurs on vanilla Clang, not only AppleClang. Thus, we exclude the specialization for all Apple platforms. ## Additional background Follow-up to #3094 #3008 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_Math.H commit a30cc5fd3213bb8118510f252fa3f5ad0028525e Author: Axel Huebl Date: Thu Jan 12 08:44:13 2023 -0800 SinCos: AppleClang Fix (#3094) ## Summary I noticed that in some situations, e.g., when building Python sources that include Python headers, `_GNU_SOURCE` is sometimes defined. This can confuse AppleClang builds of AMReX: ``` error: no member named 'sincos' in the global namespace; did you mean '__sincos'? ::sincos(x, &r.first, &r.second); ``` Adding the respective guard here should allow GCC builds on macOS but ignore the GCC-ism when building with AppleClang. Follow-up to #3008 ## Additional background First seen in pyAMReX the other week. Avoided with restructured includes, but ultimately a defect that CPython headers pull in. Seen again today by @RTSandberg Src/Base/AMReX_Math.H commit f972714bd1111991da8dfddaf65840b57dafa829 Author: Axel Huebl Date: Wed Jan 11 15:57:24 2023 -0800 CI: Doxygen (#3061) Run Doxygen and make sure it does not ~~throw warnings~~ *error while parsing* in CI. This is already run as such in downstream projects, so we need to fix our upstream errors as well. .github/workflows/style.yml .github/workflows/style/doxygen.sh Docs/Doxygen/doxygen.conf Src/Amr/AMReX_AuxBoundaryData.cpp Src/AmrCore/AMReX_ErrorList.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_Utility.H commit a324c40fc1f5e9e078e12d8a74aeae8c05b91495 Author: Weiqun Zhang Date: Wed Jan 11 15:21:54 2023 -0800 CMake: Make hypre and PETSc available only if it's not 1D. (#3093) Tools/CMake/AMReXOptions.cmake commit 64eb49bbc7c4a41233be97d6d7511fea5d112a09 Author: asalmgren Date: Tue Jan 10 15:14:23 2023 -0800 introduce amrex code of conduct (#3092) CODE_OF_CONDUCT.md commit a05384c0b7c4549b696ff6f1c140dde3c78fa27e Author: Nuno Miguel Nobre Date: Mon Jan 9 17:03:26 2023 +0000 SYCL: Reinforce device compilation preprocessor conditionals (#3090) ## Summary The changes in these two commits, although apparently redundant, guard SYCL compilations targeting CUDA or HIP devices by preventing the compiler from seeing the CUDA/HIP code already in AMReX. ## Additional background The 'vanilla' version of LLVM, which is used by hipSYCL for example, enables `__CUDA_ARCH__` and `__HIP_DEVICE_COMPILE__` when compiling device code, for both [CUDA](https://llvm.org/docs/CompileCudaWithLLVM.html#detecting-clang-vs-nvcc-from-code) and [HIP](https://reviews.llvm.org/D45441), respectively. Intel's version of LLVM is also moving in the same direction with [CUDA](https://github.com/intel/llvm/issues/7722) and [HIP](https://github.com/intel/llvm/issues/7720), although the latter is off to a rocky start. Src/Base/AMReX_Extension.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuRange.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_Math.H Src/Base/AMReX_Random.H commit 42b7e27066388dca2d1bc233fa1249df18111e29 Author: Lucas Esclapez <13371051+esclapez@users.noreply.github.com> Date: Fri Jan 6 07:53:52 2023 -0800 FAQ - parameter space overflow (#3089) Docs/sphinx_documentation/source/Faq.rst Src/EB/AMReX_EB2_IF.H Src/EB/AMReX_EB2_IF_DevicePtr.H Src/EB/CMakeLists.txt Src/EB/Make.package commit 2e0b2b463306bc8c371b38e4e3125a8ed86698e3 Author: Weiqun Zhang Date: Thu Jan 5 12:18:47 2023 -0800 New knapsack function to reduce the cost of data movement in load balancing (#3079) This adds a new function DistributionMapping::KnapSackProcessorMap that makes a new DistributionMapping given an old DistributionMapping, a vector of weights and a parameter affecting the amount of the data movement that would be needed for switching from the old to the new DistributionMapping. In this new knapsack function, some boxes will be kept on the current processes and the knapsack algorithm is applied to the rest of the boxes. This may generate a less optimal distribution. However, the new DistributionMapping is more similar to the old one, thus reducing the cost of data movement during load balancing. Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp commit c9c6ed5e098e168a0055a277fe4e0819acf8f8f7 Author: Weiqun Zhang Date: Thu Jan 5 12:18:13 2023 -0800 GPU single stream region and no sync region (#3073) Add functions and classes that allow the user to change the default behavior of MFIter and amrex functions for MultiFabs easily. Use with caution! Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuControl.cpp Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp commit 0c3fff43aa68a3f1da7f005e2fb4a6183ea6146b Author: Weiqun Zhang Date: Thu Jan 5 12:17:17 2023 -0800 Check for small cell before checking for multi-cut (#3088) This is a follow-up on #3087. There are cases with 5 faces covered and one face still open. These cells should be marked small cells before the multi-cut check. Src/EB/AMReX_EB2_3D_C.cpp commit 7a313e260ccf02ab5f08cde5ac45dddc9d1a9918 Author: Weiqun Zhang Date: Wed Jan 4 13:47:29 2023 -0800 Improve AmrMesh::ChopGrids (#3081) This tries to address the following issues. (1) Suppose that the 2d domain has 40x8 cells, the blocking factor is 2 and the max grid size is 1024. Here the max grid size is set to a big number so that we can have boxes as big as possible. If we have 4 MPI processes, the old ChopGrids will chop the domain in x-direction into 5 parts. This is far from optimal. (2) The issue in the example in (1) could be fixed by making the initial chunk size to be the min of max grid size and domain size. This would chop the domain into 4 boxes of 20x4 cells. However, this is still not ideal because boxes close to squares should be preferred to long skinny boxes. The new ChopGrids will chop the domain inot 4 boxes of 10x8 cells. Src/AmrCore/AMReX_AmrMesh.cpp commit dd46ab54580e5ad1506e9d2c18b23942a6cab9e3 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Wed Jan 4 16:29:04 2023 -0500 Add extra check for multicut cell (#3087) For some edge EB geometry cases when building the faces during an EB iteration, nodes can go from regular to covered. Additional check needs to be added to label appropriate cells as a `multicut` cell. Src/EB/AMReX_EB2_3D_C.cpp commit b648023071db8b04262b52d4ee9e77f2d07e6824 Author: Aaron M. Lattanzi <103702284+AMLattanzi@users.noreply.github.com> Date: Wed Jan 4 10:52:45 2023 -0800 Polydisperse neighbor search algorithm (#3060) ## Summary New functionality for building the particle neighbor list with multiple grids. The multi-grid approach can provide significant speed up with polydisperse particle-laden flows. ## Additional background Algorithm based upon: Shire, T., Hanley, K.J. & Stratford, K. DEM simulations of polydisperse media: efficient contact detection applied to investigate the quasi-static limit. Comp. Part. Mech. 8, 653–663 (2021). https://doi.org/10.1007/s40571-020-00361-2 Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleUtil.H commit b3241d7b368a6898d54e4a048512b13b2a3b6a52 Author: Weiqun Zhang Date: Tue Jan 3 21:14:51 2023 -0800 GPU UUID and Arena initial size (#3085) Gather GPU UUIDs to find out the number of unique devices (which has already been done for CUDA). Find out the number of processes sharing my device. Use this information, we can avoid oversubscribing GPU device memory in Arean initialization that could result in a runtime error for non-managed memory. Src/Base/AMReX.cpp Src/Base/AMReX_Arena.cpp Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp commit 490233778c869d2f7dcb4fbfa2eab16aafa7d211 Author: Weiqun Zhang Date: Tue Jan 3 11:55:08 2023 -0800 Use IsCallable to detect if F is callable (#3080) Src/Base/AMReX_GpuContainers.H commit edec77e5c43e14badaa915b4fbb1615f1f59e1a5 Author: Weiqun Zhang Date: Tue Jan 3 11:14:50 2023 -0800 GNU Make: No need to do GCC version check for make clean etc. (#3084) Tools/GNUMake/comps/gnu.mak commit 30f2290cf7c3c4f995ff19ac525ca720c2885dc3 Author: Katharina Kormann Date: Sun Jan 1 19:36:53 2023 +0100 Mlebabeclap aniso (#2640) Changes in MLEBABecLap with the goal of removing the assumption dx=dy=dz to make sure anisotropic cells are correctly treated. The changes concern the way how Feb is computed. This is only implemented for 2D. The external solvers (e.g., Hypre and PETSc) are not updated. So they may have trouble converging when the cells don't have the same size in every direction because the system solved by them is not fully consistent with MLMG. Co-authored-by: Weiqun Zhang Src/Base/AMReX_SPACE.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_Level.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_F.cpp commit 692059ae91b89245e8b8d28de117df10ddfa8ed6 Author: Weiqun Zhang Date: Sat Dec 31 17:38:26 2022 -0800 Update CHANGES for 23.01 (#3086) CHANGES commit ea7e8a507e77c3dce8187e268ddf2212d0e95a24 Author: Weiqun Zhang Date: Thu Dec 22 23:34:21 2022 -0800 amrex::fillAsync (#3076) This new function can be used to fill the elements in a vector type container (e.g., Gpu::DeviceVector). If the element type is a struct with several arithmetic types (e.g., GpuArray), the usual ParallelFor does not have good performance because of the memory access pattern. The new fillAsync function will use shared memory in that case to improve the performance. Src/Base/AMReX_GpuContainers.H commit 9c76f9ae2cc3e412ad5fb0ae9259e531a62b9a3b Author: Axel Huebl Date: Thu Dec 22 22:51:41 2022 +0100 CI: Ascent (#3078) ## Summary Add a CI entry to build Ascent in situ vis. ## Additional background cc @cyrush - https://github.com/ECP-WarpX/WarpX/pull/2483 - https://github.com/ECP-WarpX/WarpX/pull/2858 - #3077 .github/workflows/ascent.yml .github/workflows/sensei.yml commit c03306944aafdf9ee21c896fcfcab6cf7a5e4ad3 Author: Weiqun Zhang Date: Thu Dec 22 12:12:13 2022 -0800 Add forward declarations for class template friends (#3077) It seems that GCC 7.5 requires these forward declarations. Src/LinearSolvers/MLMG/AMReX_MLLinOp.H commit 2a8c71b8bf42bbc376b049aae828c4a2dfd8363b Author: Weiqun Zhang Date: Tue Dec 20 21:38:35 2022 -0800 Update GNU Make file for OLCF (#3070) OLCF_ROCM_ROOT is not defined if the amd module instead of rocm is loaded. So using LMOD_SITE_NAME seems to be a better option. Tools/GNUMake/Make.machines commit 52f4195f33df066d86736044e7150dd3aa45e03c Author: Weiqun Zhang Date: Tue Dec 20 21:38:21 2022 -0800 HIP: amdgpu-target -> offload-arch (#3069) The amdgpu-target option has been deprecated since at least v4.5. Tools/CMake/AMReXParallelBackends.cmake Tools/GNUMake/comps/hip.mak commit ae58672b451d140999944db1265fc18241b71320 Author: Weiqun Zhang Date: Tue Dec 20 14:59:37 2022 -0800 GPU kernel fusing in MLPoisson (#3071) Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H commit 4b4c443c298cd9ca2a90de15a7fa76c602d37ab5 Author: kngott Date: Tue Dec 20 15:05:02 2022 -0500 Docs: Sync Before Comms (#3075) Add a new section to the profiling section and document the sync_before_comm flag. This section can be used for any information, flags or options that applies to both Tiny and Full profiling. Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst commit c2decc83820a5457fc64aabfa2ebd1d787e4744f Author: Weiqun Zhang Date: Mon Dec 19 16:29:20 2022 -0800 Remove deprecated SYCL declaration (#3074) The macros we used do not work with the latest oneapi public release. Src/Base/AMReX_GpuAssert.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_Math.H Src/Base/AMReX_RandomEngine.H commit 96d8c53fbf096af3e6a7386a75d8b5a47beb15a7 Author: Weiqun Zhang Date: Fri Dec 16 09:24:48 2022 -0800 Update F_scripts/README.md (#3072) Tools/F_scripts/README.md commit ff1cce552060aa737f075e3c85fd961b1a284a85 Author: Weiqun Zhang Date: Tue Dec 13 19:24:17 2022 -0800 Change the type of DeriveRec::DeriveBoxMap to std::function (#3068) This gives us more flexibility. For example, lambda function with captured values can now be used. Src/Amr/AMReX_Derive.H commit 673d63122764b758a0727746944692d3fdf23dc6 Author: Axel Huebl Date: Tue Dec 13 13:09:15 2022 -0800 Add CodeQL Scanning (#3059) ## Summary [**GitHub code scanning**](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning): [evolved from LGTM.com](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/). [`codeql.yml` workflow file for GitHub Actions](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/setting-up-code-scanning-for-a-repository#setting-up-code-scanning-manually) ## Additional background - https://github.com/ECP-WarpX/WarpX/pull/3555 - https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#editing-a-code-scanning-workflow .github/workflows/codeql.yml commit ea66c3bf57a2d6ba0123e5161d946eef379efe23 Author: Weiqun Zhang Date: Tue Dec 13 10:34:24 2022 -0800 Remove old python scripts for releasing (#3066) In the old CVS days, we used to use these scripts to strip out codes that were not meant for release. Tools/Release/cleanWords.txt Tools/Release/ppCleanup.py Tools/Release/ppCleanupDir.py Tools/Release/release.py commit 7316751aa25a00ef50828947b9f164c8dc85a33a Author: Weiqun Zhang Date: Tue Dec 13 10:19:33 2022 -0800 Plotfile Reader: Allow multi-word variable names (#3067) Src/Base/AMReX_PlotFileDataImpl.cpp commit 3129db3446226ed64e1973189ded719ef8baac47 Author: Weiqun Zhang Date: Mon Dec 12 20:25:20 2022 -0800 Fix FPE in FillPatcher for RK (#3065) To avoid FPE, the interpolation of RK data should only be done inside the physical domain. Even though the FPE here is benign because the garbage values will be overwritten later, we still want to fix it so that it does not trigger signal handling. Src/AmrCore/AMReX_FillPatcher.H commit e90408d5559c551b9a911f8f2c8641938b8f05ab Author: Axel Huebl Date: Mon Dec 12 16:22:28 2022 -0800 Fix: Shared CMake builds w/ `AMReX_BUILD_SHARED_LIBS` (#3057) Forgot to honor the new option. All builds that did not set `BUILD_SHARED_LIBS` were static now. Introduced in #3013 Src/CMakeLists.txt commit d8fef14e353f914ca10df6ebfb59c53e0f6a53dc Author: Weiqun Zhang Date: Mon Dec 12 16:15:27 2022 -0800 Add PETSc CI (#3049) .github/workflows/petsc.yml commit bef5b6a282ebfd9236e53a1f262c982837299710 Author: Michael Zingale Date: Mon Dec 12 11:42:49 2022 -0500 remove plotsinglevar.py (#3063) this depended on an old Fortran library fsnapshot.so that we no longer provide. It's much easier to do this sort of plotting with yt these days. Tools/Py_util/plotsinglevar.py commit 10293db79733bba55986e0aede522bd86a8d030c Author: Michael Zingale Date: Mon Dec 12 11:40:55 2022 -0500 Remove old python build scripts from Castro / Maestro (#3064) These all originated in Castro / Maestro, and we've moved the bits we need to our codes. Many of these deal with Fortran in particular, and are no longer used at all. Tools/F_scripts/fcheck.py Tools/F_scripts/findparams.py Tools/F_scripts/makebuildinfo.py Tools/F_scripts/write_probin.py commit 414d2091da47d0ba73a67d73cbb856945c11c621 Author: Weiqun Zhang Date: Fri Dec 9 17:28:33 2022 -0800 Add math functions to amrex::Math (#3008) Add a function to return PI and the following trig functions, - `cospi(x)` returns `cos(pi*x)` - `sinpi(x)` returns `sin(pi*x)` - `sincos(x)` returns `sin(x)` and `cos(x)` in `std::pair` - `sincospi(x)` returns `sin(pi*x)` and `cos(pi*x)` in `std::pair` Because we use C++17 now, one could use structured binding on the `std::pair` returned by `sincos` and `sincospi`. auto const [sin_x, cos_x] = amrex::Math::sincos(x); Src/Base/AMReX_Math.H commit 5a92d75f2a6a7f0e764c469b1f4bbe417dcf4566 Author: Michael Zingale Date: Fri Dec 9 19:52:27 2022 -0500 remove some astro-specific python routines (#3062) these have not been used in ages and are not general for AMReX Tools/Postprocessing/python/column_depth.py Tools/Postprocessing/python/conv_slopes.py Tools/Postprocessing/python/eos_data.txt Tools/Postprocessing/python/helmeos.py Tools/Postprocessing/python/test_helmeos.py commit 821c5ba7d39b44d21cd3c3e726a2cc799111cc35 Author: Weiqun Zhang Date: Fri Dec 9 13:57:11 2022 -0800 Gpu Streams: Performance Tuning (#3058) * Change the number of GPU streams from 4 to 2. Recent tests on summit and crusher show that using 2 GPU streams seems to be better than 4 or 1 in most cases. One reason is that it reduces the cost associated with stream synchronization, which could be significant for small kernels. * In ~MFIter and ~StreamIter, there is no need to synchronize the streams that were not used. It appears that for HIP there is very little cost to sychronize a stream that does not have any GPU kerenels since the last synchronization, but that's not the case for CUDA. * Remove the dedicated stream for kernels outside MFIter, and simply use the first stream used by MFIter. * Remove synchronization in FabArray::define. It was needed when we were putting FAB in managed memory, but that's no longer the case. Src/Base/AMReX_FabArray.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuUtility.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit 57793fda870abbe8ed70b5fbb593b4f63589b442 Author: Weiqun Zhang Date: Wed Dec 7 10:27:24 2022 -0800 MLMG: template (#3035) MLMG and various linear operator classes use MultiFab as their data container. The limitations are (1) it cannot do single precision when amrex::Real is double and (2) it cannot handle data that need to be stored in multiple MultiFabs such as vectors on cell edges. Recently, we have experimented with amrex::Any that can be a type different from MultiFab. However, this involves a lot of complicated and hacky changes. In this PR, we are trying to use template to make the interface more flexible. Classes MLMG, MLPoisson, MLALaplacian, MLABecLaplacian, MLABecLap, MLCellLinop, MLLinOp, and MLCGSolver have been changed to templates. More changes will follow. For most users, this should not require any changes in their codes. However, if one has classes derived from the linear operators in MLMG, some small changes are likely needed. For example, `getNComp()` needs to be `this->getNComp()` due to C++ name lookup rules. .github/workflows/cuda.yml Docs/sphinx_documentation/source/LinearSolvers.rst Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/AMReX_PETSc.cpp Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp_temp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMGBndry.H Src/LinearSolvers/MLMG/AMReX_MLMGBndry.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H Src/LinearSolvers/MLMG/Make.package Tests/LinearSolvers/ABecLap_SP/CMakeLists.txt Tests/LinearSolvers/ABecLap_SP/GNUmakefile Tests/LinearSolvers/ABecLap_SP/Make.package Tests/LinearSolvers/ABecLap_SP/MyTest.H Tests/LinearSolvers/ABecLap_SP/MyTest.cpp Tests/LinearSolvers/ABecLap_SP/MyTestPlotfile.cpp Tests/LinearSolvers/ABecLap_SP/initProb.cpp Tests/LinearSolvers/ABecLap_SP/initProb_K.H Tests/LinearSolvers/ABecLap_SP/inputs Tests/LinearSolvers/ABecLap_SP/main.cpp Tests/LinearSolvers/ABecLaplacian_C/CMakeLists.txt Tests/LinearSolvers/ABecLaplacian_C/initProb_K.H Tests/LinearSolvers/MLMG/GNUmakefile Tests/LinearSolvers/MLMG/Make.package Tests/LinearSolvers/MLMG/fort.H Tests/LinearSolvers/MLMG/fort_3d.F90 Tests/LinearSolvers/MLMG/init_prob.cpp Tests/LinearSolvers/MLMG/inputs Tests/LinearSolvers/MLMG/inputs.boxes Tests/LinearSolvers/MLMG/main.cpp Tests/LinearSolvers/MLMG/prob_par.H Tests/LinearSolvers/MLMG/solve_with_mlmg.cpp Tests/LinearSolvers/MLMG/write_plotfile.cpp commit 1a5b308c8af78386f11c14edb72266ced66b0782 Author: Weiqun Zhang Date: Tue Dec 6 09:08:11 2022 -0800 Add Hypre CI (#3048) .github/workflows/hypre.yml commit 4a53367b19e11c42fe0a66abde4d9961a8ac3337 Author: Weiqun Zhang Date: Thu Dec 1 17:34:55 2022 -0800 Print host name in backtrace files (#3054) This can help identify which nodes generated the backtrace files. Note that the MPI rank is embedded in the file name, but it does not tell us which node it is. Src/Base/AMReX_BLBackTrace.cpp commit ccb21a03196ea604ff3316224029237a218e5efd Author: Weiqun Zhang Date: Thu Dec 1 08:50:06 2022 -0800 Update CHANGES for 22.12 (#3053) CHANGES commit 88fe04f00600407e490f66abf446da5831d1ce7e Author: Nuno Miguel Nobre Date: Tue Nov 29 03:24:30 2022 +0000 SYCL: Remove floating-point specialisations for atomic adds (#3050) Src/Base/AMReX_GpuAtomic.H commit 4d6413c45fa0e1aa6f366a02d75a9e2382c73850 Author: Brandon Runnels Date: Tue Nov 22 17:25:31 2022 -0700 Template average down (#2980) Adding templating for the average_down function Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H commit bd0cef1f097e939997ffac3e29f61b572c0a0b99 Author: Weiqun Zhang Date: Tue Nov 22 12:04:01 2022 -0800 Fix FabArray::LocalCopy SFINAE (#3047) Fix a bug in FabArray::LocalCopy SFINAE so that FAB in FabArray does not need to have typename value_type defined. Note that BaseFab has value_type, but FAB is not necessarily BaseFab. For convenience, IsBaseFab_v and IsFabArray_v are added. Src/Base/AMReX_FabArray.H Src/Base/AMReX_TypeTraits.H commit a7c63f2fc3a5dca442aa14c1d1af2f6b98e82592 Author: Michael Zingale Date: Tue Nov 22 11:51:59 2022 -0500 fix compilation of RichardsonConvergenceTest (#3046) Tools/C_util/Convergence/RichardsonConvergenceTest.cpp commit ed095ca21072b38fe6b7d8892bcc022ace531631 Author: Weiqun Zhang Date: Mon Nov 21 09:44:22 2022 -0800 Refactor Boundary Registers (#3028) Refactor boundary registers so that they can be used for single precision. - Remove MacBndry because it's no longer being used. - Make FabSet, BndryRegister, BndryData, and InterpBndryData templates with FabArray type as a parameter. The original names are now alias to FabSetT etc. - For historical reasons these classes were virtual. However, they do not need to be polymorphic anymore. - Remove unnecessary member functions. Src/AmrCore/AMReX_FluxRegister.H Src/Base/AMReX_MultiFab.H Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryData.cpp Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_BndryRegister.cpp Src/Boundary/AMReX_BoundaryFwd.H Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_FabSet.cpp Src/Boundary/AMReX_InterpBndryData.H Src/Boundary/AMReX_InterpBndryData.cpp Src/Boundary/AMReX_InterpBndryData_1D_K.H Src/Boundary/AMReX_InterpBndryData_2D_K.H Src/Boundary/AMReX_InterpBndryData_3D_K.H Src/Boundary/AMReX_LOUtil_K.H Src/Boundary/AMReX_MacBndry.H Src/Boundary/AMReX_MacBndry.cpp Src/Boundary/CMakeLists.txt Src/Boundary/Make.package Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMGBndry.H commit fb3227c8d3b92c5ca6deb6f547dee1ea3c469946 Author: Weiqun Zhang Date: Mon Nov 21 09:36:58 2022 -0800 Fix Multi-Level Open BC Solver (#3032) In the open BC solver, the enlarged level 0 grids are shifted so that the lower left corner is at (0,0,0). This allows the grids to be coarsened more than if they are not shifted. However, we did not shift the fine level grids and this was a bug. This bug is fixed here. Close #3023 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/LinearSolvers/OpenBC/AMReX_OpenBC.H Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp commit cf0afb0c152e2c942073731da7a1e0007886eed4 Author: Weiqun Zhang Date: Mon Nov 21 09:11:54 2022 -0800 Template average_down_faces (#3040) They can now be used on floats when amrex::Real is double. Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H commit e65b63e61fa0597c187fc08be50eeb11151280ba Author: Weiqun Zhang Date: Mon Nov 21 09:07:01 2022 -0800 Add FabArray::LocalCopy and LocalAdd (#3043) These are non-static member functions. LocalCopy can also be used to do mixed precision copy. Src/Base/AMReX_FabArray.H commit ccc9bd2f92fa9b2d277ac364a10c2227181599c4 Author: Weiqun Zhang Date: Mon Nov 21 09:01:23 2022 -0800 FabArray::sum (#3041) Add FabArray::sum so that we can compute sum for FabArray>. Src/Base/AMReX_FabArray.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp commit 9b6db7e1508a68310e8ee7063547c3912a82bb7b Author: Weiqun Zhang Date: Mon Nov 21 08:59:24 2022 -0800 amrex::Any::hasValue() (#3025) Add Any::hasValue() that checks whether the object contains a value. This is the analog of std::any::has_value. Src/Base/AMReX_Any.H commit f23e6a2330bf875643462105f3afc05fed28fad4 Author: Weiqun Zhang Date: Mon Nov 21 08:16:19 2022 -0800 Make YAFluxRegister template (#3030) Make YAFluxRegister a template so that it can be used for single-precision data when AMReX is built for double precision. The class is now called YAFluxRegisterT and YAFluxRegister is aliased to YAFluxRegisterT. So there are no changes in the user's code. Src/Base/AMReX_FabArray.H Src/Boundary/AMReX_BoundaryFwd.H Src/Boundary/AMReX_YAFluxRegister.H Src/Boundary/AMReX_YAFluxRegister.cpp Src/Boundary/AMReX_YAFluxRegister_1D_K.H Src/Boundary/AMReX_YAFluxRegister_2D_K.H Src/Boundary/AMReX_YAFluxRegister_3D_K.H Src/Boundary/CMakeLists.txt Src/Boundary/Make.package commit 75257bf75bcd291b78ee1ec425cd9ce11c6f37c1 Author: Weiqun Zhang Date: Mon Nov 21 08:14:24 2022 -0800 FabArray::norminf (#3039) Add FabArray::norminf so that we can compute infinity norm for single precision data when AMReX is built for double precision. Src/Base/AMReX_FabArray.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp commit de4ec8614b824cf044751c11e212cc277db151a4 Author: Weiqun Zhang Date: Mon Nov 21 08:13:05 2022 -0800 amrex::Dot (#3042) Add amrex::Dot function for computing the dot product of two FabArrays. Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_MultiFab.cpp commit 069d713539a32a216791f0a7eae1d7c8529ca685 Author: Weiqun Zhang Date: Mon Nov 21 08:11:38 2022 -0800 template FabArray::Saxpy, Xpay and LinComb (#3044) They can now be used on single precision FabArray>. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp commit 46e487c7297379eafb77ea8df7a9a0d825575978 Author: Malte Buschmann Date: Mon Nov 21 00:00:41 2022 -0500 Use std::max when coarsening a TagBox to preserve TagBox::SET values (#3045) When coarsening a TagBox in the current version, cells with the status SET are changed to BUF. This can cause undesirable effects if, e.g., TagBox::buffer is called on a coarsened TagBox. With this pull request, the {CLEAR, BUF, SET} hierarchy is preserved when coarsening a TagBox. Src/AmrCore/AMReX_TagBox.cpp commit 0d52da4dfede1eff9df65cc8c8ab4155a61d5fc6 Author: Weiqun Zhang Date: Wed Nov 16 17:51:50 2022 -0800 Fix type conversion warnings (#3034) Src/Base/AMReX_COORDSYS_1D_C.H Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_MultiFabUtil_1D_C.H commit 6d9dcd94c831b649d22d94257793ed9d7f8adf62 Author: Weiqun Zhang Date: Wed Nov 16 11:07:02 2022 -0800 Update for oneAPI 2023 (#3024) A number of things have been deprecated and need to be updated. - dpcpp -> icpx - sycl::gpu_selector -> sycl::gpu_selector_v - sycl::info::device::max_work_item_sizes -> sycl::info::device::max_work_item_sizes<3> - sycl::accessor<...,sycl::access::target::local> -> sycl_local_accessor - CL/sycl.hpp -> sycl/sycl.hpp Also remove AMREX_GPU_EXTERNAL because SYCL_EXTERNAL never worked for us. .github/workflows/intel.yml Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_GpuAssert.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_Math.H Src/Base/AMReX_RandomEngine.H Tests/GPU/AnyOf/GNUmakefile Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/sites/Make.unknown commit d8bc97f92a1a568b6e996db3b8d9715fced0464f Author: Ben Wibking Date: Sun Nov 13 12:39:58 2022 -0500 min/max- and linear combination-preserving interpolater (#3020) ## Summary This linearly interpolates onto fine grids, ensuring no new minima and maxima are created in multi-dimensions and also preserving linear combinations of components in each cell. The current implementation is only for Cartesian geometry. ## Additional background `CellConservativeLinear` can do each of these things separately (i.e., `CellConservativeLinear(0)` or `CellConservativeLinear(1)`), but does not preserve both of these properties at the same time. Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_MFInterpolater.cpp commit b0e1f8f354b19bc5ecaba0529dc78c5c5bed2ee1 Author: Andrew Myers Date: Sun Nov 13 09:03:30 2022 -0800 add a new test for particle checkpoint / restart in the native format (#3027) This fills a gap in the test suite indicated in #3026. Tests/HDF5Benchmark/main.cpp Tests/Particles/CheckpointRestart/CMakeLists.txt Tests/Particles/CheckpointRestart/GNUmakefile Tests/Particles/CheckpointRestart/Make.package Tests/Particles/CheckpointRestart/inputs Tests/Particles/CheckpointRestart/main.cpp commit 34d6e204e9bac9f97c42f9ed443c4e6a2deb0258 Author: Weiqun Zhang Date: Fri Nov 11 18:26:20 2022 -0800 template Xpay (#3031) Convert MultiFab::Xpay (y = x + a*y) to template amrex::Xpay. Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_MultiFab.cpp commit e0afc82f63ce51e0f10aef21d8ab163eb87547d1 Author: Axel Huebl Date: Tue Nov 8 16:41:39 2022 -0800 CMake: AMReX_BUILD_SHARED_LIBS (#3013) ## Summary Add a new variable `AMReX_BUILD_SHARED_LIBS` for per-project control of shared lib builds. This variable is more specific (AMReX project) than `BUILD_SHARED_LIBS` (whole build, defaults) and `USE_XSDK_DEFAULTS` (whole build, changed defaults). We need this variable so we can control the AMReX library build in superbuilds without changing all other libraries we build in other projects of the superbuild. ## Additional background - https://github.com/ECP-WarpX/impactx/pull/123 - https://github.com/ECP-WarpX/impactx/pull/208 Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.cmake commit 5e7ffd05ef36bb0fdbafa02bb7b65f58d909baf5 Author: Phil Miller Date: Tue Nov 8 13:05:59 2022 -0800 Fix mis-matched array type that broke mixed-precision restarts (#3026) ## Summary A mixed-precision build of WarpX (DP fields, SP particles) would fail to restart from a checkpoint with added real particle components, crashing while reading in data. ## Additional background Isolated with this modification of a WarpX example, and this fix tested similarly with `amr.restart = blah` ``` $ diff -u ~/repos/WarpX/Examples/Physics_applications/capacitive_discharge/inputs_2d inputs_2d --- /home/ubuntu/repos/WarpX/Examples/Physics_applications/capacitive_discharge/inputs_2d 2022-09-07 22:02:51.200174154 +0000 +++ inputs_2d 2022-11-07 21:52:04.764307593 +0000 @@ -39,6 +39,8 @@ electrons.momentum_distribution_type = maxwell_boltzmann electrons.theta = (kb*30000/(m_e*clight^2)) +electrons.save_previous_position = 1 + he_ions.species_type = helium he_ions.charge = q_e he_ions.injection_style = nuniformpercell @@ -55,24 +57,28 @@ -diagnostics.diags_names = diag1 +diagnostics.diags_names = diag1 ckpt diag1.diag_type = Full diag1.intervals = 50 diag1.intervals = 10 diag1.fields_to_plot = rho_electrons rho_he_ions + +ckpt.diag_type = Full +ckpt.intervals = 7 +ckpt.format = checkpoint ``` ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_GpuContainers.H Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Tests/Particles/AsyncIO/main.cpp Tests/Particles/ParticleTransformations/main.cpp commit 81e0635ce83297d4e78b465e620f058a0c3e3845 Author: Weiqun Zhang Date: Mon Nov 7 09:36:12 2022 -0800 Add amrex::demangle for demangling C++ names (#3022) It can turn "N5amrex3BoxE" into "amrex::Box", and "Z4mainEUlvE_" into "main::{lambda()#1}". Src/Base/AMReX_Demangle.H Src/Base/AMReX_Utility.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 0d3deeb5c75cade14c381ef620921beaa2604c11 Author: Weiqun Zhang Date: Sun Nov 6 10:36:17 2022 -0800 Add more functionalities to TypeList (#3021) Move TypeList to its own header and add more functionalities. - TypeList::size() returns the number of types in the TypeList. - TypeAt is the type at a given position of a TypeList. - ForEach applies a function to each type of a TypeList. - ForEachUntil applies a function to each type of a TypeList until true is returned. - operator+ concatenates two TypeLists. This is an existing capability. - CartesianProduct returns the Cartesian product of TypeLists. This is an existing capability with a change. The function now takes a number of TypeLists, instead of a single TypeList packed with these TypeLists. Src/Base/AMReX_CTOParallelForImpl.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_TypeList.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 6ffb2e57c8233c8f6fcdb22c8dcd4a256fc2b66b Author: kngott Date: Sat Nov 5 10:02:48 2022 -0700 Add muller to NERSC machines (#3019) Adding a NERSC test machine to the list. Tools/GNUMake/Make.machines commit c71fc4c7643be908bd2a325cb192d36b4a206578 Author: Weiqun Zhang Date: Fri Nov 4 19:03:27 2022 -0700 Fix -lquadmath in GNU Make (#3018) Some machines (e.g., arm64) may not have the quadmath library. Close #3017 Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/llvm.mak commit 73b67447d76bd2f23351adc9a10e70c04029f9ff Author: Weiqun Zhang Date: Fri Nov 4 10:02:24 2022 -0700 Fix a typo in doc (#3016) Docs/sphinx_documentation/source/AmrLevel.rst commit 2795f8a2f3ec82cce099251d145ded12ebd97c7f Author: Weiqun Zhang Date: Fri Nov 4 09:24:23 2022 -0700 Fix EB tensor solver's boundary when EB is tilted. (#3002) It was incorrect to use the connectivity of valid cells to determine the stencil involving domain boundary registers. Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_K.H commit 6044e9c2bb9a365b9708d791ac58cd4d04eff6ac Author: Weiqun Zhang Date: Wed Nov 2 17:10:12 2022 -0700 Add -Wmissing-include-dirs to gcc warning flags (#3012) .github/workflows/gcc.yml Tools/GNUMake/comps/gnu.mak commit d2dbc863ba77f35092a85d8357c568bcb200d5e7 Author: Weiqun Zhang Date: Wed Nov 2 15:05:55 2022 -0700 A better fix for the PETSc compilation issue (#3011) Use a pair of parentheses to work around PETSc's MPI macros. This is a follow-up on #3005. Src/Base/AMReX_ParallelReduce.H commit 09cb43f90f5f52ec0bbf9ec3a7d87d99e84ac707 Author: Erik Schnetter Date: Wed Nov 2 17:26:57 2022 -0400 Replace all `sprintf` by `snprintf` (#3010) ## Summary Change all `sprintf` to `snprintf`. This makes the code safer, although the current code seems to be correct. ## Additional background `sprintf` is deemed insecure and should not be used. In many cases, the code could be changed to use `std::ostringstream` instead; this would probably reduce the code size as well. Src/Amr/AMReX_AmrLevel.cpp Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Extern/ProfParser/AMReX_BLWritePlotFile.cpp commit 7db23ec9386de76392aba848de817ecb5af673af Author: Weiqun Zhang Date: Wed Nov 2 11:22:15 2022 -0700 Use std::less for comparison of pointers (#3007) In principle, using `<` for pointer comparison is UB. In practice, the compilers (e.g., clang) use `<` in their implementation. Src/Base/AMReX_CArena.H commit b816db6e085ff317222832f9cade89838b44aa78 Author: Weiqun Zhang Date: Wed Nov 2 10:28:07 2022 -0700 Add YAFluxRegister::getFineData (#3004) This will allow an application code to save and reset the fine data stored in YAFluxRegister. Src/Boundary/AMReX_YAFluxRegister.H Src/Boundary/AMReX_YAFluxRegister.cpp commit c4a4811c373d9b599bb710c7029365b1ca7f2c22 Author: Axel Huebl Date: Tue Nov 1 14:08:38 2022 -0500 C++17 Transition (#2992) ## Summary Update AMReX to require C++17 or newer. - [x] docs - [x] CMake - [x] GNUmake - [x] CI ## Additional background Requires a mature [C++17](https://en.wikipedia.org/wiki/C%2B%2B17) compiler, e.g., GCC 8, Clang 7, NVCC 11.0, MSVC 19.15 or newer. Already used since 1+ year in production by downstream codes such as Castro and WarpX. Needed for modernization and new features such as #2878 Co-authored-by: Weiqun Zhang .github/workflows/clang.yml .github/workflows/cuda.yml .github/workflows/dependencies/dependencies.sh .github/workflows/dependencies/dependencies_clang7.sh .github/workflows/dependencies/dependencies_gcc8.sh .github/workflows/dependencies/dependencies_nofortran.sh .github/workflows/dependencies/dependencies_nvcc10.sh .github/workflows/dependencies/dependencies_nvcc11.sh .github/workflows/gcc.yml Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/BuildingAMReX_Chapter.rst INSTALL Src/Amr/AMReX_StateDescriptor.cpp Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CTOParallelForImpl.H Tools/AMRProfParser/GNUmakefile Tools/CMake/AMReXTypecheck.cmake Tools/CMake/AMReX_Config.cmake Tools/GNUMake/Make.rules Tools/GNUMake/comps/armclang.mak Tools/GNUMake/comps/cray.mak Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/intel.mak Tools/GNUMake/comps/llvm-flang.mak Tools/GNUMake/comps/llvm.mak Tools/GNUMake/comps/nag.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/comps/nvhpc.mak Tools/GNUMake/comps/pgi.mak Tools/Plotfile/CMakeLists.txt commit d2b82938c171a4b1ada48839ed6891b5b0183b43 Author: Weiqun Zhang Date: Tue Nov 1 09:01:54 2022 -0700 Update CHANGES for 22.11 (#3006) CHANGES commit 5ec270b4d534a486aeabf478ae553f1df53f2e5b Author: Weiqun Zhang Date: Tue Nov 1 08:59:44 2022 -0700 Fix compilation for PETSc (#3005) We cannot include PETSc headers too early because it might redefine MPI routines as macros (https://github.com/petsc/petsc/blob/main/include/petsclog.h#L441). They break MPI calls like below, MPI_Allreduce(&tmp, &vi, 1, ParallelDescriptor::Mpi_typemap::type(), ParallelDescriptor::Mpi_op>(), comm); because of the `,` in `>`. Src/Extern/PETSc/AMReX_PETSc.cpp commit 735c3513153f1d06f783e64f455816be85fb3602 Author: Weiqun Zhang Date: Sat Oct 29 10:57:23 2022 -0700 MPI Reduce for ValLocPair (#3003) Add ParallelReduce::Min, ParallelReduce::Max, ParallelAllReduce::Min, and ParallelAllReduce::Max for ValLocPair, where TV and TI are types that have corresponding MPI types (e.g., int, Real, IntVect, Box, etc.). Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParallelReduce.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_ValLocPair.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 3ec07681574afa658f4f53117d7ab618459a514b Author: Axel Huebl Date: Wed Oct 26 16:49:40 2022 -0700 `FabArray::isDefined` (#2997) ## Summary Add a new query to `define_function_called`. ## Additional background This is a cheaper check than `ok()` for finding out if a MultiFab has been allocated or not yet, assuming that the calling code follows the convention that `define()` is called collectively. Update: It turns out you can also call `empty` inherited from `FabArrayBase`. The new API is quite explicit, which is ok, too. Co-authored-by: Weiqun Zhang Src/Base/AMReX_FabArray.H commit 7f3c90893d70ca33c6dec499436dd503c77eeddf Author: Weiqun Zhang Date: Wed Oct 26 16:40:16 2022 -0700 Make The_Device_Arena non-managed (#2998) The_Device_Arena used to be a separate Arena. We changed it to be an alias of The_Arena to avoid memory fragmentation. However, the issue is we don't have an Arena that can allocate non-managed memory unless The_Arena is not managed. Because of performance concerns, we sometimes want to allocate non-managed memory. Therefore, we make The_Device_Arena an alias if and only if The_Arena is not managed. Src/Base/AMReX_Arena.cpp Src/Base/AMReX_GpuContainers.H commit ab8c892e1dd8943a6f0f759693757c6a186668a7 Author: Weiqun Zhang Date: Wed Oct 26 15:59:39 2022 -0700 Add alias template Gpu::NonManagedDeviceVector (#2999) Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_GpuContainers.H commit b3e0a62ba4d8c66b7cc40ab439b94835a5f4247c Author: Weiqun Zhang Date: Wed Oct 26 15:02:13 2022 -0700 Pre- and Post-interpolation hook interface (#2991) Support both Fab and MultiFab versions of pre- and post-interpolation hooks. Because the pre-interp hook might modify the data, we need to make a copy to avoid modifying cached coarse data. Close #2989. Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FillPatcher.H commit 3082028e42870b1ed37f0d26160ef078580511e3 Author: Weiqun Zhang Date: Wed Oct 19 19:24:10 2022 -0700 Update GitHub Actions (#2996) https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/ ## Summary ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate .github/workflows/clang.yml .github/workflows/cuda.yml .github/workflows/docs.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/sensei.yml .github/workflows/style.yml .github/workflows/windows.yml commit 0b88bfd3718ab226fc4a03a6598d320976744346 Author: Weiqun Zhang Date: Wed Oct 19 13:39:18 2022 -0700 Add user defined BC types (#2995) Add BCType::user_1, BCType::user_2 and BCType::user_3. Previously the only "user" type is ext_dir (external Dirichlet). The BC types are passed from the user's code to FillPatch, which in turn passes them back to the user provided BC filling function. These new types will make it easy for the user to determine the user defined BC types in their BC filling functions. Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/LinearSolvers.rst Src/Base/AMReX_BC_TYPES.H Src/Base/AMReX_bc_types_mod.F90 commit 9502b99cd98cc1aa70e5f19804c50252438ec1a6 Author: Weiqun Zhang Date: Tue Oct 18 10:20:06 2022 -0700 Add BCRec::set for convenience (#2993) Src/Base/AMReX_BCRec.H commit 56b6402d238979fca6e7c57fdc644a54c4cf6fce Author: Weiqun Zhang Date: Sat Oct 15 14:59:38 2022 -0700 ParallelFor with compile time optimization of kernels with run time parameters (#2954) Branches inside ParallelFor can be very expensive. If a branch uses a lot of resources (e.g., registers), it can significantly affect the performance even if at run time the branch is never executed because it affects the GPU occupancy. For CPUs, it can affect vectorization of the kernel. The new ParallelFor functions use C++17 fold expression to generate kernel launches for all run time variants. Only one will be executed. Which one is chosen at run time depends the run time parameters. The kernel function can use constexpr if to discard unused code blocks for better run time performance. Here are two examples of how to use them. int runtime_option = ...; enum All_options : int { A0, A1, A2, A3}; // Four ParallelFors will be generated. ParallelFor(TypeList>{}, {runtime_option}, box, [=] AMREX_GPU_DEVICE (int i, int j, int k, auto control) { ... if constexpr (control.value == A0) { ... } else if constexpr (control.value == A1) { ... } else if constexpr (control.value == A2) { ... else { ... } ... }); and int A_runtime_option = ...; int B_runtime_option = ...; enum A_options : int { A0, A1, A2, A3}; enum B_options : int { B0, B1 }; // 4*2=8 ParallelFors will be generated. ParallelFor(TypeList, CompileTimeOptions > {}, {A_runtime_option, B_runtime_option}, N, [=] AMREX_GPU_DEVICE (int i, auto A_control, auto B_control) { ... if constexpr (A_control.value == A0) { ... } else if constexpr (A_control.value == A1) { ... } else if constexpr (A_control.value == A2) { ... else { ... } if constexpr (A_control.value != A3 && B_control.value == B1) { ... } ... }); Note that that due to a limitation of CUDA's extended device lambda, the constexpr if block cannot be the one that captures a variable first. If nvcc complains about it, you will have to manually capture it outside constexpr if. The data type for the parameters is int. Thank Maikel Nadolski and Alex Sinn for showing us the meta-programming techniques used here. Src/Base/AMReX_CTOParallelForImpl.H Src/Base/AMReX_GpuLaunch.H Src/Base/CMakeLists.txt Src/Base/Make.package Tests/CMakeLists.txt Tests/CTOParFor/CMakeLists.txt Tests/CTOParFor/GNUmakefile Tests/CTOParFor/Make.package Tests/CTOParFor/main.cpp commit bcbf17f1cee4cd3209552cd0cafb2558c9254f20 Author: Weiqun Zhang Date: Fri Oct 14 19:48:14 2022 -0700 2D RZ solver for WarpX: Arbitrary coefficient (#2986) The assumption in the 2D RZ solver for WarpX used to be there was no sigma_r (i.e., sigma_r == 1). In this PR, we allow arbitrary sigma_r coefficient. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 9a3cd5d985ad357ab78d8f06f397cfc741448fdc Author: Axel Huebl Date: Fri Oct 14 17:27:41 2022 -0700 CMake Docs: Fix User-Guidance (Link) (#2990) Update the user-guidance on CMake dependency linking to CMake 3.0+ (anno. 2014+). Seen in #2978 Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GPU.rst commit 1ad4144668b0656d42950be92936073c64c56db7 Author: Weiqun Zhang Date: Fri Oct 14 10:36:17 2022 -0700 Runge-Kutta support for AMR (#2974) This adds RK2, RK3 and RK4 in a new namespace RungeKutta. Together with the enhanced FillPatcher class, these functions can be used for RK time stepping in AMR simulations. A new function AmrLevel::RK is added for AmrLevel based codes. See CNS::advance in Tests/GPU/CNS/CNS_advance.cpp for an example of using the new AmrLevel::RK function. The main motivation for this PR is that ghost cell filling for high order (> 2) RK methods at coarse/fine boundary is non-trivial when there is subcycling. Co-authored-by: Jean M. Sexton Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/AmrCore/AMReX_FillPatcher.H Src/Base/AMReX_RungeKutta.H Src/Base/CMakeLists.txt Src/Base/Make.package Tests/GPU/CNS/Source/CNS.H Tests/GPU/CNS/Source/CNS.cpp Tests/GPU/CNS/Source/CNS_advance.cpp Tests/GPU/CNS/Source/diffusion/CNS_diffusion_K.H commit c841ae81ddd519c088b29523aa71b6b280da440e Author: Weiqun Zhang Date: Fri Oct 14 10:03:34 2022 -0700 Fourth-order interpolation from fine to coarse level (#2987) For fourth-order finite-difference methods with data at cell centers, we cannot use the usual averageDown function to overwrite coarse level data with fine data. We actually need to do interpolation. Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp commit 975b830a012e4677d070b46d2f92353c117ad65a Author: Weiqun Zhang Date: Fri Oct 14 09:53:22 2022 -0700 Fix EB data inconsistency when fixing small cells and multiple cuts (#2943) ## Summary For consistency, we need to call the function that zeros out the level set even if that box does not have any small cells or multiple cuts. This is because a node could exist in multiple boxes. Furthermore, a covered cell or covered face may have a node with a level set < 0. ## Additional background This is usually not an issue. However, in WarpX, we use the level set to decide whether a node is an unknown in the linear system. The inconsistency makes the solver fail in some cases. Src/EB/AMReX_EB2_3D_C.cpp commit 9c2264bb5ff60b353250b3654866aef06f93bdcc Author: Axel Huebl Date: Fri Oct 14 07:41:06 2022 -0700 `MFIter::Finalize`: Free `m_fa` (#2988) This `free` should potentially not be delayed until the destructor is called. Follow-up to #2985 #2983 Src/Base/AMReX_MFIter.cpp commit f84c7a8f77d6f80f6f8ba4ee9161ee5a73a839a5 Author: Weiqun Zhang Date: Wed Oct 12 10:44:11 2022 -0700 Fix MLMG::getGradSolution & getFluxes for inhomogeneous Neumann and Robin BC (#2984) Because of the way how inhomogeneous and Robin BC are handled, we must add the inhomogeneous fluxes back, otherwise they would be zero at those boundaries. Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp commit ed1ecd62acb3fd7d39b8a23aa4e9ad09669741bb Author: Axel Huebl Date: Wed Oct 12 08:46:34 2022 -0700 MFIter: Make Finalize Public (#2985) Follow-up to #2983 Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit 5acfe07a830305cc7cbafd1e5dd26e3c3598435b Author: Axel Huebl Date: Tue Oct 11 14:51:48 2022 -0700 MFIter::Finalize (#2983) Add a Finalize function to MFIter. The idea about this is, that we can call this already before destruction in Python, where `for` loops do not create scope. This function must be robust enough to be called again in the constructor (or we need to add an extra bool to guard that it is not called again in the destructor). Co-authored-by: Weiqun Zhang Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit 53e34d17913cc76bdd4bbaad1582dd1b04058914 Author: Andy Nonaka Date: Tue Oct 11 12:00:34 2022 -0700 fix docs; Robin BC's for MLMG (#2982) Update the MLMG Robin BC description in the docs. Docs/sphinx_documentation/source/LinearSolvers.rst commit 0019b3a41065caf6d9486000b9c6fbf86ad9837e Author: Weiqun Zhang Date: Tue Oct 11 11:00:13 2022 -0700 MLLinOp::postSolve (#2981) Add a virtual function MLLinOp::postSolve. This allows WarpX to set EB covered nodes to prescribed values in the solver's output for visualization purpose. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 2d87a4c8ad5d375008ee9b1c23a50404fe0dfa21 Author: Brandon Runnels Date: Mon Oct 10 09:49:29 2022 -0600 add templating for the cell bilinear interpolators (#2979) This templates the `mf_cell_bilin_interp` functions so that the interpolators can be used with `BaseFab`s of arbitrary type. Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H commit e4ab0485621d5566c96cae58a816860ee7d4997f Author: Weiqun Zhang Date: Wed Oct 5 12:03:41 2022 -0700 FillPatcher class (#2972) This adds a class FillPatcher for filling fine level data. It's not as general as the various FillPatch functions (e.g., FillPatchTwoLevels). However, it can reduce the amount of communication data. Suppose we use RK2 with subcycling and the refinement ratio is 2. For each step on level 0, there are two steps on level 1. With RK2, each fine step needs to call FillPatch twice. So the total number of FillPatch calls is 4 in the two fine steps. Using the free function, one ParallelCopy per FillPatch call is needed for copying coarse data for spatial interpolation. With the FillPatcher class, two ParallelCopy calls will be done to copy old and new coarse data. Then these data will be used in the four FillPatcher::fill calls. This new approach saves two ParallelCopy calls per coarse step for a two levels run. It could save more if the time stepping requires more substeps or the refinement ratio is higher. Note that many of our AMReX codes use a time stepping algorithm that needs only one FillPatch call per step. For those codes, this new approach will not save any communication for a refinement ratio of 2. However, it will save communication when the refinement ratio is 4. Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatcher.H Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Tests/Amr/Advection_AmrCore/Source/AdvancePhiAllLevels.cpp Tests/Amr/Advection_AmrCore/Source/AdvancePhiAtLevel.cpp Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tests/Amr/Advection_AmrCore/Source/Src_K/Make.package Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp commit 1bc4e4eb5a25f4bdf9933695ead86f17dfdee9ed Author: Weiqun Zhang Date: Mon Oct 3 16:50:45 2022 -0700 Remove sycl namespace alias (#2971) This causes a conflict with new compilers. Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_Math.H Src/Base/AMReX_RandomEngine.H commit de7b7f44afda2227368a30646faeeea0d4679bec Author: Weiqun Zhang Date: Mon Oct 3 14:06:58 2022 -0700 Fix Tensor Solver BC (#2930) This fixes some bugs in the physical domain BC of tensor linear solver. At the corner of two no-slip walls (e.g., (0,0)), we have u(-1,0) = -u(0,0) and u(0,-1) = -u(0,0). It's incorrect to fill the corner ghost cell with u(-1,-1) = u(-1,0) + u(0,-1) - u(0,0), because it will result in u(-1,-1) = -3 * u(0,0). In the old approach, to avoid branches in computing transverse derivatives on cell faces, we fill the ghost cells first. For example, to compute du/dy at the lo-x boundary, we use the data in i = -1 and 0, just like we compute du/dy(i) using u(i-1) and u(i) for interior faces. The problem is the normal velocity in the ghost cells outside a wall is filled with extrapolation of the Dirichlet value (which is zero) and more than 1 interior cells. Because of the high-order extrapolation, u(-1) != -u(0). This is the desired approach for computing du/dx on the wall. However, this produces incorrect results in dudy. In the new approach, we explicitly handle the boundaries in the derivative stencil. For example, to compute transverse derivatives on an inflow face, we use the boundary values directly. Co-authored-by: cgilet Src/Base/AMReX_Orientation.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp_bc.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp_grad.cpp Src/LinearSolvers/MLMG/AMReX_MLTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_K.H commit 13aa4df0f5a4af40270963ad5b42ac7ce662e045 Author: Weiqun Zhang Date: Fri Sep 30 17:48:22 2022 -0700 Disable host device for macros for SYCL/DPC++ (#2969) The host part of the AMREX_HOST_DEVICE_FOR_* macros is disabled for SYCL/DPC++. It's really slow for compilation. Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/EB/AMReX_EB2_GeometryShop.H commit 62379fbac96867437070c4852d3d741a76dc1a4b Author: Weiqun Zhang Date: Fri Sep 30 15:37:35 2022 -0700 Update CHANGES for 22.10 (#2968) CHANGES commit d65e09e4a85dd2765a8cbe0ac9eba6223c47121b Author: Roberto Porcu <53792251+rporcu@users.noreply.github.com> Date: Thu Sep 29 15:46:19 2022 -0400 Solve an issue with particles async IO when having runtime added variables (#2966) Src/Particle/AMReX_WriteBinaryParticleData.H commit cd07b0d84244d08cf2690a19e0312f349ec0aeaa Author: Weiqun Zhang Date: Wed Sep 28 09:20:42 2022 -0700 Fix int overflow in amrex::bisect (#2964) Change from (lo+hi)/2 to lo+(hi-lo)/2. Although it's very unlikely, it's possible (lo+hi), where both lo and hi are integers, could overflow. Src/Base/AMReX_Algorithm.H commit e55d6b4f5375efb22ebed9b467878e301763073b Author: Junghyeon Park Date: Thu Sep 29 01:20:15 2022 +0900 Update the SWFFT project site (#2965) Docs/sphinx_documentation/source/SWFFT.rst commit b84d7c069cef7470f195b250926ca0e84ec46fb2 Author: Weiqun Zhang Date: Mon Sep 26 16:05:10 2022 -0700 Fix MLEBNodeFDLaplacian bottom solver (#2963) MLEBNodeFDLaplacian is never singular because it has Dirichlet boundary on the EB surface. We did set the singular flag to false, but forgot about the bottom solver used a different function to query. This fixes it by overriding the isBottomSingular function. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H commit 5e84f43241edfec7754d3ebfc369154bf249d992 Author: asalmgren Date: Sun Sep 25 09:38:51 2022 -0700 make tagging routines EB_aware (#2962) Src/AmrCore/AMReX_ErrorList.cpp Src/Base/AMReX_VisMF.H Src/Particle/AMReX_ParticleInit.H commit 8b367b0071787f8688d6f7eac55f7be251de6841 Author: Weiqun Zhang Date: Sun Sep 25 09:22:13 2022 -0700 Volume weighted sum (#2961) Add a new function doing volume weighted sum across AMR levels. This may not be exactly what amrex application codes want. But it should work for many cases. Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp commit 2a3cc05dac916961b1a5ae4c18b21bacd889e7fc Author: Weiqun Zhang Date: Fri Sep 23 12:24:05 2022 -0700 CellData: data in a single cell (#2959) This adds struct CellData that allows for accessing data in a single cell in Array4. This is convenient sometimes because one can omit the i, j and k indices. It might also be faster sometimes because it can skip the repeated index calculation involving i,j,k. Src/Base/AMReX_Array4.H commit 27ef10654c4810fc7cfc0f941a3eec67b018bf34 Author: Weiqun Zhang Date: Fri Sep 23 12:23:34 2022 -0700 Quartic interpolation for cell centered data (#2960) New Interpolator for interpolation of cell centered data using a fourth-degreee polynomial. Note that the interpolation is not conservative and does not do any slope limiting. Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp commit c4b7982d067497cc97ccb501ec08720b404d957e Author: Luca Fedeli Date: Fri Sep 23 21:17:12 2022 +0200 Add GPU-compatible upper bound and lower bound algorithms to AMReX_Algorithm (#2958) Src/Base/AMReX_Algorithm.H commit 3e5cc778028030ecb06bb079c5a6045f8f5fba6e Author: Don E. Willcox Date: Tue Sep 20 17:59:48 2022 -0700 add option for makebuildsources to specify the style arguments for 'git describe'. (#2957) Tools/C_scripts/makebuildinfo_C.py commit a6e0c11989d34b976245db5719eedd0e9040f264 Author: Weiqun Zhang Date: Tue Sep 20 10:01:21 2022 -0700 Add more warnings (#2956) * Add -Wnon-virtual-dtor -Wlogical-op -Wmisleading-indentation -Wduplicated-cond -Wduplicated-branches to gcc. * Add -Wnon-virtual-dtor to clang. * Add more warnings to CI. * Fix some non-virtual dtors and some other warnings. .github/workflows/clang.yml .github/workflows/cuda.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/macos.yml Src/AmrCore/AMReX_ErrorList.H Src/EB/AMReX_distFcnElement.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/llvm.mak commit 826cd378f8ba0d844c64e1029f7914c3b066debd Author: Phil Miller Date: Thu Sep 15 17:26:00 2022 -0700 Add roundoff_lo corresponding to roundoff_hi for domains that don't start at 0 (#2950) * Lay groundwork for roundoff_lo * Add dummy implementation of roundoff_lo computation * implement bisect_prob_lo * change idx -> dxinv * use rlo instead of plo in locateParticle Co-authored-by: atmyers Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H commit 6a5a0561076f62af588d5a3d54f0deb232f3a6af Author: Weiqun Zhang Date: Thu Sep 15 13:23:40 2022 -0700 Add template parameter to ParallelFor and launch specifying block size (#2947) By default, amrex::ParallelFor launches AMREX_GPU_MAX_THREADS threads per block. We can now explicitly specfiy the block size with `ParallelFor(...)`, where BLOCK_SIZE should be a multiple of the warp size (e.g., 64, 128, etc.). A similar change has also been made to `launch`. The changes are backward compatible. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_Reduce.H commit 2cdb9df08e4668bbc9a9b6560217514518f41573 Author: Andrew Myers Date: Thu Sep 15 10:55:41 2022 -0700 Byte spread fixes (#2949) Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerI.H commit 17c94cc196d779e9f7ec48f7d004088a1c1e11c6 Author: Candace Gilet Date: Wed Sep 14 11:49:35 2022 -0400 Correct MultiFab::norm0 doxygen brief description (#2946) Src/Base/AMReX_MultiFab.H commit 0351c9958be7fdc7e3e0c419fc68d36a0c00f288 Author: Axel Huebl Date: Wed Sep 14 08:48:25 2022 -0700 CMake: HIP_PATH from ROCM_PATH (#2948) * On machines like Crusher, `ROCM_PATH` is more likely to be available then a `HIP_PATH` environment variable. This is mainly needed for our hacky ROCTX hints. * ROCTX: New Include Supposedly, there is a new include we shall use: Ref.: https://github.com/ROCm-Developer-Tools/roctracer/issues/79 * ROCtracer: Include as System library Because of GNU extensions in the roctracer include files for the legacy include. But we should make this `-isystem` anyway to be robust for the future. The 5.2 deprecated include file `` throws warnings because they rely on GNU extensions: ``` In file included from /opt/rocm/hip/../roctracer/include/ext/prof_protocol.h:27: /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:70:7: warning: anonymous structs are a GNU extension [-Wgnu-anonymous-struct] struct { ^ /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:70:7: warning: anonymous types declared in an anonymous union are an extension [-Wnested-anon-types] /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:75:7: warning: anonymous types declared in an anonymous union are an extension [-Wnested-anon-types] struct { ^ /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:82:7: warning: anonymous structs are a GNU extension [-Wgnu-anonymous-struct] struct { ^ /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:86:7: warning: anonymous structs are a GNU extension [-Wgnu-anonymous-struct] struct { ^ /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:90:7: warning: anonymous structs are a GNU extension [-Wgnu-anonymous-struct] struct { ^ /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:82:7: warning: anonymous types declared in an anonymous union are an extension [-Wnested-anon-types] struct { ^ /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:86:7: warning: anonymous types declared in an anonymous union are an extension [-Wnested-anon-types] struct { ^ /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:90:7: warning: anonymous types declared in an anonymous union are an extension [-Wnested-anon-types] struct { ^ ``` * GNUmake: Update Includes in `hip.mak` Use public prefix. .github/workflows/hip.yml Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_TinyProfiler.H Tools/CMake/AMReXParallelBackends.cmake Tools/GNUMake/comps/hip.mak commit 9aa23c202a13eee489a06030b9aeda6b89856944 Author: Cody Balos Date: Mon Sep 12 11:49:37 2022 -0700 Fix minor typo in fcompare docs (#2945) Docs/sphinx_documentation/source/Post_Processing.rst commit bfbd68f4ed31ca07572be9bf138a59cacb7e800c Author: Axel Huebl Date: Mon Sep 12 11:40:55 2022 -0700 Fix: Make Finalize->Initialize->F->I->... Work (#2944) Fix assertions in Arena::Initialize. The_BArena never dies (tm) Co-authored-by: Weiqun Zhang Src/Base/AMReX_Arena.cpp commit 67384701a808ca973ad2c24ec86cee4c7a81fd05 Author: Weiqun Zhang Date: Wed Sep 7 14:12:34 2022 -0700 Changes for Cray & Clang (#2941) * It seems that the new Cray compilers no longer define `_CRAYC`. However it does define `__cray__`. * For Clang based Cray compilers, use -O3 instead of -O2 for optimization. * Clang's vectorization pragma is very aggressive. For some codes, it makes ParallelFor with many if statements on CPU much slower than without vectorization. Unfortunately, it does not have an ivdep pragma. So we disable AMREX_PRAGMA for clang for safety. * No longer need to use -Wno-pass-failed for Clang based compilers. .github/workflows/hip.yml .github/workflows/macos.yml Src/Base/AMReX_Extension.H Tools/CMake/AMReXFlagsTargets.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/armclang.mak Tools/GNUMake/comps/cray.mak Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/llvm.mak commit 5b0c598cc71a5e914bfc4dbb7ea44313d45c8f57 Author: Weiqun Zhang Date: Wed Sep 7 09:42:57 2022 -0700 Fix a warning in packing communication send buffer (#2940) When we communication double precision data in single precision, there is a conversion from double to float in packing the send buffer. A static cast is added to fix the warning. Src/Base/AMReX_FBI.H commit 3e397bb6ba2854245a10d49a5ee37e1ba9f33f0e Author: Weiqun Zhang Date: Wed Sep 7 09:13:53 2022 -0700 Link to cublas when using CUDA and Hypre (#2933) Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp Tools/GNUMake/packages/Make.hypre commit 9525ea8892b9c0910acc2bf2ae8950f6068c34e5 Author: Weiqun Zhang Date: Wed Sep 7 09:13:20 2022 -0700 HIP: use coarse grained host memory (#2932) Src/Base/AMReX_Arena.cpp commit 7e040166efc8208e60d8796d4d99b1dd47146ef2 Author: Marco Garten Date: Wed Sep 7 08:53:20 2022 -0700 Update Testing Docs (#2937) - document `abort_on_unused_inputs` - remove duplicate superfluous argument in regtest call Docs/sphinx_documentation/source/Testing.rst commit 539427a19b20e49c4f7399c8ea0b0515fb5c79a0 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Tue Sep 6 18:13:42 2022 -0400 EB checkpoint files (#2897) * support for loading EB from checkpoint file * add support for writing chkpt file as well Co-authored-by: Weiqun Zhang Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_IndexSpace_chkpt_file.H Src/EB/AMReX_EB2_IndexSpace_chkpt_file.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EB2_Level_chkpt_file.H Src/EB/AMReX_EB2_Level_chkpt_file.cpp Src/EB/AMReX_EB_chkpt_file.H Src/EB/AMReX_EB_chkpt_file.cpp Src/EB/CMakeLists.txt Src/EB/Make.package commit 35ed6b4d343215c1ccf6e4d0a59813fc236c9f22 Author: Axel Huebl Date: Tue Sep 6 15:07:16 2022 -0700 Fix: Loading Files Again (#2936) This enables that `amrex::ParmParse::addfile` can be called multiple times. Before this, we accidentially overwrite the `FILE` static keyword. Follow-up to #2842 Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit 8f8198c2fb1868704d2b4d14b5b93d8d1d264ea0 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Tue Sep 6 13:36:35 2022 -0400 Check if boundary particles container has been created before clearance. (#2935) This fixes a segmentation fault when using more GPUs for updating particles than fluid. Src/Particle/AMReX_NeighborParticlesI.H commit fb0b31e1439b089074514f45ae900af257c66dba Author: Nuno Miguel Nobre Date: Sun Sep 4 05:18:49 2022 +0100 SYCL: Replace deprecated atomic types and operations (#2921) * SYCL: Replace deprecated atomic types and operations * Change atomic refs to device memory scope When using the relaxed memory order, the memory scope is ignored. Thus, for cosmetic reasons only, we set the memory scope to device, the broadest option when using the global address space. Co-authored-by: Weiqun Zhang Src/Base/AMReX_GpuAtomic.H commit cc3cd1470254d37f0cea4f212c2b0f6ffa8d0bee Author: Weiqun Zhang Date: Thu Sep 1 07:39:25 2022 -0700 Update CHANGES for 22.09 (#2934) CHANGES commit acc223f9918284e7d8e595d3861c5e456d84a968 Author: Weiqun Zhang Date: Tue Aug 30 16:04:43 2022 -0700 Add hypre as an option for OpenBCSolver (#2931) Src/LinearSolvers/OpenBC/AMReX_OpenBC.H Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp commit 3d29fd7d0e816f3c436112d90bdefe815e0ff72a Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Wed Aug 24 16:10:22 2022 -0400 Preserve neighbor particles when sorting particles. (#2923) Src/Particle/AMReX_ParticleContainerI.H commit 8294c3afbcbbc503f77e493196d380fbe1666d02 Author: Weiqun Zhang Date: Mon Aug 22 10:46:05 2022 -0700 Scope of NonLocalBC::ParallelCopy (#2922) Make NonLocalBC::ParallelCopy accessible in namespace amrex, because it can be useful in situations other than non-local BC. Src/Base/AMReX_MultiFab.H Src/Base/AMReX_NonLocalBC.H commit 0911fc4b2e066209a590c330bf2ddf7178dca76b Author: Weiqun Zhang Date: Sun Aug 21 18:13:07 2022 -0700 Open Boundary Poisson Solver (#2912) This adds an open boundary Poisson solver based on the James's algorithm. To use it, the user builds an amrex:OpenBCSolver object, which can be reused until the grids change, and then call OpenBCSolver::solver. Currently, this is for 3D cell-centered data only. The solver works on CPU, Nvidia GPUS, and AMD GPUs. The SYCL version of a couple of kernels for Intel GPUs are to be implemented. GNUmakefile.in Src/Base/AMReX_DistributionMapping.cpp Src/Boundary/AMReX_LOUtil_K.H Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/OpenBC/AMReX_OpenBC.H Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp Src/LinearSolvers/OpenBC/AMReX_OpenBC_K.H Src/LinearSolvers/OpenBC/Make.package commit f270b3d5db8f8b7ab010bc9134632361b8a9009c Author: Marc T. Henry de Frahan Date: Thu Aug 18 13:51:56 2022 -0600 Fix OOB access of ref ratio on HDF write header (#2919) Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp commit fa8e20f946b661bd49af2a60898ffca2c5b21cff Author: Jean M. Sexton Date: Thu Aug 18 08:57:51 2022 -0700 Add Polaris to GNUMake (#2908) Tools/GNUMake/Make.defs Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.alcf commit bd5f6a9f6a1a3a66c51eefd7950432d3bf3319a1 Author: Axel Huebl Date: Mon Aug 15 14:24:21 2022 -0700 Export GpuDevice Globals (#2918) * Export GpuDevice Globals Implement symbol export via `AMREX_EXPORT` for the global variables in `Src/Base/AMReX_GpuDevice.H`. Follow-up to #1847 #1847 Fix #2917 * Fix: Export `AMReX::m_instance` Src/Base/AMReX.H Src/Base/AMReX_GpuDevice.H commit 4f639294606d47185d31eaee4af66fc6b590e5a2 Author: asalmgren Date: Sat Aug 13 09:00:02 2022 -0700 enable LinOp to use the right Factory (fixes moving geometry problem) (#2916) Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp commit 659351846da6f930b4f04cc6cd6b9f78e7752e8a Author: Andrew Myers Date: Thu Aug 11 15:24:16 2022 -0700 Use 1 atomic instead of two per item in DenseBins::build (#2911) Src/Particle/AMReX_DenseBins.H commit d295f2299101705f7c470c813b80542296087328 Author: Nuno Miguel Nobre Date: Thu Aug 11 03:40:09 2022 +0100 [SYCL] Remove amrex::oneapi and update deprecated device descriptors (#2910) * Remove amrex::oneapi in favour of standard features * Change deprecated device descriptors Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_Scan.H commit 1bda173b489024d5f4ec79627f3f612c350e521f Author: Axel Huebl Date: Wed Aug 10 15:46:43 2022 -0600 Add: `MultiFab::sum_unique` (#2909) This provides a new method to sum values in a `MultiFab`. For non-cell-centered data, `MultiFab::sum` double counts box boundary values that are owned by multiple boxes. This provides a function that does not double count these and provides a quick way to get only the sum of physically unique values. Co-authored-by: Weiqun Zhang Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp commit 3f715d29c94b473e624aa9ff3fea9b502da25f97 Author: Candace Gilet Date: Mon Aug 8 14:40:28 2022 -0400 In MLMG::mgFcycle, assert that for EB the linop is cell-centered. (#2905) Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 59b0742b9b8c543b2896d76dd07e03c2fe4f1f94 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon Aug 8 14:17:57 2022 -0400 Clear the boundary particle indices' container before updating it. (#2907) This avoids potential segmentation faults when one grid's particles all move to other grids. Src/Particle/AMReX_NeighborParticlesI.H commit 103db6ebe2b570910ac4dbd7d6611e59d80f1a0b Author: Weiqun Zhang Date: Fri Aug 5 15:25:33 2022 -0700 EB: Add Fine Levels (#2881) Add a new function EB2::addFineLevels() that can be used to add more fine levels to the existing EB IndexSpace without changing the coarse levels. This is useful for restarting with a larger amr.max_level. The issue is we build EB at the finest level first and then coarsen it to the coarse levels. If the restart run has a different finest level, the EB on the coarse levels could be different without using this new capability. Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_IndexSpaceI.H Src/EB/AMReX_EB2_IndexSpace_STL.H Src/EB/AMReX_EB2_IndexSpace_STL.cpp Tests/EB/CNS/Source/main.cpp commit 6ebf8ffc2689e23ff2686627e660caf0a10ea315 Author: Jon Rood Date: Thu Aug 4 14:32:59 2022 -0600 Add rpath to lib64 for ZFP. (#2902) Tools/GNUMake/packages/Make.hdf5 commit ed23627d6487306e26b37ed9a97d60fd8148a935 Author: Yadong_Zeng <30739800+ruohai0925@users.noreply.github.com> Date: Thu Aug 4 16:32:21 2022 -0400 change data types from double to amrex::Real, and thus we can use single precision for the hypre IJ interface (#2896) Co-authored-by: yzeng Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit 9ed4f5955b1d5d0e400fd2f233e5e7b83db4e41b Author: Weiqun Zhang Date: Wed Aug 3 16:53:20 2022 -0700 Fix a new bug introduced in #2858 (#2901) We need to take into account that `amrex::Any` stores `MultiFab&` or `MultiFab const&`. Src/Base/AMReX_Any.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 6eaab8c1c9f0e2a21531526dfd170ebe3aad507b Author: Weiqun Zhang Date: Wed Aug 3 13:39:44 2022 -0700 MPMD Support (#2895) Add support for multiple programs multiple data (MPMD). For now, we assume there are only two programs (i.e., executables) in the MPMD mode. During the initialization, MPI_COMM_WORLD is split into two communicators. The MPMD::Copier class can be used to copy FabArray/MultiFab data between two programs. This new capability can be used by FHDeX to couple FHD with SPPARKS. Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_BoxList.H Src/Base/AMReX_MPMD.H Src/Base/AMReX_MPMD.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 94693291667bd0435819aa09cf28a293da226bf4 Author: Weiqun Zhang Date: Mon Aug 1 09:43:21 2022 -0700 MLMG interface (#2858) These changes are made to support a generic type (i.e., amrex::Any) in MLMG. This is still work in progress. But it should not break any existing codes. Src/Base/AMReX_Any.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FabArray.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_temp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit 5a3b3037950937343b7eafd292e5032cb8c7221c Author: Weiqun Zhang Date: Mon Aug 1 09:34:44 2022 -0700 Update CHANGES for 22.08 (#2894) CHANGES commit 48702b48836d9aeb0db931e23ea9cc7d4ad4ccdc Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Thu Jul 28 14:14:19 2022 -0400 Let `selectActualNeighbors` return right after starting if there are (#2886) no particles for communication. Src/Particle/AMReX_NeighborParticlesI.H commit 6a47d89fd12cb06d48e3e0d85eea415274e84a69 Author: kngott Date: Wed Jul 27 17:03:04 2022 -0700 Add Comm Sync to Redistribute (#2891) Src/Base/AMReX_FabArrayCommI.H Src/Particle/AMReX_ParticleContainerI.H commit 51542c85ac18642a2cfb69ea3df3cf544d3d6f42 Author: philip-blakely <46958218+philip-blakely@users.noreply.github.com> Date: Wed Jul 27 17:29:26 2022 +0100 Multi-materials and derived variable output (#2888) ## Summary Output small plots if only derived variables are specified. Also, make DeriveFuncFab a std::function<> instead of plain function-pointer. ## Additional background We have been implementing small-plots for outputing variables at gauges (e.g. pressure at specific gauge locations). We may want to output the derived variable pressure only, and not all state-variables. The if-condition was incorrect in this case. Further, multi-material simulations require a material index in order to compute derived variables, in addition to existing parameters. Making DeriveFuncFab a std::function is sufficient for our purposes. Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_Derive.H commit ce0fb7412dff3ceeec00941ba525e7ecf5ce8015 Author: Andrew Myers Date: Tue Jul 26 16:20:38 2022 -0700 Fix host / device sync bug in PODVector (#2890) Src/Base/AMReX_PODVector.H commit 06753e60aca7d063b28be93379c948e92afb8c5e Author: Axel Huebl Date: Tue Jul 26 12:54:35 2022 -0700 `TagBoxArray::collate`: Fujitsu Clang (#2889) `mpiFCC -Nclang` only defines `__CLANG_FUJITSU`, not `__FUJITSU` as in the classic compiler mode. Src/AmrCore/AMReX_TagBox.cpp commit 7cf77dc60e149ebe822f6b5428556f9208e150fa Author: Weiqun Zhang Date: Tue Jul 26 11:01:21 2022 -0700 MinLoc and MaxLoc Support (#2885) Add struct ValLocPair that can be used by ReduceOps/ReduceData and ParReduce to find the location of the min/max value. Add warp shuffle down function for more general types. This is needed for MinLoc/MaxLoc with CUDA < 11, because we don't use CUB for earlier versions of CUDA. The Intel GPU support is not done yet. We need to allocate enough shared local memory when the size of ValLocPair is larger than the size of unsigned long long. Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_Reduce.H commit 4b7e20057a3dff84beae21812d826d24e19f2109 Author: Weiqun Zhang Date: Thu Jul 21 10:25:57 2022 -0700 HIP: Remove the call to hipDeviceSetSharedMemConfig (#2884) AMD devices do not support shared cache banking. Thanks @afanfa for reporting this. (#2883) Src/Base/AMReX_GpuDevice.cpp commit 8e40952af9ab0600174f491c81100132f9b24c6e Author: Weiqun Zhang Date: Wed Jul 20 12:10:26 2022 -0700 Add Frontier to GNU Make (#2879) Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit b673d81723c5585a1290126233d38f50833939d7 Author: Max Katz Date: Mon Jul 18 15:14:19 2022 -0400 Add option to derefine to AMRErrorTag (#2875) This allows a refinement field to specify *derefinement* (by setting a zone's tagging value to the clear value). Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp commit 73dbf2f909bdc6c497eb5245b4e707b4814e699f Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon Jul 18 12:53:35 2022 -0400 Fix the segmentation fault in selecting actual neighbor particles. (#2877) Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H commit 40b3d2176b17785191050482a2ead5539993fac6 Author: Weiqun Zhang Date: Wed Jul 13 13:24:15 2022 -0700 Add extra braces in initialization of GpuArray (#2876) It should not be needed since C++14. But some compilers seem to need the double braces. Src/Base/AMReX_TableData.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H commit a633d2bff1db1a3335efd077a34b6a8dcfb4e793 Author: Luca Fedeli Date: Fri Jul 8 20:34:18 2022 +0200 Workaround to bypass issue observed at very large scale with Fujitsu MPI (#2874) We have observed some MPI issues at very large scale when WarpX is compiled using Fujitsu MPI (i.e., with the Fujitsu compiler). These issues seem to be related to the use of MPI Gatherv with MPI_Datatype. This PR implements a possible workaround, initially proposed by @WeiqunZhang . The idea is that, when WarpX is compiled with the Fujitsu compiler, simpler integer arrays instead of MPI_Datatype are used in the routine where the issue was observed. Src/AmrCore/AMReX_TagBox.cpp commit 7660c885d46779367344adf88af75e630a0bc77a Author: Weiqun Zhang Date: Fri Jul 8 08:48:14 2022 -0700 Allow zero components MultiFab and BaseFab (#2873) This is useful for particle I/O that does not have any mesh data. yt needs a header file associated with a MultiFab. Src/Base/AMReX_BaseFab.H commit c849dd1994388cebd78a6a1624e80bc3ab640970 Author: Weiqun Zhang Date: Fri Jul 8 08:06:37 2022 -0700 New EB optimization parameter: eb2.num_coarsen_opt (#2872) At the beginning of EB generation, we chop the entire finest domain into boxes and find out the type of the boxes. We then collect the completely covered boxes and cut boxes into two BoxArrays. This process can be costly because of the number of calls to the implicit functions. In this commit, we have introduced a new ParmParse parameter, eb2.num_coarsen_opt with a default value of zero. If for instance it is set to 3, we start the box type categorization at a resolution that is coarsened by a factor of 2^3. For the provisional cut boxes, we refine them by a factor of 2, Then we chop them into small boxes and categorize the new boxes. This process is performed recursively until we are at the original finest resolution. The users should be aware that, if eb2.num_coaren_opt is too big, this could produce in erroneous results because evaluating the implicit function on coarse boxes could miss fine structures in the EB. Thank Robert Marskar for sharing this algorithm. Src/Base/AMReX_Box.cpp Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_IndexSpaceI.H Src/EB/AMReX_EB2_IndexSpace_STL.H Src/EB/AMReX_EB2_IndexSpace_STL.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level_STL.H Src/EB/AMReX_EB2_Level_STL.cpp Tests/LinearSolvers/CellEB2/inputs.rt.2d Tests/LinearSolvers/CellEB2/inputs.rt.3d commit 557aae84902f63a84edc8b49831ee66af7d1a46a Author: Erik Date: Wed Jul 6 08:54:24 2022 -0700 point to new location of AMReX images, AMReX website repo (#2867) README.md commit cbdc6580ee3d78cccdd37172e4ba077ee181f483 Author: Axel Huebl Date: Tue Jul 5 01:41:03 2022 +0200 SENSEI 4.0: Fix Build for Particles (#2869) ## Summary This part causes a compile error now in WarpX. cc @burlen @kwryankrattiger ## Additional background X-ref: Blocks WarpX 22.07 release https://github.com/ECP-WarpX/WarpX/pull/3211 Follow-up to: - #2785 - #2834 Src/Extern/SENSEI/AMReX_AmrMeshParticleInSituBridge.H commit dc8b734b6a70583602150cfbee1b7d51f8dacdeb Author: Andrew Myers Date: Fri Jul 1 17:19:20 2022 -0700 Cache the neighbor comm tags for the CPU implementation of fillNeighbors. (#2862) * Cache the neighbor comm tags for the CPU implementation of fillNeighbors. * fix areMasksValid function Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H commit 2b42fb56a96e752d301916ca23160098c5369386 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Fri Jul 1 18:44:35 2022 -0400 Remove some hard checks in check_mvmc for 3D (#2864) Removing some hard checks in 3D coarsening logic as it appears that those are not necessarily bad states, and a soft failure to coarsen should suffice. Src/EB/AMReX_EB2_3D_C.H commit 19c70685cdb0c3322712e9f442092b1140cfe7ec Author: Erik Date: Fri Jul 1 18:24:24 2022 -0400 Carry over fix for ngbxy.smallEnd typo (#2868) This a typo that got correct in other places but didn't get fixed here. Tests/Amr/Advection_AmrCore/Source/DefineVelocity.cpp commit d736ef299b724b96b34d41103dfc5318d0ecdee4 Author: Weiqun Zhang Date: Fri Jul 1 11:00:15 2022 -0700 Update CHANGES for 22.07 (#2866) CHANGES commit be813d024e6b314e41c727734b8e53481898e08e Author: Weiqun Zhang Date: Fri Jul 1 10:29:13 2022 -0700 Hypre: add version check (#2865) These HYPRE_SetSp* are only available in hypre >= 22500. Src/Base/AMReX.cpp commit 8fb23ec17a58284af6bdafbcda3eea0d86d8ce69 Author: Jon Rood Date: Wed Jun 29 16:52:35 2022 -0600 Refactor Make.nrel to use MPT for MPI with the Intel compiler on Eagle. (#2861) Tools/GNUMake/sites/Make.nrel commit 6f9a46c7e834046970d46d684927a078671355bc Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Wed Jun 29 11:09:57 2022 -0600 Adding control APIs and namespacing for core algorithm paths like SpGEMM, SpMV, and SpTrans. (#2859) Co-authored-by: Paul Mullowney Src/Base/AMReX.cpp commit e4c83cfddc8afb1bd091c45a6ad3040d23f019bc Author: Jon Rood Date: Wed Jun 29 11:08:42 2022 -0600 Add lib64 library location for ZFP since it may exist there instead of lib. (#2860) Tools/GNUMake/packages/Make.hdf5 commit b2b9150ada12af878a07e0628be03668a9d17270 Author: Burlen Loring Date: Tue Jun 28 13:42:41 2022 -0700 update the SENSEI in situ coupling for SENSEI v4.0.0 (#2785) In this release, an install of VTK is no longer required. To compile AMReX w/ SENSEI use: ```cmake -DAMReX_SENSEI=ON -DSENSEI_DIR=//cmake ``` Note: may be `lib` or `lib64` or something else depending on your OS and is determined by CMake at configure time. See the CMake GNUInstallDirs documentation for more information. .github/workflows/sensei.yml Docs/sphinx_documentation/source/Visualization.rst Src/Extern/SENSEI/AMReX_AmrDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrInSituBridge.cpp Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrMeshInSituBridge.cpp Src/Extern/SENSEI/AMReX_AmrMeshParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrMeshParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_InSituUtils.H Src/Extern/SENSEI/AMReX_InSituUtils.cpp Src/Extern/SENSEI/AMReX_ParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_ParticleDataAdaptorI.H Tools/CMake/AMReXThirdPartyLibraries.cmake commit 2c5f475d451aede47fe2cad2bbd8681c9ca1f456 Author: Andrew Myers Date: Tue Jun 28 12:51:19 2022 -0700 Write runtime attribs to checkpoints on GPUs (#2856) Src/Particle/AMReX_WriteBinaryParticleData.H commit d2cb54668b5e49fd35a60164f40ad6f36720f806 Author: Jon Rood Date: Tue Jun 28 13:27:02 2022 -0600 Fix gnu make on Crusher for mpi_gtl_hsa (#2857) Update environment variable at OLCF for mpi_gtl_hsa. Tools/GNUMake/sites/Make.olcf commit 21fe4b3016a796b99c409760cfad7ae00a7475ba Author: Axel Huebl Date: Tue Jun 28 19:53:09 2022 +0200 CMake: FindDependency CUDAToolkit (#2849) If we install AMReX with CUDA support using a modern CMake, we need to repopulate targets such as `CUDA::curand` from `find_dependency` for downstream. Downstream users find us via `find_package` and that target link dependency showed up to be unpopulated in MFIX. Tools/CMake/AMReXConfig.cmake.in commit 027f2ff77fed33a191cfc735d8adaabb42d21743 Author: Weiqun Zhang Date: Thu Jun 23 16:15:57 2022 -0700 Fix make help (#2854) This reverts the change in #2845, which fixed an issue with `make print-%`, but broke `make help`. This is now fixed in a different way. Both `make print-%` and `make help` should work now. Tools/GNUMake/Make.rules Tools/GNUMake/sites/Make.nersc commit 3d3ad213ca4b60421c9a80328e1316b23435958f Author: kngott Date: Thu Jun 23 13:39:59 2022 -0700 NERSC Programming Environment prototype (#2848) Tools/GNUMake/sites/Make.nersc commit 487267625412e4f8a4fa1ab2492cb578955c4239 Author: Weiqun Zhang Date: Thu Jun 23 12:41:20 2022 -0700 GNU Make: No need to query mpif90 if Fortran is not used. (#2852) This minimize potential issues. Tools/GNUMake/sites/Make.unknown commit fc0d6469f4ad590d576a7109d8719b018838dd86 Author: Weiqun Zhang Date: Thu Jun 23 12:23:55 2022 -0700 Remove f90doc (#2851) We no longer use it. Tools/F_scripts/f90doc/README Tools/F_scripts/f90doc/expr_parse.pl Tools/F_scripts/f90doc/expr_parse.y Tools/F_scripts/f90doc/f90doc Tools/F_scripts/f90doc/htmling.pl Tools/F_scripts/f90doc/stmts.pl Tools/F_scripts/f90doc/typing.pl Tools/F_scripts/f90doc/utils.pl commit 5188a6a28e64dc627c3333d13bebeb0d7250b506 Author: Weiqun Zhang Date: Thu Jun 23 11:09:15 2022 -0700 Explicitly invoke python3 (#2850) According to PEP 394, a python distributor may choose to not provide the python command. In fact, that's what recent versions of macOS do. Tools/Backtrace/parse_bt.py Tools/C_scripts/describe_sources.py Tools/C_scripts/gatherbuildtime.py Tools/CompileTesting/compiletesting.py Tools/F_scripts/dep.py Tools/F_scripts/fcheck.py Tools/F_scripts/find_files_vpath.py Tools/F_scripts/findparams.py Tools/F_scripts/makebuildinfo.py Tools/F_scripts/write_probin.py Tools/GNUMake/Make.defs Tools/Postprocessing/python/column_depth.py Tools/Postprocessing/python/conv_slopes.py Tools/Postprocessing/python/dumpparthistory.py Tools/Postprocessing/python/test_helmeos.py Tools/Postprocessing/python/test_parseparticles.py Tools/Py_util/plotsinglevar.py Tools/Release/ppCleanup.py Tools/Release/ppCleanupDir.py Tools/Release/release.py Tools/libamrex/configure.py Tools/libamrex/mkconfig.py Tools/libamrex/mkpkgconfig.py Tools/libamrex/mkversionheader.py Tools/typechecker/typechecker.py commit 2d931f63cb4d611d0d23d694726889647f8a482d Author: Andrew Myers Date: Wed Jun 22 15:03:50 2022 -0500 Maintain the high end of the 'roundoff domain' in both float and double precision (#2839) * Maintain the high end of the 'roundoff domain' in both float and double precision * fix shadowing * fix warning * fix float conversion warning * fix logic * Update Src/Base/AMReX_Geometry.H * Update Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H commit 104466dee1b298e0f854f0485871437b89e6b0ec Author: Ryan Sandberg Date: Tue Jun 21 10:39:48 2022 -0700 add Ok to coordsys (#2844) Src/Base/AMReX_CoordSys.H commit bd42adec2b7a7d1b023f73a62142fefd5835b5e9 Author: Junghyeon Park Date: Wed Jun 22 02:35:12 2022 +0900 Correct the parameter name in the usage string (#2847) Tools/Postprocessing/C_Src/IntegrateComp.cpp commit 478fd8a4ac98be1b484b81819d9be6b4704262c3 Author: kngott Date: Mon Jun 20 00:38:29 2022 -0700 Double quotes (#2845) Tools/GNUMake/sites/Make.nersc commit 3ba934070e893c877a12f89c83806064f758e765 Author: Ben Wibking Date: Sat Jun 18 14:00:29 2022 +1000 Fix typo in README.md (#2846) README.md commit bf79c456a9a5fe6b22f3ba51b5973cd44b03c244 Author: Axel Huebl Date: Fri Jun 17 00:25:19 2022 +0700 ParamParse: Add Files at Runtime (#2842) Add a new runtime API to load whole files. This was currently only possible during the first initialize. Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit 6f72de283c38ef3e38ed7c5e8760a176434e6be3 Author: Weiqun Zhang Date: Wed Jun 15 09:37:15 2022 -0700 Fix a pathological case for 2d EB (#2840) What could happen is a cell might be cut a tiny bit at a corner such that 3 faces with an area fraction of one and one face with an area fraction of almost one, and the volume fraction is one. In that case, the boundary area and centroid have been set to wrong values based on an incorrect assumption. Src/EB/AMReX_EB2_2D_C.cpp commit a06cb417b9e4a1a1a32f78431c09620d107e3bab Author: Ben Wibking Date: Thu Jun 16 01:22:14 2022 +1000 add fvolumesum to GNUmakefile (#2836) * add fvolumesum to GNUmakefile and CMakeLists.txt * initialize vol as NAN Tools/Plotfile/CMakeLists.txt Tools/Plotfile/GNUmakefile Tools/Plotfile/fvolumesum.cpp commit cd166dda59b8491a5e5a19cb5c002b62556da73a Author: averytrevino <107513200+averytrevino@users.noreply.github.com> Date: Tue Jun 14 15:42:36 2022 -0700 Replaced Checkpoint with WritePlotFile to fix particle plot file (#2841) Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp commit fe37f7a0403fd225eafcef699c49809bf9859d9f Author: Phil Miller Date: Tue Jun 14 12:34:19 2022 -0700 Clamp particles shifted from plo boundary against rhi, rather than back to plo (#2814) Src/Particle/AMReX_ParticleUtil.H commit 833e61c076a708ae1680eab68713a2d40637085d Author: Axel Huebl Date: Wed Jun 15 00:47:05 2022 +0700 Fix: CMake NVTX not only Hypre (#2837) NVTX is used also without Hypre in AMReX. Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake commit adfdab0d4f03f8d201663d300c7d32723101b310 Author: kwryankrattiger <80296582+kwryankrattiger@users.noreply.github.com> Date: Tue Jun 14 10:58:50 2022 -0500 Update sensei CI container for sensei v4.0 integration (#2834) * Update sensei CI container for sensei v4.0 integration * Add simple script for building and pushing sensei container .github/workflows/docker/sensei/Dockerfile .github/workflows/docker/sensei/build-container.sh .github/workflows/docker/sensei/install_deps.sh .github/workflows/docker/sensei/install_sensei.sh .github/workflows/docker/sensei/install_vtk_minimal.sh .github/workflows/docker/sensei/vtk_use_mpi.patch commit 204bd7cf7353e7ccbfdcfdf7834d069b7da33d0f Author: Weiqun Zhang Date: Tue Jun 14 08:56:26 2022 -0700 HIP Memory Advise : Set managed memory to coarse grain (#2835) For unsafe atomics to work on managed memory, we must set it to coarse grain. For gfx90a, we can explicitly use unsafeAtomicAdd. Src/Base/AMReX_Arena.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuDevice.cpp commit e39b9647db148cca0d30010e8861a8dbe5bd966f Author: Axel Huebl Date: Tue Jun 14 22:45:22 2022 +0700 CMake: Fix `export` with `AMReX_INSTALL=OFF` (#2838) When introducing `AMReX_INSTALL`, the `export` of targets broke for `OFF`. This generalizes this to work in either case. CMakeLists.txt Tools/CMake/AMReXInstallHelpers.cmake commit 1a2fc3dbee1ce0c168cdb66710458757e92a0605 Author: Andrew Myers Date: Mon Jun 13 13:57:32 2022 -0700 make PODVector work with PolymorphicArenaAllocator (#2829) Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_PODVector.H commit 574a77d7a72a8576940b4c26472c32c198826289 Author: Candace Gilet Date: Mon Jun 13 16:09:47 2022 -0400 Re-implement FaceLinear::interp() for InterpFromCoarseLevel (#2831) Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.cpp commit eecb81ec43731d0166176277b1f4edb521da65b2 Author: Max Katz Date: Mon Jun 13 14:35:12 2022 -0400 Make regrid method of Amr class public (#2833) Src/Amr/AMReX_Amr.H commit 4c7b63997421cf4b4f2acf61a0656c000bc4075e Author: Weiqun Zhang Date: Mon Jun 13 10:38:34 2022 -0700 Fix link in readme (#2832) README.md commit a9c5335bb57a81616d17236343bc3126932ecfb8 Author: Weiqun Zhang Date: Mon Jun 13 08:48:55 2022 -0700 amrex::Any (#2827) Add amrex::Any that can be used to store any movable types. Note that std::any can only be used for copy constuctible types. Thus, std::any cannot be used to store MultiFab, whereas amrex::Any can. Src/Base/AMReX_Any.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 74183fe377df7d351e9c548f1aeaa5ec94c0dfe2 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Sat Jun 11 23:26:03 2022 -0400 Fix line integral computation (#2830) Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H commit 5ce1fda77ee366681279ba2e7f7d86d25c9923d9 Author: Andrew Myers Date: Fri Jun 10 10:59:12 2022 -0700 move ParticleContainer into it's own header file (#2822) Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_Particles.H Src/Particle/CMakeLists.txt Src/Particle/Make.package commit a5e6c6b84c2e166b863026604caee2d980e9ee26 Author: Weiqun Zhang Date: Fri Jun 10 09:09:53 2022 -0700 Fix a bug in multigrid grids (#2823) This fixes a bug introduced in #2690. We have to make sure every box in the fine BoxArray is coarsenable, otherwise average down functions will fail. For example, for a 10x10 cells domain broken into four 5x5 cells boxes, we should not add a coarse level because 5x5 is not coarsenable, even though the 10x10 domain is coarsenable. In principle, we could fix this in the average down functions instead. However, these tests with odd numbers of cells are corner cases. So we are fixing it in a relatively easy approach. Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp commit 549c2ec31ffff9ad034bed665eab7ca5c6a67be1 Author: Erik Date: Fri Jun 10 08:56:56 2022 -0700 FAQ: add some answers, split off unanswered questions (#2341) Docs/sphinx_documentation/source/Faq.rst commit 70024ca28a05dfce9dbb14b0c7afd18d2c9099b3 Author: Weiqun Zhang Date: Fri Jun 10 08:54:13 2022 -0700 Fix a typo in Readme (#2821) * Fix a typo in Readme The link to Gallery did not work because of the typo. * Update README.md Add yt to visualization software list Co-authored-by: Erik README.md commit e113ff7dcf29b67618baa88ba4c10311436e4bb7 Author: Erik Date: Thu Jun 9 10:07:41 2022 -0700 Add html, additional sections to README.md (#2775) README.md commit e491cb74e45388bc26878476d7449304e7b1057b Author: Candace Gilet Date: Thu Jun 9 12:30:21 2022 -0400 Allow StateDataPhysBCFunct to operate on face-centered data (#2819) Src/Amr/AMReX_StateData.cpp commit 7b335c5f961746ebc797489bb2391126618518ff Author: Weiqun Zhang Date: Wed Jun 8 15:24:53 2022 -0700 Fix Parser ODR (#2820) Although we define api.prefix, Bison does not use it for all the symbols. So we fix it by manually adding the prefix to yyalloc and yysymbol_kind_t. Src/Base/Parser/amrex_iparser.tab.cpp Src/Base/Parser/amrex_iparser.y Src/Base/Parser/amrex_parser.tab.cpp Src/Base/Parser/amrex_parser.y commit c8f33c76fb8a19bfd11df8393f920230e81191bf Author: Axel Huebl Date: Wed Jun 8 12:33:26 2022 -0700 CMake: Cleanup old nvToolsExt (#2817) Already searched for since #2813 Tools/CMake/AMReX_Config.cmake commit 2c1e462619fbd65354d2500c55853087af316a0e Author: Andrew Myers Date: Wed Jun 8 12:20:22 2022 -0700 Handle the case where we don't have enough device memory for the snd_buffer (#2705) Co-authored-by: Weiqun Zhang Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp Src/Base/AMReX_GpuAllocators.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H commit 18d0a2861d31c52c65752a1d5856f54e08699ce3 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Tue Jun 7 19:04:13 2022 -0400 Use amrex::math::abs in IntVect::maxDir. (#2815) Src/Base/AMReX_IntVect.H commit 6299c2cf414e9c7936fa54a89b014317ac79844c Author: Axel Huebl Date: Tue Jun 7 14:51:01 2022 -0700 CMake: 3.17+ (#2813) * CMake: 3.17+ We are using `find_package(CUDAToolkit ...)` now, which is only supported in CMake 3.17+. Technically, we want to go to 3.20+ soon as well, so that we can drop the old CUDA code paths that we used to forward device flags and CUDA architectures. * Modernize: CMake 3.17+ Required * CMake Modernize: nvToolsExt CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/BuildingAMReX_Chapter.rst Tests/CMakeTestInstall/CMakeLists.txt Tests/SpackSmokeTest/CMakeLists.txt Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/AMReX_Config.cmake commit 8a1c76c0af8961db3168d577898a6d0edaa55891 Author: ldowen <54121008+ldowen@users.noreply.github.com> Date: Mon Jun 6 17:48:51 2022 -0700 Landon/fix bug ghost particles (#2812) Src/Particle/AMReX_ParticleLocator.H commit ff9e832483f142a34da72f9bea5fed72e49fea33 Author: Andrew Myers Date: Fri Jun 3 10:54:46 2022 -0700 Follow-on to 2809; update selectActualNeighbors as well. (#2810) Src/Particle/AMReX_NeighborParticlesI.H commit 3d7f4d09dbbed1d85a83b89e473741e00a2681be Author: Andrew Myers Date: Fri Jun 3 08:16:01 2022 -0700 Generalize the type of callables that can be passed into the neighbor list build function (#2809) Src/Particle/AMReX_NeighborList.H commit c96e79e8409ba1fd9ee1b5f5e005ffaa946d1c4a Author: kngott Date: Thu Jun 2 12:16:47 2022 -0700 Add AVX2 instructions flag. (#2803) Tools/GNUMake/sites/Make.nersc commit 45e3042691ba9352d929cd1d22c5e7db3f182ce2 Author: Weiqun Zhang Date: Thu Jun 2 12:12:41 2022 -0700 Avoid M_PI because it's not in the C++ standard (#2807) Src/Base/AMReX_Geometry.H Tests/EB_CNS/Exec/Pulse/cns_prob.H Tests/LinearSolvers/LeastSquares/initEB.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleDataFor2D.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleDataFor3D.cpp Tools/Plotfile/fvolumesum.cpp commit 6e907ff7cf8ecf6154b261534e63b9c3a863e30e Author: Candace Gilet Date: Wed Jun 1 16:02:39 2022 -0400 In the array version of FillPatchTwoLevels, allow specifying an (#2800) array of component indicies for the bc functions and BCRec arrays. Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_I.H commit cc3497dee62c2293587054128e695969b66622ba Author: Erik Date: Wed Jun 1 12:40:13 2022 -0400 drop breathe plugin, upgrade sphinx ver to 5.0, remove docutils workaround (#2804) and unused python scripts for generating filelists. Convert :cpp: commands to `` in figure caption for sphinx bug workaround. .github/workflows/docs.yml Docs/sphinx_documentation/add_doxy_headers.py Docs/sphinx_documentation/make_api.py Docs/sphinx_documentation/source/Particle.rst Docs/sphinx_documentation/source/conf.py commit 605b820316277925e8f87a52fb92892ac0614148 Author: Erik Date: Wed Jun 1 12:20:14 2022 -0400 Add doxygen comment for ParticleContainer::Increment (#2776) Src/Particle/AMReX_ParticleContainerI.H commit f6c9f484bd8a20a4479fbaa5ba221235cbb68d26 Author: Weiqun Zhang Date: Wed Jun 1 08:46:34 2022 -0700 Update CHANGES for 22.06 (#2805) CHANGES commit f24582df5fec5cab9f6d368777e057d663b72cb9 Author: Weiqun Zhang Date: Tue May 31 16:38:46 2022 -0700 Solvability of bottom solver: Follow-up on #2783 (#2801) In #2783, the solvability fix at the bottom level of EB nodal projection solver was disabled because we don't have sufficient geometry information. This broke a number of tests. It appears that we should still do the best we can (i.e., use the old approach at the bottom level). Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit 1dff173b3dc14ba881a5fbe0caf9db9e52eb43e4 Author: Andrew Myers Date: Tue May 31 12:05:39 2022 -0700 Specify Sphinx version for build_and_deploy action. (#2802) .github/workflows/docs.yml commit b78921a2d80d95add9ff3ec9b498a96299ec4ed7 Author: Weiqun Zhang Date: Fri May 27 10:46:12 2022 -0700 Fix solvability issue in the nodal solver RAP approach (#2783) In the RAP approach of the nodal solver, the RHS at Neumann boundaries only includes the integral inside the domain. That is it's only half of what its "physical" value is. The usual way of subtracting a constant from the RHS does not work for RAP. We should only subtract half of the constant offset at Neumann boundaries. The computation of the constant offset needed for the solvability fix is also affected by the way how RHS is computed at Neumann boundaries. For EB, the computation of the constant offset is an integral of the RHS multiplied by the volume fraction, and the subtraction also needs to be weighted by the volume fraction. Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit 806108c5b8923f1f8aa0e816b923693860a949a4 Author: Weiqun Zhang Date: Thu May 26 14:20:35 2022 -0700 Fix document on MultiFab reduction (#2796) Docs/sphinx_documentation/source/GPU.rst commit 9ce107acd8a778d78171bc428e47f04e8725a8ef Author: Axel Huebl Date: Thu May 26 11:35:35 2022 -0700 CI: Fix HIP GPG Key (#2795) Split downloading and adding. ``` Warning: apt-key output should not be parsed (stdout is not a terminal) gpg: no valid OpenPGP data found. ``` .github/workflows/dependencies/dependencies_hip.sh commit 68a51ce34515f71bf9c9bd3784c77f8029864607 Author: Erik Date: Thu May 26 14:14:41 2022 -0400 Move 'Example Codes' section under 'Getting Started'. Fix various sphinx warnings. (#2777) Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/EB_Chapter.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GettingStarted.rst commit 72bad8bd0edfd571fce66b09bf33acdebca3df78 Author: Erik Date: Thu May 26 13:33:24 2022 -0400 User's Guide: Add content and links to SUNDIALS page. (#2788) Docs/sphinx_documentation/source/SUNDIALS_top.rst Docs/sphinx_documentation/source/TimeIntegration_Chapter.rst commit 52b72f3f13278064da78609297d4c919b8e3b5b2 Author: kngott Date: Thu May 26 10:32:05 2022 -0700 Profiler Sync Timers (#2784) Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/CMakeLists.txt Tools/GNUMake/Make.machines commit 0fd8e9667c97ca79472f602f8a5eec33fa89d5aa Author: Axel Huebl Date: Thu May 26 10:19:25 2022 -0700 Implement Serial ParallelDescriptor::Gather (#2793) This causes a bug currently in the reduced FieldProbe diags in WarpX. Src/Base/AMReX_ParallelDescriptor.H commit b3ffba60f14052fb2c4cc21d7e310eb2c43b6fa1 Author: Don E. Willcox Date: Wed May 25 19:52:03 2022 -0700 Renamed new internal class variables using m_ convention to fix compiler warnings in debug mode. (#2790) Src/Base/AMReX_TimeIntegrator.H commit 1305eb3d364dab42ae17ad3df67e696068d60415 Author: Jon Rood Date: Tue May 24 15:13:41 2022 -0600 Fix dash/underscore mistake with H5Z-ZFP. (#2792) Tools/CMake/AMReXConfig.cmake.in commit 3dda62c50f2b7a0d256b474ba13ea99b7ae954be Author: David Grote Date: Tue May 24 08:55:37 2022 -0700 Make IntVectFromLocation const (#2789) Src/Extern/amrdata/AMReX_AmrData.H Src/Extern/amrdata/AMReX_AmrData.cpp commit 5d88558d2aeff93855a5c46ec8be1a7ef8545f70 Author: Andrew Myers Date: Mon May 23 10:15:03 2022 -0700 make sure m_particles is sized properly when numLocalTilesAtLevel is called (#2782) Src/Particle/AMReX_Particles.H commit 802701647c146c070eace2536c5a9700c9e74c98 Author: Weiqun Zhang Date: Mon May 23 08:47:07 2022 -0700 Add Gpu::synchronize to Hypre interface (#2786) This fixes a bug introduced in #2754 that removed the null stream. Some of the Gpu::synchronize might not be necessary. But we have to be on the safe side because the synchronization behavior of Hypre is unclear. Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/PETSc/AMReX_PETSc.cpp commit 0540fed3576f8e57b3b110c34ac5bd8735b16b2a Author: drangara <69211175+drangara@users.noreply.github.com> Date: Thu May 19 12:12:39 2022 -0400 Fix for small cells (#2781) Move small cells outside domain extent into iterative approach. Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.cpp commit 994073b931f1a696e50c177fae3689ac0db92996 Author: Don E. Willcox Date: Wed May 18 08:50:29 2022 -0700 Add some timestep controls to the AMReX TimeIntegrator class for its integrate() driver function. (#2780) Src/Base/AMReX_TimeIntegrator.H commit 0a50776a7239119d2c6c8fa9f48e889b991125d0 Author: Axel Huebl Date: Tue May 17 16:40:55 2022 +0200 AmrCore: Include utility (#2778) Add the `` header for `std::move`. Src/AmrCore/AMReX_AmrCore.cpp commit 843a7dff266273a0f5b7b9f6cc9233a278f41fe1 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon May 16 12:56:52 2022 -0500 Fix the bug in the CMake build with AMReX_BASE_PROFILE. (#2774) Src/Base/CMakeLists.txt commit 02726a6a8440e0e6c16d5dbfeab6171a76213427 Author: Erik Date: Mon May 16 13:52:30 2022 -0400 configure value of AMReX_GPU_RDC flag for use in cmake find_package(AMReX ...) (#2770) Tools/CMake/AMReXConfig.cmake.in commit 963294be434ef9d506ee87c1e2dafd390f96a124 Author: Erik Date: Mon May 16 13:51:48 2022 -0400 CI--HIP: wget gpg key from https instead of http (#2771) * CI--HIP: wget gpg key from https instead of http * change other http to https .github/workflows/dependencies/dependencies_hip.sh commit 11cd801494b6ccfd0154a2a96cc1de28985ae3b2 Author: Erik Date: Mon May 16 12:05:06 2022 -0400 Change repo html address to Ubuntu 20.04 (#2766) .github/workflows/dependencies/dependencies_nvcc11.sh commit 903648a9e74e1c882f76bb23609de014bfdfc8e2 Author: Max Katz Date: Mon May 16 11:25:42 2022 -0400 Add an optional volume weighting to AMRErrorTag (#2772) This allows, for example, refining based on the mass in a cell rather than only on its density. A function to obtain the cell volume at runtime given an IntVect, that can be run inside a ParallelFor, is added to Geometry. Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp Src/Base/AMReX_Geometry.H commit 49c935bf129c7b639ce392a53dfa8be8a3aaaba5 Author: Axel Huebl Date: Mon May 16 07:54:12 2022 -0700 Fix: AmrCore Move (#2773) The move constructor and assignment operator for `AmrCore` with particles was broken. When moving `AmrParGDB`, its internal `m_amrcore` pointer needs to be updated, too. Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrParGDB.H commit 9473062293af7901e8ad24b103d063217f13995e Author: Andrew Myers Date: Thu May 12 17:25:01 2022 -0700 Update particle << operator after changes to id/cpu (#2769) Src/Particle/AMReX_Particle.H commit 6686b211973a79cb86f90648333c86859af1c9e1 Author: Weiqun Zhang Date: Thu May 12 17:05:39 2022 -0700 Fix maybe-uninitialized warning in calling mlock (#2768) Src/Base/AMReX_Arena.cpp commit f0c51bb0997ff278a8ac0b8408b18841c809a1c0 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Thu May 12 16:33:31 2022 -0600 this updates to recent Hypre API changes (#2765) Src/Base/AMReX.cpp commit 3be111e9686f4d41632d94e89b72d3776d8a417a Author: Weiqun Zhang Date: Thu May 12 15:21:46 2022 -0700 Fix Wstringop-overflow warning in FabConv (#2767) On Perlmutter, `g++ -O3 -march=znver3` produces lots of stringop-overflow warnings in FabConv. These warnings are false positive because the compiler does not know sizeof(amrex::Real) is either 4 or 8. This commit fixes the warnings. Close #2750 Src/Base/AMReX_FabConv.cpp commit 5aa61b26cbbe798b5fd13c17d9f9ecef7371e186 Author: Weiqun Zhang Date: Thu May 12 09:29:00 2022 -0700 Time step in the AmrLevel test (#2763) Make the dt in the AmrLevel test consistent with that in the AmrCore Test. That is we use the velocity at t+0.5*dt (here dt is from the previous step) to estimate the dt for the next step. Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp commit a629129046d77fcbaf2412a9e4d5ea86c862b91f Author: Weiqun Zhang Date: Thu May 12 08:38:26 2022 -0700 Fix the Advection_AmrCore test (#2761) The time used for computing velocity in the non-subcycling mode is incorrect. Close #2725 Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp commit 0b485513b4def2f64a0fe43ea0613c3302af58f5 Author: Andy Nonaka Date: Wed May 11 17:31:24 2022 -0700 multilevel version of writeplotfiletoascii (#2742) Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp commit 5bbb63f24753353bdcbb8c439fedbc3dfc11d15e Author: Weiqun Zhang Date: Wed May 11 14:02:31 2022 -0700 Avoid the use of null stream (#2754) The default stream (e.g., stream used outside MFIter) used to be the null stream for CUDA and HIP. By default, there is implicit synchronization between the null stream and other streams. To avoid that, the default stream in AMReX is now no longer the null stream. The behavior of Gpu::synchronize being device wide synchronization has not changed. However, for most of its use cases, it can be replaced by a new function Gpu::streamSynchronizeAll that will synchronize the activities on all AMReX streams without performing a device wide synchronization that could potentially interfere with other libraries (e.g., MPI). The behavior of [dtod|dtoh|htod]_memcpy has changed. For CUDA and HIP, these functions used to call the synchronous version of the memcpy. However, the exact synchronization behavior depends on the memory types. For SYCL/DPC++, there is no equivalent form because a queue (i.e., stream) must be specified. Furthermore, there is no guarantee of consistence across different vendor platforms. This has now changed to calling the asynchronous form using the current stream followed by a stream synchronization. Docs/sphinx_documentation/source/GPU.rst Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateData.cpp Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.cpp Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuUtility.cpp Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_Partition.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_TagParallelFor.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_VisMF.cpp Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EB_utils.cpp Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_ParticleBufferMap.cpp Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_SparseBins.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/EB_CNS/Source/CNS.cpp Tests/GPU/AnyOf/main.cpp Tests/GPU/AtomicIf/main.cpp Tests/GPU/RandomNumberGeneration/main.cpp Tests/GPU/Vector/main.cpp Tests/Particles/AsyncIO/main.cpp Tests/Particles/DenseBins/main.cpp Tests/Particles/Intersection/main.cpp Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleReduce/main.cpp Tests/Particles/ParticleTransformations/main.cpp Tests/Particles/Redistribute/main.cpp commit 894a50f9f762beb37b02194f60ce0e39097ecc5a Author: Don E. Willcox Date: Wed May 11 14:02:03 2022 -0700 add scomp and ncomp arguments to IntegratorOps functions. (#2759) Src/Base/AMReX_IntegratorBase.H commit 6264e8123699022bec4af701f9d75aa680215fa7 Author: Jon Rood Date: Wed May 11 12:39:13 2022 -0600 Add HDF5 H5Z-ZFP support in CMake (#2753) Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/AMReX_Config.H.in commit c4ad207584b0ba6cb55bf4529f2c92ce5eea20c3 Author: Axel Huebl Date: Tue May 10 07:57:29 2022 -0700 CUDA On Cray: More Robust w/o Wrapper (#2757) Searching for `curand` explicitly (required by default by AMReX) makes build more robust when we build w/o compiler wrappers. Tools/CMake/AMReXParallelBackends.cmake commit 3ca867b8cce86b9892b45ff78df8630500dc3bb4 Author: Weiqun Zhang Date: Sat May 7 16:36:48 2022 -0700 Reimplement amrex::min and max to Work around an nvcc bug (#2756) This is a workaround for an nvcc bug when compiling ERF in debug mode. Note that because std::min and max have been constexpr functions since C++14 and AMReX uses C++14 now, there is no need to use amrex::min and max anymore. But for backward compatibility, we will still keep them. Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_Algorithm.H commit ef52970b80e308552fdd0f3246ffa993c49a8caa Author: Weiqun Zhang Date: Tue May 3 22:48:17 2022 -0700 Revert "Turn on managed memory by default in The_Arena for HIP (#2734)" (#2752) This reverts commit 09e8b9b1179e3e3d7b9b6e288ebe6d2864a8cc67. The managed memory on AMD GPUs is still extremely slow. Src/Base/AMReX_Arena.cpp commit df9b6944a6d56ccdba725d9af2d7e1c5183c560a Author: Andrew Myers Date: Mon May 2 12:56:54 2022 -0700 Add methods for resetting the ParGDB of a ParticleContainer (#2732) Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_Particles.H commit 2bd5be14b179707efaabaeae0a849864349e9f8c Author: Max Katz Date: Mon May 2 12:41:13 2022 -0700 Switch nvtxRangeStart to nvtxRangePush (#2746) Src/Base/AMReX_GpuDevice.cpp commit 2c2593d6be89b9a9ad27b2a2b664fa73c90e1f17 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Mon May 2 15:36:28 2022 -0400 Eb flow diffusive solve (#2741) Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H commit 01ab678e9c468bf2606cbfb0dbb5e7006d5d2074 Author: Weiqun Zhang Date: Mon May 2 10:37:27 2022 -0700 Update CHANGES for 22.05 (#2749) CHANGES commit 72a26b78e095eb05fe740e5ab0d7f54660b854aa Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon May 2 10:38:46 2022 -0500 Update call_check_pair's interface in selectActualNeighbors to resolve the error in building MFiX. (#2748) Src/Particle/AMReX_NeighborParticlesI.H commit 50466d11e7b293bd9ec5623235c6a0fb9d0c4dcc Author: David Gardner Date: Fri Apr 29 12:17:44 2022 -0700 Update Required SUNDIALS version (#2743) Update SUNDIALS version in AMReXConfig.cmake.in to match AMReXThirdPartyLibraries.cmake Tools/CMake/AMReXConfig.cmake.in commit 3d344ec19655f2583c37c1c89e4e309d0fd6ce7f Author: Weiqun Zhang Date: Thu Apr 28 14:14:41 2022 -0700 Update CUDA repo key (#2744) Nvidia has made changes in the signing keys. https://forums.developer.nvidia.com/t/notice-cuda-linux-repository-key-rotation/212771 .github/workflows/dependencies/dependencies_nvcc10.sh .github/workflows/dependencies/dependencies_nvcc11.sh commit 3cc4a66759c9321d67ceb218c0f3b2b138a5a075 Author: Weiqun Zhang Date: Wed Apr 27 18:01:27 2022 -0700 EB: Geometry generation from STL file (#2728) This extends Hariswaran Sitaraman's ASCII STL reader by adding a GeometryShop on top of it. An IndexSpace based on the STL GeometryShop is implemented. This allows us to generate AMReX EB database from STL files. Also, we have added a binary STL reader. Docs/sphinx_documentation/source/EB.rst Src/Base/AMReX_IntConv.H Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_IndexSpace_STL.H Src/EB/AMReX_EB2_IndexSpace_STL.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level_STL.H Src/EB/AMReX_EB2_Level_STL.cpp Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/CMakeLists.txt Src/EB/Make.package commit 511b3b03810632dfc67ffba54febaa84860bacd6 Author: Weiqun Zhang Date: Wed Apr 27 13:23:52 2022 -0700 Drop CUDA 9 support (#2736) * Modify CI to test CUDA 10.2 instead of CUDA 9.2. * Remove workarounds for CUDA 9.2 * No need to test for CUDA >= 10. * Update Documentation. * Update GNU Make and CMake. .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_nvcc10.sh .github/workflows/dependencies/dependencies_nvcc11.sh Docs/sphinx_documentation/source/GPU_Chapter.rst Src/Base/AMReX_BaseFab.H Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuAsyncArray.cpp Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuElixir.cpp Src/Base/AMReX_IntVect.H Src/EB/AMReX_EB2_IF_Box.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_algoim_K.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Tests/GPU/CNS/Source/CNS_K.H Tools/CMake/AMReX_SetupCUDA.cmake Tools/GNUMake/comps/nvcc.mak commit d7f997ce7f4d3c1d535383e196436d8b859a0aaa Author: Erik Date: Wed Apr 27 14:29:14 2022 -0400 Add advection of tracer particles to Adevection_AmrCore test. (#2722) Tests/Amr/Advection_AmrCore/Exec/GNUmakefile Tests/Amr/Advection_AmrCore/Exec/Make.Adv Tests/Amr/Advection_AmrCore/Exec/inputs Tests/Amr/Advection_AmrCore/Exec/inputs-ci Tests/Amr/Advection_AmrCore/Exec/inputs_for_scaling Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tests/Amr/Advection_AmrCore/Source/DefineVelocity.cpp commit 399ff41c05cea3894007488cf87e509571633ab4 Author: Weiqun Zhang Date: Wed Apr 27 11:22:46 2022 -0700 Disable AsyncOut by default for HIP and DPC++ (#2735) This makes it the same behavior as other builds. We had it on by default because in the past our I/O functions relied on managed memory and there were issues with HIP and DPC++. Src/Base/AMReX_AsyncOut.cpp commit 09e8b9b1179e3e3d7b9b6e288ebe6d2864a8cc67 Author: Weiqun Zhang Date: Wed Apr 27 11:22:19 2022 -0700 Turn on managed memory by default in The_Arena for HIP (#2734) This makes it have the same behavior as CUDA and DPC++ builds. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_Arena.cpp commit a30e8eb12c6a7a9abb82849d14589da02210e451 Author: Weiqun Zhang Date: Wed Apr 27 08:13:41 2022 -0700 Update WarpX's MLEBNodeFDLaplacian for 2D RZ (#2733) This adds support for sigma coefficient in the z-direction for the 2D RZ LinOp MLEBNodeFDLaplacian used by WarpX. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit c768c36d6e0825df78dcb048cfe2ca592a0f738b Author: Junghyeon Park Date: Wed Apr 27 01:04:31 2022 +0900 Fix a typo (#2731) Docs/sphinx_documentation/source/Basics.rst commit fb5917b8dd62670904162a0051fe0b19f2099b85 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Tue Apr 26 09:17:56 2022 -0600 Adding more recent Hypre APIs to initialize the library. (#2729) Co-authored-by: Paul Mullowney Src/Base/AMReX.cpp commit 329f81b889a6ece31c67b1b70f207d6c6b2463b6 Author: Andrew Myers Date: Mon Apr 25 08:54:29 2022 -0700 Allow the computation of neighbor lists between particles of different types. (#2727) Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleUtil.H Tests/Particles/NeighborList/GNUmakefile Tests/Particles/NeighborList/Make.package Tests/Particles/NeighborList/inputs Tests/Particles/NeighborList/main.cpp commit 56661522b4cd5be804e3d114c7dc4e36c37204f3 Author: Akash Dhruv Date: Thu Apr 21 15:21:36 2022 -0400 add option for face linear interpolater in fortran interface (#2726) Src/F_Interfaces/AmrCore/AMReX_fillpatch_fi.cpp Src/F_Interfaces/AmrCore/AMReX_interpolater_mod.F90 commit 034b43ac787f1b94a1815b49fe6c5f24b8a85e55 Author: Jon Rood Date: Wed Apr 20 10:50:55 2022 -0600 Allow some EB code to handle single precision (#2723) Src/EB/AMReX_EB2_IF_Scale.H commit 0d136ea53aed8a20029c4997961528d25e6fa41f Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon Apr 18 13:37:47 2022 -0500 Allow `FillBoundary` to cast messages from double to single precision. (#2708) This PR allows FillBoundary to pack double-precision data into a single-precision buffer and unpack single-precision messages to double-precision FabArray, e.x. mf.template FillBoundary(...). Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_PCI.H Src/Base/AMReX_TagParallelFor.H Src/EB/AMReX_MultiCutFab.H commit d22d7e6d556f308ad6ea27bbaf8c4d311498e105 Author: Weiqun Zhang Date: Mon Apr 18 09:15:22 2022 -0700 Make EB code more single-precision friendly (#2719) Src/AmrCore/AMReX_Interp_2D_C.H Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_IF_Cylinder.H Src/EB/AMReX_EB2_IF_Ellipsoid.H Src/EB/AMReX_EB2_IF_Polynomial.H Src/EB/AMReX_EB2_IF_Spline.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EBAmrUtil.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_algoim_K.H Src/EB/AMReX_distFcnElement.cpp commit f2d7f67848de283606a20c646cc528dc240e7bac Author: Weiqun Zhang Date: Sun Apr 17 14:33:51 2022 -0700 Fix FluxRegister::SumReg for GPU builds (#2718) It's incorrect to assume that the fused version of reduce is only used on FabArrays with a simple BoxArray. FluxRegister::SumReg uses the fused version of reduce, and its BoxArray has a special BATransformer. A FluxRegister only has cells on the faces of regular Boxes, but the meta-data provided to the GPU reduce function is for regular Boxes. This results in out-of-bound errors. This commit fixes the issue by extending the meta-data caching mechanism to include special BoxArrays used by FluxRegisters. We have also refactored Batransformer's opeartor==. Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp commit b593587a3c58dc52efde2fba83b5813d44741c5b Author: Jean M. Sexton Date: Fri Apr 15 13:30:15 2022 -0700 Swap hip lib and include ordering (#2717) Update Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/hip.mak commit f63c9cf37c541c09376aa417ef6e8ae608725c42 Author: Weiqun Zhang Date: Fri Apr 15 10:06:07 2022 -0700 Fix integer overflow warnings in FabConv (#2716) Partially address #2713. Src/Base/AMReX_FabConv.cpp commit 625dd4d976cad44076ede3e9d2ffe2dda4d64322 Author: Weiqun Zhang Date: Fri Apr 15 08:48:09 2022 -0700 Make AsyncOut::Finish safe to call when async_out is not enabled. (#2715) Close #2714 Src/Base/AMReX_AsyncOut.cpp commit fc55c67f72d2842caa6bd7b908dfe82df910ec75 Author: Weiqun Zhang Date: Thu Apr 14 15:41:33 2022 -0700 flushParForInfo: remove erroneous assertion (#2712) Src/Base/AMReX_FabArrayBase.cpp commit b383b5844d29d190c930ec8a972656d5615e096d Author: Arnur Nigmetov Date: Thu Apr 14 09:47:49 2022 -0700 Replace double with Real in Advection and Parser tests. (#2711) Otherwise tests do not compile for single precision: function signatures do not match the methods they are supposed to override and std::min/max requires same types. Co-authored-by: Arnur Nigmetov Tests/MultiBlock/Advection/main.cpp Tests/Parser/main.cpp commit a537fb0516d52cdff9a0da5bfe9cf1571c100e4c Author: ldowen <54121008+ldowen@users.noreply.github.com> Date: Mon Apr 11 15:24:10 2022 -0700 Link particles in ghost cells to closest box (#2685) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleLocator.H commit c79cff9ba5640ddcb4ce4cb00e4e22df354bb821 Author: Andrew Myers Date: Mon Apr 11 12:54:39 2022 -0700 Remove make_pair from particle reduction functions. (#2707) Src/Particle/AMReX_ParticleReduce.H commit 872ce5692775fd0d659db001b838328806f5ae90 Author: Andrew Myers Date: Mon Apr 11 08:58:45 2022 -0700 Generalize the type of lambdas that can be passed into ParticleReduce (#2697) Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleUtil.H Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/ParticleReduce/main.cpp commit 7e2572f141556be19890110a1673a3e9f3280267 Author: Weiqun Zhang Date: Mon Apr 11 08:58:32 2022 -0700 Set USE_PARTICLES=TRUE in some tests. (#2706) This is a follow-up on #2703. Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/GNUmakefile Tests/Particles/NeighborParticles/GNUmakefile Tests/Particles/ParallelContext/GNUmakefile Tests/Particles/ParticleArray/GNUmakefile Tests/Particles/ParticleReduce/GNUmakefile Tests/Particles/ParticleTransformations/GNUmakefile Tests/Particles/Redistribute/GNUmakefile Tests/Particles/TypeDescriptor/GNUmakefile commit d70f70e2f56ea4b0623e6933746c4463617cbaf0 Author: Jon Rood Date: Sat Apr 9 16:51:02 2022 -0600 Return non-zero if nans were found when using fnan. (#2704) Tools/Plotfile/fnan.cpp commit 8c3c4c9b771dc267ac31cb446e9c20597f27f6b0 Author: Andrew Myers Date: Sat Apr 9 15:49:15 2022 -0700 Reorganize Make.package in Src/Particle (#2703) * reorganize Make.package in Src/Particle * now must turn on particles in HDF5 test Src/Particle/Make.package Tests/HDF5Benchmark/GNUmakefile commit e5dda4061229fc091e3c80836d0148dbc78bb2f4 Author: Brandon Runnels Date: Sat Apr 9 16:36:48 2022 -0600 FillPatchUtil edge interpolater fix (#2701) This PR addresses a compilation error that occurs when using templated FabArrays that are not MultiFab. (The static_cast of the abstract mapper Interpolater prevented compilation with derived and templated Interpolater objects.) To fix this, a templated InterpFace function was introduced that can be used by derived Interpolater, but that provides default behavior with casting for regular Interpolaters. (Note: this is similar to what is done with FillPatchInterp). Src/AmrCore/AMReX_FillPatchUtil_I.H commit a0668e9b25f15eb3307e21478893d4c03883e211 Author: Don E. Willcox Date: Fri Apr 8 15:25:18 2022 -0700 Adds SUNDIALS integrator options for flexibility. (#2700) This PR adds some options to the SUNDIALS integrator interface for greater flexibility in choosing which integration methods SUNDIALS uses internally. Docs/sphinx_documentation/source/TimeIntegration_Chapter.rst Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_TimeIntegrator.H Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H commit 1a5d8917fd49ed63a65c2ae9f499d6a9160e291d Author: Jon Rood Date: Thu Apr 7 11:22:43 2022 -0600 Only error if HDF5 is not parallel if MPI is enabled. (#2699) Tools/CMake/AMReXThirdPartyLibraries.cmake commit 84a1f7d0715b2a2f33338304bbdfc7a7e018a2b6 Author: Andrew Myers Date: Wed Apr 6 11:15:52 2022 -0700 Make the ParticleReduce functions not rely on ParIter. (#2695) Src/Particle/AMReX_ParticleReduce.H commit 321ff599c4872bce3486eb9c8d7edc1cc5353c47 Author: Andrew Myers Date: Wed Apr 6 09:52:36 2022 -0700 Use pc.WritePlotFile instead of pc.Checkpoint in Tests. (#2694) Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_Particles.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Particles/AssignDensity/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/main.cpp commit ec648a92a89ffc907a87d4b02ba4ddb8a2a932a5 Author: Weiqun Zhang Date: Tue Apr 5 07:52:00 2022 -0700 Fix various warnings (#2693) .github/workflows/cuda.yml .github/workflows/gcc.yml Src/Base/AMReX_Geometry.cpp Src/Base/Parser/GNUmakefile Src/Base/Parser/amrex_iparser.lex.cpp Src/Base/Parser/amrex_parser.lex.cpp Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_nd.F90 Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp Tests/EB/CNS/Source/diffusion/cns_eb_diff_mod.F90 Tests/Particles/ParticleIterator/main.cpp commit 917afc0e18961d602b55d550c4d5a6c1b0c3e49a Author: Weiqun Zhang Date: Mon Apr 4 14:58:11 2022 -0700 Semicoarsening in WarpX Linear Solvers (#2690) Add support for semicoarsening in two linear solvers used by WarpX, MLNodeTensorLap and EBNodeFDLaplacian. The semicoarsening approach here is different from the existing approach for the nodal projection solver. There the grids are coarsened regularly as much as possible and then we switch to coarsening in directions that are still coarsenable, whereas here we do semicoarsening first in specified direction and then switch to regular coarsening if the specified maximum semicoarsening level has not been reached. Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp commit a45414cdf3a502c0e67127c63962b5224ab3165b Author: Weiqun Zhang Date: Mon Apr 4 14:57:41 2022 -0700 FillBoundaryAndSync (#2683) Add a new function to FabArray that fills ghost cells and synchronize nodal data. Ghost regions are filled with data from the intersecting valid regions. The synchronization will override valid regions by the intersecting valid regions with a higher precedence. The smaller the global box index is, the higher precedence the box has. With periodic boundaries, for cells in the same box, those near the lower corner have higher precedence than those near the upper corner. This function produces the same result as OverrideSync followed by FillBoundary in a single pass. Also modify the implementation of OverrideSync to use this new function, which is more efficient. The nodal linear solvers use OverrideSync to sync up the nodal data. They have been updated to use the new OverrideSync function that does not use an owner mask. So we no longer need to build owner masks except for the top and bottom multigrid levels that need to use the mask in the computation of norms to avoid overcounting. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit 9cca466f0b6e53ed7ef1babf702afd9193c3741a Author: Weiqun Zhang Date: Mon Apr 4 10:14:05 2022 -0700 Simplify The_Async_Arena example (#2692) Remove the use of resize in the example of using The_Async_Arena to avoid confusion. Docs/sphinx_documentation/source/GPU.rst commit 8a2ec577e107ff8802b6f28d4c1ce8edfaa44639 Author: Weiqun Zhang Date: Fri Apr 1 18:38:00 2022 -0700 Add comment on a potential bug in using The_Async_Arena (#2691) Docs/sphinx_documentation/source/GPU.rst commit 2a4bce247e7c7794d76934d133f5100437d579a0 Author: Axel Huebl Date: Fri Apr 1 08:47:40 2022 -0700 macOS: Fix Warning (ranlib, profiler) (#2688) Fix ``` [ 54%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Particle/AMReX_ParticleBufferMap.cpp.o [ 54%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Particle/AMReX_ParticleCommunication.cpp.o [ 55%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Particle/AMReX_ParticleContainerBase.cpp.o [ 55%] Linking CXX static library ../../../lib/libamrex.a /Library/Developer/CommandLineTools/usr/bin/ranlib: file: ../../../lib/libamrex.a(AMReX_BLProfiler.cpp.o) has no symbols /Library/Developer/CommandLineTools/usr/bin/ranlib: file: ../../../lib/libamrex.a(AMReX_BLProfiler.cpp.o) has no symbols [ 55%] Built target amrex ``` when building AMReX with CMake on macOS for C++-only users. Src/Base/CMakeLists.txt commit 58c8e1e9b1e71172a2db903caae633537b26183e Author: Candace Gilet Date: Fri Apr 1 11:41:15 2022 -0400 Minor update to Docs on Linear solver curvilinear coordinate (#2684) Docs/sphinx_documentation/source/LinearSolvers.rst commit 5792f4331a6c7b981345420763f24d7b48a1a7be Author: Weiqun Zhang Date: Fri Apr 1 08:15:14 2022 -0700 Update CHANGES for 22.04 (#2689) CHANGES commit 75e2e7d09e68a0caba4eea2dba7f662b6ecee5bc Author: Axel Huebl Date: Thu Mar 31 16:54:17 2022 -0700 CI: NVHPC New Apt Repo (#2687) Update the NVHPC install instructions to the latest and greatest. Fix failing CI (dependency install). .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_nvhpc21-11.sh .github/workflows/dependencies/dependencies_nvhpc21-9.sh commit 3aafa306ad820bbcc50a4b7c14fe912406427d1d Author: philip-blakely <46958218+philip-blakely@users.noreply.github.com> Date: Thu Mar 31 01:34:59 2022 +0100 Fix parser expressions (#2682) * Fix segmentation faults in amrex::Parser if a constant was not defined. Without this fix, the following segments of code all seg-fault. With this fix, each of them will throw an exception instead. (If the setConstant lines are uncommented, the tests will pass with or without this fix.) { amrex::Print() << "Attempt 2+a" << std::endl; amrex::Parser parser("2+a"); //parser.setConstant("a", (amrex::Real)3.0); const auto parserExec = parser.compile<0>(); AMREX_ALWAYS_ASSERT(parserExec() == 5); } { amrex::Print() << "Attempt 2-a" << std::endl; amrex::Parser parser("2-a"); //parser.setConstant("a", (amrex::Real)3.0); const auto parserExec = parser.compile<0>(); AMREX_ALWAYS_ASSERT(parserExec() == -1); } { amrex::Print() << "Attempt 2*a" << std::endl; amrex::Parser parser("2*a"); //parser.setConstant("a", (amrex::Real)3.0); const auto parserExec = parser.compile<0>(); AMREX_ALWAYS_ASSERT(parserExec() == 6); } { amrex::Print() << "Attempt 3/a" << std::endl; amrex::Parser parser("3/a"); //parser.setConstant("a", (amrex::Real)3.0); const auto parserExec = parser.compile<0>(); AMREX_ALWAYS_ASSERT(parserExec() == 1); } { amrex::Print() << "Attempt -a" << std::endl; amrex::Parser parser("-a"); //parser.setConstant("a", (amrex::Real)3.0); const auto parserExec = parser.compile<0>(); AMREX_ALWAYS_ASSERT(parserExec() == -3); } Src/Base/Parser/AMReX_Parser_Exe.cpp commit 8b691cbf8ddb2973ee0e38c2d95f609562da3348 Author: Ben Wibking Date: Wed Mar 30 03:56:05 2022 +1100 fixes to build CUDA with Clang (#2681) Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_TypeTraits.H commit 9907ac19751854868b25d50d245b2045a00227f6 Author: Axel Huebl Date: Mon Mar 28 19:30:04 2022 -0700 CI: Apple Silicon (#2672) * CI: Apple Silicon Make sure we can cross-compile AMReX from Intel CPUs (x86) to "Apple Silicon"/M1 (arm64). Using fancy universal binaries here that even support both architectures at the same time. Refs.: - https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary - https://cmake.org/cmake/help/latest/envvar/CMAKE_OSX_ARCHITECTURES.html * Universal: w/o MPI - brew MPI is not a universal binary - remove ccache (nothing to ccache here between runs yet) * Fix: Unused Variable Follow-up to #1969 * macOS: Signal Code Explicit if x86 .github/workflows/macos.yml Src/Base/AMReX.cpp commit 1b73099a80e617696fc342fa8ac16e2d63ecab4b Author: Andrew Myers Date: Fri Mar 25 11:24:55 2022 -0700 Use the roundoff domain in enforcePeriodic. (#2679) Src/Base/AMReX_Geometry.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H commit 619630d180311310b6ccf6407af2877258965436 Author: Don E. Willcox Date: Thu Mar 24 19:30:47 2022 -0700 Fix some HIP warnings for IntegratorBase class (#2678) * fix unused variable warning by using AMREX_ALWAYS_ASSERT. * fix unused variable warning in IntegratorBase constructor. Src/Base/AMReX_IntegratorBase.H commit d0375d83cf233e8ddc002a6c3d6a6f0b3342a2ae Author: Weiqun Zhang Date: Thu Mar 24 16:00:51 2022 -0700 Catch CUDA 11.6 bug at compile time (#2677) Make it a compile time error if CUDA 11.6 is used to instantiate GpuBndryFuncFab. This used to be a mysterious runtime error (#2607). Note that if GpuBndryFuncFab is not used in your code, you can still use CUDA 11.6. However, the bug might still be triggered at runtime in other places. Src/Base/AMReX_BCUtil.cpp Src/Base/AMReX_PhysBCFunct.H commit 4f463b6fbba78f8d1e21e19f1c6c50f2a0d0be3c Author: Jean M. Sexton Date: Thu Mar 24 15:16:46 2022 -0700 MPI+OMP+HIP build capability (#2676) * Link explicitly to gnu OpenMP lib * Add HIP exception for AMREX_USE_OMP and _OPENMP matching Tools/CMake/AMReX_Config.H.in Tools/GNUMake/comps/hip.mak Tools/libamrex/mkconfig.py commit 95b9b145b74de944a365255191731d78b599a19d Author: drangara <69211175+drangara@users.noreply.github.com> Date: Thu Mar 24 15:33:44 2022 -0400 EB Flow (#2661) There are 2 sets of changes in this PR to support flow from/through embedded boundary faces * support for setting EB Dirichlet value for velocity in the nodal solver - this includes computation of surface integrals and adding the EB flow contribution to the divergence. * adding a new call for the method EB_computeDivergence which takes in the velocity at EB faces and adds the EB flow contribution to the divergence. Co-authored-by: Jordan Musser Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/LinearSolvers.rst Src/EB/AMReX_EBMultiFabUtil.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_algoim.H Src/EB/AMReX_algoim.cpp Src/EB/AMReX_algoim_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_eb.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 2ccb8166b22c853ffe1ae3049ca44ddc082b4236 Author: Jean M. Sexton Date: Wed Mar 23 16:47:42 2022 -0700 BackTrace workarounds (#2667) * Change -g to -gdwarf-4 for gcc 11 or newer * Use addr2line if eu-addr2line gives ? Co-authored-by: Weiqun Zhang Src/Base/AMReX_BLBackTrace.cpp Tools/GNUMake/comps/gnu.mak commit e213d767baab66574b2f29daf23e7288e6f1a8df Author: Revathi Jambunathan <41089244+RevathiJambunathan@users.noreply.github.com> Date: Tue Mar 22 22:20:30 2022 -0700 Redefine ParticleContainter multifab after setting particle boxarray and dmap (#2673) * call redefine mf after set particle BA and dmap * fix test Co-authored-by: Andrew Myers Src/Particle/AMReX_ParticleContainerBase.cpp commit ee651cd5b301bc820435fda03a5eea271dfa8701 Author: Andrew Myers Date: Tue Mar 22 19:57:05 2022 -0700 Handle the case where we lose a level in RedistributeGPU (#2670) Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particles.H Tests/Particles/Redistribute/inputs.rt.cuda.mr Tests/Particles/Redistribute/main.cpp commit 1ac5d7919b44f9c589253226c0c512acf27fd0d3 Author: Andrew Myers Date: Tue Mar 22 13:42:07 2022 -0700 Follow on to 2668, fix new bug in test. (#2671) Tests/Particles/ParticleMesh/main.cpp commit cfe53e955d05f07668a9918ff2fab4775454e49f Author: Don E. Willcox Date: Tue Mar 22 11:57:00 2022 -0700 Add override statements to avoid certain compiler warnings with GCC and Clang. (#2669) Src/Base/AMReX_FEIntegrator.H Src/Base/AMReX_RKIntegrator.H Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H commit 0082c129d340a7b4c414f96af443bbb7743006a0 Author: Axel Huebl Date: Tue Mar 22 11:46:42 2022 -0700 Doc: AMReX_TP_PROFILE (#2659) Document third-party profiling options. Doc-string taken from CMake `option()`. Docs/sphinx_documentation/source/BuildingAMReX.rst commit 336ca7002b45f3237336dfd8e2c4334437eb404b Author: Jean M. Sexton Date: Mon Mar 21 20:44:56 2022 -0700 Add int overflow assert check before Bcast wrapper (#2649) Src/Particle/AMReX_ParticleInit.H commit a2f18e83148d8031134e39649e9dec8a1698ff51 Author: Andrew Myers Date: Mon Mar 21 15:46:16 2022 -0700 Allow ParticleToMesh and MeshToParticle to access runtime particle components (#2668) Src/Particle/AMReX_ParticleUtil.H Tests/Particles/ParticleMesh/main.cpp commit 3c5afdc9dd15af28523d1c7ea16909b3bb3f67d7 Author: Weiqun Zhang Date: Mon Mar 21 12:56:44 2022 -0700 Fix bugs in the MultiBlock test (#2662) Tests/MultiBlock/Advection/main.cpp commit 36fa76c4282a31b3fbcce752c4f2b353400f6748 Author: Andrew Myers Date: Mon Mar 21 12:53:18 2022 -0700 Separate version strings for particle plotfiles and checkpoints. (#2663) Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_WriteBinaryParticleData.H commit 7b4d10daf9c4fdb011801bd5fd2562d6b438f3b0 Author: Weiqun Zhang Date: Fri Mar 18 10:14:29 2022 -0700 Fix warnings (#2665) * Unused variables in FluxRegister * Extra tokens at end of #endif directive according to gfortran Src/AmrCore/AMReX_FluxRegister.cpp Tools/CMake/AMReX_Config.H.in commit dfb0e098df468235f8f7b6fd50d06914c365177f Author: Houjun Tang Date: Wed Mar 16 17:33:50 2022 -0700 Add support for SZ compression in HDF5 output (#2644) Docs/sphinx_documentation/source/IO.rst Docs/sphinx_documentation/source/Visualization.rst Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Extern/HDF5/AMReX_ParticlesHDF5.H Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.H Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Tests/HDF5Benchmark/GNUmakefile Tests/HDF5Benchmark/inputs Tests/HDF5Benchmark/main.cpp Tests/HDF5Benchmark/sz.config Tools/GNUMake/packages/Make.hdf5 commit 04621415c77cbec25fd68ee7077a6ab50b37893d Author: Scot Breitenfeld Date: Wed Mar 16 12:11:45 2022 -0500 Removed the use of H5Aget_storage_size (#2656) Src/Extern/HDF5/AMReX_ParticleHDF5.H commit bbe5da81ee2a902c838ffc8d479dcf775e342b41 Author: Erik Date: Wed Mar 16 11:09:38 2022 -0400 Docs to install volpack via package (#2654) Docs/sphinx_documentation/source/Visualization.rst commit 3c644cb0a46a1eb730ed2eaa0c6cc00180ad017b Author: Max Katz Date: Tue Mar 15 08:58:41 2022 -0700 Add prefetchToHost and prefetchToDevice for GPU containers (#2655) Src/Base/AMReX_GpuContainers.H commit 806f2373b68efa37120c393264647178304720b7 Author: Axel Huebl Date: Fri Mar 11 11:57:22 2022 -0800 Introduce `AMReX_Version.H` (#2653) The defined `AMREX_GIT_VERSION` and `AMREX_RELEASE_NUMBER` change in every commit. This modifies the `AMReX_Config.H` file as well for every commit, which is included in every header file. Using always the latest commit counteracts `ccache` since the signature of the compilation call changes every commit. Move this define and `AMREX_RELEASE_NUMBER` to a separate version file, which is included only where needed (currently in a single `.cpp` file). Co-authored-by: Weiqun Zhang Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_Version.cpp Src/Base/CMakeLists.txt Src/Base/Make.package Tools/CMake/AMReXGenerateConfigHeader.cmake Tools/CMake/AMReX_Config.H.in Tools/CMake/AMReX_Version.H.in Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/libamrex/mkversionheader.py commit c5dce693676eb7ac505b64eb1f88fa9a8873b67c Author: Weiqun Zhang Date: Thu Mar 10 10:34:27 2022 -0800 Add move assignment operator to Fab (#2652) Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_IArrayBox.H commit fbdfc840de7c30d8edc01a0eedeb3edddc19e5b5 Author: Don E. Willcox Date: Wed Mar 9 19:33:49 2022 -0800 Added an option to set fast MRI dt directly instead of using the integer ratio of slow/fast dt. (#2651) Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_TimeIntegrator.H Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H commit 5e3404e6779629d7f3365e31ca520bbfaca0911b Author: Axel Huebl Date: Wed Mar 9 19:31:26 2022 -0800 AMReX_Config.H: AMREX_EXPORT_DYNAMIC (#2650) Forgot to add the new define there. Tools/CMake/AMReX_Config.H.in commit 6bbcbbb7829902e9038bee8c28be09800b22aab2 Author: Andrew Myers Date: Fri Mar 4 11:44:48 2022 -0800 Fix bug affecting neighbor proc calculation with mesh refinement. (#2646) Src/Particle/AMReX_NeighborParticlesI.H commit 3fe001b36504ee40dbe3131543b7265864d7e5b2 Author: Weiqun Zhang Date: Thu Mar 3 08:28:12 2022 -0800 Update flags for DPC++ AOT (#2643) Argument spir64_gen-unknown-unknown-sycldevice has been deprecated recently. We need to use spir64_gen instead. Note that we are not trying to support old oneAPI compilers. So we simply replace the deprecated argument with the new one. Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit d544b2a146cfefb008cf8d86da20f569b37927d6 Author: Axel Huebl Date: Thu Mar 3 08:27:51 2022 -0800 Doc + CMake: AMREX_EXPORT_DYNAMIC (#2635) - Add the CMake control for `AMREX_EXPORT_DYNAMIC` - Add documentation Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake commit 57fbb98d2a780c2a4ee3e2305771466013c0733d Author: Erik Date: Thu Mar 3 02:39:28 2022 -0500 Update SpackSmokeTest to run with Fortran and/or HIP. (#2629) * Update SpackSmokeTest to run with Fortran and/or HIP. * add error checks for cmake versions depending on GPU_BACKEND Tests/SpackSmokeTest/CMakeLists.txt commit 0ce37796a789db8f0f60a4169bc7db8f99dea732 Author: Weiqun Zhang Date: Wed Mar 2 18:14:28 2022 -0800 Simplify CoordSys::IsRZ and IsSPHERICAL (#2642) In the past, the enum types RZ and SPHERICAL were not defined in 3D and therefore could not be used in IsRZ and IsSPHERICAL in 3D. This is no longer the case. Src/Base/AMReX_CoordSys.H commit b982aa80781165b6962c3c7d964bb996fab28ae4 Author: Andrew Myers Date: Wed Mar 2 13:42:45 2022 -0800 Take advantage of async copies in PODVector. (#2641) Src/Base/AMReX_PODVector.H commit ab28124273f74265c948744076b9cd0c2e48c02e Author: Weiqun Zhang Date: Wed Mar 2 12:04:15 2022 -0800 HIP: atomicAddNoRet (#1809) It has been deprecated since ROCm 4.1.0. However, for performance reasons, we should still use it for versions < 5. Src/Base/AMReX_GpuAtomic.H commit 02eb74f1b62c8bf496c9b8646bbaa7cb6a20d1a4 Author: Erik Date: Wed Mar 2 11:34:46 2022 -0500 Add some doxygen comments to MLABecLaplacian (#2628) Adds some doxygen comments to ML_ABecLaplacian code. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp commit 7e73c1ed1c142b83afce7bcaec731d2dc334fbe3 Author: Weiqun Zhang Date: Tue Mar 1 10:29:18 2022 -0800 Update CHANGES for 22.03 (#2639) CHANGES commit 3199edf6152589fd04db5fe814d9e0d8a604ba2b Author: Axel Huebl Date: Mon Feb 28 09:47:58 2022 -0800 ParticleTile: push_back_real/int w/ Vector (#2634) Add more overloads to `push_back_real`/`int` to support `amrex::Vector` and its const iterators. This makes initialization routines more natural (similar to `std::vector` constructors). Src/Particle/AMReX_ParticleTile.H commit 067830e6c9fe74af9ae1e54e4359174bff6344a4 Author: Candace Gilet Date: Fri Feb 25 11:57:28 2022 -0500 Update Docs to reflect newer options in MLMG (#2620) Co-authored-by: etpalmer63 Docs/sphinx_documentation/source/LinearSolvers.rst commit aa0dc39f50f714a76a7b5df630a641cc4e411642 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Fri Feb 18 14:14:17 2022 -0600 Allow the ghost-particle exchange to communicate different variables (#2627) than particle redistribution on GPU. Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_Particles.H commit c45975c0894af19ce7c623fac767ff4807537643 Author: Andrew Myers Date: Fri Feb 18 11:28:16 2022 -0500 Fix AmrParticleContainer::AssignDensity when ncomp == AMREX_SPACEDIM+1. (#2632) Src/AmrCore/AMReX_AmrParticles.H commit 1d566b101fa3f6796c87410376f92351602b7c6c Author: Andrew Myers Date: Wed Feb 16 23:15:42 2022 -0500 Use async version of cuda/hipMemcpy in PODVector::resize() (#2631) Src/Base/AMReX_PODVector.H commit d618810d677d485f48e8de2ae12cb3e9c79ba3c1 Author: Axel Huebl Date: Wed Feb 16 12:09:42 2022 -0800 ParticleContainer::make_alike (#2630) This creates a new AMReX particle container type with same compile-time and run-time attributes. But, it can change its allocator. This is helpful when creating temporary particle buffers for filter operations and device-to-host copies. Src/Particle/AMReX_Particles.H commit 76d08651adb987e3fba6b232e806c5e7c365a8d9 Author: Andrew Myers Date: Mon Feb 14 13:10:52 2022 -0500 Update checkpoint format to account for expanded particle ids. (#2624) Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Extern/HDF5/AMReX_ParticlesHDF5.H Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H commit df93daec8da2d5df7057affacd629d18233f40be Author: Candace Gilet Date: Mon Feb 14 12:05:49 2022 -0500 Fix formatting. Note cuda-memcheck name change to compute-sanitizer (#2626) Docs/sphinx_documentation/source/Debugging.rst Docs/sphinx_documentation/source/GPU.rst commit 0f55de0e2ef6a66ae2c0e5aabc7979c8cc140e04 Author: Weiqun Zhang Date: Thu Feb 10 09:29:54 2022 -0800 Update FabArray::copyTo (#2603) Remove the two versions taking a sub box. I don't think those versions are used by any codes. This allows for simplification of the implementation. Reimplement copyTo to reduce the number of bcast to just one. Also the new versions work without managed memory. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H commit 9558c8a1684c54bc5ffd10c6b2f8f7396c462761 Author: Michael Zingale Date: Tue Feb 8 16:21:29 2022 -0500 explicitly request python3 for makebuildinfo_C.py (#2621) some systems might still have a python 2 interpreter around Tools/C_scripts/makebuildinfo_C.py commit 8e2eaa4a7caf3fed4ebd9aed975c62ae540fe842 Author: Michael Zingale Date: Tue Feb 8 11:05:20 2022 -0500 store the CUDA version in build info (#2617) For codes that use makebuildinfo_C.py, this now allows you to get the CUDA version to potentially store in output files. Tools/C_scripts/AMReX_buildInfo.H Tools/C_scripts/makebuildinfo_C.py commit 3fb8caaf78a04928944d6a535bb675b82160311f Author: ashesh2512 <36968394+ashesh2512@users.noreply.github.com> Date: Mon Feb 7 12:19:10 2022 -0700 missing check for mapped solver support when not using GPU (#2616) Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 212ffa61c1651f041393f46efbfccaabd29be3e8 Author: kngott Date: Fri Feb 4 13:10:29 2022 -0800 Remove ifdefs for older compiler version. (#2614) Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_Scan.H commit 32c9faa8b17abf8cf74736155c7978f34957808b Author: philip-blakely <46958218+philip-blakely@users.noreply.github.com> Date: Fri Feb 4 18:43:17 2022 +0000 Allow a descriptor component's BndryFuncFabDefault to be stateful. (#2612) ## Summary This allows a boundary condition function to depend on the descriptor index or other information only available when BndryFunc::setComponent() is called. ## Additional background Previously, the BndryFuncFabDefault function passed to BndryFunc::setComponent() pointed to a function that did not take descriptor index or other information. In some cases (e.g. multi-material simulations where each material is in its own Descriptor), extra information is needed to map the boundary-function call to the correct material's boundary condition. Making BndryFuncFabDefault a std::function instead of a plain function pointer makes this possible. Src/Amr/AMReX_StateDescriptor.H commit f79f40d4d0cb71bf0ca2271f76433c2c1cb0c029 Author: Weiqun Zhang Date: Thu Feb 3 20:41:33 2022 -0800 TableData: fix missing return in operator= (#2613) Src/Base/AMReX_TableData.H commit bf267b0954263f9f7d9c340c74bf50efa070f8cc Author: Don E. Willcox Date: Tue Feb 1 09:57:07 2022 -0800 SUNDIALS MRI Interface for AMReX TimeIntegrator class (#2600) This PR adds an interface to the SUNDIALS ARKODE integrator for the AMReX TimeIntegrator class to use, identical to what we have implemented for the ERF code. This will let us move that implementation into the more abstract AMReX time integration API to allow others to take advantage of it. The new capabilities are described in the Docs page for Time Integration. This capability is a work in progress and we will add features as we need them. Docs/sphinx_documentation/source/TimeIntegration_Chapter.rst Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_TimeIntegrator.H Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package commit 86a4e467b24343398dd8104da412531594172639 Author: Weiqun Zhang Date: Tue Feb 1 09:20:05 2022 -0800 Update CHANGES for 22.02 (#2610) CHANGES commit 89153db246e4193526210363bf5a0f1719ea40c3 Author: Weiqun Zhang Date: Mon Jan 31 10:54:04 2022 -0800 Fix I/O bug in Amr class (#2609) The file stream for the header must be closed before the parent directory can be renamed. Closes #2608 Src/Amr/AMReX_Amr.cpp commit 656383586c357670048001a2b37e34ba71ee7d36 Author: Erik Date: Mon Jan 31 12:18:28 2022 -0500 Spack install instructions for Building section. (#2602) Adds instructions for how to install AMReX using Spack to the Building with AMReX section. Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/BuildingAMReX_Chapter.rst Docs/sphinx_documentation/source/GettingStarted.rst commit fd6d6494bb75085b73602c3a30ec68a41383424d Author: Weiqun Zhang Date: Fri Jan 28 14:45:55 2022 -0800 Kernel fusing in Geometry (#2606) Src/Base/AMReX_Geometry.cpp commit c62334c4c5cd98ede8b416582af92b18699523de Author: Weiqun Zhang Date: Fri Jan 28 14:45:37 2022 -0800 Kernel fusing in FluxRegister (#2605) Src/AmrCore/AMReX_FluxRegister.cpp Src/Base/AMReX_TagParallelFor.H commit 2fc7776086225cf0c774bd3ea6f21a2794f3cc42 Author: Weiqun Zhang Date: Fri Jan 28 14:44:38 2022 -0800 TableData: A Multi-Dimensional Array Class (#2601) This class is somewhat similar to FArrayBox/BaseFab. The main difference is the dimension of the array in this class can be 1, 2, 3, or 4, whereas the dimension of FArrayBox/BaseFab is the spatial dimension (AMREX_SPACEDIM) plus a component dimension. This is a feature request by Maui, which will use it to store EOS data. Src/Base/AMReX_BaseFab.H Src/Base/AMReX_DataAllocator.H Src/Base/AMReX_TableData.H Src/Base/CMakeLists.txt Src/Base/Make.package commit f7c9c3bd73473736080c7a606631a54e067aed90 Author: Weiqun Zhang Date: Thu Jan 27 15:16:05 2022 -0800 FabSet::multiFab (#2604) Add public member functions to FabSet to expose the internal MultiFab data. Src/Boundary/AMReX_FabSet.H commit d38b4237444cb22cfadb4ddc63434c8b5a2c9b42 Author: Weiqun Zhang Date: Thu Jan 27 14:04:48 2022 -0800 Disable EB solver's phi-on-centroid for hip for now (#2598) * Disable EB solver's phi-on-centroid for hip for now We need this to work around the out-of-register issue with current compilers. Co-authored-by: Axel Huebl Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_F.cpp commit 874ccc6f3f8dedf842082816f6c8aa2dfceb5830 Author: Candace Gilet Date: Mon Jan 24 11:47:55 2022 -0500 Docs: Update info on print_state to include ng (#2599) Docs/sphinx_documentation/source/Debugging.rst commit 3a2a74fc4b383fdda1d85abb7cd34fe8a6888362 Author: Erik Date: Sun Jan 23 13:43:26 2022 -0500 Update locations for tutorials and minor formatting. (#2583) ## Summary Update remaining locations for tutorials referenced in the docs. ## Additional background Follow up to #2576 Docs/sphinx_documentation/source/Fortran.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GettingStarted.rst Docs/sphinx_documentation/source/LinearSolvers.rst Docs/sphinx_documentation/source/Testing.rst Docs/sphinx_documentation/source/Visualization.rst commit be204df0f3056217d2037f03d26dbcf432c295bf Author: Axel Huebl Date: Fri Jan 21 12:20:54 2022 -0800 CI: AMD `hip::device` also for Fortran (#2595) ## Summary By now (HIP 4.5), we should be able to link also against Fortran targets. Let's remove work-arounds and ditch old HIP releases, so improve maintainability and avoid surprises. ## Additional background X-ref: - https://github.com/ROCm-Developer-Tools/HIP/pull/2280 (https://github.com/ROCm-Developer-Tools/HIP/pull/2190) - https://github.com/ROCm-Developer-Tools/HIP/issues/2275 .github/workflows/dependencies/dependencies_hip.sh .github/workflows/hip.yml Tools/CMake/AMReXParallelBackends.cmake commit 947263e0076e08e3ef44db8009fd2ebd46238bd0 Author: Michael Zingale Date: Fri Jan 21 13:46:36 2022 -0500 fix vol initialization for Cartesian (#2596) Tools/Plotfile/fvolumesum.cpp commit e3b1468b8f16d11eebf4b10cca32829c78fb5d35 Author: Erik Date: Thu Jan 20 18:33:53 2022 -0500 Add headings and steps to plot vector field in ParaView doc section (#2588) Adds directions for ploting a vector field from 3 separate scalar values using the Calculator filter. Also split up the ParaView section by adding some headings. Docs/sphinx_documentation/source/Visualization.rst Docs/sphinx_documentation/source/Visualization/ParaView_vectorfield.png commit 3318e3678579dbf4d1af12b2a3d3ad7d1eb6877d Author: Weiqun Zhang Date: Thu Jan 20 10:30:08 2022 -0800 Kernel fusing in FabArrayUtility (#2593) Src/Base/AMReX_Box.H Src/Base/AMReX_FabArrayUtility.H commit b7050b410812f7772985fbe28716f4980c5bc942 Author: Candace Gilet Date: Wed Jan 19 16:16:53 2022 -0500 Change FPinfo's fact_fine_patch to always have ng=1. (#2591) This fixes a problem when using face_centered FillPatchTwoLevels with EB and an odd number of ghost cells, and fact_fine_patch's ebflag fab's box was too small. Src/Base/AMReX_FabArrayBase.cpp commit 12548009735ee00762ade0e90c52eaba3780b903 Author: Weiqun Zhang Date: Tue Jan 18 16:46:43 2022 -0800 Support arbitrary refinement ratio in FaceLinear Interpolater (#2590) Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.cpp commit fe7ef3402085c77c75e1e8a4ebcabc19f6020ea6 Author: Weiqun Zhang Date: Tue Jan 18 15:14:12 2022 -0800 Default Arena for MultiFab/iMultiFab/FabArray (#2589) * Default Arena for MultiFab/iMultiFab/FabArray Add a new constructor to MultiFab/iMultiFab/FabArray that can be used to set the default constructor that will be used when `define` is called with MFInfo containing a nullptr as Arena. Below is an example of using this. MultiFab mf(The_Pinned_Arena()); MultiFab mf2; VisMF::Read(mf, "mf"); VisMF::Read(mf2, "mf2"); // mf is allocated with The_Pinned_Arena(), whereas mf2 is allocated // with The_Arena(), in VisMF::Read. Co-authored-by: Andrew Myers Src/Base/AMReX_FabArray.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_iMultiFab.cpp commit be0d73dd6b67c9855fafb43e5c2fc8c2bfc9b3e6 Author: Erik Date: Tue Jan 18 12:23:10 2022 -0500 `add_par` precedence in Docs. (#2575) Docs/sphinx_documentation/source/Basics.rst commit 812836f08a758ef6f65caf7eea477bfc6423e72b Author: Candace Gilet Date: Mon Jan 17 20:36:00 2022 -0500 Fix typo in 1D version of AMREX_LAUNCH_DEVICE_LAMBDA_DIM (#2587) Src/Base/AMReX_GpuLaunch.H commit 89ccf0a4ca0044bd27e885b0050fdcb071b20cfe Author: Jean M. Sexton Date: Mon Jan 17 20:35:26 2022 -0500 CMake: remove nonexistent header file, update minimum sundials version (#2586) This makes CMake more consistent with expecting Sundials 6.x. Src/Extern/SUNDIALS/CMakeLists.txt Tools/CMake/AMReXThirdPartyLibraries.cmake commit 07267a0ed24dce8a482708904d57e3ae233eeed3 Author: Erik Date: Mon Jan 17 20:32:21 2022 -0500 Enable CUDA in CMakeLists.txt for Spack smoke test. (#2581) Tests/SpackSmokeTest/CMakeLists.txt commit 65f906216a5ec2abd3917213fb30e70b68bb6c68 Author: Weiqun Zhang Date: Thu Jan 13 14:39:05 2022 -0800 Fix warning from hiprand header (#2585) Src/Base/AMReX_RandomEngine.H commit 47c2b927f8b55dee7d4a80145253ac3d6c8f2204 Author: Axel Huebl Date: Thu Jan 13 11:07:36 2022 -0800 HIP: amdclang++ (#2582) On Crusher, `clang++` from AMD is named `amdclang++`. Add this to the known compiler name list and do not warn when encountered. Tools/CMake/AMReXParallelBackends.cmake commit 5576188c11d1fa834c650a6447e36dd1925897f0 Author: David Gardner Date: Thu Jan 13 10:33:02 2022 -0800 Update sundials sycl memory helper interface (#2580) Update usage of the sundials sycl memory helper for sundials 6.x Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp commit cde8e1c073d775f59ac04c8fc6152bb1a6dc8594 Author: Erik Date: Thu Jan 13 12:14:08 2022 -0500 Update locations from `amrex/Tutorials` to (#2576) `amrex-tutorials/ExampleCodes`. Docs/sphinx_documentation/source/AmrCore.rst Docs/sphinx_documentation/source/AmrCore_Chapter.rst Docs/sphinx_documentation/source/AmrLevel_Chapter.rst Docs/sphinx_documentation/source/Basics_Chapter.rst Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GettingStarted.rst Docs/sphinx_documentation/source/LinearSolvers_Chapter.rst Docs/sphinx_documentation/source/Particle.rst Docs/sphinx_documentation/source/Visualization.rst commit e34f5c67fec9745e643ed2605b82419fdddb8967 Author: Axel Huebl Date: Wed Jan 12 15:38:54 2022 -0800 HIP: Add -munsafe-fp-atomics to CMake (#2577) Otherwise atomicAdd is much slower. Same as already done in GNU Make. Tools/CMake/AMReXParallelBackends.cmake commit 83ec5fcfefae13264827a0724ab61d2d1b9a7eac Author: Weiqun Zhang Date: Wed Jan 12 14:52:56 2022 -0800 ParmPaser::queryAdd (#2573) Add a new member function, queryAdd, to ParmParse. If the name is found, the value in the ParmParse database will be stored in the reference argument. If not, the value in the argument will be added to the ParmParse database. The return value indicates if it existed previously. Many codes use ParmParse::dumpTable to store the ParmParse data in plot/checkpoint files. But for parameters that not present in inputs files or command line arguments, they would not appear in the dumped tables. The new queryAdd function can be used to address this issue. In amrex/Src, the calls to `query` are replaced by `queryAdd` so that it's easy to find their values. Src/Amr/AMReX_Amr.cpp Src/AmrCore/AMReX_AmrMesh.cpp Src/Base/AMReX.cpp Src/Base/AMReX_Arena.cpp Src/Base/AMReX_AsyncOut.cpp Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_ForkJoin.cpp Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MemProfiler.cpp Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParmParse.H Src/Base/AMReX_RKIntegrator.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_parstream.cpp Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_Level.H Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/Extern/SENSEI/AMReX_InSituBridge.cpp Src/F_Interfaces/Octree/AMReX_octree_fi.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_TracerParticles.cpp Src/Particle/AMReX_WriteBinaryParticleData.H commit 0d04b17d22d8ddb8920848eac213f6bdb858d8a0 Author: Weiqun Zhang Date: Wed Jan 12 12:22:02 2022 -0800 Fix regression test configuration (#2574) Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini commit 44388b2cd8f2ed166de70078f2f738fd689221ca Author: Erik Date: Wed Jan 12 12:55:27 2022 -0500 Links to tutorials in rst docs. (#2572) Add links to tutorials in Getting Started. In basics, change tutorials to example codes for differentiation. Also fixes broken heat equation link, and removes dead pointer. Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/GettingStarted.rst commit 405b490356e51f66fb649311594939c6d5e12067 Author: Erik Schnetter Date: Tue Jan 11 15:51:12 2022 -0500 Convert switch statement to if statement (#2570) The MPI standard does not guarantee that the thread level constants are compile-time constants; they might only be link-time constants which are not allowed in switch statements. Src/Base/AMReX_ParallelDescriptor.cpp commit af2509d7102c770931cf9219a0d649003d0a86a8 Author: Cody Balos Date: Tue Jan 11 11:10:25 2022 -0800 remove erroneous BL_ASSERT in sundials initialization (#2568) Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp Src/Extern/SUNDIALS/AMReX_Sundials.cpp commit 7314e78fecda4080f6bd3a1857d033545f7d8a56 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon Jan 10 19:11:30 2022 -0600 Support implicit function class defined on host when using device memory explicitly. (#2563) AMReX needs to use managed memory if the user-defined implicit function (IF) class's member functions are only callable by the host. This PR allows users to use IF classes defined on the host with device memory. Src/EB/AMReX_EB2_GeometryShop.H commit fa46bc9eff1306b8c3eeaf1ae32d070271606bba Author: Weiqun Zhang Date: Mon Jan 10 13:55:09 2022 -0800 HIP: Add -munsafe-fp-atomics to GNU Make (#2567) Otherwise atomicAdd is much slower. Note this is not done for CMake yet. Tools/GNUMake/comps/hip.mak commit 172cab862a1812437cbd78101592fd2ffcfd2d8f Author: Weiqun Zhang Date: Mon Jan 10 12:06:01 2022 -0800 GNU Make: Add a new machine, crusher (#2566) Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit 0278677bbc8c7fde6edf8fa5ea3cfd0d55db8b1f Author: asalmgren Date: Fri Jan 7 15:47:17 2022 -0800 Mapped solver support (#2088) This change to the AMReX_MLNodeLaplacian* files allows us to pass in a multi-component sigma to the nodal projector and this will be used as {sigmax,sigmay,sigmaz} internally. This allows us to have different weightings for the derivatives in the different coordinate directions which is useful for solves on nonuniform rectilinear meshes. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_hypre.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 1404c4a9acc65bd99a70db5be9a4ccc5ab701e7a Author: Cody Balos Date: Thu Jan 6 16:02:03 2022 -0800 Update to SUNDIALS 6 (#2551) ## Summary Update AMReX to use SUNDIALS 6. We could make this backwards compatible with SUNDIALS 5.7.0/5.8.0 without too much effort (I think), if that is desired. ## Additional background SUNDIALS 6 introduces a `SUNContext` class (needed by all/most other SUNDIALS classes) that should be associated with only one SUNDIALS simulation at a time. Thus, when using multiple threads, multiple `SUNContext` objects are needed. This is why we create one (and one `SUNMemoryHelper`) for each OpenMP thread. Src/Base/AMReX.cpp Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.H Src/Extern/SUNDIALS/AMReX_SUNMemory.H Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp Src/Extern/SUNDIALS/AMReX_Sundials.H Src/Extern/SUNDIALS/AMReX_Sundials.cpp Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package commit f36e8cbd80e189fe3ce69db5c4366023c816654a Author: Weiqun Zhang Date: Thu Jan 6 11:31:49 2022 -0800 Fix a number of warnings (#2564) Src/Base/AMReX_FileSystem.cpp Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_VisMF.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 29b6a8ebb76f43781046fc20b84f7d1616c66888 Author: kngott Date: Thu Jan 6 13:15:43 2022 -0500 Return a zero-size vector in ParallelDescriptor::Gather. (#2560) The Gather implementation that returns a vector currently returns an un-initialized, 1-element vector to the non-root ranks. This seems unintuitive as a user and requires very careful consideration of the returned vector on non-root ranks. This PR changes the non-root return to a zero-size vector instead. Src/Base/AMReX_ParallelDescriptor.H commit aa079d4dc93b5a72935e1616a0431a838adae8a1 Author: Andrew Myers Date: Thu Jan 6 12:26:52 2022 -0500 Add plotfile diff to neighbor particles test. (#2562) Tests/Particles/NeighborParticles/main.cpp commit 98a66afcf4162b5d3539642dd2cd868d661976be Author: Andrew Myers Date: Thu Jan 6 12:24:38 2022 -0500 Add option not to remove particles with negative ids when calling Redistribute. (#2561) This is useful for WarpX when interacting particles with EB walls. The default behavior of Redistribute is not changed by this PR. Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particles.H Tests/Particles/Redistribute/main.cpp commit c34082833c563ff9dab469fb610164ae57dd9a88 Author: Candace Gilet Date: Thu Jan 6 11:49:34 2022 -0500 Update FillPatchTwoLevels for single component face-centered data (#2539) Allow the non-array version of FillPatchTwoLevels to operate on face-centered data and use the valid fine values on the coarse-fine interface during the interpolation to ghost cells. This also changes the FaceLinear Interpolater to use the valid fine data on the coarse-fine interface in the interpolation to ghost cells. Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp commit 5726371bbb1a9fabb68640db67f3482d78fc33c7 Author: Weiqun Zhang Date: Wed Jan 5 10:06:05 2022 -0800 Kernel fusing in FabArray Comm (#2559) Note that amrex::Add has been moved from AMReX_FabArrayUtility.H to AMReX_FabArray.H so that it can be used by AMReX_FabArrayCommI.H. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabArrayUtility.H commit dfee5b74765e4ec7f0bc28830514168595c1d148 Author: Weiqun Zhang Date: Tue Jan 4 14:09:35 2022 -0800 Use MF ParallelFor in error tagging (#2558) Src/AmrCore/AMReX_ErrorList.cpp commit 458679dbff56a7ca18d8bbf3994a1a172fa2e5c3 Author: Weiqun Zhang Date: Mon Jan 3 11:50:26 2022 -0800 iMultiFab I/O (#2495) * iMultiFab I/O Add functions for writing and reading iMultiFab. * Update Src/Base/AMReX_VisMF.H Co-authored-by: Andrew Myers Co-authored-by: Andrew Myers Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_IntConv.H Src/Base/AMReX_IntConv.cpp Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_VectorIO.H Src/Base/AMReX_VectorIO.cpp Src/Base/AMReX_VisMF.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_VisMFBuffer.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp commit dd75ba5ec7276a494e61b5d53226c88be80b5299 Author: Weiqun Zhang Date: Mon Jan 3 09:39:35 2022 -0800 Embed more EB data in EBFArrayBox (#2550) * Embed more EB data in EBFArrayBox Before this, we only had access to EBCellFlag given an EBFArrayBox. This adds access to more EB data via EBFArrayBox. This could be useful for functions with Fab parameters (e.g., AmrLevel's derive functions). * Mark EBFArrayBox(Arena*) explicit Src/EB/AMReX_EBFArrayBox.H Src/EB/AMReX_EBFArrayBox.cpp Src/EB/AMReX_EBFabFactory.cpp Src/EB/AMReX_MultiCutFab.H Src/EB/AMReX_MultiCutFab.cpp commit d36950d8589cad77558519c00f9fa0f46015d5b5 Author: Weiqun Zhang Date: Mon Jan 3 08:36:20 2022 -0800 CpuBndryFuncFab: Face Data (#2545) For completeness, fab_filfc is implemented for CpuBndryFuncFab. However, one should use GpuBndryFuncFab that works for both GPU and CPU builds. Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_FilFC_1D_C.H Src/Base/AMReX_FilFC_2D_C.H Src/Base/AMReX_FilFC_3D_C.H Src/Base/AMReX_FilFC_C.H Src/Base/AMReX_FilFC_C.cpp Src/Base/AMReX_PhysBCFunct.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit a73ce6759c914bdf15c44e58e839a744b5878be3 Author: Weiqun Zhang Date: Sun Jan 2 14:55:14 2022 -0800 Fix bug in Parser (#2555) Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser_Y.cpp commit 39c9cb6221959a8761bebf3b372f97dc65635ff3 Author: Weiqun Zhang Date: Sat Jan 1 14:03:53 2022 -0800 Update CHANGES for 22.01 (#2554) CHANGES commit 7c606b85c6bc22e1321727ed6692dd1e591e2b72 Author: Max Katz Date: Thu Dec 30 21:24:14 2021 -0800 Allow CUDA compilation on Perlmutter if only PrgEnv-nvidia is loaded (#2553) In this scenario, there is not a CUDA module loaded by default; nvcc is provided as part of the NVIDIA HPC SDK. In the Cray PE the environment variable defined pointing to the top level of the HPC SDK is `NVIDIA_PATH` so we can use this to indicate to the make system that we should have CUDA support. Tools/GNUMake/sites/Make.nersc commit bcdb6ca488826e10cf28691b0ed2ecc776753d55 Author: kngott Date: Thu Dec 23 19:52:37 2021 -0500 Add Gatherv for USE_MPI=FALSE. (#2549) This adds a Gatherv implementation for when USE_MPI=FALSE. Src/Base/AMReX_ParallelDescriptor.H commit c5f0685554aaff7588d71a535050121dbc8c393f Author: kngott Date: Thu Dec 23 19:51:26 2021 -0500 Resize vector inside GatherLayoutDataToVector (#2548) The output vector of GatherLayoutDataToVector doesn't resize the resulting vector; the user has to pass a minimally sized vector themselves. This PR resizes the result vector inside the function. Src/Base/AMReX_ParallelDescriptor.H commit 29853f0b30252deabb343e0a919b77b480bbba60 Author: kngott Date: Thu Dec 23 15:09:32 2021 -0500 Add missing FillBoundary_nowait (#2546) The `IntVect nghost` function signature doesn't have a matching `nowait` call. This PR adds it. Src/Base/AMReX_FabArray.H commit 24e25124cc179ee7e5aed1a06c3b9800a156595d Author: kngott Date: Thu Dec 23 15:06:33 2021 -0500 Remove unused variable. (#2547) Src/Base/AMReX_FabArrayCommI.H commit 85284ea15a3c5ca7d6d8e3f1045ac1e5a48c6a19 Author: Weiqun Zhang Date: Tue Dec 21 09:48:05 2021 -0800 Fix compilation for 1D build with hypre (#2544) Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 77ccf474ba9c4123f96be0b56ef08df1985af24a Author: Michael Zingale Date: Tue Dec 21 12:47:40 2021 -0500 add a tool to compute the volume integral of a plotfile field (#2541) Tools/Plotfile/fvolumesum.cpp commit df6893521551ab14b9d87f9610786a42eafcd9d9 Author: Jean Luca Bez Date: Tue Dec 21 08:44:38 2021 -0800 Update read path for HDF5 benchmark (#2542) Update path that was not used after including the directory for read Tests/HDF5Benchmark/main.cpp commit a38ccbc90843d32cf34beb7d89c60d95af3383d3 Author: Don E. Willcox Date: Mon Dec 20 12:26:52 2021 -0800 Eliminate compiler warnings for time integration classes. (#2540) Src/Base/AMReX_FEIntegrator.H Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_RKIntegrator.H Src/Base/AMReX_TimeIntegrator.H commit e217e4a50525d4f973e1a5ba3c2510c4814cf745 Author: Weiqun Zhang Date: Mon Dec 20 11:50:42 2021 -0800 Update MLEBNodeFDLaplacian (#2538) * Update MLEBNodeFDLaplacian * Make it work for non-EB builds too. * Add option to solve in RZ coordinates. In that case, sigma is always one. * Assert that for 2d RZ with the lo-r boundary at r = 0, the domain BC is set to Neumann. Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/Make.package commit 134d33e4b7f0c9999ab9cfc2f1c129c47be31cee Author: Andy Nonaka Date: Tue Dec 14 15:38:46 2021 -0800 fix WritePlotfileToASCII compilation (#2537) Co-authored-by: Jean M. Sexton Tools/Postprocessing/C_Src/Make.package commit c6978ac1efb1afbe8625d1cc48e0c7ae68a648fc Author: Houjun Tang Date: Tue Dec 14 12:12:17 2021 -0800 Add missing const (#2536) Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp commit 9373709e34b23add981551d5446bc4810fd3b688 Author: Jean M. Sexton Date: Fri Dec 10 22:24:55 2021 -0800 Fix assert to use IntVect (#2531) Src/Base/AMReX_MultiFabUtil.H commit 8a1f7651a4723574a276f072a963cc5eb2e7dacc Author: Jean M. Sexton Date: Fri Dec 10 21:04:19 2021 -0800 Update NVector_MultiFab functionality from ATPESC-codes and add to Src/Extern/SUNDIALS (#2505) Adds NVector_MultiFab from https://github.com/AMReX-Codes/ATPESC-codes Updates NVector MultiFab wrapper for the GPU, and adds a NormHelper function Adds a check for SUNDIALS_FOUND in the CMake setup of SUNDIALS as a ThirdPartyLibrary Src/Base/AMReX_MultiFabUtil.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Src/Extern/SUNDIALS/AMReX_Sundials.H Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package Tools/CMake/AMReXThirdPartyLibraries.cmake commit 55afaaf5945ad4b55483a3fe623f50d1768d9be6 Author: Erik Date: Fri Dec 10 20:13:48 2021 -0500 Projections migrated to AMReX-Hydro (#2530) Projections have been moved to AMReX-Hydro. This PR removes them from AMReX. Docs/sphinx_documentation/source/LinearSolvers.rst GNUmakefile.in Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/LinearSolvers/Projections/Make.package Tests/LinearSolvers/MAC_Projection_EB/CMakeLists.txt Tests/LinearSolvers/MAC_Projection_EB/GNUmakefile Tests/LinearSolvers/MAC_Projection_EB/Make.package Tests/LinearSolvers/MAC_Projection_EB/README Tests/LinearSolvers/MAC_Projection_EB/inputs_3d Tests/LinearSolvers/MAC_Projection_EB/main.cpp Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini commit 50f8e4314be75003610378d159134960ad27e64b Author: Andrew Myers Date: Fri Dec 10 19:13:07 2021 -0500 Add another overload for MakeSimilarDM. (#2528) This one works with an input `ba` and `dm` instead of a `MultiFab.` Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp commit dbbac7b9649ce7feecdd80d7e46b97438fb51173 Author: Don E. Willcox Date: Fri Dec 10 12:45:28 2021 -0800 Adds `get` functions to the TimeIntegrator classes (#2529) * Add get_rhs and get_post_update functions to allow user code to retrieve std::functions for the RHS and post_update. Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_TimeIntegrator.H commit afe587b071d9f1a0f2b95125d8d3204a01cf8967 Author: Weiqun Zhang Date: Fri Dec 10 09:47:11 2021 -0800 Fix a bug in Robin BC (#2477) * Fix a bug in Robin BC When the a scalar is zero but A coefficient is not, there was a bug in the handling of Robin BC. We should reset the A coefficient to zero first before modifying it for Robin BC because we need to modify the a scalar to non-zero. Closes #2476 * Fix a bug in the detection of singularity. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp commit 4b695b7b3fe17962bb2645b0b37959adbf38e6ce Author: Erik Date: Fri Dec 10 10:54:25 2021 -0500 Doc refine grid layout (#2527) * CMake file for AmrLevel-SingleVortex * move new test to its own folder * message to indicate where build is coming from * Starting to convert Prob.f90 to cpp. * Convert Prob.f90 to Prob.cpp, runs. Testing next. * remove Adv_F.H from CMake file * Forgot to add the Prob.H file * delete unneeded ctest section * Switch AmrLevel for AmrCore b/c level not WIN32 compatible. * remove WIP comments * remove unneeded changes * Remove unneeded files. * fix words and remove comments * trying to add CUDA to Cmake WIP * some changes to the CMake file, nothing concrete * Add CUDA to CMake file. * add refine_grid_layout to sphinx docs * more add refine_grid_layout to sphinx docs Docs/sphinx_documentation/source/InputsLoadBalancing.rst commit 8d5e756a5a58b1ce08ea4ff69ad8d8e2802e8eb9 Author: Weiqun Zhang Date: Thu Dec 9 16:37:00 2021 -0800 Add ParallelCopyToGhost and fix a bug in FillPatch (#2523) * Add ParallelCopyToGhost function that copies data to ghost cells only. * There was a bug in the Array version of FillPatch for Face data. The issue is, if the destination is the same FabArray as one of the source FabArrays on the fine level, some valid data on valid box faces are overwritten with the interpolated values. This is now fixed with the use of ParallelCopyToGhost in that case. Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H commit fe08a6914b069739332405eec45a8ca79be57ac0 Author: Axel Huebl Date: Thu Dec 9 16:36:32 2021 -0800 CMake 3.22+: Policy CMP0127 (#2525) Fix a warning with CMake 3.22+. ~~We use use sometimes ;-connected AND syntax in `cmake_dependent_option`, so we need to update some of those options once we require CMake 3.22+ only.~~ We use simple syntax in `cmake_dependent_option`, so we are compatible with the extended syntax in CMake 3.22+ https://cmake.org/cmake/help/v3.22/policy/CMP0127.html First seen on Perlmutter. CMakeLists.txt commit 6b3937d41be4756291747fca2f90cdd50c5afe43 Author: Andrew Myers Date: Thu Dec 9 19:35:45 2021 -0500 Add function for make a dmap that is similar to an input MultiFab. (#2526) This was originally a PR to WarpX, but it makes more sense for it to live in AMReX: https://github.com/ECP-WarpX/WarpX/pull/2640 Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp commit fba4eba1e10f87fc2113d476766e568800c81815 Author: Erik Date: Thu Dec 9 12:16:23 2021 -0500 Specify dimensions that can be refined with `refine_grid_layout` (#2516) Adds a pp variable that can be included in the input file to specify which dimensions can be refined with `refine_grid_layout`. Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrMesh.cpp commit e88a72a7c8572ab94a078ddf6736d840bc1dbc6c Author: darylbond Date: Fri Dec 10 02:02:39 2021 +1000 avoid divide by zero in EB (#2519) Add a small number to the variable `apnorm` in the 2D EB code to avoid a divide by zero. Src/EB/AMReX_EB2_2D_C.cpp commit d52e45350f77bb70f051913f794cc205a3c76e97 Author: Luca Fedeli Date: Thu Dec 9 17:01:55 2021 +0100 Fix issue concerning Fujitsu compiler in AMReX_BLBackTrace.H (#2517) The line `#pragma omp threadprivate(bt_stack)` in `AMReX_BLBackTrace.H` can't be compiled with Fujitsu compiler (as well as with Intel compiler and Cray compiler). This PR excludes this line when the Fujitsu compiler is used. Src/Base/AMReX_BLBackTrace.H commit d2bdb9482d43b029da86203c2e6d1580b2f9c742 Author: Erik Date: Thu Dec 9 10:22:27 2021 -0500 Interpolater doxygen (#2524) Co-authored-by: atmyers Src/Particle/AMReX_ParticleInterpolators.H commit 3a7a24d05a94545596cf706a188cf5983f4d0975 Author: Weiqun Zhang Date: Wed Dec 8 15:44:37 2021 -0800 DPCPP: AMREX_INTEL_ARCH (#2459) * DPCPP: AMREX_INTEL_ARCH When AOT is on, set the target to AMREX_INTEL_ARCH, which is required if the SYCL backend is used and AOT is on. * update documentation * Apply suggestions from code review Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake commit f39e091d160cd5c0dcbcf8dc7cc00e3b28e956af Author: Andrew Myers Date: Wed Dec 8 12:27:30 2021 -0500 Doxygen comments for the new interpolator classes (#2521) Src/Particle/AMReX_ParticleInterpolators.H commit acf8694f2b2676a9e0479f3245760663fe46dd3b Author: Andrew Myers Date: Wed Dec 8 11:56:18 2021 -0500 Fix ParticleMesh test after #2475 (#2520) Remove some inadvertent changes to the ParticleMesh test. Tests/Particles/ParticleMesh/main.cpp commit 6780dbf281570c9354ed742ca9c9c4545a0b65d4 Author: asalmgren Date: Tue Dec 7 17:05:31 2021 -0800 Generalize problem size specification (#2518) * Add the option to specify geometry.prob_extent instead of geometry.prob_lo and geometry.prob_hi * add documentation of prob_extent * setting prob_hi should be inside if test * update docs Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/InputsProblemDefinition.rst Src/Base/AMReX_Geometry.cpp commit 84c7864b66247fabdcd0e12665994ea2edd94759 Author: Axel Huebl Date: Tue Dec 7 09:37:43 2021 -0800 CI: Run 2D and 1D Tests (#2475) Co-authored-by: atmyers .github/workflows/gcc.yml Src/Particle/AMReX_ParticleInterpolators.H Src/Particle/CMakeLists.txt Src/Particle/Make.package Tests/AsyncOut/multifab/main.cpp Tests/LinearSolvers/NodalPoisson/CMakeLists.txt Tests/MultiBlock/Advection/CMakeLists.txt Tests/MultiBlock/Advection/main.cpp Tests/MultiBlock/IndexType/CMakeLists.txt Tests/MultiBlock/IndexType/main.cpp Tests/Particles/AssignDensity/main.cpp Tests/Particles/AsyncIO/main.cpp Tests/Particles/GhostsAndVirtuals/CMakeLists.txt Tests/Particles/InitFromAscii/CMakeLists.txt Tests/Particles/NeighborParticles/CheckPair.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleIterator/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/main.cpp Tests/Particles/ParticleMeshMultiLevel/trilinear_deposition_K.H Tests/Particles/ParticleReduce/main.cpp Tests/Particles/ParticleTransformations/main.cpp commit 60fe729fe2ba65ebffc88c0af18743c254d3992c Author: Jean M. Sexton Date: Fri Dec 3 15:52:18 2021 -0800 Use specific targets for make uninstall (#2471) This updates the `make uninstall` rule which is used with configure to more specifically remove targets which were installed. It also adds a dependency file removal for the `make cleanconfig` rule. This removes the assumption that the directory set from `./configure --prefix=/my/example/directory` is empty, and therefore all files should be removed with `make uninstall` Co-authored-by: Weiqun Zhang GNUmakefile.in Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules commit e96ffdead46310595dfa491b1e42ae208017de1d Author: Don E. Willcox Date: Fri Dec 3 14:58:21 2021 -0800 Move Basic Time Integration into AMReX (#2503) This PR moves the latest versions of a basic general explicit time integrator into AMReX. We are already using this time integrator for the Emu (https://github.com/AMReX-Astro/Emu), ET-Integration (https://github.com/amrex-codes/ET-integration), and ERF (https://github.com/erf-model/ERF) codes and this will let us consolidate the code and let other codes can benefit from it. The time integrator includes both forward Euler and several explicit Runge-Kutta methods and example code with input parameter explanations are supplied in a new documentation webpage labeled "Time Integration." Co-authored-by: Jean M. Sexton Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/TimeIntegration_Chapter.rst Docs/sphinx_documentation/source/index.rst Src/Base/AMReX_FEIntegrator.H Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_RKIntegrator.H Src/Base/AMReX_TimeIntegrator.H Src/Base/CMakeLists.txt Src/Base/Make.package commit b6d3d1cced21648126e9465ad88ec3f6cd8b093d Author: Andrew Myers Date: Fri Dec 3 17:44:17 2021 -0500 allow the use of SuperParticle in amrex::ParticleToMesh (#2514) Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleUtil.H Tests/Particles/ParticleMesh/main.cpp commit f5c1f0371aa7ec2c53053907f856e0b64a6bc623 Author: Nathan X. Marshak <5407450+n8xm@users.noreply.github.com> Date: Fri Dec 3 15:04:01 2021 -0700 Workaround to fix amrex_particles_to_vtp.py samplefiles (#2508) ## Summary The sample input files for `amrex_paticles_to_vtp.py` were modified in order to stop the script from crashing. Based on comparison against output from the [Electrostatic PIC tutorial](https://amrex-codes.github.io/amrex/tutorials_html/Particles_Tutorial.html#electrostaticpic), I believe that the sample input files are in an outdated format. ([electrostatic_pic_particles50000.txt](https://github.com/AMReX-Codes/amrex/files/7637608/electrostatic_pic_particles50000.txt) is attached for reference.) ## Additional background `amrex_particles_to_vtp.py` skips the first four lines after the first line (containing the number of particles) of the input file. Then, the number of lines that are read is equal to exactly the number of particles specified on the first line. The sample input files under the `samplefiles/` directory do not contain these four "skipped" lines, and as a result, not only does `amrex_particles_to_vtp.py` skip four lines of valid particle data, the script crashes because it tries to read past the end of the file. (In other words, the script assumes that the file is four lines longer than it really is.) As a workaround, I've added four "dummy" lines to each file. This is NOT an ideal solution because real particle output would not contain my four "dummy" lines, and instead there would be some other information there (i.e. I believe the number of extra real and int components in the particle data/attributes). However, my workaround is adequate for the purpose of demonstrating correct usage of `amrex_particles_to_vtp.py`. I have confirmed that the resulting `.vtp` files produce sensible-looking results in ParaView. (See attached screenshots below for reference.) `particles00000`: ![particles00000](https://user-images.githubusercontent.com/5407450/144328022-71b847a8-d78a-4837-81dc-ea8d1cafd682.gif) `particles00001`: ![particles00001](https://user-images.githubusercontent.com/5407450/144328021-34077dda-fe6b-40b0-8849-4265bb4539f9.gif) `particles00002`: ![particles00002](https://user-images.githubusercontent.com/5407450/144328017-504b1158-b38c-49e9-924b-957ecc9a0fbd.gif) Co-authored-by: Nathan Marshak Tools/Py_util/amrex_particles_to_vtp/samplefiles/particles00000 Tools/Py_util/amrex_particles_to_vtp/samplefiles/particles00001 Tools/Py_util/amrex_particles_to_vtp/samplefiles/particles00002 commit 71b6368a8f3d807ce0c70d38c7ef3791bd5c431d Author: Andrew Myers Date: Fri Dec 3 15:21:36 2021 -0500 Fix AppleClang CI by ignoring unused variables in GccPlacater (#2513) Src/Base/AMReX.H Src/Base/AMReX_MultiFab.H commit e2af96c28d4e50ecac8476cbb82f048f8e82efe5 Author: Weiqun Zhang Date: Fri Dec 3 08:08:29 2021 -0800 ANSI Escape Codes (#2509) Add commonly used ANSI escape codes to AMReX_ANSIEscCode.H. They can be used to print colored output. For example, amrex::Print() << Font::Bold << FGColor::Red << "Hello world from AMReX\n" << ResetDisplay << Font::Faint << FGColor::BrightCyan << Font::Underline << "Hello world again from AMReX " << ResetDisplay << BGColor::White << FGColor::Black << amrex::Version() << ResetDisplay << std::endl; Src/Base/AMReX_ANSIEscCode.H Src/Base/AMReX_Print.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 2c307a8276ff0ad5f7636544ac7c191ffd6f6365 Author: Andrew Myers Date: Thu Dec 2 13:25:26 2021 -0500 Fix misleading documentation for ParticleTile. (#2500) Co-authored-by: etpalmer63 Docs/sphinx_documentation/source/Particle.rst commit 6396050dde3f4fb08ef65173a52350c7b33adca9 Author: Weiqun Zhang Date: Wed Dec 1 10:22:25 2021 -0800 Update CHANGES for 21.12 (#2507) CHANGES commit ef9a37d6a14a704d59f105945c203da717595cb5 Author: Weiqun Zhang Date: Tue Nov 30 18:51:24 2021 -0800 AMREX_RELEASE_NUMBER (#2448) Add a new macro `AMREX_RELEASE_NUMBER`. It is a six digits number. The first two are for the year, and the next two are for the month. The last two are usually 00, unless a patch other than the monthly release is released. If the AMReX source is a git version that's not the release version, the last release number will be used. For git version obtained with git --describe, we define something like #define AMREX_GIT_VERSION "21.10-65-gfbb2edc95551-dirty" #define AMREX_RELEASE_NUMBER 211000 For git version obtained from the CHANGES file, we define something like #define AMREX_GIT_VERSION "21.10" #define AMREX_RELEASE_NUMBER 211000 or #define AMREX_GIT_VERSION "21.10.1" #define AMREX_RELEASE_NUMBER 2110001 or #define AMREX_GIT_VERSION "21.10.11" #define AMREX_RELEASE_NUMBER 211011 Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXUtils.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs commit 1d50bfc1c1da79614c853f9823efdf33f9003cc5 Author: Andrew Myers Date: Tue Nov 30 16:53:45 2021 -0500 Add a function for detecting whether amrex::Initialize has been called or not. (#2501) Src/Base/AMReX.H Src/Base/AMReX.cpp commit 062a07100e364b25d29a966092fd9e9200d4f2b0 Author: Weiqun Zhang Date: Mon Nov 29 18:17:47 2021 -0800 Fix fcompare's --abs_tol (#2504) There is a bug that ignores the absolute errors of components that do not have the biggest relative error when --abs_tol is used. It results in something like below fcompare.gnu.ex --abs_tol 1.e-12 plt00010 plt00002 variable name absolute error relative error (||A - B||) (||A - B||/||A||) ---------------------------------------------------------------------------- level = 0 density 0 0 pressure 0 0 theta 0 0 x_velocity 4.39990436e-05 2.749038203e-06 y_velocity 4.818762188e-05 9.964503154e-06 z_velocity 4.70120484e-19 0.7986458289 PLOTFILE AGREE to specified tolerances: absolute = 1e-12 relative = 0 In this commit, this is fixed by applying the test to each component separately. A component passes if its relative error or absolute error satisfies the respective tolerance. The whole comparison passes if all components pass. Tools/Plotfile/fcompare.cpp commit d62f0d27c907324065fe4968b6dfbbb98d41fd64 Author: Andrew Myers Date: Mon Nov 29 14:13:58 2021 -0500 Fix HIP CI (#2502) Something was going wrong with the "computed default" cmake argument on this action, so this removes it. .github/workflows/hip.yml commit d32658dbbccf1e5abd82cb971bb9fec8fd87eeb1 Author: Houjun Tang Date: Mon Nov 29 07:08:22 2021 -0800 Fix an issue when some MPI ranks have no box at all (#2494) Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Tests/HDF5Benchmark/main.cpp commit 591c7856b7c876ec9f1f3a5db56ce652059c7c8b Author: Weiqun Zhang Date: Wed Nov 24 07:25:57 2021 -0800 Signed distance function in 2D (#2490) Function for computing signed distance function in 2D. Src/Base/AMReX_Scan.H Src/EB/AMReX_EB_utils.H Src/EB/AMReX_EB_utils.cpp commit 30185629176596663f9d2ee21bfd97ae4d9546b2 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon Nov 22 14:20:15 2021 -0600 Reduce ghost particles in particle communication. (#2480) * Remove duplicated particle communication when boxes are not perfectly aligned * Add a function to select the ghost particles that are neighbors to the real particles And allow `updateNeighbors()` to only communicate these ghost neighbor particles. These two steps significant reduce the particle communication time when particles are dense. * Fix CI issues. * Fix coding style issues. Co-authored-by: Hengjie Wang Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H commit 29584112a13ab086dc64edfdc99a217431ed8db7 Author: Andrew Myers Date: Sun Nov 21 10:43:12 2021 -0800 Assign initial value to prevent (likely spurious) -Wmaybe_unitialized compiler warning. (#2484) Src/Particle/AMReX_Particle.H commit 80156569892316cf413b072c3bc4a63abff28309 Author: Weiqun Zhang Date: Sun Nov 21 10:42:43 2021 -0800 Fix a corner case in fixing small cells. (#2489) During the iteration of fixing small cells, the edge types and level sets could become inconsistent if the fixed small cells are too close to periodic boundaries. The function that updates the edge types did not take that into account. This commit fixes that. Src/EB/AMReX_EB2_GeometryShop.H commit 96ba8b11befcd5361fdf70b58a9fb6e6eb5776ca Author: Candace Gilet Date: Wed Nov 17 18:39:52 2021 -0500 Fix out-of-date comment. (#2483) Src/Base/AMReX_ParmParse.H commit bf86e02eb5048c201b7014c4efa567ed3f49a1d8 Author: Erik Date: Wed Nov 17 18:39:19 2021 -0500 Remove HeatEqn Tutorial and link to new location in amrex-tutorials (#2482) Remove the Heat Equation Tutorial from the AMReX user's guide. Add link to AMReX tutorials in its place. Note, the tutorial already exists in amrex-tutorials. Docs/sphinx_documentation/source/Basics.rst commit 62774a08663f6552a5d6265200f6d52e18df824b Author: Erik Date: Wed Nov 17 18:37:48 2021 -0500 Combine debugging sections into a single chapter. (#2481) This change combines the debugging and Gpu debugging sections into a single debugging chapter. Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/Debugging.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/index.rst commit a2db70cb3cb5e74ee692ed1b568727dd66aa145d Author: David Grote Date: Wed Nov 17 15:35:50 2021 -0800 Added special case for Mac when using rpath (#2479) Tools/GNUMake/Make.defs commit 88b868099018c191e7d3ea424d3e3420b826dd69 Author: Weiqun Zhang Date: Fri Nov 12 12:42:17 2021 -0800 Fix Advection_AmrLevel test (#2474) There is no need to use pinned memory in initData anymore after #2433. Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp commit 6ff068d5d04218520fb1408b01ea734b5bca40e6 Author: Erik Date: Fri Nov 12 11:19:48 2021 -0500 Advection_AmrLevel Test: Convert Prob.f90 to Prob.cpp (#2433) Converts the only Fortran file in the AmrLevel test to cpp. Tests/Amr/Advection_AmrLevel/CMakeLists.txt Tests/Amr/Advection_AmrLevel/Exec/Make.Adv Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Make.package Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.cpp Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Make.package Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.cpp Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.f90 Tests/Amr/Advection_AmrLevel/README Tests/Amr/Advection_AmrLevel/Source/Adv_F.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Amr/Advection_AmrLevel/Source/Make.package commit 37d1af793b1ec9c20d4d9ba1c176b5936c3e01a3 Author: Weiqun Zhang Date: Thu Nov 11 16:53:31 2021 -0800 Fix multi-level EB nodal solver solvability fix (#2473) For the multi-level EB nodal solver with a singular matrix, we restrict the fine level RHS to the coarse level, and fix the coarse level's solvability by making the total sum of RHS is zero. The restriction is done with the stencil instead of simply geometric restriction because we need to take the EB into account. However, the previous way is incorrect if the sigma coefficient is not a constant, because the stencil includes sigma. But the restriction should be based on EB information only. In this commit, we fix this by constructing a special constant sigma stencil. We use this new stencil for the restriction of RHS, and for consistence the residual as well. Because we used to use the original stencil for the restriction of residual, this commit will change the answer at the tolerance level for multi-level problems. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp commit a2245756e64c1f414f35b64330fec99a9b87df0c Author: Weiqun Zhang Date: Mon Nov 8 12:36:09 2021 -0800 Disable warning on atomicAddNoRet in HIP (#2466) It seems that we may continue using the deprecated atomicAddNoRet in the near future. This disables the compiler warning. Src/Base/AMReX_GpuAtomic.H commit 798240684d9696fb00392688e3c216e21596b921 Author: Weiqun Zhang Date: Mon Nov 8 08:38:53 2021 -0800 DPCPP: workaround for the scan issue (#2462) A multi-pass algorithm is implemented. Extensive testing on early hardware shows it works for int, but there are still issues with long. Src/Base/AMReX_Scan.H commit c49b1834aa488e123cb379681577a8cf65fed68e Author: Weiqun Zhang Date: Mon Nov 8 08:28:56 2021 -0800 DPCPP: AOT in GNU Make (#2463) If INTEL_ARCH or AMREX_INTEL_ARCH is defined, use it to set target when AOT is on. Tools/GNUMake/comps/dpcpp.mak commit 3d29cc3234a319f5721da38635f37b3570da43c7 Author: Jean Luca Bez Date: Sun Nov 7 14:17:56 2021 -0800 Include optional directory to HDF5 Test Benchmark (#2461) Tests/HDF5Benchmark/inputs Tests/HDF5Benchmark/main.cpp commit 2fc0e4446aa466247de8b7efae11903cfd3b5c37 Author: Pastuzo <40230415+DUGKS@users.noreply.github.com> Date: Sun Nov 7 02:17:17 2021 +0800 Implement HOEXTRAPCC for more than single ghost cell (#2456) Implement HOEXTRAPCC for N ghost cells, N > 1. Previous HOEXTRAPCC works only for N = 1. Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_FilCC_C.cpp Src/Base/AMReX_filcc_mod.F90 commit d4669a1ce4c7abb8c15af7829b6d4f56075589a9 Author: Erik Date: Fri Nov 5 17:00:06 2021 -0400 Spack Smoke Test (#2460) Tests/SpackSmokeTest/CMakeLists.txt commit 39332fbb07478fe685c9917006df0fef1957280b Author: Weiqun Zhang Date: Fri Nov 5 10:19:11 2021 -0700 Support fmod in Parser (#2395) * Support fmod in Parser * Sneak in some minor changes Docs/sphinx_documentation/source/AmrCore.rst Docs/sphinx_documentation/source/Basics.rst Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp commit b0b7900500bc9879aefd8b649caf0346976bea50 Author: Weiqun Zhang Date: Wed Nov 3 16:25:11 2021 -0700 Fix a bug in BoxArray for SyncRegister (#2458) A bug was introduced recently in #2452. I forgot that code was used by SyncRegister in IAMR. Src/Base/AMReX_BoxArray.H commit 8023fd148ee0f23ae83127b8d0515c1323c194de Author: Scot Breitenfeld Date: Wed Nov 3 12:16:53 2021 -0500 Redo of PR #2450 (#2457) Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H commit 2ec7459dd689d1425ce45a5977f26ae7e3c9facb Author: Jean M. Sexton Date: Tue Nov 2 16:04:36 2021 -0700 HIP: ftrapv causes issues with DEBUG linking (#2455) Tools/GNUMake/comps/hip.mak commit d491ebe1c1aca7dc53558e64af24526888b79abe Author: Weiqun Zhang Date: Tue Nov 2 15:04:14 2021 -0700 DPCPP: Enable early optimizations (#2454) Early optimizations were disabled after beta9 was released to work around a compiler bug. The issue has been resolved. So we enable it again. Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit 7555541ab3c43f2c46b26b25e42a40d73fdf3656 Author: Houjun Tang Date: Tue Nov 2 14:15:07 2021 -0700 New HDF5 schema with separate datasets for different variables (#2432) Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_PlotFileUtil.cpp Src/CMakeLists.txt Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Extern/HDF5/AMReX_ParticleUtilHDF5.H Src/Extern/HDF5/AMReX_ParticlesHDF5.H Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.H Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Extern/HDF5/CMakeLists.txt Src/Extern/HDF5/Make.package Src/Particle/AMReX_ParticleUtil.cpp Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H Src/Particle/CMakeLists.txt Src/Particle/Make.package Tests/HDF5Benchmark/GNUmakefile Tests/HDF5Benchmark/inputs Tests/HDF5Benchmark/main.cpp Tools/CMake/AMReXConfig.cmake.in Tools/GNUMake/packages/Make.hdf5 commit aca6926e0a85a0fad4087bd47016edad0885252f Author: Weiqun Zhang Date: Mon Nov 1 15:15:35 2021 -0700 InterpFaceRegister (#2452) This is a new class for interpolating from coarse to fine at the coarse/fine boundary faces. It also provides masks that can be used to test if a face is at coarse/fine boundary. Src/AmrCore/AMReX_AmrCoreFwd.H Src/AmrCore/AMReX_InterpFaceReg_1D_C.H Src/AmrCore/AMReX_InterpFaceReg_2D_C.H Src/AmrCore/AMReX_InterpFaceReg_3D_C.H Src/AmrCore/AMReX_InterpFaceReg_C.H Src/AmrCore/AMReX_InterpFaceRegister.H Src/AmrCore/AMReX_InterpFaceRegister.cpp Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package Src/Base/AMReX_BoxArray.H commit 3add1f2824c5a4f33e77abc612da8a52b3f5131b Author: Weiqun Zhang Date: Mon Nov 1 13:59:36 2021 -0700 Fix WarpX EB solver's grad phi computation (#2453) Must set the Dirichlet mask on covered nodes to a negative value, because the function that computes the gradient relies on the information to detect EB. This is a bug introduced recently in #2421. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 500ab4fb3ea7a755e0e0b29c7c021495fc0707b5 Author: Weiqun Zhang Date: Mon Nov 1 10:03:45 2021 -0700 Update CHANGES for 21.11 (#2451) CHANGES commit 2cfe8354f6c6a00c7a6bb2bcefd822d59fd65518 Author: Scot Breitenfeld Date: Sat Oct 30 00:05:44 2021 -0500 Set to never write fill values to the dataset for HDF5 output (#2450) Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_WriteBinaryParticleData.H commit 6e9788139d8e1401ed304fc56e526a1bdb315b81 Author: Weiqun Zhang Date: Thu Oct 28 13:36:13 2021 -0700 Fix a new bug in GNU Make (#2447) CXX should not be set to mpiicpc if it's not an MPI build. This is a new bug introduced by me in #2435. Tools/GNUMake/sites/Make.unknown commit 210cb0e7d77d1334d009246a4cb403ae463df1dc Author: Weiqun Zhang Date: Thu Oct 28 12:54:03 2021 -0700 Lower Hypre minimum version (#2446) Lower Hypre minimum version to 2.20. This will allow users to use the provided installation on summit. The ILU smoother that requires 2.21 is still available for hypre >= 2.21. Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXThirdPartyLibraries.cmake commit 6c30ddf8fb0d1a4373b013517d43734ec4110385 Author: Axel Huebl Date: Thu Oct 28 11:28:13 2021 -0700 ROCm 4.5: Dev Packages (#2445) * ROCm 4.5: Devel Packages In ROCm 4.5, libraries were split into binary and `-devel` packages, as often done in Debian/Ubuntu packaging. We need the `-devel` packages for headers. Release notes: https://rocmdocs.amd.com/en/latest/Current_Release_Notes/Current-Release-Notes.html * Robust: Try while rollout is ongoing * What if the docs lie? Try `-dev` * Clean Up .github/workflows/dependencies/dependencies_hip.sh commit 138503ba8476c977de97628bc5bdbdd5fbb00356 Author: Weiqun Zhang Date: Thu Oct 28 08:44:46 2021 -0700 Fix use-after-free (#2443) The parallel copy meta-data cache is store in std::multimap. A CPC* might be stored multiple times in the multimap, and we need to delete the pointers before clearing the multimap. The previous way of avoiding double free was incorrect, because it had a use-after-free. In this commit, we fix it by pushing a unique copy of the pointers to a vector first while iterating the multimap. This bug only manifests if a user's code has a memory leak of MultiFab/FabArray. Src/Base/AMReX_FabArrayBase.cpp commit 08a068a638a124c7dd9e25f28a5ac02ec59fe4ae Author: Jean M. Sexton Date: Wed Oct 27 15:04:42 2021 -0700 Update MPI config checking to include Intel MPI wrappers for DPCPP (#2435) * Update MPI config checking to assume first Intel MPI wrappers, and then others * move the mpicpc check out of the usual MPI check Co-authored-by: Weiqun Zhang Tools/GNUMake/Make.defs Tools/GNUMake/sites/Make.unknown commit 8c8004c07a24dc1e20e5ac6c8e6bc07fdb0ef5a7 Author: Andrew Myers Date: Wed Oct 27 13:53:17 2021 -0700 Mark NeighborParticles as a 'selfTest' (#2442) Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini commit 68e2913f4783d3dc5fe9bc140bbaca06ff8c07d9 Author: Weiqun Zhang Date: Wed Oct 27 12:14:41 2021 -0700 GNU Make: Fix rpath when LIBRARY_LOCATION is empty. (#2441) Because of #2439, empty LIBRARY_LOCATION would result in -Xlinker -rpath='' We would like to avoid that. Tools/GNUMake/Make.defs commit 58d04c6195c0f2c47119315b37d5e99f69fe0898 Author: Weiqun Zhang Date: Wed Oct 27 12:14:09 2021 -0700 Add EBCellFlagFab::getNumRegularCells, getNumCutCells and getNumCoveredCells (#2422) Give a Box, these functions will return the number of regular, cut and covered cells, respectively. Internally, the results are cached. Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBCellFlag.cpp commit 4fa06516928ad07736cb537ddb58d8c4ecba5a4e Author: Neïl Zaim <49716072+NeilZaim@users.noreply.github.com> Date: Wed Oct 27 20:51:17 2021 +0200 ExclusiveSum on GPU: return 0 when size of pointer is 0 (#2440) Src/Base/AMReX_Scan.H commit 94b4764f098603c51bda652d78ea0dff8bee3635 Author: Weiqun Zhang Date: Wed Oct 27 10:40:56 2021 -0700 Multi-component overset solver (#2429) Modify cell-centered overset solver's constructor to allow for multiple components. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp commit 67e9aadcace3572ca840d8d276d67501663ead8b Author: Weiqun Zhang Date: Wed Oct 27 10:38:30 2021 -0700 Use -rpath=dir instead of -rpath dir (#2439) The latest dpcpp compiler does not like the latter. Tools/GNUMake/Make.defs commit 0a98df6746fcdbb51f133e1905b08d9ae75e99e9 Author: Weiqun Zhang Date: Tue Oct 26 21:13:49 2021 -0700 DPCPP: Use -g1 for non-debug build (#2438) Because `-g` makes link extremely slow, we have disabled that in GNU make since oneAPI beta6. It turns out we could still use `-g1` to get some basic debug information without slowing down link too much. Tools/GNUMake/comps/dpcpp.mak commit ce057cf7c225badbbae7f964e81f1e67745238a4 Author: Weiqun Zhang Date: Tue Oct 26 17:31:49 2021 -0700 DPCPP: Fix warning on tautological-constant-compare (#2436) Disable warning on tautological constant compare that has appeared since 2021.04. .github/workflows/intel.yml Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit 6b178313793544957f7d0c23ae7999cacead69bd Author: Weiqun Zhang Date: Tue Oct 26 17:24:03 2021 -0700 Update Tools/Postprocessing/C_Src (#2437) Tools/Postprocessing/C_Src/GNUmakefile Tools/Postprocessing/C_Src/HorizontalAvg.cpp Tools/Postprocessing/C_Src/PtwisePltTransform.cpp Tools/Postprocessing/C_Src/particle_compare.cpp commit 39da1abd86b46c64e098c43c3123fcff6c58742d Author: Axel Huebl Date: Mon Oct 25 18:02:00 2021 -0700 Tests: Safe-Guard Module Dirs (#2434) Make sure mod_file dirs are per executable. Avoid potential race conditions if people use the `setup_test` in loops in funny ways. Tests/CMakeLists.txt commit af8429793bc50cf7d5914104c05ed825dd0948c8 Author: Weiqun Zhang Date: Mon Oct 25 10:11:33 2021 -0700 Remove deprecated BaseUmap (#2425) Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BaseUmap_f.H Src/Base/AMReX_BaseUmap_nd.f90 Src/Base/Make.package commit 652af10ae992e2cf5c1812e0908d9b88ac22126c Author: Weiqun Zhang Date: Mon Oct 25 10:04:32 2021 -0700 Update WarpX EB solver (#2421) Change the coarsening strategy for much better performance. Src/EB/AMReX_EB2_2D_C.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 614b9ebbbf04eb5b4532029ff4a7ae40e6663042 Author: Weiqun Zhang Date: Fri Oct 22 14:59:39 2021 -0700 MLNodeLaplacian: Fix max coarsening level for enclosed EB (#2431) Where the domain is fully enclosed by EB, we need to make sure that it does not coarsen too much that there are no valid nodes left. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp commit 6562da035155d88663c6c98de5f9575c750a3be9 Author: Andrew Myers Date: Fri Oct 22 12:07:49 2021 -0700 Fix two IO overflow bugs uncovered by NyX. (#2417) Co-authored-by: Jean M. Sexton Src/Base/AMReX_FabConv.cpp Src/Particle/AMReX_WriteBinaryParticleData.H commit a707944d287a320e184768d85b5bb05b020f5545 Author: Weiqun Zhang Date: Fri Oct 22 12:03:19 2021 -0700 Cleanup of amrdata (#2430) * Avoid using std and amrex in amrdata headers. * Fix some warnings. Src/Extern/amrdata/AMReX_AmrData.H Src/Extern/amrdata/AMReX_AmrData.cpp Src/Extern/amrdata/AMReX_DataServices.H Src/Extern/amrdata/AMReX_DataServices.cpp Src/Extern/amrdata/AMReX_WritePlotFile.cpp Src/Extern/amrdata/AMReX_XYPlotDataList.H Src/Extern/amrdata/AMReX_XYPlotDataList.cpp commit 2d02c17651b81c3bb7d90a45c81265fdb1706c1b Author: Erik Date: Thu Oct 21 19:30:51 2021 -0400 Set options to allow other Doxygen docs to link to AMReX. (#2424) * Set options to allow other Doxygen docs to link to AMReX. * move amrex xml tag file to folder for uploading to webpage. Docs/Doxygen/doxygen.conf build_docs.sh commit f75e129df118cf448bc763d9eafd67c3cbb74825 Author: Weiqun Zhang Date: Thu Oct 21 15:57:51 2021 -0700 Remove DArena and EArena (#2426) DArena that uses the buddy memory allocation strategy and EArena that uses the best fit strategy are experimental features we explored in the past. But they have never been used because they do not work as well as the first bit based CArena. There is no reason to keep them anymore. Src/Base/AMReX_Arena.cpp Src/Base/AMReX_DArena.H Src/Base/AMReX_DArena.cpp Src/Base/AMReX_EArena.H Src/Base/AMReX_EArena.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 8a1a3023b6f0bbc715935fcabbcddbfe9ff95861 Author: Andrew Myers Date: Thu Oct 21 15:55:51 2021 -0700 Add some docstrings for the particle tile access functions. (#2423) Src/Particle/AMReX_Particles.H commit 7c88a2c139eada6601401ac23e6f0b778fab9a9c Author: Weiqun Zhang Date: Thu Oct 21 14:20:57 2021 -0700 Update MLNodeTensorLap for WarpX (#2416) Change the stencil of the linear operator so that L=D*G. This could make the solution satisfy div E = rho up to solver tolerance. Previously, the stencil was similar to that of the nodal solver for approximate projection Src/EB/AMReX_EB2_2D_C.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Tests/LinearSolvers/NodeTensorLap/MyTest.cpp commit db792398f6da8158026527759d97b5b882d1f22b Author: Andrew Myers Date: Wed Oct 20 12:30:36 2021 -0700 Restore RedistributeLocal() call to InitParticles in redistribute test (removed in #2414) (#2420) Tests/Particles/Redistribute/main.cpp commit cdc1052e9d5f518aa864c501a410ed6fbae56afe Author: Andrew Myers Date: Tue Oct 19 12:20:38 2021 -0700 Revert changes to Redistribute test (follow on to #2414) (#2415) Tests/Particles/Redistribute/inputs Tests/Particles/Redistribute/inputs.rt.cuda Tests/Particles/Redistribute/main.cpp commit a9d758ab6dc019203ae467c185553f3dc99b4e8c Author: Andrew Myers Date: Tue Oct 19 10:51:16 2021 -0700 Reduce peak memory usage when sorting particles. (#2414) Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Tests/Particles/Redistribute/inputs Tests/Particles/Redistribute/inputs.rt.cuda Tests/Particles/Redistribute/main.cpp commit 130f7304328ed1c26a72cea24dd3e96bc9426a58 Author: Candace Gilet Date: Tue Oct 19 12:24:45 2021 -0400 Add link to 'small cell problem' section that got moved to (#2412) Add link to 'small cell problem' section that got moved to AMReX-Hydro's documentation. Also add more details on EB data. Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/EB_Chapter.rst Docs/sphinx_documentation/source/conf.py commit ff747d901ea2127ed01d6053f1d80e9a45bcd381 Author: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> Date: Tue Oct 19 09:23:26 2021 -0700 Docs: MLMG solver convergence criteria (#2413) expanded the documentation with regards to convergence criteria for the MLMG solver Docs/sphinx_documentation/source/LinearSolvers.rst commit 691d9bf3fb64343a3a929d12f1463fde165ec969 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Tue Oct 19 12:22:38 2021 -0400 Extend CellQuadratic Interpolater to 3D (#2409) Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterp_C.H commit 09a0ec56378f9f6a5f68f3ac025318af746432d5 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Mon Oct 18 11:22:55 2021 -0400 Update documentation on Interpolaters (#2408) Docs/sphinx_documentation/source/AmrCore.rst commit be5294eefb2017cb0c5273054207ea1cb6d77807 Author: Weiqun Zhang Date: Thu Oct 14 16:13:05 2021 -0700 Fix 2D bug in EB2::SplineIF (#2407) Src/EB/AMReX_EB2_IF_Spline.H commit 92b4e5bc5d00b1f8c11c0eedfeaeb99ebbe38bbe Author: Weiqun Zhang Date: Thu Oct 14 16:12:34 2021 -0700 Bump up Hypre minimum version to 2.21.0 (#2406) Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXThirdPartyLibraries.cmake commit fb72b08c1914a5b1f8529dcf101c059d613ee4a0 Author: Candace Gilet Date: Thu Oct 14 19:12:12 2021 -0400 Cleanup Tools/Postprocessing/C_src (#2398) Fix memory, parsing for help, and add add automatic determination of whether f90 sources need to be included. Tools/Postprocessing/C_Src/GNUmakefile Tools/Postprocessing/C_Src/IntegrateComp.cpp Tools/Postprocessing/C_Src/PlotfileToMatLab.cpp Tools/Postprocessing/C_Src/PlotfileToTurb.cpp Tools/Postprocessing/C_Src/PtwisePltTransform.cpp commit d5a3334ba58a15a74fdb55da0eabf643a7cd0f9d Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Thu Oct 14 19:11:22 2021 -0400 Port CellQuadratic to C++ with GPU support (#2369) ## Summary The CellQuadratic interpolater has been ported from Fortran to C++ with GPU support. The errors for the CPU-only version on local are in the ~1e-15 range for AmrLevel SingleVortex 2D, ~1e-15 for IAMR 2D bubble Cartesian, and ~1e-12 for IAMR 2D bubble RZ. The errors for the GPU-enabled version on Gigan are in the ~1e-14 range for AmrLevel SingleVortex2D, ~1e-10 for IAMR 2D bubble Cartesian, and ~1e-9 for IAMR 2D bubble RZ. ## Additional background This port is heavily influenced by the CellConservativeLinear interpolater. It reuses two of the same slope kernels. With this PR, _all_ of the interpolaters now work on the GPU! Also, thanks to @cgilet for helping with debugging. Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package commit d8e6ec544c25233483bf1dc0b5aa263a9a9df730 Author: Michael J. Brazell Date: Thu Oct 14 11:03:35 2021 -0700 added bamg option to set restriction type (#2402) 0 is P transpose and 1,2 are approximate ideal restriction AIR Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit 06b30c1eaff02952e9d656da01dd090ba5b5a540 Author: Stefan Vater Date: Thu Oct 14 19:59:57 2021 +0200 Bugfix/amr mesh constructor (#1155) These resizes must consider that the vectors in AmrInfo argument might be empty. Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_AmrMesh.cpp commit 5ae9ea7e67c347d81377ff722f12720a8c5377fc Author: Weiqun Zhang Date: Thu Oct 14 10:11:40 2021 -0700 Fix dot product mask in EB nodal solver (#2404) We should set the dot product mask on all Dirichlet nodes to zero so that they do not contribute to the sum used in fixing singular systems. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp commit a81dd4512658e05c220b358324cf0d67aef13e19 Author: Candace Gilet Date: Thu Oct 14 11:38:00 2021 -0400 Move Tools/C_util/Convergence/WritePlotFile to Src/Extern/amrdata. (#2397) This contains WritePlotFile functions that take an AmrData object as a parameter, and thus Src/Extern/amrdata is the most logical place to put it when codes in Tools/C_util/Convergence and Tools/Postprocessing/C_src use it. Src/Extern/amrdata/AMReX_WritePlotFile.H Src/Extern/amrdata/AMReX_WritePlotFile.cpp Src/Extern/amrdata/CMakeLists.txt Src/Extern/amrdata/Make.package Tools/C_util/Convergence/ComputeAmrDataNorms.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedFD.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp Tools/C_util/Convergence/DiffUniform.cpp Tools/C_util/Convergence/Make.package Tools/C_util/Convergence/PltFileNormB.cpp Tools/C_util/Convergence/RichardsonConvergenceTest.cpp Tools/Postprocessing/C_Src/PlotfileToTurb.cpp Tools/Postprocessing/C_Src/PtwisePltTransform.cpp commit 2ec5203f43050dcac8a8d8a7a99670e54ef0a87a Author: Candace Gilet Date: Thu Oct 14 11:35:27 2021 -0400 Move info on sharing the command line into the ParmParse section. (#2391) Co-authored-by: etpalmer63 Docs/sphinx_documentation/source/Basics.rst commit 92af41f8a966b252f260c0f3f32d41178844ac50 Author: Weiqun Zhang Date: Thu Oct 14 08:28:51 2021 -0700 Arena Document and PrintUsage (#2403) * Arena Document and PrintUsage * Document arena initial allocation size and release threshold. * Print the number of allocations in PrintUsage. * Update GPU.rst minor spellings Co-authored-by: Erik Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_CArena.cpp commit b60a773d6c87f708a736634f6281da16e094e4cb Author: Weiqun Zhang Date: Tue Oct 12 16:03:46 2021 -0700 BArena: Make a static BArena object (#2401) Instead of creating multiple BArena objects, make a static BArena object that is local to a new function The_BArena() in an anonymous namespace. The motivation for this is to properly release the memory allocated by BArena after amrex::Finalize() is called. This is follow-up to #2389. Src/Base/AMReX_Arena.cpp commit 247b41e17aa83f653d848e77f7269e9d43bf4f73 Author: Candace Gilet Date: Tue Oct 12 16:14:54 2021 -0400 Fix inaccurate comment about AmrLevel::writePlotFile, it is no longer pure virtual (#2399) Src/Amr/AMReX_AmrLevel.H commit 48a2776d78ec2a4e2909a875fc7eab25973a81f6 Author: Erik Date: Tue Oct 12 16:09:50 2021 -0400 fix dead link. small reword (#2400) Docs/sphinx_documentation/source/index.rst commit a162b7c15518b6bd5ceb7f8fefa574e6ee3c87d5 Author: Candace Gilet Date: Tue Oct 12 13:23:19 2021 -0400 Make fcompare print usage and exit if only one plotfile is provided. (#2388) Tools/Plotfile/fcompare.cpp commit 4914d5e42c2ee02c80645722b2d336e4dba6b4b3 Author: Weiqun Zhang Date: Mon Oct 11 18:58:13 2021 -0700 Fix compilation of Tools/Postprocessing/C_Src (#2396) Broken by #2390. Tools/Postprocessing/C_Src/Make.package commit 93788e33e689b77a14290580eca0cf9f25be6523 Author: Weiqun Zhang Date: Fri Oct 8 14:54:08 2021 -0700 Add Arena::PrintUsageToFiles for debugging (#2392) Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp commit b5cc0323c87d5caa9c3ed7dc9ac639cc0d3fb012 Author: Candace Gilet Date: Fri Oct 8 16:51:29 2021 -0400 Update c util (#2390) This PR cleans out old, unused code from amrex/Tools/C_util, and ensures that what's there compiles and runs. Tools/C_util/AmrDeriveTecplot/AmrDeriveTecplot.cpp Tools/C_util/AmrDeriveTecplot/GNUmakefile Tools/C_util/AppendToPlotFile.H Tools/C_util/AppendToPlotFile.cpp Tools/C_util/AugmentPlotfile/AugmentPlotfile.cpp Tools/C_util/AugmentPlotfile/GNUmakefile Tools/C_util/Convergence/Add.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/Convergence/DebugDump.H Tools/C_util/Convergence/DebugOut.H Tools/C_util/Convergence/DebugOut.cpp Tools/C_util/Convergence/DiffFab.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedFD.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp Tools/C_util/Convergence/DiffUniform.cpp Tools/C_util/Convergence/GNUmakefile Tools/C_util/Convergence/Make.package Tools/C_util/Convergence/PltFileNorm.cpp Tools/C_util/Convergence/PltFileNormB.cpp Tools/C_util/Convergence/PltFileScalConvRate.cpp Tools/C_util/Convergence/RichardsonConvergenceTest.cpp Tools/C_util/Convergence/WritePlotFile.H Tools/C_util/Convergence/WritePlotFile.cpp Tools/C_util/DiffMultiFab/diffmultifab.cpp Tools/C_util/Make.package Tools/C_util/README Tools/C_util/Statistics/AVGDOWN_2D.F Tools/C_util/Statistics/AVGDOWN_3D.F Tools/C_util/Statistics/AVGDOWN_F.H Tools/C_util/Statistics/ComputeAmrDataStat.H Tools/C_util/Statistics/ComputeAmrDataStat.cpp Tools/C_util/Statistics/GNUmakefile Tools/C_util/Statistics/Make.package Tools/C_util/Statistics/PltFileFluxAve.H Tools/C_util/Statistics/PltFileFluxAve.cpp Tools/C_util/Statistics/PltFileList.cpp Tools/C_util/Statistics/PltFileStat.cpp Tools/C_util/Statistics/PltFileXAve.H Tools/C_util/Statistics/PltFileXAve.cpp Tools/C_util/TV_TempWrite.H Tools/C_util/ViewMF/GNUmakefile Tools/C_util/ViewMF/MFNorm.H Tools/C_util/ViewMF/MFNorm.cpp Tools/C_util/ViewMF/Make.package Tools/C_util/ViewMF/checkMFghostcells.cpp Tools/C_util/ViewMF/main.cpp Tools/C_util/ViewMF/mfMinMax.cpp Tools/C_util/ViewMF/viewMF.cpp Tools/C_util/ViewMF/viewMFcol.cpp Tools/C_util/ViewMF/viewMFdiff.cpp Tools/C_util/ViewMF/viewMFdiffHardWire.cpp Tools/C_util/dbgTools/GNUmakefile Tools/C_util/dbgTools/Make.package Tools/C_util/dbgTools/crsGrids.cpp Tools/C_util/dbgTools/intersectGrids.cpp commit d35f4c70bd536425fe49acacb7194983802ea10e Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Fri Oct 8 09:37:32 2021 -0600 Adding ILU smoothing parameters for Hypre interface. (#2382) Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit f07f8be1206a4fb7106a24ed611cc5dfcfe5da16 Author: Weiqun Zhang Date: Thu Oct 7 18:33:25 2021 -0700 MLEBNodeFDLaplacian Coarsening (#2386) Limit the maximum coarsening level so that EB objects do not disappear due to coarsening. The solver would fail to converge if the coarsening causes EB objects to disappear entirely. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit b815c734bf01dd6844c55919ddedb4e9541f48ea Author: Weiqun Zhang Date: Thu Oct 7 16:55:50 2021 -0700 Avoid free after amrex::Finalize (#2389) Previously calling Arena::free after amrex::Finalize resulted in segfault because the Arena objects have been deleted. It will cause the following code to crash in the end. int main (int argc, char* argv[]) { amrex::Initialize(argc, argv); amrex::MultiFab(...); amrex::Finalize(); } One way to fix it is to add a scope to the function body between Initialize and Finalize so that the object inside will go out of scope before amrex::Finalize is called. In this commit, we implement a way of avoiding this kind of errors even if one forgets to free everything allocated by Arenas before Finalize. Src/Base/AMReX_Arena.cpp commit c64e719c0a9a5459d2349ef9ebe3a5b8b0e49b41 Author: Candace Gilet Date: Thu Oct 7 17:50:24 2021 -0400 Command line parsing updates. (#2387) Fix command line parsing inconsistency: allow argv[1] to be of the form "arg1 = my_arg", i.e. with spaces, since later arguments allow this. Also, if argv[1] starts with '-', do not use ParmParse. This will prevent "-h" or "--help" from creating errors within ParmParse. Application code can then parse the command line to handle these cases, if desired. Src/Base/AMReX.cpp commit 691e24555db99daa19deeea8da03f951f52a2578 Author: Andrew Myers Date: Wed Oct 6 16:35:44 2021 -0700 These assertions are not needed / wrong after PR #2376 (#2384) Src/Particle/AMReX_ParticleCommunication.H commit f368906eea12d8d68f7f240c17c0cafdfaf9cd1f Author: Andrew Myers Date: Wed Oct 6 15:27:30 2021 -0700 Workaround for nvcc bug that (sometimes) incorrectly complains about unused TinyProfiler variables. (#2383) Src/Base/AMReX_BLProfiler.H commit 3eb5c9edc9d1c62e2bb059e5304edf10559eae1b Author: Andrew Myers Date: Wed Oct 6 13:34:21 2021 -0700 Add inputs for redistribute test that uses a message type other than char. (#2381) Tests/Particles/Redistribute/inputs.rt.cuda.big commit ee975b2973a8e3dfaf5856ea2525832e118e0c98 Author: Neïl Zaim <49716072+NeilZaim@users.noreply.github.com> Date: Wed Oct 6 19:04:28 2021 +0200 Add floor and ceil to parser (#2380) I wanted to use the `floor` function in the parser for a WarpX automated test that I'm developing, so I've added `floor` and `ceil` in the parser. Hopefully I've done it right (I've tested compiling and using with WarpX on my computer and it looks like both functions are working well). Docs/sphinx_documentation/source/Basics.rst Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp commit f0a8fc785ff798cac6bdfe4913ad0fa5db988dfe Author: Axel Huebl Date: Tue Oct 5 10:16:10 2021 -0700 CI: Groups Include Branch (#2379) On pushes to mainline, if mainline contains multiple branches, these branches cancelled each other. This should fix this. .github/workflows/clang.yml .github/workflows/cuda.yml .github/workflows/docs.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/sensei.yml .github/workflows/style.yml .github/workflows/windows.yml commit 64c9d92c0f4680fc4a0cc6fb48f98ee17771d5e4 Author: Andrew Myers Date: Fri Oct 1 19:58:08 2021 -0700 Fix bug in the particle communication routines when the number of chars in the message is > MAXINT. (#2376) I believe this bug was introduced in PR#1569. Src/Particle/AMReX_ParticleCommunication.H commit a7dd9da02211a88efb3c758ef7f8b806204798e1 Author: Erik Date: Fri Oct 1 19:18:13 2021 -0400 Add short decription for AMReX_TESTING flag (#2375) 1. Add the short description, "build for testing --sets MutliFab initial data to NaN" to the CMake build flags table. 2. Two other small wording changes. Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/BuildingAMReX.rst commit 91a9fae04bc2746bec48222731ac9198f3743798 Author: Weiqun Zhang Date: Fri Oct 1 15:31:33 2021 -0700 ROCm now supports __managed__ (#2377) Src/Base/AMReX_GpuQualifiers.H commit e52bccc4d9b95a791284a3e0c00fe27793cc15dd Author: Kevin Z. Zhu <86268612+KZhu-ME@users.noreply.github.com> Date: Fri Oct 1 14:30:50 2021 -0700 Fix read unitialized memory issue in MLMG solver (#2373) ## Summary Running valgrind on one of our failing circleCi tests revealed that the MLMG solver read allocated but uninitialized memory. Setting `m_phi_eb` to 0 after it is allocated fixes this issue. Attached is the valgrind logs, Line 17275 shows where valgrind detected reading of uninitialized memory. [vgout.txt](https://github.com/AMReX-Codes/amrex/files/7269235/vgout.txt) ## Additional background This was found due to one of our warpx circleCI test failing after a fetch and merge with development. This fix did not solve that issue though. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H commit b99a8433d3d5b363957b0391f5487ca95ea9e780 Author: Weiqun Zhang Date: Fri Oct 1 11:59:50 2021 -0700 Support Neumann BC in MLEBNodeFDLaplacian (#2371) Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 3ede71fa7835699784e0fca632980d89d7c23ee5 Author: Weiqun Zhang Date: Fri Oct 1 11:55:41 2021 -0700 Reset GPU RNG Seed (#2374) Add an optional parameter to amrex::ResetRandomSeed to allow the users to reset the GPU RNG seed. Slip in a typo fix. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_Random.H Src/Base/AMReX_Random.cpp commit 98a4e2338b2aef2718981b552de2d3b5db409c14 Author: Andrew Myers Date: Fri Oct 1 11:55:18 2021 -0700 Remove managed memory from neighbor particle test. (#2370) Tests/Particles/NeighborParticles/MDParticleContainer.cpp commit 1211ea30b520eb270e99d63fb80fb1d94ca8e45a Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Fri Oct 1 12:49:46 2021 -0600 NodeLaplacian Updates (#2332) In this branch, I make the number of smoothing steps a user defined parameter. Previously it was fixed at 4 for GPUs and 2 for CPUs. I also modified the NodalPoisson test to 1) use this new parameter 2) include additional parameters that control plot file dumping and the number of trials of the test to be run. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Tests/LinearSolvers/NodalPoisson/MyTest.H Tests/LinearSolvers/NodalPoisson/MyTest.cpp Tests/LinearSolvers/NodalPoisson/main.cpp commit 64ab8faa89e663cb912da9ad03a058c609e95447 Author: Weiqun Zhang Date: Fri Oct 1 07:58:07 2021 -0700 Update CHANGES for 21.10 (#2372) CHANGES commit 6f2dc1d06ee7d3f0024a9b8480ea8dde0d5fd644 Author: Weiqun Zhang Date: Wed Sep 29 15:37:17 2021 -0700 Update for oneapi 2021.4.0 (#2368) There are some namespace changes and deprecated functions in the new release. .github/workflows/intel.yml Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_Scan.H commit b11479d29b987b35089918d40747de72f994e117 Author: Axel Huebl Date: Wed Sep 29 15:09:36 2021 -0700 CI: NVHPC for CUDA (#2066) .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_nvhpc21-9.sh Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_Extension.H Tools/CMake/AMReXGenerateConfigHeader.cmake commit 3eb5af68d209d056c20464c70d9abd9d0641e61a Author: Erik Date: Wed Sep 29 15:32:29 2021 -0400 fix -B grad(phi) typo to -beta grad(phi) (#2367) Fix a typo in the rst documents for the linear solvers. Docs/sphinx_documentation/source/LinearSolvers.rst commit ba6211f431bb594e7fc20478e9309fb88e79acc9 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Wed Sep 29 14:49:54 2021 -0400 CI: Change HIP repo line to reflect latest changes (#2366) This updates the repo line for HIP to the latest version as documented on https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html#installing-a-rocm-package-from-a-debian-repository Notably, `xenial` is now `ubuntu`. .github/workflows/dependencies/dependencies_hip.sh commit 8bf9816cf6f2a50cddad288deab1e8e72ef99e76 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Wed Sep 29 14:09:41 2021 -0400 Port CellConservativeQuartic to C++ with GPU support (#2362) CellConservativeQuartic interpolater has been ported to C++ with GPU support. Errors for CPU-only version on local with AmrLevel SingleVortex 2D and 3D are exactly 0. Errors for GPU-enabled version on Gigan with AmrLevel SingleVortex 2D and 3D are on the order of 1e-14. Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp commit b219a9cbd456863b95fe78a949339ba94fce65ab Author: Axel Huebl Date: Wed Sep 29 10:13:37 2021 -0700 nvc++ w/ OMP: work-around atomic capture (#2365) ## Summary Internal compiler error in this region with NVHPC 21.9 when compiling for OpenMP (host-only). ## Additional background Nvidia bug report: 3390723. Src/Base/AMReX_FabArrayUtility.H commit 514320464dacb9677ae1b84a4aee25ca6669c22c Author: Erik Date: Wed Sep 29 11:45:33 2021 -0400 Gpu Comments to Doxygen Docs (#2354) Added some comments about `Gpu::setLaunchRegion`, `Gpu::synchronize()` and `Gpu::streamSynchronize()` from the new and previous Gpu Debugging write-up. Converted some comments to Doxygen comments. Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuDevice.H commit 5932c85cd650eac65c04e09ca0163fe29de5421b Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Wed Sep 29 00:03:00 2021 -0400 fix AmrMeshParticle cmake issue (#2364) Src/Extern/SENSEI/CMakeLists.txt commit 9b58b442cc7091e5cf18375112d60a2f3d1a1bee Author: Axel Huebl Date: Tue Sep 28 18:42:35 2021 -0700 CI: Split Linux (#2363) The CI list became too long in the Linux section. This is cumbersome if one of the tests fails due to network/runner issues, because then all need to be restarted. Split into: - Linux GCC - Linux Clang - SENSEI .github/workflows/clang.yml .github/workflows/gcc.yml .github/workflows/sensei.yml commit 3006bba653ea8d93b58e779bdf796f3bd07e9bfd Author: Weiqun Zhang Date: Tue Sep 28 14:16:16 2021 -0700 Fix bug introduced in #2327 in DPCPP backend (#2361) Src/Base/AMReX_Reduce.H commit 9256409e2aae8148ef13afcf8316a72f12f1f3b8 Author: Max Katz Date: Tue Sep 28 09:06:16 2021 -0700 Remove hardcoded CUDA version for NVHPC (#2360) Tools/GNUMake/comps/nvhpc.mak commit 26fb18b44481a76d22f182e5955e23d5dbeeef29 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Mon Sep 27 17:27:26 2021 -0400 Port CellConservativeProtected::protect() to C++ with GPU support (#2347) CellConservativeProtected interpolater has been ported to C++ with GPU support. For the CPU-only version on local, the errors are exactly 0. For the GPU-enabled version on Gigan, the errors are in ~1e-11 range for IAMR 2D bubble test case, and ~1e-6 range for IAMR 3D Poiseuille test case. Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp commit 54dc681046f8c3f19034358fe0d45b72d13a9863 Author: Axel Huebl Date: Fri Sep 24 11:25:59 2021 -0700 CMake CUDA Debug: -G by Default (#2355) ## Summary In the past, this often did not compile at all, was very sensitive to further set options, or compiled super slowly; in some cases, such as recursive function usage, apps need to increase `cudaLimitStackSize` in order to not stack overflow with device debug symbols (this costs some extra DRAM). Nonetheless, for CUDA approx. 11.0+, we see the opposite: we have very slow Debug builds with CUDA if we do not activate -G for some `LinearSolvers/MLMG` objects: `AMReX_MLNodeLaplacian.cpp` & `AMReX_MLNodeLaplacian_sten.cpp`. Thus, we default-on now to -G in Debug builds. - [x] testing: now `LinearSolvers/MLMG/AMReX_MLTensorOp.cpp` compiles a bit longer (1-2min), but that's better than the other two TUs above without `-G` in Debug ## Additional background Thanks to @WeiqunZhang and @maxpkatz to looking into this with me. Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXCUDAOptions.cmake Tools/CMake/AMReXParallelBackends.cmake commit 66c90801c63a0c682ae59f0af3d2ebb4cb57f637 Author: kngott Date: Fri Sep 24 10:17:42 2021 -0700 OverlapSync nowait and finish. (#2346) Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp commit 777a2927f0e4da2f398d54914a5bb580c8a305a4 Author: Andrew Myers Date: Thu Sep 23 14:48:49 2021 -0700 Add another overload for PC::Checkpoint (#2353) Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_Particles.H commit 8c79eec68331b7e2122517e776a20cf77dcd04fc Author: Candace Gilet Date: Thu Sep 23 14:13:56 2021 -0400 Add curly brackets between amrex::Initialize and amrex::Finalize. (#2352) This prevents segmentation faults that can occur if amrex objects (like MultiFabs) are still in scope when Finalize is called. Tools/C_util/Convergence/Add.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/Convergence/DiffFab.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedFD.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp Tools/C_util/Convergence/DiffUniform.cpp Tools/C_util/Convergence/PltFileNorm.cpp Tools/C_util/Convergence/PltFileNormB.cpp Tools/C_util/Convergence/PltFileScalConvRate.cpp Tools/C_util/Convergence/RichardsonConvergenceTest.cpp commit 368a808c57ca3dbff40c3aa496e79fdf46c730d7 Author: asalmgren Date: Tue Sep 21 12:42:11 2021 -0700 Call setLevelBC from getFluxes in case we have set up the projector class but not yet called project (#2345) Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit a6c76f123cee34ec57cfb4f11fdf53389da3c640 Author: Andrew Myers Date: Mon Sep 20 18:15:35 2021 -0700 Function to atomically set particle ids (#2344) Src/Particle/AMReX_Particle.H commit 5d6a01f90ca606ac7a46879102db744041414b8f Author: Erik Date: Mon Sep 20 13:41:41 2021 -0400 Faq part one (#2340) Docs/sphinx_documentation/source/Faq.rst Docs/sphinx_documentation/source/index.rst commit 205c9b9ef6996b34138e170985b4d79d028bd57e Author: Weiqun Zhang Date: Mon Sep 20 10:41:16 2021 -0700 Fix CUDA compilation for EBTensor test (#2342) Tests/LinearSolvers/EBTensor/MyTest.H commit 8e0a48ea2e068111e171b8649fa8d71294bfcf7c Author: Weiqun Zhang Date: Mon Sep 20 10:33:21 2021 -0700 Remove macro: CRSEGRNDOMP and AMREX_CRSEGRNDOMP (#2339) The macro was added for backward compatibility when BoxLib first transitioned to omp over tiles. This can be removed now because We no longer have any Fab level functions containing OpenMP parallel regions. Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateData.cpp Tests/EB/CNS/Exec/Make.CNS Tests/EB_CNS/Exec/Make.CNS Tests/GPU/CNS/Exec/Make.CNS commit 3cff89900f8f6c21e02f8daceacffb4fda4b728a Author: Andrew Myers Date: Mon Sep 20 10:21:41 2021 -0700 Force turning off particle tiling for GPU on Advection_AmrLevel test (#2343) This enables the same inputs file to be used for GPU / non-GPU runs. Tests/Amr/Advection_AmrLevel/Source/main.cpp commit d568a0f9bb3ac324ad52b263240ddda03906ef43 Author: Andrew Myers Date: Mon Sep 20 10:20:58 2021 -0700 Don't rely on mananaged memory in PC::InitOnePerCell (#2337) Src/Particle/AMReX_ParticleInit.H commit 350a3efc2c6408efcadb6aaf0a334944af51098b Author: Michael Kieburtz Date: Mon Sep 20 08:26:21 2021 -0700 TinyProfile parser that saves data in a machine-readable format (#2335) This script parses the saved stdout output from a simulation run and parses the TinyProfile portion at the end. It saves this output in a JSON file that can be interpreted by Hatchet (https://github.com/hatchet/hatchet). Tools/TinyProfileParser/profileparser.py commit 677bc0990c7dea0cd32e3b8ab07664791c1681e1 Author: Weiqun Zhang Date: Sat Sep 18 12:12:19 2021 -0700 Various fixes (#2338) * AMReX_EB_LeastSquares_2D_K.H: fix warning * Tests/Amr/Advection_AmrLevel: should use pinned memory on host * Tests/AsyncOut/multifab: avoid managed memory * Tests/EB_CNS: fix warning on extra semicolon * Tests/GPU/AnyOf: avoid touching managed memory on host * Tests/LinearSolvers/CellOverset: fix compilation * Tests/LinearSolvers/EBTensor: initialize data on device * Tests/Parser: print max rel. error * Tests/Particles/AsyncIO: fix typo in inputs * Tests/Particles/DenseBins: do not test serial version for dpcpp * Tests/Particles/GhostsAndVirtuals: do not do randomperbox test for dpcpp Src/EB/AMReX_EB_LeastSquares_2D_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d_K.H Tests/AsyncOut/multifab/main.cpp Tests/EB_CNS/Source/CNS.H Tests/GPU/AnyOf/main.cpp Tests/LinearSolvers/CellOverset/MyTest.cpp Tests/LinearSolvers/EBTensor/MyTest.cpp Tests/LinearSolvers/EBTensor/MyTest_2D_K.H Tests/LinearSolvers/EBTensor/MyTest_3D_K.H Tests/Parser/main.cpp Tests/Particles/AsyncIO/inputs Tests/Particles/DenseBins/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp commit f47f884a453dfb76c372f1cf064e5f1e7df5be33 Author: Weiqun Zhang Date: Fri Sep 17 09:38:44 2021 -0700 Add documentation on reduction using ParallelFor. (#2333) Co-authored-by: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Docs/sphinx_documentation/source/GPU.rst commit 57703d6b624c7b03ff4bd282c3f07ca9aa5bb0fe Author: Jean M. Sexton Date: Fri Sep 17 09:37:45 2021 -0700 Update GPU Debugging basics section (#2330) * Update GPU Debugging basics section * Update Cuda-specific section to include new tools * Added Rocm specific commands * Added some information from Intel debugging webpages * Added links in comments Co-authored-by: etpalmer63 Docs/sphinx_documentation/source/GPU.rst commit 69b69c2b281c6f1b7b9b78a5aa81effebf0e3f2f Author: Weiqun Zhang Date: Mon Sep 13 16:21:41 2021 -0700 Fix return type of MLEBNodeFDLaplacian::setEBDirichlet (#2329) Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H commit 3d5b7d2e99d635960d031d6ad2aa07c9472be2e1 Author: Weiqun Zhang Date: Mon Sep 13 10:27:22 2021 -0700 ParReduce (#2327) ## Summary * Add ParReduce functions for MultiFab/FabArray and 1D iteration space. * The implementation of ParReduce(MF) uses new ReduceOps::eval functions that launch one GPU kernel for the whole MultiFab/MultiFab. * Use the new ParReduce functions in MultiFab/FabArray reduction functions. ## Additional background The name `ParallelReduce` has been used for the namespace containing MPI reduce wrappers. So we have to use a different name. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_ParReduce.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_iMultiFab.cpp Src/Base/CMakeLists.txt Src/Base/Make.package Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp commit 18a57c000e608d9e31054e84eceb799fb2789a4b Author: kngott Date: Mon Sep 13 08:08:44 2021 -0700 Update Profiling API docs. (#2303) Based on previous discussion that TinyProfiler's REGION wasn't well documented, this adds in the discussion about the various methods and macros available for instrumentation. Co-authored-by: etpalmer63 Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst commit 177b78eec251582646050db125af007117e97aae Author: Weiqun Zhang Date: Sun Sep 12 10:05:03 2021 -0700 Remove the MLNodeLinOp test (#2328) The tutorial has been moved to amrex-tutorial. Because of recent changes, the test no longer compiles. Tests/LinearSolvers/MultiComponent/GNUmakefile Tests/LinearSolvers/MultiComponent/MCNodalLinOp.H Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/MultiComponent/Make.package Tests/LinearSolvers/MultiComponent/inputs Tests/LinearSolvers/MultiComponent/main.cpp Tools/RegressionTesting/AMReX-tests.ini commit c1c57379871fec43b3512eab09b12950673e1ae6 Author: Weiqun Zhang Date: Sun Sep 12 08:58:33 2021 -0700 Remove some versions of ParallelFor(MF) (#2326) Remove ParallelFor(MF) functions that only take `int ncomp`, but not `IntVect nghost` to avoid potential bugs. The versions we have kept are, * ParallelFor(MF const&, ...); // valid region only * ParallelFor(MF const&, IntVect const& nghost, ...); // valid+nghost * ParallelFor(MF const&, IntVect const& nghost, int ncomp, ...); // valid+nghost and loop over component The version that is removed is * ParallelFor(MF const&, int ncomp, ...); This avoids bugs like below ParallelFor(mf, mf.nGrow(), ...); // should use mf.nGrowVect() Here, `mf.nGrow()` returns `int` and therefore would be treated as the number of components in the removed version. Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MultiFabUtil.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp commit 1c5608e05856e8690f968900c88924043addf025 Author: Erik Date: Sun Sep 12 00:19:15 2021 -0400 Amrvis doc revisions (#2305) Docs/sphinx_documentation/source/Visualization.rst commit b49296b837bcdd411c9d79f2abf07de18abc5c46 Author: Brandon Runnels Date: Fri Sep 10 14:03:10 2021 -0600 Update MLMG solver to allow variable number of ghost nodes for variable refinement ratios (#2319) Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit b3e38df8297bf32fd715f9f6491314a72c801d3b Author: Erik Date: Fri Sep 10 15:17:35 2021 -0400 Downgrade to docutils v0.16 (#2324) * Downgrade to docutils v0.16 to get bullet lists to render properly with Sphinx. .github/workflows/docs.yml commit f32f822e9ace1f8e4cd711b0bf7dbd174bd94e75 Author: Andrew Myers Date: Fri Sep 10 08:41:50 2021 -0700 Fix bug introduced in PR 2322 (#2325) Src/Base/AMReX_FabArray.H commit 36672da197ff9bda10a90d454212dc1c4ac3f9ff Author: Axel Huebl Date: Fri Sep 10 08:04:44 2021 -0700 CMake: Always CUDA Perf-Neutral Debug (#2321) Always default-ON for performance-neutral debug info to CUDA builds with CMake, even in `Release` mode. Tools/CMake/AMReXCUDAOptions.cmake commit 107ef9a1e1d20c149c2325ab5e8878cf4d7df052 Author: Andrew Myers Date: Thu Sep 9 16:36:41 2021 -0700 Add version of SumBoundary that takes src_nghost (#2322) Src/Base/AMReX_FabArray.H commit 3dc694a754d499c58d6f81f14c530e9e122f5858 Author: Erik Date: Thu Sep 9 12:12:49 2021 -0400 Add comment on "is_periodic" to doxygen (#2320) Co-authored-by: atmyers Src/Base/AMReX_Geometry.H commit d187e374f11c0fbe75c37544e95cb2185410b816 Author: Weiqun Zhang Date: Tue Sep 7 14:49:07 2021 -0700 Tests/EB_CNS on GPU (#2318) * Make some member functions public for CUDA. * Update calls to EBFluxRegister to run on device. * Fix race conditions in two kernels, one by avoiding write with if test and the other with atomics. * Use ParallelFor(MF) instead of ParallelFor(Box) for performance in a few places. * Use MFInterpolater instead of Interpolater for performance. * Remove Elixirs and rely on Gpu::streamSynchronize(). Maybe later, we could switch to using The_Async_Arena for these temporary Fabs for performance. * inline -> AMREX_FORCE_INLINE * Fix some 2D issues. Tests/EB_CNS/Exec/Pulse/cns_prob.H Tests/EB_CNS/Exec/ShockRef/cns_prob.H Tests/EB_CNS/Source/CNS.H Tests/EB_CNS/Source/CNS.cpp Tests/EB_CNS/Source/CNS_K.H Tests/EB_CNS/Source/CNS_advance.cpp Tests/EB_CNS/Source/CNS_advance_box.cpp Tests/EB_CNS/Source/CNS_advance_box_eb.cpp Tests/EB_CNS/Source/CNS_setup.cpp Tests/EB_CNS/Source/CNS_tagging.H Tests/EB_CNS/Source/diffusion/CNS_diffusion_K.H Tests/EB_CNS/Source/diffusion/CNS_diffusion_eb_K.H Tests/EB_CNS/Source/hydro/CNS_divop_K.H Tests/EB_CNS/Source/hydro/CNS_flux_redistribute.cpp Tests/EB_CNS/Source/hydro/CNS_hydro_K.H Tests/EB_CNS/Source/hydro/CNS_hydro_eb_K.H commit 0dab910e02864bb643cc91d78639df06f51fe17c Author: Weiqun Zhang Date: Tue Sep 7 10:17:14 2021 -0700 Remove the Old Fusing Approach (#2308) The old fusing macros and functions are removed in this commit. The old fusing approach is based on device function pointers. It only worked on Nvidia GPUs. Furthermore, the new ParallelFor(MF) and ParallelFor(Tag) approaches are much faster than the old approach. Src/Base/AMReX.cpp Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_Gpu.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuKernelInfo.H Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/Base/CMakeLists.txt Src/Base/Make.package Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Tests/GPU/Fuse/GNUmakefile Tests/GPU/Fuse/Make.package Tests/GPU/Fuse/main.cpp commit 26a55b4a277226a00f8c9e507268b2fd3ef2f3ee Author: kngott Date: Tue Sep 7 10:16:54 2021 -0700 Amrvis build cleanup. (#2314) Src/Extern/amrdata/AMReX_DataServices.H Tools/GNUMake/Make.rules commit 30ad2cb920f59e8d1174fd3a8aa242b8b280df3e Author: Weiqun Zhang Date: Tue Sep 7 08:45:52 2021 -0700 Add const to Array's sum and product functions (#2316) Since C++14, `constexpr` does not imply `const`. This is a follow-up on #2217. Src/Base/AMReX_Array.H commit 4a79504336a34a2728e418e71f3d68dba3afed5d Author: Candace Gilet Date: Mon Sep 6 15:53:28 2021 -0400 Update hypre doc (#2315) Update documentation on using HYPRE with the nodal solver. Support for CoarseningStrategy::Sigma with hypre has been added. Docs/sphinx_documentation/source/LinearSolvers.rst commit a8fe43774d8ad7437ff71bb253175d64a168d933 Author: Andrew Myers Date: Fri Sep 3 16:33:34 2021 -0700 Fix signed/unsigned comparison in diagnostic neighbor list function. (#2312) Src/Particle/AMReX_NeighborList.H commit 168a690497396de4c6b89a36b6edb0430e51ef4c Author: Houjun Tang Date: Fri Sep 3 12:22:04 2021 -0700 HDF5 I/O and Compression (#2220) .github/workflows/linux.yml Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/HDF5Benchmark/GNUmakefile Tests/HDF5Benchmark/inputs Tests/HDF5Benchmark/main.cpp Tools/GNUMake/packages/Make.hdf5 commit 55c18a5e98841034df30f8b91300a64b1269567b Author: Weiqun Zhang Date: Fri Sep 3 10:56:40 2021 -0700 Use ParallelFor(Tag) in Linear Solver Preparation (#2297) Replace the old fusing approach with ParallelFor(Tag) in MLCellLinOp::prepareForSolve. Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H commit 941607f89f68db1dc904d2321b80ac7f622740a9 Author: Weiqun Zhang Date: Fri Sep 3 08:39:44 2021 -0700 TagBoxArray: Remove omp parallel inside GPU launch region (#2309) Src/AmrCore/AMReX_TagBox.cpp commit 0a85e99beadea4cbe4b577a2854e094a834c5e75 Author: Andy Nonaka Date: Fri Sep 3 08:39:11 2021 -0700 make sure there are no NaNs in the plotfiles (#2306) The following routines would report no difference between two input plotfiles if one of them contained NaNs, which makes for a really bad comparison for regression usage. Now they abort with an error message if a NaN is detected. ComparePlotfiles.cpp DiffSameDomainRefined.cpp DiffSameDomainRefinedComposite.cpp DiffSameDomainRefinedStag.cpp DiffSameGrid.cpp DiffSameGrid2.cpp DiffSameGridRefined.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp commit 2bf85c3e7cae74149703c0e5b5020851a2433266 Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Fri Sep 3 11:00:12 2021 -0400 add dropped CMake code for mesh+particle adaptor (#2310) Src/Extern/SENSEI/CMakeLists.txt commit 112ce5551c43c02ff5fb8ca4241ac2d22d89d3e6 Author: Weiqun Zhang Date: Fri Sep 3 07:59:19 2021 -0700 Update DiffMultiFab tool (#2307) Print the location of min and max diff. Make it work with MPI. An example of output is below, ``` diffmultifab3d.gnu.TEST.ex infile1=mf1 infile2=mf2 ngrow=0 Reading mf1 Reading mf2 Component 0 Min and max of the diff are -0.5738299806 and 0.3011247047 Min Index: (63,127,127) Max Index: (75,127,127) Min and max of 1st mf are 0.1249966244 and 1.000010767 Min and max of 2nd mf are 0.125 and 1 Component 1 Min and max of the diff are -1.273924814e-05 and 0.3954904262 Min Index: (31,127,127) Max Index: (64,127,127) Min and max of 1st mf are -1.273924814e-05 and 0.3954904262 Min and max of 2nd mf are 0 and 0 Component 2 Min and max of the diff are 0 and 0 Min and max of 1st mf are 0 and 0 Min and max of 2nd mf are 0 and 0 Writing mfdiff ``` Tools/C_util/DiffMultiFab/GNUmakefile Tools/C_util/DiffMultiFab/diffmultifab.cpp commit e31e0161658664256eec4c3ae1cf98da33273601 Author: Weiqun Zhang Date: Thu Sep 2 16:57:36 2021 -0700 Kernel fusing in linear solver's applyBC (#2295) Use ParallelFor(Tag) to fuse the kernels in the cell-centered solver's applyBC function. Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H commit b15b1cf8d282cbb2c0d0bc0c7b049a79375ea63c Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Thu Sep 2 09:13:59 2021 -0600 Adding new parallel for into normalize method of node laplacian. (#2299) Co-authored-by: Paul Mullowney Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit c428771ceb21e812b92e21bc6383c650192982f1 Author: Andrew Myers Date: Wed Sep 1 17:30:20 2021 -0700 Gpu::Atomic::If (#2301) Src/Base/AMReX_Functional.H Src/Base/AMReX_GpuAtomic.H Tests/GPU/AtomicIf/GNUmakefile Tests/GPU/AtomicIf/Make.package Tests/GPU/AtomicIf/main.cpp commit 17d5b9049942d1cf72f943ee24b9c34cff8f6dfa Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Wed Sep 1 18:47:10 2021 -0400 add adaptor and bridge for amrMesh+Particles (#2285) Src/Extern/SENSEI/AMReX_AmrMeshParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrMeshParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_AmrMeshParticleInSituBridge.H Src/Extern/SENSEI/AMReX_ParticleDataAdaptorI.H commit 05539820d5878f2548b0b84328111efdde25ec7d Author: Weiqun Zhang Date: Wed Sep 1 10:04:07 2021 -0700 Update CHANGES for 21.09 (#2302) CHANGES commit cdb54ed66dac17d8867ee5dce01718852dd1788f Author: Erik Date: Wed Sep 1 12:29:03 2021 -0400 Change list of features to bullet list in Introduction.rst (#2300) Docs/sphinx_documentation/source/Introduction.rst commit 889686189996023c06e796c2cece3e5415a4b6b1 Author: Erik Date: Mon Aug 30 18:18:22 2021 -0400 Add Docs for `fboxinfo`, `fcompare`, `fextract`, etc. to Post-Processing Tools (#2290) Docs/sphinx_documentation/source/Post_Processing.rst Docs/sphinx_documentation/source/figs/ex_fsnapshot_resize.png commit e67bc03243c7df9acec8ae2ec66f9817bbda5fea Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Mon Aug 30 18:16:16 2021 -0400 Add gradient error tagging in Tests/Advection_AmrLevel (#2296) ## Summary Add proper `FillPatch()`ing for gradient checks in `AmrLevelAdv::errorEst()`. ## Additional background While working on #2268, I "accidentally" discovered that even though gradient error tagging is coded in `Advection_AmrLevel`, this check is never actually performed. This PR tries to fix that by adapting the error tagging code from `Tests/GPU/CNS`. Tested on local for CPU (error = exactly 0) and on Gigan for NVIDIA GPU (error = 1e-14 range). Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp commit d0b275b7064e20b87272eda07ea9adef0f63cf2a Author: asalmgren Date: Fri Aug 27 16:21:41 2021 -0700 Move slope routines out of amrex (#2294) * Fix logic for 3D interpolation of cell centroids to face centroids * Remove print statement that I left in by accident * Don't use std::min on the gpu! * When using hypre, limit number of possible multigrid levels by how much the EB can be coarsened * Move slope (EB and regular) routines out of amrex and into AMReX-Hydro * Move Tests/Slopes into AMReX-Hydro as well. Src/Base/AMReX_Slopes_K.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/EB/AMReX_EB_slopes_K.H Src/EB/CMakeLists.txt Src/EB/Make.package Tests/Slopes/GNUmakefile Tests/Slopes/Make.package Tests/Slopes/MyEB.H Tests/Slopes/MyTest.H Tests/Slopes/MyTest.cpp Tests/Slopes/README.md Tests/Slopes/initData.cpp Tests/Slopes/initEB.cpp Tests/Slopes/initLinearData.cpp Tests/Slopes/initLinearDataFor2D.cpp Tests/Slopes/initLinearDataFor3D.cpp Tests/Slopes/inputs Tests/Slopes/inputs.2d.askew Tests/Slopes/inputs.2d.base Tests/Slopes/inputs.2d.fullyrotated Tests/Slopes/inputs.3d.linear.aligned.xy-x Tests/Slopes/inputs.3d.linear.aligned.xy-y Tests/Slopes/inputs.3d.linear.aligned.xz-x Tests/Slopes/inputs.3d.linear.aligned.xz-z Tests/Slopes/inputs.3d.linear.aligned.yz-y Tests/Slopes/inputs.3d.linear.aligned.yz-z Tests/Slopes/inputs.3d.linear.askew-all Tests/Slopes/inputs.3d.linear.askew-xy Tests/Slopes/inputs.3d.linear.askew-xz Tests/Slopes/inputs.3d.linear.askew-yz Tests/Slopes/main.cpp commit a94cd92ab7d73f72af9c46e2b1ac001689a4ae2f Author: Weiqun Zhang Date: Fri Aug 27 13:42:25 2021 -0700 Tweak GNU make on perlmutter (#2293) No need to throw error on `make clean`. Tools/GNUMake/sites/Make.nersc commit 806af6b0c49c6aa2e986e35f44287e952c1b637b Author: Weiqun Zhang Date: Fri Aug 27 11:58:04 2021 -0700 Use ParallelFor(Tag) in communication preparation. (#2292) Use ParallelFor(Tag) instead of the old kernel fusing approach in the initialization of masks in communication preparation. Src/Base/AMReX_FBI.H Src/Base/AMReX_PCI.H Src/Base/AMReX_TagParallelFor.H commit 6c804d92bc18e7ac53d479b0b06c674bfe02d9fc Author: Weiqun Zhang Date: Fri Aug 27 10:03:01 2021 -0700 Update inputs files of Advection_AmrLevel (#2291) There are no longer probin files. Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs.tracers Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs-ci Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs.regt Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-dpcpp-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini commit 1055b29661584f04f9270344033705b09119b625 Author: Weiqun Zhang Date: Fri Aug 27 09:03:12 2021 -0700 Optimization of thread safety check in building communication metadata (#2288) The previous approach is very slow for big boxes. Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_FabArrayBase.cpp commit b20e88cff08fcbf6d92dc5b773dd25c096e355bc Author: kngott Date: Fri Aug 27 09:02:35 2021 -0700 Fix GNU builds on Perlmutter, + some bug fixes. (#2286) Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/sites/Make.nersc commit ef4e008b231fe3c6707d61c75c1da5e4e6f68420 Author: Erik Date: Fri Aug 27 11:46:06 2021 -0400 Short range particles corr (#2271) Docs/sphinx_documentation/source/Particle.rst Docs/sphinx_documentation/source/Visualization.rst Docs/sphinx_documentation/source/index.rst commit 85b8d544e22984096113e85eef6f2d3b56e30d54 Author: Andrew Myers Date: Thu Aug 26 18:27:31 2021 -0700 Silence uninitialized value warning in valgrind in neighbor list construction. (#2287) I don't think this is an actual bug, but it's good to play nice with valgrind. Thanks to Bruce Palmer. Src/Particle/AMReX_NeighborList.H commit af631d7be0877acaf53c47ed01149a5cd557d1f2 Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Thu Aug 26 21:20:44 2021 -0400 fix bug in AmrLevel exclusion logic (#2283) ## Summary Fixes a bug in the build logic regarding optional inclusion of the AMRLEVEL components, and return a required dependency to AMReX_AmrParticleDataAdaptor.H ## Additional background This bug was introduced in https://github.com/AMReX-Codes/amrex/pull/2258 where the SENSEI adaptors and bridges that processed data from the Amr class. A key dependency was removed, in the refactor. This Pr adds that dependency back in, and moves `AMReX_AmrParticleInSituBridge.H` behind the `AMReX_AMRLEVEL` compile option where it belongs. Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptor.H Src/Extern/SENSEI/CMakeLists.txt commit 33d5e4d04c16386147ddad92b3203cd91b033e7a Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Thu Aug 26 17:09:03 2021 -0400 Port Tests/Amr/Advection_AmrLevel from Fortran to C++ GPU (#2268) ## Summary The Advection_AmrLevel test/tutorial has been fully ported from Fortran to C++ (except for `initdata()` subroutine that stays in Fortran). ## Additional background A good chunk of the code was copy-pasted and/or adapted from the AmrCore and GPU/CNS test/tutorials. All four tests {SingleVortex,UniformVelocity} x {2D,3D} have been tested to work on the CPU, NVIDIA GPU on Gigan, and AMD GPU on Spock (`fcompare` shows differences of the order of 1e-14). Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/AmrLevel.rst Docs/sphinx_documentation/source/AmrLevel/figs/flowchart.odg Docs/sphinx_documentation/source/AmrLevel/figs/flowchart.png Tests/Amr/Advection_AmrCore/Exec/inputs-ci Tests/Amr/Advection_AmrLevel/CMakeLists.txt Tests/Amr/Advection_AmrLevel/Exec/Make.Adv Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Adv_prob.cpp Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Make.package Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob_Parm.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d_K.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs-ci Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs.tracers Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/probin Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Adv_prob.cpp Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Make.package Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob_Parm.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs-ci Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs.regt Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/probdata.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/probin Tests/Amr/Advection_AmrLevel/README Tests/Amr/Advection_AmrLevel/Source/Adv.cpp Tests/Amr/Advection_AmrLevel/Source/Adv_F.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Amr/Advection_AmrLevel/Source/Kernels.H Tests/Amr/Advection_AmrLevel/Source/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_2d/Adv_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_2d/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_2d/compute_flux_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_2d/slope_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Adv_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_3d/compute_flux_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/slope_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_K/Adv_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_K/flux_2d_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/flux_3d_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/slope_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/tagging_K.H Tests/Amr/Advection_AmrLevel/Source/Src_nd/Adv_nd.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_nd/Tagging_nd.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/tagging_params.f90 Tests/Amr/Advection_AmrLevel/Source/Tagging_params.cpp Tests/Amr/Advection_AmrLevel/Source/bc_nullfill.cpp Tests/GPU/CNS/Source/CNS.H commit cc16557fbcbd9e023272c9db8830d5b837265e10 Author: Weiqun Zhang Date: Thu Aug 26 11:47:44 2021 -0700 Use ROCm 4.3 in CI (#2280) * Use ROCm 4.3 in CI Since the VOP bug has been fixed in the latest ROCm release v4.3, CI can use the latest ROCm release now instead of 4.1. * Work-Around: OpenMPI 4.0.3 on Ubuntu 20.04 "mpic++ --showme" forgets to link open-pal ``` /opt/rocm/llvm/bin/clang++ -pthread CMakeFiles/cmTC_abf94.dir/test_mpi.cpp.o -o cmTC_abf94 -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so ld.lld: error: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so: undefined reference to opal_class_init_epoch [--no-allow-shlib-undefined] ld.lld: error: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so: undefined reference to opal_list_item_t_class [--no-allow-shlib-undefined] ld.lld: error: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so: undefined reference to opal_class_initialize [--no-allow-shlib-undefined] ld.lld: error: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so: undefined reference to opal_uses_threads [--no-allow-shlib-undefined] ``` Results in `MPI_CXX` being not found. Co-authored-by: Axel Huebl .github/workflows/dependencies/dependencies_hip.sh .github/workflows/hip.yml commit 44d7b56878609c544b0f2906691786fc4d41633e Author: Weiqun Zhang Date: Thu Aug 26 08:04:52 2021 -0700 ParallelFor(Tag) (#2281) We have been using a ParallelFor function template that takes a Vector of Tags in communication functions. It launches a single GPU kernel for an irregular iteration space. This function is useful in many situations. Therefore, it is moved from a FillBoundary implementation header to its own header. And we have added two more versions of this for 1d and 3d loops. Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_TagParallelFor.H Src/Base/CMakeLists.txt Src/Base/Make.package commit fc6dcb883c4460523b7ae8164a2e26922511b857 Author: Andrew Myers Date: Wed Aug 25 17:08:44 2021 -0700 remove faulty assertion from PC::Restart() (#2279) Src/Particle/AMReX_ParticleIO.H commit 298b4892d082bf281b7cb4bab9423991e059bf3a Author: Andrew Myers Date: Wed Aug 25 16:05:06 2021 -0700 Protect against nullptr when calling PODVector dtor. (#2272) Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_PODVector.H commit 44edcc104f551b6243984b567ccd6723ac336699 Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Wed Aug 25 18:18:35 2021 -0400 remove amr adaptor/bridge when AMRLEVEL is off (#2258) Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_InSituBridge.cpp Src/Extern/SENSEI/CMakeLists.txt commit cbfe59d9d11140326e943681a8675e072cfedb91 Author: Weiqun Zhang Date: Wed Aug 25 15:17:06 2021 -0700 Parser: Hide local symbols (#2274) Previously `Parser::symbols()` returned a list of symbols including local symbols. This is problematic because WarpX makes sure that all symbols are either a registered variable or constant. But the local variables are unknown to WarpX. In this commit, the local symbols are excluded from the return value of `Parser::symbols()`. Src/Base/Parser/AMReX_IParser_Y.H Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp commit 1e5492c974dc866749e0557f329a32250e38e524 Author: Erik Date: Wed Aug 25 17:14:46 2021 -0400 Doc macros (#2275) ## Summary Convert some comments on macros to Doxygen style for inclusion in the docs. ## Additional background I did not see a straightforward way to include comments on the variables set at compile time. Docs/Doxygen/doxygen.conf Src/Base/AMReX_SPACE.H commit 742069a0445fd511a5c64271d4ca21f22bab9b59 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Wed Aug 25 17:12:51 2021 -0400 Add EditorConfig section for rst files (#2273) ## Summary Add a section for `*.rst` files for EditorConfig. ## Additional background When editing the Sphinx `rst` source files for AmrLevel (PR #2268), I found out that the root `.editorconfig` file doesn't have a section for `rst` files. This PR adds that section, roughly based on the guidelines at https://docs.typo3.org/m/typo3/docs-how-to-document/master/en-us/GeneralConventions/CodingGuidelines.html . Note that I haven't gauged the "disruption" from whitespace changes that might be caused by this PR. .editorconfig commit 83de0324465f1f62d04c51a43e7212be348ad833 Author: Andrew Myers Date: Wed Aug 25 12:04:18 2021 -0700 Fix bug in the dual grid PC restart in the case where there are no particles on some levels. (#2276) Src/Particle/AMReX_ParticleIO.H commit bc3ad8347420a188838159b52d9bdcc5e9407060 Author: Weiqun Zhang Date: Tue Aug 24 10:59:59 2021 -0700 MF ParallelFor: LinearSolvers (#2267) Replace some `*_FUSIBLE` macros with MF ParallelFor. Note that the FUSIBLE macros are still used in some boundary functions. Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_MultiMask.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_F.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H commit 57618b0d0a46f9321cd558a2707503ce128bbe82 Author: Weiqun Zhang Date: Mon Aug 23 19:32:32 2021 -0700 Fix FabArray move-ctor and move-= (#2270) A bug was introduced to the move ctor and move operator= in the MF ParallelFor PR #2073. The cached arrays were not properly moved. Src/Base/AMReX_FabArray.H commit 6df13f8bb81caf4fdf9cda141e20994cd83018b2 Author: Weiqun Zhang Date: Mon Aug 23 12:49:57 2021 -0700 Fine-tune FabArray::setBndry (#2269) Use either ParallelFor(MF) or ParallelFor(tag) depending on box sizes. Src/Base/AMReX_FabArray.H commit 60578291de339964af938b81f2d0f7db5c6bf78c Author: Weiqun Zhang Date: Fri Aug 20 19:02:33 2021 -0700 MF ParallelFor: PETSc (#2262) * Use the new MF ParallelFor in PETSc setup. * Fix some existing issues in PETSc setup, including compilation in 3D and with CUDA. Src/Base/AMReX_MFParallelForG.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/AMReX_PETSc.cpp Tools/GNUMake/packages/Make.petsc commit e488c939316a2d52f0c4d3394689c4c154c6deb4 Author: Jean M. Sexton Date: Fri Aug 20 16:31:20 2021 -0700 Add new machine to GNUMake based on Make.olcf (#2264) Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.frontier-coe commit 6c5909a99ca327ac68aef9f2233bf4ff41de5248 Author: Weiqun Zhang Date: Fri Aug 20 10:29:11 2021 -0700 MF ParallelFor: HYPRE (#2257) * Use the new MF ParallelFor in HYPRE setup and remove the use of fusible macros. * Fix typos Src/Base/AMReX_LayoutData.H Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/PETSc/AMReX_PETSc.cpp commit 8f3d3511812d78ae5d5ca4f2620331cf8e3f3faa Author: Erik Date: Fri Aug 20 11:53:39 2021 -0400 Template wording (#2261) Changes the wording for the last checkbox on this pull request template. .github/pull_request_template.md commit 5e06ff8487368901c5fa79e5c7e8c944c62f7096 Author: Erik Date: Fri Aug 20 11:52:06 2021 -0400 Update contributing md (#2259) Adds style guide and short doxygen example to CONTRIBUTING.md CONTRIBUTING.md commit f840732f77100da5ebd40605a06e44ec44799516 Author: Weiqun Zhang Date: Thu Aug 19 12:27:44 2021 -0700 Fix #2256 (#2260) In #2256, a no-allocated MultiFab was passed to the MF ParalleFor. This resulted in segfault because of how the ParallelFor was implemented. In this commit, we fix the bug by reimplementing it without relying on Fabs being allocated. Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFParallelForG.H commit ebd1bac74492dc90def7fadea9e15ea77cb0e7a9 Author: Weiqun Zhang Date: Wed Aug 18 15:20:27 2021 -0700 MF ParallelFor: MultiFabUtil (#2256) Use the new MF ParallelFor in some MultiFabUtil functions. Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/F_Interfaces/Octree/AMReX_octree_fi.cpp commit 989d98b849ebaf31f000f60ff48ffd09304ff621 Author: Weiqun Zhang Date: Tue Aug 17 15:02:01 2021 -0700 Fix Particle CI Tests (#2252) A number of CI tests had issues with mixing single precision particle data with double precision amrex::Real. .github/workflows/linux.yml Src/AmrCore/AMReX_MFInterp_1D_C.H Src/Particle/AMReX_ParticleInit.H Tests/Particles/AssignDensity/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/main.cpp Tests/Particles/ParticleReduce/main.cpp commit 4f5800cce8396eda1b4692faabe05fc26273e738 Author: Weiqun Zhang Date: Tue Aug 17 15:01:19 2021 -0700 Github Action Concurrency (#2254) Cancel previous jobs in progress if a PR is updated. .github/workflows/cuda.yml .github/workflows/docs.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml .github/workflows/macos.yml .github/workflows/style.yml .github/workflows/windows.yml commit ba6a7467e5235616be6730384302ed3b684eea05 Author: Erik Date: Tue Aug 17 16:44:25 2021 -0400 Add live links (#2255) * correct sp typo * spelling and comma * Add live links. Some other small formatting. Docs/sphinx_documentation/source/AmrLevel_Chapter.rst Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/Fortran.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Docs/sphinx_documentation/source/IO.rst Docs/sphinx_documentation/source/LinearSolvers.rst Docs/sphinx_documentation/source/LinearSolvers_Chapter.rst Docs/sphinx_documentation/source/Particle.rst Docs/sphinx_documentation/source/SWFFT.rst commit b97214dbeaf2a799a56c802a7e07d1265bcbaf5a Author: Weiqun Zhang Date: Tue Aug 17 13:12:28 2021 -0700 MF ParallelFor: MultiFab (#2249) Use the new ParallelFor in some MultiFab functions. Src/Base/AMReX_MultiFab.cpp commit 4f1ebf2213c396f077dd7c70fdf8c9e65e8caff2 Author: Andrew Myers Date: Tue Aug 17 12:10:18 2021 -0700 Reimplement AddParticlesAtLevel for GPU (#2236) * Reimplement AddParticlesAtLevel for GPU * Don't assume grid 0, tile 0 is empty. Src/Particle/AMReX_ParticleContainerI.H commit 48bad74b9ca0642fa794925156d053fd14e3a93e Author: Lucas Esclapez <13371051+esclapez@users.noreply.github.com> Date: Tue Aug 17 12:05:35 2021 -0700 Enable mixed (face) GpuBndryFuncFab (#2238) Add face-centered Fill function. Only foextrap, hoextrap, reflectodd/even supported right now. Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_FilFC_1D_C.H Src/Base/AMReX_FilFC_2D_C.H Src/Base/AMReX_FilFC_3D_C.H Src/Base/AMReX_FilFC_C.H Src/Base/AMReX_PhysBCFunct.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 68327bc3ccfbb9bf31e467d479db6a854626e071 Author: Erik Date: Tue Aug 17 13:38:22 2021 -0400 Remove eb doc section (#2253) Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/EB/areas_and_volumes.fig Docs/sphinx_documentation/source/EB/areas_and_volumes.png Docs/sphinx_documentation/source/EB/eb_fluxes.fig Docs/sphinx_documentation/source/EB/eb_fluxes.png Docs/sphinx_documentation/source/EB/redist.fig Docs/sphinx_documentation/source/EB/redist.png commit 9a97484ad7940bfa9a1aa9064f07cc326992196a Author: Jean M. Sexton Date: Tue Aug 17 10:31:52 2021 -0700 Make sure all ba are defined in test, and only call Init once per PC (#2247) * Change boxes so all ba are defined, and use a new ParticleContainer for initializing tests * Remove duplicate workaround * GhostsAndVirtuals: Cleanup - Safety: use `.at()` over `operator[]` for performance-uncritical code - Some formatting * CI: Test Particles_GhostsAndVirtuals Co-authored-by: Axel Huebl .github/workflows/linux.yml .github/workflows/macos.yml Tests/Particles/GhostsAndVirtuals/main.cpp commit 2788e558156219f84840f0ab73d42dcfd344f21e Author: Weiqun Zhang Date: Mon Aug 16 18:23:23 2021 -0700 Shorten CI tests and Enable CNS in CI (#2251) * Split macos ci tests into two checks * Split hip ci tests * Change some build types from Debug to RelWithDebInfo * Enable Assertions and FPE in some tests * Add some inputs for ci * Enable CNS in CI .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml .github/workflows/macos.yml Tests/Amr/Advection_AmrCore/CMakeLists.txt Tests/Amr/Advection_AmrCore/Exec/inputs-ci Tests/Amr/Advection_AmrLevel/CMakeLists.txt Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs-ci Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs-ci Tests/EB/CNS/CMakeLists.txt Tests/EB/CNS/Exec/Sod/inputs-ci Tests/EB_CNS/CMakeLists.txt Tests/EB_CNS/Exec/Combustor/inputs-ci Tests/GPU/CNS/CMakeLists.txt Tests/GPU/CNS/Exec/Sod/inputs-ci Tests/LinearSolvers/NodalPoisson/CMakeLists.txt Tests/LinearSolvers/NodalPoisson/inputs-ci commit 79a57600d5eec3c9a0ffdbb717462728a1d56598 Author: Weiqun Zhang Date: Mon Aug 16 10:22:17 2021 -0700 Use the new ParallelFor in some FabArrayUtility functions (#2246) Src/Base/AMReX_FabArrayUtility.H commit a4f94e7524c0348af134968a9e17c7fd08f266f7 Author: Weiqun Zhang Date: Sun Aug 15 15:31:49 2021 -0700 Use new ParallelFor in FabArray (#2245) * Add FabArrayBase::isFusingCandidate() that is fine tuned for MI100, V100 and A100. * Convert some FabArray functions to using the new ParallelFor. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp commit c94cbd9b78cc8fa559c00c45c49b9add0075cc0f Author: Axel Huebl Date: Sat Aug 14 18:50:47 2021 -0700 Fix: Slopes K Literals (#2250) * Fix: Slopes K Literals Address compile issues with GCC 9.3.0 (Manda) / 10.1.0 (Battra). Seen with GNUmake by Ann. * Slopes K: Cleanup - Add used headers - move into regular amrex:: namespace - simplify class usage in amrex:: namespace Src/Base/AMReX_Slopes_K.H commit 9f58fc5c936129424f2a64d5c2c4e4de243bc3b5 Author: Axel Huebl Date: Sat Aug 14 12:10:51 2021 -0700 CI: Build Embedded Boundaries (#2232) - [x] Add coverage for EB in CI - [x] fix DPC++ single precision build - [x] ~~fix `LinearSolvers` array bounds w/ GCC 7.5~~ -> false positive Co-authored-by: Weiqun Zhang .github/workflows/cuda.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml .github/workflows/macos.yml .github/workflows/windows.yml Src/Base/AMReX_Slopes_K.H Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EB_utils.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_eb.cpp Tests/EB/CNS/Source/CNS.H Tests/EB_CNS/Source/hydro/CNS_divop_K.H commit cc89a4bc60b54902f4b07f531a03cd2c1b834e58 Author: Andrew Myers Date: Fri Aug 13 19:34:28 2021 -0700 Also remove empty map entries when calling clearParticles (#2248) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particles.H commit e8512831302f27820e0e6a0714f83f3905e89895 Author: Maxim Shatsky Date: Fri Aug 13 19:29:28 2021 +0300 Bug fix of rarely met issue from Tools/CMake/AMReXUtils.cmake file (#2244) Everything is already described here: https://github.com/AMReX-Codes/amrex/issues/2240 Tools/CMake/AMReXUtils.cmake commit 090596bd4241497616cfbb86f168556d539ceb42 Author: Erik Date: Thu Aug 12 16:30:34 2021 -0400 Add definition of MLMG (#2242) Docs/sphinx_documentation/source/LinearSolvers.rst Src/Base/AMReX_MultiFab.H commit fb39fa7ac33ed916f42cc26d134c377113e78fa6 Author: Weiqun Zhang Date: Thu Aug 12 12:40:37 2021 -0700 Move MF ParallelFor out of experimental:: (#2241) It's still available in experimental::, but it will be removed from it eventually. Src/AmrCore/AMReX_MFInterpolater.cpp Src/Base/AMReX_MFParallelFor.H Src/EB/AMReX_EBMFInterpolater.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 9a7688371d000798c71a3ca246bd983bc9d300b6 Author: Andrew Myers Date: Thu Aug 12 08:58:11 2021 -0700 Fix some tests so that ctests works with CUDA enabled. (#2239) This also fixes and scoping bug in the particle iterator test. Tests/LinearSolvers/NodeTensorLap/CMakeLists.txt Tests/Particles/ParallelContext/CMakeLists.txt Tests/Particles/ParallelContext/inputs.rt.cuda Tests/Particles/ParticleIterator/main.cpp Tests/Particles/Redistribute/CMakeLists.txt commit d39e0b86813852d6c2a38f57f6243cdb750e08df Author: Weiqun Zhang Date: Thu Aug 12 08:53:50 2021 -0700 Option for dynamic tiling in CPU build of MF ParallelFor (#2235) Examples: experimental::ParallelFor(mf, nghost, TileSize{my_tile_size}, DynamicTiling{true}, [=] ...); experimental::ParallelFor(mf, nghost, ncomp, TileSize{my_tile_size}, DynamicTiling{true}, [=] ...); Co-authored-by: Andrew Myers Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MFParallelForC.H Src/Base/AMReX_MFParallelForG.H commit 694279b0fb23b90a40ac8ba93fe5ed90b8510123 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Wed Aug 11 21:55:14 2021 -0400 Add sum(), product(), size(), {x,y,z}{lo,hi,len}() member functions to GpuArray and Array?D (#2217) Now `GpuArray` and `Array1`/`2`/`3D` has member functions `sum()` and `product()`. These functions will take the sum or product over the entire array. For Array2D, both can optionally take in `int axis` and `int loc`. Same goes for `Array3D`, but it takes two positions. The following static member functions to return array bounds and sizes are added: * `Array1D`: `lo()`, `hi()`, `len()`, `size()` * `Array2D`: `xlo()`, `xhi()`, `xlen()`, `ylo()`, `yhi()`, `ylen()`, `size()` * `Array3D`:`xlo()`, `xhi()`, `xlen()`, `ylo()`, `yhi()`, `ylen()`, `zlo()`, `zhi()`, `zlen()`, `size()` The `*lo()` and `*hi()` functions return an `int`, and the `*len()` (per direction) and `size()` (total in all directions) functions return a `uint`. In addition, the `begin()` and `end()` member functions have also been added for `Array?D`, which return the start pointer address and the pointer address right after the last element, for iterator purposes, if needed. For `Array2`/`3D`, it's implemented as if the array is one-dimensional. Co-authored-by: etpalmer63 Co-authored-by: Weiqun Zhang Src/Base/AMReX_Array.H commit e82d45264b7840ff9b6247c4a7eb534f55ed5233 Author: Weiqun Zhang Date: Wed Aug 11 10:55:13 2021 -0700 Fix derive data in AmrLevel::writePlotFile (#2234) Previously, it only worked with single-component DeriveList. Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Derive.cpp Tests/GPU/CNS/Exec/RT/inputs Tests/GPU/CNS/Exec/RT/inputs-rt Tests/GPU/CNS/Exec/Sod/inputs Tests/GPU/CNS/Exec/Sod/inputs-rt Tests/GPU/CNS/Source/CNS_derive.cpp Tests/GPU/CNS/Source/CNS_setup.cpp commit f4c2629340bf5fdac41d869321e0514f68654789 Author: Weiqun Zhang Date: Tue Aug 10 19:41:27 2021 -0700 Add MFNodeBilinear for nodal linear interpolation (#2202) Src/Amr/AMReX_AmrLevel.cpp Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_MFInterpolater.cpp commit 0ca401daa57fd80e7ae7a12eb5fd28b9d10ce11b Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Tue Aug 10 11:33:23 2021 -0600 Adding experimental parfor into the FSmooth method (#2230) Co-authored-by: Paul Mullowney Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 075f30f8ad323c2a8726e35d52ff70f53c681a49 Author: Weiqun Zhang Date: Tue Aug 10 10:15:07 2021 -0700 Parser: Exception handling (#2233) For unknown characters, syntax errors, and unknown variables, throw exceptions with hopefully more helpful messages. Src/Base/Parser/AMReX_IParser.H Src/Base/Parser/AMReX_IParser.cpp Src/Base/Parser/AMReX_IParser_Exe.cpp Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Exe.cpp Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_iparser.l Src/Base/Parser/amrex_iparser.lex.cpp Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp commit 0528d849c6c3d17193263f1c4aa15a6b07e7ed1a Author: Weiqun Zhang Date: Mon Aug 9 18:12:57 2021 -0700 Parser: Noinline transcendental functions (#2229) It's too expensive to have them inlined. Src/Base/Parser/AMReX_Parser_Y.H commit a2ad6250aabed7042f33b7c5356274104165ef4b Author: asalmgren Date: Sun Aug 8 13:50:59 2021 -0700 Fix 2d eb slope limiting (#2226) There was a bug in the 2D EB slope limiting -- this fixes that Src/EB/AMReX_EB_slopes_K.H commit 867d1deb3b62ca99ce50a95eb190a548eb119112 Author: asalmgren Date: Sat Aug 7 13:22:58 2021 -0700 New test eb cns (#2208) This directory Tests/EB_CNS is designed to replace Tests/EB/CNS -- it is a pure-C++ version of EB/CNS and matches the current regression tests to sufficient precision. It is not GPU-ready. Tests/EB_CNS/CMakeLists.txt Tests/EB_CNS/Exec/Combustor/CNS_bcfill.cpp Tests/EB_CNS/Exec/Combustor/GNUmakefile Tests/EB_CNS/Exec/Combustor/Make.package Tests/EB_CNS/Exec/Combustor/cns_prob.H Tests/EB_CNS/Exec/Combustor/cns_prob.cpp Tests/EB_CNS/Exec/Combustor/cns_prob_parm.H Tests/EB_CNS/Exec/Combustor/cns_prob_parm.cpp Tests/EB_CNS/Exec/Combustor/inputs Tests/EB_CNS/Exec/Combustor/inputs.regt Tests/EB_CNS/Exec/Make.CNS Tests/EB_CNS/Exec/Pulse/GNUmakefile Tests/EB_CNS/Exec/Pulse/Make.package Tests/EB_CNS/Exec/Pulse/cns_prob.H Tests/EB_CNS/Exec/Pulse/cns_prob.cpp Tests/EB_CNS/Exec/Pulse/cns_prob_parm.H Tests/EB_CNS/Exec/Pulse/inputs Tests/EB_CNS/Exec/Pulse/inputs.regt Tests/EB_CNS/Exec/ShockRef/GNUmakefile Tests/EB_CNS/Exec/ShockRef/Make.package Tests/EB_CNS/Exec/ShockRef/cns_prob.H Tests/EB_CNS/Exec/ShockRef/cns_prob.cpp Tests/EB_CNS/Exec/ShockRef/cns_prob_parm.H Tests/EB_CNS/Exec/ShockRef/inputs Tests/EB_CNS/Exec/ShockRef/inputs.amr Tests/EB_CNS/Exec/ShockRef/inputs.regt Tests/EB_CNS/Exec/Sod/GNUmakefile Tests/EB_CNS/Exec/Sod/Make.package Tests/EB_CNS/Exec/Sod/cns_prob.H Tests/EB_CNS/Exec/Sod/cns_prob.cpp Tests/EB_CNS/Exec/Sod/cns_prob_parm.H Tests/EB_CNS/Exec/Sod/inputs Tests/EB_CNS/Source/CNS.H Tests/EB_CNS/Source/CNS.cpp Tests/EB_CNS/Source/CNSBld.cpp Tests/EB_CNS/Source/CNS_K.H Tests/EB_CNS/Source/CNS_advance.cpp Tests/EB_CNS/Source/CNS_advance_box.cpp Tests/EB_CNS/Source/CNS_advance_box_eb.cpp Tests/EB_CNS/Source/CNS_bcfill.cpp Tests/EB_CNS/Source/CNS_derive.H Tests/EB_CNS/Source/CNS_derive.cpp Tests/EB_CNS/Source/CNS_index_macros.H Tests/EB_CNS/Source/CNS_init_eb2.cpp Tests/EB_CNS/Source/CNS_io.cpp Tests/EB_CNS/Source/CNS_parm.H Tests/EB_CNS/Source/CNS_parm.cpp Tests/EB_CNS/Source/CNS_setup.cpp Tests/EB_CNS/Source/CNS_tagging.H Tests/EB_CNS/Source/Make.package Tests/EB_CNS/Source/diffusion/CNS_diffusion_K.H Tests/EB_CNS/Source/diffusion/CNS_diffusion_eb_K.H Tests/EB_CNS/Source/diffusion/Make.package Tests/EB_CNS/Source/hydro/CNS_divop_K.H Tests/EB_CNS/Source/hydro/CNS_flux_redistribute.cpp Tests/EB_CNS/Source/hydro/CNS_hydro_K.H Tests/EB_CNS/Source/hydro/CNS_hydro_eb_K.H Tests/EB_CNS/Source/hydro/Make.package Tests/EB_CNS/Source/main.cpp commit eed6b0c572301d793c098c3dae7286010d046182 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Sat Aug 7 12:08:36 2021 -0600 Adding experimental parfor into the restriction operator (#2204) Co-authored-by: Paul Mullowney Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit 660a17309966e07fdda41b66d7016e23795a846d Author: Weiqun Zhang Date: Sat Aug 7 10:52:26 2021 -0700 Fix PArena (#2221) In some PArena functions, we forgot to test if the memory pool is supported. ## Additional background https://github.com/AMReX-Astro/Castro/issues/1955 Src/Base/AMReX_PArena.cpp commit 468feb057586e06e069b22f62468c572f7172856 Author: Axel Huebl Date: Sat Aug 7 10:51:53 2021 -0700 Test Build Variant (#2223) The AMReX "test" build variant that is used by [regression tests](https://github.com/AMReX-Codes/regression_testing) by default was not documented, which leads to some confusion when debugging failing apps. This PR: - documents GNUmake's `TEST` & `USE_ASSERTION` options - adds a CMake option `AMReX_TESTING` that are in sync with the define `-DAMREX_TESTING` that changes FAB default values and other debug-like options. Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXSetDefines.cmake commit 2c98061cb00d552ec6262cdffc475e4c0cc92445 Author: Weiqun Zhang Date: Sat Aug 7 10:44:29 2021 -0700 EB: Fix small cells and multicuts (#2225) The previous approach of fix small cells and multicuts is not consistent. If a small cell is turned into a covered cell, we need to fix its neighbors too. In this new approach, we build the EB data iteratively. This commit also introduces a new ParmParse parameter, eb2.maxiter, that controls the maximal number of iterations. Src/Base/AMReX_GpuBuffer.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_MultiGFab.H Src/EB/AMReX_EB2_MultiGFab.cpp Src/EB/AMReX_EB2_ND_C.cpp Tests/LinearSolvers/CellEB/MyEB.H commit bca7ff1fdd2d670eda38c0304ebf8741f15b0873 Author: Jean M. Sexton Date: Thu Aug 5 13:55:59 2021 -0700 Make CreateVirtualParticles use GPU functions for aggregation_type=Cell (#2219) Src/Particle/AMReX_ParticleContainerI.H Tests/Particles/GhostsAndVirtuals/fixed_grids.init Tests/Particles/GhostsAndVirtuals/main.cpp commit cbd153a6d0b483b82b7ec5c540488cd5b97c5189 Author: asalmgren Date: Thu Aug 5 12:11:52 2021 -0700 Change getgradphi in nodalproj (#2222) * Add options to return const or non-const vector of pointers to phi and gradphi from nodal projector * oops -- left in extra const Src/LinearSolvers/Projections/AMReX_NodalProjector.H commit 6690a90f407c536c0acf43e6c8fc3862ab6d7097 Author: Jean M. Sexton Date: Tue Aug 3 16:48:21 2021 -0700 Add static_cast for ParticleReal, and check assertion (#2215) Src/Amr/AMReX_Derive.cpp Src/Particle/AMReX_NeighborParticlesCPUImpl.H Tests/Particles/AsyncIO/main.cpp Tests/Particles/NeighborParticles/MDParticleContainer.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/main.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleReduce/main.cpp Tests/Particles/ParticleTransformations/main.cpp Tests/Particles/Redistribute/main.cpp commit 3b2c8de1ea1d8b67cb9910a142249c5cc6fe7e75 Author: Weiqun Zhang Date: Tue Aug 3 15:04:09 2021 -0700 Fix EBInterpolater (#2214) Fix bug in #2203. The fine factory may not contain information on boxes outside the fine target region. Src/EB/AMReX_EBInterpolater.cpp Src/EB/AMReX_EBMFInterpolater.cpp commit ac818cd22aded63eff8358ad54e0567232163341 Author: Jean M. Sexton Date: Tue Aug 3 14:49:40 2021 -0700 Add check for OpenMP::get_max_threads()=0 case for CI warning (#2212) Src/Base/AMReX_MultiFabUtil.cpp commit 5da293757b8bc67225702d3fe1ddc6fb0c010fa4 Author: Weiqun Zhang Date: Mon Aug 2 15:15:44 2021 -0700 Add EBMFCellConsLinInterp for EB cell data interpolation (#2203) Src/Amr/AMReX_StateDescriptor.H Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_MFInterpolater.H Src/EB/AMReX_EBInterpolater.H Src/EB/AMReX_EBInterpolater.cpp Src/EB/AMReX_EBMFInterpolater.H Src/EB/AMReX_EBMFInterpolater.cpp Src/EB/CMakeLists.txt Src/EB/Make.package commit af968f828c51d7d6d07646c69fe8e53b85a7d6a1 Author: Weiqun Zhang Date: Mon Aug 2 14:05:11 2021 -0700 Bilinear interpolation for cell data (#2205) Add MFCellBilinear for cell data. Port CellBilinear to C++. Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_MFInterpolater.cpp commit ee8f6828dd1162ef6bcf123c417b31abf63f4eef Author: Axel Huebl Date: Mon Aug 2 10:08:24 2021 -0700 CMake: Prebuild & SetupCUDA (#2209) ## Summary With CMake 3.20+, we don't need to include the `SetupCUDA.cmake` scripts anymore. ## Additional background Follow-up to #2012 Tools/CMake/AMReXConfig.cmake.in commit 2695be7cad80f2b4ec25abe57925cc166e3aa824 Author: Weiqun Zhang Date: Mon Aug 2 08:59:50 2021 -0700 Reduction of MulitFab to 1D data (#2182) Add `sumToLine` function that reduces MultiFab's data to a 1D line. Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp commit b076fa56f696acc3591a2b6ee03cc6f981731cf4 Author: Weiqun Zhang Date: Mon Aug 2 08:59:15 2021 -0700 YAFluxRegister with blocking factor of 1 (#2155) Fix YAFluxRegister for the case that the blocking factor is 1. In that case, a coarse cell might have fine neighbors at both x-lo and x-hi faces. Src/Boundary/AMReX_YAFluxRegister_1D_K.H Src/Boundary/AMReX_YAFluxRegister_2D_K.H Src/Boundary/AMReX_YAFluxRegister_3D_K.H commit 3fdfac26cd238cfc0c8c4fd8a754dfd4b8cd3eeb Author: Weiqun Zhang Date: Mon Aug 2 08:32:44 2021 -0700 Fix lincc_interp (#2201) This fixes a bug in lincc_interp that was introduced more than 2 years ago when the Fortran code was converted to C++. The slope was mistakenly limited twice. For each component, we compute a limiting factor and we use the minimum of all components as the final limiting factor. That final limiting factor should be applied to the unlimited central difference slope, not the monotonized central slope. Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H commit 4ee1fd09cfdda8ad975e8e62aa70772bfe2a7455 Author: Weiqun Zhang Date: Mon Aug 2 08:31:51 2021 -0700 Add MFPCInterp for piece-wise constant interpolation (#2200) Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_MFInterpolater.cpp commit fa89d68a8909c9630d0a9034ca27fda50137119c Author: Weiqun Zhang Date: Sun Aug 1 15:22:12 2021 -0700 Fix Coarse/Fine Boundary BoxArray generated in FPInfo for FillPatch (#2207) The issue was degenerate boxes could appear when the coarse/fine boxarray was split into smaller chunks. This is usually not an issue except for a small extra cost. But in IAMR this could result in external Dirichlet pressure boundary function that has not been implemented being called. Src/Base/AMReX_FabArrayBase.cpp commit 7034f81724850188d104e34c787c302d3d372c44 Author: Weiqun Zhang Date: Sun Aug 1 14:42:40 2021 -0700 Update CHANGES for 21.08 (#2206) CHANGES commit b87ec0a8fc084dbe886973e1e3c8a4eedf6b7dfa Author: kngott Date: Thu Jul 29 09:29:22 2021 -0700 Fix bulleted list in IO Docs. (#2198) Docs/sphinx_documentation/source/IO.rst commit fbdf1d043ed572d8388eb445a7e091d0a11acde9 Author: kngott Date: Thu Jul 29 08:54:02 2021 -0700 Add AsyncVector. (#2197) Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuContainers.H Tests/GPU/Vector/main.cpp commit 0124f3b425adbe00e8ec021397d654eb411df9b0 Author: Jean M. Sexton Date: Thu Jul 29 08:08:47 2021 -0700 Make CreateVirtualParticles use GPU functions for aggregation_type=None (#2195) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_TracerParticle_mod_K.H commit 4387cf22f3a5417b17b2445b22fd3659fa0f1462 Author: Weiqun Zhang Date: Wed Jul 28 18:09:40 2021 -0700 Fix divide by zero due to roundoff errors (#2199) Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H commit 5619ef246175454d1777cd5dc4cd0013ad2966dd Author: asalmgren Date: Wed Jul 28 13:42:55 2021 -0700 Change mask for ctr to ctroid interp (#2196) Src/EB/AMReX_EBMultiFabUtil.cpp commit 282878dfbf1c823a90f3265c318e15185676710a Author: Andrew Myers Date: Wed Jul 28 12:14:45 2021 -0700 Add OpenMP support to DenseBins::build (#2193) Src/Particle/AMReX_DenseBins.H Tests/Particles/DenseBins/CMakeLists.txt Tests/Particles/DenseBins/GNUmakefile Tests/Particles/DenseBins/Make.package Tests/Particles/DenseBins/inputs Tests/Particles/DenseBins/main.cpp commit 7ac29e889e4c0949eba6dffcc1e03ce27dc2751c Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Tue Jul 27 17:15:31 2021 -0600 Integration of experimental parfor into NodeLaplacian Fapply. (#2190) Co-authored-by: Paul Mullowney Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 5bdae60f411b47b2360c755ad21b7cc550a547d8 Author: kngott Date: Tue Jul 27 16:08:41 2021 -0700 Add some missing FabArray function timers. (#2194) Src/Base/AMReX_FabArray.H commit 35edeb12e000a6dd75765c65dcb6dd93b3cd340a Author: Andrew Myers Date: Tue Jul 27 14:12:08 2021 -0700 Fix filter functions for callables that return bool. (#2192) Src/Particle/AMReX_ParticleTransformation.H Tests/Particles/ParticleTransformations/main.cpp commit 16cdf1504d9e00b906053431e8037c9f6d5b3f9c Author: Andrew Myers Date: Tue Jul 27 14:11:56 2021 -0700 Reduce code duplication in DenseBins::build (#2191) Src/Particle/AMReX_DenseBins.H commit bdebb1055d3cc84d95e379833821572a579bb838 Author: Weiqun Zhang Date: Mon Jul 26 17:58:04 2021 -0700 MFCellConsLinInterp (#2189) Add MFCellConsLinInterp that is similar to CellConsLinInterp. The difference is that MFCellConsLinInterp works on MultiFab, whereas CellConsLinInterp works on FArrayBox. The new interpolater launches fewer kernels, therefore has better performance on GPU when there are many boxes. Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_InterpBase.H Src/AmrCore/AMReX_InterpBase.cpp Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterp_C.H Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_MFInterpolater.cpp Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_MFParallelForG.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp commit 3181f76910e6ac28ec5e099cba3e192bfdbf9076 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Sun Jul 25 19:53:42 2021 -0400 Introduction of the experimental parfor into NodeLaplacian interpolation (#2183) All algorithms fully implemented including interpadd_aa, semi_interpadd_aa, interp_c, interp_rap, and interpadd_ha. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit f35a59b079878a43e64418f36be356227e7fc1b9 Author: Andrew Myers Date: Sun Jul 25 15:55:32 2021 -0700 Enable default construction for AmrParticleContainer and add isDefined() method. (#2188) A Particle container is considered "defined" if it has been given a set of grids, otherwise not. Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_ParticleContainerBase.H commit 45711e08a54d42ced219ffe00f6e8b435ebc7bf3 Author: jmsexton03 Date: Fri Jul 23 17:04:41 2021 -0700 Make CreateGhostParticles use GPU functions (#2185) Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTransformation.H Tests/Particles/GhostsAndVirtuals/CMakeLists.txt Tests/Particles/GhostsAndVirtuals/fixed_grids.init Tests/Particles/GhostsAndVirtuals/inputs Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/GhostsAndVirtuals/particle_file.init commit 27069ef1d885a35401f963ead2e3bad19cc9dc8f Author: Axel Huebl Date: Thu Jul 22 08:47:39 2021 -0700 CI: HIP Cleanup (#2186) Move the PATH setup for the AMD `clang`/`clang++` executables to a common location for the setup scripts. As a reminder, this is needed so the system clang is not taken by accident. .github/workflows/dependencies/dependencies_hip.sh .github/workflows/hip.yml commit 91fa2b7e6ff33117bd537635c8c55192359863a2 Author: Weiqun Zhang Date: Wed Jul 21 12:12:52 2021 -0700 Add ROCMClang as an LLVM compiler to cmake (#2184) * Add ROCMClang as an LLVM compiler to cmake Since CMake >= 3.21, ROCm is identified as ROCMClang. * HIP: Fix legacy CI / hipcc Until we get ROCM 4.4, we need to work around this new detection: https://gitlab.kitware.com/cmake/cmake/-/blob/v3.21.0/Modules/CMakeDetermineCompilerId.cmake#L153-159 Co-authored-by: Axel Huebl .github/workflows/hip.yml Tools/CMake/AMReXGenerateConfigHeader.cmake commit 948bd72d3e2ee4ee7ee6e1861a08f559ac4d1ffc Author: asalmgren Date: Tue Jul 20 14:53:41 2021 -0700 Fix typo (#2181) * Fix typo : The_Async_Async --> The_Async_Arena Docs/sphinx_documentation/source/GPU.rst commit fe9ae3e2a978de929686ad070c713baaf634ec1c Author: jmsexton03 Date: Tue Jul 20 12:07:50 2021 -0700 Add bounds-checking in InitFromAscii for CI null-dereference error (#2180) Src/Particle/AMReX_ParticleInit.H Tests/Particles/InitFromAscii/main.cpp commit a449f384df5ddece0d4e063fe9001e280ed64024 Author: jmsexton03 Date: Tue Jul 20 09:57:32 2021 -0700 Scale uniform random interval to account for gridbox length (#2179) Src/Particle/AMReX_ParticleInit.H commit 48d84eb342b04aae7e18217980a4514e5a9de3ea Author: Weiqun Zhang Date: Tue Jul 20 08:44:32 2021 -0700 Fix and update multifab ParallelFor (#2178) - Fix a bug in caching multifab ParallelFor meta-data. - Add new versions of multifab ParallelFor that are templated on max threads per block. Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MFParallelForG.H commit 779835b7d122f1c94be7c85486e951e8766730ec Author: Weiqun Zhang Date: Mon Jul 19 10:41:51 2021 -0700 Optimization of ParallelFor for single box FabArray (#2176) When there is only one Fab, we can simplify the MF ParallelFor funciton. Src/Base/AMReX_MFParallelForG.H commit ee8facf7e09d0b40af1fe680665660c24a19d32a Author: asalmgren Date: Mon Jul 19 09:11:47 2021 -0700 Add ns to gpu cns (#2177) Add the ability to compute and use diffusive fluxes to Tests/GPU/CNS. Default is now to include diffusive terms so the existing inputs files now have cns.do_visc=false. Tests/GPU/CNS/CMakeLists.txt Tests/GPU/CNS/Exec/RT/inputs Tests/GPU/CNS/Exec/RT/inputs-rt Tests/GPU/CNS/Exec/Sod/inputs Tests/GPU/CNS/Exec/Sod/inputs-rt Tests/GPU/CNS/Source/CNS.H Tests/GPU/CNS/Source/CNS.cpp Tests/GPU/CNS/Source/CNS_advance.cpp Tests/GPU/CNS/Source/CNS_index_macros.H Tests/GPU/CNS/Source/CNS_parm.H Tests/GPU/CNS/Source/diffusion/CNS_diffusion_K.H Tests/GPU/CNS/Source/diffusion/Make.package Tests/GPU/CNS/Source/hydro/CNS_hydro_K.H commit 388ce7e96d74eef0f082f19d95e16647a2d34ccb Author: Axel Huebl Date: Mon Jul 19 08:44:23 2021 -0700 CI: IntelLLVM known to CMake (#2136) ## Summary `IntelLLVM` is now a recognized CMake compiler, so we don't need to add a Clang-ish identification anymore. Also fix a CMake -Wdev warning (DPC++/SYCL) - beta08 work-around: `-mlong-double-64` (https://github.com/intel/llvm/issues/2187) -> still needed! - beta09 work-around: `-fno-sycl-early-optimizations` (link to upstream issue?) -> keep for now, RT tests pending to verify fix ## Additional background setting the CXX compiler ID was a work-around for the 2021.1 DPC++ release / CMake 3.19.0-3.19.1 https://gitlab.kitware.com/cmake/cmake/-/issues/21551#note_869580 .github/workflows/intel.yml Tools/CMake/AMReXSYCL.cmake commit 98c9e7197c5c94135fd9bb070aeea12b8fb58629 Author: Andrew Myers Date: Fri Jul 16 12:21:41 2021 -0700 Add default move constructors / assignment operators for other pc types. (#2174) Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_TracerParticles.H commit 6e27136f76280e565d30ff654031d1b1ac683171 Author: Weiqun Zhang Date: Thu Jul 15 16:58:58 2021 -0700 Fix warning with Intel icpc (#2173) * Fix warning with Intel icpc The classical EDG based Intel compiler gives warning about `fallthrough` when `-std=c++14` is used, even though it passes the feature test for `fallthrough`. * CI: Add Intel ICC/ICPC/IFORT Co-authored-by: Axel Huebl .github/workflows/intel.yml Src/Base/AMReX_Extension.H commit 6a49159efff4f7d742b604ffbe61f39ab4fa1a0a Author: Weiqun Zhang Date: Thu Jul 15 14:15:47 2021 -0700 Gradient in WarpX EB nodal solver (#2171) Implement function that computes the gradient of potential in the EB nodal solver for WarpX. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 1be4862218d034349ae6f24d948ad39abeac1a2c Author: Candace Gilet Date: Wed Jul 14 22:12:18 2021 -0400 Update MacProjector::project to check if m_umac is nullptr (#2166) * Remove unnecessary temporary and copy. * Update MacProjector::project to check if m_umac is nullptr. If nullptr, don't compute div(umac) or attempt to average down or update umac. * Remove tabs Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit 8a2ddee47fecb368349db09a4505fe753ddeb65d Author: Weiqun Zhang Date: Wed Jul 14 16:56:47 2021 -0700 EB nodal solver (#2145) New EB nodal solver with Dirichlet EB. This uses a finite-difference stencil. It will be used by WarpX's static solver. Currently only Dirichlet and periodic domain BC are supported. Support for Neumann BC at the domain boundary will be added later. Currently, it supports single level only. Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EBDataCollection.cpp Src/EB/AMReX_EBFabFactory.H Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/Make.package commit cc1ceb1188537b0d126be323537f8aaa4294983c Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed Jul 14 15:57:39 2021 -0700 Fix missing MPI flags in Perlmutter. (#2170) Tools/GNUMake/sites/Make.nersc commit 24a586d89da996195ef81aff2a47c7d2bf0825c7 Author: Andrew Myers Date: Wed Jul 14 14:26:25 2021 -0700 Add ssh instructions to CONTRIBUTING.md (#2168) CONTRIBUTING.md commit 6ab65e7fc0ace3cd116f2b7e2c51a3506e7d4ff1 Author: Weiqun Zhang Date: Tue Jul 13 20:46:31 2021 -0700 More reliable workaround for DPCPP reduction issue (#2167) It seems that the DPC++ reduction issue is due to writing to pinned memory on device. The workaround is to write to device memory and then memcpy. Src/Base/AMReX_Reduce.H commit d9eeb62638eb3352fb4a7eaf6ff93b9f0c668e04 Author: Weiqun Zhang Date: Mon Jul 12 17:32:14 2021 -0700 Workaround for a DPC++ reduction issue (#2165) Src/Base/AMReX_Reduce.H commit cd1f5430be29aed0a85348669394d7fbb8355dba Author: Weiqun Zhang Date: Mon Jul 12 08:45:53 2021 -0700 add ParmParse::remove (#2160) Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit 0ba4a8f628d240642431b537fd62e809ddf6aa41 Author: Weiqun Zhang Date: Mon Jul 12 08:45:18 2021 -0700 Minor optimization of reducing 4 bytes (#2161) Box length is used in many kernels. Because the length in z-direction is not needed, we only need 2 integers, not Dim3 that has 3 integers. Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_BaseFabUtility.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_Reduce.H commit 135010c8baab688b9a190f008e5098a50a91edb2 Author: Weiqun Zhang Date: Mon Jul 12 08:44:52 2021 -0700 Remove MFGhostIter (#2162) It was added several years in an experiment of overlapping communication and computation. It has never been used. Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit 30401db6fc699259b388c09ef5a5cbcc40f04639 Author: Weiqun Zhang Date: Mon Jul 12 08:44:24 2021 -0700 Fix typos with the help of codespell (#2163) .github/workflows/docker/sensei/Dockerfile CHANGES Docs/Notes/Readme.typecheck Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Docs/sphinx_documentation/source/GridCreation.rst Docs/sphinx_documentation/source/SWFFT.rst Docs/sphinx_documentation/source/Testing.rst Docs/sphinx_documentation/source/Visualization.rst Src/Amr/AMReX_Amr.H Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateDescriptor.H Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_TagBox.H Src/AmrCore/CMakeLists.txt Src/Base/AMReX_Arena.H Src/Base/AMReX_AsyncOut.H Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BaseUmap_nd.f90 Src/Base/AMReX_Box.H Src/Base/AMReX_BoxIterator.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_DArena.cpp Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FILCC_3D.F90 Src/Base/AMReX_FPC.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabConv.cpp Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_IntVect.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_Partition.H Src/Base/AMReX_Utility.cpp Src/Base/AMReX_Vector.H Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_parstream.cpp Src/Base/Parser/AMReX_IParser_Y.H Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Boundary/AMReX_BndryData.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_EB_slopes_K.H Src/EB/AMReX_EB_utils.cpp Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp Src/Extern/Conduit/AMReX_Conduit_Blueprint_ParticlesI.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/Extern/ProfParser/AMReX_BLWritePlotFile.cpp Src/Extern/ProfParser/AMReX_CommProfStats.cpp Src/Extern/SENSEI/AMReX_AmrDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrParticleInSituBridge.H Src/Extern/SENSEI/AMReX_InSituBridge.H Src/Extern/SENSEI/AMReX_InSituUtils.H Src/Extern/SENSEI/AMReX_ParticleInSituBridge.H Src/Extern/SWFFT/CheckDecomposition.c Src/Extern/SWFFT/Dfft.H Src/Extern/SWFFT/README Src/Extern/SWFFT/distribution.c Src/Extern/amrdata/AMReX_DataServices.cpp Src/F_Interfaces/Base/AMReX_mpi_reduce_int.F90 Src/F_Interfaces/Base/AMReX_mpi_reduce_real.F90 Src/LinearSolvers/MLMG/AMReX_MLALaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_ParticleArray.H Src/Particle/AMReX_Particles.H Src/SDC/AMReX_SDCstruct.cpp Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Amr/Advection_AmrLevel/Source/Src_2d/Adv_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Adv_3d.f90 Tests/DivFreePatch/main.cpp Tests/EB/CNS/Exec/Sod/inputs Tests/EB/CNS/Source/diffusion/cns_eb_diff_wall.F90 Tests/FortranInterface/Advection_F/Source/Src_2d/advect_2d_mod.F90 Tests/FortranInterface/Advection_F/Source/Src_3d/advect_3d_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/Src_2d/advect_2d_mod.F90 Tests/GPU/CNS/Source/CNS_bcfill.cpp Tests/GPU/Fuse/GNUmakefile Tests/LinearSolvers/ABecLaplacian_C/scalingtest/results.org Tests/LinearSolvers/ABecLaplacian_F/mytest.F90 Tests/LinearSolvers/LeastSquares/README.md Tests/LinearSolvers/MAC_Projection_EB/main.cpp Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Slopes/README.md Tools/CMake/AMReXCUDAOptions.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/CMake/AMReXTypecheck.cmake Tools/C_scripts/makebuildinfo_C.py Tools/C_scripts/mkdep Tools/C_scripts/strip72 Tools/C_util/AppendToPlotFile.cpp Tools/C_util/AugmentPlotfile/AugmentPlotfile.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/README Tools/F_scripts/extract.parallel Tools/F_scripts/f90doc/typing.pl Tools/F_scripts/write_probin.py Tools/GNUMake/Make.rules Tools/GNUMake/comps/cray.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/comps/nvhpc.mak Tools/GNUMake/comps/pgi.mak Tools/GNUMake/sites/Make.nersc Tools/GNUMake/sites/Make.nrel Tools/Migration/step-7-bindc/bindc.sh Tools/Postprocessing/python/dumpparthistory.py Tools/Release/ppCleanup.py Tools/Release/ppCleanupDir.py Tools/Release/release.py Tools/libamrex/mkpkgconfig.py commit bef02a4c6394d24578ceed82ab9d73ede5a637e2 Author: Weiqun Zhang Date: Sat Jul 10 15:09:36 2021 -0700 Parser Precision (#2154) Use double internally. If the arguments are all floats, the final result will be converted to float. This is necessary to avoid overflow in some WarpX single precision tests. Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Exe.H Src/Base/Parser/AMReX_Parser_Exe.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_parser.tab.h Src/Base/Parser/amrex_parser.y commit f8c2fe41d3187e69f747816f1f6ab0d38335c11e Author: asalmgren Date: Sat Jul 10 14:11:49 2021 -0700 Add to particlemeshtest (#2159) Tests/Particles/ParticleMeshMultiLevel/CMakeLists.txt Tests/Particles/ParticleMeshMultiLevel/Make.package Tests/Particles/ParticleMeshMultiLevel/main.cpp Tests/Particles/ParticleMeshMultiLevel/mypc.H Tests/Particles/ParticleMeshMultiLevel/trilinear_deposition_K.H commit d325cf8eab0d78bb356d5bcf6c90b9bef7ef63ae Author: Weiqun Zhang Date: Sat Jul 10 14:09:06 2021 -0700 IParser: Integer Parser (#2158) Add a parser for integers. Because there is rounding in integer division, it's not safe to use the real number parser for integer. The two parsers have a lot of similarity, but floating point number specific functions (e.g., `sqrt`, `sin`, etc.) are not supported in `IParser`. In addition to `/` whose result truncates towards zero, the integer parser also supports `//` whose result truncates towards negative infinity. Docs/sphinx_documentation/source/Basics.rst Src/Base/CMakeLists.txt Src/Base/Make.package Src/Base/Parser/AMReX_IParser.H Src/Base/Parser/AMReX_IParser.cpp Src/Base/Parser/AMReX_IParser_Exe.H Src/Base/Parser/AMReX_IParser_Exe.cpp Src/Base/Parser/AMReX_IParser_Y.H Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/GNUmakefile Src/Base/Parser/amrex_iparser.l Src/Base/Parser/amrex_iparser.lex.cpp Src/Base/Parser/amrex_iparser.lex.h Src/Base/Parser/amrex_iparser.tab.cpp Src/Base/Parser/amrex_iparser.tab.h Src/Base/Parser/amrex_iparser.y Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp Src/Base/Parser/amrex_parser.y Tests/Parser/main.cpp commit fb41ccbb852a89026cda4b1d26eee6f1fab56f9e Author: asalmgren Date: Fri Jul 9 14:26:29 2021 -0700 Remove single level assertions for NeighborParticleContainers on GPUs (#2157) * Remove single-level assertions for NeighborParticleContainer on GPUs Src/Particle/AMReX_NeighborParticlesGPUImpl.H commit cebf3df6c3e068a0c4f69b04198c0ba9af94d50a Author: Candace Gilet Date: Fri Jul 9 15:08:59 2021 -0400 Docs: fix typo in author name (#2156) Docs/sphinx_documentation/source/GridCreation.rst commit b0ffd259a5bf7eba82fc47a3871ae6b61c8860a5 Author: Erik Date: Thu Jul 8 15:57:11 2021 -0400 correct sp typo (#2153) Docs/sphinx_documentation/source/GettingStarted.rst commit 91ea9a0243b48a5d66c0c09e18df16e2db60e27f Author: Erik Schnetter Date: Thu Jul 8 13:44:41 2021 -0400 Disable `jn` on Windows (#2152) Disable `jn` on Windows – not just when `dpcpp` is used, but also when building with other compilers in MINGW. Src/Base/Parser/AMReX_Parser_Y.H commit 6b2ef0b507a24d2d90a04c81895c33aaf315454a Author: Mu-Hua Chien Date: Thu Jul 8 11:24:46 2021 -0400 add an option for hypre solver for Tests/Linearsolvers/NodalPoisson (#2139) Add input-rt.hypre for NodalPoisson example Tests/LinearSolvers/NodalPoisson/GNUmakefile Tests/LinearSolvers/NodalPoisson/MyTest.H Tests/LinearSolvers/NodalPoisson/MyTest.cpp Tests/LinearSolvers/NodalPoisson/inputs-rt.hypre commit 49b6ad4a85f09f86e7601a07744ed6e63f29a164 Author: Erik Date: Thu Jul 8 11:20:46 2021 -0400 Doxygen Side-Bar Navigation, Doxygen Logo and Mainpage Content (#2150) Enables side-bar navigation for Doxygen, and adds logo and content to main splash page. Docs/Doxygen/customdoxygen.css Docs/Doxygen/doxygen.conf Docs/Doxygen/footer.html Docs/Doxygen/header.html Docs/Doxygen/main.dox Docs/Doxygen/resize_AMReX_logo.png commit e7591aabfe4578ddf9339688632700a70ac17af0 Author: Weiqun Zhang Date: Wed Jul 7 20:38:29 2021 -0700 Allow empty string Parser (#2151) Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp commit 1d6b01af423bf0f39d9efba8ac59ebff75179de0 Author: Erik Date: Wed Jul 7 12:13:38 2021 -0400 Doxygen comment formatting (#2148) Src/Particle/AMReX_Particles.H commit fffd3d976c7e8a3fa1e9a59e35e7cbfbd37a2493 Author: Andrew Myers Date: Tue Jul 6 21:53:51 2021 -0700 Fix the size of the BC Vector in the multi-level version of ParticleToMesh. (#2149) Src/AmrCore/AMReX_AmrParticles.H commit 7c185502a57fb263f0819635c14f7ee84e96f0b7 Author: Andrew Myers Date: Tue Jul 6 10:56:01 2021 -0700 Make ParticleToMesh and MeshToParticle support iMultifab. (#2137) Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_iMultiFab.cpp Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleMesh.H Tests/Particles/ParticleMesh/main.cpp commit 901bac18bdc3635b992121ba1023b0b9d14f8729 Author: Nicholas Brunhart-Lupo Date: Fri Jul 2 15:42:16 2021 -0600 Fix crashes with Conduit Blueprint Code (#2144) ## Summary This PR contributes two fixes to the Conduit Blueprint code. ### Nestsets fix (Issue AMReX-Codes/amrex#2135) The first fix here is within `Nestsets`, which resolves a one-past the end access on `ref_ratio`. The problematic lines are 116 to 120, of the form `window["ratio/i"] = ref_ratio[level+1][0];` For some inputs, this is one-past the end of the `ref_ratio` array. It was advised in https://github.com/AMReX-Codes/amrex/issues/2135#issuecomment-872692909 to resolve this by removing the `+1`. ### MultiLevelToBlueprint fix (Issue AMReX-Codes/amrex#2133) The second fix resolves a bug inside `MultiLevelToBlueprint`, where the `box_offsets` array is not built correctly. The problematic code is here: https://github.com/AMReX-Codes/amrex/blob/14dde9aaf741f84cf85adb67a7cdc5bef4c0764a/Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp#L363-L376 During the construction of the `box_offsets` array the first round of the loop at line 364, with `i == 0` will size the `box_offsets` array to 1. Subsequent rounds through the loop will attempt to write to non-existent slots of the vector here: https://github.com/AMReX-Codes/amrex/blob/14dde9aaf741f84cf85adb67a7cdc5bef4c0764a/Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp#L374 This fix sizes `box_offsets` before the loop starts, and changes all accesses to the vector by index, not by appending. This appears to be the right fix as advised here: https://github.com/AMReX-Codes/amrex/issues/2133#issuecomment-871501258 Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 85021c840b13c0f89c0db7039cda66585e9b239a Author: kngott <30483578+kngott@users.noreply.github.com> Date: Fri Jul 2 08:30:43 2021 -0700 Perlmutter Build: Adjust to allow CPU build + cleanup. (#2143) Tested on PrgEnv-nvidia, PrgEnv-gnu for USE_CUDA={TRUE,FALSE} and BL_NO_FORT={TRUE,FALSE}. Tools/GNUMake/sites/Make.nersc commit ab69825ddee58a2ca834a2437376a05c0b8a76af Author: Weiqun Zhang Date: Thu Jul 1 17:06:29 2021 -0700 Fix mismatched tag warning (#2141) AmrInfo is defined as a struct, and it was declared as class in forward declaration. This is valid C++. But the Intel oneAPI compiler gives a warning that this may result in linker errors under the Microsoft C++ ABI. Src/AmrCore/AMReX_AmrCoreFwd.H commit 83087f64af364dd7e80a191fc905a890cf1f635e Author: Weiqun Zhang Date: Thu Jul 1 17:05:57 2021 -0700 Default values for members of ParserExecutor (#2142) Set the default values of pointers inside ParserExecutor to nullptr and add a conversion operator so that it can be tested for null. Src/Base/Parser/AMReX_Parser.H commit d7360e84bb68505316645c346abd1bbe3662d7ce Author: Candace Gilet Date: Thu Jul 1 13:37:23 2021 -0400 Docs: Add labels to chapters that were missing one. (#2134) Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/GettingStarted_Chapter.rst Docs/sphinx_documentation/source/ManagingGridHierarchy_Chapter.rst commit bee2568d831aa1c84e6a9fe24c3e69f3ec5b49a0 Author: Axel Huebl Date: Thu Jul 1 10:36:08 2021 -0700 CMake: HIP Modernizing & RDC (#2031) ## Summary Add `-fgpu-rdc` flags to HIP if requested via `AMReX_GPU_RDC`. Modernize HIP logic with recommended targets that avoid the flaky `hipcc` compiler scripts: https://rocmdocs.amd.com/en/latest/Installation_Guide/Using-CMake-with-AMD-ROCm.html#using-hip-in-cmake Add support for AMDs `clang++`/`clang` compiler for HIP instead of using the legacy `hipcc` perl wrapper as C++ compiler. This also increases support towards Cray compiler wrappers, which also refer to `clang++`/`clang` underneath (Spock/OLCF). Close #1688 ## Additional background Relocatable-device-code (RDC) flags are needed for `extern` device variable support (for codes that use global variables on device). Also needed when linking with Ascent. Follow-up to #2029 With HIP GPU RDC, static libs emitting & linking does get more fancy: https://github.com/ROCmSoftwarePlatform/rccl/pull/260 .github/workflows/hip.yml Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXParallelBackends.cmake commit 5c83ba4d6f5f9c4db9812bbbf0b032f94aac7205 Author: Weiqun Zhang Date: Thu Jul 1 08:10:58 2021 -0700 Update CHANGES for 21.07 (#2138) CHANGES commit 14dde9aaf741f84cf85adb67a7cdc5bef4c0764a Author: Weiqun Zhang Date: Mon Jun 28 20:29:50 2021 -0700 Parser::compileHost (#2132) Add a function that compiles the parser expression into host code only. Src/Base/Parser/AMReX_Parser.H commit 97d907762f776ff4400868029fcc53709703291c Author: Andrew Myers Date: Mon Jun 28 14:03:43 2021 -0700 Fix MR bug in NL construction - the number of cells needs to be increased by a factor of ref_fac. (#2131) Src/Particle/AMReX_NeighborParticlesI.H commit fd1fd11302601a085b2e7ff01f07d36ffa0add66 Author: Weiqun Zhang Date: Mon Jun 28 10:11:55 2021 -0700 EB Implicit Function using Parser (#2129) Add ParserIF to EB. One can use a parser function to generate geometry. For example, the following ParmParse parameters can be used in 3D to place two cylinders with a radius of 0.1 at (-0.5,0,) and (0.5,0,). ``` eb2.geom_type = parser eb2.parser_function = "max(0.01-(x+0.5)^2-y^2, 0.01-(x-0.5)^2-y^2)" ``` Put the internal data of Parser in a shared_ptr for memory safety. Docs/sphinx_documentation/source/Basics.rst Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_IF.H Src/EB/AMReX_EB2_IF_Parser.H Src/EB/CMakeLists.txt Src/EB/Make.package commit 55cb40092910183729ca81e6eef3d0ce386af317 Author: Weiqun Zhang Date: Sun Jun 27 10:52:09 2021 -0700 AMReX Version (#2128) In git fails to obtain AMReX version in the make system, grep for the version number in the CHANGES file. CHANGES Tools/GNUMake/Make.defs commit 6ca4306167ed432399a41aaa7a4a80a3f15f65e9 Author: Candace Gilet Date: Fri Jun 25 21:55:09 2021 -0400 Fix typo in 3d NodeLaplacian. (#2126) This fixes an error for cases with for dz != dy Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H commit 81535291686e88a4d2cc3bf36a77bca6a6bf48c3 Author: Weiqun Zhang Date: Fri Jun 25 14:48:53 2021 -0700 Parser (#2121) Add amrex::Parser that can be used at runtime to evaluate math expressions given by strings. It works on both host and device. This is moved from WarpX with additional performance optimization of eliminating recursion. Support for local automatic variables are added. .github/workflows/linux.yml .github/workflows/style/check_tabs.sh .github/workflows/style/check_trailing_whitespaces.sh Docs/sphinx_documentation/source/Basics.rst Src/Base/AMReX_BaseFwd.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Exe.H Src/Base/Parser/AMReX_Parser_Exe.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/GNUmakefile Src/Base/Parser/README Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp Src/Base/Parser/amrex_parser.lex.h Src/Base/Parser/amrex_parser.tab.cpp Src/Base/Parser/amrex_parser.tab.h Src/Base/Parser/amrex_parser.y Tests/CMakeLists.txt Tests/Parser/CMakeLists.txt Tests/Parser/GNUmakefile Tests/Parser/Make.package Tests/Parser/main.cpp commit 67b0794d02c2f13e2c0156b752c486da6863deea Author: Andrew Myers Date: Thu Jun 24 17:52:45 2021 -0700 Remove incorrect call to coarsen. (#2125) Src/Particle/AMReX_NeighborParticlesI.H commit 8575439bdbc52bda3eb38f12660a5d3c1e0ae584 Author: Weiqun Zhang Date: Wed Jun 23 22:59:37 2021 -0700 Avoid std inclusive and exclusive scan for _GLIBCXX_RELEASE < 10 (#2123) Src/Base/AMReX_Scan.H commit 5db3b4fff09400a7c040b6792e06b2d9051fd01a Author: Andrew Myers Date: Tue Jun 22 12:37:48 2021 -0700 Fix Arena docs. (#2118) * Fix Arena docs. * remove device arena from printstats output in docs Docs/sphinx_documentation/source/GPU.rst commit 91a4ff2ce1f0a950ee7092697ea48f9e814391d3 Author: c-wetterer-nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Tue Jun 22 14:28:39 2021 -0400 Add SENSEI Adaptor for Particles Containers (#2016) Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Extern/SENSEI/AMReX_AmrDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrInSituBridge.H Src/Extern/SENSEI/AMReX_AmrInSituBridge.cpp Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrMeshInSituBridge.H Src/Extern/SENSEI/AMReX_AmrMeshInSituBridge.cpp Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_AmrParticleInSituBridge.H Src/Extern/SENSEI/AMReX_InSituBridge.H Src/Extern/SENSEI/AMReX_InSituBridge.cpp Src/Extern/SENSEI/AMReX_InSituUtils.H Src/Extern/SENSEI/AMReX_InSituUtils.cpp Src/Extern/SENSEI/AMReX_ParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_ParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_ParticleInSituBridge.H Src/Extern/SENSEI/CMakeLists.txt Src/Extern/SENSEI/Make.package Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs Tools/GNUMake/tools/Make.sensei Tools/libamrex/configure.py commit c797f6b6979aea63cc6bda574765c29ba89e7b9c Author: kwryankrattiger <80296582+kwryankrattiger@users.noreply.github.com> Date: Thu Jun 17 15:45:09 2021 -0500 CI: Add job to test SENSEI build (#2115) .github/workflows/docker/sensei/Dockerfile .github/workflows/docker/sensei/install_deps.sh .github/workflows/docker/sensei/install_sensei.sh .github/workflows/docker/sensei/install_vtk_minimal.sh .github/workflows/docker/sensei/tools.sh .github/workflows/linux.yml commit 4eb054cf1d37f3e2fa79f01a17116072ccfbbcb8 Author: Weiqun Zhang Date: Thu Jun 17 10:22:09 2021 -0700 Fix warning in algoim (#2117) Split the constructor of ImplicitIntegral into two templates to avoid an unreachable loop warning. Src/EB/AMReX_algoim_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Tools/GNUMake/Make.defs commit 04464a4f6d1201dfd01fd248c3ad64c5a6809a74 Author: Andy Nonaka Date: Thu Jun 17 09:46:31 2021 -0700 Enhance DiffSameGrid2.cpp (#2114) This utility, when using L0 norm, now outputs an additional column with the absolute error divided by the maximum value on the coarse grid. This is useful for comparing/verifying data with local regions of very small (close to zero in a machine precision sense) values that may fluctuate based on roundoff errors. Tools/C_util/Convergence/DiffSameGrid2.cpp commit 501626901f500b45fce5102c01d15cd8f74c7187 Author: Andrew Myers Date: Wed Jun 16 15:49:57 2021 -0700 Don't rely on managed memory in ParticleCopyPlan (#2116) Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleCommunication.cpp commit baf0d4cd96fa81ace1c964a2e5601685e97d41f6 Author: Axel Huebl Date: Tue Jun 15 16:10:11 2021 -0700 HIP: Increase to C++17 (#2113) Bump HIP requirements to C++17 or newer (like DPC++). At the moment, we only test against AMD's clang++/hipcc, which supports C++17 anyway. .github/workflows/hip.yml Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReX_Config.cmake Tools/GNUMake/comps/hip.mak commit f7ae247580204b1f607e22b09c046120d2cc6e38 Author: Weiqun Zhang Date: Tue Jun 15 14:26:44 2021 -0700 Port remaining Fortran and CPU kernels in FluxRegister to GPU (#2112) Src/AmrCore/AMReX_FLUXREG_F.H Src/AmrCore/AMReX_FLUXREG_nd.F90 Src/AmrCore/AMReX_FluxRegister.H Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package Src/Base/AMReX_FBI.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H commit 659ab43420a2c4a918cda2702910771fbbc395d3 Author: Weiqun Zhang Date: Mon Jun 14 18:01:42 2021 -0700 TinyProfiler: Add spaces to print (#2110) Add more spaces so that the output looks nicer when there are multiple layers of profilers. Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit 8182a3a630e580ed9158bd1da33276380f170d41 Author: Andrew Myers Date: Mon Jun 14 14:33:08 2021 -0700 Reimplement partitionParticlesByDest (#2109) Src/Particle/AMReX_ParticleUtil.H commit d7ff203305e8c8605ca6f92e9212acf717da56de Author: Weiqun Zhang Date: Mon Jun 14 12:10:05 2021 -0700 GpuArray template parameter (#2108) Use unsigned int as the type for size instead of std::size_t so that the forward declaration does not need to include another header. Src/Base/AMReX_Array.H Src/Base/AMReX_BaseFwd.H commit c221a567dbd8cadd474146d9cc36ba5152178fb1 Author: Weiqun Zhang Date: Mon Jun 14 10:38:00 2021 -0700 Fix a bug in in-place ExclusiveSum (#2105) For ExclusiveSum versions using memcpy to get the total sum, we need to copy the last element of the input data to the host before the scan so that it's safe for in-place scan. Also use std::exclusive_scan and inclusive_scan on CPU if available. Src/Base/AMReX_Scan.H commit 8357d530311235e3f76899051ca8d44a51b971c0 Author: Luca Fedeli Date: Mon Jun 14 17:18:20 2021 +0200 add missing class (#2106) Src/Base/AMReX_BaseFwd.H commit fc9bc1e3efdaead0253d0a55dcf1abb7abc06bd7 Author: Weiqun Zhang Date: Sun Jun 13 12:42:38 2021 -0700 Add more warnings to CI and fix warnings (#2104) Also fixed a bug in device version of `Random_int`. .github/workflows/cuda.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml .github/workflows/macos.yml Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_Random.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/MultiBlock/Advection/main.cpp Tests/MultiBlock/IndexType/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleArray/main.cpp Tests/Particles/ParticleIterator/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/SparseBins/main.cpp Tests/Particles/TypeDescriptor/main.cpp Tools/CMake/AMReXParallelBackends.cmake Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/hip.mak Tools/Plotfile/fextract.cpp commit 2ff4519b6e16c1f0906e637559f62d183158734c Author: Weiqun Zhang Date: Sat Jun 12 08:07:04 2021 -0700 Missing defined in preprocessing (#2102) Src/Base/AMReX_FPC.cpp commit 2a676f8a25e6cd75ed2e3936bd0cd2e30141363f Author: Michael Zingale Date: Fri Jun 11 20:42:24 2021 -0400 add AUTO_BUILD_SOURCES to the dependencies for GNU make (#2101) AUTO_BUILD_SOURCES can be filled by an application code with the source files that are created automatically at build time to ensure they are made before compilation / dependency resolution. Tools/GNUMake/Make.rules commit 64e41a8d314fc28f6d6473095ac5bdcd256c445f Author: Weiqun Zhang Date: Fri Jun 11 17:20:08 2021 -0700 Add a number of forward declaration headers (#2100) Src/Amr/AMReX_AmrFwd.H Src/Amr/CMakeLists.txt Src/Amr/Make.package Src/AmrCore/AMReX_AmrCoreFwd.H Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package Src/Base/AMReX_Array4.H Src/Base/AMReX_BaseFwd.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/Boundary/AMReX_BoundaryFwd.H Src/Boundary/CMakeLists.txt Src/Boundary/Make.package commit 4cf09ce1a3eb2c06460866af04b0ec420300f1d0 Author: Axel Huebl Date: Wed Jun 9 20:01:14 2021 -0700 Particle Device Members: Inline (#2097) Member functions that are not `operator()` are not inlined by default. I think we should inline those getters, since they are too tiny / too deep in hot loops to make sense to indirect. (The compiler likely figured this out on its own already, at least I saw no immediate difference in a massive WarpX kernel with NVCC and looking at ptxas info.) Src/Particle/AMReX_Particle.H commit 7eaf1ee13726cfb689d4f0bffe3b5d056b09eceb Author: Axel Huebl Date: Wed Jun 9 15:19:21 2021 -0700 CI: Split Linux Workflows (#2096) Split workflows into - linux (CPU) - cuda - intel (DPC++/ICC/ICX) - hip workflows. Individual workflows can be individually restarted and this increases order a bit as well. .github/workflows/cuda.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml commit e6c52ec95486d8d26ec679d9aea561fa6266d844 Author: Axel Huebl Date: Wed Jun 9 09:17:03 2021 -0700 CMake 3.20+ CUDA: Fast Math (#2094) Honor CUDA fast math option `AMReX_CUDA_FASTMATH` (default: ON) in modernized CUDA CMake logic. Tools/CMake/AMReXParallelBackends.cmake commit 56de38905af1fe6b1eff8fe390de50bcc65449c2 Author: Axel Huebl Date: Mon Jun 7 19:20:17 2021 -0700 CMake 3.20+ CUDA: Shell-Escape --Werror (#2093) ## Summary When adding a list of flags to CMake targets, duplicated flags get by default deduplicated. This caused with our new CMake 3.20+ CUDA logic #2012 that this gets passed to the linker: `--Werror ext-lambda-captures-this cross-execution-space-call`. The CMake `SHELL:` prefix is exactly made for this instance, keeping flags that belong together also over transformations like this together: https://cmake.org/cmake/help/latest/command/target_compile_options.html#arguments Since we injected directly into the global `CMAKE_CUDA_FLAGS` flags for earlier CMake versions, this deduplication logic does not apply and thus we have not experienced it earlier. ## Additional background Seen first with CI in AMReX Tutorials (e.g. https://github.com/AMReX-Codes/amrex-tutorials/pull/12). Tools/CMake/AMReXParallelBackends.cmake commit 88b50cf2e427c5358bed05d792f557f5da303028 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Mon Jun 7 22:04:23 2021 -0400 Add link to "Building with CMake" page in INSTALL file (#2092) Add a link to "Building with CMake" page in the AMReX documentation web page into the INSTALL file. INSTALL commit 53614b4a5b8bd769918523af10341b22c57a5596 Author: Weiqun Zhang Date: Mon Jun 7 12:18:04 2021 -0700 Remove VecReduce and FabReduce (#2091) They were used for the old atomics based reduction approach. Only PeleC uses VecReduce. The use of VecReduce there can be replaced with Reduce::Sum. Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H commit 79678dd3b8c1ecf92c0aaf896c4da1ab7e297465 Author: Weiqun Zhang Date: Mon Jun 7 12:17:23 2021 -0700 Add more versions of ParallelFor(MF) (#2089) Src/Base/AMReX_Algorithm.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MFParallelForC.H Src/Base/AMReX_MFParallelForG.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 87e01fe44b483adab034072dc000aed67aa1b5a7 Author: Roberto Porcu <53792251+rporcu@users.noreply.github.com> Date: Fri Jun 4 17:53:15 2021 -0400 Fix indexes bug for particles output in GPU version of packIOData (#2087) Src/Particle/AMReX_WriteBinaryParticleData.H commit 9e3fb2f574453dc13e7e1d03a53f0ce41a9ad0e1 Author: Weiqun Zhang Date: Fri Jun 4 11:32:00 2021 -0700 Remove deprecated Reduce functions (#2086) It includes Min, Max, and MinMax with the signature like `T Sum (N n, U const* v, T init_val, BOP bop)`. The other versions are still available. Src/Base/AMReX_Reduce.H commit eb2fb8eb11d4df88d5b6c04a68f2971a6c3b97f0 Author: Weiqun Zhang Date: Thu Jun 3 21:31:05 2021 -0700 Fix some macro issues in CMake Fortran build (#2084) Include AMReX_Config.H in Fortran files that need preprocessing. Do not pass those definitions already defined in AMReX_Config.H to the Fortran compiler in the cmake build. This fixes the warnings about redefined macros. Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/Base/AMReX_FILCC_1D.F90 Src/Base/AMReX_FILCC_2D.F90 Src/Base/AMReX_FILCC_3D.F90 Src/Base/AMReX_acc_mod.F90 Src/Base/AMReX_filcc_mod.F90 Src/Base/AMReX_fort_mod.F90 Src/Base/AMReX_omp_mod.F90 Src/Boundary/AMReX_LO_UTIL.F90 Src/EB/AMReX_ebcellflag_mod.F90 Src/F_Interfaces/AmrCore/AMReX_fillpatch_mod.F90 Src/F_Interfaces/Base/AMReX_fab_mod.F90 Src/F_Interfaces/Base/AMReX_parallel_mod.F90 Src/F_Interfaces/Particle/AMReX_particlecontainer_mod.F90 Src/LinearSolvers/MLMG/AMReX_MLLinOp_nd.F90 Tests/FortranInterface/Advection_F/Source/compute_dt_mod.F90 Tests/FortranInterface/Advection_F/Source/evolve_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/compute_dt_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/evolve_mod.F90 Tools/CMake/AMReXSetDefines.cmake commit 5fd8671f9a9e7b319b03faae03c1fdfa371626f5 Author: Weiqun Zhang Date: Thu Jun 3 10:06:26 2021 -0700 Disable kernel fusing if gpu rdc is off. (#2083) Because the kernel fusing depends on rdc. Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs commit 5fca8d0268c461720d9665f0aef6a4d25bd11381 Author: Weiqun Zhang Date: Wed Jun 2 21:23:35 2021 -0700 ParallelFor for MultiFab/FabArray (#2073) Add an experimental feature that can be used to launch a single kernel to work on MultiFabs or FabArrays. An example is shown below, where `mf` is a MultiFab. ``` auto ma = mf.arrays(); amrex::ParallelFor(mf, [=] AMREX_GPU_DEVICE (int box_no, int i, int j, int k) { ma[box_no](i,j,k) = 1.0; }); Gpu::synchronize(); ``` It's still unclear how we handle tiling and OpenMP and how it performs on CPU. However, there is strong evidence that this improves performance for GPU runs when there are small boxes. The purpose of this PR is to get the feature into the AMReX repo first and do further tuning later. Since the new function is not used at all, it should not affect any codes. Src/Base/AMReX_CArena.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFParallelForC.H Src/Base/AMReX_MFParallelForG.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_TypeTraits.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/EB/AMReX_MultiCutFab.H commit 7404b8a26a60e30cb6be4caf991522676c4bc36c Author: Andrew Myers Date: Wed Jun 2 20:34:20 2021 -0700 ParticleContainer-level reduction functions that work on tuples of data at once. (#2082) Src/Particle/AMReX_ParticleReduce.H Tests/Particles/ParticleReduce/main.cpp commit ea9345194adaf8a202a1b3a48d438697d008bdad Author: Axel Huebl Date: Wed Jun 2 20:15:29 2021 -0700 AMReX_PROBINT: Depends on AMReX_AMRLEVEL (#2081) CMake: do not enable if `AMReX_AMRLEVEL` is off anyway. Tools/CMake/AMReXOptions.cmake commit 3f70dd18d624d90161d323e218ccb398f86fecea Author: Weiqun Zhang Date: Wed Jun 2 20:14:41 2021 -0700 Clean up device random engine states (#2080) Src/Base/AMReX_Random.cpp Src/Base/AMReX_RandomEngine.H commit 241e4ef89f36363c2b00bf99141527db420cea4a Author: Weiqun Zhang Date: Wed Jun 2 20:14:12 2021 -0700 Arena aliases (#2078) Make The_Device_Arena and The_Managed_Arena an alias to The_Arena unless the memory allocation type in incompatible. This should help avoid unnecessary memory fragmentation. Src/Base/AMReX_Arena.cpp commit d20237636ccec63dcfb8540cfa681e0ebd893a40 Author: Axel Huebl Date: Wed Jun 2 16:34:29 2021 -0700 Docs: RDC Option (#2079) ## Summary Add a bit more detailed text to the AMReX_GPU_RDC option. This should help people searching for the option and explain the acronym. ## Additional background Did not yet find a way to slip in "separable compilation" as a term as well... Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXOptions.cmake commit f7fd082bc8ab099d42784f69d6fe49dea3d388a7 Author: Axel Huebl Date: Wed Jun 2 13:39:30 2021 -0700 CMake CUDA: Filter out SM Before Pascal (#2077) Filter out unsupported CUDA architectures in CMake, for both legacy and CMake 3.20+ logic. Tools/CMake/AMReXUtils.cmake commit 2acc1caacf687c92e3cb07a54f7e7ba1f4251888 Author: Andrew Myers Date: Wed Jun 2 13:24:22 2021 -0700 fix bug introduced in PR 2063 (#2076) Src/Particle/AMReX_WriteBinaryParticleData.H commit 439663145ebf0d9f5e66caad738b2bd968ed9331 Author: Andrew Myers Date: Wed Jun 2 13:02:54 2021 -0700 fix signed / unsigned comparison warning (#2075) Src/Particle/AMReX_ParticleCommunication.cpp commit c099e342334c01c8fa2055b2e1e806c62bdf3035 Author: Axel Huebl Date: Wed Jun 2 10:36:22 2021 -0700 CMake 3.20+ CUDA: Policy & Include (#2074) ## Summary Do not include the deprecated CUDA setup with CMake 3.20+ and support new policy for CUDA_ARCHITECTURES. Also fix errors of the kind: ``` ERROR: set_target_properties called with incorrect number of arguments. Value in AMREX_CUDA_ARCHS is 53;60;61;70;75;80;86;86 ``` by quoting the semicolon-separated list. ## Additional background Those are regressions to #2012 that appear with our modernized logic in CMake 3.20+ with CUDA. I saw those regressions with [HiPACE++](https://github.com/Hi-PACE/hipace). CMakeLists.txt Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXTargetHelpers.cmake commit e11fa66c9755e2909280930c387020af2849c1cf Author: Weiqun Zhang Date: Tue Jun 1 15:47:46 2021 -0700 GPU RDC Support (#2061) Disable GPU RDC by default. The user can enable it by `USE_GPU_RDC=TRUE` in gnu make and `-DAMReX_GPU_RDC=ON` in cmake. Remove CUDA specific random number functions that rely on rdc. Co-authored-by: Axel Huebl Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_Arena.H Src/Base/AMReX_Random.H Src/Base/AMReX_Random.cpp Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/GNUMake/Make.defs Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/nvcc.mak Tools/Plotfile/AMReX_PPMUtil.cpp commit 70922cb7cb79bc06a8843f932367c72cf7071ef7 Author: Marc Day Date: Tue Jun 1 16:45:03 2021 -0600 Generalize fextract allowing to specify coords perpendicular to slicedir (#2035) Generalize fextract. Allow spec as coordinate + dir, so that 1D slice can be positioned as desired, within constraints of backward compatibility. Tools/Plotfile/fextract.cpp commit 702af37a91c60bf3916306884ef78319beb5b527 Author: Cody Balos Date: Tue Jun 1 13:11:23 2021 -0700 add interface between sundials SUNMemory and the Arena (#1800) ## Summary As @asalmgren @jmsexton03 @mic84 @ajnonaka @dwillcox @gardner48 @cswoodward and I discussed... this PR adds an interface between the AMReX Arena and the SUNDIALS SUNMemory API. This is currently used in Pele and will be used in Nyx also. In general, any AMReX based app that also uses SUNDIALS (and its provided data structures) will likely find this useful when targeting GPUs. CC @drummerdoc ## Additional background In SUNDIALS 5.4.0, the SUNMemory API was introduced to allow users to substitute their own memory management underneath SUNDIALS data structures. In Pele, using the AMReX arena underneath the SUNDIALS data structures results in as much as a 15% speedup. I have tried to structure the code to follow AMReX conventions. Since we also will be bringing in an interface between the MultiFab and the SUNDIALS N_Vector, I have also tried to structure the code so that it will be easy to add that interface in as well. Co-authored-by: Weiqun Zhang Co-authored-by: David Gardner Co-authored-by: Michele Rosso Co-authored-by: Jean M. Sexton Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GPU.rst GNUmakefile.in Src/Base/AMReX.cpp Src/CMakeLists.txt Src/Extern/SUNDIALS/AMReX_SUNMemory.H Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp Src/Extern/SUNDIALS/AMReX_Sundials.H Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs Tools/GNUMake/packages/Make.sundials commit ef1c484bc467486be3cdff6c90884b0352c39ab5 Author: Andrew Myers Date: Tue Jun 1 12:27:43 2021 -0700 Do not rely on managed memory in ParticleContainer::WriteAsciiFile (#2070) This function is mainly used for small runs / debugging and does not provide filters, hence it should be fine just to copy all the particles to pinned host memory before writing them to disk. Src/Particle/AMReX_ParticleIO.H commit ecffb6a400180f2b2497e66f7c655fafd6bd6eb4 Author: Axel Huebl Date: Tue Jun 1 12:03:13 2021 -0700 HIP ROCTX: USE_RPATH Control (#2064) Just rely on USE_RPATH instead for GNUmake. Follow-up to #2057 Tools/GNUMake/comps/hip.mak commit e7a725644a3eb72dba5c9cd74d35eff4d96af8a8 Author: Andrew Myers Date: Tue Jun 1 12:02:08 2021 -0700 Particle plotfile/checkpoint without managed memory. (#2063) Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H commit b5826c021316179174228e56ad74736c8823caba Author: Weiqun Zhang Date: Tue Jun 1 09:50:40 2021 -0700 Support preprocessing of AMREX_SPACEDIM in input files. (#2049) Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit 1aa8d382af3195e96eea4625dc452c832dfedc53 Author: Weiqun Zhang Date: Tue Jun 1 08:06:13 2021 -0700 Update CHANGES for 21.06 (#2072) CHANGES commit 55ad11dae86fca6737e6e7b8c1656100e5a3bead Author: Axel Huebl Date: Fri May 28 13:03:39 2021 -0700 CI: Cover Fortran (#2067) * CI: Cover Fortran It looks like at some point we changed the `AMReX_Fortran` default from ON to OFF and forgot to adjust CI. This re-enables Fortran compile tests. * CMake: Fix Install Smoke Test (Fortran) Avoid erroring out if Fortran is enabled. .github/workflows/linux.yml Tests/CMakeTestInstall/CMakeLists.txt Tools/CMake/AMReXInstallHelpers.cmake commit 1492a91642e47515b688e58047232daa148196e0 Author: Mark Meredith Date: Fri May 28 00:59:42 2021 -0400 Set HIP_LIBRARIES before using (#2065) Tools/CMake/AMReXParallelBackends.cmake commit 52847883765f041e908350c4d2c03799f6924b1c Author: Axel Huebl Date: Thu May 27 18:09:02 2021 -0700 CMake: HIP Compiler - don't fail (#2062) Potentially, users might use `CC` (Cray Prg Envs) and later `clang++` (underlying compiler) as compilers. Only warn if not met. ROCm/Cray environments are currently being developed and we want to keep things flexible for codes trying to use the compiler wrapper instead of hpcc. Tools/CMake/AMReXParallelBackends.cmake commit 2c2c6992b4169613ba0c956a2a2bd120552f446a Author: jmsexton03 Date: Thu May 27 15:38:31 2021 -0700 Add roctx ranges to TinyProfiler (#2057) ## Summary Adds roctx markup similarly to existing nvtx markup. ## Additional background New features are controlled by added compile flag USE_ROCTX. This assumes the location of roctracer/libroctracer64 and roctracer/libroctx64 are similar to other ROCm library installations such as rocrand. Example run to generate results.json ``` export HIP_PATH=/path/to/rocm/root cd amrex/Tests/GPU/CNS/Exec/Sod make USE_HIP=TRUE TINY_PROFILE=TRUE USE_ROCTX=TRUE USE_MPI=FALSE NO_CONFIG_CHECKING=TRUE srun -n 1 ${HIP_PATH}/bin/rocprof --hsa-trace --stats --timestamp on --roctx-trace ./CNS3d.*.TPROF.HIP.ex inputs ``` This results.json file can be viewed with a browser as described in the documentation using chrome://tracing/ https://github.com/ROCm-Developer-Tools/rocprofiler/blob/amd-master/doc/rocprof.md#43rd-party-visualization-tools Co-authored-by: Axel Huebl Co-authored-by: Weiqun Zhang .github/workflows/dependencies/dependencies_hip.sh Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/comps/hip.mak commit 01361ca148d01c6416e6e87ba672c83b694f3385 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Thu May 27 14:46:35 2021 -0700 Clean up some unused variables. (#2060) Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Tests/MultiBlock/IndexType/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp commit 7dcbf3946a95a96b05f6c9ca6c2ee4d7fb5e8dca Author: Mark Meredith Date: Thu May 27 17:22:44 2021 -0400 Link with HIP_LIBRARIES (#2055) * Link with HIP_LIBRARIES * Try LINK_LANGUAGE GenExpr * Only link with HIP_LIBRARIES for C++ only config * fix * Add HIP_LIBRARIES for all non-Fortran * CI: HIP no-Fortran Build * ROCm spelling Co-authored-by: Axel Huebl .github/workflows/linux.yml Tools/CMake/AMReXParallelBackends.cmake commit 0afebba6ae751a2b58fe3e1f4e8c05e2b9cf2025 Author: Lucas Esclapez <13371051+esclapez@users.noreply.github.com> Date: Thu May 27 12:46:15 2021 -0700 Extrapolator with nGrow > 1. (#2054) Extend the AMReX_Extrapolator in order to handle more than 1 layer of ghost cells. This is done by repeating the original algorithm several times while increasing the box size by 1 at every iteration. Src/Amr/AMReX_Extrapolater.H Src/Amr/AMReX_Extrapolater.cpp Src/Amr/AMReX_extrapolater_2D_K.H Src/Amr/AMReX_extrapolater_3D_K.H commit db8e47927065fcf283f6aae631f91a07b222876b Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed May 26 12:40:32 2021 -0700 Perlmutter Make: CUDA_PATH (#2059) * Perlmutter: find CUDA_ROOT or CUDA_PATH. * Cover all bases. Tools/GNUMake/sites/Make.nersc commit affbd9fe2ca8fecab786c465911067ca05a2c225 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Wed May 26 15:29:53 2021 -0400 Fix to correct CUDA flag and add missing libraries in hypre cmake (#2056) * Fix to correct CUDA flag and add missing libraries in hypre cmake * use CUDAToolkit instead of find_library Tools/CMake/AMReXThirdPartyLibraries.cmake commit 4b29e5d85c2d0eb09b21f50a05a7ef590919d18b Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed May 26 10:12:30 2021 -0700 Perlmutter make (#2058) * Perlmutter make first draft. Allow Pele to start testing. * Keep mpi_nvidia. Tools/GNUMake/sites/Make.nersc commit 33d8bbb1b15009c68bc71ee72c9b067b730148d5 Author: Weiqun Zhang Date: Tue May 25 15:16:38 2021 -0700 Fix compilation of the TensorOverset test (#2053) Tests/LinearSolvers/TensorOverset/MyTest.cpp commit dedbdce8b963167b5ae8a64ca82cf9c8ce6f5d86 Author: Weiqun Zhang Date: Tue May 25 15:16:16 2021 -0700 Remove IndexSequence (#2052) Since we use C++14 now, our own implementation of index sequence can be removed. Src/Base/AMReX_IndexSequence.H Src/Base/AMReX_Tuple.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/EB/AMReX_EB2_IF_Intersection.H Src/EB/AMReX_EB2_IF_Union.H Src/Particle/AMReX_ParticleArray.H commit 12b8518603dd480e01119aa445410d8684858472 Author: Weiqun Zhang Date: Tue May 25 11:49:43 2021 -0700 Use Cray wrappers to get mpi flags on spock (#2051) Tools/GNUMake/sites/Make.olcf commit 0589d3a76098d90e0f7d364234233643cb212699 Author: Mark Meredith Date: Tue May 25 14:30:38 2021 -0400 CMake link with roc::rocprim for HIP (#2050) * CMake link with roc::rocprim for HIP * link with HIP_LIBRARIES * Update Tools/CMake/AMReXParallelBackends.cmake Co-authored-by: Axel Huebl Co-authored-by: Axel Huebl Tools/CMake/AMReXParallelBackends.cmake commit fd5163bb48fdc6b802cf3e9a8d2c861954475e8c Author: Andrew Myers Date: Tue May 25 09:24:32 2021 -0700 Fix PODVector to work without managed memory. (#2048) In a number of places in `amrex::PODVector`, we were always running operations on the host, assuming the presence of managed memory. This fixes these to run either on the host or on the device, depending on the execution policy of the `PODVector`'s `Allocator` (meaning, for managed and device vectors we run on the device, for host and pinned we run on the host). Additionally, this improves the existing test of `PODVector` by passing it `amrex::the_arena_is_managed=0`, which is more sensitive to errors. This fixes an issue reported by @jmsexton03 regarding Nyx and the InitFromBinaryFile capabilities. Src/Base/AMReX_PODVector.H Tests/GPU/Vector/inputs Tests/GPU/Vector/main.cpp commit 11bdf9e45e39070d5072562db633a1da45d36bd6 Author: Andrew Myers Date: Tue May 25 09:13:36 2021 -0700 When building neighbor masks, perform setVal on device so it will work without managed memory. (#2046) This fixes an issue reported by Chris Lishka from Intel re: mfix. Src/Particle/AMReX_NeighborParticlesGPUImpl.H commit e6654533565fad924793bb3515dd1ee086d1bccb Author: kngott <30483578+kngott@users.noreply.github.com> Date: Tue May 25 09:11:52 2021 -0700 Async Output Docs (#2041) Docs/sphinx_documentation/source/IO.rst commit af5a869484df0ee842afcbf231473d1c472499e5 Author: Weiqun Zhang Date: Mon May 24 17:22:51 2021 -0700 Option to set amd gpu targets (#2038) In GNU make, one can set amd gpu targets with make argument AMD_ARCH or environment variable AMREX_AMD_ARCH. Also set MPI wrapper's underlying compiler to hipcc. This is needed if the machine is unknown. For machines known to the gnu make build system, we could either use mpicxx or hipcc -Ipath/to/mpi/include. Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/GNUMake/Make.defs Tools/GNUMake/comps/hip.mak Tools/GNUMake/sites/Make.frontier-coe Tools/GNUMake/sites/Make.olcf commit e28bbc2d08e79ccb9103d38a9df30aac9f335b16 Author: Weiqun Zhang Date: Mon May 24 17:19:55 2021 -0700 Non-GPU build on spock (#2047) Do not link to gpu aware mpi if it's not a GPU build. Tools/CMake/AMReXUtils.cmake Tools/GNUMake/sites/Make.olcf commit 727829666aa490873a747f3d00c2b3c7ddc91508 Author: Axel Huebl Date: Mon May 24 12:34:53 2021 -0700 Work-Around: oneAPI 2021.3.0 SYCL with oneDPL (#2025) Add recommended work-around flags to oneAPI 2021.3.0 SYCL compilers. https://software.intel.com/content/www/us/en/develop/articles/intel-oneapi-dpcpp-library-release-notes.html#inpage-nav-2-3 ``` The use of oneDPL together with the GNU C++ standard library (libstdc++) version 9 or 10 may lead to compilation errors (caused by oneTBB API changes). To overcome these issues, include oneDPL header files before the standard C++ header files, or disable parallel algorithms support in the standard library. For more information, please see Intel® oneAPI Threading Building Blocks (oneTBB) Release Notes. ``` ``` An application using Parallel STL algorithms in libstdc++ versions 9 and 10 may fail to compile due to incompatible interface changes between earlier versions of Threading Building Blocks (TBB) and oneAPI Threading Building Blocks (oneTBB). Disable support for Parallel STL algorithms by defining PSTL_USE_PARALLEL_POLICIES (in libstdc++ 9) or _GLIBCXX_USE_TBB_PAR_BACKEND (in libstdc++ 10) macro to zero before inclusion of the first standard header file in each translation unit. ``` Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in commit 646d5f63445ff46e4df23cbdb1fbe662ca1708fb Author: Axel Huebl Date: Mon May 24 11:20:34 2021 -0700 ParamParse: Find Entries under Prefix (#2043) Extend the "unused params" checks to find generally parameters under a given inputs prefix. Demonstrator & test in: https://github.com/AMReX-Codes/amrex-tutorials/pull/7 Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit 8935b4ca1e8b8ab2e69dc8f1f85c84a276e8efe0 Author: Max Katz Date: Mon May 24 09:14:23 2021 -0700 Add relative gradient to AMRErrorTag (#2044) Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp commit 25b08dc3a46b9b2b96f8e004cb15e1bf67e94bef Author: Weiqun Zhang Date: Fri May 21 10:23:39 2021 -0700 Workaround argument mismatch in MPI reduce for gfortran >= 10 (#2042) In #2040, we added -fallow-argument-mismatch as a workaround. This commit reverts the change and implements an alternative approach that does not require that compiler compiler. We just need to make sure that there are no argument mismatches in the same file. Src/F_Interfaces/Base/AMReX_fi_mpi_mod.F90 Src/F_Interfaces/Base/AMReX_mpi_reduce_int.F90 Src/F_Interfaces/Base/AMReX_mpi_reduce_real.F90 Src/F_Interfaces/Base/AMReX_parallel_mod.F90 Src/F_Interfaces/Base/Make.package Src/F_Interfaces/CMakeLists.txt Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/llvm.mak commit 2255a3c7196ca7ac9225f58edb82c150a2cc0ec9 Author: Andrew Myers Date: Thu May 20 16:17:37 2021 -0700 Wrappers for builtin clz extensions and tests thereof. (#2036) Src/Base/AMReX_Algorithm.H Src/Base/AMReX_Extension.H Tests/CLZ/CMakeLists.txt Tests/CLZ/GNUmakefile Tests/CLZ/Make.package Tests/CLZ/main.cpp Tests/CMakeLists.txt commit 90977cbfe14d79fa391fe4607c5b59513b9f1413 Author: Weiqun Zhang Date: Thu May 20 12:54:13 2021 -0700 Workaround for gfortran 10 (#2040) Starting from gfortran 10, mismatches between actual and dummy argument lists in a single file have been rejected with an error. This causes issues for MPI calls. This commit adds -fallow-argument-mismatch as a workaround. Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/llvm.mak commit 2bbfef42fcd4b5d4386a6145f48eddab7692f907 Author: Axel Huebl Date: Thu May 20 10:27:06 2021 -0700 HIP: AMREX_AMD_ARCH Env Variable (#2034) Adding an AMREX_AMD_ARCH env variable to control the AMD GPU arch from user environments & profiles. Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake commit 16714342b3c08a69438313a9007f93d8c5b7211b Author: Weiqun Zhang Date: Tue May 18 09:50:30 2021 -0700 Add spock support to GNU make (#2028) Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit a1faf316ff6556aabf76fd6d605bf4a59d6c0f59 Author: Weiqun Zhang Date: Tue May 18 08:32:04 2021 -0700 nvcc --display-error-number --diag-error errNum (#2033) Since 11.2, nvcc can display a diagnostic number for any warning message and it can turn specified warnings into errors These two new flags are added to the build system, and it's now an error to write a __device__ variable in a host function. Co-authored-by: Michele Rosso Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReX_SetupCUDA.cmake Tools/GNUMake/comps/nvcc.mak commit aff93b8a26348bb5b42d6af9c3a27ad44e9389f9 Author: Andrew Myers Date: Tue May 18 08:21:08 2021 -0700 Build neighbor masks without managed memory (#2030) Construct these vectors on the host then copy to the device for platforms without managed memory. Src/Particle/AMReX_NeighborParticlesGPUImpl.H commit 79f79aae9f5cc21c09b0353bc0612f41daed4bad Author: Weiqun Zhang Date: Mon May 17 19:13:16 2021 -0700 Option to use oneDPL (#2027) If both USE_ONEDPL and USE_DPCPP are TRUE in GNU make, use oneDPL for the pointer based inclusive and exclusive sum. Also updated the memory fence calls and fixed the compilation for the scan of long for dpcpp. Src/Base/AMReX_FBI.H Src/Base/AMReX_Scan.H Tools/GNUMake/Make.defs Tools/GNUMake/comps/dpcpp.mak commit c6c493bb458515e3ba4b856af097562dcb49a19b Author: mic84 Date: Mon May 17 18:28:54 2021 -0700 CMake: add support for CMake >= 3.20 (#2012) * CMake: group CUDA options in a standalone file * CMake: continue enabling new CUDA support * CMake: make set_mininum_cxx_compiler_version work for a generic language * CMake: rename AMReX_Utils.cmake -> AMReXUtils.cmake * CMake: fix bug * CMake: get rid of NVCC_VERSION_MAJOR and NVCC_VERSION_MINOR * CMake: some upgrades * CMake: upgrade CUDA support * CMake: fix bug * CMake: forgot to guard a set_target_property * CMake: handle CMAKE_CUDA_ARCHITECTURES * CMake: add CUDA flags * CMake: bump up minimum GCC version to 5.1. * CMake: add print for debugging * CMake: add support for PLATFORM_ID genex * Revert "CMake: add print for debugging" This reverts commit 146f645ebae1ac5525e1f5f11cbd167e479dd00f. * CMake: evaluate SHELL genex * CMake: fix printing out of CUDA flags * CMake: fix bug * Update Doc * Update Docs/sphinx_documentation/source/GPU.rst Co-authored-by: Axel Huebl Co-authored-by: Axel Huebl CMakeLists.txt Docs/sphinx_documentation/source/GPU.rst Src/CMakeLists.txt Tools/CMake/AMReXCUDAOptions.cmake Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXFlagsTargets.cmake Tools/CMake/AMReXGenexHelpers.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/CMake/AMReXUtils.cmake Tools/CMake/AMReX_Config.cmake Tools/CMake/AMReX_SetupCUDA.cmake Tools/CMake/AMReX_Utils.cmake commit 9cf1ca565447d5776bd7e7fadd01fdf2e114d53e Author: Weiqun Zhang Date: Mon May 17 17:46:06 2021 -0700 Turn on RDC support for HIP (#2029) This will allow `extern` device variables. Tools/GNUMake/comps/hip.mak commit 6f1085a23de1f7ee1a5f92dd173e984c59839511 Author: Max Katz Date: Sun May 16 11:05:05 2021 -0700 Device profilerStart and profilerStop APIs (#2026) Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp commit c6a8fa243e8449938b8b4b5b16a9e95df9487d48 Author: Emmanuel Motheau Date: Sun May 16 11:04:20 2021 -0700 Workaround to fix the AugmentPlotfile tool (#2023) Tools/C_util/AugmentPlotfile/AugmentPlotfile.cpp commit 442c6302b76ea0ce7d0018f69ae957686d4c4add Author: Weiqun Zhang Date: Fri May 14 18:46:38 2021 -0700 Use rocm 4.1.1 for hip ci (#2024) .github/workflows/dependencies/dependencies_hip.sh .github/workflows/linux.yml commit 6795c1f45833f1784b24d2277664d670f8c7171c Author: Max Katz Date: Fri May 14 17:41:34 2021 -0700 Make nvcc play nicely with CC wrapper (#2019) Tools/GNUMake/Make.defs Tools/GNUMake/comps/cray.mak commit ab84978412f2a00aabf966cbc54b87da39f9ab60 Author: Max Katz Date: Fri May 14 17:34:08 2021 -0700 Add Perlmutter to list of NERSC hosts (#2021) Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.nersc commit 65226552590664d61d32b2d3d8e0de7e691d7e62 Author: Max Katz Date: Fri May 14 17:32:59 2021 -0700 Cray ftn does not accept isystem (#2020) Tools/GNUMake/Make.defs commit d0210641e10dd257a5f2942e992e3c9ab5a4fcfb Author: Max Katz Date: Fri May 14 17:31:38 2021 -0700 Clang Cray can also show up with explicit Clang identifier (#2018) Tools/GNUMake/comps/cray.mak commit 2756788d7b06c3cc7ad5c471ee271edcfa484845 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Fri May 14 12:20:10 2021 -0700 Label function HOST_DEVICE. (#2022) Src/Base/AMReX_RealBox.H commit f7d87b8be6b9bc9373aa17e8b2805256afcd0c55 Author: Max Katz Date: Fri May 14 08:37:11 2021 -0700 Remove restrictions on nvcc host compiler (#2017) These restrictions were originally enforced back in the days of CUDA Fortran, there is no need to require this now. Tools/GNUMake/Make.defs commit 6c9a3f73f3034b5e2b299b6d0a6a7a4333dc1875 Author: Weiqun Zhang Date: Wed May 12 21:58:45 2021 -0700 Remove minBlocksPerMultiprocessor from launch bounds (#2015) After reading CUDA Programming Guide, I think we should remove minBlocksPerMultiprocessor from launch bounds, so that it's closer to what we had before. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#launch-bounds Src/Base/AMReX_GpuLaunchGlobal.H commit 19a2152a097dda52df64e2e42ba60e267fda74be Author: Weiqun Zhang Date: Wed May 12 20:53:02 2021 -0700 Loosen a HIP regresstion test's tolerance (#2014) Tools/RegressionTesting/AMReX-hip-tests.ini commit ad3eea065f3387fd374d9480a31419ba3bbb72bc Author: Weiqun Zhang Date: Wed May 12 20:49:06 2021 -0700 Launch bounds (#2013) Set launch bounds to AMREX_GPU_MAX_THREADS (256 by default) for all kernel launches. This helps resolve some HIP compiler issues we have had recently. Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuReduce.H commit 50847f55331c15f10e38f4e710c0fef292737fe5 Author: Weiqun Zhang Date: Wed May 12 07:45:20 2021 -0700 Disable HIP CI (#2011) ROCm 4.2 has a bug that causes the ci to fail. .github/workflows/linux.yml commit 73dc2422bb07c7346c29c01d1bbfa5a83352ceae Author: asalmgren Date: Tue May 11 20:45:04 2021 -0700 Test on vfrac rather than isRegular when doing the slope limiting -- we were doing this in one (#2008) place but not another place which made the behavior inconsistent. Src/EB/AMReX_EB_slopes_K.H commit 20be66e90b6685865f534ab83b1b8850ad848bcb Author: Weiqun Zhang Date: Tue May 11 16:44:04 2021 -0700 Arena initial size (#2007) Add ParmParse parameters to allow the users to change the initial size of various Arenas. In particular this allows the users to not allocate any managed memory if they set `amrex.the_arena_is_managed=0 amrex.the_managed_arena_init_size=0`. Src/Base/AMReX_Arena.cpp commit 8a650f69446681320504fe5156591dc547bcb0c9 Author: Weiqun Zhang Date: Tue May 11 16:34:48 2021 -0700 Use CPU memory in memory pools for OpenMP threads (#2006) The memory pools are primarily used by Fortran functions as scratch space inside OpenMP parallel regions. There are no reasons to allocate them in managed device memory. Src/Base/AMReX_MemPool.cpp commit bf524ee95ff4a068ddf0a27599d4ce652e25b3cb Author: Weiqun Zhang Date: Tue May 11 13:02:03 2021 -0700 No casting to unsigned long long needed in iMultiFab::sum (#2005) Because we no longer use atomics in reduction, we do not need to cast to unsigned long long in iMultiFab::sum anymore. Have to install tgtermes.sty to build the docs. .github/workflows/dependencies/documentation.sh Src/Base/AMReX_iMultiFab.cpp commit fb0c16e34b9340e68ee63013e64570240e436815 Author: Max Katz Date: Sat May 8 16:12:26 2021 -0700 Avoid call to amrex::max with only one argument in 1D (#2004) Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp commit be62957c227508cda828bd4d4b7ea2ab9d3a2b87 Author: Weiqun Zhang Date: Fri May 7 14:03:21 2021 -0700 Cray complier type detection (#2001) There are two types of Cray compilers, the classical and clang based. The version number is not a reliable way to determine the type. For example, cce/9 on Cori has both versions. Cray compilers for aarch64 on Ookami are classical. Thus we switch to grep "LLVM" in `CC --version`. This works on both Cori and Ookami. Tools/GNUMake/comps/cray.mak commit 3c3bb354d7fb8c9d4b1072d5aabab74b7cc34117 Author: Weiqun Zhang Date: Fri May 7 13:25:22 2021 -0700 Arm Clang (#2002) Add support for Arm Clang in GNU make. Tools/GNUMake/Make.defs Tools/GNUMake/comps/armclang.mak Tools/libamrex/configure.py Tools/libamrex/mkconfig.py commit 9496af4b7bc2e768608c026b06e3afe6f55ce01c Author: Andrew Myers Date: Fri May 7 10:56:30 2021 -0700 Move Morton code utility functions somewhere they can be reused. (#1999) Also ports them to GPU. Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_Morton.H Src/Base/CMakeLists.txt Src/Base/Make.package commit ce04df6c1cb870860672c218f3e3b641f1ec24ec Author: Andrew Myers Date: Wed May 5 17:05:05 2021 -0700 Templated implementation of ostream operator for both Dim3 and XDim3 (#1997) Src/Base/AMReX_Dim3.H Src/Base/AMReX_Dim3.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 8d7d25ff6ae426f31cd609a97d66bc8608fc03d1 Author: Weiqun Zhang Date: Wed May 5 16:57:15 2021 -0700 Disable HIP device assertion (#1998) It makes the compliation very slow and it does not currently work. Src/Base/AMReX.H Src/Base/AMReX_GpuQualifiers.H commit a2a23d75fafb05f7f2a974d0fe7bf5ff584a2823 Author: Andrew Myers Date: Wed May 5 14:59:01 2021 -0700 Generalize elemwiseMin and elemwiseMax. (#1996) This allows these functions to be called on `XDim3` objects, as well as any other object that has `x`, `y`, and `z` data members, instead of just `Dim3`. Additionally, it makes them variadic so they work on any number of objects, instead of just two. Src/Base/AMReX_Algorithm.H commit c1375ccb916f35caca50cc52c72535025e179124 Author: Weiqun Zhang Date: Wed May 5 14:29:04 2021 -0700 Add new parameter to BoxArray::contains(BoxArray) (#1995) In regrid, we need to figure out how many cells we need to grow the current coarse level grids in order for the fine grids to be properly nested. We used BoxArray::contains(BoxArray) in a loop and grew the coarse BoxArray in the loop. In this commit, we add a new parameter to BoxArray::contains(BoxArray) to specify the number of grow cells without having to actually grow the BoxArray. This is better than the previous version because growing BoxArray would invalidate the hash data in BoxArray. We also modify the implementation of BoxArray::contains(Box) in this commit. Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp commit b8dad4457ae795d39bfc4d9e7c3c1927102873df Author: kngott <30483578+kngott@users.noreply.github.com> Date: Tue May 4 15:50:50 2021 -0700 Remove Warnings in DEBUG in AssignParticle Test (#1993) Tests/Particles/AssignDensity/main.cpp commit c6702b5772d40bd7d34aeb65fa445510e8bce01e Author: Weiqun Zhang Date: Tue May 4 15:49:22 2021 -0700 Optimizing away temporary BoxArrays in AmrMesh::MakeNewGrids (#1994) Previously, we stored the information needed for proper nesting in BoxList. Then when they were used, they were repeatedly converted to BoxArray for hash based fast intersection. In this commit, we store them directly in BoxArray, thus removing those hidden temporaries. Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_TagBox.H Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_BoxDomain.H Src/Base/AMReX_BoxDomain.cpp commit 57e8af38fa513674ad9097f6bc6858d8eb669015 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Tue May 4 14:43:40 2021 -0700 Non-blocking SumBoundary (#1991) Non-blocking SumBoundary, using the stored MF pointer in PCData. Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp commit bc335117a000dd5df511d2dc298ce041e09bafee Author: Weiqun Zhang Date: Tue May 4 12:40:10 2021 -0700 Remove an expensive assertion in BoxArray::contains (#1992) The assertion gets extremely expensive for large runs with long skinny patches when it's used by ClusterList::intersect. Src/Base/AMReX_BoxArray.cpp commit 50280bc83decd9c04df80697db3d95ca2d70b947 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Tue May 4 12:29:26 2021 -0700 Docs: Add non-blocking comms. (#1988) Very brief discussion of non-blocking comms for the documentation (currently, I can't find any reference). Docs/sphinx_documentation/source/Basics.rst commit 741fdd27e355d2c5992facd79d41fc283c4ea777 Author: Zach Jibben Date: Mon May 3 22:12:21 2021 -0600 Define BL_USE_MPI compilation flag when MPI and Fortran are enabled (#1990) Resolves #1989 Tools/CMake/AMReXSetDefines.cmake commit b451e90934eb0a8f0d10878e4e087b3aa933d78e Author: kngott <30483578+kngott@users.noreply.github.com> Date: Mon May 3 16:10:47 2021 -0700 FBData object (#1983) Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H commit b27fc508b608eb6f78a9d9bcfe11f1ef0a226461 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Mon May 3 15:41:18 2021 -0700 Deprecate FabArray::copy() (#1985) This deprecates the legacy `FabArray::copy()` in favor of the better named `FabArray::ParallelCopy()` functions. This should help alleviate the problem of confusion between `FabArray::copy()` and `MultiFab::Copy()`, which is a local copy. Also removes "copy" calls from the rest of the repo. Src/Amr/AMReX_AuxBoundaryData.cpp Src/AmrCore/AMReX_FillPatchUtil.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtilI.H Src/Base/AMReX_VisMF.cpp Src/Boundary/AMReX_FabSet.cpp Src/EB/AMReX_EBMultiFabUtil.cpp Src/Extern/ProfParser/AMReX_BLProfUtilities.cpp Src/Extern/amrdata/AMReX_DataServices.cpp Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_TracerParticles.cpp commit ff82946b64fa3dcd4c01df52e564c83ecd752089 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Mon May 3 15:29:33 2021 -0700 PCData object for non-blocking ParallelCopy (#1973) Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H commit 783cbf183adac05004bf1b048942888243e59032 Author: Weiqun Zhang Date: Mon May 3 10:39:06 2021 -0700 Fix proper nesting issue for large error buffer (#1987) Reimplement the code that ensures proper nesting by projecting the fine grids onto the coarse. The previous implementation has a bug that causes it to fail when the error buffer is very large. In the very beginning of BoxLib, the enforcement of proper nesting was actually done in a similar way as the approach in this commit (i.e., calling TagBoxArray::setVal(BoxArray,TagBox::CLEAR) with projected down BoxArray). However, it had the issue of buffering the buffer cells, creating unnecessarily big refinement patches. That issue was fixed in a block of very complicated code that was hard to reason, also in the very beginning of BoxLib days. Since it worked, the code had not been touched at all. However, it was reported recently it failed when a very large error buffet was used. In this commit, we switched back to the original setVal approach and it is performed after the tagged cells are buffered already, thus avoiding the issue of buffering the buffer cells. Furthermore, it is performed after the tags are coarsened, thus performing less work. This also adds a check to make sure that blocking factors do not vary too much between levels. Otherwise regrid will break. Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_TagBox.cpp commit 2b397a9ba0c99376177ec33138e8ed3de707cd8a Author: Houjun Tang Date: Mon May 3 10:36:17 2021 -0700 Update HDF5 I/O to use new HDF5 Async VOL APIs (#1948) Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_ParticleUtil.cpp Src/Particle/AMReX_Particles.H Tests/HDF5Benchmark/GNUmakefile Tests/HDF5Benchmark/main.cpp commit c670d955e90912515389f4715321abe67c4cf206 Author: Mark Meredith Date: Mon May 3 11:59:13 2021 -0400 Remove unused variables (fix warnings) (#1976) Found when building MFiX Src/EB/AMReX_EB_slopes_K.H commit cff96a9f23638819cff688071ff2588d8cfadd63 Author: Weiqun Zhang Date: Sun May 2 10:26:43 2021 -0700 Update CHANGES for 21.05 (#1986) CHANGES commit 8e8b3d75376a081c02c1735fe8d1520a12bfb9f3 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Fri Apr 30 10:19:42 2021 -0600 Hypre Default Params (#1984) This PR reverts the default values for bamg_relax_type back to 6 (Symmetric Gauss Seidel). This appears to be more robust than 11 (2 stage Gauss Seidel). Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit fe2830d4dd426c96cc0454dfb337e08bd489c7e9 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Thu Apr 29 16:48:56 2021 -0600 Fix to this parameter (#1982) Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit bfca28a1f40a9774790d09c3247b39675986aaae Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Thu Apr 29 10:12:41 2021 -0600 modifying some hypre params and changing defaults. (#1981) Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit 47d580e2943dcc90998b8852ead53baaf4dd9a76 Author: cdaley Date: Wed Apr 28 19:57:11 2021 -0700 Support OpenMP offload with the NVIDIA compiler (#1980) These changes enable the ElectromagneticPIC OpenMP offload application to succeed (Particles/ElectromagneticPIC/Exec/OpenMP). It requires NVIDIA HPC SDK 21.3 or higher. Earlier versions of NVIDIA HPC SDK fail for one of two reasons: no interoperability between CUDA and OpenMP offload, or failure to support Fortran automatic arrays in OpenMP offload regions. I tested on Cori-GPU with nvhpc/21.3 and cuda/11.1.1. Co-authored-by: Christopher Daley Tools/GNUMake/Make.defs Tools/GNUMake/comps/nvhpc.mak Tools/libamrex/mkconfig.py commit a61b40464f3f16eff420d19d99cff0fa010eb2b6 Author: Weiqun Zhang Date: Tue Apr 27 11:32:55 2021 -0700 Use full path to AMReX_Config.H in gnu make target (#1979) Tools/GNUMake/Make.rules commit 30879b025d222b383aa22540598dc1891ce73478 Author: Weiqun Zhang Date: Mon Apr 26 16:43:20 2021 -0700 Fix GNU make for nvcc 11.0 (#1977) This fixes a bug introduced in #1967. It turns out nvcc 11.0 does not support -MP. Tools/GNUMake/comps/nvcc.mak commit 0340e766b71005c9675d6e9b759c66bb5c705297 Author: Weiqun Zhang Date: Mon Apr 26 13:20:16 2021 -0700 Add CI for 2D Single Precision EB HIP build (#1974) Also fix some warnings. .github/workflows/linux.yml Src/Base/AMReX_GpuLaunchGlobal.H Src/Base/AMReX_Reduce.H Src/EB/AMReX_EB2_2D_C.cpp commit 521ff492834377cc5ee91d6eb72db2471ccde3b7 Author: Weiqun Zhang Date: Mon Apr 26 13:00:09 2021 -0700 Fix dependency of f90.depends (#1975) Tools/GNUMake/Make.rules commit b1709947c0a354dc5dbefeb3421497478ef2333c Author: Weiqun Zhang Date: Mon Apr 26 09:07:35 2021 -0700 GNU Make: -MMD -MP (#1967) Use -MMD -MP instead of -M to generate dependency unless it is not supported by the compiler. In the new approach, the dependencies are generated during compilation without a separate step. Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/comps/cray.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/llvm.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/comps/pgi.mak commit e6833a86ebf35861017cfd5f16ca66ac9dc8b7ae Author: Luca Fedeli Date: Mon Apr 26 17:13:19 2021 +0200 Fix missing Real() in AMReX_EB2_2D_C.cpp (#1972) This PR fixes few missing `Real()` in `AMReX_EB2_2D_C.cpp`, which currently prevent the compilation of a function in single precision. Src/EB/AMReX_EB2_2D_C.cpp commit 32597f78f68fc8b948ceb10a22c0d01ab2b513ec Author: Weiqun Zhang Date: Mon Apr 26 08:11:58 2021 -0700 Undeprecate ReduceData::value() (#1970) ReduceData::value() with no argument was recently marked as a depracated function because it involves calling a std::function. However, the call is never made inside a loop like MFIter. So there is no practical difference in performance between ReduceData::vaule and ReduceData::value(ReduceOps&). It is hereby undeprecated to avoid warning messages for a lot of applications. Src/Base/AMReX_Reduce.H commit 5eedbae2d805d86014788427bee39e6cccb2155c Author: Shreyas Ananthan Date: Sun Apr 25 13:12:48 2021 -0600 Workaround compile error on Apple M1 (ARM64) (#1969) Fixes #1968 Src/Base/AMReX.cpp commit 41d1ea1873494d5e59c6f7da16d73b634e2a9362 Author: asalmgren Date: Fri Apr 23 10:54:14 2021 -0700 Changes to EB slopes (#1965) * 1) Change test on flag.isRegular() to (vfrac == 1) -- this only matters when we have cells with vfrac == 1 but with an EB face -- this change will allow us to use the non-EB stencil when parallel to an EB face if all the vfrac's we need are = 1, and is also consistent with how we treat slopes tangential to domain boundaries 2) pass max_order in so we can default to 4th order if max_order == 4 and there are enough cells with vfrac == 1 3) Modify the Barth-Jespersen limiting so that, for example, it doesn't use cells at i-1 when computing a slope on the i+1/2 face. NOTE: this contains a change to the interfaces to the slope routines so all code that calls these will need to be modified to pass in vfrac and max_order * Remove trailing white spaces * remove more trailing white spaces * the last of the trailing white spaces ... * Compile in 2d Src/EB/AMReX_EB_slopes_K.H commit c0570ec8d60f1a6227b5835f9105801a40a81eab Author: Axel Huebl Date: Fri Apr 23 10:24:16 2021 -0700 Docs: CMake Multiple Archs (#1964) Answer a user question with a new doc entry. Docs/sphinx_documentation/source/GPU.rst commit e85b665bcb032f4064fb9845a925c6d3d8c8aa03 Author: Weiqun Zhang Date: Fri Apr 23 09:06:37 2021 -0700 Split Large Files (#1966) Split some large files in MLMG into smaller files to improve compile time. Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_F.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp_bc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_eb.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_hypre.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp_grad.cpp Src/LinearSolvers/MLMG/Make.package commit 84cc7114097b548dce3b704b6eb9681b425f1e94 Author: Weiqun Zhang Date: Thu Apr 22 17:56:10 2021 -0700 Build Time Log (#1963) In GNU Make, if one uses `LOG_BUILD_TIME=TRUE`, a log of build time will be available in the end. Tools/C_scripts/gatherbuildtime.py Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules commit 6e5a03f8f8346e0f4840d1b5670f9afa60a0fc4a Author: Weiqun Zhang Date: Thu Apr 22 10:19:42 2021 -0700 Using rocPRIM in HIP Version of Scan (#1962) Reimplement HIP version of scan using rocprim. This uses a few things in rocprim::detail. Hopefully it will not break in a future release of HIP. Also some minor tweaks to the CUDA version of Scan. Src/Base/AMReX_GpuLaunchGlobal.H Src/Base/AMReX_Scan.H commit a0790de872d6b18b563712b9f1b18323b8d00072 Author: Weiqun Zhang Date: Tue Apr 20 16:21:37 2021 -0700 Reimplementation of ReduceOps (#1955) The new approach does reduction in two passes. Unlike the old approach, this does not require atomics, and has much better performance for types that do not have hardware support for atomics. Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_iMultiFab.cpp Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EBCellFlag.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/Particles/Intersection/main.cpp commit 1796dd911bae1ae0b58ae92e4e023bdb00a0f44a Author: mic84 Date: Mon Apr 19 17:01:36 2021 -0700 CMake: implement logic to support compiler ID IntelLLVM (new in CMake 3.20) (#1959) CMake 3.20+ has its own compiler identification for the new oneAPI LLVM-ICX/DPCXX implementation. Tools/CMake/AMReXFlagsTargets.cmake Tools/CMake/AMReXGenerateConfigHeader.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake commit 0a50fded0df7539b5e8e32b2a88966d8d671e264 Author: Weiqun Zhang Date: Mon Apr 19 14:19:53 2021 -0700 DPCPP: fix atomicAdd for float (#1958) We recently removed the template specialization for atomicAdd of float in the DPC++ backend. That was a mistake because SYCL does not support atomicAdd for float, and we have to use atomicCAS to implement it. Src/Base/AMReX_GpuAtomic.H Tests/Amr/Advection_AmrCore/Source/Src_K/slope_K.H commit c409f9e1ffc870fd4926190ffceffc7d0d5ad409 Author: Weiqun Zhang Date: Mon Apr 19 13:30:05 2021 -0700 CPU version of TagBox::buffer (#1957) Add a CPU version of TagBox::buffer. @MSABuschman reported in #1951 that TagBox::buffer has been very slow since commit #1258 if the error buffer size is large. The function was rewritten in #1258 to do the work on GPU. In this PR, the old version is reintroduced for CPU. Note that the current implementation is expected to have poor performance on GPU if it has a very large error buffer. It's still not clear how we should implement this function for GPU if a large error buffer is used. .github/workflows/linux.yml Src/AmrCore/AMReX_TagBox.H Src/AmrCore/AMReX_TagBox.cpp commit 4320729fe46992851b3f222d28833109cff0234e Author: jmsexton03 Date: Sun Apr 18 11:21:27 2021 -0700 Fix out of bound error in AsyncWrite (#1753) Fix out of bound error when the local number of fabs is zero. Co-authored-by: Weiqun Zhang Src/Base/AMReX_VisMF.cpp commit b30cae951ba2c7c004f06e3d7d382c7e79793887 Author: Weiqun Zhang Date: Sun Apr 18 08:15:16 2021 -0700 Fab data I/O (#1956) When Fab data are in GPU device memory (including managed memory), use pinned memory as buffer to avoid accessing the data on the host. Previously, MultiFab, plotfile and checkpoint writers except for the async versions relied on that the memory was managed and therefore could be used directly in I/O. Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_VisMF.cpp commit f21f15d8f7a3098d9fa13ac6b316ca518f464ac5 Author: Max Katz Date: Fri Apr 16 17:54:37 2021 -0700 Allow user to opt out of renaming MTMPI suffix (#1954) If MPI_THREAD_MULTIPLE is going to be on by default in an application, might as well not need to rename the suffix, so this gives the application the ability to just keep the suffix MPI. Tools/GNUMake/Make.defs commit 3812eadb83a667a615630a2ff8d92def964c3e58 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Fri Apr 16 19:57:47 2021 -0400 Add triangle wave function to least squares test (#1947) * Add a triangle wave test for Least Squares centroids, fix some bugs in test setup * update doc * fix formatting * fix formatting Tests/LinearSolvers/LeastSquares/Make.package Tests/LinearSolvers/LeastSquares/MyTest.H Tests/LinearSolvers/LeastSquares/MyTest.cpp Tests/LinearSolvers/LeastSquares/README.md Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/LeastSquares/initTriangleWaveData.cpp Tests/LinearSolvers/LeastSquares/initTriangleWaveDataFor2D.cpp Tests/LinearSolvers/LeastSquares/inputs.2d.trianglewave commit eaad7627ff925f8ae9b290a7d6a2d5f2fba54239 Author: asalmgren Date: Fri Apr 16 15:52:15 2021 -0700 level_0_gome --> level_0_geom (#1953) Src/AmrCore/AMReX_AmrCore.cpp commit 029a8b0608beab17f440718b5b147dc5dd24b67c Author: Weiqun Zhang Date: Fri Apr 16 13:51:19 2021 -0700 Port FluxRegister::ClearInternalBorders to GPU (#1952) Src/AmrCore/AMReX_FluxRegister.cpp commit 42da8fb5ac8df41566a045bb3332df6ad24126d8 Author: Weiqun Zhang Date: Thu Apr 15 19:19:25 2021 -0700 Fix the return of MLPoisson::makeNLinOp (#1950) Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp commit 66f99da25c6a305fae47f0b952ebe670af867b19 Author: Andrew Myers Date: Thu Apr 15 18:07:28 2021 -0700 Skip temporary in ParticleToMesh when it is not needed. (#1949) Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_ParticleMesh.H commit d2e37b06bed136e001ae62718ba5754720a01f0d Author: Andrew Myers Date: Thu Apr 15 11:32:42 2021 -0700 Multilevel version of ParticleToMesh (#1945) This uses the same algorithm as the AssignDensity method in `AmrCore/AMReX_AmrParticles.H` to handle the coarse / fine boundaries. However, it allows a user-specified lambda function to define the interpolation operation, as in the single-level `ParticleToMesh` routine. Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleUtil.H Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/CMakeLists.txt Tests/Particles/ParticleMeshMultiLevel/GNUmakefile Tests/Particles/ParticleMeshMultiLevel/Make.package Tests/Particles/ParticleMeshMultiLevel/inputs Tests/Particles/ParticleMeshMultiLevel/main.cpp commit b1d05dec0da3b6d0074af8850163837c53488ebf Author: Erik Schnetter Date: Thu Apr 15 13:32:54 2021 -0400 Correct typo in comment (#1946) Tools/CMake/AMReXOptions.cmake commit 914459f443f5ed54b5e33551802d10521042af9b Author: Don E. Willcox Date: Wed Apr 14 16:13:44 2021 -0700 fix greater->less in docs for abort_on_out_of_gpu_memory (#1944) Docs/sphinx_documentation/source/GPU.rst commit 87ee8f03c3b38c645312331fa75c92b6a932a87b Author: Weiqun Zhang Date: Tue Apr 13 16:45:37 2021 -0700 Use CUB for block reduction (#1943) Also removed some reduction functions that have been deprecated long ago. Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_Reduce.H commit ba378870a10ad475c647cad3b8650151fc896580 Author: Weiqun Zhang Date: Tue Apr 13 08:24:57 2021 -0700 Reimplementation of PrefixSum with cub (#1942) For CUDA >= 11, use cub to reimplement PrefixSum that can take two lambda functions. Both cub and thrust only provide scan functions that take iterators. Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H commit 58c3ba169a2b8fc311395a959724ea63dc37aa3c Author: Weiqun Zhang Date: Sun Apr 11 17:03:22 2021 -0700 Avoid AMREX_NO_UNIQUE_ADDRESS (#1941) * Avoid AMREX_NO_UNIQUE_ADDRESS because it causes gcc 9.3 to crash. * Fix AMREX_NO_UNIQUE_ADDRESS and AMREX_NODISCARD macros. * Adjust compiler flag in CI to allow for C++17 extension when compiling with C++14. .github/workflows/linux.yml Src/Base/AMReX_Extension.H Src/Base/AMReX_NonLocalBC.H commit 18c5e96e04c982ef886d8aa9f6e65ebee1564f35 Author: Weiqun Zhang Date: Thu Apr 8 12:58:09 2021 -0700 Rename variable to fix a shadow warning (#1939) Src/Particle/AMReX_ParticleUtil.H commit 51ba2f7ebbc2dc759845fdab073d353ab175507f Author: Weiqun Zhang Date: Thu Apr 8 11:55:51 2021 -0700 Workaround a compiler bug(?) reported in #1937 (#1938) Src/Base/AMReX_Arena.cpp commit 7c259bab6cc583a92a8d7f6b640af6618e6206f3 Author: Weiqun Zhang Date: Wed Apr 7 17:52:04 2021 -0700 Cell-Centered Overset Solver Coarsening (#1915) Currently for the cell-centered overset multigrid solver, coarsening stops when a coarsened cell would contain both regular and masked-out cells. Here, we further coarsen the grids by ignoring the overset mask and giving the masked-out cells a huge value of alpha so that their values are essentially frozen. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp commit db9303290d4912f15230c792f4e1c91bce0fb0ce Author: Weiqun Zhang Date: Wed Apr 7 15:38:40 2021 -0700 Hypre Nodal Setup (#1906) * Hypre Nodal Setup * Allow CoarseningStrategy::Sigma in Hypre nodal solver. * Port Hypre nodal setup to GPU. * For EB nodal solver, label a node ad Dirichlet if its diagonal component of the stencil is zero. * remove EB stuff in hypre nodal setup because it's no longer needed due to the change in dirichlet mask Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_Scan.H Src/EB/AMReX_EB2_Level.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp commit 8f9589682e56d7f443e42de032b027bd7b1944fa Author: mic84 Date: Wed Apr 7 14:15:50 2021 -0700 CMake: add support for BOUND_CHECK (#1934) Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in commit d5e4a3882b263c858221de1130f73e04825a2e1e Author: Luca Fedeli Date: Wed Apr 7 16:54:28 2021 +0200 Fix other missing Real() (#1933) This PR fixes few missing `Real()` in `AMReX_EB2_3D_C.cpp`, which currently prevent the compilation of some methods in single precision. Src/EB/AMReX_EB2_3D_C.cpp commit 3f3a9174286aa06d7826367b3d07b29c339d428a Author: Luca Fedeli Date: Wed Apr 7 04:13:13 2021 +0200 fix missing Real (#1932) Src/EB/AMReX_EB2_3D_C.H commit 4123e87ce3779855a638663008de8dcec5ccb99f Author: Andrew Myers Date: Tue Apr 6 12:08:23 2021 -0400 Alias HostVector to PinnedVector and update documentation. (#1931) Src/Base/AMReX_GpuContainers.H commit 4cb1c5b79906fe566d1b80777315cf4817924b1a Author: Weiqun Zhang Date: Mon Apr 5 16:56:23 2021 -0700 Make CNS work on CPU too (#1930) Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FabConv.cpp Src/Base/AMReX_IArrayBox.cpp Tests/GPU/CNS/Exec/RT/cns_prob.cpp Tests/GPU/CNS/Exec/Sod/cns_prob.cpp Tests/GPU/CNS/Source/CNS.cpp commit 7c37a75e23eabc317c7a969e21153183db616fd1 Author: Axel Huebl Date: Mon Apr 5 16:15:02 2021 -0700 std::isfinite/inf for SYCL (#1929) Wrappity wrap the non-constexpr functions: https://github.com/intel/llvm/blob/sycl/sycl/doc/extensions/C-CXX-StandardLibrary/C-CXX-StandardLibrary.rst Note that `std::isnan` works by accident/virtue of being constexpr. Src/Base/AMReX_Math.H commit 1ee06e64345a52259315181ac13a821a4ac64f09 Author: Weiqun Zhang Date: Mon Apr 5 14:10:12 2021 -0700 Update RNG for latest oneAPI (#1928) MKL now has device support for uniform distribution of unsigned int and Poisson distribution. Src/Base/AMReX_Random.H commit deddea69d9f7e2b5cac2727cadfe77e0806c194b Author: Weiqun Zhang Date: Mon Apr 5 10:13:30 2021 -0700 Use rocprim for scan (#1925) .github/workflows/dependencies/dependencies_hip.sh .github/workflows/linux.yml Src/Base/AMReX_Scan.H commit 06cec5dd8ede26ea3c0646368bdbdb32f35842ae Author: Weiqun Zhang Date: Mon Apr 5 09:31:25 2021 -0700 Use pinned memory for data transfer between host and device (#1923) The amount of data to be transferred in TagBox and STLtools is likely to be big. So pinned memory is used instead of pageable. AsyncArray is sometimes used to transfer a large amount of data. So pinned memory is used. Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_TagBox.H Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuAsyncArray.cpp Src/EB/AMReX_EB_STL_utils.H commit d3a24fe0519d8eb7cfd7043be2907e08ee534352 Author: Weiqun Zhang Date: Mon Apr 5 09:16:28 2021 -0700 Use cub::DeviceScan for CUDA >= 11 (#1924) Src/Base/AMReX_Scan.H Tools/GNUMake/Make.defs Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/nvcc.mak commit ddfbf3de3bdf5ced8f42d0828ed52d95fcea22ec Author: Weiqun Zhang Date: Mon Apr 5 09:16:07 2021 -0700 Fix compilation broken in #1920 (#1926) Tests/LinearSolvers/CellEB/MyTest.cpp Tests/LinearSolvers/CellEB2/MyTest.cpp Tests/LinearSolvers/EBConvergenceTest/MyTest.cpp Tests/LinearSolvers/EBflux_grad/MyTest.cpp Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/NodeEB/MyTest.cpp commit 5ec3d3799ff238208ae41949ce506d3491b37208 Author: Weiqun Zhang Date: Fri Apr 2 19:03:42 2021 -0700 Move Tests/CNS to Tests/EB/CNS (#1922) Tests/EB/CNS/CMakeLists.txt Tests/EB/CNS/Exec/Combustor/GNUmakefile Tests/EB/CNS/Exec/Combustor/Make.package Tests/EB/CNS/Exec/Combustor/bc_fill_nd.F90 Tests/EB/CNS/Exec/Combustor/bc_fill_nd.F90_jbb Tests/EB/CNS/Exec/Combustor/cns_prob.F90 Tests/EB/CNS/Exec/Combustor/cns_prob.F90_jbb Tests/EB/CNS/Exec/Combustor/inputs Tests/EB/CNS/Exec/Combustor/inputs.regt Tests/EB/CNS/Exec/Make.CNS Tests/EB/CNS/Exec/Pulse/GNUmakefile Tests/EB/CNS/Exec/Pulse/Make.package Tests/EB/CNS/Exec/Pulse/cns_prob.F90 Tests/EB/CNS/Exec/Pulse/inputs Tests/EB/CNS/Exec/Pulse/inputs.regt Tests/EB/CNS/Exec/ShockRef/GNUmakefile Tests/EB/CNS/Exec/ShockRef/Make.package Tests/EB/CNS/Exec/ShockRef/cns_prob.F90 Tests/EB/CNS/Exec/ShockRef/inputs Tests/EB/CNS/Exec/ShockRef/inputs.amr Tests/EB/CNS/Exec/ShockRef/inputs.regt Tests/EB/CNS/Exec/Sod/GNUmakefile Tests/EB/CNS/Exec/Sod/Make.package Tests/EB/CNS/Exec/Sod/cns_prob.F90 Tests/EB/CNS/Exec/Sod/inputs Tests/EB/CNS/Source/CNS.H Tests/EB/CNS/Source/CNS.cpp Tests/EB/CNS/Source/CNSBld.cpp Tests/EB/CNS/Source/CNS_F.H Tests/EB/CNS/Source/CNS_advance.cpp Tests/EB/CNS/Source/CNS_init_eb2.cpp Tests/EB/CNS/Source/CNS_io.cpp Tests/EB/CNS/Source/CNS_setup.cpp Tests/EB/CNS/Source/Make.package Tests/EB/CNS/Source/diffusion/Make.package Tests/EB/CNS/Source/diffusion/cns_diff_mod.F90 Tests/EB/CNS/Source/diffusion/cns_eb_diff_mod.F90 Tests/EB/CNS/Source/diffusion/cns_eb_diff_wall.F90 Tests/EB/CNS/Source/diffusion/diff_coef_mod.F90 Tests/EB/CNS/Source/fortran/CNS_derive.F90 Tests/EB/CNS/Source/fortran/CNS_divop.F90 Tests/EB/CNS/Source/fortran/CNS_dudt.F90 Tests/EB/CNS/Source/fortran/CNS_f.F90 Tests/EB/CNS/Source/fortran/CNS_nd.F90 Tests/EB/CNS/Source/fortran/CNS_physics.F90 Tests/EB/CNS/Source/fortran/CNS_tagging.F90 Tests/EB/CNS/Source/fortran/Make.package Tests/EB/CNS/Source/fortran/bc_fill_nd.F90 Tests/EB/CNS/Source/hydro/Hyp_gamma_MOL.F90 Tests/EB/CNS/Source/hydro/Hyp_gamma_MOL_EB.F90 Tests/EB/CNS/Source/hydro/Make.package Tests/EB/CNS/Source/hydro/analriem3d.F90 Tests/EB/CNS/Source/hydro/cns_eb_hyp_wall.F90 Tests/EB/CNS/Source/hydro/slope_mol_3d_gamma.F90 Tests/EB/CNS/Source/hydro/slope_mol_3d_gamma_EB.F90 Tests/EB/CNS/Source/main.cpp commit 42063585bb299c5dd220aa5b35be9e3f05594da5 Author: Weiqun Zhang Date: Fri Apr 2 18:07:50 2021 -0700 No need for AMREX_IS_TRIVIALLY_COPYABLE && AMREX_IS_TRIVIALLY_DEFAULT_CONSTRUCTIBLE (#1921) They are not needed because gcc 4 is no longer supported. Src/Base/AMReX_BaseFab.H Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_TypeTraits.H commit 77fb0251e3ad99af069d5d0bcaa80128cc50062b Author: Weiqun Zhang Date: Fri Apr 2 10:30:42 2021 -0700 Use std::make_unique instead of reset since we are using C++14 (#1920) Docs/sphinx_documentation/source/IO.rst Docs/sphinx_documentation/source/LinearSolvers.rst Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrParticles.H Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_AsyncOut.cpp Src/Base/AMReX_BackgroundThread.cpp Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_CuptiTrace.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_ForkJoin.cpp Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MemProfiler.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_PlotFileDataImpl.cpp Src/Base/AMReX_PlotFileUtil.cpp Src/Base/AMReX_iMultiFab.cpp Src/EB/AMReX_EB2_IndexSpaceI.H Src/EB/AMReX_EBFabFactory.cpp Src/Extern/HYPRE/AMReX_Hypre.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/PETSc/AMReX_PETSc.cpp Src/Extern/amrdata/AMReX_AmrData.cpp Src/F_Interfaces/Octree/AMReX_octree_fi.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/Projections/AMReX_MacProjector.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_TracerParticles.cpp Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/FillBoundaryComparison/main.cpp Tests/GPU/CNS/Source/CNS.cpp Tests/GPU/CNS/Source/CNS_io.cpp Tests/HDF5Benchmark/main.cpp Tests/LinearSolvers/CellEB/MyTest.cpp Tests/LinearSolvers/CellEB2/MyTest.cpp Tests/LinearSolvers/CellOverset/MyTest.cpp Tests/LinearSolvers/EBConvergenceTest/MyTest.cpp Tests/LinearSolvers/EBflux_grad/MyTest.cpp Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/NodeEB/MyTest.cpp Tests/LinearSolvers/TensorOverset/MyTest.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/AsyncIO/main.cpp commit 57d41ef9cc3f0db4b5d644d0169311b4f9f68a92 Author: Weiqun Zhang Date: Thu Apr 1 08:37:00 2021 -0700 Update CHANGES for 21.04 (#1919) CHANGES commit 622fbf998d44b77f44d1c682b9b33e018402051f Author: Andrew Myers Date: Thu Apr 1 11:04:12 2021 -0400 Docs typo fix. (#1918) Docs/sphinx_documentation/source/LinearSolvers_Chapter.rst commit ecf7e9812760feff4cdb2c53deb3402b68383fc8 Author: asalmgren Date: Wed Mar 31 14:02:27 2021 -0700 Allow us to set the "normalization threshold" variable at run-time (#1917) through the NodalProjector class Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit b8559fe999715afd3056cb94371bf2399a461863 Author: Weiqun Zhang Date: Tue Mar 30 17:01:58 2021 -0700 Fix a bug in BaseFab::resize. (#1916) Previously if `BaseFab::resize`'s optional `Arena*` argument is `nullptr` (which is the default) while `BaseFab`'s `Arena*` member is a different `Arena`, the `nullptr` will be interpreted as the default `Arena`, `The_Arena`, ignoring the `BaseFab`'s `Arena` has already been set. This causes the following code to allocate data using `The_Arena`, not `The_Cpu_Arena`. BaseFab fab(The_Cpu_Arena()); fab.resize(box); This PR fixes the bug. Src/Base/AMReX_BaseFab.H commit 81982da08fb25cd266b3666d358d1cbf09bb4de5 Author: asalmgren Date: Tue Mar 30 12:47:49 2021 -0700 Change the default value of m_normalization_threshold from 1e-10 to 1e-8 (#1914) This is motivated by an EB nodal BiCG bottom solver failing to converge with 1e-10 while converging with 1e-8. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H commit 92945ad3a3560031c43fe7f02b9cc252f8330708 Author: Andrew Myers Date: Tue Mar 30 12:34:53 2021 -0400 Dont zero out input multifab in amrex::ParticleToMesh (#1910) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleMesh.H commit 78eae52af97d4c6b993feec9bf76f21b9ef7288a Author: asalmgren Date: Mon Mar 29 17:00:17 2021 -0700 Revert the change which zeroed out the center coefficient if it has the wrong (#1911) sign. This breaks more cases than it fixes. Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H commit 227e378901d904040340ad38971db5142f008d15 Author: vzendejasl <59891999+vzendejasl@users.noreply.github.com> Date: Mon Mar 29 13:56:25 2021 -0700 Slopes test (#1859) Slope computation on cell centroids using the Least Square method. These tests are based from the LeastSquares Test (#1707) but instead of calculating scalar values on face centroids they are calculated on cell centroids using a linear function rather than a second order polynomial (#1707). Tests/Slopes/GNUmakefile Tests/Slopes/Make.package Tests/Slopes/MyEB.H Tests/Slopes/MyTest.H Tests/Slopes/MyTest.cpp Tests/Slopes/README.md Tests/Slopes/initData.cpp Tests/Slopes/initEB.cpp Tests/Slopes/initLinearData.cpp Tests/Slopes/initLinearDataFor2D.cpp Tests/Slopes/initLinearDataFor3D.cpp Tests/Slopes/inputs Tests/Slopes/inputs.2d.askew Tests/Slopes/inputs.2d.base Tests/Slopes/inputs.2d.fullyrotated Tests/Slopes/inputs.3d.linear.aligned.xy-x Tests/Slopes/inputs.3d.linear.aligned.xy-y Tests/Slopes/inputs.3d.linear.aligned.xz-x Tests/Slopes/inputs.3d.linear.aligned.xz-z Tests/Slopes/inputs.3d.linear.aligned.yz-y Tests/Slopes/inputs.3d.linear.aligned.yz-z Tests/Slopes/inputs.3d.linear.askew-all Tests/Slopes/inputs.3d.linear.askew-xy Tests/Slopes/inputs.3d.linear.askew-xz Tests/Slopes/inputs.3d.linear.askew-yz Tests/Slopes/main.cpp commit 0ab63f3f93950b5a90fb091c6eabbf3239886fcd Author: Weiqun Zhang Date: Mon Mar 29 09:32:23 2021 -0700 Fix bug in nodal solver that uses harmonic averageing (#1902) This is a minor bug that does not affect the final result. The convergence rate might be slightly affected. Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H commit dc8f72702c1ed4bb4443916ded01d9daff4f811f Author: asalmgren Date: Fri Mar 26 12:40:02 2021 -0700 Ensure center coefficient in an EB nodal solve has correct sign (#1905) It is possible for the center coefficient in an EB nodal solve to have the incorrect sign after coarsening -- here we protect against that by setting s0 to the max of (s0,0) -- this enables the BiCG to converge. Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H commit 9f297b9bc3a233553925feb5c0e2db784f1ddf9e Author: mic84 Date: Fri Mar 26 12:00:44 2021 -0700 CMake: add test_install (#1894) * CMake: add test_install * CMake: forgot to add test dir * CMake: specify compiler to use when running test_install * Update Doc .github/workflows/linux.yml CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Tests/CMakeTestInstall/CMakeLists.txt Tools/CMake/AMReXInstallHelpers.cmake commit 146fc573967dbb2328be15d8e499a479f218c4e8 Author: jmsexton03 Date: Thu Mar 25 16:16:33 2021 -0700 Update Make.CNS and similar files to reflect Tutorials to Tests move (#1903) Tests/Amr/Advection_AmrCore/README Tests/Amr/Advection_AmrCore/README.md Tests/Amr/Advection_AmrLevel/Exec/Make.Adv Tests/CNS/Exec/Make.CNS Tests/FortranInterface/Advection_F/Exec/Make.Adv Tests/FortranInterface/Advection_octree_F/Exec/Make.Adv Tests/GPU/CNS/Exec/Make.CNS Tests/LinearSolvers/MultiComponent/MCNodalLinOp.H Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/MultiComponent/main.cpp commit f9216a341e51404e935733a4c953525a6f503d47 Author: Andrew Myers Date: Thu Mar 25 14:42:37 2021 -0400 Fix AMREX_EXPORT (#1901) Src/EB/AMReX_EB2.H commit e57c1e8294c7c74a5e3a71821639a4cc1e48d895 Author: Andrew Myers Date: Thu Mar 25 13:52:56 2021 -0400 reimplement increment and incrementwithtotal to take advantage of GPUs and OpenMP (#1899) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_Particles.H commit 62b3d7e4cafbc5b2f6453782bd9f4500014c4b63 Author: Andrew Myers Date: Thu Mar 25 13:46:49 2021 -0400 Export more global variables for windows support (#1900) Src/EB/AMReX_EB2.cpp commit 87c81d0ca70e65c9209386c03b3c85ab204667bb Author: Axel Huebl Date: Wed Mar 24 16:48:05 2021 -0700 CI: HIPCC as C Compiler (#1897) Since `hipcc` is a C++ compiler, we should assign it with `-x c` if used as C compiler. Or... we just use the corresponding `clang` for C files. .github/workflows/linux.yml commit f2443e0bc4aa433baa2154cf9b3b7ca4a672bf65 Author: Andrew Myers Date: Wed Mar 24 18:09:52 2021 -0400 Return const& in rdata and idata and disallow returning copies for rvalues. (#1895) Src/Particle/AMReX_Particle.H commit d226b807024fedd11eec4d751367524f75c01178 Author: Axel Huebl Date: Wed Mar 24 14:36:25 2021 -0700 CI: HIP w/o MPI (#1896) HIP 4.1 introduces issues with C targets which break the MPI feature test in CMake's `FindMPI.cmake` for C. .github/workflows/linux.yml commit 7c3e95210a1877c67139ea57c62fbcc717acc1c3 Author: Andrew Myers Date: Mon Mar 22 18:13:14 2021 -0400 Remove unnecessary abort. (#1892) Src/EB/AMReX_distFcnElement.cpp commit f3ddeadbb53049034f817d8bb9a8c21951ff494a Author: Weiqun Zhang Date: Mon Mar 22 14:53:44 2021 -0700 Update ALaplacian for multi-component and reuse (#1891) Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.cpp commit d1468003318845a83fb59692bb4f06be95eb319a Author: Andrew Myers Date: Mon Mar 22 13:28:37 2021 -0400 Move tutorials (#1876) .github/workflows/linux.yml .github/workflows/macos.yml .github/workflows/windows.yml CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_NFiles.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H Tests/Amr/Advection_AmrCore/CMakeLists.txt Tests/Amr/Advection_AmrCore/Exec/GNUmakefile Tests/Amr/Advection_AmrCore/Exec/GNUmakefile_movie Tests/Amr/Advection_AmrCore/Exec/Make.Adv Tests/Amr/Advection_AmrCore/Exec/Make.package Tests/Amr/Advection_AmrCore/Exec/Prob.H Tests/Amr/Advection_AmrCore/Exec/inputs Tests/Amr/Advection_AmrCore/Exec/inputs_for_scaling Tests/Amr/Advection_AmrCore/Exec/paraview_amr101.py Tests/Amr/Advection_AmrCore/README Tests/Amr/Advection_AmrCore/README.md Tests/Amr/Advection_AmrCore/Source/AdvancePhiAllLevels.cpp Tests/Amr/Advection_AmrCore/Source/AdvancePhiAtLevel.cpp Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tests/Amr/Advection_AmrCore/Source/DefineVelocity.cpp Tests/Amr/Advection_AmrCore/Source/Kernels.H Tests/Amr/Advection_AmrCore/Source/Make.package Tests/Amr/Advection_AmrCore/Source/Src_K/Adv_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/Make.package Tests/Amr/Advection_AmrCore/Source/Src_K/compute_flux_2D_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/compute_flux_3D_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/slope_K.H Tests/Amr/Advection_AmrCore/Source/Tagging.H Tests/Amr/Advection_AmrCore/Source/bc_fill.H Tests/Amr/Advection_AmrCore/Source/face_velocity.H Tests/Amr/Advection_AmrCore/Source/main.cpp Tests/Amr/Advection_AmrLevel/CMakeLists.txt Tests/Amr/Advection_AmrLevel/Exec/Make.Adv Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/GNUmakefile Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Make.package Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs.tracers Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/probin Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/GNUmakefile Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Make.package Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs.regt Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/probdata.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/probin Tests/Amr/Advection_AmrLevel/README Tests/Amr/Advection_AmrLevel/Source/Adv_F.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Amr/Advection_AmrLevel/Source/LevelBldAdv.cpp Tests/Amr/Advection_AmrLevel/Source/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_2d/Adv_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_2d/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_2d/compute_flux_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_2d/slope_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Adv_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_3d/compute_flux_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/slope_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/Adv_nd.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_nd/Tagging_nd.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/tagging_params.f90 Tests/Amr/Advection_AmrLevel/Source/main.cpp Tests/CMakeLists.txt Tests/CNS/CMakeLists.txt Tests/CNS/Exec/Combustor/GNUmakefile Tests/CNS/Exec/Combustor/Make.package Tests/CNS/Exec/Combustor/bc_fill_nd.F90 Tests/CNS/Exec/Combustor/bc_fill_nd.F90_jbb Tests/CNS/Exec/Combustor/cns_prob.F90 Tests/CNS/Exec/Combustor/cns_prob.F90_jbb Tests/CNS/Exec/Combustor/inputs Tests/CNS/Exec/Combustor/inputs.regt Tests/CNS/Exec/Make.CNS Tests/CNS/Exec/Pulse/GNUmakefile Tests/CNS/Exec/Pulse/Make.package Tests/CNS/Exec/Pulse/cns_prob.F90 Tests/CNS/Exec/Pulse/inputs Tests/CNS/Exec/Pulse/inputs.regt Tests/CNS/Exec/ShockRef/GNUmakefile Tests/CNS/Exec/ShockRef/Make.package Tests/CNS/Exec/ShockRef/cns_prob.F90 Tests/CNS/Exec/ShockRef/inputs Tests/CNS/Exec/ShockRef/inputs.amr Tests/CNS/Exec/ShockRef/inputs.regt Tests/CNS/Exec/Sod/GNUmakefile Tests/CNS/Exec/Sod/Make.package Tests/CNS/Exec/Sod/cns_prob.F90 Tests/CNS/Exec/Sod/inputs Tests/CNS/Source/CNS.H Tests/CNS/Source/CNS.cpp Tests/CNS/Source/CNSBld.cpp Tests/CNS/Source/CNS_F.H Tests/CNS/Source/CNS_advance.cpp Tests/CNS/Source/CNS_init_eb2.cpp Tests/CNS/Source/CNS_io.cpp Tests/CNS/Source/CNS_setup.cpp Tests/CNS/Source/Make.package Tests/CNS/Source/diffusion/Make.package Tests/CNS/Source/diffusion/cns_diff_mod.F90 Tests/CNS/Source/diffusion/cns_eb_diff_mod.F90 Tests/CNS/Source/diffusion/cns_eb_diff_wall.F90 Tests/CNS/Source/diffusion/diff_coef_mod.F90 Tests/CNS/Source/fortran/CNS_derive.F90 Tests/CNS/Source/fortran/CNS_divop.F90 Tests/CNS/Source/fortran/CNS_dudt.F90 Tests/CNS/Source/fortran/CNS_f.F90 Tests/CNS/Source/fortran/CNS_nd.F90 Tests/CNS/Source/fortran/CNS_physics.F90 Tests/CNS/Source/fortran/CNS_tagging.F90 Tests/CNS/Source/fortran/Make.package Tests/CNS/Source/fortran/bc_fill_nd.F90 Tests/CNS/Source/hydro/Hyp_gamma_MOL.F90 Tests/CNS/Source/hydro/Hyp_gamma_MOL_EB.F90 Tests/CNS/Source/hydro/Make.package Tests/CNS/Source/hydro/analriem3d.F90 Tests/CNS/Source/hydro/cns_eb_hyp_wall.F90 Tests/CNS/Source/hydro/slope_mol_3d_gamma.F90 Tests/CNS/Source/hydro/slope_mol_3d_gamma_EB.F90 Tests/CNS/Source/main.cpp Tests/FortranInterface/Advection_F/CMakeLists.txt Tests/FortranInterface/Advection_F/Exec/Make.Adv Tests/FortranInterface/Advection_F/Exec/SingleVortex/GNUmakefile Tests/FortranInterface/Advection_F/Exec/SingleVortex/Make.package Tests/FortranInterface/Advection_F/Exec/SingleVortex/Prob_2d.f90 Tests/FortranInterface/Advection_F/Exec/SingleVortex/Prob_3d.f90 Tests/FortranInterface/Advection_F/Exec/SingleVortex/face_velocity_2d.F90 Tests/FortranInterface/Advection_F/Exec/SingleVortex/face_velocity_3d.F90 Tests/FortranInterface/Advection_F/Exec/SingleVortex/inputs Tests/FortranInterface/Advection_F/Exec/SingleVortex/inputs.physbc Tests/FortranInterface/Advection_F/Exec/SingleVortex/inputs.rt Tests/FortranInterface/Advection_F/README Tests/FortranInterface/Advection_F/Source/Make.package Tests/FortranInterface/Advection_F/Source/Src_2d/Make.package Tests/FortranInterface/Advection_F/Source/Src_2d/advect_2d_mod.F90 Tests/FortranInterface/Advection_F/Source/Src_2d/compute_flux_2d.f90 Tests/FortranInterface/Advection_F/Source/Src_2d/slope_2d.f90 Tests/FortranInterface/Advection_F/Source/Src_3d/Make.package Tests/FortranInterface/Advection_F/Source/Src_3d/advect_3d_mod.F90 Tests/FortranInterface/Advection_F/Source/Src_3d/compute_flux_3d.f90 Tests/FortranInterface/Advection_F/Source/Src_3d/slope_3d.f90 Tests/FortranInterface/Advection_F/Source/amr_data_mod.F90 Tests/FortranInterface/Advection_F/Source/averagedown_mod.F90 Tests/FortranInterface/Advection_F/Source/bc_mod.F90 Tests/FortranInterface/Advection_F/Source/compute_dt_mod.F90 Tests/FortranInterface/Advection_F/Source/evolve_mod.F90 Tests/FortranInterface/Advection_F/Source/fillpatch_mod.F90 Tests/FortranInterface/Advection_F/Source/fmain.F90 Tests/FortranInterface/Advection_F/Source/initdata.F90 Tests/FortranInterface/Advection_F/Source/my_amr_mod.F90 Tests/FortranInterface/Advection_F/Source/plotfile_mod.F90 Tests/FortranInterface/Advection_F/Source/tagging_mod.F90 Tests/FortranInterface/Advection_octree_F/CMakeLists.txt Tests/FortranInterface/Advection_octree_F/Exec/Make.Adv Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/GNUmakefile Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/Make.package Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/Prob.f90 Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/face_velocity_2d.F90 Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/inputs Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/inputs.rt Tests/FortranInterface/Advection_octree_F/README Tests/FortranInterface/Advection_octree_F/Source/Make.package Tests/FortranInterface/Advection_octree_F/Source/Src_2d/Make.package Tests/FortranInterface/Advection_octree_F/Source/Src_2d/advect_2d_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/Src_2d/compute_flux_2d.f90 Tests/FortranInterface/Advection_octree_F/Source/Src_2d/slope_2d.f90 Tests/FortranInterface/Advection_octree_F/Source/amr_data_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/averagedown_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/bc_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/compute_dt_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/evolve_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/fillpatch_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/fmain.F90 Tests/FortranInterface/Advection_octree_F/Source/initdata.F90 Tests/FortranInterface/Advection_octree_F/Source/my_amr_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/plotfile_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/tagging_mod.F90 Tests/GPU/CNS/CMakeLists.txt Tests/GPU/CNS/Exec/Make.CNS Tests/GPU/CNS/Exec/RT/GNUmakefile Tests/GPU/CNS/Exec/RT/Make.package Tests/GPU/CNS/Exec/RT/cns_prob.H Tests/GPU/CNS/Exec/RT/cns_prob.cpp Tests/GPU/CNS/Exec/RT/cns_prob_parm.H Tests/GPU/CNS/Exec/RT/inputs Tests/GPU/CNS/Exec/RT/inputs-rt Tests/GPU/CNS/Exec/Sod/GNUmakefile Tests/GPU/CNS/Exec/Sod/Make.package Tests/GPU/CNS/Exec/Sod/cns_prob.H Tests/GPU/CNS/Exec/Sod/cns_prob.cpp Tests/GPU/CNS/Exec/Sod/cns_prob_parm.H Tests/GPU/CNS/Exec/Sod/inputs Tests/GPU/CNS/Exec/Sod/inputs-rt Tests/GPU/CNS/Source/CNS.H Tests/GPU/CNS/Source/CNS.cpp Tests/GPU/CNS/Source/CNSBld.cpp Tests/GPU/CNS/Source/CNS_K.H Tests/GPU/CNS/Source/CNS_advance.cpp Tests/GPU/CNS/Source/CNS_bcfill.cpp Tests/GPU/CNS/Source/CNS_derive.H Tests/GPU/CNS/Source/CNS_derive.cpp Tests/GPU/CNS/Source/CNS_index_macros.H Tests/GPU/CNS/Source/CNS_io.cpp Tests/GPU/CNS/Source/CNS_parm.H Tests/GPU/CNS/Source/CNS_parm.cpp Tests/GPU/CNS/Source/CNS_setup.cpp Tests/GPU/CNS/Source/CNS_tagging.H Tests/GPU/CNS/Source/Make.package Tests/GPU/CNS/Source/diffusion/Make.package Tests/GPU/CNS/Source/hydro/CNS_hydro_K.H Tests/GPU/CNS/Source/hydro/Make.package Tests/GPU/CNS/Source/main.cpp Tests/GPU/Vector/inputs Tests/HDF5Benchmark/inputs Tests/LinearSolvers/ABecLaplacian_C/CMakeLists.txt Tests/LinearSolvers/ABecLaplacian_F/CMakeLists.txt Tests/LinearSolvers/MAC_Projection_EB/CMakeLists.txt Tests/LinearSolvers/NodalPoisson/CMakeLists.txt Tests/LinearSolvers/Nodal_Projection_EB/CMakeLists.txt Tests/LinearSolvers/NodeTensorLap/CMakeLists.txt Tests/MultiBlock/IndexType/main.cpp Tests/Particles/ParallelContext/CMakeLists.txt Tests/Particles/ParallelContext/inputs.rt Tests/Particles/ParallelContext/main.cpp Tests/Particles/Redistribute/inputs Tests/Particles/Redistribute/inputs.rt Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-dpcpp-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini Tutorials/Basic/Build_with_libamrex/GNUmakefile Tutorials/Basic/Build_with_libamrex/MyParams.H Tutorials/Basic/Build_with_libamrex/main.cpp Tutorials/Basic/Build_with_libamrex/my_func.f90 Tutorials/Basic/Build_with_libamrex/test_parameters.cpp Tutorials/Basic/HeatEquation_EX1_C/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_C/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX1_C/Exec/inputs Tutorials/Basic/HeatEquation_EX1_C/Source/Make.package Tutorials/Basic/HeatEquation_EX1_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX1_C/Source/myfunc.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/mykernel.H Tutorials/Basic/HeatEquation_EX1_CF/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_CF/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX1_CF/Exec/inputs_2d Tutorials/Basic/HeatEquation_EX1_CF/Exec/inputs_3d Tutorials/Basic/HeatEquation_EX1_CF/Source/Make.package Tutorials/Basic/HeatEquation_EX1_CF/Source/advance.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/advance_2d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/advance_3d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/myfunc.H Tutorials/Basic/HeatEquation_EX1_CF/Source/myfunc_F.H Tutorials/Basic/HeatEquation_EX1_F/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_F/GNUmakefile Tutorials/Basic/HeatEquation_EX1_F/Make.package Tutorials/Basic/HeatEquation_EX1_F/advance.f90 Tutorials/Basic/HeatEquation_EX1_F/fmain.F90 Tutorials/Basic/HeatEquation_EX1_F/init_phi.f90 Tutorials/Basic/HeatEquation_EX1_F/inputs Tutorials/Basic/HeatEquation_EX2_C/CMakeLists.txt Tutorials/Basic/HeatEquation_EX2_C/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX2_C/Exec/inputs Tutorials/Basic/HeatEquation_EX2_C/Source/Make.package Tutorials/Basic/HeatEquation_EX2_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX2_C/Source/myfunc.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/mykernel.H Tutorials/Basic/HeatEquation_EX2_CF/CMakeLists.txt Tutorials/Basic/HeatEquation_EX2_CF/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX2_CF/Exec/inputs_2d Tutorials/Basic/HeatEquation_EX2_CF/Exec/inputs_3d Tutorials/Basic/HeatEquation_EX2_CF/Source/Make.package Tutorials/Basic/HeatEquation_EX2_CF/Source/advance.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/advance_2d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/advance_3d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/myfunc.H Tutorials/Basic/HeatEquation_EX2_CF/Source/myfunc_F.H Tutorials/Basic/HeatEquation_EX3_C/CMakeLists.txt Tutorials/Basic/HeatEquation_EX3_C/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX3_C/Exec/inputs_2d Tutorials/Basic/HeatEquation_EX3_C/Exec/inputs_3d Tutorials/Basic/HeatEquation_EX3_C/Source/Make.package Tutorials/Basic/HeatEquation_EX3_C/Source/advance.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX3_C/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX3_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX3_C/Source/myfunc_F.H Tutorials/Basic/HelloWorld_C/CMakeLists.txt Tutorials/Basic/HelloWorld_C/GNUmakefile Tutorials/Basic/HelloWorld_C/Make.package Tutorials/Basic/HelloWorld_C/main.cpp Tutorials/Basic/HelloWorld_F/CMakeLists.txt Tutorials/Basic/HelloWorld_F/GNUmakefile Tutorials/Basic/HelloWorld_F/Make.package Tutorials/Basic/HelloWorld_F/fmain.f90 Tutorials/Basic/PrefixSum_MultiFab/CMakeLists.txt Tutorials/Basic/PrefixSum_MultiFab/GNUmakefile Tutorials/Basic/PrefixSum_MultiFab/Make.package Tutorials/Basic/PrefixSum_MultiFab/Parallel-Prefix-Sum.ipynb Tutorials/Basic/PrefixSum_MultiFab/README.md Tutorials/Basic/PrefixSum_MultiFab/inputs Tutorials/Basic/PrefixSum_MultiFab/main.cpp Tutorials/Basic/main_C/CMakeLists.txt Tutorials/Basic/main_C/GNUmakefile Tutorials/Basic/main_C/Make.package Tutorials/Basic/main_C/main.cpp Tutorials/Basic/main_F/CMakeLists.txt Tutorials/Basic/main_F/GNUmakefile Tutorials/Basic/main_F/Make.package Tutorials/Basic/main_F/main.F90 Tutorials/Blueprint/AssignMultiLevelDensity/CMakeLists.txt Tutorials/Blueprint/AssignMultiLevelDensity/GNUmakefile Tutorials/Blueprint/AssignMultiLevelDensity/Make.package Tutorials/Blueprint/AssignMultiLevelDensity/inputs Tutorials/Blueprint/AssignMultiLevelDensity/main.cpp Tutorials/Blueprint/CellSortedParticles/CMakeLists.txt Tutorials/Blueprint/CellSortedParticles/CellSortedPC.H Tutorials/Blueprint/CellSortedParticles/CellSortedPC.cpp Tutorials/Blueprint/CellSortedParticles/GNUmakefile Tutorials/Blueprint/CellSortedParticles/Make.package Tutorials/Blueprint/CellSortedParticles/cell_sorted_3d.F90 Tutorials/Blueprint/CellSortedParticles/cell_sorted_F.H Tutorials/Blueprint/CellSortedParticles/inputs Tutorials/Blueprint/CellSortedParticles/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/CMakeLists.txt Tutorials/Blueprint/HeatEquation_EX1_C/Exec/GNUmakefile Tutorials/Blueprint/HeatEquation_EX1_C/Exec/inputs_2d Tutorials/Blueprint/HeatEquation_EX1_C/Exec/inputs_3d Tutorials/Blueprint/HeatEquation_EX1_C/Source/Make.package Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance_2d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance_3d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/init_phi_2d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/init_phi_3d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/myfunc.H Tutorials/Blueprint/HeatEquation_EX1_C/Source/myfunc_F.H Tutorials/Blueprint/README.txt Tutorials/CMakeLists.txt Tutorials/EB/GeometryGeneration/CMakeLists.txt Tutorials/EB/GeometryGeneration/GNUmakefile Tutorials/EB/GeometryGeneration/Make.package Tutorials/EB/GeometryGeneration/main.cpp Tutorials/EB/MacProj/CMakeLists.txt Tutorials/EB/MacProj/GNUmakefile Tutorials/EB/MacProj/Make.package Tutorials/EB/MacProj/inputs Tutorials/EB/MacProj/main.cpp Tutorials/EB/Poisson/CMakeLists.txt Tutorials/EB/Poisson/GNUmakefile Tutorials/EB/Poisson/Make.package Tutorials/EB/Poisson/Poisson.H Tutorials/EB/Poisson/Poisson.cpp Tutorials/EB/Poisson/inputs Tutorials/EB/Poisson/main.cpp Tutorials/EB/STLtest/GNUmakefile Tutorials/EB/STLtest/Make.package Tutorials/EB/STLtest/airfoil.stl Tutorials/EB/STLtest/inputs Tutorials/EB/STLtest/main.cpp Tutorials/ForkJoin/MLMG/CMakeLists.txt Tutorials/ForkJoin/MLMG/GNUmakefile Tutorials/ForkJoin/MLMG/Make.package Tutorials/ForkJoin/MLMG/ff.f90 Tutorials/ForkJoin/MLMG/inputs Tutorials/ForkJoin/MLMG/main.cpp Tutorials/ForkJoin/Simple/CMakeLists.txt Tutorials/ForkJoin/Simple/GNUmakefile Tutorials/ForkJoin/Simple/Make.package Tutorials/ForkJoin/Simple/MyTest.H Tutorials/ForkJoin/Simple/MyTest.cpp Tutorials/ForkJoin/Simple/MyTest_F.H Tutorials/ForkJoin/Simple/inputs Tutorials/ForkJoin/Simple/main.cpp Tutorials/FortranInterface/Advection_octree_F2/CMakeLists.txt Tutorials/FortranInterface/Advection_octree_F2/Exec/Make.Adv Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/GNUmakefile Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/Make.package Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/Prob.f90 Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/face_velocity_2d.F90 Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/inputs Tutorials/FortranInterface/Advection_octree_F2/README Tutorials/FortranInterface/Advection_octree_F2/Source/Make.package Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/Make.package Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/advect_2d_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/compute_flux_2d.f90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/slope_2d.f90 Tutorials/FortranInterface/Advection_octree_F2/Source/amr_data_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/averagedown_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/bc_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/compute_dt_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/evolve_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/fillpatch_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/fmain.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/initdata.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/my_amr_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/plotfile_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/tagging_mod.F90 Tutorials/GPU/EBCNS/CMakeLists.txt Tutorials/GPU/EBCNS/Exec/Make.CNS Tutorials/GPU/EBCNS/Exec/Sod/GNUmakefile Tutorials/GPU/EBCNS/Exec/Sod/Make.package Tutorials/GPU/EBCNS/Exec/Sod/cns_prob.H Tutorials/GPU/EBCNS/Exec/Sod/cns_prob.cpp Tutorials/GPU/EBCNS/Exec/Sod/cns_prob_parm.H Tutorials/GPU/EBCNS/Exec/Sod/inputs Tutorials/GPU/EBCNS/README Tutorials/GPU/EBCNS/Source/CNS.H Tutorials/GPU/EBCNS/Source/CNS.cpp Tutorials/GPU/EBCNS/Source/CNSBld.cpp Tutorials/GPU/EBCNS/Source/CNS_K.H Tutorials/GPU/EBCNS/Source/CNS_advance.cpp Tutorials/GPU/EBCNS/Source/CNS_bcfill.cpp Tutorials/GPU/EBCNS/Source/CNS_derive.H Tutorials/GPU/EBCNS/Source/CNS_derive.cpp Tutorials/GPU/EBCNS/Source/CNS_index_macros.H Tutorials/GPU/EBCNS/Source/CNS_init_eb2.cpp Tutorials/GPU/EBCNS/Source/CNS_io.cpp Tutorials/GPU/EBCNS/Source/CNS_parm.H Tutorials/GPU/EBCNS/Source/CNS_parm.cpp Tutorials/GPU/EBCNS/Source/CNS_setup.cpp Tutorials/GPU/EBCNS/Source/CNS_tagging.H Tutorials/GPU/EBCNS/Source/Make.package Tutorials/GPU/EBCNS/Source/diffusion/Make.package Tutorials/GPU/EBCNS/Source/hydro/CNS_hydro_K.H Tutorials/GPU/EBCNS/Source/hydro/Make.package Tutorials/GPU/EBCNS/Source/main.cpp Tutorials/GPU/Launch/CMakeLists.txt Tutorials/GPU/Launch/GNUmakefile Tutorials/GPU/Launch/Make.package Tutorials/GPU/Launch/MyKernel_F.F90 Tutorials/GPU/Launch/MyKernel_F.H Tutorials/GPU/Launch/Readme.md Tutorials/GPU/Launch/main.cpp Tutorials/GPU/ParallelReduce/CMakeLists.txt Tutorials/GPU/ParallelReduce/GNUmakefile Tutorials/GPU/ParallelReduce/Make.package Tutorials/GPU/ParallelReduce/main.cpp Tutorials/GPU/ParallelScan/CMakeLists.txt Tutorials/GPU/ParallelScan/GNUmakefile Tutorials/GPU/ParallelScan/Make.package Tutorials/GPU/ParallelScan/main.cpp Tutorials/GPU/run.corigpu Tutorials/GPU/run.summit Tutorials/MUI/Exec_01/GNUmakefile Tutorials/MUI/Exec_02/GNUmakefile Tutorials/MUI/Exec_coupled/cmd_mpirun Tutorials/MUI/Exec_coupled/inputs Tutorials/MUI/Source_01/Make.package Tutorials/MUI/Source_01/init_phi_3d.f90 Tutorials/MUI/Source_01/main_01.cpp Tutorials/MUI/Source_01/myfunc.H Tutorials/MUI/Source_01/myfunc_F.H Tutorials/MUI/Source_02/Make.package Tutorials/MUI/Source_02/main_02.cpp Tutorials/MUI/Source_02/myfunc.H Tutorials/MUI/doc/GNUmakefile Tutorials/MUI/doc/MUIcouplingNotes.tex Tutorials/MUI/doc/iface_rect.png Tutorials/MUI/doc/vis_interface.m Tutorials/Particles/CellSortedParticles/CMakeLists.txt Tutorials/Particles/CellSortedParticles/CellSortedPC.H Tutorials/Particles/CellSortedParticles/CellSortedPC.cpp Tutorials/Particles/CellSortedParticles/GNUmakefile Tutorials/Particles/CellSortedParticles/Make.package Tutorials/Particles/CellSortedParticles/cell_sorted_3d.F90 Tutorials/Particles/CellSortedParticles/cell_sorted_F.H Tutorials/Particles/CellSortedParticles/inputs Tutorials/Particles/CellSortedParticles/main.cpp Tutorials/Particles/ElectromagneticPIC/CMakeLists.txt Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/Evolve.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/GNUmakefile Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/Make.package Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/em_pic_K.H Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/inputs Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/Evolve.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/GNUmakefile Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/Make.package Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/em_pic_3d.F90 Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/em_pic_F.H Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/inputs Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/script.sh Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/Evolve.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/GNUmakefile Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/GNUmakefile.libamrex Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/Make.package Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/em_pic_3d.F90 Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/em_pic_F.H Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/inputs Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/run.summitdev Tutorials/Particles/ElectromagneticPIC/Make.EMPIC Tutorials/Particles/ElectromagneticPIC/Source/Constants.H Tutorials/Particles/ElectromagneticPIC/Source/EMParticleContainer.H Tutorials/Particles/ElectromagneticPIC/Source/EMParticleContainerInit.cpp Tutorials/Particles/ElectromagneticPIC/Source/Evolve.H Tutorials/Particles/ElectromagneticPIC/Source/IO.H Tutorials/Particles/ElectromagneticPIC/Source/IO.cpp Tutorials/Particles/ElectromagneticPIC/Source/Make.package Tutorials/Particles/ElectromagneticPIC/Source/NodalFlags.H Tutorials/Particles/ElectromagneticPIC/Source/NodalFlags.cpp Tutorials/Particles/ElectromagneticPIC/Source/main.cpp Tutorials/Particles/ElectrostaticPIC/CMakeLists.txt Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.H Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.cpp Tutorials/Particles/ElectrostaticPIC/GNUmakefile Tutorials/Particles/ElectrostaticPIC/Make.package Tutorials/Particles/ElectrostaticPIC/PhysConst.H Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_2d.f90 Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_3d.f90 Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_F.H Tutorials/Particles/ElectrostaticPIC/inputs Tutorials/Particles/ElectrostaticPIC/main.cpp Tutorials/Particles/NeighborList/CMakeLists.txt Tutorials/Particles/NeighborList/CheckPair.H Tutorials/Particles/NeighborList/Constants.H Tutorials/Particles/NeighborList/GNUmakefile Tutorials/Particles/NeighborList/MDParticleContainer.H Tutorials/Particles/NeighborList/MDParticleContainer.cpp Tutorials/Particles/NeighborList/Make.package Tutorials/Particles/NeighborList/README.md Tutorials/Particles/NeighborList/inputs Tutorials/Particles/NeighborList/main.cpp Tutorials/Particles/NeighborList/script.sh Tutorials/README.md Tutorials/SDC/MISDC_ADR_2d/Exec/GNUmakefile Tutorials/SDC/MISDC_ADR_2d/Exec/inputs_2d Tutorials/SDC/MISDC_ADR_2d/README Tutorials/SDC/MISDC_ADR_2d/Source/Make.package Tutorials/SDC/MISDC_ADR_2d/Source/SDC_sweeper.cpp Tutorials/SDC/MISDC_ADR_2d/Source/functions_2d.f90 Tutorials/SDC/MISDC_ADR_2d/Source/init_phi_2d.f90 Tutorials/SDC/MISDC_ADR_2d/Source/main.cpp Tutorials/SDC/MISDC_ADR_2d/Source/myfunc.H Tutorials/SDC/MISDC_ADR_2d/Source/myfunc_F.H Tutorials/SWFFT/SWFFT_poisson/GNUmakefile Tutorials/SWFFT/SWFFT_poisson/Make.package Tutorials/SWFFT/SWFFT_poisson/README Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test.H Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test.cpp Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test_F.F90 Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test_F.H Tutorials/SWFFT/SWFFT_poisson/inputs.128 Tutorials/SWFFT/SWFFT_poisson/inputs.32 Tutorials/SWFFT/SWFFT_poisson/inputs.64 Tutorials/SWFFT/SWFFT_poisson/main.cpp Tutorials/SWFFT/SWFFT_poisson/run_me Tutorials/SWFFT/SWFFT_poisson/swfft_solver.cpp Tutorials/SWFFT/SWFFT_simple/GNUmakefile Tutorials/SWFFT/SWFFT_simple/Make.package Tutorials/SWFFT/SWFFT_simple/README Tutorials/SWFFT/SWFFT_simple/SWFFT_Test.H Tutorials/SWFFT/SWFFT_simple/SWFFT_Test.cpp Tutorials/SWFFT/SWFFT_simple/SWFFT_Test_F.F90 Tutorials/SWFFT/SWFFT_simple/SWFFT_Test_F.H Tutorials/SWFFT/SWFFT_simple/inputs.multipleGrids Tutorials/SWFFT/SWFFT_simple/inputs.oneGrid Tutorials/SWFFT/SWFFT_simple/main.cpp Tutorials/SWFFT/SWFFT_simple/run_me_2d Tutorials/SWFFT/SWFFT_simple/run_me_3d Tutorials/SWFFT/SWFFT_simple/swfft_compute.cpp commit 384ca2341889a8f1db78e87cb4b1735bd2d58554 Author: Axel Huebl Date: Mon Mar 22 10:09:53 2021 -0700 CMake: IPO/LTO (#1890) Add optional interprocedural optimization (IPO) aka link-time optimization (LTO) to AMReX. In my WarpX tests recently on x86_64 and ppc64le, using those options incurred a 2% performance hit (although one would expect a slight increase). Thus, they are not enabled by default. Also, enabling them with some toolchains needs users to set the linker properly. Nontheless, enabling IPO shrinks the binary size a lot. This is very valuable and worth the runtime penalty for our binary deployments, e.g. when shipping generic x86_64, aarch64 and ppc64le binaries on conda-forge to users. WarpX binary size (default build): * no IPO: 217MiB * WarpX IPO, AMReX no IPO: 155MiB * both WarpX and AMReX with IPO: 127M Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXBuildInfo.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXTypecheck.cmake Tools/CMake/AMReX_Config.cmake commit ef0eb9f5c4b065b38e3a3242a43263aba43aa874 Author: Weiqun Zhang Date: Sun Mar 21 14:35:07 2021 -0700 Header file order and iosfwd (#1883) Include amrex headers before standard C++ headers. Use iosfwd instead of iostream if we can. Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Derive.cpp Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.H Src/Base/AMReX_Array.H Src/Base/AMReX_Array4.H Src/Base/AMReX_BCRec.cpp Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_BaseFab.cpp Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxDomain.H Src/Base/AMReX_BoxDomain.cpp Src/Base/AMReX_BoxIterator.H Src/Base/AMReX_BoxIterator.cpp Src/Base/AMReX_BoxList.H Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_CoordSys.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_Dim3.H Src/Base/AMReX_Dim3.cpp Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabConv.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuComplex.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_IndexType.H Src/Base/AMReX_IndexType.cpp Src/Base/AMReX_IntConv.H Src/Base/AMReX_IntVect.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MemProfiler.H Src/Base/AMReX_MemProfiler.cpp Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_Orientation.H Src/Base/AMReX_Orientation.cpp Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_Periodicity.cpp Src/Base/AMReX_PlotFileDataImpl.H Src/Base/AMReX_PlotFileDataImpl.cpp Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_PlotFileUtil.cpp Src/Base/AMReX_Print.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_RealVect.H Src/Base/AMReX_RealVect.cpp Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_Tuple.H Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Utility.H Src/Base/AMReX_Vector.H Src/Base/AMReX_VectorIO.H Src/Base/AMReX_VectorIO.cpp Src/Base/AMReX_VisMF.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_iMultiFab.cpp Src/Base/AMReX_parmparse_fi.cpp Src/Base/AMReX_parstream.H Src/Base/AMReX_parstream.cpp Src/Base/CMakeLists.txt Src/Base/Make.package Src/Boundary/AMReX_Mask.cpp Src/EB/AMReX_EB2_IF_Base.H Src/EB/AMReX_EB2_IF_Spline.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBCellFlag.cpp Src/EB/AMReX_EBToPVD.H Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_distFcnElement.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/F_Interfaces/AmrCore/AMReX_FAmrCore.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleMPIUtil.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.cpp commit c984473e021aabdd5f5d2091dcc57481e8637c7c Author: Weiqun Zhang Date: Sun Mar 21 11:38:21 2021 -0700 Remove outdated notes (#1889) Docs/Notes/DPCPPWishlist.md Docs/Notes/HIPIssues.md Docs/Notes/Notes.io_implementation commit 15f5168a1ad9f49d87ff0ee25935e171fe1d0f19 Author: Weiqun Zhang Date: Fri Mar 19 20:32:22 2021 -0700 amrex::EnableIf_t -> std::enable_if_t (#1885) Since we require C++14 now, we can use std::enable_if_t now. Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_TypeTraits.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H commit 1dfb98c29afe574afe30101a8db82190d9eadaa6 Author: Andrew Myers Date: Fri Mar 19 23:31:57 2021 -0400 remove __attribute__((warn_unused_result)) from AMREX_NODISCARD (#1886) Src/Base/AMReX_Extension.H commit 352059135e3e87f3b35e237b1c7fb8961eabffa1 Author: Weiqun Zhang Date: Fri Mar 19 18:11:52 2021 -0700 Make FabArray::setFab safer (#1884) Instead of taking a raw pointer, it takes std::unique_ptr or rvalue FAB, so that the ownership is very clear. It also deletes the old FAB if not null. Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_ParallelDescriptor.H Src/EB/AMReX_EB2_MultiGFab.cpp Src/EB/AMReX_MultiCutFab.cpp Src/Extern/amrdata/AMReX_AmrData.cpp commit 9e1576fab02e20b19d54105e6e7a434bf6d6f040 Author: Weiqun Zhang Date: Fri Mar 19 17:00:02 2021 -0700 Fix typo in AMREX_DEBUG macro (#1881) Fix the call to CheckRcvStats. Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H commit ea9ba93abd20c9a4c79f2ae8c392cd62a553119f Author: Maikel Nadolski Date: Fri Mar 19 23:23:51 2021 +0100 Forward declare functions that are used in the implementation (#1882) This PR forward declares the functions local_copy_cpu unpack_recv_buffer_cpu local_copy_gpu unpack_recv_buffer_gpu before its first usage. Some compilers error out if a function is used before it was declared, even in template code. I am not sure, why it worked in the CI so far. Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H commit ca1839a8d9d805e915d6bd9d385a0c3087ad866d Author: Axel Huebl Date: Fri Mar 19 12:44:33 2021 -0700 Flags: RelWithDebInfo (#1879) Add warning control to CMake Build mode `RelWithDebugInfo` (currently default for WarpX). Tools/CMake/AMReXFlagsTargets.cmake commit 82664e56d3dbdefa75bec47ddd871996778805c3 Author: Axel Huebl Date: Fri Mar 19 08:47:04 2021 -0700 Mask: -Wfinal-dtor-non-final-class (#1880) ## Summary ``` Src/Boundary/AMReX_Mask.H:57:40: warning: class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class] virtual ~Mask () noexcept override final {} ^ Src/Boundary/AMReX_Mask.H:26:7: note: mark 'amrex::Mask' as 'final' to silence this warning class Mask ``` Fix the warning in TagBox, CutFab and EBCellFlagFab as well. ## Additional background Seen with Clang 10.0 on Ubuntu 20.04 LTS. Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_TagBox.H Src/Boundary/AMReX_Mask.H Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_MultiCutFab.H commit 0e1acd6c559b8939ac323446963dfafbf7430052 Author: Andrew Myers Date: Fri Mar 19 01:50:12 2021 -0400 Fix Typedescriptor test for Windows (#1878) Tests/Particles/TypeDescriptor/main.cpp commit 281ff0fb583047355e0a726805c08d5d4e20cf67 Author: Axel Huebl Date: Thu Mar 18 11:42:30 2021 -0700 Build: Windows with Shared Libs (.dll) (#1847) * CI: Win with Shared Libs (.dll) Demonstrator of shared library issues on Windows. * Remove: warn-unresolved-symbols / undefined,warning * CMake typecheckobjs lib: export symbols Make this library behave like on Unix: symbol visibility by default. * Globals: dllimport/dllexport (Win) https://stackoverflow.com/questions/54560832/cmake-windows-export-all-symbols-does-not-cover-global-variables/54568678#54568678 .github/workflows/windows.yml Src/AmrCore/AMReX_Interpolater.H Src/Base/AMReX.H Src/Base/AMReX_Extension.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_VisMF.H Src/EB/AMReX_EBInterpolater.H Src/Particle/AMReX_ParticleContainerBase.H Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXTypecheck.cmake Tools/CMake/AMReX_Config.H.in Tools/CMake/AMReX_Config.cmake Tools/GNUMake/tools/Make.sensei commit 17371b17ffd5f7794a91236de7fa464d4284fa86 Author: Andrew Myers Date: Thu Mar 18 13:24:45 2021 -0400 Remove tutorials documentation from main repo; it now lives at https://github.com/AMReX-Codes/amrex-tutorials (#1877) Docs/sphinx_tutorials/Makefile Docs/sphinx_tutorials/source/AMR_Tutorial.rst Docs/sphinx_tutorials/source/Basic_Tutorial.rst Docs/sphinx_tutorials/source/Blueprint_Tutorial.rst Docs/sphinx_tutorials/source/EB_Tutorial.rst Docs/sphinx_tutorials/source/ForkJoin_Tutorial.rst Docs/sphinx_tutorials/source/GPU_Tutorial.rst Docs/sphinx_tutorials/source/LinearSolvers_Tutorial.rst Docs/sphinx_tutorials/source/MUI_Tutorial.rst Docs/sphinx_tutorials/source/Particles_Tutorial.rst Docs/sphinx_tutorials/source/SDC_Tutorial.rst Docs/sphinx_tutorials/source/SWFFT/iface_rect.png Docs/sphinx_tutorials/source/SWFFT_Tutorial.rst Docs/sphinx_tutorials/source/_static/theme_overrides.css Docs/sphinx_tutorials/source/conf.py Docs/sphinx_tutorials/source/figs/fork_join_tasks.png Docs/sphinx_tutorials/source/figs/mf_remap_hires.png Docs/sphinx_tutorials/source/figs/nested_fork_join_tasks.png Docs/sphinx_tutorials/source/index.rst build_docs.sh commit b31d050d176c822d51dad418a868d62426c6a1b4 Author: Weiqun Zhang Date: Wed Mar 17 19:13:57 2021 -0700 2D Poisson and ALaplacian Solvers in 3D Build of AMReX (#1805) New capability of solving 2D cell-centered Poisson's and ALaplacian equation with 3D build of amrex. Src/Base/AMReX_Box.H Src/Base/AMReX_Box.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_K.H Src/LinearSolvers/MLMG/Make.package commit 14b6bff6954d677b496af065c8c9def6bacc2d43 Author: Weiqun Zhang Date: Wed Mar 17 18:26:39 2021 -0700 Add FabArray::release and Make BaseFab virtual (#1873) Add FabArray::release functions that release the ownership of the FAB at the given index and return a raw pointer to the FAB. Make BaseFab virtual so that it's safe to use the new release functions. Also declare a number of functions `[[nodiscard]]`. Fix some warnings in EB/CNS example. Src/AmrCore/AMReX_TagBox.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabFactory.H Src/Base/AMReX_IArrayBox.H Src/Boundary/AMReX_Mask.H Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBFabFactory.H Src/EB/AMReX_EBFabFactory.cpp Src/EB/AMReX_MultiCutFab.H Tutorials/EB/CNS/Source/CNS.cpp Tutorials/EB/CNS/Source/CNS_advance.cpp Tutorials/EB/CNS/Source/CNS_init_eb2.cpp commit 87b182a718574cc52a93750a4c484c2e5aa4264b Author: Weiqun Zhang Date: Wed Mar 17 15:16:57 2021 -0700 Robin BC in Linear Solver (#1848) Add Robin BC to MLABecLaplacian. There are two steps for setting up Robin BC. The first step is to call setDomainBC, and the second step is to call setLevelBC, which has been modified to take a, b and f in Robin BC, `a*phi + b*dphi/dn = f`. Docs/sphinx_documentation/source/LinearSolvers.rst Src/Boundary/AMReX_LO_BCTYPES.H Src/Boundary/AMReX_LO_BCTYPES.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit 5b235f13980b2602636eff689f95d96a99c75d06 Author: Weiqun Zhang Date: Wed Mar 17 10:06:45 2021 -0700 LevelBld (#1852) In Amr class, we need to get LevelBld* and use it to build AmrLevels. Previously, Amr called a function `LevelBld* getLevelBld()` to get that. The problem was that `getLevelBld` was not defined by amrex but it was being called by amrex. This caused issues in building amrex as a shared library on Mac and Windows, because they do not support weak symbols. In this PR, the constructors of Amr now take `LevelBld*` as an argument. This is unfortunately a breaking change. However, it's a compile time failure and it's easy to fix. See the changes in this commit for examples. Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_LevelBld.H Tutorials/Amr/Advection_AmrLevel/Source/main.cpp Tutorials/EB/CNS/Source/main.cpp Tutorials/GPU/CNS/Source/main.cpp Tutorials/GPU/EBCNS/Source/main.cpp commit 5d32c74750b59e06c5731e084a75eec6ab91aad9 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Wed Mar 17 11:24:36 2021 -0400 Fix bug with face-centroid slopes when using multigrid (#1871) When computing the Least Square slopes on face-centroid stencil with multigrid, we need to grow `m_eb_phi` and make additional checks. This adds to the changes made in https://github.com/AMReX-Codes/amrex/pull/1707 Src/EB/AMReX_EB_LeastSquares_2D_K.H Src/EB/AMReX_EB_LeastSquares_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Tests/LinearSolvers/LeastSquares/inputs.2d.askew-x.mg Tests/LinearSolvers/LeastSquares/inputs.2d.askew-y.mg Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-all.mg commit c28d91c2ee07ad6f970eb10bbc14d00af6adab4a Author: Weiqun Zhang Date: Tue Mar 16 14:59:05 2021 -0700 Avoid conflict with X11 (#1872) Bool -> B to avoid a conflict with an X11 header that defines ``` #define Bool int #define Status int #define True 1 #define False 0 ``` This should fix the compilation issue of Amrvis. Src/Base/AMReX_TypeTraits.H commit 44f77978de2bdaec40cc6136169b854a78ff167e Author: Andrew Myers Date: Tue Mar 16 11:23:18 2021 -0400 Add explanation of parameters to Redistribute docstring (#1868) Src/Particle/AMReX_Particles.H commit e1b123b2302c0377efb9ab99066a60f907c663c9 Author: Weiqun Zhang Date: Mon Mar 15 18:05:17 2021 -0700 Fix sign-compare and shadow warnings (#1867) The warning about shadowed varible in AMReX_NonLocalBC.H seems a compiler bug. Nevertheless, the source is modified to get rid of the warning. Src/Base/AMReX_NonLocalBC.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H commit 7d833e64ff06477953c9b3a2642b04e61116be93 Author: Weiqun Zhang Date: Mon Mar 15 12:42:05 2021 -0700 EditorConfig and trailing whitespaces (#1866) Trim trailing whitespaces in editorconfig for C/C++ codes and others. Allow md file to have trailing whitespaces because they are syntactically significant. .editorconfig .github/workflows/style/check_trailing_whitespaces.sh commit e5b06d59e3a6cf55c40c6498e42eafe940203cb9 Author: Weiqun Zhang Date: Mon Mar 15 12:28:06 2021 -0700 Optimization of CArena::freeUnused (#1865) Use `find` instead of manully looping over the set. Src/Base/AMReX_CArena.cpp commit fbef03f566fbbe3c583bd96db00b5cee391ad6b8 Author: Axel Huebl Date: Mon Mar 15 12:25:52 2021 -0700 CI: macOS Shared Lib (#1836) ## Summary CI for macOS: Build a shared library. This is the default in Spack. I see issues with AppleClang 12.0 raising: ``` ld: can't use -undefined warning or suppress with -twolevel_namespace clang: error: linker command failed with exit code 1 (use -v to see invocation) ``` @RemiLehe reported this again here: https://github.com/AMReX-Codes/amrex/issues/425#issuecomment-473153152 - [x] reproduce issue - [x] add fix ## Additional background Introduced in 1b6af3cb630 to fix #425 .github/workflows/dependencies/dependencies_mac.sh .github/workflows/macos.yml Src/Amr/AMReX_LevelBld.H Src/Amr/AMReX_PROB_AMR_F.H Src/Base/AMReX_Extension.H Tools/CMake/AMReX_Config.cmake commit 2789d017a7d1e4803fe9ca007dbbad611e42069f Author: Weiqun Zhang Date: Sun Mar 14 14:39:58 2021 -0700 Fix bug in #1861 (#1864) We cannot simply use std::set::erase on the free node list because the operator== only compares the starting block address. Src/Base/AMReX_CArena.cpp commit 107b5d4f36063b110539242928faba914f07e5b3 Author: Weiqun Zhang Date: Sun Mar 14 14:25:49 2021 -0700 Fix CUDA API calls in host callback functions (#1863) CUDA API calls are not allowed in host callback functions. Therefore, we have to move the call to free unused memory from `free` to `alloc`. Src/Base/AMReX_CArena.cpp commit d28caf8685f6c65639aee1d7d1731901480b3a14 Author: Weiqun Zhang Date: Sun Mar 14 14:19:48 2021 -0700 CI for Code Style (#1862) Check for tabs and trailing whitespaces at the end of line in CI. .github/workflows/style.yml .github/workflows/style/check_tabs.sh .github/workflows/style/check_trailing_whitespaces.sh .gitignore commit 486cd438aea970962a02c799c910eb8ddcd7dd71 Author: CCSE@LBNL Date: Sun Mar 14 08:38:31 2021 -0700 Remove tabs and trailing white spaces (#1860) The changes are purely white spaces only. `git diff -w development..HEAD` shows nothing. It was done with ``` find . -type d \( -name .git \ -o -path ./paper \ \) -prune -o \ -type f \( -name "*.H" -o -name "*.h" -o -name "*.hh" -o -name "*.hpp" \ -o -name "*.c" -o -name "*.cc" -o -name "*.cpp" -o -name "*.cxx" \ -o -name "*.f" -o -name "*.F" -o -name "*.f90" -o -name "*.F90" \ -o -name "*.py" \ -o -name "*.md" -o -name "*.rst" \ -o -name "*.sh" \ -o -name "*.tex" \ -o -name "*.txt" \ \) \ -exec grep -Iq . {} \; \ -exec sed -i 's/[[:blank:]]\+$//g' {} + \ -exec sed -i 's/\t/\ \ \ \ \ \ \ \ /g' {} + ``` We will add a CI check as a follow-up. CONTRIBUTING.md Docs/Migration/Migration.md Docs/sphinx_documentation/source/AMReX_Profiling_Tools_Chapter.rst Docs/sphinx_documentation/source/AmrCore.rst Docs/sphinx_documentation/source/AmrLevel.rst Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/DualGrid.rst Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/External_Profiling_Tools.rst Docs/sphinx_documentation/source/Fortran.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Docs/sphinx_documentation/source/GettingStarted.rst Docs/sphinx_documentation/source/GridCreation.rst Docs/sphinx_documentation/source/IO.rst Docs/sphinx_documentation/source/InputsLoadBalancing.rst Docs/sphinx_documentation/source/Inputs_Chapter.rst Docs/sphinx_documentation/source/Introduction.rst Docs/sphinx_documentation/source/LinearSolvers.rst Docs/sphinx_documentation/source/LinearSolvers_Chapter.rst Docs/sphinx_documentation/source/LoadBalancing.rst Docs/sphinx_documentation/source/ManagingGridHierarchy_Chapter.rst Docs/sphinx_documentation/source/Particle.rst Docs/sphinx_documentation/source/SWFFT.rst Docs/sphinx_documentation/source/Testing.rst Docs/sphinx_documentation/source/Visualization.rst Docs/sphinx_documentation/source/index.rst Docs/sphinx_tutorials/source/AMR_Tutorial.rst Docs/sphinx_tutorials/source/Basic_Tutorial.rst Docs/sphinx_tutorials/source/Blueprint_Tutorial.rst Docs/sphinx_tutorials/source/GPU_Tutorial.rst Docs/sphinx_tutorials/source/LinearSolvers_Tutorial.rst Docs/sphinx_tutorials/source/MUI_Tutorial.rst Docs/sphinx_tutorials/source/Particles_Tutorial.rst Docs/sphinx_tutorials/source/index.rst README.md Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_AuxBoundaryData.H Src/Amr/AMReX_AuxBoundaryData.cpp Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Derive.cpp Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/Amr/AMReX_extrapolater_3D_K.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_AmrParGDB.H Src/AmrCore/AMReX_AmrParticles.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_2d.F90 Src/AmrCore/AMReX_FillPatchUtil_3d.F90 Src/AmrCore/AMReX_FillPatchUtil_F.H Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_BCRec.cpp Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_BLutil_F.F90 Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseFab.cpp Src/Base/AMReX_BaseUmap_f.H Src/Base/AMReX_BlockMutex.H Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_Box.H Src/Base/AMReX_Box.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxDomain.cpp Src/Base/AMReX_BoxList.H Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_COORDSYS_1D_C.H Src/Base/AMReX_COORDSYS_2D_C.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_CuptiTrace.H Src/Base/AMReX_CuptiTrace.cpp Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_Extension.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FILCC_1D.F90 Src/Base/AMReX_FILCC_2D.F90 Src/Base/AMReX_FILCC_3D.F90 Src/Base/AMReX_FPC.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_FabConv.cpp Src/Base/AMReX_FileSystem.cpp Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuComplex.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_INT.H Src/Base/AMReX_IntConv.H Src/Base/AMReX_IntConv.cpp Src/Base/AMReX_IntVect.H Src/Base/AMReX_Lazy.H Src/Base/AMReX_Lazy.cpp Src/Base/AMReX_MFCopyDescriptor.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MemProfiler.H Src/Base/AMReX_MemProfiler.cpp Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBC.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParallelDescriptor_F.F90 Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_Partition.H Src/Base/AMReX_Periodicity.H Src/Base/AMReX_Periodicity.cpp Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_PlotFileUtil.cpp Src/Base/AMReX_Print.H Src/Base/AMReX_REAL.H Src/Base/AMReX_RealBox.cpp Src/Base/AMReX_RealVect.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_SPACE.H Src/Base/AMReX_Scan.H Src/Base/AMReX_Slopes_K.H Src/Base/AMReX_ThirdPartyProfiling.H Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Utility.H Src/Base/AMReX_Utility.cpp Src/Base/AMReX_Vector.H Src/Base/AMReX_VectorIO.H Src/Base/AMReX_VectorIO.cpp Src/Base/AMReX_VisMF.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_bc_types_mod.F90 Src/Base/AMReX_ccse-mpi.H Src/Base/AMReX_error_fi.cpp Src/Base/AMReX_error_mod.F90 Src/Base/AMReX_filcc_mod.F90 Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_iMultiFab.cpp Src/Base/AMReX_mempool_mod.F90 Src/Base/AMReX_parmparse_fi.cpp Src/Base/AMReX_parmparse_mod.F90 Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryData.cpp Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_BndryRegister.cpp Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_FabSet.cpp Src/Boundary/AMReX_InterpBndryData.H Src/Boundary/AMReX_InterpBndryData.cpp Src/Boundary/AMReX_MacBndry.H Src/Boundary/AMReX_MacBndry.cpp Src/Boundary/AMReX_Mask.H Src/Boundary/AMReX_MultiMask.H Src/Boundary/AMReX_MultiMask.cpp Src/Boundary/AMReX_YAFluxRegister.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_IF_Complement.H Src/EB/AMReX_EB2_IF_Lathe.H Src/EB/AMReX_EB2_IF_Sphere.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EBFluxRegister.cpp Src/EB/AMReX_EBInterpolater.cpp Src/EB/AMReX_EBMultiFabUtil.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EBToPVD.H Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_EB_LeastSquares_2D_K.H Src/EB/AMReX_EB_LeastSquares_3D_K.H Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_slopes_K.H Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/AMReX_MultiCutFab.H Src/EB/AMReX_WriteEBSurface.H Src/EB/AMReX_WriteEBSurface.cpp Src/EB/AMReX_ebcellflag_mod.F90 Src/Extern/Conduit/AMReX_Conduit_Blueprint_ParticlesI.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_Hypre.cpp Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.H Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/AMReX_PETSc.cpp Src/Extern/ProfParser/AMReX_AVGDOWN_1D.F Src/Extern/ProfParser/AMReX_AVGDOWN_2D.F Src/Extern/ProfParser/AMReX_AVGDOWN_3D.F Src/Extern/ProfParser/AMReX_AVGDOWN_F.H Src/Extern/ProfParser/AMReX_BLProfStats.H Src/Extern/ProfParser/AMReX_BLProfStats.cpp Src/Extern/ProfParser/AMReX_BLProfUtilities.H Src/Extern/ProfParser/AMReX_BLProfUtilities.cpp Src/Extern/ProfParser/AMReX_BLWritePlotFile.H Src/Extern/ProfParser/AMReX_BLWritePlotFile.cpp Src/Extern/ProfParser/AMReX_CommProfStats.H Src/Extern/ProfParser/AMReX_CommProfStats.cpp Src/Extern/ProfParser/AMReX_ProfParserBatch.cpp Src/Extern/ProfParser/AMReX_RegionsProfStats.H Src/Extern/ProfParser/AMReX_RegionsProfStats.cpp Src/Extern/SWFFT/AlignedAllocator.h Src/Extern/SWFFT/CheckDecomposition.c Src/Extern/SWFFT/Dfft.H Src/Extern/SWFFT/DfftC.cpp Src/Extern/SWFFT/Distribution.H Src/Extern/SWFFT/DistributionC.cpp Src/Extern/SWFFT/Error.h Src/Extern/SWFFT/TimingStats.h Src/Extern/SWFFT/complex-type.h Src/Extern/SWFFT/distribution.c Src/Extern/SWFFT/distribution_c.h Src/Extern/amrdata/AMReX_AmrData.H Src/Extern/amrdata/AMReX_AmrData.cpp Src/Extern/amrdata/AMReX_DataServices.H Src/Extern/amrdata/AMReX_DataServices.cpp Src/Extern/amrdata/AMReX_FABUTIL_1D.F Src/Extern/amrdata/AMReX_FABUTIL_2D.F Src/Extern/amrdata/AMReX_FABUTIL_3D.F Src/Extern/amrdata/AMReX_XYPlotDataList.H Src/Extern/amrdata/AMReX_XYPlotDataList.cpp Src/Extern/amrdata/CMakeLists.txt Src/Extern/hpgmg/BL_HPGMG.H Src/Extern/hpgmg/BL_HPGMG.cpp Src/F_Interfaces/AmrCore/AMReX_FAmrCore.H Src/F_Interfaces/AmrCore/AMReX_FAmrCore.cpp Src/F_Interfaces/AmrCore/AMReX_amrcore_fi.cpp Src/F_Interfaces/AmrCore/AMReX_amrcore_mod.F90 Src/F_Interfaces/AmrCore/AMReX_fillpatch_fi.cpp Src/F_Interfaces/AmrCore/AMReX_fillpatch_mod.F90 Src/F_Interfaces/AmrCore/AMReX_fluxregister_fi.cpp Src/F_Interfaces/AmrCore/AMReX_fluxregister_mod.F90 Src/F_Interfaces/AmrCore/AMReX_tagbox_fi.cpp Src/F_Interfaces/AmrCore/AMReX_tagbox_mod.F90 Src/F_Interfaces/Base/AMReX_FPhysBC.cpp Src/F_Interfaces/Base/AMReX_box_fi.cpp Src/F_Interfaces/Base/AMReX_box_mod.F90 Src/F_Interfaces/Base/AMReX_boxarray_fi.cpp Src/F_Interfaces/Base/AMReX_distromap_fi.cpp Src/F_Interfaces/Base/AMReX_distromap_mod.F90 Src/F_Interfaces/Base/AMReX_geometry_fi.cpp Src/F_Interfaces/Base/AMReX_geometry_mod.F90 Src/F_Interfaces/Base/AMReX_init_mod.F90 Src/F_Interfaces/Base/AMReX_multifab_fi.cpp Src/F_Interfaces/Base/AMReX_multifab_mod.F90 Src/F_Interfaces/Base/AMReX_multifabutil_fi.cpp Src/F_Interfaces/Base/AMReX_parallel_mod.F90 Src/F_Interfaces/Base/AMReX_physbc_fi.cpp Src/F_Interfaces/Base/AMReX_physbc_mod.F90 Src/F_Interfaces/Base/AMReX_plotfile_fi.cpp Src/F_Interfaces/Base/AMReX_vismf_fi.cpp Src/F_Interfaces/LinearSolvers/AMReX_linop_mod.F90 Src/F_Interfaces/LinearSolvers/AMReX_multigrid_fi.cpp Src/F_Interfaces/LinearSolvers/AMReX_multigrid_mod.F90 Src/F_Interfaces/LinearSolvers/AMReX_poisson_mod.F90 Src/F_Interfaces/Octree/AMReX_octree_fi.cpp Src/F_Interfaces/Particle/AMReX_particlecontainer_fi.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_nd.F90 Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMGBndry.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_BinIterator.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleMPIUtil.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particle_mod_K.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_SparseBins.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.H Src/Particle/AMReX_TracerParticles.cpp Src/SDC/AMReX_SDCquadrature.F90 Src/SDC/AMReX_SDCstruct.H Src/SDC/AMReX_SDCstruct.cpp Tests/AsyncOut/multifab/main.cpp Tests/DivFreePatch/main.cpp Tests/FillBoundaryComparison/main.cpp Tests/GPU/AnyOf/main.cpp Tests/GPU/Fuse/main.cpp Tests/GPU/Vector/main.cpp Tests/HDF5Benchmark/main.cpp Tests/LinearSolvers/ABecLaplacian_C/MyTest.H Tests/LinearSolvers/ABecLaplacian_C/MyTest.cpp Tests/LinearSolvers/ABecLaplacian_C/initProb.cpp Tests/LinearSolvers/ABecLaplacian_C/initProb_K.H Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-16.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-32.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-1.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-16.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-2.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-32.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-4.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-64.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-1.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-16.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-2.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-32.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-4.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-64.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-8.sh Tests/LinearSolvers/ABecLaplacian_F/init_prob.F90 Tests/LinearSolvers/ABecLaplacian_F/mytest.F90 Tests/LinearSolvers/CellEB/MyTest.H Tests/LinearSolvers/CellEB/MyTest.cpp Tests/LinearSolvers/CellEB/initEB.cpp Tests/LinearSolvers/CellEB2/MyTest.H Tests/LinearSolvers/CellEB2/MyTest.cpp Tests/LinearSolvers/CellEB2/initEB.cpp Tests/LinearSolvers/EBConvergenceTest/BC_2D.F90 Tests/LinearSolvers/EBConvergenceTest/BC_3D.F90 Tests/LinearSolvers/EBConvergenceTest/Convergence_Data_Gen.sh Tests/LinearSolvers/EBConvergenceTest/MyTest.H Tests/LinearSolvers/EBConvergenceTest/MyTest.cpp Tests/LinearSolvers/EBConvergenceTest/MyTest_F.H Tests/LinearSolvers/EBConvergenceTest/RHS.F90 Tests/LinearSolvers/EBConvergenceTest/acoef.F90 Tests/LinearSolvers/EBConvergenceTest/bcoef.F90 Tests/LinearSolvers/EBTensor/MyTest.cpp Tests/LinearSolvers/EBTensor/MyTestPlotfile.cpp Tests/LinearSolvers/EBTensor/MyTest_2D_K.H Tests/LinearSolvers/EBflux_grad/MyTest.H Tests/LinearSolvers/EBflux_grad/MyTest.cpp Tests/LinearSolvers/EBflux_grad/initEB.cpp Tests/LinearSolvers/LeastSquares/MyTest.cpp Tests/LinearSolvers/LeastSquares/README.md Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/LeastSquares/initEB.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleData.cpp Tests/LinearSolvers/MAC_Projection_EB/main.cpp Tests/LinearSolvers/MLMG/fort_3d.F90 Tests/LinearSolvers/MLMG/init_prob.cpp Tests/LinearSolvers/MLMG/main.cpp Tests/LinearSolvers/MultiComponent/MCNodalLinOp.H Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/MultiComponent/main.cpp Tests/LinearSolvers/Nodal_Projection_EB/main.cpp Tests/LinearSolvers/NodeEB/MyTest.cpp Tests/MultiBlock/Advection/main.cpp Tests/MultiBlock/IndexType/main.cpp Tests/Particles/AssignDensity/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/AsyncIO/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/InitFromAscii/main.cpp Tests/Particles/Intersection/main.cpp Tests/Particles/NeighborParticles/CheckPair.H Tests/Particles/NeighborParticles/Constants.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/README.md Tests/Particles/ParticleIterator/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleTransformations/main.cpp Tests/Particles/SparseBins/main.cpp Tests/Particles/TypeDescriptor/main.cpp Tools/AMRProfParser/TestCodes/AMRProfTest0.cpp Tools/AMRProfParser/TestCodes/ProfWaitTest.cpp Tools/AMRProfParser/TestCodes/SendTest0.cpp Tools/AMRProfParser/TestCodes/TokenizeTest.cpp Tools/C_util/AmrDeriveTecplot/AmrDeriveTecplot.cpp Tools/C_util/AppendToPlotFile.cpp Tools/C_util/AugmentPlotfile/AugmentPlotfile.cpp Tools/C_util/AugmentPlotfile/AugmentPlotfile_F.H Tools/C_util/Convergence/Add.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/Convergence/ComputeAmrDataNorms.H Tools/C_util/Convergence/ComputeAmrDataNorms.cpp Tools/C_util/Convergence/DebugDump.H Tools/C_util/Convergence/DebugOut.H Tools/C_util/Convergence/DebugOut.cpp Tools/C_util/Convergence/DiffFab.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedFD.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp Tools/C_util/Convergence/DiffUniform.cpp Tools/C_util/Convergence/PltFileNorm.cpp Tools/C_util/Convergence/PltFileNormB.cpp Tools/C_util/Convergence/PltFileScalConvRate.cpp Tools/C_util/Convergence/RichardsonConvergenceTest.cpp Tools/C_util/DiffMultiFab/diffmultifab.cpp Tools/C_util/Statistics/AVGDOWN_2D.F Tools/C_util/Statistics/AVGDOWN_3D.F Tools/C_util/Statistics/AVGDOWN_F.H Tools/C_util/Statistics/ComputeAmrDataStat.H Tools/C_util/Statistics/ComputeAmrDataStat.cpp Tools/C_util/Statistics/PltFileFluxAve.H Tools/C_util/Statistics/PltFileFluxAve.cpp Tools/C_util/Statistics/PltFileList.cpp Tools/C_util/Statistics/PltFileStat.cpp Tools/C_util/Statistics/PltFileXAve.H Tools/C_util/Statistics/PltFileXAve.cpp Tools/C_util/TV_TempWrite.H Tools/C_util/ViewMF/MFNorm.cpp Tools/C_util/ViewMF/checkMFghostcells.cpp Tools/C_util/ViewMF/main.cpp Tools/C_util/ViewMF/mfMinMax.cpp Tools/C_util/ViewMF/viewMF.cpp Tools/C_util/ViewMF/viewMFcol.cpp Tools/C_util/ViewMF/viewMFdiff.cpp Tools/C_util/ViewMF/viewMFdiffHardWire.cpp Tools/C_util/WritePlotFile.H Tools/C_util/WritePlotFile.cpp Tools/C_util/dbgTools/crsGrids.cpp Tools/C_util/dbgTools/intersectGrids.cpp Tools/CompileTesting/compiletesting.py Tools/EBSurfaceTools/ConvertEBSurface.cpp Tools/EBSurfaceTools/isoToVTK.py Tools/F_scripts/fcheck.py Tools/GNUMake/README.md Tools/Plotfile/AMReX_PPMUtil.cpp Tools/Plotfile/fcompare.cpp Tools/Postprocessing/C_Src/HorizontalAvg.cpp Tools/Postprocessing/C_Src/IntegrateComp.cpp Tools/Postprocessing/C_Src/MultiFabToMatLab.cpp Tools/Postprocessing/C_Src/PlotfileToMatLab.cpp Tools/Postprocessing/C_Src/PlotfileToTurb.cpp Tools/Postprocessing/C_Src/PtwisePltTransform.cpp Tools/Postprocessing/C_Src/PtwisePltTransform_nd.f90 Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp Tools/Postprocessing/C_Src/cube_extract.cpp Tools/Postprocessing/C_Src/particle_compare.cpp Tools/Postprocessing/python/column_depth.py Tools/Postprocessing/python/conv_slopes.py Tools/Postprocessing/python/dumpparthistory.py Tools/Postprocessing/python/eos_data.txt Tools/Postprocessing/python/helmeos.py Tools/Postprocessing/python/parseparticles.py Tools/Postprocessing/python/test_helmeos.py Tools/Postprocessing/python/test_parseparticles.py Tools/Py_util/amrex_particles_to_vtp/amrex_binary_particles_to_vtp.py Tools/Py_util/amrex_particles_to_vtp/write_pview_file.py Tools/Py_util/plotsinglevar.py Tools/Release/ppCleanup.py Tools/Release/ppCleanupDir.py Tools/Release/release.py Tools/typechecker/typechecker.py Tutorials/Amr/Advection_AmrCore/Exec/Prob.H Tutorials/Amr/Advection_AmrCore/Exec/paraview_amr101.py Tutorials/Amr/Advection_AmrCore/README.md Tutorials/Amr/Advection_AmrCore/Source/AdvancePhiAllLevels.cpp Tutorials/Amr/Advection_AmrCore/Source/AdvancePhiAtLevel.cpp Tutorials/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tutorials/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tutorials/Amr/Advection_AmrCore/Source/DefineVelocity.cpp Tutorials/Amr/Advection_AmrCore/Source/Src_K/Adv_K.H Tutorials/Amr/Advection_AmrCore/Source/Src_K/compute_flux_3D_K.H Tutorials/Amr/Advection_AmrCore/Source/face_velocity.H Tutorials/Amr/Advection_AmrCore/Source/main.cpp Tutorials/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.f90 Tutorials/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d.f90 Tutorials/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d.f90 Tutorials/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.f90 Tutorials/Amr/Advection_AmrLevel/Source/Adv_F.H Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tutorials/Amr/Advection_AmrLevel/Source/LevelBldAdv.cpp Tutorials/Amr/Advection_AmrLevel/Source/Src_2d/Adv_2d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_2d/compute_flux_2d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_2d/slope_2d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_3d/Adv_3d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_3d/compute_flux_3d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_3d/slope_3d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_nd/Tagging_nd.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_nd/tagging_params.f90 Tutorials/Amr/Advection_AmrLevel/Source/main.cpp Tutorials/Basic/Build_with_libamrex/main.cpp Tutorials/Basic/Build_with_libamrex/test_parameters.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX1_C/Source/myfunc.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/advance.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/advance_3d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/myfunc.H Tutorials/Basic/HeatEquation_EX1_CF/Source/myfunc_F.H Tutorials/Basic/HeatEquation_EX1_F/advance.f90 Tutorials/Basic/HeatEquation_EX1_F/fmain.F90 Tutorials/Basic/HeatEquation_EX1_F/init_phi.f90 Tutorials/Basic/HeatEquation_EX2_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX2_C/Source/myfunc.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/mykernel.H Tutorials/Basic/HeatEquation_EX2_CF/Source/advance.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/advance_3d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/myfunc.H Tutorials/Basic/HeatEquation_EX2_CF/Source/myfunc_F.H Tutorials/Basic/HeatEquation_EX3_C/Source/advance.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX3_C/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX3_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX3_C/Source/myfunc_F.H Tutorials/Basic/main_C/main.cpp Tutorials/Basic/main_F/main.F90 Tutorials/Blueprint/AssignMultiLevelDensity/main.cpp Tutorials/Blueprint/CellSortedParticles/CellSortedPC.H Tutorials/Blueprint/CellSortedParticles/CellSortedPC.cpp Tutorials/Blueprint/CellSortedParticles/cell_sorted_3d.F90 Tutorials/Blueprint/CellSortedParticles/cell_sorted_F.H Tutorials/Blueprint/CellSortedParticles/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance_3d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/init_phi_2d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/init_phi_3d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/myfunc.H Tutorials/Blueprint/HeatEquation_EX1_C/Source/myfunc_F.H Tutorials/Blueprint/README.txt Tutorials/EB/CNS/Exec/Combustor/bc_fill_nd.F90 Tutorials/EB/CNS/Exec/Pulse/cns_prob.F90 Tutorials/EB/CNS/Exec/ShockRef/cns_prob.F90 Tutorials/EB/CNS/Exec/Sod/cns_prob.F90 Tutorials/EB/CNS/Source/CNS.H Tutorials/EB/CNS/Source/CNS.cpp Tutorials/EB/CNS/Source/CNS_F.H Tutorials/EB/CNS/Source/CNS_advance.cpp Tutorials/EB/CNS/Source/CNS_init_eb2.cpp Tutorials/EB/CNS/Source/CNS_io.cpp Tutorials/EB/CNS/Source/CNS_setup.cpp Tutorials/EB/CNS/Source/diffusion/cns_diff_mod.F90 Tutorials/EB/CNS/Source/diffusion/cns_eb_diff_wall.F90 Tutorials/EB/CNS/Source/diffusion/diff_coef_mod.F90 Tutorials/EB/CNS/Source/fortran/CNS_divop.F90 Tutorials/EB/CNS/Source/fortran/CNS_dudt.F90 Tutorials/EB/CNS/Source/fortran/CNS_f.F90 Tutorials/EB/CNS/Source/fortran/CNS_nd.F90 Tutorials/EB/CNS/Source/fortran/bc_fill_nd.F90 Tutorials/EB/CNS/Source/hydro/Hyp_gamma_MOL.F90 Tutorials/EB/CNS/Source/hydro/Hyp_gamma_MOL_EB.F90 Tutorials/EB/CNS/Source/hydro/analriem3d.F90 Tutorials/EB/CNS/Source/hydro/cns_eb_hyp_wall.F90 Tutorials/EB/CNS/Source/hydro/slope_mol_3d_gamma.F90 Tutorials/EB/CNS/Source/hydro/slope_mol_3d_gamma_EB.F90 Tutorials/EB/CNS/Source/main.cpp Tutorials/EB/GeometryGeneration/main.cpp Tutorials/EB/MacProj/main.cpp Tutorials/EB/Poisson/main.cpp Tutorials/ForkJoin/MLMG/ff.f90 Tutorials/FortranInterface/Advection_F/Exec/SingleVortex/Prob_2d.f90 Tutorials/FortranInterface/Advection_F/Exec/SingleVortex/Prob_3d.f90 Tutorials/FortranInterface/Advection_F/Exec/SingleVortex/face_velocity_2d.F90 Tutorials/FortranInterface/Advection_F/Exec/SingleVortex/face_velocity_3d.F90 Tutorials/FortranInterface/Advection_F/Source/Src_2d/advect_2d_mod.F90 Tutorials/FortranInterface/Advection_F/Source/Src_2d/compute_flux_2d.f90 Tutorials/FortranInterface/Advection_F/Source/Src_2d/slope_2d.f90 Tutorials/FortranInterface/Advection_F/Source/Src_3d/advect_3d_mod.F90 Tutorials/FortranInterface/Advection_F/Source/Src_3d/compute_flux_3d.f90 Tutorials/FortranInterface/Advection_F/Source/Src_3d/slope_3d.f90 Tutorials/FortranInterface/Advection_F/Source/amr_data_mod.F90 Tutorials/FortranInterface/Advection_F/Source/averagedown_mod.F90 Tutorials/FortranInterface/Advection_F/Source/bc_mod.F90 Tutorials/FortranInterface/Advection_F/Source/compute_dt_mod.F90 Tutorials/FortranInterface/Advection_F/Source/evolve_mod.F90 Tutorials/FortranInterface/Advection_F/Source/fillpatch_mod.F90 Tutorials/FortranInterface/Advection_F/Source/fmain.F90 Tutorials/FortranInterface/Advection_F/Source/initdata.F90 Tutorials/FortranInterface/Advection_F/Source/my_amr_mod.F90 Tutorials/FortranInterface/Advection_F/Source/plotfile_mod.F90 Tutorials/FortranInterface/Advection_F/Source/tagging_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Exec/SingleVortex/Prob.f90 Tutorials/FortranInterface/Advection_octree_F/Exec/SingleVortex/face_velocity_2d.F90 Tutorials/FortranInterface/Advection_octree_F/Source/Src_2d/advect_2d_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/Src_2d/compute_flux_2d.f90 Tutorials/FortranInterface/Advection_octree_F/Source/Src_2d/slope_2d.f90 Tutorials/FortranInterface/Advection_octree_F/Source/amr_data_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/averagedown_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/bc_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/compute_dt_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/evolve_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/fillpatch_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/my_amr_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/Prob.f90 Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/face_velocity_2d.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/advect_2d_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/compute_flux_2d.f90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/slope_2d.f90 Tutorials/FortranInterface/Advection_octree_F2/Source/amr_data_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/averagedown_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/bc_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/compute_dt_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/evolve_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/fillpatch_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/my_amr_mod.F90 Tutorials/GPU/CNS/Source/CNS.H Tutorials/GPU/CNS/Source/CNS.cpp Tutorials/GPU/CNS/Source/CNS_K.H Tutorials/GPU/CNS/Source/CNS_advance.cpp Tutorials/GPU/CNS/Source/CNS_io.cpp Tutorials/GPU/CNS/Source/CNS_setup.cpp Tutorials/GPU/CNS/Source/hydro/CNS_hydro_K.H Tutorials/GPU/CNS/Source/main.cpp Tutorials/GPU/EBCNS/Source/CNS.H Tutorials/GPU/EBCNS/Source/CNS.cpp Tutorials/GPU/EBCNS/Source/CNS_advance.cpp Tutorials/GPU/EBCNS/Source/CNS_init_eb2.cpp Tutorials/GPU/EBCNS/Source/CNS_io.cpp Tutorials/GPU/EBCNS/Source/CNS_setup.cpp Tutorials/GPU/EBCNS/Source/hydro/CNS_hydro_K.H Tutorials/GPU/EBCNS/Source/main.cpp Tutorials/GPU/Launch/MyKernel_F.F90 Tutorials/GPU/ParallelScan/main.cpp Tutorials/MUI/Source_01/init_phi_3d.f90 Tutorials/MUI/Source_01/main_01.cpp Tutorials/MUI/Source_02/main_02.cpp Tutorials/MUI/doc/MUIcouplingNotes.tex Tutorials/Particles/CellSortedParticles/CellSortedPC.H Tutorials/Particles/CellSortedParticles/CellSortedPC.cpp Tutorials/Particles/CellSortedParticles/cell_sorted_3d.F90 Tutorials/Particles/CellSortedParticles/cell_sorted_F.H Tutorials/Particles/CellSortedParticles/main.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/em_pic_K.H Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/em_pic_3d.F90 Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Source/EMParticleContainer.H Tutorials/Particles/ElectromagneticPIC/Source/EMParticleContainerInit.cpp Tutorials/Particles/ElectromagneticPIC/Source/main.cpp Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.H Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.cpp Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_2d.f90 Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_3d.f90 Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_F.H Tutorials/Particles/ElectrostaticPIC/main.cpp Tutorials/Particles/NeighborList/CheckPair.H Tutorials/Particles/NeighborList/Constants.H Tutorials/Particles/NeighborList/MDParticleContainer.H Tutorials/Particles/NeighborList/MDParticleContainer.cpp Tutorials/Particles/NeighborList/README.md Tutorials/Particles/NeighborList/main.cpp Tutorials/SDC/MISDC_ADR_2d/Source/SDC_sweeper.cpp Tutorials/SDC/MISDC_ADR_2d/Source/functions_2d.f90 Tutorials/SDC/MISDC_ADR_2d/Source/init_phi_2d.f90 Tutorials/SDC/MISDC_ADR_2d/Source/main.cpp Tutorials/SDC/MISDC_ADR_2d/Source/myfunc.H Tutorials/SDC/MISDC_ADR_2d/Source/myfunc_F.H Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test.H Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test.cpp Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test_F.F90 Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test_F.H Tutorials/SWFFT/SWFFT_poisson/main.cpp Tutorials/SWFFT/SWFFT_poisson/swfft_solver.cpp Tutorials/SWFFT/SWFFT_simple/SWFFT_Test.H Tutorials/SWFFT/SWFFT_simple/SWFFT_Test.cpp Tutorials/SWFFT/SWFFT_simple/SWFFT_Test_F.H Tutorials/SWFFT/SWFFT_simple/main.cpp Tutorials/SWFFT/SWFFT_simple/swfft_compute.cpp commit d84f787548d8e5b89466cd44fbad9d1656e7d08a Author: Weiqun Zhang Date: Sun Mar 14 06:15:13 2021 -0700 Free unused memory (#1861) If Arena is asked to allocate more than what's available in the system, try to free unused memory first. Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp commit 85609ec18f8dd3b86eed3bf805e48515e5ce6e7d Author: kngott <30483578+kngott@users.noreply.github.com> Date: Sat Mar 13 17:52:38 2021 -0800 Fortran Interfaces for new FillPatch (#1793) Fortran Interfaces for new FillPatch. Compiles, and I believe the indexing is correct, but didn't test a Fortran implementation. Src/F_Interfaces/AmrCore/AMReX_fillpatch_fi.cpp Src/F_Interfaces/AmrCore/AMReX_fillpatch_mod.F90 Src/F_Interfaces/AmrCore/AMReX_interpolater_mod.F90 commit 9b17f925887a9058a3efb91af26a5ac082d5f2c6 Author: Weiqun Zhang Date: Fri Mar 12 12:14:08 2021 -0800 The_Device_Arena -> The_Arena (#1858) Replace The_Device_Arena with The_Arena in various places. Using The_Arena should help reduce the memory footprint. We used The_Device_Arena in the early days to avoid the penalty of page fault caused by touching managed memory on CPU. We are much better in that regard now. In fact, AMReX itself no longer depends on managed memory. Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H commit 7e31cfd67ad9739b7779ee47d1a2750acf9c2cc1 Author: Weiqun Zhang Date: Fri Mar 12 12:13:42 2021 -0800 Arena Release Threshold (#1857) Release memory from Arena back to the system if it's above a threshold. For the pinned arena, the release threshold is set to the size of the global memory. For others, it's set to a huge number. These can be controlled with ParmParse parameters, `amrex.the_arena_release_threshold` and `the_[device|pinned|managed|async]_release_threshold`. The reason for setting the pinned arena's release threshold to the size of the global memory is that we received a report in the past that the pinned arena used by the communication functions wasted a huge amount of memory in the case that more and more grids were added over the run time. Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp Src/Base/AMReX_DArena.H Src/Base/AMReX_DArena.cpp Src/Base/AMReX_EArena.H Src/Base/AMReX_EArena.cpp Src/Base/AMReX_PArena.cpp commit fafaccd7d4a7d0fbe90ec3507402e1988cfd00a1 Author: Maikel Nadolski Date: Fri Mar 12 17:07:57 2021 +0100 Replace is_pod with is_standard_layout && is_trivial (#1856) [`std::is_pod`](https://en.cppreference.com/w/cpp/types/is_pod) is deprecated in C++20 and should be replaced by `std::is_standard_layout && std::is_trivial`. This is the only occurrence of `std::is_pod` that I found in AMReX. If unchanged, this will trigger warnings in future compilers. This triggered a warning in our internal CI when using gcc 10.2 and the `-std=c++20` option. Src/Base/AMReX_GpuAsyncArray.H commit 7d594d2affc9188963ceb55a9ebca597d02462da Author: Maikel Nadolski Date: Thu Mar 11 17:49:00 2021 +0100 Feature/non local parallel copy (#1842) This PR introduces "Multiblock" capabilities to AMReX. Multiblock refers to having multiple distinct computational domains, known as blocks, that have some relation to each other. This PR proposes a generic NonLocalBC::ParallelCopy that performs copies between two FabArrays that might live on different blocks. The ParallelCopy respects - An index mapping such as already present in NonLocalBC (DTOS) - A projection function that will be called on either sender or receiver side to perform actions like e.g. swapping two components or changing the sign of velocity components The proposed usage is ``` NonLocalBC::ParallelCopy(dest, destbox, src, scomp, dcomp, ncomp, ngrow, index_mapping, fab_projection); ``` Notes - In the current implementation, `DTOS` needs to be a map between index spaces where inverse images of boxes are boxes again. - The name `NonLocalBC::ParallelCopy` is confusing if it does something else than `amrex::ParallelCopy`. Alternatives? - Parts of the implementation might be shared with other places in AMReX, such as in communication procedures of FabArray. But this is out of scope. - In addition to `SwapComponents`, a more generic `PermuteComponents` is useful. - The test is not commented yet. Src/Base/AMReX_Extension.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBC.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_TypeTraits.H Src/Base/CMakeLists.txt Src/Base/Make.package Tests/CMakeLists.txt Tests/MultiBlock/Advection/CMakeLists.txt Tests/MultiBlock/Advection/GNUmakefile Tests/MultiBlock/Advection/Make.package Tests/MultiBlock/Advection/main.cpp Tests/MultiBlock/IndexType/CMakeLists.txt Tests/MultiBlock/IndexType/GNUmakefile Tests/MultiBlock/IndexType/Make.package Tests/MultiBlock/IndexType/main.cpp commit 4893049d2588261f235c1e05ef7d0121a9ad6b6c Author: Pankaj Jha Date: Wed Mar 10 14:35:57 2021 -0800 Include missing library 'pthread' for linking to amrex library (#1854) - [HOTFIX ] Include missing library 'pthread' for linking to amrex library in the tutorial 'Basic/Build_with_libamrex' Tutorials/Basic/Build_with_libamrex/GNUmakefile commit 6ec00666843dcc0ae794358c872dd92e30a3cd34 Author: Andrew Myers Date: Mon Mar 8 17:05:52 2021 -0500 Enable expanding MPI / GPU macros for doxygen. (#1853) Docs/Doxygen/doxygen.conf commit 9001a7486ffcdd68cec9f2c1212559c4982865d8 Author: Weiqun Zhang Date: Mon Mar 8 08:15:45 2021 -0800 Revert the use of codeplay_host_task (#1851) It results int NaNs. Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuElixir.cpp commit fff2224879807f73c5a4acac19609f5630f77a8e Author: asalmgren Date: Sun Mar 7 10:01:11 2021 -0800 Add harmonic averaging (#1850) Mostly adding functionality, but also fixed a bug for 2d multicomponent cc->fc averaging Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H commit 146f3f2b7596cbee73d2b2eb8d48d4576ecc3f7c Author: Candace Gilet Date: Sat Mar 6 21:36:28 2021 -0500 Fix typo in GNUMake/Make.defs introduced in PR#1846 (#1849) Tools/GNUMake/Make.defs commit 0fc630c5bbc08b65feda01cae000b2314772d240 Author: Weiqun Zhang Date: Fri Mar 5 14:53:57 2021 -0800 AMREX_NO_PROBINIT (#1846) For Amr/AmrLevel based codes, there is now a compile time option AMREX_NO_PROBINIT=[TRUE|FALSE] for the support of the probin file and amrex_probinit function. The default behavior of requiring a user defined amrex_probinit function unless amr.probin_file is empty does not change unless amrex is built into a shared library on non-GNU/Linux system. The reason for the inconsistency is there is no portable way of supporting a weak symbol (.e.g, undefined amrex_probinit in amrex itself) in a shared library. This will partially solve the shared library build issue on Mac and Windows. Co-authored-by: Michele Rosso Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Tools/CMake/AMReXGenerateConfigHeader.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs Tools/libamrex/configure.py commit e7a264ddc0b008310d80c98ecf578e292f3cf13d Author: Maikel Nadolski Date: Fri Mar 5 19:53:28 2021 +0100 Use unique_ptr instead of naked pointer (#1839) This commit replaces naked pointer to char with `std::unique_ptr` in `NonLocalBC::CommHandler`. This will lead to proper cleanup in case of any exceptions. Note, that `Comm_nowait` and `Comm_finish` are *not* marked noexcept. Furthermore, future PRs will add interpolation and some kind of projection for non-local data transactions. Error handling for those is still unclear. Maybe, we want to mark everything noexcept. This is merely an implementation change without nearly any noticable difference in user code. Src/Base/AMReX_Extension.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_NonLocalBCImpl.H commit ef3a757eae988bff7948a47c346a6fb47ffd9ac0 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Fri Mar 5 09:43:54 2021 -0800 Fix early returns in ParallelCopy. (#1845) Fixes a new segfault bug due to pc_src.size() being called when pc_src isn't set. Instead, tracks all ParallelCopy_nowait early returns and checks once at the top of the ParallelCopy_finish. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H commit 6c1296b2466263a5aba46b5df16aceda47457b11 Author: Candace Gilet Date: Fri Mar 5 11:49:42 2021 -0500 Revert "Change default for Amr::probin_file -> "" (#1837)" (#1844) This reverts commit 2519768caedcf46a05b0cfd2b552dff2148465f7. Src/Amr/AMReX_Amr.cpp commit 2fb82bf64838aba949f7a641988a0c58774e5a88 Author: mic84 Date: Thu Mar 4 13:25:13 2021 -0800 CMake: move AMReX_INSTALL option to AMReXOptions.cmake (#1840) * CMake: add missing AMReX_INSTALL option * CMake: remove AMReX_INSTALL from root CMakeLists.txt CMakeLists.txt Tools/CMake/AMReXOptions.cmake commit e01e1c73e670defda422b0f6a7b0a5fc45fb17af Author: Weiqun Zhang Date: Thu Mar 4 12:23:03 2021 -0800 Add EditorConfig (#1841) .editorconfig commit 076be729a1cc8fabc5c18f6836e134f85fa8b0c7 Author: asalmgren Date: Thu Mar 4 08:00:53 2021 -0800 Migrate more of Tutorials/LinearSolvers to Tests/LinearSolvers (#1835) Tests/LinearSolvers/ABecLaplacian_C/CMakeLists.txt Tests/LinearSolvers/ABecLaplacian_C/GNUmakefile Tests/LinearSolvers/ABecLaplacian_C/Make.package Tests/LinearSolvers/ABecLaplacian_C/MyTest.H Tests/LinearSolvers/ABecLaplacian_C/MyTest.cpp Tests/LinearSolvers/ABecLaplacian_C/MyTestPlotfile.cpp Tests/LinearSolvers/ABecLaplacian_C/initProb.cpp Tests/LinearSolvers/ABecLaplacian_C/initProb_K.H Tests/LinearSolvers/ABecLaplacian_C/inputs Tests/LinearSolvers/ABecLaplacian_C/inputs-inhomNeumann Tests/LinearSolvers/ABecLaplacian_C/inputs-rt-abeclap-com Tests/LinearSolvers/ABecLaplacian_C/inputs-rt-poisson-lev Tests/LinearSolvers/ABecLaplacian_C/inputs.hypre Tests/LinearSolvers/ABecLaplacian_C/inputs.petsc Tests/LinearSolvers/ABecLaplacian_C/main.cpp Tests/LinearSolvers/ABecLaplacian_C/scalingtest/inputs.test Tests/LinearSolvers/ABecLaplacian_C/scalingtest/main.diff Tests/LinearSolvers/ABecLaplacian_C/scalingtest/results.org Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-1.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-1024.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-128.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-16.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-2.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-2048.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-256.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-32.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-4.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-512.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-64.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/inputs.test Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-1.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-16.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-2.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-32.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-4.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-64.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-1.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-16.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-2.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-32.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-4.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-64.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/main.diff Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/results.org Tests/LinearSolvers/ABecLaplacian_F/CMakeLists.txt Tests/LinearSolvers/ABecLaplacian_F/GNUmakefile Tests/LinearSolvers/ABecLaplacian_F/Make.package Tests/LinearSolvers/ABecLaplacian_F/README Tests/LinearSolvers/ABecLaplacian_F/init_prob.F90 Tests/LinearSolvers/ABecLaplacian_F/inputs Tests/LinearSolvers/ABecLaplacian_F/inputs-rt-abeclap-lev Tests/LinearSolvers/ABecLaplacian_F/inputs-rt-poisson-com Tests/LinearSolvers/ABecLaplacian_F/main.F90 Tests/LinearSolvers/ABecLaplacian_F/mytest.F90 Tests/LinearSolvers/MultiComponent/GNUmakefile Tests/LinearSolvers/MultiComponent/MCNodalLinOp.H Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/MultiComponent/Make.package Tests/LinearSolvers/MultiComponent/inputs Tests/LinearSolvers/MultiComponent/main.cpp Tests/LinearSolvers/NodalPoisson/CMakeLists.txt Tests/LinearSolvers/NodalPoisson/GNUmakefile Tests/LinearSolvers/NodalPoisson/Make.package Tests/LinearSolvers/NodalPoisson/MyTest.H Tests/LinearSolvers/NodalPoisson/MyTest.cpp Tests/LinearSolvers/NodalPoisson/MyTestPlotfile.cpp Tests/LinearSolvers/NodalPoisson/inputs-rt Tests/LinearSolvers/NodalPoisson/main.cpp Tests/LinearSolvers/Nodal_Projection_EB/CMakeLists.txt Tests/LinearSolvers/Nodal_Projection_EB/GNUmakefile Tests/LinearSolvers/Nodal_Projection_EB/Make.package Tests/LinearSolvers/Nodal_Projection_EB/README Tests/LinearSolvers/Nodal_Projection_EB/inputs_3d Tests/LinearSolvers/Nodal_Projection_EB/main.cpp Tests/LinearSolvers/NodeTensorLap/CMakeLists.txt Tests/LinearSolvers/NodeTensorLap/GNUmakefile Tests/LinearSolvers/NodeTensorLap/Make.package Tests/LinearSolvers/NodeTensorLap/MyTest.H Tests/LinearSolvers/NodeTensorLap/MyTest.cpp Tests/LinearSolvers/NodeTensorLap/MyTestPlotfile.cpp Tests/LinearSolvers/NodeTensorLap/main.cpp Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-dpcpp-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini commit 2519768caedcf46a05b0cfd2b552dff2148465f7 Author: Candace Gilet Date: Thu Mar 4 11:00:18 2021 -0500 Change default for Amr::probin_file -> "" (#1837) Src/Amr/AMReX_Amr.cpp commit 86c58645a181907962bd9a72d7c12089d68c1a3a Author: Weiqun Zhang Date: Wed Mar 3 21:47:44 2021 -0800 Fix BoxArray in doxygen (#1838) Src/Base/AMReX_BoxArray.H commit 0ad9fd9c31e9ddfbbdbb57e659d7864b2f2a6ed8 Author: asalmgren Date: Wed Mar 3 14:35:12 2021 -0800 Move Tutorials/LinearSolvers/MAC_Projection_EB to Tests/LinearSolvers/MAC_Projection_EB (#1834) Tests/LinearSolvers/MAC_Projection_EB/CMakeLists.txt Tests/LinearSolvers/MAC_Projection_EB/GNUmakefile Tests/LinearSolvers/MAC_Projection_EB/Make.package Tests/LinearSolvers/MAC_Projection_EB/README Tests/LinearSolvers/MAC_Projection_EB/inputs_3d Tests/LinearSolvers/MAC_Projection_EB/main.cpp Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini commit 42c927d1cae12f90a841c07ccd88275d8a66d5cc Author: Axel Huebl Date: Wed Mar 3 09:16:46 2021 -0800 AMReX Install: Config Header (#1833) Found another header that still installed itself :) Tools/CMake/AMReXGenerateConfigHeader.cmake commit dd2ec76a5cc5db01caa17bdf51efb603c019346d Author: Weiqun Zhang Date: Tue Mar 2 10:30:48 2021 -0800 Fix ncomp bug in linear solver (#1832) Need to give m_ncomp a default value, otherwise default constructor followed by define(...) will not work. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H commit 28fd2ee530e4c0229f04e33e041989f8535c9d33 Author: Axel Huebl Date: Tue Mar 2 09:41:39 2021 -0800 CMake: AMReX_INSTALL (#1831) Add an `AMReX_INSTALL` option that controls `install()` generation. Such options are typically used in superbuilds by dependent projects to supress the install of AMReX headers, helpers and libs, e.g. if AMReX is fully consumed as private dependency. CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst commit b7ddf2d2677fce63a567612978e01ced288dbda2 Author: Weiqun Zhang Date: Mon Mar 1 11:18:19 2021 -0800 Update CHANGES for 21.03 (#1830) CHANGES commit 2794f2f8f89d7461e4d99f19edc805c0feb8d7fe Author: Weiqun Zhang Date: Mon Mar 1 08:24:18 2021 -0800 Fix a sign compare warning (#1827) Src/Base/AMReX_GpuLaunchFunctsG.H commit 0518bb82022da8e6d6250d65b11f7cdaa8b31b24 Author: asalmgren Date: Sun Feb 28 12:58:34 2021 -0800 Ncomp for average cc to fc (#1826) Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H commit 90efb8b28714a496a3cd9188bdba4794579c59a4 Author: asalmgren Date: Sun Feb 28 09:01:48 2021 -0800 Add n_comp as an optional argument to the constructor of AMReX_MLABecLaplacian (#1825) Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp commit 662231a896991dd4fc110aa2fae6740a1f511c59 Author: Weiqun Zhang Date: Sat Feb 27 08:37:59 2021 -0800 Arena Query Functions and Bug Fix in FArrayBox & IArrayBox (#1823) Add Arena::isDeviceAccessible, isHostAccessible, isManaged, isDevice, and isPinned. Use them in various places. Fix bugs in FArrayBox & IArrayBox that touch device memory allocated by The_Async_Arena() on the host. Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_BArena.H Src/Base/AMReX_BArena.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_PArena.H Src/Base/AMReX_PArena.cpp Src/Base/AMReX_VisMF.cpp commit 272e9fc5e93b54f125e907b29e3b9db149c4ae63 Author: Weiqun Zhang Date: Sat Feb 27 08:09:28 2021 -0800 DPCPP: Fix bug in computing the number of active threads (#1822) Thank Intel for the help on debugging! Src/Base/AMReX_GpuLaunchFunctsG.H commit da6eca54acd9fc291fec189a9ee75b54d9727cab Author: mic84 Date: Fri Feb 26 09:51:54 2021 -0800 CMake: add -Wno-pass-failed to Clang-based compilers (#1815) * CMake: add -Wno-pass-failed to Clang-based compilers * CMake: forgot to add flag in one place * CMake: need that flag for Release too Tools/CMake/AMReXFlagsTargets.cmake Tools/CMake/AMReXSYCL.cmake commit f208bc92bc0c9c0b744423cf3cc847db23e5a0ba Author: drangara <69211175+drangara@users.noreply.github.com> Date: Fri Feb 26 11:55:37 2021 -0500 Cover multiple cuts (#1810) ## Summary When building the edge types, cover the cut edges that touch opposite regular corners in order to prevent the `too many cuts in a face error`. Logic is only enabled when flag is set to true in inputs file (default is false). ## Additional background When trying to resolve complex geometries with sharp edges one can run into cases where there are multiple EB cuts on a face. In this change, when the flag `cover_multiple_cuts` is set, we cover appropriate edges in order to avoid this error. Manually tested geometry generation on some complex geometries that were previously failing to resolve. Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_MultiGFab.H Src/EB/AMReX_EB2_MultiGFab.cpp commit 35d27f7b8ba4ac3b77faadf12af5132ecd1f24cc Author: Weiqun Zhang Date: Fri Feb 26 07:33:30 2021 -0800 Fix sign-compare warnings (#1817) Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_VisMF.cpp Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_distFcnElement.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_StructOfArrays.H Tools/GNUMake/comps/gnu.mak commit 8cdf250b04aa1c06ae745133100be2988e9a35aa Author: Erik Schnetter Date: Thu Feb 25 22:52:50 2021 -0500 AMReX_GpuFuse.H: Avoid compiler warning (#1816) Src/Base/AMReX_GpuFuse.H commit a2c92323ee40c46a1848438230ee2b6ac1cfefd8 Author: Erik Schnetter Date: Thu Feb 25 17:03:58 2021 -0500 Avoid compiler warning in FabArray::defined (#1814) Avoid compiler warning in `FabArray::defined`: Convert `size_t` to `int`. Src/Base/AMReX_FabArray.H commit 327c62b150bb23ce01633a4d7ae060291156c62d Author: Weiqun Zhang Date: Wed Feb 24 13:42:23 2021 -0800 Add regression testing script for HIP (#1812) Tools/RegressionTesting/AMReX-hip-tests.ini commit f2e46722e8a6d2a8fd383d268b14579c4bae2929 Author: Weiqun Zhang Date: Wed Feb 24 09:16:25 2021 -0800 Add Axel to the list of core developers (#1811) .zenodo.json CONTRIBUTING.md commit 47389357d5de37d84be69d9b06a294e64cb2a96b Author: Weiqun Zhang Date: Mon Feb 22 09:30:43 2021 -0800 Update docs for GitHub Discussions (#1808) CONTRIBUTING.md Docs/sphinx_documentation/source/index.rst commit 13aa70f0ae5e156b9afdec8ed9f8ef13ba86a0fb Author: Weiqun Zhang Date: Mon Feb 22 08:40:22 2021 -0800 The_Async_Arena and Elixir::append (#1804) Add The_Async_Arena that uses CUDA steam-ordered memory allocator, if available. If the feature is not available, use The_Arena to allocate and free memory and the free is done with a host callback function. The purpose of this Arena is to be used by a temporary FArrayBox inside an MFIter loop. Add Elixir::append that appends an rvalue Elixir to an existing one to reduce the number of host callback calls. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuElixir.H Src/Base/AMReX_GpuElixir.cpp Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_PArena.H Src/Base/AMReX_PArena.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 110422d41aae2eee79772d890a52de4ee54f1d2c Author: kngott <30483578+kngott@users.noreply.github.com> Date: Sun Feb 21 20:13:08 2021 -0800 Add function name output option to TinyProfiler. (#1803) Use tiny_profiler.verbose=1 to output "Entering " and "Leaving " output. Primarily for hunting down hang issues. Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit 85cad7492dfa14851cc47ed4c55e4d2a56d6d56c Author: Axel Huebl Date: Fri Feb 19 15:40:58 2021 -0800 CMake: More Particle Tests (#1801) Tests/Particles/AssignDensity/CMakeLists.txt Tests/Particles/AssignMultiLevelDensity/CMakeLists.txt Tests/Particles/AsyncIO/CMakeLists.txt Tests/Particles/AsyncIO/inputs Tests/Particles/GhostsAndVirtuals/CMakeLists.txt Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/InitFromAscii/CMakeLists.txt Tests/Particles/InitFromAscii/main.cpp Tests/Particles/Intersection/CMakeLists.txt Tests/Particles/Intersection/main.cpp Tests/Particles/ParallelContext/CMakeLists.txt Tests/Particles/ParticleArray/CMakeLists.txt Tests/Particles/ParticleIterator/CMakeLists.txt Tests/Particles/SparseBins/CMakeLists.txt Tests/Particles/TypeDescriptor/CMakeLists.txt commit 093275a3bbf26e45faa43971418bac597b8a7b85 Author: mic84 Date: Wed Feb 17 13:36:06 2021 -0800 Extrapolater: make assertion less restrictive (#1799) Src/Amr/AMReX_Extrapolater.cpp commit 86e26ca7bdcb64d5e694f325973541f721c568b9 Author: Weiqun Zhang Date: Wed Feb 17 10:57:36 2021 -0800 DPCPP: Use codeplay_host_task (#1797) In AsyncArray and Elixir, replace streamSynchronize with SYCL 2020's host_task. Note that the function name in the current oneAPI release is codeplay_host_task, because it started as a Codeplay extension. This will probably change in the future, since the extension has been accepted into the standard. Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuElixir.cpp commit d29a5709545b0d44051e82a895be37f5f821cbc6 Author: Weiqun Zhang Date: Wed Feb 17 10:47:06 2021 -0800 Remove functions about the number of host callbacks (#1798) We used to use the number of host callbacks to limit the number of GPU streams. But we no longer do that anymore, and the default number of streams has been reduced to 4. So the functions keeping track of the number of host callbacks can now be removed. Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuElixir.cpp Src/Base/AMReX_MFIter.cpp commit 23c2fa054f21263220d1f1d384889af9593fc330 Author: Andrew Myers Date: Wed Feb 17 10:10:33 2021 -0800 Add ParticleArray classes (#1796) Add a class for storing arrays of particle data that can be toggled between AoS and SoA. Src/Base/AMReX_PODVector.H Src/Particle/AMReX_ParticleArray.H Src/Particle/CMakeLists.txt Src/Particle/Make.package Tests/Particles/ParticleArray/GNUmakefile Tests/Particles/ParticleArray/Make.package Tests/Particles/ParticleArray/main.cpp commit e611121acca7696320f3757982de857cbab74926 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Tue Feb 16 16:43:21 2021 -0800 ParallelCopy_nowait & ParallelCopy_finish (#1765) Most direct implementation of `ParallelCopy_nowait` and `ParallelCopy_finish`, with minimal changes. Previous version had a max component limit per message, set by the input parameter `fabarray.maxcomp`, 25 by default. This is preserved. If `ncomp > fabarray.maxcomp`, `ParallelCopy_nowait` calls `ParallelCopy_finish` internally for all iterations of the loop except the final iteration. For most apps using GPUs that haven't been manually setting `fabarray.maxcomp`, this will almost certainly mean behavior identical to `FillBoundary_nowait`, as `ncomp > 25` would require very small `MultiFabs` to fit on current GPUs. Keeps `ParallelCopy`, which simply calls `ParallelCopy_nowait` followed by `ParallelCopy_finish`. No doubt, there are potential optimizations available: e.g. if `ncomp>fabarray.maxcomp` and `ncomp%fabarray.maxcomp != 0` returning from `nowait` with a message with a full `fabarray.maxcomp` components may allow the largest possible overlap. But, this version gets the feature running while we discuss additional improvements such as this. Plan to follow up with a scaling study on the effect of message size on `ParallelCopy` timing on Summit by exploiting the `fabarray.maxcomp` feature to see if rewriting with more controlled buffer sizes yields enough benefit to justify a rewrite. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H commit 305770490a946deede9eee31ddd3814fe077ff67 Author: Weiqun Zhang Date: Tue Feb 16 14:40:47 2021 -0800 Fix a new bug in GNU Make (#1795) Must set CXXSTD for nvcc because we need to use it later. Tools/GNUMake/comps/nvcc.mak commit 446fb1e9ee4647604a56a68063f64cb7d94cfd50 Author: Weiqun Zhang Date: Tue Feb 16 10:50:25 2021 -0800 Add FabArray::tags() to return the tags. (#1794) Src/Base/AMReX_FabArray.H commit 4a332b5c3875a5eba705ad0d33f76d7f592bef80 Author: Andrew Myers Date: Tue Feb 16 09:30:16 2021 -0800 Bump minimum C++ standard from 11 to 14. (#1787) This is well-supported now and will allow a lot of the code in amrex to be simplified. I have also bumped the version on the Cuda 9 CI test to Cuda 9.2 and GCC 6.5.0, since the old combination did not work with C++14. .github/workflows/dependencies/dependencies_nvcc9.sh .github/workflows/linux.yml Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/BuildingAMReX_Chapter.rst INSTALL Tools/CMake/AMReX_Config.cmake Tools/GNUMake/comps/cray.mak Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/intel.mak Tools/GNUMake/comps/llvm-flang.mak Tools/GNUMake/comps/nag.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/comps/nvhpc.mak Tools/GNUMake/comps/pgi.mak Tutorials/Basic/Build_with_libamrex/GNUmakefile commit 3ed36ff23c1d408a855573eadcce2686b51160d8 Author: asalmgren Date: Mon Feb 15 15:07:55 2021 -0800 fix table formatting in GPU.rst (#1790) Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/Post_Processing.rst Docs/sphinx_documentation/source/Post_Processing_Chapter.rst commit e8b4d0739e4f21b44169a195a7aa40eaf0092c1b Author: asalmgren Date: Mon Feb 15 15:03:15 2021 -0800 Add an interface to the flux redistribution that takes the arguments (#1789) as Array4s rather than FABs or MultiFabs. Docs/sphinx_documentation/source/Post_Processing.rst Src/EB/AMReX_EB_utils.H Src/EB/AMReX_EB_utils.cpp commit 15a84595c5b88f4e289be505869786400bac49e3 Author: Max Katz Date: Mon Feb 15 15:00:58 2021 -0800 Update the preferred short name of NVIDIA HPC SDK to nvhpc (#1788) `nvhpc` is the preferred short name of the NVIDIA HPC SDK (for example when used in modules). Tools/GNUMake/Make.defs Tools/GNUMake/comps/nvhpc.mak commit 492c5524375858d46f4dd6ea97589cffef749c42 Author: Weiqun Zhang Date: Mon Feb 15 14:58:59 2021 -0800 Fix sphinx config for sphinx 3.5.0 (#1792) https://github.com/sphinx-doc/sphinx/issues/8885 Docs/sphinx_documentation/source/conf.py Docs/sphinx_tutorials/source/conf.py commit df5bbe922563bf3ae9eb97a566014205e1a1b2f4 Author: Weiqun Zhang Date: Thu Feb 11 10:07:38 2021 -0800 BoxArray::simplified() & simplified_list() (#1786) Make them public so that they can be used by application codes to regenerate grids covering exactly the same region but with a different max grid size. However, it should be emphasized that they only work with regular BoxArrays without anything like coarsening. Src/Base/AMReX_BoxArray.H commit 7742e1c99f5e7a5ac7a4fa3530dba3a30d51daeb Author: mic84 Date: Wed Feb 10 15:21:10 2021 -0800 CMake: bump up minimum required HYPRE version (#1785) Tools/CMake/AMReXThirdPartyLibraries.cmake commit 932836ad422ad742a7c3c9b6375bad9cffec76d7 Author: asalmgren Date: Tue Feb 9 18:17:01 2021 -0800 fix bug in EB redistribution which was accidentally including (#1784) contributions from outside a non-periodic domain Src/EB/AMReX_EB_utils.cpp commit d2c88194fac2b9e35e469cf332034bedbf361cf9 Author: Axel Huebl Date: Tue Feb 9 00:02:41 2021 -0800 Windows: WINDOWS_EXPORT_ALL_SYMBOLS (#1782) I am seeing some linker issues downstream when building Python bindings with WarpX on Windows. This nice option makes sure we don't need to add dll export symbols all over the place when building a shared AMReX library (dll). Kudos to the ROOT team at CERN for contributing this feature years ago to CMake. Tools/CMake/AMReXBuildInfo.cmake Tools/CMake/AMReX_Config.cmake commit 6219fa1df9b1980327da3f40bbcf4457a93a6602 Author: Weiqun Zhang Date: Mon Feb 8 18:35:42 2021 -0800 Fix precision issue in ParmParse::add (#1783) When adding values to ParmParse, we need to call `std::setprecision` so as not to lose precision. Src/Base/AMReX_ParmParse.cpp commit ba0f1c0e77515debd83fe651abd3045d87567dab Author: Weiqun Zhang Date: Mon Feb 8 14:57:30 2021 -0800 Fix bug in #1774 (#1779) A bug was introduced in #1774 that resulted in incorrect weights when making nodal RHS from cell-centered RHS. Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H commit d44e353bb9f7745973502e4d7b16bbde3d29ed09 Author: Andrew Myers Date: Mon Feb 8 11:45:59 2021 -0800 Fix up neighbor list for 2D (#1781) Src/Particle/AMReX_NeighborList.H commit 591f48db3c0539fa639e8d846cd4a036a7e63c53 Author: Weiqun Zhang Date: Sat Feb 6 17:38:19 2021 -0800 Remove managed memory from CNS (#1780) Tutorials/GPU/CNS/Exec/RT/cns_prob.cpp Tutorials/GPU/CNS/Exec/RT/cns_prob_parm.H Tutorials/GPU/CNS/Exec/Sod/cns_prob.cpp Tutorials/GPU/CNS/Exec/Sod/cns_prob_parm.H Tutorials/GPU/CNS/Source/CNS.H Tutorials/GPU/CNS/Source/CNS.cpp Tutorials/GPU/CNS/Source/CNS_advance.cpp Tutorials/GPU/CNS/Source/CNS_derive.cpp Tutorials/GPU/CNS/Source/CNS_parm.H Tutorials/GPU/CNS/Source/CNS_setup.cpp commit d31f9b4a36a1cb912d4dd52bb1807117fe9d79f6 Author: Weiqun Zhang Date: Fri Feb 5 19:17:43 2021 -0800 Overset Solver with Refinement Ratio of 4 (#1778) We need to define overset mask for on coarse MG levels of fine AMR levels. We didn't need to do it for refinement ratio of 2 because there are no coarse MG levels on fine AMR levels. Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp commit eb9b4242a07028a77bd95986f5a5686c81dce428 Author: Andrew Myers Date: Thu Feb 4 12:11:37 2021 -0800 Fix a circular include involving PODVector. (#1776) Src/Base/AMReX_PODVector.H commit 5e08d985d3a91ab8ec10d9aec8c318b38ff10849 Author: Andrew Myers Date: Thu Feb 4 12:09:15 2021 -0800 add missing template arguments to the particle HDF5 IO methods (#1777) Src/Particle/AMReX_ParticleHDF5.H commit f8808fcb56e172f9548954240468cf3d18223851 Author: Weiqun Zhang Date: Thu Feb 4 10:28:54 2021 -0800 Refinement Ratio of 4 Support in Nodal Solver (#1774) * Refinement Ratio of 4 Support in Nodal Solver This adds refinement ratio of 4 support in non-EB nodal linear solver. Roundoff errors are expected for multi-level solves with a refinement ratio of 2. * MacProjector refinement ratio of 4 bug Move setLevelBC to project and call it only if it has not been called. This fixes a refinement ratio of 4 bug, because setLevelBC should not be called before coarse/fine ratio is set. Src/Base/AMReX_Box.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp Tutorials/LinearSolvers/NodalPoisson/MyTest.H Tutorials/LinearSolvers/NodalPoisson/MyTest.cpp commit c686e458ffccfa0032f231753c65ea514df8f122 Author: Candace Gilet Date: Wed Feb 3 23:48:45 2021 -0500 For FillPatchTwoLevels, BCs should be set on the coarse box (#1772) since they are only used on the coarse level (when computing slopes). This corrects certain cases where a level of refinement is very close to, but not actually touching an ext_dir or hoextrap boundary. Src/Amr/AMReX_AmrLevel.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H commit ebf64aa13ff7d57013473bdd4df0acf7be144193 Author: Michael Zingale Date: Wed Feb 3 23:34:09 2021 -0500 remove f90cat.py (#1775) it was only ever used with Castro as an experiment Tools/F_scripts/f90cat.py commit c9265f4c85057e94b39db780df42491f20137d19 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed Feb 3 16:06:45 2021 -0800 Make.unknown MPI Detection Update (#1770) Simplify/robust-ify Make.unknown's MPI detection. Tools/GNUMake/sites/Make.unknown commit 39fe5870a75b7dfd9f3b0cdc531ab63f92a9f15b Author: Axel Huebl Date: Wed Feb 3 13:45:56 2021 -0800 MSVC: Proper __cplusplus macro (#1773) Modern MSVC versions finally set the __cplusplus macro correctly, but we need to request this with an additional flag. Tools/CMake/AMReX_Config.cmake commit b00475d69aec7304d18129d6b3ce9c2a2f4d835a Author: Michael Zingale Date: Tue Feb 2 16:03:12 2021 -0500 remove any trailing / from HIP_PATH via realpath (#1771) Tools/GNUMake/comps/hip.mak commit c68493447deeaf1675a6cdfef323d0adb62f4bbe Author: Andrew Myers Date: Tue Feb 2 08:55:09 2021 -0800 Use ParIter instead of MakeMFIter in WriteBinaryParticleDataAsync. (#1767) Src/Particle/AMReX_WriteBinaryParticleData.H commit d2270fab703098760806d9b9297a7502e8918733 Author: Axel Huebl Date: Tue Feb 2 08:35:34 2021 -0800 IntVect::toArray() (#1768) ## Summary Add a ::toArray() conversion member to IntVect that can be used to convert back to an array type and cast the result on the way, e.g. to unsigned. ## Additional background Functions in https://github.com/ECP-WarpX/WarpX/pull/1660 Src/Base/AMReX_IntVect.H commit 4a425335e40fbae69261987e8a97bc094712fb9f Author: Weiqun Zhang Date: Mon Feb 1 08:37:34 2021 -0800 Update CHANGES for 21.02 (#1766) CHANGES commit 466e3071713ced99cdfce4426c283d75da21b2c5 Author: Andrew Myers Date: Sat Jan 30 13:21:11 2021 -0800 Add a Github action that tries to build tutorials with SP particles and DP mesh. (#1764) Currently this combination is only caught by the Nyx CI. .github/workflows/linux.yml commit 302fad36faaa8f5bb65c207c608ca332b1a07116 Author: Candace Gilet Date: Sat Jan 30 16:18:05 2021 -0500 Abort when MLMG is detected as failing regardless of verbosity setting. (#1762) Move Abort() outside if( verbose >0 ) block. Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit f649f04fef6922374f2d2b31474ac87eda35a5f6 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Sat Jan 30 13:16:45 2021 -0800 Anyof (#1757) * DPC++ versions of AnyOf. Written to match CUDA/HIP version. Performance not explicitly tested. * Also, adjustment to AMREX_DEVICE_PRINTF to remove the need for AMREX_DEVICE_COMPILE. Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_Reduce.H Tests/GPU/AnyOf/GNUmakefile Tests/GPU/AnyOf/Make.package Tests/GPU/AnyOf/inputs Tests/GPU/AnyOf/main.cpp commit 896916428ca28b012c7c4ed53812898f2c26f815 Author: jmsexton03 Date: Fri Jan 29 13:32:24 2021 -0800 Use ParticleReal for particle interpolation (#1763) Src/Particle/AMReX_TracerParticles.cpp commit adbb5695469183897014c05824bd44772252f398 Author: Andrew Myers Date: Fri Jan 29 11:59:32 2021 -0800 Fix new logic error with aggregation_type. (#1761) Src/Particle/AMReX_ParticleContainerBase.cpp commit ef3e68bbc1380065f9515ea5c0236fe2f18cda5f Author: Andrew Myers Date: Fri Jan 29 11:51:30 2021 -0800 Remove obsolete interpolation functions from AMReX (#1754) Src/Particle/AMReX_Particle.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.cpp commit 5df10f661e060de45fca677026325fc278b5c70c Author: Andrew Myers Date: Fri Jan 29 11:06:53 2021 -0800 Move more PC methods that don't need the template parameters into base class. (#1760) Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_Particles.H commit f45ede1d49940035f84dc9a8562fef0a2ad3b720 Author: Andrew Myers Date: Thu Jan 28 13:28:26 2021 -0800 make pc.Geom always refer to the particle one. (#1470) Src/Particle/AMReX_Particles.H commit c143555e74e4096ec25400369651bf884b8561b3 Author: Andrew Myers Date: Wed Jan 27 16:04:37 2021 -0800 Fix 'potential null dereference warning' in StructOfArrays (#1759) Src/Particle/AMReX_StructOfArrays.H commit 2cd12635b7d7c2d41ee5ebce07c8003ef7fbe1b0 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed Jan 27 14:31:30 2021 -0800 Gpu Docs Update First Pass (#1758) First pass at updating the GPU docs. More detailed updates to follow. Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst commit 0b79c5f7d855ed0078b63bfe34cf982c0956f06e Author: Andrew Myers Date: Wed Jan 27 09:29:29 2021 -0800 Move DataPrefix and Version... (#1756) * DataPrefix and Version do not need the template arguments, so moving them to a base class. Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H Src/Particle/CMakeLists.txt Src/Particle/Make.package commit b95a0f30003d6eb97fb6e33a5fde5a902221495d Author: Andrew Myers Date: Wed Jan 27 09:19:47 2021 -0800 Remove unused move random methods. (#1755) These methods are not used, and if needed, they should be implemented at the application level. Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_Particles.H commit 02c52cda1b70c77384807d898a7a5c1f705c96c4 Author: Andrew Myers Date: Tue Jan 26 08:50:02 2021 -0800 Add IntVect versions of numParticlesOutOfRange. (#1751) Src/Particle/AMReX_ParticleUtil.H commit 9896c7643f72bc091b119e63ab6f46e5d1a058d2 Author: Axel Huebl Date: Mon Jan 25 16:52:20 2021 -0800 Fix ParIter Ctor (#1752) With user-defined allocators, this lead to: ``` Src/Particle/AMReX_ParIter.H:114:84: error: type ‘amrex::ParIterBase’ is not a direct base of ‘amrex::ParIter<0, 0, 4, 0, amrex::PinnedArenaAllocator>’ 114 | : ParIterBase(pc,level) | ``` in non-const contexts. Src/Particle/AMReX_ParIter.H commit b560557c4f5876337c0eb47df82f0ead37cf52b6 Author: asalmgren Date: Mon Jan 25 13:40:42 2021 -0800 Improve the interpolation from cell centroids to face centroids: (#1750) 1) if the centroids are aligned tangentially to the face but not at equal distance to the face, use just those two cells for interpolation 2) in the case where the face centroid is at 0 but the cell centroids are not aligned, instead of testing on an area fraction tangential to the face to decide which way to go, in 2d, now test on the minimum volume for each option and choose the option with the larger minimum volume -- this should avoid using covered or very small cells. (3D already has logic to avoid using covered cells) Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H commit 3069f1ba4dd899f40ac6092fff2916466af8d980 Author: asalmgren Date: Mon Jan 25 13:40:25 2021 -0800 Fix EB slopes in 3D when two of the domain boundaries with ext_dir bc's meet at an edge -- (#1749) this case was handled incorrectly before Src/EB/AMReX_EB_slopes_K.H commit e93c5705a6c013ffc602a16e877d22ea3afbf1fc Author: Don E. Willcox Date: Sun Jan 24 20:23:00 2021 -0800 In ParticleToMesh and MeshToParticle, use nGrowVect() to accommodate different a different number of grow cells in each dimension. (#1748) Src/Particle/AMReX_ParticleMesh.H commit 162bca40a4ecec77a5d8d1df68b1864bfd15ec6a Author: asalmgren Date: Sat Jan 23 12:50:54 2021 -0800 Add additional tests when interpolating from cell centroids to face centroids (#1746) when the face centroid is exactly at the center of the face we need an additional test to know which way we can look transversely and not hit covered cells. Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H commit cbbd1c5bd623fc59b67a96e1d5cd5339f2b24cbb Author: Weiqun Zhang Date: Fri Jan 22 09:40:57 2021 -0800 Fix single precision build of CNS (#1745) Tutorials/GPU/CNS/Source/main.cpp commit e2b1108dde79e258f4d06e584e745e0ffd2bbd5f Author: Andrew Myers Date: Thu Jan 21 17:37:54 2021 -0800 Add missing template parameters in ParIter.H. (#1744) Src/Particle/AMReX_ParIter.H commit 1656d55a6e001f139f88c5ada93068a7b862b59c Author: Andrew Myers Date: Thu Jan 21 15:23:39 2021 -0800 Template copy/add particles based on src type. (#1741) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_Particles.H commit a63941a8f13fd972b860297de2132185a739418e Author: Axel Huebl Date: Thu Jan 21 15:00:09 2021 -0800 HIP: New Warp Size Macro (#1742) Use new the HIP macro for the warp size on various AMD devices. Src/Base/AMReX_GpuDevice.H commit c2beebd492f1efb202d09cef2f62e5283998e1b7 Author: Andrew Myers Date: Wed Jan 20 17:21:05 2021 -0800 Add allocator template parameter to ParticleContainer (#1416) Src/AmrCore/AMReX_AmrParticles.H Src/Base/AMReX_TypeTraits.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_Particles.H commit bcf034a796031d362b85dba9c88fad24f30b36e6 Author: asalmgren Date: Wed Jan 20 15:21:05 2021 -0800 Reduce compiler warnings (#1740) Co-authored-by: Andrew Myers Src/EB/AMReX_EB_slopes_K.H Src/Particle/AMReX_NeighborList.H commit b2c1b6077a07a23201aa65156ef2a1033943482f Author: Andrew Myers Date: Wed Jan 20 11:44:24 2021 -0800 Fix some new warnings about 'pointless comparisons' with 0 for unsigned types. (#1737) Src/Particle/AMReX_StructOfArrays.H commit 1775ce65c769d0502f727aa1f1955366a172ff2c Author: drangara <69211175+drangara@users.noreply.github.com> Date: Wed Jan 20 14:16:39 2021 -0500 Add ability to calculate slopes on face centroids (#1707) And apply the explicit stencil when computing the laplacian These changes allow the computation of EB viscous laplacian operator when phi is defined on cell centroids by using the Least squares method for calculating slopes on face centroids. Currently only works when Dirichlet BC is set. Src/EB/AMReX_EB_LeastSquares_2D_K.H Src/EB/AMReX_EB_LeastSquares_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Tests/LinearSolvers/LeastSquares/GNUmakefile Tests/LinearSolvers/LeastSquares/Make.package Tests/LinearSolvers/LeastSquares/MyEB.H Tests/LinearSolvers/LeastSquares/MyTest.H Tests/LinearSolvers/LeastSquares/MyTest.cpp Tests/LinearSolvers/LeastSquares/README.md Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/LeastSquares/initEB.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleData.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleDataFor2D.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleDataFor3D.cpp Tests/LinearSolvers/LeastSquares/inputs.2d.askew-x Tests/LinearSolvers/LeastSquares/inputs.2d.askew-y Tests/LinearSolvers/LeastSquares/inputs.2d.base Tests/LinearSolvers/LeastSquares/inputs.2d.fullyrotated Tests/LinearSolvers/LeastSquares/inputs.2d.other Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.xy-x Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.xy-y Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.xz-x Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.xz-z Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.yz-y Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.yz-z Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-all Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-xy Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-xz Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-yz Tests/LinearSolvers/LeastSquares/main.cpp commit 9bda4f80dea74dcef0ccd4cc9dfa60c693613228 Author: Weiqun Zhang Date: Wed Jan 20 09:53:21 2021 -0800 HIP indirect function and kernel fusing (#1739) Add a GNU Make flag, HIP_INDIRECT_FUNCTION=[TRUE|FALSE], and modify the Gpu kernel fusing test. Currently HIP does not support indirect function calls via device function pointers, which our GPU fusing implementation depends on. Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_Reduce.H Tests/GPU/Fuse/GNUmakefile Tests/GPU/Fuse/main.cpp Tools/GNUMake/Make.defs commit 9437e36505e115da96cebebab973d5499f357bb2 Author: Robert Maynard Date: Tue Jan 19 13:39:04 2021 -0500 When using CMake 3.15+ allow for MSVC runtime library changes (#1736) Enabling of CMP0091 needs to go before the first `project` call as that computes if MSVC_RUNTIME_LIBRARY information is used CMakeLists.txt Tools/CMake/AMReX_Config.cmake commit 5791c90593389dbf23502d10c52f01b35e77bfa6 Author: Weiqun Zhang Date: Mon Jan 18 20:49:38 2021 -0800 RoundRobin: option to not sort processes (#1721) Add an optional parameter `sort` to RoundRobin constructors. This makes it consistent with other DistributionMapping strategies. And this fixes an incorrect behavior when RoundRobin is used by KnapSack that has been given `sort=false`. This should be able to fix the hanging issue observed in WarpX simulations with load balancing enabled and more MPI processes than the number of boxes. Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp commit 5d3e31a08af987cf9e127ab07375483c74724743 Author: Weiqun Zhang Date: Sun Jan 17 12:02:00 2021 -0800 EB: edge centroid (#1730) Add edge centroid to the database and make it available via EBFArrayBoxFactory. Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_IF_Rotation.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EB2_ND_C.cpp Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EBDataCollection.cpp Src/EB/AMReX_EBFabFactory.H Src/EB/AMReX_EB_utils.cpp Src/EB/CMakeLists.txt Src/EB/Make.package commit 8b123354e9929bc2515b032c2069e2d924e88e59 Author: Weiqun Zhang Date: Sun Jan 17 11:58:36 2021 -0800 No Fortran needed for building plotfile tools (#1711) Tools/Plotfile/GNUmakefile commit d2ff769fa0ad57243a3c9d0010eb68e2ab049005 Author: Emmanuel Motheau Date: Sun Jan 17 11:57:30 2021 -0800 fix RegridOnly to regrid even with single level grids (#1722) Src/Amr/AMReX_Amr.cpp commit 832ce5e8bcdd224c8946392dfaeb8915e0e02eed Author: Andy Nonaka Date: Sat Jan 16 11:54:48 2021 -0800 Consolidate agglomerate (#1733) a few lines in the docs about consolidation and agglomeration options in the linear solvers Docs/sphinx_documentation/source/LinearSolvers.rst commit eab09708761036232dd650c5ee724af7f28249f4 Author: Axel Huebl Date: Sat Jan 16 11:52:55 2021 -0800 Doc: Break into Debugger (#1732) Describe how to disable AMReX' signal and backtrace handling. Thanks to @sayerhs for pointing me to the right place (again :) ). Docs/sphinx_documentation/source/Basics.rst commit f6b0abc7c32368bafffe8e81a2de18c492c8971a Author: Max Katz Date: Sat Jan 16 11:50:44 2021 -0800 Only add Hypre interfaces to MLMG in 2D/3D (#1734) This is necessary for compiling with both Hypre and MLMG in 1D. Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp commit 889af2f8a8e1ec191627f1fb5b28ac9ed6d5aa9a Author: Max Katz Date: Fri Jan 15 18:13:37 2021 -0800 Don't include Hypre files in GNU make for 1D (#1731) These files cannot be compiled in 1D, so there is no point in adding them. CMake treats this as fatal, but for Castro we would like to use the AMReX Hypre GNU make package even without the actual interfaces. Src/Extern/HYPRE/Make.package commit 549bb3d5675f7cac9fd2d116f8cde1daaa477a09 Author: Andrew Myers Date: Fri Jan 15 17:05:38 2021 -0800 Always assert that tiling is false in RedistributeGPU (#1729) Src/Particle/AMReX_ParticleContainerI.H commit 55b62cb42fb6afcfd2e78c8d68fab80c39180a9a Author: Axel Huebl Date: Fri Jan 15 16:33:01 2021 -0800 Define: _OPENMP -> AMREX_USE_OMP (#1560) Replace the define `_OPENMP` with `AMREX_USE_OMP` for all parallel "backend" implementations and control of MFIter loops. This avoids accidentially enabling OpenMP for the parallel compute components, i.e. when a users explicitly uses OpenMP in auxiliary or implicit dependent functionality, but does not request to also OpenMP-parallelize those sections. Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/AmrCore.rst Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/GPU.rst Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_Extrapolater.cpp Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.cpp Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseFab.cpp Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_Extension.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuUtility.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_OpenMP.H Src/Base/AMReX_PCI.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_PhysBCFunct.H Src/Base/AMReX_Print.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_Utility.cpp Src/Base/AMReX_iMultiFab.cpp Src/Base/AMReX_omp_mod.F90 Src/Boundary/AMReX_BndryRegister.cpp Src/Boundary/AMReX_FabSet.cpp Src/Boundary/AMReX_InterpBndryData.cpp Src/Boundary/AMReX_MultiMask.H Src/Boundary/AMReX_MultiMask.cpp Src/Boundary/AMReX_YAFluxRegister.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EBAmrUtil.cpp Src/EB/AMReX_EBCellFlag.cpp Src/EB/AMReX_EBFluxRegister.cpp Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EB_utils.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_MultiCutFab.cpp Src/EB/AMReX_algoim.cpp Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/PETSc/AMReX_PETSc.cpp Src/Extern/ProfParser/AMReX_BLProfStats.cpp Src/Extern/ProfParser/AMReX_CommProfStats.cpp Src/Extern/ProfParser/AMReX_RegionsProfStats.cpp Src/Extern/hpgmg/BL_HPGMG.cpp Src/F_Interfaces/Octree/AMReX_octree_fi.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMGBndry.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_TracerParticles.cpp Tests/FillBoundaryComparison/main.cpp Tests/LinearSolvers/CellEB2/MyTest.cpp Tests/LinearSolvers/CellOverset/MyTest.cpp Tests/LinearSolvers/MLMG/init_prob.cpp Tests/LinearSolvers/TensorOverset/MyTest.cpp Tests/Particles/ParticleIterator/main.cpp Tools/Postprocessing/C_Src/IntegrateComp.cpp Tutorials/Amr/Advection_AmrCore/Source/AdvancePhiAllLevels.cpp Tutorials/Amr/Advection_AmrCore/Source/AdvancePhiAtLevel.cpp Tutorials/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tutorials/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tutorials/Amr/Advection_AmrCore/Source/DefineVelocity.cpp Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tutorials/Blueprint/CellSortedParticles/CellSortedPC.cpp Tutorials/EB/CNS/Source/CNS.cpp Tutorials/EB/CNS/Source/CNS_advance.cpp Tutorials/EB/Poisson/Poisson.cpp Tutorials/GPU/CNS/Source/CNS.cpp Tutorials/GPU/EBCNS/Source/CNS.cpp Tutorials/LinearSolvers/ABecLaplacian_C/initProb.cpp Tutorials/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tutorials/LinearSolvers/NodalPoisson/MyTest.cpp Tutorials/LinearSolvers/NodeTensorLap/MyTest.cpp Tutorials/Particles/CellSortedParticles/CellSortedPC.cpp commit 2536f1ae9f7b7e73bbeee71d5642aa22e02432fc Author: Robert Maynard Date: Fri Jan 15 14:12:12 2021 -0500 When using CMake 3.15 allow for MSVC runtime library changes (#1724) Tools/CMake/AMReX_Config.cmake commit 40dcbf1f69cb81fc875e4e1db49d33c3292bbe9f Author: Mark Meredith Date: Fri Jan 15 12:04:44 2021 -0500 fix -Werror=sign-compare (#1699) Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_StructOfArrays.H commit 633c971c3258d4a662f91d5620e44834bf8d4c92 Author: asalmgren Date: Thu Jan 14 17:25:58 2021 -0800 Quiet the compiler warnings coming from EB/AMReX_EB_slopes_K.H (#1728) Src/EB/AMReX_EB_slopes_K.H commit fd6f5ce3c255d449d5de3d664269d64d069cbdc9 Author: mic84 Date: Thu Jan 14 15:55:57 2021 -0800 CMake: link tests exes against alias library (#1727) Tests/CMakeLists.txt commit 873c4801ad9b1f262a8865f6b9fe198a4ead29ca Author: Jon Rood Date: Thu Jan 14 15:36:17 2021 -0700 Update Make.nrel to accomodate MPT MPI with CUDA enabled (#1726) For our main computer at NREL, Eagle, we mostly use HPE's MPT for MPI. Until these changes, CUDA did not work with MPT when using the GNU makefiles. Tools/GNUMake/Make.defs Tools/GNUMake/sites/Make.nrel commit bcbdb65b525644fcb1cd61dc57d4e77c19b8eac6 Author: Weiqun Zhang Date: Thu Jan 14 13:38:11 2021 -0800 Fix CNS tutorial for HIP (#1725) Use AMREX_GPU_HOST_DEVICE instead of AMREX_GPU_DEVICE because of we cannot detect the lambda type in HIP. Tutorials/GPU/CNS/Source/CNS.cpp commit 10f3a2348e8cb4d0f52e90d273f70426ef1fcbbc Author: Weiqun Zhang Date: Thu Jan 14 11:14:01 2021 -0800 Cell-centered Hypre and Petsc setup on GPU (#1709) Do cell-centered hypre and Petsc solver setup on GPU. Docs/sphinx_documentation/source/LinearSolvers.rst Src/Base/AMReX.cpp Src/Base/AMReX_BaseFabUtility.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_Habec_K.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.H Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/AMReX_PETSc.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.H Tools/GNUMake/packages/Make.hypre Tutorials/LinearSolvers/ABecLaplacian_C/initProb.cpp commit dd0a3ceb3c62f68ad096b0692352acc776c352ae Author: mic84 Date: Wed Jan 13 12:59:28 2021 -0800 Tests: replace D_DECL with AMREX_D_DECL (#1723) Tests/HDF5Benchmark/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/AsyncIO/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/Intersection/main.cpp Tests/Particles/Redistribute/main.cpp Tests/Particles/SparseBins/main.cpp commit 07488476533e1bf4731f1bd7c17c1b14c0fd13d2 Author: Matt Larsen Date: Mon Jan 11 13:35:37 2021 -0800 fixing assert (#1720) Co-authored-by: Matt Larsen Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 95661e553fbd92c4d41ffa88438272a3ca4f385f Author: Andrew Myers Date: Mon Jan 11 11:38:54 2021 -0800 Fix compiler warnings in CreateVirtualParticles (#1718) Src/Particle/AMReX_ParticleContainerI.H commit f35b856849f22909e2e4395ac73edb263b108434 Author: Andrew Myers Date: Mon Jan 11 11:17:50 2021 -0800 Fix compiler warnings in ParticleInit.H (#1719) Src/Particle/AMReX_ParticleInit.H commit cd56f543e5372230314f1f0acfb6566a754376ed Author: Weiqun Zhang Date: Sun Jan 10 12:46:38 2021 -0800 ParmParse::hasUnusedInputs & getUnusedInputs (#1716) Add functions to find if there are unused `ParmParse` parameters starting with `prefix.`. If `prefix` is empty, all unused `ParmParse` parameters are included. Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit cf17e95632b8bc26107a9ecd9e8dcae75db47c4e Author: asalmgren Date: Sat Jan 9 13:39:48 2021 -0800 There is no longer a CVODE tutorial (#1715) Docs/sphinx_tutorials/source/CVODE_Tutorial.rst Docs/sphinx_tutorials/source/index.rst commit 4e7f200eb91b1d8689b38f31d2ea1e29535094c0 Author: mic84 Date: Fri Jan 8 17:00:27 2021 -0800 CMake: offer option to exclude Src/Amr from build (#1714) Docs/sphinx_documentation/source/BuildingAMReX.rst Src/CMakeLists.txt Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake commit 42f045ea6d39b7a1f02b2a92612352cff585a9b7 Author: Weiqun Zhang Date: Fri Jan 8 16:45:55 2021 -0800 Always have Abort_device etc. defined for GPU build (#1712) It used to be these functions are defined only if NDEBUG is defined. However, the cmake build system does not store `NDEBUG` in `AMReX_Config.H. It's possible that AMReX is built with `NDEBUG` and `Abort_device` is not defined, but the user's code may be built without `NDEBUG`. Then `amrex::Abort` defined in `AMReX.H` would fail to compile. Always having these device functions defined can solve the issue. Src/Base/AMReX.H Src/Base/AMReX.cpp commit 147934bcca2551e24c16bb0b5d4a50575a8a2eec Author: Weiqun Zhang Date: Fri Jan 8 14:57:07 2021 -0800 Prefetch not supported on Windows (#1713) On-demand page migration is not supported on Windows. Src/Base/AMReX_BaseFab.H commit 7eb484bb8fdaec3c46f66921b9baefe7a3b00580 Author: Emmanuel Motheau Date: Fri Jan 8 13:10:39 2021 -0800 Update write plot file to ascii tool (#1700) A flag (fast = 1) has been added to the WritePlotfileToAscii tool to propose to export data via Amrex::Print() instead of the 4 i,j,k,n imbricated loops. Indeed, on large plotfiles, exporting data to an ascii file is extremely slow with the procedure originally implemented in the tool. Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp commit 3daa8ced947e315f8c3cf7506e2eec45c6f8fbf4 Author: Weiqun Zhang Date: Fri Jan 8 10:34:38 2021 -0800 Tweak some floating point comparisons (#1710) Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_RealBox.H Src/Base/AMReX_RealBox.cpp Src/EB/AMReX_EB_utils.cpp commit 6a773594766d842c866cd3cbb74f2fe6f09884d6 Author: Weiqun Zhang Date: Thu Jan 7 14:48:37 2021 -0800 Make bisect more robust (#1708) If the lower and upper bounds are almost equal, break out of the bisect iteration. Src/Base/AMReX_Algorithm.H Src/Base/AMReX_Geometry.cpp commit be30569c55ce9ecf1f6576b4d94cd9040b3d8490 Author: Weiqun Zhang Date: Thu Jan 7 14:17:12 2021 -0800 Support single precision build with configure (#1706) * Add new arguments `--single-precision` and `--single-precision-particles` to `configure`. * Add amrex::almostEqual for comparing floating point numbers. * Fix double to float conversion warnings. * Add CI testing single precision build .github/workflows/linux.yml Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_AuxBoundaryData.cpp Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_extrapolater_3D_K.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_ErrorList.cpp Src/Base/AMReX_Algorithm.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_COORDSYS_2D_C.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FilCC_C.cpp Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_MFCopyDescriptor.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_RealBox.cpp Src/Base/AMReX_VisMF.cpp Src/Boundary/AMReX_InterpBndryData_3D_K.H Src/Boundary/AMReX_MacBndry.cpp Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMGBndry.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.cpp Tests/FillBoundaryComparison/main.cpp Tools/C_util/WritePlotFile.cpp Tools/libamrex/configure.py Tutorials/Amr/Advection_AmrCore/Source/main.cpp Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tutorials/Amr/Advection_AmrLevel/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/main.cpp Tutorials/EB/CNS/Source/CNS_advance.cpp Tutorials/EB/CNS/Source/main.cpp Tutorials/GPU/CNS/Source/main.cpp Tutorials/GPU/EBCNS/Source/main.cpp Tutorials/LinearSolvers/MAC_Projection_EB/main.cpp Tutorials/LinearSolvers/Nodal_Projection_EB/main.cpp commit f84b7b891cfa55affc25426288b4d3881e4be2a7 Author: Andrew Myers Date: Wed Jan 6 15:34:03 2021 -0800 Fix logic in applying periodic bcs when subcycling is on (#1705) Src/Particle/AMReX_ParticleUtil.H commit 20119b405e406040637daff2f4a591670de9093d Author: Axel Huebl Date: Wed Jan 6 11:55:01 2021 -0800 CMake<3.18: Fix Linker Regression (#1701) Roll-back to explicit `-Wl,` linker flags for CMake<3.18 Tools/CMake/AMReX_Config.cmake commit 887a4ce2a3e995592d9dcfd4318682526c297546 Author: Weiqun Zhang Date: Wed Jan 6 10:13:23 2021 -0800 ParallelDescriptor::ReduceReal wrappers (#1703) Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp commit c36b253b295032f83e13edd9de78c40e0e660a10 Author: Luca Fedeli Date: Tue Jan 5 17:47:19 2021 +0100 Fix several warnings related to double-->float conversions (#1698) ## Summary I tried to compile `WarpX` in single precision and I noticed several warnings related to casts from `double` to `float` in `AMReX`. This PR should fix several of these warnings. ## Additional background I compiled `WarpX` with `make DIM=3 PRECISION=FLOAT USE_SINGLE_PRECISION_PARTICLES=TRUE` using the compiler `g++ (Ubuntu 10.2.0-13ubuntu1) 10.2.0` Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H commit 6c6924e66f26586d03d4be661739e7ab600986fa Author: Robert Maynard Date: Tue Jan 5 11:21:23 2021 -0500 Support when CMake 3.18 policy 105 is set to new (#1696) When AMReX is using a newer version of CMake (3.18+) it sets policy `105` to new, causing `target_link_options` rules to be applied to CUDA device linking. When that happens we need to make sure that host only linking flags are excluded. Tools/CMake/AMReX_Config.cmake commit 8aa9810b22c37855f6b0602403138e7ffcea4a7b Author: kngott <30483578+kngott@users.noreply.github.com> Date: Tue Jan 5 08:20:42 2021 -0800 Update FabReduce-isum test in ParallelReduce tutorial. (#1695) Tutorials/GPU/ParallelReduce/main.cpp commit 00c116c09110fb93428624da5cf5a7f0cdaabe43 Author: Matt Larsen Date: Tue Jan 5 08:09:58 2021 -0800 changing blueprint ghost type to int (#1694) This MR changes the type of the ghost field used by conduit blueprint from `double` to `int`. Ascent expects integers, and this reduces the storage space required for the field. Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 5baf0f7fee46a02caa349d4f8aaae26c192acd05 Author: Axel Huebl Date: Mon Jan 4 16:10:03 2021 -0800 CI: Brew Returns Non-Zero If Already Installed (#1697) Homebrew has the unusual behavior to return a non-zero error code in `brew install` if the package is already installed. Since the base image in CI is somewhat fluid and also the status that `brew update` leaves is very dynamic, we just change this ourselves now to be a non-error. .github/workflows/dependencies/dependencies_mac.sh commit 7eafbe22b93d3d9404efd1de5ea830be4915e372 Author: Weiqun Zhang Date: Sun Jan 3 10:12:03 2021 -0800 Optimized function for filling BaseFab of GpuArray (#1692) Add amrex::fill for BaseFab of GpuArray (a.k.a. AoS). The GPU implementation uses shared memory. Tests show it's more than 10x faster than a naive implementation for GpuArray. Also added are a few functions in DPC++ so that I do not have to look up the documentation every time I need to use things like threadIdx and blockIdx. Src/Base/AMReX_BaseFabUtility.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_MultiFabUtil.H commit 693700e6afb62df0531601cb48c2a5ab272784ea Author: Weiqun Zhang Date: Sat Jan 2 09:42:15 2021 -0800 BaseFab::minmax (#1691) Add BaseFab::minmax function and use it in VisMF. Src/Base/AMReX_BaseFab.H Src/Base/AMReX_VisMF.cpp commit 9ce1e4e19e64d7ae57d49a4dd0d17aab5a77a376 Author: Emmanuel Motheau Date: Sat Jan 2 09:40:45 2021 -0800 New feature in the fextract tool to directly export in the csv format (#1634) This PR adds a feature to the fextract tool to directly export in the csv format. Docs/sphinx_documentation/source/Post_Processing.rst Tools/Plotfile/fextract.cpp commit 3e38f80ba5e42cd66d98692edd4cef7075661a5b Author: Weiqun Zhang Date: Sat Jan 2 09:25:05 2021 -0800 Update CHANGES (#1693) CHANGES commit 63a567f7b00e7e7592686a8268ba796c3b886481 Author: Weiqun Zhang Date: Mon Dec 28 09:23:53 2020 -0800 DPCPP: GNU Make USE_GPU variable (#1689) Set USE_GPU=TRUE for dpcpp in Make.defs just like cuda and hip. Tools/GNUMake/Make.defs commit 6309a8b7a142d9bdcaedf9e95df15d6db8d9a8f5 Author: Weiqun Zhang Date: Sat Dec 26 12:20:00 2020 -0800 Optimization of MLCellLinOp::applyBC (#1690) Launch one kernel per box to work on all faces and all components. If the number of boxes exceeds the kernel fusing threshold, the kernels from all boxes will be fused. .github/workflows/linux.yml Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_Box.H Src/Base/AMReX_GpuContainers.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/Particle/AMReX_ParticleTransformation.H commit 8bcc31cf0b4319afe633c1c791cdbb30b7bd7118 Author: Max Katz Date: Thu Dec 24 09:12:05 2020 -0800 Remove old launch macros (#1687) Removes some old macros that were used exclusively for the GPU pragma. Also removes `box_threads_and_blocks()` which was only used by that functionality. Src/Base/AMReX_BLFort.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp commit d3fd6462b288249b992baa5f4e9c03d6d8945979 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed Dec 23 17:12:26 2020 -0800 unsigned long long iMultiFab sum. (#1684) Changes iMultiFab sum to use hardware supported unsigned long long. Using ParallelReduce, this changes sum time on Cori (default CUDA, 10.1) from 0.07546 to 0.001185, 63x increase. Src/Base/AMReX_iMultiFab.cpp commit 021a9a3d9a5678984afd64fdffdea0c5d4912c44 Author: Weiqun Zhang Date: Wed Dec 23 14:44:23 2020 -0800 Sanitize the input of mkconfig (#1685) The mkconfig.py script takes CPPFLAGS and generates AMReX_Config.H. If CPPFLAGS contains -I flags (as advised by homebrew) and the input is not santized, it would generate ill-formed code. Tools/libamrex/mkconfig.py commit b0977fd7aadb9021196807e9ffe1c8bfa2aaa53c Author: Andy Nonaka Date: Wed Dec 23 14:38:33 2020 -0800 WritePlotfileToASCII Dimensionality Checking (#1686) Added dimensionality error checking to WritePlotfileToASCII.cpp Also supports older FBoxLib plotfile types now. Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp commit c1713daf9e6ea60b54eb201ab0b18161a2e28c28 Author: Weiqun Zhang Date: Wed Dec 23 14:22:42 2020 -0800 Gpu::Reduce cleanup (#1681) Remove unused parameter in blockReduce_partial. Move the test for full block into Gpu::Handler class. Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuTypes.H commit cf82b6bcd2fdae99c1d78e856e4ee06bcb762995 Author: Weiqun Zhang Date: Wed Dec 23 14:20:28 2020 -0800 New versions of Reduce::Min, Max and MinMax taking callable (#1679) ## Summary Previously, Reduce::Min, Max and MinMax take a pointer to the data. New versions these functions taking a callable object is added to provide more flexibility. Note that Reduce::Sum already has the callable version. Also modify the implementation of the CPU versions of these functions to use OpenMP reduction instead of atomic when we can. Add optional initial value to some functions. Fix the order of template parameters for consistence. ## Checklist The proposed changes: - [x] add new capabilities to AMReX Src/Base/AMReX_Reduce.H commit c98be7422796a0e29cb539f805d91d19c2a5ba01 Author: Johannes Blaschke Date: Wed Dec 23 11:43:48 2020 -0800 update SumNeighbors to reflect recent Particle API changes (#1680) Src/Particle/AMReX_NeighborParticlesCPUImpl.H commit 7e48821802e61c999dcd6bdc7c0471d82e2fee55 Author: Weiqun Zhang Date: Wed Dec 23 10:28:22 2020 -0800 Gpu::AsyncArray -> Gpu::Buffer (#1676) because there is no need for a host callback function to free memory. Src/Base/AMReX_FabArrayUtility.H commit d080afba2402f0f46d03d80ab6d3264ef1968bad Author: Max Katz Date: Tue Dec 22 20:31:14 2020 -0800 Only issue prefetch if arena uses managed memory (#1678) ## Summary Before issuing a managed memory prefetch we check that the arena uses managed memory. This allows the CUDA implementation to still work with `amrex.the_arena_is_managed=0` while calling this API. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FabArrayUtility.H commit 88abc1e88eb34f7f7db0ecf110d75347a9d384a0 Author: Weiqun Zhang Date: Tue Dec 22 15:22:00 2020 -0800 CUDA CI: add flags to catch common errors (#1677) ## Summary Add `--Werror ext-lambda-captures-this` (nvcc 11 only) and `--Werror cross-execution-space-call` to CUDA CI tests. .github/workflows/linux.yml commit 0f820789c57213729b6598f3fa912bd751916dea Author: Weiqun Zhang Date: Tue Dec 22 15:19:49 2020 -0800 GNU Make: Fix the library flags (#1641) Previously for GPU build, the GNU Make system was very sensitive on which variable to use for adding extra `libraries`. Before a certain location, `LIBRARIES` should be used, but after that `libraries` should be used. This is very error prone. This PR tries to fix this. Now the user should be able to use `LIBRARIES` anywhere. The use of `libraries` should be avoided because it is still the case that it can only be used after a certain location in the make file. Nevertheless, this PR is backward compatible and it does not force the use of `LIBRARIES`. Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules commit a4a5adc298abf8ff6b74306fea7448fdff8c7502 Author: Weiqun Zhang Date: Tue Dec 22 14:54:48 2020 -0800 DPCPP: add sync to mimic CUDA NULL stream behavior (#1675) Src/Base/AMReX_GpuDevice.H commit 6c18f6c72f7a965aeed920aba3bf3b6bd6cc64d0 Author: Axel Huebl Date: Tue Dec 22 14:51:07 2020 -0800 NVCC 11.2: CUDA/C++17 (#1665) This adds CI coverage for C++17 builds with NVCC. Several apps expressed interest to transition to C++17 code-bases in the near future and this adds test coverage. Co-authored-by: Weiqun Zhang .github/workflows/linux.yml Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_ParticleLocator.H Tutorials/GPU/CNS/Source/CNS.cpp commit c8d01767451b03df74afb6f3dfd3da609ca1ffdb Author: Andrew Myers Date: Tue Dec 22 14:31:51 2020 -0800 Remove noexcept from this functor to work around cuda 11 bug. (#1674) Also uses variadic form of `amrex::max`. Tutorials/Particles/NeighborList/MDParticleContainer.cpp commit c777f74b02667f8c9a4929d00f7fb9e835900c53 Author: Weiqun Zhang Date: Tue Dec 22 13:13:36 2020 -0800 PolymorphicArray4 (#1643) PolymorphicArray4 is a thin layer around Array4. It allows us to access both AoS and SoA data with operator(). For example, constexpr int ncomp = 2; Box box(...); BaseFab soa_fab(box, ncomp); auto soa_array = makePolymorphic(soa_fab.array()); int i = ...; int j = ...; int k = ...; soa_array(i,j,k,0) = 0; soa_array(i,j,k,1) = 1; BaseFab> aos_fab(box,1); auto aos_array = makePolymorphic(aos_fab.array()); aos_array(i,j,k,0) = 0; aos_array(i,j,k,1) = 1; Src/Base/AMReX_Array.H Src/Base/AMReX_Array4.H commit afda0382ee0598ea0018894c0a3495c693225bc2 Author: Weiqun Zhang Date: Tue Dec 22 09:33:54 2020 -0800 Remove the check on ref ratio being <= 12 (#1672) Src/Amr/AMReX_Amr.cpp Src/AmrCore/AMReX_AmrMesh.cpp commit 1a8e3e70691a4cdf2183a455d13f2ac529705535 Author: Axel Huebl Date: Mon Dec 21 22:44:25 2020 -0800 CI: OpenMP on Windows (Clang) (#1671) - fix MSVC entry back to MSVC - add OpenMP to Clang .github/workflows/windows.yml commit 6680398a1a4f00424a35cd6d989ffa0f22963197 Author: Weiqun Zhang Date: Mon Dec 21 21:01:01 2020 -0800 Unsigned long long for counting number of particles on GPU (#1670) Because long does not have native atomicAdd support, it's much slower than doing reduction using unsigned long long. Src/Particle/AMReX_ParticleContainerI.H commit c9a571f79d44daba2cae2a4be743be8e29ea348d Author: mic84 Date: Mon Dec 21 18:49:32 2020 -0800 CMake: dowstream projects must enable Fortran if AMReX_FORTRAN=ON (#1669) * CMake: disable Fortran by default * CMake: dowstream projects must enable Fortran if AMReX_FORTRAN=ON Docs/sphinx_documentation/source/BuildingAMReX.rst Src/CMakeLists.txt Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake commit 4e8386ecf2a5575c7e5e7c7d8c741d4f361bb658 Author: Johannes Blaschke Date: Mon Dec 21 16:16:12 2020 -0800 fix neighbor particle 'GetNeighbors' return type (#1668) Src/Particle/AMReX_NeighborParticles.H commit 754451b9325690e63d0ab45f1d68ba1bae5e284a Author: kngott <30483578+kngott@users.noreply.github.com> Date: Mon Dec 21 14:36:16 2020 -0800 FaceLinear fused interpolater variation (with performance comparison!) (#1663) Add a `interp_arr` version of `FaceLinear` to test generality of the new API (and performance comparison, as `FaceDivFree `stencil can't be used in regular API for comparison). Also includes a bit of `interp_arr` cleanup. Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp commit 7b9eb82bec95a5c0e729b3086d614cbdfe2bd71b Author: Andrew Myers Date: Mon Dec 21 14:24:42 2020 -0800 Allow alternate form of "check_pair" function (#1667) This allows users to use a different form of `check_pair` functor that passes the particle indices. This facilitates the creation of "half" neighbor lists. Src/Particle/AMReX_NeighborList.H commit 10ed0e029b6278a8dcf2e4af46623a53f00f1641 Author: Weiqun Zhang Date: Mon Dec 21 13:42:47 2020 -0800 ParallelFor for Reduction (#1658) Add capability for `ParallelFor` to safely do reduction using `deviceReduceSum`, `Min`, etc. The user passes `Gpu::KernelInfo{}.setReduction(true)` to notify `ParallelFor` that this is a parallel reduction, and gives `ParallelFor` a callable that takes `Gpu::Handler`. A `Gpu::Handler` is needed to call `deviceReduceSum`. Also add `Gpu::Buffer` class, whose data pointer can be used as a device destination for `deviceReduceSum`. It also has a `copyToHost` method to copy the device result back to the host. See `Tutorials/GPU/ParallelReduce` for examples of how to use `ParallelFor` for reduction. Also note that the reduction function is OpenMP CPU threads safe. Thus the same code can run on with OpenMP when it is not built for GPU. Co-authored-by: Andrew Myers Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.cpp Src/Base/AMReX_Box.H Src/Base/AMReX_Gpu.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuAsyncArray.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuKernelInfo.H Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/EB/AMReX_EB2_GeometryShop.H Tutorials/GPU/ParallelReduce/GNUmakefile Tutorials/GPU/ParallelReduce/main.cpp commit c8cdfa61f443ecde21debc3ad52665bf48eded8b Author: Andrew Myers Date: Sat Dec 19 11:32:26 2020 -0800 Revert tiling change in #1655 (#1664) Src/Particle/AMReX_ParticleContainerI.H commit 875ebff912b5d3f4edc0e6ad7a983a56374a9e69 Author: Axel Huebl Date: Fri Dec 18 20:36:28 2020 -0800 CI: CUDA 9.1 & 11.0 (#1653) Add a recent CUDA 11 release for cutting-edge coverage, too. .github/workflows/dependencies/dependencies_nvcc11.sh .github/workflows/dependencies/dependencies_nvcc9.sh .github/workflows/linux.yml commit 25d06e82804e3e1b4f3f1b0cea4e9979ed37373c Author: Axel Huebl Date: Fri Dec 18 15:29:58 2020 -0800 CI: Lib + Plotfiles Install (#1662) Test the new install of plotfile tools works in CI. .github/workflows/linux.yml commit cc61f30f4266407225bd8c7354565931b275a07f Author: mic84 Date: Fri Dec 18 13:56:37 2020 -0800 CMake: streamline install of plotfile tools (#1661) CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Src/CMakeLists.txt Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXInstallHelpers.cmake Tools/Plotfile/CMakeLists.txt commit 1c291c0b6bb52be354b77b2abb646216dcb62ff7 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Fri Dec 18 10:39:38 2020 -0800 Add new clean commands to docs. (#1660) Docs/sphinx_documentation/source/BuildingAMReX.rst commit 3b897e1b7e04222ae718c00010085908f7693b41 Author: mic84 Date: Thu Dec 17 17:14:36 2020 -0800 Fix how fextract handle the tolerance options (#1659) Tools/Plotfile/fextract.cpp commit de1701ed9d6cbdf61a8c4c3015c1cf5a9b00f7f5 Author: jmsexton03 Date: Thu Dec 17 17:00:36 2020 -0500 AssignDensity and AssignCellDensitySingleLevel copy -> ParallelCopy (#1633) Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_ParticleContainerI.H commit 07a21fd454317b7b434240d548765c4c4ae4464a Author: Andrew Myers Date: Thu Dec 17 13:29:53 2020 -0800 Have AssignCellDensitySingleLevel respect gpu launch guards. (#1655) Src/Particle/AMReX_ParticleContainerI.H commit e21901db47b85704e29b3800fcad7318b9544e76 Author: Weiqun Zhang Date: Thu Dec 17 12:27:08 2020 -0800 Clean up GPU pragma and CUDA Fortran (#1656) Clean up the remaining GPU pragma and CUDA Fortran. This also fixes a bug in #1650 for codes using the Fortran 77 version of filcc. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Src/Base/AMReX_BCUtil.cpp Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_PhysBCFunct.H Src/Base/AMReX_PhysBCFunct.cpp Src/Base/AMReX_filcc_mod.F90 Src/Base/AMReX_fort_mod.F90 Src/Base/Make.package Tools/GNUMake/Make.defs Tools/GNUMake/comps/hpcsdk.mak Tools/GNUMake/comps/ibm.mak Tools/GNUMake/comps/pgi.mak Tutorials/GPU/Launch/Readme.md commit 63db5cbe23e2ddc419e6895ec69610388519ca9d Author: mic84 Date: Thu Dec 17 11:14:35 2020 -0800 Add features to fextract (#1657) Add the following features to `fextract`: - user option to set the number of decimals in the output of reals - user option to set a cut-off tolerance: anything smaller than the tolerance gets printed as zero - user option to disable the job summary report at the end of the output The job summary report is disabled by default in contrast with the current behaviour. Tools/Plotfile/fextract.cpp commit 1f8891e49f23397b5633b34dae4437619d3b46a8 Author: Andrew Myers Date: Thu Dec 17 10:40:55 2020 -0800 Fix parallel add bug when both USE_GPU and USE_OMP are defined. (#1654) We cannot assume copies are thread safe any time AMREX_USE_GPU is defined, whether or not USE_OMP is also on. Note that this only would have been an issue when compiling with OMP but running on only one thread. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX Src/Base/AMReX_FabArrayBase.cpp commit e63397244cd19a67d1e76fdedce73667d41e8f2c Author: Andy Nonaka Date: Thu Dec 17 10:38:23 2020 -0800 Write a plotfile to ASCII (#1638) This utility reads in a single-level plotfile, copies it to a MultiFab with a single box, then writes out all the data in 'i j k comp ' format to the screen The user can modify this cpp file to write out on certain components, coordinates, row/column formatting, etc. Usage: ./WritePlotfileToASCII2d.gnu.MPI.ex infile=inputFileName Docs/sphinx_documentation/source/Post_Processing.rst Docs/sphinx_documentation/source/Post_Processing_Chapter.rst Docs/sphinx_documentation/source/index.rst Tools/Postprocessing/C_Src/GNUmakefile Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp commit 7b24139e3214b417b54f8d6238131dc484bc6e66 Author: mic84 Date: Wed Dec 16 12:31:01 2020 -0800 CMake: AMRDATA needs Fortran (#1645) Tools/CMake/AMReXOptions.cmake commit 75373313e44803c3732096a2dee413b52ed2d62e Author: mic84 Date: Wed Dec 16 12:30:36 2020 -0800 CMake: fix few quirks with GPU backends variables (#1561) Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake commit b851ce78a2f3f26d53eb101aa0a0266284f44bfb Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed Dec 16 12:23:39 2020 -0800 FaceLinear: Avoid accessing face outside of coarse box. (#1651) ## Summary Fix an index out-of-bound trigger in FaceLinear interpolater. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H commit 08e27d80f3cb0151566b05f964fb4fc30296f56c Author: Axel Huebl Date: Wed Dec 16 12:09:48 2020 -0800 CI: fix macOS (#1652) * CI: fix macOS brew did again some updates and things fail. let's revert our current work-around. * Brew: Add gfortran not pulled automatically anymore .github/workflows/dependencies/dependencies_mac.sh commit d7b6b97354c02f2e37761f8afae0fca36e7a4550 Author: Candace Gilet Date: Wed Dec 16 14:13:52 2020 -0500 Fix NodalProjector::getGradPhi for multilevel case (#1642) ## Summary Two bug fixes for multilevel in NodalProjector: 1. Ensure NodalProjector::getGradPhi() returns the correct result. Previously could return \sum{sigma grad phi} / (2**dim sigma_coarse). 2. Average down the complete projected velocity at the end. For alpha!=1, this ensures the projected velocity is averaged down correctly. This could change the results of multilevel regression tests. If the application uses the affected outputs from NodalProjector as is, then changes are expected. If the application subsequently averages down the outputs, then changes greater than roundoff level are not expected. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 70b7ab9999522a51666027ae3f7f3dc527541290 Author: Max Katz Date: Wed Dec 16 09:10:42 2020 -0800 Remove GPU pragma support from GNU make build system (#1650) This is no longer used. Tools/F_scripts/gpu_fortran.py Tools/F_scripts/write_cuda_headers.py Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/sites/Make.llnl commit affcccf3c900d1d29349aaeb0c5f21c9dd63e637 Author: Max Katz Date: Wed Dec 16 09:08:45 2020 -0800 Remove GPU pragma macros from ArrayLim (#1649) These are no longer used. Src/Base/AMReX_ArrayLim.H commit 78eae95a21df8fa1d0e2a7c86b0437dd68e8aa7a Author: Max Katz Date: Wed Dec 16 09:06:31 2020 -0800 Remove custom MFIter reduction code (#1648) This is no longer in use by any of the astro codes. Src/Base/AMReX_BLFort.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit 2f9277a64058c91a5d0153b77bd2476bd1f85690 Author: Axel Huebl Date: Tue Dec 15 17:06:11 2020 -0800 CMake: Fix Typos in SYCL Warnings (#1647) Just a typo and a missing space in a warning. Tools/CMake/AMReXOptions.cmake commit 006569febeae1329fc0a3e0c2fb17c77e42539c7 Author: Weiqun Zhang Date: Tue Dec 15 16:30:54 2020 -0800 DPCPP MPI wrapper (#1646) ## Summary Replace mpiicpx with mpiicpc because mpiicpc is shipped by Intel, whereas mpiicpx is a script that is only available on JLSE testbeds. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Co-authored-by: Michele Rosso Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/Make.defs Tutorials/GPU/CNS/Source/CNS_bcfill.cpp Tutorials/GPU/CNS/Source/hydro/CNS_hydro_K.H commit cd2e06e47cb586d0b8e2ef3eb0c534548634ad97 Author: Weiqun Zhang Date: Tue Dec 15 10:44:19 2020 -0800 Remove support for old versions of DPCPP (#1644) Src/Base/AMReX_Extension.H Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_RandomEngine.H Src/Base/AMReX_Scan.H commit cd31efb8aee323dafddb6040c0613667cfb5382c Author: Weiqun Zhang Date: Tue Dec 15 09:14:51 2020 -0800 HIP and DPCPP support in configure (#1640) Tools/GNUMake/comps/hip.mak Tools/libamrex/configure.py Tools/libamrex/mkconfig.py commit 66e52d44979d14859e007d769fb1b550ec591cc7 Author: Weiqun Zhang Date: Mon Dec 14 15:45:16 2020 -0800 Fused FillBoundary (#1570) ## Summary Add a new FillBoundary function for a Vector of FabArrays. There are two implementations in this PR. The one being used simply calls `FillBoundary_nowait` and `FillBoundary_finish`. The other version aggregates the MPI messages from multiple FabArrays into one. Tests on summit shows the latter is slower. But I don't want to lose the code, so it is kept for future investigation. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Array4.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_Box.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp commit 6a212600436d83fcfb656ce72d6c810c1b7c1cdd Author: kngott <30483578+kngott@users.noreply.github.com> Date: Mon Dec 14 13:51:13 2020 -0800 Divergence Free Stencil and API (#1483) API for multi-dimensional, face-based, coarse-to-fine interpolation (InterpFromCoarse and FillPatchTwoLevels). Includes a variation of Vanella et. al. stencil (doi:10.1016/j.jcp.2010.05.003, section 3.2), but instead of interpolating from a far coarse cell to solve interior closure problem (highly biased result dependent on chosen coarse cell), this uses least squares with initial guess equal to the average of fine face values across the cell. (Mathematica notebooks of solution are available). Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_1d.F90 Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_GpuLaunch.H Tests/DivFreePatch/GNUmakefile Tests/DivFreePatch/Make.package Tests/DivFreePatch/inputs Tests/DivFreePatch/main.cpp commit 78d6f64fadc606d65e30d4da5f218e941bc7ea59 Author: Weiqun Zhang Date: Mon Dec 14 10:19:51 2020 -0800 Constant coefficient MacProjector (#1636) Optimization of constant coefficient MAC projection by using constant coefficient Poisson solver. Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit 4794c1d1c2e1fd83bc5ac6be8ff71c4893861128 Author: Weiqun Zhang Date: Mon Dec 14 08:39:07 2020 -0800 Option to allow multiple active MFIters (#1631) Add MFIter::allowMultipleMFIters(bool) so that users can disable assertion on the number of active MFIters. This is needed by FLASH because multiple MFIters (one for each level) are built in its iterator. Docs/sphinx_documentation/source/Basics.rst Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/F_Interfaces/Base/AMReX_multifab_fi.cpp Src/F_Interfaces/Base/AMReX_multifab_mod.F90 commit 8e73aa00126634cd83d4ca9c46fbac20da924e4e Author: drangara <69211175+drangara@users.noreply.github.com> Date: Mon Dec 14 11:37:42 2020 -0500 Convert AMReX_eb_to_pvd.F90 to cpp (#1627) ## Summary Convert the fortran file that writes the EB surface for paraview visualization to cpp ## Additional background Tested with the visualization files produced by `mfix/benchamrk/07-hopper` and `mfix/tutorials/clr/prototype`. Paraview visualization seems to look "good". When comparing the actual data from the files I do see some differences b/w the fortran and cpp version, but I believe the differences can be attributed to the precision of datatypes used in the fortran vs cpp code. These changes should not affect the solution results, just affects visualization. Src/EB/AMReX_EBToPVD.H Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_WriteEBSurface.cpp Src/EB/AMReX_WriteEB_F.H Src/EB/AMReX_eb_to_pvd.F90 Src/EB/CMakeLists.txt Src/EB/Make.package commit c0a931f0ca46d62210e5b8f2e92c6952cfd60685 Author: Weiqun Zhang Date: Sat Dec 12 14:42:57 2020 -0800 Fix EB bug in #1629 (#1635) For EB nodal solver with constant sigma, the domain ghost cells of sigma should be set to zero. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit f6f883bfedf2ecf3900c7f79f1f4df3ec92cca76 Author: Weiqun Zhang Date: Sat Dec 12 10:45:05 2020 -0800 Nodal solver with constant coefficient (#1629) * Add specialization for constant coefficient case in the nodal solver. * Update NodalProjector. * Fix some warnings in 1D. Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 85a5f3be91eb9d58845f96c2907af8ee65661f87 Author: Andrew Myers Date: Sat Dec 12 09:44:58 2020 -0800 fix bug in pshifting neighbors introduced in #1590 (#1632) Src/Particle/AMReX_ParticleCommunication.H commit 52d32c6a07cd14498129ec3f03ae69a06a27573e Author: asalmgren Date: Fri Dec 11 21:10:38 2020 -0800 When we create a new level of refinement, we need to set the "level_s… (#1630) …teps" of that new level appropriately. ## Summary ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Amr/AMReX_Amr.cpp commit 389a0cc49f018ea86df0067bc8182336426fe414 Author: Andrew Myers Date: Fri Dec 11 16:24:59 2020 -0800 Expose the index of the neighboring particle to users of the iterator. (#1628) The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Particle/AMReX_NeighborList.H commit 73509be24826d16680289e5555cdb8db1e2217e5 Author: Weiqun Zhang Date: Thu Dec 10 10:02:10 2020 -0800 Documentation on avoiding continue and return inside GPU launch macros. (#1617) Docs/sphinx_documentation/source/GPU.rst commit 728db75fa70fcf10fb7fcb53210ba988ce24f1e4 Author: Weiqun Zhang Date: Thu Dec 10 10:01:16 2020 -0800 Implement overset mask for MLPoisson too (#1604) Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H Src/LinearSolvers/MLMG/Make.package commit 8db1187e6faa4d6628d47ddb28dcc23200516451 Author: Axel Huebl Date: Thu Dec 10 09:47:03 2020 -0800 Warning in SP: large ICC float constant (#1626) Fix an ICC warning for a too-large float constant when compiling for single precision in WarpX. Src/Amr/AMReX_Amr.cpp commit cd88adc2e64456116fcaf0dcce4669921bdb2296 Author: Weiqun Zhang Date: Wed Dec 9 18:57:46 2020 -0800 Fuse nodal jacobi smoother for GPU (#1625) ## Summary Fuse nodal jacobi smoother kernels by switch the order of the sweep and MFIter loops. For Amr-Wind's ABL test with `amr.n_cell = 256 256 64` and `amr.max_grid_size = 32` on one V100, the nodal project is about 2x faster. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit b846969f308e0402e188b95756b478261a31a8d2 Author: Matt Larsen Date: Wed Dec 9 16:26:11 2020 -0800 don't use geom.size() as the number of levels (#1624) ## Summary The conduit adapter was using the `geom.size()` to determine the number of levels, instead of believing `n_levels`. There was a case where the max levels was greater than the actual refinement level. In this case, geoms was the size of the max_levels in the input deck, but mfs was the size of the active levels. resolves #1623 The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 923752e81d08a87d8edd7aaf01e92359125e4f86 Author: OscarAntepara <52221614+OscarAntepara@users.noreply.github.com> Date: Wed Dec 9 08:15:17 2020 -0800 Adding eb slopes limiter (#1594) * Adding eb slope limiters * Adding comments and making 3d adjustments * Fixing face centroids computation in the z-direction and correcting the comments Src/EB/AMReX_EB_slopes_K.H commit ffa5387dd6d873987386cd9ebbea3bab9592c9fd Author: Andrew Myers Date: Tue Dec 8 13:36:03 2020 -0800 Support for SoA data for neighbor particles. (#1590) Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_Particles.H Tests/Particles/NeighborParticles/MDParticleContainer.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/main.cpp Tests/Particles/Redistribute/main.cpp commit a28b02f1d194a0bce76cf784c6dc98cad1339015 Author: Axel Huebl Date: Tue Dec 8 11:13:42 2020 -0800 CMake: Verify DPC++ Compiler ID and at best warn (#1619) * CMake: less stringed MPI+DPCPP Check `find_package(MPI)` is capable to find Intel's MPI with a dpcpp CXX compiler. I would relax this check. Also, `CMAKE_CXX_COMPILER` can be an absolute path, e.g. `-DCMAKE_CXX_COMPILER=$(which dpcpp)`, which is not covered by the original syntax. * DPCPP Compiler Check: Only Warn on ID * Update recommendation * keep leading \n (consistency) Tools/CMake/AMReXOptions.cmake commit e0896cbd9e0432f642d484675e2a4f090ce93e33 Author: Houjun Date: Tue Dec 8 10:53:32 2020 -0800 Fix a bug that causes errors when writing HDF5 plotfile and particle (#1621) * Add HDF5 optimizations * Fix an issue that causes incorrect HDF5 data write with Tutorials/Basic/HeatEquation_EX1_C example code * Minor HDF5 tuning parameter change Co-authored-by: Houjun Tang Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_ParticleHDF5.H commit 82ca8fa508cad15c3f63c6233c8b028da5b4e347 Author: Michael Zingale Date: Tue Dec 8 12:34:11 2020 -0500 the A64fx Cray 10.0.1 compilers need the old-style options (#1620) ## Summary The default cray compilers on the Ookami A64fx machine use the old-style options for cray compilers. This logic allows us to build there. ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/comps/cray.mak commit 91a48a24777dabe03b1ff32bfce05536ce806200 Author: Weiqun Zhang Date: Mon Dec 7 20:00:29 2020 -0800 Fortran Inteface: Porting more MultiFab functions (#1609) Port IntVect ghost cell version of MultiFab::Add, Subtruct, Multiply, Divide, Saxpy, and LinComb to Fortran Interface. Src/F_Interfaces/Base/AMReX_multifab_fi.cpp Src/F_Interfaces/Base/AMReX_multifab_mod.F90 commit 975135ebdec9e0d96f9225a0196fade8ccbd2196 Author: Andrew Myers Date: Mon Dec 7 14:46:19 2020 -0800 [CI] turn off CLEAN option to keep existing files in the deploy target. (#1618) .github/workflows/docs.yml commit 8d45dd95dde0b6cb579c41b13ed3b0b69add297b Author: Axel Huebl Date: Mon Dec 7 13:40:33 2020 -0800 Conduit: Fix unused variables (#1615) ## Summary Removes four (4) unused variable warnings. cc @cyrush @mclarsen ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 774ec481be4fdf28bd80f5a381af9b1c97d5b007 Author: Axel Huebl Date: Mon Dec 7 13:32:59 2020 -0800 DPC++: Re-Enable with Work-around. (#1613) My former colleagues shared a work-around that we can use to overwrite the compiler identification: https://gitlab.kitware.com/cmake/cmake/-/issues/21551#note_869580 This regression will also be fixed in CMake 3.19.2 .github/workflows/linux.yml commit ef6ebdbe62366e78d542fa13739e6d0557a3cbf4 Author: Andrew Myers Date: Mon Dec 7 11:48:19 2020 -0800 deploy to docs to seperate 'build' directory. (#1614) .github/workflows/docs.yml commit 2bbaf1bd81b0b8fee3dbd1bc95f2fc5c22009ac3 Author: Andrew Myers Date: Mon Dec 7 10:55:56 2020 -0800 fix and simplify script that builds the documentation (#1612) build_docs.sh commit 6fb23a385bf2fa0bc2e3b64a24e4940b4c0ea16d Author: Andrew Myers Date: Mon Dec 7 10:32:11 2020 -0800 need to specify the target directory on the website repository (#1611) .github/workflows/docs.yml commit 0a402eef6c0f7bdcb609e786d3140a991e99b7b3 Author: Andrew Myers Date: Mon Dec 7 08:59:32 2020 -0800 Migrate building and deploying the documentation to Github CI. (#1603) .github/workflows/dependencies/documentation.sh .github/workflows/docs.yml .travis.yml Docs/sphinx_tutorials/Makefile build_and_deploy.sh build_docs.sh deploy_key.enc commit 81a56bd628b92745c2baef58d8e7b044306cc34c Author: Weiqun Zhang Date: Sun Dec 6 18:46:48 2020 -0800 Add check for plotfile tools (#1610) .github/workflows/linux.yml Tools/Plotfile/fboxinfo.cpp Tools/Plotfile/fextract.cpp Tools/Plotfile/fextrema.cpp commit 65615e97ef62a2cac302ecbe30482d35f0806542 Author: Weiqun Zhang Date: Sun Dec 6 16:05:16 2020 -0800 Fix error in #1599 (#1608) The proposed change fixes an error in #1599. Tools/Plotfile/fsnapshot.cpp commit 88236accbcc263cbf3d4b103189bc7b522e06dfd Author: Andrew Myers Date: Sun Dec 6 14:33:50 2020 -0800 Remove alternative operators from AMReX (#1599) Operators like `and`, `or`, and `not` are allowed in the C++ standard; however, they cause problems on Windows and with our clang Windows CI. This PR removes them from AMReX in favor of `&&`, `||`, and `!`, and also configures the CI to throw a compilation error if they are re-introduced into the codebase. Application codes are still free to use these if they want. Co-authored-by: Weiqun Zhang .github/workflows/linux.yml .github/workflows/macos.yml Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrParGDB.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_Array4.H Src/Base/AMReX_AsyncOut.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_CuptiTrace.cpp Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_FilCC_C.cpp Src/Base/AMReX_FileSystem.cpp Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_PCI.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_PlotFileDataImpl.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_Slopes_K.H Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Vector.H Src/Base/AMReX_VisMF.cpp Src/Boundary/AMReX_InterpBndryData_2D_K.H Src/Boundary/AMReX_InterpBndryData_3D_K.H Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EBAmrUtil.cpp Src/EB/AMReX_EBFluxRegister_2D_C.H Src/EB/AMReX_EBFluxRegister_3D_C.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EB_slopes_K.H Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_WriteEBSurface.cpp Src/EB/AMReX_algoim_K.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/LinearSolvers/CellOverset/MyTest.cpp Tests/LinearSolvers/EBTensor/MyTest.cpp Tests/LinearSolvers/NodalOverset/MyTest.cpp Tests/LinearSolvers/TensorOverset/MyTest.cpp Tools/Plotfile/fcompare.cpp Tools/Plotfile/fsnapshot.cpp Tools/Postprocessing/C_Src/particle_compare.cpp Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tutorials/LinearSolvers/ABecLaplacian_C/MyTest.cpp Tutorials/Particles/CellSortedParticles/CellSortedPC.cpp Tutorials/Particles/NeighborList/MDParticleContainer.cpp commit ad7974de5b3e68fe2f8fa7c114ffaa460ad166a2 Author: hsitaram Date: Sun Dec 6 14:08:33 2020 -0700 Eb from stl (#1592) ## Summary This is a work-in-progress PR that adds a preliminary set of tools and helper functions for using triangulated STL files in amrex based applications. Coupling with AMReX's EB framework will be added in later pull requests. This PR is done early so as to provide a software framework for incremental changes. As of now, I have added one helper function that provides a nodal multifab with a marker value indicating inside or outside the STL geometry. I guess it can be used for particle-wall collisions. I have also added a test case in the EB folder that generates a nodal multifab with the blanked values for an airfoil. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Co-authored-by: Hariswaran Sitaraman Co-authored-by: Hariswaran Sitaraman Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/CMakeLists.txt Src/EB/Make.package Tutorials/EB/STLtest/GNUmakefile Tutorials/EB/STLtest/Make.package Tutorials/EB/STLtest/airfoil.stl Tutorials/EB/STLtest/inputs Tutorials/EB/STLtest/main.cpp commit 8dc48ad8385cd395310567bf72a36470236e0ce2 Author: Weiqun Zhang Date: Fri Dec 4 17:18:32 2020 -0800 DPCPP: run stencil rap on device (#1606) An earlier compiler bug has been fixed. The stencil rap kernel can run on GPU now. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit 0490136d291cc3b354558f091391277f16a46c5a Author: Weiqun Zhang Date: Fri Dec 4 17:11:29 2020 -0800 Remove DPCPP hacks (#1605) Because the Intel graphics driver has increased the kernel parameters to 2K by default, these hacks are no longer needed. Src/Base/AMReX_GpuLaunchFunctsG.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_Level.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp commit 13d67fd18e736e96bfe4e67a6387750380900384 Author: Axel Huebl Date: Fri Dec 4 10:25:55 2020 -0800 CI: Disable DPC++ oneAPI 2020-11 (#1602) Someone patched in a dedicated CMake compiler ID for Intel DPC++ but did not add full compiler support. Release: 2021.1.1 Ref.: https://gitlab.kitware.com/cmake/cmake/-/issues/21551 .github/workflows/linux.yml commit a168f5927cee0d87ff839b23128c8fc878622313 Author: mic84 Date: Thu Dec 3 16:19:42 2020 -0800 CMake: bring back check on CUDA computing capabilities (#1600) Tools/CMake/AMReX_SetupCUDA.cmake commit 2a070723e249e3e4cf2befd61a576330fc280d80 Author: Andrew Myers Date: Thu Dec 3 14:05:14 2020 -0800 move the memcpy wrapper into amrex::Gpu (#1598) Src/Base/AMReX_GpuUtility.H commit 5b436831f47158a50738dcd64a6fe579c9ef8861 Author: Andrew Myers Date: Thu Dec 3 12:35:20 2020 -0800 Add wrapper for memcpy. (#1597) `std::memcpy` does not seem to work in device code in our HIP CI. This adds a wrapper that reverts to the global namespace `memcpy` in HIP device code. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_GpuUtility.H commit ef665e2a2fcc245162767b3613aba48bbc59f958 Author: Axel Huebl Date: Thu Dec 3 11:16:38 2020 -0800 Fix: Windows and/or includes (iso646) (#1593) ## Summary Add missing includes to `AMReX_Extension.H`, which pulls the `` include on Windows for support for `and`/`or`. ## Additional background Seen with Clang 11.0.0 on x64 Windows with Visual Studio 2017 on conda-forge during a WarpX build. See also #947 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Extension.H Src/Base/AMReX_Vector.H Src/Particle/AMReX_ParticleTile.H commit 5c1addce382f86f0701e6bddc209522cfe70c52d Author: Axel Huebl Date: Thu Dec 3 10:33:42 2020 -0800 CI: Tutorial with OMP (#1591) Making sure the OpenMP Tutorials build as well. .github/workflows/linux.yml Tutorials/Particles/ElectromagneticPIC/CMakeLists.txt commit 0d000b5397b5f6ef94924d85b65846c5811faf86 Author: Weiqun Zhang Date: Thu Dec 3 10:23:15 2020 -0800 AMReX_Config.H (#1566) Based on discussions with @sayerhs, @ax3l, and @mic84, We have decided to make the following changes to the build systems. All headers in Src/ will include AMReX_Config.H, which will be generated by the build systems (both gnu make and cmake). The file will include all the macros we currently pass as compiler options. The purpose of this is to solve some inconsistency issues in the current systems. We support both super build and using AMReX as a library. Moreover, we do not want to force libamrex users to use the same build tool that is used to build the library. In this PR, we are put all the macro definitions in AMReX_Config.H that is included by every AMReX header. Co-authored-by: Axel Huebl Co-authored-by: Michele Rosso Src/Amr/AMReX_Amr.H Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AuxBoundaryData.H Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Extrapolater.H Src/Amr/AMReX_LevelBld.H Src/Amr/AMReX_PROB_AMR_F.H Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_extrapolater_1D_K.H Src/Amr/AMReX_extrapolater_2D_K.H Src/Amr/AMReX_extrapolater_3D_K.H Src/Amr/AMReX_extrapolater_K.H Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrParGDB.H Src/AmrCore/AMReX_AmrParticles.H Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_FLUXREG_F.H Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_F.H Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FluxReg_1D_C.H Src/AmrCore/AMReX_FluxReg_2D_C.H Src/AmrCore/AMReX_FluxReg_3D_C.H Src/AmrCore/AMReX_FluxReg_C.H Src/AmrCore/AMReX_FluxRegister.H Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_TagBox.H Src/Base/AMReX.H Src/Base/AMReX_Algorithm.H Src/Base/AMReX_Arena.H Src/Base/AMReX_Array.H Src/Base/AMReX_Array4.H Src/Base/AMReX_ArrayLim.H Src/Base/AMReX_AsyncOut.H Src/Base/AMReX_BArena.H Src/Base/AMReX_BCRec.H Src/Base/AMReX_BCUtil.H Src/Base/AMReX_BC_TYPES.H Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_BLFort.H Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_BLassert.H Src/Base/AMReX_BackgroundThread.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseFabUtility.H Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BaseUmap_f.H Src/Base/AMReX_BlockMutex.H Src/Base/AMReX_Box.H Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxDomain.H Src/Base/AMReX_BoxIterator.H Src/Base/AMReX_BoxList.H Src/Base/AMReX_CArena.H Src/Base/AMReX_CONSTANTS.H Src/Base/AMReX_COORDSYS_1D_C.H Src/Base/AMReX_COORDSYS_2D_C.H Src/Base/AMReX_COORDSYS_3D_C.H Src/Base/AMReX_COORDSYS_C.H Src/Base/AMReX_CoordSys.H Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_CuptiTrace.H Src/Base/AMReX_DArena.H Src/Base/AMReX_Dim3.H Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_EArena.H Src/Base/AMReX_Exception.H Src/Base/AMReX_Extension.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FPC.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_FabConv.H Src/Base/AMReX_FabFactory.H Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_FilCC_C.H Src/Base/AMReX_FilND_C.H Src/Base/AMReX_FileSystem.H Src/Base/AMReX_ForkJoin.H Src/Base/AMReX_Functional.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_Gpu.H Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuAssert.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuComplex.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuElixir.H Src/Base/AMReX_GpuError.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuKernelInfo.H Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchGlobal.H Src/Base/AMReX_GpuLaunchMacrosC.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuRange.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_INT.H Src/Base/AMReX_IndexSequence.H Src/Base/AMReX_IndexType.H Src/Base/AMReX_IntConv.H Src/Base/AMReX_IntVect.H Src/Base/AMReX_LayoutData.H Src/Base/AMReX_Lazy.H Src/Base/AMReX_Loop.H Src/Base/AMReX_MFCopyDescriptor.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_Machine.H Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MakeType.H Src/Base/AMReX_Math.H Src/Base/AMReX_MemPool.H Src/Base/AMReX_MemProfiler.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H Src/Base/AMReX_MultiFabUtil_C.H Src/Base/AMReX_MultiFabUtil_nd_C.H Src/Base/AMReX_NFiles.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_OpenMP.H Src/Base/AMReX_Orientation.H Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelReduce.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_Partition.H Src/Base/AMReX_Periodicity.H Src/Base/AMReX_PhysBCFunct.H Src/Base/AMReX_PlotFileDataImpl.H Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_Print.H Src/Base/AMReX_REAL.H Src/Base/AMReX_Random.H Src/Base/AMReX_RandomEngine.H Src/Base/AMReX_RealBox.H Src/Base/AMReX_RealVect.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_SPACE.H Src/Base/AMReX_Scan.H Src/Base/AMReX_Slopes_K.H Src/Base/AMReX_ThirdPartyProfiling.H Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Utility.H Src/Base/AMReX_Vector.H Src/Base/AMReX_VectorIO.H Src/Base/AMReX_VisMF.H Src/Base/AMReX_ccse-mpi.H Src/Base/AMReX_filcc_f.H Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_parstream.H Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_BoundCond.H Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_InterpBndryData.H Src/Boundary/AMReX_InterpBndryData_1D_K.H Src/Boundary/AMReX_InterpBndryData_2D_K.H Src/Boundary/AMReX_InterpBndryData_3D_K.H Src/Boundary/AMReX_InterpBndryData_K.H Src/Boundary/AMReX_LOUtil_K.H Src/Boundary/AMReX_LO_BCTYPES.H Src/Boundary/AMReX_MacBndry.H Src/Boundary/AMReX_Mask.H Src/Boundary/AMReX_MultiMask.H Src/Boundary/AMReX_YAFluxRegister.H Src/Boundary/AMReX_YAFluxRegister_1D_K.H Src/Boundary/AMReX_YAFluxRegister_2D_K.H Src/Boundary/AMReX_YAFluxRegister_3D_K.H Src/Boundary/AMReX_YAFluxRegister_K.H Src/CMakeLists.txt Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Graph.H Src/EB/AMReX_EB2_IF.H Src/EB/AMReX_EB2_IF_AllRegular.H Src/EB/AMReX_EB2_IF_Base.H Src/EB/AMReX_EB2_IF_Box.H Src/EB/AMReX_EB2_IF_Complement.H Src/EB/AMReX_EB2_IF_Cylinder.H Src/EB/AMReX_EB2_IF_Difference.H Src/EB/AMReX_EB2_IF_Ellipsoid.H Src/EB/AMReX_EB2_IF_Extrusion.H Src/EB/AMReX_EB2_IF_Intersection.H Src/EB/AMReX_EB2_IF_Lathe.H Src/EB/AMReX_EB2_IF_Plane.H Src/EB/AMReX_EB2_IF_Polynomial.H Src/EB/AMReX_EB2_IF_Rotation.H Src/EB/AMReX_EB2_IF_Scale.H Src/EB/AMReX_EB2_IF_Sphere.H Src/EB/AMReX_EB2_IF_Spline.H Src/EB/AMReX_EB2_IF_Torus.H Src/EB/AMReX_EB2_IF_Translation.H Src/EB/AMReX_EB2_IF_Union.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_MultiGFab.H Src/EB/AMReX_EBAmrUtil.H Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EBFArrayBox.H Src/EB/AMReX_EBFabFactory.H Src/EB/AMReX_EBFluxRegister.H Src/EB/AMReX_EBFluxRegister_2D_C.H Src/EB/AMReX_EBFluxRegister_3D_C.H Src/EB/AMReX_EBFluxRegister_C.H Src/EB/AMReX_EBInterpolater.H Src/EB/AMReX_EBMultiFabUtil.H Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EBMultiFabUtil_C.H Src/EB/AMReX_EBSupport.H Src/EB/AMReX_EB_slopes_K.H Src/EB/AMReX_EB_utils.H Src/EB/AMReX_MultiCutFab.H Src/EB/AMReX_WriteEBSurface.H Src/EB/AMReX_WriteEB_F.H Src/EB/AMReX_algoim.H Src/EB/AMReX_algoim_K.H Src/EB/AMReX_distFcnElement.H Src/Extern/Conduit/AMReX_Conduit_Blueprint.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_Habec_K.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap2.H Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/Extern/HYPRE/CMakeLists.txt Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/CMakeLists.txt Src/Extern/ProfParser/AMReX_AVGDOWN_F.H Src/Extern/ProfParser/AMReX_BLProfStats.H Src/Extern/ProfParser/AMReX_BLProfUtilities.H Src/Extern/ProfParser/AMReX_BLWritePlotFile.H Src/Extern/ProfParser/AMReX_CommProfStats.H Src/Extern/ProfParser/AMReX_RegionsProfStats.H Src/Extern/ProfParser/CMakeLists.txt Src/Extern/SENSEI/AMReX_AmrDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrInSituBridge.H Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrMeshInSituBridge.H Src/Extern/SENSEI/AMReX_InSituBridge.H Src/Extern/SENSEI/AMReX_InSituUtils.H Src/Extern/SENSEI/CMakeLists.txt Src/Extern/SWFFT/Dfft.H Src/Extern/amrdata/AMReX_AmrData.H Src/Extern/amrdata/AMReX_AmrvisConstants.H Src/Extern/amrdata/AMReX_DataServices.H Src/Extern/amrdata/AMReX_XYPlotDataList.H Src/Extern/hpgmg/BL_HPGMG.H Src/F_Interfaces/AmrCore/AMReX_FAmrCore.H Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.H Src/F_Interfaces/Base/AMReX_FPhysBC.H Src/F_Interfaces/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp_F.H Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMGBndry.H Src/LinearSolvers/MLMG/AMReX_MLMG_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLTensor_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_K.H Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_BinIterator.H Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleBufferMap.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleMPIUtil.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particle_mod_K.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_SparseBins.H Src/Particle/AMReX_StructOfArrays.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.H Src/Particle/AMReX_WriteBinaryParticleData.H Src/Particle/CMakeLists.txt Src/SDC/AMReX_SDCstruct.H Tools/CMake/AMReXGenerateConfigHeader.cmake Tools/CMake/AMReXInstallHelpers.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/CMake/AMReX_Config.cmake Tools/CMake/AMReX_Defines.cmake Tools/CMake/AMReX_Utils.cmake Tools/CMake/modify_installed_headers.cmake Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/libamrex/mkconfig.py Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H commit 5fb87ea35fa521e9ce97dc9d36f21eda5e2343be Author: Shreyas Ananthan Date: Thu Dec 3 09:19:23 2020 -0700 Restore reproducibility of solutions with MacProjector reuse (#1596) Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit ae4cc9a1488719e284409b078887866fe65709d9 Author: mic84 Date: Wed Dec 2 15:12:33 2020 -0800 CMake: output missing components list when AMReX is not found (#1572) Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXInstallHelpers.cmake commit 75c0896594cd38d883d7907466657eac3ca1a188 Author: Axel Huebl Date: Wed Dec 2 15:09:17 2020 -0800 CI: Windows + Clang (#1588) Add Clang as Windows target. This compiler offers potentially a route to modern OpenMP. (MSVC: 2.0 standard, LLVM/Clang: 3.1 standard) .github/workflows/windows.yml commit 91bdc795d598b8bbf58d640d572f27dfa5ae31a5 Author: Shreyas Ananthan Date: Wed Dec 2 14:44:56 2020 -0700 MacProjector: Update setDomainBC API (#1589) Allow `setDomainBC` to be called before `setUMAC` has been called. Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit 820a30d7966426a96e24e3963b37f3789eec258f Author: Andrew Myers Date: Wed Dec 2 13:26:03 2020 -0800 Fix InitNRandomPerCell for single precision particles (#1582) * due to roundoff errors, we need to redraw these numbers sometimes when single precision particles are used. * missing ; Src/Particle/AMReX_ParticleInit.H commit dc0ec74c4d120509f9754fb7d43af6d275d9c1a4 Author: Shreyas Ananthan Date: Wed Dec 2 13:35:05 2020 -0700 DPCPP: Update CMake options to allow building with MPI+DPCPP (#1586) Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake commit 89220ed73d9fe74618386f8b1bfafd30760d6fde Author: Andrew Myers Date: Tue Dec 1 10:31:23 2020 -0800 Reimplement NumParticlesAtLevel (#1581) This removes a workaround that had this function fall back to CPUs for CUDA (but not HIP or DPC++). Additionally, this re-implements the function in terms of ReduceTuple to avoid multiple memcpy operations in the case of multiple boxes. Src/Particle/AMReX_ParticleContainerI.H Tests/Particles/ParticleReduce/main.cpp commit bbcce736ea9497cdb4722e8b4f4c2ee0d7c193ca Author: Robert Maynard Date: Tue Dec 1 13:24:16 2020 -0500 eval_genex correctly replaces the matching compiler version (#1580) Previously it would place the resulting output in the variable `1` instead of `_in`. Additionally it would try to add an extra `>`. Tools/CMake/AMReXGenexHelpers.cmake commit 9a523cf95dc3046081b0830cf435a3455929728d Author: Robert Maynard Date: Tue Dec 1 13:20:19 2020 -0500 Add msvc cuda support (#1573) ## Summary This set of changes has allowed me to build AMReX on windows using CUDA 11.X and VS2019. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_ErrorList.cpp Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_Extension.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_MultiFab.H Tools/CMake/AMReX_Config.cmake commit 1cb99250ecd4da445cfc8afe325664030c49c4f0 Author: Weiqun Zhang Date: Tue Dec 1 10:12:38 2020 -0800 Update CHANGES (#1584) CHANGES commit 94ca530cb43c497cf22dfbfbd13dfd087f7d910d Author: Andrew Myers Date: Mon Nov 30 21:55:26 2020 -0500 Fix bug affecting async IO + single precision particles (#1583) The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Particle/AMReX_WriteBinaryParticleData.H commit d9af9ab7f134b04bc5e3cc57362e82a64fecef0a Author: Weiqun Zhang Date: Mon Nov 30 12:49:50 2020 -0800 Fix warnings for DPCPP (#1579) Fix unused variable. Do not pass `-fsycl-unnamed-lambda` to `mpiicpx` for linking to get rid of warning on unknown argument. Src/Base/AMReX_FBI.H Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tutorials/GPU/CNS/Source/CNS.H commit 07afa8ec8f59403ac54c2b2c566c1c88b1863112 Author: Weiqun Zhang Date: Mon Nov 30 10:25:56 2020 -0800 Template specialization of MPI communication functions for char (#1569) * Add template specialization of MPI communication functions for char. Select a proper data type according to the size. * Use the new functions in FabArray and Particle communication. And this also fixes a bug in Particle communication. (The counts should be divided by the size of data type.) * Due to a gcc bug on explicit specialization in a namespace, workaround was implemented in AMReX_ParallelDescriptor.H and .cpp. Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Particle/AMReX_ParticleCommunication.H commit 50c40c46382a83cdc98fd59e337c5037aba07c1e Author: Weiqun Zhang Date: Mon Nov 30 10:16:16 2020 -0800 EB2::PolyIF (#1567) The existing PolynomialIF contains a dynamic size vector. This makes it hard to support it on GPU. We have used an undocumented feature to get this run on GPU. However, this code can no longer compile, because the GNU make system now makes cross execution space calls an error. So we disable GPU support for PolynomialIF and add a new PolyIF class with fixed size array and therefore GPU support. Src/EB/AMReX_EB2_IF_Polynomial.H commit 26e4d30f346cb89921502e409da1b933a60f9c81 Author: Weiqun Zhang Date: Mon Nov 30 10:11:37 2020 -0800 Add .gitattributes (#1578) Make github detect *.H as C++. This file is borrowed from Castro. .gitattributes commit a77968674f3d9ec1d4298d99a506dd1951f88c67 Author: Shreyas Ananthan Date: Mon Nov 30 09:17:48 2020 -0700 Tools/Plotfile: Fix build errors in single precision mode (#1577) This PR fixes one build error and one warning observed when compiling `Tools/Plotfile` utilities with `-DAMREX_USE_FLOAT` Tools/Plotfile/fcompare.cpp Tools/Plotfile/fextract.cpp commit 82eceae29419ceec24450926af1f5445ac425d00 Author: Shreyas Ananthan Date: Mon Nov 30 09:14:54 2020 -0700 Arena: Add accessor to query Arena attributes through info object (#1575) ## Summary This PR adds an accessor to the `Arena` class that can be used to query the attributes of the Arena (e.g., `device_use_managed_memory` flag) from the application codes. ## Additional background Currently, there is no easy way to determine if `amrex::The_Arena()` is using managed memory. Applications would have to use a combination of `AMREX_USE_HIP` compile-time definition and `amrex.the_arena_is_managed` ParmParse variable to determine this. Testing individual pointers with `amrex::Gpu::isManaged()` is possible only on CUDA/HIP but not DPC++. Also it is costly compared to just querying the `Arena` object. Allowing read-only access to `ArenaInfo` object will allow a wide variety of checks in application code. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [X] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Arena.H commit acdb7eac1d5a1744e87c0caafd1c43cc5e1171e9 Author: Luca Fedeli Date: Mon Nov 30 16:58:32 2020 +0100 Fix some warnings related to double-->float conversions (#1571) I tried to compile `WarpX` in single precision and I noticed several warnings related to casts from `double` to `float` in `AMReX`. This PR should fix some of these warnings. I plan to continue in the next days with other PRs, if you agree. Src/AmrCore/AMReX_AmrMesh.H Src/Base/AMReX_CoordSys.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_nd_C.H commit abdab63376660238396626ee064130a5c1826024 Author: Shreyas Ananthan Date: Sun Nov 29 14:04:44 2020 -0700 Fix assertion check for MacProjection (#1576) Fixes bug introduced in #1574 Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit abeb357db57261fe8a8d811e161da0c6c2814d92 Author: Shreyas Ananthan Date: Sun Nov 29 09:53:42 2020 -0700 Allow MacProjection reuse (#1574) This commit modifies the MacProjector interface to allow reuse of the MacProjector. The changes to the API are backwards compatible. - Add a new constructor that does not require specification of beta, umac etc. - Add `initProjector` to explicitly initialize the projection instance at app level - Add methods to set MAC velocities, divU etc. Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit cc8f323c426f3957619c93477af2b9e32e3b49c2 Author: Axel Huebl Date: Mon Nov 23 22:27:06 2020 -0800 CI: Make DPC++ Pretty (#1568) * CI: Make DPC++ Pretty Just combing the show pony. * oneAPI Activation: still needs set +e .github/workflows/linux.yml commit f068164b513c12828dac069510d64f8b4912028e Author: mic84 Date: Mon Nov 23 14:48:58 2020 -0800 Remove SUNDIALS support (#1559) ## Summary SUNDIALS support in AMReX consisted of only few Fortran interfaces to SUNDIALS functions for the convenience of app codes. Since there are no AMReX functionalities depending on SUNDIALS, support for SUNDIALS will be removed. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/SUNDIALS.rst Docs/sphinx_documentation/source/SUNDIALS_CVODE.rst Docs/sphinx_documentation/source/SUNDIALS_top.rst Docs/sphinx_tutorials/source/SUNDIALS_Tutorial.rst Src/CMakeLists.txt Src/Extern/CVODE/Make.package Src/Extern/CVODE/cvode_interface.f90 Src/Extern/CVODE/fnvector_serial.f90 Src/Extern/CVODE/integrator_stats.f90 Src/Extern/CVODE/sundials_fdlsmat.f90 Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package Src/Extern/SUNDIALS/arkode_interface.f90 Src/Extern/SUNDIALS/cvode_interface.f90 Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/FindSUNDIALS.cmake Tools/GNUMake/Make.defs Tools/GNUMake/packages/Make.cvode Tools/GNUMake/packages/Make.sundials Tutorials/CMakeLists.txt Tutorials/CVODE/EX1_F/GNUmakefile Tutorials/CVODE/EX1_F/Make.package Tutorials/CVODE/EX1_F/inputs Tutorials/CVODE/EX1_F/integrate_ode.f90 Tutorials/CVODE/EX1_F/main.cpp Tutorials/CVODE/EX1_F/myfunc_F.H Tutorials/CVODE/EX1_F/ode_mod.f90 Tutorials/CVODE/EX2_F/GNUmakefile Tutorials/CVODE/EX2_F/Make.package Tutorials/CVODE/EX2_F/inputs Tutorials/CVODE/EX2_F/integrate_ode_no_jac.f90 Tutorials/CVODE/EX2_F/integrate_ode_with_jac.f90 Tutorials/CVODE/EX2_F/main.cpp Tutorials/CVODE/EX2_F/myfunc_F.H Tutorials/CVODE/EX2_F/ode_mod.f90 Tutorials/SUNDIALS/EX-CUSOLVER/GNUmakefile Tutorials/SUNDIALS/EX-CUSOLVER/Make.CVODE Tutorials/SUNDIALS/EX-CUSOLVER/Make.package Tutorials/SUNDIALS/EX-CUSOLVER/README.md Tutorials/SUNDIALS/EX-CUSOLVER/extern_probin.template Tutorials/SUNDIALS/EX-CUSOLVER/inputs Tutorials/SUNDIALS/EX-CUSOLVER/inputs_128 Tutorials/SUNDIALS/EX-CUSOLVER/inputs_256 Tutorials/SUNDIALS/EX-CUSOLVER/inputs_32 Tutorials/SUNDIALS/EX-CUSOLVER/inputs_64 Tutorials/SUNDIALS/EX-CUSOLVER/main.cpp Tutorials/SUNDIALS/EX-CUSOLVER/make_cuda.sh Tutorials/SUNDIALS/EX-CUSOLVER/make_cuda_cusolver.sh Tutorials/SUNDIALS/EX-CUSOLVER/make_serial.sh Tutorials/SUNDIALS/EX-CUSOLVER/react_cuda.cpp Tutorials/SUNDIALS/EX-CUSOLVER/react_cuda_cusolver.cpp Tutorials/SUNDIALS/EX-CUSOLVER/react_serial.cpp Tutorials/SUNDIALS/EX-CUSOLVER/react_system.F90 Tutorials/SUNDIALS/EX-CUSOLVER/react_utils.F90 Tutorials/SUNDIALS/EX-CUSOLVER/test_react.H Tutorials/SUNDIALS/EX-CUSOLVER/test_react_F.H Tutorials/SUNDIALS/EX1_C/CMakeLists.txt Tutorials/SUNDIALS/EX1_C/GNUmakefile Tutorials/SUNDIALS/EX1_C/Make.package Tutorials/SUNDIALS/EX1_C/SetIC.f90 Tutorials/SUNDIALS/EX1_C/inputs Tutorials/SUNDIALS/EX1_C/inputs_2box Tutorials/SUNDIALS/EX1_C/inputs_non_vectorized Tutorials/SUNDIALS/EX1_C/main.cpp Tutorials/SUNDIALS/EX1_C/myfunc_F.H Tutorials/SUNDIALS/EX1_CUDA/GNUmakefile Tutorials/SUNDIALS/EX1_CUDA/Make.package Tutorials/SUNDIALS/EX1_CUDA/SetIC.f90 Tutorials/SUNDIALS/EX1_CUDA/inputs Tutorials/SUNDIALS/EX1_CUDA/main.cpp Tutorials/SUNDIALS/EX1_CUDA/myfunc_F.H Tutorials/SUNDIALS/EX1_CUDA/ode_mod.f90 Tutorials/SUNDIALS/EX1_F/GNUmakefile Tutorials/SUNDIALS/EX1_F/Make.package Tutorials/SUNDIALS/EX1_F/inputs Tutorials/SUNDIALS/EX1_F/integrate_ode.f90 Tutorials/SUNDIALS/EX1_F/main.cpp Tutorials/SUNDIALS/EX1_F/myfunc_F.H Tutorials/SUNDIALS/EX1_F/ode_mod.f90 Tutorials/SUNDIALS/EX2_F/GNUmakefile Tutorials/SUNDIALS/EX2_F/Make.package Tutorials/SUNDIALS/EX2_F/inputs Tutorials/SUNDIALS/EX2_F/integrate_ode_no_jac.f90 Tutorials/SUNDIALS/EX2_F/integrate_ode_with_jac.f90 Tutorials/SUNDIALS/EX2_F/main.cpp Tutorials/SUNDIALS/EX2_F/myfunc_F.H Tutorials/SUNDIALS/EX2_F/ode_mod.f90 commit e1ff59c8d9207a135f25284380a1ba8fc15abb1f Author: Roberto Porcu <53792251+rporcu@users.noreply.github.com> Date: Sun Nov 22 23:19:06 2020 -0500 allow MLEBABecLap to solve for multiple components (#1557) ## Summary MLLinOp::getNComp() is returning 1 and right now MLEBABecLap is working only with MultiFabs having 1 component. ## Additional background This PR has the goal to generalize MLEBABecLap solver for a generic N number of components. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Co-authored-by: Weiqun Zhang Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp commit 1f8163c397e1aa33fd633219166b87d4633a0b46 Author: Weiqun Zhang Date: Sun Nov 22 19:40:00 2020 -0800 DPCPP: MPI (#1556) Allow MPI for dpcpp build in gnu make assuming mpiicpx is used. Tools/GNUMake/Make.defs Tools/GNUMake/Make.machines Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/sites/Make.alcf commit a85fd8d7971f9448b8a00e796a392bbe84ae0f97 Author: Axel Huebl Date: Fri Nov 20 10:28:25 2020 -0800 CMake: OpenMP for Fortran (#1563) * CMake: OpenMP for Fortran Fix missing activation. * Update Tools/CMake/AMReXParallelBackends.cmake Co-authored-by: mic84 * CMakeL OpenMP Fortran fix Config, too Co-authored-by: mic84 Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXParallelBackends.cmake commit 7a0fea34be764441014e2ba7dac44fecbdbbabfb Author: Weiqun Zhang Date: Thu Nov 19 11:01:07 2020 -0800 Non-local boundary conditions (#1544) * Non-local boundary conditions * Rotate90 fills the lo-x and lo-y boundary regions by rotating the data around (x=0,y=0) by 90 degrees in either direction. It also fills the corner of lo-x and lo-y boundary region by rotating the data by 180 degrees. * Rotate180 fills the lo-x boundary by rotating the data around (x=0,y=L_y/2) by 180 degrees. * Fill the polar boundaries of the spherical coordinates (theta, phi, r). The lo-x boundary is filled with f(-x,y) = f(x,mod(y+pi,2*pi)), and the hi-x boundary is filled with f(pi+x,y) = f(pi-x,mod(y+pi,2*pi)). * GPU support for NonLocalBC Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/CMakeLists.txt Src/Base/Make.package commit b5e08d92541f24e3ee3cb8d46f5b49c0147538bd Author: Weiqun Zhang Date: Thu Nov 19 10:28:53 2020 -0800 Remove deprecated MFUtil::convert function. One could use amrex::cast for FabArray casting. (#1523) Src/Base/AMReX_MultiFabUtilI.H commit 907baaede901d52601f120d606224bccd04d9611 Author: Andrew Myers Date: Thu Nov 19 13:24:16 2020 -0500 If TheZeroVector() is passed in to SortParticlesByBin, do nothing instead of crashing. (#1564) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_Particles.H commit 6038a3102fc55db861d69b242eecc35119675b96 Author: Weiqun Zhang Date: Wed Nov 18 09:49:08 2020 -0800 Scan size type (#1521) Make the size type a template parameter to support the scan of more than INT_MAX elements. Src/Base/AMReX_Scan.H commit ff92f1c129ea149eead13dc264eabf15a705e74d Author: mic84 Date: Wed Nov 18 09:14:51 2020 -0800 CMake: AMReX_MPI options depends on SYCL, not HIP (#1555) Tools/CMake/AMReXOptions.cmake commit 7cf969f1b656f9ab17975c90952360fb400dd712 Author: Axel Huebl Date: Wed Nov 18 09:14:25 2020 -0800 CMake: Fix Single-Precision Switch (#1550) * CMake: Fix Single-Precision Switch Since the latest refactoring, single-recision builds were not activated. This fixes it. * AMReX_Config.H: add AMREX_SINGLE_PRECISION_PARTICLES Src/Particle/CMakeLists.txt Tools/CMake/AMReX_Config.H.in Tools/CMake/AMReX_Defines.cmake commit 14c3003f8f4dda9ef9db0c04ba613e378677358a Author: Weiqun Zhang Date: Wed Nov 18 09:03:38 2020 -0800 Refactor FabArray's communication functions (#1553) Remove duplication and allow for more flexibility for extension. Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_iMultiFab.cpp Src/Boundary/AMReX_YAFluxRegister.cpp commit 079f599f74057ea93d8c60b82760b8747b21849e Author: Weiqun Zhang Date: Tue Nov 17 18:40:51 2020 -0800 DPCPP: Fortran (#1554) Switch from gfortran to ifx that comes with oneapi in GNU make. Tools/GNUMake/comps/dpcpp.mak commit e38742e322f6fbdf0e0fb090271d229fc2c834b9 Author: mic84 Date: Tue Nov 17 13:26:24 2020 -0800 CMake: improvements to handling of buildInfo (#1552) Tools/CMake/AMReXBuildInfo.cmake commit 477df17dd96607c385fc9410c4fe7746f121b58d Author: Weiqun Zhang Date: Tue Nov 17 11:19:19 2020 -0800 Fix BaseFab::maxabs (#1546) ## Summary For the host version, call host function LoopOnCpu instead of host device function Loop because the function we pass to it is a host function. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_BaseFab.H commit cd6f4d0bdbc389f672e3d6a3b8d286729caca3b6 Author: Andrew Myers Date: Tue Nov 17 13:33:00 2020 -0500 Fix bug in the 'intarray' method of StructOfArrays (#1548) Thanks to @rporcu for finding this. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Particle/AMReX_StructOfArrays.H commit b329cfe0bc4036aae7b69dc94b7bb34f8ab040c9 Author: Axel Huebl Date: Mon Nov 16 21:07:35 2020 -0800 CMake: generate_buildinfo target (#1545) This commit generates a static library `buildInfo::` for the passed user target. This target is then automatically linked. Also, the generated `.cpp` file is now scoped properly, avoiding collisions. This solves the following corner cases: - workflows with OBJECT libraries, that cannot be passed as targets since they have no `PRE_BUILD` custom command support - workflows with multiple targets that are not linked together (those call `generate_buildinfo` multiple times) - workflows with multiple targets that are linked together (those call `generate_buildinfo` once and re-use the target) Tools/CMake/AMReXBuildInfo.cmake commit 5076d33c583d88e0fb78214470715c2d09c083cd Author: Shreyas Ananthan Date: Mon Nov 16 10:00:02 2020 -0700 fcompare: fix logic error when comparing both rel/abs tolerances (#1542) PR #1537 introduced the option to allow tolerance checks on both absolute and relative tolerances. However, the check used `and` instead of `or` to allow tests to pass when either absolute or relative error was below user-specified tolerance. ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/Plotfile/fcompare.cpp commit 95b97799f2e611d19a4b685f725bfa3e5e12dc7f Author: Weiqun Zhang Date: Sun Nov 15 17:11:12 2020 -0800 nvcc --Werror cross-execution-space-call (#1540) * nvcc --Werror cross-execution-space-call * GNU make: Turn on the flag by default. Use `GPU_ERROR_CROSS_EXECUTION_SPACE_CALL=FALSE` to turn it off. * CMake: Add AMReX_ERROR_CROSS_EXECUTION_SPACE_CALL, which if off by default (just like AMReX_ERROR_CAPTURE_THIS). * update cmake documentation Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReX_SetupCUDA.cmake Tools/GNUMake/Make.defs Tools/GNUMake/comps/nvcc.mak commit aef49304278572eb3c4a5db75462c8c213d35221 Author: Max Katz Date: Sun Nov 15 11:22:29 2020 -0800 Don't throw C++ std error when not config checking (#1543) Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/nvcc.mak commit c80007df2f90e373f3d947fa7f621ab902378f3c Author: Shreyas Ananthan Date: Thu Nov 12 17:36:06 2020 -0700 fcompare: Add option for an absolute tolerance check (#1537) This PR adds a command line option to `fcompare` that takes in a user-specified tolerance for absolute error and adds logic to compare both absolute and relative errors against user-specified tolerances. Since `-a` is already used as a short-form for `--allow_diff_grids`, I have only used `--abs_tol` as the option without a short version when parsing command line options. Tools/Plotfile/fcompare.cpp commit cdb236ee435635f8007d2be97c0f6f1df13ecba0 Author: Weiqun Zhang Date: Thu Nov 12 08:50:35 2020 -0800 Remove call to cudaDeviceSetCacheConfig (#1536) ## Summary No longer set CUDA device cache configuration to prefer L1 cache. This does not appear to affect any kernels in a negative way. With more shared memory available, reduction functions are faster in some cases. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_GpuDevice.cpp commit f55a950e0977a4b062bba6ff9787ba84e9f7da92 Author: Weiqun Zhang Date: Thu Nov 12 08:11:02 2020 -0800 Fix multi-component nodal solver tutorial (#1535) ## Summary `MultiFab::setVal` should not be called inside `MFIter`. ## Additional background This was caught by #1530. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tutorials/LinearSolvers/MultiComponent/MCNodalLinOp.cpp commit a01697fda4a2f38a1e7e3c849bc52baca2c2999d Author: Axel Huebl Date: Tue Nov 10 18:18:03 2020 -0800 Ascent: Check Bounds Particle Reals (#1515) Access of particle variable names with a bound-check, so users definitely pass the right amount of names. Src/Extern/Conduit/AMReX_Conduit_Blueprint_ParticlesI.H commit d9f60b4cb934852cb87653fbc84df22c98cfe0e0 Author: Andrew Myers Date: Tue Nov 10 09:02:30 2020 -0800 Allow for particle containers constructed with the same AmrCore... (#1532) Src/AmrCore/AMReX_AmrParGDB.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_Particles.H commit 10c2a5e5949213d607305f771edab433285166d7 Author: asalmgren Date: Mon Nov 9 17:58:38 2020 -0800 Update the docs to include hypre.adjust_singular_matrix (#1534) ## Summary adds documentation of hypre.adjust_singular_matrix ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [X] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/LinearSolvers.rst commit 59b8bf1cc1ebf6dac76602a24a4ce696080b070c Author: Weiqun Zhang Date: Mon Nov 9 15:59:56 2020 -0800 DPC++: fix scan (#1533) * Use atomics instead of volatile to read status written to global memory by other blocks * Workaround `Random()` bug in the scan test * Limit the memory usage in the scan test Src/Base/AMReX_Scan.H Tutorials/GPU/ParallelScan/main.cpp commit 15ab2368225b81777e2810e0e02b1293364bb575 Author: Weiqun Zhang Date: Mon Nov 9 14:02:21 2020 -0800 Fix #1530 (#1531) `FillPatchIterator` is derived from `MFIter`. We need to reset `MFIter::depth` so that the ctor of `FillPatchIterator` can start `MFIter`. Src/Amr/AMReX_AmrLevel.cpp commit b94e1d2fca2bffd2b0e826eebfca7d63290efd40 Author: Weiqun Zhang Date: Mon Nov 9 10:30:06 2020 -0800 Assert no nested MFIters (#1530) ## Summary Add assertion to catch nested MFIters (e.g., MultiFab functions are called inside MFIter). ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit ad7c4eac3475fb7ddfc2abe8c74400d0274f8398 Author: Weiqun Zhang Date: Mon Nov 9 10:29:06 2020 -0800 Add anonymous namespace back to parameters in AMReX_Amr.cpp (#1529) Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp commit eb29a4f6b40e6458323db681795c1a12e1403c1c Author: Weiqun Zhang Date: Mon Nov 9 07:51:46 2020 -0800 Free MPI_Datatype defined by AMReX (#1527) ## Summary This fixes a one-time memory leak in MPI_Datatypes defined by AMReX as reported in #1525. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_ParallelDescriptor.cpp commit 0fd351b118b695d9c439c33a0f1ff9c6f3f00aeb Author: Weiqun Zhang Date: Sun Nov 8 10:27:36 2020 -0800 Update OLCF makefile (#1528) ## Summary * Remove summitdev and peak. * Environment variable OLCF_MODULEPATH_ROOT is used in addition to host name to detect OLCF machines. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit 6cc5104e57548e575717051b04361ffed0fee9c2 Author: Weiqun Zhang Date: Sun Nov 8 10:26:56 2020 -0800 Remove getGridSize that is no longer used. (#1524) Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunch.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 86abbe0221c1c3c53e69d740cdc24190ce1376c5 Author: Weiqun Zhang Date: Sat Nov 7 14:57:59 2020 -0800 Support CUDA arch < 6.0 (#1512) ## Summary * Implement atomicAdd for CUDA arch < 6.0. * Remove the CUDA arch check in CMake. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_BaseFab.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Tools/CMake/AMReX_SetupCUDA.cmake commit ec4424eb66366c5b798639a100b30c6c01938f79 Author: Axel Huebl Date: Sat Nov 7 14:56:34 2020 -0800 Change Spelling: Env AMREX_CUDA_ARCH (#1522) ## Summary Make the _environment variable_ that sets a default CUDA architecture all-caps, as this is way more common in Unix. Typical Values: `7.0` or `Volta` (i.e. for V100) ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReX_SetupCUDA.cmake commit 1b00f34ef1e170d4e2b398045e9dd45b4ba8ec34 Author: Weiqun Zhang Date: Fri Nov 6 18:18:27 2020 -0800 make clean & make cleanconfig (#1520) ## Summary `make clean` now does what `make realclean` does. `make realclean` is kept. `make cleanconfig` is introduced to do what `make clean` used to do. For most users, `make clean` is the one that should be used. Also add some tests in makefiles to avoid error messages in case when `make clean` is run on a makefile with the default not suitable for the system. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/README.md Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/sites/Make.unknown commit d9ac9a6c844a5055cdd2b332d101096da3ae5d72 Author: mic84 Date: Fri Nov 6 13:47:58 2020 -0800 CMake: fix options name in Test directory (#1514) Tests/CMakeLists.txt Tests/HDF5Benchmark/CMakeLists.txt commit 6dc26f704a386a03120f71395f00186049499264 Author: mic84 Date: Fri Nov 6 13:47:15 2020 -0800 CMake: add component check for Fortran (#1511) Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXConfig.cmake.in commit 9f2822bdcdd5604eea85a6206413684fb0137ea4 Author: mic84 Date: Fri Nov 6 13:44:41 2020 -0800 CMake: fix some quirks with options output (#1516) Tools/CMake/AMReXOptions.cmake commit 79d991ca8518943155e312a9a7566380e563bc93 Author: Shreyas Ananthan Date: Fri Nov 6 12:41:36 2020 -0700 hypre: Adjust singular matrices before hypre solves (#1519) Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp commit fe6058b946fb4334b74c42fedde919836f82066d Author: Shreyas Ananthan Date: Fri Nov 6 12:04:55 2020 -0700 hypre: Update nodal Laplacian interface to accept custom options (#1518) HypreNodeLap class creates the hypre IJ instance in its constructor. Therefore, the custom option namespace specified by user must be part of the constructor arguments. Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit 7b7c1c8e6addc152fb9c0059577799bfe8dc6d93 Author: Weiqun Zhang Date: Thu Nov 5 16:27:28 2020 -0800 DPC++: sync after htod_memcpy on null stream (#1517) ## Summary DPC++ does not have the concept of null stream. Adding a sync after htod_memcpy on the "null" stream will help eliminate potential bugs (e.g., htod_memcpy is called before MFIter that uses non-null stream). ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_Reduce.H commit 8035b01d86a7f8f4542bec6aa167dc864c8b3209 Author: Shreyas Ananthan Date: Thu Nov 5 15:06:36 2020 -0700 CMake: Fix ENABLE_HYPRE to AMReX_HYPRE (#1513) Fix the variable being checked to enable HYPRE. ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/CMake/AMReXThirdPartyLibraries.cmake commit 9aa75b1844e4ecc6c860a2d5d37b57bccedf73ed Author: Max Katz Date: Thu Nov 5 08:59:20 2020 -0800 Fix some build issues at LLNL (#1510) Tools/GNUMake/sites/Make.llnl commit 7ca532e5b299df1d7e9137e61f9271e4341c6189 Author: Andrew Myers Date: Thu Nov 5 08:54:57 2020 -0800 Add timer to neighbor list construction (#1509) Src/Particle/AMReX_NeighborList.H commit 86c6b3d00ae8d8e232b345b7060d9188fc838136 Author: mic84 Date: Wed Nov 4 19:39:39 2020 -0800 CMake: overhaul options (#1490) * Doc: move ENABLE_CUDA_FASTMATH to the GPU options section * CMake: change all options to include AMReX_ namespace * CMake: remove ENABLE_ACC option * CMake: make AMReX_SPACEDIM a multi-valued string * Doc: add option AMReX_ENABLE_TESTS * CMake: rename AMReX_Options.cmake to AMReXOptions.cmake * CMake: refactor GPU-related options * CMake: MPI must be turned off if DPCPP is enabled * CMake: set via precision via selection of value * CMake: forgot to update Particles/CMakeLists.txt * CMake: append prefix to TL_PROFILE * Update Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Co-authored-by: Axel Huebl * Update Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Co-authored-by: Axel Huebl * Update Docs/sphinx_documentation/source/BuildingAMReX.rst Co-authored-by: Axel Huebl * CMake: components in Config file shall not have namespace prefix * Revert "CMake: components in Config file shall not have namespace prefix" This reverts commit 924cf15f2a926467f3924a76f80595c0d1e53a7a. * Update Tools/CMake/AMReX_SetupCUDA.cmake Co-authored-by: Axel Huebl * CMake: update Doc * Update Tools/CMake/AMReX_SetupCUDA.cmake Co-authored-by: Axel Huebl Co-authored-by: Axel Huebl .github/workflows/linux.yml .github/workflows/macos.yml .github/workflows/windows.yml CHANGES CMakeLists.txt Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/Visualization.rst Src/Amr/CMakeLists.txt Src/AmrCore/CMakeLists.txt Src/Base/CMakeLists.txt Src/Boundary/CMakeLists.txt Src/CMakeLists.txt Src/EB/CMakeLists.txt Src/Extern/HYPRE/CMakeLists.txt Src/Extern/PETSc/CMakeLists.txt Src/Extern/ProfParser/CMakeLists.txt Src/Extern/amrdata/CMakeLists.txt Src/F_Interfaces/CMakeLists.txt Src/LinearSolvers/CMakeLists.txt Src/Particle/CMakeLists.txt Tests/CMakeLists.txt Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXInstallHelpers.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSYCL.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/AMReX_Config.cmake Tools/CMake/AMReX_Defines.cmake Tools/CMake/AMReX_Options.cmake Tools/CMake/AMReX_SetupCUDA.cmake Tools/Plotfile/CMakeLists.txt Tutorials/Amr/Advection_AmrCore/CMakeLists.txt Tutorials/Amr/Advection_AmrLevel/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_C/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_CF/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_F/CMakeLists.txt Tutorials/Basic/HeatEquation_EX2_CF/CMakeLists.txt Tutorials/Basic/HeatEquation_EX3_C/CMakeLists.txt Tutorials/Basic/HelloWorld_F/CMakeLists.txt Tutorials/Basic/main_C/CMakeLists.txt Tutorials/Basic/main_F/CMakeLists.txt Tutorials/Blueprint/AssignMultiLevelDensity/CMakeLists.txt Tutorials/Blueprint/CellSortedParticles/CMakeLists.txt Tutorials/Blueprint/HeatEquation_EX1_C/CMakeLists.txt Tutorials/CMakeLists.txt Tutorials/EB/CNS/CMakeLists.txt Tutorials/EB/GeometryGeneration/CMakeLists.txt Tutorials/ForkJoin/MLMG/CMakeLists.txt Tutorials/ForkJoin/Simple/MyTest.cpp Tutorials/FortranInterface/Advection_F/CMakeLists.txt Tutorials/FortranInterface/Advection_octree_F/CMakeLists.txt Tutorials/FortranInterface/Advection_octree_F2/CMakeLists.txt Tutorials/GPU/CNS/CMakeLists.txt Tutorials/GPU/EBCNS/CMakeLists.txt Tutorials/GPU/Launch/CMakeLists.txt Tutorials/LinearSolvers/ABecLaplacian_C/CMakeLists.txt Tutorials/LinearSolvers/ABecLaplacian_F/CMakeLists.txt Tutorials/LinearSolvers/MAC_Projection_EB/CMakeLists.txt Tutorials/LinearSolvers/Nodal_Projection_EB/CMakeLists.txt Tutorials/Particles/CellSortedParticles/CMakeLists.txt Tutorials/Particles/ElectromagneticPIC/CMakeLists.txt Tutorials/Particles/ElectrostaticPIC/CMakeLists.txt Tutorials/Particles/NeighborList/CMakeLists.txt Tutorials/SUNDIALS/EX1_C/CMakeLists.txt commit 22e6f93f2f66d24e364f35226b3f0d990182e316 Author: Weiqun Zhang Date: Wed Nov 4 17:00:42 2020 -0800 New Reduce::Sum function taking lambda function (#1508) ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Reduce.H commit ae9d0c696dad43b4477517728bf0f2ef978165db Author: Weiqun Zhang Date: Wed Nov 4 14:50:18 2020 -0800 GNU Make: AMREX_CUDA_ARCH (#1507) ## Summary Option to use AMREX_CUDA_ARCH in addition to CUDA_ARCH to set CUDA arch. If AMREX_CUDA is set, it will be used over CUDA_ARCH. ## Additional background The CMake system is also planning to support AMREX_CUDA_ARCH. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [x] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/GNUMake/Make.defs commit 1ab6d82e749b039b0417fa6c539c04d464ba95ac Author: Andrew Myers Date: Wed Nov 4 10:54:39 2020 -0800 Fix bug in new assertion (#1506) Src/Particle/AMReX_ParticleContainerI.H commit db41d3d75449b162ebe4113656da49536149e165 Author: Max Katz Date: Wed Nov 4 09:23:41 2020 -0800 Add option for device sync at beginning and end of TinyProfiler region (#1505) ## Summary Now when setting `tiny_profiler.device_synchronize_on_region_end = 1` in the inputs file, we will synchronize before calling nvtxRangePop() and nvtxRangePush(), which means that TINY_PROFILE regions will include the full kernel time, rather than just the kernel launch time. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit 81fdb9a6254fee22daf3c32843c619e08d527bbc Author: Max Katz Date: Wed Nov 4 07:50:41 2020 -0800 Guard MFIter reduction macros against USE_GPU_PRAGMA (#1504) Src/Base/AMReX_BLFort.H commit 461ca5b42dfeb3ac24ab0aa344d3fc00129cc8b4 Author: Weiqun Zhang Date: Tue Nov 3 15:39:57 2020 -0800 GNU make: adjust multiple target rules (#1500) ## Summary Remove `/` from the object file name so that it works for multiple targets in different directories. This should not affect its current usage. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.rules commit 3ed4b614a91e8b49d3e6f9b434493cf8f1c9464a Author: mic84 Date: Tue Nov 3 14:09:34 2020 -0800 CMake; bump up hypre minimi required version (#1503) Tools/CMake/AMReXThirdPartyLibraries.cmake commit f4c041f181dd1bb5743483e4b86f19cbb6e40fb2 Author: Andrew Myers Date: Tue Nov 3 13:54:38 2020 -0800 assert if aos.size() != soa.size() (#1502) Src/Particle/AMReX_ParticleContainerI.H commit b5a506997ce5db41bc46d6d64a238053fa2ea1ea Author: Weiqun Zhang Date: Tue Nov 3 13:53:23 2020 -0800 Fix warnings in AMReX_Machine.cpp (#1499) Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_Machine.H Src/Base/AMReX_Machine.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp commit e61266427dc11ffdfd2bad6577ac204e19ad5e88 Author: Andrew Myers Date: Tue Nov 3 13:52:40 2020 -0800 Use Long type for pindex to avoid potential overflow. (#1501) Src/Particle/AMReX_ParticleContainerI.H commit 7117a8f9006ad2d647c3e6412aee3ec3047d6def Author: Weiqun Zhang Date: Tue Nov 3 09:01:25 2020 -0800 Update GNU make for Cori GPU node (#1497) ## Summary Set CUDA_ARCH to 80 for A100 on Cori. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.nersc commit 24cacefc388e9811f52c1ebbceeb06cca90ba9b6 Author: Axel Huebl Date: Tue Nov 3 08:07:50 2020 -0800 Lin. Solvers (MLMG): Include Order (#1496) ## Summary Include own headers before stdlib headers to catch missing includes early. This avoid problems with less common compilers and environments. Carved out of #1485 ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit ee3076a8306269bf6d18ffee3f4d4944c660241d Author: Max Katz Date: Tue Nov 3 08:06:58 2020 -0800 GNUMake: Merge Ascent and Peak with Summit (#1493) ## Summary These are all systems with the same architecture so we can share the build setup. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/sites/Make.olcf commit 43f112fe4b9deda918f3d3961d2979e124d0645d Author: Weiqun Zhang Date: Mon Nov 2 13:00:42 2020 -0800 Update CHANGES for AMReX 20.11 (#1494) CHANGES commit db23e834e9cdc24884b689e0bdbd285ba6124562 Author: Shreyas Ananthan Date: Mon Nov 2 11:00:19 2020 -0700 HYPRE: Fix issues with hypre IJ interface (#1495) This commit fixes two issues with the hypre IJ interface - Fixes a memory leak in HypreIJIface as it was calling parse_inputs in constructor - Sets default number of iterations and tolerance for BoomerAMG when used as a preconditioner Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit f29a0c9d1b8e6356f8007a0bcb9cfc1000f2a0ac Author: Weiqun Zhang Date: Fri Oct 30 15:45:23 2020 -0700 Change default hypre interface to ij for non-EB (#1492) Src/LinearSolvers/MLMG/AMReX_MLMG.H commit 93ffe6391b8a18c749610cabe05c785b6325378d Author: Weiqun Zhang Date: Fri Oct 30 10:39:39 2020 -0700 Fix DEBUG build for DPC++ (#1491) This fixes a bug introduced in #1489. Src/Base/AMReX.cpp commit 82c7a83ff6dd43be71d5960ad422f4db88d929e7 Author: Weiqun Zhang Date: Fri Oct 30 08:45:24 2020 -0700 Changes to make hipcc --save-temps work. (#1489) ## Summary It does not like AMREX_GPU_DEVICE_EXTERNAL. Also changed is the device version of `Error`, `Abort` and `Warning`. If `NDEBUG` is defined, they do not do anything. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX.H Src/Base/AMReX.cpp Tools/GNUMake/Make.defs commit 69c496f8e8e65f7fe4e9cd5afd4e9e89fa4ad9ba Author: Weiqun Zhang Date: Thu Oct 29 12:32:24 2020 -0700 Fix GPU race condition for EB (#1487) A GPU race condition was introduced in #1451 trying to fix faces and cells next to covered cells that were converted from small cut cells. It is fixed by launching a separate kernel for each direction. Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.cpp commit 147149885e7ca02dcfdc88f1f4953bf26dc10dc8 Author: Axel Huebl Date: Thu Oct 29 11:30:41 2020 -0700 Ascent: Use Default Name for Mesh Ghost (#1488) ## Summary Use the default name for ghost cells in Ascent. The current name is non-default and causes issues at domain-boundaries during volume rendering (unless one overwrites the `ghost_field_name` keyword in the options). Thanks to @mclarsen and @cyrush for debugging and hinting this with me. ## Additional background Seen with WarpX. Pre-PR: ![replay_000400(9)](https://user-images.githubusercontent.com/1353258/97613195-9eed1c80-19d5-11eb-9934-cafeffc51458.png) Post-PR: ![replay_000400(10)](https://user-images.githubusercontent.com/1353258/97614230-eb852780-19d6-11eb-8dcd-4ca4d86d2e21.png) ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 32052a90a11da0dec54c353e0d6e29758e52b9a1 Author: Axel Huebl Date: Wed Oct 28 16:08:03 2020 -0700 CI: Check More Warnings (#1486) ## Summary Checks for more warnings that are likely to uncover nifty bugs: `-Wshadow -Woverloaded-virtual -Wunreachable-code` Also checks for superfluous `;` with `-Wextra-semi` (only a flag in Clang). ## Additional background Does not yet add `-Wall -Wextra -Wpedantic` because there are quite a few places where we need to add casts to avoid signed-unsigned-comparisons, etc. WarpX' CI covers currently: * GNU: `-Wall -Wextra -Wpedantic -Wshadow -Woverloaded-virtual -Wunreachable-code` * Clang: `-Wall -Wextra -Wpedantic -Wshadow -Woverloaded-virtual -Wextra-semi -Wunreachable-code` Even with super-builds, we current include AMReX with `-isystem` and build AMReX with default compiler flags; this enables us to suppress most AMReX build warnings besides the ones that drop-through from macros defined in headers, which atm. are all fixed for the code sections we use. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate .github/workflows/linux.yml .github/workflows/macos.yml Src/Particle/AMReX_NeighborParticles.H Tutorials/Basic/PrefixSum_MultiFab/main.cpp Tutorials/Particles/CellSortedParticles/cell_sorted_F.H Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/em_pic_F.H Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.cpp Tutorials/Particles/NeighborList/MDParticleContainer.cpp commit da1063cc5c16dcf519e74d630f898e7b23711c76 Author: Weiqun Zhang Date: Wed Oct 28 14:03:39 2020 -0700 HIP GNU Make: need to override COMP for regression tests (#1467) Tools/GNUMake/Make.defs commit ee020a50235341408297ff7018c536486eb30754 Author: Axel Huebl Date: Wed Oct 28 13:38:34 2020 -0700 CI: -Werror for host compilers (#1484) ## Summary Add more warning coverage in CI for CMake builds. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate .github/workflows/linux.yml .github/workflows/macos.yml commit 0186e2f103f800280d8edfc5dffcd9280f132f42 Author: Weiqun Zhang Date: Mon Oct 26 11:18:29 2020 -0700 Fix warning in FabSet for single precision (#1481) * Fix warning in FabSet for single precision * Constify constants Co-authored-by: Axel Huebl Src/Boundary/AMReX_FabSet.cpp commit 6f07fe4e84a4dd57a0e89c2d2cb84c9953fda85d Author: Weiqun Zhang Date: Fri Oct 23 16:23:17 2020 -0700 Workaround for Mac CI (#1482) .github/workflows/dependencies/dependencies_mac.sh commit 61ef93db305fccfe81404686d6878f1f805fee02 Author: Axel Huebl Date: Fri Oct 23 13:29:11 2020 -0700 CMake 3.18+: CUDA Arch Policy (OLD) (#1480) Keep the old policy for now to avoid setting the code generation flags twice. We should be able to transition this well, but I need to first find out how device LTO generation flags are handled here, if at all, and how to restore the default-detection of the local GPU architecture, e.g. on Summits head-nodes. CMakeLists.txt commit f4d73901d209caa80951b4842f0e46848104a98f Author: asalmgren Date: Fri Oct 23 10:13:49 2020 -0700 More fixes to documenting hypre (previous text got lost). (#1477) Docs/sphinx_documentation/source/LinearSolvers.rst commit f42ab5658751c51d6792d7dd7f1999c4c3310143 Author: Luca Fedeli Date: Thu Oct 22 17:23:11 2020 +0200 Add fill method to GpuArray (#1472) ## Summary I would like to propose to add a `fill` method to `GpuArray`. This PR implements just that. ## Additional background I am wondering if it would be better to replace `i < N` with `i < amrex::max(N,std::size_t{1})`, in order to deal with the case `N=0`. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [X] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Array.H commit 0cfd7d349279040c1d9b9283f970b333296594d9 Author: asalmgren Date: Thu Oct 22 08:21:53 2020 -0700 Update the docs to reflect the new hypre options (#1473) Docs/sphinx_documentation/source/LinearSolvers.rst commit 6071cf447613beba8753fb0a5409dbe4e299111a Author: Weiqun Zhang Date: Tue Oct 20 18:10:01 2020 -0700 Gpu::Atomic::AddNoRet (#1469) Add Gpu::Atomic::AddNoRet to use HIP's atomicAddNoRet for float, which is much faster than atomicAdd for float that is currently implemented with CAS. Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_TypeTraits.H Src/EB/AMReX_EB_utils.cpp Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_Particle_mod_K.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/ParticleMesh/main.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/em_pic_K.H commit a004707203dcbd3e28c5e637c623d569301c64cd Author: Andy Nonaka Date: Tue Oct 20 18:01:29 2020 -0700 Enforce singular solvable (#1471) ## Summary Option to make the code NOT enforce solvability for singular problems. This is accomplished by calling linop.setEnforceSingularSolvable(false) The default behavior is true, so no results will change in existing codes ## Additional background ## Checklist The proposed changes: - [x] add new capabilities to AMReX Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 8350a1641044aa222d922e85bd082be5734e2a6a Author: mic84 Date: Tue Oct 20 17:15:07 2020 -0700 CMake: improve HDF5 support (#1468) * CMake: look only for the C component of HDF5 * CMake: use only C-related variables from FindHDF5 * CMake: add Tests/HDF5Benchmark to CTest * CMake: fix typo * Update Tools/CMake/AMReXThirdPartyLibraries.cmake * CMake: use hdf5 imported target if available Co-authored-by: Axel Huebl Tests/CMakeLists.txt Tests/HDF5Benchmark/CMakeLists.txt Tools/CMake/AMReXThirdPartyLibraries.cmake commit c0710a2bc4725aa8e7cab57690b27c73285436ae Author: Shreyas Ananthan Date: Sun Oct 18 14:16:56 2020 -0600 Use rocrand from HIP APT repository (#1466) .github/workflows/dependencies/dependencies_hip.sh commit 4f0052023f6c68ca2898ed99552e641a2e4dcc8f Author: Shreyas Ananthan Date: Sun Oct 18 14:16:03 2020 -0600 Address warnings in CUDA builds (#1465) Src/Base/AMReX.cpp Src/Base/AMReX_FBI.H Tools/Plotfile/fsnapshot.cpp commit aef02b9a4745005856a45417632b964a760dadb0 Author: mic84 Date: Sun Oct 18 11:03:49 2020 -0700 CMake: FindPETSc must not overwrite CMAKE_Fortran_FLAGS (#1464) ## Summary FindPETSc used to set ```CMAKE_Fortran_FLAGS``` to the list of PETSc include directories. This caused problems with the compilation of Fortran files (see Issue #1463 ). This PR fixes this. ## Additional background ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/CMake/FindPETSc.cmake commit f64fb79d0f8a04a785e7c21f49dd81330ce55692 Author: Axel Huebl Date: Fri Oct 16 18:56:55 2020 -0700 CI: rocRAND v1.8.2 (#1462) Building the development branch is a bit unstable. .github/workflows/dependencies/dependencies_hip.sh commit 0cea78bf2f4f24568a1d71d20e8b2614fe19012c Author: mic84 Date: Fri Oct 16 17:33:42 2020 -0700 CMake: some cleanup (#1460) * CMake: provide function to retrieve AMReX version * CMake: remove unused helper functions CMakeLists.txt Tools/CMake/AMReX_Utils.cmake commit 931cf5993b8ad2d5fda1a853ebabb36dbe2ae5c2 Author: Weiqun Zhang Date: Fri Oct 16 17:30:50 2020 -0700 DPCPP: Scan (#1461) ## Summary Need to use global address space when doing atomic inc on virtual block id. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Scan.H commit 31ca23418f488fe0b309ef3fc3e99c522ced4921 Author: mic84 Date: Fri Oct 16 14:53:58 2020 -0700 [WIP] CMake: add support for dpc++ beta10 (#1459) * CMake: retrieve dpcpp version * CMake: modify dpc++ flags to account for beta10 version Tools/CMake/AMReXSYCL.cmake commit d036a6df1c6f8f7bb458c7e5253a91f4b6f96b83 Author: Max Katz Date: Fri Oct 16 14:21:28 2020 -0700 Fix potential out of bounds access in #1441 (#1458) ## Summary The user may have given us a vector that has too many values, so we just ignore the remainder. ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_ErrorList.H commit 18b293dad9199deb329efecddd56ccce709371f5 Author: Weiqun Zhang Date: Fri Oct 16 10:12:19 2020 -0700 DPCPP beta10 (#1456) ## Summary * Update for DPCPP beta10 * Fix some warnings * Remove some device versions of Random() for HIP and DPC++ ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Extension.H Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_Random.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_RandomEngine.H Src/Base/AMReX_Scan.H Src/Particle/AMReX_BinIterator.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H Tools/GNUMake/comps/dpcpp.mak commit 15643f4b0ab28dc6b3277c5c44c18ac1f80e1345 Author: mic84 Date: Thu Oct 15 16:21:13 2020 -0700 CMake: prevent in-source builds (#1453) * CMake: prevent in-source builds * CMake: make error message more detailed CMakeLists.txt commit c53b1bbe88bed6b731bae4e11fd9ed0883a6b814 Author: Andy Nonaka Date: Thu Oct 15 14:29:41 2020 -0700 change char * to char const* so SWFFT code compiles with USE_CUDA=TRUE (#1454) Src/Extern/SWFFT/distribution.c commit 48b0a20210b9825c1c403e1e0b8a741ba913dffa Author: Shreyas Ananthan Date: Wed Oct 14 15:00:12 2020 -0600 Make HypreABecLap3::getSolution public for CUDA builds (#1452) Src/Extern/HYPRE/AMReX_HypreABecLap3.H commit 5843b838c1a86ea7a7978d0b6edd9f5e0c03d24a Author: asalmgren Date: Wed Oct 14 09:10:35 2020 -0700 Fix when small_volfrac is used -- we need to adjust the data in cut c… (#1451) Fix when small_volfrac is used -- we need to adjust the data in cut cell neighbors as well as regular neighbors This addresses issue #1450 ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.cpp commit 79f3a94fd50b6e9bcac6f99d0b03ffb47e9a5a87 Author: Andrew Myers Date: Wed Oct 14 00:39:23 2020 -0700 Allow users to set a Geometry object for ParticleContainer independently of the AmrCore / AmrLevel object. (#1446) Src/AmrCore/AMReX_AmrParGDB.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_Particles.H commit ff17a9295b54a36fa78c4f3c2350c3b3b5dd1338 Author: Houjun Date: Tue Oct 13 12:14:50 2020 -0700 HDF5 plotfile write bug fix and performance improvement (#1448) * Add HDF5 optimizations * Fix an issue that causes incorrect HDF5 data write with Tutorials/Basic/HeatEquation_EX1_C example code Co-authored-by: Houjun Tang Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_ParticleHDF5.H Tests/HDF5Benchmark/GNUmakefile commit 6649d227425fdc8ecbed301c4383c8dc74800323 Author: mic84 Date: Tue Oct 13 11:52:09 2020 -0700 CMake: add HIP support (#1316) * CMake: first basic attempt at HIP support * Update Tools/CMake/AMReXParallelBackends.cmake Co-authored-by: Axel Huebl * CI: Add HIP-Clang * fix typo: GCC 9.3 on focal * Fix sources: set two lists * fix target properties: append two lists * CMake: some changes to debug issue with generated files * CI: only run HIP check for now * CMake: fix typo * CMake: let's try this * CMake: other small modifications * CMake: attemp #1 * CMake: attemp #2 * CI: build just the bare minimum for now * CMake: attempt #3 * CMake: missing defines for HIP builds * CMake: defines were still missing :-P * CMake: AMREX_HIP_PLATFORM is actually the HIP_COMPILER * CMake: let's see if this works * Install rocRAND * CMake: re-factor and update defines for HIP compilation * CMake: list against hiprand too * CMake: find and link to hiprand * CMake: add -DNDEBUG to any type of build when HIP is on * CMake: we must manually add rocrand too * CMake: let's see if it works with Fortran enabled * Revert "CMake: let's see if it works with Fortran enabled" This reverts commit 065071539fd166f3bee994c2621d86c773180e3d. * CMake: try this * CMake: this should work too * Revert "Revert "CMake: let's see if it works with Fortran enabled"" This reverts commit 6a92adcc4ddfb06b4130f10d29cc49b298a66e73. * CMake: no fortran for now * CI: turn on linear solvers in HIP check * CI: turn on particles in HIP checks * Revert "CI: turn on linear solvers in HIP check" This reverts commit c723560a7b401266d6259af340130b1065e6be59. The reason is that Linear Solvers tutorials are not HIP-aware yet. * CMake: hipify tutorials * CMake: temporary fix * Revert "CMake: temporary fix" This reverts commit bd6724ed73f7afc82a4dc60c60d4390c6a83808d. * CMake: fix HIP compilation for Tutorials * CMake: some cleanup * CMake: add HIP options to pass in architecture and extra flags * CMake: oops * CMake: try to enable linear solvers * CMake: update ROCm version number in CI check name * CMake: enable Fortran * CMake: disable tutorials for the time being * CMake: commit custom FindHIP.cmake to debug the issue * CMake: trying this * CMake: now it should use the local FindHIP.cmake * CMake: now custom FindHIP.cmake should be able to find helper files * CMake: let's see if we really need this * CMake: let's try this * CMake: temporary fix * Revert "CMake: disable tutorials for the time being" This reverts commit 885b9113b72f54bca9eaf74f4e5b6b90b17032ff. * CMake: Tutorials/Particles/CellSortedParticles do not work with HIP * CMake: remove local FindHIP.cmake. * CMake: HIP arch flags are now PUBLIC and inheritable * CMake: no use for setup_target_for_hip_compilation anymore * CMake: remove unnecessary options for HIP * CMake: check that HIP_COMPILER is the same as CMAKE_CXX_COMPILER * CMake: better this * CI: uncomment all checks * CMake: fix visibility of arch flags for HIP * Doc: add subsection on HIP+CMake * Update Docs/sphinx_documentation/source/GPU.rst Co-authored-by: Shreyas Ananthan * CMake: fix incorrect configuration option passed to cmake * CI: target 'tutorials' no longer exists Co-authored-by: Axel Huebl Co-authored-by: Shreyas Ananthan .github/workflows/dependencies/dependencies_hip.sh .github/workflows/linux.yml Docs/sphinx_documentation/source/GPU.rst Src/CMakeLists.txt Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/CMake/AMReX_Defines.cmake Tools/CMake/AMReX_Options.cmake Tutorials/CMakeLists.txt Tutorials/Particles/CellSortedParticles/CMakeLists.txt commit 6dc6ce8f55a5ef19f349c267b7bb9fc4bb616b07 Author: Candace Gilet Date: Tue Oct 13 09:48:50 2020 -0400 volume multifab must be defined before calling GetVolume(volume) (#1449) Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 5e6d555d07d3862b5b88b6c38559a59918541194 Author: mic84 Date: Mon Oct 12 17:38:22 2020 -0700 NodalProjector: use volume-weighted average down (#1444) Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 95ddfdb2c1434b1705d84174047f06cd3004174d Author: Weiqun Zhang Date: Wed Oct 7 19:02:08 2020 -0700 Fix EB interpolation from cell centers to faces (#1443) ##Summary There were race conditions that made it fail on AMD GPUs. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H commit 2c7b5004fb98d92bf185ecc73b6429dfd7ed8b6f Author: Weiqun Zhang Date: Wed Oct 7 15:16:51 2020 -0700 Fix bug in hypre overset solver (#1442) ## Summary ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/PETSc/AMReX_PETSc.cpp commit 3972c6d582f024c4a4d108ff02e66d117e6db1e6 Author: Max Katz Date: Wed Oct 7 15:33:35 2020 -0400 Allow tagging value to vary by level (#1441) ## Summary The new error tagging scheme from #1166 is modified to allow the threshold value to vary by level. Using the example there, if we did: ``` amr.refinement_indicators = flame_tracer lo_temp amr.refine.flame_tracer.max_level = 3 amr.refine.flame_tracer.value_greater = 1.e-6 1.e-5 amr.refine.flame_tracer.field_name = Y(H) amr.refine.lo_temp.max_level = 2 amr.refine.lo_temp.value_less = 1000. amr.refine.lo_temp.field_name = temp ``` Then level 0 would be tagged for refinement for Y(H) >= 1.e-6, while level 1 and all higher levels would be tagged for refinement for Y(H) >= 1.e-5. (If a value for every level is not provided, we assume the last value holds for all remaining levels, similar to how we treat quantities like amr.n_error_buf.) ## Additional background This will help for cases where a field varies substantially on the domain and you want to have better control over refinement so that you don't have too many zones. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp commit 92399587f20875a98655ccbd06a083965cfc9c5c Author: Weiqun Zhang Date: Tue Oct 6 13:30:52 2020 -0700 Hypre with overset (#1439) * Hypre with overset Add overset mask support for hypre. * Keep bottom_verbose restricted to amrex side. Hypre side's verbosity will be controlled by parmparse parameters. Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_Hypre.cpp Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Tests/LinearSolvers/CellOverset/MyTest.H Tests/LinearSolvers/CellOverset/MyTest.cpp commit 6a20c352cf54ed9a1393676b1bf19c57941db2af Author: Axel Huebl Date: Mon Oct 5 16:17:43 2020 -0700 GNUmake: -Titan (OLCF) (#1440) R.I.P. Titan. Btw, only found because a colleague of mine could not compile WarpX. Their hostname is called -titan. Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit 9992d844099c3df81d9499519ffeb7f93a201dc6 Author: Shreyas Ananthan Date: Sun Oct 4 18:19:24 2020 -0600 Hypre IJ interface: Enable access to additional solvers and preconditioners available in Hypre (#1437) This PR extends the AMReX _hypre_ IJ interface to allow access to additional solvers and preconditioners available in _hypre_. - Refactored `HypreABecLap3` and `HypreNodeLap` (IJ matrix/vector ParCSR interfaces) and created a new IJ interface class that provides a unified way to access Hypre ParCSR solvers. - In addition to BoomerAMG, adds support for 7 other [ParCSR solvers](https://hypre.readthedocs.io/en/latest/ch-solvers.html) (GMRES (4 variants), PCG, BiCGSTAB, and Hybrid). Also adds support for BoomerAMG and Euclid as preconditioners for solvers that can use a preconditioner. - Adds support to parse user options to configure hypre library via AMReX ParmParse interface and input file, using a custom namespace (default is `hypre`) - Adds `MLMG::setHypreOptionsNamespace` that allow applications to customize the namespace for different types of linear solvers (e.g., `mac_proj.hypre.