commit 4fc7ef352fe10ce57f999e713a1a5e4cb8e22d13 Author: siddanib <45137944+siddanib@users.noreply.github.com> Date: Wed Feb 7 18:30:29 2024 -0800 Minor new functions in AMReX_MPMD to provide flexibility for python binding (#3748) ## Summary New functions (*void* **Initialize_without_split**, int **AppNum**) are created in Src/Base/AMReX_MPMD.* to provide additional flexibility of calling MPI_Comm_split equivalent in python instead of it being called inside of AMReX, which is the default behavior of *MPI_Comm* **Initialize**. The proposed changes: - [ ] add minor new functions to AMReX_MPMD --------- Co-authored-by: Bhargav Siddani Src/Base/AMReX_MPMD.H Src/Base/AMReX_MPMD.cpp commit 928a485af2949b2d41b20adc4585690908e41970 Author: Weiqun Zhang Date: Tue Feb 6 17:20:00 2024 -0800 Box::numPts() returns 0 for empty boxes (#3747) This is now necessary because the returned long int is converted to std::uint64_t in ParallelFor. This is a follow-up on #3742. Src/Base/AMReX_Box.H commit a8bc0bfbcf10ce005a8f1c1067c36626d843f1ce Author: Weiqun Zhang Date: Mon Feb 5 21:22:13 2024 -0800 Fix typos (#3746) Docs/sphinx_documentation/source/Debugging.rst Docs/sphinx_documentation/source/GPU.rst Src/AmrCore/AMReX_AmrCore.cpp Src/Base/AMReX_BaseFab.H Src/EB/AMReX_EB2_Level.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp commit fb2f9254d7cffc5bfb3531a96a861e2cdeafe337 Author: Weiqun Zhang Date: Mon Feb 5 17:15:24 2024 -0800 New Linear Solver: Curl of Curl (#3682) Add a new linear solver for curl (alpha curl E) + beta E = rhs in 2D & 3D, where E is an array of 3 MultiFabs on edges, and alpha and beta are scalars. It supports periodic, homogeneous Dirichlet, and symmetry boundaries. At the symmetry boundary, the normal component changes the sign, whereas the transverse components do not. --------- Co-authored-by: Axel Huebl Src/Base/AMReX_FabDataType.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_LO_BCTYPES.H Src/Boundary/AMReX_LO_BCTYPES.cpp Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLCurlCurl.H Src/LinearSolvers/MLMG/AMReX_MLCurlCurl.cpp Src/LinearSolvers/MLMG/AMReX_MLCurlCurl_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/Make.package Tests/LinearSolvers/ABecLaplacian_C/MyTestPlotfile.cpp Tests/LinearSolvers/CurlCurl/CMakeLists.txt Tests/LinearSolvers/CurlCurl/GNUmakefile Tests/LinearSolvers/CurlCurl/Make.package Tests/LinearSolvers/CurlCurl/MyTest.H Tests/LinearSolvers/CurlCurl/MyTest.cpp Tests/LinearSolvers/CurlCurl/initProb.cpp Tests/LinearSolvers/CurlCurl/initProb_K.H Tests/LinearSolvers/CurlCurl/inputs Tests/LinearSolvers/CurlCurl/main.cpp commit e46bd91f99f401440af74ca58cb2b21a184dd825 Author: Weiqun Zhang Date: Mon Feb 5 14:57:23 2024 -0800 Use long integer in GPU kernels (#3742) In the current implementation of ParallelFor, we use `int` for the linear cell index in the flatten one-dimension view. This limits the size of the box to less than 2^30 cells (half of INT_MAX). The factor of half comes from the grid-stride loop. This limitation has not been a serious issue until recently because GPUs did not have that much memory. However, the total memory on the latest GPUs has increased quite a lot. Issues have been reported by users. In this PR, we have started using `std::uint64_t` as the linear cell index. An issue of using `std::uint64_t` is that 64-bit integer division is very expensive. Fortunately, we are able to "steal" the fast division code from https://github.com/NVIDIA/cutlass. Streaming tests have shown very good performance. On A100, the FArrayBox version of the triad test achieves 1.76 TB/s, which is the same as the rate of the much simpler 1D vector version. In fact, it's slightly faster than the rate of 1.72 TB/s from the current version in the development branch. We have not made all kernel launches in AMReX safe for large sizes. Nevertheless, this PR is the first step, and it will be followed up by more PRs. Src/Base/AMReX_BaseFabUtility.H Src/Base/AMReX_Box.H Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_INT.H Src/Base/AMReX_Math.H commit 296ed40e16ae1877640f5b78e9162dbd4ba1c279 Author: Axel Huebl Date: Thu Feb 1 09:09:09 2024 -0800 Add `ParticleIDWrapper::make_invalid()` (#3735) Add faster functions to ParticleIDWrapper: make_invalid, make_valid and is_valid. Src/Particle/AMReX_Particle.H commit 52393d3fafc835fa379f0420aa2de4ccdddf155a Author: Weiqun Zhang Date: Thu Feb 1 07:27:10 2024 -0800 Update CHANGES for 24.02 (#3741) CHANGES commit 689144d157a0106faf3d0ae89f8d90b0250cf975 Author: Axel Huebl Date: Wed Jan 31 09:05:36 2024 -0800 Update AoS Restrict (#3738) ## Summary Move into type to mitigate MSVC compile errors: ``` Src\Particle\AMReX_ParticleTile.H(48,5): error C2219: syntax error: type qualifier must be after '*' ``` ## Additional background Follow-up to #3736. Seen in https://github.com/ECP-WarpX/WarpX/pull/4654 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleTile.H commit d861bdff6ab7b31ada277f87c3b4a57b5883a0b0 Author: Weiqun Zhang Date: Tue Jan 30 22:50:30 2024 -0800 Disable m_aos for SoA Particle (#3736) Change the type of m_aos for SoA particle to void*. Because void* cannot be dereferenced, this can avoid bugs that use m_aos on SoA particles. Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleTile.H commit 09fc44a69d2a7e64605921c6b167cd7b9d7a0dad Author: Andrew Myers Date: Tue Jan 30 17:52:09 2024 -0800 Add helper function for setting id and cpu simultaneously. (#3733) Also sneaking in a small ignore_unused fix. Close #3731. Co-authored-by: Axel Huebl Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleIO.H commit 33dea1f728817941740ca569cff0bfb268d3849f Author: Weiqun Zhang Date: Tue Jan 30 17:26:08 2024 -0800 Disable SYCL on Nvidia and AMD CIs (#3726) The Codeplay APT has stopped working for a few weeks now. It fails with error messages like The following packages have unmet dependencies: oneapi-nvidia-12.0 : Depends: intel-basekit (>= 2024.0.2) but it is not going to be installed E: Unable to correct problems, you have held broken packages. Error: Process completed with exit code 1. To avoid confusion, let's disable them for now. .github/workflows/intel.yml commit dbc0c6f155a94c976a1f812dc14f193af888ba69 Author: Axel Huebl Date: Tue Jan 30 17:25:17 2024 -0800 ParticleCopyPlan for SoA Particles (#3732) ## Summary Update `ParticleCopyPlan::build` for pure SoA particle layout. ## Additional background - [x] testing on GPU in https://github.com/ECP-WarpX/WarpX/pull/4653 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Andrew Myers Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleTransformation.H commit 47c4f95ffe75e7c8aad6793068abbfebded1c0f2 Author: Andrew Myers Date: Tue Jan 30 16:42:25 2024 -0800 Do not use std::forward twice on the same object in AmrParticleLocator (#3734) Bug found by @WeiqunZhang The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleLocator.H commit 4aeeabef098633c699282887c7bbde953fa2bc2c Author: Axel Huebl Date: Tue Jan 30 11:02:08 2024 -0800 Particle Copy Plan: Default Vals (#3729) ## Summary Valgrind finds those to be uninitialized in GPU runs with 1 MPI rank. ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleCommunication.H commit ab99ea69089f4fffbfd727ed965d5ceb3d905baa Author: Weiqun Zhang Date: Sat Jan 27 11:17:04 2024 -0800 GitHub Actions: 4 vCPUs are available now on Ubuntu and Windows (#3703) .github/workflows/apps.yml .github/workflows/ascent.yml .github/workflows/bittree.yml .github/workflows/clang.yml .github/workflows/codeql.yml .github/workflows/cuda.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/hypre.yml .github/workflows/intel.yml .github/workflows/petsc.yml .github/workflows/sensei.yml .github/workflows/smoke.yml .github/workflows/sundials.yml .github/workflows/windows.yml commit 41397e2e704955c9408ce58b6b3d7b94c8b68853 Author: Weiqun Zhang Date: Fri Jan 26 15:27:11 2024 -0800 Fix circular header file dependency (#3725) AMReX_GpuContainers.H included AMReX_Scan.H, which in turn includes AMReX_Gpu.H, which in turn includes AMReX_GpuContainers.H. In this PR, AMReX_Scan.H is removed from AMReX_GpuContainers.H, because it does not need it. Thanks to @zingale and clang-tidy. Src/Base/AMReX_BaseFab.H Src/Base/AMReX_GpuContainers.H Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_ParticleCommunication.H commit d5cc57909f518a8e204e5fbf3d439418e69beb08 Author: Andrew Myers Date: Thu Jan 25 17:58:13 2024 -0800 Follow-on to #3701 (#3702) The parenthesis are not needed and some compilers warn about them. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_Extension.H commit 501f92dae309105cf4355f4a4f99172dfa5fdcb7 Author: Andrew Myers Date: Thu Jan 25 13:26:31 2024 -0800 Provide portable Gpu::Atomic::Multiply and Gpu::Atomic::Divide implemented with CAS. (#3724) The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Weiqun Zhang Src/Base/AMReX_Functional.H Src/Base/AMReX_GpuAtomic.H commit 30738c689207dcf9b62a108b6efddde93142b5d9 Author: Miren Radia <32646026+mirenradia@users.noreply.github.com> Date: Wed Jan 24 16:54:40 2024 +0000 Plotfile Tools: Add missing option to fcompare usage print (#3722) This small PR just adds the `--abort_if_not_all_found` option to the usage print out of `fcompare`. Tools/Plotfile/fcompare.cpp commit c5c62c00fd2de9159f9f79e6c9a80400dd6c9ede Author: Remi Lehe Date: Wed Jan 24 08:51:08 2024 -0800 Update documentation for STL files (#3723) The documentation mentions the parameter ``eb2.stl_file``. However, this parameter is ignored by AMReX **unless** one sets ``eb2.geom_type = stl`` (which is not mentioned in the current version of the documentation). Docs/sphinx_documentation/source/EB.rst commit d121723c7491acedf8e2e010cfb1bf3773224b95 Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Wed Jan 24 00:25:29 2024 +0100 Add TypeMultiplier, MakeZeroTuple and IdentityTuple (#3718) ## Summary This PR adds TypeMultiplier and MakeZeroTuple from https://github.com/Hi-PACE/hipace/pull/1052 to AMReX. TypeMultiplier can be used to shorten ReduceOps and ReduceData definitions where the same type is used many times. MakeZeroTuple can be used to initialize a GpuTuple to zero (the default constructor would leave the values uninitialized). IdentityTuple can be used to initialize a GpuTuple to the identity elements of each operation in a ReduceOps. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_Reduce.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_TypeList.H commit 0ccffb42ab1d7679496c177106723877d4638bf1 Author: Revathi Jambunathan Date: Tue Jan 23 15:01:22 2024 -0800 Minor cleanup of commented code and alignment (#3719) ## Summary ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_TracerParticle_mod_K.H commit 3e2a3c2940f4db6c740fdff863ed9b5cee36790b Author: Andrew Myers Date: Tue Jan 23 12:55:41 2024 -0800 Interpolation routines for tracers with mapped_z. (#3714) This code is from @asalmgren and @RevathiJambunathan . The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Revathi Jambunathan <41089244+RevathiJambunathan@users.noreply.github.com> Co-authored-by: Ann Almgren Src/Particle/AMReX_TracerParticle_mod_K.H commit 73b215557c0e842c3e829b683939bbb7a7e12373 Author: Andrew Myers Date: Tue Jan 23 10:35:12 2024 -0800 relax constraint that real_comp_names.size() == pc.NumRealComps() + NStructReal for pure SoA plotfiles (#3717) The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/Particles/CheckpointRestartSOA/main.cpp commit 2fc8a34ced0c0acae04298c69ec23803ec05e9cd Author: Andrew Myers Date: Tue Jan 23 07:29:23 2024 -0800 Fix warnings in DenseBins::build with serial bin policy (#3716) Src/Particle/AMReX_DenseBins.H commit 0c59bad946fdf5ee86f0a48204324c5efaad7284 Author: Weiqun Zhang Date: Mon Jan 22 15:22:54 2024 -0800 lockAdd: case of 2D plane in 3D (#3700) ## Summary In HiPACE++, atomicAdd is used on 2d x & y planes even though AMREX_SPACEDIM is 3. In that case, we would have all threads competing for a single lock in the previous implementation of lockAdd. This PR fixes this use case by having locks associated with the y-direction when the number of cells in the z-direction is 1. ## Additional background https://github.com/Hi-PACE/hipace/pull/1059 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_BaseFab.H commit 022f97ea9ebbc28fca25f38f3eda26ce68b6230b Author: Weiqun Zhang Date: Fri Jan 19 21:01:26 2024 -0800 Update Build and Deploy action (#3712) Make some changes because a new version of JamesIves/github-pages-deploy-action is used now. .github/workflows/docs.yml commit 6f1c74f7a9295f7b217a2ef8a631cbe136bc689e Author: Weiqun Zhang Date: Fri Jan 19 17:50:23 2024 -0800 Revert "add a dependabot.yml to checkout github actions versions (#3704)" (#3710) This reverts commit 53a581cc802734846f1c102e593bddb828922cf0. It creates too much noise for forks, and it cannot be disabled for forks. Another issue. It appears that it just called docs.yml and push a new branch "main" to AMReX-Codes/amrex, not AMReX-Codes/AMReX-Codes.github.io. .github/dependabot.yml commit 4be40506dbb26d18f1ac46b5c0add4d94aa1f51e Author: Andrew Myers Date: Fri Jan 19 16:29:00 2024 -0800 Clean up interpolation routines in AMReX_TracerParticle_mod_K.H and AMReX_Particle_mod_K.H (#3679) The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Revathi Jambunathan <41089244+RevathiJambunathan@users.noreply.github.com> Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_Particle_mod_K.H Src/Particle/AMReX_TracerParticle_mod_K.H Tests/Particles/AssignDensity/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp commit 9996b68e9ce9e1d995cbc894b591eb5217227cc0 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Jan 19 23:09:07 2024 +0000 Bump actions/upload-artifact from 3 to 4 (#3709) .github/workflows/apps.yml .github/workflows/bittree.yml .github/workflows/clang.yml .github/workflows/codeql.yml .github/workflows/cuda.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/hypre.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/petsc.yml .github/workflows/post-pr.yml .github/workflows/smoke.yml .github/workflows/sundials.yml commit 1647cff047c59cffb1acea56c9fbe5be1d9d58e9 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Jan 19 14:39:58 2024 -0800 Bump github/codeql-action from 2 to 3 (#3708) .github/workflows/codeql.yml commit abc1cfb268a8b3dd011e6d431963054a7eb55037 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Jan 19 14:00:57 2024 -0800 Bump actions/checkout from 3 to 4 (#3707) .github/workflows/apps.yml .github/workflows/ascent.yml .github/workflows/clang.yml .github/workflows/cleanup-cache-postpr.yml .github/workflows/cleanup-cache.yml .github/workflows/codeql.yml .github/workflows/codespell.yml .github/workflows/cuda.yml .github/workflows/docs.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/hypre.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/petsc.yml .github/workflows/sensei.yml .github/workflows/smoke.yml .github/workflows/style.yml .github/workflows/sundials.yml .github/workflows/windows.yml commit 03bf4cf0485e280074a94f3574ac3f916f19e667 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Jan 19 13:14:52 2024 -0800 Bump actions/cache from 3 to 4 (#3706) .github/workflows/apps.yml .github/workflows/bittree.yml .github/workflows/clang.yml .github/workflows/codeql.yml .github/workflows/cuda.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/hypre.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/petsc.yml .github/workflows/smoke.yml .github/workflows/sundials.yml commit cf287fe1c176a0c04bef3f9e3fe59defd1b093ec Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Jan 19 13:12:58 2024 -0800 Bump JamesIves/github-pages-deploy-action from 3.7.1 to 4.5.0 (#3705) Bumps [JamesIves/github-pages-deploy-action](https://github.com/jamesives/github-pages-deploy-action) from 3.7.1 to 4.5.0.
Release notes

Sourced from JamesIves/github-pages-deploy-action's releases.

v4.5.0

What's Changed

New Contributors

Full Changelog: https://github.com/JamesIves/github-pages-deploy-action/compare/v4.4.3...v4.5.0

v4.4.3

What's Changed

... (truncated)

Commits
  • 65b5dfd Deploy Production Code for Commit 32bb2a8d7bda92f28b1fd223fc5793ab27f725eb 🚀
  • 32bb2a8 Merge branch 'dev' into releases/v4
  • 7879c16 feat: update to Node 20
  • 1c34508 build(deps-dev): bump @​types/node from 20.9.0 to 20.10.0 (#1479)
  • b957d05 Bump actions/checkout from 3 to 4 (#1429)
  • 3a4632d build(deps-dev): bump @​types/node from 20.4.9 to 20.9.0 (#1474)
  • 9de0269 build(deps): bump actions/setup-node from 3 to 4 (#1467)
  • 810af53 Update README.md
  • 24c1e2a doc: .gitignore matches not being deployed (#1459)
  • 0769273 fix: remove base.Dockerfile as it's not needed
  • Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=JamesIves/github-pages-deploy-action&package-manager=github_actions&previous-version=3.7.1&new-version=4.5.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) ---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> .github/workflows/docs.yml commit 53a581cc802734846f1c102e593bddb828922cf0 Author: Michael Zingale Date: Fri Jan 19 12:43:59 2024 -0500 add a dependabot.yml to checkout github actions versions (#3704) .github/dependabot.yml commit 9b799efb04e06912be7c051861cd60a0d7b82579 Author: Andrew Myers Date: Tue Jan 16 16:44:51 2024 -0800 Add macro for loop unrolling across compilers (#3701) Src/Base/AMReX_Extension.H commit 03f59365ab29fc619d836a5f0fd3e9249eeacd89 Author: Luca Fedeli Date: Tue Jan 16 06:00:31 2024 +0100 Add a linspace-like function in AMReX_Algorithm.H (#3698) ## Summary I would like to propose to add a linspace-like function in AMReX_Algorithm.H , since filling a container with equally spaced numbers is a task that occurs rather frequently. This PR proposes a possible implementation. The container is modified in place only if it has at least 2 elements. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [X] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_Algorithm.H commit b96b731d4a6fa496f80e54c2c0476a50b9dd4e86 Author: Ben Wibking Date: Mon Jan 15 13:11:26 2024 -0500 use amrex::Gpu::memcpy for packParticleIDs (#3699) std::memcpy does not exist in HIP, so we use amrex::Gpu::memcpy instead for packParticleIDs when writing binary particle data. Src/Particle/AMReX_WriteBinaryParticleData.H commit 255d30f387cf2c1a7eff5a31f703c94de803e8d8 Author: Weiqun Zhang Date: Fri Jan 12 14:48:35 2024 -0800 BaseFab::lockAdd: Faster version of BaseFab::atomicAdd for OpenMP (#3696) For WarpX's Gordon Bell runs on Fugaku, Stephan Jaure of ATOS optimized atomicAdd using pthread spin locks. This commit implements Stephan's approach using OpenMP. In AMReX::Initialize, we create a number of OMP locks. When BaseFab::lockAdd is called, we loop over planes in z-direction and try to acquire a lock with omp_test_lock. If it's successful, we can access the data in that z-plane without worrying about race conditions. This allows us to use simd instructions instead of omp atomic adds. If it's not successful, we will try a different z-plane. The process stops till all planes are processed. Src/Base/AMReX.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_OpenMP.H Src/Base/AMReX_OpenMP.cpp commit d44001045c8074b620fcf176bda5ba35f08267f9 Author: Weiqun Zhang Date: Fri Jan 12 14:48:13 2024 -0800 GMRES (#3648) Add a GMRES solver class template This is still work in progress. So the interface may not be stable yet. Src/Base/AMReX_TableData.H Src/LinearSolvers/AMReX_GMRES.H Src/LinearSolvers/AMReX_GMRES_MLMG.H Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/Make.package Src/LinearSolvers/Make.package Src/LinearSolvers/OpenBC/Make.package Tests/LinearSolvers/ABecLaplacian_C/GNUmakefile Tests/LinearSolvers/ABecLaplacian_C/MyTest.H Tests/LinearSolvers/ABecLaplacian_C/MyTest.cpp Tests/LinearSolvers/ABecLaplacian_C/MyTestPlotfile.cpp Tests/LinearSolvers/ABecLaplacian_C/inputs.gmres commit 8692f601b428c90725423eb2c5693e12ccc4f457 Author: Andrew Myers Date: Thu Jan 11 21:52:24 2024 -0800 Add special named flag for invalid particles (#3688) This lets you do `p.m_idcpu = LongParticleIDs::InvalidParticleID;` instead of `amrex::ParticleIDWrapper{p.m_idcpu} = -1;` The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Axel Huebl Src/Particle/AMReX_Particle.H commit e780f33fed28fbd9aee951e09c9aecf42e596a97 Author: David Grote Date: Thu Jan 11 12:16:08 2024 -0800 Fix BL_PROFILE_TINY_FLUSH (#3695) ## Summary This fixes the BL_PROFILE_TINY_FLUSH macro. The `amrex` prefix was left off of the call to `TinyProfiler::MemoryFinalize`. This produced a compilation error that `TinyProfiler` was not defined. Note that this is in code that does not have `using namespace amrex`. ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_BLProfiler.H commit 656bb64515bda87807c97e82321cb998aa7bf16f Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Tue Jan 9 21:24:04 2024 +0100 Align GpuComplex to its size (#3691) ## Summary As discussed in #3677, this PR makes the alignment of `amrex::GpuComplex` stricter to allow for coalesced memory accesses of arrays of GpuComplex by nvidia GPUs such as A100. Note that this may break `reinterpret_cast` from an array allocated as `std::complex` to `amrex::GpuComplex`, but not the other way around. ## Additional background Typical allocators (malloc, amrex CArena) give memory aligned to 16 bytes and CUDA allocators aligned to 256 bytes, which is sufficient for `amrex::GpuComplex`. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_GpuComplex.H commit e0b77e1fce491021696d22f1496138e8d9942d65 Author: Weiqun Zhang Date: Mon Jan 8 09:18:49 2024 -0800 Fix Advection_AmrCore test (#3690) The particle container should not be a static member, because it's a member of a class derived from AmrCore, not AmrLevel. The issue of its being a static member is that it is not deleted properly (unless we explicitly do it). Note that in the AmrLevel case we must make it a static member because all AmrLevel objects need to share the same particle container. Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp commit f1ec8df75c562d2a4822cea84d284cf8e72c2e14 Author: Weiqun Zhang Date: Thu Jan 4 08:58:46 2024 -0800 Install Codeplay packages via APT (#3687) Codeplay now provides packages via APT. Token is no longer needed. .github/workflows/dependencies/dependencies_codeplay.sh .github/workflows/dependencies/dependencies_hip.sh .github/workflows/intel.yml commit a068330e6c66b5d9a7c6ca0e1c874f318e73f4cc Author: Weiqun Zhang Date: Tue Jan 2 07:42:38 2024 -0800 Update CHANGES for 24.01 (#3686) CHANGES commit 75571e2dcbf2417529c5ed8e24113580e8e1f3f1 Author: Weiqun Zhang Date: Wed Dec 20 19:04:57 2023 -0800 MLMG: Use free functions instead of MF member functions (#3681) Note that the use of unqualified functions (e.g., setVal instead of amrex::setVal) is intentional. With ADL, these calls in MLMG could work with user defined data. Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H commit 3407e877a7b219546c0ef12bef287845f5cb9f05 Author: Weiqun Zhang Date: Wed Dec 20 12:11:53 2023 -0800 Add a few free functions for MLMG (#3680) These are useful when we use Array as the data type for MLMG. Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_TypeTraits.H commit 85462ce91e038451894daef45d339c4374c0b6ac Author: Ankith Anil Das <46082093+ankithadas@users.noreply.github.com> Date: Thu Dec 21 05:58:00 2023 +1100 Eliminating Matrix operations in MLMG CG bottom solver if initial vector is zero (#3668) A matrix multiplication and a few copy operations can be avoided if the input vector is zero. MLMG calls all the the bottom solvers with zeroed `x` vector, and thus the initial residual calculation `b - Ax` is `b`. Furthermore, it also eliminates the memory requirement of storing the initial vector. Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLMG.H commit ef38229189e3213f992a2e89dbe304fb49db9287 Author: Max Katz Date: Mon Dec 18 23:21:03 2023 -0500 Add a for loop that is unrolled at compile time (#3674) ## Summary The constexpr_for function is fully unrolled at compile time. This is useful for relatively short loops where some of the functions inside the loop are known to possible to evaluate at compile time and may be relatively expensive, so evaluating them at compile time rather than at runtime may be beneficial for performance reasons. ## Additional background This has been used in AMReX-Astro/Microphysics successfully in the context of evaluating some nuclear reaction network quantities at compile time. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_Loop.H commit d1e55fbab7c5d5cb72d3c448a91430a568628b9e Author: Andrew Myers Date: Mon Dec 18 12:01:08 2023 -0800 Add PTD version of getParticleCell (#3675) The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleUtil.H commit 78c4fdaf95175f992b87e670780f5cf0d01d0783 Author: Andrew Myers Date: Mon Dec 18 11:42:06 2023 -0800 Improve ParIter docs (#3676) Show how to get the ParIter type from the ParticleContainer - should be less confusing to new users. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Docs/sphinx_documentation/source/Particle.rst commit 3497f016d9204adf2b406659100adb2803118468 Author: Weiqun Zhang Date: Fri Dec 15 17:27:48 2023 -0800 Fix CI for ROCm 6.0 (#3673) Need to explicitly install hiprand package in CI because it's now a standalone project, not a submodule for rocRand according to the release notes. .github/workflows/dependencies/dependencies_hip.sh commit 554b1ca3b00c3a56cbed4861bfa16c8074a78fc5 Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Fri Dec 15 01:21:16 2023 +0100 PureSoA IdCpu fixes (#3671) ## Summary I noticed a few issues in AMReX while trying to update HiPACE++ for #3585. Additionally, I would like to point out that `ParticleTile` has `push_back_real` and `push_back_int` functions but for PureSoA there is no `push_back_idcpu`, however this is not added in this PR. ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_StructOfArrays.H commit 0c6f2b4ad81178a6e14f26ebb5f2ad6c642c785c Author: Weiqun Zhang Date: Thu Dec 14 14:55:44 2023 -0800 CMake: AMReX_PARALLEL_LINK_JOBS (#3628) Add AMReX_PARALLEL_LINK_JOBS option to the CMake build system and use it for SYCL builds. Adjust SYCL's RelWithDebInfo mode to the recommendation of Intel VTune. This is a follow-up on #3498 that made similar changes to GNU Make. .github/workflows/intel.yml Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXFlagsTargets.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake commit 0a208bbc4635abacd9f087b62aa3cdffc25db113 Author: Weiqun Zhang Date: Thu Dec 14 11:04:37 2023 -0800 Clang-Tidy in CI: Keep Going after Errors (#3667) Add `-k` to the make command running clang-tidy. With that, the jobs will keep going and show all the clang-tidy check errors instead of stopping on the first error. .github/workflows/bittree.yml .github/workflows/clang.yml .github/workflows/gcc.yml .github/workflows/hypre.yml .github/workflows/petsc.yml .github/workflows/smoke.yml .github/workflows/sundials.yml commit d988c985950bc0e6733db672a62cd19b8c046c9a Author: Edward Basso Date: Thu Dec 14 09:45:24 2023 -0800 Delete empty below comments on classes and functions (#3669) Deleting the empty lines between the Doxygen comments and the class declarations makes Intellisense actually recognize the connections. Src/Amr/AMReX_Amr.H Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_Derive.H Src/Amr/AMReX_LevelBld.H Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateDescriptor.H Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_FillPatcher.H Src/AmrCore/AMReX_FluxRegister.H Src/AmrCore/AMReX_InterpFaceRegister.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_TagBox.H Src/Base/AMReX_Arena.H Src/Base/AMReX_BArena.H Src/Base/AMReX_BCRec.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BoxDomain.H Src/Base/AMReX_BoxList.H Src/Base/AMReX_CArena.H Src/Base/AMReX_CoordSys.H Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FPC.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_IndexType.H Src/Base/AMReX_IntVect.H Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_NFiles.H Src/Base/AMReX_Orientation.H Src/Base/AMReX_PArena.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_RealVect.H Src/Base/AMReX_RungeKutta.H Src/Base/AMReX_Vector.H Src/Base/AMReX_VisMF.H Src/Boundary/AMReX_BoundCond.H Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_Mask.H Src/Boundary/AMReX_YAFluxRegister.H Src/EB/AMReX_EBFluxRegister.H Src/Extern/Bittree/AMReX_Bittree.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Src/Extern/SUNDIALS/AMReX_SUNMemory.H Src/Extern/SUNDIALS/AMReX_Sundials_Core.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H commit b903e8896afbb202cf5dca786ba75b0cb598ad02 Author: Weiqun Zhang Date: Wed Dec 13 11:53:12 2023 -0800 Documentation for Profiling: Hot Spots and Load Balance (#3622) Add more documentation on identifying hot spots and load imbalance in profiling results. --------- Co-authored-by: Andrew Myers Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst commit cc4c4ffeb28c1db62768eebe68ac81f34c0112f2 Author: Andrew Myers Date: Wed Dec 13 10:05:00 2023 -0800 Fix warnings in SortParticlesForDeposition (#3664) Src/Particle/AMReX_ParticleUtil.H commit 064db4eaa5992788555ebd5e52824b885f8d0340 Author: Weiqun Zhang Date: Tue Dec 12 15:48:52 2023 -0800 Fix Resize Issue of Fab with the Async Arena (#3663) ## Summary Previously there was an issue with resizing Fabs using The_Async_Arena. The issue was the previous allocation during resize might be done on a different stream. This commit fixes the issue and makes the following patterns work. FArrayBox tmp0(The_Async_Arena()); FArrayBox tmp1(The_Async_Arena()); FArrayBox tmp2; for (MFIter ...) { tmp0.resize(box,ncomp,The_Async_Arena()); tmp1.resize(box,ncomp); tmp2.resize(box,ncomp,The_Async_Arena()); } ## Additional background https://github.com/AMReX-Astro/Castro/pull/2677 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_Arena.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_PArena.H commit ecaa46d0be4b5c79b8806e48e3469000d8bb7252 Author: Axel Huebl Date: Mon Dec 11 20:38:06 2023 -0800 Fix SuperParticle `push_back` (#3661) ## Summary From #3585 commit: fixes a segfault for the legacy particle layout: ``` Thread 1 "python3" received signal SIGSEGV, Segmentation fault. 0x00007ffff5951e5c in amrex::ParticleTile, 2, 1, std::allocator>::push_back<2, 1, 0> (this=0x555555f206e0, sp=...) at /home/axel/src/pyamrex/build/_deps/fetchedamrex-src/Src/Particle/AMReX_ParticleTile.H:916 916 m_soa_tile.GetIdCPUData()[np] = sp.m_idcpu; ``` ## Additional background X-ref: https://github.com/AMReX-Codes/pyamrex/pull/232 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleTile.H commit d884f44493c0b0dcd15660aa64fb42a1bdbce0e5 Author: Axel Huebl Date: Mon Dec 11 11:48:26 2023 -0800 Pure SoA Particle: Separate Array for IdCPU (#3585) ## Summary This addresses a regression we see when moving to pure SoA particles: - slightly slower read/write to Ids when needed, e.g., for sorting - issues going up to the full 64bit range ## Additional background Once finished, this will close #3569. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Andrew Myers Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_StructOfArrays.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/Particles/RedistributeSOA/main.cpp commit 43d71da32fa49d9d92f87f1f3e29fd67eb6952bb Author: Weiqun Zhang Date: Fri Dec 8 11:25:59 2023 -0800 Limit the scope of gpu_rand_generator (#3659) Move gpu_rand_generator from namespace amrex to anonymous namespace. The variable is only used in AMReX_Random.cpp, so there are no reasons for it to be in the amrex namespace. Src/Base/AMReX_Random.cpp commit ccd635716e70b70baf65459897b3fcb01dd0dc49 Author: Weiqun Zhang Date: Fri Dec 8 11:00:29 2023 -0800 Fix a typo in doxygen for NonLocalBC::FillBoundary (#3658) Src/Base/AMReX_NonLocalBC.H commit edb4c25027efbbc465c88d453441dcd7115d8651 Author: Weiqun Zhang Date: Wed Dec 6 09:44:58 2023 -0800 GNU Make: Fix name collision for aurora (#3656) Tools/GNUMake/Make.machines commit efd77ffab39c4e32e2b9e4a4fd446a49e22f0c9a Author: Ann Almgren Date: Sat Dec 2 16:51:48 2023 -0800 two separate fixes -- particle_compare and ref_ratio=1 (#3655) ## Summary ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [X] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/AmrCore/AMReX_AmrMesh.cpp Tools/Postprocessing/C_Src/particle_compare.cpp commit 2f1b1d1206eec72dbbc364c5c95c5e515f4fe479 Author: Candace Gilet Date: Sat Dec 2 17:47:38 2023 -0500 Clarify documentation on setEBDirchlet() and fix link to AMReX-Hydro (#3652) ## Summary ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Docs/sphinx_documentation/source/LinearSolvers.rst Docs/sphinx_documentation/source/conf.py commit 72c333d39ce9a7ba2e2bec85b2d9ef4f5230ca9b Author: Weiqun Zhang Date: Sat Dec 2 14:46:30 2023 -0800 Robustify the Cache Cleanup Scripts (#3650) Make changes to handle the cases where the workflow names contain spaces. Note that none of the workflow names in amrex has spaces. .github/workflows/cleanup-cache-postpr.yml .github/workflows/cleanup-cache.yml commit 6e8c125428a5390dfd04aff738a35588e68de4e8 Author: Weiqun Zhang Date: Sat Dec 2 14:45:59 2023 -0800 Disable CodeQL scheduled jobs on forks (#3649) .github/workflows/codeql.yml commit eb388ae90a7534b32c67f0410cd595a39bc362f9 Author: Andrew Myers Date: Sat Dec 2 14:45:29 2023 -0800 Work around compiler bug in nvcc 12.2 by using functor instead of lambda (#3653) This fixes an issue seen in ERF on garuda. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleIO.H commit 2f47fa7361bbf5793503cfb31b717bece889bde0 Author: Weiqun Zhang Date: Fri Dec 1 09:54:52 2023 -0800 Update CHANGES for 23.12 (#3651) CHANGES commit 9b733ec45cd93a80234a7c98248b6eb4816589d5 Author: Edward Basso Date: Wed Nov 29 17:43:44 2023 -0600 solve_cg: avoid use of MF `z` (#3637) ## Summary The method `solve_cg` is modified so that the MF `z` is no longer used, thus saving on memory usage. ## Additional background This is similar to the PR that changed `solve_bicgstab` that avoided use of `s`. Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H commit 0cd1385bde604099e5822d636f85d724d74d756e Author: Axel Huebl Date: Wed Nov 29 15:41:35 2023 -0800 Fix: nosmt OMP Threads Default (#3647) ## Summary Fix that `OMP_NUM_THREADS` was ignored in non-verbose runs. ## Additional background Follow-up to #3607 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_OpenMP.cpp commit 44c5c7a5da23c1a5e7e899612b1df696cea7e4fe Author: Andrew Myers Date: Wed Nov 29 15:07:44 2023 -0800 When checking for periodic outs on GPU, copy full particle data (#3646) Fixes a bug in mapped coordinates when GPUs are enabled. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleUtil.H commit 4b8dd0373188ca6505e3a9dfaaafbaf4b7fe5ffe Author: Edward Basso Date: Tue Nov 28 19:07:39 2023 -0600 Comments on FAB-related classes and methods (#3636) New or updated comments for various FAB-related classes and methods. Descriptions are tweaked to match standard comment conventions and make IntelliSense happy. For example, some descriptions get moved to be just before their respective class declarations. Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BoxArray.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabConv.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp commit 4b64003c80099627e9da856801517844f2bb21b8 Author: Edward Basso Date: Mon Nov 27 15:58:46 2023 -0600 Comments: correct some typos (#3641) ## Summary This PR fixes some typos and spelling errors in the comments that codespell seems to have missed. CHANGES Src/AmrCore/AMReX_FillPatcher.H Src/AmrCore/AMReX_InterpFaceRegister.H Src/Base/AMReX_BCRec.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_iMultiFab.H Src/EB/AMReX_EB_utils.cpp Src/F_Interfaces/Base/AMReX_parallel_mod.F90 Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit 60b45bc19b691c1187f4cfd9cd39b2744bf7e35a Author: Weiqun Zhang Date: Mon Nov 27 13:57:39 2023 -0800 MLEBABecLap: Support Robin BC at Domain Boundaries (#3617) Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H commit 9e35dc19489dc5d312e92781cb0471d282cf8370 Author: Axel Huebl Date: Mon Nov 20 21:04:03 2023 -0800 Ascent: SoA Particle Support (#3350) ## Summary Add support for pure SoA layouted particle containers for Ascent. ## Additional background Follow-up to #2878. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Andrew Myers Src/Extern/Conduit/AMReX_Conduit_Blueprint.H Src/Extern/Conduit/AMReX_Conduit_Blueprint_ParticlesI.H Tests/Particles/Ascent_Insitu_SOA/CMakeLists.txt Tests/Particles/Ascent_Insitu_SOA/GNUmakefile Tests/Particles/Ascent_Insitu_SOA/inputs.rt Tests/Particles/Ascent_Insitu_SOA/main.cpp commit c51e0a3071f3f1d2a11b6c339bff58d4ba5e30e8 Author: Axel Huebl Date: Mon Nov 20 13:59:06 2023 -0800 CI: Ascent + Conduit (#3639) ## Summary For some reason, `AMReX_ASCENT` does not trigger `AMReX_CONDUIT` to be set, too. cc @cyrush ## Additional background X-ref: #3350 .github/workflows/ascent.yml commit d75c04b422dae567c9d3c74482b5be84d002ee57 Author: Edward Basso Date: Mon Nov 20 13:57:50 2023 -0800 solve_bicgstab: use fewer MFs (#3635) ## Summary This PR cuts the number of MFs used in `solve_bicgstab`, saving on memory and LocalCopy operations. In particular, the MFs `ph` and `sh` are removed. ## Additional background This is a follow up to avoid-use-of-s and other PRs to improve `solve_bicgstab`. My own testing has shown that this PR gives the same results as before, but regression testing should be done to verify this in all cases. Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H commit 175b99d913dc2748e43c53192737170c770fe0e8 Author: Edward Basso Date: Fri Nov 17 10:22:21 2023 -0800 solve_bicgstab: cut use of `s` (#3629) ## Summary The MF named `s` seems to be an unnecessary usage of memory as the residue `r` can fulfill its roles in the algorithm. This PR replaces `s` with `r` and `LinComb` with `Saxpy` as appropriate. ## Additional background This PR is part of a larger request to improve `solve_bicgstab` and `solve_cg`. Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H commit d93f34416da3dfb5b50e5a8b0cbb3b6f78bd0569 Author: Ankith Anil Das <46082093+ankithadas@users.noreply.github.com> Date: Fri Nov 17 04:57:41 2023 +1100 Bug fix for amrex::Subtract when called with interger nghost (#3634) ## Summary `amrex::Subtract` leads to an infinite recursion when invoked with an integer value for nghost. To address this issue, explicit casting of the integer to `IntVect` has been incorporated. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_FabArrayUtility.H commit d171635b3a54fb9e8896c9562bbedaa5f3a894cd Author: Edward Basso Date: Thu Nov 16 08:49:21 2023 -0800 Fix typo in `MLMGT::getGradSolution` when `MF` is different from `AMF` (#3631) ## Summary This PR fixes a small typo in `MLMGT::getGradSolution` that causes an error when `AMR` is distinct from `MF`. A test demonstrating the fix is added for the case where `MF`=`MultiFab` and `AMF`=`fMultiFab`. ## Additional background This typo was noticed when working on generalizing the `MLMG` template to the WIP class `FabArraySet`. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX Src/LinearSolvers/MLMG/AMReX_MLMG.H Tests/LinearSolvers/EBflux_grad/MyTest.H Tests/LinearSolvers/EBflux_grad/MyTest.cpp commit a9da2a54de5d704dda746c8b036de469ea19f4e9 Author: Weiqun Zhang Date: Wed Nov 15 12:56:44 2023 -0800 SUNDIALS: Use sunrealtype instead of realtype (#3632) The latter has been deprecated. Src/Extern/SUNDIALS/AMReX_Sundials.H Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H commit 15a0bb9a8c1b34136632b16c5511375e9d56b184 Author: Weiqun Zhang Date: Tue Nov 14 19:36:40 2023 -0800 SYCL: Use get_multi_ptr instead of get_pointer (#3630) The latter has been deprecated in SYCL 2020. .github/workflows/intel.yml Src/Base/AMReX_GpuLaunchFunctsG.H commit af1e1be8d7d41de9b999b4973b2024281a28f23e Author: Weiqun Zhang Date: Tue Nov 14 11:13:47 2023 -0800 Plotfile Tools: GPU support (#3626) ## Summary Add GPU support for fcompare, fextrema, fsnapshot and fvolumesum. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_PlotFileDataImpl.cpp Tools/Plotfile/fextrema.cpp Tools/Plotfile/fsnapshot.cpp Tools/Plotfile/fvolumesum.cpp commit b7408ea6e8feca7eab2f7cf30606d066b6699814 Author: Edward Basso Date: Mon Nov 13 18:56:02 2023 -0800 solve_cg: use linop.make instead of MF constructor (#3627) ## Summary This PR replaces the explicit use of `MF` constructors in `MLCGSolverT::solve_cg` with calls to the make method of the linear operator associated with the `MLCGSolverT` object. ## Additional background This is a similar to the PR on `solve_bicgstab`. Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H commit fa3743fd1fdd5e3f1b12d431b0f6bb4b15bb7b95 Author: Weiqun Zhang Date: Mon Nov 13 15:00:56 2023 -0800 CArena: shrink_in_place and operator<< (#3621) ## Summary Implement CArena::shrink_in_place, which is used by PODVector::shrink_to_fit. It avoids a new memory allocation and data movement. Add operator<< to CArena. This helps debugging. ## Additional background Follow-up on #3426. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp commit 6e2b831245f6fdac9a714c64417bd5ab21fb613d Author: Edward Basso Date: Mon Nov 13 15:00:36 2023 -0800 solve_bicgstab: use linop.make instead of MF constructor (#3619) ## Summary This PR replaces the explicit use of MF constructors in ```MLCGSolverT::solve_bicgstab``` with calls to the `make` method of the linear operator associated with the MLCGSolverT object. ## Additional background The use of `MLLinOpT::make` allows for inheritance of MLCGSolverT without an override of `solve_bicgstab` even if the MF class lacks a constructor with the same arguments as those MultiFab. For the MLMG template classes, `make` should generally be used instead of explicit MF constructors. Another PR to change this in `solve_cg` will follow once this is fully vetted and approved. Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H commit d36463103daed09a40cdea235041a6ab79ff280c Author: Ben Wibking Date: Tue Nov 7 00:19:10 2023 -0500 replace AMREX_DEVICE_COMPILE with AMREX_IF_ON_DEVICE and AMREX_IF_ON_HOST (#3591) ## Summary This adds the macros `AMREX_IF_ON_DEVICE((code_for_device))` and `AMREX_IF_ON_HOST((code_for_host))` that are compatible with single-pass host/device compilation (as used by `nvc++ -cuda`), as well as backward compatible with all other compilers. This also replaces all uses of `AMREX_DEVICE_COMPILE` with these macros. Fixes https://github.com/AMReX-Codes/amrex/issues/3586. ## Additional background Single-pass compilation evalutes the preprocessor macros once for each source file. This means that preprocessor conditionals cannot be used to choose between host and device code. In particular, NVHPC with `-cuda` does not support `__CUDA_ARCH__`, instead requiring the use of the `if target` construct. This creates portable macros that work for either single-pass or two-pass compilation, but requires restructuring of any code that uses AMREX_DEVICE_COMPILE so that the code appears as a macro argument. This PR will allow using NVHPC with `-cuda` as the unified host/device compiler for AMReX. In the future, single-pass compilers for other backends may be available, e.g., SYCL (https://dl.acm.org/doi/abs/10.1145/3585341.3585351). AMReX can be configured to build with `nvc++ -cuda` using CMake: ``` cmake .. -DAMReX_GPU_BACKEND=CUDA -DCMAKE_C_COMPILER=nvc -DCMAKE_CXX_COMPILER=nvc++ -DCMAKE_CUDA_COMPILER=nvc++ -DCMAKE_CUDA_COMPILER_ID=NVCXX -DCMAKE_CUDA_ARCHITECTURES=80 -DCMAKE_CUDA_COMPILER_FORCED=ON -DCMAKE_CUDA_COMPILE_FEATURES=cuda_std_17 -DAMReX_GPU_RDC=OFF -DCMAKE_CXX_FLAGS="-cuda --gcc-toolchain=$(which gcc)" -DCMAKE_CUDA_FLAGS="-cuda --gcc-toolchain=$(which gcc)" -DAMReX_ENABLE_TESTS=ON -DCMAKE_CUDA_HOST_LINK_LAUNCHER=nvc++ -DCMAKE_CUDA_LINK_EXECUTABLE=" -o " ``` CMake hacks (https://github.com/NVIDIA/cub/blob/0fc3c3701632a4be906765b73be20a9ad0da603d/cmake/CubCompilerHacks.cmake) are tested with CMake 3.22.1 and NVHPC 23.5, 23.7, and 23.9 (earlier versions do not work). However, it currently fails to link the executables for the tests due to a [compiler/linker bug](https://forums.developer.nvidia.com/t/nvc-cuda-fails-to-link-code-when-using-device-curand-functions/270401/5). (Note that by default, `nvcc` preserves denormals, whereas `nvc++` does not. Also, `nvc++` generates relocatable device code by default, whereas `nvcc` does not.) ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX.H Src/Base/AMReX_Algorithm.H Src/Base/AMReX_Array4.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuRange.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_Math.H Src/Base/AMReX_Random.H Src/Base/AMReX_TableData.H Src/Base/Parser/AMReX_IParser.H Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser_Y.H Src/EB/AMReX_EB2_GeometryShop.H commit a7afcba3cffd86acc748edf39a8a2f33c973bf5d Author: Axel Huebl Date: Thu Nov 2 08:45:19 2023 -0700 `amrex.omp_threads`: Can Avoid SMT (#3607) ## Summary In all our applications in BLAST, the OpenMP default to use all [logical cores on modern CPUs](https://en.wikipedia.org/wiki/Simultaneous_multithreading) results in significantly slower performance than just using the physical cores with AMReX. Thus, we introduce a new option `amrex.omp_threads` that enables control over the OpenMP threads at startup and has - for most popular systems - an implementation to find out the actual number of physical threads and default to it. For codes, users that change the default to `amrex.omp_threads = nosmt`, the `OMP_NUM_THREADS` variable will still take precedence. This is a bit unusual (because CLI options usually have higher precedence than env vars - and they do if the user provides a number here), but done intentionally: this way, codes like WarpX can set the `nosmt` default and HPC job scripts will set the exact, preferably benchmarked number of threads as usual without surprises. - [x] document ## Tests Performed for AMReX OMP Backend Tests were performed with very small examples, WarpX 3D LWFA test as checked in or AMReX AMRCore 3d test. - [x] Ubuntu 22.04 Laptop w/ 12th Gen Intel i9-12900H: @ax3l - 20 logical cores; the first 12 logical cores use 2x SMT/HT - 20 virtual (default) -> 14 physical (`amrex.omp_threads = nosmt`) - faster runtime! - [x] Perlmutter (SUSE Linux Enterprise 15.4, kernel 5.14.21) - [CPU node](https://docs.nersc.gov/systems/perlmutter/architecture/) with 2x [AMD EPYC 7763](https://www.amd.com/en/products/cpu/amd-epyc-7763) - 2x SMT - 256 default, 128 with `amrex.omp_threads = nosmt` - faster runtime! - [x] Frontier (SUSE Linux Enterprise 15.4, kernel 5.14.21) - 1x AMD EPYC 7763 64-Core Processor (w/ 2x SMT enabled) - 2x SMT - 128 default - 64 with `amrex.omp_threads = nosmt` - faster runtime! - The ideal result might also be lower, due to first cores used by OS and [low-noise cores](https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#low-noise-mode-layout) after that. But that is an orthogonal question and should be set in job scripts: `#SBATCH --ntasks-per-node=8` `#SBATCH --cpus-per-task=7` `#SBATCH --gpus-per-task=1` - [x] Summit (RHEL 8.2, kernel 4.18.0) - 2x IBM Power9 (each 22 physical cores each, each 6 disabled/hidden for OS?, 4x SMT enabled; cpuinfo says 128 total) - 4x SMT - 128 default, 32 with `amrex.omp_threads = nosmt` - faster runtime! - [x] [Lassen](https://hpc.llnl.gov/hardware/compute-platforms/lassen) (RHEL 7.9, kernel 4.14.0) - 2x IBM Power9 (each 22 physical cores, each 2 reserved for OS?, 4x SMT enabled) - 4x SMT - 160 default, 44 with `amrex.omp_threads = nosmt` - faster runtime! - The ideal result might be even down to 40, but that is an orthogonal question and should be set in job scripts. - [x] macOS M1 (arm64/aarch64) mini: - no SMT/HT - 8 default, 8 with `amrex.omp_threads = nosmt` - [x] macOS (OSX Ventura 13.5.2, 2.8 GHz Quad-Core Intel Core i7-8569U) Intel x86_64 @n01r - 2x SMT - 8 default, 4 with `amrex.omp_threads = nosmt` - faster runtime! - [x] macOS (OSX Ventura 13.5.2) M1 Max on mac studio @RTSandberg - no SMT/HT - 10 default, 10 with `amrex.omp_threads = nosmt` - [ ] some BSD/FreeBSD system? - no user requests - low priority, we just keep the default for now - [ ] Windows... looking for a system ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/InputsComputeBackends.rst Docs/sphinx_documentation/source/Inputs_Chapter.rst Src/Base/AMReX.cpp Src/Base/AMReX_OpenMP.H Src/Base/AMReX_OpenMP.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 606a94c69d640a03d5fbf35d11f1c46bf9676e47 Author: Axel Huebl Date: Wed Nov 1 15:11:03 2023 -0700 [Breaking] Prefix `amrex_` to each plotfile Tool (#3600) ## Summary Packaging AMReX for Conda, we received feedback that our plotfile tool names are too generic and can collide with other packages. Thus, we propose to rename them with a common prefix. - [x] decide prefix or suffix (suffix: `` completion will work; prefix: easier for new users) - [x] decide name: `plt`/`amrex`/`amr`/... - Weiqun and Axel prefer `amrex_` so far - [ ] also change for GNUmake - [ ] update [test harness](https://github.com/AMReX-Codes/regression_testing) - [ ] update user-facing documentation for tools - [x] ~update [tutorials](https://github.com/AMReX-Codes/amrex-tutorials)?~ ## Additional background https://github.com/conda-forge/staged-recipes/pull/24294 ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Weiqun Zhang .github/workflows/clang.yml .github/workflows/gcc.yml Tools/Plotfile/CMakeLists.txt commit f5400421a4b1ba84be3dda98040a49ff2ef89149 Author: Weiqun Zhang Date: Wed Nov 1 12:02:02 2023 -0700 FillRandom: Use MKL host API (#3536) Src/Base/AMReX_Random.cpp Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit 1269a5bf41a8c4f0a832581532fda6e723d9aadc Author: Rakesh Roy <137397847+rakesroy@users.noreply.github.com> Date: Thu Nov 2 00:17:30 2023 +0530 use hipPointerAttribute_t.type as HIP is removing hipPointerAttribute_t.memoryType (#3610) ## Summary This replaces hipPointerAttribute_t.memoryType with hipPointerAttribute_t.type. ## Additional background In ROCm6.0 hipPointerAttribute_t.memoryType will be removed from HIP. Instead hipPointerAttribute_t.type to be used. This is causing build failure in https://github.com/Exawind/amr-wind.git. hipPointerAttribute_t.type has been existing since ROCm5.5, so this change will be backward compatible till ROCm5.5. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Co-authored-by: Rakesh Roy Src/Base/AMReX_GpuUtility.H commit ae7b64bcf6a4ed36dd03e17357cae83b0a394912 Author: Weiqun Zhang Date: Wed Nov 1 08:55:58 2023 -0700 Update CHANGES for 23.11 (#3613) CHANGES commit 601cc4ee80e0d7435a5b099202459bbb1f16b7a6 Author: Klaus Date: Tue Oct 31 14:07:41 2023 -0500 Give FlashFluxRegisters ways to accumulate data in registers (#3597) I introduce variant methods that allow adding to previously stored data in Flash(-X) flux registers, rather than just copying over the older data. For Flash-X'ers: Something like this is needed to get flux correction working in the nontelescoping variant of the current Spark Hydro implementation. ## Summary ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Weiqun Zhang Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.H Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.cpp Src/F_Interfaces/AmrCore/AMReX_flash_fluxregister_fi.cpp Src/F_Interfaces/AmrCore/AMReX_flash_fluxregister_mod.F90 commit be6c6415467d09da6109d27cfa218868abc1f9db Author: Axel Huebl Date: Thu Oct 26 09:50:01 2023 -0700 `AMReXBuildInfo.cmake`: AMReX_DIR (#3609) ## Summary The `AMReX_DIR` points if set to the CMake module path root. The old logic did not work for me in a situation (ImpactX) where: - AMReX is pre-installed but - found through a superbuild of another transient lib (ABLASTR) ## Additional background Follow-up to #3599 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Tools/CMake/AMReXBuildInfo.cmake commit 152fdac6aa4c88ddaf43a9b10952364a885b02a7 Author: Bruce Perry <53018946+baperry2@users.noreply.github.com> Date: Mon Oct 23 11:18:33 2023 -0600 update doc for amrex::Abort on GPU (#3605) The docs on the behavior of amrex::Abort and related functions appear to be out of date, so this clarifies that when running on GPU these functions do not do anything when called from within GPU kernels unless you compile in debug mode or with `USE_ASSERTION=TRUE`. Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/GPU.rst commit da79aff8053058371a78d4bf85488384242368ee Author: Jordan Musser Date: Mon Oct 23 11:21:38 2023 -0400 Add runtime particle components to HDF5 wrapper (#3596) ## Summary Add runtime components when computing the sizes/offsets for particle HDF5 files. ## Additional background ## Checklist The proposed changes: - [ x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Jordan Musser Co-authored-by: Andrew Myers Src/Extern/HDF5/AMReX_ParticleHDF5.H commit 54c337f36164227a9b7dd74d5e647cf3627a23d0 Author: Axel Huebl Date: Thu Oct 19 18:03:56 2023 -0700 Windows: Fix Installed AMReXBuildInfo.cmake (#3606) ## Summary Account for different CMake path on Unix vs. Windows. https://cmake.org/cmake/help/latest/command/find_package.html#config-mode-search-procedure ## Additional background Follow-up to #3599 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Tools/CMake/AMReXBuildInfo.cmake commit 7c5ff6911763109c92f48f2f3e9fea06b2e868b8 Author: Weiqun Zhang Date: Thu Oct 19 15:04:52 2023 -0700 Print AMReX version at the beginning of Initialize (#3604) Oftentimes, a GPU run dies at the first GPU kernel inside amrex::Initialize before printing a message showing AMReX with a certain version has been initialized. Adding a message at the beginning of amrex::Initialize can provide useful information. Src/Base/AMReX.cpp commit c7b1ac7b836369daabc1dc00e3a3447e8dd56384 Author: Axel Huebl Date: Thu Oct 19 12:32:40 2023 -0700 Install Move Tools to `shared/amrex` (#3599) ## Summary Packaging AMReX for Conda, we realized that we break with common conventions on Unix systems by installing a top-level `Tools/` directory. This moves its content in both the build directory and the install destination to `share/amrex/` and `/AMReXCMakeModules/`, respectively. ## Additional background https://github.com/conda-forge/staged-recipes/pull/24294 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Weiqun Zhang Tools/CMake/AMReXBuildInfo.cmake Tools/CMake/AMReXInstallHelpers.cmake Tools/CMake/AMReXTypecheck.cmake commit 88d08113d7ba5ad6a75e0f47b560b4ccd00084ec Author: Jon Rood Date: Thu Oct 19 13:17:14 2023 -0600 Revert "Add ability for GCC 8 in CMake to build fgradient which uses std::filesystem" (#3601) Reverts AMReX-Codes/amrex#3523 Tools/CMake/FindFilesystem.cmake Tools/Plotfile/CMakeLists.txt commit 83bc468e2434855952b0ba010bac9e8949a0c7a0 Author: Weiqun Zhang Date: Thu Oct 19 12:16:15 2023 -0700 Avoid std::filesystem (#3602) It has caused a lot of issues on systems with a default gcc <= 8. PR #3523 was trying to the issue. But unfortunately it does not always work. So we are going to simply avoid std::filesystem. Tools/Plotfile/fgradient.cpp commit 7ee29121ed70d7e255ad98a8b1690d345cb4fb33 Author: Weiqun Zhang Date: Mon Oct 16 12:52:06 2023 -0700 Fix Assertion in MLEBNodeFDLaplacian (#3594) ## Summary MLEBNodeFDLaplacian is never singular. MLNodeLinOp::m_is_bottom_singular should be ignored. This bug caused an assertion error (although it should not affect result if there is no assertion). Also to prevent this from happening again, m_is_bottom_singular has been made private. ## Additional background https://github.com/ECP-WarpX/WarpX/pull/4363 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H commit ecaa14456e3f5916e6ef2b08095d61f61f19eeae Author: Weiqun Zhang Date: Wed Oct 11 16:22:23 2023 -0700 Fix a memory "leak" in VisMF's persistent streams (#3592) ## Summary VisMF uses a std::map to keep track of the streams it opens. Each PersistentIFStream object also has a vector used as IO buffer. When a stream is closed, we need to free to memory used by the buffer. The issue was it used std::vector::clear(), which only changes size() without reducing capacity() at all. So the memory was never freed. The issue is now fixed by std::vector::swap. ## Additional background Thank Simon Guichandut (@simonguichandut) for reporting this! Thank Eric Johnson (@yut23) for pinning down the location of the leak with the help of the Massif heap profiler! ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_VisMFBuffer.H commit 77d4d1fe5ce68a1e71095093ce856e061f24fc07 Author: Axel Huebl Date: Mon Oct 9 10:21:25 2023 -0700 RealVect Static: Export (#3589) ## Summary Export global symbols for Windows. Seen in WarpX on conda-forge: ``` lld-link: warning: ignoring unknown argument '-lpthreads' lld-link: error: undefined symbol: public: static class amrex::RealVect const amrex::RealVect::Zero >>> referenced by libwarpx.rz.NOMPI.OMP.DP.PDP.OPMD.PSATD.QED.lib(SpectralKSpaceRZ.cpp.obj):(public: __cdecl SpectralKSpaceRZ::SpectralKSpaceRZ(class amrex::BoxArray const &, class amrex::DistributionMapping const &, class amrex::RealVect)) ``` ## Additional background https://github.com/conda-forge/warpx-feedstock/pull/72 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_RealVect.H commit 01b30b79186b329b1b344463af9505058ae8b3e4 Author: Ann Almgren Date: Mon Oct 9 10:20:06 2023 -0700 change MaxCnt from 4 to max(4,max_level+1) for how many iterations we… (#3588) … allow in creation of the initial grid hierarchy Src/Amr/AMReX_Amr.cpp commit d39abbc396d0a9cc123245947b560954a3e9f16d Author: Weiqun Zhang Date: Sun Oct 8 11:32:25 2023 -0700 Add Bittree CI (#3577) Also fix some clang-tidy warnings. Note that the 3D test does not actually work as expected. It has zero cells. We will try to fix in the future, because it's out of the scope of this PR adding CI. .github/workflows/bittree.yml .github/workflows/cleanup-cache.yml Src/Extern/Bittree/AMReX_Bittree.H Src/Extern/Bittree/AMReX_Bittree.cpp commit f817d776f544410c315cf6007564f88f6b16fc2b Author: Weiqun Zhang Date: Sun Oct 8 11:32:13 2023 -0700 BCType::ext_dir_cc (#3581) Add a new BC type `BCType::ext_dir_cc` to address the issue in #3580. For cell conservation interpolation, the boundary location for `BCType::ext_dir` is on the domain face. Applications assuming boundary location at cell centers can now use `BCType::ext_dir_cc`. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Docs/sphinx_documentation/source/Basics.rst Src/Amr/AMReX_StateDescriptor.cpp Src/Base/AMReX_BCUtil.H Src/Base/AMReX_BCUtil.cpp Src/Base/AMReX_BC_TYPES.H Src/Base/AMReX_FILCC_1D.F90 Src/Base/AMReX_FILCC_2D.F90 Src/Base/AMReX_FILCC_3D.F90 Src/Base/AMReX_FilCC_C.cpp Src/Base/AMReX_bc_types.fi Src/Base/AMReX_bc_types_mod.F90 Src/Base/AMReX_filcc_mod.F90 Tests/FortranInterface/Advection_F/Source/fillpatch_mod.F90 Tests/GPU/CNS/Source/CNS_bcfill.cpp commit fd2dfa282a30c497b692398cc0db9b18e6c4de85 Author: Weiqun Zhang Date: Fri Oct 6 09:46:30 2023 -0700 Disable CCache in Windows CIs (#3566) CCache has stopped working on Windows since 2023-09-20 for unknown reasons. After a couple of hours of investigation, I still could not get it to work. So let's disable it for now. .github/workflows/cleanup-cache.yml .github/workflows/windows.yml commit a9dad648b5336d33ede4ee4e1b9b4456cbce44af Author: Weiqun Zhang Date: Fri Oct 6 09:45:29 2023 -0700 Fix ICC CI by Freeing up Disk Space (#3583) The ICC CI has been failing in the last few days with following message, `System.IO.IOException: No space left on device`. So we need to free up more disk space. .github/workflows/intel.yml commit d197c8f5db9eb8123804ec849bd5b67728a4172f Author: Axel Huebl Date: Thu Oct 5 09:26:05 2023 -0700 Docs: Link pyAMReX (#3582) Link Python interfaces after Fortran interfaces. Docs/sphinx_documentation/source/Python_Chapter.rst Docs/sphinx_documentation/source/index.rst commit 87556458d867cb9cb2f148b4bcfc920c67bc918a Author: Weiqun Zhang Date: Tue Oct 3 17:13:36 2023 -0700 NodeABecLaplacian: Reuse (#3579) Add reuse capability to NodeABecLaplacian. More specifically, we need to average down coefficients again. .codespell-ignore-words Src/AmrCore/AMReX_AmrMesh.H Src/Base/AMReX_CoordSys.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_NonLocalBC.H Src/EB/AMReX_EB_StateRedistribute.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/HYPRE/AMReX_HypreSolver.H Src/LinearSolvers/MLMG/AMReX_MLNodeABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeABecLaplacian.cpp Tests/HDF5Benchmark/sz.config Tools/CMake/AMReXBuildInfo.cmake Tools/CMake/AMReXClangTidy.cmake Tools/C_scripts/mkdep commit e470d3350ed34686d46a461f7c5e704a0e027efc Author: drangara <69211175+drangara@users.noreply.github.com> Date: Mon Oct 2 13:12:50 2023 -0400 simplify how 2d surface integrals are computed (#3571) Previously, the 2D surface integrals were generated after calculating the intersection points of the EB on each of the cell faces. With this change, the intersection points are determined directly from `bcent`, `bnorm` and `barea`. This approach seems to simplify the need to handle special cases. Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_eb.cpp commit 8515ea846f85656c26d842401af1b29958de7c17 Author: Akash Dhruv Date: Sun Oct 1 13:34:10 2023 -0500 Adding bittree interface to improve regridding performance in octree mode (#3555) ## Summary This PR introduces dependency on [Bittree library](https://github.com/Flash-X/Bittree) to improve regridding performance in octree mode. ## Additional background Testing and development of this feature is done in sync with Flash-X and is recorded in this [reproducibility capsule](https://github.com/Lab-Notebooks/AMReX-Bittree-Performance). This PR is primarily created to consolidate development work and avoid creating multiple branches. At present using AMReX+Bittree improves regridding performance [by a factor of 2](https://github.com/Lab-Notebooks/AMReX-Bittree-Performance/blob/14faa2212c4e5dba7fd99a6526c6937414f9c109/analysis/Performance.ipynb) at > 20000 ranks. We hope to improve performance further using a bittree-based distribution mapping and therefore adding a new function `AmrMesh::MakeDistributionMap` as a place-holder. Continuation of #2893 and #3547 ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Weiqun Zhang Co-authored-by: Tom Klosterman GNUmakefile.in Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/Make.package Src/Extern/Bittree/AMReX_Bittree.H Src/Extern/Bittree/AMReX_Bittree.cpp Src/Extern/Bittree/CMakeLists.txt Src/Extern/Bittree/Make.package Tests/Amr/Advection_AmrCore/Exec/GNUmakefile Tests/Amr/Advection_AmrCore/Exec/inputs_bittree Tools/GNUMake/Make.defs Tools/GNUMake/packages/Make.bittree Tools/libamrex/configure.py commit 703a2041f758978bcb4894932265ab2f22812247 Author: Weiqun Zhang Date: Sun Oct 1 11:30:06 2023 -0700 MLNodeABecLaplacian (#3559) Add a new Linear Opearator class MLNodeABecLaplacian. It solves `(alpha * a - beta * (del dot b grad)) phi = rhs`, where a, phi and rhs are nodal MultiFabs and b is cell-centered. It works in 2D and 3D, CPU and GPU. So far it works only for single level. There is an example of using it at Tests/LinearSolvers/ABecLaplacian_C as prob_type=4. The test problem converges at the second order. --------- Co-authored-by: Andy Nonaka Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLNodeABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/Make.package Tests/LinearSolvers/ABecLaplacian_C/MyTest.H Tests/LinearSolvers/ABecLaplacian_C/MyTest.cpp Tests/LinearSolvers/ABecLaplacian_C/MyTestPlotfile.cpp Tests/LinearSolvers/ABecLaplacian_C/initProb.cpp Tests/LinearSolvers/ABecLaplacian_C/initProb_K.H Tests/LinearSolvers/ABecLaplacian_C/inputs-node commit 65e7a1ce5e959668542af694b6e4d838ee95d15a Author: Weiqun Zhang Date: Sun Oct 1 11:28:49 2023 -0700 Fix Boundary Centroid in a Corner Case in 2D (#3568) Due to roundoff errors, It's possible to have a cell with area fraction of 1-epsilon, 1, 1, and 1 at its four faces. In this corner case, the previous way of computing boundary centroid was not correct. In this PR, it's fixed by using the level set. Src/EB/AMReX_EB2_2D_C.cpp commit 388738dc788cc4a72c33a5d5d9940ef3a37a76b7 Author: Weiqun Zhang Date: Sun Oct 1 11:03:12 2023 -0700 Update CHANGES for 23.10 (#3576) CHANGES commit b95482d3b2ad6cab8318c867eb3b71b4f9ee11ac Author: Weiqun Zhang Date: Fri Sep 29 17:23:48 2023 -0700 Free Disk Space in Github CIs (#3574) Add a new script that removes some packages we do not need. It can free up about 7 GB of space on Ubuntu. Use it in nvhpc and sycl/amd CIs. These CIs often run out of space. We don't want to use it in every CI, because it takes a few minutes to run the script. .github/workflows/cuda.yml .github/workflows/dependencies/ubuntu_free_disk_space.sh .github/workflows/intel.yml commit 89352170b8a993f7d9e67bf2369da541c8502470 Author: Weiqun Zhang Date: Fri Sep 29 17:12:07 2023 -0700 Documentation: Fix default values (#3562) Boolean flag `amrex.use_gpu_aware_mpi` has `int` as its C++ type. So its default value should be 0 not False. For some other boolean flags with `bool` as their C++ type, the default value should be `false`, not `False`. Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/InputsPlotFiles.rst Docs/sphinx_documentation/source/LinearSolvers.rst Docs/sphinx_documentation/source/Particle.rst commit 71e9f93d3b7d23ec7b4792513a3b1ca1ce6c41c3 Author: prakash <120606615+moprak-nrel@users.noreply.github.com> Date: Fri Sep 29 10:39:24 2023 -0600 bugfix typo in AMReX_SundialsIntegrator.H Nvar vs NVar, the declared/used variable is NVar (#3573) Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H commit 6c8122d437f9210e05857e4590717de1e87ad50e Author: Weiqun Zhang Date: Thu Sep 28 08:39:57 2023 -0700 Code Spell (#3563) Add a github action and fix various spellings. --------- Co-authored-by: Jean M. Sexton .codespell-ignore-words .codespellrc .github/workflows/cleanup-cache-postpr.yml .github/workflows/cleanup-cache.yml .github/workflows/codespell.yml Docs/sphinx_documentation/source/AmrCore.rst Src/AmrCore/AMReX_FillPatcher.H Src/AmrCore/AMReX_FluxRegister.H Src/Base/AMReX.cpp Src/Base/AMReX_ANSIEscCode.H Src/Base/AMReX_Arena.H Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CTOParallelForImpl.H Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_IntVect.H Src/Base/AMReX_MultiFabUtilI.H Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParReduce.H Src/Base/AMReX_Utility.cpp Src/Base/AMReX_VisMF.H Src/Boundary/AMReX_InterpBndryData.H Src/Boundary/AMReX_YAFluxRegister.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_utils.H Src/Extern/HDF5/AMReX_ParticlesHDF5.H Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrMeshParticleInSituBridge.H Src/Extern/SUNDIALS/AMReX_Sundials_Core.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainer.H Tests/FortranInterface/Advection_F/Source/my_amr_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/my_amr_mod.F90 Tests/GPU/Vector/main.cpp Tests/HDF5Benchmark/sz.config Tests/LinearSolvers/Hypre/MyTest.cpp Tests/Parser2/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/C_scripts/mmclt.py Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/comps/hip.mak commit d45933c3616d1306a287d8991b8507a3be0e511e Author: Andrew Myers Date: Wed Sep 27 18:42:37 2023 -0700 Revert #3565 (#3567) This needs to be fixed on the FLASH-X side instead. Src/F_Interfaces/Particle/AMReX_particlecontainer_fi.cpp commit 1ac263f708beab3ad6ff77d7b2f5f6d5483bb685 Author: Andrew Myers Date: Wed Sep 27 14:32:09 2023 -0700 Fix regression in Fortran Interfaces for Particles (#3565) The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/F_Interfaces/Particle/AMReX_particlecontainer_fi.cpp commit f08b40b34bd466c9c9177f0707665af271c9770e Author: Weiqun Zhang Date: Mon Sep 25 13:47:09 2023 -0700 Add Fortran interface for average_down_faces (#3553) Src/Base/AMReX_MultiFabUtil.cpp Src/F_Interfaces/Base/AMReX_multifabutil_fi.cpp Src/F_Interfaces/Base/AMReX_multifabutil_mod.F90 commit 2e99628138df3b5b0ecf50b0c1201d5547f821a0 Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Fri Sep 22 22:01:27 2023 +0200 PureSoA: Disable AoS Access (#3290) ## Summary When using a PureSoA particle container, trying to access the AoS components will now result in a compile-time error. Previously it still compiled, and I suspect it even allocated memory for AoS. New error message: ``` /hipace/src/Hipace.cpp:1499:39: error: ‘struct amrex::ThisParticleTileHasNoAoS’ has no member named ‘begin’ 1499 | const auto& pos_structs = aos.begin() + nreal; | ^~~~~ ``` Using this I found more places that needed to be updated to account for PureSoA. PureSoA still has the following issues: - [x] I believe it’s not possible to use the binning functions (in `DenseBins.H`) as they require a particle pointer. - [ ] The `ParticleTileData` struct that is used to access SoA data can be very large, over 300 bytes in some cases. I recommend changing the array-of-pointers to a full 2D array. This way ParticleTileData could be as small as 24 bytes (real pointer, int pointer, stride). ## Additional background Follow-up to #2878. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Axel Huebl Src/Extern/SENSEI/AMReX_ParticleDataAdaptorI.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleTile.H Tests/Particles/RedistributeSOA/main.cpp Tests/Particles/SENSEI_Insitu_SOA/main.cpp commit b2052f2d2e5ce44317450ba13de705a3e01ef0ea Author: Andrew Myers Date: Mon Sep 18 17:54:35 2023 -0700 Another terrain fix for MPI (#3557) The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleContainerI.H commit cbf064a37163c1c7b58f999df828ada4b65278a7 Author: Andrew Myers Date: Mon Sep 18 16:03:51 2023 -0700 Fix Increment, OK, and EnforcePeriodicWhere for terrain-fitted particles. (#3556) Follow-on to #3499. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Co-authored-by: Ann Almgren Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H commit e9f443528fae75ccb971eba313e244e7dde85062 Author: ndeak Date: Mon Sep 18 13:36:55 2023 -0600 Added cvode functionality to SUNDIALS integrator (#3436) ## Summary ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Nicholas Deak Co-authored-by: Ann Almgren Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H commit ad2a89a955bdcdb3d9f57b67f9cc04e4db3ce16b Author: Axel Huebl Date: Mon Sep 18 11:05:54 2023 -0700 ParmParse::addfile needs Init (#3440) ## Summary `ParmParse` can be used before AMReX is initialized for most functionality, which is greatly helpful for us. One function that cannot be used before init is reading from a file, because it needs an MPI context to broadcast. Add a clean error message instead of a segfault. ## Additional background cc @ezoni @RevathiJambunathan ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_ParmParse.cpp commit 69b019f35c59255f9ce3b51ed41d672552df7ddf Author: Ann Almgren Date: Mon Sep 18 10:25:08 2023 -0700 make the same changes to ApplyInitialRedistribution as to ApplyMLRedi… (#3554) …stribution Src/EB/AMReX_EB_RedistributionApply.cpp commit 48b3ec7cb7ad99823bd85fad83c13c3cfd5ecdd4 Author: Weiqun Zhang Date: Sun Sep 17 16:39:22 2023 -0700 Reset EB Fab Type (#3552) This is a follow-up on #3549. After extending EB data outside the domain, we need to update the EB Fab type. Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBCellFlag.cpp Src/EB/AMReX_EBDataCollection.cpp commit 1aea95c021a8cbf0cab5e9e81f02a1b6e66249b5 Author: Ann Almgren Date: Sun Sep 17 10:55:29 2023 -0700 We weren't defining cent_hat out far enough to allow us to use ith wh… (#3548) …en using the grown stencil for slopes inside the state redistribution call; this commit fixes the size of the arrays and bounds of the loops to calculate what we need Src/EB/AMReX_EB_RedistributionApply.cpp Src/EB/AMReX_EB_StateRedistUtils.cpp commit 9c75779aaa023779feb040da6d63302acbc53968 Author: Weiqun Zhang Date: Sun Sep 17 10:52:16 2023 -0700 Fix #3549 (#3551) Area fractions are stored in MultiCutFabs that we must check if a fab is OK in MFIter for loop. Additionally, we could skip the call to extendDataOutsideDomain if the EB is all regular. Src/EB/AMReX_EBDataCollection.cpp commit dda52685d294bd4b99a8f8ff854178a1b0dcbbcd Author: Weiqun Zhang Date: Sat Sep 16 21:23:11 2023 -0700 EB Data outside domain (#3549) We do not guarantee there is valid EB information outside the domain, unless the user specifies a large enough number of ghost cells when building EB. However, this makes things difficult for application codes with AMR, because the number of ghost cells needed depends on the number of AMR levels. In this commit, we try to fill some basic EB information (EBCellFlag, volume fraction and area fraction) for all cells outside the domain. Co-authored-by: Ann Almgren Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EBDataCollection.cpp commit e06025b7913a13840243ac2268d2da77e9ddc2dc Author: Weiqun Zhang Date: Sat Sep 16 16:27:02 2023 -0700 Add Fortran inteface for FillCoarsePatch for face variables (#3542) ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/F_Interfaces/AmrCore/AMReX_fillpatch_fi.cpp Src/F_Interfaces/AmrCore/AMReX_fillpatch_mod.F90 commit 64d04be781dc335148d20a6a4e2cfeb7b23c2f0b Author: Weiqun Zhang Date: Sat Sep 16 16:25:04 2023 -0700 print_state/printCell: Make it work without managed memory (#3543) Src/Base/AMReX_FabArrayUtility.H commit 6eb91be02036c8fd4879e26af7c8098edeb9b4c7 Author: Weiqun Zhang Date: Fri Sep 15 10:12:58 2023 -0700 FillPatch Fortran Interface: Fix incorrect size of Vector (#3546) This fixes a very old bug. Src/F_Interfaces/AmrCore/AMReX_fillpatch_fi.cpp commit a8192ec9d8b86e32641163e028b616aec3e0ef0b Author: Depei Song <39750942+FR13ndSDP@users.noreply.github.com> Date: Fri Sep 15 22:48:27 2023 +0800 Eliminate warning from make: unused variable (#3544) Src/Base/AMReX_NFiles.cpp commit 078905a16294df1c97b540d111ac82c43c885bef Author: Weiqun Zhang Date: Thu Sep 14 20:48:51 2023 -0700 ReduceOps: reset result readiness flag (#3545) In #3421, a change was made to avoid a common mistake of calling ReduceData::value multiple times. Unfortunately, it breaks codes that using one ReduceOps with multiple ReduceData objects. This PR fixes both cases. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_Reduce.H commit 582929664d74969183c6339f376f9ef6d7b41f7d Author: Weiqun Zhang Date: Thu Sep 14 08:54:15 2023 -0700 Fix Fortran interface for FillPatch for face variables (#3541) ## Summary The stride was wrong when copying MultiFab pointers from a Fortran array to a Vector of Array of MultiFab pointers. This also fixes cases when the user passes t_new followed by t_old, instead of the other order. The issue was then teps was negative. This was a minor issue, because the FillPatch function in C++ does not care about the order. ## Additional background https://github.com/AMReX-Codes/amrex/issues/327#issuecomment-1716356603 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/F_Interfaces/AmrCore/AMReX_fillpatch_fi.cpp Src/F_Interfaces/AmrCore/AMReX_fillpatch_mod.F90 commit 4c79a836ae02a006ecb9767679baab340fbcec37 Author: philip-blakely <46958218+philip-blakely@users.noreply.github.com> Date: Wed Sep 13 16:59:23 2023 +0100 Support multiple CUDA architectures at compilation (#3535) ## Summary This allows users to specify multiple (space-separated) values for CUDA_ARCH in Makefiles. This may be useful for users on heterogeneous networks who want to have a single executable that can run on multiple CUDA hardware types. ## Additional background The changes here are done for Makefiles only, and have only been tested fully for the nvcc compiler. The correct compiler-command syntax has also been implemented for PGI and NVHPC compilers, but not tested (beyond manual checking of the command-line) because I don't have access to these. One CI test is updated to compile for two different CUDA architectures simultaneously and passes. Trying this in another case resulted in a too-large executable for the available disk. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate .github/workflows/cuda.yml Tools/GNUMake/comps/ibm.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/comps/nvhpc.mak Tools/GNUMake/comps/pgi.mak commit ae65b3afc481d497ff095cb90a2ac5cd70e1d6a9 Author: Weiqun Zhang Date: Tue Sep 12 09:35:52 2023 -0700 Fix the author list in CITATION (#3538) CITATION commit b98bdae9fb67e5d9aafc488de92c53001bd323ec Author: Candace Gilet Date: Thu Sep 7 14:40:32 2023 -0400 Add some Doxygen comments for HyperSolver (#3534) Src/Extern/HYPRE/AMReX_HypreSolver.H Tests/LinearSolvers/Hypre/MyTest.cpp commit f3598d2f7b6beca566e19d918b6e9976f8ed5a88 Author: Jon Rood Date: Tue Sep 5 22:18:32 2023 -0600 Add Kestrel machine and remove Rhodes machine from Make.nrel (#3533) Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.nrel commit dd3bcb841b282642cdb6a44bfa48edd8d8badbc8 Author: Eric T. Johnson Date: Tue Sep 5 23:54:42 2023 -0400 Explicitly flush when writing to the terse run log (#3532) ## Summary The log files are block-buffered by default, so the (very small) writes to the terse run log were only getting flushed to disk every couple hundred lines or so. ## Additional background The standard run log gets flushed at the end of `printGridInfo()` whenever there's a regrid (it also prints a lot more data, so it fills the buffer much faster). Src/Amr/AMReX_Amr.cpp commit 30fc3f050d73a9204409bea08c61275a07178bb4 Author: Weiqun Zhang Date: Tue Sep 5 14:13:49 2023 -0700 Missing header in AMReX_GpuComplex.H (#3531) amrex::abs is used in AMReX_GpuComplex.H, so we need to include AMReX_Algorithm.H. Src/Base/AMReX_GpuComplex.H commit 36dac3aba5911e751e8634d2b79990491025d920 Author: Nicole Date: Tue Sep 5 12:09:40 2023 -0600 Add global domain id offset to conduit wrapper (#3524) ## Summary Make sure all domains have unique domain ids by adding a global domain offset ## Additional background Currently meshes and particles will have non unique domain ids. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp Src/Extern/Conduit/AMReX_Conduit_Blueprint_ParticlesI.H commit 9a85e502923e4b3bf6dfa12fcb335c6786784154 Author: Weiqun Zhang Date: Fri Sep 1 13:57:41 2023 -0700 Update CHANGES for 23.09 (#3529) CHANGES commit b7e8a262dd9b4b8aa02b0ac5963e0644c33754da Author: Axel Huebl Date: Fri Sep 1 11:01:58 2023 -0700 Fix InitRandomPerBox for 1D & 2D (#3527) ## Summary Fixing an array bounds violation for `RealBox` in <3D for `InitRandomPerBox`. RealBox: SPACE include added for nD macros used within it. ## Additional background Seen in pyAMReX 1D and 2D builds. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_RealBox.H Src/Particle/AMReX_ParticleInit.H commit 6d862f29c93ca1f7246f7feecac6770f5d3e2356 Author: Jon Rood Date: Fri Sep 1 11:26:45 2023 -0600 Add ability for GCC 8 in CMake to build fgradient which uses std::filesystem (#3523) ## Summary Since GCC 8 is still a base compiler on several Linux distributions, I think it should probably continue to be supported. This allows `fgradient` to link to the std::filesystem library necessary for GCC < 9 and succeed in building. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Co-authored-by: Ann Almgren Tools/CMake/FindFilesystem.cmake Tools/Plotfile/CMakeLists.txt commit 35068ea42c85571e30be02dc6ddff47e848c9189 Author: Axel Huebl Date: Fri Sep 1 09:39:56 2023 -0700 Fix: Include Guard 1D for MLPoisson (#3528) ## Summary ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H commit c6fb58635a4ea28eebaec78873ed68244ec3cf66 Author: Weiqun Zhang Date: Fri Sep 1 09:38:38 2023 -0700 Workaround for Intel compiler classic (#3526) It seems that they have issues with constexpr std::string_view. https://github.com/xsdk-project/xsdk-issues/issues/221 Src/Base/Parser/AMReX_Parser_Y.H commit de7c6189623bc3ba178a380e703e3173f33f78ef Author: Axel Huebl Date: Wed Aug 30 10:01:15 2023 -0700 Constness in buildInfo: num_modules (#3522) Clang-Tidy: ``` warning: variable 'num_modules' of type 'int' can be declared 'const' [misc-const-correctness] ``` Co-authored-by: Weiqun Zhang Tools/CMake/AMReXBuildInfo.cmake Tools/CMake/AMReX_buildInfo.cpp.in Tools/C_scripts/makebuildinfo_C.py commit 5975a2e83efdc69356de0d6159646269b8cf9a0c Author: Weiqun Zhang Date: Wed Aug 30 08:57:21 2023 -0700 MLABecLaplacian: Tweak kernel fusing (#3521) Always fuse the kernels in MLABecLaplacian instead of relying on the heuristics in FabArrayBase::isFusingCandidate. This improves the performance on GPUs because it reduces the register pressure. Because these kernels are called many times during MLMG::solve, the one-time cost of copying the metadata to GPU device memory is amortized. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H commit 22908d8ad2141ed096401e9be9f067e8e589ec31 Author: Weiqun Zhang Date: Mon Aug 28 16:11:27 2023 -0700 UniqueString tweak (#3520) # Summary Previously UniqueString used MPI_Wtime, which could have a very low resolution (1e-5s) on some machines. In this PR, we switch to use amrex::second(), which uses the highest resolution steady clock (usually 1.e-9s). We have also changed a hardwired number according to resolution of the clock so that the digits are not wasted on trailing zeros. ## Additional background https://github.com/ECP-WarpX/impactx/pull/424 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_Utility.cpp commit f7c098609585162f0fd575bf91b4390bf675a041 Author: Simon Guichandut <40605210+simonguichandut@users.noreply.github.com> Date: Mon Aug 28 12:21:29 2023 -0400 faverage fix (#3519) ## Summary Make it so that Tools/Plotfile/faverage.cpp only *requires* density to be in the plotfile if the user wants to do a weighted average (-f option). Also cleans up the help message a bit. ## Additional background In the new [amrex-astro-diag](https://github.com/AMReX-Astro/amrex-astro-diag) set of tools, we generate new plotfiles without density, but would still like to be able to average. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Tools/Plotfile/faverage.cpp commit c45770c9f9b2c5fa98c675a439c502e78912bf47 Author: Simon Wilkinson Date: Sat Aug 26 21:53:20 2023 +0100 Empty level fix (#3517) ## Summary Closes #3516 . This works for me in practice, and is more convenient than filtering the levels on the caller's side. ## Additional background Idea is to simply not consider the empty levels in the linear solver. Src/LinearSolvers/MLMG/AMReX_MLLinOp.H commit d4761e89240203bfe385252c598967a14d4a06db Author: Axel Huebl Date: Fri Aug 25 17:47:36 2023 -0700 Simplify filterParticles Kernel (#3510) ## Summary On Summit, generation of this kernel shows compiler issues with `nvcc` 11.3.109: it compiles without warnings but leads to a segmentation fault at runtime. The fix for the compiler bug is to implement the trivial lambda that is passed to `copyParticles` w/o capture: `[]`. We also add explicit capture to a few other lambdas, to simplify compiler intake complexity. This simplifies the kernel generation, which also solves the issue seen with WarpX [for this line](https://github.com/ECP-WarpX/WarpX/blob/43d2ac7b54546c87d2cb540df7a2b3cb57592b84/Source/Diagnostics/WarpXOpenPMD.cpp#L585). Co-authored-by: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTransformation.H commit a45eadea6a08159a426c342d71472fb589a0fbc3 Author: Weiqun Zhang Date: Fri Aug 25 12:23:02 2023 -0700 ParticleLocator: Make Assignor optional template parameter (#3515) So that it is backward compatible. Follow-up to #3499. Src/Particle/AMReX_ParticleLocator.H commit bad61aca0ad852ca6e4fdf075d6f59b4869f2d8c Author: Andrew Myers Date: Fri Aug 25 23:37:21 2023 +0900 Follow-on to 3499 (#3514) A few more places needed to be generalized for terrain-fitted particles in ERF. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_ParticleContainer.H commit de86b7d2224ed8117cf4f585f735a8e00572abf1 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Thu Aug 24 16:38:16 2023 -0700 GPU Launch Min Blocks (#3503) Adding another launch configuration which specifies the minimum number of coresident blocks. This has shown good performance advantages for some Pele kernels. Co-authored-by: Paul Mullowney Src/Base/AMReX_GpuLaunchGlobal.H commit 0c9b1f00f2e053d26a498e5f5a4d393d4e592dac Author: Weiqun Zhang Date: Thu Aug 24 16:37:37 2023 -0700 Remove unsafe std::forward (#3513) These are unsafe because they are used inside a loop. Src/AmrCore/AMReX_AmrParticles.H Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleArray.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/Particles/ParticleTransformations/main.cpp commit f515cfde8fd215386d764cb24060746a6e9cebd8 Author: Andrew Myers Date: Fri Aug 25 08:34:00 2023 +0900 Generatize particle-to-cell assignment function (#3499) This is to support terrain-fitted coordinates in ERF. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Ann Almgren Src/AmrCore/AMReX_AmrParticles.H Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleUtil.H commit d4d0e9031d036070fad6d1d3bc96a8a32b48b3a9 Author: Weiqun Zhang Date: Tue Aug 22 20:16:22 2023 -0700 Fix Bug in FaceLinear Interpolater (#3483) The bug was introduced in #2539. The issue was when FaceLinear was used by functions other than FillPatchTwoLevels, the assumption of the fine box was a refined version of the coarse box could be wrong. The bug did not affect FillPatchFromTwoLevels. Co-authored-by: Ann Almgren Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.cpp commit 84bd01575dfca776f93e963d9008c1bab97d11e4 Author: Ann Almgren Date: Tue Aug 22 16:33:36 2023 -0700 Add test on whether we are in the valid box when updating drho_as_crs… (#3506) …e in StateRedistribute -- this is analogous to the test on 'inside' in FluxRedistribute Src/EB/AMReX_EB_StateRedistribute.cpp commit 0f59c07949e15659c8d1d27facdfd7aea0c3e590 Author: Axel Huebl Date: Tue Aug 22 15:40:05 2023 -0700 Fix backtrace nullptrs (#3505) Follow-up to #3502. - [x] now tested :) Src/Base/AMReX_BLBackTrace.cpp commit 68c49cc1293dc1ab7feca6da9969ceb40656d825 Author: Marc T. Henry de Frahan Date: Tue Aug 22 16:39:34 2023 -0600 Skip ubsan lib on OSX (#3504) I get a linking error with `ld: library not found for -lubsan` when `FSANITIZER = TRUE` when building on OSX with llvm. This PR fixes that error. Tools/GNUMake/comps/llvm.mak commit 5fabca4afb1e9fce3890116d6cb3a766aa26c1bb Author: Miren Radia <32646026+mirenradia@users.noreply.github.com> Date: Tue Aug 22 19:13:15 2023 +0100 SYCL: Add table of GNU Make config vars to docs (#3400) This PR adds a table of GNU make config vars to the docs. Docs/sphinx_documentation/source/GPU.rst commit e63308431a58cebd1aabd279cd12115089667113 Author: Axel Huebl Date: Mon Aug 21 18:36:42 2023 -0600 Fix: Backtraces will NULL Addresses (#3502) ## Summary The handing of invalid stack addresses in backtraces was incomplete, failing to run for stack addresses down a nullptr address with the error: ``` sh: -c: line 0: syntax error near unexpected token `(' sh: -c: line 0: `/usr/bin/addr2line -Cpfie /gpfs/alpine/aph114/proj-shared/huebl/2023-08-21_ms-debugging-rz/002_btd_patch_pr4208/./warpx.rz (nil)' ``` - [x] seen on Summit (OLCF) - [ ] tested to fix the issue ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_BLBackTrace.cpp commit 07c433264ab2ca2a357701e9d107222e90506625 Author: Christoph Bauinger Date: Mon Aug 21 20:52:39 2023 +0200 Update gpu elixir and async array synchronization for sycl (#3498) Updated the way the GpuElixir and GpuAsyncArray synchronize. Instead of a Gpu::streamSynchronize(); it now uses a host task by default since it was implemented in SYCL 2020. Removed the codeplay-extension version. In addition, the option SYCL_PARALLEL_LINK_JOBS was added to dpcpp.mak. Setting it to a positive integer then allows linking the SYCL code with threads, speeding up the whole compilation process, especially when using AOT in SYCL. Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuElixir.cpp Tools/GNUMake/comps/dpcpp.mak commit dda0d979ca2ec912f7dd2eb630399f33ee7990e0 Author: Weiqun Zhang Date: Sat Aug 19 12:36:40 2023 -0700 SundialsIntegrator: Fix memory deallocation (#3501) ## Summary nv_many_array is allocated with new[]. So it must be deallocated with delete[]. Therefore we cannot call Sundials's N_VDestroyVectorArray that uses std::free. ## Additional background https://github.com/AMReX-Codes/amrex/pull/3436#issuecomment-1646938731 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H commit a4fb1d4a88cc4ae97cc1e0d30b6bee6f20353741 Author: Weiqun Zhang Date: Fri Aug 18 16:25:16 2023 -0700 FillRandom & FillRandomNormal (#3500) Add new functions that fill a MultiFab or a 1d vector with random numbers from a uniform or normal distribution. The functions use CUDA and HIP's host APIs for random number generations. Compared with our existing capability of using device APIs, the new functions are slightly (about 15%) faster on Nvidia GPUs, whereas they are 4x faster on the AMD GPU we tested. For Intel GPUs, we are still using the device APIs, because we have not figured out how to use its host APIs. More specially, we are missing some LDFLAGS at link time. The functions work for CPU builds too, and for normal distribution they are about 2x faster than using `amrex::RandomNormal` because the latter only uses one of two random numbers generated by the Box-Muller transform. The functions work for MultiFabs with any index types (e.g., nodal). However, if the numbers on shared nodes need to be consistent across boxes and MPI processes, it's the user's responsibility to call OverrideSync. (This gives the user a chance to cache the mask used by OverrideSync.) It's also the user's responsibility to fix data consistency between overlapping ghost and valid points, if that is needed. Src/Base/AMReX_GpuError.H Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_Random.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_RandomEngine.H Tools/GNUMake/Make.defs commit 435281069858cb8df4ef5e7f291dde7cc5d7c019 Author: Donald E. Willcox Date: Wed Aug 16 11:41:12 2023 -0700 Update position in 2nd pass for tracer advect with Ucc. (#3496) ## Summary In the Ucc advection routine for tracer particles, the 2nd advection pass did not increment the particle position. This change fixes the issue, and it is now consistent with the leapfrog approach used in the Umac advection routine. ## Additional background ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_TracerParticles.cpp commit 3396b1df117532ec9a7eafdcaad4bdd18ac49a0f Author: Ann Almgren Date: Mon Aug 14 12:37:03 2023 -0700 Move use of bcrec back onto GPU (#3494) This fixes a bug introduced in the last PR when running on GPUs Src/EB/AMReX_EB_StateRedistribute.cpp commit 3291c074b67ad2c0fbe223389644bcd0e4db70d7 Author: Ann Almgren Date: Sun Aug 13 15:17:23 2023 -0700 fix bug in StateRedist and add multi-level functionality for FluxRedi… (#3491) …st and StateRedist This PR has two purposes. The first is to fix a bug in StateRedistribution which occurred when the EB was near a fine-fine boundary. In this case we weren't calculating certain arrays in a wide enough band of ghost cells. This has been fixed by enlarging two of the arrays and calculating them on the correct number of ghost cells. The second aspect of this PR is to make available the multi-level StateRedistribution and FluxRedistribution algorithms. Using these will require application codes to make changes in the calling routines. This PR does not break the way application codes currently call the single-level routines. Src/EB/AMReX_EB_Redistribution.H Src/EB/AMReX_EB_RedistributionApply.cpp Src/EB/AMReX_EB_StateRedistUtils.cpp Src/EB/AMReX_EB_StateRedistribute.cpp commit 97bd0892b32e04b62dfe16dc452626bffe5abbe2 Author: Weiqun Zhang Date: Sat Aug 12 14:26:16 2023 -0700 fgradient: A new plotfile tool for computing gradient (#3490) Docs/sphinx_documentation/source/Post_Processing.rst Src/Base/AMReX_PhysBCFunct.H Tools/Plotfile/CMakeLists.txt Tools/Plotfile/GNUmakefile Tools/Plotfile/fgradient.cpp commit 9bceb99ba8b0334ad7f6e55d68004b3f88ef49a8 Author: Nuno Nobre Date: Sat Aug 12 02:11:35 2023 +0100 CI: Remove redundancy in nvcc dependencies files (#3387) Hi @ax3l, @WeiqunZhang, I hope you're enjoying your time in Switzerland. :) I've removed some redundancy around the nvcc dependencies files, mostly for consistency with the changes I did to the hip dependencies file in #3341. It should all be pretty self-explanatory. Cheers, -Nuno .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_llvm_cuda11_clang15.sh .github/workflows/dependencies/dependencies_llvm_cuda11_ubuntu22.sh .github/workflows/dependencies/dependencies_nvcc.sh .github/workflows/dependencies/dependencies_nvcc12.sh .github/workflows/hypre.yml .github/workflows/intel.yml .github/workflows/sundials.yml commit a12d498a0c195e2f50427e90b7d07a32090c2d65 Author: Axel Huebl Date: Fri Aug 11 18:09:41 2023 -0700 CMake: HIP Debug with -O1 (#3487) ## Summary Same work-around as in GNUmake, see https://github.com/AMReX-Codes/amrex/pull/3311 - [ ] testing ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Tools/CMake/AMReXParallelBackends.cmake commit 2ed67c2a83105e7ed79067b89379b8bcdb91cb1e Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Sat Aug 12 01:19:23 2023 +0200 Fix overflow in ParallelFor(Long n, ...) (#3489) ## Summary When using the `void ParallelFor (T n, L&& f)` overload with T equal to `long long` or `unsigned long long`, an overflow could occur if `n` was larger than uint max. ## Additional background The overflow occurs in the Gpu part of the ParallelFor function itself when calculating `stride = blockDim.x*gridDim.x`. In this PR the execution config is changed to limit gridDim.x to prevent the overflow. ParallelFor will still iterate over all elements because the for loop inside the kernel will make a single thread iterate over multiple elements if gridDim.x needed to be reduced. Note: Some SYCL functions look to still be prone to overflow https://github.com/AMReX-Codes/amrex/blob/cc7a5c1171bb2634a9a71ce3c18cfd0be72f581c/Src/Base/AMReX_GpuLaunchFunctsG.H#L196 (should be at least unsigned int). Src/Base/AMReX_GpuLaunch.H commit 6535763b217858280296a306b81efe5bd50009a7 Author: Kevin Gott Date: Thu Aug 10 10:18:55 2023 -0700 Adjust build for upstream Mpich and OpenMPI modules on Perlmutter. (#3486) ## Summary Upstream MPICH and OpenMPI are available now on Perlmutter. To access them: module load mpich module load openmpi With openmpi, srun also requires the flag --mpi=pmix, or equivalent. (https://docs.nersc.gov/development/programming-models/mpi/openmpi/) With these adjustments, HeatEquation runs fine, albeit with some errors at start and/or end, which may be srun flag related. Tools/GNUMake/sites/Make.nersc commit d92232a0c17b846ac85851ca5c950d1cd916e3e3 Author: Weiqun Zhang Date: Tue Aug 8 09:44:15 2023 -0700 CArena: Implement alloc_in_place (#3426) First we try to see if CArena has free space that can be used to extend the given pointer to the specified size range. If that's unsuccessful, we use the normal alloc function to allocate memory. If in-place allocation is successful during vector resizing, we can avoid the cost of copying data to the new allocation address. Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp commit 346a4d1c915f2960cc0b0cbd2383cc341da4f56c Author: Weiqun Zhang Date: Mon Aug 7 18:18:40 2023 -0700 Fix a bug in #3466 (#3485) Src/Base/AMReX_Array.H commit fdce60c55e7e0eead0c93becc78b8c5d7437fd1a Author: Mahesh Natarajan Date: Mon Aug 7 12:03:48 2023 -0700 Option to force regrid level zero (#3484) ## Summary This PR gives the user an option to force regrid level zero. ## Additional background This is needed for moving EB cases. Src/Amr/AMReX_Amr.cpp commit afdf25d71f40ad112fb7a094161655f09c1357d1 Author: Weiqun Zhang Date: Mon Aug 7 12:02:38 2023 -0700 Clang-Tidy: Add more checks (#3466) Most of the changes were done by clang-tidy --fix for readability-braces-around-statements. .clang-tidy Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_Algorithm.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_Array.H Src/Base/AMReX_AsyncOut.cpp Src/Base/AMReX_BCUtil.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_Box.H Src/Base/AMReX_Box.cpp Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxDomain.cpp Src/Base/AMReX_BoxIterator.H Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_FabConv.cpp Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_GpuComplex.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_IndexType.cpp Src/Base/AMReX_IntConv.H Src/Base/AMReX_IntVect.cpp Src/Base/AMReX_LayoutData.H Src/Base/AMReX_Lazy.cpp Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_MPMD.H Src/Base/AMReX_Machine.cpp Src/Base/AMReX_Math.H Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MemProfiler.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtilI.H Src/Base/AMReX_NonLocalBC.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_Orientation.cpp Src/Base/AMReX_PArena.cpp Src/Base/AMReX_PCI.H Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_PhysBCFunct.H Src/Base/AMReX_PlotFileUtil.cpp Src/Base/AMReX_Random.cpp Src/Base/AMReX_RealBox.H Src/Base/AMReX_RealBox.cpp Src/Base/AMReX_RealVect.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H Src/Base/AMReX_TagParallelFor.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_TypeList.H Src/Base/AMReX_Utility.cpp Src/Base/AMReX_Vector.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_iMultiFab.cpp Src/Base/AMReX_parstream.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Tests/Amr/Advection_AmrCore/Source/Src_K/compute_flux_2D_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/compute_flux_3D_K.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob_Parm.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob_Parm.H Tests/Amr/Advection_AmrLevel/Source/Src_K/Adv_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/flux_2d_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/flux_3d_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/slope_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/tagging_K.H Tests/EB/CNS/Source/CNS.H commit e7d3d5a4fad6559ada6bfaa011438f9e7e317ca7 Author: Andrew Myers Date: Sun Aug 6 15:12:18 2023 -0700 Fix typo introduced in PR 3452 (#3482) Fixes new issue introduced in PR #3452. Bug reported by @hklion. Src/Particle/AMReX_ParticleMesh.H commit 7ec2e0b89352baf5c3fa8bfb644de64ab0a21e94 Author: Weiqun Zhang Date: Fri Aug 4 17:16:30 2023 -0700 Clang-Tidy changes in Src/LinearSolvers (#3475) Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_F.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp_bc.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_eb.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp commit 0dddf9e3074a9c43a5347e01b973060f224f5a6b Author: Ann Almgren Date: Fri Aug 4 14:12:05 2023 -0700 Update FluxRedist and comments for StateItracker (#3481) This updates the FluxRedistribution scheme to operate correctly when the EB surface intersects or is near a periodic boundary. Src/EB/AMReX_EB_FluxRedistribute.cpp Src/EB/AMReX_EB_StateRedistItracker.cpp commit 6e5cb926052fa8a1392f96a9b89b1069328e574c Author: Weiqun Zhang Date: Fri Aug 4 10:00:15 2023 -0700 Clang-Tidy changes in Src/EB (#3473) Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EBFluxRegister.cpp Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_EB_FluxRedistribute.cpp Src/EB/AMReX_EB_LeastSquares_2D_K.H Src/EB/AMReX_EB_LeastSquares_3D_K.H Src/EB/AMReX_EB_Redistribution.cpp Src/EB/AMReX_EB_RedistributionApply.cpp Src/EB/AMReX_EB_Slopes_2D_K.H Src/EB/AMReX_EB_Slopes_3D_K.H Src/EB/AMReX_EB_StateRedistItracker.cpp Src/EB/AMReX_EB_StateRedistSlopeLimiter_K.H Src/EB/AMReX_EB_StateRedistUtils.cpp Src/EB/AMReX_EB_StateRedistribute.cpp Src/EB/AMReX_EB_chkpt_file.cpp Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_WriteEBSurface.cpp Src/EB/AMReX_algoim.cpp Src/EB/AMReX_algoim_K.H commit 95ce6910beef3abe3e2766e2469551f197b5a253 Author: Weiqun Zhang Date: Fri Aug 4 09:37:00 2023 -0700 Clang-Tidy changes in Src/Boundary (#3471) Src/Boundary/AMReX_InterpBndryData.H Src/Boundary/AMReX_Mask.cpp Src/Boundary/AMReX_YAFluxRegister.H commit 559766e63c71ca1851c6b95b0451bf79dcd2bed8 Author: Weiqun Zhang Date: Fri Aug 4 09:36:35 2023 -0700 Clang-Tidy changes in Src/F_Interfaces (#3474) Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.cpp Src/F_Interfaces/AmrCore/AMReX_fluxregister_fi.cpp Src/F_Interfaces/Base/AMReX_distromap_fi.cpp Src/F_Interfaces/Base/AMReX_geometry_fi.cpp Src/F_Interfaces/LinearSolvers/AMReX_abeclaplacian_fi.cpp Src/F_Interfaces/LinearSolvers/AMReX_poisson_fi.cpp commit 933ec977634500ab9bf5e65e950b3df1fd80cba3 Author: Weiqun Zhang Date: Fri Aug 4 09:35:35 2023 -0700 Clang-Tidy changes in Src/Particle (#3476) Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleArray.H Src/Particle/AMReX_ParticleBufferMap.cpp Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleMPIUtil.cpp Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_ParticleUtil.cpp Src/Particle/AMReX_Particle_mod_K.H Src/Particle/AMReX_SparseBins.H Src/Particle/AMReX_StructOfArrays.H Src/Particle/AMReX_TracerParticles.cpp Src/Particle/AMReX_WriteBinaryParticleData.H commit c1ec94d852b57d1f727c3a3963a959e269ea718c Author: Weiqun Zhang Date: Fri Aug 4 09:23:33 2023 -0700 Clang-Tidy changes in Src/Extern (#3472) Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/Extern/PETSc/AMReX_PETSc.cpp Src/Extern/ProfParser/AMReX_ProfParserBatch.cpp Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H Src/Extern/SUNDIALS/AMReX_Sundials_Core.cpp commit 97ca4965b5379d3e132edb9df6a4b6e3fbbb412a Author: Weiqun Zhang Date: Fri Aug 4 09:07:00 2023 -0700 Clang-Tidy changes in Tests (#3477) Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tests/Amr/Advection_AmrCore/Source/DefineVelocity.cpp Tests/Amr/Advection_AmrCore/Source/Tagging.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/DivFreePatch/main.cpp Tests/EB/CNS/Source/CNS.cpp Tests/EB/CNS/Source/CNS_advance.cpp Tests/EB_CNS/Source/CNS.cpp Tests/GPU/CNS/Source/CNS.cpp Tests/LinearSolvers/CellEB/MyTest.cpp Tests/LinearSolvers/NodalPoisson/main.cpp Tests/LinearSolvers/Nodal_Projection_EB/main.cpp Tests/MultiBlock/IndexType/main.cpp Tests/Parser2/fn.cpp Tests/Particles/AssignDensity/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/AsyncIO/main.cpp Tests/Particles/CheckpointRestart/main.cpp Tests/Particles/CheckpointRestartSOA/main.cpp Tests/Particles/DenseBins/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/InitRandom/main.cpp Tests/Particles/Intersection/main.cpp Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/main.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleIterator/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/main.cpp Tests/Particles/ParticleReduce/main.cpp Tests/Particles/ParticleTransformations/main.cpp Tests/Particles/Redistribute/main.cpp Tests/Particles/RedistributeSOA/main.cpp Tests/Particles/SOAParticle/GNUmakefile Tests/Particles/SOAParticle/Make.package Tests/Particles/SOAParticle/main.cpp Tests/Particles/SparseBins/main.cpp commit 753b75e122c31a89fa34bf807f334f8433a16975 Author: Weiqun Zhang Date: Fri Aug 4 08:51:39 2023 -0700 Clang-Tidy changes in Tools (#3478) Tools/Plotfile/fcompare.cpp Tools/Plotfile/fextract.cpp Tools/Plotfile/fnan.cpp Tools/Plotfile/fsnapshot.cpp Tools/Postprocessing/C_Src/particle_compare.cpp commit a7f2cd9c90d81269de9a89c7b9dd8f6c4fff1fea Author: Weiqun Zhang Date: Thu Aug 3 22:53:15 2023 -0700 Change the default value of The_Pinned_Arena release threshold (#3479) The old default of the size of device memory is too big for some machines (e.g., Perlmutter 80 GB nodes). Some WarpX runs were out of memory during restart. It is now set to half of the device memory. Src/Base/AMReX_Arena.cpp commit 1f1a661440d47efdb51eb9735f8b50c0e0ce3b0c Author: Andrew Myers Date: Thu Aug 3 16:30:18 2023 -0700 Handle ThetaGPU in GNU make system (#3469) ThetaGPU gets incorrectly set to use the configuration for Theta, which does not work because the cray compiler wrappers are not available on ThetaGPU. However, Make.unknown works fine. So this PR reverts to using that on ThetaGPU. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Tools/GNUMake/Make.machines commit 1ce0303ea1e4827099c16669bc7087c3894dd467 Author: Weiqun Zhang Date: Thu Aug 3 13:37:21 2023 -0700 Clang-Tidy changes in Src/AmrCore (#3468) Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_AmrParticles.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil.cpp Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_TagBox.cpp commit bee6554ebb126682277f92a179ddb16c5b5600f8 Author: Weiqun Zhang Date: Thu Aug 3 13:02:36 2023 -0700 Clang-Tidy changes in Src/Amr (#3467) Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_AuxBoundaryData.H Src/Amr/AMReX_AuxBoundaryData.cpp Src/Amr/AMReX_Derive.cpp Src/Amr/AMReX_Extrapolater.cpp Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.cpp Src/Amr/AMReX_extrapolater_2D_K.H Src/Amr/AMReX_extrapolater_3D_K.H commit 3af462ce7c99a426b2c8f0952f2485d4fdde8b11 Author: Weiqun Zhang Date: Thu Aug 3 11:58:45 2023 -0700 Fix a bug in EB tensor solver's cross term (#3465) Found by clang-tidy misc-redundant-expression. Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H commit 67b8a5b04b7f2272ae9cc190052dcd0f1f9399d8 Author: Luca Fedeli Date: Wed Aug 2 17:04:52 2023 +0200 Add misc-misplaced-const check to clang-tidy CI test (#3464) This PR adds the [misc-misplaced-const](https://releases.llvm.org/14.0.0/tools/clang/tools/extra/docs/clang-tidy/checks/misc-misplaced-const.html) check to clang-tidy CI test. I am not sure if you want to enforce this in AMReX, but I find the constructs it protects against quite misleading. .clang-tidy commit 2de66d76894b121b8bb35627816f49499f24bf1a Author: Andrew Myers Date: Tue Aug 1 14:21:13 2023 -0700 Suppress implicit return warnings with nvcc (#3463) A bug in nvcc results in spurious warnings about implicit return statements when `if constexpr` is involved: https://stackoverflow.com/questions/64523302/cuda-missing-return-statement-at-end-of-non-void-function-in-constexpr-if-fun Suppress those for now. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReX_SetupCUDA.cmake Tools/GNUMake/comps/nvcc.mak commit 2736683ec201dcbc81ca310add721c949503d5c1 Author: Andrew Myers Date: Tue Aug 1 14:05:42 2023 -0700 Silence "unreachable loop" warning for GPU builds (#3462) This silences warnings like `Src/Base/AMReX_PODVector.H(126): warning #128-D: loop is not reachable` The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_PODVector.H commit ebe604a59eba07c15e3b3a667bf9da957aa5a55e Author: Axel Huebl Date: Tue Aug 1 10:30:59 2023 -0700 PlotFileUtil: Add Direct Includes (#3446) Adds the used direct includes to `AMReX_PlotFileUtil.H`. Src/Base/AMReX_PlotFileUtil.H commit b905668d2fef9cd595352aa26bafe794aae0088b Author: Weiqun Zhang Date: Tue Aug 1 08:28:54 2023 -0700 Update CHANGES for 23.08 (#3461) CHANGES commit 5141e75e5b29d5f42ca537103563aad3ee76ee70 Author: Weiqun Zhang Date: Mon Jul 31 23:44:54 2023 -0700 Fix CI Setup (#3456) Two CIs (one GCC and one Clang/CUDA) have been failing because of libunwind package conflict issues. This PR fixes them by (1) switching to clang-tidy-15 in the GCC test and (2) explicitly installing libunwind-15 and libunwind-15-dev in Clang test. .github/workflows/dependencies/dependencies_llvm_cuda11_ubuntu22.sh .github/workflows/gcc.yml Src/F_Interfaces/Base/AMReX_init_fi.cpp commit cf8d9f8e233be2ac68aeb65a95e992adfb483939 Author: Weiqun Zhang Date: Mon Jul 31 08:55:32 2023 -0700 Fix a bug in AmrMesh::ChopGrids (#3460) Fix a corner case for AmrMesh::ChopGrids when there is only one cell in one dimension. Close #3459 Src/AmrCore/AMReX_AmrMesh.cpp commit 98d22d21af45b15104647c143c8dad61f9693aae Author: Andrew Myers Date: Fri Jul 28 20:17:56 2023 -0700 Remove use of managed memory from TracerParticleContainer::TimeStamp (#3457) Src/Particle/AMReX_TracerParticles.cpp commit bba6d9c2bc68be493c8615f6a72a6d7fa857d604 Author: Grant Johnson <69021085+johnson452@users.noreply.github.com> Date: Thu Jul 27 20:47:26 2023 -0700 Mac Arm64 Architecture fpe handling (#3447) ## Summary Handles floating pointing exception when compiled on arch64 and Apple (for M1 and M2 chips). Only works for overflow currently, there is an apparent limitation to these architectures for catching division by zero. ## Additional background SIGILL instead of SIGFPE is needed to catch arch64 signals, see ref: https://developer.apple.com/forums/thread/689159 --------- Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/Debugging.rst Src/Base/AMReX.cpp Src/Base/AMReX_BLBackTrace.cpp Tools/CMake/AMReXFlagsTargets.cmake commit f41e619c29eccd752b05314f226bb0dd61baecec Author: Weiqun Zhang Date: Thu Jul 27 20:20:16 2023 -0700 SYCL: Remove Workaround no longer needed (#3455) Src/EB/AMReX_algoim_K.H commit d9df80bcc8630da349b15c567a02a6b03ed5a2c5 Author: Axel Huebl Date: Thu Jul 27 18:52:52 2023 -0700 SENSEI: SoA Particle Support (#3349) ## Summary Add support for pure SoA layouted particle containers for SENSEI. ## Additional background Follow-up to #2878. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Andrew Myers Src/Extern/SENSEI/AMReX_AmrMeshParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrMeshParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_AmrMeshParticleInSituBridge.H Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_AmrParticleInSituBridge.H Src/Extern/SENSEI/AMReX_ParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_ParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_ParticleInSituBridge.H Tests/CMakeLists.txt Tests/Particles/SENSEI_Insitu_SOA/CMakeLists.txt Tests/Particles/SENSEI_Insitu_SOA/GNUmakefile Tests/Particles/SENSEI_Insitu_SOA/inputs.rt Tests/Particles/SENSEI_Insitu_SOA/main.cpp commit 2628ec85f8004cb25469334637895172dddfb639 Author: Jon Rood Date: Thu Jul 27 17:54:45 2023 -0600 Warn if USE_GPU_RDC=TRUE when HIP_SAVE_TEMPS=TRUE (#3454) USE_GPU_RDC=FALSE is necessary to save the assembly files for AMD GPU kernels. Tools/GNUMake/Make.defs commit b459e58395d0fcc75e3753f6430fb686bc9a5985 Author: Weiqun Zhang Date: Thu Jul 27 16:01:20 2023 -0700 SYCL: Allocate 3/4 of total global memory by default (#3431) In the past, we had issues with allocating too big a chunk of memory on Intel GPUs, so we only allocated 1 GB by default. This is no longer the case. Src/Base/AMReX_Arena.cpp commit f5bf0e412cef9250c859ae115ce60dde7d0d178d Author: Weiqun Zhang Date: Thu Jul 27 15:51:36 2023 -0700 GPU Assignment (#3382) When there are multiple GPU devices visible to an MPI process, we need to make a decision on which device to use. In the current development branch, we create a local sub-communicator using something like MPI_COMM_TYPE_SHARED and assign the devices in a round-robin way based on the local rank. This however does not always work. In this WarpX issue (https://github.com/ECP-WarpX/WarpX/issues/3967), it was reported that all processes in a node used the same GPU because the sub-communicator had only one process in it. In this PR, we use MPI_Get_processor_name as an alternative way of obtaining the ranks within an actual node and use that for GPU assignment. If both approaches fail, we will fall back to use the rank in the global communicator modulo the number of devices. In this PR, we have also removed AMREX_GPUS_PER_SOCKET and AMREX_GPUS_PER_NODE. They have not been used for some time now. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config_ND.H.in Tools/GNUMake/Make.defs Tools/GNUMake/sites/Make.alcf Tools/GNUMake/sites/Make.llnl Tools/GNUMake/sites/Make.nersc Tools/GNUMake/sites/Make.nrel Tools/GNUMake/sites/Make.olcf commit 34c0ae3949688215a02c4ee36c427524c7863f5b Author: Andrew Myers Date: Thu Jul 27 09:13:54 2023 -0700 call_f refactor (#3452) Using `if constexpr` in these functions allows us to substantially reduce the number of overloads needed. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleUtil.H commit 49dd703558c51530f4e145e99d1640f32f68422b Author: Houjun Tang Date: Wed Jul 26 14:31:07 2023 -0700 HDF5 bug fix for vol-async and compression (#3451) ## Summary - Fix a couple of issues with vol-async and SZ compression, regular HDF5 I/O should not be affected. - Changed BL_USE_MPI to AMREX_USE_MPI. ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Tests/HDF5Benchmark/GNUmakefile commit b75dd9d5279263768c418ce19285f87de37209f9 Author: Andrew Myers Date: Tue Jul 25 22:09:52 2023 -0700 Add second template parameter to other AoS for swap (#3450) This is also needed for: https://github.com/ECP-WarpX/WarpX/actions/runs/5649976693/job/15305447602?pr=3474 https://github.com/ECP-WarpX/WarpX/actions/runs/5649976693/job/15305447804?pr=3474 Co-authored-by: Weiqun Zhang Src/Particle/AMReX_ArrayOfStructs.H commit 8aa7dc6ef40b272b31f1052b00ff3f1be53c848f Author: Andrew Myers Date: Tue Jul 25 21:47:01 2023 -0700 Refactor call_f in ParticleTransformation and WriteBinaryParticleData to use constexpr if (#3448) This results in much easier-to-interpret error messages, useful in finding #3449 and #3450 Co-authored-by: Weiqun Zhang Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_WriteBinaryParticleData.H commit a3b69fa32e519860228ce3afca10a9dc3a1dca9a Author: Andrew Myers Date: Tue Jul 25 20:02:27 2023 -0700 Fix this version of WritePlotFile to use SuperParticle, as expected by WriteBinaryParticleData (#3449) Fixes these issues seen in the WarpX CI: https://github.com/ECP-WarpX/WarpX/actions/runs/5649976693/job/15305447602?pr=3474 https://github.com/ECP-WarpX/WarpX/actions/runs/5649976693/job/15305447804?pr=3474 Src/Particle/AMReX_ParticleIO.H commit e0583fbd54ccaf3268d29fb7f4983244434fbdd1 Author: Axel Huebl Date: Mon Jul 24 20:45:43 2023 -0700 PODVector: Add a Missing STL Include (#3445) Used by `std::reverse_iterator`. Src/Base/AMReX_PODVector.H commit 5dfb0400581e0e2deb7bb0dc11b8f7efb14c8d17 Author: Weiqun Zhang Date: Mon Jul 24 20:43:04 2023 -0700 Disable Managed Memory for The_Arena by default. (#3438) It used to be that The_Arena was managed for CUDA and SYCL, but not for HIP. The users can still turn it on with `amrex.the_arena_is_managed=1`. They can also use The_Managed_Arena explicitly. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_Arena.cpp commit ed1b3a16b23f235f586987d985baf4f4c502df54 Author: Weiqun Zhang Date: Mon Jul 24 18:43:49 2023 -0700 SYCL RelWithDebInfo Build: Add -g1 (#3429) Add `-g1` to RelWithDebInfo build with oneAPI SYCL compiler (i.e., IntelLLVM). This is because with `-g` the link step can be very slow and the final executable can be order of magnitude bigger than that generated with `-g1`. This actually passes `-g -g1` to the compiler. The first flag `-g` comes from CMake with the RelWithDebInfo build type. Here, the order of the two flags matter, because the latter flag `-g1` overrides `-g`. Tools/CMake/AMReXFlagsTargets.cmake commit 14f698b7c772c895b7e2e871b794c1b9f87625f5 Author: Weiqun Zhang Date: Mon Jul 24 18:42:55 2023 -0700 Clang Tidy: +modernize-use-using (#3420) .clang-tidy Src/Amr/AMReX_Amr.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_Derive.H Src/Amr/AMReX_StateDescriptor.H Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_TagBox.H Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxDomain.H Src/Base/AMReX_BoxList.H Src/Base/AMReX_CArena.H Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FEIntegrator.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FileSystem.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_INT.H Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_Lazy.H Src/Base/AMReX_MFCopyDescriptor.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_PCI.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_PhysBCFunct.H Src/Base/AMReX_REAL.H Src/Base/AMReX_RKIntegrator.H Src/Base/AMReX_Scan.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_VisMFBuffer.H Src/Base/AMReX_ccse-mpi.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.H Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H Src/F_Interfaces/AmrCore/AMReX_fillpatch_fi.cpp Src/F_Interfaces/Base/AMReX_init_fi.cpp Src/LinearSolvers/OpenBC/AMReX_OpenBC.H Src/Particle/AMReX_ParticleArray.H Src/Particle/AMReX_ParticleContainer.H Tests/HDF5Benchmark/main.cpp Tests/Particles/AssignDensity/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/CheckpointRestart/main.cpp Tests/Particles/CheckpointRestartSOA/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/InitFromAscii/main.cpp Tests/Particles/InitRandom/main.cpp Tests/Particles/ParticleIterator/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/mypc.H commit 81103e064186785fa34a17d9cde42840b4d1347a Author: Weiqun Zhang Date: Mon Jul 24 11:49:14 2023 -0700 Intel CI: Update Warning Flags (#3443) No need for -Wno-missing-brace and -Wno-sign-compares anymore. These warnings in MKL have been fixed in oneAPI 2023.2.0. .github/workflows/intel.yml commit 0eb494d34427552ac934fa4c1b636503c8ab3a7a Author: Candace Gilet Date: Sun Jul 23 14:40:42 2023 -0400 Update inaccurate comment. (#3442) Src/Base/AMReX_MultiFabUtil.cpp commit 6faf7ee37ca5899bcb184cbabb1368abf877cc86 Author: Candace Gilet Date: Sun Jul 23 14:40:09 2023 -0400 In MLEBABecLap::compGrad, add missing call to addInhomogNeumannFlux (#3441) For inhomogeneous Neumann BC, we convert the inhomogeneous part into RHS and treat the BC as homogeneous Neumann during the solve. Thus, the flux is zero there before inhomogeneous flux is added. Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp commit e41dbfa9bd5ef476e9c5393d2f2f0010ea7accce Author: Axel Huebl Date: Fri Jul 21 15:39:12 2023 -0700 Doxygen: SoA Get[Real/Int]Data (#3439) ## Summary There are subtle differences with respect to access to runtime-added SoA attributes in the overloads of these functions - adding a doc string as in pyAMReX to clarify for users. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [x] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_StructOfArrays.H commit 587542a395e4bd1991c3e731b289ca89fe43746e Author: Weiqun Zhang Date: Thu Jul 20 20:21:17 2023 -0700 ParmParse: Line Continuation (#3437) In #3433, we added an option to disable multi-line support to avoid certain types of errors. Since multi-line support seems to be desirable feature, we add a new feature that uses `\` as line continuation. To avoid confusion, we have decided to stop supporting the old style of multi-line values and make it a runtime error. Hopefully it does not cause too much trouble for users. Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit 0fda3dcc202b6b202cc651428ebb3e67e5d0451b Author: Weiqun Zhang Date: Thu Jul 20 15:33:04 2023 -0700 SYCL: Get Pointer Type (#3434) This SYCL 2020 feature has been available in recent compilers. Src/Base/AMReX_GpuUtility.H commit abc6696d6eca0e37f87c01f895b36abfa8bf9175 Author: Weiqun Zhang Date: Thu Jul 20 10:47:35 2023 -0700 ParmParse: Option to disable multi-line support (#3433) By default, ParmParse supports multi-line values. For example, ``` plot_vars = dens vx vy vx energy entropy ``` This can be disabled by calling setMultiLineSupport(false) before amrex::Initialize(). This can avoid errors in inputs like, ``` algo.current_deposition = direct # Enable galilean psatd.use_default_v_galilean # Unfortunately we forgot = 1 ``` With multi-line support, this is unfortunately equivalent to ``` algo.current_deposition = direct psatd.use_default_v_galilean ``` With multi-line support disabled, it will abort. Note that even after multi-line support is disabled, one is still allowed to have ``` f = "x + y + sin(z)" ``` because here what's inside the pair of double quotes is considered a single string. Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit dbfa3b0daa3ce6144ab44ba7b2540637988c3e68 Author: Weiqun Zhang Date: Wed Jul 19 14:03:47 2023 -0700 SYCL: UUID (#3432) This was implemented a couple months ago, but I didn't have time to check the correctness until now. Src/Base/AMReX_GpuDevice.cpp commit 20f9d300fc8cb3dbf0986544afa73055fc8af088 Author: Weiqun Zhang Date: Wed Jul 19 10:12:42 2023 -0700 Re-Enable FPE Trap for SYCL (#3430) FPE trapping was disabled for SYCL in the past due to a bug in the Intel graphics compiler. The bug has long been fixed. Also remove the macro for PGI, because we don't need to support PGI < 16 anymore. More information about the bug. The JIT compiler computed `sqrt(-1)` and was caught by AMReX's signal handler. Src/Base/AMReX.cpp commit b523c3297d00526ae11a44585106ac454743c6cf Author: Weiqun Zhang Date: Tue Jul 18 18:50:39 2023 -0700 Memory Efficient AoS Particle Sorting (#3427) Because of the size of AoS particle, creating a temporary vector for all the AoS data during reording might use too much memory. In fact, this is usually the memory usage bottleneck in WarpX simulations. In this new approach, we divide the AoS into smaller chunks and reorder them one at a time, thus significantly reducing the peak memory usage. Thanks @AlexanderSinn for the suggestion. Src/Particle/AMReX_ParticleContainerI.H commit fc32f3e875af8416000c66c43e93fc5170a9b083 Author: Weiqun Zhang Date: Tue Jul 18 18:05:15 2023 -0700 Use MPI_IN_PLACE (#3428) Src/Base/AMReX_ParallelReduce.H commit 10b6cb26d0ac359922276ce08eeb728a14624b70 Author: Weiqun Zhang Date: Mon Jul 17 14:30:40 2023 -0700 PODVector Updates (#3425) Remove deprecated and unused PolymorphicAllocator. It has been replaced by PolymorphicArenaAllocator. Restrict PODVector's Allocator to std::allocator and AMReX's various Arena based allocators. This simplifies the implementation of PODVector, because std::allocator is stateless and Arena based allocators are simple even when it's polymorphic. Fix a few issues of PODVectors with a PolymorphicArenaAllocator. For example, copy assignment operator should copy the Allocator. Copy constructor should consider the possibility that the other PODVector has a different type of Arena. Add placeholders for potentially growing and shrinking memory allocation in-place that will be implemented in a follow-up PR. Update PODVector's growth strategy. Hopefully this helps to reduce the memory consumption. * Always try to grow in-place. * For constructors, assign, operator=, resize & reserve, allocate the specified size without extra capacity. * For push_back & emplace_back, grow the capacity by a factor that is 1.5 by default, when there is need to grow. * For insert, the capacity grows either by a factor that is 1.5 by default or to the new size, whichever is greater, when there is need to grow. Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_Arena.H Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_PODVector.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainer.H commit 01b750d909b0e10c7d9197360d0545cbb8bd48d8 Author: Ann Almgren Date: Sat Jul 15 21:40:03 2023 -0700 generalize interpolaters to not need or use extra values in a directi… (#3415) …on for which the refinement ratio is 1 The previous version of the interpolation routines in AmrCore would grow the "CoarseBox" in all directions even if one of the directions had refinement ratio 1. This PR changes that behavior so the CoarseBox is not grown, and slopes are not computed, in any direction for which the refinement ratio is 1. Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterpolater.cpp commit 5a90c264fad82e0d09fe3958a679dddca3571940 Author: Weiqun Zhang Date: Sat Jul 15 10:55:25 2023 -0700 Fix Fortran module installation (#3411) It was broken in #3309, in which multi-D support was added. Tools/CMake/AMReXInstallHelpers.cmake commit 5376bf85b83fd8f23bd983dea52d828c86dc8f67 Author: Weiqun Zhang Date: Sat Jul 15 10:54:36 2023 -0700 Fix allocateSlice for EB (#3391) The slice data has a different BoxArray, so we cannot use the original Factory. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_MultiFabUtil.cpp commit 36f42eaa3fa61f2e8c8c9a35c0e8f6ba09c0c0d4 Author: Weiqun Zhang Date: Sat Jul 15 10:48:08 2023 -0700 Update Intel compiler installation (#3423) .github/workflows/dependencies/dependencies_dpcpp.sh commit b803adbf6a00685985895a7367f46beb3aa8bbe1 Author: Candace Gilet Date: Fri Jul 14 20:50:26 2023 -0400 Improve comments and documentation on FaceDivFree Interpolater. (#3422) Docs/sphinx_documentation/source/AmrCore.rst Src/AmrCore/AMReX_Interpolater.H commit 500e5fc7409e7b4a8162f704bb5bd3073072e90a Author: Candace Gilet Date: Fri Jul 14 16:29:52 2023 -0400 Add the option to throw an exception on MLMG failure rather than aborting (#3424) Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/LinearSolvers.rst Src/LinearSolvers/MLMG/AMReX_MLMG.H commit 2a555f86eb8aeea64dcd414151f966907fc6e623 Author: Weiqun Zhang Date: Thu Jul 13 09:28:56 2023 -0700 Make ReduceData::value safer (#3421) Although ReduceData::value is meant to be called only once, the user might call it multiple times. This will result in the wrong result for sum reduction because of how the value function is implemented. In this PR, we fix this defect by allowing the function be called multiple times. Src/Base/AMReX_Reduce.H commit 5117d21af7902a8032a004f09ca495b5be4fd752 Author: Weiqun Zhang Date: Wed Jul 12 15:50:28 2023 -0700 Use atomic load in SYCL version of the single-pass scan (#3419) Otherwise the single-pass scan of long ints will hang because it does not load the most recent data even though the member function is volatile. Src/Base/AMReX_Scan.H Tools/GNUMake/comps/dpcpp.mak commit 59a310629397b69d4d5242ee0eba11ef8718f2aa Author: Miren Radia <32646026+mirenradia@users.noreply.github.com> Date: Wed Jul 12 20:24:52 2023 +0100 GPU: Add Device::deviceVendor() function and fix #3416 (#3418) ## Summary This adds a function that returns the GPU's vendor. ## Additional background This function is only really relevant for SYCL (although only Intel GPUs are officially supported with the SYCL backend AFAIK). For CUDA and HIP it is inferred at compile time. This function is used to disable SIGSEGV handling by default on Intel GPUs to fix #3416. In theory, we could only default disable SIGSEGV handling for specific Intel GPU architectures (e.g. using the experimental [oneAPI SYCL Device Architecture extension](https://github.com/intel/llvm/blob/48be219e238ca2354a1e9e9989752aa7a60b5627/sycl/doc/extensions/experimental/sycl_ext_oneapi_device_architecture.asciidoc)) since integrated Intel GPUs are not affected by #3416 but this would need to be updated every time Intel releases a new GPU architecture which still uses SIGSEGV for managed memory. It is simpler to disable it by default on all Intel GPUs. Src/Base/AMReX.cpp Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp commit 6dfdd48dc476142cf43e92617b1a4a0d10516e21 Author: Weiqun Zhang Date: Tue Jul 11 17:12:01 2023 -0700 New feature in EB: Add regular coarse levels (#3414) This feature could be useful when the EB is so thin that it is only resolved at the finest level and the usual way of generate coarse levels fails with multi-cut or multi-value cells. When the EB is fully refined to the finest level and there is no subcycling, no EB information is really needed at coarse levels. amrex::TagCutCells has been updated to work with the new feature. Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_IndexSpaceI.H Src/EB/AMReX_EB2_IndexSpace_STL.H Src/EB/AMReX_EB2_IndexSpace_STL.cpp Src/EB/AMReX_EB2_IndexSpace_chkpt_file.H Src/EB/AMReX_EB2_IndexSpace_chkpt_file.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EBAmrUtil.cpp Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EBDataCollection.cpp Src/EB/AMReX_EBFabFactory.H Src/EB/AMReX_EBFabFactory.cpp commit 3701d00778d0f69ea5a111687c44d90a8d8d6c6e Author: Ann Almgren Date: Tue Jul 11 15:32:06 2023 -0700 add access function for EBFactory (moved out of IAMR) (#3417) Src/Amr/AMReX_AmrLevel.H commit b41f85ab36acf9640bfc397977bb8c67d1ec108d Author: Weiqun Zhang Date: Fri Jul 7 19:30:11 2023 -0700 Fix a warning (#3413) Src/Base/AMReX_VisMF.cpp commit bc806f0f15cee3b943b1fbc3a5f32da88cc24f0d Author: Weiqun Zhang Date: Thu Jul 6 15:17:15 2023 -0700 Fix clang-tidy warning seen in incflo (#3408) Src/Base/AMReX_PhysBCFunct.H Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2.cpp commit 7fed45f3d97d34f6c1e0c479ef3828338458c705 Author: Weiqun Zhang Date: Thu Jul 6 11:50:55 2023 -0700 Quiet some clang-tidy warnings (#3407) Src/Base/AMReX.cpp Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2.cpp commit 2cd314f421beea714835ca91ce615e597980ed4d Author: Miren Radia <32646026+mirenradia@users.noreply.github.com> Date: Thu Jul 6 17:16:40 2023 +0100 DataServices: Qualify math functions with `std::` namespace specifier (#3406) ## Summary As in the title. This fixes #3405. ## Additional background I did briefly look at using `powi`, but I don't think the exponents are known at compile time. I also added some auto-generated source files to .gitignore. .gitignore Src/Extern/amrdata/AMReX_DataServices.cpp commit 28df482cebf26dd0f4e8bb5ee7faa9cb463415bb Author: Weiqun Zhang Date: Wed Jul 5 10:29:08 2023 -0700 Update CleanUpCache workflow (#3403) The pr_number artifact is only available for `pull_request`. The logic there will become incorrect if another event like `schedule` is added, because it only assumes it's either `push` or `pull_request`. .github/workflows/cleanup-cache.yml commit 9154e8417c3a458ef537a60fc241ed69e291ad45 Author: Miren Radia <32646026+mirenradia@users.noreply.github.com> Date: Wed Jul 5 17:25:10 2023 +0100 Add some profilers to AmrLevel and FillPatcher (#3402) ## Summary Pretty much as in the title. ## Additional background There are profilers in most of the other `AmrLevel` functions that look like they might be worth profiling. Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/AmrCore/AMReX_FillPatcher.H commit 88f03408f18a8c5b6785082b56c88c974e8001a1 Author: Weiqun Zhang Date: Mon Jul 3 22:17:45 2023 -0700 Update the clang-tidy/ccache script (#3399) Add and argument specifying a unique identify. This makes it easier for the script to be used by other codes (e.g., AMReX-Hydro) in their CIs. Also increase the cache size for some CIs. ROCm 5.6 seems to need more ccache spaces. .github/workflows/clang.yml .github/workflows/hip.yml Tools/C_scripts/mmclt.py commit 2589f265eeb4a5af021bb9845617cab50b09da97 Author: Weiqun Zhang Date: Mon Jul 3 15:00:25 2023 -0700 Fix some warnings (#3398) Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_VisMF.cpp Src/EB/AMReX_EB_FluxRedistribute.cpp commit e80199a810de8968576e6df9a614f0ad68067be7 Author: Weiqun Zhang Date: Mon Jul 3 13:13:55 2023 -0700 Update Git workflow in CONTRIBUTING.md (#3397) Use `origin` as the name of the official amrex repo and `` as the name of a developer's fork. This is less confusing than using names like `upstream` for the official repo and `origin` for the fork. Remove the git rebase instruction, which is arguably out of the scope of CONTRIBUTING.md. CONTRIBUTING.md commit 9fc635426caeef7d2c0b6bf7ae7700bd5e6600ec Author: Ann Almgren Date: Sun Jul 2 20:08:08 2023 -0700 move redist into amrex (#3378) This PR moves all redistribution functionality from AMReX-Hydro to amrex/Src/EB. It also adds re-redistribution to the routines in EB (copied from Tests/EB_CNS). (It also moves the slope routines since those are needed by state redistribution.) This change will require changes in any code that currently uses AMReX-Hydro for redistribution. Additional changes will be required in application codes that want to enable re-redistribution. Once this PR is merged in amrex, all routines in AMReX-Hydro/Slopes and AMReX-Hydro/Redistribution will be deleted. Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/EB/areas_and_volumes.fig Docs/sphinx_documentation/source/EB/areas_and_volumes.png Docs/sphinx_documentation/source/EB/eb_fluxes.fig Docs/sphinx_documentation/source/EB/eb_fluxes.png Docs/sphinx_documentation/source/EB/redist.fig Docs/sphinx_documentation/source/EB/redist.png Docs/sphinx_documentation/source/LinearSolvers.rst Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_Slopes_K.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/Boundary/AMReX_YAFluxRegister.H Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBFArrayBox.cpp Src/EB/AMReX_EBFluxRegister.H Src/EB/AMReX_EBFluxRegister.cpp Src/EB/AMReX_EBFluxRegister_2D_C.H Src/EB/AMReX_EBFluxRegister_3D_C.H Src/EB/AMReX_EBInterpolater.cpp Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EB_FluxRedistribute.cpp Src/EB/AMReX_EB_Redistribution.H Src/EB/AMReX_EB_Redistribution.cpp Src/EB/AMReX_EB_RedistributionApply.cpp Src/EB/AMReX_EB_Slopes_2D_K.H Src/EB/AMReX_EB_Slopes_3D_K.H Src/EB/AMReX_EB_Slopes_K.H Src/EB/AMReX_EB_StateRedistItracker.cpp Src/EB/AMReX_EB_StateRedistSlopeLimiter_K.H Src/EB/AMReX_EB_StateRedistUtils.cpp Src/EB/AMReX_EB_StateRedistribute.cpp Src/EB/AMReX_EB_utils.H Src/EB/AMReX_EB_utils.cpp Src/EB/CMakeLists.txt Src/EB/Make.package Tests/EB_CNS/Source/CNS.H Tests/EB_CNS/Source/CNS.cpp Tests/EB_CNS/Source/CNSBld.cpp Tests/EB_CNS/Source/CNS_advance_box_eb.cpp Tests/EB_CNS/Source/CNS_bcfill.cpp Tests/EB_CNS/Source/CNS_derive.H Tests/EB_CNS/Source/CNS_init_eb2.cpp Tests/EB_CNS/Source/CNS_parm.cpp Tests/EB_CNS/Source/Make.package Tests/EB_CNS/Source/hydro/CNS_flux_redistribute.cpp Tests/EB_CNS/Source/main.cpp commit 22a1a3a1d4b0aaa81d3deaa0320e4a4aebb2d99d Author: Miren Radia <32646026+mirenradia@users.noreply.github.com> Date: Sat Jul 1 20:40:01 2023 +0100 Amr: Add printing of simulation time (#3392) A very small PR to add the printing of simulation time in `Amr::timeStep` (for `Amr::verbose > 0`). Though the time is already printed out every coarse timestep [here](https://github.com/AMReX-Codes/amrex/blob/9c256b12b89f86b30fe1a6f03dd79db351d56fb3/Src/Amr/AMReX_Amr.cpp#L2174), with deep AMR hierarchies, it can be a long amount of [wall]time between these printouts, hence I would find it useful to see the current simulation time more often. Src/Amr/AMReX_Amr.cpp commit 97724ebdc79fd7a239ea6e6ade3a4d2303d79c82 Author: Andrew Myers Date: Sat Jul 1 12:37:50 2023 -0700 Add function for setting growth factor (#3394) Add the ability to set the custom vector growth factor through a function, instead of only as a runtime input. Also reducing the min allowed factor, as there seems to be demand for that. Src/Base/AMReX_PODVector.H Src/Base/AMReX_PODVector.cpp commit e8120c8935c86dbc0ff14cb60610e5e3b8033be5 Author: Cyrus Harrison Date: Sat Jul 1 12:33:27 2023 -0700 github actions update ascent ci to use 0.9.2 (#3395) Updates Ascent Github Actions CI to use 0.9.2 (latest release) .github/workflows/ascent.yml commit 60b23d9d54287d4b4616add69783ab19729664af Author: Weiqun Zhang Date: Sat Jul 1 10:58:56 2023 -0700 Update CHANGES for 23.07 (#3396) CHANGES commit 9c256b12b89f86b30fe1a6f03dd79db351d56fb3 Author: Andrew Myers Date: Thu Jun 29 16:24:53 2023 -0700 Allow users to change the default vector growth stategy (#3389) This allows users to set at runtime an alternate to the default vector growth factor of 1.5. This could help save memory in simulations that use a lot of particles and are close to the capacity of the GPU device. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Weiqun Zhang Src/Base/AMReX.cpp Src/Base/AMReX_PODVector.H Src/Base/AMReX_PODVector.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 4a4e8b3fb1b9248b32aeeb62682ed3632791a74b Author: Mukul Dave Date: Wed Jun 28 09:04:56 2023 -0700 Communications arena implementation (#3388) ## Summary Implement a communications arena for comm buffers to replace `the_fa_arena`. It creates a separate arena when GPU-aware MPI is used and `the_arena` is not managed. ## Additional background The motivation for this is a communication performance degradation that is observed for GPU-aware MPI with `amrex.the_arena_is_managed=0`. @WeiqunZhang has a hypothesis that this may be due to the need for frequent re-registering of comm buffer pointers when using the same device arena as the other compute data. Hence a separate arena in this case would alleviate this issue. `the_fa_arena` is eliminated in this PR and the communication buffer directly uses `the_comms_arena` to simplify the code. ## Performance tests The above stated performance degradation is particularly observed with the `GPU/CNS/Exec/Sod` code under `Tests` and is alleviated by using a separate comms arena as seen in the performance data below. `original` refers to the state before we made the change in #3362 related to `the_fa_arena` pointing to the device arena which allowed `amrex.the_arena_is_managed=1` with GPU-aware MPI without a significant performance hit. It is compared with the current development branch and the proposed comms arena implementation. The data pointing to the performance improvement from this PR is highlighted. ![Screenshot 2023-06-27 at 4 11 54 PM](https://github.com/AMReX-Codes/amrex/assets/18251677/ae16b822-0178-4679-a90f-255cad6c5451) In other tests such as the `ABecLaplacian` linear solve or the ERF code, using `amrex.the_arena_is_managed=0` did not show a significant performance hit and using this comms arena implementation did not harm the performance either. More comprehensive tests would be required to determine the effect on other codes and platforms. --------- Co-authored-by: Mukul Dave Co-authored-by: Weiqun Zhang Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_NonLocalBC.cpp commit 0236a3732cc8e399e01eacd0253a782da40ce1f7 Author: Nuno Nobre Date: Tue Jun 27 15:56:33 2023 +0100 CI: oneAPI SYCL for AMD GPUs (#3341) .github/workflows/dependencies/dependencies_codeplay.sh .github/workflows/dependencies/dependencies_hip.sh .github/workflows/intel.yml Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake commit 4d6fdde3cbc596e172331796d1e103bf33a4a338 Author: Jean M. Sexton Date: Mon Jun 26 14:24:19 2023 -0700 Adjust caption for growntilebox picture (#3386) Clarifies caption describing boxes in MFIter with Tiling documentation Docs/sphinx_documentation/source/Basics.rst commit e2d0d86facde61e19dc0276eb44bad435c30e36e Author: Miren Radia <32646026+mirenradia@users.noreply.github.com> Date: Mon Jun 26 15:44:54 2023 +0100 GPU: Always deallocate gpu_rand_state in Finalize() (#3384) ## Summary This fixes #3383. ## Additional background I noticed this problem when I tried to build an application with multiple `Initialize()` and `Finalize()` calls with HIP e.g. something like: ```cpp amrex::Initialize(MPI_COMM_WORLD); amrex::Finalize(); amrex::Initialize(MPI_COMM_WORLD); amrex::Finalize(); ``` I'm not sure if this officially supported but it works fine with CUDA (and with MPI so long as you only initialize and finalize this once outside of the above code) but not with HIP which fails at the second initialization [here](https://github.com/AMReX-Codes/amrex/blob/239d4d89933488e1d8f6e39ceae211ab690aa855/Src/Base/AMReX_Random.cpp#L36) with the error message ``` amrex::Abort::0::CArena::free: unknown pointer !!! SIGABRT ``` ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX.cpp commit 93e3d1d990bd876a9de5f0d323bb04b94c948f34 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Sun Jun 25 15:28:06 2023 -0700 Hypre ILU options (#3381) Adding more parameter hooks for ILU smoothers. These options are supported on AMD and Nvidia architectures as of Hypre commit 18b9880 (Jun-15-2023) Co-authored-by: Paul Mullowney Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit 239d4d89933488e1d8f6e39ceae211ab690aa855 Author: Weiqun Zhang Date: Tue Jun 20 09:39:59 2023 -0700 Amr Class: Synchronization of StateData's Time (#3375) The synchronization of StateData's time is done in a virtual function, AmrLevel::postCoarseTime. However, the issue is the override version of the virtual function often forget to do it. In this PR, the job is instead done in the Amr class directly. This should make the code more robust. Close #3374 Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp commit d28b9ab1cd64e648372ae679a6be01c733899525 Author: Siddhartha Menon <42359201+Sqvid@users.noreply.github.com> Date: Fri Jun 16 17:54:26 2023 +0100 Fix typo in profiler macro (#3371) Tests/EB_CNS/Source/CNS_advance_box.cpp commit 02ac1f39bce4c6b0aa76aba40de1ac26c92dd464 Author: Weiqun Zhang Date: Thu Jun 15 15:58:33 2023 -0700 EB 2D: Fix levelset on nodes adjacent to covered edges (#3370) ## Summary This fixes inconsistency in 2D EB data. When an edge is covered, its edge length is set to zero. If the level set value on its nodes is not set to less than zero, the EB solver LEBNodeFDLaplacian will treat them as normal nodes and that will cause floating point issues. Note that we have already been doing this kind of fix in 3D. ## Additional background This issue was reported by @RemiLehe. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_Level.H commit 6303e0abfae93ce1a76dcfa6bd37f9a4942d7ca2 Author: Weiqun Zhang Date: Wed Jun 14 21:39:28 2023 -0700 User's Guide: Fix Instruction on Making Movies with VisIt (#3368) Use `ls -1v` instead of `ls -1` to get a list of sorted plotfile headers. We need to use `-v` because the length of plotfiles may be different. Thank @JerryNiOnly for identifying the issue! (#3363) Docs/sphinx_documentation/source/Visualization.rst commit c31bf0db65f15e8ff5d9ec469256bef4a7b0df0b Author: Weiqun Zhang Date: Wed Jun 14 16:27:56 2023 -0700 Add workflow for oneAPI SYCL on Nvidia GPUs (#3360) The Codeplay token used in this workflow is saved as a secret. Because of how GitHub secrets work, the workflow will only run when a PR is merged, not when it's trigger by a PR from an outside fork. --------- Co-authored-by: Nuno Miguel Nobre .github/workflows/dependencies/dependencies_codeplay.sh .github/workflows/intel.yml commit 9c412974de2ac0496878acf1f7e8d6c426f535e7 Author: Axel Huebl Date: Wed Jun 14 15:10:07 2023 -0700 CI: Simplify App Checkout (#3365) ## Summary Based on API in `actions/checkout`: https://github.com/actions/checkout ## Additional background Follow-up to #3348 Co-authored-by: Weiqun Zhang .github/workflows/apps.yml commit 41a6700392400bab9a688ca6892a6d638483f6e0 Author: Axel Huebl Date: Wed Jun 14 10:04:37 2023 -0700 CMake: Enforce >= C++17 for ROCm (#3355) ## Summary Check if this adds the C++17 flags again or if there is an undocumented change in defaults that we need to manually address for AMD's Clang fork'd compilers now. ## Additional background For #3337 Tools/CMake/AMReXParallelBackends.cmake commit 96b811d2cb9f24e87428e4ebc9904b4dcdf73e7f Author: Mukul Dave Date: Tue Jun 13 08:04:39 2023 -0700 Use device arena for the_fa_arena when activating GPU-aware MPI (#3362) ## Summary This change suggested by @WeiqunZhang points `the_fa_arena` to `The_Device_Arena` when activating GPU-aware MPI. This obviates the need for setting `the_arena_is_managed=0` to take advantage of GPU-aware MPI since it does not work well with managed memory. ## Additional background The motivation for this PR is that this was an long-pending change but the immediate trigger was finding that GPU-aware MPI can reduce communication times significantly but that currently needs setting `the_arena_is_managed=0`. Not setting this for GPU-aware MPI currently results in degraded performance. Past discussion on GPU-aware MPI: #2967 ## Preliminary performance test Running 100 steps on 8 GPUs over 2 Perlmutter A100 nodes with `Tests/GPU/CNS/Exec/Sod`, `amr.n_cell = 128^3` per GPU, `amr.max_grid_size = 128`, `amrex.use_profiler_syncs = 1` and setting optimal GPU affinities. ### Without `amrex.use_gpu_aware_mpi=1` ``` FabArray::ParallelCopy_nowait() 200 0.133 0.1779 0.2067 17.82% FabArray::ParallelCopy_finish() 200 0.07822 0.1193 0.1786 15.40% ``` ### With `amrex.use_gpu_aware_mpi=1` ``` FabArray::ParallelCopy_nowait() 200 0.05655 0.07633 0.1034 11.20% FabArray::ParallelCopy_finish() 200 0.03969 0.06087 0.09024 9.77% ``` Co-authored-by: Mukul Dave Src/Base/AMReX_FabArrayBase.cpp commit 0375e78e7838405190c240f6ceb5013a38debc92 Author: Weiqun Zhang Date: Mon Jun 12 14:01:27 2023 -0700 Coverity Scan (#3361) ## Summary Fix issues found by Coverity Scan, including infinite loop bugs in the kernels of MLALaplacian. The scan will be done nightly on a local workstation. Add coverity scan and license badges to README.md. Remove the cmake README.md because it's broken. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate README.md Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateData.cpp Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabConv.cpp Src/Base/AMReX_ForkJoin.cpp Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_LayoutData.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_PlotFileDataImpl.cpp Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_Tuple.H Src/Base/AMReX_Utility.H Src/Base/AMReX_Utility.cpp Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_ccse-mpi.H Src/Base/AMReX_iMultiFab.cpp Src/Boundary/AMReX_Mask.cpp Src/Boundary/AMReX_YAFluxRegister.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_MultiCutFab.H Src/EB/AMReX_MultiCutFab.cpp Src/EB/AMReX_algoim_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_ParticleUtil.cpp Tools/GNUMake/Make.defs commit 65ba959170fb2f29c3cd6e641b01a2ac208e1233 Author: Weiqun Zhang Date: Mon Jun 12 12:42:20 2023 -0700 CMake: AMReX_TEST_TYPE (#3357) Add a new option, AMReX_TEST_TYPE=[All|Small]. Be default, all tests are built when AMReX_ENABLE_TEST is on. When the type is small, only four tests are enabled. If a CI job takes too long to compile and run all the tests, we could switch to the small test type. Re-enable tests and change the test type to Small for the nvhpc CI. We had to disable it due to space limit in. It should no longer be a problem for the small set of tests. .github/workflows/cuda.yml CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Tests/CMakeLists.txt Tools/CMake/AMReXOptions.cmake commit cc7a5c1171bb2634a9a71ce3c18cfd0be72f581c Author: Nuno Nobre Date: Fri Jun 9 18:23:59 2023 +0100 CI: Tweak Intel compiler workflow (#3354) .github/workflows/intel.yml commit 7c7a8fb1fff518d1f003bafa42fb37841c1d8885 Author: Weiqun Zhang Date: Wed Jun 7 20:39:33 2023 -0700 Fix ccache setup for building CUDA with clang (#3359) Need to set `CCACHE_DEPEND=1`. .github/workflows/cuda.yml commit 05dc073d4f5bae35a561d37cf7a258eb3bb430df Author: Weiqun Zhang Date: Wed Jun 7 16:27:22 2023 -0700 CMake: Enable AMReX_PARTICLES by default (#3358) This adds very little time to building the amrex library because most of the particle codes are in headers. Note that `configure` using GNU Make has had particle on by default already. Also note that this does not affect amrex built with spack because it explicitly sets to the particles variant to false. .github/workflows/ascent.yml .github/workflows/clang.yml .github/workflows/codeql.yml .github/workflows/cuda.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/sensei.yml .github/workflows/smoke.yml .github/workflows/windows.yml Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXOptions.cmake commit 1f3afaa42d0dbba95a1b4a8dee0cbab0286d0455 Author: Andy Nonaka Date: Tue Jun 6 18:11:07 2023 -0700 updated petsc build info (#3353) improved the petsc build instructions Docs/sphinx_documentation/source/LinearSolvers.rst commit 97e351a3fc01aabcbdeb13d077d3783bb926b3c5 Author: Weiqun Zhang Date: Tue Jun 6 10:29:42 2023 -0700 NVHPC CI: Disable Tests (#3352) With Tests, the CI uses more space than available in a GitHub runner. Disable tests to avoid `LLVM ERROR: IO failure on output stream: No space left on device`. .github/workflows/cuda.yml commit 58ab3dd4a9bd3febd91eab32d0ef56015508a7e3 Author: Weiqun Zhang Date: Tue Jun 6 09:43:06 2023 -0700 Add AMReX Apps to CI (#3348) Castro and WarpX are added in this commit. We can add more in the future. This provides more coverage. The issues in #3342 could have been avoided. .github/workflows/apps.yml .github/workflows/cleanup-cache.yml commit 5449a6981799c67237091d89182c149544a80e05 Author: Weiqun Zhang Date: Mon Jun 5 21:45:38 2023 -0700 Fix Warning in #3338 (#3347) Explicitly convert amrex::Long to int. Src/Particle/AMReX_ParticleTile.H commit 996ff7a12acb07b3abb4fcfae045af0a8d15bfca Author: Weiqun Zhang Date: Mon Jun 5 21:15:17 2023 -0700 Make sure amrex::Long is wider than int. (#3345) This should work on both Windows and 32 bits systems. Close #3344 Src/Base/AMReX_INT.H commit 7bd7ba1c62077bb85ec16450e32d77579250a583 Author: Weiqun Zhang Date: Mon Jun 5 19:36:11 2023 -0700 Fix some issues in #3342 (#3346) Src/AmrCore/AMReX_ErrorList.H Src/Base/AMReX_PhysBCFunct.H commit d9bae8ce9e69a962154a9340a0fb8ae9895c1fde Author: Axel Huebl Date: Mon Jun 5 18:35:38 2023 -0700 Particle Ids: Add int Constants (#3338) ## Summary Ideally we do not need them, because 2B particles generated per GPU is very low, but we do need this temporarily until we find a way to generalize large id values again for pure SoA particle layouts. cc @AlexanderSinn ## Additional background Follow-up to #2878. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTile.H commit a2a22dabc2c4c451807495e75558048c41467482 Author: Weiqun Zhang Date: Mon Jun 5 11:07:28 2023 -0700 Fix Clang-Tidy Header Filter (#3342) Fix regex in the clang-tidy header filter. The issue was the old regex filtered out headers like AMReX_Any.H (because of its letter n) and AMReX_IntVect.H (because of it's letter t before .H) in addition to files ending with nolint.H. Fix warnings. .clang-tidy .github/workflows/cuda.yml Src/Amr/AMReX_Derive.H Src/Amr/AMReX_LevelBld.H Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp Src/Base/AMReX_Any.H Src/Base/AMReX_Arena.H Src/Base/AMReX_AsyncOut.H Src/Base/AMReX_BArena.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxDomain.H Src/Base/AMReX_BoxList.H Src/Base/AMReX_CArena.H Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FabConv.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_GpuUtility.cpp Src/Base/AMReX_IntVect.H Src/Base/AMReX_Machine.H Src/Base/AMReX_Morton.H Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtilI.H Src/Base/AMReX_Orientation.H Src/Base/AMReX_PArena.H Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_Periodicity.H Src/Base/AMReX_PhysBCFunct.H Src/Base/AMReX_PhysBCFunct.cpp Src/Base/AMReX_Print.H Src/Base/AMReX_RealBox.H Src/Base/AMReX_RealVect.H Src/Base/AMReX_TagParallelFor.H Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Utility.H Src/Base/AMReX_Utility.cpp Src/Boundary/AMReX_BoundCond.H Src/Boundary/AMReX_FabSet.H Src/EB/AMReX_EB2_IF_AllRegular.H Src/EB/AMReX_EB2_IF_Box.H Src/EB/AMReX_EB2_IF_Complement.H Src/EB/AMReX_EB2_IF_Cylinder.H Src/EB/AMReX_EB2_IF_DevicePtr.H Src/EB/AMReX_EB2_IF_Difference.H Src/EB/AMReX_EB2_IF_Ellipsoid.H Src/EB/AMReX_EB2_IF_Extrusion.H Src/EB/AMReX_EB2_IF_Intersection.H Src/EB/AMReX_EB2_IF_Lathe.H Src/EB/AMReX_EB2_IF_Parser.H Src/EB/AMReX_EB2_IF_Plane.H Src/EB/AMReX_EB2_IF_Polynomial.H Src/EB/AMReX_EB2_IF_Rotation.H Src/EB/AMReX_EB2_IF_Scale.H Src/EB/AMReX_EB2_IF_Sphere.H Src/EB/AMReX_EB2_IF_Spline.H Src/EB/AMReX_EB2_IF_Torus.H Src/EB/AMReX_EB2_IF_Translation.H Src/EB/AMReX_EB2_IF_Union.H Src/EB/AMReX_EB2_IndexSpace_chkpt_file.H Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_EB_chkpt_file.H Src/EB/AMReX_EB_utils.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_algoim_K.H Src/EB/AMReX_distFcnElement.H Src/Extern/HYPRE/AMReX_HypreSolver.H Src/Extern/PETSc/AMReX_PETSc.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_hypre.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.H Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_SparseBins.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d_K.H Tests/Amr/Advection_AmrLevel/Source/Adv.cpp Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tests/Amr/Advection_AmrLevel/Source/bc_nullfill.cpp Tests/LinearSolvers/ABecLap_SP/MyTest.H Tests/LinearSolvers/ABecLaplacian_C/MyTest.H Tests/LinearSolvers/NodalPoisson/MyTest.H commit 79f6283b5cb4b6bbafae43f946f9d919f68e1b6e Author: Axel Huebl Date: Mon Jun 5 10:43:18 2023 -0700 CI: Clang CUDA (#3184) Compile with vanilla Clang/LLVM's CUDA C++ frontend & its LLVM PTX backend. Vanilla LLVM is used in a variety of tools that we want to use for research. Co-authored-by: Weiqun Zhang .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_llvm_cuda11_ubuntu22.sh commit 572f226f4c9894f0067fae48c8fb1d6a5ca9049e Author: Nuno Nobre Date: Sun Jun 4 23:43:07 2023 +0100 Reinforce __CUDACC__ conditionals w/ AMREX_USE_CUDA (#3331) Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_GpuDevice.cpp commit bd008705611a06d2e86f76d93d4714913dc8c8bb Author: Nuno Nobre Date: Sun Jun 4 22:21:30 2023 +0100 Documentation tweaks (#3340) Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/IO.rst Docs/sphinx_documentation/source/LinearSolvers.rst Docs/sphinx_documentation/source/Particle.rst Docs/sphinx_documentation/source/Testing.rst Docs/sphinx_documentation/source/TimeIntegration_Chapter.rst commit 08326158c80e5ad05c57009c8170bb9b76e170c5 Author: Weiqun Zhang Date: Fri Jun 2 17:03:34 2023 -0700 CMake: Link option when HIP is a language (#3335) ## Summary Although HIP is treated as C++ in AMReX CMake, we should add the link options `-Xoffload-linker --whole-archive -fgpu-rdc` when the link language is HIP, not just CXX. Otherwise, an AMReX application code that treats HIP as a CMake supported language will not get those flags. ## Additional background https://github.com/xsdk-project/xsdk-examples/issues/50 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Tools/CMake/AMReXParallelBackends.cmake commit 02ce32baa5f243bbf012ba2824cc4c44682c6748 Author: Weiqun Zhang Date: Thu Jun 1 09:07:26 2023 -0700 Update CHANGES for 23.06 (#3339) CHANGES commit 2e1106e246c442514d072f76b049d4f0b02216a2 Author: Andrew Myers Date: Wed May 31 16:02:59 2023 -0700 Implement Checkpoint, Restart, and Plotfile for pure SOA (#3332) Follow-up to #2878. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Co-authored-by: Axel Huebl Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/Particles/CheckpointRestartSOA/CMakeLists.txt Tests/Particles/CheckpointRestartSOA/GNUmakefile Tests/Particles/CheckpointRestartSOA/Make.package Tests/Particles/CheckpointRestartSOA/inputs Tests/Particles/CheckpointRestartSOA/main.cpp Tests/Particles/CheckpointRestartSOA_AsyncIO/CMakeLists.txt Tests/Particles/CheckpointRestartSOA_AsyncIO/GNUmakefile Tests/Particles/CheckpointRestartSOA_AsyncIO/Make.package Tests/Particles/CheckpointRestartSOA_AsyncIO/inputs Tests/Particles/CheckpointRestartSOA_AsyncIO/main.cpp commit 07f87b0a9047a4306a056f60d71d6d49388a3497 Author: Axel Huebl Date: Mon May 29 09:12:32 2023 -0700 Clang: -Wno-null-conversion for Lexer (#3333) ## Summary Silence a warning in the lexer used in the parser for Clang 12. ## Additional background ``` /g/g90/huebl1/src/warpx/build_lassen/_deps/fetchedamrex-src/Src/Base/Parser/amrex_iparser.lex.nolint.H:2010:68: warning: implicit conversion of NULL constant to 'bool' [-Wnull-conversion] while ((yy_buffer_stack) ? yy_buffer_stack[yy_buffer_stack_top] : (__null)) { ... ``` et al. Src/Base/Parser/amrex_parser.lex.cpp commit 76d6d344e12d5089d659c943f9989c34f31f97a9 Author: Andrew Myers Date: Wed May 24 18:47:14 2023 -0700 Fix Pure SOA RedistributeGPU: size from ptile, not the AoS (#3330) Follow-up to $2878. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleContainerI.H commit b53312854d596c1e2d56febcb405e532295d00c0 Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Thu May 25 01:15:54 2023 +0200 cleaning and fixes for PureSoA ParticleTile (#3327) ## Summary * fix return by value/reference mismatch of `id()` and `cpu()` with `decltype(auto)` * update `packParticleData` and `unpackParticleData` for PureSoA * change `getSuperParticle` and add `setSuperParticle` for PureSoA. Remaining issue: super particle has redundant/unused fields for positions, id and cpu * update `ConstParticleTileData` for PureSoA * don’t call `getParticleTileData` in `ParticleTile::id()` etc. as this could be very slow if used in a for loop * use if constexpr instead if SFINAE in a number of places * change PODVector constructor so that `getParticleTileData()` won't allocate memory if no runtime components are used ## Additional background Follow-up to #2878. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_PODVector.H Src/F_Interfaces/Particle/AMReX_particlecontainer_fi.cpp Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_StructOfArrays.H commit d65dd906bd53c83eab8c88ea0e9205aa6dc6410b Author: Axel Huebl Date: Tue May 23 20:24:52 2023 -0700 RealBox: Accept RealVect Setters (#3328) Add overloads to RealVect. Src/Base/AMReX_RealBox.H commit 4167cbe534a00d65568203396964eeea3d449f1e Author: Andrew Myers Date: Mon May 22 21:13:57 2023 -0700 Port InitRandom for pure SoA (#3325) This is useful for porting and testing other features, like Checkpoint / Restart. I've also added a test. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleInit.H Tests/Particles/InitRandom/CMakeLists.txt Tests/Particles/InitRandom/GNUmakefile Tests/Particles/InitRandom/Make.package Tests/Particles/InitRandom/inputs Tests/Particles/InitRandom/main.cpp commit b625d63c8bf18ccaec51ea2c942b306d10b4971c Author: Weiqun Zhang Date: Mon May 22 16:22:07 2023 -0700 FillBoundary for Spherical Coordinates (#3286) This adds a FillBoundary function for spherical coordinates (theta, phi, r). The function can also be used to implement other cases .clang-tidy Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_IntVect.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBC.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/EB/AMReX_EB_STL_utils.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Tests/Amr/Advection_AmrCore/Source/AdvancePhiAllLevels.cpp Tests/MultiBlock/Advection/main.cpp Tools/Plotfile/fcompare.cpp commit 1881aca2c98ac71d8cedfe90ca1c62cf90b99aa4 Author: Axel Huebl Date: Mon May 22 16:03:00 2023 -0700 Fix: AMReXConfig.cmake Multi-Dim (#3324) ## Summary Fix two regressions from #3309 in the installed `AMReXConfig.cmake`: ``` CMake Error at lib/cmake/AMReX/AMReXConfig.cmake:257 (add_library): add_library cannot create ALIAS target "AMReX::amrex" because another target with the same name already exists. ``` and setting properties for newer CUDA. ## Additional background Regression to #3309, spotted by @drangara, @ajnonaka, wdf et al. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Tools/CMake/AMReXConfig.cmake.in commit 210a71a9f843e421a48dc783482eb84db1e15da8 Author: Weiqun Zhang Date: Mon May 22 13:22:08 2023 -0700 HypreSolver: a new interface for Hypre (#3269) Add a new interface class for Hypre. The existing Hypre interface classes are all tightly coupled with MLMG, and are therefore not usable for linear systems not supported by MLMG. In that case, one can use this new class. --------- Co-authored-by: Jean M. Sexton .github/workflows/hypre.yml Src/Base/AMReX_FabArray.H Src/EB/AMReX_MultiCutFab.H Src/Extern/HYPRE/AMReX_Hypre.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/HYPRE/AMReX_HypreSolver.H Src/Extern/HYPRE/CMakeLists.txt Src/Extern/HYPRE/Make.package Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Tests/LinearSolvers/Hypre/GNUmakefile Tests/LinearSolvers/Hypre/Make.package Tests/LinearSolvers/Hypre/MyTest.H Tests/LinearSolvers/Hypre/MyTest.cpp Tests/LinearSolvers/Hypre/initEB.cpp Tests/LinearSolvers/Hypre/inputs.2d Tests/LinearSolvers/Hypre/main.cpp commit f986296e949765d675b06a3f4ca2ffefc1b91b6f Author: Weiqun Zhang Date: Mon May 22 13:19:37 2023 -0700 Add a new CI for multi-d builds (#3321) In this CI, we also run installation tests and the spack smoke test. This is a follow-up on #3309. .clang-tidy .github/workflows/cleanup-cache.yml .github/workflows/smoke.yml Src/Amr/AMReX_Amr.cpp Src/AmrCore/AMReX_Cluster.cpp Src/Base/AMReX_CArena.cpp Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FileSystem.cpp Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_Utility.cpp Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H commit 342799dd78ec14b689507ede31d2acf2e7c3b217 Author: Weiqun Zhang Date: Mon May 22 10:16:43 2023 -0700 Fix Tests/EB_CNS in 2D (#3323) Close #3322 Tests/EB_CNS/Source/CNS_tagging.H commit 084165c236ab2780219c0e792e099f10365011c4 Author: Weiqun Zhang Date: Fri May 19 11:27:42 2023 -0700 Fix CodeQL warnings (#3318) Docs/sphinx_documentation/source/conf.py Src/Base/AMReX_Box.H Src/Base/AMReX_FArrayBox.H Src/EB/AMReX_EB2_IF_Spline.H Src/Particle/AMReX_ParticleContainerI.H Tools/C_scripts/describe_sources.py Tools/C_scripts/gatherbuildtime.py Tools/C_scripts/makebuildinfo_C.py Tools/CompileTesting/compiletesting.py Tools/EBSurfaceTools/isoToVTK.py Tools/Postprocessing/python/dumpparthistory.py Tools/Postprocessing/python/parseparticles.py Tools/Postprocessing/python/test_parseparticles.py Tools/Py_util/amrex_particles_to_vtp/amrex_binary_particles_to_vtp.py Tools/Py_util/amrex_particles_to_vtp/write_pview_file.py Tools/libamrex/mkpkgconfig.py Tools/libamrex/mkversionheader.py commit f4f6ae9184d26e4f906a66bd63ad5efe4e68bb7f Author: David Grote Date: Thu May 18 18:02:56 2023 -0700 Add get/set of AMRErrorTagInfo in AMRErrorTag (#3317) ## Summary This add Get/Set routines of the AMRErrorTagInfo object in AMRErrorTag. ## Additional background The purpose of this change is to allow dynamically changing the error tag info objects (without creating whole new ones). The larger goal is to allow more control over whether a regard is done or not. Src/AmrCore/AMReX_ErrorList.H commit ac8574131dd152ce40853e17af948360def806f4 Author: Eric T. Johnson Date: Thu May 18 18:58:18 2023 -0400 GNU Make: allow skipping paths when running clang-tidy (#3316) Downstream projects can set `CLANG_TIDY_IGNORE_SOURCES` to a space-separated list of paths under which clang-tidy should not be run. Tools/GNUMake/Make.rules Tools/GNUMake/tools/Make.clang-tidy commit ca706580b5fd28296608a18ef7ab29a9149b4438 Author: Axel Huebl Date: Thu May 18 11:42:15 2023 -0700 Multi-Dim Buildsystem Support (#3309) ## Summary This updates the CMake build system to allow to build multiple AMReX libraries *at once*, by building multiple, uniquely-named `(lib)amrex_[1|2|3]d.[so|a|dll|lib]` artifacts. Developers can request this by specifying a list of dims instead of a single one during configure time, e.g., `-DAMReX_SPACEDIM="1;2;3"`. Providing this functionality will significantly reduce deployment burdens for: - conda packages - spack packages - Python modules & packages - probably Julia packages (Johannes, blink twice when you read this) - HPC modules because all dimension-related variants can be shipped _at once_ and _do not produce multiple, conflicting packages_ anymore. Apps can use the same mechanism to pick up the dimensional variants and build multiple artifacts. (This is something we already do, but cannot generalize to package mangers just yet.) ### Backward Compatibility To stay backwards compatible, the *last* (or traditionally, *only*) passed dimensionality will define a few legacy artifacts: - [x] the `amrex::amrex`/`amrex` CMake targets - [x] the `libamrex.[so|a]`/`amrex.[dll|lib]` artifact (symlink) - [x] the `AMReX_Config.H` file: defaults to the last dim in `AMReX_SPACEDIM` and defines `AMREX_SPACEDIM` via a public CLI define now ## Additional background I recently wrote an analysis piece lamenting that [compile time software variants that define exclusive features](https://bssw.io/blog_posts/rethinking-software-variants) are a productivity and usability challenge. One particular challenge we face in the shipping of WarpX and pyAMReX is that users (and developers) would like to seamlessly switch between the dims (1-3D) of our codes. Switching between compile-time and runtime description for such elementary workflows can and should be avoided. Generally, this problem can be solved with more general interfaces in AMReX. Practically, this is a bit of work that we have not yet started. Weiqun: > Other than EB and Linear Solvers, most of the amrex stuff built for 3d could work for 1d and 2d, including i/o. ## Things this does not yet solve As before, this does not solve namespace and/or symbol clashes. We should introduce unique name spaces and rework many globals in AMReX for this, but this is an orthogonal issue. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [x] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Amr/CMakeLists.txt Src/AmrCore/CMakeLists.txt Src/Base/CMakeLists.txt Src/Boundary/CMakeLists.txt Src/CMakeLists.txt Src/EB/CMakeLists.txt Src/Extern/Conduit/CMakeLists.txt Src/Extern/HDF5/CMakeLists.txt Src/Extern/HYPRE/CMakeLists.txt Src/Extern/PETSc/CMakeLists.txt Src/Extern/ProfParser/CMakeLists.txt Src/Extern/SENSEI/CMakeLists.txt Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/amrdata/CMakeLists.txt Src/F_Interfaces/CMakeLists.txt Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/Particle/CMakeLists.txt Tests/Amr/Advection_AmrCore/CMakeLists.txt Tests/Amr/Advection_AmrLevel/CMakeLists.txt Tests/AsyncOut/multifab/CMakeLists.txt Tests/CLZ/CMakeLists.txt Tests/CMakeLists.txt Tests/CMakeTestInstall/CMakeLists.txt Tests/CTOParFor/CMakeLists.txt Tests/EB/CNS/CMakeLists.txt Tests/EB_CNS/CMakeLists.txt Tests/FillBoundaryComparison/CMakeLists.txt Tests/FortranInterface/Advection_F/CMakeLists.txt Tests/FortranInterface/Advection_octree_F/CMakeLists.txt Tests/GPU/CNS/CMakeLists.txt Tests/HDF5Benchmark/CMakeLists.txt Tests/LinearSolvers/ABecLap_SP/CMakeLists.txt Tests/LinearSolvers/ABecLap_SP/MyTest.H Tests/LinearSolvers/ABecLap_SP/MyTest.cpp Tests/LinearSolvers/ABecLaplacian_C/CMakeLists.txt Tests/LinearSolvers/ABecLaplacian_F/CMakeLists.txt Tests/LinearSolvers/NodalPoisson/CMakeLists.txt Tests/LinearSolvers/Nodal_Projection_EB/CMakeLists.txt Tests/LinearSolvers/NodeTensorLap/CMakeLists.txt Tests/MultiBlock/Advection/CMakeLists.txt Tests/MultiBlock/IndexType/CMakeLists.txt Tests/Parser/CMakeLists.txt Tests/Parser2/CMakeLists.txt Tests/Particles/AssignDensity/CMakeLists.txt Tests/Particles/AssignMultiLevelDensity/CMakeLists.txt Tests/Particles/AsyncIO/CMakeLists.txt Tests/Particles/CheckpointRestart/CMakeLists.txt Tests/Particles/DenseBins/CMakeLists.txt Tests/Particles/GhostsAndVirtuals/CMakeLists.txt Tests/Particles/InitFromAscii/CMakeLists.txt Tests/Particles/Intersection/CMakeLists.txt Tests/Particles/NeighborList/CMakeLists.txt Tests/Particles/NeighborParticles/CMakeLists.txt Tests/Particles/ParallelContext/CMakeLists.txt Tests/Particles/ParticleArray/CMakeLists.txt Tests/Particles/ParticleIterator/CMakeLists.txt Tests/Particles/ParticleMesh/CMakeLists.txt Tests/Particles/ParticleMeshMultiLevel/CMakeLists.txt Tests/Particles/ParticleReduce/CMakeLists.txt Tests/Particles/ParticleTransformations/CMakeLists.txt Tests/Particles/Redistribute/CMakeLists.txt Tests/Particles/RedistributeSOA/CMakeLists.txt Tests/Particles/SOAParticle/CMakeLists.txt Tests/Particles/SparseBins/CMakeLists.txt Tests/Particles/TypeDescriptor/CMakeLists.txt Tests/Reinit/CMakeLists.txt Tests/RoundoffDomain/CMakeLists.txt Tests/complementIn/CMakeLists.txt Tools/CMake/AMReXClangTidy.cmake Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXGenerateConfigHeader.cmake Tools/CMake/AMReXInstallHelpers.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/AMReX_Config.H.in Tools/CMake/AMReX_Config.cmake Tools/CMake/AMReX_Config_ND.H.in Tools/CMake/AMReX_ThirdPartyProfilers.cmake commit 32526c32d3176653a1cb43c9e8f32f16ddf91376 Author: Eric T. Johnson Date: Thu May 18 12:00:25 2023 -0400 GNU Make: use CLANG_TIDY_CONFIG_FILE if it's set (#3315) If a downstream project defined `CLANG_TIDY_CONFIG_FILE` with their own config, it didn't actually get added to the clang-tidy arguments. This PR makes `CLANG_TIDY_CONFIG_FILE` default to the amrex config file instead. Tools/GNUMake/tools/Make.clang-tidy commit 23b157942e41d2b8ac4681551816592d2295f83a Author: Weiqun Zhang Date: Wed May 17 16:40:49 2023 -0700 Refactor Parser (#3265) Refactor Parser to allow for further optimization. One issue of the old version is its types have different sizes. That makes it hard to swap nodes that are different types. In the current version, even though the types still have different sizes, the malloc calls allocate enough memory so that we can cast from one type to another. Another issue is that there are a lot more AST node types in the old version than in the current versions. Many of them are introduced for optimization. In the current version, a lot of AST node types including subtraction have been removed. So `(- a b)` (i.e., `a-b`) now becomes `(+ a (* -1 b))`. The seems less efficient. However this makes it much easier to optimize the AST because of the associative property of add. Note that after the AST is generated there is a compilation step that flattens the AST into a sequence of instructions. The removed types (e.g., sub) and special instructions are now added in the compilation step. The new version could be significantly faster for many non-simplifed expressions. For example, `sin(x) + y**2*y*sin(x)` is about 2x faster, because the new version optimizes it to `(1+y**2)*sin(x)`. However, there could be no performance improvement if the expression is already simplified. In addition to the refactor and optimization, we have also added a few new math functions, atan2, asinh, acosh, and atanh, for completeness. I have tested the new version with thousands of expressions. The new version will have roundoff errors with the old version. Docs/sphinx_documentation/source/Basics.rst Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Exe.H Src/Base/Parser/AMReX_Parser_Exe.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_iparser.lex.h Src/Base/Parser/amrex_iparser.lex.nolint.H Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.h Src/Base/Parser/amrex_parser.lex.nolint.H Src/Base/Parser/amrex_parser.tab.nolint.H Src/Base/Parser/amrex_parser.y Tests/CMakeLists.txt Tests/Parser/main.cpp Tests/Parser2/CMakeLists.txt Tests/Parser2/GNUmakefile Tests/Parser2/Make.package Tests/Parser2/fn.cpp Tests/Parser2/main.cpp commit 81e1ac377d54ee78ab6ee1ff40e765600c37861b Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Mon May 15 20:26:31 2023 +0200 Enable binning for ParticleTileData (#3307) ## Summary To use binning with ParticleTileData, replace ``` using PlasmaBins = amrex::DenseBins; ``` with ``` using PlasmaBins = amrex::DenseBins; ``` and ``` bins.build(pti.numParticles(), pti.GetArrayOfStructs().begin(), … ``` with ``` bins.build(pti.numParticles(), pti.GetParticleTile().getParticleTileData(), … ``` Using ParticleTileData is necessary for PureSoA. ## Additional background The previous call_f ``` template AMREX_GPU_HOST_DEVICE auto call_f(F const& f, T* pstruct_ptr, I& index) noexcept -> decltype(f(pstruct_ptr,index)) { return f(pstruct_ptr[index],index); } ``` had an issue where `pstruct_ptr` and `pstruct_ptr[index]` was mixed up. I fixed this here which may cause some backwards compatibility issues. Otherwise binning should work as before when using a `pstruct_ptr`. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Axel Huebl Src/Particle/AMReX_BinIterator.H Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_SparseBins.H commit d448c1784cb86ba286009522c0735b5aad5d0780 Author: Axel Huebl Date: Mon May 15 08:58:31 2023 -0700 ParticleTile: More SoA Updates (#3305) ## Summary More updates to `ParticleTile` and simplifications for pure SoA support. ## Additional background Isolated from #3290 and author credited to @AlexanderSinn. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Co-authored-by: Alexander Sinn Src/Particle/AMReX_ParticleTile.H commit 1a96800dc229e90d41a0a2dea9f6ae608f6c1736 Author: Weiqun Zhang Date: Fri May 12 13:07:05 2023 -0700 HIP Debug Build (#3311) Use -O1 instead of -O0, because the HIP compiler has various issues with the latter. For example, it segfaults when compiling WarpX. It fails with errors of size exceeding limit at link time when compiling Tests/Amr/Advection_AmrCore/Exec. Now both codes work (with GNU Make). Add -munsafe-fp-atomics in debug mode too. The changes are for GNU Make only. Even with `-O1`, CMake still does not work. .github/workflows/codeql.yml Tools/GNUMake/Make.defs Tools/GNUMake/comps/hip.mak commit ab567b81cbf1382fc46f8f1c239d89478f72995c Author: Weiqun Zhang Date: Thu May 11 13:09:48 2023 -0700 Run PODVector::push_back on GPU (#3308) There are two versions of PODVector::push_back. The `const&` version previously ran on CPU, whereas the `&&` version on GPU. It seems that they should have the same behavior. Thus the `const&` version now runs on GPU. (We may not even need to have two versions, because the move (i.e., `&&`) version does not move. It calls another function that takes `const&`.) Src/Base/AMReX_PODVector.H commit 1e2154f614e5ccfd5401d9a21eae8d4d0189214e Author: Axel Huebl Date: Tue May 9 10:17:26 2023 -0700 ParticleInit: Implicit Int Float Conversion (#3304) ## Summary Fix ICX warning: ``` implicit conversion from 'int' to 'amrex::Real' (aka 'float') changes value from 2147483647 to 2147483648 [-Wimplicit-const-int-float-conversion] ``` ## Additional background First seen in https://github.com/ECP-WarpX/impactx/pull/318 during built of pyAMReX. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleInit.H commit 85a89e2653692d85842a6fadc1a0f64960d46b13 Author: Andrew Myers Date: Mon May 8 18:17:56 2023 -0700 More constexpr if for pure soa in RedistributeCPU (#3299) The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleContainerI.H commit bc3e82e9b8d471484076328790fa06dfe7b08f09 Author: Weiqun Zhang Date: Mon May 8 17:56:48 2023 -0700 Cache CodeQL build (#3297) .github/workflows/cleanup-cache.yml .github/workflows/codeql.yml commit 528604e9b2f6bf2261c682de4bd7b244076dc81c Author: Weiqun Zhang Date: Mon May 8 17:56:06 2023 -0700 MLEBNodeFDLaplacian: Implement the alpha term (#3301) In 2d rz, MLEBNodeFDLaplacian solves del dot (simga grad phi) - alpha/r^2 phi = rhs. The alpha term was not implemented previously. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit d01b15ccbd49269b0ae0af58de457d35c57291ad Author: Andrew Myers Date: Mon May 8 17:01:40 2023 -0700 Fixing the ParIter numParticles functions for pure SOA (#3303) The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParIter.H commit 68aa69a11750b3f6ac36ec13b26249f0be80d527 Author: Weiqun Zhang Date: Thu May 4 15:37:47 2023 -0700 Add .mailmap (#3295) The list is incomplete. Feel free to submit a PR to update it. --------- Co-authored-by: Axel Huebl .mailmap commit a393d7ff7e320cefeeb55e31f1b0e0b5ac2d90ca Author: Andrew Myers Date: Thu May 4 12:25:59 2023 -0700 Implement StructOfArray::empty() and ParticleTile::empty() for pure SoA (#3296) The implementation of `ParticleTile::empty()` was wrong for pure SoA, which led to problems in ImpactX: https://github.com/ECP-WarpX/impactx/pull/348/ The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_StructOfArrays.H commit c2f49bf9cd8689d3d4064d57ab10cf1fe1c2b587 Author: Michael Zingale Date: Thu May 4 11:15:01 2023 -0400 port the old BoxLib faverage tool to AMReX (#3293) this can compute the lateral average (optionally density weighted) of a variable from a plotfile. It assumes that the vertical direction is the last dimension. Docs/sphinx_documentation/source/Post_Processing.rst Tools/Plotfile/faverage.cpp commit 0916f68560214a990b220a442fd895c940c92f54 Author: Weiqun Zhang Date: Wed May 3 16:34:01 2023 -0700 GNU Make: add -pthread for HIP (#3294) It appears that hipcc no longer adds -pthread at link time since rocm/5.5. So we need to add it. Also add a CI test using GNU Make to compile HIP code that could have caught this. .github/workflows/dependencies/dependencies_hip.sh .github/workflows/hip.yml Tests/LinearSolvers/NodeEB/MyTest.cpp Tools/GNUMake/comps/hip.mak commit 85a247b8a9d6c1e8edd1aa56d2f2de519dacdcd5 Author: Weiqun Zhang Date: Mon May 1 14:43:40 2023 -0700 Github Cache: Path and Hypre (#3288) Use ~/.cache/ccache instead of ~/.cache as path, because the system might store all kinds of caches in ~/.cache, but we only want to save ccache's cache. (Currently, this is not an issue, but it might be in the future.) Cache hypre cuda build. .github/workflows/clang.yml .github/workflows/cuda.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/hypre.yml .github/workflows/intel.yml .github/workflows/petsc.yml .github/workflows/sundials.yml commit a3e175871e0ebba2a0fd7a96e85481f1b334285d Author: Weiqun Zhang Date: Mon May 1 07:41:59 2023 -0700 Update CHANGES for 23.05 (#3287) CHANGES commit cf6b46bd95a35a9e0d4e5c83ddc8394e6295e123 Author: Michael Zingale Date: Fri Apr 28 23:56:01 2023 -0400 some spell checking on the docs (#3285) Docs/Migration/Migration.md Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/LinearSolvers.rst Docs/sphinx_documentation/source/SUNDIALS_top.rst Docs/sphinx_documentation/source/Visualization.rst commit ee492f47704ae8b094ad9bb4b6f758d82606a61e Author: Andrew Myers Date: Thu Apr 27 12:14:23 2023 -0700 Fix increment for pure SoA (follow on to #3278) (#3284) The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleContainerI.H commit 8d6ef280c7b2180416d1bc9bbb31354cbbc45abd Author: Andrew Myers Date: Thu Apr 27 12:14:04 2023 -0700 Fix RealVect version of pos() for pure SoA (#3283) The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleTile.H commit 40b0ed570dc150e72956349ac72e7571e483d9b4 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Thu Apr 27 12:27:47 2023 -0400 handle divide by 0 in signed distance function (#3279) In some cases we were hitting a division by `c_norm = 0`. This attempts to handle those cases. Src/EB/AMReX_EB_utils.cpp commit a6e89ecc117525acd462a2077103e9f12f02da10 Author: Andrew Myers Date: Wed Apr 26 17:12:11 2023 -0700 Implement increment for pure SoA (#3278) The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleContainerI.H commit a5a0715876ee92105ce1511b9ac98ef6e4689724 Author: Axel Huebl Date: Wed Apr 26 16:25:14 2023 -0700 Work-Around: CUDA 12.1 Non-Default Codelines (#3275) ## Summary Set `AMReX_CUDA_SHOW_CODELINES` to be `FALSE`/`OFF` by default for CUDA 12.1, since we know that `--source-in-ptx` has a NVCC regression in debug mode. ## Additional background - [x] rebase after #3274 was merged - [x] close #3215 Tools/CMake/AMReXCUDAOptions.cmake commit e7e2e1ee8faa8e529ddbe682f10851f00ae93f80 Author: Andrew Myers Date: Wed Apr 26 15:45:10 2023 -0700 Update SortParticlesForDeposition for pure SoA (#3277) I also fixed a typo in the SoA particle `pos` routine that returned a `RealVect`. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleUtil.H commit ba691558ecd709d95aacdaa4ee5eead3f33e73c1 Author: Axel Huebl Date: Wed Apr 26 07:48:13 2023 -0700 CMake: Fix CUDA Options Order (#3274) ## Summary Some options in the CMake CUDA logic check the CUDA version. Thus, the CUDA language needs to be enabled and the compiler be found already. This fixes this order. ## Additional background Preparation of a work-around for #3215 CMakeLists.txt commit 9adadfb8d881b7dd252ce1b00f6570284b8e1e3a Author: Axel Huebl Date: Tue Apr 25 13:58:40 2023 -0700 Doc: Real/ParticleReal Doxygen (#3270) ## Summary Add missing doxygen string, so users can click through on important AMReX types when they look at API documentation. ## Additional background First seen in missing links in https://impactx.readthedocs.io/en/latest/_static/doxyhtml/classimpactx_1_1_impact_x_particle_container.html#a8cabfd7912cd98a8871bc75734474472 Src/Base/AMReX_REAL.H commit adfb055e983bed362c077867f05993e6c2586b96 Author: Weiqun Zhang Date: Tue Apr 25 11:16:35 2023 -0700 Fix bug in EBNodeFDLap (#3271) In this linear solver for WarpX, we missed a corner case where the edge with a length of exactly 1 has one end exactly on the EB surface. It is fixed by using the implicit function to detect covered nodes. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 6a85487dce1bbb4f5d36d3ce1dcd3d04f7797e06 Author: Jean M. Sexton Date: Mon Apr 24 20:33:01 2023 -0400 Generalize some CUDA-specific language in GPU docs (#3272) Docs/sphinx_documentation/source/GPU.rst Tools/GNUMake/Make.defs commit 024a3845b55531c2294849b56f357cccfcd0a820 Author: Weiqun Zhang Date: Mon Apr 24 10:33:10 2023 -0700 Minor updates (#3268) * Update GPU documentation. * Fix clang-tidy warning. * Add hdf to the ignore list in the Fortran dependency script. Docs/sphinx_documentation/source/GPU_Chapter.rst Src/Base/AMReX_GpuAsyncArray.H Tools/F_scripts/dep.py commit 9d2f762f4897b5f585544baf87a19edd0f0c9560 Author: Cyrus Harrison Date: Mon Apr 24 07:42:49 2023 -0700 update ci to use ascent 0.9.1 (#3266) Updates CI Testing to use Ascent 0.9.1 release. .github/workflows/ascent.yml commit 9718b9c135ed493b580d96b32b258ba162939900 Author: Andrew Myers Date: Thu Apr 20 11:23:34 2023 -0700 Ensure that particles are always < rhi after applying periodic bcs. (#3263) We either need to iterate or just fix it this way, which is symmetric with the way rlo is handled. This relies on `rhi` being the largest point that is *inside* the domain, which is consistent with the ASSERT below. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleUtil.H commit b0a037918f78a5e9f32988c4b0016093929954e7 Author: Andrew Myers Date: Tue Apr 18 08:36:04 2023 -0700 Fix default template parameter for ParConstIter (#3262) Follow-on to #2878 The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParIter.H commit c1e8e73b63dc8b6732c6cdfe51914f7ae144575b Author: Axel Huebl Date: Mon Apr 17 18:57:49 2023 -0700 Conduit: Update ParticleTile API (#3260) ## Summary Update to use the new `ParticleTile` template parameters. ## Additional background Follow-up to #2878. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint_ParticlesI.H commit 6b294a5cd3e5c55b36c7d0afe2993501775eb42f Author: thierry <70574092+Thierry992@users.noreply.github.com> Date: Mon Apr 17 11:40:34 2023 -0700 Support SoA-Only Particles (#2878) Add support for storing particle positions and ids SoA style. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [x] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Andrew Myers Co-authored-by: Axel Huebl Src/AmrCore/AMReX_AmrParticles.H Src/Base/AMReX_Array.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_TypeTraits.H Src/EB/AMReX_EB2.H Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_MakeParticle.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_WriteBinaryParticleData.H Src/Particle/CMakeLists.txt Src/Particle/Make.package Tests/Particles/ParticleTransformations/main.cpp Tests/Particles/RedistributeSOA/CMakeLists.txt Tests/Particles/RedistributeSOA/GNUmakefile Tests/Particles/RedistributeSOA/Make.package Tests/Particles/RedistributeSOA/inputs Tests/Particles/RedistributeSOA/inputs.rt Tests/Particles/RedistributeSOA/inputs.rt.cuda Tests/Particles/RedistributeSOA/inputs.rt.cuda.big Tests/Particles/RedistributeSOA/inputs.rt.cuda.mr Tests/Particles/RedistributeSOA/inputs.rt.cuda.nonperiodic Tests/Particles/RedistributeSOA/inputs.rt.cuda.sort Tests/Particles/RedistributeSOA/main.cpp Tests/Particles/SOAParticle/CMakeLists.txt Tests/Particles/SOAParticle/main.cpp commit 27ef4ea56945af7f250a6cec9634ebdcaf863b0b Author: Weiqun Zhang Date: Wed Apr 12 13:41:55 2023 -0700 Roundoff errors in computeRoundoffDomain (#3255) Making changes to let the Intel CI tests pass. This is not really an issue in practice. Src/Base/AMReX_Geometry.cpp commit 2f47d132d71c49abbeac5e081fe9ce886c621d54 Author: Luca Fedeli Date: Wed Apr 12 21:40:31 2023 +0200 Add powi function in AMReX_Math.H (#3251) ## Summary This PR adds a replacement of `std::pow` (`powi`) that is intended to be used when the exponent is integer and known at compile time. If fast math optimizations are not enabled, `powi` can be one order of magnitude faster than `std::pow`. ## Additional background Adding this function to `AMReX_Math.H` was discussed with @WeiqunZhang and @ax3l in https://github.com/ECP-WarpX/WarpX/pull/3824 ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [X] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_Math.H commit bdb1552d8c248f4fccd2e1bb642d4515eff7a31e Author: pramenku <7664080+pramenku@users.noreply.github.com> Date: Wed Apr 12 21:59:55 2023 +0530 update hiprand header path (#3252) Need to update hiprand header path to "" so it will work with future rocm release. It will work with 5.4 release too. So to be ready well ahead with future rocm changes, requesting the change, In file included from /home/master/amr-wind/submods/amrex/Src/Base/AMReX_RandomEngine.H:11: /opt/rocm-xxxx/include/hiprand.hpp:16:2: error: "This file is deprecated. Use the header file from /opt/rocm-xxx/include/hiprand/hiprand.hpp by using #include " #error "This file is deprecated. Use the header file from /opt/rocm-xxxx/include/hiprand/hiprand.hpp by using #include " ^ The proposed changes: It will help user to be ready to use future rocm release alongwith previous releases like 5.4. Src/Base/AMReX_RandomEngine.H commit 1cb5f01a7339b52e98b5d098e956456a6faf79ab Author: Luca Fedeli Date: Wed Apr 12 17:11:58 2023 +0200 Fix a typo in a comment in AMReX_Math.H (#3253) Src/Base/AMReX_Math.H commit 15038d466c05fc607a0dd6ad0c192f8ed025281a Author: Weiqun Zhang Date: Tue Apr 11 13:36:21 2023 -0700 Add Tests/RoundoffDomain (#3248) The new test exposes an issue with Intel compiler. So we have to rewrite the computeRoundofDomain function in a slightly different way to avoid roundoff difference between `auto r = f(x-y)` and `z = x-y; auto r = f(z)`. Also fix an assertion issue in #3247. Src/Base/AMReX_Geometry.cpp Src/Particle/AMReX_ParticleUtil.H Tests/CMakeLists.txt Tests/RoundoffDomain/CMakeLists.txt Tests/RoundoffDomain/GNUmakefile Tests/RoundoffDomain/Make.package Tests/RoundoffDomain/main.cpp commit a54fee8706c812925c4ec7c29cecbc3e068a09bc Author: Weiqun Zhang Date: Mon Apr 10 17:27:26 2023 -0700 Installation using configure: Forgot to install Parser headers (#3249) GNUmakefile.in commit bb47ec9a6e4fbaea36839d94cd2bf7bdba038168 Author: Weiqun Zhang Date: Mon Apr 10 15:50:07 2023 -0700 Fix Roundoff Domain (#3247) There was a flaw in amrex::Geometry::computeRoundoffDomain. If probhi is close to zero, std::nextafter towards negative infinity will be a very small number (e.g., 1.e-300). So the function will be able to find the lowest value of roundoff_hi that's outside the domain within a reasonable number of iterations. In the new implementation, we use bisection to find the lowest value of roundoff_lo that's inside the domain and the highest value fo roundoff_hi that's inside the domain, up to a tolerance. X-ref: - #3243 - #3199 Src/Base/AMReX_Geometry.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H commit 8cae8024915ff8d31fe655665d17783fd34cda8f Author: Weiqun Zhang Date: Sat Apr 8 09:02:40 2023 -0700 Adjust GitHub Cache Size Again (#3246) The sizes seemed to be too small, and we had a lot of cache misses. .github/workflows/clang.yml .github/workflows/cuda.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/hypre.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/sundials.yml .github/workflows/windows.yml commit d21ba63af5269c5ba69f0869473d61ddc3b406c3 Author: Weiqun Zhang Date: Fri Apr 7 17:23:06 2023 -0700 CMake Tests Setup (#3244) Previously most of the tests even if compiled with MPI were run on only one process because they did not specify the number of tasks explicitly in their setup. For the three tests that did specify, they were run twice during CI, first on a single process and then on two processes, due to the setup in Tests/CMakeLists.txt. The setup was similar for OpenMP. This has changed. For tests compiled with OMP, they are run with two OMP threads, otherwise they are run with two MPI processes if they are built with MPI. This should make the CI tests faster. .github/workflows/clang.yml .github/workflows/macos.yml Tests/CMakeLists.txt Tests/Particles/GhostsAndVirtuals/CMakeLists.txt Tests/Particles/InitFromAscii/CMakeLists.txt Tests/Particles/NeighborParticles/CMakeLists.txt Tests/Particles/ParallelContext/CMakeLists.txt Tests/Particles/ParticleReduce/CMakeLists.txt Tests/Particles/Redistribute/CMakeLists.txt Tests/Particles/TypeDescriptor/CMakeLists.txt commit cfc4c822e3c5747c3212b84fcb9eda33302bfff0 Author: Axel Huebl Date: Fri Apr 7 16:27:17 2023 -0700 ParticleTileData: No Restrict in Storage (#3245) ## Summary Remove the `restrict` qualifier from `GpuArray` members in `ParticleTileData`. We see compilation problems with, among others Clang 14.0.6 and AppleClang, for this construct. We expect that using `restrict` (`AMREX_RESTRICT`) on APIs as well as aliased pointers in these arrays before access should be sufficient to prevent that the compiler thinks that individual arrays alias each other in hot loops. ## Additional background - WarpX build regressions since 23.03, e.g., - https://github.com/conda-forge/warpx-feedstock/pull/63 - https://github.com/conda-forge/warpx-feedstock/pull/64 - https://github.com/ECP-WarpX/WarpX/pull/3817 - independently, about to be remove in https://github.com/AMReX-Codes/amrex/pull/2878 Src/Particle/AMReX_ParticleTile.H commit eab8beead3e9a8d7cb2d34e29429144d272a8ce2 Author: Andrew Myers Date: Fri Apr 7 13:12:58 2023 -0700 Fix periodic boundary bug in #3199 (#3243) PR #3199 broke the Nyx regression tests. The issue is we require the particle positions to be in [plo, phi). This won't happen if rhi is exactly equal to phi. The fix is to make sure rhi is always slightly inside the domain. EDIT - fixed this in a different way. Now, when applying periodic boundaries, the particle positions will be set to be strictly less than rhi. Src/Particle/AMReX_ParticleUtil.H commit 2f68f00d5f56d03dc4c9930f688c89c5621c33b6 Author: Axel Huebl Date: Thu Apr 6 19:30:39 2023 -0700 CI: Retry Apt Repos (#3242) Since Intel Repos are often out-of-sync with each others for apt packages provided (checksum errors), we try to pause and retry up to five times to get them installed. Also set apt retries to 3 for `apt update`. .github/workflows/dependencies/dependencies.sh .github/workflows/dependencies/dependencies_clang-tidy.sh .github/workflows/dependencies/dependencies_clang.sh .github/workflows/dependencies/dependencies_dpcpp.sh .github/workflows/dependencies/dependencies_gcc.sh .github/workflows/dependencies/dependencies_hip.sh .github/workflows/dependencies/dependencies_nofortran.sh .github/workflows/dependencies/dependencies_nvcc11.sh .github/workflows/dependencies/dependencies_nvcc12.sh .github/workflows/dependencies/dependencies_nvhpc.sh .github/workflows/dependencies/documentation.sh commit 7d6d87a24988c27a9f0f721917f9b815ab4bf267 Author: haymanpf <73078585+haymanpf@users.noreply.github.com> Date: Fri Apr 7 12:29:46 2023 +1200 Silence SWFFT (#3240) This PR adds a check for AMReX verbosity level to SWFFT output statements. ## Additional background SWFFT somewhat impolitely outputs a lot of information to stdout and stderr. Certainly it's less than ideal to modify external code, but for anyone running an FFT more than once per run, the excess output can be troublesome. Co-authored-by: Peter Hayman Src/Extern/SWFFT/Make.package Src/Extern/SWFFT/TimingStats.h Src/Extern/SWFFT/distribution.c Src/Extern/SWFFT/verbosity.cpp Src/Extern/SWFFT/verbosity.h commit d7052c6751e8c4f7a9712006d4302dc34056675b Author: aveksler1 <124003120+aveksler1@users.noreply.github.com> Date: Thu Apr 6 16:38:05 2023 -0700 Support complete elliptic integrals of 1st and 2nd type in Parser (#3225) We want to use complete elliptic integrals from `cmath`, `comp_ellint_1` and `comp_ellint_2` in the parser in order to make it easier to load in magnetic fields from coils in WarpX. I added these to the parser, and tested against the same functions in scipy - the results match. Since clang does not support special functions in C++17 (https://en.cppreference.com/w/cpp/compiler_support/17#C.2B.2B17_library_features) I implemented it for GCC compilers only, following the structure of how `jn` is implemented in the parser. Co-authored-by: Avigdor Veksler Docs/sphinx_documentation/source/Basics.rst Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_iparser.lex.cpp Src/Base/Parser/amrex_iparser.lex.h Src/Base/Parser/amrex_iparser.lex.nolint.H Src/Base/Parser/amrex_iparser.tab.h Src/Base/Parser/amrex_iparser.tab.nolint.H Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp Src/Base/Parser/amrex_parser.lex.h Src/Base/Parser/amrex_parser.lex.nolint.H Src/Base/Parser/amrex_parser.tab.h Src/Base/Parser/amrex_parser.tab.nolint.H commit ce3401fb0bbed39a879bf5f571c63a1824569613 Author: David Grote Date: Thu Apr 6 14:28:57 2023 -0700 Rework handling of roundoff domain extent (#3199) ## Summary This reworks previous changes that created the round off domain which was done to avoid round off errors when locating particles in the domain, specifically when handling periodic boundaries. This PR changes how this is done so that the roundoff domain makes as small a change as possible. ## Additional background Previous PRs #2839 and #2950 created a round off domain that has a slightly smaller extent that the actual domain. Any particles that ended up outside the round off domain but inside the actual domain were shifted to be on the boundary of the round off domain. The main purpose was to handle problems with periodic boundary with mixed precision (single precision particles and double precision grid). In that case, it is possible for a particle to have `z < lo` and `z+(hi-lo) > hi`. It is also possible for a particle to have `z < hi` but `(z-lo)*dzinv > ihi` putting it in a grid cell beyond the domain. Unfortunately, the roundoff domains was being set further inside that was needed and would affect valid particles that would not have those issues. While the problem is small, the effects can be noticeable when examining simulation properties, for example charge conservation in WarpX. This PR changes the roundoff domain so that only problematic particles are affected. The code now uses `std::nextafter` to find the exact place where the problems occurs. At the lower end, it ensures that the casted position will be greater than ProbLo. At the upper end, it ensures that `(z-lo)*dzinv <= ihi`. One comment is that this assumes that particle grid coordinates will always be calculated at the higher precision, with expressions like `iz = (int)(z - lo)`, without `lo` being cast to `ParticleReal`. Co-authored-by: atmyers Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp commit 1e73fa8ec7c31fbaf994fbb4f59792b90ae78433 Author: Axel Huebl Date: Tue Apr 4 21:43:15 2023 -0700 CI: oneAPI -Wmissing-braces (#3239) ## Summary The latest oneAPI release, 2023.1.10, ships with `-Wmissing-braces` warnings in the public MKL headers... ``` /opt/intel/oneapi/mkl/2023.1.0/include/oneapi/mkl/rng/device/detail/mrg32k3a_impl.hpp:163:33: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces] std::uint32_t temp[3][3] = {1, 0, 0, 0, 1, 0, 0, 0, 1}; ^~~~~~~ { } /opt/intel/oneapi/mkl/2023.1.0/include/oneapi/mkl/rng/device/detail/mrg32k3a_impl.hpp:163:42: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces] std::uint32_t temp[3][3] = {1, 0, 0, 0, 1, 0, 0, 0, 1}; ^~~~~~~ { } /opt/intel/oneapi/mkl/2023.1.0/include/oneapi/mkl/rng/device/detail/mrg32k3a_impl.hpp:163:51: error: suggest braces around initialization of subobject [-Werror,-Wmissing-braces] std::uint32_t temp[3][3] = {1, 0, 0, 0, 1, 0, 0, 0, 1}; ^~~~~~~ { } ``` ## Additional background Broke our CI. attn @rscohn2 FYI - let's report this? :) ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate .github/workflows/intel.yml commit bacaa10a76366ed40b90b422cf5c7da43565281a Author: Weiqun Zhang Date: Sun Apr 2 21:00:31 2023 -0700 Notes on Cache Cleanup (#3237) Add notes on cache cleanup workflows. This could be useful for debugging and reasoning about how things work. Fix a minor issue that only affects forks. To clean up cache left by push events for a feature branch on a fork, we should use github.event.workflow_run.head_branch (i.e., the feature branch) instead of github.ref (i.e., the default branch). This is because the cache is associated with the feature branch, whereas the cleanup workflow is run on the default branch. This is not a problem for the main repo because a direct push is not allowed. In the case of "indrect" push due to a PR merge, the head branch is the default branch for us because we have only one branch. .github/workflows/cache-cleanup-notes.md .github/workflows/cleanup-cache.yml commit 8325f8e2eeaa6cde20d30c4cbf12874f5cb1e548 Author: Weiqun Zhang Date: Sun Apr 2 20:56:58 2023 -0700 Adjust GitHub Cache Size (#3236) Testing using the current development branch shows that ccache will generate about 2 GB data, if we start from scratch. We have adjusted the cache size limit for individual jobs so that the whole set at its limit will use roughly 3 GB of space. This allows us to fit one default branch and 2 additional PR branches' data in the 10 GB available to us on GitHub. .github/workflows/clang.yml .github/workflows/cleanup-cache.yml .github/workflows/cuda.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/hypre.yml .github/workflows/intel.yml .github/workflows/petsc.yml .github/workflows/sundials.yml .github/workflows/windows.yml commit 604af7751cc5588c88101e647a54b468a4c3c742 Author: Weiqun Zhang Date: Fri Mar 31 19:13:10 2023 -0700 Ccache: Use depend mode for Intel SYCL compiler (#3234) The Intel SYCL compiler generates temporary files during compilations that result in cache misses for ccache. The ccache depend mode forgoes the preprocessed file information, and thus works around the issue. .github/workflows/intel.yml Tools/GNUMake/Make.rules Tools/GNUMake/comps/dpcpp.mak commit 615d0880406448bf124ed5a5af2b49152466e0f0 Author: Axel Huebl Date: Fri Mar 31 19:12:09 2023 -0700 Fix: queryAdd for resizing vectors (#3220) ## Summary I think there is a bug in `ParmParse::queryAdd` and `ParmParse::queryarr` for `std::vector` arguments. The passed array is only up-sized if needed, but not down-sized if the key exists. This leads to bugs for long default values and shorter user-provided arrays. Reproducer: ```C++ std::vector cos_coef = ez.default_cos_coef; // long default vector pp_element.queryAdd("cos_coefficients", cos_coef); // can be shorter user input amrex::Print() << "cos_coef.size()" << cos_coef.size() << "\n"; // size of the default vector ``` ## Additional background First seen in https://github.com/ECP-WarpX/impactx/pull/322 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Weiqun Zhang Src/Base/AMReX_ParmParse.H commit 03016933ab7c77fdd16f3de7920c0d1d8d7d14ad Author: Weiqun Zhang Date: Fri Mar 31 18:39:53 2023 -0700 GitHub Cache Management (#3229) The total size of all caches in a GitHub repo is limited to 10 GB. So we need to proactively manage the caches to avoid undesired cache eviction. GitHub cache has its scope associated with branch. When a new PR is opened, the last used cache of the default branch (i.e., development for AMReX) is restored for the PR's actions. After the actions finish, the caches will be saved under a unique name every time and only that PR has access to it. (Presumably this is for security reason to avoid cache poison.) If the actions are run many times, a large number of caches will be accumulated. This uses a lot of space, even though we only need to keep the most recent cache. In this PR, we implement the following approach to keep only the last used cache for each job and remove all caches associated with a PR when it's closed. We have added a workflow that is triggered by the events of completed workflows (e.g., LinuxClang workflow that has a number of jobs using cache.) This new workflow runs on the default branch and it's therefore safe to be granted write permission. Each triggering workflow saves the PR number as an artifact, if it's a pull request run. This is a way to pass the PR number to the cache cleanup workflow, which will clean up old caches associated with the jobs in the triggering workflow. When a PR is closed or merged, a post PR workflow is trigger, and it will in turn trigger another workflow that cleans up all the caches associated with the PR. After the merge, the actions will be run on the default branch. The newly added workflows will also clean up the old caches in the default branch. .github/workflows/clang.yml .github/workflows/cleanup-cache-postpr.yml .github/workflows/cleanup-cache.yml .github/workflows/cuda.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/hypre.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/petsc.yml .github/workflows/post-pr.yml .github/workflows/sundials.yml .github/workflows/windows.yml commit c0e81dc9d0ace42fb412632d7902a3ad98db5dee Author: Axel Huebl Date: Fri Mar 31 18:24:08 2023 -0700 CI: CMake-Easyinstall Unused (#3230) ## Summary Not used anymore. ## Additional background git.io service is discontinued and shows elevated internal error rates. .github/workflows/dependencies/dependencies_hip.sh commit fb022f2cfe3a9a4a25919e3045ad7024e52237f1 Author: Weiqun Zhang Date: Fri Mar 31 17:40:28 2023 -0700 NVHPC CI (#3235) Use the default version whatever that is. Right now, if we explicitly ask for 23.1, the latest version (i.e., 23.3) gets installed as well. Then we run out of space in GitHub CI. .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_nvhpc.sh commit 0ec0d0dd3c0bf68ff61f5abf3493d4eaa4c70d7b Author: Weiqun Zhang Date: Fri Mar 31 14:47:26 2023 -0700 CodeQL: Add query filters (#3233) This should reduce the number of warnings and fix the SARIF error in CI. .github/workflows/codeql.yml .github/workflows/codeql/codeql-config.yml commit 5631b6ee4abdeb3eefac842f889270ed47c17f38 Author: Weiqun Zhang Date: Fri Mar 31 14:38:20 2023 -0700 Update CHANGES for 23.04 (#3232) CHANGES commit d781598bc94b4601c1799e0c15f46595ef204f35 Author: Weiqun Zhang Date: Thu Mar 30 10:41:10 2023 -0700 Speedup clang tidy (#3219) We make ccache save a log file, from which we find out which files need a rebuild. Instead of running clang-tidy on every file, we now only run it on those files that had a miss in ccache. This significantly speeds up clang-tidy in CIs. Note that we treat clang-tidy warnings as errors in CIs. So if there is a hit in the ccache's cache, the file is free of clang-tidy warnings. Also note that we set CCACHE_EXTRAFILES to the clang-tidy configuration file. So any changes to the config file will invalidate the cache as it should. If a CI job successfully passes the compilation stage with ccache enabled, but fails at the clang-tidy stage, the ccache's cache has been updated. One might think this will cause issues because we now have files that have been cached by ccache and meanwhile have failed the clang-tidy checks. However, this is not an issue for Github CIs. This is because ccache's cache will not be saved as a Github cache, if a CI fails. Fix various clang-tidy warnings in Tests/. They were not reported before because clang-tidy was not used on the tests. .github/workflows/clang.yml .github/workflows/gcc.yml .github/workflows/hypre.yml .github/workflows/macos.yml .github/workflows/petsc.yml .github/workflows/sundials.yml Src/AmrCore/AMReX_AmrParticles.H Src/Base/AMReX_Any.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_Lazy.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/Parser/AMReX_Parser_Exe.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_IF_Complement.H Src/EB/AMReX_EB2_IF_Difference.H Src/EB/AMReX_EB2_IF_Extrusion.H Src/EB/AMReX_EB2_IF_Lathe.H Src/EB/AMReX_EB2_IF_Plane.H Src/EB/AMReX_EB2_IF_Polynomial.H Src/EB/AMReX_EB2_IF_Rotation.H Src/EB/AMReX_EB2_IF_Scale.H Src/EB/AMReX_EB2_IF_Translation.H Src/Particle/AMReX_BinIterator.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleArray.H Src/Particle/AMReX_ParticleInterpolators.H Tests/Amr/Advection_AmrCore/Source/AdvancePhiAllLevels.cpp Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tests/Amr/Advection_AmrCore/Source/Src_K/Adv_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/compute_flux_2D_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/compute_flux_3D_K.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d_K.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Amr/Advection_AmrLevel/Source/LevelBldAdv.cpp Tests/AsyncOut/multifab/main.cpp Tests/EB/CNS/Source/CNS.H Tests/EB/CNS/Source/CNS.cpp Tests/EB/CNS/Source/CNSBld.cpp Tests/EB/CNS/Source/CNS_advance.cpp Tests/EB/CNS/Source/CNS_init_eb2.cpp Tests/EB/CNS/Source/main.cpp Tests/LinearSolvers/ABecLap_SP/MyTestPlotfile.cpp Tests/LinearSolvers/NodalPoisson/MyTestPlotfile.cpp Tests/LinearSolvers/NodeTensorLap/CMakeLists.txt Tests/MultiBlock/Advection/main.cpp Tests/MultiBlock/IndexType/main.cpp Tests/Parser/main.cpp Tests/Particles/AssignDensity/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/AsyncIO/main.cpp Tests/Particles/CheckpointRestart/main.cpp Tests/Particles/DenseBins/main.cpp Tests/Particles/InitFromAscii/main.cpp Tests/Particles/Intersection/main.cpp Tests/Particles/NeighborList/main.cpp Tests/Particles/NeighborParticles/MDParticleContainer.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/main.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleIterator/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/main.cpp Tests/Particles/ParticleReduce/main.cpp Tests/Particles/ParticleTransformations/main.cpp Tests/Particles/Redistribute/main.cpp Tests/Particles/SparseBins/main.cpp Tools/C_scripts/mmclt.py Tools/GNUMake/Make.rules Tools/Plotfile/fboxinfo.cpp Tools/Plotfile/fcompare.cpp Tools/Plotfile/fextract.cpp Tools/Plotfile/fsnapshot.cpp commit b41c44166e12705cc08657d6bc33744f23bc8741 Author: Weiqun Zhang Date: Wed Mar 29 13:23:08 2023 -0700 Use ccache in Github Actions (#3218) This works very well for CUDA and HIP builds. However, the benefit is relatively small for GCC, Clang and Mac builds because most of their time are spent on clang-tidy and running tests. There are two remaining issues. Ccache works with Intel classic compiler icpc, but it does not work with the Intel SYCL compiler, icpx. On Windows, MSVC works, but I cannot get ClangCl to work. We can revisit these two issues in the future. In earlier versions of this PR, we compiled ccache from source because the package version on Ubuntu 20.04 is too old. Because of that, we had to move `env: {CXXFLAGS: ...}` down to AMReX's `Build & Install` section to avoid the flags being picked up when compiling ccache. In the current version, we download the binary version of ccache 4.8. However, we did not move the env setup back. It seems to make sense to keep it in the `Build & Install` section. To use github cache, we need to provide a primary key and optionally restore keys. If the primary key is found, it will be used, but the change to the cache during the CI will NOT be saved at the end of the job. That's how github cache works. If it is not found, partially matched restore keys will be considered and the latest one will be used. At the end, the cache (whether it's updated or not) will be saved under the name of the primary key. Below is an example of our setup. ``` with: path: ~/.cache key: ccache-${{ github.workflow }}-${{ github.job }}-git-${{ github.sha }} restore-keys: | ccache-clang.yml-${{ env.cache-name }}-git- ``` Here, we use the workflow string and job string as part of the names for the keys. We also include the git commit hash `${{github.sha}}` in the name of the primary key. In this setup, the CI will always fail to find the primary key. This is what we want because it guarantees the latest ccache data will be saved. Furthermore, the most recently saved cache from the last CI job will be used to restore the cache during initialization. .github/workflows/clang.yml .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_ccache.sh .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/hypre.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/petsc.yml .github/workflows/sundials.yml .github/workflows/windows.yml commit 831d1487fa19feb04b4330976a8dd90c604624c5 Author: Ann Almgren Date: Tue Mar 28 21:13:53 2023 -0700 fixes to make clang-tidy happy (#3227) Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EBDataCollection.cpp commit 4478d678c13ea00f1163385e05fd5cd09bf9d78e Author: Ann Almgren Date: Tue Mar 28 16:47:47 2023 -0700 fix for clang-tidy (#3224) Src/Base/AMReX_FabArray.H commit 6f10c63358d1e7d93dbce000dad1d09ae77c0931 Author: hsitaram Date: Tue Mar 28 15:49:20 2023 -0600 bug fix in creating EB from STL files (#3223) ## Summary Triangle data from an STL file needs to be broadcasted across MPI ranks ## Additional background The STL file is currently read only by ioprocessor and the tri_data on host is set only on ioprocessor. FillFab function would only fill the correct data for processor 0 Co-authored-by: Hariswaran Sitaraman Src/EB/AMReX_EB_STL_utils.cpp commit 27ebdfbbbb428b27d1608677f1afed712bb4bda6 Author: Ann Almgren Date: Tue Mar 28 13:49:46 2023 -0700 fixes suggested by clang-tidy (#3222) Src/Base/AMReX_BCRec.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_IntegratorBase.H commit 4fab19a02359d05f011b5a5ee4aacfda4b86f640 Author: Lucas Esclapez <13371051+esclapez@users.noreply.github.com> Date: Mon Mar 27 08:56:27 2023 -0700 Fix bug in MLEBABecLap getEBFluxes(). (#3217) Bug introduced when adding anisotropy to MLEBABecLap, bareascaling goes on the denominator when calculating the EBfluxes. Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H commit 7aad6ca7f64956c136b68d5fedfbc5e28ab1aab7 Author: Weiqun Zhang Date: Thu Mar 23 10:44:32 2023 -0700 Add comments to FluxRegister::OverwriteFlux (#3216) Src/AmrCore/AMReX_FluxRegister.H commit 82e38a8f5ebef4622805db256183cdf01970326d Author: Weiqun Zhang Date: Wed Mar 22 16:53:51 2023 -0700 Add build option for Intel SYCL AOT GRF mode (#3173) By default, the Intel SYCL compiler uses up to 128 registers. In the large register file mode, it's up to 256 registers. There is also the auto-large mode. For CMake, one can use `-DAMReX_SYCL_AOT_GRF_MODE=[Large|AutoLarge]` to choose the large register file or auto-large mode. AOT must also be enabled with `-DAMReX_SYCL_AOT=TRUE`. For GNU Make, one can use `SYCL_AOT_GRF_MODE=[Large|AutoLarge]`, and it depends on AOT being enabled by `SYCL_AOT=TRUE`. Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit ea670122d5956506aa66827193ad601c86b72d06 Author: Weiqun Zhang Date: Wed Mar 22 15:38:48 2023 -0700 SYCL sub-group size (#3180) Add option to specify SYCL sub-group size. The default is 32. It can be changed to 16 with `-DAMReX_SYCL_SUB_GROUP_SIZE=16` in CMake and `SYCL_SUB_GROUP_SIZE=16` in GNU Make. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_GpuDevice.H Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs commit f7d36cb62ca480c41d866e7db702064ec4912539 Author: Weiqun Zhang Date: Wed Mar 22 14:56:19 2023 -0700 Warning about CPU core oversubscription (#3211) If the user forgets to set OMP_NUM_THREADS, OMP might be oversubscribing CPU cores. If that's the case (to the best of our knowledge) and verbosity is on, we print out a warning. We use `std::thread::hardware_concurrency` to detect the number of cores available. Note that this is only a hint according to the C++ standard. To detect the number of MPI ranks per node, we use OMP_COMM_TYPE_NODE for Open MPI and MPI_COMM_TYPE_SHARED for others. Since MPI_COMM_TYPE_SHARED might group processes across nodes, the return value of `ParallelDescriptor::NProcsPerNode` might also be wrong. Nevertheless, both `std::thread::hardware_concurrency` and `ParallelDescriptor::NProcsPerNode` seem to work well. Src/Base/AMReX.cpp Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp commit 9d4ca1ed28c3833ca5564ab5e695c3aa5f40ad7f Author: Weiqun Zhang Date: Wed Mar 22 12:39:52 2023 -0700 Clang-Tidy VII (#3203) Fix warnings in the interfaces for Hypre, PETSc and SUNDIALS. .github/workflows/clang.yml .github/workflows/dependencies/dependencies_clang7.sh .github/workflows/hypre.yml .github/workflows/petsc.yml .github/workflows/sundials.yml Src/Base/AMReX_CuptiTrace.cpp Src/EB/AMReX_EB2_IF_Difference.H Src/EB/AMReX_EB2_IF_Lathe.H Src/EB/AMReX_EB2_IF_Scale.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_Hypre.cpp Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.H Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/AMReX_PETSc.cpp Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Src/Extern/SUNDIALS/AMReX_SUNMemory.H Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp Src/Extern/SUNDIALS/AMReX_Sundials_Core.H Src/Extern/SUNDIALS/AMReX_Sundials_Core.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleIO.H Tests/LinearSolvers/ABecLaplacian_C/MyTest.cpp Tests/LinearSolvers/ABecLaplacian_C/MyTestPlotfile.cpp Tests/LinearSolvers/CellEB/MyEB.H Tools/GNUMake/packages/Make.hypre Tools/GNUMake/packages/Make.petsc commit f22705a9a2c63e13e5abb0dd6d99dd3158c911ed Author: Weiqun Zhang Date: Tue Mar 21 18:29:07 2023 -0700 Clang-Tidy VI (#3202) * Update GNU Make. One can use `USE_CLANG_TIDY=TRUE` to enable clang-tidy check. For example, to enable clang-tidy check, specify the clang-tidy command (if it's the default clang-tidy), and treat warnings as errors, one can run make USE_CLANG_TIDY=TRUE CLANG_TIDY=clang-tidy-12 CLANG_TIDY_WARN_ERROR=TRUE By default amrex/.clang-tidy is used as the config file. One can change it with CLANG_TIDY_CONFIG_FILE=my_clang-tidy-conf. If you want to let clang-tidy fix errors for you, you can run make USE_CLANG_TIDY=TRUE CLANG_TIDY="clang-tidy --fix-errors" To avoid race conditions, the make job should not use `-j` (unless it's `-j1`) when fixing errors. * Add clang-tidy to GNU Make CIs. * Fix warnings in Fortran interface, particle and plotfile tools. .github/workflows/clang.yml .github/workflows/gcc.yml Src/AmrCore/AMReX_AmrParticles.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_Box.H Src/Base/AMReX_PODVector.H Src/F_Interfaces/AmrCore/AMReX_FAmrCore.H Src/F_Interfaces/AmrCore/AMReX_FAmrCore.cpp Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.H Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.cpp Src/F_Interfaces/AmrCore/AMReX_fillpatch_fi.cpp Src/F_Interfaces/Base/AMReX_FPhysBC.H Src/F_Interfaces/Base/AMReX_init_fi.cpp Src/F_Interfaces/LinearSolvers/AMReX_abeclaplacian_fi.cpp Src/F_Interfaces/LinearSolvers/AMReX_poisson_fi.cpp Src/F_Interfaces/Octree/AMReX_octree_fi.cpp Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_TracerParticles.H Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/sites/Make.unknown Tools/GNUMake/tools/Make.clang-tidy Tools/Plotfile/AMReX_PPMUtil.cpp Tools/Plotfile/fboxinfo.cpp Tools/Plotfile/fcompare.cpp Tools/Plotfile/fextrema.cpp Tools/Plotfile/fsnapshot.cpp Tools/Plotfile/fvolumesum.cpp commit d50cf55330b18a320754b5a85bbaca244bff62b7 Author: Weiqun Zhang Date: Tue Mar 21 11:36:58 2023 -0700 Clang-Tidy V (#3201) This is the fifth batch of changes fixing clang-tidy warnings. The changes are most related to single precision, 1D, 2D, and OMP. Enable clang-tidy in GCC and Clang CIs. Switch to Ubuntu 22.04 and GCC 12 in one test. .clang-tidy .github/workflows/clang.yml .github/workflows/dependencies/dependencies_clang-tidy.sh .github/workflows/dependencies/dependencies_clang.sh .github/workflows/dependencies/dependencies_gcc.sh .github/workflows/dependencies/dependencies_gcc8.sh .github/workflows/gcc.yml .github/workflows/sundials.yml Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_AuxBoundaryData.cpp Src/Amr/AMReX_Extrapolater.H Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/Amr/AMReX_extrapolater_2D_K.H Src/Amr/AMReX_extrapolater_3D_K.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_InterpFaceReg_2D_C.H Src/AmrCore/AMReX_InterpFaceReg_3D_C.H Src/AmrCore/AMReX_InterpFaceRegister.cpp Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/Base/AMReX_BaseFab.cpp Src/Base/AMReX_Box.H Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp Src/Base/AMReX_CoordSys.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FilCC_C.cpp Src/Base/AMReX_FileSystem.H Src/Base/AMReX_FileSystem.cpp Src/Base/AMReX_Geometry.H Src/Base/AMReX_Gpu.H Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuKernelInfo.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_LayoutData.H Src/Base/AMReX_MFParallelForC.H Src/Base/AMReX_MFParallelForG.H Src/Base/AMReX_MPMD.H Src/Base/AMReX_Math.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H Src/Base/AMReX_MultiFabUtil_nd_C.H Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_OpenMP.H Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_PhysBCFunct.cpp Src/Base/AMReX_RealBox.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_RungeKutta.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_ccse-mpi.H Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_YAFluxRegister.H Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Graph.H Src/EB/AMReX_EB2_IF_AllRegular.H Src/EB/AMReX_EB2_IF_Box.H Src/EB/AMReX_EB2_IF_Complement.H Src/EB/AMReX_EB2_IF_Cylinder.H Src/EB/AMReX_EB2_IF_DevicePtr.H Src/EB/AMReX_EB2_IF_Difference.H Src/EB/AMReX_EB2_IF_Ellipsoid.H Src/EB/AMReX_EB2_IF_Extrusion.H Src/EB/AMReX_EB2_IF_Intersection.H Src/EB/AMReX_EB2_IF_Lathe.H Src/EB/AMReX_EB2_IF_Parser.H Src/EB/AMReX_EB2_IF_Plane.H Src/EB/AMReX_EB2_IF_Polynomial.H Src/EB/AMReX_EB2_IF_Rotation.H Src/EB/AMReX_EB2_IF_Scale.H Src/EB/AMReX_EB2_IF_Sphere.H Src/EB/AMReX_EB2_IF_Spline.H Src/EB/AMReX_EB2_IF_Torus.H Src/EB/AMReX_EB2_IF_Translation.H Src/EB/AMReX_EB2_IF_Union.H Src/EB/AMReX_EB2_IndexSpace_STL.H Src/EB/AMReX_EB2_IndexSpace_chkpt_file.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level_STL.H Src/EB/AMReX_EB2_Level_chkpt_file.H Src/EB/AMReX_EB2_MultiGFab.H Src/EB/AMReX_EBFluxRegister.cpp Src/EB/AMReX_EBFluxRegister_2D_C.H Src/EB/AMReX_EBFluxRegister_3D_C.H Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_EB_LeastSquares_3D_K.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_chkpt_file.H Src/EB/AMReX_EB_chkpt_file.cpp Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_algoim.H Src/EB/AMReX_algoim_K.H Src/EB/AMReX_distFcnElement.cpp Src/Extern/HYPRE/AMReX_Habec_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMGBndry.H Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_eb.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp_grad.cpp Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp Src/LinearSolvers/OpenBC/AMReX_OpenBC_K.H Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_ParticleUtil.cpp Src/Particle/AMReX_Particle_mod_K.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.cpp commit 5f8030af0ddfe99139968bcf896b5662cccf94a4 Author: Michael Zingale Date: Sun Mar 19 18:34:07 2023 -0400 fix typo in debugging docs (#3210) ## Summary ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Docs/sphinx_documentation/source/Debugging.rst commit c4e2bbdf777e11bd6a99cd28cc00b409592c9002 Author: Weiqun Zhang Date: Fri Mar 17 20:28:10 2023 -0700 Add Jean's name to the development team (#3209) .zenodo.json commit fe0ae235cd39697b1d9d6bcc9df6bbac4e0ead86 Author: Weiqun Zhang Date: Fri Mar 17 14:31:57 2023 -0700 Fix issues in ParticleContainerBase (#3208) * The move constructor and assignment operator were not safe, because the m_gdb member could be the address of m_gdb_object. After the move that can no longer be true. This is fixed by making m_gdb_object a unique_ptr. Moving a unique_ptr does not change its pointer inside. * Fix SetParticleBoxArray, SetParticleDistributionMapping and SetParticleGeometry. The issue was these functions took a reference, which could be aliasing what was inside the m_gdb_object. ParGDB has a move assignment operator that could render the reference pointing to an invalid object. So now these functions take by value. This was not an issue before #3200, because the move assignment of ParGDB did copy instead of move. Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp commit 254d3faba017c9b13800d68499abcfe7d0405128 Author: Weiqun Zhang Date: Thu Mar 16 16:45:12 2023 -0700 Fix #3200 (#3206) It was a mistake to delete the move assignment operator of ParticleIDWrapper and ParticleCPUWrapper. Src/Particle/AMReX_Particle.H commit adcf4d4721b44bd47e4de614f62ddeffd6d71a47 Author: Weiqun Zhang Date: Thu Mar 16 08:18:22 2023 -0700 Clang-Tidy IV (#3200) This is the fourth batch of changes fixing clang-tidy warnings. Most of them are in particle code. Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrParGDB.H Src/Base/AMReX_GpuAtomic.H Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleBufferMap.H Src/Particle/AMReX_ParticleBufferMap.cpp Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleMPIUtil.cpp Src/Particle/AMReX_ParticleUtil.cpp Src/Particle/AMReX_StructOfArrays.H Src/Particle/AMReX_TracerParticles.H Src/Particle/AMReX_TracerParticles.cpp Src/Particle/AMReX_WriteBinaryParticleData.H commit 0f4f9877c81e827eb850ba5dc6f0980360552e04 Author: Weiqun Zhang Date: Wed Mar 15 13:59:10 2023 -0700 Clang-Tidy III (#3197) This is the third batch of changes fixing clang-tidy warnings. Most of the warnings are in EB code. Use Clang 14 in one of the Clang CI tests that used to use Clang 7. .clang-tidy .github/workflows/clang.yml .github/workflows/dependencies/dependencies_clang.sh Src/Amr/AMReX_AmrLevel.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_PlotFileUtil.cpp Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Graph.H Src/EB/AMReX_EB2_IF_Box.H Src/EB/AMReX_EB2_IF_Cylinder.H Src/EB/AMReX_EB2_IF_Ellipsoid.H Src/EB/AMReX_EB2_IF_Parser.H Src/EB/AMReX_EB2_IF_Sphere.H Src/EB/AMReX_EB2_IF_Torus.H Src/EB/AMReX_EB2_IndexSpace_STL.H Src/EB/AMReX_EB2_IndexSpace_STL.cpp Src/EB/AMReX_EB2_IndexSpace_chkpt_file.H Src/EB/AMReX_EB2_IndexSpace_chkpt_file.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EB2_Level_STL.cpp Src/EB/AMReX_EB2_Level_chkpt_file.cpp Src/EB/AMReX_EB2_MultiGFab.cpp Src/EB/AMReX_EB2_ND_C.cpp Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBCellFlag.cpp Src/EB/AMReX_EBFArrayBox.H Src/EB/AMReX_EBFArrayBox.cpp Src/EB/AMReX_EBFabFactory.H Src/EB/AMReX_EBFabFactory.cpp Src/EB/AMReX_EBFluxRegister.H Src/EB/AMReX_EBInterpolater.H Src/EB/AMReX_EBInterpolater.cpp Src/EB/AMReX_EBMFInterpolater.H Src/EB/AMReX_EBMFInterpolater.cpp Src/EB/AMReX_EBMultiFabUtil.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBToPVD.H Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_chkpt_file.H Src/EB/AMReX_EB_chkpt_file.cpp Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_MultiCutFab.H Src/EB/AMReX_MultiCutFab.cpp Src/EB/AMReX_WriteEBSurface.cpp Src/EB/AMReX_algoim.cpp Src/EB/AMReX_distFcnElement.H Src/EB/AMReX_distFcnElement.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_F.cpp Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp_bc.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp commit fdcbb7a0152baf96e81a500035fd139e9c11f40c Author: Andrew Myers Date: Wed Mar 15 13:58:18 2023 -0700 Add assert that we are not inside threaded region when InitRandom is called. (#3204) Src/Base/AMReX_Random.cpp commit d54067a4857042be858a36a346f5fe7a6fe7f99a Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Wed Mar 15 21:39:44 2023 +0100 Sorting for faster current deposition (#3198) ## Summary For current deposition in codes such as HiPACE++ and WarpX, it is mush better to sort by `x -> y -> z -> ppc` instead of `ppc -> x -> y -> z`. ## Additional background To use this, pass in {0,0,0} (sort by cell), {-1,-1,-1} (sort by node) or {-2, -2, -2} (sort by cell and node) as a bin_size into SortParticlesByBin. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H commit aaab804f1438b5eeef3f613ca527c62bcabb5425 Author: David Gardner Date: Mon Mar 13 19:29:44 2023 -0700 Update SUNDIALS NVector constructors (#3165) ## Summary Update MultiFab NVector wrappers to accept a sundials context object. ## Additional background Profiling (and in the future error handling) in sundials requires objects to be create with a non-NULL context. This PR updates the MultiFab NVector wrappers with an optional input for the sundials context that defaults to the context created at initialization for the corresponding OpenMP thread. Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Src/Extern/SUNDIALS/AMReX_SUNMemory.H Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp Src/Extern/SUNDIALS/AMReX_Sundials.H Src/Extern/SUNDIALS/AMReX_Sundials_Core.H Src/Extern/SUNDIALS/AMReX_Sundials_Core.cpp Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package commit 6d30e83c944e4e6167178f2d145df2a3e67d2b24 Author: Weiqun Zhang Date: Fri Mar 10 20:48:40 2023 -0800 Clang-Tidy II (#3190) This is the second batch of changes fixing clang-tidy warnings. Add CMake option AMReX_CLANG_TIDY_WERROR to update warnings to errors. Use clang-tidy in CI: GNU@8.4 C++17 Release [lib]. .clang-tidy .github/workflows/gcc.yml Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_AuxBoundaryData.H Src/Amr/AMReX_AuxBoundaryData.cpp Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Derive.cpp Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FillPatcher.H Src/AmrCore/AMReX_InterpBase.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_TagBox.H Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_Array.H Src/Base/AMReX_BackgroundThread.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_Box.cpp Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabFactory.H Src/Base/AMReX_ForkJoin.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_LayoutData.H Src/Base/AMReX_MFCopyDescriptor.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_MPMD.H Src/Base/AMReX_MPMD.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBC.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_PCI.H Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParallelReduce.H Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_Utility.cpp Src/Base/AMReX_VisMF.H Src/Base/AMReX_VisMF.cpp Src/Base/Parser/AMReX_IParser.H Src/Base/Parser/AMReX_Parser.H Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_Mask.H Src/Boundary/AMReX_Mask.cpp Src/Boundary/AMReX_YAFluxRegister.H Src/Extern/amrdata/AMReX_AmrData.cpp Src/Extern/amrdata/AMReX_DataServices.cpp Src/F_Interfaces/Base/AMReX_init_fi.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp_F.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/OpenBC/AMReX_OpenBC.H Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp Tools/CMake/AMReXClangTidy.cmake Tools/CMake/AMReXOptions.cmake commit 5b7fa9867d0260fc3407ee9404927c5c99fc691e Author: Weiqun Zhang Date: Tue Mar 7 18:27:30 2023 -0800 Fix an issue in #3176 (#3188) We need to add another version of SetTag now, otherwise it breaks WarpX. Src/Base/AMReX_FabArray.H commit 041a3ce16cd274bbe3eff012ab4d2d78db009b31 Author: Weiqun Zhang Date: Tue Mar 7 13:42:27 2023 -0800 ReduceToPlane (#3187) * Add a function template for reducing MultiFab data to a plane. * Add KeyValuePair as an alias to ValLocPair. * Add more versions of MPI reduce functions for KeyValuePair/ValLocPair. Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_ParallelReduce.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_ValLocPair.H commit 184d997fa1a7023e7a79ce6ef887ed755ab9489d Author: Weiqun Zhang Date: Tue Mar 7 13:05:48 2023 -0800 Clang-Tidy (#3176) Add a configuration file .clang-tidy for clang-tidy. Add support to it in CMake. It can be enabled by `-DAMReX_CLANG_TIDY=ON`. Fix many warnings raised by clang-tidy. Most of them are fixed by `clang-tidy --fix-errors`. To bypass clang-tidy, some unfixable codes have been moved to `*.nolint.H`. A number of issues have been found and fixed. * There was use-after-move in testing OpenMP thread safety for communication. * The MultiFab tags were not handled correctly during vector resize. * Parentheses were missing in the definition of AMREX_REAL_LOWEST. .clang-tidy .github/workflows/style/check_tabs.sh .github/workflows/style/check_trailing_whitespaces.sh Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_AuxBoundaryData.cpp Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Derive.cpp Src/Amr/AMReX_Extrapolater.cpp Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FluxRegister.H Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_InterpBase.H Src/AmrCore/AMReX_InterpBase.cpp Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_TagBox.H Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_Algorithm.H Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_Array.H Src/Base/AMReX_Array4.H Src/Base/AMReX_AsyncOut.cpp Src/Base/AMReX_BCRec.H Src/Base/AMReX_BC_TYPES.H Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseFab.cpp Src/Base/AMReX_Box.H Src/Base/AMReX_Box.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxDomain.cpp Src/Base/AMReX_BoxIterator.H Src/Base/AMReX_BoxList.H Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp Src/Base/AMReX_CoordSys.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_DataAllocator.H Src/Base/AMReX_Demangle.H Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabConv.H Src/Base/AMReX_FabConv.cpp Src/Base/AMReX_FabFactory.H Src/Base/AMReX_FileSystem.H Src/Base/AMReX_FileSystem.cpp Src/Base/AMReX_ForkJoin.H Src/Base/AMReX_ForkJoin.cpp Src/Base/AMReX_Geometry.H Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuControl.cpp Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuElixir.H Src/Base/AMReX_GpuElixir.cpp Src/Base/AMReX_GpuKernelInfo.H Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunch.nolint.H Src/Base/AMReX_GpuLaunchMacrosC.H Src/Base/AMReX_GpuLaunchMacrosC.nolint.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuLaunchMacrosG.nolint.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_GpuRange.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_GpuUtility.cpp Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_IndexType.H Src/Base/AMReX_IntConv.cpp Src/Base/AMReX_IntVect.cpp Src/Base/AMReX_Loop.H Src/Base/AMReX_Loop.nolint.H Src/Base/AMReX_MFCopyDescriptor.H Src/Base/AMReX_MFCopyDescriptor.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_MPMD.H Src/Base/AMReX_MPMD.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBC.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_Orientation.H Src/Base/AMReX_PArena.cpp Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParallelReduce.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_Periodicity.cpp Src/Base/AMReX_PlotFileDataImpl.H Src/Base/AMReX_PlotFileDataImpl.cpp Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_PlotFileUtil.cpp Src/Base/AMReX_REAL.H Src/Base/AMReX_Random.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_Utility.H Src/Base/AMReX_Utility.cpp Src/Base/AMReX_Vector.H Src/Base/AMReX_VectorIO.cpp Src/Base/AMReX_VisMF.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_iMultiFab.cpp Src/Base/AMReX_parmparse_fi.cpp Src/Base/CMakeLists.txt Src/Base/Make.package Src/Base/Parser/AMReX_IParser.H Src/Base/Parser/AMReX_IParser.cpp Src/Base/Parser/AMReX_IParser_Exe.H Src/Base/Parser/AMReX_IParser_Exe.cpp Src/Base/Parser/AMReX_IParser_Y.H Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Exe.H Src/Base/Parser/AMReX_Parser_Exe.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/GNUmakefile Src/Base/Parser/amrex_iparser.lex.cpp Src/Base/Parser/amrex_iparser.lex.nolint.H Src/Base/Parser/amrex_iparser.tab.cpp Src/Base/Parser/amrex_iparser.tab.nolint.H Src/Base/Parser/amrex_parser.lex.cpp Src/Base/Parser/amrex_parser.lex.nolint.H Src/Base/Parser/amrex_parser.tab.cpp Src/Base/Parser/amrex_parser.tab.nolint.H Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_Mask.H Src/Boundary/AMReX_Mask.cpp Src/Boundary/AMReX_MultiMask.cpp Src/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/OpenBC/AMReX_OpenBC.H Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp Tools/CMake/AMReXClangTidy.cmake Tools/CMake/AMReXOptions.cmake commit 1ffc6e427e3aea9302602190c8fc1cfbea0c9725 Author: Weiqun Zhang Date: Mon Mar 6 08:48:05 2023 -0800 ParticleContainer: Resize runtime components (#3186) Add two new functions ResizeRuntimeRealComp and ResizeRuntimeIntComp to ParticelContainer. Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerI.H commit 6a7da99093780cc9b93b20e284f478018014ec63 Author: Nuno Miguel Nobre Date: Sun Mar 5 18:54:32 2023 +0000 SYCL: Tweak #3164 and switch to multi_ptr for sincos (#3185) * Use std::isnan and std::isinf instead of ::isnan and ::isinf for HIP and CUDA. * Add sycl::isinf and sycl::isfinite back to amrex::Math because Intel compiler's aggressive fp mode makes std::isinf and std::isfinite always return false in release build. * Use sycl::isinf and sycl::isnan in amrex::isnan and amrex::isinf when SYCL is on for the same reason as above. * Fix call to sycl::sincos. According to the SYCL standard the pointer should be a multi_ptr instead of a raw pointer. Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_Math.H commit b96f6e5bd13d156998edead7d92b21235bbea99d Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Wed Mar 1 20:33:34 2023 +0100 Fix race condition with TinyProfiler for memory (#3169) Fix #3156 by giving every Arena its own map for profiling. Src/Base/AMReX.cpp Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit 3ffd848fca27b8ca8a4938e44a9a0dd64095328a Author: Weiqun Zhang Date: Wed Mar 1 07:59:55 2023 -0800 Update CHANGES for 23.03 (#3183) CHANGES commit d0d2fb461204d1c014007ec669f109c5ba441ffd Author: Weiqun Zhang Date: Wed Mar 1 05:35:36 2023 -0800 SYCL: Fix #3172 (#3181) This may not be our bug. But using sycl::reqd_work_group_size does not work for the 2d range cases. So we have to remove that. Src/Base/AMReX_GpuLaunchMacrosG.H commit 32a65ffa8aba81ca6fc8e527cca1cff7e6a5c2ba Author: Michael Zingale Date: Mon Feb 27 10:46:26 2023 -0500 fix recently introduced tmp_build_dir build bug (#3179) the make variable was renamed from TmpBuildDir to TMP_BUILD_DIR breaking application codes. This adds an alias to the old make variable name so code compile again. addresses bug in #3177 closes #3178 Tools/GNUMake/Make.defs commit ef5c8b765f7821cf10f1040f122474b0344b796c Author: Miren Radia <32646026+mirenradia@users.noreply.github.com> Date: Sun Feb 26 21:13:53 2023 +0000 GNU Make: Allow customizing tmp_build_dir (#3177) ## Summary This allows the user to change the default temporary build directory to something other than `tmp_build_dir`. ## Additional background This would allow for users to build multiple AMReX applications using the same build directory and thus reduce compile times without having to use `ccache`. Of course, there might be a small risk in terms of users potentially accidentally mixing build configurations hence I would suggest not advertising this feature or at least advising caution. Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules commit 8014c80f478256adb99ac4bf04ca79e0f8ce3b79 Author: Miren Radia <32646026+mirenradia@users.noreply.github.com> Date: Sun Feb 26 21:10:35 2023 +0000 GNU Make: Allow choice of Intel compiler variant (#3175) ## Summary This allows the user to specify `COMP = intel-classic` or `COMP = intel-llvm` if they wish to use a specific one. ## Additional background Note that the existing behaviour is maintained if `COMP = intel` (i.e. intel-llvm is chosen if the `icpx` command exists and intel-classic is chosen otherwise following #3126). However, the `AMREX_FCOMP`, `AMREX_CCOMP` and `lowercase_comp` makefile variables are changed to one of `intel-llvm` or `intel-classic` depending on which is used. This further allows the user to use information about the used variant in their `Make.local`. For example they may wish to set something like ``` CXX = mpiicpc -cxx=icpx ``` in the case `AMREX_CCOMP = intel-llvm`, `USE_MPI = TRUE` and they are using Intel MPI. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [x] include documentation in the code and/or rst files, if appropriate Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/GNUMake/Make.defs Tools/GNUMake/comps/intel-classic.mak Tools/GNUMake/comps/intel-llvm.mak Tools/GNUMake/tools/Make.vtune Tools/libamrex/configure.py Tools/libamrex/mkconfig.py commit e42172a2bcd041bfad3df41af0b9c87ba574f93b Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Sun Feb 26 14:54:26 2023 -0500 Hypre setup and solve profile (#3168) I've wrapped the Hypre setup and solve functions so they can be more easily profiled. Co-authored-by: Paul Mullowney Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit 3d9bacd961c43a88ed6592d58ec091abf5ee3b90 Author: Weiqun Zhang Date: Sun Feb 26 05:10:01 2023 -0800 CMake: Parallel HDF5 Detection (#3170) ## Summary It turns out the words in the settings summary of HDF5 came from the words used when HDF5 was configured with CMake. The issue is `h5pcc -showconfig` may show `Parallel HDF5: ON`, but the current versions of CMake uses `Parallel HDF5: yes` to detect Parallel HDF5. This commit adds a more reliable way of detecting Parallel HDF5. ## Additional background - https://github.com/spack/spack/issues/35546 - https://github.com/HDFGroup/hdf5/issues/2488 - https://gitlab.kitware.com/cmake/cmake/-/merge_requests/8234 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Tools/CMake/AMReXThirdPartyLibraries.cmake commit 42acecf9c7f7651bda3b172eba71dfefc7fe2be9 Author: Weiqun Zhang Date: Thu Feb 23 08:27:38 2023 -0800 TableData: Relax the static assertion (#3174) Relax the requirement of T to trivially copyable (so that htod memcpy will work) and trivially destructible. After this change, TableData> should work. Src/Base/AMReX_TableData.H commit a3c4c9adc20852ef812785ee8d1422c78a3fdf10 Author: Weiqun Zhang Date: Wed Feb 22 19:41:29 2023 -0800 SYCL: Group and subgroup size (#3172) * Use [[sycl::reqd_work_group_size]] when we know the group size at compile time. I have not seen perfromance difference, but it should not hurt. * Replace [[intel::reqd_sub_group_size]] with [[sycl::reqd_sub_group_size]] since the Intel extension is in the standard now. Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_MFParallelForG.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_TagParallelFor.H commit 606589d8ccc2ac43cb6f9f25a54b53709c23d230 Author: Miren Radia <32646026+mirenradia@users.noreply.github.com> Date: Wed Feb 22 17:49:45 2023 +0000 Add Make.local-pre to gitignore (#3171) Given that `Make.local` files are in the `.gitignore`, this should probably be there too. .gitignore commit d8e6401642ff61aa72f1169ec1f004af406600f1 Author: Weiqun Zhang Date: Sun Feb 19 09:38:00 2023 -0800 New functions for getting data in cell or line. (#3166) Add get_cell_data and get_line_data to get the data in a cell or a line, respectively. Note that we already had get_slice_data. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_MultiFabUtil.H commit b1e9669ffc3e7ee05fdbaf20f4dd6ce4b4522326 Author: Weiqun Zhang Date: Sat Feb 18 09:46:16 2023 -0800 SYCL: Optimization of small reduction (#3167) For small reductions, it's faster to do the second pass on CPU. The change is for SYCL only. Src/Base/AMReX_Reduce.H commit 3e72725a55d0be2d0f26079261ed6a724b8ade14 Author: Weiqun Zhang Date: Fri Feb 17 16:00:28 2023 -0800 amrex::Math: No longer need to use sycl math functions (#3164) We introduced `amrex::Math::floor`, etc. because their std versions were not supported in device code by the Intel compiler at that time. Otherwise, the user would have to call either sycl::floor or std::floor with the help of macros. This is no longer the case. Now we can switch to use the std versions. The amrex::Math versions are kept for backward compatibility. The particle code is not updated because it's undergoing some large changes. Docs/sphinx_documentation/source/GPU.rst Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_InterpFaceReg_2D_C.H Src/AmrCore/AMReX_InterpFaceReg_3D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/Base/AMReX_Algorithm.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_COORDSYS_1D_C.H Src/Base/AMReX_COORDSYS_2D_C.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_GpuComplex.H Src/Base/AMReX_IntVect.H Src/Base/AMReX_Math.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil_nd_C.H Src/Base/AMReX_RealVect.H Src/Base/Parser/AMReX_IParser_Y.H Src/Base/Parser/AMReX_Parser_Y.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_chkpt_file.cpp Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_algoim_K.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/slope_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/slope_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/tagging_K.H Tests/EB_CNS/Source/CNS_K.H Tests/EB_CNS/Source/CNS_tagging.H Tests/EB_CNS/Source/hydro/CNS_divop_K.H Tests/EB_CNS/Source/hydro/CNS_hydro_K.H Tests/EB_CNS/Source/hydro/CNS_hydro_eb_K.H Tests/GPU/CNS/Source/CNS_K.H Tests/GPU/CNS/Source/CNS_tagging.H Tests/GPU/CNS/Source/hydro/CNS_hydro_K.H Tests/Particles/GhostsAndVirtuals/main.cpp commit 0a8eeb864a13ce3ad3aaee591bde18fdc94fd4f7 Author: Axel Huebl Date: Fri Feb 17 09:19:46 2023 -0800 gitignore: Editors & IDEs (#3163) ## Summary Add a few typical editor and IDE project directories to `.gitignore`. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate .gitignore commit 5729dd1c7eb2461f971584cbe1b50216727dfbcc Author: Weiqun Zhang Date: Thu Feb 16 14:09:55 2023 -0800 SYCL Device Info (#3161) Print out more SYCL device information if available. Currently, free memory and max memory band width are not available yet. Src/Base/AMReX_GpuDevice.cpp commit cbe2b291d3edd114dad171eaa6786150fb6684f4 Author: Weiqun Zhang Date: Mon Feb 13 15:51:57 2023 -0800 SYCL AOT: Fix warning (#3159) Finally set the proper flags without generating warnings about unused compiler arguments. Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit 2f88f61297d0048d3266b41c27ecf453f67160c5 Author: Weiqun Zhang Date: Mon Feb 13 13:54:53 2023 -0800 GNU Make: make it an error if intel gpu arch for aot is unknown (#3157) This is a follow-up on #3155 to make it consistent with CMake, and also because `-device *` does not actually work on the testbed. Tools/GNUMake/comps/dpcpp.mak commit e45a6a624be02f4f5bd1fdf759df778d469678bf Author: Weiqun Zhang Date: Mon Feb 13 10:03:42 2023 -0800 SYCL: Fix AOT (#3155) This reverts part of #3123 because the AOT argument is a compile option. Also we must provide AMReX_INTEL_ARCH when SYCL AOT is on. Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit 774f5443a7703524a9a054a59d2efa955fdde3c9 Author: Weiqun Zhang Date: Sun Feb 12 11:29:49 2023 -0800 CI: Add sundials (#3153) Also fixes warnings. .github/workflows/dependencies/dependencies_nvcc12.sh .github/workflows/sundials.yml Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Src/Extern/SUNDIALS/AMReX_SUNMemory.H Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp commit dbe1ebd464103524a14a093ff4504f58aadede51 Author: Weiqun Zhang Date: Sat Feb 11 19:12:57 2023 -0800 Add static_assert to Mpi_typemap (#3154) Src/Base/AMReX_ParallelDescriptor.H commit bb39ad028a071e0265b55f68b3b9444abb574dd9 Author: Weiqun Zhang Date: Sat Feb 11 11:07:58 2023 -0800 SUNDIALS: Replace SUNDIALS math macros with std math functions (#3151) This fixes https://github.com/xsdk-project/xsdk-issues/issues/211. Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp commit e52fb91f805b2b076fb81b72b00fa30eb93e1a44 Author: Ann Almgren Date: Fri Feb 10 18:00:16 2023 -0800 Fix case where a norm is 1e-15 but not technically zero (#3149) Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H commit ccada32844229697d91ca4bdaf3a29cc7f5f87ed Author: Andrew Myers Date: Fri Feb 10 17:15:07 2023 -0800 Fix bug in async IO for particles (#3150) Without this fix the following will fail: > amrex/Tests/Particles/NeighborParticles(development)$ make -j8 TINY_PROFILE=TRUE BL_NO_FORT=TRUE USE_MPI=FALSE DEBUG=TRUE > $ ./main3d.gnu.DEBUG.TPROF.CUDA.ex inputs amrex.async_out=1 amrex.the_arena_is_managed=0 > Initializing CUDA... > CUDA initialized with 1 device. > AMReX (23.02-21-g858e72dcc651) initialized > 0::Assertion `index >= 0 && index < NReal + static_cast(m_runtime_rdata.size())' failed, file "../../..//Src/Particle/AMReX_StructOfArrays.H", line 43 !!! > SIGABRT The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_WriteBinaryParticleData.H commit d4822cc06b9afe1a6cd46027f936199959d9762c Author: Weiqun Zhang Date: Fri Feb 10 15:27:42 2023 -0800 DPCPP -> SYCL (#3140) The name dpcpp has been deprecated by Intel. This PR replaces DPCPP with SYCL. For backward compatibility, we have kept the AMREX_USE_DPCPP macro. However, build options like AMReX_DPCPP_AOT for cmake and DPCPP_AOT for GNU Make have been replaced by AMReX_SYCL_AOT and SYCL_AOT. Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Docs/sphinx_documentation/source/Introduction.rst Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.cpp Src/Base/AMReX_Arena.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseFabUtility.H Src/Base/AMReX_BlockMutex.H Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuAssert.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuAsyncArray.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuElixir.cpp Src/Base/AMReX_GpuError.H Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuRange.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_MFParallelForG.H Src/Base/AMReX_Math.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_Random.cpp Src/Base/AMReX_RandomEngine.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H Src/Base/AMReX_TagParallelFor.H Src/Base/Parser/AMReX_Parser_Y.H Src/EB/AMReX_algoim_K.H Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp_bc.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp Tests/GPU/AnyOf/GNUmakefile Tests/Parser/GNUmakefile Tests/Particles/DenseBins/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSYCL.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/sites/Make.unknown Tools/RegressionTesting/AMReX-dpcpp-tests.ini Tools/RegressionTesting/AMReX-sycl-tests.ini Tools/libamrex/configure.py Tools/libamrex/mkconfig.py commit 522f1bfb237c6967fe2ac228131e3b917242b7ea Author: Weiqun Zhang Date: Fri Feb 10 14:07:09 2023 -0800 Add MPI support to GpuComplex (#3148) This creates a specialization of Mpi_typemap for GpuComplex. Needed by https://github.com/ECP-WarpX/WarpX/pull/3618/ ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_ParallelDescriptor.H Src/EB/AMReX_distFcnElement.cpp commit 74a5398a941c9f9e864ed8a9d1706d4d895c7ca7 Author: Weiqun Zhang Date: Fri Feb 10 13:45:40 2023 -0800 GNU Make: Add support for Intel LLVM compiler in CPU builds (#3126) icpc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. Src/Base/AMReX_Extension.H Tools/GNUMake/comps/intel-classic.mak Tools/GNUMake/comps/intel-llvm.mak Tools/GNUMake/comps/intel.mak commit 087cf8be30e910414535f163d7d2e411328fe119 Author: Weiqun Zhang Date: Fri Feb 10 13:28:54 2023 -0800 SYCL: Fix the order of destruction (#3146) Although we have not observed any issues, we should destroy the SYCL context, device, and queues in the reverse order of their construction. Src/Base/AMReX_GpuDevice.cpp commit 9ecf6e47764a81d44e05b63eb49ec8e084e4944e Author: Weiqun Zhang Date: Fri Feb 10 13:28:01 2023 -0800 Adjust subgroup size for Intel GPUs (#3139) We can now use 32 as the subgroup size for Intel GPUs. This improves performance in most situations. Src/Base/AMReX_GpuDevice.H commit fc004c395e9ddb37389b04f37cdd165492a48fce Author: Weiqun Zhang Date: Fri Feb 10 13:27:03 2023 -0800 Runtime parameters for Signal handling controls (#3125) Add amrex.handle_sigsegv, handle_sigint, handle_sigabrt, and handle_sigfpe for more fine controls. Disable sigsegv handling for certain Intel GPUs because it's incompatible with the managed memory. Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_GpuDevice.cpp commit 489cadbe49759080911bd3b28f1423ca88ac2ba9 Author: Weiqun Zhang Date: Fri Feb 10 13:26:22 2023 -0800 FillPatcher: Update interpolation in time (#3106) The previous implementation was susceptible to roundoff errors. It was observed in 7 levels PeleC runs that the code could fail because the times in StateData were out of sync. The updated logic is closer to that in the FillPatch function and the StateData class. Src/AmrCore/AMReX_FillPatcher.H commit d322dc977efd49c1f2c293c7b4dae775bf78e1b1 Author: Weiqun Zhang Date: Fri Feb 10 12:58:46 2023 -0800 AmrLevel::FillPatch: Abort if grids are not properly nested for GPU or EB (#3098) Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp commit 858e72dcc65175e8e1ee5a8984fc611ba193182a Author: Weiqun Zhang Date: Fri Feb 10 11:54:48 2023 -0800 Fix FabArray::shift (#3147) ## Summary The cached metadata used by fused GPU kernel launches needs to be updated in FabArray::shift because the BoxArray has changed. ## Additional background Close #3145 Src/Base/AMReX_FabArray.H commit 285f5b5d10200127404d3b8b28642eec28798ef0 Author: Andrew Myers Date: Fri Feb 10 11:54:17 2023 -0800 Fix nl test (#3142) This fixes a bug introduced in PR #3060 related to building neighbor lists between particles of different types. I also made it so that CI tests would have caught this bug. Closes #3127 Src/Particle/AMReX_NeighborParticlesI.H Tests/Particles/NeighborList/CMakeLists.txt Tests/Particles/NeighborList/main.cpp commit 1f7a2f52b84dda058b2645a97f5a4135bf7a8b31 Author: Weiqun Zhang Date: Thu Feb 9 16:33:17 2023 -0800 Fix bug in Device::Finalize: Forgot to clear streams (#3143) ## Summary We forgot to clear the non-owning gpu_stream vector. They still held the old streams after we destroyed the streams in Device::Finalize. This bug affects codes that call amrex::Initialize and amrex::Finalize repeatedly. Note that we have the non-owning gpu stream vector for OMP CPU thread safety. The bug was introduced in #1064. ## Additional background #3141 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_GpuDevice.cpp commit 5cd5852f6276a5ac8f5f1bd99ef17531595d25df Author: Lucas Esclapez <13371051+esclapez@users.noreply.github.com> Date: Thu Feb 9 16:14:57 2023 -0800 Add a compute face-centered velocity gradient in MLEBTensor (#3133) Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H commit 23d6b40ef6102d8927a3d859cd7eaed36dbe25a9 Author: Axel Huebl Date: Thu Feb 9 14:35:39 2023 -0800 Test: Reinit AMReX (#3141) ## Summary Run a test that inits and finalizes AMReX multiple times. ```console cmake -S . -B build -DAMReX_GPU_BACKEND=CUDA -DAMReX_ENABLE_TESTS=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo cmake --build build -j 8 ctest --test-dir build --verbose -R Reinit ``` ```console build/Tests/Reinit cuda-gdb -ex r -ex bt --args ./Test_Reinit inputs ``` ## Additional background This is often done in pytest tests, where the same process is reused for multiple tests. Needed for pyAMReX, ImpactX, WarpX et al. This also demonstrates a bug for GPU testing. Tests/CMakeLists.txt Tests/Reinit/CMakeLists.txt Tests/Reinit/GNUmakefile Tests/Reinit/Make.package Tests/Reinit/inputs Tests/Reinit/main.cpp commit 0c444b898f8abc1f1f7dd81cb06e2b29faabb935 Author: Weiqun Zhang Date: Wed Feb 8 12:57:59 2023 -0800 Functional: Less -> Minimum, Greater -> Maximum (#3138) The functional classes for returning the lesser and greater of two arguments have been misnamed, because std::less and std::greater are for comparison and they return bool. To avoid confusion, they have been renamed. Src/Base/AMReX_Functional.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_ParallelReduce.H commit 188aaf6654bf27a1d4a0eec338ac044c60b58840 Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Wed Feb 8 19:57:17 2023 +0100 Initialize TinyProfiler earlier only for memory (#3137) Initialize TinyProfiler earlier only for memory and finalize it later only for memory as well. Src/Base/AMReX.cpp Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit 6dc1f61f3567614e67fb27bad3907eff398cc712 Author: Weiqun Zhang Date: Tue Feb 7 19:04:22 2023 -0800 Fix #3105: missing include (#3134) Src/Base/AMReX_TinyProfiler.H commit d6923bff490b80ebd807056ac49b52a1e1a22cd9 Author: Weiqun Zhang Date: Tue Feb 7 12:59:10 2023 -0800 PODVector: Add some stream synchronization (#3130) The correctness of some functions depends on the asynchronous behavior of memcpy (especially when pageable host memory is involved). It seems that that is not well defined in the SYCL standard. So for safety, stream synchronization is added to a few places. Src/Base/AMReX_PODVector.H commit 50d7cd1980c4c7657fc425e746918863020c110f Author: Weiqun Zhang Date: Tue Feb 7 12:55:25 2023 -0800 SENSIE CI: run on ubuntu-20.04. 18.04 is deprecated. (#3131) .github/workflows/sensei.yml commit 35dd973033f00e3ff1c4b86ef63037b8496699fe Author: AlexanderSinn <64009254+AlexanderSinn@users.noreply.github.com> Date: Tue Feb 7 19:50:02 2023 +0100 Profile memory allocations of the CArena using TinyProfiler (#3105) ## Summary The following per-section exclusive metrics of all CArenas are collected: * Nalloc Number of memory allocations inside a profiler section. * Nfree How many of the allocations counted in Nalloc where freed before `Finalize`. Works if the memory was freed in a different section compared to where it was allocated. * AvgMem The amount of memory allocated in a section averaged over execution time, taking into account how long it takes until the memory is freed again. * MaxMem The maximum amount of memory allocated by a section at any given time. In contrast to the other metrics, MaxMem is not additive between two sections, only satisfying the triangle inequality. If more the one rank is used, Nalloc and Nfree are summed between all ranks while for AvgMem and MaxMem the minimum, average and maximum value across all ranks is printed. Sample output from HiPACE++ (1 Rank) ``` Device Memory Usage: --------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem --------------------------------------------------------------- The_Device_Arena::Initialize() 1 1 4 B 8192 KiB --------------------------------------------------------------- Managed Memory Usage: --------------------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem --------------------------------------------------------------------------- The_Arena::Initialize() 1 1 43 KiB 29 GiB sortBeamParticlesByBox() 44 44 7377 MiB 11 GiB BeamParticleContainer::InitParticles 10 10 17 MiB 7390 MiB PlasmaParticleContainer::InitParticles 34 34 4554 MiB 4737 MiB Fields::AllocData() 3 3 2820 MiB 2820 MiB Hipace::ExplicitMGSolveBxBy() 45 45 1332 MiB 1366 MiB DenseBins::buildGPU 36 36 1182 MiB 1210 MiB FFTPoissonSolverDirichlet::define() 3 3 383 MiB 383 MiB AnyDST::CreatePlan() 2 2 255 MiB 255 MiB ResizeRandomSeed 1 1 40 MiB 40 MiB AdaptiveTimeStep::Calculate() 4 4 6 B 972 KiB hpmg::MultiGrid::solve1() 17320 17320 108 KiB 486 KiB Hipace::InitData() 7 7 273 B 243 KiB MultiPlasma::InitData() 2 2 0 B 243 KiB main() 3 3 79 B 80 B Fields::Copy() 1 1 62 B 64 B DepositCurrent_PlasmaParticleContainer() 10000 10000 1 B 16 B --------------------------------------------------------------------------- Pinned Memory Usage: ----------------------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------------------- Hipace::Evolve() 1 1 1682 MiB 1718 MiB The_Pinned_Arena::Initialize() 1 1 52 B 8192 KiB Hipace::ExplicitMGSolveBxBy() 1 1 998 B 1024 B sortBeamParticlesByBox() 24 24 127 B 256 B Hipace::InitData() 37 37 207 B 240 B main() 41 41 79 B 96 B Fields::Copy() 1 1 62 B 64 B AdaptiveTimeStep::Calculate() 4 4 0 B 32 B MultiPlasma::InitData() 2 2 0 B 16 B hpmg::MultiGrid::solve1() 17320 17320 6 B 16 B ----------------------------------------------------------------- Cpu (amrex_mempool) Memory Usage: ----------------------------------------------------- Name Nalloc Nfree AvgMem MaxMem ----------------------------------------------------- amrex_mempool_init() 1 1 25 B 8192 KiB ----------------------------------------------------- ``` 4 Ranks: ``` Device Memory Usage: --------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max --------------------------------------------------------------------------------------------------------------------- The_Device_Arena::Initialize() 4 4 4 B 4 B 5 B 8192 KiB 8192 KiB 8192 KiB --------------------------------------------------------------------------------------------------------------------- Managed Memory Usage: -------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max -------------------------------------------------------------------------------------------------------------------------------- The_Arena::Initialize() 4 4 16 KiB 19 KiB 23 KiB 29 GiB 29 GiB 29 GiB sortBeamParticlesByBox() 657 657 1458 MiB 1915 MiB 2808 MiB 4221 MiB 5987 MiB 11 GiB MultiBeam::PackLocalGhostParticles() 10 10 0 B 500 MiB 2003 MiB 0 B 2078 MiB 8313 MiB BeamParticleContainer::InitParticles 10 10 0 B 1252 KiB 5010 KiB 0 B 1847 MiB 7390 MiB PlasmaParticleContainer::InitParticles 136 136 4556 MiB 4557 MiB 4557 MiB 4737 MiB 4737 MiB 4737 MiB Hipace::Wait() 301 301 1836 KiB 4227 KiB 5413 KiB 3695 MiB 3695 MiB 3695 MiB Fields::AllocData() 12 12 2820 MiB 2820 MiB 2820 MiB 2820 MiB 2820 MiB 2820 MiB Hipace::ExplicitMGSolveBxBy() 180 180 1011 MiB 1176 MiB 1363 MiB 1366 MiB 1366 MiB 1366 MiB DenseBins::buildGPU 744 744 240 MiB 283 MiB 325 MiB 605 MiB 605 MiB 605 MiB FFTPoissonSolverDirichlet::define() 12 12 383 MiB 383 MiB 383 MiB 383 MiB 383 MiB 383 MiB AnyDST::CreatePlan() 8 8 255 MiB 255 MiB 255 MiB 255 MiB 255 MiB 255 MiB ResizeRandomSeed 4 4 40 MiB 40 MiB 40 MiB 40 MiB 40 MiB 40 MiB AdaptiveTimeStep::Calculate() 66 66 2 B 3 B 5 B 972 KiB 972 KiB 972 KiB hpmg::MultiGrid::solve1() 154398 154398 86 KiB 99 KiB 111 KiB 486 KiB 486 KiB 486 KiB Hipace::InitData() 28 28 271 B 271 B 272 B 243 KiB 243 KiB 243 KiB MultiPlasma::InitData() 8 8 0 B 0 B 0 B 243 KiB 243 KiB 243 KiB Hipace::Notify() 196 196 0 B 0 B 2 B 128 B 128 B 128 B main() 12 12 79 B 79 B 79 B 80 B 80 B 80 B Fields::Copy() 4 4 47 B 54 B 63 B 64 B 64 B 64 B DepositCurrent_PlasmaParticleContainer() 80000 80000 1 B 1 B 1 B 16 B 16 B 16 B -------------------------------------------------------------------------------------------------------------------------------- Pinned Memory Usage: ---------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max ---------------------------------------------------------------------------------------------------------------------- Hipace::Notify() 245 245 554 MiB 932 MiB 1166 MiB 3051 MiB 3051 MiB 3051 MiB Hipace::Wait() 245 245 9 MiB 20 MiB 32 MiB 3051 MiB 3051 MiB 3051 MiB Hipace::Evolve() 4 4 317 MiB 369 MiB 428 MiB 429 MiB 429 MiB 429 MiB The_Pinned_Arena::Initialize() 4 4 21 B 24 B 27 B 8192 KiB 8192 KiB 8192 KiB Hipace::ExplicitMGSolveBxBy() 4 4 757 B 881 B 1022 B 1024 B 1024 B 1024 B sortBeamParticlesByBox() 576 576 287 B 287 B 287 B 416 B 416 B 416 B Hipace::InitData() 148 148 207 B 207 B 207 B 240 B 240 B 240 B main() 164 164 79 B 79 B 79 B 96 B 96 B 96 B Fields::Copy() 4 4 47 B 54 B 63 B 64 B 64 B 64 B AdaptiveTimeStep::Calculate() 66 66 0 B 0 B 0 B 32 B 32 B 32 B MultiPlasma::InitData() 8 8 0 B 0 B 0 B 16 B 16 B 16 B hpmg::MultiGrid::solve1() 154398 154398 5 B 5 B 7 B 16 B 16 B 16 B ---------------------------------------------------------------------------------------------------------------------- Cpu (amrex_mempool) Memory Usage: ----------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max ----------------------------------------------------------------------------------------------------------- amrex_mempool_init() 4 4 9 B 9 B 11 B 8192 KiB 8192 KiB 8192 KiB ----------------------------------------------------------------------------------------------------------- ``` ## Additional background A major difficulty in implementing this is that different omp threads could be in different functions. This means that each thread needs its own profiler function stack. Additionally multiple threads may share the same TinyProfiler object. To clean up the Profiler output, profiler sections were added to arena preallocate/initialize functions. Additionally `amrex::DeallocateRandomSeedDevArray()` was moved before `BL_TINY_PROFILE_FINALIZE()` so the deallocation is not missing in the profile. Src/Base/AMReX.cpp Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit a7322024263f62d779e7d7a905659de090267e20 Author: Weiqun Zhang Date: Mon Feb 6 10:30:07 2023 -0800 NeighborParticles: memcpy -> memcpy_async (#3129) There are no reasons to use memcpy, which contains a stream sync. Moreover, the scan function later has to do a stream sync internally because it has to allocate some temporary space. Src/Particle/AMReX_NeighborParticlesI.H commit e81a9c232c2608de90b3514ece0f211687b41cfe Author: Weiqun Zhang Date: Fri Feb 3 19:28:05 2023 -0800 Rename AMReX_CUDA_MAX_THREADS to AMReX_GPU_MAX_THREADS (#3115) Rename cmake option AMReX_CUDA_MAX_THREADS to AMReX_GPU_MAX_THREADS. This option can now be used for all GPU backends. GNU Make has also been updated for this. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_GpuControl.H Tools/CMake/AMReXCUDAOptions.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs commit 8595daac71aab852070f045856e1b28028e5f7b5 Author: Axel Huebl Date: Fri Feb 3 16:36:05 2023 -0800 NVHPC: CUDA (#3124) In my last commit #3122, I accidentially switched NVHPC to compile for OpenMP instead of the CUDA backend. .github/workflows/cuda.yml commit 48f5ffc629cfcded97f9214dabc551248ab11d26 Author: Weiqun Zhang Date: Fri Feb 3 14:50:33 2023 -0800 Update Build Systems for oneAPI (#3123) * For DEBUG builds, link with -fsycl-link-huge-device-code, otherwise it might crash due to device code larger than 2GB. * The AOT arguments are for the linker only. So they are removed from compile options. * Change the default of AMReX_INTEL_ARCH to * (i.e., all devices). This does not seem to increase link time or executable size that much. The movtivation for this is that the users do not have to set the device target to some magic numbers. * Add new CI to test EB. Also switch to use Intel MPI. * Fix a mistake in GNU Make. DPCPP_SPLIT_KERNEL can be used independent of AOT. .github/workflows/dependencies/dependencies_dpcpp.sh .github/workflows/intel.yml Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit 25fcdfc433ac747cf1022fa85f858b36cb771f45 Author: Axel Huebl Date: Thu Feb 2 17:01:13 2023 -0800 NVHPC: OpenMP Atomic Capture Fixed (#2378) Src/Base/AMReX_FabArrayUtility.H commit 2ad56772eac9095b9678de5ced1e5440c4e579c9 Author: Axel Huebl Date: Thu Feb 2 13:07:57 2023 -0800 CI: NVHPC 23.1 (#3122) Update CI to Test against the latest NVHPC release, v23.1 See #2378 .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_nvhpc23-1.sh Src/Particle/AMReX_ParticleArray.H commit e6e2fbc3cd753e82bcd95d213a4adf436f94796f Author: Weiqun Zhang Date: Wed Feb 1 17:36:48 2023 -0800 oneAPI: Work around compiler bug (#3121) This makes Tests/LinearSolvers/NodeEB/ 3D test work with oneAPI. Src/EB/AMReX_algoim_K.H commit 4e8ca9aa0667b81cdebd144c55f85e16e25f785d Author: Aaron M. Lattanzi <103702284+AMLattanzi@users.noreply.github.com> Date: Wed Feb 1 18:16:32 2023 -0700 Multi-grid NeighborList GPU Sync (#3120) Synchronizing at the end of building the neighbor list avoids memory mis-references on GPU with the multi-grid approach. Src/Particle/AMReX_NeighborList.H commit 78dd7a0585840c2920b4e86d3e355a8b156d8497 Author: Andrew Myers Date: Wed Feb 1 13:06:25 2023 -0800 Modify the neighbor particles test to also test selectActualNeighbors. (#3118) The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Particle/AMReX_NeighborParticlesI.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/inputs Tests/Particles/NeighborParticles/main.cpp commit c09ed7de228d9c20d2240f0b732ca3535df13691 Author: Weiqun Zhang Date: Wed Feb 1 13:05:21 2023 -0800 Remove Atomic::Inc and Dec (#3117) The implementation for Atomic::Inc and Dec in our SYCL backend use compare and exchange, because they do not exist in SYCL standard. This turns out to be very inefficient. Although both CUDA and HIP have efficient implementation of atomicInc and atomicDec, we are removing Atomic::Inc and Dec from amrex to avoid surprises. In all cases that we have used these functions, they can be replaced with Atomic::Add. Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_Scan.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleCommunication.H commit eefe2463884081a27025973d4a99dc909657b4f0 Author: Weiqun Zhang Date: Wed Feb 1 09:09:38 2023 -0800 Update CHANGES for 23.02 (#3119) CHANGES commit 225c605680e825e1f0793780746b4112d4262381 Author: Cyrus Harrison Date: Fri Jan 27 15:15:46 2023 -0800 update github actions to use ascent 0.9.0 release (#3114) Updates ascent in situ github actions CI test to use new ascent release 0.9.0. .github/workflows/ascent.yml commit b45ea138416ef76e1c3122dd08fe54da04645427 Author: Weiqun Zhang Date: Fri Jan 27 13:24:48 2023 -0800 Documentation: Masks (#3113) Add documentation on various masks. Docs/sphinx_documentation/source/Basics.rst commit 756468d421064b03605dafd262e8b707e751d0b6 Author: Axel Huebl Date: Thu Jan 26 08:48:55 2023 -0800 CMake 3.18+: Stay OLD CUDA_ARCHITECTURES Policy (#3112) ## Summary For now... We removed this guard in WarpX & ImpactX a while ago. Need to revisit how we want to add the logic, probably by jumping to CMake 3.20+ requirements and ditching legacy CUDA logic. ## Additional background Regression from switching to CMake 3.18+, which then enables new policies. Introduced in #3107 - https://cmake.org/cmake/help/latest/policy/CMP0104.html - https://cmake.org/cmake/help/v3.24/prop_tgt/CUDA_ARCHITECTURES.html#prop_tgt:CUDA_ARCHITECTURES ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate CMakeLists.txt Tools/CMake/AMReXUtils.cmake commit a232be4e275bec58ab26f302f6423e736c75a840 Author: Weiqun Zhang Date: Tue Jan 24 13:54:53 2023 -0800 OpenBCSolver: Fix reuse (#3111) When the open BC solver is reused, we must update the level BC. Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp commit a83c130b34ffca97e43e7451e2df6a948830fd54 Author: Axel Huebl Date: Tue Jan 24 12:53:29 2023 -0800 CMake: Fix AMD Flags for ROCm 5.5 and Bump cmake minimum version to 3.18 (#3107) ## First Commit: Shell-Escape Tokens - [x] The new ROCm 5.5 flag `-Xoffload-linker --whole-archive` is a flag with an argument. CMake will deduplicate the command line if it encounters another `-Xoffload-linker` on it, which will break compilations in the future. This marks the whole string as a token that can only be deduplicated as a whole. Ref.: https://cmake.org/cmake/help/v3.25/command/target_link_options.html#option-de-duplication ## Second Commit: Only ROCm 5.5+ - [x] Do not add the flags `-Xoffload-linker --whole-archive` in earlier ROCm releases, avoiding compile-time warnings. ## Third Commit: Clean Up CI - [x] Remove warning suppression introduced in #3097 ## Additional background Follow-up to @david-salinas's #3097 .github/workflows/hip.yml CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/BuildingAMReX_Chapter.rst Tests/CMakeTestInstall/CMakeLists.txt Tests/SpackSmokeTest/CMakeLists.txt Tools/CMake/AMReXParallelBackends.cmake commit 1c18d00a48a685441636f2e6e90715817d55ee97 Author: Marc T. Henry de Frahan Date: Tue Jan 24 09:34:36 2023 -0700 Check component and derive names for `canDerive` and `get` (#3109) `canDerive` and `get` in the `DeriveList` class only check the derive name but not the variable component names. This code adds a check on the variable names. @esclapez did this internally for PeleLMeX and we need it for PeleC as well so I thought this should just belong in AMReX. Src/Amr/AMReX_Derive.cpp commit e472c108e72174112d387e9828f272277a16f68e Author: Weiqun Zhang Date: Mon Jan 23 21:08:51 2023 -0800 CMake clean up (#3100) * Clean up oneAPI compiler setup. We no longer support early beta versions. * Move AMREX_USE_EB macro definition from AMReXSetDefines.cmake to Src/EB/CMakeLists.txt. The former is not supposed to set defines for optional components (as mentioned in its comments). * AMReXParallelBackends.cmake: unset temporary variable _genex. * Update documentation for CMAKE_CXX_STANDARD. C++14 is no longer supported. * Update oneAPI CI test. Use icx and ifx instead of clang and gfortran for C and Fortran, respectively. .github/workflows/dependencies/dependencies_dpcpp.sh .github/workflows/intel.yml Docs/sphinx_documentation/source/BuildingAMReX.rst Src/EB/CMakeLists.txt Tools/CMake/AMReXFlagsTargets.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSYCL.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXUtils.cmake commit d61ec5efdeb698edaece3986a332c24700d406c5 Author: Weiqun Zhang Date: Mon Jan 23 17:36:41 2023 -0800 Replace sprintf with snprintf to fix compiler warnings (#3108) Tests/HDF5Benchmark/main.cpp Tests/Particles/CheckpointRestart/main.cpp commit b487434b948fcdc848498062cb094ce53d5414ad Author: Weiqun Zhang Date: Thu Jan 19 10:26:21 2023 -0800 Fix comment for OwnerMask (#3102) Src/Base/AMReX_iMultiFab.H commit 12730dec3e9a64dba5f7585b360b470a5d722b41 Author: Weiqun Zhang Date: Thu Jan 19 08:43:34 2023 -0800 ParmParse: support nan and inf (#3101) When the ParamParse table is dumped, if a floating point number is a NaN, it will be printed as nan. If it's infinity, it will be printed as inf. However, if one tried to read it back, the code will crash because ParmParse did not recognize nan and inf as valid floating point numbers. This commit adds support for nan and inf in ParmParse. Src/Base/AMReX_ParmParse.cpp commit a5063a5d705b5192c772d5b969d13a36b97da0da Author: Axel Huebl Date: Tue Jan 17 16:22:15 2023 -0800 CI: actions/checkout@v3 (#3099) Update leftover actions that use the deprecated v2. .github/workflows/ascent.yml .github/workflows/style.yml commit d6f8b2cce0fba1967485902b100bbfc80cb07d35 Author: David Salinas Date: Fri Jan 13 17:30:21 2023 -0500 CMake: add offload linker option --whole-archive for ROCm hipcc (#3097) ## Summary Hipcc linking will be relying on clang to offload all code objects from archives, so we need to add -Xoffload-linker --whole-archive option. ## Additional background Existing hipcc would offload code objects in archive files. Now it will rely on clang to do this, but we need to tell the offload linker to include all symbols/functions in an archive file. Co-authored-by: David Salinas Co-authored-by: Weiqun Zhang .github/workflows/hip.yml Tools/CMake/AMReXParallelBackends.cmake commit d5ddf3b22e949e7cc346a2f8660d1c73edfa391a Author: Axel Huebl Date: Thu Jan 12 16:45:07 2023 -0800 SinCos Guard: Apple Define (#3096) ## Summary The issue also occurs on vanilla Clang, not only AppleClang. Thus, we exclude the specialization for all Apple platforms. ## Additional background Follow-up to #3094 #3008 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_Math.H commit a30cc5fd3213bb8118510f252fa3f5ad0028525e Author: Axel Huebl Date: Thu Jan 12 08:44:13 2023 -0800 SinCos: AppleClang Fix (#3094) ## Summary I noticed that in some situations, e.g., when building Python sources that include Python headers, `_GNU_SOURCE` is sometimes defined. This can confuse AppleClang builds of AMReX: ``` error: no member named 'sincos' in the global namespace; did you mean '__sincos'? ::sincos(x, &r.first, &r.second); ``` Adding the respective guard here should allow GCC builds on macOS but ignore the GCC-ism when building with AppleClang. Follow-up to #3008 ## Additional background First seen in pyAMReX the other week. Avoided with restructured includes, but ultimately a defect that CPython headers pull in. Seen again today by @RTSandberg Src/Base/AMReX_Math.H commit f972714bd1111991da8dfddaf65840b57dafa829 Author: Axel Huebl Date: Wed Jan 11 15:57:24 2023 -0800 CI: Doxygen (#3061) Run Doxygen and make sure it does not ~~throw warnings~~ *error while parsing* in CI. This is already run as such in downstream projects, so we need to fix our upstream errors as well. .github/workflows/style.yml .github/workflows/style/doxygen.sh Docs/Doxygen/doxygen.conf Src/Amr/AMReX_AuxBoundaryData.cpp Src/AmrCore/AMReX_ErrorList.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_Utility.H commit a324c40fc1f5e9e078e12d8a74aeae8c05b91495 Author: Weiqun Zhang Date: Wed Jan 11 15:21:54 2023 -0800 CMake: Make hypre and PETSc available only if it's not 1D. (#3093) Tools/CMake/AMReXOptions.cmake commit 64eb49bbc7c4a41233be97d6d7511fea5d112a09 Author: Ann Almgren Date: Tue Jan 10 15:14:23 2023 -0800 introduce amrex code of conduct (#3092) CODE_OF_CONDUCT.md commit a05384c0b7c4549b696ff6f1c140dde3c78fa27e Author: Nuno Miguel Nobre Date: Mon Jan 9 17:03:26 2023 +0000 SYCL: Reinforce device compilation preprocessor conditionals (#3090) ## Summary The changes in these two commits, although apparently redundant, guard SYCL compilations targeting CUDA or HIP devices by preventing the compiler from seeing the CUDA/HIP code already in AMReX. ## Additional background The 'vanilla' version of LLVM, which is used by hipSYCL for example, enables `__CUDA_ARCH__` and `__HIP_DEVICE_COMPILE__` when compiling device code, for both [CUDA](https://llvm.org/docs/CompileCudaWithLLVM.html#detecting-clang-vs-nvcc-from-code) and [HIP](https://reviews.llvm.org/D45441), respectively. Intel's version of LLVM is also moving in the same direction with [CUDA](https://github.com/intel/llvm/issues/7722) and [HIP](https://github.com/intel/llvm/issues/7720), although the latter is off to a rocky start. Src/Base/AMReX_Extension.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuRange.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_Math.H Src/Base/AMReX_Random.H commit 42b7e27066388dca2d1bc233fa1249df18111e29 Author: Lucas Esclapez <13371051+esclapez@users.noreply.github.com> Date: Fri Jan 6 07:53:52 2023 -0800 FAQ - parameter space overflow (#3089) Docs/sphinx_documentation/source/Faq.rst Src/EB/AMReX_EB2_IF.H Src/EB/AMReX_EB2_IF_DevicePtr.H Src/EB/CMakeLists.txt Src/EB/Make.package commit 2e0b2b463306bc8c371b38e4e3125a8ed86698e3 Author: Weiqun Zhang Date: Thu Jan 5 12:18:47 2023 -0800 New knapsack function to reduce the cost of data movement in load balancing (#3079) This adds a new function DistributionMapping::KnapSackProcessorMap that makes a new DistributionMapping given an old DistributionMapping, a vector of weights and a parameter affecting the amount of the data movement that would be needed for switching from the old to the new DistributionMapping. In this new knapsack function, some boxes will be kept on the current processes and the knapsack algorithm is applied to the rest of the boxes. This may generate a less optimal distribution. However, the new DistributionMapping is more similar to the old one, thus reducing the cost of data movement during load balancing. Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp commit c9c6ed5e098e168a0055a277fe4e0819acf8f8f7 Author: Weiqun Zhang Date: Thu Jan 5 12:18:13 2023 -0800 GPU single stream region and no sync region (#3073) Add functions and classes that allow the user to change the default behavior of MFIter and amrex functions for MultiFabs easily. Use with caution! Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuControl.cpp Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp commit 0c3fff43aa68a3f1da7f005e2fb4a6183ea6146b Author: Weiqun Zhang Date: Thu Jan 5 12:17:17 2023 -0800 Check for small cell before checking for multi-cut (#3088) This is a follow-up on #3087. There are cases with 5 faces covered and one face still open. These cells should be marked small cells before the multi-cut check. Src/EB/AMReX_EB2_3D_C.cpp commit 7a313e260ccf02ab5f08cde5ac45dddc9d1a9918 Author: Weiqun Zhang Date: Wed Jan 4 13:47:29 2023 -0800 Improve AmrMesh::ChopGrids (#3081) This tries to address the following issues. (1) Suppose that the 2d domain has 40x8 cells, the blocking factor is 2 and the max grid size is 1024. Here the max grid size is set to a big number so that we can have boxes as big as possible. If we have 4 MPI processes, the old ChopGrids will chop the domain in x-direction into 5 parts. This is far from optimal. (2) The issue in the example in (1) could be fixed by making the initial chunk size to be the min of max grid size and domain size. This would chop the domain into 4 boxes of 20x4 cells. However, this is still not ideal because boxes close to squares should be preferred to long skinny boxes. The new ChopGrids will chop the domain inot 4 boxes of 10x8 cells. Src/AmrCore/AMReX_AmrMesh.cpp commit dd46ab54580e5ad1506e9d2c18b23942a6cab9e3 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Wed Jan 4 16:29:04 2023 -0500 Add extra check for multicut cell (#3087) For some edge EB geometry cases when building the faces during an EB iteration, nodes can go from regular to covered. Additional check needs to be added to label appropriate cells as a `multicut` cell. Src/EB/AMReX_EB2_3D_C.cpp commit b648023071db8b04262b52d4ee9e77f2d07e6824 Author: Aaron M. Lattanzi <103702284+AMLattanzi@users.noreply.github.com> Date: Wed Jan 4 10:52:45 2023 -0800 Polydisperse neighbor search algorithm (#3060) ## Summary New functionality for building the particle neighbor list with multiple grids. The multi-grid approach can provide significant speed up with polydisperse particle-laden flows. ## Additional background Algorithm based upon: Shire, T., Hanley, K.J. & Stratford, K. DEM simulations of polydisperse media: efficient contact detection applied to investigate the quasi-static limit. Comp. Part. Mech. 8, 653–663 (2021). https://doi.org/10.1007/s40571-020-00361-2 Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleUtil.H commit b3241d7b368a6898d54e4a048512b13b2a3b6a52 Author: Weiqun Zhang Date: Tue Jan 3 21:14:51 2023 -0800 GPU UUID and Arena initial size (#3085) Gather GPU UUIDs to find out the number of unique devices (which has already been done for CUDA). Find out the number of processes sharing my device. Use this information, we can avoid oversubscribing GPU device memory in Arean initialization that could result in a runtime error for non-managed memory. Src/Base/AMReX.cpp Src/Base/AMReX_Arena.cpp Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp commit 490233778c869d2f7dcb4fbfa2eab16aafa7d211 Author: Weiqun Zhang Date: Tue Jan 3 11:55:08 2023 -0800 Use IsCallable to detect if F is callable (#3080) Src/Base/AMReX_GpuContainers.H commit edec77e5c43e14badaa915b4fbb1615f1f59e1a5 Author: Weiqun Zhang Date: Tue Jan 3 11:14:50 2023 -0800 GNU Make: No need to do GCC version check for make clean etc. (#3084) Tools/GNUMake/comps/gnu.mak commit 30f2290cf7c3c4f995ff19ac525ca720c2885dc3 Author: Katharina Kormann Date: Sun Jan 1 19:36:53 2023 +0100 Mlebabeclap aniso (#2640) Changes in MLEBABecLap with the goal of removing the assumption dx=dy=dz to make sure anisotropic cells are correctly treated. The changes concern the way how Feb is computed. This is only implemented for 2D. The external solvers (e.g., Hypre and PETSc) are not updated. So they may have trouble converging when the cells don't have the same size in every direction because the system solved by them is not fully consistent with MLMG. Co-authored-by: Weiqun Zhang Src/Base/AMReX_SPACE.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_Level.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_F.cpp commit 692059ae91b89245e8b8d28de117df10ddfa8ed6 Author: Weiqun Zhang Date: Sat Dec 31 17:38:26 2022 -0800 Update CHANGES for 23.01 (#3086) CHANGES commit ea7e8a507e77c3dce8187e268ddf2212d0e95a24 Author: Weiqun Zhang Date: Thu Dec 22 23:34:21 2022 -0800 amrex::fillAsync (#3076) This new function can be used to fill the elements in a vector type container (e.g., Gpu::DeviceVector). If the element type is a struct with several arithmetic types (e.g., GpuArray), the usual ParallelFor does not have good performance because of the memory access pattern. The new fillAsync function will use shared memory in that case to improve the performance. Src/Base/AMReX_GpuContainers.H commit 9c76f9ae2cc3e412ad5fb0ae9259e531a62b9a3b Author: Axel Huebl Date: Thu Dec 22 22:51:41 2022 +0100 CI: Ascent (#3078) ## Summary Add a CI entry to build Ascent in situ vis. ## Additional background cc @cyrush - https://github.com/ECP-WarpX/WarpX/pull/2483 - https://github.com/ECP-WarpX/WarpX/pull/2858 - #3077 .github/workflows/ascent.yml .github/workflows/sensei.yml commit c03306944aafdf9ee21c896fcfcab6cf7a5e4ad3 Author: Weiqun Zhang Date: Thu Dec 22 12:12:13 2022 -0800 Add forward declarations for class template friends (#3077) It seems that GCC 7.5 requires these forward declarations. Src/LinearSolvers/MLMG/AMReX_MLLinOp.H commit 2a8c71b8bf42bbc376b049aae828c4a2dfd8363b Author: Weiqun Zhang Date: Tue Dec 20 21:38:35 2022 -0800 Update GNU Make file for OLCF (#3070) OLCF_ROCM_ROOT is not defined if the amd module instead of rocm is loaded. So using LMOD_SITE_NAME seems to be a better option. Tools/GNUMake/Make.machines commit 52f4195f33df066d86736044e7150dd3aa45e03c Author: Weiqun Zhang Date: Tue Dec 20 21:38:21 2022 -0800 HIP: amdgpu-target -> offload-arch (#3069) The amdgpu-target option has been deprecated since at least v4.5. Tools/CMake/AMReXParallelBackends.cmake Tools/GNUMake/comps/hip.mak commit ae58672b451d140999944db1265fc18241b71320 Author: Weiqun Zhang Date: Tue Dec 20 14:59:37 2022 -0800 GPU kernel fusing in MLPoisson (#3071) Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H commit 4b4c443c298cd9ca2a90de15a7fa76c602d37ab5 Author: Kevin Gott Date: Tue Dec 20 15:05:02 2022 -0500 Docs: Sync Before Comms (#3075) Add a new section to the profiling section and document the sync_before_comm flag. This section can be used for any information, flags or options that applies to both Tiny and Full profiling. Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst commit c2decc83820a5457fc64aabfa2ebd1d787e4744f Author: Weiqun Zhang Date: Mon Dec 19 16:29:20 2022 -0800 Remove deprecated SYCL declaration (#3074) The macros we used do not work with the latest oneapi public release. Src/Base/AMReX_GpuAssert.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_Math.H Src/Base/AMReX_RandomEngine.H commit 96d8c53fbf096af3e6a7386a75d8b5a47beb15a7 Author: Weiqun Zhang Date: Fri Dec 16 09:24:48 2022 -0800 Update F_scripts/README.md (#3072) Tools/F_scripts/README.md commit ff1cce552060aa737f075e3c85fd961b1a284a85 Author: Weiqun Zhang Date: Tue Dec 13 19:24:17 2022 -0800 Change the type of DeriveRec::DeriveBoxMap to std::function (#3068) This gives us more flexibility. For example, lambda function with captured values can now be used. Src/Amr/AMReX_Derive.H commit 673d63122764b758a0727746944692d3fdf23dc6 Author: Axel Huebl Date: Tue Dec 13 13:09:15 2022 -0800 Add CodeQL Scanning (#3059) ## Summary [**GitHub code scanning**](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning): [evolved from LGTM.com](https://github.blog/2022-08-15-the-next-step-for-lgtm-com-github-code-scanning/). [`codeql.yml` workflow file for GitHub Actions](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/setting-up-code-scanning-for-a-repository#setting-up-code-scanning-manually) ## Additional background - https://github.com/ECP-WarpX/WarpX/pull/3555 - https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#editing-a-code-scanning-workflow .github/workflows/codeql.yml commit ea66c3bf57a2d6ba0123e5161d946eef379efe23 Author: Weiqun Zhang Date: Tue Dec 13 10:34:24 2022 -0800 Remove old python scripts for releasing (#3066) In the old CVS days, we used to use these scripts to strip out codes that were not meant for release. Tools/Release/cleanWords.txt Tools/Release/ppCleanup.py Tools/Release/ppCleanupDir.py Tools/Release/release.py commit 7316751aa25a00ef50828947b9f164c8dc85a33a Author: Weiqun Zhang Date: Tue Dec 13 10:19:33 2022 -0800 Plotfile Reader: Allow multi-word variable names (#3067) Src/Base/AMReX_PlotFileDataImpl.cpp commit 3129db3446226ed64e1973189ded719ef8baac47 Author: Weiqun Zhang Date: Mon Dec 12 20:25:20 2022 -0800 Fix FPE in FillPatcher for RK (#3065) To avoid FPE, the interpolation of RK data should only be done inside the physical domain. Even though the FPE here is benign because the garbage values will be overwritten later, we still want to fix it so that it does not trigger signal handling. Src/AmrCore/AMReX_FillPatcher.H commit e90408d5559c551b9a911f8f2c8641938b8f05ab Author: Axel Huebl Date: Mon Dec 12 16:22:28 2022 -0800 Fix: Shared CMake builds w/ `AMReX_BUILD_SHARED_LIBS` (#3057) Forgot to honor the new option. All builds that did not set `BUILD_SHARED_LIBS` were static now. Introduced in #3013 Src/CMakeLists.txt commit d8fef14e353f914ca10df6ebfb59c53e0f6a53dc Author: Weiqun Zhang Date: Mon Dec 12 16:15:27 2022 -0800 Add PETSc CI (#3049) .github/workflows/petsc.yml commit bef5b6a282ebfd9236e53a1f262c982837299710 Author: Michael Zingale Date: Mon Dec 12 11:42:49 2022 -0500 remove plotsinglevar.py (#3063) this depended on an old Fortran library fsnapshot.so that we no longer provide. It's much easier to do this sort of plotting with yt these days. Tools/Py_util/plotsinglevar.py commit 10293db79733bba55986e0aede522bd86a8d030c Author: Michael Zingale Date: Mon Dec 12 11:40:55 2022 -0500 Remove old python build scripts from Castro / Maestro (#3064) These all originated in Castro / Maestro, and we've moved the bits we need to our codes. Many of these deal with Fortran in particular, and are no longer used at all. Tools/F_scripts/fcheck.py Tools/F_scripts/findparams.py Tools/F_scripts/makebuildinfo.py Tools/F_scripts/write_probin.py commit 414d2091da47d0ba73a67d73cbb856945c11c621 Author: Weiqun Zhang Date: Fri Dec 9 17:28:33 2022 -0800 Add math functions to amrex::Math (#3008) Add a function to return PI and the following trig functions, - `cospi(x)` returns `cos(pi*x)` - `sinpi(x)` returns `sin(pi*x)` - `sincos(x)` returns `sin(x)` and `cos(x)` in `std::pair` - `sincospi(x)` returns `sin(pi*x)` and `cos(pi*x)` in `std::pair` Because we use C++17 now, one could use structured binding on the `std::pair` returned by `sincos` and `sincospi`. auto const [sin_x, cos_x] = amrex::Math::sincos(x); Src/Base/AMReX_Math.H commit 5a92d75f2a6a7f0e764c469b1f4bbe417dcf4566 Author: Michael Zingale Date: Fri Dec 9 19:52:27 2022 -0500 remove some astro-specific python routines (#3062) these have not been used in ages and are not general for AMReX Tools/Postprocessing/python/column_depth.py Tools/Postprocessing/python/conv_slopes.py Tools/Postprocessing/python/eos_data.txt Tools/Postprocessing/python/helmeos.py Tools/Postprocessing/python/test_helmeos.py commit 821c5ba7d39b44d21cd3c3e726a2cc799111cc35 Author: Weiqun Zhang Date: Fri Dec 9 13:57:11 2022 -0800 Gpu Streams: Performance Tuning (#3058) * Change the number of GPU streams from 4 to 2. Recent tests on summit and crusher show that using 2 GPU streams seems to be better than 4 or 1 in most cases. One reason is that it reduces the cost associated with stream synchronization, which could be significant for small kernels. * In ~MFIter and ~StreamIter, there is no need to synchronize the streams that were not used. It appears that for HIP there is very little cost to sychronize a stream that does not have any GPU kerenels since the last synchronization, but that's not the case for CUDA. * Remove the dedicated stream for kernels outside MFIter, and simply use the first stream used by MFIter. * Remove synchronization in FabArray::define. It was needed when we were putting FAB in managed memory, but that's no longer the case. Src/Base/AMReX_FabArray.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuUtility.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit 57793fda870abbe8ed70b5fbb593b4f63589b442 Author: Weiqun Zhang Date: Wed Dec 7 10:27:24 2022 -0800 MLMG: template (#3035) MLMG and various linear operator classes use MultiFab as their data container. The limitations are (1) it cannot do single precision when amrex::Real is double and (2) it cannot handle data that need to be stored in multiple MultiFabs such as vectors on cell edges. Recently, we have experimented with amrex::Any that can be a type different from MultiFab. However, this involves a lot of complicated and hacky changes. In this PR, we are trying to use template to make the interface more flexible. Classes MLMG, MLPoisson, MLALaplacian, MLABecLaplacian, MLABecLap, MLCellLinop, MLLinOp, and MLCGSolver have been changed to templates. More changes will follow. For most users, this should not require any changes in their codes. However, if one has classes derived from the linear operators in MLMG, some small changes are likely needed. For example, `getNComp()` needs to be `this->getNComp()` due to C++ name lookup rules. .github/workflows/cuda.yml Docs/sphinx_documentation/source/LinearSolvers.rst Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/AMReX_PETSc.cpp Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp_temp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMGBndry.H Src/LinearSolvers/MLMG/AMReX_MLMGBndry.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H Src/LinearSolvers/MLMG/Make.package Tests/LinearSolvers/ABecLap_SP/CMakeLists.txt Tests/LinearSolvers/ABecLap_SP/GNUmakefile Tests/LinearSolvers/ABecLap_SP/Make.package Tests/LinearSolvers/ABecLap_SP/MyTest.H Tests/LinearSolvers/ABecLap_SP/MyTest.cpp Tests/LinearSolvers/ABecLap_SP/MyTestPlotfile.cpp Tests/LinearSolvers/ABecLap_SP/initProb.cpp Tests/LinearSolvers/ABecLap_SP/initProb_K.H Tests/LinearSolvers/ABecLap_SP/inputs Tests/LinearSolvers/ABecLap_SP/main.cpp Tests/LinearSolvers/ABecLaplacian_C/CMakeLists.txt Tests/LinearSolvers/ABecLaplacian_C/initProb_K.H Tests/LinearSolvers/MLMG/GNUmakefile Tests/LinearSolvers/MLMG/Make.package Tests/LinearSolvers/MLMG/fort.H Tests/LinearSolvers/MLMG/fort_3d.F90 Tests/LinearSolvers/MLMG/init_prob.cpp Tests/LinearSolvers/MLMG/inputs Tests/LinearSolvers/MLMG/inputs.boxes Tests/LinearSolvers/MLMG/main.cpp Tests/LinearSolvers/MLMG/prob_par.H Tests/LinearSolvers/MLMG/solve_with_mlmg.cpp Tests/LinearSolvers/MLMG/write_plotfile.cpp commit 1a5b308c8af78386f11c14edb72266ced66b0782 Author: Weiqun Zhang Date: Tue Dec 6 09:08:11 2022 -0800 Add Hypre CI (#3048) .github/workflows/hypre.yml commit 4a53367b19e11c42fe0a66abde4d9961a8ac3337 Author: Weiqun Zhang Date: Thu Dec 1 17:34:55 2022 -0800 Print host name in backtrace files (#3054) This can help identify which nodes generated the backtrace files. Note that the MPI rank is embedded in the file name, but it does not tell us which node it is. Src/Base/AMReX_BLBackTrace.cpp commit ccb21a03196ea604ff3316224029237a218e5efd Author: Weiqun Zhang Date: Thu Dec 1 08:50:06 2022 -0800 Update CHANGES for 22.12 (#3053) CHANGES commit 88fe04f00600407e490f66abf446da5831d1ce7e Author: Nuno Miguel Nobre Date: Tue Nov 29 03:24:30 2022 +0000 SYCL: Remove floating-point specialisations for atomic adds (#3050) Src/Base/AMReX_GpuAtomic.H commit 4d6413c45fa0e1aa6f366a02d75a9e2382c73850 Author: Brandon Runnels Date: Tue Nov 22 17:25:31 2022 -0700 Template average down (#2980) Adding templating for the average_down function Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H commit bd0cef1f097e939997ffac3e29f61b572c0a0b99 Author: Weiqun Zhang Date: Tue Nov 22 12:04:01 2022 -0800 Fix FabArray::LocalCopy SFINAE (#3047) Fix a bug in FabArray::LocalCopy SFINAE so that FAB in FabArray does not need to have typename value_type defined. Note that BaseFab has value_type, but FAB is not necessarily BaseFab. For convenience, IsBaseFab_v and IsFabArray_v are added. Src/Base/AMReX_FabArray.H Src/Base/AMReX_TypeTraits.H commit a7c63f2fc3a5dca442aa14c1d1af2f6b98e82592 Author: Michael Zingale Date: Tue Nov 22 11:51:59 2022 -0500 fix compilation of RichardsonConvergenceTest (#3046) Tools/C_util/Convergence/RichardsonConvergenceTest.cpp commit ed095ca21072b38fe6b7d8892bcc022ace531631 Author: Weiqun Zhang Date: Mon Nov 21 09:44:22 2022 -0800 Refactor Boundary Registers (#3028) Refactor boundary registers so that they can be used for single precision. - Remove MacBndry because it's no longer being used. - Make FabSet, BndryRegister, BndryData, and InterpBndryData templates with FabArray type as a parameter. The original names are now alias to FabSetT etc. - For historical reasons these classes were virtual. However, they do not need to be polymorphic anymore. - Remove unnecessary member functions. Src/AmrCore/AMReX_FluxRegister.H Src/Base/AMReX_MultiFab.H Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryData.cpp Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_BndryRegister.cpp Src/Boundary/AMReX_BoundaryFwd.H Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_FabSet.cpp Src/Boundary/AMReX_InterpBndryData.H Src/Boundary/AMReX_InterpBndryData.cpp Src/Boundary/AMReX_InterpBndryData_1D_K.H Src/Boundary/AMReX_InterpBndryData_2D_K.H Src/Boundary/AMReX_InterpBndryData_3D_K.H Src/Boundary/AMReX_LOUtil_K.H Src/Boundary/AMReX_MacBndry.H Src/Boundary/AMReX_MacBndry.cpp Src/Boundary/CMakeLists.txt Src/Boundary/Make.package Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMGBndry.H commit fb3227c8d3b92c5ca6deb6f547dee1ea3c469946 Author: Weiqun Zhang Date: Mon Nov 21 09:36:58 2022 -0800 Fix Multi-Level Open BC Solver (#3032) In the open BC solver, the enlarged level 0 grids are shifted so that the lower left corner is at (0,0,0). This allows the grids to be coarsened more than if they are not shifted. However, we did not shift the fine level grids and this was a bug. This bug is fixed here. Close #3023 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/LinearSolvers/OpenBC/AMReX_OpenBC.H Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp commit cf0afb0c152e2c942073731da7a1e0007886eed4 Author: Weiqun Zhang Date: Mon Nov 21 09:11:54 2022 -0800 Template average_down_faces (#3040) They can now be used on floats when amrex::Real is double. Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H commit e65b63e61fa0597c187fc08be50eeb11151280ba Author: Weiqun Zhang Date: Mon Nov 21 09:07:01 2022 -0800 Add FabArray::LocalCopy and LocalAdd (#3043) These are non-static member functions. LocalCopy can also be used to do mixed precision copy. Src/Base/AMReX_FabArray.H commit ccc9bd2f92fa9b2d277ac364a10c2227181599c4 Author: Weiqun Zhang Date: Mon Nov 21 09:01:23 2022 -0800 FabArray::sum (#3041) Add FabArray::sum so that we can compute sum for FabArray>. Src/Base/AMReX_FabArray.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp commit 9b6db7e1508a68310e8ee7063547c3912a82bb7b Author: Weiqun Zhang Date: Mon Nov 21 08:59:24 2022 -0800 amrex::Any::hasValue() (#3025) Add Any::hasValue() that checks whether the object contains a value. This is the analog of std::any::has_value. Src/Base/AMReX_Any.H commit f23e6a2330bf875643462105f3afc05fed28fad4 Author: Weiqun Zhang Date: Mon Nov 21 08:16:19 2022 -0800 Make YAFluxRegister template (#3030) Make YAFluxRegister a template so that it can be used for single-precision data when AMReX is built for double precision. The class is now called YAFluxRegisterT and YAFluxRegister is aliased to YAFluxRegisterT. So there are no changes in the user's code. Src/Base/AMReX_FabArray.H Src/Boundary/AMReX_BoundaryFwd.H Src/Boundary/AMReX_YAFluxRegister.H Src/Boundary/AMReX_YAFluxRegister.cpp Src/Boundary/AMReX_YAFluxRegister_1D_K.H Src/Boundary/AMReX_YAFluxRegister_2D_K.H Src/Boundary/AMReX_YAFluxRegister_3D_K.H Src/Boundary/CMakeLists.txt Src/Boundary/Make.package commit 75257bf75bcd291b78ee1ec425cd9ce11c6f37c1 Author: Weiqun Zhang Date: Mon Nov 21 08:14:24 2022 -0800 FabArray::norminf (#3039) Add FabArray::norminf so that we can compute infinity norm for single precision data when AMReX is built for double precision. Src/Base/AMReX_FabArray.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp commit de4ec8614b824cf044751c11e212cc277db151a4 Author: Weiqun Zhang Date: Mon Nov 21 08:13:05 2022 -0800 amrex::Dot (#3042) Add amrex::Dot function for computing the dot product of two FabArrays. Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_MultiFab.cpp commit 069d713539a32a216791f0a7eae1d7c8529ca685 Author: Weiqun Zhang Date: Mon Nov 21 08:11:38 2022 -0800 template FabArray::Saxpy, Xpay and LinComb (#3044) They can now be used on single precision FabArray>. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp commit 46e487c7297379eafb77ea8df7a9a0d825575978 Author: Malte Buschmann Date: Mon Nov 21 00:00:41 2022 -0500 Use std::max when coarsening a TagBox to preserve TagBox::SET values (#3045) When coarsening a TagBox in the current version, cells with the status SET are changed to BUF. This can cause undesirable effects if, e.g., TagBox::buffer is called on a coarsened TagBox. With this pull request, the {CLEAR, BUF, SET} hierarchy is preserved when coarsening a TagBox. Src/AmrCore/AMReX_TagBox.cpp commit 0d52da4dfede1eff9df65cc8c8ab4155a61d5fc6 Author: Weiqun Zhang Date: Wed Nov 16 17:51:50 2022 -0800 Fix type conversion warnings (#3034) Src/Base/AMReX_COORDSYS_1D_C.H Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_MultiFabUtil_1D_C.H commit 6d9dcd94c831b649d22d94257793ed9d7f8adf62 Author: Weiqun Zhang Date: Wed Nov 16 11:07:02 2022 -0800 Update for oneAPI 2023 (#3024) A number of things have been deprecated and need to be updated. - dpcpp -> icpx - sycl::gpu_selector -> sycl::gpu_selector_v - sycl::info::device::max_work_item_sizes -> sycl::info::device::max_work_item_sizes<3> - sycl::accessor<...,sycl::access::target::local> -> sycl_local_accessor - CL/sycl.hpp -> sycl/sycl.hpp Also remove AMREX_GPU_EXTERNAL because SYCL_EXTERNAL never worked for us. .github/workflows/intel.yml Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_GpuAssert.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_Math.H Src/Base/AMReX_RandomEngine.H Tests/GPU/AnyOf/GNUmakefile Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/sites/Make.unknown commit d8bc97f92a1a568b6e996db3b8d9715fced0464f Author: Ben Wibking Date: Sun Nov 13 12:39:58 2022 -0500 min/max- and linear combination-preserving interpolater (#3020) ## Summary This linearly interpolates onto fine grids, ensuring no new minima and maxima are created in multi-dimensions and also preserving linear combinations of components in each cell. The current implementation is only for Cartesian geometry. ## Additional background `CellConservativeLinear` can do each of these things separately (i.e., `CellConservativeLinear(0)` or `CellConservativeLinear(1)`), but does not preserve both of these properties at the same time. Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_MFInterpolater.cpp commit b0e1f8f354b19bc5ecaba0529dc78c5c5bed2ee1 Author: Andrew Myers Date: Sun Nov 13 09:03:30 2022 -0800 add a new test for particle checkpoint / restart in the native format (#3027) This fills a gap in the test suite indicated in #3026. Tests/HDF5Benchmark/main.cpp Tests/Particles/CheckpointRestart/CMakeLists.txt Tests/Particles/CheckpointRestart/GNUmakefile Tests/Particles/CheckpointRestart/Make.package Tests/Particles/CheckpointRestart/inputs Tests/Particles/CheckpointRestart/main.cpp commit 34d6e204e9bac9f97c42f9ed443c4e6a2deb0258 Author: Weiqun Zhang Date: Fri Nov 11 18:26:20 2022 -0800 template Xpay (#3031) Convert MultiFab::Xpay (y = x + a*y) to template amrex::Xpay. Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_MultiFab.cpp commit e0afc82f63ce51e0f10aef21d8ab163eb87547d1 Author: Axel Huebl Date: Tue Nov 8 16:41:39 2022 -0800 CMake: AMReX_BUILD_SHARED_LIBS (#3013) ## Summary Add a new variable `AMReX_BUILD_SHARED_LIBS` for per-project control of shared lib builds. This variable is more specific (AMReX project) than `BUILD_SHARED_LIBS` (whole build, defaults) and `USE_XSDK_DEFAULTS` (whole build, changed defaults). We need this variable so we can control the AMReX library build in superbuilds without changing all other libraries we build in other projects of the superbuild. ## Additional background - https://github.com/ECP-WarpX/impactx/pull/123 - https://github.com/ECP-WarpX/impactx/pull/208 Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.cmake commit 5e7ffd05ef36bb0fdbafa02bb7b65f58d909baf5 Author: Phil Miller Date: Tue Nov 8 13:05:59 2022 -0800 Fix mis-matched array type that broke mixed-precision restarts (#3026) ## Summary A mixed-precision build of WarpX (DP fields, SP particles) would fail to restart from a checkpoint with added real particle components, crashing while reading in data. ## Additional background Isolated with this modification of a WarpX example, and this fix tested similarly with `amr.restart = blah` ``` $ diff -u ~/repos/WarpX/Examples/Physics_applications/capacitive_discharge/inputs_2d inputs_2d --- /home/ubuntu/repos/WarpX/Examples/Physics_applications/capacitive_discharge/inputs_2d 2022-09-07 22:02:51.200174154 +0000 +++ inputs_2d 2022-11-07 21:52:04.764307593 +0000 @@ -39,6 +39,8 @@ electrons.momentum_distribution_type = maxwell_boltzmann electrons.theta = (kb*30000/(m_e*clight^2)) +electrons.save_previous_position = 1 + he_ions.species_type = helium he_ions.charge = q_e he_ions.injection_style = nuniformpercell @@ -55,24 +57,28 @@ -diagnostics.diags_names = diag1 +diagnostics.diags_names = diag1 ckpt diag1.diag_type = Full diag1.intervals = 50 diag1.intervals = 10 diag1.fields_to_plot = rho_electrons rho_he_ions + +ckpt.diag_type = Full +ckpt.intervals = 7 +ckpt.format = checkpoint ``` ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate Src/Base/AMReX_GpuContainers.H Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Tests/Particles/AsyncIO/main.cpp Tests/Particles/ParticleTransformations/main.cpp commit 81e0635ce83297d4e78b465e620f058a0c3e3845 Author: Weiqun Zhang Date: Mon Nov 7 09:36:12 2022 -0800 Add amrex::demangle for demangling C++ names (#3022) It can turn "N5amrex3BoxE" into "amrex::Box", and "Z4mainEUlvE_" into "main::{lambda()#1}". Src/Base/AMReX_Demangle.H Src/Base/AMReX_Utility.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 0d3deeb5c75cade14c381ef620921beaa2604c11 Author: Weiqun Zhang Date: Sun Nov 6 10:36:17 2022 -0800 Add more functionalities to TypeList (#3021) Move TypeList to its own header and add more functionalities. - TypeList::size() returns the number of types in the TypeList. - TypeAt is the type at a given position of a TypeList. - ForEach applies a function to each type of a TypeList. - ForEachUntil applies a function to each type of a TypeList until true is returned. - operator+ concatenates two TypeLists. This is an existing capability. - CartesianProduct returns the Cartesian product of TypeLists. This is an existing capability with a change. The function now takes a number of TypeLists, instead of a single TypeList packed with these TypeLists. Src/Base/AMReX_CTOParallelForImpl.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_TypeList.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 6ffb2e57c8233c8f6fcdb22c8dcd4a256fc2b66b Author: Kevin Gott Date: Sat Nov 5 10:02:48 2022 -0700 Add muller to NERSC machines (#3019) Adding a NERSC test machine to the list. Tools/GNUMake/Make.machines commit c71fc4c7643be908bd2a325cb192d36b4a206578 Author: Weiqun Zhang Date: Fri Nov 4 19:03:27 2022 -0700 Fix -lquadmath in GNU Make (#3018) Some machines (e.g., arm64) may not have the quadmath library. Close #3017 Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/llvm.mak commit 73b67447d76bd2f23351adc9a10e70c04029f9ff Author: Weiqun Zhang Date: Fri Nov 4 10:02:24 2022 -0700 Fix a typo in doc (#3016) Docs/sphinx_documentation/source/AmrLevel.rst commit 2795f8a2f3ec82cce099251d145ded12ebd97c7f Author: Weiqun Zhang Date: Fri Nov 4 09:24:23 2022 -0700 Fix EB tensor solver's boundary when EB is tilted. (#3002) It was incorrect to use the connectivity of valid cells to determine the stencil involving domain boundary registers. Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_K.H commit 6044e9c2bb9a365b9708d791ac58cd4d04eff6ac Author: Weiqun Zhang Date: Wed Nov 2 17:10:12 2022 -0700 Add -Wmissing-include-dirs to gcc warning flags (#3012) .github/workflows/gcc.yml Tools/GNUMake/comps/gnu.mak commit d2dbc863ba77f35092a85d8357c568bcb200d5e7 Author: Weiqun Zhang Date: Wed Nov 2 15:05:55 2022 -0700 A better fix for the PETSc compilation issue (#3011) Use a pair of parentheses to work around PETSc's MPI macros. This is a follow-up on #3005. Src/Base/AMReX_ParallelReduce.H commit 09cb43f90f5f52ec0bbf9ec3a7d87d99e84ac707 Author: Erik Schnetter Date: Wed Nov 2 17:26:57 2022 -0400 Replace all `sprintf` by `snprintf` (#3010) ## Summary Change all `sprintf` to `snprintf`. This makes the code safer, although the current code seems to be correct. ## Additional background `sprintf` is deemed insecure and should not be used. In many cases, the code could be changed to use `std::ostringstream` instead; this would probably reduce the code size as well. Src/Amr/AMReX_AmrLevel.cpp Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Extern/ProfParser/AMReX_BLWritePlotFile.cpp commit 7db23ec9386de76392aba848de817ecb5af673af Author: Weiqun Zhang Date: Wed Nov 2 11:22:15 2022 -0700 Use std::less for comparison of pointers (#3007) In principle, using `<` for pointer comparison is UB. In practice, the compilers (e.g., clang) use `<` in their implementation. Src/Base/AMReX_CArena.H commit b816db6e085ff317222832f9cade89838b44aa78 Author: Weiqun Zhang Date: Wed Nov 2 10:28:07 2022 -0700 Add YAFluxRegister::getFineData (#3004) This will allow an application code to save and reset the fine data stored in YAFluxRegister. Src/Boundary/AMReX_YAFluxRegister.H Src/Boundary/AMReX_YAFluxRegister.cpp commit c4a4811c373d9b599bb710c7029365b1ca7f2c22 Author: Axel Huebl Date: Tue Nov 1 14:08:38 2022 -0500 C++17 Transition (#2992) ## Summary Update AMReX to require C++17 or newer. - [x] docs - [x] CMake - [x] GNUmake - [x] CI ## Additional background Requires a mature [C++17](https://en.wikipedia.org/wiki/C%2B%2B17) compiler, e.g., GCC 8, Clang 7, NVCC 11.0, MSVC 19.15 or newer. Already used since 1+ year in production by downstream codes such as Castro and WarpX. Needed for modernization and new features such as #2878 Co-authored-by: Weiqun Zhang .github/workflows/clang.yml .github/workflows/cuda.yml .github/workflows/dependencies/dependencies.sh .github/workflows/dependencies/dependencies_clang7.sh .github/workflows/dependencies/dependencies_gcc8.sh .github/workflows/dependencies/dependencies_nofortran.sh .github/workflows/dependencies/dependencies_nvcc10.sh .github/workflows/dependencies/dependencies_nvcc11.sh .github/workflows/gcc.yml Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/BuildingAMReX_Chapter.rst INSTALL Src/Amr/AMReX_StateDescriptor.cpp Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CTOParallelForImpl.H Tools/AMRProfParser/GNUmakefile Tools/CMake/AMReXTypecheck.cmake Tools/CMake/AMReX_Config.cmake Tools/GNUMake/Make.rules Tools/GNUMake/comps/armclang.mak Tools/GNUMake/comps/cray.mak Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/intel.mak Tools/GNUMake/comps/llvm-flang.mak Tools/GNUMake/comps/llvm.mak Tools/GNUMake/comps/nag.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/comps/nvhpc.mak Tools/GNUMake/comps/pgi.mak Tools/Plotfile/CMakeLists.txt commit d2b82938c171a4b1ada48839ed6891b5b0183b43 Author: Weiqun Zhang Date: Tue Nov 1 09:01:54 2022 -0700 Update CHANGES for 22.11 (#3006) CHANGES commit 5ec270b4d534a486aeabf478ae553f1df53f2e5b Author: Weiqun Zhang Date: Tue Nov 1 08:59:44 2022 -0700 Fix compilation for PETSc (#3005) We cannot include PETSc headers too early because it might redefine MPI routines as macros (https://github.com/petsc/petsc/blob/main/include/petsclog.h#L441). They break MPI calls like below, MPI_Allreduce(&tmp, &vi, 1, ParallelDescriptor::Mpi_typemap::type(), ParallelDescriptor::Mpi_op>(), comm); because of the `,` in `>`. Src/Extern/PETSc/AMReX_PETSc.cpp commit 735c3513153f1d06f783e64f455816be85fb3602 Author: Weiqun Zhang Date: Sat Oct 29 10:57:23 2022 -0700 MPI Reduce for ValLocPair (#3003) Add ParallelReduce::Min, ParallelReduce::Max, ParallelAllReduce::Min, and ParallelAllReduce::Max for ValLocPair, where TV and TI are types that have corresponding MPI types (e.g., int, Real, IntVect, Box, etc.). Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParallelReduce.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_ValLocPair.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 3ec07681574afa658f4f53117d7ab618459a514b Author: Axel Huebl Date: Wed Oct 26 16:49:40 2022 -0700 `FabArray::isDefined` (#2997) ## Summary Add a new query to `define_function_called`. ## Additional background This is a cheaper check than `ok()` for finding out if a MultiFab has been allocated or not yet, assuming that the calling code follows the convention that `define()` is called collectively. Update: It turns out you can also call `empty` inherited from `FabArrayBase`. The new API is quite explicit, which is ok, too. Co-authored-by: Weiqun Zhang Src/Base/AMReX_FabArray.H commit 7f3c90893d70ca33c6dec499436dd503c77eeddf Author: Weiqun Zhang Date: Wed Oct 26 16:40:16 2022 -0700 Make The_Device_Arena non-managed (#2998) The_Device_Arena used to be a separate Arena. We changed it to be an alias of The_Arena to avoid memory fragmentation. However, the issue is we don't have an Arena that can allocate non-managed memory unless The_Arena is not managed. Because of performance concerns, we sometimes want to allocate non-managed memory. Therefore, we make The_Device_Arena an alias if and only if The_Arena is not managed. Src/Base/AMReX_Arena.cpp Src/Base/AMReX_GpuContainers.H commit ab8c892e1dd8943a6f0f759693757c6a186668a7 Author: Weiqun Zhang Date: Wed Oct 26 15:59:39 2022 -0700 Add alias template Gpu::NonManagedDeviceVector (#2999) Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_GpuContainers.H commit b3e0a62ba4d8c66b7cc40ab439b94835a5f4247c Author: Weiqun Zhang Date: Wed Oct 26 15:02:13 2022 -0700 Pre- and Post-interpolation hook interface (#2991) Support both Fab and MultiFab versions of pre- and post-interpolation hooks. Because the pre-interp hook might modify the data, we need to make a copy to avoid modifying cached coarse data. Close #2989. Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FillPatcher.H commit 3082028e42870b1ed37f0d26160ef078580511e3 Author: Weiqun Zhang Date: Wed Oct 19 19:24:10 2022 -0700 Update GitHub Actions (#2996) https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/ ## Summary ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate .github/workflows/clang.yml .github/workflows/cuda.yml .github/workflows/docs.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/sensei.yml .github/workflows/style.yml .github/workflows/windows.yml commit 0b88bfd3718ab226fc4a03a6598d320976744346 Author: Weiqun Zhang Date: Wed Oct 19 13:39:18 2022 -0700 Add user defined BC types (#2995) Add BCType::user_1, BCType::user_2 and BCType::user_3. Previously the only "user" type is ext_dir (external Dirichlet). The BC types are passed from the user's code to FillPatch, which in turn passes them back to the user provided BC filling function. These new types will make it easy for the user to determine the user defined BC types in their BC filling functions. Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/LinearSolvers.rst Src/Base/AMReX_BC_TYPES.H Src/Base/AMReX_bc_types_mod.F90 commit 9502b99cd98cc1aa70e5f19804c50252438ec1a6 Author: Weiqun Zhang Date: Tue Oct 18 10:20:06 2022 -0700 Add BCRec::set for convenience (#2993) Src/Base/AMReX_BCRec.H commit 56b6402d238979fca6e7c57fdc644a54c4cf6fce Author: Weiqun Zhang Date: Sat Oct 15 14:59:38 2022 -0700 ParallelFor with compile time optimization of kernels with run time parameters (#2954) Branches inside ParallelFor can be very expensive. If a branch uses a lot of resources (e.g., registers), it can significantly affect the performance even if at run time the branch is never executed because it affects the GPU occupancy. For CPUs, it can affect vectorization of the kernel. The new ParallelFor functions use C++17 fold expression to generate kernel launches for all run time variants. Only one will be executed. Which one is chosen at run time depends the run time parameters. The kernel function can use constexpr if to discard unused code blocks for better run time performance. Here are two examples of how to use them. int runtime_option = ...; enum All_options : int { A0, A1, A2, A3}; // Four ParallelFors will be generated. ParallelFor(TypeList>{}, {runtime_option}, box, [=] AMREX_GPU_DEVICE (int i, int j, int k, auto control) { ... if constexpr (control.value == A0) { ... } else if constexpr (control.value == A1) { ... } else if constexpr (control.value == A2) { ... else { ... } ... }); and int A_runtime_option = ...; int B_runtime_option = ...; enum A_options : int { A0, A1, A2, A3}; enum B_options : int { B0, B1 }; // 4*2=8 ParallelFors will be generated. ParallelFor(TypeList, CompileTimeOptions > {}, {A_runtime_option, B_runtime_option}, N, [=] AMREX_GPU_DEVICE (int i, auto A_control, auto B_control) { ... if constexpr (A_control.value == A0) { ... } else if constexpr (A_control.value == A1) { ... } else if constexpr (A_control.value == A2) { ... else { ... } if constexpr (A_control.value != A3 && B_control.value == B1) { ... } ... }); Note that that due to a limitation of CUDA's extended device lambda, the constexpr if block cannot be the one that captures a variable first. If nvcc complains about it, you will have to manually capture it outside constexpr if. The data type for the parameters is int. Thank Maikel Nadolski and Alex Sinn for showing us the meta-programming techniques used here. Src/Base/AMReX_CTOParallelForImpl.H Src/Base/AMReX_GpuLaunch.H Src/Base/CMakeLists.txt Src/Base/Make.package Tests/CMakeLists.txt Tests/CTOParFor/CMakeLists.txt Tests/CTOParFor/GNUmakefile Tests/CTOParFor/Make.package Tests/CTOParFor/main.cpp commit bcbf17f1cee4cd3209552cd0cafb2558c9254f20 Author: Weiqun Zhang Date: Fri Oct 14 19:48:14 2022 -0700 2D RZ solver for WarpX: Arbitrary coefficient (#2986) The assumption in the 2D RZ solver for WarpX used to be there was no sigma_r (i.e., sigma_r == 1). In this PR, we allow arbitrary sigma_r coefficient. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 9a3cd5d985ad357ab78d8f06f397cfc741448fdc Author: Axel Huebl Date: Fri Oct 14 17:27:41 2022 -0700 CMake Docs: Fix User-Guidance (Link) (#2990) Update the user-guidance on CMake dependency linking to CMake 3.0+ (anno. 2014+). Seen in #2978 Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GPU.rst commit 1ad4144668b0656d42950be92936073c64c56db7 Author: Weiqun Zhang Date: Fri Oct 14 10:36:17 2022 -0700 Runge-Kutta support for AMR (#2974) This adds RK2, RK3 and RK4 in a new namespace RungeKutta. Together with the enhanced FillPatcher class, these functions can be used for RK time stepping in AMR simulations. A new function AmrLevel::RK is added for AmrLevel based codes. See CNS::advance in Tests/GPU/CNS/CNS_advance.cpp for an example of using the new AmrLevel::RK function. The main motivation for this PR is that ghost cell filling for high order (> 2) RK methods at coarse/fine boundary is non-trivial when there is subcycling. Co-authored-by: Jean M. Sexton Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/AmrCore/AMReX_FillPatcher.H Src/Base/AMReX_RungeKutta.H Src/Base/CMakeLists.txt Src/Base/Make.package Tests/GPU/CNS/Source/CNS.H Tests/GPU/CNS/Source/CNS.cpp Tests/GPU/CNS/Source/CNS_advance.cpp Tests/GPU/CNS/Source/diffusion/CNS_diffusion_K.H commit c841ae81ddd519c088b29523aa71b6b280da440e Author: Weiqun Zhang Date: Fri Oct 14 10:03:34 2022 -0700 Fourth-order interpolation from fine to coarse level (#2987) For fourth-order finite-difference methods with data at cell centers, we cannot use the usual averageDown function to overwrite coarse level data with fine data. We actually need to do interpolation. Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp commit 975b830a012e4677d070b46d2f92353c117ad65a Author: Weiqun Zhang Date: Fri Oct 14 09:53:22 2022 -0700 Fix EB data inconsistency when fixing small cells and multiple cuts (#2943) ## Summary For consistency, we need to call the function that zeros out the level set even if that box does not have any small cells or multiple cuts. This is because a node could exist in multiple boxes. Furthermore, a covered cell or covered face may have a node with a level set < 0. ## Additional background This is usually not an issue. However, in WarpX, we use the level set to decide whether a node is an unknown in the linear system. The inconsistency makes the solver fail in some cases. Src/EB/AMReX_EB2_3D_C.cpp commit 9c2264bb5ff60b353250b3654866aef06f93bdcc Author: Axel Huebl Date: Fri Oct 14 07:41:06 2022 -0700 `MFIter::Finalize`: Free `m_fa` (#2988) This `free` should potentially not be delayed until the destructor is called. Follow-up to #2985 #2983 Src/Base/AMReX_MFIter.cpp commit f84c7a8f77d6f80f6f8ba4ee9161ee5a73a839a5 Author: Weiqun Zhang Date: Wed Oct 12 10:44:11 2022 -0700 Fix MLMG::getGradSolution & getFluxes for inhomogeneous Neumann and Robin BC (#2984) Because of the way how inhomogeneous and Robin BC are handled, we must add the inhomogeneous fluxes back, otherwise they would be zero at those boundaries. Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp commit ed1ecd62acb3fd7d39b8a23aa4e9ad09669741bb Author: Axel Huebl Date: Wed Oct 12 08:46:34 2022 -0700 MFIter: Make Finalize Public (#2985) Follow-up to #2983 Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit 5acfe07a830305cc7cbafd1e5dd26e3c3598435b Author: Axel Huebl Date: Tue Oct 11 14:51:48 2022 -0700 MFIter::Finalize (#2983) Add a Finalize function to MFIter. The idea about this is, that we can call this already before destruction in Python, where `for` loops do not create scope. This function must be robust enough to be called again in the constructor (or we need to add an extra bool to guard that it is not called again in the destructor). Co-authored-by: Weiqun Zhang Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit 53e34d17913cc76bdd4bbaad1582dd1b04058914 Author: Andy Nonaka Date: Tue Oct 11 12:00:34 2022 -0700 fix docs; Robin BC's for MLMG (#2982) Update the MLMG Robin BC description in the docs. Docs/sphinx_documentation/source/LinearSolvers.rst commit 0019b3a41065caf6d9486000b9c6fbf86ad9837e Author: Weiqun Zhang Date: Tue Oct 11 11:00:13 2022 -0700 MLLinOp::postSolve (#2981) Add a virtual function MLLinOp::postSolve. This allows WarpX to set EB covered nodes to prescribed values in the solver's output for visualization purpose. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 2d87a4c8ad5d375008ee9b1c23a50404fe0dfa21 Author: Brandon Runnels Date: Mon Oct 10 09:49:29 2022 -0600 add templating for the cell bilinear interpolators (#2979) This templates the `mf_cell_bilin_interp` functions so that the interpolators can be used with `BaseFab`s of arbitrary type. Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H commit e4ab0485621d5566c96cae58a816860ee7d4997f Author: Weiqun Zhang Date: Wed Oct 5 12:03:41 2022 -0700 FillPatcher class (#2972) This adds a class FillPatcher for filling fine level data. It's not as general as the various FillPatch functions (e.g., FillPatchTwoLevels). However, it can reduce the amount of communication data. Suppose we use RK2 with subcycling and the refinement ratio is 2. For each step on level 0, there are two steps on level 1. With RK2, each fine step needs to call FillPatch twice. So the total number of FillPatch calls is 4 in the two fine steps. Using the free function, one ParallelCopy per FillPatch call is needed for copying coarse data for spatial interpolation. With the FillPatcher class, two ParallelCopy calls will be done to copy old and new coarse data. Then these data will be used in the four FillPatcher::fill calls. This new approach saves two ParallelCopy calls per coarse step for a two levels run. It could save more if the time stepping requires more substeps or the refinement ratio is higher. Note that many of our AMReX codes use a time stepping algorithm that needs only one FillPatch call per step. For those codes, this new approach will not save any communication for a refinement ratio of 2. However, it will save communication when the refinement ratio is 4. Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatcher.H Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Tests/Amr/Advection_AmrCore/Source/AdvancePhiAllLevels.cpp Tests/Amr/Advection_AmrCore/Source/AdvancePhiAtLevel.cpp Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tests/Amr/Advection_AmrCore/Source/Src_K/Make.package Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp commit 1bc4e4eb5a25f4bdf9933695ead86f17dfdee9ed Author: Weiqun Zhang Date: Mon Oct 3 16:50:45 2022 -0700 Remove sycl namespace alias (#2971) This causes a conflict with new compilers. Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_Math.H Src/Base/AMReX_RandomEngine.H commit de7b7f44afda2227368a30646faeeea0d4679bec Author: Weiqun Zhang Date: Mon Oct 3 14:06:58 2022 -0700 Fix Tensor Solver BC (#2930) This fixes some bugs in the physical domain BC of tensor linear solver. At the corner of two no-slip walls (e.g., (0,0)), we have u(-1,0) = -u(0,0) and u(0,-1) = -u(0,0). It's incorrect to fill the corner ghost cell with u(-1,-1) = u(-1,0) + u(0,-1) - u(0,0), because it will result in u(-1,-1) = -3 * u(0,0). In the old approach, to avoid branches in computing transverse derivatives on cell faces, we fill the ghost cells first. For example, to compute du/dy at the lo-x boundary, we use the data in i = -1 and 0, just like we compute du/dy(i) using u(i-1) and u(i) for interior faces. The problem is the normal velocity in the ghost cells outside a wall is filled with extrapolation of the Dirichlet value (which is zero) and more than 1 interior cells. Because of the high-order extrapolation, u(-1) != -u(0). This is the desired approach for computing du/dx on the wall. However, this produces incorrect results in dudy. In the new approach, we explicitly handle the boundaries in the derivative stencil. For example, to compute transverse derivatives on an inflow face, we use the boundary values directly. Co-authored-by: cgilet Src/Base/AMReX_Orientation.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp_bc.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp_grad.cpp Src/LinearSolvers/MLMG/AMReX_MLTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_K.H commit 13aa4df0f5a4af40270963ad5b42ac7ce662e045 Author: Weiqun Zhang Date: Fri Sep 30 17:48:22 2022 -0700 Disable host device for macros for SYCL/DPC++ (#2969) The host part of the AMREX_HOST_DEVICE_FOR_* macros is disabled for SYCL/DPC++. It's really slow for compilation. Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/EB/AMReX_EB2_GeometryShop.H commit 62379fbac96867437070c4852d3d741a76dc1a4b Author: Weiqun Zhang Date: Fri Sep 30 15:37:35 2022 -0700 Update CHANGES for 22.10 (#2968) CHANGES commit d65e09e4a85dd2765a8cbe0ac9eba6223c47121b Author: Roberto Porcu <53792251+rporcu@users.noreply.github.com> Date: Thu Sep 29 15:46:19 2022 -0400 Solve an issue with particles async IO when having runtime added variables (#2966) Src/Particle/AMReX_WriteBinaryParticleData.H commit cd07b0d84244d08cf2690a19e0312f349ec0aeaa Author: Weiqun Zhang Date: Wed Sep 28 09:20:42 2022 -0700 Fix int overflow in amrex::bisect (#2964) Change from (lo+hi)/2 to lo+(hi-lo)/2. Although it's very unlikely, it's possible (lo+hi), where both lo and hi are integers, could overflow. Src/Base/AMReX_Algorithm.H commit e55d6b4f5375efb22ebed9b467878e301763073b Author: Junghyeon Park Date: Thu Sep 29 01:20:15 2022 +0900 Update the SWFFT project site (#2965) Docs/sphinx_documentation/source/SWFFT.rst commit b84d7c069cef7470f195b250926ca0e84ec46fb2 Author: Weiqun Zhang Date: Mon Sep 26 16:05:10 2022 -0700 Fix MLEBNodeFDLaplacian bottom solver (#2963) MLEBNodeFDLaplacian is never singular because it has Dirichlet boundary on the EB surface. We did set the singular flag to false, but forgot about the bottom solver used a different function to query. This fixes it by overriding the isBottomSingular function. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H commit 5e84f43241edfec7754d3ebfc369154bf249d992 Author: Ann Almgren Date: Sun Sep 25 09:38:51 2022 -0700 make tagging routines EB_aware (#2962) Src/AmrCore/AMReX_ErrorList.cpp Src/Base/AMReX_VisMF.H Src/Particle/AMReX_ParticleInit.H commit 8b367b0071787f8688d6f7eac55f7be251de6841 Author: Weiqun Zhang Date: Sun Sep 25 09:22:13 2022 -0700 Volume weighted sum (#2961) Add a new function doing volume weighted sum across AMR levels. This may not be exactly what amrex application codes want. But it should work for many cases. Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp commit 2a3cc05dac916961b1a5ae4c18b21bacd889e7fc Author: Weiqun Zhang Date: Fri Sep 23 12:24:05 2022 -0700 CellData: data in a single cell (#2959) This adds struct CellData that allows for accessing data in a single cell in Array4. This is convenient sometimes because one can omit the i, j and k indices. It might also be faster sometimes because it can skip the repeated index calculation involving i,j,k. Src/Base/AMReX_Array4.H commit 27ef10654c4810fc7cfc0f941a3eec67b018bf34 Author: Weiqun Zhang Date: Fri Sep 23 12:23:34 2022 -0700 Quartic interpolation for cell centered data (#2960) New Interpolator for interpolation of cell centered data using a fourth-degreee polynomial. Note that the interpolation is not conservative and does not do any slope limiting. Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp commit c4b7982d067497cc97ccb501ec08720b404d957e Author: Luca Fedeli Date: Fri Sep 23 21:17:12 2022 +0200 Add GPU-compatible upper bound and lower bound algorithms to AMReX_Algorithm (#2958) Src/Base/AMReX_Algorithm.H commit 3e5cc778028030ecb06bb079c5a6045f8f5fba6e Author: Donald E. Willcox Date: Tue Sep 20 17:59:48 2022 -0700 add option for makebuildsources to specify the style arguments for 'git describe'. (#2957) Tools/C_scripts/makebuildinfo_C.py commit a6e0c11989d34b976245db5719eedd0e9040f264 Author: Weiqun Zhang Date: Tue Sep 20 10:01:21 2022 -0700 Add more warnings (#2956) * Add -Wnon-virtual-dtor -Wlogical-op -Wmisleading-indentation -Wduplicated-cond -Wduplicated-branches to gcc. * Add -Wnon-virtual-dtor to clang. * Add more warnings to CI. * Fix some non-virtual dtors and some other warnings. .github/workflows/clang.yml .github/workflows/cuda.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/macos.yml Src/AmrCore/AMReX_ErrorList.H Src/EB/AMReX_distFcnElement.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/llvm.mak commit 826cd378f8ba0d844c64e1029f7914c3b066debd Author: Phil Miller Date: Thu Sep 15 17:26:00 2022 -0700 Add roundoff_lo corresponding to roundoff_hi for domains that don't start at 0 (#2950) * Lay groundwork for roundoff_lo * Add dummy implementation of roundoff_lo computation * implement bisect_prob_lo * change idx -> dxinv * use rlo instead of plo in locateParticle Co-authored-by: atmyers Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H commit 6a5a0561076f62af588d5a3d54f0deb232f3a6af Author: Weiqun Zhang Date: Thu Sep 15 13:23:40 2022 -0700 Add template parameter to ParallelFor and launch specifying block size (#2947) By default, amrex::ParallelFor launches AMREX_GPU_MAX_THREADS threads per block. We can now explicitly specfiy the block size with `ParallelFor(...)`, where BLOCK_SIZE should be a multiple of the warp size (e.g., 64, 128, etc.). A similar change has also been made to `launch`. The changes are backward compatible. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_Reduce.H commit 2cdb9df08e4668bbc9a9b6560217514518f41573 Author: Andrew Myers Date: Thu Sep 15 10:55:41 2022 -0700 Byte spread fixes (#2949) Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_ParticleContainerI.H commit 17c94cc196d779e9f7ec48f7d004088a1c1e11c6 Author: Candace Gilet Date: Wed Sep 14 11:49:35 2022 -0400 Correct MultiFab::norm0 doxygen brief description (#2946) Src/Base/AMReX_MultiFab.H commit 0351c9958be7fdc7e3e0c419fc68d36a0c00f288 Author: Axel Huebl Date: Wed Sep 14 08:48:25 2022 -0700 CMake: HIP_PATH from ROCM_PATH (#2948) * On machines like Crusher, `ROCM_PATH` is more likely to be available then a `HIP_PATH` environment variable. This is mainly needed for our hacky ROCTX hints. * ROCTX: New Include Supposedly, there is a new include we shall use: Ref.: https://github.com/ROCm-Developer-Tools/roctracer/issues/79 * ROCtracer: Include as System library Because of GNU extensions in the roctracer include files for the legacy include. But we should make this `-isystem` anyway to be robust for the future. The 5.2 deprecated include file `` throws warnings because they rely on GNU extensions: ``` In file included from /opt/rocm/hip/../roctracer/include/ext/prof_protocol.h:27: /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:70:7: warning: anonymous structs are a GNU extension [-Wgnu-anonymous-struct] struct { ^ /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:70:7: warning: anonymous types declared in an anonymous union are an extension [-Wnested-anon-types] /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:75:7: warning: anonymous types declared in an anonymous union are an extension [-Wnested-anon-types] struct { ^ /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:82:7: warning: anonymous structs are a GNU extension [-Wgnu-anonymous-struct] struct { ^ /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:86:7: warning: anonymous structs are a GNU extension [-Wgnu-anonymous-struct] struct { ^ /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:90:7: warning: anonymous structs are a GNU extension [-Wgnu-anonymous-struct] struct { ^ /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:82:7: warning: anonymous types declared in an anonymous union are an extension [-Wnested-anon-types] struct { ^ /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:86:7: warning: anonymous types declared in an anonymous union are an extension [-Wnested-anon-types] struct { ^ /opt/rocm/hip/../roctracer/include/ext/../../../include/roctracer/ext/prof_protocol.h:90:7: warning: anonymous types declared in an anonymous union are an extension [-Wnested-anon-types] struct { ^ ``` * GNUmake: Update Includes in `hip.mak` Use public prefix. .github/workflows/hip.yml Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_TinyProfiler.H Tools/CMake/AMReXParallelBackends.cmake Tools/GNUMake/comps/hip.mak commit 9aa23c202a13eee489a06030b9aeda6b89856944 Author: Cody Balos Date: Mon Sep 12 11:49:37 2022 -0700 Fix minor typo in fcompare docs (#2945) Docs/sphinx_documentation/source/Post_Processing.rst commit bfbd68f4ed31ca07572be9bf138a59cacb7e800c Author: Axel Huebl Date: Mon Sep 12 11:40:55 2022 -0700 Fix: Make Finalize->Initialize->F->I->... Work (#2944) Fix assertions in Arena::Initialize. The_BArena never dies (tm) Co-authored-by: Weiqun Zhang Src/Base/AMReX_Arena.cpp commit 67384701a808ca973ad2c24ec86cee4c7a81fd05 Author: Weiqun Zhang Date: Wed Sep 7 14:12:34 2022 -0700 Changes for Cray & Clang (#2941) * It seems that the new Cray compilers no longer define `_CRAYC`. However it does define `__cray__`. * For Clang based Cray compilers, use -O3 instead of -O2 for optimization. * Clang's vectorization pragma is very aggressive. For some codes, it makes ParallelFor with many if statements on CPU much slower than without vectorization. Unfortunately, it does not have an ivdep pragma. So we disable AMREX_PRAGMA for clang for safety. * No longer need to use -Wno-pass-failed for Clang based compilers. .github/workflows/hip.yml .github/workflows/macos.yml Src/Base/AMReX_Extension.H Tools/CMake/AMReXFlagsTargets.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/armclang.mak Tools/GNUMake/comps/cray.mak Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/llvm.mak commit 5b0c598cc71a5e914bfc4dbb7ea44313d45c8f57 Author: Weiqun Zhang Date: Wed Sep 7 09:42:57 2022 -0700 Fix a warning in packing communication send buffer (#2940) When we communication double precision data in single precision, there is a conversion from double to float in packing the send buffer. A static cast is added to fix the warning. Src/Base/AMReX_FBI.H commit 3e397bb6ba2854245a10d49a5ee37e1ba9f33f0e Author: Weiqun Zhang Date: Wed Sep 7 09:13:53 2022 -0700 Link to cublas when using CUDA and Hypre (#2933) Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp Tools/GNUMake/packages/Make.hypre commit 9525ea8892b9c0910acc2bf2ae8950f6068c34e5 Author: Weiqun Zhang Date: Wed Sep 7 09:13:20 2022 -0700 HIP: use coarse grained host memory (#2932) Src/Base/AMReX_Arena.cpp commit 7e040166efc8208e60d8796d4d99b1dd47146ef2 Author: Marco Garten Date: Wed Sep 7 08:53:20 2022 -0700 Update Testing Docs (#2937) - document `abort_on_unused_inputs` - remove duplicate superfluous argument in regtest call Docs/sphinx_documentation/source/Testing.rst commit 539427a19b20e49c4f7399c8ea0b0515fb5c79a0 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Tue Sep 6 18:13:42 2022 -0400 EB checkpoint files (#2897) * support for loading EB from checkpoint file * add support for writing chkpt file as well Co-authored-by: Weiqun Zhang Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_IndexSpace_chkpt_file.H Src/EB/AMReX_EB2_IndexSpace_chkpt_file.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EB2_Level_chkpt_file.H Src/EB/AMReX_EB2_Level_chkpt_file.cpp Src/EB/AMReX_EB_chkpt_file.H Src/EB/AMReX_EB_chkpt_file.cpp Src/EB/CMakeLists.txt Src/EB/Make.package commit 35ed6b4d343215c1ccf6e4d0a59813fc236c9f22 Author: Axel Huebl Date: Tue Sep 6 15:07:16 2022 -0700 Fix: Loading Files Again (#2936) This enables that `amrex::ParmParse::addfile` can be called multiple times. Before this, we accidentially overwrite the `FILE` static keyword. Follow-up to #2842 Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit 8f8198c2fb1868704d2b4d14b5b93d8d1d264ea0 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Tue Sep 6 13:36:35 2022 -0400 Check if boundary particles container has been created before clearance. (#2935) This fixes a segmentation fault when using more GPUs for updating particles than fluid. Src/Particle/AMReX_NeighborParticlesI.H commit fb0b31e1439b089074514f45ae900af257c66dba Author: Nuno Miguel Nobre Date: Sun Sep 4 05:18:49 2022 +0100 SYCL: Replace deprecated atomic types and operations (#2921) * SYCL: Replace deprecated atomic types and operations * Change atomic refs to device memory scope When using the relaxed memory order, the memory scope is ignored. Thus, for cosmetic reasons only, we set the memory scope to device, the broadest option when using the global address space. Co-authored-by: Weiqun Zhang Src/Base/AMReX_GpuAtomic.H commit cc3cd1470254d37f0cea4f212c2b0f6ffa8d0bee Author: Weiqun Zhang Date: Thu Sep 1 07:39:25 2022 -0700 Update CHANGES for 22.09 (#2934) CHANGES commit acc223f9918284e7d8e595d3861c5e456d84a968 Author: Weiqun Zhang Date: Tue Aug 30 16:04:43 2022 -0700 Add hypre as an option for OpenBCSolver (#2931) Src/LinearSolvers/OpenBC/AMReX_OpenBC.H Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp commit 3d29fd7d0e816f3c436112d90bdefe815e0ff72a Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Wed Aug 24 16:10:22 2022 -0400 Preserve neighbor particles when sorting particles. (#2923) Src/Particle/AMReX_ParticleContainerI.H commit 8294c3afbcbbc503f77e493196d380fbe1666d02 Author: Weiqun Zhang Date: Mon Aug 22 10:46:05 2022 -0700 Scope of NonLocalBC::ParallelCopy (#2922) Make NonLocalBC::ParallelCopy accessible in namespace amrex, because it can be useful in situations other than non-local BC. Src/Base/AMReX_MultiFab.H Src/Base/AMReX_NonLocalBC.H commit 0911fc4b2e066209a590c330bf2ddf7178dca76b Author: Weiqun Zhang Date: Sun Aug 21 18:13:07 2022 -0700 Open Boundary Poisson Solver (#2912) This adds an open boundary Poisson solver based on the James's algorithm. To use it, the user builds an amrex:OpenBCSolver object, which can be reused until the grids change, and then call OpenBCSolver::solver. Currently, this is for 3D cell-centered data only. The solver works on CPU, Nvidia GPUS, and AMD GPUs. The SYCL version of a couple of kernels for Intel GPUs are to be implemented. GNUmakefile.in Src/Base/AMReX_DistributionMapping.cpp Src/Boundary/AMReX_LOUtil_K.H Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/OpenBC/AMReX_OpenBC.H Src/LinearSolvers/OpenBC/AMReX_OpenBC.cpp Src/LinearSolvers/OpenBC/AMReX_OpenBC_K.H Src/LinearSolvers/OpenBC/Make.package commit f270b3d5db8f8b7ab010bc9134632361b8a9009c Author: Marc T. Henry de Frahan Date: Thu Aug 18 13:51:56 2022 -0600 Fix OOB access of ref ratio on HDF write header (#2919) Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp commit fa8e20f946b661bd49af2a60898ffca2c5b21cff Author: Jean M. Sexton Date: Thu Aug 18 08:57:51 2022 -0700 Add Polaris to GNUMake (#2908) Tools/GNUMake/Make.defs Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.alcf commit bd5f6a9f6a1a3a66c51eefd7950432d3bf3319a1 Author: Axel Huebl Date: Mon Aug 15 14:24:21 2022 -0700 Export GpuDevice Globals (#2918) * Export GpuDevice Globals Implement symbol export via `AMREX_EXPORT` for the global variables in `Src/Base/AMReX_GpuDevice.H`. Follow-up to #1847 #1847 Fix #2917 * Fix: Export `AMReX::m_instance` Src/Base/AMReX.H Src/Base/AMReX_GpuDevice.H commit 4f639294606d47185d31eaee4af66fc6b590e5a2 Author: Ann Almgren Date: Sat Aug 13 09:00:02 2022 -0700 enable LinOp to use the right Factory (fixes moving geometry problem) (#2916) Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp commit 659351846da6f930b4f04cc6cd6b9f78e7752e8a Author: Andrew Myers Date: Thu Aug 11 15:24:16 2022 -0700 Use 1 atomic instead of two per item in DenseBins::build (#2911) Src/Particle/AMReX_DenseBins.H commit d295f2299101705f7c470c813b80542296087328 Author: Nuno Miguel Nobre Date: Thu Aug 11 03:40:09 2022 +0100 [SYCL] Remove amrex::oneapi and update deprecated device descriptors (#2910) * Remove amrex::oneapi in favour of standard features * Change deprecated device descriptors Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_Scan.H commit 1bda173b489024d5f4ec79627f3f612c350e521f Author: Axel Huebl Date: Wed Aug 10 15:46:43 2022 -0600 Add: `MultiFab::sum_unique` (#2909) This provides a new method to sum values in a `MultiFab`. For non-cell-centered data, `MultiFab::sum` double counts box boundary values that are owned by multiple boxes. This provides a function that does not double count these and provides a quick way to get only the sum of physically unique values. Co-authored-by: Weiqun Zhang Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp commit 3f715d29c94b473e624aa9ff3fea9b502da25f97 Author: Candace Gilet Date: Mon Aug 8 14:40:28 2022 -0400 In MLMG::mgFcycle, assert that for EB the linop is cell-centered. (#2905) Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 59b0742b9b8c543b2896d76dd07e03c2fe4f1f94 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon Aug 8 14:17:57 2022 -0400 Clear the boundary particle indices' container before updating it. (#2907) This avoids potential segmentation faults when one grid's particles all move to other grids. Src/Particle/AMReX_NeighborParticlesI.H commit 103db6ebe2b570910ac4dbd7d6611e59d80f1a0b Author: Weiqun Zhang Date: Fri Aug 5 15:25:33 2022 -0700 EB: Add Fine Levels (#2881) Add a new function EB2::addFineLevels() that can be used to add more fine levels to the existing EB IndexSpace without changing the coarse levels. This is useful for restarting with a larger amr.max_level. The issue is we build EB at the finest level first and then coarsen it to the coarse levels. If the restart run has a different finest level, the EB on the coarse levels could be different without using this new capability. Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_IndexSpaceI.H Src/EB/AMReX_EB2_IndexSpace_STL.H Src/EB/AMReX_EB2_IndexSpace_STL.cpp Tests/EB/CNS/Source/main.cpp commit 6ebf8ffc2689e23ff2686627e660caf0a10ea315 Author: Jon Rood Date: Thu Aug 4 14:32:59 2022 -0600 Add rpath to lib64 for ZFP. (#2902) Tools/GNUMake/packages/Make.hdf5 commit ed23627d6487306e26b37ed9a97d60fd8148a935 Author: Yadong_Zeng <30739800+ruohai0925@users.noreply.github.com> Date: Thu Aug 4 16:32:21 2022 -0400 change data types from double to amrex::Real, and thus we can use single precision for the hypre IJ interface (#2896) Co-authored-by: yzeng Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit 9ed4f5955b1d5d0e400fd2f233e5e7b83db4e41b Author: Weiqun Zhang Date: Wed Aug 3 16:53:20 2022 -0700 Fix a new bug introduced in #2858 (#2901) We need to take into account that `amrex::Any` stores `MultiFab&` or `MultiFab const&`. Src/Base/AMReX_Any.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 6eaab8c1c9f0e2a21531526dfd170ebe3aad507b Author: Weiqun Zhang Date: Wed Aug 3 13:39:44 2022 -0700 MPMD Support (#2895) Add support for multiple programs multiple data (MPMD). For now, we assume there are only two programs (i.e., executables) in the MPMD mode. During the initialization, MPI_COMM_WORLD is split into two communicators. The MPMD::Copier class can be used to copy FabArray/MultiFab data between two programs. This new capability can be used by FHDeX to couple FHD with SPPARKS. Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_BoxList.H Src/Base/AMReX_MPMD.H Src/Base/AMReX_MPMD.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 94693291667bd0435819aa09cf28a293da226bf4 Author: Weiqun Zhang Date: Mon Aug 1 09:43:21 2022 -0700 MLMG interface (#2858) These changes are made to support a generic type (i.e., amrex::Any) in MLMG. This is still work in progress. But it should not break any existing codes. Src/Base/AMReX_Any.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FabArray.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_temp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit 5a3b3037950937343b7eafd292e5032cb8c7221c Author: Weiqun Zhang Date: Mon Aug 1 09:34:44 2022 -0700 Update CHANGES for 22.08 (#2894) CHANGES commit 48702b48836d9aeb0db931e23ea9cc7d4ad4ccdc Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Thu Jul 28 14:14:19 2022 -0400 Let `selectActualNeighbors` return right after starting if there are (#2886) no particles for communication. Src/Particle/AMReX_NeighborParticlesI.H commit 6a47d89fd12cb06d48e3e0d85eea415274e84a69 Author: Kevin Gott Date: Wed Jul 27 17:03:04 2022 -0700 Add Comm Sync to Redistribute (#2891) Src/Base/AMReX_FabArrayCommI.H Src/Particle/AMReX_ParticleContainerI.H commit 51542c85ac18642a2cfb69ea3df3cf544d3d6f42 Author: philip-blakely <46958218+philip-blakely@users.noreply.github.com> Date: Wed Jul 27 17:29:26 2022 +0100 Multi-materials and derived variable output (#2888) ## Summary Output small plots if only derived variables are specified. Also, make DeriveFuncFab a std::function<> instead of plain function-pointer. ## Additional background We have been implementing small-plots for outputing variables at gauges (e.g. pressure at specific gauge locations). We may want to output the derived variable pressure only, and not all state-variables. The if-condition was incorrect in this case. Further, multi-material simulations require a material index in order to compute derived variables, in addition to existing parameters. Making DeriveFuncFab a std::function is sufficient for our purposes. Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_Derive.H commit ce0fb7412dff3ceeec00941ba525e7ecf5ce8015 Author: Andrew Myers Date: Tue Jul 26 16:20:38 2022 -0700 Fix host / device sync bug in PODVector (#2890) Src/Base/AMReX_PODVector.H commit 06753e60aca7d063b28be93379c948e92afb8c5e Author: Axel Huebl Date: Tue Jul 26 12:54:35 2022 -0700 `TagBoxArray::collate`: Fujitsu Clang (#2889) `mpiFCC -Nclang` only defines `__CLANG_FUJITSU`, not `__FUJITSU` as in the classic compiler mode. Src/AmrCore/AMReX_TagBox.cpp commit 7cf77dc60e149ebe822f6b5428556f9208e150fa Author: Weiqun Zhang Date: Tue Jul 26 11:01:21 2022 -0700 MinLoc and MaxLoc Support (#2885) Add struct ValLocPair that can be used by ReduceOps/ReduceData and ParReduce to find the location of the min/max value. Add warp shuffle down function for more general types. This is needed for MinLoc/MaxLoc with CUDA < 11, because we don't use CUB for earlier versions of CUDA. The Intel GPU support is not done yet. We need to allocate enough shared local memory when the size of ValLocPair is larger than the size of unsigned long long. Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_Reduce.H commit 4b7e20057a3dff84beae21812d826d24e19f2109 Author: Weiqun Zhang Date: Thu Jul 21 10:25:57 2022 -0700 HIP: Remove the call to hipDeviceSetSharedMemConfig (#2884) AMD devices do not support shared cache banking. Thanks @afanfa for reporting this. (#2883) Src/Base/AMReX_GpuDevice.cpp commit 8e40952af9ab0600174f491c81100132f9b24c6e Author: Weiqun Zhang Date: Wed Jul 20 12:10:26 2022 -0700 Add Frontier to GNU Make (#2879) Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit b673d81723c5585a1290126233d38f50833939d7 Author: Max Katz Date: Mon Jul 18 15:14:19 2022 -0400 Add option to derefine to AMRErrorTag (#2875) This allows a refinement field to specify *derefinement* (by setting a zone's tagging value to the clear value). Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp commit 73dbf2f909bdc6c497eb5245b4e707b4814e699f Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon Jul 18 12:53:35 2022 -0400 Fix the segmentation fault in selecting actual neighbor particles. (#2877) Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H commit 40b3d2176b17785191050482a2ead5539993fac6 Author: Weiqun Zhang Date: Wed Jul 13 13:24:15 2022 -0700 Add extra braces in initialization of GpuArray (#2876) It should not be needed since C++14. But some compilers seem to need the double braces. Src/Base/AMReX_TableData.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H commit a633d2bff1db1a3335efd077a34b6a8dcfb4e793 Author: Luca Fedeli Date: Fri Jul 8 20:34:18 2022 +0200 Workaround to bypass issue observed at very large scale with Fujitsu MPI (#2874) We have observed some MPI issues at very large scale when WarpX is compiled using Fujitsu MPI (i.e., with the Fujitsu compiler). These issues seem to be related to the use of MPI Gatherv with MPI_Datatype. This PR implements a possible workaround, initially proposed by @WeiqunZhang . The idea is that, when WarpX is compiled with the Fujitsu compiler, simpler integer arrays instead of MPI_Datatype are used in the routine where the issue was observed. Src/AmrCore/AMReX_TagBox.cpp commit 7660c885d46779367344adf88af75e630a0bc77a Author: Weiqun Zhang Date: Fri Jul 8 08:48:14 2022 -0700 Allow zero components MultiFab and BaseFab (#2873) This is useful for particle I/O that does not have any mesh data. yt needs a header file associated with a MultiFab. Src/Base/AMReX_BaseFab.H commit c849dd1994388cebd78a6a1624e80bc3ab640970 Author: Weiqun Zhang Date: Fri Jul 8 08:06:37 2022 -0700 New EB optimization parameter: eb2.num_coarsen_opt (#2872) At the beginning of EB generation, we chop the entire finest domain into boxes and find out the type of the boxes. We then collect the completely covered boxes and cut boxes into two BoxArrays. This process can be costly because of the number of calls to the implicit functions. In this commit, we have introduced a new ParmParse parameter, eb2.num_coarsen_opt with a default value of zero. If for instance it is set to 3, we start the box type categorization at a resolution that is coarsened by a factor of 2^3. For the provisional cut boxes, we refine them by a factor of 2, Then we chop them into small boxes and categorize the new boxes. This process is performed recursively until we are at the original finest resolution. The users should be aware that, if eb2.num_coaren_opt is too big, this could produce in erroneous results because evaluating the implicit function on coarse boxes could miss fine structures in the EB. Thank Robert Marskar for sharing this algorithm. Src/Base/AMReX_Box.cpp Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_IndexSpaceI.H Src/EB/AMReX_EB2_IndexSpace_STL.H Src/EB/AMReX_EB2_IndexSpace_STL.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level_STL.H Src/EB/AMReX_EB2_Level_STL.cpp Tests/LinearSolvers/CellEB2/inputs.rt.2d Tests/LinearSolvers/CellEB2/inputs.rt.3d commit 557aae84902f63a84edc8b49831ee66af7d1a46a Author: Erik Date: Wed Jul 6 08:54:24 2022 -0700 point to new location of AMReX images, AMReX website repo (#2867) README.md commit cbdc6580ee3d78cccdd37172e4ba077ee181f483 Author: Axel Huebl Date: Tue Jul 5 01:41:03 2022 +0200 SENSEI 4.0: Fix Build for Particles (#2869) ## Summary This part causes a compile error now in WarpX. cc @burlen @kwryankrattiger ## Additional background X-ref: Blocks WarpX 22.07 release https://github.com/ECP-WarpX/WarpX/pull/3211 Follow-up to: - #2785 - #2834 Src/Extern/SENSEI/AMReX_AmrMeshParticleInSituBridge.H commit dc8b734b6a70583602150cfbee1b7d51f8dacdeb Author: Andrew Myers Date: Fri Jul 1 17:19:20 2022 -0700 Cache the neighbor comm tags for the CPU implementation of fillNeighbors. (#2862) * Cache the neighbor comm tags for the CPU implementation of fillNeighbors. * fix areMasksValid function Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H commit 2b42fb56a96e752d301916ca23160098c5369386 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Fri Jul 1 18:44:35 2022 -0400 Remove some hard checks in check_mvmc for 3D (#2864) Removing some hard checks in 3D coarsening logic as it appears that those are not necessarily bad states, and a soft failure to coarsen should suffice. Src/EB/AMReX_EB2_3D_C.H commit 19c70685cdb0c3322712e9f442092b1140cfe7ec Author: Erik Date: Fri Jul 1 18:24:24 2022 -0400 Carry over fix for ngbxy.smallEnd typo (#2868) This a typo that got correct in other places but didn't get fixed here. Tests/Amr/Advection_AmrCore/Source/DefineVelocity.cpp commit d736ef299b724b96b34d41103dfc5318d0ecdee4 Author: Weiqun Zhang Date: Fri Jul 1 11:00:15 2022 -0700 Update CHANGES for 22.07 (#2866) CHANGES commit be813d024e6b314e41c727734b8e53481898e08e Author: Weiqun Zhang Date: Fri Jul 1 10:29:13 2022 -0700 Hypre: add version check (#2865) These HYPRE_SetSp* are only available in hypre >= 22500. Src/Base/AMReX.cpp commit 8fb23ec17a58284af6bdafbcda3eea0d86d8ce69 Author: Jon Rood Date: Wed Jun 29 16:52:35 2022 -0600 Refactor Make.nrel to use MPT for MPI with the Intel compiler on Eagle. (#2861) Tools/GNUMake/sites/Make.nrel commit 6f9a46c7e834046970d46d684927a078671355bc Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Wed Jun 29 11:09:57 2022 -0600 Adding control APIs and namespacing for core algorithm paths like SpGEMM, SpMV, and SpTrans. (#2859) Co-authored-by: Paul Mullowney Src/Base/AMReX.cpp commit e4c83cfddc8afb1bd091c45a6ad3040d23f019bc Author: Jon Rood Date: Wed Jun 29 11:08:42 2022 -0600 Add lib64 library location for ZFP since it may exist there instead of lib. (#2860) Tools/GNUMake/packages/Make.hdf5 commit b2b9150ada12af878a07e0628be03668a9d17270 Author: Burlen Loring Date: Tue Jun 28 13:42:41 2022 -0700 update the SENSEI in situ coupling for SENSEI v4.0.0 (#2785) In this release, an install of VTK is no longer required. To compile AMReX w/ SENSEI use: ```cmake -DAMReX_SENSEI=ON -DSENSEI_DIR=//cmake ``` Note: may be `lib` or `lib64` or something else depending on your OS and is determined by CMake at configure time. See the CMake GNUInstallDirs documentation for more information. .github/workflows/sensei.yml Docs/sphinx_documentation/source/Visualization.rst Src/Extern/SENSEI/AMReX_AmrDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrInSituBridge.cpp Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrMeshInSituBridge.cpp Src/Extern/SENSEI/AMReX_AmrMeshParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrMeshParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_InSituUtils.H Src/Extern/SENSEI/AMReX_InSituUtils.cpp Src/Extern/SENSEI/AMReX_ParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_ParticleDataAdaptorI.H Tools/CMake/AMReXThirdPartyLibraries.cmake commit 2c5f475d451aede47fe2cad2bbd8681c9ca1f456 Author: Andrew Myers Date: Tue Jun 28 12:51:19 2022 -0700 Write runtime attribs to checkpoints on GPUs (#2856) Src/Particle/AMReX_WriteBinaryParticleData.H commit d2cb54668b5e49fd35a60164f40ad6f36720f806 Author: Jon Rood Date: Tue Jun 28 13:27:02 2022 -0600 Fix gnu make on Crusher for mpi_gtl_hsa (#2857) Update environment variable at OLCF for mpi_gtl_hsa. Tools/GNUMake/sites/Make.olcf commit 21fe4b3016a796b99c409760cfad7ae00a7475ba Author: Axel Huebl Date: Tue Jun 28 19:53:09 2022 +0200 CMake: FindDependency CUDAToolkit (#2849) If we install AMReX with CUDA support using a modern CMake, we need to repopulate targets such as `CUDA::curand` from `find_dependency` for downstream. Downstream users find us via `find_package` and that target link dependency showed up to be unpopulated in MFIX. Tools/CMake/AMReXConfig.cmake.in commit 027f2ff77fed33a191cfc735d8adaabb42d21743 Author: Weiqun Zhang Date: Thu Jun 23 16:15:57 2022 -0700 Fix make help (#2854) This reverts the change in #2845, which fixed an issue with `make print-%`, but broke `make help`. This is now fixed in a different way. Both `make print-%` and `make help` should work now. Tools/GNUMake/Make.rules Tools/GNUMake/sites/Make.nersc commit 3d3ad213ca4b60421c9a80328e1316b23435958f Author: Kevin Gott Date: Thu Jun 23 13:39:59 2022 -0700 NERSC Programming Environment prototype (#2848) Tools/GNUMake/sites/Make.nersc commit 487267625412e4f8a4fa1ab2492cb578955c4239 Author: Weiqun Zhang Date: Thu Jun 23 12:41:20 2022 -0700 GNU Make: No need to query mpif90 if Fortran is not used. (#2852) This minimize potential issues. Tools/GNUMake/sites/Make.unknown commit fc0d6469f4ad590d576a7109d8719b018838dd86 Author: Weiqun Zhang Date: Thu Jun 23 12:23:55 2022 -0700 Remove f90doc (#2851) We no longer use it. Tools/F_scripts/f90doc/README Tools/F_scripts/f90doc/expr_parse.pl Tools/F_scripts/f90doc/expr_parse.y Tools/F_scripts/f90doc/f90doc Tools/F_scripts/f90doc/htmling.pl Tools/F_scripts/f90doc/stmts.pl Tools/F_scripts/f90doc/typing.pl Tools/F_scripts/f90doc/utils.pl commit 5188a6a28e64dc627c3333d13bebeb0d7250b506 Author: Weiqun Zhang Date: Thu Jun 23 11:09:15 2022 -0700 Explicitly invoke python3 (#2850) According to PEP 394, a python distributor may choose to not provide the python command. In fact, that's what recent versions of macOS do. Tools/Backtrace/parse_bt.py Tools/C_scripts/describe_sources.py Tools/C_scripts/gatherbuildtime.py Tools/CompileTesting/compiletesting.py Tools/F_scripts/dep.py Tools/F_scripts/fcheck.py Tools/F_scripts/find_files_vpath.py Tools/F_scripts/findparams.py Tools/F_scripts/makebuildinfo.py Tools/F_scripts/write_probin.py Tools/GNUMake/Make.defs Tools/Postprocessing/python/column_depth.py Tools/Postprocessing/python/conv_slopes.py Tools/Postprocessing/python/dumpparthistory.py Tools/Postprocessing/python/test_helmeos.py Tools/Postprocessing/python/test_parseparticles.py Tools/Py_util/plotsinglevar.py Tools/Release/ppCleanup.py Tools/Release/ppCleanupDir.py Tools/Release/release.py Tools/libamrex/configure.py Tools/libamrex/mkconfig.py Tools/libamrex/mkpkgconfig.py Tools/libamrex/mkversionheader.py Tools/typechecker/typechecker.py commit 2d931f63cb4d611d0d23d694726889647f8a482d Author: Andrew Myers Date: Wed Jun 22 15:03:50 2022 -0500 Maintain the high end of the 'roundoff domain' in both float and double precision (#2839) * Maintain the high end of the 'roundoff domain' in both float and double precision * fix shadowing * fix warning * fix float conversion warning * fix logic * Update Src/Base/AMReX_Geometry.H * Update Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H commit 104466dee1b298e0f854f0485871437b89e6b0ec Author: Ryan Sandberg Date: Tue Jun 21 10:39:48 2022 -0700 add Ok to coordsys (#2844) Src/Base/AMReX_CoordSys.H commit bd42adec2b7a7d1b023f73a62142fefd5835b5e9 Author: Junghyeon Park Date: Wed Jun 22 02:35:12 2022 +0900 Correct the parameter name in the usage string (#2847) Tools/Postprocessing/C_Src/IntegrateComp.cpp commit 478fd8a4ac98be1b484b81819d9be6b4704262c3 Author: Kevin Gott Date: Mon Jun 20 00:38:29 2022 -0700 Double quotes (#2845) Tools/GNUMake/sites/Make.nersc commit 3ba934070e893c877a12f89c83806064f758e765 Author: Ben Wibking Date: Sat Jun 18 14:00:29 2022 +1000 Fix typo in README.md (#2846) README.md commit bf79c456a9a5fe6b22f3ba51b5973cd44b03c244 Author: Axel Huebl Date: Fri Jun 17 00:25:19 2022 +0700 ParamParse: Add Files at Runtime (#2842) Add a new runtime API to load whole files. This was currently only possible during the first initialize. Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit 6f72de283c38ef3e38ed7c5e8760a176434e6be3 Author: Weiqun Zhang Date: Wed Jun 15 09:37:15 2022 -0700 Fix a pathological case for 2d EB (#2840) What could happen is a cell might be cut a tiny bit at a corner such that 3 faces with an area fraction of one and one face with an area fraction of almost one, and the volume fraction is one. In that case, the boundary area and centroid have been set to wrong values based on an incorrect assumption. Src/EB/AMReX_EB2_2D_C.cpp commit a06cb417b9e4a1a1a32f78431c09620d107e3bab Author: Ben Wibking Date: Thu Jun 16 01:22:14 2022 +1000 add fvolumesum to GNUmakefile (#2836) * add fvolumesum to GNUmakefile and CMakeLists.txt * initialize vol as NAN Tools/Plotfile/CMakeLists.txt Tools/Plotfile/GNUmakefile Tools/Plotfile/fvolumesum.cpp commit cd166dda59b8491a5e5a19cb5c002b62556da73a Author: averytrevino <107513200+averytrevino@users.noreply.github.com> Date: Tue Jun 14 15:42:36 2022 -0700 Replaced Checkpoint with WritePlotFile to fix particle plot file (#2841) Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp commit fe37f7a0403fd225eafcef699c49809bf9859d9f Author: Phil Miller Date: Tue Jun 14 12:34:19 2022 -0700 Clamp particles shifted from plo boundary against rhi, rather than back to plo (#2814) Src/Particle/AMReX_ParticleUtil.H commit 833e61c076a708ae1680eab68713a2d40637085d Author: Axel Huebl Date: Wed Jun 15 00:47:05 2022 +0700 Fix: CMake NVTX not only Hypre (#2837) NVTX is used also without Hypre in AMReX. Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake commit adfdab0d4f03f8d201663d300c7d32723101b310 Author: kwryankrattiger <80296582+kwryankrattiger@users.noreply.github.com> Date: Tue Jun 14 10:58:50 2022 -0500 Update sensei CI container for sensei v4.0 integration (#2834) * Update sensei CI container for sensei v4.0 integration * Add simple script for building and pushing sensei container .github/workflows/docker/sensei/Dockerfile .github/workflows/docker/sensei/build-container.sh .github/workflows/docker/sensei/install_deps.sh .github/workflows/docker/sensei/install_sensei.sh .github/workflows/docker/sensei/install_vtk_minimal.sh .github/workflows/docker/sensei/vtk_use_mpi.patch commit 204bd7cf7353e7ccbfdcfdf7834d069b7da33d0f Author: Weiqun Zhang Date: Tue Jun 14 08:56:26 2022 -0700 HIP Memory Advise : Set managed memory to coarse grain (#2835) For unsafe atomics to work on managed memory, we must set it to coarse grain. For gfx90a, we can explicitly use unsafeAtomicAdd. Src/Base/AMReX_Arena.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuDevice.cpp commit e39b9647db148cca0d30010e8861a8dbe5bd966f Author: Axel Huebl Date: Tue Jun 14 22:45:22 2022 +0700 CMake: Fix `export` with `AMReX_INSTALL=OFF` (#2838) When introducing `AMReX_INSTALL`, the `export` of targets broke for `OFF`. This generalizes this to work in either case. CMakeLists.txt Tools/CMake/AMReXInstallHelpers.cmake commit 1a2fc3dbee1ce0c168cdb66710458757e92a0605 Author: Andrew Myers Date: Mon Jun 13 13:57:32 2022 -0700 make PODVector work with PolymorphicArenaAllocator (#2829) Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_PODVector.H commit 574a77d7a72a8576940b4c26472c32c198826289 Author: Candace Gilet Date: Mon Jun 13 16:09:47 2022 -0400 Re-implement FaceLinear::interp() for InterpFromCoarseLevel (#2831) Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.cpp commit eecb81ec43731d0166176277b1f4edb521da65b2 Author: Max Katz Date: Mon Jun 13 14:35:12 2022 -0400 Make regrid method of Amr class public (#2833) Src/Amr/AMReX_Amr.H commit 4c7b63997421cf4b4f2acf61a0656c000bc4075e Author: Weiqun Zhang Date: Mon Jun 13 10:38:34 2022 -0700 Fix link in readme (#2832) README.md commit a9c5335bb57a81616d17236343bc3126932ecfb8 Author: Weiqun Zhang Date: Mon Jun 13 08:48:55 2022 -0700 amrex::Any (#2827) Add amrex::Any that can be used to store any movable types. Note that std::any can only be used for copy constuctible types. Thus, std::any cannot be used to store MultiFab, whereas amrex::Any can. Src/Base/AMReX_Any.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 74183fe377df7d351e9c548f1aeaa5ec94c0dfe2 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Sat Jun 11 23:26:03 2022 -0400 Fix line integral computation (#2830) Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H commit 5ce1fda77ee366681279ba2e7f7d86d25c9923d9 Author: Andrew Myers Date: Fri Jun 10 10:59:12 2022 -0700 move ParticleContainer into it's own header file (#2822) Src/Particle/AMReX_ParticleContainer.H Src/Particle/AMReX_Particles.H Src/Particle/CMakeLists.txt Src/Particle/Make.package commit a5e6c6b84c2e166b863026604caee2d980e9ee26 Author: Weiqun Zhang Date: Fri Jun 10 09:09:53 2022 -0700 Fix a bug in multigrid grids (#2823) This fixes a bug introduced in #2690. We have to make sure every box in the fine BoxArray is coarsenable, otherwise average down functions will fail. For example, for a 10x10 cells domain broken into four 5x5 cells boxes, we should not add a coarse level because 5x5 is not coarsenable, even though the 10x10 domain is coarsenable. In principle, we could fix this in the average down functions instead. However, these tests with odd numbers of cells are corner cases. So we are fixing it in a relatively easy approach. Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp commit 549c2ec31ffff9ad034bed665eab7ca5c6a67be1 Author: Erik Date: Fri Jun 10 08:56:56 2022 -0700 FAQ: add some answers, split off unanswered questions (#2341) Docs/sphinx_documentation/source/Faq.rst commit 70024ca28a05dfce9dbb14b0c7afd18d2c9099b3 Author: Weiqun Zhang Date: Fri Jun 10 08:54:13 2022 -0700 Fix a typo in Readme (#2821) * Fix a typo in Readme The link to Gallery did not work because of the typo. * Update README.md Add yt to visualization software list Co-authored-by: Erik README.md commit e113ff7dcf29b67618baa88ba4c10311436e4bb7 Author: Erik Date: Thu Jun 9 10:07:41 2022 -0700 Add html, additional sections to README.md (#2775) README.md commit e491cb74e45388bc26878476d7449304e7b1057b Author: Candace Gilet Date: Thu Jun 9 12:30:21 2022 -0400 Allow StateDataPhysBCFunct to operate on face-centered data (#2819) Src/Amr/AMReX_StateData.cpp commit 7b335c5f961746ebc797489bb2391126618518ff Author: Weiqun Zhang Date: Wed Jun 8 15:24:53 2022 -0700 Fix Parser ODR (#2820) Although we define api.prefix, Bison does not use it for all the symbols. So we fix it by manually adding the prefix to yyalloc and yysymbol_kind_t. Src/Base/Parser/amrex_iparser.tab.cpp Src/Base/Parser/amrex_iparser.y Src/Base/Parser/amrex_parser.tab.cpp Src/Base/Parser/amrex_parser.y commit c8f33c76fb8a19bfd11df8393f920230e81191bf Author: Axel Huebl Date: Wed Jun 8 12:33:26 2022 -0700 CMake: Cleanup old nvToolsExt (#2817) Already searched for since #2813 Tools/CMake/AMReX_Config.cmake commit 2c1e462619fbd65354d2500c55853087af316a0e Author: Andrew Myers Date: Wed Jun 8 12:20:22 2022 -0700 Handle the case where we don't have enough device memory for the snd_buffer (#2705) Co-authored-by: Weiqun Zhang Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp Src/Base/AMReX_GpuAllocators.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H commit 18d0a2861d31c52c65752a1d5856f54e08699ce3 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Tue Jun 7 19:04:13 2022 -0400 Use amrex::math::abs in IntVect::maxDir. (#2815) Src/Base/AMReX_IntVect.H commit 6299c2cf414e9c7936fa54a89b014317ac79844c Author: Axel Huebl Date: Tue Jun 7 14:51:01 2022 -0700 CMake: 3.17+ (#2813) * CMake: 3.17+ We are using `find_package(CUDAToolkit ...)` now, which is only supported in CMake 3.17+. Technically, we want to go to 3.20+ soon as well, so that we can drop the old CUDA code paths that we used to forward device flags and CUDA architectures. * Modernize: CMake 3.17+ Required * CMake Modernize: nvToolsExt CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/BuildingAMReX_Chapter.rst Tests/CMakeTestInstall/CMakeLists.txt Tests/SpackSmokeTest/CMakeLists.txt Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/AMReX_Config.cmake commit 8a1c76c0af8961db3168d577898a6d0edaa55891 Author: ldowen <54121008+ldowen@users.noreply.github.com> Date: Mon Jun 6 17:48:51 2022 -0700 Landon/fix bug ghost particles (#2812) Src/Particle/AMReX_ParticleLocator.H commit ff9e832483f142a34da72f9bea5fed72e49fea33 Author: Andrew Myers Date: Fri Jun 3 10:54:46 2022 -0700 Follow-on to 2809; update selectActualNeighbors as well. (#2810) Src/Particle/AMReX_NeighborParticlesI.H commit 3d7f4d09dbbed1d85a83b89e473741e00a2681be Author: Andrew Myers Date: Fri Jun 3 08:16:01 2022 -0700 Generalize the type of callables that can be passed into the neighbor list build function (#2809) Src/Particle/AMReX_NeighborList.H commit c96e79e8409ba1fd9ee1b5f5e005ffaa946d1c4a Author: Kevin Gott Date: Thu Jun 2 12:16:47 2022 -0700 Add AVX2 instructions flag. (#2803) Tools/GNUMake/sites/Make.nersc commit 45e3042691ba9352d929cd1d22c5e7db3f182ce2 Author: Weiqun Zhang Date: Thu Jun 2 12:12:41 2022 -0700 Avoid M_PI because it's not in the C++ standard (#2807) Src/Base/AMReX_Geometry.H Tests/EB_CNS/Exec/Pulse/cns_prob.H Tests/LinearSolvers/LeastSquares/initEB.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleDataFor2D.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleDataFor3D.cpp Tools/Plotfile/fvolumesum.cpp commit 6e907ff7cf8ecf6154b261534e63b9c3a863e30e Author: Candace Gilet Date: Wed Jun 1 16:02:39 2022 -0400 In the array version of FillPatchTwoLevels, allow specifying an (#2800) array of component indicies for the bc functions and BCRec arrays. Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_I.H commit cc3497dee62c2293587054128e695969b66622ba Author: Erik Date: Wed Jun 1 12:40:13 2022 -0400 drop breathe plugin, upgrade sphinx ver to 5.0, remove docutils workaround (#2804) and unused python scripts for generating filelists. Convert :cpp: commands to `` in figure caption for sphinx bug workaround. .github/workflows/docs.yml Docs/sphinx_documentation/add_doxy_headers.py Docs/sphinx_documentation/make_api.py Docs/sphinx_documentation/source/Particle.rst Docs/sphinx_documentation/source/conf.py commit 605b820316277925e8f87a52fb92892ac0614148 Author: Erik Date: Wed Jun 1 12:20:14 2022 -0400 Add doxygen comment for ParticleContainer::Increment (#2776) Src/Particle/AMReX_ParticleContainerI.H commit f6c9f484bd8a20a4479fbaa5ba221235cbb68d26 Author: Weiqun Zhang Date: Wed Jun 1 08:46:34 2022 -0700 Update CHANGES for 22.06 (#2805) CHANGES commit f24582df5fec5cab9f6d368777e057d663b72cb9 Author: Weiqun Zhang Date: Tue May 31 16:38:46 2022 -0700 Solvability of bottom solver: Follow-up on #2783 (#2801) In #2783, the solvability fix at the bottom level of EB nodal projection solver was disabled because we don't have sufficient geometry information. This broke a number of tests. It appears that we should still do the best we can (i.e., use the old approach at the bottom level). Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit 1dff173b3dc14ba881a5fbe0caf9db9e52eb43e4 Author: Andrew Myers Date: Tue May 31 12:05:39 2022 -0700 Specify Sphinx version for build_and_deploy action. (#2802) .github/workflows/docs.yml commit b78921a2d80d95add9ff3ec9b498a96299ec4ed7 Author: Weiqun Zhang Date: Fri May 27 10:46:12 2022 -0700 Fix solvability issue in the nodal solver RAP approach (#2783) In the RAP approach of the nodal solver, the RHS at Neumann boundaries only includes the integral inside the domain. That is it's only half of what its "physical" value is. The usual way of subtracting a constant from the RHS does not work for RAP. We should only subtract half of the constant offset at Neumann boundaries. The computation of the constant offset needed for the solvability fix is also affected by the way how RHS is computed at Neumann boundaries. For EB, the computation of the constant offset is an integral of the RHS multiplied by the volume fraction, and the subtraction also needs to be weighted by the volume fraction. Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit 806108c5b8923f1f8aa0e816b923693860a949a4 Author: Weiqun Zhang Date: Thu May 26 14:20:35 2022 -0700 Fix document on MultiFab reduction (#2796) Docs/sphinx_documentation/source/GPU.rst commit 9ce107acd8a778d78171bc428e47f04e8725a8ef Author: Axel Huebl Date: Thu May 26 11:35:35 2022 -0700 CI: Fix HIP GPG Key (#2795) Split downloading and adding. ``` Warning: apt-key output should not be parsed (stdout is not a terminal) gpg: no valid OpenPGP data found. ``` .github/workflows/dependencies/dependencies_hip.sh commit 68a51ce34515f71bf9c9bd3784c77f8029864607 Author: Erik Date: Thu May 26 14:14:41 2022 -0400 Move 'Example Codes' section under 'Getting Started'. Fix various sphinx warnings. (#2777) Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/EB_Chapter.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GettingStarted.rst commit 72bad8bd0edfd571fce66b09bf33acdebca3df78 Author: Erik Date: Thu May 26 13:33:24 2022 -0400 User's Guide: Add content and links to SUNDIALS page. (#2788) Docs/sphinx_documentation/source/SUNDIALS_top.rst Docs/sphinx_documentation/source/TimeIntegration_Chapter.rst commit 52b72f3f13278064da78609297d4c919b8e3b5b2 Author: Kevin Gott Date: Thu May 26 10:32:05 2022 -0700 Profiler Sync Timers (#2784) Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/CMakeLists.txt Tools/GNUMake/Make.machines commit 0fd8e9667c97ca79472f602f8a5eec33fa89d5aa Author: Axel Huebl Date: Thu May 26 10:19:25 2022 -0700 Implement Serial ParallelDescriptor::Gather (#2793) This causes a bug currently in the reduced FieldProbe diags in WarpX. Src/Base/AMReX_ParallelDescriptor.H commit b3ffba60f14052fb2c4cc21d7e310eb2c43b6fa1 Author: Donald E. Willcox Date: Wed May 25 19:52:03 2022 -0700 Renamed new internal class variables using m_ convention to fix compiler warnings in debug mode. (#2790) Src/Base/AMReX_TimeIntegrator.H commit 1305eb3d364dab42ae17ad3df67e696068d60415 Author: Jon Rood Date: Tue May 24 15:13:41 2022 -0600 Fix dash/underscore mistake with H5Z-ZFP. (#2792) Tools/CMake/AMReXConfig.cmake.in commit 3dda62c50f2b7a0d256b474ba13ea99b7ae954be Author: David Grote Date: Tue May 24 08:55:37 2022 -0700 Make IntVectFromLocation const (#2789) Src/Extern/amrdata/AMReX_AmrData.H Src/Extern/amrdata/AMReX_AmrData.cpp commit 5d88558d2aeff93855a5c46ec8be1a7ef8545f70 Author: Andrew Myers Date: Mon May 23 10:15:03 2022 -0700 make sure m_particles is sized properly when numLocalTilesAtLevel is called (#2782) Src/Particle/AMReX_Particles.H commit 802701647c146c070eace2536c5a9700c9e74c98 Author: Weiqun Zhang Date: Mon May 23 08:47:07 2022 -0700 Add Gpu::synchronize to Hypre interface (#2786) This fixes a bug introduced in #2754 that removed the null stream. Some of the Gpu::synchronize might not be necessary. But we have to be on the safe side because the synchronization behavior of Hypre is unclear. Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/PETSc/AMReX_PETSc.cpp commit 0540fed3576f8e57b3b110c34ac5bd8735b16b2a Author: drangara <69211175+drangara@users.noreply.github.com> Date: Thu May 19 12:12:39 2022 -0400 Fix for small cells (#2781) Move small cells outside domain extent into iterative approach. Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.cpp commit 994073b931f1a696e50c177fae3689ac0db92996 Author: Donald E. Willcox Date: Wed May 18 08:50:29 2022 -0700 Add some timestep controls to the AMReX TimeIntegrator class for its integrate() driver function. (#2780) Src/Base/AMReX_TimeIntegrator.H commit 0a50776a7239119d2c6c8fa9f48e889b991125d0 Author: Axel Huebl Date: Tue May 17 16:40:55 2022 +0200 AmrCore: Include utility (#2778) Add the `` header for `std::move`. Src/AmrCore/AMReX_AmrCore.cpp commit 843a7dff266273a0f5b7b9f6cc9233a278f41fe1 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon May 16 12:56:52 2022 -0500 Fix the bug in the CMake build with AMReX_BASE_PROFILE. (#2774) Src/Base/CMakeLists.txt commit 02726a6a8440e0e6c16d5dbfeab6171a76213427 Author: Erik Date: Mon May 16 13:52:30 2022 -0400 configure value of AMReX_GPU_RDC flag for use in cmake find_package(AMReX ...) (#2770) Tools/CMake/AMReXConfig.cmake.in commit 963294be434ef9d506ee87c1e2dafd390f96a124 Author: Erik Date: Mon May 16 13:51:48 2022 -0400 CI--HIP: wget gpg key from https instead of http (#2771) * CI--HIP: wget gpg key from https instead of http * change other http to https .github/workflows/dependencies/dependencies_hip.sh commit 11cd801494b6ccfd0154a2a96cc1de28985ae3b2 Author: Erik Date: Mon May 16 12:05:06 2022 -0400 Change repo html address to Ubuntu 20.04 (#2766) .github/workflows/dependencies/dependencies_nvcc11.sh commit 903648a9e74e1c882f76bb23609de014bfdfc8e2 Author: Max Katz Date: Mon May 16 11:25:42 2022 -0400 Add an optional volume weighting to AMRErrorTag (#2772) This allows, for example, refining based on the mass in a cell rather than only on its density. A function to obtain the cell volume at runtime given an IntVect, that can be run inside a ParallelFor, is added to Geometry. Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp Src/Base/AMReX_Geometry.H commit 49c935bf129c7b639ce392a53dfa8be8a3aaaba5 Author: Axel Huebl Date: Mon May 16 07:54:12 2022 -0700 Fix: AmrCore Move (#2773) The move constructor and assignment operator for `AmrCore` with particles was broken. When moving `AmrParGDB`, its internal `m_amrcore` pointer needs to be updated, too. Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrParGDB.H commit 9473062293af7901e8ad24b103d063217f13995e Author: Andrew Myers Date: Thu May 12 17:25:01 2022 -0700 Update particle << operator after changes to id/cpu (#2769) Src/Particle/AMReX_Particle.H commit 6686b211973a79cb86f90648333c86859af1c9e1 Author: Weiqun Zhang Date: Thu May 12 17:05:39 2022 -0700 Fix maybe-uninitialized warning in calling mlock (#2768) Src/Base/AMReX_Arena.cpp commit f0c51bb0997ff278a8ac0b8408b18841c809a1c0 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Thu May 12 16:33:31 2022 -0600 this updates to recent Hypre API changes (#2765) Src/Base/AMReX.cpp commit 3be111e9686f4d41632d94e89b72d3776d8a417a Author: Weiqun Zhang Date: Thu May 12 15:21:46 2022 -0700 Fix Wstringop-overflow warning in FabConv (#2767) On Perlmutter, `g++ -O3 -march=znver3` produces lots of stringop-overflow warnings in FabConv. These warnings are false positive because the compiler does not know sizeof(amrex::Real) is either 4 or 8. This commit fixes the warnings. Close #2750 Src/Base/AMReX_FabConv.cpp commit 5aa61b26cbbe798b5fd13c17d9f9ecef7371e186 Author: Weiqun Zhang Date: Thu May 12 09:29:00 2022 -0700 Time step in the AmrLevel test (#2763) Make the dt in the AmrLevel test consistent with that in the AmrCore Test. That is we use the velocity at t+0.5*dt (here dt is from the previous step) to estimate the dt for the next step. Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp commit a629129046d77fcbaf2412a9e4d5ea86c862b91f Author: Weiqun Zhang Date: Thu May 12 08:38:26 2022 -0700 Fix the Advection_AmrCore test (#2761) The time used for computing velocity in the non-subcycling mode is incorrect. Close #2725 Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp commit 0b485513b4def2f64a0fe43ea0613c3302af58f5 Author: Andy Nonaka Date: Wed May 11 17:31:24 2022 -0700 multilevel version of writeplotfiletoascii (#2742) Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp commit 5bbb63f24753353bdcbb8c439fedbc3dfc11d15e Author: Weiqun Zhang Date: Wed May 11 14:02:31 2022 -0700 Avoid the use of null stream (#2754) The default stream (e.g., stream used outside MFIter) used to be the null stream for CUDA and HIP. By default, there is implicit synchronization between the null stream and other streams. To avoid that, the default stream in AMReX is now no longer the null stream. The behavior of Gpu::synchronize being device wide synchronization has not changed. However, for most of its use cases, it can be replaced by a new function Gpu::streamSynchronizeAll that will synchronize the activities on all AMReX streams without performing a device wide synchronization that could potentially interfere with other libraries (e.g., MPI). The behavior of [dtod|dtoh|htod]_memcpy has changed. For CUDA and HIP, these functions used to call the synchronous version of the memcpy. However, the exact synchronization behavior depends on the memory types. For SYCL/DPC++, there is no equivalent form because a queue (i.e., stream) must be specified. Furthermore, there is no guarantee of consistence across different vendor platforms. This has now changed to calling the asynchronous form using the current stream followed by a stream synchronization. Docs/sphinx_documentation/source/GPU.rst Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateData.cpp Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.cpp Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuUtility.cpp Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_Partition.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_TagParallelFor.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_VisMF.cpp Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EB_utils.cpp Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_ParticleBufferMap.cpp Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_SparseBins.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/EB_CNS/Source/CNS.cpp Tests/GPU/AnyOf/main.cpp Tests/GPU/AtomicIf/main.cpp Tests/GPU/RandomNumberGeneration/main.cpp Tests/GPU/Vector/main.cpp Tests/Particles/AsyncIO/main.cpp Tests/Particles/DenseBins/main.cpp Tests/Particles/Intersection/main.cpp Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleReduce/main.cpp Tests/Particles/ParticleTransformations/main.cpp Tests/Particles/Redistribute/main.cpp commit 894a50f9f762beb37b02194f60ce0e39097ecc5a Author: Donald E. Willcox Date: Wed May 11 14:02:03 2022 -0700 add scomp and ncomp arguments to IntegratorOps functions. (#2759) Src/Base/AMReX_IntegratorBase.H commit 6264e8123699022bec4af701f9d75aa680215fa7 Author: Jon Rood Date: Wed May 11 12:39:13 2022 -0600 Add HDF5 H5Z-ZFP support in CMake (#2753) Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/AMReX_Config.H.in commit c4ad207584b0ba6cb55bf4529f2c92ce5eea20c3 Author: Axel Huebl Date: Tue May 10 07:57:29 2022 -0700 CUDA On Cray: More Robust w/o Wrapper (#2757) Searching for `curand` explicitly (required by default by AMReX) makes build more robust when we build w/o compiler wrappers. Tools/CMake/AMReXParallelBackends.cmake commit 3ca867b8cce86b9892b45ff78df8630500dc3bb4 Author: Weiqun Zhang Date: Sat May 7 16:36:48 2022 -0700 Reimplement amrex::min and max to Work around an nvcc bug (#2756) This is a workaround for an nvcc bug when compiling ERF in debug mode. Note that because std::min and max have been constexpr functions since C++14 and AMReX uses C++14 now, there is no need to use amrex::min and max anymore. But for backward compatibility, we will still keep them. Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_Algorithm.H commit ef52970b80e308552fdd0f3246ffa993c49a8caa Author: Weiqun Zhang Date: Tue May 3 22:48:17 2022 -0700 Revert "Turn on managed memory by default in The_Arena for HIP (#2734)" (#2752) This reverts commit 09e8b9b1179e3e3d7b9b6e288ebe6d2864a8cc67. The managed memory on AMD GPUs is still extremely slow. Src/Base/AMReX_Arena.cpp commit df9b6944a6d56ccdba725d9af2d7e1c5183c560a Author: Andrew Myers Date: Mon May 2 12:56:54 2022 -0700 Add methods for resetting the ParGDB of a ParticleContainer (#2732) Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_Particles.H commit 2bd5be14b179707efaabaeae0a849864349e9f8c Author: Max Katz Date: Mon May 2 12:41:13 2022 -0700 Switch nvtxRangeStart to nvtxRangePush (#2746) Src/Base/AMReX_GpuDevice.cpp commit 2c2593d6be89b9a9ad27b2a2b664fa73c90e1f17 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Mon May 2 15:36:28 2022 -0400 Eb flow diffusive solve (#2741) Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H commit 01ab678e9c468bf2606cbfb0dbb5e7006d5d2074 Author: Weiqun Zhang Date: Mon May 2 10:37:27 2022 -0700 Update CHANGES for 22.05 (#2749) CHANGES commit 72a26b78e095eb05fe740e5ab0d7f54660b854aa Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon May 2 10:38:46 2022 -0500 Update call_check_pair's interface in selectActualNeighbors to resolve the error in building MFiX. (#2748) Src/Particle/AMReX_NeighborParticlesI.H commit 50466d11e7b293bd9ec5623235c6a0fb9d0c4dcc Author: David Gardner Date: Fri Apr 29 12:17:44 2022 -0700 Update Required SUNDIALS version (#2743) Update SUNDIALS version in AMReXConfig.cmake.in to match AMReXThirdPartyLibraries.cmake Tools/CMake/AMReXConfig.cmake.in commit 3d344ec19655f2583c37c1c89e4e309d0fd6ce7f Author: Weiqun Zhang Date: Thu Apr 28 14:14:41 2022 -0700 Update CUDA repo key (#2744) Nvidia has made changes in the signing keys. https://forums.developer.nvidia.com/t/notice-cuda-linux-repository-key-rotation/212771 .github/workflows/dependencies/dependencies_nvcc10.sh .github/workflows/dependencies/dependencies_nvcc11.sh commit 3cc4a66759c9321d67ceb218c0f3b2b138a5a075 Author: Weiqun Zhang Date: Wed Apr 27 18:01:27 2022 -0700 EB: Geometry generation from STL file (#2728) This extends Hariswaran Sitaraman's ASCII STL reader by adding a GeometryShop on top of it. An IndexSpace based on the STL GeometryShop is implemented. This allows us to generate AMReX EB database from STL files. Also, we have added a binary STL reader. Docs/sphinx_documentation/source/EB.rst Src/Base/AMReX_IntConv.H Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_IndexSpace_STL.H Src/EB/AMReX_EB2_IndexSpace_STL.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level_STL.H Src/EB/AMReX_EB2_Level_STL.cpp Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/CMakeLists.txt Src/EB/Make.package commit 511b3b03810632dfc67ffba54febaa84860bacd6 Author: Weiqun Zhang Date: Wed Apr 27 13:23:52 2022 -0700 Drop CUDA 9 support (#2736) * Modify CI to test CUDA 10.2 instead of CUDA 9.2. * Remove workarounds for CUDA 9.2 * No need to test for CUDA >= 10. * Update Documentation. * Update GNU Make and CMake. .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_nvcc10.sh .github/workflows/dependencies/dependencies_nvcc11.sh Docs/sphinx_documentation/source/GPU_Chapter.rst Src/Base/AMReX_BaseFab.H Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuAsyncArray.cpp Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuElixir.cpp Src/Base/AMReX_IntVect.H Src/EB/AMReX_EB2_IF_Box.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_algoim_K.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Tests/GPU/CNS/Source/CNS_K.H Tools/CMake/AMReX_SetupCUDA.cmake Tools/GNUMake/comps/nvcc.mak commit d7f997ce7f4d3c1d535383e196436d8b859a0aaa Author: Erik Date: Wed Apr 27 14:29:14 2022 -0400 Add advection of tracer particles to Adevection_AmrCore test. (#2722) Tests/Amr/Advection_AmrCore/Exec/GNUmakefile Tests/Amr/Advection_AmrCore/Exec/Make.Adv Tests/Amr/Advection_AmrCore/Exec/inputs Tests/Amr/Advection_AmrCore/Exec/inputs-ci Tests/Amr/Advection_AmrCore/Exec/inputs_for_scaling Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tests/Amr/Advection_AmrCore/Source/DefineVelocity.cpp commit 399ff41c05cea3894007488cf87e509571633ab4 Author: Weiqun Zhang Date: Wed Apr 27 11:22:46 2022 -0700 Disable AsyncOut by default for HIP and DPC++ (#2735) This makes it the same behavior as other builds. We had it on by default because in the past our I/O functions relied on managed memory and there were issues with HIP and DPC++. Src/Base/AMReX_AsyncOut.cpp commit 09e8b9b1179e3e3d7b9b6e288ebe6d2864a8cc67 Author: Weiqun Zhang Date: Wed Apr 27 11:22:19 2022 -0700 Turn on managed memory by default in The_Arena for HIP (#2734) This makes it have the same behavior as CUDA and DPC++ builds. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_Arena.cpp commit a30e8eb12c6a7a9abb82849d14589da02210e451 Author: Weiqun Zhang Date: Wed Apr 27 08:13:41 2022 -0700 Update WarpX's MLEBNodeFDLaplacian for 2D RZ (#2733) This adds support for sigma coefficient in the z-direction for the 2D RZ LinOp MLEBNodeFDLaplacian used by WarpX. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit c768c36d6e0825df78dcb048cfe2ca592a0f738b Author: Junghyeon Park Date: Wed Apr 27 01:04:31 2022 +0900 Fix a typo (#2731) Docs/sphinx_documentation/source/Basics.rst commit fb5917b8dd62670904162a0051fe0b19f2099b85 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Tue Apr 26 09:17:56 2022 -0600 Adding more recent Hypre APIs to initialize the library. (#2729) Co-authored-by: Paul Mullowney Src/Base/AMReX.cpp commit 329f81b889a6ece31c67b1b70f207d6c6b2463b6 Author: Andrew Myers Date: Mon Apr 25 08:54:29 2022 -0700 Allow the computation of neighbor lists between particles of different types. (#2727) Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleUtil.H Tests/Particles/NeighborList/GNUmakefile Tests/Particles/NeighborList/Make.package Tests/Particles/NeighborList/inputs Tests/Particles/NeighborList/main.cpp commit 56661522b4cd5be804e3d114c7dc4e36c37204f3 Author: Akash Dhruv Date: Thu Apr 21 15:21:36 2022 -0400 add option for face linear interpolater in fortran interface (#2726) Src/F_Interfaces/AmrCore/AMReX_fillpatch_fi.cpp Src/F_Interfaces/AmrCore/AMReX_interpolater_mod.F90 commit 034b43ac787f1b94a1815b49fe6c5f24b8a85e55 Author: Jon Rood Date: Wed Apr 20 10:50:55 2022 -0600 Allow some EB code to handle single precision (#2723) Src/EB/AMReX_EB2_IF_Scale.H commit 0d136ea53aed8a20029c4997961528d25e6fa41f Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon Apr 18 13:37:47 2022 -0500 Allow `FillBoundary` to cast messages from double to single precision. (#2708) This PR allows FillBoundary to pack double-precision data into a single-precision buffer and unpack single-precision messages to double-precision FabArray, e.x. mf.template FillBoundary(...). Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_PCI.H Src/Base/AMReX_TagParallelFor.H Src/EB/AMReX_MultiCutFab.H commit d22d7e6d556f308ad6ea27bbaf8c4d311498e105 Author: Weiqun Zhang Date: Mon Apr 18 09:15:22 2022 -0700 Make EB code more single-precision friendly (#2719) Src/AmrCore/AMReX_Interp_2D_C.H Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_IF_Cylinder.H Src/EB/AMReX_EB2_IF_Ellipsoid.H Src/EB/AMReX_EB2_IF_Polynomial.H Src/EB/AMReX_EB2_IF_Spline.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EBAmrUtil.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_algoim_K.H Src/EB/AMReX_distFcnElement.cpp commit f2d7f67848de283606a20c646cc528dc240e7bac Author: Weiqun Zhang Date: Sun Apr 17 14:33:51 2022 -0700 Fix FluxRegister::SumReg for GPU builds (#2718) It's incorrect to assume that the fused version of reduce is only used on FabArrays with a simple BoxArray. FluxRegister::SumReg uses the fused version of reduce, and its BoxArray has a special BATransformer. A FluxRegister only has cells on the faces of regular Boxes, but the meta-data provided to the GPU reduce function is for regular Boxes. This results in out-of-bound errors. This commit fixes the issue by extending the meta-data caching mechanism to include special BoxArrays used by FluxRegisters. We have also refactored Batransformer's opeartor==. Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp commit b593587a3c58dc52efde2fba83b5813d44741c5b Author: Jean M. Sexton Date: Fri Apr 15 13:30:15 2022 -0700 Swap hip lib and include ordering (#2717) Update Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/hip.mak commit f63c9cf37c541c09376aa417ef6e8ae608725c42 Author: Weiqun Zhang Date: Fri Apr 15 10:06:07 2022 -0700 Fix integer overflow warnings in FabConv (#2716) Partially address #2713. Src/Base/AMReX_FabConv.cpp commit 625dd4d976cad44076ede3e9d2ffe2dda4d64322 Author: Weiqun Zhang Date: Fri Apr 15 08:48:09 2022 -0700 Make AsyncOut::Finish safe to call when async_out is not enabled. (#2715) Close #2714 Src/Base/AMReX_AsyncOut.cpp commit fc55c67f72d2842caa6bd7b908dfe82df910ec75 Author: Weiqun Zhang Date: Thu Apr 14 15:41:33 2022 -0700 flushParForInfo: remove erroneous assertion (#2712) Src/Base/AMReX_FabArrayBase.cpp commit b383b5844d29d190c930ec8a972656d5615e096d Author: Arnur Nigmetov Date: Thu Apr 14 09:47:49 2022 -0700 Replace double with Real in Advection and Parser tests. (#2711) Otherwise tests do not compile for single precision: function signatures do not match the methods they are supposed to override and std::min/max requires same types. Co-authored-by: Arnur Nigmetov Tests/MultiBlock/Advection/main.cpp Tests/Parser/main.cpp commit a537fb0516d52cdff9a0da5bfe9cf1571c100e4c Author: ldowen <54121008+ldowen@users.noreply.github.com> Date: Mon Apr 11 15:24:10 2022 -0700 Link particles in ghost cells to closest box (#2685) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleLocator.H commit c79cff9ba5640ddcb4ce4cb00e4e22df354bb821 Author: Andrew Myers Date: Mon Apr 11 12:54:39 2022 -0700 Remove make_pair from particle reduction functions. (#2707) Src/Particle/AMReX_ParticleReduce.H commit 872ce5692775fd0d659db001b838328806f5ae90 Author: Andrew Myers Date: Mon Apr 11 08:58:45 2022 -0700 Generalize the type of lambdas that can be passed into ParticleReduce (#2697) Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleUtil.H Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/ParticleReduce/main.cpp commit 7e2572f141556be19890110a1673a3e9f3280267 Author: Weiqun Zhang Date: Mon Apr 11 08:58:32 2022 -0700 Set USE_PARTICLES=TRUE in some tests. (#2706) This is a follow-up on #2703. Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/GNUmakefile Tests/Particles/NeighborParticles/GNUmakefile Tests/Particles/ParallelContext/GNUmakefile Tests/Particles/ParticleArray/GNUmakefile Tests/Particles/ParticleReduce/GNUmakefile Tests/Particles/ParticleTransformations/GNUmakefile Tests/Particles/Redistribute/GNUmakefile Tests/Particles/TypeDescriptor/GNUmakefile commit d70f70e2f56ea4b0623e6933746c4463617cbaf0 Author: Jon Rood Date: Sat Apr 9 16:51:02 2022 -0600 Return non-zero if nans were found when using fnan. (#2704) Tools/Plotfile/fnan.cpp commit 8c3c4c9b771dc267ac31cb446e9c20597f27f6b0 Author: Andrew Myers Date: Sat Apr 9 15:49:15 2022 -0700 Reorganize Make.package in Src/Particle (#2703) * reorganize Make.package in Src/Particle * now must turn on particles in HDF5 test Src/Particle/Make.package Tests/HDF5Benchmark/GNUmakefile commit e5dda4061229fc091e3c80836d0148dbc78bb2f4 Author: Brandon Runnels Date: Sat Apr 9 16:36:48 2022 -0600 FillPatchUtil edge interpolater fix (#2701) This PR addresses a compilation error that occurs when using templated FabArrays that are not MultiFab. (The static_cast of the abstract mapper Interpolater prevented compilation with derived and templated Interpolater objects.) To fix this, a templated InterpFace function was introduced that can be used by derived Interpolater, but that provides default behavior with casting for regular Interpolaters. (Note: this is similar to what is done with FillPatchInterp). Src/AmrCore/AMReX_FillPatchUtil_I.H commit a0668e9b25f15eb3307e21478893d4c03883e211 Author: Donald E. Willcox Date: Fri Apr 8 15:25:18 2022 -0700 Adds SUNDIALS integrator options for flexibility. (#2700) This PR adds some options to the SUNDIALS integrator interface for greater flexibility in choosing which integration methods SUNDIALS uses internally. Docs/sphinx_documentation/source/TimeIntegration_Chapter.rst Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_TimeIntegrator.H Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H commit 1a5d8917fd49ed63a65c2ae9f499d6a9160e291d Author: Jon Rood Date: Thu Apr 7 11:22:43 2022 -0600 Only error if HDF5 is not parallel if MPI is enabled. (#2699) Tools/CMake/AMReXThirdPartyLibraries.cmake commit 84a1f7d0715b2a2f33338304bbdfc7a7e018a2b6 Author: Andrew Myers Date: Wed Apr 6 11:15:52 2022 -0700 Make the ParticleReduce functions not rely on ParIter. (#2695) Src/Particle/AMReX_ParticleReduce.H commit 321ff599c4872bce3486eb9c8d7edc1cc5353c47 Author: Andrew Myers Date: Wed Apr 6 09:52:36 2022 -0700 Use pc.WritePlotFile instead of pc.Checkpoint in Tests. (#2694) Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_Particles.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Particles/AssignDensity/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/main.cpp commit ec648a92a89ffc907a87d4b02ba4ddb8a2a932a5 Author: Weiqun Zhang Date: Tue Apr 5 07:52:00 2022 -0700 Fix various warnings (#2693) .github/workflows/cuda.yml .github/workflows/gcc.yml Src/Base/AMReX_Geometry.cpp Src/Base/Parser/GNUmakefile Src/Base/Parser/amrex_iparser.lex.cpp Src/Base/Parser/amrex_parser.lex.cpp Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_nd.F90 Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp Tests/EB/CNS/Source/diffusion/cns_eb_diff_mod.F90 Tests/Particles/ParticleIterator/main.cpp commit 917afc0e18961d602b55d550c4d5a6c1b0c3e49a Author: Weiqun Zhang Date: Mon Apr 4 14:58:11 2022 -0700 Semicoarsening in WarpX Linear Solvers (#2690) Add support for semicoarsening in two linear solvers used by WarpX, MLNodeTensorLap and EBNodeFDLaplacian. The semicoarsening approach here is different from the existing approach for the nodal projection solver. There the grids are coarsened regularly as much as possible and then we switch to coarsening in directions that are still coarsenable, whereas here we do semicoarsening first in specified direction and then switch to regular coarsening if the specified maximum semicoarsening level has not been reached. Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp commit a45414cdf3a502c0e67127c63962b5224ab3165b Author: Weiqun Zhang Date: Mon Apr 4 14:57:41 2022 -0700 FillBoundaryAndSync (#2683) Add a new function to FabArray that fills ghost cells and synchronize nodal data. Ghost regions are filled with data from the intersecting valid regions. The synchronization will override valid regions by the intersecting valid regions with a higher precedence. The smaller the global box index is, the higher precedence the box has. With periodic boundaries, for cells in the same box, those near the lower corner have higher precedence than those near the upper corner. This function produces the same result as OverrideSync followed by FillBoundary in a single pass. Also modify the implementation of OverrideSync to use this new function, which is more efficient. The nodal linear solvers use OverrideSync to sync up the nodal data. They have been updated to use the new OverrideSync function that does not use an owner mask. So we no longer need to build owner masks except for the top and bottom multigrid levels that need to use the mask in the computation of norms to avoid overcounting. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit 9cca466f0b6e53ed7ef1babf702afd9193c3741a Author: Weiqun Zhang Date: Mon Apr 4 10:14:05 2022 -0700 Simplify The_Async_Arena example (#2692) Remove the use of resize in the example of using The_Async_Arena to avoid confusion. Docs/sphinx_documentation/source/GPU.rst commit 8a2ec577e107ff8802b6f28d4c1ce8edfaa44639 Author: Weiqun Zhang Date: Fri Apr 1 18:38:00 2022 -0700 Add comment on a potential bug in using The_Async_Arena (#2691) Docs/sphinx_documentation/source/GPU.rst commit 2a4bce247e7c7794d76934d133f5100437d579a0 Author: Axel Huebl Date: Fri Apr 1 08:47:40 2022 -0700 macOS: Fix Warning (ranlib, profiler) (#2688) Fix ``` [ 54%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Particle/AMReX_ParticleBufferMap.cpp.o [ 54%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Particle/AMReX_ParticleCommunication.cpp.o [ 55%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Particle/AMReX_ParticleContainerBase.cpp.o [ 55%] Linking CXX static library ../../../lib/libamrex.a /Library/Developer/CommandLineTools/usr/bin/ranlib: file: ../../../lib/libamrex.a(AMReX_BLProfiler.cpp.o) has no symbols /Library/Developer/CommandLineTools/usr/bin/ranlib: file: ../../../lib/libamrex.a(AMReX_BLProfiler.cpp.o) has no symbols [ 55%] Built target amrex ``` when building AMReX with CMake on macOS for C++-only users. Src/Base/CMakeLists.txt commit 58c8e1e9b1e71172a2db903caae633537b26183e Author: Candace Gilet Date: Fri Apr 1 11:41:15 2022 -0400 Minor update to Docs on Linear solver curvilinear coordinate (#2684) Docs/sphinx_documentation/source/LinearSolvers.rst commit 5792f4331a6c7b981345420763f24d7b48a1a7be Author: Weiqun Zhang Date: Fri Apr 1 08:15:14 2022 -0700 Update CHANGES for 22.04 (#2689) CHANGES commit 75e2e7d09e68a0caba4eea2dba7f662b6ecee5bc Author: Axel Huebl Date: Thu Mar 31 16:54:17 2022 -0700 CI: NVHPC New Apt Repo (#2687) Update the NVHPC install instructions to the latest and greatest. Fix failing CI (dependency install). .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_nvhpc21-11.sh .github/workflows/dependencies/dependencies_nvhpc21-9.sh commit 3aafa306ad820bbcc50a4b7c14fe912406427d1d Author: philip-blakely <46958218+philip-blakely@users.noreply.github.com> Date: Thu Mar 31 01:34:59 2022 +0100 Fix parser expressions (#2682) * Fix segmentation faults in amrex::Parser if a constant was not defined. Without this fix, the following segments of code all seg-fault. With this fix, each of them will throw an exception instead. (If the setConstant lines are uncommented, the tests will pass with or without this fix.) { amrex::Print() << "Attempt 2+a" << std::endl; amrex::Parser parser("2+a"); //parser.setConstant("a", (amrex::Real)3.0); const auto parserExec = parser.compile<0>(); AMREX_ALWAYS_ASSERT(parserExec() == 5); } { amrex::Print() << "Attempt 2-a" << std::endl; amrex::Parser parser("2-a"); //parser.setConstant("a", (amrex::Real)3.0); const auto parserExec = parser.compile<0>(); AMREX_ALWAYS_ASSERT(parserExec() == -1); } { amrex::Print() << "Attempt 2*a" << std::endl; amrex::Parser parser("2*a"); //parser.setConstant("a", (amrex::Real)3.0); const auto parserExec = parser.compile<0>(); AMREX_ALWAYS_ASSERT(parserExec() == 6); } { amrex::Print() << "Attempt 3/a" << std::endl; amrex::Parser parser("3/a"); //parser.setConstant("a", (amrex::Real)3.0); const auto parserExec = parser.compile<0>(); AMREX_ALWAYS_ASSERT(parserExec() == 1); } { amrex::Print() << "Attempt -a" << std::endl; amrex::Parser parser("-a"); //parser.setConstant("a", (amrex::Real)3.0); const auto parserExec = parser.compile<0>(); AMREX_ALWAYS_ASSERT(parserExec() == -3); } Src/Base/Parser/AMReX_Parser_Exe.cpp commit 8b691cbf8ddb2973ee0e38c2d95f609562da3348 Author: Ben Wibking Date: Wed Mar 30 03:56:05 2022 +1100 fixes to build CUDA with Clang (#2681) Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_TypeTraits.H commit 9907ac19751854868b25d50d245b2045a00227f6 Author: Axel Huebl Date: Mon Mar 28 19:30:04 2022 -0700 CI: Apple Silicon (#2672) * CI: Apple Silicon Make sure we can cross-compile AMReX from Intel CPUs (x86) to "Apple Silicon"/M1 (arm64). Using fancy universal binaries here that even support both architectures at the same time. Refs.: - https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary - https://cmake.org/cmake/help/latest/envvar/CMAKE_OSX_ARCHITECTURES.html * Universal: w/o MPI - brew MPI is not a universal binary - remove ccache (nothing to ccache here between runs yet) * Fix: Unused Variable Follow-up to #1969 * macOS: Signal Code Explicit if x86 .github/workflows/macos.yml Src/Base/AMReX.cpp commit 1b73099a80e617696fc342fa8ac16e2d63ecab4b Author: Andrew Myers Date: Fri Mar 25 11:24:55 2022 -0700 Use the roundoff domain in enforcePeriodic. (#2679) Src/Base/AMReX_Geometry.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H commit 619630d180311310b6ccf6407af2877258965436 Author: Donald E. Willcox Date: Thu Mar 24 19:30:47 2022 -0700 Fix some HIP warnings for IntegratorBase class (#2678) * fix unused variable warning by using AMREX_ALWAYS_ASSERT. * fix unused variable warning in IntegratorBase constructor. Src/Base/AMReX_IntegratorBase.H commit d0375d83cf233e8ddc002a6c3d6a6f0b3342a2ae Author: Weiqun Zhang Date: Thu Mar 24 16:00:51 2022 -0700 Catch CUDA 11.6 bug at compile time (#2677) Make it a compile time error if CUDA 11.6 is used to instantiate GpuBndryFuncFab. This used to be a mysterious runtime error (#2607). Note that if GpuBndryFuncFab is not used in your code, you can still use CUDA 11.6. However, the bug might still be triggered at runtime in other places. Src/Base/AMReX_BCUtil.cpp Src/Base/AMReX_PhysBCFunct.H commit 4f463b6fbba78f8d1e21e19f1c6c50f2a0d0be3c Author: Jean M. Sexton Date: Thu Mar 24 15:16:46 2022 -0700 MPI+OMP+HIP build capability (#2676) * Link explicitly to gnu OpenMP lib * Add HIP exception for AMREX_USE_OMP and _OPENMP matching Tools/CMake/AMReX_Config.H.in Tools/GNUMake/comps/hip.mak Tools/libamrex/mkconfig.py commit 95b9b145b74de944a365255191731d78b599a19d Author: drangara <69211175+drangara@users.noreply.github.com> Date: Thu Mar 24 15:33:44 2022 -0400 EB Flow (#2661) There are 2 sets of changes in this PR to support flow from/through embedded boundary faces * support for setting EB Dirichlet value for velocity in the nodal solver - this includes computation of surface integrals and adding the EB flow contribution to the divergence. * adding a new call for the method EB_computeDivergence which takes in the velocity at EB faces and adds the EB flow contribution to the divergence. Co-authored-by: Jordan Musser Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/LinearSolvers.rst Src/EB/AMReX_EBMultiFabUtil.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_algoim.H Src/EB/AMReX_algoim.cpp Src/EB/AMReX_algoim_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_eb.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 2ccb8166b22c853ffe1ae3049ca44ddc082b4236 Author: Jean M. Sexton Date: Wed Mar 23 16:47:42 2022 -0700 BackTrace workarounds (#2667) * Change -g to -gdwarf-4 for gcc 11 or newer * Use addr2line if eu-addr2line gives ? Co-authored-by: Weiqun Zhang Src/Base/AMReX_BLBackTrace.cpp Tools/GNUMake/comps/gnu.mak commit e213d767baab66574b2f29daf23e7288e6f1a8df Author: Revathi Jambunathan Date: Tue Mar 22 22:20:30 2022 -0700 Redefine ParticleContainter multifab after setting particle boxarray and dmap (#2673) * call redefine mf after set particle BA and dmap * fix test Co-authored-by: Andrew Myers Src/Particle/AMReX_ParticleContainerBase.cpp commit ee651cd5b301bc820435fda03a5eea271dfa8701 Author: Andrew Myers Date: Tue Mar 22 19:57:05 2022 -0700 Handle the case where we lose a level in RedistributeGPU (#2670) Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particles.H Tests/Particles/Redistribute/inputs.rt.cuda.mr Tests/Particles/Redistribute/main.cpp commit 1ac5d7919b44f9c589253226c0c512acf27fd0d3 Author: Andrew Myers Date: Tue Mar 22 13:42:07 2022 -0700 Follow on to 2668, fix new bug in test. (#2671) Tests/Particles/ParticleMesh/main.cpp commit cfe53e955d05f07668a9918ff2fab4775454e49f Author: Donald E. Willcox Date: Tue Mar 22 11:57:00 2022 -0700 Add override statements to avoid certain compiler warnings with GCC and Clang. (#2669) Src/Base/AMReX_FEIntegrator.H Src/Base/AMReX_RKIntegrator.H Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H commit 0082c129d340a7b4c414f96af443bbb7743006a0 Author: Axel Huebl Date: Tue Mar 22 11:46:42 2022 -0700 Doc: AMReX_TP_PROFILE (#2659) Document third-party profiling options. Doc-string taken from CMake `option()`. Docs/sphinx_documentation/source/BuildingAMReX.rst commit 336ca7002b45f3237336dfd8e2c4334437eb404b Author: Jean M. Sexton Date: Mon Mar 21 20:44:56 2022 -0700 Add int overflow assert check before Bcast wrapper (#2649) Src/Particle/AMReX_ParticleInit.H commit a2f18e83148d8031134e39649e9dec8a1698ff51 Author: Andrew Myers Date: Mon Mar 21 15:46:16 2022 -0700 Allow ParticleToMesh and MeshToParticle to access runtime particle components (#2668) Src/Particle/AMReX_ParticleUtil.H Tests/Particles/ParticleMesh/main.cpp commit 3c5afdc9dd15af28523d1c7ea16909b3bb3f67d7 Author: Weiqun Zhang Date: Mon Mar 21 12:56:44 2022 -0700 Fix bugs in the MultiBlock test (#2662) Tests/MultiBlock/Advection/main.cpp commit 36fa76c4282a31b3fbcce752c4f2b353400f6748 Author: Andrew Myers Date: Mon Mar 21 12:53:18 2022 -0700 Separate version strings for particle plotfiles and checkpoints. (#2663) Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_WriteBinaryParticleData.H commit 7b4d10daf9c4fdb011801bd5fd2562d6b438f3b0 Author: Weiqun Zhang Date: Fri Mar 18 10:14:29 2022 -0700 Fix warnings (#2665) * Unused variables in FluxRegister * Extra tokens at end of #endif directive according to gfortran Src/AmrCore/AMReX_FluxRegister.cpp Tools/CMake/AMReX_Config.H.in commit dfb0e098df468235f8f7b6fd50d06914c365177f Author: Houjun Tang Date: Wed Mar 16 17:33:50 2022 -0700 Add support for SZ compression in HDF5 output (#2644) Docs/sphinx_documentation/source/IO.rst Docs/sphinx_documentation/source/Visualization.rst Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Extern/HDF5/AMReX_ParticlesHDF5.H Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.H Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Tests/HDF5Benchmark/GNUmakefile Tests/HDF5Benchmark/inputs Tests/HDF5Benchmark/main.cpp Tests/HDF5Benchmark/sz.config Tools/GNUMake/packages/Make.hdf5 commit 04621415c77cbec25fd68ee7077a6ab50b37893d Author: Scot Breitenfeld Date: Wed Mar 16 12:11:45 2022 -0500 Removed the use of H5Aget_storage_size (#2656) Src/Extern/HDF5/AMReX_ParticleHDF5.H commit bbe5da81ee2a902c838ffc8d479dcf775e342b41 Author: Erik Date: Wed Mar 16 11:09:38 2022 -0400 Docs to install volpack via package (#2654) Docs/sphinx_documentation/source/Visualization.rst commit 3c644cb0a46a1eb730ed2eaa0c6cc00180ad017b Author: Max Katz Date: Tue Mar 15 08:58:41 2022 -0700 Add prefetchToHost and prefetchToDevice for GPU containers (#2655) Src/Base/AMReX_GpuContainers.H commit 806f2373b68efa37120c393264647178304720b7 Author: Axel Huebl Date: Fri Mar 11 11:57:22 2022 -0800 Introduce `AMReX_Version.H` (#2653) The defined `AMREX_GIT_VERSION` and `AMREX_RELEASE_NUMBER` change in every commit. This modifies the `AMReX_Config.H` file as well for every commit, which is included in every header file. Using always the latest commit counteracts `ccache` since the signature of the compilation call changes every commit. Move this define and `AMREX_RELEASE_NUMBER` to a separate version file, which is included only where needed (currently in a single `.cpp` file). Co-authored-by: Weiqun Zhang Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_Version.cpp Src/Base/CMakeLists.txt Src/Base/Make.package Tools/CMake/AMReXGenerateConfigHeader.cmake Tools/CMake/AMReX_Config.H.in Tools/CMake/AMReX_Version.H.in Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/libamrex/mkversionheader.py commit c5dce693676eb7ac505b64eb1f88fa9a8873b67c Author: Weiqun Zhang Date: Thu Mar 10 10:34:27 2022 -0800 Add move assignment operator to Fab (#2652) Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_IArrayBox.H commit fbdfc840de7c30d8edc01a0eedeb3edddc19e5b5 Author: Donald E. Willcox Date: Wed Mar 9 19:33:49 2022 -0800 Added an option to set fast MRI dt directly instead of using the integer ratio of slow/fast dt. (#2651) Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_TimeIntegrator.H Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H commit 5e3404e6779629d7f3365e31ca520bbfaca0911b Author: Axel Huebl Date: Wed Mar 9 19:31:26 2022 -0800 AMReX_Config.H: AMREX_EXPORT_DYNAMIC (#2650) Forgot to add the new define there. Tools/CMake/AMReX_Config.H.in commit 6bbcbbb7829902e9038bee8c28be09800b22aab2 Author: Andrew Myers Date: Fri Mar 4 11:44:48 2022 -0800 Fix bug affecting neighbor proc calculation with mesh refinement. (#2646) Src/Particle/AMReX_NeighborParticlesI.H commit 3fe001b36504ee40dbe3131543b7265864d7e5b2 Author: Weiqun Zhang Date: Thu Mar 3 08:28:12 2022 -0800 Update flags for DPC++ AOT (#2643) Argument spir64_gen-unknown-unknown-sycldevice has been deprecated recently. We need to use spir64_gen instead. Note that we are not trying to support old oneAPI compilers. So we simply replace the deprecated argument with the new one. Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit d544b2a146cfefb008cf8d86da20f569b37927d6 Author: Axel Huebl Date: Thu Mar 3 08:27:51 2022 -0800 Doc + CMake: AMREX_EXPORT_DYNAMIC (#2635) - Add the CMake control for `AMREX_EXPORT_DYNAMIC` - Add documentation Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake commit 57fbb98d2a780c2a4ee3e2305771466013c0733d Author: Erik Date: Thu Mar 3 02:39:28 2022 -0500 Update SpackSmokeTest to run with Fortran and/or HIP. (#2629) * Update SpackSmokeTest to run with Fortran and/or HIP. * add error checks for cmake versions depending on GPU_BACKEND Tests/SpackSmokeTest/CMakeLists.txt commit 0ce37796a789db8f0f60a4169bc7db8f99dea732 Author: Weiqun Zhang Date: Wed Mar 2 18:14:28 2022 -0800 Simplify CoordSys::IsRZ and IsSPHERICAL (#2642) In the past, the enum types RZ and SPHERICAL were not defined in 3D and therefore could not be used in IsRZ and IsSPHERICAL in 3D. This is no longer the case. Src/Base/AMReX_CoordSys.H commit b982aa80781165b6962c3c7d964bb996fab28ae4 Author: Andrew Myers Date: Wed Mar 2 13:42:45 2022 -0800 Take advantage of async copies in PODVector. (#2641) Src/Base/AMReX_PODVector.H commit ab28124273f74265c948744076b9cd0c2e48c02e Author: Weiqun Zhang Date: Wed Mar 2 12:04:15 2022 -0800 HIP: atomicAddNoRet (#1809) It has been deprecated since ROCm 4.1.0. However, for performance reasons, we should still use it for versions < 5. Src/Base/AMReX_GpuAtomic.H commit 02eb74f1b62c8bf496c9b8646bbaa7cb6a20d1a4 Author: Erik Date: Wed Mar 2 11:34:46 2022 -0500 Add some doxygen comments to MLABecLaplacian (#2628) Adds some doxygen comments to ML_ABecLaplacian code. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp commit 7e73c1ed1c142b83afce7bcaec731d2dc334fbe3 Author: Weiqun Zhang Date: Tue Mar 1 10:29:18 2022 -0800 Update CHANGES for 22.03 (#2639) CHANGES commit 3199edf6152589fd04db5fe814d9e0d8a604ba2b Author: Axel Huebl Date: Mon Feb 28 09:47:58 2022 -0800 ParticleTile: push_back_real/int w/ Vector (#2634) Add more overloads to `push_back_real`/`int` to support `amrex::Vector` and its const iterators. This makes initialization routines more natural (similar to `std::vector` constructors). Src/Particle/AMReX_ParticleTile.H commit 067830e6c9fe74af9ae1e54e4359174bff6344a4 Author: Candace Gilet Date: Fri Feb 25 11:57:28 2022 -0500 Update Docs to reflect newer options in MLMG (#2620) Co-authored-by: etpalmer63 Docs/sphinx_documentation/source/LinearSolvers.rst commit aa0dc39f50f714a76a7b5df630a641cc4e411642 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Fri Feb 18 14:14:17 2022 -0600 Allow the ghost-particle exchange to communicate different variables (#2627) than particle redistribution on GPU. Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_Particles.H commit c45975c0894af19ce7c623fac767ff4807537643 Author: Andrew Myers Date: Fri Feb 18 11:28:16 2022 -0500 Fix AmrParticleContainer::AssignDensity when ncomp == AMREX_SPACEDIM+1. (#2632) Src/AmrCore/AMReX_AmrParticles.H commit 1d566b101fa3f6796c87410376f92351602b7c6c Author: Andrew Myers Date: Wed Feb 16 23:15:42 2022 -0500 Use async version of cuda/hipMemcpy in PODVector::resize() (#2631) Src/Base/AMReX_PODVector.H commit d618810d677d485f48e8de2ae12cb3e9c79ba3c1 Author: Axel Huebl Date: Wed Feb 16 12:09:42 2022 -0800 ParticleContainer::make_alike (#2630) This creates a new AMReX particle container type with same compile-time and run-time attributes. But, it can change its allocator. This is helpful when creating temporary particle buffers for filter operations and device-to-host copies. Src/Particle/AMReX_Particles.H commit 76d08651adb987e3fba6b232e806c5e7c365a8d9 Author: Andrew Myers Date: Mon Feb 14 13:10:52 2022 -0500 Update checkpoint format to account for expanded particle ids. (#2624) Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Extern/HDF5/AMReX_ParticlesHDF5.H Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H commit df93daec8da2d5df7057affacd629d18233f40be Author: Candace Gilet Date: Mon Feb 14 12:05:49 2022 -0500 Fix formatting. Note cuda-memcheck name change to compute-sanitizer (#2626) Docs/sphinx_documentation/source/Debugging.rst Docs/sphinx_documentation/source/GPU.rst commit 0f55de0e2ef6a66ae2c0e5aabc7979c8cc140e04 Author: Weiqun Zhang Date: Thu Feb 10 09:29:54 2022 -0800 Update FabArray::copyTo (#2603) Remove the two versions taking a sub box. I don't think those versions are used by any codes. This allows for simplification of the implementation. Reimplement copyTo to reduce the number of bcast to just one. Also the new versions work without managed memory. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H commit 9558c8a1684c54bc5ffd10c6b2f8f7396c462761 Author: Michael Zingale Date: Tue Feb 8 16:21:29 2022 -0500 explicitly request python3 for makebuildinfo_C.py (#2621) some systems might still have a python 2 interpreter around Tools/C_scripts/makebuildinfo_C.py commit 8e2eaa4a7caf3fed4ebd9aed975c62ae540fe842 Author: Michael Zingale Date: Tue Feb 8 11:05:20 2022 -0500 store the CUDA version in build info (#2617) For codes that use makebuildinfo_C.py, this now allows you to get the CUDA version to potentially store in output files. Tools/C_scripts/AMReX_buildInfo.H Tools/C_scripts/makebuildinfo_C.py commit 3fb8caaf78a04928944d6a535bb675b82160311f Author: ashesh2512 <36968394+ashesh2512@users.noreply.github.com> Date: Mon Feb 7 12:19:10 2022 -0700 missing check for mapped solver support when not using GPU (#2616) Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 212ffa61c1651f041393f46efbfccaabd29be3e8 Author: Kevin Gott Date: Fri Feb 4 13:10:29 2022 -0800 Remove ifdefs for older compiler version. (#2614) Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_Scan.H commit 32c9faa8b17abf8cf74736155c7978f34957808b Author: philip-blakely <46958218+philip-blakely@users.noreply.github.com> Date: Fri Feb 4 18:43:17 2022 +0000 Allow a descriptor component's BndryFuncFabDefault to be stateful. (#2612) ## Summary This allows a boundary condition function to depend on the descriptor index or other information only available when BndryFunc::setComponent() is called. ## Additional background Previously, the BndryFuncFabDefault function passed to BndryFunc::setComponent() pointed to a function that did not take descriptor index or other information. In some cases (e.g. multi-material simulations where each material is in its own Descriptor), extra information is needed to map the boundary-function call to the correct material's boundary condition. Making BndryFuncFabDefault a std::function instead of a plain function pointer makes this possible. Src/Amr/AMReX_StateDescriptor.H commit f79f40d4d0cb71bf0ca2271f76433c2c1cb0c029 Author: Weiqun Zhang Date: Thu Feb 3 20:41:33 2022 -0800 TableData: fix missing return in operator= (#2613) Src/Base/AMReX_TableData.H commit bf267b0954263f9f7d9c340c74bf50efa070f8cc Author: Donald E. Willcox Date: Tue Feb 1 09:57:07 2022 -0800 SUNDIALS MRI Interface for AMReX TimeIntegrator class (#2600) This PR adds an interface to the SUNDIALS ARKODE integrator for the AMReX TimeIntegrator class to use, identical to what we have implemented for the ERF code. This will let us move that implementation into the more abstract AMReX time integration API to allow others to take advantage of it. The new capabilities are described in the Docs page for Time Integration. This capability is a work in progress and we will add features as we need them. Docs/sphinx_documentation/source/TimeIntegration_Chapter.rst Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_TimeIntegrator.H Src/Extern/SUNDIALS/AMReX_SundialsIntegrator.H Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package commit 86a4e467b24343398dd8104da412531594172639 Author: Weiqun Zhang Date: Tue Feb 1 09:20:05 2022 -0800 Update CHANGES for 22.02 (#2610) CHANGES commit 89153db246e4193526210363bf5a0f1719ea40c3 Author: Weiqun Zhang Date: Mon Jan 31 10:54:04 2022 -0800 Fix I/O bug in Amr class (#2609) The file stream for the header must be closed before the parent directory can be renamed. Closes #2608 Src/Amr/AMReX_Amr.cpp commit 656383586c357670048001a2b37e34ba71ee7d36 Author: Erik Date: Mon Jan 31 12:18:28 2022 -0500 Spack install instructions for Building section. (#2602) Adds instructions for how to install AMReX using Spack to the Building with AMReX section. Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/BuildingAMReX_Chapter.rst Docs/sphinx_documentation/source/GettingStarted.rst commit fd6d6494bb75085b73602c3a30ec68a41383424d Author: Weiqun Zhang Date: Fri Jan 28 14:45:55 2022 -0800 Kernel fusing in Geometry (#2606) Src/Base/AMReX_Geometry.cpp commit c62334c4c5cd98ede8b416582af92b18699523de Author: Weiqun Zhang Date: Fri Jan 28 14:45:37 2022 -0800 Kernel fusing in FluxRegister (#2605) Src/AmrCore/AMReX_FluxRegister.cpp Src/Base/AMReX_TagParallelFor.H commit 2fc7776086225cf0c774bd3ea6f21a2794f3cc42 Author: Weiqun Zhang Date: Fri Jan 28 14:44:38 2022 -0800 TableData: A Multi-Dimensional Array Class (#2601) This class is somewhat similar to FArrayBox/BaseFab. The main difference is the dimension of the array in this class can be 1, 2, 3, or 4, whereas the dimension of FArrayBox/BaseFab is the spatial dimension (AMREX_SPACEDIM) plus a component dimension. This is a feature request by Maui, which will use it to store EOS data. Src/Base/AMReX_BaseFab.H Src/Base/AMReX_DataAllocator.H Src/Base/AMReX_TableData.H Src/Base/CMakeLists.txt Src/Base/Make.package commit f7c9c3bd73473736080c7a606631a54e067aed90 Author: Weiqun Zhang Date: Thu Jan 27 15:16:05 2022 -0800 FabSet::multiFab (#2604) Add public member functions to FabSet to expose the internal MultiFab data. Src/Boundary/AMReX_FabSet.H commit d38b4237444cb22cfadb4ddc63434c8b5a2c9b42 Author: Weiqun Zhang Date: Thu Jan 27 14:04:48 2022 -0800 Disable EB solver's phi-on-centroid for hip for now (#2598) * Disable EB solver's phi-on-centroid for hip for now We need this to work around the out-of-register issue with current compilers. Co-authored-by: Axel Huebl Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_F.cpp commit 874ccc6f3f8dedf842082816f6c8aa2dfceb5830 Author: Candace Gilet Date: Mon Jan 24 11:47:55 2022 -0500 Docs: Update info on print_state to include ng (#2599) Docs/sphinx_documentation/source/Debugging.rst commit 3a2a74fc4b383fdda1d85abb7cd34fe8a6888362 Author: Erik Date: Sun Jan 23 13:43:26 2022 -0500 Update locations for tutorials and minor formatting. (#2583) ## Summary Update remaining locations for tutorials referenced in the docs. ## Additional background Follow up to #2576 Docs/sphinx_documentation/source/Fortran.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GettingStarted.rst Docs/sphinx_documentation/source/LinearSolvers.rst Docs/sphinx_documentation/source/Testing.rst Docs/sphinx_documentation/source/Visualization.rst commit be204df0f3056217d2037f03d26dbcf432c295bf Author: Axel Huebl Date: Fri Jan 21 12:20:54 2022 -0800 CI: AMD `hip::device` also for Fortran (#2595) ## Summary By now (HIP 4.5), we should be able to link also against Fortran targets. Let's remove work-arounds and ditch old HIP releases, so improve maintainability and avoid surprises. ## Additional background X-ref: - https://github.com/ROCm-Developer-Tools/HIP/pull/2280 (https://github.com/ROCm-Developer-Tools/HIP/pull/2190) - https://github.com/ROCm-Developer-Tools/HIP/issues/2275 .github/workflows/dependencies/dependencies_hip.sh .github/workflows/hip.yml Tools/CMake/AMReXParallelBackends.cmake commit 947263e0076e08e3ef44db8009fd2ebd46238bd0 Author: Michael Zingale Date: Fri Jan 21 13:46:36 2022 -0500 fix vol initialization for Cartesian (#2596) Tools/Plotfile/fvolumesum.cpp commit e3b1468b8f16d11eebf4b10cca32829c78fb5d35 Author: Erik Date: Thu Jan 20 18:33:53 2022 -0500 Add headings and steps to plot vector field in ParaView doc section (#2588) Adds directions for ploting a vector field from 3 separate scalar values using the Calculator filter. Also split up the ParaView section by adding some headings. Docs/sphinx_documentation/source/Visualization.rst Docs/sphinx_documentation/source/Visualization/ParaView_vectorfield.png commit 3318e3678579dbf4d1af12b2a3d3ad7d1eb6877d Author: Weiqun Zhang Date: Thu Jan 20 10:30:08 2022 -0800 Kernel fusing in FabArrayUtility (#2593) Src/Base/AMReX_Box.H Src/Base/AMReX_FabArrayUtility.H commit b7050b410812f7772985fbe28716f4980c5bc942 Author: Candace Gilet Date: Wed Jan 19 16:16:53 2022 -0500 Change FPinfo's fact_fine_patch to always have ng=1. (#2591) This fixes a problem when using face_centered FillPatchTwoLevels with EB and an odd number of ghost cells, and fact_fine_patch's ebflag fab's box was too small. Src/Base/AMReX_FabArrayBase.cpp commit 12548009735ee00762ade0e90c52eaba3780b903 Author: Weiqun Zhang Date: Tue Jan 18 16:46:43 2022 -0800 Support arbitrary refinement ratio in FaceLinear Interpolater (#2590) Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.cpp commit fe7ef3402085c77c75e1e8a4ebcabc19f6020ea6 Author: Weiqun Zhang Date: Tue Jan 18 15:14:12 2022 -0800 Default Arena for MultiFab/iMultiFab/FabArray (#2589) * Default Arena for MultiFab/iMultiFab/FabArray Add a new constructor to MultiFab/iMultiFab/FabArray that can be used to set the default constructor that will be used when `define` is called with MFInfo containing a nullptr as Arena. Below is an example of using this. MultiFab mf(The_Pinned_Arena()); MultiFab mf2; VisMF::Read(mf, "mf"); VisMF::Read(mf2, "mf2"); // mf is allocated with The_Pinned_Arena(), whereas mf2 is allocated // with The_Arena(), in VisMF::Read. Co-authored-by: Andrew Myers Src/Base/AMReX_FabArray.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_iMultiFab.cpp commit be0d73dd6b67c9855fafb43e5c2fc8c2bfc9b3e6 Author: Erik Date: Tue Jan 18 12:23:10 2022 -0500 `add_par` precedence in Docs. (#2575) Docs/sphinx_documentation/source/Basics.rst commit 812836f08a758ef6f65caf7eea477bfc6423e72b Author: Candace Gilet Date: Mon Jan 17 20:36:00 2022 -0500 Fix typo in 1D version of AMREX_LAUNCH_DEVICE_LAMBDA_DIM (#2587) Src/Base/AMReX_GpuLaunch.H commit 89ccf0a4ca0044bd27e885b0050fdcb071b20cfe Author: Jean M. Sexton Date: Mon Jan 17 20:35:26 2022 -0500 CMake: remove nonexistent header file, update minimum sundials version (#2586) This makes CMake more consistent with expecting Sundials 6.x. Src/Extern/SUNDIALS/CMakeLists.txt Tools/CMake/AMReXThirdPartyLibraries.cmake commit 07267a0ed24dce8a482708904d57e3ae233eeed3 Author: Erik Date: Mon Jan 17 20:32:21 2022 -0500 Enable CUDA in CMakeLists.txt for Spack smoke test. (#2581) Tests/SpackSmokeTest/CMakeLists.txt commit 65f906216a5ec2abd3917213fb30e70b68bb6c68 Author: Weiqun Zhang Date: Thu Jan 13 14:39:05 2022 -0800 Fix warning from hiprand header (#2585) Src/Base/AMReX_RandomEngine.H commit 47c2b927f8b55dee7d4a80145253ac3d6c8f2204 Author: Axel Huebl Date: Thu Jan 13 11:07:36 2022 -0800 HIP: amdclang++ (#2582) On Crusher, `clang++` from AMD is named `amdclang++`. Add this to the known compiler name list and do not warn when encountered. Tools/CMake/AMReXParallelBackends.cmake commit 5576188c11d1fa834c650a6447e36dd1925897f0 Author: David Gardner Date: Thu Jan 13 10:33:02 2022 -0800 Update sundials sycl memory helper interface (#2580) Update usage of the sundials sycl memory helper for sundials 6.x Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp commit cde8e1c073d775f59ac04c8fc6152bb1a6dc8594 Author: Erik Date: Thu Jan 13 12:14:08 2022 -0500 Update locations from `amrex/Tutorials` to (#2576) `amrex-tutorials/ExampleCodes`. Docs/sphinx_documentation/source/AmrCore.rst Docs/sphinx_documentation/source/AmrCore_Chapter.rst Docs/sphinx_documentation/source/AmrLevel_Chapter.rst Docs/sphinx_documentation/source/Basics_Chapter.rst Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GettingStarted.rst Docs/sphinx_documentation/source/LinearSolvers_Chapter.rst Docs/sphinx_documentation/source/Particle.rst Docs/sphinx_documentation/source/Visualization.rst commit e34f5c67fec9745e643ed2605b82419fdddb8967 Author: Axel Huebl Date: Wed Jan 12 15:38:54 2022 -0800 HIP: Add -munsafe-fp-atomics to CMake (#2577) Otherwise atomicAdd is much slower. Same as already done in GNU Make. Tools/CMake/AMReXParallelBackends.cmake commit 83ec5fcfefae13264827a0724ab61d2d1b9a7eac Author: Weiqun Zhang Date: Wed Jan 12 14:52:56 2022 -0800 ParmPaser::queryAdd (#2573) Add a new member function, queryAdd, to ParmParse. If the name is found, the value in the ParmParse database will be stored in the reference argument. If not, the value in the argument will be added to the ParmParse database. The return value indicates if it existed previously. Many codes use ParmParse::dumpTable to store the ParmParse data in plot/checkpoint files. But for parameters that not present in inputs files or command line arguments, they would not appear in the dumped tables. The new queryAdd function can be used to address this issue. In amrex/Src, the calls to `query` are replaced by `queryAdd` so that it's easy to find their values. Src/Amr/AMReX_Amr.cpp Src/AmrCore/AMReX_AmrMesh.cpp Src/Base/AMReX.cpp Src/Base/AMReX_Arena.cpp Src/Base/AMReX_AsyncOut.cpp Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_ForkJoin.cpp Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MemProfiler.cpp Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParmParse.H Src/Base/AMReX_RKIntegrator.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_parstream.cpp Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_Level.H Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/Extern/SENSEI/AMReX_InSituBridge.cpp Src/F_Interfaces/Octree/AMReX_octree_fi.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_TracerParticles.cpp Src/Particle/AMReX_WriteBinaryParticleData.H commit 0d04b17d22d8ddb8920848eac213f6bdb858d8a0 Author: Weiqun Zhang Date: Wed Jan 12 12:22:02 2022 -0800 Fix regression test configuration (#2574) Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini commit 44388b2cd8f2ed166de70078f2f738fd689221ca Author: Erik Date: Wed Jan 12 12:55:27 2022 -0500 Links to tutorials in rst docs. (#2572) Add links to tutorials in Getting Started. In basics, change tutorials to example codes for differentiation. Also fixes broken heat equation link, and removes dead pointer. Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/GettingStarted.rst commit 405b490356e51f66fb649311594939c6d5e12067 Author: Erik Schnetter Date: Tue Jan 11 15:51:12 2022 -0500 Convert switch statement to if statement (#2570) The MPI standard does not guarantee that the thread level constants are compile-time constants; they might only be link-time constants which are not allowed in switch statements. Src/Base/AMReX_ParallelDescriptor.cpp commit af2509d7102c770931cf9219a0d649003d0a86a8 Author: Cody Balos Date: Tue Jan 11 11:10:25 2022 -0800 remove erroneous BL_ASSERT in sundials initialization (#2568) Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp Src/Extern/SUNDIALS/AMReX_Sundials.cpp commit 7314e78fecda4080f6bd3a1857d033545f7d8a56 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon Jan 10 19:11:30 2022 -0600 Support implicit function class defined on host when using device memory explicitly. (#2563) AMReX needs to use managed memory if the user-defined implicit function (IF) class's member functions are only callable by the host. This PR allows users to use IF classes defined on the host with device memory. Src/EB/AMReX_EB2_GeometryShop.H commit fa46bc9eff1306b8c3eeaf1ae32d070271606bba Author: Weiqun Zhang Date: Mon Jan 10 13:55:09 2022 -0800 HIP: Add -munsafe-fp-atomics to GNU Make (#2567) Otherwise atomicAdd is much slower. Note this is not done for CMake yet. Tools/GNUMake/comps/hip.mak commit 172cab862a1812437cbd78101592fd2ffcfd2d8f Author: Weiqun Zhang Date: Mon Jan 10 12:06:01 2022 -0800 GNU Make: Add a new machine, crusher (#2566) Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit 0278677bbc8c7fde6edf8fa5ea3cfd0d55db8b1f Author: Ann Almgren Date: Fri Jan 7 15:47:17 2022 -0800 Mapped solver support (#2088) This change to the AMReX_MLNodeLaplacian* files allows us to pass in a multi-component sigma to the nodal projector and this will be used as {sigmax,sigmay,sigmaz} internally. This allows us to have different weightings for the derivatives in the different coordinate directions which is useful for solves on nonuniform rectilinear meshes. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_hypre.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 1404c4a9acc65bd99a70db5be9a4ccc5ab701e7a Author: Cody Balos Date: Thu Jan 6 16:02:03 2022 -0800 Update to SUNDIALS 6 (#2551) ## Summary Update AMReX to use SUNDIALS 6. We could make this backwards compatible with SUNDIALS 5.7.0/5.8.0 without too much effort (I think), if that is desired. ## Additional background SUNDIALS 6 introduces a `SUNContext` class (needed by all/most other SUNDIALS classes) that should be associated with only one SUNDIALS simulation at a time. Thus, when using multiple threads, multiple `SUNContext` objects are needed. This is why we create one (and one `SUNMemoryHelper`) for each OpenMP thread. Src/Base/AMReX.cpp Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.H Src/Extern/SUNDIALS/AMReX_SUNMemory.H Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp Src/Extern/SUNDIALS/AMReX_Sundials.H Src/Extern/SUNDIALS/AMReX_Sundials.cpp Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package commit f36e8cbd80e189fe3ce69db5c4366023c816654a Author: Weiqun Zhang Date: Thu Jan 6 11:31:49 2022 -0800 Fix a number of warnings (#2564) Src/Base/AMReX_FileSystem.cpp Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_VisMF.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 29b6a8ebb76f43781046fc20b84f7d1616c66888 Author: Kevin Gott Date: Thu Jan 6 13:15:43 2022 -0500 Return a zero-size vector in ParallelDescriptor::Gather. (#2560) The Gather implementation that returns a vector currently returns an un-initialized, 1-element vector to the non-root ranks. This seems unintuitive as a user and requires very careful consideration of the returned vector on non-root ranks. This PR changes the non-root return to a zero-size vector instead. Src/Base/AMReX_ParallelDescriptor.H commit aa079d4dc93b5a72935e1616a0431a838adae8a1 Author: Andrew Myers Date: Thu Jan 6 12:26:52 2022 -0500 Add plotfile diff to neighbor particles test. (#2562) Tests/Particles/NeighborParticles/main.cpp commit 98a66afcf4162b5d3539642dd2cd868d661976be Author: Andrew Myers Date: Thu Jan 6 12:24:38 2022 -0500 Add option not to remove particles with negative ids when calling Redistribute. (#2561) This is useful for WarpX when interacting particles with EB walls. The default behavior of Redistribute is not changed by this PR. Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particles.H Tests/Particles/Redistribute/main.cpp commit c34082833c563ff9dab469fb610164ae57dd9a88 Author: Candace Gilet Date: Thu Jan 6 11:49:34 2022 -0500 Update FillPatchTwoLevels for single component face-centered data (#2539) Allow the non-array version of FillPatchTwoLevels to operate on face-centered data and use the valid fine values on the coarse-fine interface during the interpolation to ghost cells. This also changes the FaceLinear Interpolater to use the valid fine data on the coarse-fine interface in the interpolation to ghost cells. Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp commit 5726371bbb1a9fabb68640db67f3482d78fc33c7 Author: Weiqun Zhang Date: Wed Jan 5 10:06:05 2022 -0800 Kernel fusing in FabArray Comm (#2559) Note that amrex::Add has been moved from AMReX_FabArrayUtility.H to AMReX_FabArray.H so that it can be used by AMReX_FabArrayCommI.H. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabArrayUtility.H commit dfee5b74765e4ec7f0bc28830514168595c1d148 Author: Weiqun Zhang Date: Tue Jan 4 14:09:35 2022 -0800 Use MF ParallelFor in error tagging (#2558) Src/AmrCore/AMReX_ErrorList.cpp commit 458679dbff56a7ca18d8bbf3994a1a172fa2e5c3 Author: Weiqun Zhang Date: Mon Jan 3 11:50:26 2022 -0800 iMultiFab I/O (#2495) * iMultiFab I/O Add functions for writing and reading iMultiFab. * Update Src/Base/AMReX_VisMF.H Co-authored-by: Andrew Myers Co-authored-by: Andrew Myers Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_IntConv.H Src/Base/AMReX_IntConv.cpp Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_VectorIO.H Src/Base/AMReX_VectorIO.cpp Src/Base/AMReX_VisMF.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_VisMFBuffer.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp commit dd75ba5ec7276a494e61b5d53226c88be80b5299 Author: Weiqun Zhang Date: Mon Jan 3 09:39:35 2022 -0800 Embed more EB data in EBFArrayBox (#2550) * Embed more EB data in EBFArrayBox Before this, we only had access to EBCellFlag given an EBFArrayBox. This adds access to more EB data via EBFArrayBox. This could be useful for functions with Fab parameters (e.g., AmrLevel's derive functions). * Mark EBFArrayBox(Arena*) explicit Src/EB/AMReX_EBFArrayBox.H Src/EB/AMReX_EBFArrayBox.cpp Src/EB/AMReX_EBFabFactory.cpp Src/EB/AMReX_MultiCutFab.H Src/EB/AMReX_MultiCutFab.cpp commit d36950d8589cad77558519c00f9fa0f46015d5b5 Author: Weiqun Zhang Date: Mon Jan 3 08:36:20 2022 -0800 CpuBndryFuncFab: Face Data (#2545) For completeness, fab_filfc is implemented for CpuBndryFuncFab. However, one should use GpuBndryFuncFab that works for both GPU and CPU builds. Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_FilFC_1D_C.H Src/Base/AMReX_FilFC_2D_C.H Src/Base/AMReX_FilFC_3D_C.H Src/Base/AMReX_FilFC_C.H Src/Base/AMReX_FilFC_C.cpp Src/Base/AMReX_PhysBCFunct.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit a73ce6759c914bdf15c44e58e839a744b5878be3 Author: Weiqun Zhang Date: Sun Jan 2 14:55:14 2022 -0800 Fix bug in Parser (#2555) Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser_Y.cpp commit 39c9cb6221959a8761bebf3b372f97dc65635ff3 Author: Weiqun Zhang Date: Sat Jan 1 14:03:53 2022 -0800 Update CHANGES for 22.01 (#2554) CHANGES commit 7c606b85c6bc22e1321727ed6692dd1e591e2b72 Author: Max Katz Date: Thu Dec 30 21:24:14 2021 -0800 Allow CUDA compilation on Perlmutter if only PrgEnv-nvidia is loaded (#2553) In this scenario, there is not a CUDA module loaded by default; nvcc is provided as part of the NVIDIA HPC SDK. In the Cray PE the environment variable defined pointing to the top level of the HPC SDK is `NVIDIA_PATH` so we can use this to indicate to the make system that we should have CUDA support. Tools/GNUMake/sites/Make.nersc commit bcdb6ca488826e10cf28691b0ed2ecc776753d55 Author: Kevin Gott Date: Thu Dec 23 19:52:37 2021 -0500 Add Gatherv for USE_MPI=FALSE. (#2549) This adds a Gatherv implementation for when USE_MPI=FALSE. Src/Base/AMReX_ParallelDescriptor.H commit c5f0685554aaff7588d71a535050121dbc8c393f Author: Kevin Gott Date: Thu Dec 23 19:51:26 2021 -0500 Resize vector inside GatherLayoutDataToVector (#2548) The output vector of GatherLayoutDataToVector doesn't resize the resulting vector; the user has to pass a minimally sized vector themselves. This PR resizes the result vector inside the function. Src/Base/AMReX_ParallelDescriptor.H commit 29853f0b30252deabb343e0a919b77b480bbba60 Author: Kevin Gott Date: Thu Dec 23 15:09:32 2021 -0500 Add missing FillBoundary_nowait (#2546) The `IntVect nghost` function signature doesn't have a matching `nowait` call. This PR adds it. Src/Base/AMReX_FabArray.H commit 24e25124cc179ee7e5aed1a06c3b9800a156595d Author: Kevin Gott Date: Thu Dec 23 15:06:33 2021 -0500 Remove unused variable. (#2547) Src/Base/AMReX_FabArrayCommI.H commit 85284ea15a3c5ca7d6d8e3f1045ac1e5a48c6a19 Author: Weiqun Zhang Date: Tue Dec 21 09:48:05 2021 -0800 Fix compilation for 1D build with hypre (#2544) Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 77ccf474ba9c4123f96be0b56ef08df1985af24a Author: Michael Zingale Date: Tue Dec 21 12:47:40 2021 -0500 add a tool to compute the volume integral of a plotfile field (#2541) Tools/Plotfile/fvolumesum.cpp commit df6893521551ab14b9d87f9610786a42eafcd9d9 Author: Jean Luca Bez Date: Tue Dec 21 08:44:38 2021 -0800 Update read path for HDF5 benchmark (#2542) Update path that was not used after including the directory for read Tests/HDF5Benchmark/main.cpp commit a38ccbc90843d32cf34beb7d89c60d95af3383d3 Author: Donald E. Willcox Date: Mon Dec 20 12:26:52 2021 -0800 Eliminate compiler warnings for time integration classes. (#2540) Src/Base/AMReX_FEIntegrator.H Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_RKIntegrator.H Src/Base/AMReX_TimeIntegrator.H commit e217e4a50525d4f973e1a5ba3c2510c4814cf745 Author: Weiqun Zhang Date: Mon Dec 20 11:50:42 2021 -0800 Update MLEBNodeFDLaplacian (#2538) * Update MLEBNodeFDLaplacian * Make it work for non-EB builds too. * Add option to solve in RZ coordinates. In that case, sigma is always one. * Assert that for 2d RZ with the lo-r boundary at r = 0, the domain BC is set to Neumann. Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/Make.package commit 134d33e4b7f0c9999ab9cfc2f1c129c47be31cee Author: Andy Nonaka Date: Tue Dec 14 15:38:46 2021 -0800 fix WritePlotfileToASCII compilation (#2537) Co-authored-by: Jean M. Sexton Tools/Postprocessing/C_Src/Make.package commit c6978ac1efb1afbe8625d1cc48e0c7ae68a648fc Author: Houjun Tang Date: Tue Dec 14 12:12:17 2021 -0800 Add missing const (#2536) Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp commit 9373709e34b23add981551d5446bc4810fd3b688 Author: Jean M. Sexton Date: Fri Dec 10 22:24:55 2021 -0800 Fix assert to use IntVect (#2531) Src/Base/AMReX_MultiFabUtil.H commit 8a1f7651a4723574a276f072a963cc5eb2e7dacc Author: Jean M. Sexton Date: Fri Dec 10 21:04:19 2021 -0800 Update NVector_MultiFab functionality from ATPESC-codes and add to Src/Extern/SUNDIALS (#2505) Adds NVector_MultiFab from https://github.com/AMReX-Codes/ATPESC-codes Updates NVector MultiFab wrapper for the GPU, and adds a NormHelper function Adds a check for SUNDIALS_FOUND in the CMake setup of SUNDIALS as a ThirdPartyLibrary Src/Base/AMReX_MultiFabUtil.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.H Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp Src/Extern/SUNDIALS/AMReX_Sundials.H Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package Tools/CMake/AMReXThirdPartyLibraries.cmake commit 55afaaf5945ad4b55483a3fe623f50d1768d9be6 Author: Erik Date: Fri Dec 10 20:13:48 2021 -0500 Projections migrated to AMReX-Hydro (#2530) Projections have been moved to AMReX-Hydro. This PR removes them from AMReX. Docs/sphinx_documentation/source/LinearSolvers.rst GNUmakefile.in Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/LinearSolvers/Projections/Make.package Tests/LinearSolvers/MAC_Projection_EB/CMakeLists.txt Tests/LinearSolvers/MAC_Projection_EB/GNUmakefile Tests/LinearSolvers/MAC_Projection_EB/Make.package Tests/LinearSolvers/MAC_Projection_EB/README Tests/LinearSolvers/MAC_Projection_EB/inputs_3d Tests/LinearSolvers/MAC_Projection_EB/main.cpp Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini commit 50f8e4314be75003610378d159134960ad27e64b Author: Andrew Myers Date: Fri Dec 10 19:13:07 2021 -0500 Add another overload for MakeSimilarDM. (#2528) This one works with an input `ba` and `dm` instead of a `MultiFab.` Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp commit dbbac7b9649ce7feecdd80d7e46b97438fb51173 Author: Donald E. Willcox Date: Fri Dec 10 12:45:28 2021 -0800 Adds `get` functions to the TimeIntegrator classes (#2529) * Add get_rhs and get_post_update functions to allow user code to retrieve std::functions for the RHS and post_update. Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_TimeIntegrator.H commit afe587b071d9f1a0f2b95125d8d3204a01cf8967 Author: Weiqun Zhang Date: Fri Dec 10 09:47:11 2021 -0800 Fix a bug in Robin BC (#2477) * Fix a bug in Robin BC When the a scalar is zero but A coefficient is not, there was a bug in the handling of Robin BC. We should reset the A coefficient to zero first before modifying it for Robin BC because we need to modify the a scalar to non-zero. Closes #2476 * Fix a bug in the detection of singularity. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp commit 4b695b7b3fe17962bb2645b0b37959adbf38e6ce Author: Erik Date: Fri Dec 10 10:54:25 2021 -0500 Doc refine grid layout (#2527) * CMake file for AmrLevel-SingleVortex * move new test to its own folder * message to indicate where build is coming from * Starting to convert Prob.f90 to cpp. * Convert Prob.f90 to Prob.cpp, runs. Testing next. * remove Adv_F.H from CMake file * Forgot to add the Prob.H file * delete unneeded ctest section * Switch AmrLevel for AmrCore b/c level not WIN32 compatible. * remove WIP comments * remove unneeded changes * Remove unneeded files. * fix words and remove comments * trying to add CUDA to Cmake WIP * some changes to the CMake file, nothing concrete * Add CUDA to CMake file. * add refine_grid_layout to sphinx docs * more add refine_grid_layout to sphinx docs Docs/sphinx_documentation/source/InputsLoadBalancing.rst commit 8d5e756a5a58b1ce08ea4ff69ad8d8e2802e8eb9 Author: Weiqun Zhang Date: Thu Dec 9 16:37:00 2021 -0800 Add ParallelCopyToGhost and fix a bug in FillPatch (#2523) * Add ParallelCopyToGhost function that copies data to ghost cells only. * There was a bug in the Array version of FillPatch for Face data. The issue is, if the destination is the same FabArray as one of the source FabArrays on the fine level, some valid data on valid box faces are overwritten with the interpolated values. This is now fixed with the use of ParallelCopyToGhost in that case. Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H commit fe08a6914b069739332405eec45a8ca79be57ac0 Author: Axel Huebl Date: Thu Dec 9 16:36:32 2021 -0800 CMake 3.22+: Policy CMP0127 (#2525) Fix a warning with CMake 3.22+. ~~We use use sometimes ;-connected AND syntax in `cmake_dependent_option`, so we need to update some of those options once we require CMake 3.22+ only.~~ We use simple syntax in `cmake_dependent_option`, so we are compatible with the extended syntax in CMake 3.22+ https://cmake.org/cmake/help/v3.22/policy/CMP0127.html First seen on Perlmutter. CMakeLists.txt commit 6b3937d41be4756291747fca2f90cdd50c5afe43 Author: Andrew Myers Date: Thu Dec 9 19:35:45 2021 -0500 Add function for make a dmap that is similar to an input MultiFab. (#2526) This was originally a PR to WarpX, but it makes more sense for it to live in AMReX: https://github.com/ECP-WarpX/WarpX/pull/2640 Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp commit fba4eba1e10f87fc2113d476766e568800c81815 Author: Erik Date: Thu Dec 9 12:16:23 2021 -0500 Specify dimensions that can be refined with `refine_grid_layout` (#2516) Adds a pp variable that can be included in the input file to specify which dimensions can be refined with `refine_grid_layout`. Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrMesh.cpp commit e88a72a7c8572ab94a078ddf6736d840bc1dbc6c Author: darylbond Date: Fri Dec 10 02:02:39 2021 +1000 avoid divide by zero in EB (#2519) Add a small number to the variable `apnorm` in the 2D EB code to avoid a divide by zero. Src/EB/AMReX_EB2_2D_C.cpp commit d52e45350f77bb70f051913f794cc205a3c76e97 Author: Luca Fedeli Date: Thu Dec 9 17:01:55 2021 +0100 Fix issue concerning Fujitsu compiler in AMReX_BLBackTrace.H (#2517) The line `#pragma omp threadprivate(bt_stack)` in `AMReX_BLBackTrace.H` can't be compiled with Fujitsu compiler (as well as with Intel compiler and Cray compiler). This PR excludes this line when the Fujitsu compiler is used. Src/Base/AMReX_BLBackTrace.H commit d2bdb9482d43b029da86203c2e6d1580b2f9c742 Author: Erik Date: Thu Dec 9 10:22:27 2021 -0500 Interpolater doxygen (#2524) Co-authored-by: atmyers Src/Particle/AMReX_ParticleInterpolators.H commit 3a7a24d05a94545596cf706a188cf5983f4d0975 Author: Weiqun Zhang Date: Wed Dec 8 15:44:37 2021 -0800 DPCPP: AMREX_INTEL_ARCH (#2459) * DPCPP: AMREX_INTEL_ARCH When AOT is on, set the target to AMREX_INTEL_ARCH, which is required if the SYCL backend is used and AOT is on. * update documentation * Apply suggestions from code review Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake commit f39e091d160cd5c0dcbcf8dc7cc00e3b28e956af Author: Andrew Myers Date: Wed Dec 8 12:27:30 2021 -0500 Doxygen comments for the new interpolator classes (#2521) Src/Particle/AMReX_ParticleInterpolators.H commit acf8694f2b2676a9e0479f3245760663fe46dd3b Author: Andrew Myers Date: Wed Dec 8 11:56:18 2021 -0500 Fix ParticleMesh test after #2475 (#2520) Remove some inadvertent changes to the ParticleMesh test. Tests/Particles/ParticleMesh/main.cpp commit 6780dbf281570c9354ed742ca9c9c4545a0b65d4 Author: Ann Almgren Date: Tue Dec 7 17:05:31 2021 -0800 Generalize problem size specification (#2518) * Add the option to specify geometry.prob_extent instead of geometry.prob_lo and geometry.prob_hi * add documentation of prob_extent * setting prob_hi should be inside if test * update docs Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/InputsProblemDefinition.rst Src/Base/AMReX_Geometry.cpp commit 84c7864b66247fabdcd0e12665994ea2edd94759 Author: Axel Huebl Date: Tue Dec 7 09:37:43 2021 -0800 CI: Run 2D and 1D Tests (#2475) Co-authored-by: atmyers .github/workflows/gcc.yml Src/Particle/AMReX_ParticleInterpolators.H Src/Particle/CMakeLists.txt Src/Particle/Make.package Tests/AsyncOut/multifab/main.cpp Tests/LinearSolvers/NodalPoisson/CMakeLists.txt Tests/MultiBlock/Advection/CMakeLists.txt Tests/MultiBlock/Advection/main.cpp Tests/MultiBlock/IndexType/CMakeLists.txt Tests/MultiBlock/IndexType/main.cpp Tests/Particles/AssignDensity/main.cpp Tests/Particles/AsyncIO/main.cpp Tests/Particles/GhostsAndVirtuals/CMakeLists.txt Tests/Particles/InitFromAscii/CMakeLists.txt Tests/Particles/NeighborParticles/CheckPair.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleIterator/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/main.cpp Tests/Particles/ParticleMeshMultiLevel/trilinear_deposition_K.H Tests/Particles/ParticleReduce/main.cpp Tests/Particles/ParticleTransformations/main.cpp commit 60fe729fe2ba65ebffc88c0af18743c254d3992c Author: Jean M. Sexton Date: Fri Dec 3 15:52:18 2021 -0800 Use specific targets for make uninstall (#2471) This updates the `make uninstall` rule which is used with configure to more specifically remove targets which were installed. It also adds a dependency file removal for the `make cleanconfig` rule. This removes the assumption that the directory set from `./configure --prefix=/my/example/directory` is empty, and therefore all files should be removed with `make uninstall` Co-authored-by: Weiqun Zhang GNUmakefile.in Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules commit e96ffdead46310595dfa491b1e42ae208017de1d Author: Donald E. Willcox Date: Fri Dec 3 14:58:21 2021 -0800 Move Basic Time Integration into AMReX (#2503) This PR moves the latest versions of a basic general explicit time integrator into AMReX. We are already using this time integrator for the Emu (https://github.com/AMReX-Astro/Emu), ET-Integration (https://github.com/amrex-codes/ET-integration), and ERF (https://github.com/erf-model/ERF) codes and this will let us consolidate the code and let other codes can benefit from it. The time integrator includes both forward Euler and several explicit Runge-Kutta methods and example code with input parameter explanations are supplied in a new documentation webpage labeled "Time Integration." Co-authored-by: Jean M. Sexton Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/TimeIntegration_Chapter.rst Docs/sphinx_documentation/source/index.rst Src/Base/AMReX_FEIntegrator.H Src/Base/AMReX_IntegratorBase.H Src/Base/AMReX_RKIntegrator.H Src/Base/AMReX_TimeIntegrator.H Src/Base/CMakeLists.txt Src/Base/Make.package commit b6d3d1cced21648126e9465ad88ec3f6cd8b093d Author: Andrew Myers Date: Fri Dec 3 17:44:17 2021 -0500 allow the use of SuperParticle in amrex::ParticleToMesh (#2514) Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleUtil.H Tests/Particles/ParticleMesh/main.cpp commit f5c1f0371aa7ec2c53053907f856e0b64a6bc623 Author: Nathan X. Marshak <5407450+n8xm@users.noreply.github.com> Date: Fri Dec 3 15:04:01 2021 -0700 Workaround to fix amrex_particles_to_vtp.py samplefiles (#2508) ## Summary The sample input files for `amrex_paticles_to_vtp.py` were modified in order to stop the script from crashing. Based on comparison against output from the [Electrostatic PIC tutorial](https://amrex-codes.github.io/amrex/tutorials_html/Particles_Tutorial.html#electrostaticpic), I believe that the sample input files are in an outdated format. ([electrostatic_pic_particles50000.txt](https://github.com/AMReX-Codes/amrex/files/7637608/electrostatic_pic_particles50000.txt) is attached for reference.) ## Additional background `amrex_particles_to_vtp.py` skips the first four lines after the first line (containing the number of particles) of the input file. Then, the number of lines that are read is equal to exactly the number of particles specified on the first line. The sample input files under the `samplefiles/` directory do not contain these four "skipped" lines, and as a result, not only does `amrex_particles_to_vtp.py` skip four lines of valid particle data, the script crashes because it tries to read past the end of the file. (In other words, the script assumes that the file is four lines longer than it really is.) As a workaround, I've added four "dummy" lines to each file. This is NOT an ideal solution because real particle output would not contain my four "dummy" lines, and instead there would be some other information there (i.e. I believe the number of extra real and int components in the particle data/attributes). However, my workaround is adequate for the purpose of demonstrating correct usage of `amrex_particles_to_vtp.py`. I have confirmed that the resulting `.vtp` files produce sensible-looking results in ParaView. (See attached screenshots below for reference.) `particles00000`: ![particles00000](https://user-images.githubusercontent.com/5407450/144328022-71b847a8-d78a-4837-81dc-ea8d1cafd682.gif) `particles00001`: ![particles00001](https://user-images.githubusercontent.com/5407450/144328021-34077dda-fe6b-40b0-8849-4265bb4539f9.gif) `particles00002`: ![particles00002](https://user-images.githubusercontent.com/5407450/144328017-504b1158-b38c-49e9-924b-957ecc9a0fbd.gif) Co-authored-by: Nathan Marshak Tools/Py_util/amrex_particles_to_vtp/samplefiles/particles00000 Tools/Py_util/amrex_particles_to_vtp/samplefiles/particles00001 Tools/Py_util/amrex_particles_to_vtp/samplefiles/particles00002 commit 71b6368a8f3d807ce0c70d38c7ef3791bd5c431d Author: Andrew Myers Date: Fri Dec 3 15:21:36 2021 -0500 Fix AppleClang CI by ignoring unused variables in GccPlacater (#2513) Src/Base/AMReX.H Src/Base/AMReX_MultiFab.H commit e2af96c28d4e50ecac8476cbb82f048f8e82efe5 Author: Weiqun Zhang Date: Fri Dec 3 08:08:29 2021 -0800 ANSI Escape Codes (#2509) Add commonly used ANSI escape codes to AMReX_ANSIEscCode.H. They can be used to print colored output. For example, amrex::Print() << Font::Bold << FGColor::Red << "Hello world from AMReX\n" << ResetDisplay << Font::Faint << FGColor::BrightCyan << Font::Underline << "Hello world again from AMReX " << ResetDisplay << BGColor::White << FGColor::Black << amrex::Version() << ResetDisplay << std::endl; Src/Base/AMReX_ANSIEscCode.H Src/Base/AMReX_Print.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 2c307a8276ff0ad5f7636544ac7c191ffd6f6365 Author: Andrew Myers Date: Thu Dec 2 13:25:26 2021 -0500 Fix misleading documentation for ParticleTile. (#2500) Co-authored-by: etpalmer63 Docs/sphinx_documentation/source/Particle.rst commit 6396050dde3f4fb08ef65173a52350c7b33adca9 Author: Weiqun Zhang Date: Wed Dec 1 10:22:25 2021 -0800 Update CHANGES for 21.12 (#2507) CHANGES commit ef9a37d6a14a704d59f105945c203da717595cb5 Author: Weiqun Zhang Date: Tue Nov 30 18:51:24 2021 -0800 AMREX_RELEASE_NUMBER (#2448) Add a new macro `AMREX_RELEASE_NUMBER`. It is a six digits number. The first two are for the year, and the next two are for the month. The last two are usually 00, unless a patch other than the monthly release is released. If the AMReX source is a git version that's not the release version, the last release number will be used. For git version obtained with git --describe, we define something like #define AMREX_GIT_VERSION "21.10-65-gfbb2edc95551-dirty" #define AMREX_RELEASE_NUMBER 211000 For git version obtained from the CHANGES file, we define something like #define AMREX_GIT_VERSION "21.10" #define AMREX_RELEASE_NUMBER 211000 or #define AMREX_GIT_VERSION "21.10.1" #define AMREX_RELEASE_NUMBER 2110001 or #define AMREX_GIT_VERSION "21.10.11" #define AMREX_RELEASE_NUMBER 211011 Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXUtils.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs commit 1d50bfc1c1da79614c853f9823efdf33f9003cc5 Author: Andrew Myers Date: Tue Nov 30 16:53:45 2021 -0500 Add a function for detecting whether amrex::Initialize has been called or not. (#2501) Src/Base/AMReX.H Src/Base/AMReX.cpp commit 062a07100e364b25d29a966092fd9e9200d4f2b0 Author: Weiqun Zhang Date: Mon Nov 29 18:17:47 2021 -0800 Fix fcompare's --abs_tol (#2504) There is a bug that ignores the absolute errors of components that do not have the biggest relative error when --abs_tol is used. It results in something like below fcompare.gnu.ex --abs_tol 1.e-12 plt00010 plt00002 variable name absolute error relative error (||A - B||) (||A - B||/||A||) ---------------------------------------------------------------------------- level = 0 density 0 0 pressure 0 0 theta 0 0 x_velocity 4.39990436e-05 2.749038203e-06 y_velocity 4.818762188e-05 9.964503154e-06 z_velocity 4.70120484e-19 0.7986458289 PLOTFILE AGREE to specified tolerances: absolute = 1e-12 relative = 0 In this commit, this is fixed by applying the test to each component separately. A component passes if its relative error or absolute error satisfies the respective tolerance. The whole comparison passes if all components pass. Tools/Plotfile/fcompare.cpp commit d62f0d27c907324065fe4968b6dfbbb98d41fd64 Author: Andrew Myers Date: Mon Nov 29 14:13:58 2021 -0500 Fix HIP CI (#2502) Something was going wrong with the "computed default" cmake argument on this action, so this removes it. .github/workflows/hip.yml commit d32658dbbccf1e5abd82cb971bb9fec8fd87eeb1 Author: Houjun Tang Date: Mon Nov 29 07:08:22 2021 -0800 Fix an issue when some MPI ranks have no box at all (#2494) Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Tests/HDF5Benchmark/main.cpp commit 591c7856b7c876ec9f1f3a5db56ce652059c7c8b Author: Weiqun Zhang Date: Wed Nov 24 07:25:57 2021 -0800 Signed distance function in 2D (#2490) Function for computing signed distance function in 2D. Src/Base/AMReX_Scan.H Src/EB/AMReX_EB_utils.H Src/EB/AMReX_EB_utils.cpp commit 30185629176596663f9d2ee21bfd97ae4d9546b2 Author: hengjiew <86926839+hengjiew@users.noreply.github.com> Date: Mon Nov 22 14:20:15 2021 -0600 Reduce ghost particles in particle communication. (#2480) * Remove duplicated particle communication when boxes are not perfectly aligned * Add a function to select the ghost particles that are neighbors to the real particles And allow `updateNeighbors()` to only communicate these ghost neighbor particles. These two steps significant reduce the particle communication time when particles are dense. * Fix CI issues. * Fix coding style issues. Co-authored-by: Hengjie Wang Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H commit 29584112a13ab086dc64edfdc99a217431ed8db7 Author: Andrew Myers Date: Sun Nov 21 10:43:12 2021 -0800 Assign initial value to prevent (likely spurious) -Wmaybe_unitialized compiler warning. (#2484) Src/Particle/AMReX_Particle.H commit 80156569892316cf413b072c3bc4a63abff28309 Author: Weiqun Zhang Date: Sun Nov 21 10:42:43 2021 -0800 Fix a corner case in fixing small cells. (#2489) During the iteration of fixing small cells, the edge types and level sets could become inconsistent if the fixed small cells are too close to periodic boundaries. The function that updates the edge types did not take that into account. This commit fixes that. Src/EB/AMReX_EB2_GeometryShop.H commit 96ba8b11befcd5361fdf70b58a9fb6e6eb5776ca Author: Candace Gilet Date: Wed Nov 17 18:39:52 2021 -0500 Fix out-of-date comment. (#2483) Src/Base/AMReX_ParmParse.H commit bf86e02eb5048c201b7014c4efa567ed3f49a1d8 Author: Erik Date: Wed Nov 17 18:39:19 2021 -0500 Remove HeatEqn Tutorial and link to new location in amrex-tutorials (#2482) Remove the Heat Equation Tutorial from the AMReX user's guide. Add link to AMReX tutorials in its place. Note, the tutorial already exists in amrex-tutorials. Docs/sphinx_documentation/source/Basics.rst commit 62774a08663f6552a5d6265200f6d52e18df824b Author: Erik Date: Wed Nov 17 18:37:48 2021 -0500 Combine debugging sections into a single chapter. (#2481) This change combines the debugging and Gpu debugging sections into a single debugging chapter. Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/Debugging.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/index.rst commit a2db70cb3cb5e74ee692ed1b568727dd66aa145d Author: David Grote Date: Wed Nov 17 15:35:50 2021 -0800 Added special case for Mac when using rpath (#2479) Tools/GNUMake/Make.defs commit 88b868099018c191e7d3ea424d3e3420b826dd69 Author: Weiqun Zhang Date: Fri Nov 12 12:42:17 2021 -0800 Fix Advection_AmrLevel test (#2474) There is no need to use pinned memory in initData anymore after #2433. Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp commit 6ff068d5d04218520fb1408b01ea734b5bca40e6 Author: Erik Date: Fri Nov 12 11:19:48 2021 -0500 Advection_AmrLevel Test: Convert Prob.f90 to Prob.cpp (#2433) Converts the only Fortran file in the AmrLevel test to cpp. Tests/Amr/Advection_AmrLevel/CMakeLists.txt Tests/Amr/Advection_AmrLevel/Exec/Make.Adv Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Make.package Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.cpp Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Make.package Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.cpp Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.f90 Tests/Amr/Advection_AmrLevel/README Tests/Amr/Advection_AmrLevel/Source/Adv_F.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Amr/Advection_AmrLevel/Source/Make.package commit 37d1af793b1ec9c20d4d9ba1c176b5936c3e01a3 Author: Weiqun Zhang Date: Thu Nov 11 16:53:31 2021 -0800 Fix multi-level EB nodal solver solvability fix (#2473) For the multi-level EB nodal solver with a singular matrix, we restrict the fine level RHS to the coarse level, and fix the coarse level's solvability by making the total sum of RHS is zero. The restriction is done with the stencil instead of simply geometric restriction because we need to take the EB into account. However, the previous way is incorrect if the sigma coefficient is not a constant, because the stencil includes sigma. But the restriction should be based on EB information only. In this commit, we fix this by constructing a special constant sigma stencil. We use this new stencil for the restriction of RHS, and for consistence the residual as well. Because we used to use the original stencil for the restriction of residual, this commit will change the answer at the tolerance level for multi-level problems. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp commit a2245756e64c1f414f35b64330fec99a9b87df0c Author: Weiqun Zhang Date: Mon Nov 8 12:36:09 2021 -0800 Disable warning on atomicAddNoRet in HIP (#2466) It seems that we may continue using the deprecated atomicAddNoRet in the near future. This disables the compiler warning. Src/Base/AMReX_GpuAtomic.H commit 798240684d9696fb00392688e3c216e21596b921 Author: Weiqun Zhang Date: Mon Nov 8 08:38:53 2021 -0800 DPCPP: workaround for the scan issue (#2462) A multi-pass algorithm is implemented. Extensive testing on early hardware shows it works for int, but there are still issues with long. Src/Base/AMReX_Scan.H commit c49b1834aa488e123cb379681577a8cf65fed68e Author: Weiqun Zhang Date: Mon Nov 8 08:28:56 2021 -0800 DPCPP: AOT in GNU Make (#2463) If INTEL_ARCH or AMREX_INTEL_ARCH is defined, use it to set target when AOT is on. Tools/GNUMake/comps/dpcpp.mak commit 3d29cc3234a319f5721da38635f37b3570da43c7 Author: Jean Luca Bez Date: Sun Nov 7 14:17:56 2021 -0800 Include optional directory to HDF5 Test Benchmark (#2461) Tests/HDF5Benchmark/inputs Tests/HDF5Benchmark/main.cpp commit 2fc0e4446aa466247de8b7efae11903cfd3b5c37 Author: Pastuzo <40230415+DUGKS@users.noreply.github.com> Date: Sun Nov 7 02:17:17 2021 +0800 Implement HOEXTRAPCC for more than single ghost cell (#2456) Implement HOEXTRAPCC for N ghost cells, N > 1. Previous HOEXTRAPCC works only for N = 1. Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_FilCC_C.cpp Src/Base/AMReX_filcc_mod.F90 commit d4669a1ce4c7abb8c15af7829b6d4f56075589a9 Author: Erik Date: Fri Nov 5 17:00:06 2021 -0400 Spack Smoke Test (#2460) Tests/SpackSmokeTest/CMakeLists.txt commit 39332fbb07478fe685c9917006df0fef1957280b Author: Weiqun Zhang Date: Fri Nov 5 10:19:11 2021 -0700 Support fmod in Parser (#2395) * Support fmod in Parser * Sneak in some minor changes Docs/sphinx_documentation/source/AmrCore.rst Docs/sphinx_documentation/source/Basics.rst Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp commit b0b7900500bc9879aefd8b649caf0346976bea50 Author: Weiqun Zhang Date: Wed Nov 3 16:25:11 2021 -0700 Fix a bug in BoxArray for SyncRegister (#2458) A bug was introduced recently in #2452. I forgot that code was used by SyncRegister in IAMR. Src/Base/AMReX_BoxArray.H commit 8023fd148ee0f23ae83127b8d0515c1323c194de Author: Scot Breitenfeld Date: Wed Nov 3 12:16:53 2021 -0500 Redo of PR #2450 (#2457) Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H commit 2ec7459dd689d1425ce45a5977f26ae7e3c9facb Author: Jean M. Sexton Date: Tue Nov 2 16:04:36 2021 -0700 HIP: ftrapv causes issues with DEBUG linking (#2455) Tools/GNUMake/comps/hip.mak commit d491ebe1c1aca7dc53558e64af24526888b79abe Author: Weiqun Zhang Date: Tue Nov 2 15:04:14 2021 -0700 DPCPP: Enable early optimizations (#2454) Early optimizations were disabled after beta9 was released to work around a compiler bug. The issue has been resolved. So we enable it again. Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit 7555541ab3c43f2c46b26b25e42a40d73fdf3656 Author: Houjun Tang Date: Tue Nov 2 14:15:07 2021 -0700 New HDF5 schema with separate datasets for different variables (#2432) Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_PlotFileUtil.cpp Src/CMakeLists.txt Src/Extern/HDF5/AMReX_ParticleHDF5.H Src/Extern/HDF5/AMReX_ParticleUtilHDF5.H Src/Extern/HDF5/AMReX_ParticlesHDF5.H Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.H Src/Extern/HDF5/AMReX_PlotFileUtilHDF5.cpp Src/Extern/HDF5/AMReX_WriteBinaryParticleDataHDF5.H Src/Extern/HDF5/CMakeLists.txt Src/Extern/HDF5/Make.package Src/Particle/AMReX_ParticleUtil.cpp Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H Src/Particle/CMakeLists.txt Src/Particle/Make.package Tests/HDF5Benchmark/GNUmakefile Tests/HDF5Benchmark/inputs Tests/HDF5Benchmark/main.cpp Tools/CMake/AMReXConfig.cmake.in Tools/GNUMake/packages/Make.hdf5 commit aca6926e0a85a0fad4087bd47016edad0885252f Author: Weiqun Zhang Date: Mon Nov 1 15:15:35 2021 -0700 InterpFaceRegister (#2452) This is a new class for interpolating from coarse to fine at the coarse/fine boundary faces. It also provides masks that can be used to test if a face is at coarse/fine boundary. Src/AmrCore/AMReX_AmrCoreFwd.H Src/AmrCore/AMReX_InterpFaceReg_1D_C.H Src/AmrCore/AMReX_InterpFaceReg_2D_C.H Src/AmrCore/AMReX_InterpFaceReg_3D_C.H Src/AmrCore/AMReX_InterpFaceReg_C.H Src/AmrCore/AMReX_InterpFaceRegister.H Src/AmrCore/AMReX_InterpFaceRegister.cpp Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package Src/Base/AMReX_BoxArray.H commit 3add1f2824c5a4f33e77abc612da8a52b3f5131b Author: Weiqun Zhang Date: Mon Nov 1 13:59:36 2021 -0700 Fix WarpX EB solver's grad phi computation (#2453) Must set the Dirichlet mask on covered nodes to a negative value, because the function that computes the gradient relies on the information to detect EB. This is a bug introduced recently in #2421. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 500ab4fb3ea7a755e0e0b29c7c021495fc0707b5 Author: Weiqun Zhang Date: Mon Nov 1 10:03:45 2021 -0700 Update CHANGES for 21.11 (#2451) CHANGES commit 2cfe8354f6c6a00c7a6bb2bcefd822d59fd65518 Author: Scot Breitenfeld Date: Sat Oct 30 00:05:44 2021 -0500 Set to never write fill values to the dataset for HDF5 output (#2450) Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_WriteBinaryParticleData.H commit 6e9788139d8e1401ed304fc56e526a1bdb315b81 Author: Weiqun Zhang Date: Thu Oct 28 13:36:13 2021 -0700 Fix a new bug in GNU Make (#2447) CXX should not be set to mpiicpc if it's not an MPI build. This is a new bug introduced by me in #2435. Tools/GNUMake/sites/Make.unknown commit 210cb0e7d77d1334d009246a4cb403ae463df1dc Author: Weiqun Zhang Date: Thu Oct 28 12:54:03 2021 -0700 Lower Hypre minimum version (#2446) Lower Hypre minimum version to 2.20. This will allow users to use the provided installation on summit. The ILU smoother that requires 2.21 is still available for hypre >= 2.21. Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXThirdPartyLibraries.cmake commit 6c30ddf8fb0d1a4373b013517d43734ec4110385 Author: Axel Huebl Date: Thu Oct 28 11:28:13 2021 -0700 ROCm 4.5: Dev Packages (#2445) * ROCm 4.5: Devel Packages In ROCm 4.5, libraries were split into binary and `-devel` packages, as often done in Debian/Ubuntu packaging. We need the `-devel` packages for headers. Release notes: https://rocmdocs.amd.com/en/latest/Current_Release_Notes/Current-Release-Notes.html * Robust: Try while rollout is ongoing * What if the docs lie? Try `-dev` * Clean Up .github/workflows/dependencies/dependencies_hip.sh commit 138503ba8476c977de97628bc5bdbdd5fbb00356 Author: Weiqun Zhang Date: Thu Oct 28 08:44:46 2021 -0700 Fix use-after-free (#2443) The parallel copy meta-data cache is store in std::multimap. A CPC* might be stored multiple times in the multimap, and we need to delete the pointers before clearing the multimap. The previous way of avoiding double free was incorrect, because it had a use-after-free. In this commit, we fix it by pushing a unique copy of the pointers to a vector first while iterating the multimap. This bug only manifests if a user's code has a memory leak of MultiFab/FabArray. Src/Base/AMReX_FabArrayBase.cpp commit 08a068a638a124c7dd9e25f28a5ac02ec59fe4ae Author: Jean M. Sexton Date: Wed Oct 27 15:04:42 2021 -0700 Update MPI config checking to include Intel MPI wrappers for DPCPP (#2435) * Update MPI config checking to assume first Intel MPI wrappers, and then others * move the mpicpc check out of the usual MPI check Co-authored-by: Weiqun Zhang Tools/GNUMake/Make.defs Tools/GNUMake/sites/Make.unknown commit 8c8004c07a24dc1e20e5ac6c8e6bc07fdb0ef5a7 Author: Andrew Myers Date: Wed Oct 27 13:53:17 2021 -0700 Mark NeighborParticles as a 'selfTest' (#2442) Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini commit 68e2913f4783d3dc5fe9bc140bbaca06ff8c07d9 Author: Weiqun Zhang Date: Wed Oct 27 12:14:41 2021 -0700 GNU Make: Fix rpath when LIBRARY_LOCATION is empty. (#2441) Because of #2439, empty LIBRARY_LOCATION would result in -Xlinker -rpath='' We would like to avoid that. Tools/GNUMake/Make.defs commit 58d04c6195c0f2c47119315b37d5e99f69fe0898 Author: Weiqun Zhang Date: Wed Oct 27 12:14:09 2021 -0700 Add EBCellFlagFab::getNumRegularCells, getNumCutCells and getNumCoveredCells (#2422) Give a Box, these functions will return the number of regular, cut and covered cells, respectively. Internally, the results are cached. Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBCellFlag.cpp commit 4fa06516928ad07736cb537ddb58d8c4ecba5a4e Author: Neïl Zaim <49716072+NeilZaim@users.noreply.github.com> Date: Wed Oct 27 20:51:17 2021 +0200 ExclusiveSum on GPU: return 0 when size of pointer is 0 (#2440) Src/Base/AMReX_Scan.H commit 94b4764f098603c51bda652d78ea0dff8bee3635 Author: Weiqun Zhang Date: Wed Oct 27 10:40:56 2021 -0700 Multi-component overset solver (#2429) Modify cell-centered overset solver's constructor to allow for multiple components. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp commit 67e9aadcace3572ca840d8d276d67501663ead8b Author: Weiqun Zhang Date: Wed Oct 27 10:38:30 2021 -0700 Use -rpath=dir instead of -rpath dir (#2439) The latest dpcpp compiler does not like the latter. Tools/GNUMake/Make.defs commit 0a98df6746fcdbb51f133e1905b08d9ae75e99e9 Author: Weiqun Zhang Date: Tue Oct 26 21:13:49 2021 -0700 DPCPP: Use -g1 for non-debug build (#2438) Because `-g` makes link extremely slow, we have disabled that in GNU make since oneAPI beta6. It turns out we could still use `-g1` to get some basic debug information without slowing down link too much. Tools/GNUMake/comps/dpcpp.mak commit ce057cf7c225badbbae7f964e81f1e67745238a4 Author: Weiqun Zhang Date: Tue Oct 26 17:31:49 2021 -0700 DPCPP: Fix warning on tautological-constant-compare (#2436) Disable warning on tautological constant compare that has appeared since 2021.04. .github/workflows/intel.yml Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/comps/dpcpp.mak commit 6b178313793544957f7d0c23ae7999cacead69bd Author: Weiqun Zhang Date: Tue Oct 26 17:24:03 2021 -0700 Update Tools/Postprocessing/C_Src (#2437) Tools/Postprocessing/C_Src/GNUmakefile Tools/Postprocessing/C_Src/HorizontalAvg.cpp Tools/Postprocessing/C_Src/PtwisePltTransform.cpp Tools/Postprocessing/C_Src/particle_compare.cpp commit 39da1abd86b46c64e098c43c3123fcff6c58742d Author: Axel Huebl Date: Mon Oct 25 18:02:00 2021 -0700 Tests: Safe-Guard Module Dirs (#2434) Make sure mod_file dirs are per executable. Avoid potential race conditions if people use the `setup_test` in loops in funny ways. Tests/CMakeLists.txt commit af8429793bc50cf7d5914104c05ed825dd0948c8 Author: Weiqun Zhang Date: Mon Oct 25 10:11:33 2021 -0700 Remove deprecated BaseUmap (#2425) Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BaseUmap_f.H Src/Base/AMReX_BaseUmap_nd.f90 Src/Base/Make.package commit 652af10ae992e2cf5c1812e0908d9b88ac22126c Author: Weiqun Zhang Date: Mon Oct 25 10:04:32 2021 -0700 Update WarpX EB solver (#2421) Change the coarsening strategy for much better performance. Src/EB/AMReX_EB2_2D_C.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 614b9ebbbf04eb5b4532029ff4a7ae40e6663042 Author: Weiqun Zhang Date: Fri Oct 22 14:59:39 2021 -0700 MLNodeLaplacian: Fix max coarsening level for enclosed EB (#2431) Where the domain is fully enclosed by EB, we need to make sure that it does not coarsen too much that there are no valid nodes left. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp commit 6562da035155d88663c6c98de5f9575c750a3be9 Author: Andrew Myers Date: Fri Oct 22 12:07:49 2021 -0700 Fix two IO overflow bugs uncovered by NyX. (#2417) Co-authored-by: Jean M. Sexton Src/Base/AMReX_FabConv.cpp Src/Particle/AMReX_WriteBinaryParticleData.H commit a707944d287a320e184768d85b5bb05b020f5545 Author: Weiqun Zhang Date: Fri Oct 22 12:03:19 2021 -0700 Cleanup of amrdata (#2430) * Avoid using std and amrex in amrdata headers. * Fix some warnings. Src/Extern/amrdata/AMReX_AmrData.H Src/Extern/amrdata/AMReX_AmrData.cpp Src/Extern/amrdata/AMReX_DataServices.H Src/Extern/amrdata/AMReX_DataServices.cpp Src/Extern/amrdata/AMReX_WritePlotFile.cpp Src/Extern/amrdata/AMReX_XYPlotDataList.H Src/Extern/amrdata/AMReX_XYPlotDataList.cpp commit 2d02c17651b81c3bb7d90a45c81265fdb1706c1b Author: Erik Date: Thu Oct 21 19:30:51 2021 -0400 Set options to allow other Doxygen docs to link to AMReX. (#2424) * Set options to allow other Doxygen docs to link to AMReX. * move amrex xml tag file to folder for uploading to webpage. Docs/Doxygen/doxygen.conf build_docs.sh commit f75e129df118cf448bc763d9eafd67c3cbb74825 Author: Weiqun Zhang Date: Thu Oct 21 15:57:51 2021 -0700 Remove DArena and EArena (#2426) DArena that uses the buddy memory allocation strategy and EArena that uses the best fit strategy are experimental features we explored in the past. But they have never been used because they do not work as well as the first bit based CArena. There is no reason to keep them anymore. Src/Base/AMReX_Arena.cpp Src/Base/AMReX_DArena.H Src/Base/AMReX_DArena.cpp Src/Base/AMReX_EArena.H Src/Base/AMReX_EArena.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 8a1a3023b6f0bbc715935fcabbcddbfe9ff95861 Author: Andrew Myers Date: Thu Oct 21 15:55:51 2021 -0700 Add some docstrings for the particle tile access functions. (#2423) Src/Particle/AMReX_Particles.H commit 7c88a2c139eada6601401ac23e6f0b778fab9a9c Author: Weiqun Zhang Date: Thu Oct 21 14:20:57 2021 -0700 Update MLNodeTensorLap for WarpX (#2416) Change the stencil of the linear operator so that L=D*G. This could make the solution satisfy div E = rho up to solver tolerance. Previously, the stencil was similar to that of the nodal solver for approximate projection Src/EB/AMReX_EB2_2D_C.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Tests/LinearSolvers/NodeTensorLap/MyTest.cpp commit db792398f6da8158026527759d97b5b882d1f22b Author: Andrew Myers Date: Wed Oct 20 12:30:36 2021 -0700 Restore RedistributeLocal() call to InitParticles in redistribute test (removed in #2414) (#2420) Tests/Particles/Redistribute/main.cpp commit cdc1052e9d5f518aa864c501a410ed6fbae56afe Author: Andrew Myers Date: Tue Oct 19 12:20:38 2021 -0700 Revert changes to Redistribute test (follow on to #2414) (#2415) Tests/Particles/Redistribute/inputs Tests/Particles/Redistribute/inputs.rt.cuda Tests/Particles/Redistribute/main.cpp commit a9d758ab6dc019203ae467c185553f3dc99b4e8c Author: Andrew Myers Date: Tue Oct 19 10:51:16 2021 -0700 Reduce peak memory usage when sorting particles. (#2414) Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Tests/Particles/Redistribute/inputs Tests/Particles/Redistribute/inputs.rt.cuda Tests/Particles/Redistribute/main.cpp commit 130f7304328ed1c26a72cea24dd3e96bc9426a58 Author: Candace Gilet Date: Tue Oct 19 12:24:45 2021 -0400 Add link to 'small cell problem' section that got moved to (#2412) Add link to 'small cell problem' section that got moved to AMReX-Hydro's documentation. Also add more details on EB data. Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/EB_Chapter.rst Docs/sphinx_documentation/source/conf.py commit ff747d901ea2127ed01d6053f1d80e9a45bcd381 Author: Roelof Groenewald <40245517+roelof-groenewald@users.noreply.github.com> Date: Tue Oct 19 09:23:26 2021 -0700 Docs: MLMG solver convergence criteria (#2413) expanded the documentation with regards to convergence criteria for the MLMG solver Docs/sphinx_documentation/source/LinearSolvers.rst commit 691d9bf3fb64343a3a929d12f1463fde165ec969 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Tue Oct 19 12:22:38 2021 -0400 Extend CellQuadratic Interpolater to 3D (#2409) Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterp_C.H commit 09a0ec56378f9f6a5f68f3ac025318af746432d5 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Mon Oct 18 11:22:55 2021 -0400 Update documentation on Interpolaters (#2408) Docs/sphinx_documentation/source/AmrCore.rst commit be5294eefb2017cb0c5273054207ea1cb6d77807 Author: Weiqun Zhang Date: Thu Oct 14 16:13:05 2021 -0700 Fix 2D bug in EB2::SplineIF (#2407) Src/EB/AMReX_EB2_IF_Spline.H commit 92b4e5bc5d00b1f8c11c0eedfeaeb99ebbe38bbe Author: Weiqun Zhang Date: Thu Oct 14 16:12:34 2021 -0700 Bump up Hypre minimum version to 2.21.0 (#2406) Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXThirdPartyLibraries.cmake commit fb72b08c1914a5b1f8529dcf101c059d613ee4a0 Author: Candace Gilet Date: Thu Oct 14 19:12:12 2021 -0400 Cleanup Tools/Postprocessing/C_src (#2398) Fix memory, parsing for help, and add add automatic determination of whether f90 sources need to be included. Tools/Postprocessing/C_Src/GNUmakefile Tools/Postprocessing/C_Src/IntegrateComp.cpp Tools/Postprocessing/C_Src/PlotfileToMatLab.cpp Tools/Postprocessing/C_Src/PlotfileToTurb.cpp Tools/Postprocessing/C_Src/PtwisePltTransform.cpp commit d5a3334ba58a15a74fdb55da0eabf643a7cd0f9d Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Thu Oct 14 19:11:22 2021 -0400 Port CellQuadratic to C++ with GPU support (#2369) ## Summary The CellQuadratic interpolater has been ported from Fortran to C++ with GPU support. The errors for the CPU-only version on local are in the ~1e-15 range for AmrLevel SingleVortex 2D, ~1e-15 for IAMR 2D bubble Cartesian, and ~1e-12 for IAMR 2D bubble RZ. The errors for the GPU-enabled version on Gigan are in the ~1e-14 range for AmrLevel SingleVortex2D, ~1e-10 for IAMR 2D bubble Cartesian, and ~1e-9 for IAMR 2D bubble RZ. ## Additional background This port is heavily influenced by the CellConservativeLinear interpolater. It reuses two of the same slope kernels. With this PR, _all_ of the interpolaters now work on the GPU! Also, thanks to @cgilet for helping with debugging. Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package commit d8e6ec544c25233483bf1dc0b5aa263a9a9df730 Author: Michael J. Brazell Date: Thu Oct 14 11:03:35 2021 -0700 added bamg option to set restriction type (#2402) 0 is P transpose and 1,2 are approximate ideal restriction AIR Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit 06b30c1eaff02952e9d656da01dd090ba5b5a540 Author: Stefan Vater Date: Thu Oct 14 19:59:57 2021 +0200 Bugfix/amr mesh constructor (#1155) These resizes must consider that the vectors in AmrInfo argument might be empty. Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_AmrMesh.cpp commit 5ae9ea7e67c347d81377ff722f12720a8c5377fc Author: Weiqun Zhang Date: Thu Oct 14 10:11:40 2021 -0700 Fix dot product mask in EB nodal solver (#2404) We should set the dot product mask on all Dirichlet nodes to zero so that they do not contribute to the sum used in fixing singular systems. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp commit a81dd4512658e05c220b358324cf0d67aef13e19 Author: Candace Gilet Date: Thu Oct 14 11:38:00 2021 -0400 Move Tools/C_util/Convergence/WritePlotFile to Src/Extern/amrdata. (#2397) This contains WritePlotFile functions that take an AmrData object as a parameter, and thus Src/Extern/amrdata is the most logical place to put it when codes in Tools/C_util/Convergence and Tools/Postprocessing/C_src use it. Src/Extern/amrdata/AMReX_WritePlotFile.H Src/Extern/amrdata/AMReX_WritePlotFile.cpp Src/Extern/amrdata/CMakeLists.txt Src/Extern/amrdata/Make.package Tools/C_util/Convergence/ComputeAmrDataNorms.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedFD.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp Tools/C_util/Convergence/DiffUniform.cpp Tools/C_util/Convergence/Make.package Tools/C_util/Convergence/PltFileNormB.cpp Tools/C_util/Convergence/RichardsonConvergenceTest.cpp Tools/Postprocessing/C_Src/PlotfileToTurb.cpp Tools/Postprocessing/C_Src/PtwisePltTransform.cpp commit 2ec5203f43050dcac8a8d8a7a99670e54ef0a87a Author: Candace Gilet Date: Thu Oct 14 11:35:27 2021 -0400 Move info on sharing the command line into the ParmParse section. (#2391) Co-authored-by: etpalmer63 Docs/sphinx_documentation/source/Basics.rst commit 92af41f8a966b252f260c0f3f32d41178844ac50 Author: Weiqun Zhang Date: Thu Oct 14 08:28:51 2021 -0700 Arena Document and PrintUsage (#2403) * Arena Document and PrintUsage * Document arena initial allocation size and release threshold. * Print the number of allocations in PrintUsage. * Update GPU.rst minor spellings Co-authored-by: Erik Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_CArena.cpp commit b60a773d6c87f708a736634f6281da16e094e4cb Author: Weiqun Zhang Date: Tue Oct 12 16:03:46 2021 -0700 BArena: Make a static BArena object (#2401) Instead of creating multiple BArena objects, make a static BArena object that is local to a new function The_BArena() in an anonymous namespace. The motivation for this is to properly release the memory allocated by BArena after amrex::Finalize() is called. This is follow-up to #2389. Src/Base/AMReX_Arena.cpp commit 247b41e17aa83f653d848e77f7269e9d43bf4f73 Author: Candace Gilet Date: Tue Oct 12 16:14:54 2021 -0400 Fix inaccurate comment about AmrLevel::writePlotFile, it is no longer pure virtual (#2399) Src/Amr/AMReX_AmrLevel.H commit 48a2776d78ec2a4e2909a875fc7eab25973a81f6 Author: Erik Date: Tue Oct 12 16:09:50 2021 -0400 fix dead link. small reword (#2400) Docs/sphinx_documentation/source/index.rst commit a162b7c15518b6bd5ceb7f8fefa574e6ee3c87d5 Author: Candace Gilet Date: Tue Oct 12 13:23:19 2021 -0400 Make fcompare print usage and exit if only one plotfile is provided. (#2388) Tools/Plotfile/fcompare.cpp commit 4914d5e42c2ee02c80645722b2d336e4dba6b4b3 Author: Weiqun Zhang Date: Mon Oct 11 18:58:13 2021 -0700 Fix compilation of Tools/Postprocessing/C_Src (#2396) Broken by #2390. Tools/Postprocessing/C_Src/Make.package commit 93788e33e689b77a14290580eca0cf9f25be6523 Author: Weiqun Zhang Date: Fri Oct 8 14:54:08 2021 -0700 Add Arena::PrintUsageToFiles for debugging (#2392) Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp commit b5cc0323c87d5caa9c3ed7dc9ac639cc0d3fb012 Author: Candace Gilet Date: Fri Oct 8 16:51:29 2021 -0400 Update c util (#2390) This PR cleans out old, unused code from amrex/Tools/C_util, and ensures that what's there compiles and runs. Tools/C_util/AmrDeriveTecplot/AmrDeriveTecplot.cpp Tools/C_util/AmrDeriveTecplot/GNUmakefile Tools/C_util/AppendToPlotFile.H Tools/C_util/AppendToPlotFile.cpp Tools/C_util/AugmentPlotfile/AugmentPlotfile.cpp Tools/C_util/AugmentPlotfile/GNUmakefile Tools/C_util/Convergence/Add.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/Convergence/DebugDump.H Tools/C_util/Convergence/DebugOut.H Tools/C_util/Convergence/DebugOut.cpp Tools/C_util/Convergence/DiffFab.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedFD.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp Tools/C_util/Convergence/DiffUniform.cpp Tools/C_util/Convergence/GNUmakefile Tools/C_util/Convergence/Make.package Tools/C_util/Convergence/PltFileNorm.cpp Tools/C_util/Convergence/PltFileNormB.cpp Tools/C_util/Convergence/PltFileScalConvRate.cpp Tools/C_util/Convergence/RichardsonConvergenceTest.cpp Tools/C_util/Convergence/WritePlotFile.H Tools/C_util/Convergence/WritePlotFile.cpp Tools/C_util/DiffMultiFab/diffmultifab.cpp Tools/C_util/Make.package Tools/C_util/README Tools/C_util/Statistics/AVGDOWN_2D.F Tools/C_util/Statistics/AVGDOWN_3D.F Tools/C_util/Statistics/AVGDOWN_F.H Tools/C_util/Statistics/ComputeAmrDataStat.H Tools/C_util/Statistics/ComputeAmrDataStat.cpp Tools/C_util/Statistics/GNUmakefile Tools/C_util/Statistics/Make.package Tools/C_util/Statistics/PltFileFluxAve.H Tools/C_util/Statistics/PltFileFluxAve.cpp Tools/C_util/Statistics/PltFileList.cpp Tools/C_util/Statistics/PltFileStat.cpp Tools/C_util/Statistics/PltFileXAve.H Tools/C_util/Statistics/PltFileXAve.cpp Tools/C_util/TV_TempWrite.H Tools/C_util/ViewMF/GNUmakefile Tools/C_util/ViewMF/MFNorm.H Tools/C_util/ViewMF/MFNorm.cpp Tools/C_util/ViewMF/Make.package Tools/C_util/ViewMF/checkMFghostcells.cpp Tools/C_util/ViewMF/main.cpp Tools/C_util/ViewMF/mfMinMax.cpp Tools/C_util/ViewMF/viewMF.cpp Tools/C_util/ViewMF/viewMFcol.cpp Tools/C_util/ViewMF/viewMFdiff.cpp Tools/C_util/ViewMF/viewMFdiffHardWire.cpp Tools/C_util/dbgTools/GNUmakefile Tools/C_util/dbgTools/Make.package Tools/C_util/dbgTools/crsGrids.cpp Tools/C_util/dbgTools/intersectGrids.cpp commit d35f4c70bd536425fe49acacb7194983802ea10e Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Fri Oct 8 09:37:32 2021 -0600 Adding ILU smoothing parameters for Hypre interface. (#2382) Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit f07f8be1206a4fb7106a24ed611cc5dfcfe5da16 Author: Weiqun Zhang Date: Thu Oct 7 18:33:25 2021 -0700 MLEBNodeFDLaplacian Coarsening (#2386) Limit the maximum coarsening level so that EB objects do not disappear due to coarsening. The solver would fail to converge if the coarsening causes EB objects to disappear entirely. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit b815c734bf01dd6844c55919ddedb4e9541f48ea Author: Weiqun Zhang Date: Thu Oct 7 16:55:50 2021 -0700 Avoid free after amrex::Finalize (#2389) Previously calling Arena::free after amrex::Finalize resulted in segfault because the Arena objects have been deleted. It will cause the following code to crash in the end. int main (int argc, char* argv[]) { amrex::Initialize(argc, argv); amrex::MultiFab(...); amrex::Finalize(); } One way to fix it is to add a scope to the function body between Initialize and Finalize so that the object inside will go out of scope before amrex::Finalize is called. In this commit, we implement a way of avoiding this kind of errors even if one forgets to free everything allocated by Arenas before Finalize. Src/Base/AMReX_Arena.cpp commit c64e719c0a9a5459d2349ef9ebe3a5b8b0e49b41 Author: Candace Gilet Date: Thu Oct 7 17:50:24 2021 -0400 Command line parsing updates. (#2387) Fix command line parsing inconsistency: allow argv[1] to be of the form "arg1 = my_arg", i.e. with spaces, since later arguments allow this. Also, if argv[1] starts with '-', do not use ParmParse. This will prevent "-h" or "--help" from creating errors within ParmParse. Application code can then parse the command line to handle these cases, if desired. Src/Base/AMReX.cpp commit 691e24555db99daa19deeea8da03f951f52a2578 Author: Andrew Myers Date: Wed Oct 6 16:35:44 2021 -0700 These assertions are not needed / wrong after PR #2376 (#2384) Src/Particle/AMReX_ParticleCommunication.H commit f368906eea12d8d68f7f240c17c0cafdfaf9cd1f Author: Andrew Myers Date: Wed Oct 6 15:27:30 2021 -0700 Workaround for nvcc bug that (sometimes) incorrectly complains about unused TinyProfiler variables. (#2383) Src/Base/AMReX_BLProfiler.H commit 3eb5c9edc9d1c62e2bb059e5304edf10559eae1b Author: Andrew Myers Date: Wed Oct 6 13:34:21 2021 -0700 Add inputs for redistribute test that uses a message type other than char. (#2381) Tests/Particles/Redistribute/inputs.rt.cuda.big commit ee975b2973a8e3dfaf5856ea2525832e118e0c98 Author: Neïl Zaim <49716072+NeilZaim@users.noreply.github.com> Date: Wed Oct 6 19:04:28 2021 +0200 Add floor and ceil to parser (#2380) I wanted to use the `floor` function in the parser for a WarpX automated test that I'm developing, so I've added `floor` and `ceil` in the parser. Hopefully I've done it right (I've tested compiling and using with WarpX on my computer and it looks like both functions are working well). Docs/sphinx_documentation/source/Basics.rst Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp commit f0a8fc785ff798cac6bdfe4913ad0fa5db988dfe Author: Axel Huebl Date: Tue Oct 5 10:16:10 2021 -0700 CI: Groups Include Branch (#2379) On pushes to mainline, if mainline contains multiple branches, these branches cancelled each other. This should fix this. .github/workflows/clang.yml .github/workflows/cuda.yml .github/workflows/docs.yml .github/workflows/gcc.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/macos.yml .github/workflows/sensei.yml .github/workflows/style.yml .github/workflows/windows.yml commit 64c9d92c0f4680fc4a0cc6fb48f98ee17771d5e4 Author: Andrew Myers Date: Fri Oct 1 19:58:08 2021 -0700 Fix bug in the particle communication routines when the number of chars in the message is > MAXINT. (#2376) I believe this bug was introduced in PR#1569. Src/Particle/AMReX_ParticleCommunication.H commit a7dd9da02211a88efb3c758ef7f8b806204798e1 Author: Erik Date: Fri Oct 1 19:18:13 2021 -0400 Add short decription for AMReX_TESTING flag (#2375) 1. Add the short description, "build for testing --sets MutliFab initial data to NaN" to the CMake build flags table. 2. Two other small wording changes. Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/BuildingAMReX.rst commit 91a9fae04bc2746bec48222731ac9198f3743798 Author: Weiqun Zhang Date: Fri Oct 1 15:31:33 2021 -0700 ROCm now supports __managed__ (#2377) Src/Base/AMReX_GpuQualifiers.H commit e52bccc4d9b95a791284a3e0c00fe27793cc15dd Author: Kevin Z. Zhu <86268612+KZhu-ME@users.noreply.github.com> Date: Fri Oct 1 14:30:50 2021 -0700 Fix read unitialized memory issue in MLMG solver (#2373) ## Summary Running valgrind on one of our failing circleCi tests revealed that the MLMG solver read allocated but uninitialized memory. Setting `m_phi_eb` to 0 after it is allocated fixes this issue. Attached is the valgrind logs, Line 17275 shows where valgrind detected reading of uninitialized memory. [vgout.txt](https://github.com/AMReX-Codes/amrex/files/7269235/vgout.txt) ## Additional background This was found due to one of our warpx circleCI test failing after a fetch and merge with development. This fix did not solve that issue though. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H commit b99a8433d3d5b363957b0391f5487ca95ea9e780 Author: Weiqun Zhang Date: Fri Oct 1 11:59:50 2021 -0700 Support Neumann BC in MLEBNodeFDLaplacian (#2371) Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 3ede71fa7835699784e0fca632980d89d7c23ee5 Author: Weiqun Zhang Date: Fri Oct 1 11:55:41 2021 -0700 Reset GPU RNG Seed (#2374) Add an optional parameter to amrex::ResetRandomSeed to allow the users to reset the GPU RNG seed. Slip in a typo fix. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_Random.H Src/Base/AMReX_Random.cpp commit 98a4e2338b2aef2718981b552de2d3b5db409c14 Author: Andrew Myers Date: Fri Oct 1 11:55:18 2021 -0700 Remove managed memory from neighbor particle test. (#2370) Tests/Particles/NeighborParticles/MDParticleContainer.cpp commit 1211ea30b520eb270e99d63fb80fb1d94ca8e45a Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Fri Oct 1 12:49:46 2021 -0600 NodeLaplacian Updates (#2332) In this branch, I make the number of smoothing steps a user defined parameter. Previously it was fixed at 4 for GPUs and 2 for CPUs. I also modified the NodalPoisson test to 1) use this new parameter 2) include additional parameters that control plot file dumping and the number of trials of the test to be run. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Tests/LinearSolvers/NodalPoisson/MyTest.H Tests/LinearSolvers/NodalPoisson/MyTest.cpp Tests/LinearSolvers/NodalPoisson/main.cpp commit 64ab8faa89e663cb912da9ad03a058c609e95447 Author: Weiqun Zhang Date: Fri Oct 1 07:58:07 2021 -0700 Update CHANGES for 21.10 (#2372) CHANGES commit 6f2dc1d06ee7d3f0024a9b8480ea8dde0d5fd644 Author: Weiqun Zhang Date: Wed Sep 29 15:37:17 2021 -0700 Update for oneapi 2021.4.0 (#2368) There are some namespace changes and deprecated functions in the new release. .github/workflows/intel.yml Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_Scan.H commit b11479d29b987b35089918d40747de72f994e117 Author: Axel Huebl Date: Wed Sep 29 15:09:36 2021 -0700 CI: NVHPC for CUDA (#2066) .github/workflows/cuda.yml .github/workflows/dependencies/dependencies_nvhpc21-9.sh Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_Extension.H Tools/CMake/AMReXGenerateConfigHeader.cmake commit 3eb5af68d209d056c20464c70d9abd9d0641e61a Author: Erik Date: Wed Sep 29 15:32:29 2021 -0400 fix -B grad(phi) typo to -beta grad(phi) (#2367) Fix a typo in the rst documents for the linear solvers. Docs/sphinx_documentation/source/LinearSolvers.rst commit ba6211f431bb594e7fc20478e9309fb88e79acc9 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Wed Sep 29 14:49:54 2021 -0400 CI: Change HIP repo line to reflect latest changes (#2366) This updates the repo line for HIP to the latest version as documented on https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html#installing-a-rocm-package-from-a-debian-repository Notably, `xenial` is now `ubuntu`. .github/workflows/dependencies/dependencies_hip.sh commit 8bf9816cf6f2a50cddad288deab1e8e72ef99e76 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Wed Sep 29 14:09:41 2021 -0400 Port CellConservativeQuartic to C++ with GPU support (#2362) CellConservativeQuartic interpolater has been ported to C++ with GPU support. Errors for CPU-only version on local with AmrLevel SingleVortex 2D and 3D are exactly 0. Errors for GPU-enabled version on Gigan with AmrLevel SingleVortex 2D and 3D are on the order of 1e-14. Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp commit b219a9cbd456863b95fe78a949339ba94fce65ab Author: Axel Huebl Date: Wed Sep 29 10:13:37 2021 -0700 nvc++ w/ OMP: work-around atomic capture (#2365) ## Summary Internal compiler error in this region with NVHPC 21.9 when compiling for OpenMP (host-only). ## Additional background Nvidia bug report: 3390723. Src/Base/AMReX_FabArrayUtility.H commit 514320464dacb9677ae1b84a4aee25ca6669c22c Author: Erik Date: Wed Sep 29 11:45:33 2021 -0400 Gpu Comments to Doxygen Docs (#2354) Added some comments about `Gpu::setLaunchRegion`, `Gpu::synchronize()` and `Gpu::streamSynchronize()` from the new and previous Gpu Debugging write-up. Converted some comments to Doxygen comments. Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuDevice.H commit 5932c85cd650eac65c04e09ca0163fe29de5421b Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Wed Sep 29 00:03:00 2021 -0400 fix AmrMeshParticle cmake issue (#2364) Src/Extern/SENSEI/CMakeLists.txt commit 9b58b442cc7091e5cf18375112d60a2f3d1a1bee Author: Axel Huebl Date: Tue Sep 28 18:42:35 2021 -0700 CI: Split Linux (#2363) The CI list became too long in the Linux section. This is cumbersome if one of the tests fails due to network/runner issues, because then all need to be restarted. Split into: - Linux GCC - Linux Clang - SENSEI .github/workflows/clang.yml .github/workflows/gcc.yml .github/workflows/sensei.yml commit 3006bba653ea8d93b58e779bdf796f3bd07e9bfd Author: Weiqun Zhang Date: Tue Sep 28 14:16:16 2021 -0700 Fix bug introduced in #2327 in DPCPP backend (#2361) Src/Base/AMReX_Reduce.H commit 9256409e2aae8148ef13afcf8316a72f12f1f3b8 Author: Max Katz Date: Tue Sep 28 09:06:16 2021 -0700 Remove hardcoded CUDA version for NVHPC (#2360) Tools/GNUMake/comps/nvhpc.mak commit 26fb18b44481a76d22f182e5955e23d5dbeeef29 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Mon Sep 27 17:27:26 2021 -0400 Port CellConservativeProtected::protect() to C++ with GPU support (#2347) CellConservativeProtected interpolater has been ported to C++ with GPU support. For the CPU-only version on local, the errors are exactly 0. For the GPU-enabled version on Gigan, the errors are in ~1e-11 range for IAMR 2D bubble test case, and ~1e-6 range for IAMR 3D Poiseuille test case. Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp commit 54dc681046f8c3f19034358fe0d45b72d13a9863 Author: Axel Huebl Date: Fri Sep 24 11:25:59 2021 -0700 CMake CUDA Debug: -G by Default (#2355) ## Summary In the past, this often did not compile at all, was very sensitive to further set options, or compiled super slowly; in some cases, such as recursive function usage, apps need to increase `cudaLimitStackSize` in order to not stack overflow with device debug symbols (this costs some extra DRAM). Nonetheless, for CUDA approx. 11.0+, we see the opposite: we have very slow Debug builds with CUDA if we do not activate -G for some `LinearSolvers/MLMG` objects: `AMReX_MLNodeLaplacian.cpp` & `AMReX_MLNodeLaplacian_sten.cpp`. Thus, we default-on now to -G in Debug builds. - [x] testing: now `LinearSolvers/MLMG/AMReX_MLTensorOp.cpp` compiles a bit longer (1-2min), but that's better than the other two TUs above without `-G` in Debug ## Additional background Thanks to @WeiqunZhang and @maxpkatz to looking into this with me. Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXCUDAOptions.cmake Tools/CMake/AMReXParallelBackends.cmake commit 66c90801c63a0c682ae59f0af3d2ebb4cb57f637 Author: Kevin Gott Date: Fri Sep 24 10:17:42 2021 -0700 OverlapSync nowait and finish. (#2346) Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp commit 777a2927f0e4da2f398d54914a5bb580c8a305a4 Author: Andrew Myers Date: Thu Sep 23 14:48:49 2021 -0700 Add another overload for PC::Checkpoint (#2353) Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_Particles.H commit 8c79eec68331b7e2122517e776a20cf77dcd04fc Author: Candace Gilet Date: Thu Sep 23 14:13:56 2021 -0400 Add curly brackets between amrex::Initialize and amrex::Finalize. (#2352) This prevents segmentation faults that can occur if amrex objects (like MultiFabs) are still in scope when Finalize is called. Tools/C_util/Convergence/Add.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/Convergence/DiffFab.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedFD.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp Tools/C_util/Convergence/DiffUniform.cpp Tools/C_util/Convergence/PltFileNorm.cpp Tools/C_util/Convergence/PltFileNormB.cpp Tools/C_util/Convergence/PltFileScalConvRate.cpp Tools/C_util/Convergence/RichardsonConvergenceTest.cpp commit 368a808c57ca3dbff40c3aa496e79fdf46c730d7 Author: Ann Almgren Date: Tue Sep 21 12:42:11 2021 -0700 Call setLevelBC from getFluxes in case we have set up the projector class but not yet called project (#2345) Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit a6c76f123cee34ec57cfb4f11fdf53389da3c640 Author: Andrew Myers Date: Mon Sep 20 18:15:35 2021 -0700 Function to atomically set particle ids (#2344) Src/Particle/AMReX_Particle.H commit 5d6a01f90ca606ac7a46879102db744041414b8f Author: Erik Date: Mon Sep 20 13:41:41 2021 -0400 Faq part one (#2340) Docs/sphinx_documentation/source/Faq.rst Docs/sphinx_documentation/source/index.rst commit 205c9b9ef6996b34138e170985b4d79d028bd57e Author: Weiqun Zhang Date: Mon Sep 20 10:41:16 2021 -0700 Fix CUDA compilation for EBTensor test (#2342) Tests/LinearSolvers/EBTensor/MyTest.H commit 8e0a48ea2e068111e171b8649fa8d71294bfcf7c Author: Weiqun Zhang Date: Mon Sep 20 10:33:21 2021 -0700 Remove macro: CRSEGRNDOMP and AMREX_CRSEGRNDOMP (#2339) The macro was added for backward compatibility when BoxLib first transitioned to omp over tiles. This can be removed now because We no longer have any Fab level functions containing OpenMP parallel regions. Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateData.cpp Tests/EB/CNS/Exec/Make.CNS Tests/EB_CNS/Exec/Make.CNS Tests/GPU/CNS/Exec/Make.CNS commit 3cff89900f8f6c21e02f8daceacffb4fda4b728a Author: Andrew Myers Date: Mon Sep 20 10:21:41 2021 -0700 Force turning off particle tiling for GPU on Advection_AmrLevel test (#2343) This enables the same inputs file to be used for GPU / non-GPU runs. Tests/Amr/Advection_AmrLevel/Source/main.cpp commit d568a0f9bb3ac324ad52b263240ddda03906ef43 Author: Andrew Myers Date: Mon Sep 20 10:20:58 2021 -0700 Don't rely on mananaged memory in PC::InitOnePerCell (#2337) Src/Particle/AMReX_ParticleInit.H commit 350a3efc2c6408efcadb6aaf0a334944af51098b Author: Michael Kieburtz Date: Mon Sep 20 08:26:21 2021 -0700 TinyProfile parser that saves data in a machine-readable format (#2335) This script parses the saved stdout output from a simulation run and parses the TinyProfile portion at the end. It saves this output in a JSON file that can be interpreted by Hatchet (https://github.com/hatchet/hatchet). Tools/TinyProfileParser/profileparser.py commit 677bc0990c7dea0cd32e3b8ab07664791c1681e1 Author: Weiqun Zhang Date: Sat Sep 18 12:12:19 2021 -0700 Various fixes (#2338) * AMReX_EB_LeastSquares_2D_K.H: fix warning * Tests/Amr/Advection_AmrLevel: should use pinned memory on host * Tests/AsyncOut/multifab: avoid managed memory * Tests/EB_CNS: fix warning on extra semicolon * Tests/GPU/AnyOf: avoid touching managed memory on host * Tests/LinearSolvers/CellOverset: fix compilation * Tests/LinearSolvers/EBTensor: initialize data on device * Tests/Parser: print max rel. error * Tests/Particles/AsyncIO: fix typo in inputs * Tests/Particles/DenseBins: do not test serial version for dpcpp * Tests/Particles/GhostsAndVirtuals: do not do randomperbox test for dpcpp Src/EB/AMReX_EB_LeastSquares_2D_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d_K.H Tests/AsyncOut/multifab/main.cpp Tests/EB_CNS/Source/CNS.H Tests/GPU/AnyOf/main.cpp Tests/LinearSolvers/CellOverset/MyTest.cpp Tests/LinearSolvers/EBTensor/MyTest.cpp Tests/LinearSolvers/EBTensor/MyTest_2D_K.H Tests/LinearSolvers/EBTensor/MyTest_3D_K.H Tests/Parser/main.cpp Tests/Particles/AsyncIO/inputs Tests/Particles/DenseBins/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp commit f47f884a453dfb76c372f1cf064e5f1e7df5be33 Author: Weiqun Zhang Date: Fri Sep 17 09:38:44 2021 -0700 Add documentation on reduction using ParallelFor. (#2333) Co-authored-by: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Docs/sphinx_documentation/source/GPU.rst commit 57703d6b624c7b03ff4bd282c3f07ca9aa5bb0fe Author: Jean M. Sexton Date: Fri Sep 17 09:37:45 2021 -0700 Update GPU Debugging basics section (#2330) * Update GPU Debugging basics section * Update Cuda-specific section to include new tools * Added Rocm specific commands * Added some information from Intel debugging webpages * Added links in comments Co-authored-by: etpalmer63 Docs/sphinx_documentation/source/GPU.rst commit 69b69c2b281c6f1b7b9b78a5aa81effebf0e3f2f Author: Weiqun Zhang Date: Mon Sep 13 16:21:41 2021 -0700 Fix return type of MLEBNodeFDLaplacian::setEBDirichlet (#2329) Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H commit 3d5b7d2e99d635960d031d6ad2aa07c9472be2e1 Author: Weiqun Zhang Date: Mon Sep 13 10:27:22 2021 -0700 ParReduce (#2327) ## Summary * Add ParReduce functions for MultiFab/FabArray and 1D iteration space. * The implementation of ParReduce(MF) uses new ReduceOps::eval functions that launch one GPU kernel for the whole MultiFab/MultiFab. * Use the new ParReduce functions in MultiFab/FabArray reduction functions. ## Additional background The name `ParallelReduce` has been used for the namespace containing MPI reduce wrappers. So we have to use a different name. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_ParReduce.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_iMultiFab.cpp Src/Base/CMakeLists.txt Src/Base/Make.package Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp commit 18a57c000e608d9e31054e84eceb799fb2789a4b Author: Kevin Gott Date: Mon Sep 13 08:08:44 2021 -0700 Update Profiling API docs. (#2303) Based on previous discussion that TinyProfiler's REGION wasn't well documented, this adds in the discussion about the various methods and macros available for instrumentation. Co-authored-by: etpalmer63 Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst commit 177b78eec251582646050db125af007117e97aae Author: Weiqun Zhang Date: Sun Sep 12 10:05:03 2021 -0700 Remove the MLNodeLinOp test (#2328) The tutorial has been moved to amrex-tutorial. Because of recent changes, the test no longer compiles. Tests/LinearSolvers/MultiComponent/GNUmakefile Tests/LinearSolvers/MultiComponent/MCNodalLinOp.H Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/MultiComponent/Make.package Tests/LinearSolvers/MultiComponent/inputs Tests/LinearSolvers/MultiComponent/main.cpp Tools/RegressionTesting/AMReX-tests.ini commit c1c57379871fec43b3512eab09b12950673e1ae6 Author: Weiqun Zhang Date: Sun Sep 12 08:58:33 2021 -0700 Remove some versions of ParallelFor(MF) (#2326) Remove ParallelFor(MF) functions that only take `int ncomp`, but not `IntVect nghost` to avoid potential bugs. The versions we have kept are, * ParallelFor(MF const&, ...); // valid region only * ParallelFor(MF const&, IntVect const& nghost, ...); // valid+nghost * ParallelFor(MF const&, IntVect const& nghost, int ncomp, ...); // valid+nghost and loop over component The version that is removed is * ParallelFor(MF const&, int ncomp, ...); This avoids bugs like below ParallelFor(mf, mf.nGrow(), ...); // should use mf.nGrowVect() Here, `mf.nGrow()` returns `int` and therefore would be treated as the number of components in the removed version. Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MultiFabUtil.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp commit 1c5608e05856e8690f968900c88924043addf025 Author: Erik Date: Sun Sep 12 00:19:15 2021 -0400 Amrvis doc revisions (#2305) Docs/sphinx_documentation/source/Visualization.rst commit b49296b837bcdd411c9d79f2abf07de18abc5c46 Author: Brandon Runnels Date: Fri Sep 10 14:03:10 2021 -0600 Update MLMG solver to allow variable number of ghost nodes for variable refinement ratios (#2319) Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit b3e38df8297bf32fd715f9f6491314a72c801d3b Author: Erik Date: Fri Sep 10 15:17:35 2021 -0400 Downgrade to docutils v0.16 (#2324) * Downgrade to docutils v0.16 to get bullet lists to render properly with Sphinx. .github/workflows/docs.yml commit f32f822e9ace1f8e4cd711b0bf7dbd174bd94e75 Author: Andrew Myers Date: Fri Sep 10 08:41:50 2021 -0700 Fix bug introduced in PR 2322 (#2325) Src/Base/AMReX_FabArray.H commit 36672da197ff9bda10a90d454212dc1c4ac3f9ff Author: Axel Huebl Date: Fri Sep 10 08:04:44 2021 -0700 CMake: Always CUDA Perf-Neutral Debug (#2321) Always default-ON for performance-neutral debug info to CUDA builds with CMake, even in `Release` mode. Tools/CMake/AMReXCUDAOptions.cmake commit 107ef9a1e1d20c149c2325ab5e8878cf4d7df052 Author: Andrew Myers Date: Thu Sep 9 16:36:41 2021 -0700 Add version of SumBoundary that takes src_nghost (#2322) Src/Base/AMReX_FabArray.H commit 3dc694a754d499c58d6f81f14c530e9e122f5858 Author: Erik Date: Thu Sep 9 12:12:49 2021 -0400 Add comment on "is_periodic" to doxygen (#2320) Co-authored-by: atmyers Src/Base/AMReX_Geometry.H commit d187e374f11c0fbe75c37544e95cb2185410b816 Author: Weiqun Zhang Date: Tue Sep 7 14:49:07 2021 -0700 Tests/EB_CNS on GPU (#2318) * Make some member functions public for CUDA. * Update calls to EBFluxRegister to run on device. * Fix race conditions in two kernels, one by avoiding write with if test and the other with atomics. * Use ParallelFor(MF) instead of ParallelFor(Box) for performance in a few places. * Use MFInterpolater instead of Interpolater for performance. * Remove Elixirs and rely on Gpu::streamSynchronize(). Maybe later, we could switch to using The_Async_Arena for these temporary Fabs for performance. * inline -> AMREX_FORCE_INLINE * Fix some 2D issues. Tests/EB_CNS/Exec/Pulse/cns_prob.H Tests/EB_CNS/Exec/ShockRef/cns_prob.H Tests/EB_CNS/Source/CNS.H Tests/EB_CNS/Source/CNS.cpp Tests/EB_CNS/Source/CNS_K.H Tests/EB_CNS/Source/CNS_advance.cpp Tests/EB_CNS/Source/CNS_advance_box.cpp Tests/EB_CNS/Source/CNS_advance_box_eb.cpp Tests/EB_CNS/Source/CNS_setup.cpp Tests/EB_CNS/Source/CNS_tagging.H Tests/EB_CNS/Source/diffusion/CNS_diffusion_K.H Tests/EB_CNS/Source/diffusion/CNS_diffusion_eb_K.H Tests/EB_CNS/Source/hydro/CNS_divop_K.H Tests/EB_CNS/Source/hydro/CNS_flux_redistribute.cpp Tests/EB_CNS/Source/hydro/CNS_hydro_K.H Tests/EB_CNS/Source/hydro/CNS_hydro_eb_K.H commit 0dab910e02864bb643cc91d78639df06f51fe17c Author: Weiqun Zhang Date: Tue Sep 7 10:17:14 2021 -0700 Remove the Old Fusing Approach (#2308) The old fusing macros and functions are removed in this commit. The old fusing approach is based on device function pointers. It only worked on Nvidia GPUs. Furthermore, the new ParallelFor(MF) and ParallelFor(Tag) approaches are much faster than the old approach. Src/Base/AMReX.cpp Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_Gpu.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuKernelInfo.H Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/Base/CMakeLists.txt Src/Base/Make.package Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Tests/GPU/Fuse/GNUmakefile Tests/GPU/Fuse/Make.package Tests/GPU/Fuse/main.cpp commit 26a55b4a277226a00f8c9e507268b2fd3ef2f3ee Author: Kevin Gott Date: Tue Sep 7 10:16:54 2021 -0700 Amrvis build cleanup. (#2314) Src/Extern/amrdata/AMReX_DataServices.H Tools/GNUMake/Make.rules commit 30ad2cb920f59e8d1174fd3a8aa242b8b280df3e Author: Weiqun Zhang Date: Tue Sep 7 08:45:52 2021 -0700 Add const to Array's sum and product functions (#2316) Since C++14, `constexpr` does not imply `const`. This is a follow-up on #2217. Src/Base/AMReX_Array.H commit 4a79504336a34a2728e418e71f3d68dba3afed5d Author: Candace Gilet Date: Mon Sep 6 15:53:28 2021 -0400 Update hypre doc (#2315) Update documentation on using HYPRE with the nodal solver. Support for CoarseningStrategy::Sigma with hypre has been added. Docs/sphinx_documentation/source/LinearSolvers.rst commit a8fe43774d8ad7437ff71bb253175d64a168d933 Author: Andrew Myers Date: Fri Sep 3 16:33:34 2021 -0700 Fix signed/unsigned comparison in diagnostic neighbor list function. (#2312) Src/Particle/AMReX_NeighborList.H commit 168a690497396de4c6b89a36b6edb0430e51ef4c Author: Houjun Tang Date: Fri Sep 3 12:22:04 2021 -0700 HDF5 I/O and Compression (#2220) .github/workflows/linux.yml Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/HDF5Benchmark/GNUmakefile Tests/HDF5Benchmark/inputs Tests/HDF5Benchmark/main.cpp Tools/GNUMake/packages/Make.hdf5 commit 55c18a5e98841034df30f8b91300a64b1269567b Author: Weiqun Zhang Date: Fri Sep 3 10:56:40 2021 -0700 Use ParallelFor(Tag) in Linear Solver Preparation (#2297) Replace the old fusing approach with ParallelFor(Tag) in MLCellLinOp::prepareForSolve. Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H commit 941607f89f68db1dc904d2321b80ac7f622740a9 Author: Weiqun Zhang Date: Fri Sep 3 08:39:44 2021 -0700 TagBoxArray: Remove omp parallel inside GPU launch region (#2309) Src/AmrCore/AMReX_TagBox.cpp commit 0a85e99beadea4cbe4b577a2854e094a834c5e75 Author: Andy Nonaka Date: Fri Sep 3 08:39:11 2021 -0700 make sure there are no NaNs in the plotfiles (#2306) The following routines would report no difference between two input plotfiles if one of them contained NaNs, which makes for a really bad comparison for regression usage. Now they abort with an error message if a NaN is detected. ComparePlotfiles.cpp DiffSameDomainRefined.cpp DiffSameDomainRefinedComposite.cpp DiffSameDomainRefinedStag.cpp DiffSameGrid.cpp DiffSameGrid2.cpp DiffSameGridRefined.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp commit 2bf85c3e7cae74149703c0e5b5020851a2433266 Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Fri Sep 3 11:00:12 2021 -0400 add dropped CMake code for mesh+particle adaptor (#2310) Src/Extern/SENSEI/CMakeLists.txt commit 112ce5551c43c02ff5fb8ca4241ac2d22d89d3e6 Author: Weiqun Zhang Date: Fri Sep 3 07:59:19 2021 -0700 Update DiffMultiFab tool (#2307) Print the location of min and max diff. Make it work with MPI. An example of output is below, ``` diffmultifab3d.gnu.TEST.ex infile1=mf1 infile2=mf2 ngrow=0 Reading mf1 Reading mf2 Component 0 Min and max of the diff are -0.5738299806 and 0.3011247047 Min Index: (63,127,127) Max Index: (75,127,127) Min and max of 1st mf are 0.1249966244 and 1.000010767 Min and max of 2nd mf are 0.125 and 1 Component 1 Min and max of the diff are -1.273924814e-05 and 0.3954904262 Min Index: (31,127,127) Max Index: (64,127,127) Min and max of 1st mf are -1.273924814e-05 and 0.3954904262 Min and max of 2nd mf are 0 and 0 Component 2 Min and max of the diff are 0 and 0 Min and max of 1st mf are 0 and 0 Min and max of 2nd mf are 0 and 0 Writing mfdiff ``` Tools/C_util/DiffMultiFab/GNUmakefile Tools/C_util/DiffMultiFab/diffmultifab.cpp commit e31e0161658664256eec4c3ae1cf98da33273601 Author: Weiqun Zhang Date: Thu Sep 2 16:57:36 2021 -0700 Kernel fusing in linear solver's applyBC (#2295) Use ParallelFor(Tag) to fuse the kernels in the cell-centered solver's applyBC function. Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H commit b15b1cf8d282cbb2c0d0bc0c7b049a79375ea63c Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Thu Sep 2 09:13:59 2021 -0600 Adding new parallel for into normalize method of node laplacian. (#2299) Co-authored-by: Paul Mullowney Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit c428771ceb21e812b92e21bc6383c650192982f1 Author: Andrew Myers Date: Wed Sep 1 17:30:20 2021 -0700 Gpu::Atomic::If (#2301) Src/Base/AMReX_Functional.H Src/Base/AMReX_GpuAtomic.H Tests/GPU/AtomicIf/GNUmakefile Tests/GPU/AtomicIf/Make.package Tests/GPU/AtomicIf/main.cpp commit 17d5b9049942d1cf72f943ee24b9c34cff8f6dfa Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Wed Sep 1 18:47:10 2021 -0400 add adaptor and bridge for amrMesh+Particles (#2285) Src/Extern/SENSEI/AMReX_AmrMeshParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrMeshParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_AmrMeshParticleInSituBridge.H Src/Extern/SENSEI/AMReX_ParticleDataAdaptorI.H commit 05539820d5878f2548b0b84328111efdde25ec7d Author: Weiqun Zhang Date: Wed Sep 1 10:04:07 2021 -0700 Update CHANGES for 21.09 (#2302) CHANGES commit cdb54ed66dac17d8867ee5dce01718852dd1788f Author: Erik Date: Wed Sep 1 12:29:03 2021 -0400 Change list of features to bullet list in Introduction.rst (#2300) Docs/sphinx_documentation/source/Introduction.rst commit 889686189996023c06e796c2cece3e5415a4b6b1 Author: Erik Date: Mon Aug 30 18:18:22 2021 -0400 Add Docs for `fboxinfo`, `fcompare`, `fextract`, etc. to Post-Processing Tools (#2290) Docs/sphinx_documentation/source/Post_Processing.rst Docs/sphinx_documentation/source/figs/ex_fsnapshot_resize.png commit e67bc03243c7df9acec8ae2ec66f9817bbda5fea Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Mon Aug 30 18:16:16 2021 -0400 Add gradient error tagging in Tests/Advection_AmrLevel (#2296) ## Summary Add proper `FillPatch()`ing for gradient checks in `AmrLevelAdv::errorEst()`. ## Additional background While working on #2268, I "accidentally" discovered that even though gradient error tagging is coded in `Advection_AmrLevel`, this check is never actually performed. This PR tries to fix that by adapting the error tagging code from `Tests/GPU/CNS`. Tested on local for CPU (error = exactly 0) and on Gigan for NVIDIA GPU (error = 1e-14 range). Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp commit d0b275b7064e20b87272eda07ea9adef0f63cf2a Author: Ann Almgren Date: Fri Aug 27 16:21:41 2021 -0700 Move slope routines out of amrex (#2294) * Fix logic for 3D interpolation of cell centroids to face centroids * Remove print statement that I left in by accident * Don't use std::min on the gpu! * When using hypre, limit number of possible multigrid levels by how much the EB can be coarsened * Move slope (EB and regular) routines out of amrex and into AMReX-Hydro * Move Tests/Slopes into AMReX-Hydro as well. Src/Base/AMReX_Slopes_K.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/EB/AMReX_EB_slopes_K.H Src/EB/CMakeLists.txt Src/EB/Make.package Tests/Slopes/GNUmakefile Tests/Slopes/Make.package Tests/Slopes/MyEB.H Tests/Slopes/MyTest.H Tests/Slopes/MyTest.cpp Tests/Slopes/README.md Tests/Slopes/initData.cpp Tests/Slopes/initEB.cpp Tests/Slopes/initLinearData.cpp Tests/Slopes/initLinearDataFor2D.cpp Tests/Slopes/initLinearDataFor3D.cpp Tests/Slopes/inputs Tests/Slopes/inputs.2d.askew Tests/Slopes/inputs.2d.base Tests/Slopes/inputs.2d.fullyrotated Tests/Slopes/inputs.3d.linear.aligned.xy-x Tests/Slopes/inputs.3d.linear.aligned.xy-y Tests/Slopes/inputs.3d.linear.aligned.xz-x Tests/Slopes/inputs.3d.linear.aligned.xz-z Tests/Slopes/inputs.3d.linear.aligned.yz-y Tests/Slopes/inputs.3d.linear.aligned.yz-z Tests/Slopes/inputs.3d.linear.askew-all Tests/Slopes/inputs.3d.linear.askew-xy Tests/Slopes/inputs.3d.linear.askew-xz Tests/Slopes/inputs.3d.linear.askew-yz Tests/Slopes/main.cpp commit a94cd92ab7d73f72af9c46e2b1ac001689a4ae2f Author: Weiqun Zhang Date: Fri Aug 27 13:42:25 2021 -0700 Tweak GNU make on perlmutter (#2293) No need to throw error on `make clean`. Tools/GNUMake/sites/Make.nersc commit 806af6b0c49c6aa2e986e35f44287e952c1b637b Author: Weiqun Zhang Date: Fri Aug 27 11:58:04 2021 -0700 Use ParallelFor(Tag) in communication preparation. (#2292) Use ParallelFor(Tag) instead of the old kernel fusing approach in the initialization of masks in communication preparation. Src/Base/AMReX_FBI.H Src/Base/AMReX_PCI.H Src/Base/AMReX_TagParallelFor.H commit 6c804d92bc18e7ac53d479b0b06c674bfe02d9fc Author: Weiqun Zhang Date: Fri Aug 27 10:03:01 2021 -0700 Update inputs files of Advection_AmrLevel (#2291) There are no longer probin files. Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs.tracers Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs-ci Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs.regt Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-dpcpp-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini commit 1055b29661584f04f9270344033705b09119b625 Author: Weiqun Zhang Date: Fri Aug 27 09:03:12 2021 -0700 Optimization of thread safety check in building communication metadata (#2288) The previous approach is very slow for big boxes. Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_FabArrayBase.cpp commit b20e88cff08fcbf6d92dc5b773dd25c096e355bc Author: Kevin Gott Date: Fri Aug 27 09:02:35 2021 -0700 Fix GNU builds on Perlmutter, + some bug fixes. (#2286) Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/sites/Make.nersc commit ef4e008b231fe3c6707d61c75c1da5e4e6f68420 Author: Erik Date: Fri Aug 27 11:46:06 2021 -0400 Short range particles corr (#2271) Docs/sphinx_documentation/source/Particle.rst Docs/sphinx_documentation/source/Visualization.rst Docs/sphinx_documentation/source/index.rst commit 85b8d544e22984096113e85eef6f2d3b56e30d54 Author: Andrew Myers Date: Thu Aug 26 18:27:31 2021 -0700 Silence uninitialized value warning in valgrind in neighbor list construction. (#2287) I don't think this is an actual bug, but it's good to play nice with valgrind. Thanks to Bruce Palmer. Src/Particle/AMReX_NeighborList.H commit af631d7be0877acaf53c47ed01149a5cd557d1f2 Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Thu Aug 26 21:20:44 2021 -0400 fix bug in AmrLevel exclusion logic (#2283) ## Summary Fixes a bug in the build logic regarding optional inclusion of the AMRLEVEL components, and return a required dependency to AMReX_AmrParticleDataAdaptor.H ## Additional background This bug was introduced in https://github.com/AMReX-Codes/amrex/pull/2258 where the SENSEI adaptors and bridges that processed data from the Amr class. A key dependency was removed, in the refactor. This Pr adds that dependency back in, and moves `AMReX_AmrParticleInSituBridge.H` behind the `AMReX_AMRLEVEL` compile option where it belongs. Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptor.H Src/Extern/SENSEI/CMakeLists.txt commit 33d5e4d04c16386147ddad92b3203cd91b033e7a Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Thu Aug 26 17:09:03 2021 -0400 Port Tests/Amr/Advection_AmrLevel from Fortran to C++ GPU (#2268) ## Summary The Advection_AmrLevel test/tutorial has been fully ported from Fortran to C++ (except for `initdata()` subroutine that stays in Fortran). ## Additional background A good chunk of the code was copy-pasted and/or adapted from the AmrCore and GPU/CNS test/tutorials. All four tests {SingleVortex,UniformVelocity} x {2D,3D} have been tested to work on the CPU, NVIDIA GPU on Gigan, and AMD GPU on Spock (`fcompare` shows differences of the order of 1e-14). Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/AmrLevel.rst Docs/sphinx_documentation/source/AmrLevel/figs/flowchart.odg Docs/sphinx_documentation/source/AmrLevel/figs/flowchart.png Tests/Amr/Advection_AmrCore/Exec/inputs-ci Tests/Amr/Advection_AmrLevel/CMakeLists.txt Tests/Amr/Advection_AmrLevel/Exec/Make.Adv Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Adv_prob.cpp Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Make.package Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob_Parm.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d_K.H Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs-ci Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs.tracers Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/probin Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Adv_prob.cpp Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Make.package Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob_Parm.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d_K.H Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs-ci Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs.regt Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/probdata.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/probin Tests/Amr/Advection_AmrLevel/README Tests/Amr/Advection_AmrLevel/Source/Adv.cpp Tests/Amr/Advection_AmrLevel/Source/Adv_F.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Amr/Advection_AmrLevel/Source/Kernels.H Tests/Amr/Advection_AmrLevel/Source/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_2d/Adv_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_2d/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_2d/compute_flux_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_2d/slope_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Adv_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_3d/compute_flux_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/slope_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_K/Adv_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_K/flux_2d_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/flux_3d_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/slope_K.H Tests/Amr/Advection_AmrLevel/Source/Src_K/tagging_K.H Tests/Amr/Advection_AmrLevel/Source/Src_nd/Adv_nd.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_nd/Tagging_nd.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/tagging_params.f90 Tests/Amr/Advection_AmrLevel/Source/Tagging_params.cpp Tests/Amr/Advection_AmrLevel/Source/bc_nullfill.cpp Tests/GPU/CNS/Source/CNS.H commit cc16557fbcbd9e023272c9db8830d5b837265e10 Author: Weiqun Zhang Date: Thu Aug 26 11:47:44 2021 -0700 Use ROCm 4.3 in CI (#2280) * Use ROCm 4.3 in CI Since the VOP bug has been fixed in the latest ROCm release v4.3, CI can use the latest ROCm release now instead of 4.1. * Work-Around: OpenMPI 4.0.3 on Ubuntu 20.04 "mpic++ --showme" forgets to link open-pal ``` /opt/rocm/llvm/bin/clang++ -pthread CMakeFiles/cmTC_abf94.dir/test_mpi.cpp.o -o cmTC_abf94 -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so ld.lld: error: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so: undefined reference to opal_class_init_epoch [--no-allow-shlib-undefined] ld.lld: error: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so: undefined reference to opal_list_item_t_class [--no-allow-shlib-undefined] ld.lld: error: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so: undefined reference to opal_class_initialize [--no-allow-shlib-undefined] ld.lld: error: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so: undefined reference to opal_uses_threads [--no-allow-shlib-undefined] ``` Results in `MPI_CXX` being not found. Co-authored-by: Axel Huebl .github/workflows/dependencies/dependencies_hip.sh .github/workflows/hip.yml commit 44d7b56878609c544b0f2906691786fc4d41633e Author: Weiqun Zhang Date: Thu Aug 26 08:04:52 2021 -0700 ParallelFor(Tag) (#2281) We have been using a ParallelFor function template that takes a Vector of Tags in communication functions. It launches a single GPU kernel for an irregular iteration space. This function is useful in many situations. Therefore, it is moved from a FillBoundary implementation header to its own header. And we have added two more versions of this for 1d and 3d loops. Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_TagParallelFor.H Src/Base/CMakeLists.txt Src/Base/Make.package commit fc6dcb883c4460523b7ae8164a2e26922511b857 Author: Andrew Myers Date: Wed Aug 25 17:08:44 2021 -0700 remove faulty assertion from PC::Restart() (#2279) Src/Particle/AMReX_ParticleIO.H commit 298b4892d082bf281b7cb4bab9423991e059bf3a Author: Andrew Myers Date: Wed Aug 25 16:05:06 2021 -0700 Protect against nullptr when calling PODVector dtor. (#2272) Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_PODVector.H commit 44edcc104f551b6243984b567ccd6723ac336699 Author: Corey Wetterer-Nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Wed Aug 25 18:18:35 2021 -0400 remove amr adaptor/bridge when AMRLEVEL is off (#2258) Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_InSituBridge.cpp Src/Extern/SENSEI/CMakeLists.txt commit cbfe59d9d11140326e943681a8675e072cfedb91 Author: Weiqun Zhang Date: Wed Aug 25 15:17:06 2021 -0700 Parser: Hide local symbols (#2274) Previously `Parser::symbols()` returned a list of symbols including local symbols. This is problematic because WarpX makes sure that all symbols are either a registered variable or constant. But the local variables are unknown to WarpX. In this commit, the local symbols are excluded from the return value of `Parser::symbols()`. Src/Base/Parser/AMReX_IParser_Y.H Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp commit 1e5492c974dc866749e0557f329a32250e38e524 Author: Erik Date: Wed Aug 25 17:14:46 2021 -0400 Doc macros (#2275) ## Summary Convert some comments on macros to Doxygen style for inclusion in the docs. ## Additional background I did not see a straightforward way to include comments on the variables set at compile time. Docs/Doxygen/doxygen.conf Src/Base/AMReX_SPACE.H commit 742069a0445fd511a5c64271d4ca21f22bab9b59 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Wed Aug 25 17:12:51 2021 -0400 Add EditorConfig section for rst files (#2273) ## Summary Add a section for `*.rst` files for EditorConfig. ## Additional background When editing the Sphinx `rst` source files for AmrLevel (PR #2268), I found out that the root `.editorconfig` file doesn't have a section for `rst` files. This PR adds that section, roughly based on the guidelines at https://docs.typo3.org/m/typo3/docs-how-to-document/master/en-us/GeneralConventions/CodingGuidelines.html . Note that I haven't gauged the "disruption" from whitespace changes that might be caused by this PR. .editorconfig commit 83de0324465f1f62d04c51a43e7212be348ad833 Author: Andrew Myers Date: Wed Aug 25 12:04:18 2021 -0700 Fix bug in the dual grid PC restart in the case where there are no particles on some levels. (#2276) Src/Particle/AMReX_ParticleIO.H commit bc3ad8347420a188838159b52d9bdcc5e9407060 Author: Weiqun Zhang Date: Tue Aug 24 10:59:59 2021 -0700 MF ParallelFor: LinearSolvers (#2267) Replace some `*_FUSIBLE` macros with MF ParallelFor. Note that the FUSIBLE macros are still used in some boundary functions. Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_MultiMask.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_F.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H commit 57618b0d0a46f9321cd558a2707503ce128bbe82 Author: Weiqun Zhang Date: Mon Aug 23 19:32:32 2021 -0700 Fix FabArray move-ctor and move-= (#2270) A bug was introduced to the move ctor and move operator= in the MF ParallelFor PR #2073. The cached arrays were not properly moved. Src/Base/AMReX_FabArray.H commit 6df13f8bb81caf4fdf9cda141e20994cd83018b2 Author: Weiqun Zhang Date: Mon Aug 23 12:49:57 2021 -0700 Fine-tune FabArray::setBndry (#2269) Use either ParallelFor(MF) or ParallelFor(tag) depending on box sizes. Src/Base/AMReX_FabArray.H commit 60578291de339964af938b81f2d0f7db5c6bf78c Author: Weiqun Zhang Date: Fri Aug 20 19:02:33 2021 -0700 MF ParallelFor: PETSc (#2262) * Use the new MF ParallelFor in PETSc setup. * Fix some existing issues in PETSc setup, including compilation in 3D and with CUDA. Src/Base/AMReX_MFParallelForG.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/AMReX_PETSc.cpp Tools/GNUMake/packages/Make.petsc commit e488c939316a2d52f0c4d3394689c4c154c6deb4 Author: Jean M. Sexton Date: Fri Aug 20 16:31:20 2021 -0700 Add new machine to GNUMake based on Make.olcf (#2264) Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.frontier-coe commit 6c5909a99ca327ac68aef9f2233bf4ff41de5248 Author: Weiqun Zhang Date: Fri Aug 20 10:29:11 2021 -0700 MF ParallelFor: HYPRE (#2257) * Use the new MF ParallelFor in HYPRE setup and remove the use of fusible macros. * Fix typos Src/Base/AMReX_LayoutData.H Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/PETSc/AMReX_PETSc.cpp commit 8f3d3511812d78ae5d5ca4f2620331cf8e3f3faa Author: Erik Date: Fri Aug 20 11:53:39 2021 -0400 Template wording (#2261) Changes the wording for the last checkbox on this pull request template. .github/pull_request_template.md commit 5e06ff8487368901c5fa79e5c7e8c944c62f7096 Author: Erik Date: Fri Aug 20 11:52:06 2021 -0400 Update contributing md (#2259) Adds style guide and short doxygen example to CONTRIBUTING.md CONTRIBUTING.md commit f840732f77100da5ebd40605a06e44ec44799516 Author: Weiqun Zhang Date: Thu Aug 19 12:27:44 2021 -0700 Fix #2256 (#2260) In #2256, a no-allocated MultiFab was passed to the MF ParalleFor. This resulted in segfault because of how the ParallelFor was implemented. In this commit, we fix the bug by reimplementing it without relying on Fabs being allocated. Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFParallelForG.H commit ebd1bac74492dc90def7fadea9e15ea77cb0e7a9 Author: Weiqun Zhang Date: Wed Aug 18 15:20:27 2021 -0700 MF ParallelFor: MultiFabUtil (#2256) Use the new MF ParallelFor in some MultiFabUtil functions. Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/F_Interfaces/Octree/AMReX_octree_fi.cpp commit 989d98b849ebaf31f000f60ff48ffd09304ff621 Author: Weiqun Zhang Date: Tue Aug 17 15:02:01 2021 -0700 Fix Particle CI Tests (#2252) A number of CI tests had issues with mixing single precision particle data with double precision amrex::Real. .github/workflows/linux.yml Src/AmrCore/AMReX_MFInterp_1D_C.H Src/Particle/AMReX_ParticleInit.H Tests/Particles/AssignDensity/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/main.cpp Tests/Particles/ParticleReduce/main.cpp commit 4f5800cce8396eda1b4692faabe05fc26273e738 Author: Weiqun Zhang Date: Tue Aug 17 15:01:19 2021 -0700 Github Action Concurrency (#2254) Cancel previous jobs in progress if a PR is updated. .github/workflows/cuda.yml .github/workflows/docs.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml .github/workflows/macos.yml .github/workflows/style.yml .github/workflows/windows.yml commit ba6a7467e5235616be6730384302ed3b684eea05 Author: Erik Date: Tue Aug 17 16:44:25 2021 -0400 Add live links (#2255) * correct sp typo * spelling and comma * Add live links. Some other small formatting. Docs/sphinx_documentation/source/AmrLevel_Chapter.rst Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/Fortran.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Docs/sphinx_documentation/source/IO.rst Docs/sphinx_documentation/source/LinearSolvers.rst Docs/sphinx_documentation/source/LinearSolvers_Chapter.rst Docs/sphinx_documentation/source/Particle.rst Docs/sphinx_documentation/source/SWFFT.rst commit b97214dbeaf2a799a56c802a7e07d1265bcbaf5a Author: Weiqun Zhang Date: Tue Aug 17 13:12:28 2021 -0700 MF ParallelFor: MultiFab (#2249) Use the new ParallelFor in some MultiFab functions. Src/Base/AMReX_MultiFab.cpp commit 4f1ebf2213c396f077dd7c70fdf8c9e65e8caff2 Author: Andrew Myers Date: Tue Aug 17 12:10:18 2021 -0700 Reimplement AddParticlesAtLevel for GPU (#2236) * Reimplement AddParticlesAtLevel for GPU * Don't assume grid 0, tile 0 is empty. Src/Particle/AMReX_ParticleContainerI.H commit 48bad74b9ca0642fa794925156d053fd14e3a93e Author: Lucas Esclapez <13371051+esclapez@users.noreply.github.com> Date: Tue Aug 17 12:05:35 2021 -0700 Enable mixed (face) GpuBndryFuncFab (#2238) Add face-centered Fill function. Only foextrap, hoextrap, reflectodd/even supported right now. Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_FilFC_1D_C.H Src/Base/AMReX_FilFC_2D_C.H Src/Base/AMReX_FilFC_3D_C.H Src/Base/AMReX_FilFC_C.H Src/Base/AMReX_PhysBCFunct.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 68327bc3ccfbb9bf31e467d479db6a854626e071 Author: Erik Date: Tue Aug 17 13:38:22 2021 -0400 Remove eb doc section (#2253) Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/EB/areas_and_volumes.fig Docs/sphinx_documentation/source/EB/areas_and_volumes.png Docs/sphinx_documentation/source/EB/eb_fluxes.fig Docs/sphinx_documentation/source/EB/eb_fluxes.png Docs/sphinx_documentation/source/EB/redist.fig Docs/sphinx_documentation/source/EB/redist.png commit 9a97484ad7940bfa9a1aa9064f07cc326992196a Author: Jean M. Sexton Date: Tue Aug 17 10:31:52 2021 -0700 Make sure all ba are defined in test, and only call Init once per PC (#2247) * Change boxes so all ba are defined, and use a new ParticleContainer for initializing tests * Remove duplicate workaround * GhostsAndVirtuals: Cleanup - Safety: use `.at()` over `operator[]` for performance-uncritical code - Some formatting * CI: Test Particles_GhostsAndVirtuals Co-authored-by: Axel Huebl .github/workflows/linux.yml .github/workflows/macos.yml Tests/Particles/GhostsAndVirtuals/main.cpp commit 2788e558156219f84840f0ab73d42dcfd344f21e Author: Weiqun Zhang Date: Mon Aug 16 18:23:23 2021 -0700 Shorten CI tests and Enable CNS in CI (#2251) * Split macos ci tests into two checks * Split hip ci tests * Change some build types from Debug to RelWithDebInfo * Enable Assertions and FPE in some tests * Add some inputs for ci * Enable CNS in CI .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml .github/workflows/macos.yml Tests/Amr/Advection_AmrCore/CMakeLists.txt Tests/Amr/Advection_AmrCore/Exec/inputs-ci Tests/Amr/Advection_AmrLevel/CMakeLists.txt Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs-ci Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs-ci Tests/EB/CNS/CMakeLists.txt Tests/EB/CNS/Exec/Sod/inputs-ci Tests/EB_CNS/CMakeLists.txt Tests/EB_CNS/Exec/Combustor/inputs-ci Tests/GPU/CNS/CMakeLists.txt Tests/GPU/CNS/Exec/Sod/inputs-ci Tests/LinearSolvers/NodalPoisson/CMakeLists.txt Tests/LinearSolvers/NodalPoisson/inputs-ci commit 79a57600d5eec3c9a0ffdbb717462728a1d56598 Author: Weiqun Zhang Date: Mon Aug 16 10:22:17 2021 -0700 Use the new ParallelFor in some FabArrayUtility functions (#2246) Src/Base/AMReX_FabArrayUtility.H commit a4f94e7524c0348af134968a9e17c7fd08f266f7 Author: Weiqun Zhang Date: Sun Aug 15 15:31:49 2021 -0700 Use new ParallelFor in FabArray (#2245) * Add FabArrayBase::isFusingCandidate() that is fine tuned for MI100, V100 and A100. * Convert some FabArray functions to using the new ParallelFor. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp commit c94cbd9b78cc8fa559c00c45c49b9add0075cc0f Author: Axel Huebl Date: Sat Aug 14 18:50:47 2021 -0700 Fix: Slopes K Literals (#2250) * Fix: Slopes K Literals Address compile issues with GCC 9.3.0 (Manda) / 10.1.0 (Battra). Seen with GNUmake by Ann. * Slopes K: Cleanup - Add used headers - move into regular amrex:: namespace - simplify class usage in amrex:: namespace Src/Base/AMReX_Slopes_K.H commit 9f58fc5c936129424f2a64d5c2c4e4de243bc3b5 Author: Axel Huebl Date: Sat Aug 14 12:10:51 2021 -0700 CI: Build Embedded Boundaries (#2232) - [x] Add coverage for EB in CI - [x] fix DPC++ single precision build - [x] ~~fix `LinearSolvers` array bounds w/ GCC 7.5~~ -> false positive Co-authored-by: Weiqun Zhang .github/workflows/cuda.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml .github/workflows/macos.yml .github/workflows/windows.yml Src/Base/AMReX_Slopes_K.H Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EB_utils.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_eb.cpp Tests/EB/CNS/Source/CNS.H Tests/EB_CNS/Source/hydro/CNS_divop_K.H commit cc89a4bc60b54902f4b07f531a03cd2c1b834e58 Author: Andrew Myers Date: Fri Aug 13 19:34:28 2021 -0700 Also remove empty map entries when calling clearParticles (#2248) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particles.H commit e8512831302f27820e0e6a0714f83f3905e89895 Author: Maxim Shatsky Date: Fri Aug 13 19:29:28 2021 +0300 Bug fix of rarely met issue from Tools/CMake/AMReXUtils.cmake file (#2244) Everything is already described here: https://github.com/AMReX-Codes/amrex/issues/2240 Tools/CMake/AMReXUtils.cmake commit 090596bd4241497616cfbb86f168556d539ceb42 Author: Erik Date: Thu Aug 12 16:30:34 2021 -0400 Add definition of MLMG (#2242) Docs/sphinx_documentation/source/LinearSolvers.rst Src/Base/AMReX_MultiFab.H commit fb39fa7ac33ed916f42cc26d134c377113e78fa6 Author: Weiqun Zhang Date: Thu Aug 12 12:40:37 2021 -0700 Move MF ParallelFor out of experimental:: (#2241) It's still available in experimental::, but it will be removed from it eventually. Src/AmrCore/AMReX_MFInterpolater.cpp Src/Base/AMReX_MFParallelFor.H Src/EB/AMReX_EBMFInterpolater.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 9a7688371d000798c71a3ca246bd983bc9d300b6 Author: Andrew Myers Date: Thu Aug 12 08:58:11 2021 -0700 Fix some tests so that ctests works with CUDA enabled. (#2239) This also fixes and scoping bug in the particle iterator test. Tests/LinearSolvers/NodeTensorLap/CMakeLists.txt Tests/Particles/ParallelContext/CMakeLists.txt Tests/Particles/ParallelContext/inputs.rt.cuda Tests/Particles/ParticleIterator/main.cpp Tests/Particles/Redistribute/CMakeLists.txt commit d39e0b86813852d6c2a38f57f6243cdb750e08df Author: Weiqun Zhang Date: Thu Aug 12 08:53:50 2021 -0700 Option for dynamic tiling in CPU build of MF ParallelFor (#2235) Examples: experimental::ParallelFor(mf, nghost, TileSize{my_tile_size}, DynamicTiling{true}, [=] ...); experimental::ParallelFor(mf, nghost, ncomp, TileSize{my_tile_size}, DynamicTiling{true}, [=] ...); Co-authored-by: Andrew Myers Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MFParallelForC.H Src/Base/AMReX_MFParallelForG.H commit 694279b0fb23b90a40ac8ba93fe5ed90b8510123 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Wed Aug 11 21:55:14 2021 -0400 Add sum(), product(), size(), {x,y,z}{lo,hi,len}() member functions to GpuArray and Array?D (#2217) Now `GpuArray` and `Array1`/`2`/`3D` has member functions `sum()` and `product()`. These functions will take the sum or product over the entire array. For Array2D, both can optionally take in `int axis` and `int loc`. Same goes for `Array3D`, but it takes two positions. The following static member functions to return array bounds and sizes are added: * `Array1D`: `lo()`, `hi()`, `len()`, `size()` * `Array2D`: `xlo()`, `xhi()`, `xlen()`, `ylo()`, `yhi()`, `ylen()`, `size()` * `Array3D`:`xlo()`, `xhi()`, `xlen()`, `ylo()`, `yhi()`, `ylen()`, `zlo()`, `zhi()`, `zlen()`, `size()` The `*lo()` and `*hi()` functions return an `int`, and the `*len()` (per direction) and `size()` (total in all directions) functions return a `uint`. In addition, the `begin()` and `end()` member functions have also been added for `Array?D`, which return the start pointer address and the pointer address right after the last element, for iterator purposes, if needed. For `Array2`/`3D`, it's implemented as if the array is one-dimensional. Co-authored-by: etpalmer63 Co-authored-by: Weiqun Zhang Src/Base/AMReX_Array.H commit e82d45264b7840ff9b6247c4a7eb534f55ed5233 Author: Weiqun Zhang Date: Wed Aug 11 10:55:13 2021 -0700 Fix derive data in AmrLevel::writePlotFile (#2234) Previously, it only worked with single-component DeriveList. Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Derive.cpp Tests/GPU/CNS/Exec/RT/inputs Tests/GPU/CNS/Exec/RT/inputs-rt Tests/GPU/CNS/Exec/Sod/inputs Tests/GPU/CNS/Exec/Sod/inputs-rt Tests/GPU/CNS/Source/CNS_derive.cpp Tests/GPU/CNS/Source/CNS_setup.cpp commit f4c2629340bf5fdac41d869321e0514f68654789 Author: Weiqun Zhang Date: Tue Aug 10 19:41:27 2021 -0700 Add MFNodeBilinear for nodal linear interpolation (#2202) Src/Amr/AMReX_AmrLevel.cpp Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_MFInterpolater.cpp commit 0ca401daa57fd80e7ae7a12eb5fd28b9d10ce11b Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Tue Aug 10 11:33:23 2021 -0600 Adding experimental parfor into the FSmooth method (#2230) Co-authored-by: Paul Mullowney Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 075f30f8ad323c2a8726e35d52ff70f53c681a49 Author: Weiqun Zhang Date: Tue Aug 10 10:15:07 2021 -0700 Parser: Exception handling (#2233) For unknown characters, syntax errors, and unknown variables, throw exceptions with hopefully more helpful messages. Src/Base/Parser/AMReX_IParser.H Src/Base/Parser/AMReX_IParser.cpp Src/Base/Parser/AMReX_IParser_Exe.cpp Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Exe.cpp Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_iparser.l Src/Base/Parser/amrex_iparser.lex.cpp Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp commit 0528d849c6c3d17193263f1c4aa15a6b07e7ed1a Author: Weiqun Zhang Date: Mon Aug 9 18:12:57 2021 -0700 Parser: Noinline transcendental functions (#2229) It's too expensive to have them inlined. Src/Base/Parser/AMReX_Parser_Y.H commit a2ad6250aabed7042f33b7c5356274104165ef4b Author: Ann Almgren Date: Sun Aug 8 13:50:59 2021 -0700 Fix 2d eb slope limiting (#2226) There was a bug in the 2D EB slope limiting -- this fixes that Src/EB/AMReX_EB_slopes_K.H commit 867d1deb3b62ca99ce50a95eb190a548eb119112 Author: Ann Almgren Date: Sat Aug 7 13:22:58 2021 -0700 New test eb cns (#2208) This directory Tests/EB_CNS is designed to replace Tests/EB/CNS -- it is a pure-C++ version of EB/CNS and matches the current regression tests to sufficient precision. It is not GPU-ready. Tests/EB_CNS/CMakeLists.txt Tests/EB_CNS/Exec/Combustor/CNS_bcfill.cpp Tests/EB_CNS/Exec/Combustor/GNUmakefile Tests/EB_CNS/Exec/Combustor/Make.package Tests/EB_CNS/Exec/Combustor/cns_prob.H Tests/EB_CNS/Exec/Combustor/cns_prob.cpp Tests/EB_CNS/Exec/Combustor/cns_prob_parm.H Tests/EB_CNS/Exec/Combustor/cns_prob_parm.cpp Tests/EB_CNS/Exec/Combustor/inputs Tests/EB_CNS/Exec/Combustor/inputs.regt Tests/EB_CNS/Exec/Make.CNS Tests/EB_CNS/Exec/Pulse/GNUmakefile Tests/EB_CNS/Exec/Pulse/Make.package Tests/EB_CNS/Exec/Pulse/cns_prob.H Tests/EB_CNS/Exec/Pulse/cns_prob.cpp Tests/EB_CNS/Exec/Pulse/cns_prob_parm.H Tests/EB_CNS/Exec/Pulse/inputs Tests/EB_CNS/Exec/Pulse/inputs.regt Tests/EB_CNS/Exec/ShockRef/GNUmakefile Tests/EB_CNS/Exec/ShockRef/Make.package Tests/EB_CNS/Exec/ShockRef/cns_prob.H Tests/EB_CNS/Exec/ShockRef/cns_prob.cpp Tests/EB_CNS/Exec/ShockRef/cns_prob_parm.H Tests/EB_CNS/Exec/ShockRef/inputs Tests/EB_CNS/Exec/ShockRef/inputs.amr Tests/EB_CNS/Exec/ShockRef/inputs.regt Tests/EB_CNS/Exec/Sod/GNUmakefile Tests/EB_CNS/Exec/Sod/Make.package Tests/EB_CNS/Exec/Sod/cns_prob.H Tests/EB_CNS/Exec/Sod/cns_prob.cpp Tests/EB_CNS/Exec/Sod/cns_prob_parm.H Tests/EB_CNS/Exec/Sod/inputs Tests/EB_CNS/Source/CNS.H Tests/EB_CNS/Source/CNS.cpp Tests/EB_CNS/Source/CNSBld.cpp Tests/EB_CNS/Source/CNS_K.H Tests/EB_CNS/Source/CNS_advance.cpp Tests/EB_CNS/Source/CNS_advance_box.cpp Tests/EB_CNS/Source/CNS_advance_box_eb.cpp Tests/EB_CNS/Source/CNS_bcfill.cpp Tests/EB_CNS/Source/CNS_derive.H Tests/EB_CNS/Source/CNS_derive.cpp Tests/EB_CNS/Source/CNS_index_macros.H Tests/EB_CNS/Source/CNS_init_eb2.cpp Tests/EB_CNS/Source/CNS_io.cpp Tests/EB_CNS/Source/CNS_parm.H Tests/EB_CNS/Source/CNS_parm.cpp Tests/EB_CNS/Source/CNS_setup.cpp Tests/EB_CNS/Source/CNS_tagging.H Tests/EB_CNS/Source/Make.package Tests/EB_CNS/Source/diffusion/CNS_diffusion_K.H Tests/EB_CNS/Source/diffusion/CNS_diffusion_eb_K.H Tests/EB_CNS/Source/diffusion/Make.package Tests/EB_CNS/Source/hydro/CNS_divop_K.H Tests/EB_CNS/Source/hydro/CNS_flux_redistribute.cpp Tests/EB_CNS/Source/hydro/CNS_hydro_K.H Tests/EB_CNS/Source/hydro/CNS_hydro_eb_K.H Tests/EB_CNS/Source/hydro/Make.package Tests/EB_CNS/Source/main.cpp commit eed6b0c572301d793c098c3dae7286010d046182 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Sat Aug 7 12:08:36 2021 -0600 Adding experimental parfor into the restriction operator (#2204) Co-authored-by: Paul Mullowney Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit 660a17309966e07fdda41b66d7016e23795a846d Author: Weiqun Zhang Date: Sat Aug 7 10:52:26 2021 -0700 Fix PArena (#2221) In some PArena functions, we forgot to test if the memory pool is supported. ## Additional background https://github.com/AMReX-Astro/Castro/issues/1955 Src/Base/AMReX_PArena.cpp commit 468feb057586e06e069b22f62468c572f7172856 Author: Axel Huebl Date: Sat Aug 7 10:51:53 2021 -0700 Test Build Variant (#2223) The AMReX "test" build variant that is used by [regression tests](https://github.com/AMReX-Codes/regression_testing) by default was not documented, which leads to some confusion when debugging failing apps. This PR: - documents GNUmake's `TEST` & `USE_ASSERTION` options - adds a CMake option `AMReX_TESTING` that are in sync with the define `-DAMREX_TESTING` that changes FAB default values and other debug-like options. Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXSetDefines.cmake commit 2c98061cb00d552ec6262cdffc475e4c0cc92445 Author: Weiqun Zhang Date: Sat Aug 7 10:44:29 2021 -0700 EB: Fix small cells and multicuts (#2225) The previous approach of fix small cells and multicuts is not consistent. If a small cell is turned into a covered cell, we need to fix its neighbors too. In this new approach, we build the EB data iteratively. This commit also introduces a new ParmParse parameter, eb2.maxiter, that controls the maximal number of iterations. Src/Base/AMReX_GpuBuffer.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_MultiGFab.H Src/EB/AMReX_EB2_MultiGFab.cpp Src/EB/AMReX_EB2_ND_C.cpp Tests/LinearSolvers/CellEB/MyEB.H commit bca7ff1fdd2d670eda38c0304ebf8741f15b0873 Author: Jean M. Sexton Date: Thu Aug 5 13:55:59 2021 -0700 Make CreateVirtualParticles use GPU functions for aggregation_type=Cell (#2219) Src/Particle/AMReX_ParticleContainerI.H Tests/Particles/GhostsAndVirtuals/fixed_grids.init Tests/Particles/GhostsAndVirtuals/main.cpp commit cbd153a6d0b483b82b7ec5c540488cd5b97c5189 Author: Ann Almgren Date: Thu Aug 5 12:11:52 2021 -0700 Change getgradphi in nodalproj (#2222) * Add options to return const or non-const vector of pointers to phi and gradphi from nodal projector * oops -- left in extra const Src/LinearSolvers/Projections/AMReX_NodalProjector.H commit 6690a90f407c536c0acf43e6c8fc3862ab6d7097 Author: Jean M. Sexton Date: Tue Aug 3 16:48:21 2021 -0700 Add static_cast for ParticleReal, and check assertion (#2215) Src/Amr/AMReX_Derive.cpp Src/Particle/AMReX_NeighborParticlesCPUImpl.H Tests/Particles/AsyncIO/main.cpp Tests/Particles/NeighborParticles/MDParticleContainer.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/main.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleReduce/main.cpp Tests/Particles/ParticleTransformations/main.cpp Tests/Particles/Redistribute/main.cpp commit 3b2c8de1ea1d8b67cb9910a142249c5cc6fe7e75 Author: Weiqun Zhang Date: Tue Aug 3 15:04:09 2021 -0700 Fix EBInterpolater (#2214) Fix bug in #2203. The fine factory may not contain information on boxes outside the fine target region. Src/EB/AMReX_EBInterpolater.cpp Src/EB/AMReX_EBMFInterpolater.cpp commit ac818cd22aded63eff8358ad54e0567232163341 Author: Jean M. Sexton Date: Tue Aug 3 14:49:40 2021 -0700 Add check for OpenMP::get_max_threads()=0 case for CI warning (#2212) Src/Base/AMReX_MultiFabUtil.cpp commit 5da293757b8bc67225702d3fe1ddc6fb0c010fa4 Author: Weiqun Zhang Date: Mon Aug 2 15:15:44 2021 -0700 Add EBMFCellConsLinInterp for EB cell data interpolation (#2203) Src/Amr/AMReX_StateDescriptor.H Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_MFInterpolater.H Src/EB/AMReX_EBInterpolater.H Src/EB/AMReX_EBInterpolater.cpp Src/EB/AMReX_EBMFInterpolater.H Src/EB/AMReX_EBMFInterpolater.cpp Src/EB/CMakeLists.txt Src/EB/Make.package commit af968f828c51d7d6d07646c69fe8e53b85a7d6a1 Author: Weiqun Zhang Date: Mon Aug 2 14:05:11 2021 -0700 Bilinear interpolation for cell data (#2205) Add MFCellBilinear for cell data. Port CellBilinear to C++. Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_MFInterpolater.cpp commit ee8f6828dd1162ef6bcf123c417b31abf63f4eef Author: Axel Huebl Date: Mon Aug 2 10:08:24 2021 -0700 CMake: Prebuild & SetupCUDA (#2209) ## Summary With CMake 3.20+, we don't need to include the `SetupCUDA.cmake` scripts anymore. ## Additional background Follow-up to #2012 Tools/CMake/AMReXConfig.cmake.in commit 2695be7cad80f2b4ec25abe57925cc166e3aa824 Author: Weiqun Zhang Date: Mon Aug 2 08:59:50 2021 -0700 Reduction of MulitFab to 1D data (#2182) Add `sumToLine` function that reduces MultiFab's data to a 1D line. Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp commit b076fa56f696acc3591a2b6ee03cc6f981731cf4 Author: Weiqun Zhang Date: Mon Aug 2 08:59:15 2021 -0700 YAFluxRegister with blocking factor of 1 (#2155) Fix YAFluxRegister for the case that the blocking factor is 1. In that case, a coarse cell might have fine neighbors at both x-lo and x-hi faces. Src/Boundary/AMReX_YAFluxRegister_1D_K.H Src/Boundary/AMReX_YAFluxRegister_2D_K.H Src/Boundary/AMReX_YAFluxRegister_3D_K.H commit 3fdfac26cd238cfc0c8c4fd8a754dfd4b8cd3eeb Author: Weiqun Zhang Date: Mon Aug 2 08:32:44 2021 -0700 Fix lincc_interp (#2201) This fixes a bug in lincc_interp that was introduced more than 2 years ago when the Fortran code was converted to C++. The slope was mistakenly limited twice. For each component, we compute a limiting factor and we use the minimum of all components as the final limiting factor. That final limiting factor should be applied to the unlimited central difference slope, not the monotonized central slope. Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H commit 4ee1fd09cfdda8ad975e8e62aa70772bfe2a7455 Author: Weiqun Zhang Date: Mon Aug 2 08:31:51 2021 -0700 Add MFPCInterp for piece-wise constant interpolation (#2200) Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_MFInterpolater.cpp commit fa89d68a8909c9630d0a9034ca27fda50137119c Author: Weiqun Zhang Date: Sun Aug 1 15:22:12 2021 -0700 Fix Coarse/Fine Boundary BoxArray generated in FPInfo for FillPatch (#2207) The issue was degenerate boxes could appear when the coarse/fine boxarray was split into smaller chunks. This is usually not an issue except for a small extra cost. But in IAMR this could result in external Dirichlet pressure boundary function that has not been implemented being called. Src/Base/AMReX_FabArrayBase.cpp commit 7034f81724850188d104e34c787c302d3d372c44 Author: Weiqun Zhang Date: Sun Aug 1 14:42:40 2021 -0700 Update CHANGES for 21.08 (#2206) CHANGES commit b87ec0a8fc084dbe886973e1e3c8a4eedf6b7dfa Author: Kevin Gott Date: Thu Jul 29 09:29:22 2021 -0700 Fix bulleted list in IO Docs. (#2198) Docs/sphinx_documentation/source/IO.rst commit fbdf1d043ed572d8388eb445a7e091d0a11acde9 Author: Kevin Gott Date: Thu Jul 29 08:54:02 2021 -0700 Add AsyncVector. (#2197) Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuContainers.H Tests/GPU/Vector/main.cpp commit 0124f3b425adbe00e8ec021397d654eb411df9b0 Author: Jean M. Sexton Date: Thu Jul 29 08:08:47 2021 -0700 Make CreateVirtualParticles use GPU functions for aggregation_type=None (#2195) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_TracerParticle_mod_K.H commit 4387cf22f3a5417b17b2445b22fd3659fa0f1462 Author: Weiqun Zhang Date: Wed Jul 28 18:09:40 2021 -0700 Fix divide by zero due to roundoff errors (#2199) Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H commit 5619ef246175454d1777cd5dc4cd0013ad2966dd Author: Ann Almgren Date: Wed Jul 28 13:42:55 2021 -0700 Change mask for ctr to ctroid interp (#2196) Src/EB/AMReX_EBMultiFabUtil.cpp commit 282878dfbf1c823a90f3265c318e15185676710a Author: Andrew Myers Date: Wed Jul 28 12:14:45 2021 -0700 Add OpenMP support to DenseBins::build (#2193) Src/Particle/AMReX_DenseBins.H Tests/Particles/DenseBins/CMakeLists.txt Tests/Particles/DenseBins/GNUmakefile Tests/Particles/DenseBins/Make.package Tests/Particles/DenseBins/inputs Tests/Particles/DenseBins/main.cpp commit 7ac29e889e4c0949eba6dffcc1e03ce27dc2751c Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Tue Jul 27 17:15:31 2021 -0600 Integration of experimental parfor into NodeLaplacian Fapply. (#2190) Co-authored-by: Paul Mullowney Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp commit 5bdae60f411b47b2360c755ad21b7cc550a547d8 Author: Kevin Gott Date: Tue Jul 27 16:08:41 2021 -0700 Add some missing FabArray function timers. (#2194) Src/Base/AMReX_FabArray.H commit 35edeb12e000a6dd75765c65dcb6dd93b3cd340a Author: Andrew Myers Date: Tue Jul 27 14:12:08 2021 -0700 Fix filter functions for callables that return bool. (#2192) Src/Particle/AMReX_ParticleTransformation.H Tests/Particles/ParticleTransformations/main.cpp commit 16cdf1504d9e00b906053431e8037c9f6d5b3f9c Author: Andrew Myers Date: Tue Jul 27 14:11:56 2021 -0700 Reduce code duplication in DenseBins::build (#2191) Src/Particle/AMReX_DenseBins.H commit bdebb1055d3cc84d95e379833821572a579bb838 Author: Weiqun Zhang Date: Mon Jul 26 17:58:04 2021 -0700 MFCellConsLinInterp (#2189) Add MFCellConsLinInterp that is similar to CellConsLinInterp. The difference is that MFCellConsLinInterp works on MultiFab, whereas CellConsLinInterp works on FArrayBox. The new interpolater launches fewer kernels, therefore has better performance on GPU when there are many boxes. Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_InterpBase.H Src/AmrCore/AMReX_InterpBase.cpp Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_MFInterp_1D_C.H Src/AmrCore/AMReX_MFInterp_2D_C.H Src/AmrCore/AMReX_MFInterp_3D_C.H Src/AmrCore/AMReX_MFInterp_C.H Src/AmrCore/AMReX_MFInterpolater.H Src/AmrCore/AMReX_MFInterpolater.cpp Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_MFParallelForG.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp commit 3181f76910e6ac28ec5e099cba3e192bfdbf9076 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Sun Jul 25 19:53:42 2021 -0400 Introduction of the experimental parfor into NodeLaplacian interpolation (#2183) All algorithms fully implemented including interpadd_aa, semi_interpadd_aa, interp_c, interp_rap, and interpadd_ha. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit f35a59b079878a43e64418f36be356227e7fc1b9 Author: Andrew Myers Date: Sun Jul 25 15:55:32 2021 -0700 Enable default construction for AmrParticleContainer and add isDefined() method. (#2188) A Particle container is considered "defined" if it has been given a set of grids, otherwise not. Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_ParticleContainerBase.H commit 45711e08a54d42ced219ffe00f6e8b435ebc7bf3 Author: Jean M. Sexton Date: Fri Jul 23 17:04:41 2021 -0700 Make CreateGhostParticles use GPU functions (#2185) Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTransformation.H Tests/Particles/GhostsAndVirtuals/CMakeLists.txt Tests/Particles/GhostsAndVirtuals/fixed_grids.init Tests/Particles/GhostsAndVirtuals/inputs Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/GhostsAndVirtuals/particle_file.init commit 27069ef1d885a35401f963ead2e3bad19cc9dc8f Author: Axel Huebl Date: Thu Jul 22 08:47:39 2021 -0700 CI: HIP Cleanup (#2186) Move the PATH setup for the AMD `clang`/`clang++` executables to a common location for the setup scripts. As a reminder, this is needed so the system clang is not taken by accident. .github/workflows/dependencies/dependencies_hip.sh .github/workflows/hip.yml commit 91fa2b7e6ff33117bd537635c8c55192359863a2 Author: Weiqun Zhang Date: Wed Jul 21 12:12:52 2021 -0700 Add ROCMClang as an LLVM compiler to cmake (#2184) * Add ROCMClang as an LLVM compiler to cmake Since CMake >= 3.21, ROCm is identified as ROCMClang. * HIP: Fix legacy CI / hipcc Until we get ROCM 4.4, we need to work around this new detection: https://gitlab.kitware.com/cmake/cmake/-/blob/v3.21.0/Modules/CMakeDetermineCompilerId.cmake#L153-159 Co-authored-by: Axel Huebl .github/workflows/hip.yml Tools/CMake/AMReXGenerateConfigHeader.cmake commit 948bd72d3e2ee4ee7ee6e1861a08f559ac4d1ffc Author: Ann Almgren Date: Tue Jul 20 14:53:41 2021 -0700 Fix typo (#2181) * Fix typo : The_Async_Async --> The_Async_Arena Docs/sphinx_documentation/source/GPU.rst commit fe9ae3e2a978de929686ad070c713baaf634ec1c Author: Jean M. Sexton Date: Tue Jul 20 12:07:50 2021 -0700 Add bounds-checking in InitFromAscii for CI null-dereference error (#2180) Src/Particle/AMReX_ParticleInit.H Tests/Particles/InitFromAscii/main.cpp commit a449f384df5ddece0d4e063fe9001e280ed64024 Author: Jean M. Sexton Date: Tue Jul 20 09:57:32 2021 -0700 Scale uniform random interval to account for gridbox length (#2179) Src/Particle/AMReX_ParticleInit.H commit 48d84eb342b04aae7e18217980a4514e5a9de3ea Author: Weiqun Zhang Date: Tue Jul 20 08:44:32 2021 -0700 Fix and update multifab ParallelFor (#2178) - Fix a bug in caching multifab ParallelFor meta-data. - Add new versions of multifab ParallelFor that are templated on max threads per block. Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MFParallelForG.H commit 779835b7d122f1c94be7c85486e951e8766730ec Author: Weiqun Zhang Date: Mon Jul 19 10:41:51 2021 -0700 Optimization of ParallelFor for single box FabArray (#2176) When there is only one Fab, we can simplify the MF ParallelFor funciton. Src/Base/AMReX_MFParallelForG.H commit ee8facf7e09d0b40af1fe680665660c24a19d32a Author: Ann Almgren Date: Mon Jul 19 09:11:47 2021 -0700 Add ns to gpu cns (#2177) Add the ability to compute and use diffusive fluxes to Tests/GPU/CNS. Default is now to include diffusive terms so the existing inputs files now have cns.do_visc=false. Tests/GPU/CNS/CMakeLists.txt Tests/GPU/CNS/Exec/RT/inputs Tests/GPU/CNS/Exec/RT/inputs-rt Tests/GPU/CNS/Exec/Sod/inputs Tests/GPU/CNS/Exec/Sod/inputs-rt Tests/GPU/CNS/Source/CNS.H Tests/GPU/CNS/Source/CNS.cpp Tests/GPU/CNS/Source/CNS_advance.cpp Tests/GPU/CNS/Source/CNS_index_macros.H Tests/GPU/CNS/Source/CNS_parm.H Tests/GPU/CNS/Source/diffusion/CNS_diffusion_K.H Tests/GPU/CNS/Source/diffusion/Make.package Tests/GPU/CNS/Source/hydro/CNS_hydro_K.H commit 388ce7e96d74eef0f082f19d95e16647a2d34ccb Author: Axel Huebl Date: Mon Jul 19 08:44:23 2021 -0700 CI: IntelLLVM known to CMake (#2136) ## Summary `IntelLLVM` is now a recognized CMake compiler, so we don't need to add a Clang-ish identification anymore. Also fix a CMake -Wdev warning (DPC++/SYCL) - beta08 work-around: `-mlong-double-64` (https://github.com/intel/llvm/issues/2187) -> still needed! - beta09 work-around: `-fno-sycl-early-optimizations` (link to upstream issue?) -> keep for now, RT tests pending to verify fix ## Additional background setting the CXX compiler ID was a work-around for the 2021.1 DPC++ release / CMake 3.19.0-3.19.1 https://gitlab.kitware.com/cmake/cmake/-/issues/21551#note_869580 .github/workflows/intel.yml Tools/CMake/AMReXSYCL.cmake commit 98c9e7197c5c94135fd9bb070aeea12b8fb58629 Author: Andrew Myers Date: Fri Jul 16 12:21:41 2021 -0700 Add default move constructors / assignment operators for other pc types. (#2174) Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_TracerParticles.H commit 6e27136f76280e565d30ff654031d1b1ac683171 Author: Weiqun Zhang Date: Thu Jul 15 16:58:58 2021 -0700 Fix warning with Intel icpc (#2173) * Fix warning with Intel icpc The classical EDG based Intel compiler gives warning about `fallthrough` when `-std=c++14` is used, even though it passes the feature test for `fallthrough`. * CI: Add Intel ICC/ICPC/IFORT Co-authored-by: Axel Huebl .github/workflows/intel.yml Src/Base/AMReX_Extension.H commit 6a49159efff4f7d742b604ffbe61f39ab4fa1a0a Author: Weiqun Zhang Date: Thu Jul 15 14:15:47 2021 -0700 Gradient in WarpX EB nodal solver (#2171) Implement function that computes the gradient of potential in the EB nodal solver for WarpX. Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp commit 1be4862218d034349ae6f24d948ad39abeac1a2c Author: Candace Gilet Date: Wed Jul 14 22:12:18 2021 -0400 Update MacProjector::project to check if m_umac is nullptr (#2166) * Remove unnecessary temporary and copy. * Update MacProjector::project to check if m_umac is nullptr. If nullptr, don't compute div(umac) or attempt to average down or update umac. * Remove tabs Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit 8a2ddee47fecb368349db09a4505fe753ddeb65d Author: Weiqun Zhang Date: Wed Jul 14 16:56:47 2021 -0700 EB nodal solver (#2145) New EB nodal solver with Dirichlet EB. This uses a finite-difference stencil. It will be used by WarpX's static solver. Currently only Dirichlet and periodic domain BC are supported. Support for Neumann BC at the domain boundary will be added later. Currently, it supports single level only. Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EBDataCollection.cpp Src/EB/AMReX_EBFabFactory.H Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/Make.package commit cc1ceb1188537b0d126be323537f8aaa4294983c Author: Kevin Gott Date: Wed Jul 14 15:57:39 2021 -0700 Fix missing MPI flags in Perlmutter. (#2170) Tools/GNUMake/sites/Make.nersc commit 24a586d89da996195ef81aff2a47c7d2bf0825c7 Author: Andrew Myers Date: Wed Jul 14 14:26:25 2021 -0700 Add ssh instructions to CONTRIBUTING.md (#2168) CONTRIBUTING.md commit 6ab65e7fc0ace3cd116f2b7e2c51a3506e7d4ff1 Author: Weiqun Zhang Date: Tue Jul 13 20:46:31 2021 -0700 More reliable workaround for DPCPP reduction issue (#2167) It seems that the DPC++ reduction issue is due to writing to pinned memory on device. The workaround is to write to device memory and then memcpy. Src/Base/AMReX_Reduce.H commit d9eeb62638eb3352fb4a7eaf6ff93b9f0c668e04 Author: Weiqun Zhang Date: Mon Jul 12 17:32:14 2021 -0700 Workaround for a DPC++ reduction issue (#2165) Src/Base/AMReX_Reduce.H commit cd1f5430be29aed0a85348669394d7fbb8355dba Author: Weiqun Zhang Date: Mon Jul 12 08:45:53 2021 -0700 add ParmParse::remove (#2160) Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit 0ba4a8f628d240642431b537fd62e809ddf6aa41 Author: Weiqun Zhang Date: Mon Jul 12 08:45:18 2021 -0700 Minor optimization of reducing 4 bytes (#2161) Box length is used in many kernels. Because the length in z-direction is not needed, we only need 2 integers, not Dim3 that has 3 integers. Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_BaseFabUtility.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_Reduce.H commit 135010c8baab688b9a190f008e5098a50a91edb2 Author: Weiqun Zhang Date: Mon Jul 12 08:44:52 2021 -0700 Remove MFGhostIter (#2162) It was added several years in an experiment of overlapping communication and computation. It has never been used. Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit 30401db6fc699259b388c09ef5a5cbcc40f04639 Author: Weiqun Zhang Date: Mon Jul 12 08:44:24 2021 -0700 Fix typos with the help of codespell (#2163) .github/workflows/docker/sensei/Dockerfile CHANGES Docs/Notes/Readme.typecheck Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Docs/sphinx_documentation/source/GridCreation.rst Docs/sphinx_documentation/source/SWFFT.rst Docs/sphinx_documentation/source/Testing.rst Docs/sphinx_documentation/source/Visualization.rst Src/Amr/AMReX_Amr.H Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateDescriptor.H Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_TagBox.H Src/AmrCore/CMakeLists.txt Src/Base/AMReX_Arena.H Src/Base/AMReX_AsyncOut.H Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BaseUmap_nd.f90 Src/Base/AMReX_Box.H Src/Base/AMReX_BoxIterator.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_DArena.cpp Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FILCC_3D.F90 Src/Base/AMReX_FPC.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabConv.cpp Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_IntVect.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_Partition.H Src/Base/AMReX_Utility.cpp Src/Base/AMReX_Vector.H Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_parstream.cpp Src/Base/Parser/AMReX_IParser_Y.H Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Boundary/AMReX_BndryData.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_EB_slopes_K.H Src/EB/AMReX_EB_utils.cpp Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp Src/Extern/Conduit/AMReX_Conduit_Blueprint_ParticlesI.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/Extern/ProfParser/AMReX_BLWritePlotFile.cpp Src/Extern/ProfParser/AMReX_CommProfStats.cpp Src/Extern/SENSEI/AMReX_AmrDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrParticleInSituBridge.H Src/Extern/SENSEI/AMReX_InSituBridge.H Src/Extern/SENSEI/AMReX_InSituUtils.H Src/Extern/SENSEI/AMReX_ParticleInSituBridge.H Src/Extern/SWFFT/CheckDecomposition.c Src/Extern/SWFFT/Dfft.H Src/Extern/SWFFT/README Src/Extern/SWFFT/distribution.c Src/Extern/amrdata/AMReX_DataServices.cpp Src/F_Interfaces/Base/AMReX_mpi_reduce_int.F90 Src/F_Interfaces/Base/AMReX_mpi_reduce_real.F90 Src/LinearSolvers/MLMG/AMReX_MLALaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_ParticleArray.H Src/Particle/AMReX_Particles.H Src/SDC/AMReX_SDCstruct.cpp Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Amr/Advection_AmrLevel/Source/Src_2d/Adv_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Adv_3d.f90 Tests/DivFreePatch/main.cpp Tests/EB/CNS/Exec/Sod/inputs Tests/EB/CNS/Source/diffusion/cns_eb_diff_wall.F90 Tests/FortranInterface/Advection_F/Source/Src_2d/advect_2d_mod.F90 Tests/FortranInterface/Advection_F/Source/Src_3d/advect_3d_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/Src_2d/advect_2d_mod.F90 Tests/GPU/CNS/Source/CNS_bcfill.cpp Tests/GPU/Fuse/GNUmakefile Tests/LinearSolvers/ABecLaplacian_C/scalingtest/results.org Tests/LinearSolvers/ABecLaplacian_F/mytest.F90 Tests/LinearSolvers/LeastSquares/README.md Tests/LinearSolvers/MAC_Projection_EB/main.cpp Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Slopes/README.md Tools/CMake/AMReXCUDAOptions.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/CMake/AMReXTypecheck.cmake Tools/C_scripts/makebuildinfo_C.py Tools/C_scripts/mkdep Tools/C_scripts/strip72 Tools/C_util/AppendToPlotFile.cpp Tools/C_util/AugmentPlotfile/AugmentPlotfile.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/README Tools/F_scripts/extract.parallel Tools/F_scripts/f90doc/typing.pl Tools/F_scripts/write_probin.py Tools/GNUMake/Make.rules Tools/GNUMake/comps/cray.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/comps/nvhpc.mak Tools/GNUMake/comps/pgi.mak Tools/GNUMake/sites/Make.nersc Tools/GNUMake/sites/Make.nrel Tools/Migration/step-7-bindc/bindc.sh Tools/Postprocessing/python/dumpparthistory.py Tools/Release/ppCleanup.py Tools/Release/ppCleanupDir.py Tools/Release/release.py Tools/libamrex/mkpkgconfig.py commit bef02a4c6394d24578ceed82ab9d73ede5a637e2 Author: Weiqun Zhang Date: Sat Jul 10 15:09:36 2021 -0700 Parser Precision (#2154) Use double internally. If the arguments are all floats, the final result will be converted to float. This is necessary to avoid overflow in some WarpX single precision tests. Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Exe.H Src/Base/Parser/AMReX_Parser_Exe.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/amrex_parser.tab.h Src/Base/Parser/amrex_parser.y commit f8c2fe41d3187e69f747816f1f6ab0d38335c11e Author: Ann Almgren Date: Sat Jul 10 14:11:49 2021 -0700 Add to particlemeshtest (#2159) Tests/Particles/ParticleMeshMultiLevel/CMakeLists.txt Tests/Particles/ParticleMeshMultiLevel/Make.package Tests/Particles/ParticleMeshMultiLevel/main.cpp Tests/Particles/ParticleMeshMultiLevel/mypc.H Tests/Particles/ParticleMeshMultiLevel/trilinear_deposition_K.H commit d325cf8eab0d78bb356d5bcf6c90b9bef7ef63ae Author: Weiqun Zhang Date: Sat Jul 10 14:09:06 2021 -0700 IParser: Integer Parser (#2158) Add a parser for integers. Because there is rounding in integer division, it's not safe to use the real number parser for integer. The two parsers have a lot of similarity, but floating point number specific functions (e.g., `sqrt`, `sin`, etc.) are not supported in `IParser`. In addition to `/` whose result truncates towards zero, the integer parser also supports `//` whose result truncates towards negative infinity. Docs/sphinx_documentation/source/Basics.rst Src/Base/CMakeLists.txt Src/Base/Make.package Src/Base/Parser/AMReX_IParser.H Src/Base/Parser/AMReX_IParser.cpp Src/Base/Parser/AMReX_IParser_Exe.H Src/Base/Parser/AMReX_IParser_Exe.cpp Src/Base/Parser/AMReX_IParser_Y.H Src/Base/Parser/AMReX_IParser_Y.cpp Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/GNUmakefile Src/Base/Parser/amrex_iparser.l Src/Base/Parser/amrex_iparser.lex.cpp Src/Base/Parser/amrex_iparser.lex.h Src/Base/Parser/amrex_iparser.tab.cpp Src/Base/Parser/amrex_iparser.tab.h Src/Base/Parser/amrex_iparser.y Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp Src/Base/Parser/amrex_parser.y Tests/Parser/main.cpp commit fb41ccbb852a89026cda4b1d26eee6f1fab56f9e Author: Ann Almgren Date: Fri Jul 9 14:26:29 2021 -0700 Remove single level assertions for NeighborParticleContainers on GPUs (#2157) * Remove single-level assertions for NeighborParticleContainer on GPUs Src/Particle/AMReX_NeighborParticlesGPUImpl.H commit cebf3df6c3e068a0c4f69b04198c0ba9af94d50a Author: Candace Gilet Date: Fri Jul 9 15:08:59 2021 -0400 Docs: fix typo in author name (#2156) Docs/sphinx_documentation/source/GridCreation.rst commit b0ffd259a5bf7eba82fc47a3871ae6b61c8860a5 Author: Erik Date: Thu Jul 8 15:57:11 2021 -0400 correct sp typo (#2153) Docs/sphinx_documentation/source/GettingStarted.rst commit 91ea9a0243b48a5d66c0c09e18df16e2db60e27f Author: Erik Schnetter Date: Thu Jul 8 13:44:41 2021 -0400 Disable `jn` on Windows (#2152) Disable `jn` on Windows – not just when `dpcpp` is used, but also when building with other compilers in MINGW. Src/Base/Parser/AMReX_Parser_Y.H commit 6b2ef0b507a24d2d90a04c81895c33aaf315454a Author: Mu-Hua Chien Date: Thu Jul 8 11:24:46 2021 -0400 add an option for hypre solver for Tests/Linearsolvers/NodalPoisson (#2139) Add input-rt.hypre for NodalPoisson example Tests/LinearSolvers/NodalPoisson/GNUmakefile Tests/LinearSolvers/NodalPoisson/MyTest.H Tests/LinearSolvers/NodalPoisson/MyTest.cpp Tests/LinearSolvers/NodalPoisson/inputs-rt.hypre commit 49b6ad4a85f09f86e7601a07744ed6e63f29a164 Author: Erik Date: Thu Jul 8 11:20:46 2021 -0400 Doxygen Side-Bar Navigation, Doxygen Logo and Mainpage Content (#2150) Enables side-bar navigation for Doxygen, and adds logo and content to main splash page. Docs/Doxygen/customdoxygen.css Docs/Doxygen/doxygen.conf Docs/Doxygen/footer.html Docs/Doxygen/header.html Docs/Doxygen/main.dox Docs/Doxygen/resize_AMReX_logo.png commit e7591aabfe4578ddf9339688632700a70ac17af0 Author: Weiqun Zhang Date: Wed Jul 7 20:38:29 2021 -0700 Allow empty string Parser (#2151) Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp commit 1d6b01af423bf0f39d9efba8ac59ebff75179de0 Author: Erik Date: Wed Jul 7 12:13:38 2021 -0400 Doxygen comment formatting (#2148) Src/Particle/AMReX_Particles.H commit fffd3d976c7e8a3fa1e9a59e35e7cbfbd37a2493 Author: Andrew Myers Date: Tue Jul 6 21:53:51 2021 -0700 Fix the size of the BC Vector in the multi-level version of ParticleToMesh. (#2149) Src/AmrCore/AMReX_AmrParticles.H commit 7c185502a57fb263f0819635c14f7ee84e96f0b7 Author: Andrew Myers Date: Tue Jul 6 10:56:01 2021 -0700 Make ParticleToMesh and MeshToParticle support iMultifab. (#2137) Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_iMultiFab.cpp Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleMesh.H Tests/Particles/ParticleMesh/main.cpp commit 901bac18bdc3635b992121ba1023b0b9d14f8729 Author: Nicholas Brunhart-Lupo Date: Fri Jul 2 15:42:16 2021 -0600 Fix crashes with Conduit Blueprint Code (#2144) ## Summary This PR contributes two fixes to the Conduit Blueprint code. ### Nestsets fix (Issue AMReX-Codes/amrex#2135) The first fix here is within `Nestsets`, which resolves a one-past the end access on `ref_ratio`. The problematic lines are 116 to 120, of the form `window["ratio/i"] = ref_ratio[level+1][0];` For some inputs, this is one-past the end of the `ref_ratio` array. It was advised in https://github.com/AMReX-Codes/amrex/issues/2135#issuecomment-872692909 to resolve this by removing the `+1`. ### MultiLevelToBlueprint fix (Issue AMReX-Codes/amrex#2133) The second fix resolves a bug inside `MultiLevelToBlueprint`, where the `box_offsets` array is not built correctly. The problematic code is here: https://github.com/AMReX-Codes/amrex/blob/14dde9aaf741f84cf85adb67a7cdc5bef4c0764a/Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp#L363-L376 During the construction of the `box_offsets` array the first round of the loop at line 364, with `i == 0` will size the `box_offsets` array to 1. Subsequent rounds through the loop will attempt to write to non-existent slots of the vector here: https://github.com/AMReX-Codes/amrex/blob/14dde9aaf741f84cf85adb67a7cdc5bef4c0764a/Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp#L374 This fix sizes `box_offsets` before the loop starts, and changes all accesses to the vector by index, not by appending. This appears to be the right fix as advised here: https://github.com/AMReX-Codes/amrex/issues/2133#issuecomment-871501258 Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 85021c840b13c0f89c0db7039cda66585e9b239a Author: Kevin Gott Date: Fri Jul 2 08:30:43 2021 -0700 Perlmutter Build: Adjust to allow CPU build + cleanup. (#2143) Tested on PrgEnv-nvidia, PrgEnv-gnu for USE_CUDA={TRUE,FALSE} and BL_NO_FORT={TRUE,FALSE}. Tools/GNUMake/sites/Make.nersc commit ab69825ddee58a2ca834a2437376a05c0b8a76af Author: Weiqun Zhang Date: Thu Jul 1 17:06:29 2021 -0700 Fix mismatched tag warning (#2141) AmrInfo is defined as a struct, and it was declared as class in forward declaration. This is valid C++. But the Intel oneAPI compiler gives a warning that this may result in linker errors under the Microsoft C++ ABI. Src/AmrCore/AMReX_AmrCoreFwd.H commit 83087f64af364dd7e80a191fc905a890cf1f635e Author: Weiqun Zhang Date: Thu Jul 1 17:05:57 2021 -0700 Default values for members of ParserExecutor (#2142) Set the default values of pointers inside ParserExecutor to nullptr and add a conversion operator so that it can be tested for null. Src/Base/Parser/AMReX_Parser.H commit d7360e84bb68505316645c346abd1bbe3662d7ce Author: Candace Gilet Date: Thu Jul 1 13:37:23 2021 -0400 Docs: Add labels to chapters that were missing one. (#2134) Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/GettingStarted_Chapter.rst Docs/sphinx_documentation/source/ManagingGridHierarchy_Chapter.rst commit bee2568d831aa1c84e6a9fe24c3e69f3ec5b49a0 Author: Axel Huebl Date: Thu Jul 1 10:36:08 2021 -0700 CMake: HIP Modernizing & RDC (#2031) ## Summary Add `-fgpu-rdc` flags to HIP if requested via `AMReX_GPU_RDC`. Modernize HIP logic with recommended targets that avoid the flaky `hipcc` compiler scripts: https://rocmdocs.amd.com/en/latest/Installation_Guide/Using-CMake-with-AMD-ROCm.html#using-hip-in-cmake Add support for AMDs `clang++`/`clang` compiler for HIP instead of using the legacy `hipcc` perl wrapper as C++ compiler. This also increases support towards Cray compiler wrappers, which also refer to `clang++`/`clang` underneath (Spock/OLCF). Close #1688 ## Additional background Relocatable-device-code (RDC) flags are needed for `extern` device variable support (for codes that use global variables on device). Also needed when linking with Ascent. Follow-up to #2029 With HIP GPU RDC, static libs emitting & linking does get more fancy: https://github.com/ROCmSoftwarePlatform/rccl/pull/260 .github/workflows/hip.yml Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXParallelBackends.cmake commit 5c83ba4d6f5f9c4db9812bbbf0b032f94aac7205 Author: Weiqun Zhang Date: Thu Jul 1 08:10:58 2021 -0700 Update CHANGES for 21.07 (#2138) CHANGES commit 14dde9aaf741f84cf85adb67a7cdc5bef4c0764a Author: Weiqun Zhang Date: Mon Jun 28 20:29:50 2021 -0700 Parser::compileHost (#2132) Add a function that compiles the parser expression into host code only. Src/Base/Parser/AMReX_Parser.H commit 97d907762f776ff4400868029fcc53709703291c Author: Andrew Myers Date: Mon Jun 28 14:03:43 2021 -0700 Fix MR bug in NL construction - the number of cells needs to be increased by a factor of ref_fac. (#2131) Src/Particle/AMReX_NeighborParticlesI.H commit fd1fd11302601a085b2e7ff01f07d36ffa0add66 Author: Weiqun Zhang Date: Mon Jun 28 10:11:55 2021 -0700 EB Implicit Function using Parser (#2129) Add ParserIF to EB. One can use a parser function to generate geometry. For example, the following ParmParse parameters can be used in 3D to place two cylinders with a radius of 0.1 at (-0.5,0,) and (0.5,0,). ``` eb2.geom_type = parser eb2.parser_function = "max(0.01-(x+0.5)^2-y^2, 0.01-(x-0.5)^2-y^2)" ``` Put the internal data of Parser in a shared_ptr for memory safety. Docs/sphinx_documentation/source/Basics.rst Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/EB/AMReX_EB2.cpp Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_IF.H Src/EB/AMReX_EB2_IF_Parser.H Src/EB/CMakeLists.txt Src/EB/Make.package commit 55cb40092910183729ca81e6eef3d0ce386af317 Author: Weiqun Zhang Date: Sun Jun 27 10:52:09 2021 -0700 AMReX Version (#2128) In git fails to obtain AMReX version in the make system, grep for the version number in the CHANGES file. CHANGES Tools/GNUMake/Make.defs commit 6ca4306167ed432399a41aaa7a4a80a3f15f65e9 Author: Candace Gilet Date: Fri Jun 25 21:55:09 2021 -0400 Fix typo in 3d NodeLaplacian. (#2126) This fixes an error for cases with for dz != dy Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H commit 81535291686e88a4d2cc3bf36a77bca6a6bf48c3 Author: Weiqun Zhang Date: Fri Jun 25 14:48:53 2021 -0700 Parser (#2121) Add amrex::Parser that can be used at runtime to evaluate math expressions given by strings. It works on both host and device. This is moved from WarpX with additional performance optimization of eliminating recursion. Support for local automatic variables are added. .github/workflows/linux.yml .github/workflows/style/check_tabs.sh .github/workflows/style/check_trailing_whitespaces.sh Docs/sphinx_documentation/source/Basics.rst Src/Base/AMReX_BaseFwd.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/Base/Parser/AMReX_Parser.H Src/Base/Parser/AMReX_Parser.cpp Src/Base/Parser/AMReX_Parser_Exe.H Src/Base/Parser/AMReX_Parser_Exe.cpp Src/Base/Parser/AMReX_Parser_Y.H Src/Base/Parser/AMReX_Parser_Y.cpp Src/Base/Parser/GNUmakefile Src/Base/Parser/README Src/Base/Parser/amrex_parser.l Src/Base/Parser/amrex_parser.lex.cpp Src/Base/Parser/amrex_parser.lex.h Src/Base/Parser/amrex_parser.tab.cpp Src/Base/Parser/amrex_parser.tab.h Src/Base/Parser/amrex_parser.y Tests/CMakeLists.txt Tests/Parser/CMakeLists.txt Tests/Parser/GNUmakefile Tests/Parser/Make.package Tests/Parser/main.cpp commit 67b0794d02c2f13e2c0156b752c486da6863deea Author: Andrew Myers Date: Thu Jun 24 17:52:45 2021 -0700 Remove incorrect call to coarsen. (#2125) Src/Particle/AMReX_NeighborParticlesI.H commit 8575439bdbc52bda3eb38f12660a5d3c1e0ae584 Author: Weiqun Zhang Date: Wed Jun 23 22:59:37 2021 -0700 Avoid std inclusive and exclusive scan for _GLIBCXX_RELEASE < 10 (#2123) Src/Base/AMReX_Scan.H commit 5db3b4fff09400a7c040b6792e06b2d9051fd01a Author: Andrew Myers Date: Tue Jun 22 12:37:48 2021 -0700 Fix Arena docs. (#2118) * Fix Arena docs. * remove device arena from printstats output in docs Docs/sphinx_documentation/source/GPU.rst commit 91a4ff2ce1f0a950ee7092697ea48f9e814391d3 Author: c-wetterer-nelson <78513275+c-wetterer-nelson@users.noreply.github.com> Date: Tue Jun 22 14:28:39 2021 -0400 Add SENSEI Adaptor for Particles Containers (#2016) Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Extern/SENSEI/AMReX_AmrDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrInSituBridge.H Src/Extern/SENSEI/AMReX_AmrInSituBridge.cpp Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.cpp Src/Extern/SENSEI/AMReX_AmrMeshInSituBridge.H Src/Extern/SENSEI/AMReX_AmrMeshInSituBridge.cpp Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_AmrParticleInSituBridge.H Src/Extern/SENSEI/AMReX_InSituBridge.H Src/Extern/SENSEI/AMReX_InSituBridge.cpp Src/Extern/SENSEI/AMReX_InSituUtils.H Src/Extern/SENSEI/AMReX_InSituUtils.cpp Src/Extern/SENSEI/AMReX_ParticleDataAdaptor.H Src/Extern/SENSEI/AMReX_ParticleDataAdaptorI.H Src/Extern/SENSEI/AMReX_ParticleInSituBridge.H Src/Extern/SENSEI/CMakeLists.txt Src/Extern/SENSEI/Make.package Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs Tools/GNUMake/tools/Make.sensei Tools/libamrex/configure.py commit c797f6b6979aea63cc6bda574765c29ba89e7b9c Author: kwryankrattiger <80296582+kwryankrattiger@users.noreply.github.com> Date: Thu Jun 17 15:45:09 2021 -0500 CI: Add job to test SENSEI build (#2115) .github/workflows/docker/sensei/Dockerfile .github/workflows/docker/sensei/install_deps.sh .github/workflows/docker/sensei/install_sensei.sh .github/workflows/docker/sensei/install_vtk_minimal.sh .github/workflows/docker/sensei/tools.sh .github/workflows/linux.yml commit 4eb054cf1d37f3e2fa79f01a17116072ccfbbcb8 Author: Weiqun Zhang Date: Thu Jun 17 10:22:09 2021 -0700 Fix warning in algoim (#2117) Split the constructor of ImplicitIntegral into two templates to avoid an unreachable loop warning. Src/EB/AMReX_algoim_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Tools/GNUMake/Make.defs commit 04464a4f6d1201dfd01fd248c3ad64c5a6809a74 Author: Andy Nonaka Date: Thu Jun 17 09:46:31 2021 -0700 Enhance DiffSameGrid2.cpp (#2114) This utility, when using L0 norm, now outputs an additional column with the absolute error divided by the maximum value on the coarse grid. This is useful for comparing/verifying data with local regions of very small (close to zero in a machine precision sense) values that may fluctuate based on roundoff errors. Tools/C_util/Convergence/DiffSameGrid2.cpp commit 501626901f500b45fce5102c01d15cd8f74c7187 Author: Andrew Myers Date: Wed Jun 16 15:49:57 2021 -0700 Don't rely on managed memory in ParticleCopyPlan (#2116) Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleCommunication.cpp commit baf0d4cd96fa81ace1c964a2e5601685e97d41f6 Author: Axel Huebl Date: Tue Jun 15 16:10:11 2021 -0700 HIP: Increase to C++17 (#2113) Bump HIP requirements to C++17 or newer (like DPC++). At the moment, we only test against AMD's clang++/hipcc, which supports C++17 anyway. .github/workflows/hip.yml Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReX_Config.cmake Tools/GNUMake/comps/hip.mak commit f7ae247580204b1f607e22b09c046120d2cc6e38 Author: Weiqun Zhang Date: Tue Jun 15 14:26:44 2021 -0700 Port remaining Fortran and CPU kernels in FluxRegister to GPU (#2112) Src/AmrCore/AMReX_FLUXREG_F.H Src/AmrCore/AMReX_FLUXREG_nd.F90 Src/AmrCore/AMReX_FluxRegister.H Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package Src/Base/AMReX_FBI.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H commit 659ab43420a2c4a918cda2702910771fbbc395d3 Author: Weiqun Zhang Date: Mon Jun 14 18:01:42 2021 -0700 TinyProfiler: Add spaces to print (#2110) Add more spaces so that the output looks nicer when there are multiple layers of profilers. Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit 8182a3a630e580ed9158bd1da33276380f170d41 Author: Andrew Myers Date: Mon Jun 14 14:33:08 2021 -0700 Reimplement partitionParticlesByDest (#2109) Src/Particle/AMReX_ParticleUtil.H commit d7ff203305e8c8605ca6f92e9212acf717da56de Author: Weiqun Zhang Date: Mon Jun 14 12:10:05 2021 -0700 GpuArray template parameter (#2108) Use unsigned int as the type for size instead of std::size_t so that the forward declaration does not need to include another header. Src/Base/AMReX_Array.H Src/Base/AMReX_BaseFwd.H commit c221a567dbd8cadd474146d9cc36ba5152178fb1 Author: Weiqun Zhang Date: Mon Jun 14 10:38:00 2021 -0700 Fix a bug in in-place ExclusiveSum (#2105) For ExclusiveSum versions using memcpy to get the total sum, we need to copy the last element of the input data to the host before the scan so that it's safe for in-place scan. Also use std::exclusive_scan and inclusive_scan on CPU if available. Src/Base/AMReX_Scan.H commit 8357d530311235e3f76899051ca8d44a51b971c0 Author: Luca Fedeli Date: Mon Jun 14 17:18:20 2021 +0200 add missing class (#2106) Src/Base/AMReX_BaseFwd.H commit fc9bc1e3efdaead0253d0a55dcf1abb7abc06bd7 Author: Weiqun Zhang Date: Sun Jun 13 12:42:38 2021 -0700 Add more warnings to CI and fix warnings (#2104) Also fixed a bug in device version of `Random_int`. .github/workflows/cuda.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml .github/workflows/macos.yml Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_Random.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/MultiBlock/Advection/main.cpp Tests/MultiBlock/IndexType/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/ParallelContext/main.cpp Tests/Particles/ParticleArray/main.cpp Tests/Particles/ParticleIterator/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/SparseBins/main.cpp Tests/Particles/TypeDescriptor/main.cpp Tools/CMake/AMReXParallelBackends.cmake Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/hip.mak Tools/Plotfile/fextract.cpp commit 2ff4519b6e16c1f0906e637559f62d183158734c Author: Weiqun Zhang Date: Sat Jun 12 08:07:04 2021 -0700 Missing defined in preprocessing (#2102) Src/Base/AMReX_FPC.cpp commit 2a676f8a25e6cd75ed2e3936bd0cd2e30141363f Author: Michael Zingale Date: Fri Jun 11 20:42:24 2021 -0400 add AUTO_BUILD_SOURCES to the dependencies for GNU make (#2101) AUTO_BUILD_SOURCES can be filled by an application code with the source files that are created automatically at build time to ensure they are made before compilation / dependency resolution. Tools/GNUMake/Make.rules commit 64e41a8d314fc28f6d6473095ac5bdcd256c445f Author: Weiqun Zhang Date: Fri Jun 11 17:20:08 2021 -0700 Add a number of forward declaration headers (#2100) Src/Amr/AMReX_AmrFwd.H Src/Amr/CMakeLists.txt Src/Amr/Make.package Src/AmrCore/AMReX_AmrCoreFwd.H Src/AmrCore/CMakeLists.txt Src/AmrCore/Make.package Src/Base/AMReX_Array4.H Src/Base/AMReX_BaseFwd.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/Boundary/AMReX_BoundaryFwd.H Src/Boundary/CMakeLists.txt Src/Boundary/Make.package commit 4cf09ce1a3eb2c06460866af04b0ec420300f1d0 Author: Axel Huebl Date: Wed Jun 9 20:01:14 2021 -0700 Particle Device Members: Inline (#2097) Member functions that are not `operator()` are not inlined by default. I think we should inline those getters, since they are too tiny / too deep in hot loops to make sense to indirect. (The compiler likely figured this out on its own already, at least I saw no immediate difference in a massive WarpX kernel with NVCC and looking at ptxas info.) Src/Particle/AMReX_Particle.H commit 7eaf1ee13726cfb689d4f0bffe3b5d056b09eceb Author: Axel Huebl Date: Wed Jun 9 15:19:21 2021 -0700 CI: Split Linux Workflows (#2096) Split workflows into - linux (CPU) - cuda - intel (DPC++/ICC/ICX) - hip workflows. Individual workflows can be individually restarted and this increases order a bit as well. .github/workflows/cuda.yml .github/workflows/hip.yml .github/workflows/intel.yml .github/workflows/linux.yml commit e6c52ec95486d8d26ec679d9aea561fa6266d844 Author: Axel Huebl Date: Wed Jun 9 09:17:03 2021 -0700 CMake 3.20+ CUDA: Fast Math (#2094) Honor CUDA fast math option `AMReX_CUDA_FASTMATH` (default: ON) in modernized CUDA CMake logic. Tools/CMake/AMReXParallelBackends.cmake commit 56de38905af1fe6b1eff8fe390de50bcc65449c2 Author: Axel Huebl Date: Mon Jun 7 19:20:17 2021 -0700 CMake 3.20+ CUDA: Shell-Escape --Werror (#2093) ## Summary When adding a list of flags to CMake targets, duplicated flags get by default deduplicated. This caused with our new CMake 3.20+ CUDA logic #2012 that this gets passed to the linker: `--Werror ext-lambda-captures-this cross-execution-space-call`. The CMake `SHELL:` prefix is exactly made for this instance, keeping flags that belong together also over transformations like this together: https://cmake.org/cmake/help/latest/command/target_compile_options.html#arguments Since we injected directly into the global `CMAKE_CUDA_FLAGS` flags for earlier CMake versions, this deduplication logic does not apply and thus we have not experienced it earlier. ## Additional background Seen first with CI in AMReX Tutorials (e.g. https://github.com/AMReX-Codes/amrex-tutorials/pull/12). Tools/CMake/AMReXParallelBackends.cmake commit 88b50cf2e427c5358bed05d792f557f5da303028 Author: Wileam Y. Phan <50928756+wyphan@users.noreply.github.com> Date: Mon Jun 7 22:04:23 2021 -0400 Add link to "Building with CMake" page in INSTALL file (#2092) Add a link to "Building with CMake" page in the AMReX documentation web page into the INSTALL file. INSTALL commit 53614b4a5b8bd769918523af10341b22c57a5596 Author: Weiqun Zhang Date: Mon Jun 7 12:18:04 2021 -0700 Remove VecReduce and FabReduce (#2091) They were used for the old atomics based reduction approach. Only PeleC uses VecReduce. The use of VecReduce there can be replaced with Reduce::Sum. Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H commit 79678dd3b8c1ecf92c0aaf896c4da1ab7e297465 Author: Weiqun Zhang Date: Mon Jun 7 12:17:23 2021 -0700 Add more versions of ParallelFor(MF) (#2089) Src/Base/AMReX_Algorithm.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFParallelFor.H Src/Base/AMReX_MFParallelForC.H Src/Base/AMReX_MFParallelForG.H Src/Base/CMakeLists.txt Src/Base/Make.package commit 87e01fe44b483adab034072dc000aed67aa1b5a7 Author: Roberto Porcu <53792251+rporcu@users.noreply.github.com> Date: Fri Jun 4 17:53:15 2021 -0400 Fix indexes bug for particles output in GPU version of packIOData (#2087) Src/Particle/AMReX_WriteBinaryParticleData.H commit 9e3fb2f574453dc13e7e1d03a53f0ce41a9ad0e1 Author: Weiqun Zhang Date: Fri Jun 4 11:32:00 2021 -0700 Remove deprecated Reduce functions (#2086) It includes Min, Max, and MinMax with the signature like `T Sum (N n, U const* v, T init_val, BOP bop)`. The other versions are still available. Src/Base/AMReX_Reduce.H commit eb2fb8eb11d4df88d5b6c04a68f2971a6c3b97f0 Author: Weiqun Zhang Date: Thu Jun 3 21:31:05 2021 -0700 Fix some macro issues in CMake Fortran build (#2084) Include AMReX_Config.H in Fortran files that need preprocessing. Do not pass those definitions already defined in AMReX_Config.H to the Fortran compiler in the cmake build. This fixes the warnings about redefined macros. Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_3D.F90 Src/Base/AMReX_FILCC_1D.F90 Src/Base/AMReX_FILCC_2D.F90 Src/Base/AMReX_FILCC_3D.F90 Src/Base/AMReX_acc_mod.F90 Src/Base/AMReX_filcc_mod.F90 Src/Base/AMReX_fort_mod.F90 Src/Base/AMReX_omp_mod.F90 Src/Boundary/AMReX_LO_UTIL.F90 Src/EB/AMReX_ebcellflag_mod.F90 Src/F_Interfaces/AmrCore/AMReX_fillpatch_mod.F90 Src/F_Interfaces/Base/AMReX_fab_mod.F90 Src/F_Interfaces/Base/AMReX_parallel_mod.F90 Src/F_Interfaces/Particle/AMReX_particlecontainer_mod.F90 Src/LinearSolvers/MLMG/AMReX_MLLinOp_nd.F90 Tests/FortranInterface/Advection_F/Source/compute_dt_mod.F90 Tests/FortranInterface/Advection_F/Source/evolve_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/compute_dt_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/evolve_mod.F90 Tools/CMake/AMReXSetDefines.cmake commit 5fd8671f9a9e7b319b03faae03c1fdfa371626f5 Author: Weiqun Zhang Date: Thu Jun 3 10:06:26 2021 -0700 Disable kernel fusing if gpu rdc is off. (#2083) Because the kernel fusing depends on rdc. Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs commit 5fca8d0268c461720d9665f0aef6a4d25bd11381 Author: Weiqun Zhang Date: Wed Jun 2 21:23:35 2021 -0700 ParallelFor for MultiFab/FabArray (#2073) Add an experimental feature that can be used to launch a single kernel to work on MultiFabs or FabArrays. An example is shown below, where `mf` is a MultiFab. ``` auto ma = mf.arrays(); amrex::ParallelFor(mf, [=] AMREX_GPU_DEVICE (int box_no, int i, int j, int k) { ma[box_no](i,j,k) = 1.0; }); Gpu::synchronize(); ``` It's still unclear how we handle tiling and OpenMP and how it performs on CPU. However, there is strong evidence that this improves performance for GPU runs when there are small boxes. The purpose of this PR is to get the feature into the AMReX repo first and do further tuning later. Since the new function is not used at all, it should not affect any codes. Src/Base/AMReX_CArena.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFParallelForC.H Src/Base/AMReX_MFParallelForG.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_TypeTraits.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/EB/AMReX_MultiCutFab.H commit 7404b8a26a60e30cb6be4caf991522676c4bc36c Author: Andrew Myers Date: Wed Jun 2 20:34:20 2021 -0700 ParticleContainer-level reduction functions that work on tuples of data at once. (#2082) Src/Particle/AMReX_ParticleReduce.H Tests/Particles/ParticleReduce/main.cpp commit ea9345194adaf8a202a1b3a48d438697d008bdad Author: Axel Huebl Date: Wed Jun 2 20:15:29 2021 -0700 AMReX_PROBINT: Depends on AMReX_AMRLEVEL (#2081) CMake: do not enable if `AMReX_AMRLEVEL` is off anyway. Tools/CMake/AMReXOptions.cmake commit 3f70dd18d624d90161d323e218ccb398f86fecea Author: Weiqun Zhang Date: Wed Jun 2 20:14:41 2021 -0700 Clean up device random engine states (#2080) Src/Base/AMReX_Random.cpp Src/Base/AMReX_RandomEngine.H commit 241e4ef89f36363c2b00bf99141527db420cea4a Author: Weiqun Zhang Date: Wed Jun 2 20:14:12 2021 -0700 Arena aliases (#2078) Make The_Device_Arena and The_Managed_Arena an alias to The_Arena unless the memory allocation type in incompatible. This should help avoid unnecessary memory fragmentation. Src/Base/AMReX_Arena.cpp commit d20237636ccec63dcfb8540cfa681e0ebd893a40 Author: Axel Huebl Date: Wed Jun 2 16:34:29 2021 -0700 Docs: RDC Option (#2079) ## Summary Add a bit more detailed text to the AMReX_GPU_RDC option. This should help people searching for the option and explain the acronym. ## Additional background Did not yet find a way to slip in "separable compilation" as a term as well... Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXOptions.cmake commit f7fd082bc8ab099d42784f69d6fe49dea3d388a7 Author: Axel Huebl Date: Wed Jun 2 13:39:30 2021 -0700 CMake CUDA: Filter out SM Before Pascal (#2077) Filter out unsupported CUDA architectures in CMake, for both legacy and CMake 3.20+ logic. Tools/CMake/AMReXUtils.cmake commit 2acc1caacf687c92e3cb07a54f7e7ba1f4251888 Author: Andrew Myers Date: Wed Jun 2 13:24:22 2021 -0700 fix bug introduced in PR 2063 (#2076) Src/Particle/AMReX_WriteBinaryParticleData.H commit 439663145ebf0d9f5e66caad738b2bd968ed9331 Author: Andrew Myers Date: Wed Jun 2 13:02:54 2021 -0700 fix signed / unsigned comparison warning (#2075) Src/Particle/AMReX_ParticleCommunication.cpp commit c099e342334c01c8fa2055b2e1e806c62bdf3035 Author: Axel Huebl Date: Wed Jun 2 10:36:22 2021 -0700 CMake 3.20+ CUDA: Policy & Include (#2074) ## Summary Do not include the deprecated CUDA setup with CMake 3.20+ and support new policy for CUDA_ARCHITECTURES. Also fix errors of the kind: ``` ERROR: set_target_properties called with incorrect number of arguments. Value in AMREX_CUDA_ARCHS is 53;60;61;70;75;80;86;86 ``` by quoting the semicolon-separated list. ## Additional background Those are regressions to #2012 that appear with our modernized logic in CMake 3.20+ with CUDA. I saw those regressions with [HiPACE++](https://github.com/Hi-PACE/hipace). CMakeLists.txt Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXTargetHelpers.cmake commit e11fa66c9755e2909280930c387020af2849c1cf Author: Weiqun Zhang Date: Tue Jun 1 15:47:46 2021 -0700 GPU RDC Support (#2061) Disable GPU RDC by default. The user can enable it by `USE_GPU_RDC=TRUE` in gnu make and `-DAMReX_GPU_RDC=ON` in cmake. Remove CUDA specific random number functions that rely on rdc. Co-authored-by: Axel Huebl Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_Arena.H Src/Base/AMReX_Random.H Src/Base/AMReX_Random.cpp Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/GNUMake/Make.defs Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/nvcc.mak Tools/Plotfile/AMReX_PPMUtil.cpp commit 70922cb7cb79bc06a8843f932367c72cf7071ef7 Author: Marc Day Date: Tue Jun 1 16:45:03 2021 -0600 Generalize fextract allowing to specify coords perpendicular to slicedir (#2035) Generalize fextract. Allow spec as coordinate + dir, so that 1D slice can be positioned as desired, within constraints of backward compatibility. Tools/Plotfile/fextract.cpp commit 702af37a91c60bf3916306884ef78319beb5b527 Author: Cody Balos Date: Tue Jun 1 13:11:23 2021 -0700 add interface between sundials SUNMemory and the Arena (#1800) ## Summary As @asalmgren @jmsexton03 @mic84 @ajnonaka @dwillcox @gardner48 @cswoodward and I discussed... this PR adds an interface between the AMReX Arena and the SUNDIALS SUNMemory API. This is currently used in Pele and will be used in Nyx also. In general, any AMReX based app that also uses SUNDIALS (and its provided data structures) will likely find this useful when targeting GPUs. CC @drummerdoc ## Additional background In SUNDIALS 5.4.0, the SUNMemory API was introduced to allow users to substitute their own memory management underneath SUNDIALS data structures. In Pele, using the AMReX arena underneath the SUNDIALS data structures results in as much as a 15% speedup. I have tried to structure the code to follow AMReX conventions. Since we also will be bringing in an interface between the MultiFab and the SUNDIALS N_Vector, I have also tried to structure the code so that it will be easy to add that interface in as well. Co-authored-by: Weiqun Zhang Co-authored-by: David Gardner Co-authored-by: Michele Rosso Co-authored-by: Jean M. Sexton Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GPU.rst GNUmakefile.in Src/Base/AMReX.cpp Src/CMakeLists.txt Src/Extern/SUNDIALS/AMReX_SUNMemory.H Src/Extern/SUNDIALS/AMReX_SUNMemory.cpp Src/Extern/SUNDIALS/AMReX_Sundials.H Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs Tools/GNUMake/packages/Make.sundials commit ef1c484bc467486be3cdff6c90884b0352c39ab5 Author: Andrew Myers Date: Tue Jun 1 12:27:43 2021 -0700 Do not rely on managed memory in ParticleContainer::WriteAsciiFile (#2070) This function is mainly used for small runs / debugging and does not provide filters, hence it should be fine just to copy all the particles to pinned host memory before writing them to disk. Src/Particle/AMReX_ParticleIO.H commit ecffb6a400180f2b2497e66f7c655fafd6bd6eb4 Author: Axel Huebl Date: Tue Jun 1 12:03:13 2021 -0700 HIP ROCTX: USE_RPATH Control (#2064) Just rely on USE_RPATH instead for GNUmake. Follow-up to #2057 Tools/GNUMake/comps/hip.mak commit e7a725644a3eb72dba5c9cd74d35eff4d96af8a8 Author: Andrew Myers Date: Tue Jun 1 12:02:08 2021 -0700 Particle plotfile/checkpoint without managed memory. (#2063) Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H commit b5826c021316179174228e56ad74736c8823caba Author: Weiqun Zhang Date: Tue Jun 1 09:50:40 2021 -0700 Support preprocessing of AMREX_SPACEDIM in input files. (#2049) Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit 1aa8d382af3195e96eea4625dc452c832dfedc53 Author: Weiqun Zhang Date: Tue Jun 1 08:06:13 2021 -0700 Update CHANGES for 21.06 (#2072) CHANGES commit 55ad11dae86fca6737e6e7b8c1656100e5a3bead Author: Axel Huebl Date: Fri May 28 13:03:39 2021 -0700 CI: Cover Fortran (#2067) * CI: Cover Fortran It looks like at some point we changed the `AMReX_Fortran` default from ON to OFF and forgot to adjust CI. This re-enables Fortran compile tests. * CMake: Fix Install Smoke Test (Fortran) Avoid erroring out if Fortran is enabled. .github/workflows/linux.yml Tests/CMakeTestInstall/CMakeLists.txt Tools/CMake/AMReXInstallHelpers.cmake commit 1492a91642e47515b688e58047232daa148196e0 Author: Mark Meredith Date: Fri May 28 00:59:42 2021 -0400 Set HIP_LIBRARIES before using (#2065) Tools/CMake/AMReXParallelBackends.cmake commit 52847883765f041e908350c4d2c03799f6924b1c Author: Axel Huebl Date: Thu May 27 18:09:02 2021 -0700 CMake: HIP Compiler - don't fail (#2062) Potentially, users might use `CC` (Cray Prg Envs) and later `clang++` (underlying compiler) as compilers. Only warn if not met. ROCm/Cray environments are currently being developed and we want to keep things flexible for codes trying to use the compiler wrapper instead of hpcc. Tools/CMake/AMReXParallelBackends.cmake commit 2c2c6992b4169613ba0c956a2a2bd120552f446a Author: Jean M. Sexton Date: Thu May 27 15:38:31 2021 -0700 Add roctx ranges to TinyProfiler (#2057) ## Summary Adds roctx markup similarly to existing nvtx markup. ## Additional background New features are controlled by added compile flag USE_ROCTX. This assumes the location of roctracer/libroctracer64 and roctracer/libroctx64 are similar to other ROCm library installations such as rocrand. Example run to generate results.json ``` export HIP_PATH=/path/to/rocm/root cd amrex/Tests/GPU/CNS/Exec/Sod make USE_HIP=TRUE TINY_PROFILE=TRUE USE_ROCTX=TRUE USE_MPI=FALSE NO_CONFIG_CHECKING=TRUE srun -n 1 ${HIP_PATH}/bin/rocprof --hsa-trace --stats --timestamp on --roctx-trace ./CNS3d.*.TPROF.HIP.ex inputs ``` This results.json file can be viewed with a browser as described in the documentation using chrome://tracing/ https://github.com/ROCm-Developer-Tools/rocprofiler/blob/amd-master/doc/rocprof.md#43rd-party-visualization-tools Co-authored-by: Axel Huebl Co-authored-by: Weiqun Zhang .github/workflows/dependencies/dependencies_hip.sh Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/comps/hip.mak commit 01361ca148d01c6416e6e87ba672c83b694f3385 Author: Kevin Gott Date: Thu May 27 14:46:35 2021 -0700 Clean up some unused variables. (#2060) Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Tests/MultiBlock/IndexType/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp commit 7dcbf3946a95a96b05f6c9ca6c2ee4d7fb5e8dca Author: Mark Meredith Date: Thu May 27 17:22:44 2021 -0400 Link with HIP_LIBRARIES (#2055) * Link with HIP_LIBRARIES * Try LINK_LANGUAGE GenExpr * Only link with HIP_LIBRARIES for C++ only config * fix * Add HIP_LIBRARIES for all non-Fortran * CI: HIP no-Fortran Build * ROCm spelling Co-authored-by: Axel Huebl .github/workflows/linux.yml Tools/CMake/AMReXParallelBackends.cmake commit 0afebba6ae751a2b58fe3e1f4e8c05e2b9cf2025 Author: Lucas Esclapez <13371051+esclapez@users.noreply.github.com> Date: Thu May 27 12:46:15 2021 -0700 Extrapolator with nGrow > 1. (#2054) Extend the AMReX_Extrapolator in order to handle more than 1 layer of ghost cells. This is done by repeating the original algorithm several times while increasing the box size by 1 at every iteration. Src/Amr/AMReX_Extrapolater.H Src/Amr/AMReX_Extrapolater.cpp Src/Amr/AMReX_extrapolater_2D_K.H Src/Amr/AMReX_extrapolater_3D_K.H commit db8e47927065fcf283f6aae631f91a07b222876b Author: Kevin Gott Date: Wed May 26 12:40:32 2021 -0700 Perlmutter Make: CUDA_PATH (#2059) * Perlmutter: find CUDA_ROOT or CUDA_PATH. * Cover all bases. Tools/GNUMake/sites/Make.nersc commit affbd9fe2ca8fecab786c465911067ca05a2c225 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Wed May 26 15:29:53 2021 -0400 Fix to correct CUDA flag and add missing libraries in hypre cmake (#2056) * Fix to correct CUDA flag and add missing libraries in hypre cmake * use CUDAToolkit instead of find_library Tools/CMake/AMReXThirdPartyLibraries.cmake commit 4b29e5d85c2d0eb09b21f50a05a7ef590919d18b Author: Kevin Gott Date: Wed May 26 10:12:30 2021 -0700 Perlmutter make (#2058) * Perlmutter make first draft. Allow Pele to start testing. * Keep mpi_nvidia. Tools/GNUMake/sites/Make.nersc commit 33d8bbb1b15009c68bc71ee72c9b067b730148d5 Author: Weiqun Zhang Date: Tue May 25 15:16:38 2021 -0700 Fix compilation of the TensorOverset test (#2053) Tests/LinearSolvers/TensorOverset/MyTest.cpp commit dedbdce8b963167b5ae8a64ca82cf9c8ce6f5d86 Author: Weiqun Zhang Date: Tue May 25 15:16:16 2021 -0700 Remove IndexSequence (#2052) Since we use C++14 now, our own implementation of index sequence can be removed. Src/Base/AMReX_IndexSequence.H Src/Base/AMReX_Tuple.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/EB/AMReX_EB2_IF_Intersection.H Src/EB/AMReX_EB2_IF_Union.H Src/Particle/AMReX_ParticleArray.H commit 12b8518603dd480e01119aa445410d8684858472 Author: Weiqun Zhang Date: Tue May 25 11:49:43 2021 -0700 Use Cray wrappers to get mpi flags on spock (#2051) Tools/GNUMake/sites/Make.olcf commit 0589d3a76098d90e0f7d364234233643cb212699 Author: Mark Meredith Date: Tue May 25 14:30:38 2021 -0400 CMake link with roc::rocprim for HIP (#2050) * CMake link with roc::rocprim for HIP * link with HIP_LIBRARIES * Update Tools/CMake/AMReXParallelBackends.cmake Co-authored-by: Axel Huebl Co-authored-by: Axel Huebl Tools/CMake/AMReXParallelBackends.cmake commit fd5163bb48fdc6b802cf3e9a8d2c861954475e8c Author: Andrew Myers Date: Tue May 25 09:24:32 2021 -0700 Fix PODVector to work without managed memory. (#2048) In a number of places in `amrex::PODVector`, we were always running operations on the host, assuming the presence of managed memory. This fixes these to run either on the host or on the device, depending on the execution policy of the `PODVector`'s `Allocator` (meaning, for managed and device vectors we run on the device, for host and pinned we run on the host). Additionally, this improves the existing test of `PODVector` by passing it `amrex::the_arena_is_managed=0`, which is more sensitive to errors. This fixes an issue reported by @jmsexton03 regarding Nyx and the InitFromBinaryFile capabilities. Src/Base/AMReX_PODVector.H Tests/GPU/Vector/inputs Tests/GPU/Vector/main.cpp commit 11bdf9e45e39070d5072562db633a1da45d36bd6 Author: Andrew Myers Date: Tue May 25 09:13:36 2021 -0700 When building neighbor masks, perform setVal on device so it will work without managed memory. (#2046) This fixes an issue reported by Chris Lishka from Intel re: mfix. Src/Particle/AMReX_NeighborParticlesGPUImpl.H commit e6654533565fad924793bb3515dd1ee086d1bccb Author: Kevin Gott Date: Tue May 25 09:11:52 2021 -0700 Async Output Docs (#2041) Docs/sphinx_documentation/source/IO.rst commit af5a869484df0ee842afcbf231473d1c472499e5 Author: Weiqun Zhang Date: Mon May 24 17:22:51 2021 -0700 Option to set amd gpu targets (#2038) In GNU make, one can set amd gpu targets with make argument AMD_ARCH or environment variable AMREX_AMD_ARCH. Also set MPI wrapper's underlying compiler to hipcc. This is needed if the machine is unknown. For machines known to the gnu make build system, we could either use mpicxx or hipcc -Ipath/to/mpi/include. Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/GNUMake/Make.defs Tools/GNUMake/comps/hip.mak Tools/GNUMake/sites/Make.frontier-coe Tools/GNUMake/sites/Make.olcf commit e28bbc2d08e79ccb9103d38a9df30aac9f335b16 Author: Weiqun Zhang Date: Mon May 24 17:19:55 2021 -0700 Non-GPU build on spock (#2047) Do not link to gpu aware mpi if it's not a GPU build. Tools/CMake/AMReXUtils.cmake Tools/GNUMake/sites/Make.olcf commit 727829666aa490873a747f3d00c2b3c7ddc91508 Author: Axel Huebl Date: Mon May 24 12:34:53 2021 -0700 Work-Around: oneAPI 2021.3.0 SYCL with oneDPL (#2025) Add recommended work-around flags to oneAPI 2021.3.0 SYCL compilers. https://software.intel.com/content/www/us/en/develop/articles/intel-oneapi-dpcpp-library-release-notes.html#inpage-nav-2-3 ``` The use of oneDPL together with the GNU C++ standard library (libstdc++) version 9 or 10 may lead to compilation errors (caused by oneTBB API changes). To overcome these issues, include oneDPL header files before the standard C++ header files, or disable parallel algorithms support in the standard library. For more information, please see Intel® oneAPI Threading Building Blocks (oneTBB) Release Notes. ``` ``` An application using Parallel STL algorithms in libstdc++ versions 9 and 10 may fail to compile due to incompatible interface changes between earlier versions of Threading Building Blocks (TBB) and oneAPI Threading Building Blocks (oneTBB). Disable support for Parallel STL algorithms by defining PSTL_USE_PARALLEL_POLICIES (in libstdc++ 9) or _GLIBCXX_USE_TBB_PAR_BACKEND (in libstdc++ 10) macro to zero before inclusion of the first standard header file in each translation unit. ``` Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in commit 646d5f63445ff46e4df23cbdb1fbe662ca1708fb Author: Axel Huebl Date: Mon May 24 11:20:34 2021 -0700 ParamParse: Find Entries under Prefix (#2043) Extend the "unused params" checks to find generally parameters under a given inputs prefix. Demonstrator & test in: https://github.com/AMReX-Codes/amrex-tutorials/pull/7 Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit 8935b4ca1e8b8ab2e69dc8f1f85c84a276e8efe0 Author: Max Katz Date: Mon May 24 09:14:23 2021 -0700 Add relative gradient to AMRErrorTag (#2044) Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp commit 25b08dc3a46b9b2b96f8e004cb15e1bf67e94bef Author: Weiqun Zhang Date: Fri May 21 10:23:39 2021 -0700 Workaround argument mismatch in MPI reduce for gfortran >= 10 (#2042) In #2040, we added -fallow-argument-mismatch as a workaround. This commit reverts the change and implements an alternative approach that does not require that compiler compiler. We just need to make sure that there are no argument mismatches in the same file. Src/F_Interfaces/Base/AMReX_fi_mpi_mod.F90 Src/F_Interfaces/Base/AMReX_mpi_reduce_int.F90 Src/F_Interfaces/Base/AMReX_mpi_reduce_real.F90 Src/F_Interfaces/Base/AMReX_parallel_mod.F90 Src/F_Interfaces/Base/Make.package Src/F_Interfaces/CMakeLists.txt Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/llvm.mak commit 2255a3c7196ca7ac9225f58edb82c150a2cc0ec9 Author: Andrew Myers Date: Thu May 20 16:17:37 2021 -0700 Wrappers for builtin clz extensions and tests thereof. (#2036) Src/Base/AMReX_Algorithm.H Src/Base/AMReX_Extension.H Tests/CLZ/CMakeLists.txt Tests/CLZ/GNUmakefile Tests/CLZ/Make.package Tests/CLZ/main.cpp Tests/CMakeLists.txt commit 90977cbfe14d79fa391fe4607c5b59513b9f1413 Author: Weiqun Zhang Date: Thu May 20 12:54:13 2021 -0700 Workaround for gfortran 10 (#2040) Starting from gfortran 10, mismatches between actual and dummy argument lists in a single file have been rejected with an error. This causes issues for MPI calls. This commit adds -fallow-argument-mismatch as a workaround. Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/llvm.mak commit 2bbfef42fcd4b5d4386a6145f48eddab7692f907 Author: Axel Huebl Date: Thu May 20 10:27:06 2021 -0700 HIP: AMREX_AMD_ARCH Env Variable (#2034) Adding an AMREX_AMD_ARCH env variable to control the AMD GPU arch from user environments & profiles. Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReXOptions.cmake commit 16714342b3c08a69438313a9007f93d8c5b7211b Author: Weiqun Zhang Date: Tue May 18 09:50:30 2021 -0700 Add spock support to GNU make (#2028) Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit a1faf316ff6556aabf76fd6d605bf4a59d6c0f59 Author: Weiqun Zhang Date: Tue May 18 08:32:04 2021 -0700 nvcc --display-error-number --diag-error errNum (#2033) Since 11.2, nvcc can display a diagnostic number for any warning message and it can turn specified warnings into errors These two new flags are added to the build system, and it's now an error to write a __device__ variable in a host function. Co-authored-by: Michele Rosso Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReX_SetupCUDA.cmake Tools/GNUMake/comps/nvcc.mak commit aff93b8a26348bb5b42d6af9c3a27ad44e9389f9 Author: Andrew Myers Date: Tue May 18 08:21:08 2021 -0700 Build neighbor masks without managed memory (#2030) Construct these vectors on the host then copy to the device for platforms without managed memory. Src/Particle/AMReX_NeighborParticlesGPUImpl.H commit 79f79aae9f5cc21c09b0353bc0612f41daed4bad Author: Weiqun Zhang Date: Mon May 17 19:13:16 2021 -0700 Option to use oneDPL (#2027) If both USE_ONEDPL and USE_DPCPP are TRUE in GNU make, use oneDPL for the pointer based inclusive and exclusive sum. Also updated the memory fence calls and fixed the compilation for the scan of long for dpcpp. Src/Base/AMReX_FBI.H Src/Base/AMReX_Scan.H Tools/GNUMake/Make.defs Tools/GNUMake/comps/dpcpp.mak commit c6c493bb458515e3ba4b856af097562dcb49a19b Author: Michele Rosso Date: Mon May 17 18:28:54 2021 -0700 CMake: add support for CMake >= 3.20 (#2012) * CMake: group CUDA options in a standalone file * CMake: continue enabling new CUDA support * CMake: make set_mininum_cxx_compiler_version work for a generic language * CMake: rename AMReX_Utils.cmake -> AMReXUtils.cmake * CMake: fix bug * CMake: get rid of NVCC_VERSION_MAJOR and NVCC_VERSION_MINOR * CMake: some upgrades * CMake: upgrade CUDA support * CMake: fix bug * CMake: forgot to guard a set_target_property * CMake: handle CMAKE_CUDA_ARCHITECTURES * CMake: add CUDA flags * CMake: bump up minimum GCC version to 5.1. * CMake: add print for debugging * CMake: add support for PLATFORM_ID genex * Revert "CMake: add print for debugging" This reverts commit 146f645ebae1ac5525e1f5f11cbd167e479dd00f. * CMake: evaluate SHELL genex * CMake: fix printing out of CUDA flags * CMake: fix bug * Update Doc * Update Docs/sphinx_documentation/source/GPU.rst Co-authored-by: Axel Huebl Co-authored-by: Axel Huebl CMakeLists.txt Docs/sphinx_documentation/source/GPU.rst Src/CMakeLists.txt Tools/CMake/AMReXCUDAOptions.cmake Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXFlagsTargets.cmake Tools/CMake/AMReXGenexHelpers.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/CMake/AMReXUtils.cmake Tools/CMake/AMReX_Config.cmake Tools/CMake/AMReX_SetupCUDA.cmake Tools/CMake/AMReX_Utils.cmake commit 9cf1ca565447d5776bd7e7fadd01fdf2e114d53e Author: Weiqun Zhang Date: Mon May 17 17:46:06 2021 -0700 Turn on RDC support for HIP (#2029) This will allow `extern` device variables. Tools/GNUMake/comps/hip.mak commit 6f1085a23de1f7ee1a5f92dd173e984c59839511 Author: Max Katz Date: Sun May 16 11:05:05 2021 -0700 Device profilerStart and profilerStop APIs (#2026) Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp commit c6a8fa243e8449938b8b4b5b16a9e95df9487d48 Author: Emmanuel Motheau Date: Sun May 16 11:04:20 2021 -0700 Workaround to fix the AugmentPlotfile tool (#2023) Tools/C_util/AugmentPlotfile/AugmentPlotfile.cpp commit 442c6302b76ea0ce7d0018f69ae957686d4c4add Author: Weiqun Zhang Date: Fri May 14 18:46:38 2021 -0700 Use rocm 4.1.1 for hip ci (#2024) .github/workflows/dependencies/dependencies_hip.sh .github/workflows/linux.yml commit 6795c1f45833f1784b24d2277664d670f8c7171c Author: Max Katz Date: Fri May 14 17:41:34 2021 -0700 Make nvcc play nicely with CC wrapper (#2019) Tools/GNUMake/Make.defs Tools/GNUMake/comps/cray.mak commit ab84978412f2a00aabf966cbc54b87da39f9ab60 Author: Max Katz Date: Fri May 14 17:34:08 2021 -0700 Add Perlmutter to list of NERSC hosts (#2021) Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.nersc commit 65226552590664d61d32b2d3d8e0de7e691d7e62 Author: Max Katz Date: Fri May 14 17:32:59 2021 -0700 Cray ftn does not accept isystem (#2020) Tools/GNUMake/Make.defs commit d0210641e10dd257a5f2942e992e3c9ab5a4fcfb Author: Max Katz Date: Fri May 14 17:31:38 2021 -0700 Clang Cray can also show up with explicit Clang identifier (#2018) Tools/GNUMake/comps/cray.mak commit 2756788d7b06c3cc7ad5c471ee271edcfa484845 Author: Kevin Gott Date: Fri May 14 12:20:10 2021 -0700 Label function HOST_DEVICE. (#2022) Src/Base/AMReX_RealBox.H commit f7d87b8be6b9bc9373aa17e8b2805256afcd0c55 Author: Max Katz Date: Fri May 14 08:37:11 2021 -0700 Remove restrictions on nvcc host compiler (#2017) These restrictions were originally enforced back in the days of CUDA Fortran, there is no need to require this now. Tools/GNUMake/Make.defs commit 6c9a3f73f3034b5e2b299b6d0a6a7a4333dc1875 Author: Weiqun Zhang Date: Wed May 12 21:58:45 2021 -0700 Remove minBlocksPerMultiprocessor from launch bounds (#2015) After reading CUDA Programming Guide, I think we should remove minBlocksPerMultiprocessor from launch bounds, so that it's closer to what we had before. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#launch-bounds Src/Base/AMReX_GpuLaunchGlobal.H commit 19a2152a097dda52df64e2e42ba60e267fda74be Author: Weiqun Zhang Date: Wed May 12 20:53:02 2021 -0700 Loosen a HIP regresstion test's tolerance (#2014) Tools/RegressionTesting/AMReX-hip-tests.ini commit ad3eea065f3387fd374d9480a31419ba3bbb72bc Author: Weiqun Zhang Date: Wed May 12 20:49:06 2021 -0700 Launch bounds (#2013) Set launch bounds to AMREX_GPU_MAX_THREADS (256 by default) for all kernel launches. This helps resolve some HIP compiler issues we have had recently. Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuReduce.H commit 50847f55331c15f10e38f4e710c0fef292737fe5 Author: Weiqun Zhang Date: Wed May 12 07:45:20 2021 -0700 Disable HIP CI (#2011) ROCm 4.2 has a bug that causes the ci to fail. .github/workflows/linux.yml commit 73dc2422bb07c7346c29c01d1bbfa5a83352ceae Author: Ann Almgren Date: Tue May 11 20:45:04 2021 -0700 Test on vfrac rather than isRegular when doing the slope limiting -- we were doing this in one (#2008) place but not another place which made the behavior inconsistent. Src/EB/AMReX_EB_slopes_K.H commit 20be66e90b6685865f534ab83b1b8850ad848bcb Author: Weiqun Zhang Date: Tue May 11 16:44:04 2021 -0700 Arena initial size (#2007) Add ParmParse parameters to allow the users to change the initial size of various Arenas. In particular this allows the users to not allocate any managed memory if they set `amrex.the_arena_is_managed=0 amrex.the_managed_arena_init_size=0`. Src/Base/AMReX_Arena.cpp commit 8a650f69446681320504fe5156591dc547bcb0c9 Author: Weiqun Zhang Date: Tue May 11 16:34:48 2021 -0700 Use CPU memory in memory pools for OpenMP threads (#2006) The memory pools are primarily used by Fortran functions as scratch space inside OpenMP parallel regions. There are no reasons to allocate them in managed device memory. Src/Base/AMReX_MemPool.cpp commit bf524ee95ff4a068ddf0a27599d4ce652e25b3cb Author: Weiqun Zhang Date: Tue May 11 13:02:03 2021 -0700 No casting to unsigned long long needed in iMultiFab::sum (#2005) Because we no longer use atomics in reduction, we do not need to cast to unsigned long long in iMultiFab::sum anymore. Have to install tgtermes.sty to build the docs. .github/workflows/dependencies/documentation.sh Src/Base/AMReX_iMultiFab.cpp commit fb0c16e34b9340e68ee63013e64570240e436815 Author: Max Katz Date: Sat May 8 16:12:26 2021 -0700 Avoid call to amrex::max with only one argument in 1D (#2004) Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp commit be62957c227508cda828bd4d4b7ea2ab9d3a2b87 Author: Weiqun Zhang Date: Fri May 7 14:03:21 2021 -0700 Cray complier type detection (#2001) There are two types of Cray compilers, the classical and clang based. The version number is not a reliable way to determine the type. For example, cce/9 on Cori has both versions. Cray compilers for aarch64 on Ookami are classical. Thus we switch to grep "LLVM" in `CC --version`. This works on both Cori and Ookami. Tools/GNUMake/comps/cray.mak commit 3c3bb354d7fb8c9d4b1072d5aabab74b7cc34117 Author: Weiqun Zhang Date: Fri May 7 13:25:22 2021 -0700 Arm Clang (#2002) Add support for Arm Clang in GNU make. Tools/GNUMake/Make.defs Tools/GNUMake/comps/armclang.mak Tools/libamrex/configure.py Tools/libamrex/mkconfig.py commit 9496af4b7bc2e768608c026b06e3afe6f55ce01c Author: Andrew Myers Date: Fri May 7 10:56:30 2021 -0700 Move Morton code utility functions somewhere they can be reused. (#1999) Also ports them to GPU. Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_Morton.H Src/Base/CMakeLists.txt Src/Base/Make.package commit ce04df6c1cb870860672c218f3e3b641f1ec24ec Author: Andrew Myers Date: Wed May 5 17:05:05 2021 -0700 Templated implementation of ostream operator for both Dim3 and XDim3 (#1997) Src/Base/AMReX_Dim3.H Src/Base/AMReX_Dim3.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 8d7d25ff6ae426f31cd609a97d66bc8608fc03d1 Author: Weiqun Zhang Date: Wed May 5 16:57:15 2021 -0700 Disable HIP device assertion (#1998) It makes the compliation very slow and it does not currently work. Src/Base/AMReX.H Src/Base/AMReX_GpuQualifiers.H commit a2a23d75fafb05f7f2a974d0fe7bf5ff584a2823 Author: Andrew Myers Date: Wed May 5 14:59:01 2021 -0700 Generalize elemwiseMin and elemwiseMax. (#1996) This allows these functions to be called on `XDim3` objects, as well as any other object that has `x`, `y`, and `z` data members, instead of just `Dim3`. Additionally, it makes them variadic so they work on any number of objects, instead of just two. Src/Base/AMReX_Algorithm.H commit c1375ccb916f35caca50cc52c72535025e179124 Author: Weiqun Zhang Date: Wed May 5 14:29:04 2021 -0700 Add new parameter to BoxArray::contains(BoxArray) (#1995) In regrid, we need to figure out how many cells we need to grow the current coarse level grids in order for the fine grids to be properly nested. We used BoxArray::contains(BoxArray) in a loop and grew the coarse BoxArray in the loop. In this commit, we add a new parameter to BoxArray::contains(BoxArray) to specify the number of grow cells without having to actually grow the BoxArray. This is better than the previous version because growing BoxArray would invalidate the hash data in BoxArray. We also modify the implementation of BoxArray::contains(Box) in this commit. Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp commit b8dad4457ae795d39bfc4d9e7c3c1927102873df Author: Kevin Gott Date: Tue May 4 15:50:50 2021 -0700 Remove Warnings in DEBUG in AssignParticle Test (#1993) Tests/Particles/AssignDensity/main.cpp commit c6702b5772d40bd7d34aeb65fa445510e8bce01e Author: Weiqun Zhang Date: Tue May 4 15:49:22 2021 -0700 Optimizing away temporary BoxArrays in AmrMesh::MakeNewGrids (#1994) Previously, we stored the information needed for proper nesting in BoxList. Then when they were used, they were repeatedly converted to BoxArray for hash based fast intersection. In this commit, we store them directly in BoxArray, thus removing those hidden temporaries. Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_TagBox.H Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_BoxDomain.H Src/Base/AMReX_BoxDomain.cpp commit 57e8af38fa513674ad9097f6bc6858d8eb669015 Author: Kevin Gott Date: Tue May 4 14:43:40 2021 -0700 Non-blocking SumBoundary (#1991) Non-blocking SumBoundary, using the stored MF pointer in PCData. Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp commit bc335117a000dd5df511d2dc298ce041e09bafee Author: Weiqun Zhang Date: Tue May 4 12:40:10 2021 -0700 Remove an expensive assertion in BoxArray::contains (#1992) The assertion gets extremely expensive for large runs with long skinny patches when it's used by ClusterList::intersect. Src/Base/AMReX_BoxArray.cpp commit 50280bc83decd9c04df80697db3d95ca2d70b947 Author: Kevin Gott Date: Tue May 4 12:29:26 2021 -0700 Docs: Add non-blocking comms. (#1988) Very brief discussion of non-blocking comms for the documentation (currently, I can't find any reference). Docs/sphinx_documentation/source/Basics.rst commit 741fdd27e355d2c5992facd79d41fc283c4ea777 Author: Zach Jibben Date: Mon May 3 22:12:21 2021 -0600 Define BL_USE_MPI compilation flag when MPI and Fortran are enabled (#1990) Resolves #1989 Tools/CMake/AMReXSetDefines.cmake commit b451e90934eb0a8f0d10878e4e087b3aa933d78e Author: Kevin Gott Date: Mon May 3 16:10:47 2021 -0700 FBData object (#1983) Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H commit b27fc508b608eb6f78a9d9bcfe11f1ef0a226461 Author: Kevin Gott Date: Mon May 3 15:41:18 2021 -0700 Deprecate FabArray::copy() (#1985) This deprecates the legacy `FabArray::copy()` in favor of the better named `FabArray::ParallelCopy()` functions. This should help alleviate the problem of confusion between `FabArray::copy()` and `MultiFab::Copy()`, which is a local copy. Also removes "copy" calls from the rest of the repo. Src/Amr/AMReX_AuxBoundaryData.cpp Src/AmrCore/AMReX_FillPatchUtil.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtilI.H Src/Base/AMReX_VisMF.cpp Src/Boundary/AMReX_FabSet.cpp Src/EB/AMReX_EBMultiFabUtil.cpp Src/Extern/ProfParser/AMReX_BLProfUtilities.cpp Src/Extern/amrdata/AMReX_DataServices.cpp Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_TracerParticles.cpp commit ff82946b64fa3dcd4c01df52e564c83ecd752089 Author: Kevin Gott Date: Mon May 3 15:29:33 2021 -0700 PCData object for non-blocking ParallelCopy (#1973) Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H commit 783cbf183adac05004bf1b048942888243e59032 Author: Weiqun Zhang Date: Mon May 3 10:39:06 2021 -0700 Fix proper nesting issue for large error buffer (#1987) Reimplement the code that ensures proper nesting by projecting the fine grids onto the coarse. The previous implementation has a bug that causes it to fail when the error buffer is very large. In the very beginning of BoxLib, the enforcement of proper nesting was actually done in a similar way as the approach in this commit (i.e., calling TagBoxArray::setVal(BoxArray,TagBox::CLEAR) with projected down BoxArray). However, it had the issue of buffering the buffer cells, creating unnecessarily big refinement patches. That issue was fixed in a block of very complicated code that was hard to reason, also in the very beginning of BoxLib days. Since it worked, the code had not been touched at all. However, it was reported recently it failed when a very large error buffet was used. In this commit, we switched back to the original setVal approach and it is performed after the tagged cells are buffered already, thus avoiding the issue of buffering the buffer cells. Furthermore, it is performed after the tags are coarsened, thus performing less work. This also adds a check to make sure that blocking factors do not vary too much between levels. Otherwise regrid will break. Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_TagBox.cpp commit 2b397a9ba0c99376177ec33138e8ed3de707cd8a Author: Houjun Tang Date: Mon May 3 10:36:17 2021 -0700 Update HDF5 I/O to use new HDF5 Async VOL APIs (#1948) Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_ParticleUtil.cpp Src/Particle/AMReX_Particles.H Tests/HDF5Benchmark/GNUmakefile Tests/HDF5Benchmark/main.cpp commit c670d955e90912515389f4715321abe67c4cf206 Author: Mark Meredith Date: Mon May 3 11:59:13 2021 -0400 Remove unused variables (fix warnings) (#1976) Found when building MFiX Src/EB/AMReX_EB_slopes_K.H commit cff96a9f23638819cff688071ff2588d8cfadd63 Author: Weiqun Zhang Date: Sun May 2 10:26:43 2021 -0700 Update CHANGES for 21.05 (#1986) CHANGES commit 8e8b3d75376a081c02c1735fe8d1520a12bfb9f3 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Fri Apr 30 10:19:42 2021 -0600 Hypre Default Params (#1984) This PR reverts the default values for bamg_relax_type back to 6 (Symmetric Gauss Seidel). This appears to be more robust than 11 (2 stage Gauss Seidel). Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit fe2830d4dd426c96cc0454dfb337e08bd489c7e9 Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Thu Apr 29 16:48:56 2021 -0600 Fix to this parameter (#1982) Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit bfca28a1f40a9774790d09c3247b39675986aaae Author: PaulMullowney <60452402+PaulMullowney@users.noreply.github.com> Date: Thu Apr 29 10:12:41 2021 -0600 modifying some hypre params and changing defaults. (#1981) Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit 47d580e2943dcc90998b8852ead53baaf4dd9a76 Author: cdaley Date: Wed Apr 28 19:57:11 2021 -0700 Support OpenMP offload with the NVIDIA compiler (#1980) These changes enable the ElectromagneticPIC OpenMP offload application to succeed (Particles/ElectromagneticPIC/Exec/OpenMP). It requires NVIDIA HPC SDK 21.3 or higher. Earlier versions of NVIDIA HPC SDK fail for one of two reasons: no interoperability between CUDA and OpenMP offload, or failure to support Fortran automatic arrays in OpenMP offload regions. I tested on Cori-GPU with nvhpc/21.3 and cuda/11.1.1. Co-authored-by: Christopher Daley Tools/GNUMake/Make.defs Tools/GNUMake/comps/nvhpc.mak Tools/libamrex/mkconfig.py commit a61b40464f3f16eff420d19d99cff0fa010eb2b6 Author: Weiqun Zhang Date: Tue Apr 27 11:32:55 2021 -0700 Use full path to AMReX_Config.H in gnu make target (#1979) Tools/GNUMake/Make.rules commit 30879b025d222b383aa22540598dc1891ce73478 Author: Weiqun Zhang Date: Mon Apr 26 16:43:20 2021 -0700 Fix GNU make for nvcc 11.0 (#1977) This fixes a bug introduced in #1967. It turns out nvcc 11.0 does not support -MP. Tools/GNUMake/comps/nvcc.mak commit 0340e766b71005c9675d6e9b759c66bb5c705297 Author: Weiqun Zhang Date: Mon Apr 26 13:20:16 2021 -0700 Add CI for 2D Single Precision EB HIP build (#1974) Also fix some warnings. .github/workflows/linux.yml Src/Base/AMReX_GpuLaunchGlobal.H Src/Base/AMReX_Reduce.H Src/EB/AMReX_EB2_2D_C.cpp commit 521ff492834377cc5ee91d6eb72db2471ccde3b7 Author: Weiqun Zhang Date: Mon Apr 26 13:00:09 2021 -0700 Fix dependency of f90.depends (#1975) Tools/GNUMake/Make.rules commit b1709947c0a354dc5dbefeb3421497478ef2333c Author: Weiqun Zhang Date: Mon Apr 26 09:07:35 2021 -0700 GNU Make: -MMD -MP (#1967) Use -MMD -MP instead of -M to generate dependency unless it is not supported by the compiler. In the new approach, the dependencies are generated during compilation without a separate step. Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/comps/cray.mak Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/llvm.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/comps/pgi.mak commit e6833a86ebf35861017cfd5f16ca66ac9dc8b7ae Author: Luca Fedeli Date: Mon Apr 26 17:13:19 2021 +0200 Fix missing Real() in AMReX_EB2_2D_C.cpp (#1972) This PR fixes few missing `Real()` in `AMReX_EB2_2D_C.cpp`, which currently prevent the compilation of a function in single precision. Src/EB/AMReX_EB2_2D_C.cpp commit 32597f78f68fc8b948ceb10a22c0d01ab2b513ec Author: Weiqun Zhang Date: Mon Apr 26 08:11:58 2021 -0700 Undeprecate ReduceData::value() (#1970) ReduceData::value() with no argument was recently marked as a depracated function because it involves calling a std::function. However, the call is never made inside a loop like MFIter. So there is no practical difference in performance between ReduceData::vaule and ReduceData::value(ReduceOps&). It is hereby undeprecated to avoid warning messages for a lot of applications. Src/Base/AMReX_Reduce.H commit 5eedbae2d805d86014788427bee39e6cccb2155c Author: Shreyas Ananthan Date: Sun Apr 25 13:12:48 2021 -0600 Workaround compile error on Apple M1 (ARM64) (#1969) Fixes #1968 Src/Base/AMReX.cpp commit 41d1ea1873494d5e59c6f7da16d73b634e2a9362 Author: Ann Almgren Date: Fri Apr 23 10:54:14 2021 -0700 Changes to EB slopes (#1965) * 1) Change test on flag.isRegular() to (vfrac == 1) -- this only matters when we have cells with vfrac == 1 but with an EB face -- this change will allow us to use the non-EB stencil when parallel to an EB face if all the vfrac's we need are = 1, and is also consistent with how we treat slopes tangential to domain boundaries 2) pass max_order in so we can default to 4th order if max_order == 4 and there are enough cells with vfrac == 1 3) Modify the Barth-Jespersen limiting so that, for example, it doesn't use cells at i-1 when computing a slope on the i+1/2 face. NOTE: this contains a change to the interfaces to the slope routines so all code that calls these will need to be modified to pass in vfrac and max_order * Remove trailing white spaces * remove more trailing white spaces * the last of the trailing white spaces ... * Compile in 2d Src/EB/AMReX_EB_slopes_K.H commit c0570ec8d60f1a6227b5835f9105801a40a81eab Author: Axel Huebl Date: Fri Apr 23 10:24:16 2021 -0700 Docs: CMake Multiple Archs (#1964) Answer a user question with a new doc entry. Docs/sphinx_documentation/source/GPU.rst commit e85b665bcb032f4064fb9845a925c6d3d8c8aa03 Author: Weiqun Zhang Date: Fri Apr 23 09:06:37 2021 -0700 Split Large Files (#1966) Split some large files in MLMG into smaller files to improve compile time. Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_F.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp_bc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_eb.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_hypre.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp_grad.cpp Src/LinearSolvers/MLMG/Make.package commit 84cc7114097b548dce3b704b6eb9681b425f1e94 Author: Weiqun Zhang Date: Thu Apr 22 17:56:10 2021 -0700 Build Time Log (#1963) In GNU Make, if one uses `LOG_BUILD_TIME=TRUE`, a log of build time will be available in the end. Tools/C_scripts/gatherbuildtime.py Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules commit 6e5a03f8f8346e0f4840d1b5670f9afa60a0fc4a Author: Weiqun Zhang Date: Thu Apr 22 10:19:42 2021 -0700 Using rocPRIM in HIP Version of Scan (#1962) Reimplement HIP version of scan using rocprim. This uses a few things in rocprim::detail. Hopefully it will not break in a future release of HIP. Also some minor tweaks to the CUDA version of Scan. Src/Base/AMReX_GpuLaunchGlobal.H Src/Base/AMReX_Scan.H commit a0790de872d6b18b563712b9f1b18323b8d00072 Author: Weiqun Zhang Date: Tue Apr 20 16:21:37 2021 -0700 Reimplementation of ReduceOps (#1955) The new approach does reduction in two passes. Unlike the old approach, this does not require atomics, and has much better performance for types that do not have hardware support for atomics. Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_iMultiFab.cpp Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EBCellFlag.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/Particles/Intersection/main.cpp commit 1796dd911bae1ae0b58ae92e4e023bdb00a0f44a Author: Michele Rosso Date: Mon Apr 19 17:01:36 2021 -0700 CMake: implement logic to support compiler ID IntelLLVM (new in CMake 3.20) (#1959) CMake 3.20+ has its own compiler identification for the new oneAPI LLVM-ICX/DPCXX implementation. Tools/CMake/AMReXFlagsTargets.cmake Tools/CMake/AMReXGenerateConfigHeader.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake commit 0a50fded0df7539b5e8e32b2a88966d8d671e264 Author: Weiqun Zhang Date: Mon Apr 19 14:19:53 2021 -0700 DPCPP: fix atomicAdd for float (#1958) We recently removed the template specialization for atomicAdd of float in the DPC++ backend. That was a mistake because SYCL does not support atomicAdd for float, and we have to use atomicCAS to implement it. Src/Base/AMReX_GpuAtomic.H Tests/Amr/Advection_AmrCore/Source/Src_K/slope_K.H commit c409f9e1ffc870fd4926190ffceffc7d0d5ad409 Author: Weiqun Zhang Date: Mon Apr 19 13:30:05 2021 -0700 CPU version of TagBox::buffer (#1957) Add a CPU version of TagBox::buffer. @MSABuschman reported in #1951 that TagBox::buffer has been very slow since commit #1258 if the error buffer size is large. The function was rewritten in #1258 to do the work on GPU. In this PR, the old version is reintroduced for CPU. Note that the current implementation is expected to have poor performance on GPU if it has a very large error buffer. It's still not clear how we should implement this function for GPU if a large error buffer is used. .github/workflows/linux.yml Src/AmrCore/AMReX_TagBox.H Src/AmrCore/AMReX_TagBox.cpp commit 4320729fe46992851b3f222d28833109cff0234e Author: Jean M. Sexton Date: Sun Apr 18 11:21:27 2021 -0700 Fix out of bound error in AsyncWrite (#1753) Fix out of bound error when the local number of fabs is zero. Co-authored-by: Weiqun Zhang Src/Base/AMReX_VisMF.cpp commit b30cae951ba2c7c004f06e3d7d382c7e79793887 Author: Weiqun Zhang Date: Sun Apr 18 08:15:16 2021 -0700 Fab data I/O (#1956) When Fab data are in GPU device memory (including managed memory), use pinned memory as buffer to avoid accessing the data on the host. Previously, MultiFab, plotfile and checkpoint writers except for the async versions relied on that the memory was managed and therefore could be used directly in I/O. Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_VisMF.cpp commit f21f15d8f7a3098d9fa13ac6b316ca518f464ac5 Author: Max Katz Date: Fri Apr 16 17:54:37 2021 -0700 Allow user to opt out of renaming MTMPI suffix (#1954) If MPI_THREAD_MULTIPLE is going to be on by default in an application, might as well not need to rename the suffix, so this gives the application the ability to just keep the suffix MPI. Tools/GNUMake/Make.defs commit 3812eadb83a667a615630a2ff8d92def964c3e58 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Fri Apr 16 19:57:47 2021 -0400 Add triangle wave function to least squares test (#1947) * Add a triangle wave test for Least Squares centroids, fix some bugs in test setup * update doc * fix formatting * fix formatting Tests/LinearSolvers/LeastSquares/Make.package Tests/LinearSolvers/LeastSquares/MyTest.H Tests/LinearSolvers/LeastSquares/MyTest.cpp Tests/LinearSolvers/LeastSquares/README.md Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/LeastSquares/initTriangleWaveData.cpp Tests/LinearSolvers/LeastSquares/initTriangleWaveDataFor2D.cpp Tests/LinearSolvers/LeastSquares/inputs.2d.trianglewave commit eaad7627ff925f8ae9b290a7d6a2d5f2fba54239 Author: Ann Almgren Date: Fri Apr 16 15:52:15 2021 -0700 level_0_gome --> level_0_geom (#1953) Src/AmrCore/AMReX_AmrCore.cpp commit 029a8b0608beab17f440718b5b147dc5dd24b67c Author: Weiqun Zhang Date: Fri Apr 16 13:51:19 2021 -0700 Port FluxRegister::ClearInternalBorders to GPU (#1952) Src/AmrCore/AMReX_FluxRegister.cpp commit 42da8fb5ac8df41566a045bb3332df6ad24126d8 Author: Weiqun Zhang Date: Thu Apr 15 19:19:25 2021 -0700 Fix the return of MLPoisson::makeNLinOp (#1950) Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp commit 66f99da25c6a305fae47f0b952ebe670af867b19 Author: Andrew Myers Date: Thu Apr 15 18:07:28 2021 -0700 Skip temporary in ParticleToMesh when it is not needed. (#1949) Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_ParticleMesh.H commit d2e37b06bed136e001ae62718ba5754720a01f0d Author: Andrew Myers Date: Thu Apr 15 11:32:42 2021 -0700 Multilevel version of ParticleToMesh (#1945) This uses the same algorithm as the AssignDensity method in `AmrCore/AMReX_AmrParticles.H` to handle the coarse / fine boundaries. However, it allows a user-specified lambda function to define the interpolation operation, as in the single-level `ParticleToMesh` routine. Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleUtil.H Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleMeshMultiLevel/CMakeLists.txt Tests/Particles/ParticleMeshMultiLevel/GNUmakefile Tests/Particles/ParticleMeshMultiLevel/Make.package Tests/Particles/ParticleMeshMultiLevel/inputs Tests/Particles/ParticleMeshMultiLevel/main.cpp commit b1d05dec0da3b6d0074af8850163837c53488ebf Author: Erik Schnetter Date: Thu Apr 15 13:32:54 2021 -0400 Correct typo in comment (#1946) Tools/CMake/AMReXOptions.cmake commit 914459f443f5ed54b5e33551802d10521042af9b Author: Donald E. Willcox Date: Wed Apr 14 16:13:44 2021 -0700 fix greater->less in docs for abort_on_out_of_gpu_memory (#1944) Docs/sphinx_documentation/source/GPU.rst commit 87ee8f03c3b38c645312331fa75c92b6a932a87b Author: Weiqun Zhang Date: Tue Apr 13 16:45:37 2021 -0700 Use CUB for block reduction (#1943) Also removed some reduction functions that have been deprecated long ago. Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_Reduce.H commit ba378870a10ad475c647cad3b8650151fc896580 Author: Weiqun Zhang Date: Tue Apr 13 08:24:57 2021 -0700 Reimplementation of PrefixSum with cub (#1942) For CUDA >= 11, use cub to reimplement PrefixSum that can take two lambda functions. Both cub and thrust only provide scan functions that take iterators. Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H commit 58c3ba169a2b8fc311395a959724ea63dc37aa3c Author: Weiqun Zhang Date: Sun Apr 11 17:03:22 2021 -0700 Avoid AMREX_NO_UNIQUE_ADDRESS (#1941) * Avoid AMREX_NO_UNIQUE_ADDRESS because it causes gcc 9.3 to crash. * Fix AMREX_NO_UNIQUE_ADDRESS and AMREX_NODISCARD macros. * Adjust compiler flag in CI to allow for C++17 extension when compiling with C++14. .github/workflows/linux.yml Src/Base/AMReX_Extension.H Src/Base/AMReX_NonLocalBC.H commit 18c5e96e04c982ef886d8aa9f6e65ebee1564f35 Author: Weiqun Zhang Date: Thu Apr 8 12:58:09 2021 -0700 Rename variable to fix a shadow warning (#1939) Src/Particle/AMReX_ParticleUtil.H commit 51ba2f7ebbc2dc759845fdab073d353ab175507f Author: Weiqun Zhang Date: Thu Apr 8 11:55:51 2021 -0700 Workaround a compiler bug(?) reported in #1937 (#1938) Src/Base/AMReX_Arena.cpp commit 7c259bab6cc583a92a8d7f6b640af6618e6206f3 Author: Weiqun Zhang Date: Wed Apr 7 17:52:04 2021 -0700 Cell-Centered Overset Solver Coarsening (#1915) Currently for the cell-centered overset multigrid solver, coarsening stops when a coarsened cell would contain both regular and masked-out cells. Here, we further coarsen the grids by ignoring the overset mask and giving the masked-out cells a huge value of alpha so that their values are essentially frozen. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp commit db9303290d4912f15230c792f4e1c91bce0fb0ce Author: Weiqun Zhang Date: Wed Apr 7 15:38:40 2021 -0700 Hypre Nodal Setup (#1906) * Hypre Nodal Setup * Allow CoarseningStrategy::Sigma in Hypre nodal solver. * Port Hypre nodal setup to GPU. * For EB nodal solver, label a node ad Dirichlet if its diagonal component of the stencil is zero. * remove EB stuff in hypre nodal setup because it's no longer needed due to the change in dirichlet mask Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_Scan.H Src/EB/AMReX_EB2_Level.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp commit 8f9589682e56d7f443e42de032b027bd7b1944fa Author: Michele Rosso Date: Wed Apr 7 14:15:50 2021 -0700 CMake: add support for BOUND_CHECK (#1934) Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in commit d5e4a3882b263c858221de1130f73e04825a2e1e Author: Luca Fedeli Date: Wed Apr 7 16:54:28 2021 +0200 Fix other missing Real() (#1933) This PR fixes few missing `Real()` in `AMReX_EB2_3D_C.cpp`, which currently prevent the compilation of some methods in single precision. Src/EB/AMReX_EB2_3D_C.cpp commit 3f3a9174286aa06d7826367b3d07b29c339d428a Author: Luca Fedeli Date: Wed Apr 7 04:13:13 2021 +0200 fix missing Real (#1932) Src/EB/AMReX_EB2_3D_C.H commit 4123e87ce3779855a638663008de8dcec5ccb99f Author: Andrew Myers Date: Tue Apr 6 12:08:23 2021 -0400 Alias HostVector to PinnedVector and update documentation. (#1931) Src/Base/AMReX_GpuContainers.H commit 4cb1c5b79906fe566d1b80777315cf4817924b1a Author: Weiqun Zhang Date: Mon Apr 5 16:56:23 2021 -0700 Make CNS work on CPU too (#1930) Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FabConv.cpp Src/Base/AMReX_IArrayBox.cpp Tests/GPU/CNS/Exec/RT/cns_prob.cpp Tests/GPU/CNS/Exec/Sod/cns_prob.cpp Tests/GPU/CNS/Source/CNS.cpp commit 7c37a75e23eabc317c7a969e21153183db616fd1 Author: Axel Huebl Date: Mon Apr 5 16:15:02 2021 -0700 std::isfinite/inf for SYCL (#1929) Wrappity wrap the non-constexpr functions: https://github.com/intel/llvm/blob/sycl/sycl/doc/extensions/C-CXX-StandardLibrary/C-CXX-StandardLibrary.rst Note that `std::isnan` works by accident/virtue of being constexpr. Src/Base/AMReX_Math.H commit 1ee06e64345a52259315181ac13a821a4ac64f09 Author: Weiqun Zhang Date: Mon Apr 5 14:10:12 2021 -0700 Update RNG for latest oneAPI (#1928) MKL now has device support for uniform distribution of unsigned int and Poisson distribution. Src/Base/AMReX_Random.H commit deddea69d9f7e2b5cac2727cadfe77e0806c194b Author: Weiqun Zhang Date: Mon Apr 5 10:13:30 2021 -0700 Use rocprim for scan (#1925) .github/workflows/dependencies/dependencies_hip.sh .github/workflows/linux.yml Src/Base/AMReX_Scan.H commit 06cec5dd8ede26ea3c0646368bdbdb32f35842ae Author: Weiqun Zhang Date: Mon Apr 5 09:31:25 2021 -0700 Use pinned memory for data transfer between host and device (#1923) The amount of data to be transferred in TagBox and STLtools is likely to be big. So pinned memory is used instead of pageable. AsyncArray is sometimes used to transfer a large amount of data. So pinned memory is used. Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_TagBox.H Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuAsyncArray.cpp Src/EB/AMReX_EB_STL_utils.H commit d3a24fe0519d8eb7cfd7043be2907e08ee534352 Author: Weiqun Zhang Date: Mon Apr 5 09:16:28 2021 -0700 Use cub::DeviceScan for CUDA >= 11 (#1924) Src/Base/AMReX_Scan.H Tools/GNUMake/Make.defs Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/nvcc.mak commit ddfbf3de3bdf5ced8f42d0828ed52d95fcea22ec Author: Weiqun Zhang Date: Mon Apr 5 09:16:07 2021 -0700 Fix compilation broken in #1920 (#1926) Tests/LinearSolvers/CellEB/MyTest.cpp Tests/LinearSolvers/CellEB2/MyTest.cpp Tests/LinearSolvers/EBConvergenceTest/MyTest.cpp Tests/LinearSolvers/EBflux_grad/MyTest.cpp Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/NodeEB/MyTest.cpp commit 5ec3d3799ff238208ae41949ce506d3491b37208 Author: Weiqun Zhang Date: Fri Apr 2 19:03:42 2021 -0700 Move Tests/CNS to Tests/EB/CNS (#1922) Tests/EB/CNS/CMakeLists.txt Tests/EB/CNS/Exec/Combustor/GNUmakefile Tests/EB/CNS/Exec/Combustor/Make.package Tests/EB/CNS/Exec/Combustor/bc_fill_nd.F90 Tests/EB/CNS/Exec/Combustor/bc_fill_nd.F90_jbb Tests/EB/CNS/Exec/Combustor/cns_prob.F90 Tests/EB/CNS/Exec/Combustor/cns_prob.F90_jbb Tests/EB/CNS/Exec/Combustor/inputs Tests/EB/CNS/Exec/Combustor/inputs.regt Tests/EB/CNS/Exec/Make.CNS Tests/EB/CNS/Exec/Pulse/GNUmakefile Tests/EB/CNS/Exec/Pulse/Make.package Tests/EB/CNS/Exec/Pulse/cns_prob.F90 Tests/EB/CNS/Exec/Pulse/inputs Tests/EB/CNS/Exec/Pulse/inputs.regt Tests/EB/CNS/Exec/ShockRef/GNUmakefile Tests/EB/CNS/Exec/ShockRef/Make.package Tests/EB/CNS/Exec/ShockRef/cns_prob.F90 Tests/EB/CNS/Exec/ShockRef/inputs Tests/EB/CNS/Exec/ShockRef/inputs.amr Tests/EB/CNS/Exec/ShockRef/inputs.regt Tests/EB/CNS/Exec/Sod/GNUmakefile Tests/EB/CNS/Exec/Sod/Make.package Tests/EB/CNS/Exec/Sod/cns_prob.F90 Tests/EB/CNS/Exec/Sod/inputs Tests/EB/CNS/Source/CNS.H Tests/EB/CNS/Source/CNS.cpp Tests/EB/CNS/Source/CNSBld.cpp Tests/EB/CNS/Source/CNS_F.H Tests/EB/CNS/Source/CNS_advance.cpp Tests/EB/CNS/Source/CNS_init_eb2.cpp Tests/EB/CNS/Source/CNS_io.cpp Tests/EB/CNS/Source/CNS_setup.cpp Tests/EB/CNS/Source/Make.package Tests/EB/CNS/Source/diffusion/Make.package Tests/EB/CNS/Source/diffusion/cns_diff_mod.F90 Tests/EB/CNS/Source/diffusion/cns_eb_diff_mod.F90 Tests/EB/CNS/Source/diffusion/cns_eb_diff_wall.F90 Tests/EB/CNS/Source/diffusion/diff_coef_mod.F90 Tests/EB/CNS/Source/fortran/CNS_derive.F90 Tests/EB/CNS/Source/fortran/CNS_divop.F90 Tests/EB/CNS/Source/fortran/CNS_dudt.F90 Tests/EB/CNS/Source/fortran/CNS_f.F90 Tests/EB/CNS/Source/fortran/CNS_nd.F90 Tests/EB/CNS/Source/fortran/CNS_physics.F90 Tests/EB/CNS/Source/fortran/CNS_tagging.F90 Tests/EB/CNS/Source/fortran/Make.package Tests/EB/CNS/Source/fortran/bc_fill_nd.F90 Tests/EB/CNS/Source/hydro/Hyp_gamma_MOL.F90 Tests/EB/CNS/Source/hydro/Hyp_gamma_MOL_EB.F90 Tests/EB/CNS/Source/hydro/Make.package Tests/EB/CNS/Source/hydro/analriem3d.F90 Tests/EB/CNS/Source/hydro/cns_eb_hyp_wall.F90 Tests/EB/CNS/Source/hydro/slope_mol_3d_gamma.F90 Tests/EB/CNS/Source/hydro/slope_mol_3d_gamma_EB.F90 Tests/EB/CNS/Source/main.cpp commit 42063585bb299c5dd220aa5b35be9e3f05594da5 Author: Weiqun Zhang Date: Fri Apr 2 18:07:50 2021 -0700 No need for AMREX_IS_TRIVIALLY_COPYABLE && AMREX_IS_TRIVIALLY_DEFAULT_CONSTRUCTIBLE (#1921) They are not needed because gcc 4 is no longer supported. Src/Base/AMReX_BaseFab.H Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_TypeTraits.H commit 77fb0251e3ad99af069d5d0bcaa80128cc50062b Author: Weiqun Zhang Date: Fri Apr 2 10:30:42 2021 -0700 Use std::make_unique instead of reset since we are using C++14 (#1920) Docs/sphinx_documentation/source/IO.rst Docs/sphinx_documentation/source/LinearSolvers.rst Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrParticles.H Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_AsyncOut.cpp Src/Base/AMReX_BackgroundThread.cpp Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_CuptiTrace.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_ForkJoin.cpp Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MemProfiler.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_PlotFileDataImpl.cpp Src/Base/AMReX_PlotFileUtil.cpp Src/Base/AMReX_iMultiFab.cpp Src/EB/AMReX_EB2_IndexSpaceI.H Src/EB/AMReX_EBFabFactory.cpp Src/Extern/HYPRE/AMReX_Hypre.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/PETSc/AMReX_PETSc.cpp Src/Extern/amrdata/AMReX_AmrData.cpp Src/F_Interfaces/Octree/AMReX_octree_fi.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/Projections/AMReX_MacProjector.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_TracerParticles.cpp Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/FillBoundaryComparison/main.cpp Tests/GPU/CNS/Source/CNS.cpp Tests/GPU/CNS/Source/CNS_io.cpp Tests/HDF5Benchmark/main.cpp Tests/LinearSolvers/CellEB/MyTest.cpp Tests/LinearSolvers/CellEB2/MyTest.cpp Tests/LinearSolvers/CellOverset/MyTest.cpp Tests/LinearSolvers/EBConvergenceTest/MyTest.cpp Tests/LinearSolvers/EBflux_grad/MyTest.cpp Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/NodeEB/MyTest.cpp Tests/LinearSolvers/TensorOverset/MyTest.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/AsyncIO/main.cpp commit 57d41ef9cc3f0db4b5d644d0169311b4f9f68a92 Author: Weiqun Zhang Date: Thu Apr 1 08:37:00 2021 -0700 Update CHANGES for 21.04 (#1919) CHANGES commit 622fbf998d44b77f44d1c682b9b33e018402051f Author: Andrew Myers Date: Thu Apr 1 11:04:12 2021 -0400 Docs typo fix. (#1918) Docs/sphinx_documentation/source/LinearSolvers_Chapter.rst commit ecf7e9812760feff4cdb2c53deb3402b68383fc8 Author: Ann Almgren Date: Wed Mar 31 14:02:27 2021 -0700 Allow us to set the "normalization threshold" variable at run-time (#1917) through the NodalProjector class Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit b8559fe999715afd3056cb94371bf2399a461863 Author: Weiqun Zhang Date: Tue Mar 30 17:01:58 2021 -0700 Fix a bug in BaseFab::resize. (#1916) Previously if `BaseFab::resize`'s optional `Arena*` argument is `nullptr` (which is the default) while `BaseFab`'s `Arena*` member is a different `Arena`, the `nullptr` will be interpreted as the default `Arena`, `The_Arena`, ignoring the `BaseFab`'s `Arena` has already been set. This causes the following code to allocate data using `The_Arena`, not `The_Cpu_Arena`. BaseFab fab(The_Cpu_Arena()); fab.resize(box); This PR fixes the bug. Src/Base/AMReX_BaseFab.H commit 81982da08fb25cd266b3666d358d1cbf09bb4de5 Author: Ann Almgren Date: Tue Mar 30 12:47:49 2021 -0700 Change the default value of m_normalization_threshold from 1e-10 to 1e-8 (#1914) This is motivated by an EB nodal BiCG bottom solver failing to converge with 1e-10 while converging with 1e-8. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H commit 92945ad3a3560031c43fe7f02b9cc252f8330708 Author: Andrew Myers Date: Tue Mar 30 12:34:53 2021 -0400 Dont zero out input multifab in amrex::ParticleToMesh (#1910) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleMesh.H commit 78eae52af97d4c6b993feec9bf76f21b9ef7288a Author: Ann Almgren Date: Mon Mar 29 17:00:17 2021 -0700 Revert the change which zeroed out the center coefficient if it has the wrong (#1911) sign. This breaks more cases than it fixes. Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H commit 227e378901d904040340ad38971db5142f008d15 Author: vzendejasl <59891999+vzendejasl@users.noreply.github.com> Date: Mon Mar 29 13:56:25 2021 -0700 Slopes test (#1859) Slope computation on cell centroids using the Least Square method. These tests are based from the LeastSquares Test (#1707) but instead of calculating scalar values on face centroids they are calculated on cell centroids using a linear function rather than a second order polynomial (#1707). Tests/Slopes/GNUmakefile Tests/Slopes/Make.package Tests/Slopes/MyEB.H Tests/Slopes/MyTest.H Tests/Slopes/MyTest.cpp Tests/Slopes/README.md Tests/Slopes/initData.cpp Tests/Slopes/initEB.cpp Tests/Slopes/initLinearData.cpp Tests/Slopes/initLinearDataFor2D.cpp Tests/Slopes/initLinearDataFor3D.cpp Tests/Slopes/inputs Tests/Slopes/inputs.2d.askew Tests/Slopes/inputs.2d.base Tests/Slopes/inputs.2d.fullyrotated Tests/Slopes/inputs.3d.linear.aligned.xy-x Tests/Slopes/inputs.3d.linear.aligned.xy-y Tests/Slopes/inputs.3d.linear.aligned.xz-x Tests/Slopes/inputs.3d.linear.aligned.xz-z Tests/Slopes/inputs.3d.linear.aligned.yz-y Tests/Slopes/inputs.3d.linear.aligned.yz-z Tests/Slopes/inputs.3d.linear.askew-all Tests/Slopes/inputs.3d.linear.askew-xy Tests/Slopes/inputs.3d.linear.askew-xz Tests/Slopes/inputs.3d.linear.askew-yz Tests/Slopes/main.cpp commit 0ab63f3f93950b5a90fb091c6eabbf3239886fcd Author: Weiqun Zhang Date: Mon Mar 29 09:32:23 2021 -0700 Fix bug in nodal solver that uses harmonic averageing (#1902) This is a minor bug that does not affect the final result. The convergence rate might be slightly affected. Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H commit dc8f72702c1ed4bb4443916ded01d9daff4f811f Author: Ann Almgren Date: Fri Mar 26 12:40:02 2021 -0700 Ensure center coefficient in an EB nodal solve has correct sign (#1905) It is possible for the center coefficient in an EB nodal solve to have the incorrect sign after coarsening -- here we protect against that by setting s0 to the max of (s0,0) -- this enables the BiCG to converge. Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H commit 9f297b9bc3a233553925feb5c0e2db784f1ddf9e Author: Michele Rosso Date: Fri Mar 26 12:00:44 2021 -0700 CMake: add test_install (#1894) * CMake: add test_install * CMake: forgot to add test dir * CMake: specify compiler to use when running test_install * Update Doc .github/workflows/linux.yml CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Tests/CMakeTestInstall/CMakeLists.txt Tools/CMake/AMReXInstallHelpers.cmake commit 146fc573967dbb2328be15d8e499a479f218c4e8 Author: Jean M. Sexton Date: Thu Mar 25 16:16:33 2021 -0700 Update Make.CNS and similar files to reflect Tutorials to Tests move (#1903) Tests/Amr/Advection_AmrCore/README Tests/Amr/Advection_AmrCore/README.md Tests/Amr/Advection_AmrLevel/Exec/Make.Adv Tests/CNS/Exec/Make.CNS Tests/FortranInterface/Advection_F/Exec/Make.Adv Tests/FortranInterface/Advection_octree_F/Exec/Make.Adv Tests/GPU/CNS/Exec/Make.CNS Tests/LinearSolvers/MultiComponent/MCNodalLinOp.H Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/MultiComponent/main.cpp commit f9216a341e51404e935733a4c953525a6f503d47 Author: Andrew Myers Date: Thu Mar 25 14:42:37 2021 -0400 Fix AMREX_EXPORT (#1901) Src/EB/AMReX_EB2.H commit e57c1e8294c7c74a5e3a71821639a4cc1e48d895 Author: Andrew Myers Date: Thu Mar 25 13:52:56 2021 -0400 reimplement increment and incrementwithtotal to take advantage of GPUs and OpenMP (#1899) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_Particles.H commit 62b3d7e4cafbc5b2f6453782bd9f4500014c4b63 Author: Andrew Myers Date: Thu Mar 25 13:46:49 2021 -0400 Export more global variables for windows support (#1900) Src/EB/AMReX_EB2.cpp commit 87c81d0ca70e65c9209386c03b3c85ab204667bb Author: Axel Huebl Date: Wed Mar 24 16:48:05 2021 -0700 CI: HIPCC as C Compiler (#1897) Since `hipcc` is a C++ compiler, we should assign it with `-x c` if used as C compiler. Or... we just use the corresponding `clang` for C files. .github/workflows/linux.yml commit f2443e0bc4aa433baa2154cf9b3b7ca4a672bf65 Author: Andrew Myers Date: Wed Mar 24 18:09:52 2021 -0400 Return const& in rdata and idata and disallow returning copies for rvalues. (#1895) Src/Particle/AMReX_Particle.H commit d226b807024fedd11eec4d751367524f75c01178 Author: Axel Huebl Date: Wed Mar 24 14:36:25 2021 -0700 CI: HIP w/o MPI (#1896) HIP 4.1 introduces issues with C targets which break the MPI feature test in CMake's `FindMPI.cmake` for C. .github/workflows/linux.yml commit 7c3e95210a1877c67139ea57c62fbcc717acc1c3 Author: Andrew Myers Date: Mon Mar 22 18:13:14 2021 -0400 Remove unnecessary abort. (#1892) Src/EB/AMReX_distFcnElement.cpp commit f3ddeadbb53049034f817d8bb9a8c21951ff494a Author: Weiqun Zhang Date: Mon Mar 22 14:53:44 2021 -0700 Update ALaplacian for multi-component and reuse (#1891) Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.cpp commit d1468003318845a83fb59692bb4f06be95eb319a Author: Andrew Myers Date: Mon Mar 22 13:28:37 2021 -0400 Move tutorials (#1876) .github/workflows/linux.yml .github/workflows/macos.yml .github/workflows/windows.yml CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_NFiles.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleUtil.H Tests/Amr/Advection_AmrCore/CMakeLists.txt Tests/Amr/Advection_AmrCore/Exec/GNUmakefile Tests/Amr/Advection_AmrCore/Exec/GNUmakefile_movie Tests/Amr/Advection_AmrCore/Exec/Make.Adv Tests/Amr/Advection_AmrCore/Exec/Make.package Tests/Amr/Advection_AmrCore/Exec/Prob.H Tests/Amr/Advection_AmrCore/Exec/inputs Tests/Amr/Advection_AmrCore/Exec/inputs_for_scaling Tests/Amr/Advection_AmrCore/Exec/paraview_amr101.py Tests/Amr/Advection_AmrCore/README Tests/Amr/Advection_AmrCore/README.md Tests/Amr/Advection_AmrCore/Source/AdvancePhiAllLevels.cpp Tests/Amr/Advection_AmrCore/Source/AdvancePhiAtLevel.cpp Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tests/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tests/Amr/Advection_AmrCore/Source/DefineVelocity.cpp Tests/Amr/Advection_AmrCore/Source/Kernels.H Tests/Amr/Advection_AmrCore/Source/Make.package Tests/Amr/Advection_AmrCore/Source/Src_K/Adv_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/Make.package Tests/Amr/Advection_AmrCore/Source/Src_K/compute_flux_2D_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/compute_flux_3D_K.H Tests/Amr/Advection_AmrCore/Source/Src_K/slope_K.H Tests/Amr/Advection_AmrCore/Source/Tagging.H Tests/Amr/Advection_AmrCore/Source/bc_fill.H Tests/Amr/Advection_AmrCore/Source/face_velocity.H Tests/Amr/Advection_AmrCore/Source/main.cpp Tests/Amr/Advection_AmrLevel/CMakeLists.txt Tests/Amr/Advection_AmrLevel/Exec/Make.Adv Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/GNUmakefile Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Make.package Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d.f90 Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/inputs.tracers Tests/Amr/Advection_AmrLevel/Exec/SingleVortex/probin Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/GNUmakefile Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Make.package Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_2d.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/face_velocity_3d.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/inputs.regt Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/probdata.f90 Tests/Amr/Advection_AmrLevel/Exec/UniformVelocity/probin Tests/Amr/Advection_AmrLevel/README Tests/Amr/Advection_AmrLevel/Source/Adv_F.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tests/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tests/Amr/Advection_AmrLevel/Source/LevelBldAdv.cpp Tests/Amr/Advection_AmrLevel/Source/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_2d/Adv_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_2d/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_2d/compute_flux_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_2d/slope_2d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Adv_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_3d/compute_flux_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_3d/slope_3d.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/Adv_nd.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/Make.package Tests/Amr/Advection_AmrLevel/Source/Src_nd/Tagging_nd.f90 Tests/Amr/Advection_AmrLevel/Source/Src_nd/tagging_params.f90 Tests/Amr/Advection_AmrLevel/Source/main.cpp Tests/CMakeLists.txt Tests/CNS/CMakeLists.txt Tests/CNS/Exec/Combustor/GNUmakefile Tests/CNS/Exec/Combustor/Make.package Tests/CNS/Exec/Combustor/bc_fill_nd.F90 Tests/CNS/Exec/Combustor/bc_fill_nd.F90_jbb Tests/CNS/Exec/Combustor/cns_prob.F90 Tests/CNS/Exec/Combustor/cns_prob.F90_jbb Tests/CNS/Exec/Combustor/inputs Tests/CNS/Exec/Combustor/inputs.regt Tests/CNS/Exec/Make.CNS Tests/CNS/Exec/Pulse/GNUmakefile Tests/CNS/Exec/Pulse/Make.package Tests/CNS/Exec/Pulse/cns_prob.F90 Tests/CNS/Exec/Pulse/inputs Tests/CNS/Exec/Pulse/inputs.regt Tests/CNS/Exec/ShockRef/GNUmakefile Tests/CNS/Exec/ShockRef/Make.package Tests/CNS/Exec/ShockRef/cns_prob.F90 Tests/CNS/Exec/ShockRef/inputs Tests/CNS/Exec/ShockRef/inputs.amr Tests/CNS/Exec/ShockRef/inputs.regt Tests/CNS/Exec/Sod/GNUmakefile Tests/CNS/Exec/Sod/Make.package Tests/CNS/Exec/Sod/cns_prob.F90 Tests/CNS/Exec/Sod/inputs Tests/CNS/Source/CNS.H Tests/CNS/Source/CNS.cpp Tests/CNS/Source/CNSBld.cpp Tests/CNS/Source/CNS_F.H Tests/CNS/Source/CNS_advance.cpp Tests/CNS/Source/CNS_init_eb2.cpp Tests/CNS/Source/CNS_io.cpp Tests/CNS/Source/CNS_setup.cpp Tests/CNS/Source/Make.package Tests/CNS/Source/diffusion/Make.package Tests/CNS/Source/diffusion/cns_diff_mod.F90 Tests/CNS/Source/diffusion/cns_eb_diff_mod.F90 Tests/CNS/Source/diffusion/cns_eb_diff_wall.F90 Tests/CNS/Source/diffusion/diff_coef_mod.F90 Tests/CNS/Source/fortran/CNS_derive.F90 Tests/CNS/Source/fortran/CNS_divop.F90 Tests/CNS/Source/fortran/CNS_dudt.F90 Tests/CNS/Source/fortran/CNS_f.F90 Tests/CNS/Source/fortran/CNS_nd.F90 Tests/CNS/Source/fortran/CNS_physics.F90 Tests/CNS/Source/fortran/CNS_tagging.F90 Tests/CNS/Source/fortran/Make.package Tests/CNS/Source/fortran/bc_fill_nd.F90 Tests/CNS/Source/hydro/Hyp_gamma_MOL.F90 Tests/CNS/Source/hydro/Hyp_gamma_MOL_EB.F90 Tests/CNS/Source/hydro/Make.package Tests/CNS/Source/hydro/analriem3d.F90 Tests/CNS/Source/hydro/cns_eb_hyp_wall.F90 Tests/CNS/Source/hydro/slope_mol_3d_gamma.F90 Tests/CNS/Source/hydro/slope_mol_3d_gamma_EB.F90 Tests/CNS/Source/main.cpp Tests/FortranInterface/Advection_F/CMakeLists.txt Tests/FortranInterface/Advection_F/Exec/Make.Adv Tests/FortranInterface/Advection_F/Exec/SingleVortex/GNUmakefile Tests/FortranInterface/Advection_F/Exec/SingleVortex/Make.package Tests/FortranInterface/Advection_F/Exec/SingleVortex/Prob_2d.f90 Tests/FortranInterface/Advection_F/Exec/SingleVortex/Prob_3d.f90 Tests/FortranInterface/Advection_F/Exec/SingleVortex/face_velocity_2d.F90 Tests/FortranInterface/Advection_F/Exec/SingleVortex/face_velocity_3d.F90 Tests/FortranInterface/Advection_F/Exec/SingleVortex/inputs Tests/FortranInterface/Advection_F/Exec/SingleVortex/inputs.physbc Tests/FortranInterface/Advection_F/Exec/SingleVortex/inputs.rt Tests/FortranInterface/Advection_F/README Tests/FortranInterface/Advection_F/Source/Make.package Tests/FortranInterface/Advection_F/Source/Src_2d/Make.package Tests/FortranInterface/Advection_F/Source/Src_2d/advect_2d_mod.F90 Tests/FortranInterface/Advection_F/Source/Src_2d/compute_flux_2d.f90 Tests/FortranInterface/Advection_F/Source/Src_2d/slope_2d.f90 Tests/FortranInterface/Advection_F/Source/Src_3d/Make.package Tests/FortranInterface/Advection_F/Source/Src_3d/advect_3d_mod.F90 Tests/FortranInterface/Advection_F/Source/Src_3d/compute_flux_3d.f90 Tests/FortranInterface/Advection_F/Source/Src_3d/slope_3d.f90 Tests/FortranInterface/Advection_F/Source/amr_data_mod.F90 Tests/FortranInterface/Advection_F/Source/averagedown_mod.F90 Tests/FortranInterface/Advection_F/Source/bc_mod.F90 Tests/FortranInterface/Advection_F/Source/compute_dt_mod.F90 Tests/FortranInterface/Advection_F/Source/evolve_mod.F90 Tests/FortranInterface/Advection_F/Source/fillpatch_mod.F90 Tests/FortranInterface/Advection_F/Source/fmain.F90 Tests/FortranInterface/Advection_F/Source/initdata.F90 Tests/FortranInterface/Advection_F/Source/my_amr_mod.F90 Tests/FortranInterface/Advection_F/Source/plotfile_mod.F90 Tests/FortranInterface/Advection_F/Source/tagging_mod.F90 Tests/FortranInterface/Advection_octree_F/CMakeLists.txt Tests/FortranInterface/Advection_octree_F/Exec/Make.Adv Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/GNUmakefile Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/Make.package Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/Prob.f90 Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/face_velocity_2d.F90 Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/inputs Tests/FortranInterface/Advection_octree_F/Exec/SingleVortex/inputs.rt Tests/FortranInterface/Advection_octree_F/README Tests/FortranInterface/Advection_octree_F/Source/Make.package Tests/FortranInterface/Advection_octree_F/Source/Src_2d/Make.package Tests/FortranInterface/Advection_octree_F/Source/Src_2d/advect_2d_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/Src_2d/compute_flux_2d.f90 Tests/FortranInterface/Advection_octree_F/Source/Src_2d/slope_2d.f90 Tests/FortranInterface/Advection_octree_F/Source/amr_data_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/averagedown_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/bc_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/compute_dt_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/evolve_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/fillpatch_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/fmain.F90 Tests/FortranInterface/Advection_octree_F/Source/initdata.F90 Tests/FortranInterface/Advection_octree_F/Source/my_amr_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/plotfile_mod.F90 Tests/FortranInterface/Advection_octree_F/Source/tagging_mod.F90 Tests/GPU/CNS/CMakeLists.txt Tests/GPU/CNS/Exec/Make.CNS Tests/GPU/CNS/Exec/RT/GNUmakefile Tests/GPU/CNS/Exec/RT/Make.package Tests/GPU/CNS/Exec/RT/cns_prob.H Tests/GPU/CNS/Exec/RT/cns_prob.cpp Tests/GPU/CNS/Exec/RT/cns_prob_parm.H Tests/GPU/CNS/Exec/RT/inputs Tests/GPU/CNS/Exec/RT/inputs-rt Tests/GPU/CNS/Exec/Sod/GNUmakefile Tests/GPU/CNS/Exec/Sod/Make.package Tests/GPU/CNS/Exec/Sod/cns_prob.H Tests/GPU/CNS/Exec/Sod/cns_prob.cpp Tests/GPU/CNS/Exec/Sod/cns_prob_parm.H Tests/GPU/CNS/Exec/Sod/inputs Tests/GPU/CNS/Exec/Sod/inputs-rt Tests/GPU/CNS/Source/CNS.H Tests/GPU/CNS/Source/CNS.cpp Tests/GPU/CNS/Source/CNSBld.cpp Tests/GPU/CNS/Source/CNS_K.H Tests/GPU/CNS/Source/CNS_advance.cpp Tests/GPU/CNS/Source/CNS_bcfill.cpp Tests/GPU/CNS/Source/CNS_derive.H Tests/GPU/CNS/Source/CNS_derive.cpp Tests/GPU/CNS/Source/CNS_index_macros.H Tests/GPU/CNS/Source/CNS_io.cpp Tests/GPU/CNS/Source/CNS_parm.H Tests/GPU/CNS/Source/CNS_parm.cpp Tests/GPU/CNS/Source/CNS_setup.cpp Tests/GPU/CNS/Source/CNS_tagging.H Tests/GPU/CNS/Source/Make.package Tests/GPU/CNS/Source/diffusion/Make.package Tests/GPU/CNS/Source/hydro/CNS_hydro_K.H Tests/GPU/CNS/Source/hydro/Make.package Tests/GPU/CNS/Source/main.cpp Tests/GPU/Vector/inputs Tests/HDF5Benchmark/inputs Tests/LinearSolvers/ABecLaplacian_C/CMakeLists.txt Tests/LinearSolvers/ABecLaplacian_F/CMakeLists.txt Tests/LinearSolvers/MAC_Projection_EB/CMakeLists.txt Tests/LinearSolvers/NodalPoisson/CMakeLists.txt Tests/LinearSolvers/Nodal_Projection_EB/CMakeLists.txt Tests/LinearSolvers/NodeTensorLap/CMakeLists.txt Tests/MultiBlock/IndexType/main.cpp Tests/Particles/ParallelContext/CMakeLists.txt Tests/Particles/ParallelContext/inputs.rt Tests/Particles/ParallelContext/main.cpp Tests/Particles/Redistribute/inputs Tests/Particles/Redistribute/inputs.rt Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-dpcpp-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini Tutorials/Basic/Build_with_libamrex/GNUmakefile Tutorials/Basic/Build_with_libamrex/MyParams.H Tutorials/Basic/Build_with_libamrex/main.cpp Tutorials/Basic/Build_with_libamrex/my_func.f90 Tutorials/Basic/Build_with_libamrex/test_parameters.cpp Tutorials/Basic/HeatEquation_EX1_C/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_C/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX1_C/Exec/inputs Tutorials/Basic/HeatEquation_EX1_C/Source/Make.package Tutorials/Basic/HeatEquation_EX1_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX1_C/Source/myfunc.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/mykernel.H Tutorials/Basic/HeatEquation_EX1_CF/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_CF/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX1_CF/Exec/inputs_2d Tutorials/Basic/HeatEquation_EX1_CF/Exec/inputs_3d Tutorials/Basic/HeatEquation_EX1_CF/Source/Make.package Tutorials/Basic/HeatEquation_EX1_CF/Source/advance.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/advance_2d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/advance_3d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/myfunc.H Tutorials/Basic/HeatEquation_EX1_CF/Source/myfunc_F.H Tutorials/Basic/HeatEquation_EX1_F/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_F/GNUmakefile Tutorials/Basic/HeatEquation_EX1_F/Make.package Tutorials/Basic/HeatEquation_EX1_F/advance.f90 Tutorials/Basic/HeatEquation_EX1_F/fmain.F90 Tutorials/Basic/HeatEquation_EX1_F/init_phi.f90 Tutorials/Basic/HeatEquation_EX1_F/inputs Tutorials/Basic/HeatEquation_EX2_C/CMakeLists.txt Tutorials/Basic/HeatEquation_EX2_C/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX2_C/Exec/inputs Tutorials/Basic/HeatEquation_EX2_C/Source/Make.package Tutorials/Basic/HeatEquation_EX2_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX2_C/Source/myfunc.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/mykernel.H Tutorials/Basic/HeatEquation_EX2_CF/CMakeLists.txt Tutorials/Basic/HeatEquation_EX2_CF/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX2_CF/Exec/inputs_2d Tutorials/Basic/HeatEquation_EX2_CF/Exec/inputs_3d Tutorials/Basic/HeatEquation_EX2_CF/Source/Make.package Tutorials/Basic/HeatEquation_EX2_CF/Source/advance.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/advance_2d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/advance_3d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/myfunc.H Tutorials/Basic/HeatEquation_EX2_CF/Source/myfunc_F.H Tutorials/Basic/HeatEquation_EX3_C/CMakeLists.txt Tutorials/Basic/HeatEquation_EX3_C/Exec/GNUmakefile Tutorials/Basic/HeatEquation_EX3_C/Exec/inputs_2d Tutorials/Basic/HeatEquation_EX3_C/Exec/inputs_3d Tutorials/Basic/HeatEquation_EX3_C/Source/Make.package Tutorials/Basic/HeatEquation_EX3_C/Source/advance.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX3_C/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX3_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX3_C/Source/myfunc_F.H Tutorials/Basic/HelloWorld_C/CMakeLists.txt Tutorials/Basic/HelloWorld_C/GNUmakefile Tutorials/Basic/HelloWorld_C/Make.package Tutorials/Basic/HelloWorld_C/main.cpp Tutorials/Basic/HelloWorld_F/CMakeLists.txt Tutorials/Basic/HelloWorld_F/GNUmakefile Tutorials/Basic/HelloWorld_F/Make.package Tutorials/Basic/HelloWorld_F/fmain.f90 Tutorials/Basic/PrefixSum_MultiFab/CMakeLists.txt Tutorials/Basic/PrefixSum_MultiFab/GNUmakefile Tutorials/Basic/PrefixSum_MultiFab/Make.package Tutorials/Basic/PrefixSum_MultiFab/Parallel-Prefix-Sum.ipynb Tutorials/Basic/PrefixSum_MultiFab/README.md Tutorials/Basic/PrefixSum_MultiFab/inputs Tutorials/Basic/PrefixSum_MultiFab/main.cpp Tutorials/Basic/main_C/CMakeLists.txt Tutorials/Basic/main_C/GNUmakefile Tutorials/Basic/main_C/Make.package Tutorials/Basic/main_C/main.cpp Tutorials/Basic/main_F/CMakeLists.txt Tutorials/Basic/main_F/GNUmakefile Tutorials/Basic/main_F/Make.package Tutorials/Basic/main_F/main.F90 Tutorials/Blueprint/AssignMultiLevelDensity/CMakeLists.txt Tutorials/Blueprint/AssignMultiLevelDensity/GNUmakefile Tutorials/Blueprint/AssignMultiLevelDensity/Make.package Tutorials/Blueprint/AssignMultiLevelDensity/inputs Tutorials/Blueprint/AssignMultiLevelDensity/main.cpp Tutorials/Blueprint/CellSortedParticles/CMakeLists.txt Tutorials/Blueprint/CellSortedParticles/CellSortedPC.H Tutorials/Blueprint/CellSortedParticles/CellSortedPC.cpp Tutorials/Blueprint/CellSortedParticles/GNUmakefile Tutorials/Blueprint/CellSortedParticles/Make.package Tutorials/Blueprint/CellSortedParticles/cell_sorted_3d.F90 Tutorials/Blueprint/CellSortedParticles/cell_sorted_F.H Tutorials/Blueprint/CellSortedParticles/inputs Tutorials/Blueprint/CellSortedParticles/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/CMakeLists.txt Tutorials/Blueprint/HeatEquation_EX1_C/Exec/GNUmakefile Tutorials/Blueprint/HeatEquation_EX1_C/Exec/inputs_2d Tutorials/Blueprint/HeatEquation_EX1_C/Exec/inputs_3d Tutorials/Blueprint/HeatEquation_EX1_C/Source/Make.package Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance_2d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance_3d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/init_phi_2d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/init_phi_3d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/myfunc.H Tutorials/Blueprint/HeatEquation_EX1_C/Source/myfunc_F.H Tutorials/Blueprint/README.txt Tutorials/CMakeLists.txt Tutorials/EB/GeometryGeneration/CMakeLists.txt Tutorials/EB/GeometryGeneration/GNUmakefile Tutorials/EB/GeometryGeneration/Make.package Tutorials/EB/GeometryGeneration/main.cpp Tutorials/EB/MacProj/CMakeLists.txt Tutorials/EB/MacProj/GNUmakefile Tutorials/EB/MacProj/Make.package Tutorials/EB/MacProj/inputs Tutorials/EB/MacProj/main.cpp Tutorials/EB/Poisson/CMakeLists.txt Tutorials/EB/Poisson/GNUmakefile Tutorials/EB/Poisson/Make.package Tutorials/EB/Poisson/Poisson.H Tutorials/EB/Poisson/Poisson.cpp Tutorials/EB/Poisson/inputs Tutorials/EB/Poisson/main.cpp Tutorials/EB/STLtest/GNUmakefile Tutorials/EB/STLtest/Make.package Tutorials/EB/STLtest/airfoil.stl Tutorials/EB/STLtest/inputs Tutorials/EB/STLtest/main.cpp Tutorials/ForkJoin/MLMG/CMakeLists.txt Tutorials/ForkJoin/MLMG/GNUmakefile Tutorials/ForkJoin/MLMG/Make.package Tutorials/ForkJoin/MLMG/ff.f90 Tutorials/ForkJoin/MLMG/inputs Tutorials/ForkJoin/MLMG/main.cpp Tutorials/ForkJoin/Simple/CMakeLists.txt Tutorials/ForkJoin/Simple/GNUmakefile Tutorials/ForkJoin/Simple/Make.package Tutorials/ForkJoin/Simple/MyTest.H Tutorials/ForkJoin/Simple/MyTest.cpp Tutorials/ForkJoin/Simple/MyTest_F.H Tutorials/ForkJoin/Simple/inputs Tutorials/ForkJoin/Simple/main.cpp Tutorials/FortranInterface/Advection_octree_F2/CMakeLists.txt Tutorials/FortranInterface/Advection_octree_F2/Exec/Make.Adv Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/GNUmakefile Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/Make.package Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/Prob.f90 Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/face_velocity_2d.F90 Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/inputs Tutorials/FortranInterface/Advection_octree_F2/README Tutorials/FortranInterface/Advection_octree_F2/Source/Make.package Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/Make.package Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/advect_2d_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/compute_flux_2d.f90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/slope_2d.f90 Tutorials/FortranInterface/Advection_octree_F2/Source/amr_data_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/averagedown_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/bc_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/compute_dt_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/evolve_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/fillpatch_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/fmain.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/initdata.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/my_amr_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/plotfile_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/tagging_mod.F90 Tutorials/GPU/EBCNS/CMakeLists.txt Tutorials/GPU/EBCNS/Exec/Make.CNS Tutorials/GPU/EBCNS/Exec/Sod/GNUmakefile Tutorials/GPU/EBCNS/Exec/Sod/Make.package Tutorials/GPU/EBCNS/Exec/Sod/cns_prob.H Tutorials/GPU/EBCNS/Exec/Sod/cns_prob.cpp Tutorials/GPU/EBCNS/Exec/Sod/cns_prob_parm.H Tutorials/GPU/EBCNS/Exec/Sod/inputs Tutorials/GPU/EBCNS/README Tutorials/GPU/EBCNS/Source/CNS.H Tutorials/GPU/EBCNS/Source/CNS.cpp Tutorials/GPU/EBCNS/Source/CNSBld.cpp Tutorials/GPU/EBCNS/Source/CNS_K.H Tutorials/GPU/EBCNS/Source/CNS_advance.cpp Tutorials/GPU/EBCNS/Source/CNS_bcfill.cpp Tutorials/GPU/EBCNS/Source/CNS_derive.H Tutorials/GPU/EBCNS/Source/CNS_derive.cpp Tutorials/GPU/EBCNS/Source/CNS_index_macros.H Tutorials/GPU/EBCNS/Source/CNS_init_eb2.cpp Tutorials/GPU/EBCNS/Source/CNS_io.cpp Tutorials/GPU/EBCNS/Source/CNS_parm.H Tutorials/GPU/EBCNS/Source/CNS_parm.cpp Tutorials/GPU/EBCNS/Source/CNS_setup.cpp Tutorials/GPU/EBCNS/Source/CNS_tagging.H Tutorials/GPU/EBCNS/Source/Make.package Tutorials/GPU/EBCNS/Source/diffusion/Make.package Tutorials/GPU/EBCNS/Source/hydro/CNS_hydro_K.H Tutorials/GPU/EBCNS/Source/hydro/Make.package Tutorials/GPU/EBCNS/Source/main.cpp Tutorials/GPU/Launch/CMakeLists.txt Tutorials/GPU/Launch/GNUmakefile Tutorials/GPU/Launch/Make.package Tutorials/GPU/Launch/MyKernel_F.F90 Tutorials/GPU/Launch/MyKernel_F.H Tutorials/GPU/Launch/Readme.md Tutorials/GPU/Launch/main.cpp Tutorials/GPU/ParallelReduce/CMakeLists.txt Tutorials/GPU/ParallelReduce/GNUmakefile Tutorials/GPU/ParallelReduce/Make.package Tutorials/GPU/ParallelReduce/main.cpp Tutorials/GPU/ParallelScan/CMakeLists.txt Tutorials/GPU/ParallelScan/GNUmakefile Tutorials/GPU/ParallelScan/Make.package Tutorials/GPU/ParallelScan/main.cpp Tutorials/GPU/run.corigpu Tutorials/GPU/run.summit Tutorials/MUI/Exec_01/GNUmakefile Tutorials/MUI/Exec_02/GNUmakefile Tutorials/MUI/Exec_coupled/cmd_mpirun Tutorials/MUI/Exec_coupled/inputs Tutorials/MUI/Source_01/Make.package Tutorials/MUI/Source_01/init_phi_3d.f90 Tutorials/MUI/Source_01/main_01.cpp Tutorials/MUI/Source_01/myfunc.H Tutorials/MUI/Source_01/myfunc_F.H Tutorials/MUI/Source_02/Make.package Tutorials/MUI/Source_02/main_02.cpp Tutorials/MUI/Source_02/myfunc.H Tutorials/MUI/doc/GNUmakefile Tutorials/MUI/doc/MUIcouplingNotes.tex Tutorials/MUI/doc/iface_rect.png Tutorials/MUI/doc/vis_interface.m Tutorials/Particles/CellSortedParticles/CMakeLists.txt Tutorials/Particles/CellSortedParticles/CellSortedPC.H Tutorials/Particles/CellSortedParticles/CellSortedPC.cpp Tutorials/Particles/CellSortedParticles/GNUmakefile Tutorials/Particles/CellSortedParticles/Make.package Tutorials/Particles/CellSortedParticles/cell_sorted_3d.F90 Tutorials/Particles/CellSortedParticles/cell_sorted_F.H Tutorials/Particles/CellSortedParticles/inputs Tutorials/Particles/CellSortedParticles/main.cpp Tutorials/Particles/ElectromagneticPIC/CMakeLists.txt Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/Evolve.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/GNUmakefile Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/Make.package Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/em_pic_K.H Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/inputs Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/Evolve.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/GNUmakefile Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/Make.package Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/em_pic_3d.F90 Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/em_pic_F.H Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/inputs Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/script.sh Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/Evolve.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/GNUmakefile Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/GNUmakefile.libamrex Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/Make.package Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/em_pic_3d.F90 Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/em_pic_F.H Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/inputs Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/run.summitdev Tutorials/Particles/ElectromagneticPIC/Make.EMPIC Tutorials/Particles/ElectromagneticPIC/Source/Constants.H Tutorials/Particles/ElectromagneticPIC/Source/EMParticleContainer.H Tutorials/Particles/ElectromagneticPIC/Source/EMParticleContainerInit.cpp Tutorials/Particles/ElectromagneticPIC/Source/Evolve.H Tutorials/Particles/ElectromagneticPIC/Source/IO.H Tutorials/Particles/ElectromagneticPIC/Source/IO.cpp Tutorials/Particles/ElectromagneticPIC/Source/Make.package Tutorials/Particles/ElectromagneticPIC/Source/NodalFlags.H Tutorials/Particles/ElectromagneticPIC/Source/NodalFlags.cpp Tutorials/Particles/ElectromagneticPIC/Source/main.cpp Tutorials/Particles/ElectrostaticPIC/CMakeLists.txt Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.H Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.cpp Tutorials/Particles/ElectrostaticPIC/GNUmakefile Tutorials/Particles/ElectrostaticPIC/Make.package Tutorials/Particles/ElectrostaticPIC/PhysConst.H Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_2d.f90 Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_3d.f90 Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_F.H Tutorials/Particles/ElectrostaticPIC/inputs Tutorials/Particles/ElectrostaticPIC/main.cpp Tutorials/Particles/NeighborList/CMakeLists.txt Tutorials/Particles/NeighborList/CheckPair.H Tutorials/Particles/NeighborList/Constants.H Tutorials/Particles/NeighborList/GNUmakefile Tutorials/Particles/NeighborList/MDParticleContainer.H Tutorials/Particles/NeighborList/MDParticleContainer.cpp Tutorials/Particles/NeighborList/Make.package Tutorials/Particles/NeighborList/README.md Tutorials/Particles/NeighborList/inputs Tutorials/Particles/NeighborList/main.cpp Tutorials/Particles/NeighborList/script.sh Tutorials/README.md Tutorials/SDC/MISDC_ADR_2d/Exec/GNUmakefile Tutorials/SDC/MISDC_ADR_2d/Exec/inputs_2d Tutorials/SDC/MISDC_ADR_2d/README Tutorials/SDC/MISDC_ADR_2d/Source/Make.package Tutorials/SDC/MISDC_ADR_2d/Source/SDC_sweeper.cpp Tutorials/SDC/MISDC_ADR_2d/Source/functions_2d.f90 Tutorials/SDC/MISDC_ADR_2d/Source/init_phi_2d.f90 Tutorials/SDC/MISDC_ADR_2d/Source/main.cpp Tutorials/SDC/MISDC_ADR_2d/Source/myfunc.H Tutorials/SDC/MISDC_ADR_2d/Source/myfunc_F.H Tutorials/SWFFT/SWFFT_poisson/GNUmakefile Tutorials/SWFFT/SWFFT_poisson/Make.package Tutorials/SWFFT/SWFFT_poisson/README Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test.H Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test.cpp Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test_F.F90 Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test_F.H Tutorials/SWFFT/SWFFT_poisson/inputs.128 Tutorials/SWFFT/SWFFT_poisson/inputs.32 Tutorials/SWFFT/SWFFT_poisson/inputs.64 Tutorials/SWFFT/SWFFT_poisson/main.cpp Tutorials/SWFFT/SWFFT_poisson/run_me Tutorials/SWFFT/SWFFT_poisson/swfft_solver.cpp Tutorials/SWFFT/SWFFT_simple/GNUmakefile Tutorials/SWFFT/SWFFT_simple/Make.package Tutorials/SWFFT/SWFFT_simple/README Tutorials/SWFFT/SWFFT_simple/SWFFT_Test.H Tutorials/SWFFT/SWFFT_simple/SWFFT_Test.cpp Tutorials/SWFFT/SWFFT_simple/SWFFT_Test_F.F90 Tutorials/SWFFT/SWFFT_simple/SWFFT_Test_F.H Tutorials/SWFFT/SWFFT_simple/inputs.multipleGrids Tutorials/SWFFT/SWFFT_simple/inputs.oneGrid Tutorials/SWFFT/SWFFT_simple/main.cpp Tutorials/SWFFT/SWFFT_simple/run_me_2d Tutorials/SWFFT/SWFFT_simple/run_me_3d Tutorials/SWFFT/SWFFT_simple/swfft_compute.cpp commit 384ca2341889a8f1db78e87cb4b1735bd2d58554 Author: Axel Huebl Date: Mon Mar 22 10:09:53 2021 -0700 CMake: IPO/LTO (#1890) Add optional interprocedural optimization (IPO) aka link-time optimization (LTO) to AMReX. In my WarpX tests recently on x86_64 and ppc64le, using those options incurred a 2% performance hit (although one would expect a slight increase). Thus, they are not enabled by default. Also, enabling them with some toolchains needs users to set the linker properly. Nontheless, enabling IPO shrinks the binary size a lot. This is very valuable and worth the runtime penalty for our binary deployments, e.g. when shipping generic x86_64, aarch64 and ppc64le binaries on conda-forge to users. WarpX binary size (default build): * no IPO: 217MiB * WarpX IPO, AMReX no IPO: 155MiB * both WarpX and AMReX with IPO: 127M Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXBuildInfo.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXTypecheck.cmake Tools/CMake/AMReX_Config.cmake commit ef0eb9f5c4b065b38e3a3242a43263aba43aa874 Author: Weiqun Zhang Date: Sun Mar 21 14:35:07 2021 -0700 Header file order and iosfwd (#1883) Include amrex headers before standard C++ headers. Use iosfwd instead of iostream if we can. Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Derive.cpp Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.H Src/Base/AMReX_Array.H Src/Base/AMReX_Array4.H Src/Base/AMReX_BCRec.cpp Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_BaseFab.cpp Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxDomain.H Src/Base/AMReX_BoxDomain.cpp Src/Base/AMReX_BoxIterator.H Src/Base/AMReX_BoxIterator.cpp Src/Base/AMReX_BoxList.H Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_CoordSys.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_Dim3.H Src/Base/AMReX_Dim3.cpp Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabConv.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuComplex.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_IndexType.H Src/Base/AMReX_IndexType.cpp Src/Base/AMReX_IntConv.H Src/Base/AMReX_IntVect.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MemProfiler.H Src/Base/AMReX_MemProfiler.cpp Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_Orientation.H Src/Base/AMReX_Orientation.cpp Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_Periodicity.cpp Src/Base/AMReX_PlotFileDataImpl.H Src/Base/AMReX_PlotFileDataImpl.cpp Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_PlotFileUtil.cpp Src/Base/AMReX_Print.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_RealVect.H Src/Base/AMReX_RealVect.cpp Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_Tuple.H Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Utility.H Src/Base/AMReX_Vector.H Src/Base/AMReX_VectorIO.H Src/Base/AMReX_VectorIO.cpp Src/Base/AMReX_VisMF.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_iMultiFab.cpp Src/Base/AMReX_parmparse_fi.cpp Src/Base/AMReX_parstream.H Src/Base/AMReX_parstream.cpp Src/Base/CMakeLists.txt Src/Base/Make.package Src/Boundary/AMReX_Mask.cpp Src/EB/AMReX_EB2_IF_Base.H Src/EB/AMReX_EB2_IF_Spline.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBCellFlag.cpp Src/EB/AMReX_EBToPVD.H Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_distFcnElement.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/F_Interfaces/AmrCore/AMReX_FAmrCore.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleMPIUtil.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.cpp commit c984473e021aabdd5f5d2091dcc57481e8637c7c Author: Weiqun Zhang Date: Sun Mar 21 11:38:21 2021 -0700 Remove outdated notes (#1889) Docs/Notes/DPCPPWishlist.md Docs/Notes/HIPIssues.md Docs/Notes/Notes.io_implementation commit 15f5168a1ad9f49d87ff0ee25935e171fe1d0f19 Author: Weiqun Zhang Date: Fri Mar 19 20:32:22 2021 -0700 amrex::EnableIf_t -> std::enable_if_t (#1885) Since we require C++14 now, we can use std::enable_if_t now. Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_TypeTraits.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H commit 1dfb98c29afe574afe30101a8db82190d9eadaa6 Author: Andrew Myers Date: Fri Mar 19 23:31:57 2021 -0400 remove __attribute__((warn_unused_result)) from AMREX_NODISCARD (#1886) Src/Base/AMReX_Extension.H commit 352059135e3e87f3b35e237b1c7fb8961eabffa1 Author: Weiqun Zhang Date: Fri Mar 19 18:11:52 2021 -0700 Make FabArray::setFab safer (#1884) Instead of taking a raw pointer, it takes std::unique_ptr or rvalue FAB, so that the ownership is very clear. It also deletes the old FAB if not null. Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_ParallelDescriptor.H Src/EB/AMReX_EB2_MultiGFab.cpp Src/EB/AMReX_MultiCutFab.cpp Src/Extern/amrdata/AMReX_AmrData.cpp commit 9e1576fab02e20b19d54105e6e7a434bf6d6f040 Author: Weiqun Zhang Date: Fri Mar 19 17:00:02 2021 -0700 Fix typo in AMREX_DEBUG macro (#1881) Fix the call to CheckRcvStats. Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H commit ea9ba93abd20c9a4c79f2ae8c392cd62a553119f Author: Maikel Nadolski Date: Fri Mar 19 23:23:51 2021 +0100 Forward declare functions that are used in the implementation (#1882) This PR forward declares the functions local_copy_cpu unpack_recv_buffer_cpu local_copy_gpu unpack_recv_buffer_gpu before its first usage. Some compilers error out if a function is used before it was declared, even in template code. I am not sure, why it worked in the CI so far. Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H commit ca1839a8d9d805e915d6bd9d385a0c3087ad866d Author: Axel Huebl Date: Fri Mar 19 12:44:33 2021 -0700 Flags: RelWithDebInfo (#1879) Add warning control to CMake Build mode `RelWithDebugInfo` (currently default for WarpX). Tools/CMake/AMReXFlagsTargets.cmake commit 82664e56d3dbdefa75bec47ddd871996778805c3 Author: Axel Huebl Date: Fri Mar 19 08:47:04 2021 -0700 Mask: -Wfinal-dtor-non-final-class (#1880) ## Summary ``` Src/Boundary/AMReX_Mask.H:57:40: warning: class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class] virtual ~Mask () noexcept override final {} ^ Src/Boundary/AMReX_Mask.H:26:7: note: mark 'amrex::Mask' as 'final' to silence this warning class Mask ``` Fix the warning in TagBox, CutFab and EBCellFlagFab as well. ## Additional background Seen with Clang 10.0 on Ubuntu 20.04 LTS. Co-authored-by: Weiqun Zhang Src/AmrCore/AMReX_TagBox.H Src/Boundary/AMReX_Mask.H Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_MultiCutFab.H commit 0e1acd6c559b8939ac323446963dfafbf7430052 Author: Andrew Myers Date: Fri Mar 19 01:50:12 2021 -0400 Fix Typedescriptor test for Windows (#1878) Tests/Particles/TypeDescriptor/main.cpp commit 281ff0fb583047355e0a726805c08d5d4e20cf67 Author: Axel Huebl Date: Thu Mar 18 11:42:30 2021 -0700 Build: Windows with Shared Libs (.dll) (#1847) * CI: Win with Shared Libs (.dll) Demonstrator of shared library issues on Windows. * Remove: warn-unresolved-symbols / undefined,warning * CMake typecheckobjs lib: export symbols Make this library behave like on Unix: symbol visibility by default. * Globals: dllimport/dllexport (Win) https://stackoverflow.com/questions/54560832/cmake-windows-export-all-symbols-does-not-cover-global-variables/54568678#54568678 .github/workflows/windows.yml Src/AmrCore/AMReX_Interpolater.H Src/Base/AMReX.H Src/Base/AMReX_Extension.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_VisMF.H Src/EB/AMReX_EBInterpolater.H Src/Particle/AMReX_ParticleContainerBase.H Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReXTypecheck.cmake Tools/CMake/AMReX_Config.H.in Tools/CMake/AMReX_Config.cmake Tools/GNUMake/tools/Make.sensei commit 17371b17ffd5f7794a91236de7fa464d4284fa86 Author: Andrew Myers Date: Thu Mar 18 13:24:45 2021 -0400 Remove tutorials documentation from main repo; it now lives at https://github.com/AMReX-Codes/amrex-tutorials (#1877) Docs/sphinx_tutorials/Makefile Docs/sphinx_tutorials/source/AMR_Tutorial.rst Docs/sphinx_tutorials/source/Basic_Tutorial.rst Docs/sphinx_tutorials/source/Blueprint_Tutorial.rst Docs/sphinx_tutorials/source/EB_Tutorial.rst Docs/sphinx_tutorials/source/ForkJoin_Tutorial.rst Docs/sphinx_tutorials/source/GPU_Tutorial.rst Docs/sphinx_tutorials/source/LinearSolvers_Tutorial.rst Docs/sphinx_tutorials/source/MUI_Tutorial.rst Docs/sphinx_tutorials/source/Particles_Tutorial.rst Docs/sphinx_tutorials/source/SDC_Tutorial.rst Docs/sphinx_tutorials/source/SWFFT/iface_rect.png Docs/sphinx_tutorials/source/SWFFT_Tutorial.rst Docs/sphinx_tutorials/source/_static/theme_overrides.css Docs/sphinx_tutorials/source/conf.py Docs/sphinx_tutorials/source/figs/fork_join_tasks.png Docs/sphinx_tutorials/source/figs/mf_remap_hires.png Docs/sphinx_tutorials/source/figs/nested_fork_join_tasks.png Docs/sphinx_tutorials/source/index.rst build_docs.sh commit b31d050d176c822d51dad418a868d62426c6a1b4 Author: Weiqun Zhang Date: Wed Mar 17 19:13:57 2021 -0700 2D Poisson and ALaplacian Solvers in 3D Build of AMReX (#1805) New capability of solving 2D cell-centered Poisson's and ALaplacian equation with 3D build of amrex. Src/Base/AMReX_Box.H Src/Base/AMReX_Box.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_K.H Src/LinearSolvers/MLMG/Make.package commit 14b6bff6954d677b496af065c8c9def6bacc2d43 Author: Weiqun Zhang Date: Wed Mar 17 18:26:39 2021 -0700 Add FabArray::release and Make BaseFab virtual (#1873) Add FabArray::release functions that release the ownership of the FAB at the given index and return a raw pointer to the FAB. Make BaseFab virtual so that it's safe to use the new release functions. Also declare a number of functions `[[nodiscard]]`. Fix some warnings in EB/CNS example. Src/AmrCore/AMReX_TagBox.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabFactory.H Src/Base/AMReX_IArrayBox.H Src/Boundary/AMReX_Mask.H Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBFabFactory.H Src/EB/AMReX_EBFabFactory.cpp Src/EB/AMReX_MultiCutFab.H Tutorials/EB/CNS/Source/CNS.cpp Tutorials/EB/CNS/Source/CNS_advance.cpp Tutorials/EB/CNS/Source/CNS_init_eb2.cpp commit 87b182a718574cc52a93750a4c484c2e5aa4264b Author: Weiqun Zhang Date: Wed Mar 17 15:16:57 2021 -0700 Robin BC in Linear Solver (#1848) Add Robin BC to MLABecLaplacian. There are two steps for setting up Robin BC. The first step is to call setDomainBC, and the second step is to call setLevelBC, which has been modified to take a, b and f in Robin BC, `a*phi + b*dphi/dn = f`. Docs/sphinx_documentation/source/LinearSolvers.rst Src/Boundary/AMReX_LO_BCTYPES.H Src/Boundary/AMReX_LO_BCTYPES.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit 5b235f13980b2602636eff689f95d96a99c75d06 Author: Weiqun Zhang Date: Wed Mar 17 10:06:45 2021 -0700 LevelBld (#1852) In Amr class, we need to get LevelBld* and use it to build AmrLevels. Previously, Amr called a function `LevelBld* getLevelBld()` to get that. The problem was that `getLevelBld` was not defined by amrex but it was being called by amrex. This caused issues in building amrex as a shared library on Mac and Windows, because they do not support weak symbols. In this PR, the constructors of Amr now take `LevelBld*` as an argument. This is unfortunately a breaking change. However, it's a compile time failure and it's easy to fix. See the changes in this commit for examples. Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_LevelBld.H Tutorials/Amr/Advection_AmrLevel/Source/main.cpp Tutorials/EB/CNS/Source/main.cpp Tutorials/GPU/CNS/Source/main.cpp Tutorials/GPU/EBCNS/Source/main.cpp commit 5d32c74750b59e06c5731e084a75eec6ab91aad9 Author: drangara <69211175+drangara@users.noreply.github.com> Date: Wed Mar 17 11:24:36 2021 -0400 Fix bug with face-centroid slopes when using multigrid (#1871) When computing the Least Square slopes on face-centroid stencil with multigrid, we need to grow `m_eb_phi` and make additional checks. This adds to the changes made in https://github.com/AMReX-Codes/amrex/pull/1707 Src/EB/AMReX_EB_LeastSquares_2D_K.H Src/EB/AMReX_EB_LeastSquares_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Tests/LinearSolvers/LeastSquares/inputs.2d.askew-x.mg Tests/LinearSolvers/LeastSquares/inputs.2d.askew-y.mg Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-all.mg commit c28d91c2ee07ad6f970eb10bbc14d00af6adab4a Author: Weiqun Zhang Date: Tue Mar 16 14:59:05 2021 -0700 Avoid conflict with X11 (#1872) Bool -> B to avoid a conflict with an X11 header that defines ``` #define Bool int #define Status int #define True 1 #define False 0 ``` This should fix the compilation issue of Amrvis. Src/Base/AMReX_TypeTraits.H commit 44f77978de2bdaec40cc6136169b854a78ff167e Author: Andrew Myers Date: Tue Mar 16 11:23:18 2021 -0400 Add explanation of parameters to Redistribute docstring (#1868) Src/Particle/AMReX_Particles.H commit e1b123b2302c0377efb9ab99066a60f907c663c9 Author: Weiqun Zhang Date: Mon Mar 15 18:05:17 2021 -0700 Fix sign-compare and shadow warnings (#1867) The warning about shadowed varible in AMReX_NonLocalBC.H seems a compiler bug. Nevertheless, the source is modified to get rid of the warning. Src/Base/AMReX_NonLocalBC.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H commit 7d833e64ff06477953c9b3a2642b04e61116be93 Author: Weiqun Zhang Date: Mon Mar 15 12:42:05 2021 -0700 EditorConfig and trailing whitespaces (#1866) Trim trailing whitespaces in editorconfig for C/C++ codes and others. Allow md file to have trailing whitespaces because they are syntactically significant. .editorconfig .github/workflows/style/check_trailing_whitespaces.sh commit e5b06d59e3a6cf55c40c6498e42eafe940203cb9 Author: Weiqun Zhang Date: Mon Mar 15 12:28:06 2021 -0700 Optimization of CArena::freeUnused (#1865) Use `find` instead of manully looping over the set. Src/Base/AMReX_CArena.cpp commit fbef03f566fbbe3c583bd96db00b5cee391ad6b8 Author: Axel Huebl Date: Mon Mar 15 12:25:52 2021 -0700 CI: macOS Shared Lib (#1836) ## Summary CI for macOS: Build a shared library. This is the default in Spack. I see issues with AppleClang 12.0 raising: ``` ld: can't use -undefined warning or suppress with -twolevel_namespace clang: error: linker command failed with exit code 1 (use -v to see invocation) ``` @RemiLehe reported this again here: https://github.com/AMReX-Codes/amrex/issues/425#issuecomment-473153152 - [x] reproduce issue - [x] add fix ## Additional background Introduced in 1b6af3cb630 to fix #425 .github/workflows/dependencies/dependencies_mac.sh .github/workflows/macos.yml Src/Amr/AMReX_LevelBld.H Src/Amr/AMReX_PROB_AMR_F.H Src/Base/AMReX_Extension.H Tools/CMake/AMReX_Config.cmake commit 2789d017a7d1e4803fe9ca007dbbad611e42069f Author: Weiqun Zhang Date: Sun Mar 14 14:39:58 2021 -0700 Fix bug in #1861 (#1864) We cannot simply use std::set::erase on the free node list because the operator== only compares the starting block address. Src/Base/AMReX_CArena.cpp commit 107b5d4f36063b110539242928faba914f07e5b3 Author: Weiqun Zhang Date: Sun Mar 14 14:25:49 2021 -0700 Fix CUDA API calls in host callback functions (#1863) CUDA API calls are not allowed in host callback functions. Therefore, we have to move the call to free unused memory from `free` to `alloc`. Src/Base/AMReX_CArena.cpp commit d28caf8685f6c65639aee1d7d1731901480b3a14 Author: Weiqun Zhang Date: Sun Mar 14 14:19:48 2021 -0700 CI for Code Style (#1862) Check for tabs and trailing whitespaces at the end of line in CI. .github/workflows/style.yml .github/workflows/style/check_tabs.sh .github/workflows/style/check_trailing_whitespaces.sh .gitignore commit 486cd438aea970962a02c799c910eb8ddcd7dd71 Author: CCSE@LBNL Date: Sun Mar 14 08:38:31 2021 -0700 Remove tabs and trailing white spaces (#1860) The changes are purely white spaces only. `git diff -w development..HEAD` shows nothing. It was done with ``` find . -type d \( -name .git \ -o -path ./paper \ \) -prune -o \ -type f \( -name "*.H" -o -name "*.h" -o -name "*.hh" -o -name "*.hpp" \ -o -name "*.c" -o -name "*.cc" -o -name "*.cpp" -o -name "*.cxx" \ -o -name "*.f" -o -name "*.F" -o -name "*.f90" -o -name "*.F90" \ -o -name "*.py" \ -o -name "*.md" -o -name "*.rst" \ -o -name "*.sh" \ -o -name "*.tex" \ -o -name "*.txt" \ \) \ -exec grep -Iq . {} \; \ -exec sed -i 's/[[:blank:]]\+$//g' {} + \ -exec sed -i 's/\t/\ \ \ \ \ \ \ \ /g' {} + ``` We will add a CI check as a follow-up. CONTRIBUTING.md Docs/Migration/Migration.md Docs/sphinx_documentation/source/AMReX_Profiling_Tools_Chapter.rst Docs/sphinx_documentation/source/AmrCore.rst Docs/sphinx_documentation/source/AmrLevel.rst Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/DualGrid.rst Docs/sphinx_documentation/source/EB.rst Docs/sphinx_documentation/source/External_Profiling_Tools.rst Docs/sphinx_documentation/source/Fortran.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Docs/sphinx_documentation/source/GettingStarted.rst Docs/sphinx_documentation/source/GridCreation.rst Docs/sphinx_documentation/source/IO.rst Docs/sphinx_documentation/source/InputsLoadBalancing.rst Docs/sphinx_documentation/source/Inputs_Chapter.rst Docs/sphinx_documentation/source/Introduction.rst Docs/sphinx_documentation/source/LinearSolvers.rst Docs/sphinx_documentation/source/LinearSolvers_Chapter.rst Docs/sphinx_documentation/source/LoadBalancing.rst Docs/sphinx_documentation/source/ManagingGridHierarchy_Chapter.rst Docs/sphinx_documentation/source/Particle.rst Docs/sphinx_documentation/source/SWFFT.rst Docs/sphinx_documentation/source/Testing.rst Docs/sphinx_documentation/source/Visualization.rst Docs/sphinx_documentation/source/index.rst Docs/sphinx_tutorials/source/AMR_Tutorial.rst Docs/sphinx_tutorials/source/Basic_Tutorial.rst Docs/sphinx_tutorials/source/Blueprint_Tutorial.rst Docs/sphinx_tutorials/source/GPU_Tutorial.rst Docs/sphinx_tutorials/source/LinearSolvers_Tutorial.rst Docs/sphinx_tutorials/source/MUI_Tutorial.rst Docs/sphinx_tutorials/source/Particles_Tutorial.rst Docs/sphinx_tutorials/source/index.rst README.md Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_AuxBoundaryData.H Src/Amr/AMReX_AuxBoundaryData.cpp Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Derive.cpp Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_StateDescriptor.cpp Src/Amr/AMReX_extrapolater_3D_K.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrMesh.cpp Src/AmrCore/AMReX_AmrParGDB.H Src/AmrCore/AMReX_AmrParticles.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_2d.F90 Src/AmrCore/AMReX_FillPatchUtil_3d.F90 Src/AmrCore/AMReX_FillPatchUtil_F.H Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_INTERP_2D.F90 Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_BCRec.cpp Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_BLutil_F.F90 Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseFab.cpp Src/Base/AMReX_BaseUmap_f.H Src/Base/AMReX_BlockMutex.H Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_Box.H Src/Base/AMReX_Box.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxDomain.cpp Src/Base/AMReX_BoxList.H Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_COORDSYS_1D_C.H Src/Base/AMReX_COORDSYS_2D_C.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_CuptiTrace.H Src/Base/AMReX_CuptiTrace.cpp Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_Extension.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FILCC_1D.F90 Src/Base/AMReX_FILCC_2D.F90 Src/Base/AMReX_FILCC_3D.F90 Src/Base/AMReX_FPC.cpp Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_FabConv.cpp Src/Base/AMReX_FileSystem.cpp Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuComplex.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_INT.H Src/Base/AMReX_IntConv.H Src/Base/AMReX_IntConv.cpp Src/Base/AMReX_IntVect.H Src/Base/AMReX_Lazy.H Src/Base/AMReX_Lazy.cpp Src/Base/AMReX_MFCopyDescriptor.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MemProfiler.H Src/Base/AMReX_MemProfiler.cpp Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_NFiles.H Src/Base/AMReX_NFiles.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBC.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelContext.cpp Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_ParallelDescriptor_F.F90 Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_Partition.H Src/Base/AMReX_Periodicity.H Src/Base/AMReX_Periodicity.cpp Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_PlotFileUtil.cpp Src/Base/AMReX_Print.H Src/Base/AMReX_REAL.H Src/Base/AMReX_RealBox.cpp Src/Base/AMReX_RealVect.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_SPACE.H Src/Base/AMReX_Scan.H Src/Base/AMReX_Slopes_K.H Src/Base/AMReX_ThirdPartyProfiling.H Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Utility.H Src/Base/AMReX_Utility.cpp Src/Base/AMReX_Vector.H Src/Base/AMReX_VectorIO.H Src/Base/AMReX_VectorIO.cpp Src/Base/AMReX_VisMF.H Src/Base/AMReX_VisMF.cpp Src/Base/AMReX_bc_types_mod.F90 Src/Base/AMReX_ccse-mpi.H Src/Base/AMReX_error_fi.cpp Src/Base/AMReX_error_mod.F90 Src/Base/AMReX_filcc_mod.F90 Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_iMultiFab.cpp Src/Base/AMReX_mempool_mod.F90 Src/Base/AMReX_parmparse_fi.cpp Src/Base/AMReX_parmparse_mod.F90 Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryData.cpp Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_BndryRegister.cpp Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_FabSet.cpp Src/Boundary/AMReX_InterpBndryData.H Src/Boundary/AMReX_InterpBndryData.cpp Src/Boundary/AMReX_MacBndry.H Src/Boundary/AMReX_MacBndry.cpp Src/Boundary/AMReX_Mask.H Src/Boundary/AMReX_MultiMask.H Src/Boundary/AMReX_MultiMask.cpp Src/Boundary/AMReX_YAFluxRegister.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_IF_Complement.H Src/EB/AMReX_EB2_IF_Lathe.H Src/EB/AMReX_EB2_IF_Sphere.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EBFluxRegister.cpp Src/EB/AMReX_EBInterpolater.cpp Src/EB/AMReX_EBMultiFabUtil.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EBToPVD.H Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_EB_LeastSquares_2D_K.H Src/EB/AMReX_EB_LeastSquares_3D_K.H Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_slopes_K.H Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/AMReX_MultiCutFab.H Src/EB/AMReX_WriteEBSurface.H Src/EB/AMReX_WriteEBSurface.cpp Src/EB/AMReX_ebcellflag_mod.F90 Src/Extern/Conduit/AMReX_Conduit_Blueprint_ParticlesI.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_Hypre.cpp Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.H Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/AMReX_PETSc.cpp Src/Extern/ProfParser/AMReX_AVGDOWN_1D.F Src/Extern/ProfParser/AMReX_AVGDOWN_2D.F Src/Extern/ProfParser/AMReX_AVGDOWN_3D.F Src/Extern/ProfParser/AMReX_AVGDOWN_F.H Src/Extern/ProfParser/AMReX_BLProfStats.H Src/Extern/ProfParser/AMReX_BLProfStats.cpp Src/Extern/ProfParser/AMReX_BLProfUtilities.H Src/Extern/ProfParser/AMReX_BLProfUtilities.cpp Src/Extern/ProfParser/AMReX_BLWritePlotFile.H Src/Extern/ProfParser/AMReX_BLWritePlotFile.cpp Src/Extern/ProfParser/AMReX_CommProfStats.H Src/Extern/ProfParser/AMReX_CommProfStats.cpp Src/Extern/ProfParser/AMReX_ProfParserBatch.cpp Src/Extern/ProfParser/AMReX_RegionsProfStats.H Src/Extern/ProfParser/AMReX_RegionsProfStats.cpp Src/Extern/SWFFT/AlignedAllocator.h Src/Extern/SWFFT/CheckDecomposition.c Src/Extern/SWFFT/Dfft.H Src/Extern/SWFFT/DfftC.cpp Src/Extern/SWFFT/Distribution.H Src/Extern/SWFFT/DistributionC.cpp Src/Extern/SWFFT/Error.h Src/Extern/SWFFT/TimingStats.h Src/Extern/SWFFT/complex-type.h Src/Extern/SWFFT/distribution.c Src/Extern/SWFFT/distribution_c.h Src/Extern/amrdata/AMReX_AmrData.H Src/Extern/amrdata/AMReX_AmrData.cpp Src/Extern/amrdata/AMReX_DataServices.H Src/Extern/amrdata/AMReX_DataServices.cpp Src/Extern/amrdata/AMReX_FABUTIL_1D.F Src/Extern/amrdata/AMReX_FABUTIL_2D.F Src/Extern/amrdata/AMReX_FABUTIL_3D.F Src/Extern/amrdata/AMReX_XYPlotDataList.H Src/Extern/amrdata/AMReX_XYPlotDataList.cpp Src/Extern/amrdata/CMakeLists.txt Src/Extern/hpgmg/BL_HPGMG.H Src/Extern/hpgmg/BL_HPGMG.cpp Src/F_Interfaces/AmrCore/AMReX_FAmrCore.H Src/F_Interfaces/AmrCore/AMReX_FAmrCore.cpp Src/F_Interfaces/AmrCore/AMReX_amrcore_fi.cpp Src/F_Interfaces/AmrCore/AMReX_amrcore_mod.F90 Src/F_Interfaces/AmrCore/AMReX_fillpatch_fi.cpp Src/F_Interfaces/AmrCore/AMReX_fillpatch_mod.F90 Src/F_Interfaces/AmrCore/AMReX_fluxregister_fi.cpp Src/F_Interfaces/AmrCore/AMReX_fluxregister_mod.F90 Src/F_Interfaces/AmrCore/AMReX_tagbox_fi.cpp Src/F_Interfaces/AmrCore/AMReX_tagbox_mod.F90 Src/F_Interfaces/Base/AMReX_FPhysBC.cpp Src/F_Interfaces/Base/AMReX_box_fi.cpp Src/F_Interfaces/Base/AMReX_box_mod.F90 Src/F_Interfaces/Base/AMReX_boxarray_fi.cpp Src/F_Interfaces/Base/AMReX_distromap_fi.cpp Src/F_Interfaces/Base/AMReX_distromap_mod.F90 Src/F_Interfaces/Base/AMReX_geometry_fi.cpp Src/F_Interfaces/Base/AMReX_geometry_mod.F90 Src/F_Interfaces/Base/AMReX_init_mod.F90 Src/F_Interfaces/Base/AMReX_multifab_fi.cpp Src/F_Interfaces/Base/AMReX_multifab_mod.F90 Src/F_Interfaces/Base/AMReX_multifabutil_fi.cpp Src/F_Interfaces/Base/AMReX_parallel_mod.F90 Src/F_Interfaces/Base/AMReX_physbc_fi.cpp Src/F_Interfaces/Base/AMReX_physbc_mod.F90 Src/F_Interfaces/Base/AMReX_plotfile_fi.cpp Src/F_Interfaces/Base/AMReX_vismf_fi.cpp Src/F_Interfaces/LinearSolvers/AMReX_linop_mod.F90 Src/F_Interfaces/LinearSolvers/AMReX_multigrid_fi.cpp Src/F_Interfaces/LinearSolvers/AMReX_multigrid_mod.F90 Src/F_Interfaces/LinearSolvers/AMReX_poisson_mod.F90 Src/F_Interfaces/Octree/AMReX_octree_fi.cpp Src/F_Interfaces/Particle/AMReX_particlecontainer_fi.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_nd.F90 Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMGBndry.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_BinIterator.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleMPIUtil.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particle_mod_K.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_SparseBins.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.H Src/Particle/AMReX_TracerParticles.cpp Src/SDC/AMReX_SDCquadrature.F90 Src/SDC/AMReX_SDCstruct.H Src/SDC/AMReX_SDCstruct.cpp Tests/AsyncOut/multifab/main.cpp Tests/DivFreePatch/main.cpp Tests/FillBoundaryComparison/main.cpp Tests/GPU/AnyOf/main.cpp Tests/GPU/Fuse/main.cpp Tests/GPU/Vector/main.cpp Tests/HDF5Benchmark/main.cpp Tests/LinearSolvers/ABecLaplacian_C/MyTest.H Tests/LinearSolvers/ABecLaplacian_C/MyTest.cpp Tests/LinearSolvers/ABecLaplacian_C/initProb.cpp Tests/LinearSolvers/ABecLaplacian_C/initProb_K.H Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-16.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-32.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-1.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-16.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-2.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-32.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-4.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-64.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-1.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-16.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-2.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-32.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-4.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-64.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-8.sh Tests/LinearSolvers/ABecLaplacian_F/init_prob.F90 Tests/LinearSolvers/ABecLaplacian_F/mytest.F90 Tests/LinearSolvers/CellEB/MyTest.H Tests/LinearSolvers/CellEB/MyTest.cpp Tests/LinearSolvers/CellEB/initEB.cpp Tests/LinearSolvers/CellEB2/MyTest.H Tests/LinearSolvers/CellEB2/MyTest.cpp Tests/LinearSolvers/CellEB2/initEB.cpp Tests/LinearSolvers/EBConvergenceTest/BC_2D.F90 Tests/LinearSolvers/EBConvergenceTest/BC_3D.F90 Tests/LinearSolvers/EBConvergenceTest/Convergence_Data_Gen.sh Tests/LinearSolvers/EBConvergenceTest/MyTest.H Tests/LinearSolvers/EBConvergenceTest/MyTest.cpp Tests/LinearSolvers/EBConvergenceTest/MyTest_F.H Tests/LinearSolvers/EBConvergenceTest/RHS.F90 Tests/LinearSolvers/EBConvergenceTest/acoef.F90 Tests/LinearSolvers/EBConvergenceTest/bcoef.F90 Tests/LinearSolvers/EBTensor/MyTest.cpp Tests/LinearSolvers/EBTensor/MyTestPlotfile.cpp Tests/LinearSolvers/EBTensor/MyTest_2D_K.H Tests/LinearSolvers/EBflux_grad/MyTest.H Tests/LinearSolvers/EBflux_grad/MyTest.cpp Tests/LinearSolvers/EBflux_grad/initEB.cpp Tests/LinearSolvers/LeastSquares/MyTest.cpp Tests/LinearSolvers/LeastSquares/README.md Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/LeastSquares/initEB.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleData.cpp Tests/LinearSolvers/MAC_Projection_EB/main.cpp Tests/LinearSolvers/MLMG/fort_3d.F90 Tests/LinearSolvers/MLMG/init_prob.cpp Tests/LinearSolvers/MLMG/main.cpp Tests/LinearSolvers/MultiComponent/MCNodalLinOp.H Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/MultiComponent/main.cpp Tests/LinearSolvers/Nodal_Projection_EB/main.cpp Tests/LinearSolvers/NodeEB/MyTest.cpp Tests/MultiBlock/Advection/main.cpp Tests/MultiBlock/IndexType/main.cpp Tests/Particles/AssignDensity/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/AsyncIO/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/InitFromAscii/main.cpp Tests/Particles/Intersection/main.cpp Tests/Particles/NeighborParticles/CheckPair.H Tests/Particles/NeighborParticles/Constants.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/README.md Tests/Particles/ParticleIterator/main.cpp Tests/Particles/ParticleMesh/main.cpp Tests/Particles/ParticleTransformations/main.cpp Tests/Particles/SparseBins/main.cpp Tests/Particles/TypeDescriptor/main.cpp Tools/AMRProfParser/TestCodes/AMRProfTest0.cpp Tools/AMRProfParser/TestCodes/ProfWaitTest.cpp Tools/AMRProfParser/TestCodes/SendTest0.cpp Tools/AMRProfParser/TestCodes/TokenizeTest.cpp Tools/C_util/AmrDeriveTecplot/AmrDeriveTecplot.cpp Tools/C_util/AppendToPlotFile.cpp Tools/C_util/AugmentPlotfile/AugmentPlotfile.cpp Tools/C_util/AugmentPlotfile/AugmentPlotfile_F.H Tools/C_util/Convergence/Add.cpp Tools/C_util/Convergence/ComparePlotfiles.cpp Tools/C_util/Convergence/ComputeAmrDataNorms.H Tools/C_util/Convergence/ComputeAmrDataNorms.cpp Tools/C_util/Convergence/DebugDump.H Tools/C_util/Convergence/DebugOut.H Tools/C_util/Convergence/DebugOut.cpp Tools/C_util/Convergence/DiffFab.cpp Tools/C_util/Convergence/DiffSameDomainRefined.cpp Tools/C_util/Convergence/DiffSameDomainRefinedComposite.cpp Tools/C_util/Convergence/DiffSameDomainRefinedFD.cpp Tools/C_util/Convergence/DiffSameDomainRefinedStag.cpp Tools/C_util/Convergence/DiffSameGrid.cpp Tools/C_util/Convergence/DiffSameGrid2.cpp Tools/C_util/Convergence/DiffSameGridRefined.cpp Tools/C_util/Convergence/DiffUniform.cpp Tools/C_util/Convergence/PltFileNorm.cpp Tools/C_util/Convergence/PltFileNormB.cpp Tools/C_util/Convergence/PltFileScalConvRate.cpp Tools/C_util/Convergence/RichardsonConvergenceTest.cpp Tools/C_util/DiffMultiFab/diffmultifab.cpp Tools/C_util/Statistics/AVGDOWN_2D.F Tools/C_util/Statistics/AVGDOWN_3D.F Tools/C_util/Statistics/AVGDOWN_F.H Tools/C_util/Statistics/ComputeAmrDataStat.H Tools/C_util/Statistics/ComputeAmrDataStat.cpp Tools/C_util/Statistics/PltFileFluxAve.H Tools/C_util/Statistics/PltFileFluxAve.cpp Tools/C_util/Statistics/PltFileList.cpp Tools/C_util/Statistics/PltFileStat.cpp Tools/C_util/Statistics/PltFileXAve.H Tools/C_util/Statistics/PltFileXAve.cpp Tools/C_util/TV_TempWrite.H Tools/C_util/ViewMF/MFNorm.cpp Tools/C_util/ViewMF/checkMFghostcells.cpp Tools/C_util/ViewMF/main.cpp Tools/C_util/ViewMF/mfMinMax.cpp Tools/C_util/ViewMF/viewMF.cpp Tools/C_util/ViewMF/viewMFcol.cpp Tools/C_util/ViewMF/viewMFdiff.cpp Tools/C_util/ViewMF/viewMFdiffHardWire.cpp Tools/C_util/WritePlotFile.H Tools/C_util/WritePlotFile.cpp Tools/C_util/dbgTools/crsGrids.cpp Tools/C_util/dbgTools/intersectGrids.cpp Tools/CompileTesting/compiletesting.py Tools/EBSurfaceTools/ConvertEBSurface.cpp Tools/EBSurfaceTools/isoToVTK.py Tools/F_scripts/fcheck.py Tools/GNUMake/README.md Tools/Plotfile/AMReX_PPMUtil.cpp Tools/Plotfile/fcompare.cpp Tools/Postprocessing/C_Src/HorizontalAvg.cpp Tools/Postprocessing/C_Src/IntegrateComp.cpp Tools/Postprocessing/C_Src/MultiFabToMatLab.cpp Tools/Postprocessing/C_Src/PlotfileToMatLab.cpp Tools/Postprocessing/C_Src/PlotfileToTurb.cpp Tools/Postprocessing/C_Src/PtwisePltTransform.cpp Tools/Postprocessing/C_Src/PtwisePltTransform_nd.f90 Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp Tools/Postprocessing/C_Src/cube_extract.cpp Tools/Postprocessing/C_Src/particle_compare.cpp Tools/Postprocessing/python/column_depth.py Tools/Postprocessing/python/conv_slopes.py Tools/Postprocessing/python/dumpparthistory.py Tools/Postprocessing/python/eos_data.txt Tools/Postprocessing/python/helmeos.py Tools/Postprocessing/python/parseparticles.py Tools/Postprocessing/python/test_helmeos.py Tools/Postprocessing/python/test_parseparticles.py Tools/Py_util/amrex_particles_to_vtp/amrex_binary_particles_to_vtp.py Tools/Py_util/amrex_particles_to_vtp/write_pview_file.py Tools/Py_util/plotsinglevar.py Tools/Release/ppCleanup.py Tools/Release/ppCleanupDir.py Tools/Release/release.py Tools/typechecker/typechecker.py Tutorials/Amr/Advection_AmrCore/Exec/Prob.H Tutorials/Amr/Advection_AmrCore/Exec/paraview_amr101.py Tutorials/Amr/Advection_AmrCore/README.md Tutorials/Amr/Advection_AmrCore/Source/AdvancePhiAllLevels.cpp Tutorials/Amr/Advection_AmrCore/Source/AdvancePhiAtLevel.cpp Tutorials/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tutorials/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tutorials/Amr/Advection_AmrCore/Source/DefineVelocity.cpp Tutorials/Amr/Advection_AmrCore/Source/Src_K/Adv_K.H Tutorials/Amr/Advection_AmrCore/Source/Src_K/compute_flux_3D_K.H Tutorials/Amr/Advection_AmrCore/Source/face_velocity.H Tutorials/Amr/Advection_AmrCore/Source/main.cpp Tutorials/Amr/Advection_AmrLevel/Exec/SingleVortex/Prob.f90 Tutorials/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_2d.f90 Tutorials/Amr/Advection_AmrLevel/Exec/SingleVortex/face_velocity_3d.f90 Tutorials/Amr/Advection_AmrLevel/Exec/UniformVelocity/Prob.f90 Tutorials/Amr/Advection_AmrLevel/Source/Adv_F.H Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tutorials/Amr/Advection_AmrLevel/Source/LevelBldAdv.cpp Tutorials/Amr/Advection_AmrLevel/Source/Src_2d/Adv_2d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_2d/compute_flux_2d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_2d/slope_2d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_3d/Adv_3d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_3d/compute_flux_3d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_3d/slope_3d.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_nd/Tagging_nd.f90 Tutorials/Amr/Advection_AmrLevel/Source/Src_nd/tagging_params.f90 Tutorials/Amr/Advection_AmrLevel/Source/main.cpp Tutorials/Basic/Build_with_libamrex/main.cpp Tutorials/Basic/Build_with_libamrex/test_parameters.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX1_C/Source/myfunc.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/advance.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/advance_3d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX1_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/myfunc.H Tutorials/Basic/HeatEquation_EX1_CF/Source/myfunc_F.H Tutorials/Basic/HeatEquation_EX1_F/advance.f90 Tutorials/Basic/HeatEquation_EX1_F/fmain.F90 Tutorials/Basic/HeatEquation_EX1_F/init_phi.f90 Tutorials/Basic/HeatEquation_EX2_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX2_C/Source/myfunc.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/mykernel.H Tutorials/Basic/HeatEquation_EX2_CF/Source/advance.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/advance_3d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX2_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/myfunc.H Tutorials/Basic/HeatEquation_EX2_CF/Source/myfunc_F.H Tutorials/Basic/HeatEquation_EX3_C/Source/advance.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/init_phi_2d.f90 Tutorials/Basic/HeatEquation_EX3_C/Source/init_phi_3d.f90 Tutorials/Basic/HeatEquation_EX3_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/myfunc.H Tutorials/Basic/HeatEquation_EX3_C/Source/myfunc_F.H Tutorials/Basic/main_C/main.cpp Tutorials/Basic/main_F/main.F90 Tutorials/Blueprint/AssignMultiLevelDensity/main.cpp Tutorials/Blueprint/CellSortedParticles/CellSortedPC.H Tutorials/Blueprint/CellSortedParticles/CellSortedPC.cpp Tutorials/Blueprint/CellSortedParticles/cell_sorted_3d.F90 Tutorials/Blueprint/CellSortedParticles/cell_sorted_F.H Tutorials/Blueprint/CellSortedParticles/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/advance_3d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/init_phi_2d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/init_phi_3d.f90 Tutorials/Blueprint/HeatEquation_EX1_C/Source/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/myfunc.H Tutorials/Blueprint/HeatEquation_EX1_C/Source/myfunc_F.H Tutorials/Blueprint/README.txt Tutorials/EB/CNS/Exec/Combustor/bc_fill_nd.F90 Tutorials/EB/CNS/Exec/Pulse/cns_prob.F90 Tutorials/EB/CNS/Exec/ShockRef/cns_prob.F90 Tutorials/EB/CNS/Exec/Sod/cns_prob.F90 Tutorials/EB/CNS/Source/CNS.H Tutorials/EB/CNS/Source/CNS.cpp Tutorials/EB/CNS/Source/CNS_F.H Tutorials/EB/CNS/Source/CNS_advance.cpp Tutorials/EB/CNS/Source/CNS_init_eb2.cpp Tutorials/EB/CNS/Source/CNS_io.cpp Tutorials/EB/CNS/Source/CNS_setup.cpp Tutorials/EB/CNS/Source/diffusion/cns_diff_mod.F90 Tutorials/EB/CNS/Source/diffusion/cns_eb_diff_wall.F90 Tutorials/EB/CNS/Source/diffusion/diff_coef_mod.F90 Tutorials/EB/CNS/Source/fortran/CNS_divop.F90 Tutorials/EB/CNS/Source/fortran/CNS_dudt.F90 Tutorials/EB/CNS/Source/fortran/CNS_f.F90 Tutorials/EB/CNS/Source/fortran/CNS_nd.F90 Tutorials/EB/CNS/Source/fortran/bc_fill_nd.F90 Tutorials/EB/CNS/Source/hydro/Hyp_gamma_MOL.F90 Tutorials/EB/CNS/Source/hydro/Hyp_gamma_MOL_EB.F90 Tutorials/EB/CNS/Source/hydro/analriem3d.F90 Tutorials/EB/CNS/Source/hydro/cns_eb_hyp_wall.F90 Tutorials/EB/CNS/Source/hydro/slope_mol_3d_gamma.F90 Tutorials/EB/CNS/Source/hydro/slope_mol_3d_gamma_EB.F90 Tutorials/EB/CNS/Source/main.cpp Tutorials/EB/GeometryGeneration/main.cpp Tutorials/EB/MacProj/main.cpp Tutorials/EB/Poisson/main.cpp Tutorials/ForkJoin/MLMG/ff.f90 Tutorials/FortranInterface/Advection_F/Exec/SingleVortex/Prob_2d.f90 Tutorials/FortranInterface/Advection_F/Exec/SingleVortex/Prob_3d.f90 Tutorials/FortranInterface/Advection_F/Exec/SingleVortex/face_velocity_2d.F90 Tutorials/FortranInterface/Advection_F/Exec/SingleVortex/face_velocity_3d.F90 Tutorials/FortranInterface/Advection_F/Source/Src_2d/advect_2d_mod.F90 Tutorials/FortranInterface/Advection_F/Source/Src_2d/compute_flux_2d.f90 Tutorials/FortranInterface/Advection_F/Source/Src_2d/slope_2d.f90 Tutorials/FortranInterface/Advection_F/Source/Src_3d/advect_3d_mod.F90 Tutorials/FortranInterface/Advection_F/Source/Src_3d/compute_flux_3d.f90 Tutorials/FortranInterface/Advection_F/Source/Src_3d/slope_3d.f90 Tutorials/FortranInterface/Advection_F/Source/amr_data_mod.F90 Tutorials/FortranInterface/Advection_F/Source/averagedown_mod.F90 Tutorials/FortranInterface/Advection_F/Source/bc_mod.F90 Tutorials/FortranInterface/Advection_F/Source/compute_dt_mod.F90 Tutorials/FortranInterface/Advection_F/Source/evolve_mod.F90 Tutorials/FortranInterface/Advection_F/Source/fillpatch_mod.F90 Tutorials/FortranInterface/Advection_F/Source/fmain.F90 Tutorials/FortranInterface/Advection_F/Source/initdata.F90 Tutorials/FortranInterface/Advection_F/Source/my_amr_mod.F90 Tutorials/FortranInterface/Advection_F/Source/plotfile_mod.F90 Tutorials/FortranInterface/Advection_F/Source/tagging_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Exec/SingleVortex/Prob.f90 Tutorials/FortranInterface/Advection_octree_F/Exec/SingleVortex/face_velocity_2d.F90 Tutorials/FortranInterface/Advection_octree_F/Source/Src_2d/advect_2d_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/Src_2d/compute_flux_2d.f90 Tutorials/FortranInterface/Advection_octree_F/Source/Src_2d/slope_2d.f90 Tutorials/FortranInterface/Advection_octree_F/Source/amr_data_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/averagedown_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/bc_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/compute_dt_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/evolve_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/fillpatch_mod.F90 Tutorials/FortranInterface/Advection_octree_F/Source/my_amr_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/Prob.f90 Tutorials/FortranInterface/Advection_octree_F2/Exec/SingleVortex/face_velocity_2d.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/advect_2d_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/compute_flux_2d.f90 Tutorials/FortranInterface/Advection_octree_F2/Source/Src_2d/slope_2d.f90 Tutorials/FortranInterface/Advection_octree_F2/Source/amr_data_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/averagedown_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/bc_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/compute_dt_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/evolve_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/fillpatch_mod.F90 Tutorials/FortranInterface/Advection_octree_F2/Source/my_amr_mod.F90 Tutorials/GPU/CNS/Source/CNS.H Tutorials/GPU/CNS/Source/CNS.cpp Tutorials/GPU/CNS/Source/CNS_K.H Tutorials/GPU/CNS/Source/CNS_advance.cpp Tutorials/GPU/CNS/Source/CNS_io.cpp Tutorials/GPU/CNS/Source/CNS_setup.cpp Tutorials/GPU/CNS/Source/hydro/CNS_hydro_K.H Tutorials/GPU/CNS/Source/main.cpp Tutorials/GPU/EBCNS/Source/CNS.H Tutorials/GPU/EBCNS/Source/CNS.cpp Tutorials/GPU/EBCNS/Source/CNS_advance.cpp Tutorials/GPU/EBCNS/Source/CNS_init_eb2.cpp Tutorials/GPU/EBCNS/Source/CNS_io.cpp Tutorials/GPU/EBCNS/Source/CNS_setup.cpp Tutorials/GPU/EBCNS/Source/hydro/CNS_hydro_K.H Tutorials/GPU/EBCNS/Source/main.cpp Tutorials/GPU/Launch/MyKernel_F.F90 Tutorials/GPU/ParallelScan/main.cpp Tutorials/MUI/Source_01/init_phi_3d.f90 Tutorials/MUI/Source_01/main_01.cpp Tutorials/MUI/Source_02/main_02.cpp Tutorials/MUI/doc/MUIcouplingNotes.tex Tutorials/Particles/CellSortedParticles/CellSortedPC.H Tutorials/Particles/CellSortedParticles/CellSortedPC.cpp Tutorials/Particles/CellSortedParticles/cell_sorted_3d.F90 Tutorials/Particles/CellSortedParticles/cell_sorted_F.H Tutorials/Particles/CellSortedParticles/main.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/em_pic_K.H Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Exec/OpenACC/em_pic_3d.F90 Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/EMParticleContainer.cpp Tutorials/Particles/ElectromagneticPIC/Source/EMParticleContainer.H Tutorials/Particles/ElectromagneticPIC/Source/EMParticleContainerInit.cpp Tutorials/Particles/ElectromagneticPIC/Source/main.cpp Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.H Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.cpp Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_2d.f90 Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_3d.f90 Tutorials/Particles/ElectrostaticPIC/electrostatic_pic_F.H Tutorials/Particles/ElectrostaticPIC/main.cpp Tutorials/Particles/NeighborList/CheckPair.H Tutorials/Particles/NeighborList/Constants.H Tutorials/Particles/NeighborList/MDParticleContainer.H Tutorials/Particles/NeighborList/MDParticleContainer.cpp Tutorials/Particles/NeighborList/README.md Tutorials/Particles/NeighborList/main.cpp Tutorials/SDC/MISDC_ADR_2d/Source/SDC_sweeper.cpp Tutorials/SDC/MISDC_ADR_2d/Source/functions_2d.f90 Tutorials/SDC/MISDC_ADR_2d/Source/init_phi_2d.f90 Tutorials/SDC/MISDC_ADR_2d/Source/main.cpp Tutorials/SDC/MISDC_ADR_2d/Source/myfunc.H Tutorials/SDC/MISDC_ADR_2d/Source/myfunc_F.H Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test.H Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test.cpp Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test_F.F90 Tutorials/SWFFT/SWFFT_poisson/SWFFT_Test_F.H Tutorials/SWFFT/SWFFT_poisson/main.cpp Tutorials/SWFFT/SWFFT_poisson/swfft_solver.cpp Tutorials/SWFFT/SWFFT_simple/SWFFT_Test.H Tutorials/SWFFT/SWFFT_simple/SWFFT_Test.cpp Tutorials/SWFFT/SWFFT_simple/SWFFT_Test_F.H Tutorials/SWFFT/SWFFT_simple/main.cpp Tutorials/SWFFT/SWFFT_simple/swfft_compute.cpp commit d84f787548d8e5b89466cd44fbad9d1656e7d08a Author: Weiqun Zhang Date: Sun Mar 14 06:15:13 2021 -0700 Free unused memory (#1861) If Arena is asked to allocate more than what's available in the system, try to free unused memory first. Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp commit 85609ec18f8dd3b86eed3bf805e48515e5ce6e7d Author: Kevin Gott Date: Sat Mar 13 17:52:38 2021 -0800 Fortran Interfaces for new FillPatch (#1793) Fortran Interfaces for new FillPatch. Compiles, and I believe the indexing is correct, but didn't test a Fortran implementation. Src/F_Interfaces/AmrCore/AMReX_fillpatch_fi.cpp Src/F_Interfaces/AmrCore/AMReX_fillpatch_mod.F90 Src/F_Interfaces/AmrCore/AMReX_interpolater_mod.F90 commit 9b17f925887a9058a3efb91af26a5ac082d5f2c6 Author: Weiqun Zhang Date: Fri Mar 12 12:14:08 2021 -0800 The_Device_Arena -> The_Arena (#1858) Replace The_Device_Arena with The_Arena in various places. Using The_Arena should help reduce the memory footprint. We used The_Device_Arena in the early days to avoid the penalty of page fault caused by touching managed memory on CPU. We are much better in that regard now. In fact, AMReX itself no longer depends on managed memory. Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H commit 7e31cfd67ad9739b7779ee47d1a2750acf9c2cc1 Author: Weiqun Zhang Date: Fri Mar 12 12:13:42 2021 -0800 Arena Release Threshold (#1857) Release memory from Arena back to the system if it's above a threshold. For the pinned arena, the release threshold is set to the size of the global memory. For others, it's set to a huge number. These can be controlled with ParmParse parameters, `amrex.the_arena_release_threshold` and `the_[device|pinned|managed|async]_release_threshold`. The reason for setting the pinned arena's release threshold to the size of the global memory is that we received a report in the past that the pinned arena used by the communication functions wasted a huge amount of memory in the case that more and more grids were added over the run time. Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_CArena.H Src/Base/AMReX_CArena.cpp Src/Base/AMReX_DArena.H Src/Base/AMReX_DArena.cpp Src/Base/AMReX_EArena.H Src/Base/AMReX_EArena.cpp Src/Base/AMReX_PArena.cpp commit fafaccd7d4a7d0fbe90ec3507402e1988cfd00a1 Author: Maikel Nadolski Date: Fri Mar 12 17:07:57 2021 +0100 Replace is_pod with is_standard_layout && is_trivial (#1856) [`std::is_pod`](https://en.cppreference.com/w/cpp/types/is_pod) is deprecated in C++20 and should be replaced by `std::is_standard_layout && std::is_trivial`. This is the only occurrence of `std::is_pod` that I found in AMReX. If unchanged, this will trigger warnings in future compilers. This triggered a warning in our internal CI when using gcc 10.2 and the `-std=c++20` option. Src/Base/AMReX_GpuAsyncArray.H commit 7d594d2affc9188963ceb55a9ebca597d02462da Author: Maikel Nadolski Date: Thu Mar 11 17:49:00 2021 +0100 Feature/non local parallel copy (#1842) This PR introduces "Multiblock" capabilities to AMReX. Multiblock refers to having multiple distinct computational domains, known as blocks, that have some relation to each other. This PR proposes a generic NonLocalBC::ParallelCopy that performs copies between two FabArrays that might live on different blocks. The ParallelCopy respects - An index mapping such as already present in NonLocalBC (DTOS) - A projection function that will be called on either sender or receiver side to perform actions like e.g. swapping two components or changing the sign of velocity components The proposed usage is ``` NonLocalBC::ParallelCopy(dest, destbox, src, scomp, dcomp, ncomp, ngrow, index_mapping, fab_projection); ``` Notes - In the current implementation, `DTOS` needs to be a map between index spaces where inverse images of boxes are boxes again. - The name `NonLocalBC::ParallelCopy` is confusing if it does something else than `amrex::ParallelCopy`. Alternatives? - Parts of the implementation might be shared with other places in AMReX, such as in communication procedures of FabArray. But this is out of scope. - In addition to `SwapComponents`, a more generic `PermuteComponents` is useful. - The test is not commented yet. Src/Base/AMReX_Extension.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBC.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_TypeTraits.H Src/Base/CMakeLists.txt Src/Base/Make.package Tests/CMakeLists.txt Tests/MultiBlock/Advection/CMakeLists.txt Tests/MultiBlock/Advection/GNUmakefile Tests/MultiBlock/Advection/Make.package Tests/MultiBlock/Advection/main.cpp Tests/MultiBlock/IndexType/CMakeLists.txt Tests/MultiBlock/IndexType/GNUmakefile Tests/MultiBlock/IndexType/Make.package Tests/MultiBlock/IndexType/main.cpp commit 4893049d2588261f235c1e05ef7d0121a9ad6b6c Author: Pankaj Jha Date: Wed Mar 10 14:35:57 2021 -0800 Include missing library 'pthread' for linking to amrex library (#1854) - [HOTFIX ] Include missing library 'pthread' for linking to amrex library in the tutorial 'Basic/Build_with_libamrex' Tutorials/Basic/Build_with_libamrex/GNUmakefile commit 6ec00666843dcc0ae794358c872dd92e30a3cd34 Author: Andrew Myers Date: Mon Mar 8 17:05:52 2021 -0500 Enable expanding MPI / GPU macros for doxygen. (#1853) Docs/Doxygen/doxygen.conf commit 9001a7486ffcdd68cec9f2c1212559c4982865d8 Author: Weiqun Zhang Date: Mon Mar 8 08:15:45 2021 -0800 Revert the use of codeplay_host_task (#1851) It results int NaNs. Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuElixir.cpp commit fff2224879807f73c5a4acac19609f5630f77a8e Author: Ann Almgren Date: Sun Mar 7 10:01:11 2021 -0800 Add harmonic averaging (#1850) Mostly adding functionality, but also fixed a bug for 2d multicomponent cc->fc averaging Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H commit 146f3f2b7596cbee73d2b2eb8d48d4576ecc3f7c Author: Candace Gilet Date: Sat Mar 6 21:36:28 2021 -0500 Fix typo in GNUMake/Make.defs introduced in PR#1846 (#1849) Tools/GNUMake/Make.defs commit 0fc630c5bbc08b65feda01cae000b2314772d240 Author: Weiqun Zhang Date: Fri Mar 5 14:53:57 2021 -0800 AMREX_NO_PROBINIT (#1846) For Amr/AmrLevel based codes, there is now a compile time option AMREX_NO_PROBINIT=[TRUE|FALSE] for the support of the probin file and amrex_probinit function. The default behavior of requiring a user defined amrex_probinit function unless amr.probin_file is empty does not change unless amrex is built into a shared library on non-GNU/Linux system. The reason for the inconsistency is there is no portable way of supporting a weak symbol (.e.g, undefined amrex_probinit in amrex itself) in a shared library. This will partially solve the shared library build issue on Mac and Windows. Co-authored-by: Michele Rosso Docs/sphinx_documentation/source/BuildingAMReX.rst Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp Tools/CMake/AMReXGenerateConfigHeader.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/GNUMake/Make.defs Tools/libamrex/configure.py commit e7a264ddc0b008310d80c98ecf578e292f3cf13d Author: Maikel Nadolski Date: Fri Mar 5 19:53:28 2021 +0100 Use unique_ptr instead of naked pointer (#1839) This commit replaces naked pointer to char with `std::unique_ptr` in `NonLocalBC::CommHandler`. This will lead to proper cleanup in case of any exceptions. Note, that `Comm_nowait` and `Comm_finish` are *not* marked noexcept. Furthermore, future PRs will add interpolation and some kind of projection for non-local data transactions. Error handling for those is still unclear. Maybe, we want to mark everything noexcept. This is merely an implementation change without nearly any noticable difference in user code. Src/Base/AMReX_Extension.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_NonLocalBCImpl.H commit ef3a757eae988bff7948a47c346a6fb47ffd9ac0 Author: Kevin Gott Date: Fri Mar 5 09:43:54 2021 -0800 Fix early returns in ParallelCopy. (#1845) Fixes a new segfault bug due to pc_src.size() being called when pc_src isn't set. Instead, tracks all ParallelCopy_nowait early returns and checks once at the top of the ParallelCopy_finish. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H commit 6c1296b2466263a5aba46b5df16aceda47457b11 Author: Candace Gilet Date: Fri Mar 5 11:49:42 2021 -0500 Revert "Change default for Amr::probin_file -> "" (#1837)" (#1844) This reverts commit 2519768caedcf46a05b0cfd2b552dff2148465f7. Src/Amr/AMReX_Amr.cpp commit 2fb82bf64838aba949f7a641988a0c58774e5a88 Author: Michele Rosso Date: Thu Mar 4 13:25:13 2021 -0800 CMake: move AMReX_INSTALL option to AMReXOptions.cmake (#1840) * CMake: add missing AMReX_INSTALL option * CMake: remove AMReX_INSTALL from root CMakeLists.txt CMakeLists.txt Tools/CMake/AMReXOptions.cmake commit e01e1c73e670defda422b0f6a7b0a5fc45fb17af Author: Weiqun Zhang Date: Thu Mar 4 12:23:03 2021 -0800 Add EditorConfig (#1841) .editorconfig commit 076be729a1cc8fabc5c18f6836e134f85fa8b0c7 Author: Ann Almgren Date: Thu Mar 4 08:00:53 2021 -0800 Migrate more of Tutorials/LinearSolvers to Tests/LinearSolvers (#1835) Tests/LinearSolvers/ABecLaplacian_C/CMakeLists.txt Tests/LinearSolvers/ABecLaplacian_C/GNUmakefile Tests/LinearSolvers/ABecLaplacian_C/Make.package Tests/LinearSolvers/ABecLaplacian_C/MyTest.H Tests/LinearSolvers/ABecLaplacian_C/MyTest.cpp Tests/LinearSolvers/ABecLaplacian_C/MyTestPlotfile.cpp Tests/LinearSolvers/ABecLaplacian_C/initProb.cpp Tests/LinearSolvers/ABecLaplacian_C/initProb_K.H Tests/LinearSolvers/ABecLaplacian_C/inputs Tests/LinearSolvers/ABecLaplacian_C/inputs-inhomNeumann Tests/LinearSolvers/ABecLaplacian_C/inputs-rt-abeclap-com Tests/LinearSolvers/ABecLaplacian_C/inputs-rt-poisson-lev Tests/LinearSolvers/ABecLaplacian_C/inputs.hypre Tests/LinearSolvers/ABecLaplacian_C/inputs.petsc Tests/LinearSolvers/ABecLaplacian_C/main.cpp Tests/LinearSolvers/ABecLaplacian_C/scalingtest/inputs.test Tests/LinearSolvers/ABecLaplacian_C/scalingtest/main.diff Tests/LinearSolvers/ABecLaplacian_C/scalingtest/results.org Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-1.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-1024.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-128.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-16.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-2.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-2048.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-256.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-32.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-4.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-512.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-64.sh Tests/LinearSolvers/ABecLaplacian_C/scalingtest/run-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/inputs.test Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-1.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-16.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-2.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-32.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-4.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-64.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-mpi-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-1.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-16.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-2.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-32.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-4.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-64.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/knl-omp-8.sh Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/main.diff Tests/LinearSolvers/ABecLaplacian_C/threadmultiple_test/results.org Tests/LinearSolvers/ABecLaplacian_F/CMakeLists.txt Tests/LinearSolvers/ABecLaplacian_F/GNUmakefile Tests/LinearSolvers/ABecLaplacian_F/Make.package Tests/LinearSolvers/ABecLaplacian_F/README Tests/LinearSolvers/ABecLaplacian_F/init_prob.F90 Tests/LinearSolvers/ABecLaplacian_F/inputs Tests/LinearSolvers/ABecLaplacian_F/inputs-rt-abeclap-lev Tests/LinearSolvers/ABecLaplacian_F/inputs-rt-poisson-com Tests/LinearSolvers/ABecLaplacian_F/main.F90 Tests/LinearSolvers/ABecLaplacian_F/mytest.F90 Tests/LinearSolvers/MultiComponent/GNUmakefile Tests/LinearSolvers/MultiComponent/MCNodalLinOp.H Tests/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tests/LinearSolvers/MultiComponent/Make.package Tests/LinearSolvers/MultiComponent/inputs Tests/LinearSolvers/MultiComponent/main.cpp Tests/LinearSolvers/NodalPoisson/CMakeLists.txt Tests/LinearSolvers/NodalPoisson/GNUmakefile Tests/LinearSolvers/NodalPoisson/Make.package Tests/LinearSolvers/NodalPoisson/MyTest.H Tests/LinearSolvers/NodalPoisson/MyTest.cpp Tests/LinearSolvers/NodalPoisson/MyTestPlotfile.cpp Tests/LinearSolvers/NodalPoisson/inputs-rt Tests/LinearSolvers/NodalPoisson/main.cpp Tests/LinearSolvers/Nodal_Projection_EB/CMakeLists.txt Tests/LinearSolvers/Nodal_Projection_EB/GNUmakefile Tests/LinearSolvers/Nodal_Projection_EB/Make.package Tests/LinearSolvers/Nodal_Projection_EB/README Tests/LinearSolvers/Nodal_Projection_EB/inputs_3d Tests/LinearSolvers/Nodal_Projection_EB/main.cpp Tests/LinearSolvers/NodeTensorLap/CMakeLists.txt Tests/LinearSolvers/NodeTensorLap/GNUmakefile Tests/LinearSolvers/NodeTensorLap/Make.package Tests/LinearSolvers/NodeTensorLap/MyTest.H Tests/LinearSolvers/NodeTensorLap/MyTest.cpp Tests/LinearSolvers/NodeTensorLap/MyTestPlotfile.cpp Tests/LinearSolvers/NodeTensorLap/main.cpp Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-dpcpp-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini commit 2519768caedcf46a05b0cfd2b552dff2148465f7 Author: Candace Gilet Date: Thu Mar 4 11:00:18 2021 -0500 Change default for Amr::probin_file -> "" (#1837) Src/Amr/AMReX_Amr.cpp commit 86c58645a181907962bd9a72d7c12089d68c1a3a Author: Weiqun Zhang Date: Wed Mar 3 21:47:44 2021 -0800 Fix BoxArray in doxygen (#1838) Src/Base/AMReX_BoxArray.H commit 0ad9fd9c31e9ddfbbdbb57e659d7864b2f2a6ed8 Author: Ann Almgren Date: Wed Mar 3 14:35:12 2021 -0800 Move Tutorials/LinearSolvers/MAC_Projection_EB to Tests/LinearSolvers/MAC_Projection_EB (#1834) Tests/LinearSolvers/MAC_Projection_EB/CMakeLists.txt Tests/LinearSolvers/MAC_Projection_EB/GNUmakefile Tests/LinearSolvers/MAC_Projection_EB/Make.package Tests/LinearSolvers/MAC_Projection_EB/README Tests/LinearSolvers/MAC_Projection_EB/inputs_3d Tests/LinearSolvers/MAC_Projection_EB/main.cpp Tools/RegressionTesting/AMReX-cuda-tests.ini Tools/RegressionTesting/AMReX-hip-tests.ini Tools/RegressionTesting/AMReX-tests.ini commit 42c927d1cae12f90a841c07ccd88275d8a66d5cc Author: Axel Huebl Date: Wed Mar 3 09:16:46 2021 -0800 AMReX Install: Config Header (#1833) Found another header that still installed itself :) Tools/CMake/AMReXGenerateConfigHeader.cmake commit dd2ec76a5cc5db01caa17bdf51efb603c019346d Author: Weiqun Zhang Date: Tue Mar 2 10:30:48 2021 -0800 Fix ncomp bug in linear solver (#1832) Need to give m_ncomp a default value, otherwise default constructor followed by define(...) will not work. Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H commit 28fd2ee530e4c0229f04e33e041989f8535c9d33 Author: Axel Huebl Date: Tue Mar 2 09:41:39 2021 -0800 CMake: AMReX_INSTALL (#1831) Add an `AMReX_INSTALL` option that controls `install()` generation. Such options are typically used in superbuilds by dependent projects to supress the install of AMReX headers, helpers and libs, e.g. if AMReX is fully consumed as private dependency. CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst commit b7ddf2d2677fce63a567612978e01ced288dbda2 Author: Weiqun Zhang Date: Mon Mar 1 11:18:19 2021 -0800 Update CHANGES for 21.03 (#1830) CHANGES commit 2794f2f8f89d7461e4d99f19edc805c0feb8d7fe Author: Weiqun Zhang Date: Mon Mar 1 08:24:18 2021 -0800 Fix a sign compare warning (#1827) Src/Base/AMReX_GpuLaunchFunctsG.H commit 0518bb82022da8e6d6250d65b11f7cdaa8b31b24 Author: Ann Almgren Date: Sun Feb 28 12:58:34 2021 -0800 Ncomp for average cc to fc (#1826) Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H commit 90efb8b28714a496a3cd9188bdba4794579c59a4 Author: Ann Almgren Date: Sun Feb 28 09:01:48 2021 -0800 Add n_comp as an optional argument to the constructor of AMReX_MLABecLaplacian (#1825) Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp commit 662231a896991dd4fc110aa2fae6740a1f511c59 Author: Weiqun Zhang Date: Sat Feb 27 08:37:59 2021 -0800 Arena Query Functions and Bug Fix in FArrayBox & IArrayBox (#1823) Add Arena::isDeviceAccessible, isHostAccessible, isManaged, isDevice, and isPinned. Use them in various places. Fix bugs in FArrayBox & IArrayBox that touch device memory allocated by The_Async_Arena() on the host. Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_BArena.H Src/Base/AMReX_BArena.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_PArena.H Src/Base/AMReX_PArena.cpp Src/Base/AMReX_VisMF.cpp commit 272e9fc5e93b54f125e907b29e3b9db149c4ae63 Author: Weiqun Zhang Date: Sat Feb 27 08:09:28 2021 -0800 DPCPP: Fix bug in computing the number of active threads (#1822) Thank Intel for the help on debugging! Src/Base/AMReX_GpuLaunchFunctsG.H commit da6eca54acd9fc291fec189a9ee75b54d9727cab Author: Michele Rosso Date: Fri Feb 26 09:51:54 2021 -0800 CMake: add -Wno-pass-failed to Clang-based compilers (#1815) * CMake: add -Wno-pass-failed to Clang-based compilers * CMake: forgot to add flag in one place * CMake: need that flag for Release too Tools/CMake/AMReXFlagsTargets.cmake Tools/CMake/AMReXSYCL.cmake commit f208bc92bc0c9c0b744423cf3cc847db23e5a0ba Author: drangara <69211175+drangara@users.noreply.github.com> Date: Fri Feb 26 11:55:37 2021 -0500 Cover multiple cuts (#1810) ## Summary When building the edge types, cover the cut edges that touch opposite regular corners in order to prevent the `too many cuts in a face error`. Logic is only enabled when flag is set to true in inputs file (default is false). ## Additional background When trying to resolve complex geometries with sharp edges one can run into cases where there are multiple EB cuts on a face. In this change, when the flag `cover_multiple_cuts` is set, we cover appropriate edges in order to avoid this error. Manually tested geometry generation on some complex geometries that were previously failing to resolve. Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_MultiGFab.H Src/EB/AMReX_EB2_MultiGFab.cpp commit 35d27f7b8ba4ac3b77faadf12af5132ecd1f24cc Author: Weiqun Zhang Date: Fri Feb 26 07:33:30 2021 -0800 Fix sign-compare warnings (#1817) Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_ParmParse.cpp Src/Base/AMReX_VisMF.cpp Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_distFcnElement.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_StructOfArrays.H Tools/GNUMake/comps/gnu.mak commit 8cdf250b04aa1c06ae745133100be2988e9a35aa Author: Erik Schnetter Date: Thu Feb 25 22:52:50 2021 -0500 AMReX_GpuFuse.H: Avoid compiler warning (#1816) Src/Base/AMReX_GpuFuse.H commit a2c92323ee40c46a1848438230ee2b6ac1cfefd8 Author: Erik Schnetter Date: Thu Feb 25 17:03:58 2021 -0500 Avoid compiler warning in FabArray::defined (#1814) Avoid compiler warning in `FabArray::defined`: Convert `size_t` to `int`. Src/Base/AMReX_FabArray.H commit 327c62b150bb23ce01633a4d7ae060291156c62d Author: Weiqun Zhang Date: Wed Feb 24 13:42:23 2021 -0800 Add regression testing script for HIP (#1812) Tools/RegressionTesting/AMReX-hip-tests.ini commit f2e46722e8a6d2a8fd383d268b14579c4bae2929 Author: Weiqun Zhang Date: Wed Feb 24 09:16:25 2021 -0800 Add Axel to the list of core developers (#1811) .zenodo.json CONTRIBUTING.md commit 47389357d5de37d84be69d9b06a294e64cb2a96b Author: Weiqun Zhang Date: Mon Feb 22 09:30:43 2021 -0800 Update docs for GitHub Discussions (#1808) CONTRIBUTING.md Docs/sphinx_documentation/source/index.rst commit 13aa70f0ae5e156b9afdec8ed9f8ef13ba86a0fb Author: Weiqun Zhang Date: Mon Feb 22 08:40:22 2021 -0800 The_Async_Arena and Elixir::append (#1804) Add The_Async_Arena that uses CUDA steam-ordered memory allocator, if available. If the feature is not available, use The_Arena to allocate and free memory and the free is done with a host callback function. The purpose of this Arena is to be used by a temporary FArrayBox inside an MFIter loop. Add Elixir::append that appends an rvalue Elixir to an existing one to reduce the number of host callback calls. Docs/sphinx_documentation/source/GPU.rst Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuElixir.H Src/Base/AMReX_GpuElixir.cpp Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_PArena.H Src/Base/AMReX_PArena.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 110422d41aae2eee79772d890a52de4ee54f1d2c Author: Kevin Gott Date: Sun Feb 21 20:13:08 2021 -0800 Add function name output option to TinyProfiler. (#1803) Use tiny_profiler.verbose=1 to output "Entering " and "Leaving " output. Primarily for hunting down hang issues. Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit 85cad7492dfa14851cc47ed4c55e4d2a56d6d56c Author: Axel Huebl Date: Fri Feb 19 15:40:58 2021 -0800 CMake: More Particle Tests (#1801) Tests/Particles/AssignDensity/CMakeLists.txt Tests/Particles/AssignMultiLevelDensity/CMakeLists.txt Tests/Particles/AsyncIO/CMakeLists.txt Tests/Particles/AsyncIO/inputs Tests/Particles/GhostsAndVirtuals/CMakeLists.txt Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/InitFromAscii/CMakeLists.txt Tests/Particles/InitFromAscii/main.cpp Tests/Particles/Intersection/CMakeLists.txt Tests/Particles/Intersection/main.cpp Tests/Particles/ParallelContext/CMakeLists.txt Tests/Particles/ParticleArray/CMakeLists.txt Tests/Particles/ParticleIterator/CMakeLists.txt Tests/Particles/SparseBins/CMakeLists.txt Tests/Particles/TypeDescriptor/CMakeLists.txt commit 093275a3bbf26e45faa43971418bac597b8a7b85 Author: Michele Rosso Date: Wed Feb 17 13:36:06 2021 -0800 Extrapolater: make assertion less restrictive (#1799) Src/Amr/AMReX_Extrapolater.cpp commit 86e26ca7bdcb64d5e694f325973541f721c568b9 Author: Weiqun Zhang Date: Wed Feb 17 10:57:36 2021 -0800 DPCPP: Use codeplay_host_task (#1797) In AsyncArray and Elixir, replace streamSynchronize with SYCL 2020's host_task. Note that the function name in the current oneAPI release is codeplay_host_task, because it started as a Codeplay extension. This will probably change in the future, since the extension has been accepted into the standard. Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuElixir.cpp commit d29a5709545b0d44051e82a895be37f5f821cbc6 Author: Weiqun Zhang Date: Wed Feb 17 10:47:06 2021 -0800 Remove functions about the number of host callbacks (#1798) We used to use the number of host callbacks to limit the number of GPU streams. But we no longer do that anymore, and the default number of streams has been reduced to 4. So the functions keeping track of the number of host callbacks can now be removed. Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_GpuElixir.cpp Src/Base/AMReX_MFIter.cpp commit 23c2fa054f21263220d1f1d384889af9593fc330 Author: Andrew Myers Date: Wed Feb 17 10:10:33 2021 -0800 Add ParticleArray classes (#1796) Add a class for storing arrays of particle data that can be toggled between AoS and SoA. Src/Base/AMReX_PODVector.H Src/Particle/AMReX_ParticleArray.H Src/Particle/CMakeLists.txt Src/Particle/Make.package Tests/Particles/ParticleArray/GNUmakefile Tests/Particles/ParticleArray/Make.package Tests/Particles/ParticleArray/main.cpp commit e611121acca7696320f3757982de857cbab74926 Author: Kevin Gott Date: Tue Feb 16 16:43:21 2021 -0800 ParallelCopy_nowait & ParallelCopy_finish (#1765) Most direct implementation of `ParallelCopy_nowait` and `ParallelCopy_finish`, with minimal changes. Previous version had a max component limit per message, set by the input parameter `fabarray.maxcomp`, 25 by default. This is preserved. If `ncomp > fabarray.maxcomp`, `ParallelCopy_nowait` calls `ParallelCopy_finish` internally for all iterations of the loop except the final iteration. For most apps using GPUs that haven't been manually setting `fabarray.maxcomp`, this will almost certainly mean behavior identical to `FillBoundary_nowait`, as `ncomp > 25` would require very small `MultiFabs` to fit on current GPUs. Keeps `ParallelCopy`, which simply calls `ParallelCopy_nowait` followed by `ParallelCopy_finish`. No doubt, there are potential optimizations available: e.g. if `ncomp>fabarray.maxcomp` and `ncomp%fabarray.maxcomp != 0` returning from `nowait` with a message with a full `fabarray.maxcomp` components may allow the largest possible overlap. But, this version gets the feature running while we discuss additional improvements such as this. Plan to follow up with a scaling study on the effect of message size on `ParallelCopy` timing on Summit by exploiting the `fabarray.maxcomp` feature to see if rewriting with more controlled buffer sizes yields enough benefit to justify a rewrite. Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H commit 305770490a946deede9eee31ddd3814fe077ff67 Author: Weiqun Zhang Date: Tue Feb 16 14:40:47 2021 -0800 Fix a new bug in GNU Make (#1795) Must set CXXSTD for nvcc because we need to use it later. Tools/GNUMake/comps/nvcc.mak commit 446fb1e9ee4647604a56a68063f64cb7d94cfd50 Author: Weiqun Zhang Date: Tue Feb 16 10:50:25 2021 -0800 Add FabArray::tags() to return the tags. (#1794) Src/Base/AMReX_FabArray.H commit 4a332b5c3875a5eba705ad0d33f76d7f592bef80 Author: Andrew Myers Date: Tue Feb 16 09:30:16 2021 -0800 Bump minimum C++ standard from 11 to 14. (#1787) This is well-supported now and will allow a lot of the code in amrex to be simplified. I have also bumped the version on the Cuda 9 CI test to Cuda 9.2 and GCC 6.5.0, since the old combination did not work with C++14. .github/workflows/dependencies/dependencies_nvcc9.sh .github/workflows/linux.yml Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/BuildingAMReX_Chapter.rst INSTALL Tools/CMake/AMReX_Config.cmake Tools/GNUMake/comps/cray.mak Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/intel.mak Tools/GNUMake/comps/llvm-flang.mak Tools/GNUMake/comps/nag.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/comps/nvhpc.mak Tools/GNUMake/comps/pgi.mak Tutorials/Basic/Build_with_libamrex/GNUmakefile commit 3ed36ff23c1d408a855573eadcce2686b51160d8 Author: Ann Almgren Date: Mon Feb 15 15:07:55 2021 -0800 fix table formatting in GPU.rst (#1790) Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/Post_Processing.rst Docs/sphinx_documentation/source/Post_Processing_Chapter.rst commit e8b4d0739e4f21b44169a195a7aa40eaf0092c1b Author: Ann Almgren Date: Mon Feb 15 15:03:15 2021 -0800 Add an interface to the flux redistribution that takes the arguments (#1789) as Array4s rather than FABs or MultiFabs. Docs/sphinx_documentation/source/Post_Processing.rst Src/EB/AMReX_EB_utils.H Src/EB/AMReX_EB_utils.cpp commit 15a84595c5b88f4e289be505869786400bac49e3 Author: Max Katz Date: Mon Feb 15 15:00:58 2021 -0800 Update the preferred short name of NVIDIA HPC SDK to nvhpc (#1788) `nvhpc` is the preferred short name of the NVIDIA HPC SDK (for example when used in modules). Tools/GNUMake/Make.defs Tools/GNUMake/comps/nvhpc.mak commit 492c5524375858d46f4dd6ea97589cffef749c42 Author: Weiqun Zhang Date: Mon Feb 15 14:58:59 2021 -0800 Fix sphinx config for sphinx 3.5.0 (#1792) https://github.com/sphinx-doc/sphinx/issues/8885 Docs/sphinx_documentation/source/conf.py Docs/sphinx_tutorials/source/conf.py commit df5bbe922563bf3ae9eb97a566014205e1a1b2f4 Author: Weiqun Zhang Date: Thu Feb 11 10:07:38 2021 -0800 BoxArray::simplified() & simplified_list() (#1786) Make them public so that they can be used by application codes to regenerate grids covering exactly the same region but with a different max grid size. However, it should be emphasized that they only work with regular BoxArrays without anything like coarsening. Src/Base/AMReX_BoxArray.H commit 7742e1c99f5e7a5ac7a4fa3530dba3a30d51daeb Author: Michele Rosso Date: Wed Feb 10 15:21:10 2021 -0800 CMake: bump up minimum required HYPRE version (#1785) Tools/CMake/AMReXThirdPartyLibraries.cmake commit 932836ad422ad742a7c3c9b6375bad9cffec76d7 Author: Ann Almgren Date: Tue Feb 9 18:17:01 2021 -0800 fix bug in EB redistribution which was accidentally including (#1784) contributions from outside a non-periodic domain Src/EB/AMReX_EB_utils.cpp commit d2c88194fac2b9e35e469cf332034bedbf361cf9 Author: Axel Huebl Date: Tue Feb 9 00:02:41 2021 -0800 Windows: WINDOWS_EXPORT_ALL_SYMBOLS (#1782) I am seeing some linker issues downstream when building Python bindings with WarpX on Windows. This nice option makes sure we don't need to add dll export symbols all over the place when building a shared AMReX library (dll). Kudos to the ROOT team at CERN for contributing this feature years ago to CMake. Tools/CMake/AMReXBuildInfo.cmake Tools/CMake/AMReX_Config.cmake commit 6219fa1df9b1980327da3f40bbcf4457a93a6602 Author: Weiqun Zhang Date: Mon Feb 8 18:35:42 2021 -0800 Fix precision issue in ParmParse::add (#1783) When adding values to ParmParse, we need to call `std::setprecision` so as not to lose precision. Src/Base/AMReX_ParmParse.cpp commit ba0f1c0e77515debd83fe651abd3045d87567dab Author: Weiqun Zhang Date: Mon Feb 8 14:57:30 2021 -0800 Fix bug in #1774 (#1779) A bug was introduced in #1774 that resulted in incorrect weights when making nodal RHS from cell-centered RHS. Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H commit d44e353bb9f7745973502e4d7b16bbde3d29ed09 Author: Andrew Myers Date: Mon Feb 8 11:45:59 2021 -0800 Fix up neighbor list for 2D (#1781) Src/Particle/AMReX_NeighborList.H commit 591f48db3c0539fa639e8d846cd4a036a7e63c53 Author: Weiqun Zhang Date: Sat Feb 6 17:38:19 2021 -0800 Remove managed memory from CNS (#1780) Tutorials/GPU/CNS/Exec/RT/cns_prob.cpp Tutorials/GPU/CNS/Exec/RT/cns_prob_parm.H Tutorials/GPU/CNS/Exec/Sod/cns_prob.cpp Tutorials/GPU/CNS/Exec/Sod/cns_prob_parm.H Tutorials/GPU/CNS/Source/CNS.H Tutorials/GPU/CNS/Source/CNS.cpp Tutorials/GPU/CNS/Source/CNS_advance.cpp Tutorials/GPU/CNS/Source/CNS_derive.cpp Tutorials/GPU/CNS/Source/CNS_parm.H Tutorials/GPU/CNS/Source/CNS_setup.cpp commit d31f9b4a36a1cb912d4dd52bb1807117fe9d79f6 Author: Weiqun Zhang Date: Fri Feb 5 19:17:43 2021 -0800 Overset Solver with Refinement Ratio of 4 (#1778) We need to define overset mask for on coarse MG levels of fine AMR levels. We didn't need to do it for refinement ratio of 2 because there are no coarse MG levels on fine AMR levels. Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp commit eb9b4242a07028a77bd95986f5a5686c81dce428 Author: Andrew Myers Date: Thu Feb 4 12:11:37 2021 -0800 Fix a circular include involving PODVector. (#1776) Src/Base/AMReX_PODVector.H commit 5e08d985d3a91ab8ec10d9aec8c318b38ff10849 Author: Andrew Myers Date: Thu Feb 4 12:09:15 2021 -0800 add missing template arguments to the particle HDF5 IO methods (#1777) Src/Particle/AMReX_ParticleHDF5.H commit f8808fcb56e172f9548954240468cf3d18223851 Author: Weiqun Zhang Date: Thu Feb 4 10:28:54 2021 -0800 Refinement Ratio of 4 Support in Nodal Solver (#1774) * Refinement Ratio of 4 Support in Nodal Solver This adds refinement ratio of 4 support in non-EB nodal linear solver. Roundoff errors are expected for multi-level solves with a refinement ratio of 2. * MacProjector refinement ratio of 4 bug Move setLevelBC to project and call it only if it has not been called. This fixes a refinement ratio of 4 bug, because setLevelBC should not be called before coarse/fine ratio is set. Src/Base/AMReX_Box.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp Tutorials/LinearSolvers/NodalPoisson/MyTest.H Tutorials/LinearSolvers/NodalPoisson/MyTest.cpp commit c686e458ffccfa0032f231753c65ea514df8f122 Author: Candace Gilet Date: Wed Feb 3 23:48:45 2021 -0500 For FillPatchTwoLevels, BCs should be set on the coarse box (#1772) since they are only used on the coarse level (when computing slopes). This corrects certain cases where a level of refinement is very close to, but not actually touching an ext_dir or hoextrap boundary. Src/Amr/AMReX_AmrLevel.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H commit ebf64aa13ff7d57013473bdd4df0acf7be144193 Author: Michael Zingale Date: Wed Feb 3 23:34:09 2021 -0500 remove f90cat.py (#1775) it was only ever used with Castro as an experiment Tools/F_scripts/f90cat.py commit c9265f4c85057e94b39db780df42491f20137d19 Author: Kevin Gott Date: Wed Feb 3 16:06:45 2021 -0800 Make.unknown MPI Detection Update (#1770) Simplify/robust-ify Make.unknown's MPI detection. Tools/GNUMake/sites/Make.unknown commit 39fe5870a75b7dfd9f3b0cdc531ab63f92a9f15b Author: Axel Huebl Date: Wed Feb 3 13:45:56 2021 -0800 MSVC: Proper __cplusplus macro (#1773) Modern MSVC versions finally set the __cplusplus macro correctly, but we need to request this with an additional flag. Tools/CMake/AMReX_Config.cmake commit b00475d69aec7304d18129d6b3ce9c2a2f4d835a Author: Michael Zingale Date: Tue Feb 2 16:03:12 2021 -0500 remove any trailing / from HIP_PATH via realpath (#1771) Tools/GNUMake/comps/hip.mak commit c68493447deeaf1675a6cdfef323d0adb62f4bbe Author: Andrew Myers Date: Tue Feb 2 08:55:09 2021 -0800 Use ParIter instead of MakeMFIter in WriteBinaryParticleDataAsync. (#1767) Src/Particle/AMReX_WriteBinaryParticleData.H commit d2270fab703098760806d9b9297a7502e8918733 Author: Axel Huebl Date: Tue Feb 2 08:35:34 2021 -0800 IntVect::toArray() (#1768) ## Summary Add a ::toArray() conversion member to IntVect that can be used to convert back to an array type and cast the result on the way, e.g. to unsigned. ## Additional background Functions in https://github.com/ECP-WarpX/WarpX/pull/1660 Src/Base/AMReX_IntVect.H commit 4a425335e40fbae69261987e8a97bc094712fb9f Author: Weiqun Zhang Date: Mon Feb 1 08:37:34 2021 -0800 Update CHANGES for 21.02 (#1766) CHANGES commit 466e3071713ced99cdfce4426c283d75da21b2c5 Author: Andrew Myers Date: Sat Jan 30 13:21:11 2021 -0800 Add a Github action that tries to build tutorials with SP particles and DP mesh. (#1764) Currently this combination is only caught by the Nyx CI. .github/workflows/linux.yml commit 302fad36faaa8f5bb65c207c608ca332b1a07116 Author: Candace Gilet Date: Sat Jan 30 16:18:05 2021 -0500 Abort when MLMG is detected as failing regardless of verbosity setting. (#1762) Move Abort() outside if( verbose >0 ) block. Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit f649f04fef6922374f2d2b31474ac87eda35a5f6 Author: Kevin Gott Date: Sat Jan 30 13:16:45 2021 -0800 Anyof (#1757) * DPC++ versions of AnyOf. Written to match CUDA/HIP version. Performance not explicitly tested. * Also, adjustment to AMREX_DEVICE_PRINTF to remove the need for AMREX_DEVICE_COMPILE. Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_Reduce.H Tests/GPU/AnyOf/GNUmakefile Tests/GPU/AnyOf/Make.package Tests/GPU/AnyOf/inputs Tests/GPU/AnyOf/main.cpp commit 896916428ca28b012c7c4ed53812898f2c26f815 Author: Jean M. Sexton Date: Fri Jan 29 13:32:24 2021 -0800 Use ParticleReal for particle interpolation (#1763) Src/Particle/AMReX_TracerParticles.cpp commit adbb5695469183897014c05824bd44772252f398 Author: Andrew Myers Date: Fri Jan 29 11:59:32 2021 -0800 Fix new logic error with aggregation_type. (#1761) Src/Particle/AMReX_ParticleContainerBase.cpp commit ef3e68bbc1380065f9515ea5c0236fe2f18cda5f Author: Andrew Myers Date: Fri Jan 29 11:51:30 2021 -0800 Remove obsolete interpolation functions from AMReX (#1754) Src/Particle/AMReX_Particle.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.cpp commit 5df10f661e060de45fca677026325fc278b5c70c Author: Andrew Myers Date: Fri Jan 29 11:06:53 2021 -0800 Move more PC methods that don't need the template parameters into base class. (#1760) Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_Particles.H commit f45ede1d49940035f84dc9a8562fef0a2ad3b720 Author: Andrew Myers Date: Thu Jan 28 13:28:26 2021 -0800 make pc.Geom always refer to the particle one. (#1470) Src/Particle/AMReX_Particles.H commit c143555e74e4096ec25400369651bf884b8561b3 Author: Andrew Myers Date: Wed Jan 27 16:04:37 2021 -0800 Fix 'potential null dereference warning' in StructOfArrays (#1759) Src/Particle/AMReX_StructOfArrays.H commit 2cd12635b7d7c2d41ee5ebce07c8003ef7fbe1b0 Author: Kevin Gott Date: Wed Jan 27 14:31:30 2021 -0800 Gpu Docs Update First Pass (#1758) First pass at updating the GPU docs. More detailed updates to follow. Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst commit 0b79c5f7d855ed0078b63bfe34cf982c0956f06e Author: Andrew Myers Date: Wed Jan 27 09:29:29 2021 -0800 Move DataPrefix and Version... (#1756) * DataPrefix and Version do not need the template arguments, so moving them to a base class. Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleContainerBase.H Src/Particle/AMReX_ParticleContainerBase.cpp Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_WriteBinaryParticleData.H Src/Particle/CMakeLists.txt Src/Particle/Make.package commit b95a0f30003d6eb97fb6e33a5fde5a902221495d Author: Andrew Myers Date: Wed Jan 27 09:19:47 2021 -0800 Remove unused move random methods. (#1755) These methods are not used, and if needed, they should be implemented at the application level. Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_Particles.H commit 02c52cda1b70c77384807d898a7a5c1f705c96c4 Author: Andrew Myers Date: Tue Jan 26 08:50:02 2021 -0800 Add IntVect versions of numParticlesOutOfRange. (#1751) Src/Particle/AMReX_ParticleUtil.H commit 9896c7643f72bc091b119e63ab6f46e5d1a058d2 Author: Axel Huebl Date: Mon Jan 25 16:52:20 2021 -0800 Fix ParIter Ctor (#1752) With user-defined allocators, this lead to: ``` Src/Particle/AMReX_ParIter.H:114:84: error: type ‘amrex::ParIterBase’ is not a direct base of ‘amrex::ParIter<0, 0, 4, 0, amrex::PinnedArenaAllocator>’ 114 | : ParIterBase(pc,level) | ``` in non-const contexts. Src/Particle/AMReX_ParIter.H commit b560557c4f5876337c0eb47df82f0ead37cf52b6 Author: Ann Almgren Date: Mon Jan 25 13:40:42 2021 -0800 Improve the interpolation from cell centroids to face centroids: (#1750) 1) if the centroids are aligned tangentially to the face but not at equal distance to the face, use just those two cells for interpolation 2) in the case where the face centroid is at 0 but the cell centroids are not aligned, instead of testing on an area fraction tangential to the face to decide which way to go, in 2d, now test on the minimum volume for each option and choose the option with the larger minimum volume -- this should avoid using covered or very small cells. (3D already has logic to avoid using covered cells) Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H commit 3069f1ba4dd899f40ac6092fff2916466af8d980 Author: Ann Almgren Date: Mon Jan 25 13:40:25 2021 -0800 Fix EB slopes in 3D when two of the domain boundaries with ext_dir bc's meet at an edge -- (#1749) this case was handled incorrectly before Src/EB/AMReX_EB_slopes_K.H commit e93c5705a6c013ffc602a16e877d22ea3afbf1fc Author: Donald E. Willcox Date: Sun Jan 24 20:23:00 2021 -0800 In ParticleToMesh and MeshToParticle, use nGrowVect() to accommodate different a different number of grow cells in each dimension. (#1748) Src/Particle/AMReX_ParticleMesh.H commit 162bca40a4ecec77a5d8d1df68b1864bfd15ec6a Author: Ann Almgren Date: Sat Jan 23 12:50:54 2021 -0800 Add additional tests when interpolating from cell centroids to face centroids (#1746) when the face centroid is exactly at the center of the face we need an additional test to know which way we can look transversely and not hit covered cells. Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H commit cbbd1c5bd623fc59b67a96e1d5cd5339f2b24cbb Author: Weiqun Zhang Date: Fri Jan 22 09:40:57 2021 -0800 Fix single precision build of CNS (#1745) Tutorials/GPU/CNS/Source/main.cpp commit e2b1108dde79e258f4d06e584e745e0ffd2bbd5f Author: Andrew Myers Date: Thu Jan 21 17:37:54 2021 -0800 Add missing template parameters in ParIter.H. (#1744) Src/Particle/AMReX_ParIter.H commit 1656d55a6e001f139f88c5ada93068a7b862b59c Author: Andrew Myers Date: Thu Jan 21 15:23:39 2021 -0800 Template copy/add particles based on src type. (#1741) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_Particles.H commit a63941a8f13fd972b860297de2132185a739418e Author: Axel Huebl Date: Thu Jan 21 15:00:09 2021 -0800 HIP: New Warp Size Macro (#1742) Use new the HIP macro for the warp size on various AMD devices. Src/Base/AMReX_GpuDevice.H commit c2beebd492f1efb202d09cef2f62e5283998e1b7 Author: Andrew Myers Date: Wed Jan 20 17:21:05 2021 -0800 Add allocator template parameter to ParticleContainer (#1416) Src/AmrCore/AMReX_AmrParticles.H Src/Base/AMReX_TypeTraits.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_Particles.H commit bcf034a796031d362b85dba9c88fad24f30b36e6 Author: Ann Almgren Date: Wed Jan 20 15:21:05 2021 -0800 Reduce compiler warnings (#1740) Co-authored-by: Andrew Myers Src/EB/AMReX_EB_slopes_K.H Src/Particle/AMReX_NeighborList.H commit b2c1b6077a07a23201aa65156ef2a1033943482f Author: Andrew Myers Date: Wed Jan 20 11:44:24 2021 -0800 Fix some new warnings about 'pointless comparisons' with 0 for unsigned types. (#1737) Src/Particle/AMReX_StructOfArrays.H commit 1775ce65c769d0502f727aa1f1955366a172ff2c Author: drangara <69211175+drangara@users.noreply.github.com> Date: Wed Jan 20 14:16:39 2021 -0500 Add ability to calculate slopes on face centroids (#1707) And apply the explicit stencil when computing the laplacian These changes allow the computation of EB viscous laplacian operator when phi is defined on cell centroids by using the Least squares method for calculating slopes on face centroids. Currently only works when Dirichlet BC is set. Src/EB/AMReX_EB_LeastSquares_2D_K.H Src/EB/AMReX_EB_LeastSquares_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Tests/LinearSolvers/LeastSquares/GNUmakefile Tests/LinearSolvers/LeastSquares/Make.package Tests/LinearSolvers/LeastSquares/MyEB.H Tests/LinearSolvers/LeastSquares/MyTest.H Tests/LinearSolvers/LeastSquares/MyTest.cpp Tests/LinearSolvers/LeastSquares/README.md Tests/LinearSolvers/LeastSquares/initData.cpp Tests/LinearSolvers/LeastSquares/initEB.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleData.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleDataFor2D.cpp Tests/LinearSolvers/LeastSquares/initPoiseuilleDataFor3D.cpp Tests/LinearSolvers/LeastSquares/inputs.2d.askew-x Tests/LinearSolvers/LeastSquares/inputs.2d.askew-y Tests/LinearSolvers/LeastSquares/inputs.2d.base Tests/LinearSolvers/LeastSquares/inputs.2d.fullyrotated Tests/LinearSolvers/LeastSquares/inputs.2d.other Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.xy-x Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.xy-y Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.xz-x Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.xz-z Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.yz-y Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.aligned.yz-z Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-all Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-xy Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-xz Tests/LinearSolvers/LeastSquares/inputs.3d.poiseuille.askew-yz Tests/LinearSolvers/LeastSquares/main.cpp commit 9bda4f80dea74dcef0ccd4cc9dfa60c693613228 Author: Weiqun Zhang Date: Wed Jan 20 09:53:21 2021 -0800 HIP indirect function and kernel fusing (#1739) Add a GNU Make flag, HIP_INDIRECT_FUNCTION=[TRUE|FALSE], and modify the Gpu kernel fusing test. Currently HIP does not support indirect function calls via device function pointers, which our GPU fusing implementation depends on. Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuFuse.cpp Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_Reduce.H Tests/GPU/Fuse/GNUmakefile Tests/GPU/Fuse/main.cpp Tools/GNUMake/Make.defs commit 9437e36505e115da96cebebab973d5499f357bb2 Author: Robert Maynard Date: Tue Jan 19 13:39:04 2021 -0500 When using CMake 3.15+ allow for MSVC runtime library changes (#1736) Enabling of CMP0091 needs to go before the first `project` call as that computes if MSVC_RUNTIME_LIBRARY information is used CMakeLists.txt Tools/CMake/AMReX_Config.cmake commit 5791c90593389dbf23502d10c52f01b35e77bfa6 Author: Weiqun Zhang Date: Mon Jan 18 20:49:38 2021 -0800 RoundRobin: option to not sort processes (#1721) Add an optional parameter `sort` to RoundRobin constructors. This makes it consistent with other DistributionMapping strategies. And this fixes an incorrect behavior when RoundRobin is used by KnapSack that has been given `sort=false`. This should be able to fix the hanging issue observed in WarpX simulations with load balancing enabled and more MPI processes than the number of boxes. Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_DistributionMapping.cpp commit 5d3e31a08af987cf9e127ab07375483c74724743 Author: Weiqun Zhang Date: Sun Jan 17 12:02:00 2021 -0800 EB: edge centroid (#1730) Add edge centroid to the database and make it available via EBFArrayBoxFactory. Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_IF_Rotation.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EB2_ND_C.cpp Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EBDataCollection.cpp Src/EB/AMReX_EBFabFactory.H Src/EB/AMReX_EB_utils.cpp Src/EB/CMakeLists.txt Src/EB/Make.package commit 8b123354e9929bc2515b032c2069e2d924e88e59 Author: Weiqun Zhang Date: Sun Jan 17 11:58:36 2021 -0800 No Fortran needed for building plotfile tools (#1711) Tools/Plotfile/GNUmakefile commit d2ff769fa0ad57243a3c9d0010eb68e2ab049005 Author: Emmanuel Motheau Date: Sun Jan 17 11:57:30 2021 -0800 fix RegridOnly to regrid even with single level grids (#1722) Src/Amr/AMReX_Amr.cpp commit 832ce5e8bcdd224c8946392dfaeb8915e0e02eed Author: Andy Nonaka Date: Sat Jan 16 11:54:48 2021 -0800 Consolidate agglomerate (#1733) a few lines in the docs about consolidation and agglomeration options in the linear solvers Docs/sphinx_documentation/source/LinearSolvers.rst commit eab09708761036232dd650c5ee724af7f28249f4 Author: Axel Huebl Date: Sat Jan 16 11:52:55 2021 -0800 Doc: Break into Debugger (#1732) Describe how to disable AMReX' signal and backtrace handling. Thanks to @sayerhs for pointing me to the right place (again :) ). Docs/sphinx_documentation/source/Basics.rst commit f6b0abc7c32368bafffe8e81a2de18c492c8971a Author: Max Katz Date: Sat Jan 16 11:50:44 2021 -0800 Only add Hypre interfaces to MLMG in 2D/3D (#1734) This is necessary for compiling with both Hypre and MLMG in 1D. Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp commit 889af2f8a8e1ec191627f1fb5b28ac9ed6d5aa9a Author: Max Katz Date: Fri Jan 15 18:13:37 2021 -0800 Don't include Hypre files in GNU make for 1D (#1731) These files cannot be compiled in 1D, so there is no point in adding them. CMake treats this as fatal, but for Castro we would like to use the AMReX Hypre GNU make package even without the actual interfaces. Src/Extern/HYPRE/Make.package commit 549bb3d5675f7cac9fd2d116f8cde1daaa477a09 Author: Andrew Myers Date: Fri Jan 15 17:05:38 2021 -0800 Always assert that tiling is false in RedistributeGPU (#1729) Src/Particle/AMReX_ParticleContainerI.H commit 55b62cb42fb6afcfd2e78c8d68fab80c39180a9a Author: Axel Huebl Date: Fri Jan 15 16:33:01 2021 -0800 Define: _OPENMP -> AMREX_USE_OMP (#1560) Replace the define `_OPENMP` with `AMREX_USE_OMP` for all parallel "backend" implementations and control of MFIter loops. This avoids accidentially enabling OpenMP for the parallel compute components, i.e. when a users explicitly uses OpenMP in auxiliary or implicit dependent functionality, but does not request to also OpenMP-parallelize those sections. Co-authored-by: Weiqun Zhang Docs/sphinx_documentation/source/AmrCore.rst Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/GPU.rst Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_Extrapolater.cpp Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_StateDescriptor.cpp Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_ErrorList.cpp Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FluxRegister.cpp Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.cpp Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_BLProfiler.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseFab.cpp Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_Extension.H Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuUtility.cpp Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/Base/AMReX_MemPool.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_OpenMP.H Src/Base/AMReX_PCI.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_PhysBCFunct.H Src/Base/AMReX_Print.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_TinyProfiler.cpp Src/Base/AMReX_Utility.cpp Src/Base/AMReX_iMultiFab.cpp Src/Base/AMReX_omp_mod.F90 Src/Boundary/AMReX_BndryRegister.cpp Src/Boundary/AMReX_FabSet.cpp Src/Boundary/AMReX_InterpBndryData.cpp Src/Boundary/AMReX_MultiMask.H Src/Boundary/AMReX_MultiMask.cpp Src/Boundary/AMReX_YAFluxRegister.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_Level.cpp Src/EB/AMReX_EBAmrUtil.cpp Src/EB/AMReX_EBCellFlag.cpp Src/EB/AMReX_EBFluxRegister.cpp Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EB_utils.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_MultiCutFab.cpp Src/EB/AMReX_algoim.cpp Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/PETSc/AMReX_PETSc.cpp Src/Extern/ProfParser/AMReX_BLProfStats.cpp Src/Extern/ProfParser/AMReX_CommProfStats.cpp Src/Extern/ProfParser/AMReX_RegionsProfStats.cpp Src/Extern/hpgmg/BL_HPGMG.cpp Src/F_Interfaces/Octree/AMReX_octree_fi.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMGBndry.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_TracerParticles.cpp Tests/FillBoundaryComparison/main.cpp Tests/LinearSolvers/CellEB2/MyTest.cpp Tests/LinearSolvers/CellOverset/MyTest.cpp Tests/LinearSolvers/MLMG/init_prob.cpp Tests/LinearSolvers/TensorOverset/MyTest.cpp Tests/Particles/ParticleIterator/main.cpp Tools/Postprocessing/C_Src/IntegrateComp.cpp Tutorials/Amr/Advection_AmrCore/Source/AdvancePhiAllLevels.cpp Tutorials/Amr/Advection_AmrCore/Source/AdvancePhiAtLevel.cpp Tutorials/Amr/Advection_AmrCore/Source/AmrCoreAdv.H Tutorials/Amr/Advection_AmrCore/Source/AmrCoreAdv.cpp Tutorials/Amr/Advection_AmrCore/Source/DefineVelocity.cpp Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tutorials/Blueprint/CellSortedParticles/CellSortedPC.cpp Tutorials/EB/CNS/Source/CNS.cpp Tutorials/EB/CNS/Source/CNS_advance.cpp Tutorials/EB/Poisson/Poisson.cpp Tutorials/GPU/CNS/Source/CNS.cpp Tutorials/GPU/EBCNS/Source/CNS.cpp Tutorials/LinearSolvers/ABecLaplacian_C/initProb.cpp Tutorials/LinearSolvers/MultiComponent/MCNodalLinOp.cpp Tutorials/LinearSolvers/NodalPoisson/MyTest.cpp Tutorials/LinearSolvers/NodeTensorLap/MyTest.cpp Tutorials/Particles/CellSortedParticles/CellSortedPC.cpp commit 2536f1ae9f7b7e73bbeee71d5642aa22e02432fc Author: Robert Maynard Date: Fri Jan 15 14:12:12 2021 -0500 When using CMake 3.15 allow for MSVC runtime library changes (#1724) Tools/CMake/AMReX_Config.cmake commit 40dcbf1f69cb81fc875e4e1db49d33c3292bbe9f Author: Mark Meredith Date: Fri Jan 15 12:04:44 2021 -0500 fix -Werror=sign-compare (#1699) Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_StructOfArrays.H commit 633c971c3258d4a662f91d5620e44834bf8d4c92 Author: Ann Almgren Date: Thu Jan 14 17:25:58 2021 -0800 Quiet the compiler warnings coming from EB/AMReX_EB_slopes_K.H (#1728) Src/EB/AMReX_EB_slopes_K.H commit fd6f5ce3c255d449d5de3d664269d64d069cbdc9 Author: Michele Rosso Date: Thu Jan 14 15:55:57 2021 -0800 CMake: link tests exes against alias library (#1727) Tests/CMakeLists.txt commit 873c4801ad9b1f262a8865f6b9fe198a4ead29ca Author: Jon Rood Date: Thu Jan 14 15:36:17 2021 -0700 Update Make.nrel to accomodate MPT MPI with CUDA enabled (#1726) For our main computer at NREL, Eagle, we mostly use HPE's MPT for MPI. Until these changes, CUDA did not work with MPT when using the GNU makefiles. Tools/GNUMake/Make.defs Tools/GNUMake/sites/Make.nrel commit bcbdb65b525644fcb1cd61dc57d4e77c19b8eac6 Author: Weiqun Zhang Date: Thu Jan 14 13:38:11 2021 -0800 Fix CNS tutorial for HIP (#1725) Use AMREX_GPU_HOST_DEVICE instead of AMREX_GPU_DEVICE because of we cannot detect the lambda type in HIP. Tutorials/GPU/CNS/Source/CNS.cpp commit 10f3a2348e8cb4d0f52e90d273f70426ef1fcbbc Author: Weiqun Zhang Date: Thu Jan 14 11:14:01 2021 -0800 Cell-centered Hypre and Petsc setup on GPU (#1709) Do cell-centered hypre and Petsc solver setup on GPU. Docs/sphinx_documentation/source/LinearSolvers.rst Src/Base/AMReX.cpp Src/Base/AMReX_BaseFabUtility.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_Habec_K.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap2.H Src/Extern/HYPRE/AMReX_HypreABecLap2.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/AMReX_PETSc.cpp Src/LinearSolvers/MLMG/AMReX_MLMG.H Tools/GNUMake/packages/Make.hypre Tutorials/LinearSolvers/ABecLaplacian_C/initProb.cpp commit dd0a3ceb3c62f68ad096b0692352acc776c352ae Author: Michele Rosso Date: Wed Jan 13 12:59:28 2021 -0800 Tests: replace D_DECL with AMREX_D_DECL (#1723) Tests/HDF5Benchmark/main.cpp Tests/Particles/AssignMultiLevelDensity/main.cpp Tests/Particles/AsyncIO/main.cpp Tests/Particles/GhostsAndVirtuals/main.cpp Tests/Particles/Intersection/main.cpp Tests/Particles/Redistribute/main.cpp Tests/Particles/SparseBins/main.cpp commit 07488476533e1bf4731f1bd7c17c1b14c0fd13d2 Author: Matt Larsen Date: Mon Jan 11 13:35:37 2021 -0800 fixing assert (#1720) Co-authored-by: Matt Larsen Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 95661e553fbd92c4d41ffa88438272a3ca4f385f Author: Andrew Myers Date: Mon Jan 11 11:38:54 2021 -0800 Fix compiler warnings in CreateVirtualParticles (#1718) Src/Particle/AMReX_ParticleContainerI.H commit f35b856849f22909e2e4395ac73edb263b108434 Author: Andrew Myers Date: Mon Jan 11 11:17:50 2021 -0800 Fix compiler warnings in ParticleInit.H (#1719) Src/Particle/AMReX_ParticleInit.H commit cd56f543e5372230314f1f0acfb6566a754376ed Author: Weiqun Zhang Date: Sun Jan 10 12:46:38 2021 -0800 ParmParse::hasUnusedInputs & getUnusedInputs (#1716) Add functions to find if there are unused `ParmParse` parameters starting with `prefix.`. If `prefix` is empty, all unused `ParmParse` parameters are included. Src/Base/AMReX_ParmParse.H Src/Base/AMReX_ParmParse.cpp commit cf17e95632b8bc26107a9ecd9e8dcae75db47c4e Author: Ann Almgren Date: Sat Jan 9 13:39:48 2021 -0800 There is no longer a CVODE tutorial (#1715) Docs/sphinx_tutorials/source/CVODE_Tutorial.rst Docs/sphinx_tutorials/source/index.rst commit 4e7f200eb91b1d8689b38f31d2ea1e29535094c0 Author: Michele Rosso Date: Fri Jan 8 17:00:27 2021 -0800 CMake: offer option to exclude Src/Amr from build (#1714) Docs/sphinx_documentation/source/BuildingAMReX.rst Src/CMakeLists.txt Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake commit 42f045ea6d39b7a1f02b2a92612352cff585a9b7 Author: Weiqun Zhang Date: Fri Jan 8 16:45:55 2021 -0800 Always have Abort_device etc. defined for GPU build (#1712) It used to be these functions are defined only if NDEBUG is defined. However, the cmake build system does not store `NDEBUG` in `AMReX_Config.H. It's possible that AMReX is built with `NDEBUG` and `Abort_device` is not defined, but the user's code may be built without `NDEBUG`. Then `amrex::Abort` defined in `AMReX.H` would fail to compile. Always having these device functions defined can solve the issue. Src/Base/AMReX.H Src/Base/AMReX.cpp commit 147934bcca2551e24c16bb0b5d4a50575a8a2eec Author: Weiqun Zhang Date: Fri Jan 8 14:57:07 2021 -0800 Prefetch not supported on Windows (#1713) On-demand page migration is not supported on Windows. Src/Base/AMReX_BaseFab.H commit 7eb484bb8fdaec3c46f66921b9baefe7a3b00580 Author: Emmanuel Motheau Date: Fri Jan 8 13:10:39 2021 -0800 Update write plot file to ascii tool (#1700) A flag (fast = 1) has been added to the WritePlotfileToAscii tool to propose to export data via Amrex::Print() instead of the 4 i,j,k,n imbricated loops. Indeed, on large plotfiles, exporting data to an ascii file is extremely slow with the procedure originally implemented in the tool. Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp commit 3daa8ced947e315f8c3cf7506e2eec45c6f8fbf4 Author: Weiqun Zhang Date: Fri Jan 8 10:34:38 2021 -0800 Tweak some floating point comparisons (#1710) Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_RealBox.H Src/Base/AMReX_RealBox.cpp Src/EB/AMReX_EB_utils.cpp commit 6a773594766d842c866cd3cbb74f2fe6f09884d6 Author: Weiqun Zhang Date: Thu Jan 7 14:48:37 2021 -0800 Make bisect more robust (#1708) If the lower and upper bounds are almost equal, break out of the bisect iteration. Src/Base/AMReX_Algorithm.H Src/Base/AMReX_Geometry.cpp commit be30569c55ce9ecf1f6576b4d94cd9040b3d8490 Author: Weiqun Zhang Date: Thu Jan 7 14:17:12 2021 -0800 Support single precision build with configure (#1706) * Add new arguments `--single-precision` and `--single-precision-particles` to `configure`. * Add amrex::almostEqual for comparing floating point numbers. * Fix double to float conversion warnings. * Add CI testing single precision build .github/workflows/linux.yml Src/Amr/AMReX_Amr.cpp Src/Amr/AMReX_AmrLevel.cpp Src/Amr/AMReX_AuxBoundaryData.cpp Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateData.cpp Src/Amr/AMReX_extrapolater_3D_K.H Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_ErrorList.cpp Src/Base/AMReX_Algorithm.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BoxList.cpp Src/Base/AMReX_COORDSYS_2D_C.H Src/Base/AMReX_CoordSys.cpp Src/Base/AMReX_DistributionMapping.cpp Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FilCC_C.cpp Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_MFCopyDescriptor.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_RealBox.cpp Src/Base/AMReX_VisMF.cpp Src/Boundary/AMReX_InterpBndryData_3D_K.H Src/Boundary/AMReX_MacBndry.cpp Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMGBndry.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.cpp Tests/FillBoundaryComparison/main.cpp Tools/C_util/WritePlotFile.cpp Tools/libamrex/configure.py Tutorials/Amr/Advection_AmrCore/Source/main.cpp Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tutorials/Amr/Advection_AmrLevel/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX1_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_C/Source/main.cpp Tutorials/Basic/HeatEquation_EX2_CF/Source/main.cpp Tutorials/Basic/HeatEquation_EX3_C/Source/main.cpp Tutorials/Blueprint/HeatEquation_EX1_C/Source/main.cpp Tutorials/EB/CNS/Source/CNS_advance.cpp Tutorials/EB/CNS/Source/main.cpp Tutorials/GPU/CNS/Source/main.cpp Tutorials/GPU/EBCNS/Source/main.cpp Tutorials/LinearSolvers/MAC_Projection_EB/main.cpp Tutorials/LinearSolvers/Nodal_Projection_EB/main.cpp commit f84b7b891cfa55affc25426288b4d3881e4be2a7 Author: Andrew Myers Date: Wed Jan 6 15:34:03 2021 -0800 Fix logic in applying periodic bcs when subcycling is on (#1705) Src/Particle/AMReX_ParticleUtil.H commit 20119b405e406040637daff2f4a591670de9093d Author: Axel Huebl Date: Wed Jan 6 11:55:01 2021 -0800 CMake<3.18: Fix Linker Regression (#1701) Roll-back to explicit `-Wl,` linker flags for CMake<3.18 Tools/CMake/AMReX_Config.cmake commit 887a4ce2a3e995592d9dcfd4318682526c297546 Author: Weiqun Zhang Date: Wed Jan 6 10:13:23 2021 -0800 ParallelDescriptor::ReduceReal wrappers (#1703) Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp commit c36b253b295032f83e13edd9de78c40e0e660a10 Author: Luca Fedeli Date: Tue Jan 5 17:47:19 2021 +0100 Fix several warnings related to double-->float conversions (#1698) ## Summary I tried to compile `WarpX` in single precision and I noticed several warnings related to casts from `double` to `float` in `AMReX`. This PR should fix several of these warnings. ## Additional background I compiled `WarpX` with `make DIM=3 PRECISION=FLOAT USE_SINGLE_PRECISION_PARTICLES=TRUE` using the compiler `g++ (Ubuntu 10.2.0-13ubuntu1) 10.2.0` Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleIO.H commit 6c6924e66f26586d03d4be661739e7ab600986fa Author: Robert Maynard Date: Tue Jan 5 11:21:23 2021 -0500 Support when CMake 3.18 policy 105 is set to new (#1696) When AMReX is using a newer version of CMake (3.18+) it sets policy `105` to new, causing `target_link_options` rules to be applied to CUDA device linking. When that happens we need to make sure that host only linking flags are excluded. Tools/CMake/AMReX_Config.cmake commit 8aa9810b22c37855f6b0602403138e7ffcea4a7b Author: Kevin Gott Date: Tue Jan 5 08:20:42 2021 -0800 Update FabReduce-isum test in ParallelReduce tutorial. (#1695) Tutorials/GPU/ParallelReduce/main.cpp commit 00c116c09110fb93428624da5cf5a7f0cdaabe43 Author: Matt Larsen Date: Tue Jan 5 08:09:58 2021 -0800 changing blueprint ghost type to int (#1694) This MR changes the type of the ghost field used by conduit blueprint from `double` to `int`. Ascent expects integers, and this reduces the storage space required for the field. Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 5baf0f7fee46a02caa349d4f8aaae26c192acd05 Author: Axel Huebl Date: Mon Jan 4 16:10:03 2021 -0800 CI: Brew Returns Non-Zero If Already Installed (#1697) Homebrew has the unusual behavior to return a non-zero error code in `brew install` if the package is already installed. Since the base image in CI is somewhat fluid and also the status that `brew update` leaves is very dynamic, we just change this ourselves now to be a non-error. .github/workflows/dependencies/dependencies_mac.sh commit 7eafbe22b93d3d9404efd1de5ea830be4915e372 Author: Weiqun Zhang Date: Sun Jan 3 10:12:03 2021 -0800 Optimized function for filling BaseFab of GpuArray (#1692) Add amrex::fill for BaseFab of GpuArray (a.k.a. AoS). The GPU implementation uses shared memory. Tests show it's more than 10x faster than a naive implementation for GpuArray. Also added are a few functions in DPC++ so that I do not have to look up the documentation every time I need to use things like threadIdx and blockIdx. Src/Base/AMReX_BaseFabUtility.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_MultiFabUtil.H commit 693700e6afb62df0531601cb48c2a5ab272784ea Author: Weiqun Zhang Date: Sat Jan 2 09:42:15 2021 -0800 BaseFab::minmax (#1691) Add BaseFab::minmax function and use it in VisMF. Src/Base/AMReX_BaseFab.H Src/Base/AMReX_VisMF.cpp commit 9ce1e4e19e64d7ae57d49a4dd0d17aab5a77a376 Author: Emmanuel Motheau Date: Sat Jan 2 09:40:45 2021 -0800 New feature in the fextract tool to directly export in the csv format (#1634) This PR adds a feature to the fextract tool to directly export in the csv format. Docs/sphinx_documentation/source/Post_Processing.rst Tools/Plotfile/fextract.cpp commit 3e38f80ba5e42cd66d98692edd4cef7075661a5b Author: Weiqun Zhang Date: Sat Jan 2 09:25:05 2021 -0800 Update CHANGES (#1693) CHANGES commit 63a567f7b00e7e7592686a8268ba796c3b886481 Author: Weiqun Zhang Date: Mon Dec 28 09:23:53 2020 -0800 DPCPP: GNU Make USE_GPU variable (#1689) Set USE_GPU=TRUE for dpcpp in Make.defs just like cuda and hip. Tools/GNUMake/Make.defs commit 6309a8b7a142d9bdcaedf9e95df15d6db8d9a8f5 Author: Weiqun Zhang Date: Sat Dec 26 12:20:00 2020 -0800 Optimization of MLCellLinOp::applyBC (#1690) Launch one kernel per box to work on all faces and all components. If the number of boxes exceeds the kernel fusing threshold, the kernels from all boxes will be fused. .github/workflows/linux.yml Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_Box.H Src/Base/AMReX_GpuContainers.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.cpp Src/Particle/AMReX_ParticleTransformation.H commit 8bcc31cf0b4319afe633c1c791cdbb30b7bd7118 Author: Max Katz Date: Thu Dec 24 09:12:05 2020 -0800 Remove old launch macros (#1687) Removes some old macros that were used exclusively for the GPU pragma. Also removes `box_threads_and_blocks()` which was only used by that functionality. Src/Base/AMReX_BLFort.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp commit d3fd6462b288249b992baa5f4e9c03d6d8945979 Author: Kevin Gott Date: Wed Dec 23 17:12:26 2020 -0800 unsigned long long iMultiFab sum. (#1684) Changes iMultiFab sum to use hardware supported unsigned long long. Using ParallelReduce, this changes sum time on Cori (default CUDA, 10.1) from 0.07546 to 0.001185, 63x increase. Src/Base/AMReX_iMultiFab.cpp commit 021a9a3d9a5678984afd64fdffdea0c5d4912c44 Author: Weiqun Zhang Date: Wed Dec 23 14:44:23 2020 -0800 Sanitize the input of mkconfig (#1685) The mkconfig.py script takes CPPFLAGS and generates AMReX_Config.H. If CPPFLAGS contains -I flags (as advised by homebrew) and the input is not santized, it would generate ill-formed code. Tools/libamrex/mkconfig.py commit b0977fd7aadb9021196807e9ffe1c8bfa2aaa53c Author: Andy Nonaka Date: Wed Dec 23 14:38:33 2020 -0800 WritePlotfileToASCII Dimensionality Checking (#1686) Added dimensionality error checking to WritePlotfileToASCII.cpp Also supports older FBoxLib plotfile types now. Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp commit c1713daf9e6ea60b54eb201ab0b18161a2e28c28 Author: Weiqun Zhang Date: Wed Dec 23 14:22:42 2020 -0800 Gpu::Reduce cleanup (#1681) Remove unused parameter in blockReduce_partial. Move the test for full block into Gpu::Handler class. Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuTypes.H commit cf82b6bcd2fdae99c1d78e856e4ee06bcb762995 Author: Weiqun Zhang Date: Wed Dec 23 14:20:28 2020 -0800 New versions of Reduce::Min, Max and MinMax taking callable (#1679) ## Summary Previously, Reduce::Min, Max and MinMax take a pointer to the data. New versions these functions taking a callable object is added to provide more flexibility. Note that Reduce::Sum already has the callable version. Also modify the implementation of the CPU versions of these functions to use OpenMP reduction instead of atomic when we can. Add optional initial value to some functions. Fix the order of template parameters for consistence. ## Checklist The proposed changes: - [x] add new capabilities to AMReX Src/Base/AMReX_Reduce.H commit c98be7422796a0e29cb539f805d91d19c2a5ba01 Author: Johannes Blaschke Date: Wed Dec 23 11:43:48 2020 -0800 update SumNeighbors to reflect recent Particle API changes (#1680) Src/Particle/AMReX_NeighborParticlesCPUImpl.H commit 7e48821802e61c999dcd6bdc7c0471d82e2fee55 Author: Weiqun Zhang Date: Wed Dec 23 10:28:22 2020 -0800 Gpu::AsyncArray -> Gpu::Buffer (#1676) because there is no need for a host callback function to free memory. Src/Base/AMReX_FabArrayUtility.H commit d080afba2402f0f46d03d80ab6d3264ef1968bad Author: Max Katz Date: Tue Dec 22 20:31:14 2020 -0800 Only issue prefetch if arena uses managed memory (#1678) ## Summary Before issuing a managed memory prefetch we check that the arena uses managed memory. This allows the CUDA implementation to still work with `amrex.the_arena_is_managed=0` while calling this API. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX Src/Base/AMReX_BaseFab.H Src/Base/AMReX_FabArrayUtility.H commit 88abc1e88eb34f7f7db0ecf110d75347a9d384a0 Author: Weiqun Zhang Date: Tue Dec 22 15:22:00 2020 -0800 CUDA CI: add flags to catch common errors (#1677) ## Summary Add `--Werror ext-lambda-captures-this` (nvcc 11 only) and `--Werror cross-execution-space-call` to CUDA CI tests. .github/workflows/linux.yml commit 0f820789c57213729b6598f3fa912bd751916dea Author: Weiqun Zhang Date: Tue Dec 22 15:19:49 2020 -0800 GNU Make: Fix the library flags (#1641) Previously for GPU build, the GNU Make system was very sensitive on which variable to use for adding extra `libraries`. Before a certain location, `LIBRARIES` should be used, but after that `libraries` should be used. This is very error prone. This PR tries to fix this. Now the user should be able to use `LIBRARIES` anywhere. The use of `libraries` should be avoided because it is still the case that it can only be used after a certain location in the make file. Nevertheless, this PR is backward compatible and it does not force the use of `LIBRARIES`. Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules commit a4a5adc298abf8ff6b74306fea7448fdff8c7502 Author: Weiqun Zhang Date: Tue Dec 22 14:54:48 2020 -0800 DPCPP: add sync to mimic CUDA NULL stream behavior (#1675) Src/Base/AMReX_GpuDevice.H commit 6c18f6c72f7a965aeed920aba3bf3b6bd6cc64d0 Author: Axel Huebl Date: Tue Dec 22 14:51:07 2020 -0800 NVCC 11.2: CUDA/C++17 (#1665) This adds CI coverage for C++17 builds with NVCC. Several apps expressed interest to transition to C++17 code-bases in the near future and this adds test coverage. Co-authored-by: Weiqun Zhang .github/workflows/linux.yml Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_ParticleLocator.H Tutorials/GPU/CNS/Source/CNS.cpp commit c8d01767451b03df74afb6f3dfd3da609ca1ffdb Author: Andrew Myers Date: Tue Dec 22 14:31:51 2020 -0800 Remove noexcept from this functor to work around cuda 11 bug. (#1674) Also uses variadic form of `amrex::max`. Tutorials/Particles/NeighborList/MDParticleContainer.cpp commit c777f74b02667f8c9a4929d00f7fb9e835900c53 Author: Weiqun Zhang Date: Tue Dec 22 13:13:36 2020 -0800 PolymorphicArray4 (#1643) PolymorphicArray4 is a thin layer around Array4. It allows us to access both AoS and SoA data with operator(). For example, constexpr int ncomp = 2; Box box(...); BaseFab soa_fab(box, ncomp); auto soa_array = makePolymorphic(soa_fab.array()); int i = ...; int j = ...; int k = ...; soa_array(i,j,k,0) = 0; soa_array(i,j,k,1) = 1; BaseFab> aos_fab(box,1); auto aos_array = makePolymorphic(aos_fab.array()); aos_array(i,j,k,0) = 0; aos_array(i,j,k,1) = 1; Src/Base/AMReX_Array.H Src/Base/AMReX_Array4.H commit afda0382ee0598ea0018894c0a3495c693225bc2 Author: Weiqun Zhang Date: Tue Dec 22 09:33:54 2020 -0800 Remove the check on ref ratio being <= 12 (#1672) Src/Amr/AMReX_Amr.cpp Src/AmrCore/AMReX_AmrMesh.cpp commit 1a8e3e70691a4cdf2183a455d13f2ac529705535 Author: Axel Huebl Date: Mon Dec 21 22:44:25 2020 -0800 CI: OpenMP on Windows (Clang) (#1671) - fix MSVC entry back to MSVC - add OpenMP to Clang .github/workflows/windows.yml commit 6680398a1a4f00424a35cd6d989ffa0f22963197 Author: Weiqun Zhang Date: Mon Dec 21 21:01:01 2020 -0800 Unsigned long long for counting number of particles on GPU (#1670) Because long does not have native atomicAdd support, it's much slower than doing reduction using unsigned long long. Src/Particle/AMReX_ParticleContainerI.H commit c9a571f79d44daba2cae2a4be743be8e29ea348d Author: Michele Rosso Date: Mon Dec 21 18:49:32 2020 -0800 CMake: dowstream projects must enable Fortran if AMReX_FORTRAN=ON (#1669) * CMake: disable Fortran by default * CMake: dowstream projects must enable Fortran if AMReX_FORTRAN=ON Docs/sphinx_documentation/source/BuildingAMReX.rst Src/CMakeLists.txt Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake commit 4e8386ecf2a5575c7e5e7c7d8c741d4f361bb658 Author: Johannes Blaschke Date: Mon Dec 21 16:16:12 2020 -0800 fix neighbor particle 'GetNeighbors' return type (#1668) Src/Particle/AMReX_NeighborParticles.H commit 754451b9325690e63d0ab45f1d68ba1bae5e284a Author: Kevin Gott Date: Mon Dec 21 14:36:16 2020 -0800 FaceLinear fused interpolater variation (with performance comparison!) (#1663) Add a `interp_arr` version of `FaceLinear` to test generality of the new API (and performance comparison, as `FaceDivFree `stencil can't be used in regular API for comparison). Also includes a bit of `interp_arr` cleanup. Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp commit 7b9eb82bec95a5c0e729b3086d614cbdfe2bd71b Author: Andrew Myers Date: Mon Dec 21 14:24:42 2020 -0800 Allow alternate form of "check_pair" function (#1667) This allows users to use a different form of `check_pair` functor that passes the particle indices. This facilitates the creation of "half" neighbor lists. Src/Particle/AMReX_NeighborList.H commit 10ed0e029b6278a8dcf2e4af46623a53f00f1641 Author: Weiqun Zhang Date: Mon Dec 21 13:42:47 2020 -0800 ParallelFor for Reduction (#1658) Add capability for `ParallelFor` to safely do reduction using `deviceReduceSum`, `Min`, etc. The user passes `Gpu::KernelInfo{}.setReduction(true)` to notify `ParallelFor` that this is a parallel reduction, and gives `ParallelFor` a callable that takes `Gpu::Handler`. A `Gpu::Handler` is needed to call `deviceReduceSum`. Also add `Gpu::Buffer` class, whose data pointer can be used as a device destination for `deviceReduceSum`. It also has a `copyToHost` method to copy the device result back to the host. See `Tutorials/GPU/ParallelReduce` for examples of how to use `ParallelFor` for reduction. Also note that the reduction function is OpenMP CPU threads safe. Thus the same code can run on with OpenMP when it is not built for GPU. Co-authored-by: Andrew Myers Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX.cpp Src/Base/AMReX_Box.H Src/Base/AMReX_Gpu.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuAsyncArray.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuBuffer.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuKernelInfo.H Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_Scan.H Src/Base/CMakeLists.txt Src/Base/Make.package Src/EB/AMReX_EB2_GeometryShop.H Tutorials/GPU/ParallelReduce/GNUmakefile Tutorials/GPU/ParallelReduce/main.cpp commit c8cdfa61f443ecde21debc3ad52665bf48eded8b Author: Andrew Myers Date: Sat Dec 19 11:32:26 2020 -0800 Revert tiling change in #1655 (#1664) Src/Particle/AMReX_ParticleContainerI.H commit 875ebff912b5d3f4edc0e6ad7a983a56374a9e69 Author: Axel Huebl Date: Fri Dec 18 20:36:28 2020 -0800 CI: CUDA 9.1 & 11.0 (#1653) Add a recent CUDA 11 release for cutting-edge coverage, too. .github/workflows/dependencies/dependencies_nvcc11.sh .github/workflows/dependencies/dependencies_nvcc9.sh .github/workflows/linux.yml commit 25d06e82804e3e1b4f3f1b0cea4e9979ed37373c Author: Axel Huebl Date: Fri Dec 18 15:29:58 2020 -0800 CI: Lib + Plotfiles Install (#1662) Test the new install of plotfile tools works in CI. .github/workflows/linux.yml commit cc61f30f4266407225bd8c7354565931b275a07f Author: Michele Rosso Date: Fri Dec 18 13:56:37 2020 -0800 CMake: streamline install of plotfile tools (#1661) CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Src/CMakeLists.txt Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXInstallHelpers.cmake Tools/Plotfile/CMakeLists.txt commit 1c291c0b6bb52be354b77b2abb646216dcb62ff7 Author: Kevin Gott Date: Fri Dec 18 10:39:38 2020 -0800 Add new clean commands to docs. (#1660) Docs/sphinx_documentation/source/BuildingAMReX.rst commit 3b897e1b7e04222ae718c00010085908f7693b41 Author: Michele Rosso Date: Thu Dec 17 17:14:36 2020 -0800 Fix how fextract handle the tolerance options (#1659) Tools/Plotfile/fextract.cpp commit de1701ed9d6cbdf61a8c4c3015c1cf5a9b00f7f5 Author: Jean M. Sexton Date: Thu Dec 17 17:00:36 2020 -0500 AssignDensity and AssignCellDensitySingleLevel copy -> ParallelCopy (#1633) Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_ParticleContainerI.H commit 07a21fd454317b7b434240d548765c4c4ae4464a Author: Andrew Myers Date: Thu Dec 17 13:29:53 2020 -0800 Have AssignCellDensitySingleLevel respect gpu launch guards. (#1655) Src/Particle/AMReX_ParticleContainerI.H commit e21901db47b85704e29b3800fcad7318b9544e76 Author: Weiqun Zhang Date: Thu Dec 17 12:27:08 2020 -0800 Clean up GPU pragma and CUDA Fortran (#1656) Clean up the remaining GPU pragma and CUDA Fortran. This also fixes a bug in #1650 for codes using the Fortran 77 version of filcc. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Src/Base/AMReX_BCUtil.cpp Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_PhysBCFunct.H Src/Base/AMReX_PhysBCFunct.cpp Src/Base/AMReX_filcc_mod.F90 Src/Base/AMReX_fort_mod.F90 Src/Base/Make.package Tools/GNUMake/Make.defs Tools/GNUMake/comps/hpcsdk.mak Tools/GNUMake/comps/ibm.mak Tools/GNUMake/comps/pgi.mak Tutorials/GPU/Launch/Readme.md commit 63db5cbe23e2ddc419e6895ec69610388519ca9d Author: Michele Rosso Date: Thu Dec 17 11:14:35 2020 -0800 Add features to fextract (#1657) Add the following features to `fextract`: - user option to set the number of decimals in the output of reals - user option to set a cut-off tolerance: anything smaller than the tolerance gets printed as zero - user option to disable the job summary report at the end of the output The job summary report is disabled by default in contrast with the current behaviour. Tools/Plotfile/fextract.cpp commit 1f8891e49f23397b5633b34dae4437619d3b46a8 Author: Andrew Myers Date: Thu Dec 17 10:40:55 2020 -0800 Fix parallel add bug when both USE_GPU and USE_OMP are defined. (#1654) We cannot assume copies are thread safe any time AMREX_USE_GPU is defined, whether or not USE_OMP is also on. Note that this only would have been an issue when compiling with OMP but running on only one thread. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX Src/Base/AMReX_FabArrayBase.cpp commit e63397244cd19a67d1e76fdedce73667d41e8f2c Author: Andy Nonaka Date: Thu Dec 17 10:38:23 2020 -0800 Write a plotfile to ASCII (#1638) This utility reads in a single-level plotfile, copies it to a MultiFab with a single box, then writes out all the data in 'i j k comp ' format to the screen The user can modify this cpp file to write out on certain components, coordinates, row/column formatting, etc. Usage: ./WritePlotfileToASCII2d.gnu.MPI.ex infile=inputFileName Docs/sphinx_documentation/source/Post_Processing.rst Docs/sphinx_documentation/source/Post_Processing_Chapter.rst Docs/sphinx_documentation/source/index.rst Tools/Postprocessing/C_Src/GNUmakefile Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp commit 7b24139e3214b417b54f8d6238131dc484bc6e66 Author: Michele Rosso Date: Wed Dec 16 12:31:01 2020 -0800 CMake: AMRDATA needs Fortran (#1645) Tools/CMake/AMReXOptions.cmake commit 75373313e44803c3732096a2dee413b52ed2d62e Author: Michele Rosso Date: Wed Dec 16 12:30:36 2020 -0800 CMake: fix few quirks with GPU backends variables (#1561) Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake commit b851ce78a2f3f26d53eb101aa0a0266284f44bfb Author: Kevin Gott Date: Wed Dec 16 12:23:39 2020 -0800 FaceLinear: Avoid accessing face outside of coarse box. (#1651) ## Summary Fix an index out-of-bound trigger in FaceLinear interpolater. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H commit 08e27d80f3cb0151566b05f964fb4fc30296f56c Author: Axel Huebl Date: Wed Dec 16 12:09:48 2020 -0800 CI: fix macOS (#1652) * CI: fix macOS brew did again some updates and things fail. let's revert our current work-around. * Brew: Add gfortran not pulled automatically anymore .github/workflows/dependencies/dependencies_mac.sh commit d7b6b97354c02f2e37761f8afae0fca36e7a4550 Author: Candace Gilet Date: Wed Dec 16 14:13:52 2020 -0500 Fix NodalProjector::getGradPhi for multilevel case (#1642) ## Summary Two bug fixes for multilevel in NodalProjector: 1. Ensure NodalProjector::getGradPhi() returns the correct result. Previously could return \sum{sigma grad phi} / (2**dim sigma_coarse). 2. Average down the complete projected velocity at the end. For alpha!=1, this ensures the projected velocity is averaged down correctly. This could change the results of multilevel regression tests. If the application uses the affected outputs from NodalProjector as is, then changes are expected. If the application subsequently averages down the outputs, then changes greater than roundoff level are not expected. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 70b7ab9999522a51666027ae3f7f3dc527541290 Author: Max Katz Date: Wed Dec 16 09:10:42 2020 -0800 Remove GPU pragma support from GNU make build system (#1650) This is no longer used. Tools/F_scripts/gpu_fortran.py Tools/F_scripts/write_cuda_headers.py Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/sites/Make.llnl commit affcccf3c900d1d29349aaeb0c5f21c9dd63e637 Author: Max Katz Date: Wed Dec 16 09:08:45 2020 -0800 Remove GPU pragma macros from ArrayLim (#1649) These are no longer used. Src/Base/AMReX_ArrayLim.H commit 78eae95a21df8fa1d0e2a7c86b0437dd68e8aa7a Author: Max Katz Date: Wed Dec 16 09:06:31 2020 -0800 Remove custom MFIter reduction code (#1648) This is no longer in use by any of the astro codes. Src/Base/AMReX_BLFort.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit 2f9277a64058c91a5d0153b77bd2476bd1f85690 Author: Axel Huebl Date: Tue Dec 15 17:06:11 2020 -0800 CMake: Fix Typos in SYCL Warnings (#1647) Just a typo and a missing space in a warning. Tools/CMake/AMReXOptions.cmake commit 006569febeae1329fc0a3e0c2fb17c77e42539c7 Author: Weiqun Zhang Date: Tue Dec 15 16:30:54 2020 -0800 DPCPP MPI wrapper (#1646) ## Summary Replace mpiicpx with mpiicpc because mpiicpc is shipped by Intel, whereas mpiicpx is a script that is only available on JLSE testbeds. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Co-authored-by: Michele Rosso Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/Make.defs Tutorials/GPU/CNS/Source/CNS_bcfill.cpp Tutorials/GPU/CNS/Source/hydro/CNS_hydro_K.H commit cd2e06e47cb586d0b8e2ef3eb0c534548634ad97 Author: Weiqun Zhang Date: Tue Dec 15 10:44:19 2020 -0800 Remove support for old versions of DPCPP (#1644) Src/Base/AMReX_Extension.H Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_RandomEngine.H Src/Base/AMReX_Scan.H commit cd31efb8aee323dafddb6040c0613667cfb5382c Author: Weiqun Zhang Date: Tue Dec 15 09:14:51 2020 -0800 HIP and DPCPP support in configure (#1640) Tools/GNUMake/comps/hip.mak Tools/libamrex/configure.py Tools/libamrex/mkconfig.py commit 66e52d44979d14859e007d769fb1b550ec591cc7 Author: Weiqun Zhang Date: Mon Dec 14 15:45:16 2020 -0800 Fused FillBoundary (#1570) ## Summary Add a new FillBoundary function for a Vector of FabArrays. There are two implementations in this PR. The one being used simply calls `FillBoundary_nowait` and `FillBoundary_finish`. The other version aggregates the MPI messages from multiple FabArrays into one. Tests on summit shows the latter is slower. But I don't want to lose the code, so it is kept for future investigation. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Array4.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_Box.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp commit 6a212600436d83fcfb656ce72d6c810c1b7c1cdd Author: Kevin Gott Date: Mon Dec 14 13:51:13 2020 -0800 Divergence Free Stencil and API (#1483) API for multi-dimensional, face-based, coarse-to-fine interpolation (InterpFromCoarse and FillPatchTwoLevels). Includes a variation of Vanella et. al. stencil (doi:10.1016/j.jcp.2010.05.003, section 3.2), but instead of interpolating from a far coarse cell to solve interior closure problem (highly biased result dependent on chosen coarse cell), this uses least squares with initial guess equal to the average of fine face values across the cell. (Mathematica notebooks of solution are available). Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_1d.F90 Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_GpuLaunch.H Tests/DivFreePatch/GNUmakefile Tests/DivFreePatch/Make.package Tests/DivFreePatch/inputs Tests/DivFreePatch/main.cpp commit 78d6f64fadc606d65e30d4da5f218e941bc7ea59 Author: Weiqun Zhang Date: Mon Dec 14 10:19:51 2020 -0800 Constant coefficient MacProjector (#1636) Optimization of constant coefficient MAC projection by using constant coefficient Poisson solver. Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit 4794c1d1c2e1fd83bc5ac6be8ff71c4893861128 Author: Weiqun Zhang Date: Mon Dec 14 08:39:07 2020 -0800 Option to allow multiple active MFIters (#1631) Add MFIter::allowMultipleMFIters(bool) so that users can disable assertion on the number of active MFIters. This is needed by FLASH because multiple MFIters (one for each level) are built in its iterator. Docs/sphinx_documentation/source/Basics.rst Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/F_Interfaces/Base/AMReX_multifab_fi.cpp Src/F_Interfaces/Base/AMReX_multifab_mod.F90 commit 8e73aa00126634cd83d4ca9c46fbac20da924e4e Author: drangara <69211175+drangara@users.noreply.github.com> Date: Mon Dec 14 11:37:42 2020 -0500 Convert AMReX_eb_to_pvd.F90 to cpp (#1627) ## Summary Convert the fortran file that writes the EB surface for paraview visualization to cpp ## Additional background Tested with the visualization files produced by `mfix/benchamrk/07-hopper` and `mfix/tutorials/clr/prototype`. Paraview visualization seems to look "good". When comparing the actual data from the files I do see some differences b/w the fortran and cpp version, but I believe the differences can be attributed to the precision of datatypes used in the fortran vs cpp code. These changes should not affect the solution results, just affects visualization. Src/EB/AMReX_EBToPVD.H Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_WriteEBSurface.cpp Src/EB/AMReX_WriteEB_F.H Src/EB/AMReX_eb_to_pvd.F90 Src/EB/CMakeLists.txt Src/EB/Make.package commit c0a931f0ca46d62210e5b8f2e92c6952cfd60685 Author: Weiqun Zhang Date: Sat Dec 12 14:42:57 2020 -0800 Fix EB bug in #1629 (#1635) For EB nodal solver with constant sigma, the domain ghost cells of sigma should be set to zero. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit f6f883bfedf2ecf3900c7f79f1f4df3ec92cca76 Author: Weiqun Zhang Date: Sat Dec 12 10:45:05 2020 -0800 Nodal solver with constant coefficient (#1629) * Add specialization for constant coefficient case in the nodal solver. * Update NodalProjector. * Fix some warnings in 1D. Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 85a5f3be91eb9d58845f96c2907af8ee65661f87 Author: Andrew Myers Date: Sat Dec 12 09:44:58 2020 -0800 fix bug in pshifting neighbors introduced in #1590 (#1632) Src/Particle/AMReX_ParticleCommunication.H commit 52d32c6a07cd14498129ec3f03ae69a06a27573e Author: Ann Almgren Date: Fri Dec 11 21:10:38 2020 -0800 When we create a new level of refinement, we need to set the "level_s… (#1630) …teps" of that new level appropriately. ## Summary ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Amr/AMReX_Amr.cpp commit 389a0cc49f018ea86df0067bc8182336426fe414 Author: Andrew Myers Date: Fri Dec 11 16:24:59 2020 -0800 Expose the index of the neighboring particle to users of the iterator. (#1628) The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Particle/AMReX_NeighborList.H commit 73509be24826d16680289e5555cdb8db1e2217e5 Author: Weiqun Zhang Date: Thu Dec 10 10:02:10 2020 -0800 Documentation on avoiding continue and return inside GPU launch macros. (#1617) Docs/sphinx_documentation/source/GPU.rst commit 728db75fa70fcf10fb7fcb53210ba988ce24f1e4 Author: Weiqun Zhang Date: Thu Dec 10 10:01:16 2020 -0800 Implement overset mask for MLPoisson too (#1604) Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H Src/LinearSolvers/MLMG/Make.package commit 8db1187e6faa4d6628d47ddb28dcc23200516451 Author: Axel Huebl Date: Thu Dec 10 09:47:03 2020 -0800 Warning in SP: large ICC float constant (#1626) Fix an ICC warning for a too-large float constant when compiling for single precision in WarpX. Src/Amr/AMReX_Amr.cpp commit cd88adc2e64456116fcaf0dcce4669921bdb2296 Author: Weiqun Zhang Date: Wed Dec 9 18:57:46 2020 -0800 Fuse nodal jacobi smoother for GPU (#1625) ## Summary Fuse nodal jacobi smoother kernels by switch the order of the sweep and MFIter loops. For Amr-Wind's ABL test with `amr.n_cell = 256 256 64` and `amr.max_grid_size = 32` on one V100, the nodal project is about 2x faster. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit b846969f308e0402e188b95756b478261a31a8d2 Author: Matt Larsen Date: Wed Dec 9 16:26:11 2020 -0800 don't use geom.size() as the number of levels (#1624) ## Summary The conduit adapter was using the `geom.size()` to determine the number of levels, instead of believing `n_levels`. There was a case where the max levels was greater than the actual refinement level. In this case, geoms was the size of the max_levels in the input deck, but mfs was the size of the active levels. resolves #1623 The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 923752e81d08a87d8edd7aaf01e92359125e4f86 Author: OscarAntepara <52221614+OscarAntepara@users.noreply.github.com> Date: Wed Dec 9 08:15:17 2020 -0800 Adding eb slopes limiter (#1594) * Adding eb slope limiters * Adding comments and making 3d adjustments * Fixing face centroids computation in the z-direction and correcting the comments Src/EB/AMReX_EB_slopes_K.H commit ffa5387dd6d873987386cd9ebbea3bab9592c9fd Author: Andrew Myers Date: Tue Dec 8 13:36:03 2020 -0800 Support for SoA data for neighbor particles. (#1590) Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_Particles.H Tests/Particles/NeighborParticles/MDParticleContainer.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/main.cpp Tests/Particles/Redistribute/main.cpp commit a28b02f1d194a0bce76cf784c6dc98cad1339015 Author: Axel Huebl Date: Tue Dec 8 11:13:42 2020 -0800 CMake: Verify DPC++ Compiler ID and at best warn (#1619) * CMake: less stringed MPI+DPCPP Check `find_package(MPI)` is capable to find Intel's MPI with a dpcpp CXX compiler. I would relax this check. Also, `CMAKE_CXX_COMPILER` can be an absolute path, e.g. `-DCMAKE_CXX_COMPILER=$(which dpcpp)`, which is not covered by the original syntax. * DPCPP Compiler Check: Only Warn on ID * Update recommendation * keep leading \n (consistency) Tools/CMake/AMReXOptions.cmake commit e0896cbd9e0432f642d484675e2a4f090ce93e33 Author: Houjun Date: Tue Dec 8 10:53:32 2020 -0800 Fix a bug that causes errors when writing HDF5 plotfile and particle (#1621) * Add HDF5 optimizations * Fix an issue that causes incorrect HDF5 data write with Tutorials/Basic/HeatEquation_EX1_C example code * Minor HDF5 tuning parameter change Co-authored-by: Houjun Tang Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_ParticleHDF5.H commit 82ca8fa508cad15c3f63c6233c8b028da5b4e347 Author: Michael Zingale Date: Tue Dec 8 12:34:11 2020 -0500 the A64fx Cray 10.0.1 compilers need the old-style options (#1620) ## Summary The default cray compilers on the Ookami A64fx machine use the old-style options for cray compilers. This logic allows us to build there. ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/comps/cray.mak commit 91a48a24777dabe03b1ff32bfce05536ce806200 Author: Weiqun Zhang Date: Mon Dec 7 20:00:29 2020 -0800 Fortran Inteface: Porting more MultiFab functions (#1609) Port IntVect ghost cell version of MultiFab::Add, Subtruct, Multiply, Divide, Saxpy, and LinComb to Fortran Interface. Src/F_Interfaces/Base/AMReX_multifab_fi.cpp Src/F_Interfaces/Base/AMReX_multifab_mod.F90 commit 975135ebdec9e0d96f9225a0196fade8ccbd2196 Author: Andrew Myers Date: Mon Dec 7 14:46:19 2020 -0800 [CI] turn off CLEAN option to keep existing files in the deploy target. (#1618) .github/workflows/docs.yml commit 8d45dd95dde0b6cb579c41b13ed3b0b69add297b Author: Axel Huebl Date: Mon Dec 7 13:40:33 2020 -0800 Conduit: Fix unused variables (#1615) ## Summary Removes four (4) unused variable warnings. cc @cyrush @mclarsen ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 774ec481be4fdf28bd80f5a381af9b1c97d5b007 Author: Axel Huebl Date: Mon Dec 7 13:32:59 2020 -0800 DPC++: Re-Enable with Work-around. (#1613) My former colleagues shared a work-around that we can use to overwrite the compiler identification: https://gitlab.kitware.com/cmake/cmake/-/issues/21551#note_869580 This regression will also be fixed in CMake 3.19.2 .github/workflows/linux.yml commit ef6ebdbe62366e78d542fa13739e6d0557a3cbf4 Author: Andrew Myers Date: Mon Dec 7 11:48:19 2020 -0800 deploy to docs to seperate 'build' directory. (#1614) .github/workflows/docs.yml commit 2bbaf1bd81b0b8fee3dbd1bc95f2fc5c22009ac3 Author: Andrew Myers Date: Mon Dec 7 10:55:56 2020 -0800 fix and simplify script that builds the documentation (#1612) build_docs.sh commit 6fb23a385bf2fa0bc2e3b64a24e4940b4c0ea16d Author: Andrew Myers Date: Mon Dec 7 10:32:11 2020 -0800 need to specify the target directory on the website repository (#1611) .github/workflows/docs.yml commit 0a402eef6c0f7bdcb609e786d3140a991e99b7b3 Author: Andrew Myers Date: Mon Dec 7 08:59:32 2020 -0800 Migrate building and deploying the documentation to Github CI. (#1603) .github/workflows/dependencies/documentation.sh .github/workflows/docs.yml .travis.yml Docs/sphinx_tutorials/Makefile build_and_deploy.sh build_docs.sh deploy_key.enc commit 81a56bd628b92745c2baef58d8e7b044306cc34c Author: Weiqun Zhang Date: Sun Dec 6 18:46:48 2020 -0800 Add check for plotfile tools (#1610) .github/workflows/linux.yml Tools/Plotfile/fboxinfo.cpp Tools/Plotfile/fextract.cpp Tools/Plotfile/fextrema.cpp commit 65615e97ef62a2cac302ecbe30482d35f0806542 Author: Weiqun Zhang Date: Sun Dec 6 16:05:16 2020 -0800 Fix error in #1599 (#1608) The proposed change fixes an error in #1599. Tools/Plotfile/fsnapshot.cpp commit 88236accbcc263cbf3d4b103189bc7b522e06dfd Author: Andrew Myers Date: Sun Dec 6 14:33:50 2020 -0800 Remove alternative operators from AMReX (#1599) Operators like `and`, `or`, and `not` are allowed in the C++ standard; however, they cause problems on Windows and with our clang Windows CI. This PR removes them from AMReX in favor of `&&`, `||`, and `!`, and also configures the CI to throw a compilation error if they are re-introduced into the codebase. Application codes are still free to use these if they want. Co-authored-by: Weiqun Zhang .github/workflows/linux.yml .github/workflows/macos.yml Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrParGDB.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_Array4.H Src/Base/AMReX_AsyncOut.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_CuptiTrace.cpp Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_FilCC_C.cpp Src/Base/AMReX_FileSystem.cpp Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_PCI.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_PlotFileDataImpl.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_Slopes_K.H Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Vector.H Src/Base/AMReX_VisMF.cpp Src/Boundary/AMReX_InterpBndryData_2D_K.H Src/Boundary/AMReX_InterpBndryData_3D_K.H Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EBAmrUtil.cpp Src/EB/AMReX_EBFluxRegister_2D_C.H Src/EB/AMReX_EBFluxRegister_3D_C.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EB_slopes_K.H Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_WriteEBSurface.cpp Src/EB/AMReX_algoim_K.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/LinearSolvers/CellOverset/MyTest.cpp Tests/LinearSolvers/EBTensor/MyTest.cpp Tests/LinearSolvers/NodalOverset/MyTest.cpp Tests/LinearSolvers/TensorOverset/MyTest.cpp Tools/Plotfile/fcompare.cpp Tools/Plotfile/fsnapshot.cpp Tools/Postprocessing/C_Src/particle_compare.cpp Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tutorials/LinearSolvers/ABecLaplacian_C/MyTest.cpp Tutorials/Particles/CellSortedParticles/CellSortedPC.cpp Tutorials/Particles/NeighborList/MDParticleContainer.cpp commit ad7974de5b3e68fe2f8fa7c114ffaa460ad166a2 Author: hsitaram Date: Sun Dec 6 14:08:33 2020 -0700 Eb from stl (#1592) ## Summary This is a work-in-progress PR that adds a preliminary set of tools and helper functions for using triangulated STL files in amrex based applications. Coupling with AMReX's EB framework will be added in later pull requests. This PR is done early so as to provide a software framework for incremental changes. As of now, I have added one helper function that provides a nodal multifab with a marker value indicating inside or outside the STL geometry. I guess it can be used for particle-wall collisions. I have also added a test case in the EB folder that generates a nodal multifab with the blanked values for an airfoil. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Co-authored-by: Hariswaran Sitaraman Co-authored-by: Hariswaran Sitaraman Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/CMakeLists.txt Src/EB/Make.package Tutorials/EB/STLtest/GNUmakefile Tutorials/EB/STLtest/Make.package Tutorials/EB/STLtest/airfoil.stl Tutorials/EB/STLtest/inputs Tutorials/EB/STLtest/main.cpp commit 8dc48ad8385cd395310567bf72a36470236e0ce2 Author: Weiqun Zhang Date: Fri Dec 4 17:18:32 2020 -0800 DPCPP: run stencil rap on device (#1606) An earlier compiler bug has been fixed. The stencil rap kernel can run on GPU now. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit 0490136d291cc3b354558f091391277f16a46c5a Author: Weiqun Zhang Date: Fri Dec 4 17:11:29 2020 -0800 Remove DPCPP hacks (#1605) Because the Intel graphics driver has increased the kernel parameters to 2K by default, these hacks are no longer needed. Src/Base/AMReX_GpuLaunchFunctsG.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_Level.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp commit 13d67fd18e736e96bfe4e67a6387750380900384 Author: Axel Huebl Date: Fri Dec 4 10:25:55 2020 -0800 CI: Disable DPC++ oneAPI 2020-11 (#1602) Someone patched in a dedicated CMake compiler ID for Intel DPC++ but did not add full compiler support. Release: 2021.1.1 Ref.: https://gitlab.kitware.com/cmake/cmake/-/issues/21551 .github/workflows/linux.yml commit a168f5927cee0d87ff839b23128c8fc878622313 Author: Michele Rosso Date: Thu Dec 3 16:19:42 2020 -0800 CMake: bring back check on CUDA computing capabilities (#1600) Tools/CMake/AMReX_SetupCUDA.cmake commit 2a070723e249e3e4cf2befd61a576330fc280d80 Author: Andrew Myers Date: Thu Dec 3 14:05:14 2020 -0800 move the memcpy wrapper into amrex::Gpu (#1598) Src/Base/AMReX_GpuUtility.H commit 5b436831f47158a50738dcd64a6fe579c9ef8861 Author: Andrew Myers Date: Thu Dec 3 12:35:20 2020 -0800 Add wrapper for memcpy. (#1597) `std::memcpy` does not seem to work in device code in our HIP CI. This adds a wrapper that reverts to the global namespace `memcpy` in HIP device code. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_GpuUtility.H commit ef665e2a2fcc245162767b3613aba48bbc59f958 Author: Axel Huebl Date: Thu Dec 3 11:16:38 2020 -0800 Fix: Windows and/or includes (iso646) (#1593) ## Summary Add missing includes to `AMReX_Extension.H`, which pulls the `` include on Windows for support for `and`/`or`. ## Additional background Seen with Clang 11.0.0 on x64 Windows with Visual Studio 2017 on conda-forge during a WarpX build. See also #947 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Extension.H Src/Base/AMReX_Vector.H Src/Particle/AMReX_ParticleTile.H commit 5c1addce382f86f0701e6bddc209522cfe70c52d Author: Axel Huebl Date: Thu Dec 3 10:33:42 2020 -0800 CI: Tutorial with OMP (#1591) Making sure the OpenMP Tutorials build as well. .github/workflows/linux.yml Tutorials/Particles/ElectromagneticPIC/CMakeLists.txt commit 0d000b5397b5f6ef94924d85b65846c5811faf86 Author: Weiqun Zhang Date: Thu Dec 3 10:23:15 2020 -0800 AMReX_Config.H (#1566) Based on discussions with @sayerhs, @ax3l, and @mic84, We have decided to make the following changes to the build systems. All headers in Src/ will include AMReX_Config.H, which will be generated by the build systems (both gnu make and cmake). The file will include all the macros we currently pass as compiler options. The purpose of this is to solve some inconsistency issues in the current systems. We support both super build and using AMReX as a library. Moreover, we do not want to force libamrex users to use the same build tool that is used to build the library. In this PR, we are put all the macro definitions in AMReX_Config.H that is included by every AMReX header. Co-authored-by: Axel Huebl Co-authored-by: Michele Rosso Src/Amr/AMReX_Amr.H Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AuxBoundaryData.H Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Extrapolater.H Src/Amr/AMReX_LevelBld.H Src/Amr/AMReX_PROB_AMR_F.H Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_extrapolater_1D_K.H Src/Amr/AMReX_extrapolater_2D_K.H Src/Amr/AMReX_extrapolater_3D_K.H Src/Amr/AMReX_extrapolater_K.H Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrParGDB.H Src/AmrCore/AMReX_AmrParticles.H Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_FLUXREG_F.H Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_F.H Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FluxReg_1D_C.H Src/AmrCore/AMReX_FluxReg_2D_C.H Src/AmrCore/AMReX_FluxReg_3D_C.H Src/AmrCore/AMReX_FluxReg_C.H Src/AmrCore/AMReX_FluxRegister.H Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_TagBox.H Src/Base/AMReX.H Src/Base/AMReX_Algorithm.H Src/Base/AMReX_Arena.H Src/Base/AMReX_Array.H Src/Base/AMReX_Array4.H Src/Base/AMReX_ArrayLim.H Src/Base/AMReX_AsyncOut.H Src/Base/AMReX_BArena.H Src/Base/AMReX_BCRec.H Src/Base/AMReX_BCUtil.H Src/Base/AMReX_BC_TYPES.H Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_BLFort.H Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_BLassert.H Src/Base/AMReX_BackgroundThread.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseFabUtility.H Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BaseUmap_f.H Src/Base/AMReX_BlockMutex.H Src/Base/AMReX_Box.H Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxDomain.H Src/Base/AMReX_BoxIterator.H Src/Base/AMReX_BoxList.H Src/Base/AMReX_CArena.H Src/Base/AMReX_CONSTANTS.H Src/Base/AMReX_COORDSYS_1D_C.H Src/Base/AMReX_COORDSYS_2D_C.H Src/Base/AMReX_COORDSYS_3D_C.H Src/Base/AMReX_COORDSYS_C.H Src/Base/AMReX_CoordSys.H Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_CuptiTrace.H Src/Base/AMReX_DArena.H Src/Base/AMReX_Dim3.H Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_EArena.H Src/Base/AMReX_Exception.H Src/Base/AMReX_Extension.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FPC.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_FabConv.H Src/Base/AMReX_FabFactory.H Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_FilCC_C.H Src/Base/AMReX_FilND_C.H Src/Base/AMReX_FileSystem.H Src/Base/AMReX_ForkJoin.H Src/Base/AMReX_Functional.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_Gpu.H Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuAssert.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuComplex.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuElixir.H Src/Base/AMReX_GpuError.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuKernelInfo.H Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchGlobal.H Src/Base/AMReX_GpuLaunchMacrosC.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuRange.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_INT.H Src/Base/AMReX_IndexSequence.H Src/Base/AMReX_IndexType.H Src/Base/AMReX_IntConv.H Src/Base/AMReX_IntVect.H Src/Base/AMReX_LayoutData.H Src/Base/AMReX_Lazy.H Src/Base/AMReX_Loop.H Src/Base/AMReX_MFCopyDescriptor.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_Machine.H Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MakeType.H Src/Base/AMReX_Math.H Src/Base/AMReX_MemPool.H Src/Base/AMReX_MemProfiler.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H Src/Base/AMReX_MultiFabUtil_C.H Src/Base/AMReX_MultiFabUtil_nd_C.H Src/Base/AMReX_NFiles.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_OpenMP.H Src/Base/AMReX_Orientation.H Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelReduce.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_Partition.H Src/Base/AMReX_Periodicity.H Src/Base/AMReX_PhysBCFunct.H Src/Base/AMReX_PlotFileDataImpl.H Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_Print.H Src/Base/AMReX_REAL.H Src/Base/AMReX_Random.H Src/Base/AMReX_RandomEngine.H Src/Base/AMReX_RealBox.H Src/Base/AMReX_RealVect.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_SPACE.H Src/Base/AMReX_Scan.H Src/Base/AMReX_Slopes_K.H Src/Base/AMReX_ThirdPartyProfiling.H Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Utility.H Src/Base/AMReX_Vector.H Src/Base/AMReX_VectorIO.H Src/Base/AMReX_VisMF.H Src/Base/AMReX_ccse-mpi.H Src/Base/AMReX_filcc_f.H Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_parstream.H Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_BoundCond.H Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_InterpBndryData.H Src/Boundary/AMReX_InterpBndryData_1D_K.H Src/Boundary/AMReX_InterpBndryData_2D_K.H Src/Boundary/AMReX_InterpBndryData_3D_K.H Src/Boundary/AMReX_InterpBndryData_K.H Src/Boundary/AMReX_LOUtil_K.H Src/Boundary/AMReX_LO_BCTYPES.H Src/Boundary/AMReX_MacBndry.H Src/Boundary/AMReX_Mask.H Src/Boundary/AMReX_MultiMask.H Src/Boundary/AMReX_YAFluxRegister.H Src/Boundary/AMReX_YAFluxRegister_1D_K.H Src/Boundary/AMReX_YAFluxRegister_2D_K.H Src/Boundary/AMReX_YAFluxRegister_3D_K.H Src/Boundary/AMReX_YAFluxRegister_K.H Src/CMakeLists.txt Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Graph.H Src/EB/AMReX_EB2_IF.H Src/EB/AMReX_EB2_IF_AllRegular.H Src/EB/AMReX_EB2_IF_Base.H Src/EB/AMReX_EB2_IF_Box.H Src/EB/AMReX_EB2_IF_Complement.H Src/EB/AMReX_EB2_IF_Cylinder.H Src/EB/AMReX_EB2_IF_Difference.H Src/EB/AMReX_EB2_IF_Ellipsoid.H Src/EB/AMReX_EB2_IF_Extrusion.H Src/EB/AMReX_EB2_IF_Intersection.H Src/EB/AMReX_EB2_IF_Lathe.H Src/EB/AMReX_EB2_IF_Plane.H Src/EB/AMReX_EB2_IF_Polynomial.H Src/EB/AMReX_EB2_IF_Rotation.H Src/EB/AMReX_EB2_IF_Scale.H Src/EB/AMReX_EB2_IF_Sphere.H Src/EB/AMReX_EB2_IF_Spline.H Src/EB/AMReX_EB2_IF_Torus.H Src/EB/AMReX_EB2_IF_Translation.H Src/EB/AMReX_EB2_IF_Union.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_MultiGFab.H Src/EB/AMReX_EBAmrUtil.H Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EBFArrayBox.H Src/EB/AMReX_EBFabFactory.H Src/EB/AMReX_EBFluxRegister.H Src/EB/AMReX_EBFluxRegister_2D_C.H Src/EB/AMReX_EBFluxRegister_3D_C.H Src/EB/AMReX_EBFluxRegister_C.H Src/EB/AMReX_EBInterpolater.H Src/EB/AMReX_EBMultiFabUtil.H Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EBMultiFabUtil_C.H Src/EB/AMReX_EBSupport.H Src/EB/AMReX_EB_slopes_K.H Src/EB/AMReX_EB_utils.H Src/EB/AMReX_MultiCutFab.H Src/EB/AMReX_WriteEBSurface.H Src/EB/AMReX_WriteEB_F.H Src/EB/AMReX_algoim.H Src/EB/AMReX_algoim_K.H Src/EB/AMReX_distFcnElement.H Src/Extern/Conduit/AMReX_Conduit_Blueprint.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_Habec_K.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap2.H Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/Extern/HYPRE/CMakeLists.txt Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/CMakeLists.txt Src/Extern/ProfParser/AMReX_AVGDOWN_F.H Src/Extern/ProfParser/AMReX_BLProfStats.H Src/Extern/ProfParser/AMReX_BLProfUtilities.H Src/Extern/ProfParser/AMReX_BLWritePlotFile.H Src/Extern/ProfParser/AMReX_CommProfStats.H Src/Extern/ProfParser/AMReX_RegionsProfStats.H Src/Extern/ProfParser/CMakeLists.txt Src/Extern/SENSEI/AMReX_AmrDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrInSituBridge.H Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrMeshInSituBridge.H Src/Extern/SENSEI/AMReX_InSituBridge.H Src/Extern/SENSEI/AMReX_InSituUtils.H Src/Extern/SENSEI/CMakeLists.txt Src/Extern/SWFFT/Dfft.H Src/Extern/amrdata/AMReX_AmrData.H Src/Extern/amrdata/AMReX_AmrvisConstants.H Src/Extern/amrdata/AMReX_DataServices.H Src/Extern/amrdata/AMReX_XYPlotDataList.H Src/Extern/hpgmg/BL_HPGMG.H Src/F_Interfaces/AmrCore/AMReX_FAmrCore.H Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.H Src/F_Interfaces/Base/AMReX_FPhysBC.H Src/F_Interfaces/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp_F.H Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMGBndry.H Src/LinearSolvers/MLMG/AMReX_MLMG_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLTensor_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_K.H Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_BinIterator.H Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleBufferMap.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleMPIUtil.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particle_mod_K.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_SparseBins.H Src/Particle/AMReX_StructOfArrays.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.H Src/Particle/AMReX_WriteBinaryParticleData.H Src/Particle/CMakeLists.txt Src/SDC/AMReX_SDCstruct.H Tools/CMake/AMReXGenerateConfigHeader.cmake Tools/CMake/AMReXInstallHelpers.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/CMake/AMReX_Config.cmake Tools/CMake/AMReX_Defines.cmake Tools/CMake/AMReX_Utils.cmake Tools/CMake/modify_installed_headers.cmake Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/libamrex/mkconfig.py Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H commit 5fb87ea35fa521e9ce97dc9d36f21eda5e2343be Author: Shreyas Ananthan Date: Thu Dec 3 09:19:23 2020 -0700 Restore reproducibility of solutions with MacProjector reuse (#1596) Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit ae4cc9a1488719e284409b078887866fe65709d9 Author: Michele Rosso Date: Wed Dec 2 15:12:33 2020 -0800 CMake: output missing components list when AMReX is not found (#1572) Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXInstallHelpers.cmake commit 75c0896594cd38d883d7907466657eac3ca1a188 Author: Axel Huebl Date: Wed Dec 2 15:09:17 2020 -0800 CI: Windows + Clang (#1588) Add Clang as Windows target. This compiler offers potentially a route to modern OpenMP. (MSVC: 2.0 standard, LLVM/Clang: 3.1 standard) .github/workflows/windows.yml commit 91bdc795d598b8bbf58d640d572f27dfa5ae31a5 Author: Shreyas Ananthan Date: Wed Dec 2 14:44:56 2020 -0700 MacProjector: Update setDomainBC API (#1589) Allow `setDomainBC` to be called before `setUMAC` has been called. Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit 820a30d7966426a96e24e3963b37f3789eec258f Author: Andrew Myers Date: Wed Dec 2 13:26:03 2020 -0800 Fix InitNRandomPerCell for single precision particles (#1582) * due to roundoff errors, we need to redraw these numbers sometimes when single precision particles are used. * missing ; Src/Particle/AMReX_ParticleInit.H commit dc0ec74c4d120509f9754fb7d43af6d275d9c1a4 Author: Shreyas Ananthan Date: Wed Dec 2 13:35:05 2020 -0700 DPCPP: Update CMake options to allow building with MPI+DPCPP (#1586) Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake commit 89220ed73d9fe74618386f8b1bfafd30760d6fde Author: Andrew Myers Date: Tue Dec 1 10:31:23 2020 -0800 Reimplement NumParticlesAtLevel (#1581) This removes a workaround that had this function fall back to CPUs for CUDA (but not HIP or DPC++). Additionally, this re-implements the function in terms of ReduceTuple to avoid multiple memcpy operations in the case of multiple boxes. Src/Particle/AMReX_ParticleContainerI.H Tests/Particles/ParticleReduce/main.cpp commit bbcce736ea9497cdb4722e8b4f4c2ee0d7c193ca Author: Robert Maynard Date: Tue Dec 1 13:24:16 2020 -0500 eval_genex correctly replaces the matching compiler version (#1580) Previously it would place the resulting output in the variable `1` instead of `_in`. Additionally it would try to add an extra `>`. Tools/CMake/AMReXGenexHelpers.cmake commit 9a523cf95dc3046081b0830cf435a3455929728d Author: Robert Maynard Date: Tue Dec 1 13:20:19 2020 -0500 Add msvc cuda support (#1573) ## Summary This set of changes has allowed me to build AMReX on windows using CUDA 11.X and VS2019. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_ErrorList.cpp Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_Extension.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_MultiFab.H Tools/CMake/AMReX_Config.cmake commit 1cb99250ecd4da445cfc8afe325664030c49c4f0 Author: Weiqun Zhang Date: Tue Dec 1 10:12:38 2020 -0800 Update CHANGES (#1584) CHANGES commit 94ca530cb43c497cf22dfbfbd13dfd087f7d910d Author: Andrew Myers Date: Mon Nov 30 21:55:26 2020 -0500 Fix bug affecting async IO + single precision particles (#1583) The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Particle/AMReX_WriteBinaryParticleData.H commit d9af9ab7f134b04bc5e3cc57362e82a64fecef0a Author: Weiqun Zhang Date: Mon Nov 30 12:49:50 2020 -0800 Fix warnings for DPCPP (#1579) Fix unused variable. Do not pass `-fsycl-unnamed-lambda` to `mpiicpx` for linking to get rid of warning on unknown argument. Src/Base/AMReX_FBI.H Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tutorials/GPU/CNS/Source/CNS.H commit 07afa8ec8f59403ac54c2b2c566c1c88b1863112 Author: Weiqun Zhang Date: Mon Nov 30 10:25:56 2020 -0800 Template specialization of MPI communication functions for char (#1569) * Add template specialization of MPI communication functions for char. Select a proper data type according to the size. * Use the new functions in FabArray and Particle communication. And this also fixes a bug in Particle communication. (The counts should be divided by the size of data type.) * Due to a gcc bug on explicit specialization in a namespace, workaround was implemented in AMReX_ParallelDescriptor.H and .cpp. Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Particle/AMReX_ParticleCommunication.H commit 50c40c46382a83cdc98fd59e337c5037aba07c1e Author: Weiqun Zhang Date: Mon Nov 30 10:16:16 2020 -0800 EB2::PolyIF (#1567) The existing PolynomialIF contains a dynamic size vector. This makes it hard to support it on GPU. We have used an undocumented feature to get this run on GPU. However, this code can no longer compile, because the GNU make system now makes cross execution space calls an error. So we disable GPU support for PolynomialIF and add a new PolyIF class with fixed size array and therefore GPU support. Src/EB/AMReX_EB2_IF_Polynomial.H commit 26e4d30f346cb89921502e409da1b933a60f9c81 Author: Weiqun Zhang Date: Mon Nov 30 10:11:37 2020 -0800 Add .gitattributes (#1578) Make github detect *.H as C++. This file is borrowed from Castro. .gitattributes commit a77968674f3d9ec1d4298d99a506dd1951f88c67 Author: Shreyas Ananthan Date: Mon Nov 30 09:17:48 2020 -0700 Tools/Plotfile: Fix build errors in single precision mode (#1577) This PR fixes one build error and one warning observed when compiling `Tools/Plotfile` utilities with `-DAMREX_USE_FLOAT` Tools/Plotfile/fcompare.cpp Tools/Plotfile/fextract.cpp commit 82eceae29419ceec24450926af1f5445ac425d00 Author: Shreyas Ananthan Date: Mon Nov 30 09:14:54 2020 -0700 Arena: Add accessor to query Arena attributes through info object (#1575) ## Summary This PR adds an accessor to the `Arena` class that can be used to query the attributes of the Arena (e.g., `device_use_managed_memory` flag) from the application codes. ## Additional background Currently, there is no easy way to determine if `amrex::The_Arena()` is using managed memory. Applications would have to use a combination of `AMREX_USE_HIP` compile-time definition and `amrex.the_arena_is_managed` ParmParse variable to determine this. Testing individual pointers with `amrex::Gpu::isManaged()` is possible only on CUDA/HIP but not DPC++. Also it is costly compared to just querying the `Arena` object. Allowing read-only access to `ArenaInfo` object will allow a wide variety of checks in application code. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [X] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Arena.H commit acdb7eac1d5a1744e87c0caafd1c43cc5e1171e9 Author: Luca Fedeli Date: Mon Nov 30 16:58:32 2020 +0100 Fix some warnings related to double-->float conversions (#1571) I tried to compile `WarpX` in single precision and I noticed several warnings related to casts from `double` to `float` in `AMReX`. This PR should fix some of these warnings. I plan to continue in the next days with other PRs, if you agree. Src/AmrCore/AMReX_AmrMesh.H Src/Base/AMReX_CoordSys.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_nd_C.H commit abdab63376660238396626ee064130a5c1826024 Author: Shreyas Ananthan Date: Sun Nov 29 14:04:44 2020 -0700 Fix assertion check for MacProjection (#1576) Fixes bug introduced in #1574 Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit abeb357db57261fe8a8d811e161da0c6c2814d92 Author: Shreyas Ananthan Date: Sun Nov 29 09:53:42 2020 -0700 Allow MacProjection reuse (#1574) This commit modifies the MacProjector interface to allow reuse of the MacProjector. The changes to the API are backwards compatible. - Add a new constructor that does not require specification of beta, umac etc. - Add `initProjector` to explicitly initialize the projection instance at app level - Add methods to set MAC velocities, divU etc. Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit cc8f323c426f3957619c93477af2b9e32e3b49c2 Author: Axel Huebl Date: Mon Nov 23 22:27:06 2020 -0800 CI: Make DPC++ Pretty (#1568) * CI: Make DPC++ Pretty Just combing the show pony. * oneAPI Activation: still needs set +e .github/workflows/linux.yml commit f068164b513c12828dac069510d64f8b4912028e Author: Michele Rosso Date: Mon Nov 23 14:48:58 2020 -0800 Remove SUNDIALS support (#1559) ## Summary SUNDIALS support in AMReX consisted of only few Fortran interfaces to SUNDIALS functions for the convenience of app codes. Since there are no AMReX functionalities depending on SUNDIALS, support for SUNDIALS will be removed. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/SUNDIALS.rst Docs/sphinx_documentation/source/SUNDIALS_CVODE.rst Docs/sphinx_documentation/source/SUNDIALS_top.rst Docs/sphinx_tutorials/source/SUNDIALS_Tutorial.rst Src/CMakeLists.txt Src/Extern/CVODE/Make.package Src/Extern/CVODE/cvode_interface.f90 Src/Extern/CVODE/fnvector_serial.f90 Src/Extern/CVODE/integrator_stats.f90 Src/Extern/CVODE/sundials_fdlsmat.f90 Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package Src/Extern/SUNDIALS/arkode_interface.f90 Src/Extern/SUNDIALS/cvode_interface.f90 Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/FindSUNDIALS.cmake Tools/GNUMake/Make.defs Tools/GNUMake/packages/Make.cvode Tools/GNUMake/packages/Make.sundials Tutorials/CMakeLists.txt Tutorials/CVODE/EX1_F/GNUmakefile Tutorials/CVODE/EX1_F/Make.package Tutorials/CVODE/EX1_F/inputs Tutorials/CVODE/EX1_F/integrate_ode.f90 Tutorials/CVODE/EX1_F/main.cpp Tutorials/CVODE/EX1_F/myfunc_F.H Tutorials/CVODE/EX1_F/ode_mod.f90 Tutorials/CVODE/EX2_F/GNUmakefile Tutorials/CVODE/EX2_F/Make.package Tutorials/CVODE/EX2_F/inputs Tutorials/CVODE/EX2_F/integrate_ode_no_jac.f90 Tutorials/CVODE/EX2_F/integrate_ode_with_jac.f90 Tutorials/CVODE/EX2_F/main.cpp Tutorials/CVODE/EX2_F/myfunc_F.H Tutorials/CVODE/EX2_F/ode_mod.f90 Tutorials/SUNDIALS/EX-CUSOLVER/GNUmakefile Tutorials/SUNDIALS/EX-CUSOLVER/Make.CVODE Tutorials/SUNDIALS/EX-CUSOLVER/Make.package Tutorials/SUNDIALS/EX-CUSOLVER/README.md Tutorials/SUNDIALS/EX-CUSOLVER/extern_probin.template Tutorials/SUNDIALS/EX-CUSOLVER/inputs Tutorials/SUNDIALS/EX-CUSOLVER/inputs_128 Tutorials/SUNDIALS/EX-CUSOLVER/inputs_256 Tutorials/SUNDIALS/EX-CUSOLVER/inputs_32 Tutorials/SUNDIALS/EX-CUSOLVER/inputs_64 Tutorials/SUNDIALS/EX-CUSOLVER/main.cpp Tutorials/SUNDIALS/EX-CUSOLVER/make_cuda.sh Tutorials/SUNDIALS/EX-CUSOLVER/make_cuda_cusolver.sh Tutorials/SUNDIALS/EX-CUSOLVER/make_serial.sh Tutorials/SUNDIALS/EX-CUSOLVER/react_cuda.cpp Tutorials/SUNDIALS/EX-CUSOLVER/react_cuda_cusolver.cpp Tutorials/SUNDIALS/EX-CUSOLVER/react_serial.cpp Tutorials/SUNDIALS/EX-CUSOLVER/react_system.F90 Tutorials/SUNDIALS/EX-CUSOLVER/react_utils.F90 Tutorials/SUNDIALS/EX-CUSOLVER/test_react.H Tutorials/SUNDIALS/EX-CUSOLVER/test_react_F.H Tutorials/SUNDIALS/EX1_C/CMakeLists.txt Tutorials/SUNDIALS/EX1_C/GNUmakefile Tutorials/SUNDIALS/EX1_C/Make.package Tutorials/SUNDIALS/EX1_C/SetIC.f90 Tutorials/SUNDIALS/EX1_C/inputs Tutorials/SUNDIALS/EX1_C/inputs_2box Tutorials/SUNDIALS/EX1_C/inputs_non_vectorized Tutorials/SUNDIALS/EX1_C/main.cpp Tutorials/SUNDIALS/EX1_C/myfunc_F.H Tutorials/SUNDIALS/EX1_CUDA/GNUmakefile Tutorials/SUNDIALS/EX1_CUDA/Make.package Tutorials/SUNDIALS/EX1_CUDA/SetIC.f90 Tutorials/SUNDIALS/EX1_CUDA/inputs Tutorials/SUNDIALS/EX1_CUDA/main.cpp Tutorials/SUNDIALS/EX1_CUDA/myfunc_F.H Tutorials/SUNDIALS/EX1_CUDA/ode_mod.f90 Tutorials/SUNDIALS/EX1_F/GNUmakefile Tutorials/SUNDIALS/EX1_F/Make.package Tutorials/SUNDIALS/EX1_F/inputs Tutorials/SUNDIALS/EX1_F/integrate_ode.f90 Tutorials/SUNDIALS/EX1_F/main.cpp Tutorials/SUNDIALS/EX1_F/myfunc_F.H Tutorials/SUNDIALS/EX1_F/ode_mod.f90 Tutorials/SUNDIALS/EX2_F/GNUmakefile Tutorials/SUNDIALS/EX2_F/Make.package Tutorials/SUNDIALS/EX2_F/inputs Tutorials/SUNDIALS/EX2_F/integrate_ode_no_jac.f90 Tutorials/SUNDIALS/EX2_F/integrate_ode_with_jac.f90 Tutorials/SUNDIALS/EX2_F/main.cpp Tutorials/SUNDIALS/EX2_F/myfunc_F.H Tutorials/SUNDIALS/EX2_F/ode_mod.f90 commit e1ff59c8d9207a135f25284380a1ba8fc15abb1f Author: Roberto Porcu <53792251+rporcu@users.noreply.github.com> Date: Sun Nov 22 23:19:06 2020 -0500 allow MLEBABecLap to solve for multiple components (#1557) ## Summary MLLinOp::getNComp() is returning 1 and right now MLEBABecLap is working only with MultiFabs having 1 component. ## Additional background This PR has the goal to generalize MLEBABecLap solver for a generic N number of components. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Co-authored-by: Weiqun Zhang Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp commit 1f8163c397e1aa33fd633219166b87d4633a0b46 Author: Weiqun Zhang Date: Sun Nov 22 19:40:00 2020 -0800 DPCPP: MPI (#1556) Allow MPI for dpcpp build in gnu make assuming mpiicpx is used. Tools/GNUMake/Make.defs Tools/GNUMake/Make.machines Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/sites/Make.alcf commit a85fd8d7971f9448b8a00e796a392bbe84ae0f97 Author: Axel Huebl Date: Fri Nov 20 10:28:25 2020 -0800 CMake: OpenMP for Fortran (#1563) * CMake: OpenMP for Fortran Fix missing activation. * Update Tools/CMake/AMReXParallelBackends.cmake Co-authored-by: mic84 * CMakeL OpenMP Fortran fix Config, too Co-authored-by: mic84 Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXParallelBackends.cmake commit 7a0fea34be764441014e2ba7dac44fecbdbbabfb Author: Weiqun Zhang Date: Thu Nov 19 11:01:07 2020 -0800 Non-local boundary conditions (#1544) * Non-local boundary conditions * Rotate90 fills the lo-x and lo-y boundary regions by rotating the data around (x=0,y=0) by 90 degrees in either direction. It also fills the corner of lo-x and lo-y boundary region by rotating the data by 180 degrees. * Rotate180 fills the lo-x boundary by rotating the data around (x=0,y=L_y/2) by 180 degrees. * Fill the polar boundaries of the spherical coordinates (theta, phi, r). The lo-x boundary is filled with f(-x,y) = f(x,mod(y+pi,2*pi)), and the hi-x boundary is filled with f(pi+x,y) = f(pi-x,mod(y+pi,2*pi)). * GPU support for NonLocalBC Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/CMakeLists.txt Src/Base/Make.package commit b5e08d92541f24e3ee3cb8d46f5b49c0147538bd Author: Weiqun Zhang Date: Thu Nov 19 10:28:53 2020 -0800 Remove deprecated MFUtil::convert function. One could use amrex::cast for FabArray casting. (#1523) Src/Base/AMReX_MultiFabUtilI.H commit 907baaede901d52601f120d606224bccd04d9611 Author: Andrew Myers Date: Thu Nov 19 13:24:16 2020 -0500 If TheZeroVector() is passed in to SortParticlesByBin, do nothing instead of crashing. (#1564) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_Particles.H commit 6038a3102fc55db861d69b242eecc35119675b96 Author: Weiqun Zhang Date: Wed Nov 18 09:49:08 2020 -0800 Scan size type (#1521) Make the size type a template parameter to support the scan of more than INT_MAX elements. Src/Base/AMReX_Scan.H commit ff92f1c129ea149eead13dc264eabf15a705e74d Author: Michele Rosso Date: Wed Nov 18 09:14:51 2020 -0800 CMake: AMReX_MPI options depends on SYCL, not HIP (#1555) Tools/CMake/AMReXOptions.cmake commit 7cf969f1b656f9ab17975c90952360fb400dd712 Author: Axel Huebl Date: Wed Nov 18 09:14:25 2020 -0800 CMake: Fix Single-Precision Switch (#1550) * CMake: Fix Single-Precision Switch Since the latest refactoring, single-recision builds were not activated. This fixes it. * AMReX_Config.H: add AMREX_SINGLE_PRECISION_PARTICLES Src/Particle/CMakeLists.txt Tools/CMake/AMReX_Config.H.in Tools/CMake/AMReX_Defines.cmake commit 14c3003f8f4dda9ef9db0c04ba613e378677358a Author: Weiqun Zhang Date: Wed Nov 18 09:03:38 2020 -0800 Refactor FabArray's communication functions (#1553) Remove duplication and allow for more flexibility for extension. Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_iMultiFab.cpp Src/Boundary/AMReX_YAFluxRegister.cpp commit 079f599f74057ea93d8c60b82760b8747b21849e Author: Weiqun Zhang Date: Tue Nov 17 18:40:51 2020 -0800 DPCPP: Fortran (#1554) Switch from gfortran to ifx that comes with oneapi in GNU make. Tools/GNUMake/comps/dpcpp.mak commit e38742e322f6fbdf0e0fb090271d229fc2c834b9 Author: Michele Rosso Date: Tue Nov 17 13:26:24 2020 -0800 CMake: improvements to handling of buildInfo (#1552) Tools/CMake/AMReXBuildInfo.cmake commit 477df17dd96607c385fc9410c4fe7746f121b58d Author: Weiqun Zhang Date: Tue Nov 17 11:19:19 2020 -0800 Fix BaseFab::maxabs (#1546) ## Summary For the host version, call host function LoopOnCpu instead of host device function Loop because the function we pass to it is a host function. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_BaseFab.H commit cd6f4d0bdbc389f672e3d6a3b8d286729caca3b6 Author: Andrew Myers Date: Tue Nov 17 13:33:00 2020 -0500 Fix bug in the 'intarray' method of StructOfArrays (#1548) Thanks to @rporcu for finding this. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Particle/AMReX_StructOfArrays.H commit b329cfe0bc4036aae7b69dc94b7bb34f8ab040c9 Author: Axel Huebl Date: Mon Nov 16 21:07:35 2020 -0800 CMake: generate_buildinfo target (#1545) This commit generates a static library `buildInfo::` for the passed user target. This target is then automatically linked. Also, the generated `.cpp` file is now scoped properly, avoiding collisions. This solves the following corner cases: - workflows with OBJECT libraries, that cannot be passed as targets since they have no `PRE_BUILD` custom command support - workflows with multiple targets that are not linked together (those call `generate_buildinfo` multiple times) - workflows with multiple targets that are linked together (those call `generate_buildinfo` once and re-use the target) Tools/CMake/AMReXBuildInfo.cmake commit 5076d33c583d88e0fb78214470715c2d09c083cd Author: Shreyas Ananthan Date: Mon Nov 16 10:00:02 2020 -0700 fcompare: fix logic error when comparing both rel/abs tolerances (#1542) PR #1537 introduced the option to allow tolerance checks on both absolute and relative tolerances. However, the check used `and` instead of `or` to allow tests to pass when either absolute or relative error was below user-specified tolerance. ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/Plotfile/fcompare.cpp commit 95b97799f2e611d19a4b685f725bfa3e5e12dc7f Author: Weiqun Zhang Date: Sun Nov 15 17:11:12 2020 -0800 nvcc --Werror cross-execution-space-call (#1540) * nvcc --Werror cross-execution-space-call * GNU make: Turn on the flag by default. Use `GPU_ERROR_CROSS_EXECUTION_SPACE_CALL=FALSE` to turn it off. * CMake: Add AMReX_ERROR_CROSS_EXECUTION_SPACE_CALL, which if off by default (just like AMReX_ERROR_CAPTURE_THIS). * update cmake documentation Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReX_SetupCUDA.cmake Tools/GNUMake/Make.defs Tools/GNUMake/comps/nvcc.mak commit aef49304278572eb3c4a5db75462c8c213d35221 Author: Max Katz Date: Sun Nov 15 11:22:29 2020 -0800 Don't throw C++ std error when not config checking (#1543) Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/nvcc.mak commit c80007df2f90e373f3d947fa7f621ab902378f3c Author: Shreyas Ananthan Date: Thu Nov 12 17:36:06 2020 -0700 fcompare: Add option for an absolute tolerance check (#1537) This PR adds a command line option to `fcompare` that takes in a user-specified tolerance for absolute error and adds logic to compare both absolute and relative errors against user-specified tolerances. Since `-a` is already used as a short-form for `--allow_diff_grids`, I have only used `--abs_tol` as the option without a short version when parsing command line options. Tools/Plotfile/fcompare.cpp commit cdb236ee435635f8007d2be97c0f6f1df13ecba0 Author: Weiqun Zhang Date: Thu Nov 12 08:50:35 2020 -0800 Remove call to cudaDeviceSetCacheConfig (#1536) ## Summary No longer set CUDA device cache configuration to prefer L1 cache. This does not appear to affect any kernels in a negative way. With more shared memory available, reduction functions are faster in some cases. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_GpuDevice.cpp commit f55a950e0977a4b062bba6ff9787ba84e9f7da92 Author: Weiqun Zhang Date: Thu Nov 12 08:11:02 2020 -0800 Fix multi-component nodal solver tutorial (#1535) ## Summary `MultiFab::setVal` should not be called inside `MFIter`. ## Additional background This was caught by #1530. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tutorials/LinearSolvers/MultiComponent/MCNodalLinOp.cpp commit a01697fda4a2f38a1e7e3c849bc52baca2c2999d Author: Axel Huebl Date: Tue Nov 10 18:18:03 2020 -0800 Ascent: Check Bounds Particle Reals (#1515) Access of particle variable names with a bound-check, so users definitely pass the right amount of names. Src/Extern/Conduit/AMReX_Conduit_Blueprint_ParticlesI.H commit d9f60b4cb934852cb87653fbc84df22c98cfe0e0 Author: Andrew Myers Date: Tue Nov 10 09:02:30 2020 -0800 Allow for particle containers constructed with the same AmrCore... (#1532) Src/AmrCore/AMReX_AmrParGDB.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_Particles.H commit 10c2a5e5949213d607305f771edab433285166d7 Author: Ann Almgren Date: Mon Nov 9 17:58:38 2020 -0800 Update the docs to include hypre.adjust_singular_matrix (#1534) ## Summary adds documentation of hypre.adjust_singular_matrix ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [X] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/LinearSolvers.rst commit 59b8bf1cc1ebf6dac76602a24a4ce696080b070c Author: Weiqun Zhang Date: Mon Nov 9 15:59:56 2020 -0800 DPC++: fix scan (#1533) * Use atomics instead of volatile to read status written to global memory by other blocks * Workaround `Random()` bug in the scan test * Limit the memory usage in the scan test Src/Base/AMReX_Scan.H Tutorials/GPU/ParallelScan/main.cpp commit 15ab2368225b81777e2810e0e02b1293364bb575 Author: Weiqun Zhang Date: Mon Nov 9 14:02:21 2020 -0800 Fix #1530 (#1531) `FillPatchIterator` is derived from `MFIter`. We need to reset `MFIter::depth` so that the ctor of `FillPatchIterator` can start `MFIter`. Src/Amr/AMReX_AmrLevel.cpp commit b94e1d2fca2bffd2b0e826eebfca7d63290efd40 Author: Weiqun Zhang Date: Mon Nov 9 10:30:06 2020 -0800 Assert no nested MFIters (#1530) ## Summary Add assertion to catch nested MFIters (e.g., MultiFab functions are called inside MFIter). ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit ad7c4eac3475fb7ddfc2abe8c74400d0274f8398 Author: Weiqun Zhang Date: Mon Nov 9 10:29:06 2020 -0800 Add anonymous namespace back to parameters in AMReX_Amr.cpp (#1529) Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp commit eb29a4f6b40e6458323db681795c1a12e1403c1c Author: Weiqun Zhang Date: Mon Nov 9 07:51:46 2020 -0800 Free MPI_Datatype defined by AMReX (#1527) ## Summary This fixes a one-time memory leak in MPI_Datatypes defined by AMReX as reported in #1525. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_ParallelDescriptor.cpp commit 0fd351b118b695d9c439c33a0f1ff9c6f3f00aeb Author: Weiqun Zhang Date: Sun Nov 8 10:27:36 2020 -0800 Update OLCF makefile (#1528) ## Summary * Remove summitdev and peak. * Environment variable OLCF_MODULEPATH_ROOT is used in addition to host name to detect OLCF machines. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit 6cc5104e57548e575717051b04361ffed0fee9c2 Author: Weiqun Zhang Date: Sun Nov 8 10:26:56 2020 -0800 Remove getGridSize that is no longer used. (#1524) Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunch.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 86abbe0221c1c3c53e69d740cdc24190ce1376c5 Author: Weiqun Zhang Date: Sat Nov 7 14:57:59 2020 -0800 Support CUDA arch < 6.0 (#1512) ## Summary * Implement atomicAdd for CUDA arch < 6.0. * Remove the CUDA arch check in CMake. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_BaseFab.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Tools/CMake/AMReX_SetupCUDA.cmake commit ec4424eb66366c5b798639a100b30c6c01938f79 Author: Axel Huebl Date: Sat Nov 7 14:56:34 2020 -0800 Change Spelling: Env AMREX_CUDA_ARCH (#1522) ## Summary Make the _environment variable_ that sets a default CUDA architecture all-caps, as this is way more common in Unix. Typical Values: `7.0` or `Volta` (i.e. for V100) ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReX_SetupCUDA.cmake commit 1b00f34ef1e170d4e2b398045e9dd45b4ba8ec34 Author: Weiqun Zhang Date: Fri Nov 6 18:18:27 2020 -0800 make clean & make cleanconfig (#1520) ## Summary `make clean` now does what `make realclean` does. `make realclean` is kept. `make cleanconfig` is introduced to do what `make clean` used to do. For most users, `make clean` is the one that should be used. Also add some tests in makefiles to avoid error messages in case when `make clean` is run on a makefile with the default not suitable for the system. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/README.md Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/sites/Make.unknown commit d9ac9a6c844a5055cdd2b332d101096da3ae5d72 Author: Michele Rosso Date: Fri Nov 6 13:47:58 2020 -0800 CMake: fix options name in Test directory (#1514) Tests/CMakeLists.txt Tests/HDF5Benchmark/CMakeLists.txt commit 6dc26f704a386a03120f71395f00186049499264 Author: Michele Rosso Date: Fri Nov 6 13:47:15 2020 -0800 CMake: add component check for Fortran (#1511) Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXConfig.cmake.in commit 9f2822bdcdd5604eea85a6206413684fb0137ea4 Author: Michele Rosso Date: Fri Nov 6 13:44:41 2020 -0800 CMake: fix some quirks with options output (#1516) Tools/CMake/AMReXOptions.cmake commit 79d991ca8518943155e312a9a7566380e563bc93 Author: Shreyas Ananthan Date: Fri Nov 6 12:41:36 2020 -0700 hypre: Adjust singular matrices before hypre solves (#1519) Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp commit fe6058b946fb4334b74c42fedde919836f82066d Author: Shreyas Ananthan Date: Fri Nov 6 12:04:55 2020 -0700 hypre: Update nodal Laplacian interface to accept custom options (#1518) HypreNodeLap class creates the hypre IJ instance in its constructor. Therefore, the custom option namespace specified by user must be part of the constructor arguments. Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit 7b7c1c8e6addc152fb9c0059577799bfe8dc6d93 Author: Weiqun Zhang Date: Thu Nov 5 16:27:28 2020 -0800 DPC++: sync after htod_memcpy on null stream (#1517) ## Summary DPC++ does not have the concept of null stream. Adding a sync after htod_memcpy on the "null" stream will help eliminate potential bugs (e.g., htod_memcpy is called before MFIter that uses non-null stream). ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_Reduce.H commit 8035b01d86a7f8f4542bec6aa167dc864c8b3209 Author: Shreyas Ananthan Date: Thu Nov 5 15:06:36 2020 -0700 CMake: Fix ENABLE_HYPRE to AMReX_HYPRE (#1513) Fix the variable being checked to enable HYPRE. ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/CMake/AMReXThirdPartyLibraries.cmake commit 9aa75b1844e4ecc6c860a2d5d37b57bccedf73ed Author: Max Katz Date: Thu Nov 5 08:59:20 2020 -0800 Fix some build issues at LLNL (#1510) Tools/GNUMake/sites/Make.llnl commit 7ca532e5b299df1d7e9137e61f9271e4341c6189 Author: Andrew Myers Date: Thu Nov 5 08:54:57 2020 -0800 Add timer to neighbor list construction (#1509) Src/Particle/AMReX_NeighborList.H commit 86c6b3d00ae8d8e232b345b7060d9188fc838136 Author: Michele Rosso Date: Wed Nov 4 19:39:39 2020 -0800 CMake: overhaul options (#1490) * Doc: move ENABLE_CUDA_FASTMATH to the GPU options section * CMake: change all options to include AMReX_ namespace * CMake: remove ENABLE_ACC option * CMake: make AMReX_SPACEDIM a multi-valued string * Doc: add option AMReX_ENABLE_TESTS * CMake: rename AMReX_Options.cmake to AMReXOptions.cmake * CMake: refactor GPU-related options * CMake: MPI must be turned off if DPCPP is enabled * CMake: set via precision via selection of value * CMake: forgot to update Particles/CMakeLists.txt * CMake: append prefix to TL_PROFILE * Update Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Co-authored-by: Axel Huebl * Update Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Co-authored-by: Axel Huebl * Update Docs/sphinx_documentation/source/BuildingAMReX.rst Co-authored-by: Axel Huebl * CMake: components in Config file shall not have namespace prefix * Revert "CMake: components in Config file shall not have namespace prefix" This reverts commit 924cf15f2a926467f3924a76f80595c0d1e53a7a. * Update Tools/CMake/AMReX_SetupCUDA.cmake Co-authored-by: Axel Huebl * CMake: update Doc * Update Tools/CMake/AMReX_SetupCUDA.cmake Co-authored-by: Axel Huebl Co-authored-by: Axel Huebl .github/workflows/linux.yml .github/workflows/macos.yml .github/workflows/windows.yml CHANGES CMakeLists.txt Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/Visualization.rst Src/Amr/CMakeLists.txt Src/AmrCore/CMakeLists.txt Src/Base/CMakeLists.txt Src/Boundary/CMakeLists.txt Src/CMakeLists.txt Src/EB/CMakeLists.txt Src/Extern/HYPRE/CMakeLists.txt Src/Extern/PETSc/CMakeLists.txt Src/Extern/ProfParser/CMakeLists.txt Src/Extern/amrdata/CMakeLists.txt Src/F_Interfaces/CMakeLists.txt Src/LinearSolvers/CMakeLists.txt Src/Particle/CMakeLists.txt Tests/CMakeLists.txt Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXInstallHelpers.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSYCL.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/AMReX_Config.cmake Tools/CMake/AMReX_Defines.cmake Tools/CMake/AMReX_Options.cmake Tools/CMake/AMReX_SetupCUDA.cmake Tools/Plotfile/CMakeLists.txt Tutorials/Amr/Advection_AmrCore/CMakeLists.txt Tutorials/Amr/Advection_AmrLevel/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_C/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_CF/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_F/CMakeLists.txt Tutorials/Basic/HeatEquation_EX2_CF/CMakeLists.txt Tutorials/Basic/HeatEquation_EX3_C/CMakeLists.txt Tutorials/Basic/HelloWorld_F/CMakeLists.txt Tutorials/Basic/main_C/CMakeLists.txt Tutorials/Basic/main_F/CMakeLists.txt Tutorials/Blueprint/AssignMultiLevelDensity/CMakeLists.txt Tutorials/Blueprint/CellSortedParticles/CMakeLists.txt Tutorials/Blueprint/HeatEquation_EX1_C/CMakeLists.txt Tutorials/CMakeLists.txt Tutorials/EB/CNS/CMakeLists.txt Tutorials/EB/GeometryGeneration/CMakeLists.txt Tutorials/ForkJoin/MLMG/CMakeLists.txt Tutorials/ForkJoin/Simple/MyTest.cpp Tutorials/FortranInterface/Advection_F/CMakeLists.txt Tutorials/FortranInterface/Advection_octree_F/CMakeLists.txt Tutorials/FortranInterface/Advection_octree_F2/CMakeLists.txt Tutorials/GPU/CNS/CMakeLists.txt Tutorials/GPU/EBCNS/CMakeLists.txt Tutorials/GPU/Launch/CMakeLists.txt Tutorials/LinearSolvers/ABecLaplacian_C/CMakeLists.txt Tutorials/LinearSolvers/ABecLaplacian_F/CMakeLists.txt Tutorials/LinearSolvers/MAC_Projection_EB/CMakeLists.txt Tutorials/LinearSolvers/Nodal_Projection_EB/CMakeLists.txt Tutorials/Particles/CellSortedParticles/CMakeLists.txt Tutorials/Particles/ElectromagneticPIC/CMakeLists.txt Tutorials/Particles/ElectrostaticPIC/CMakeLists.txt Tutorials/Particles/NeighborList/CMakeLists.txt Tutorials/SUNDIALS/EX1_C/CMakeLists.txt commit 22e6f93f2f66d24e364f35226b3f0d990182e316 Author: Weiqun Zhang Date: Wed Nov 4 17:00:42 2020 -0800 New Reduce::Sum function taking lambda function (#1508) ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Reduce.H commit ae9d0c696dad43b4477517728bf0f2ef978165db Author: Weiqun Zhang Date: Wed Nov 4 14:50:18 2020 -0800 GNU Make: AMREX_CUDA_ARCH (#1507) ## Summary Option to use AMREX_CUDA_ARCH in addition to CUDA_ARCH to set CUDA arch. If AMREX_CUDA is set, it will be used over CUDA_ARCH. ## Additional background The CMake system is also planning to support AMREX_CUDA_ARCH. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [x] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/GNUMake/Make.defs commit 1ab6d82e749b039b0417fa6c539c04d464ba95ac Author: Andrew Myers Date: Wed Nov 4 10:54:39 2020 -0800 Fix bug in new assertion (#1506) Src/Particle/AMReX_ParticleContainerI.H commit db41d3d75449b162ebe4113656da49536149e165 Author: Max Katz Date: Wed Nov 4 09:23:41 2020 -0800 Add option for device sync at beginning and end of TinyProfiler region (#1505) ## Summary Now when setting `tiny_profiler.device_synchronize_on_region_end = 1` in the inputs file, we will synchronize before calling nvtxRangePop() and nvtxRangePush(), which means that TINY_PROFILE regions will include the full kernel time, rather than just the kernel launch time. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit 81fdb9a6254fee22daf3c32843c619e08d527bbc Author: Max Katz Date: Wed Nov 4 07:50:41 2020 -0800 Guard MFIter reduction macros against USE_GPU_PRAGMA (#1504) Src/Base/AMReX_BLFort.H commit 461ca5b42dfeb3ac24ab0aa344d3fc00129cc8b4 Author: Weiqun Zhang Date: Tue Nov 3 15:39:57 2020 -0800 GNU make: adjust multiple target rules (#1500) ## Summary Remove `/` from the object file name so that it works for multiple targets in different directories. This should not affect its current usage. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.rules commit 3ed4b614a91e8b49d3e6f9b434493cf8f1c9464a Author: Michele Rosso Date: Tue Nov 3 14:09:34 2020 -0800 CMake; bump up hypre minimi required version (#1503) Tools/CMake/AMReXThirdPartyLibraries.cmake commit f4c041f181dd1bb5743483e4b86f19cbb6e40fb2 Author: Andrew Myers Date: Tue Nov 3 13:54:38 2020 -0800 assert if aos.size() != soa.size() (#1502) Src/Particle/AMReX_ParticleContainerI.H commit b5a506997ce5db41bc46d6d64a238053fa2ea1ea Author: Weiqun Zhang Date: Tue Nov 3 13:53:23 2020 -0800 Fix warnings in AMReX_Machine.cpp (#1499) Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_Machine.H Src/Base/AMReX_Machine.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp commit e61266427dc11ffdfd2bad6577ac204e19ad5e88 Author: Andrew Myers Date: Tue Nov 3 13:52:40 2020 -0800 Use Long type for pindex to avoid potential overflow. (#1501) Src/Particle/AMReX_ParticleContainerI.H commit 7117a8f9006ad2d647c3e6412aee3ec3047d6def Author: Weiqun Zhang Date: Tue Nov 3 09:01:25 2020 -0800 Update GNU make for Cori GPU node (#1497) ## Summary Set CUDA_ARCH to 80 for A100 on Cori. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.nersc commit 24cacefc388e9811f52c1ebbceeb06cca90ba9b6 Author: Axel Huebl Date: Tue Nov 3 08:07:50 2020 -0800 Lin. Solvers (MLMG): Include Order (#1496) ## Summary Include own headers before stdlib headers to catch missing includes early. This avoid problems with less common compilers and environments. Carved out of #1485 ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit ee3076a8306269bf6d18ffee3f4d4944c660241d Author: Max Katz Date: Tue Nov 3 08:06:58 2020 -0800 GNUMake: Merge Ascent and Peak with Summit (#1493) ## Summary These are all systems with the same architecture so we can share the build setup. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/sites/Make.olcf commit 43f112fe4b9deda918f3d3961d2979e124d0645d Author: Weiqun Zhang Date: Mon Nov 2 13:00:42 2020 -0800 Update CHANGES for AMReX 20.11 (#1494) CHANGES commit db23e834e9cdc24884b689e0bdbd285ba6124562 Author: Shreyas Ananthan Date: Mon Nov 2 11:00:19 2020 -0700 HYPRE: Fix issues with hypre IJ interface (#1495) This commit fixes two issues with the hypre IJ interface - Fixes a memory leak in HypreIJIface as it was calling parse_inputs in constructor - Sets default number of iterations and tolerance for BoomerAMG when used as a preconditioner Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit f29a0c9d1b8e6356f8007a0bcb9cfc1000f2a0ac Author: Weiqun Zhang Date: Fri Oct 30 15:45:23 2020 -0700 Change default hypre interface to ij for non-EB (#1492) Src/LinearSolvers/MLMG/AMReX_MLMG.H commit 93ffe6391b8a18c749610cabe05c785b6325378d Author: Weiqun Zhang Date: Fri Oct 30 10:39:39 2020 -0700 Fix DEBUG build for DPC++ (#1491) This fixes a bug introduced in #1489. Src/Base/AMReX.cpp commit 82c7a83ff6dd43be71d5960ad422f4db88d929e7 Author: Weiqun Zhang Date: Fri Oct 30 08:45:24 2020 -0700 Changes to make hipcc --save-temps work. (#1489) ## Summary It does not like AMREX_GPU_DEVICE_EXTERNAL. Also changed is the device version of `Error`, `Abort` and `Warning`. If `NDEBUG` is defined, they do not do anything. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX.H Src/Base/AMReX.cpp Tools/GNUMake/Make.defs commit 69c496f8e8e65f7fe4e9cd5afd4e9e89fa4ad9ba Author: Weiqun Zhang Date: Thu Oct 29 12:32:24 2020 -0700 Fix GPU race condition for EB (#1487) A GPU race condition was introduced in #1451 trying to fix faces and cells next to covered cells that were converted from small cut cells. It is fixed by launching a separate kernel for each direction. Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.cpp commit 147149885e7ca02dcfdc88f1f4953bf26dc10dc8 Author: Axel Huebl Date: Thu Oct 29 11:30:41 2020 -0700 Ascent: Use Default Name for Mesh Ghost (#1488) ## Summary Use the default name for ghost cells in Ascent. The current name is non-default and causes issues at domain-boundaries during volume rendering (unless one overwrites the `ghost_field_name` keyword in the options). Thanks to @mclarsen and @cyrush for debugging and hinting this with me. ## Additional background Seen with WarpX. Pre-PR: ![replay_000400(9)](https://user-images.githubusercontent.com/1353258/97613195-9eed1c80-19d5-11eb-9934-cafeffc51458.png) Post-PR: ![replay_000400(10)](https://user-images.githubusercontent.com/1353258/97614230-eb852780-19d6-11eb-8dcd-4ca4d86d2e21.png) ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 32052a90a11da0dec54c353e0d6e29758e52b9a1 Author: Axel Huebl Date: Wed Oct 28 16:08:03 2020 -0700 CI: Check More Warnings (#1486) ## Summary Checks for more warnings that are likely to uncover nifty bugs: `-Wshadow -Woverloaded-virtual -Wunreachable-code` Also checks for superfluous `;` with `-Wextra-semi` (only a flag in Clang). ## Additional background Does not yet add `-Wall -Wextra -Wpedantic` because there are quite a few places where we need to add casts to avoid signed-unsigned-comparisons, etc. WarpX' CI covers currently: * GNU: `-Wall -Wextra -Wpedantic -Wshadow -Woverloaded-virtual -Wunreachable-code` * Clang: `-Wall -Wextra -Wpedantic -Wshadow -Woverloaded-virtual -Wextra-semi -Wunreachable-code` Even with super-builds, we current include AMReX with `-isystem` and build AMReX with default compiler flags; this enables us to suppress most AMReX build warnings besides the ones that drop-through from macros defined in headers, which atm. are all fixed for the code sections we use. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate .github/workflows/linux.yml .github/workflows/macos.yml Src/Particle/AMReX_NeighborParticles.H Tutorials/Basic/PrefixSum_MultiFab/main.cpp Tutorials/Particles/CellSortedParticles/cell_sorted_F.H Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/em_pic_F.H Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.cpp Tutorials/Particles/NeighborList/MDParticleContainer.cpp commit da1063cc5c16dcf519e74d630f898e7b23711c76 Author: Weiqun Zhang Date: Wed Oct 28 14:03:39 2020 -0700 HIP GNU Make: need to override COMP for regression tests (#1467) Tools/GNUMake/Make.defs commit ee020a50235341408297ff7018c536486eb30754 Author: Axel Huebl Date: Wed Oct 28 13:38:34 2020 -0700 CI: -Werror for host compilers (#1484) ## Summary Add more warning coverage in CI for CMake builds. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate .github/workflows/linux.yml .github/workflows/macos.yml commit 0186e2f103f800280d8edfc5dffcd9280f132f42 Author: Weiqun Zhang Date: Mon Oct 26 11:18:29 2020 -0700 Fix warning in FabSet for single precision (#1481) * Fix warning in FabSet for single precision * Constify constants Co-authored-by: Axel Huebl Src/Boundary/AMReX_FabSet.cpp commit 6f07fe4e84a4dd57a0e89c2d2cb84c9953fda85d Author: Weiqun Zhang Date: Fri Oct 23 16:23:17 2020 -0700 Workaround for Mac CI (#1482) .github/workflows/dependencies/dependencies_mac.sh commit 61ef93db305fccfe81404686d6878f1f805fee02 Author: Axel Huebl Date: Fri Oct 23 13:29:11 2020 -0700 CMake 3.18+: CUDA Arch Policy (OLD) (#1480) Keep the old policy for now to avoid setting the code generation flags twice. We should be able to transition this well, but I need to first find out how device LTO generation flags are handled here, if at all, and how to restore the default-detection of the local GPU architecture, e.g. on Summits head-nodes. CMakeLists.txt commit f4d73901d209caa80951b4842f0e46848104a98f Author: Ann Almgren Date: Fri Oct 23 10:13:49 2020 -0700 More fixes to documenting hypre (previous text got lost). (#1477) Docs/sphinx_documentation/source/LinearSolvers.rst commit f42ab5658751c51d6792d7dd7f1999c4c3310143 Author: Luca Fedeli Date: Thu Oct 22 17:23:11 2020 +0200 Add fill method to GpuArray (#1472) ## Summary I would like to propose to add a `fill` method to `GpuArray`. This PR implements just that. ## Additional background I am wondering if it would be better to replace `i < N` with `i < amrex::max(N,std::size_t{1})`, in order to deal with the case `N=0`. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [X] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Array.H commit 0cfd7d349279040c1d9b9283f970b333296594d9 Author: Ann Almgren Date: Thu Oct 22 08:21:53 2020 -0700 Update the docs to reflect the new hypre options (#1473) Docs/sphinx_documentation/source/LinearSolvers.rst commit 6071cf447613beba8753fb0a5409dbe4e299111a Author: Weiqun Zhang Date: Tue Oct 20 18:10:01 2020 -0700 Gpu::Atomic::AddNoRet (#1469) Add Gpu::Atomic::AddNoRet to use HIP's atomicAddNoRet for float, which is much faster than atomicAdd for float that is currently implemented with CAS. Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_TypeTraits.H Src/EB/AMReX_EB_utils.cpp Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_Particle_mod_K.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/ParticleMesh/main.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/em_pic_K.H commit a004707203dcbd3e28c5e637c623d569301c64cd Author: Andy Nonaka Date: Tue Oct 20 18:01:29 2020 -0700 Enforce singular solvable (#1471) ## Summary Option to make the code NOT enforce solvability for singular problems. This is accomplished by calling linop.setEnforceSingularSolvable(false) The default behavior is true, so no results will change in existing codes ## Additional background ## Checklist The proposed changes: - [x] add new capabilities to AMReX Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 8350a1641044aa222d922e85bd082be5734e2a6a Author: Michele Rosso Date: Tue Oct 20 17:15:07 2020 -0700 CMake: improve HDF5 support (#1468) * CMake: look only for the C component of HDF5 * CMake: use only C-related variables from FindHDF5 * CMake: add Tests/HDF5Benchmark to CTest * CMake: fix typo * Update Tools/CMake/AMReXThirdPartyLibraries.cmake * CMake: use hdf5 imported target if available Co-authored-by: Axel Huebl Tests/CMakeLists.txt Tests/HDF5Benchmark/CMakeLists.txt Tools/CMake/AMReXThirdPartyLibraries.cmake commit c0710a2bc4725aa8e7cab57690b27c73285436ae Author: Shreyas Ananthan Date: Sun Oct 18 14:16:56 2020 -0600 Use rocrand from HIP APT repository (#1466) .github/workflows/dependencies/dependencies_hip.sh commit 4f0052023f6c68ca2898ed99552e641a2e4dcc8f Author: Shreyas Ananthan Date: Sun Oct 18 14:16:03 2020 -0600 Address warnings in CUDA builds (#1465) Src/Base/AMReX.cpp Src/Base/AMReX_FBI.H Tools/Plotfile/fsnapshot.cpp commit aef02b9a4745005856a45417632b964a760dadb0 Author: Michele Rosso Date: Sun Oct 18 11:03:49 2020 -0700 CMake: FindPETSc must not overwrite CMAKE_Fortran_FLAGS (#1464) ## Summary FindPETSc used to set ```CMAKE_Fortran_FLAGS``` to the list of PETSc include directories. This caused problems with the compilation of Fortran files (see Issue #1463 ). This PR fixes this. ## Additional background ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/CMake/FindPETSc.cmake commit f64fb79d0f8a04a785e7c21f49dd81330ce55692 Author: Axel Huebl Date: Fri Oct 16 18:56:55 2020 -0700 CI: rocRAND v1.8.2 (#1462) Building the development branch is a bit unstable. .github/workflows/dependencies/dependencies_hip.sh commit 0cea78bf2f4f24568a1d71d20e8b2614fe19012c Author: Michele Rosso Date: Fri Oct 16 17:33:42 2020 -0700 CMake: some cleanup (#1460) * CMake: provide function to retrieve AMReX version * CMake: remove unused helper functions CMakeLists.txt Tools/CMake/AMReX_Utils.cmake commit 931cf5993b8ad2d5fda1a853ebabb36dbe2ae5c2 Author: Weiqun Zhang Date: Fri Oct 16 17:30:50 2020 -0700 DPCPP: Scan (#1461) ## Summary Need to use global address space when doing atomic inc on virtual block id. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Scan.H commit 31ca23418f488fe0b309ef3fc3e99c522ced4921 Author: Michele Rosso Date: Fri Oct 16 14:53:58 2020 -0700 [WIP] CMake: add support for dpc++ beta10 (#1459) * CMake: retrieve dpcpp version * CMake: modify dpc++ flags to account for beta10 version Tools/CMake/AMReXSYCL.cmake commit d036a6df1c6f8f7bb458c7e5253a91f4b6f96b83 Author: Max Katz Date: Fri Oct 16 14:21:28 2020 -0700 Fix potential out of bounds access in #1441 (#1458) ## Summary The user may have given us a vector that has too many values, so we just ignore the remainder. ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_ErrorList.H commit 18b293dad9199deb329efecddd56ccce709371f5 Author: Weiqun Zhang Date: Fri Oct 16 10:12:19 2020 -0700 DPCPP beta10 (#1456) ## Summary * Update for DPCPP beta10 * Fix some warnings * Remove some device versions of Random() for HIP and DPC++ ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Extension.H Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_Random.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_RandomEngine.H Src/Base/AMReX_Scan.H Src/Particle/AMReX_BinIterator.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H Tools/GNUMake/comps/dpcpp.mak commit 15643f4b0ab28dc6b3277c5c44c18ac1f80e1345 Author: Michele Rosso Date: Thu Oct 15 16:21:13 2020 -0700 CMake: prevent in-source builds (#1453) * CMake: prevent in-source builds * CMake: make error message more detailed CMakeLists.txt commit c53b1bbe88bed6b731bae4e11fd9ed0883a6b814 Author: Andy Nonaka Date: Thu Oct 15 14:29:41 2020 -0700 change char * to char const* so SWFFT code compiles with USE_CUDA=TRUE (#1454) Src/Extern/SWFFT/distribution.c commit 48b0a20210b9825c1c403e1e0b8a741ba913dffa Author: Shreyas Ananthan Date: Wed Oct 14 15:00:12 2020 -0600 Make HypreABecLap3::getSolution public for CUDA builds (#1452) Src/Extern/HYPRE/AMReX_HypreABecLap3.H commit 5843b838c1a86ea7a7978d0b6edd9f5e0c03d24a Author: Ann Almgren Date: Wed Oct 14 09:10:35 2020 -0700 Fix when small_volfrac is used -- we need to adjust the data in cut c… (#1451) Fix when small_volfrac is used -- we need to adjust the data in cut cell neighbors as well as regular neighbors This addresses issue #1450 ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.cpp commit 79f3a94fd50b6e9bcac6f99d0b03ffb47e9a5a87 Author: Andrew Myers Date: Wed Oct 14 00:39:23 2020 -0700 Allow users to set a Geometry object for ParticleContainer independently of the AmrCore / AmrLevel object. (#1446) Src/AmrCore/AMReX_AmrParGDB.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_Particles.H commit ff17a9295b54a36fa78c4f3c2350c3b3b5dd1338 Author: Houjun Date: Tue Oct 13 12:14:50 2020 -0700 HDF5 plotfile write bug fix and performance improvement (#1448) * Add HDF5 optimizations * Fix an issue that causes incorrect HDF5 data write with Tutorials/Basic/HeatEquation_EX1_C example code Co-authored-by: Houjun Tang Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_ParticleHDF5.H Tests/HDF5Benchmark/GNUmakefile commit 6649d227425fdc8ecbed301c4383c8dc74800323 Author: Michele Rosso Date: Tue Oct 13 11:52:09 2020 -0700 CMake: add HIP support (#1316) * CMake: first basic attempt at HIP support * Update Tools/CMake/AMReXParallelBackends.cmake Co-authored-by: Axel Huebl * CI: Add HIP-Clang * fix typo: GCC 9.3 on focal * Fix sources: set two lists * fix target properties: append two lists * CMake: some changes to debug issue with generated files * CI: only run HIP check for now * CMake: fix typo * CMake: let's try this * CMake: other small modifications * CMake: attemp #1 * CMake: attemp #2 * CI: build just the bare minimum for now * CMake: attempt #3 * CMake: missing defines for HIP builds * CMake: defines were still missing :-P * CMake: AMREX_HIP_PLATFORM is actually the HIP_COMPILER * CMake: let's see if this works * Install rocRAND * CMake: re-factor and update defines for HIP compilation * CMake: list against hiprand too * CMake: find and link to hiprand * CMake: add -DNDEBUG to any type of build when HIP is on * CMake: we must manually add rocrand too * CMake: let's see if it works with Fortran enabled * Revert "CMake: let's see if it works with Fortran enabled" This reverts commit 065071539fd166f3bee994c2621d86c773180e3d. * CMake: try this * CMake: this should work too * Revert "Revert "CMake: let's see if it works with Fortran enabled"" This reverts commit 6a92adcc4ddfb06b4130f10d29cc49b298a66e73. * CMake: no fortran for now * CI: turn on linear solvers in HIP check * CI: turn on particles in HIP checks * Revert "CI: turn on linear solvers in HIP check" This reverts commit c723560a7b401266d6259af340130b1065e6be59. The reason is that Linear Solvers tutorials are not HIP-aware yet. * CMake: hipify tutorials * CMake: temporary fix * Revert "CMake: temporary fix" This reverts commit bd6724ed73f7afc82a4dc60c60d4390c6a83808d. * CMake: fix HIP compilation for Tutorials * CMake: some cleanup * CMake: add HIP options to pass in architecture and extra flags * CMake: oops * CMake: try to enable linear solvers * CMake: update ROCm version number in CI check name * CMake: enable Fortran * CMake: disable tutorials for the time being * CMake: commit custom FindHIP.cmake to debug the issue * CMake: trying this * CMake: now it should use the local FindHIP.cmake * CMake: now custom FindHIP.cmake should be able to find helper files * CMake: let's see if we really need this * CMake: let's try this * CMake: temporary fix * Revert "CMake: disable tutorials for the time being" This reverts commit 885b9113b72f54bca9eaf74f4e5b6b90b17032ff. * CMake: Tutorials/Particles/CellSortedParticles do not work with HIP * CMake: remove local FindHIP.cmake. * CMake: HIP arch flags are now PUBLIC and inheritable * CMake: no use for setup_target_for_hip_compilation anymore * CMake: remove unnecessary options for HIP * CMake: check that HIP_COMPILER is the same as CMAKE_CXX_COMPILER * CMake: better this * CI: uncomment all checks * CMake: fix visibility of arch flags for HIP * Doc: add subsection on HIP+CMake * Update Docs/sphinx_documentation/source/GPU.rst Co-authored-by: Shreyas Ananthan * CMake: fix incorrect configuration option passed to cmake * CI: target 'tutorials' no longer exists Co-authored-by: Axel Huebl Co-authored-by: Shreyas Ananthan .github/workflows/dependencies/dependencies_hip.sh .github/workflows/linux.yml Docs/sphinx_documentation/source/GPU.rst Src/CMakeLists.txt Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/CMake/AMReX_Defines.cmake Tools/CMake/AMReX_Options.cmake Tutorials/CMakeLists.txt Tutorials/Particles/CellSortedParticles/CMakeLists.txt commit 6dc6ce8f55a5ef19f349c267b7bb9fc4bb616b07 Author: Candace Gilet Date: Tue Oct 13 09:48:50 2020 -0400 volume multifab must be defined before calling GetVolume(volume) (#1449) Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 5e6d555d07d3862b5b88b6c38559a59918541194 Author: Michele Rosso Date: Mon Oct 12 17:38:22 2020 -0700 NodalProjector: use volume-weighted average down (#1444) Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 95ddfdb2c1434b1705d84174047f06cd3004174d Author: Weiqun Zhang Date: Wed Oct 7 19:02:08 2020 -0700 Fix EB interpolation from cell centers to faces (#1443) ##Summary There were race conditions that made it fail on AMD GPUs. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H commit 2c7b5004fb98d92bf185ecc73b6429dfd7ed8b6f Author: Weiqun Zhang Date: Wed Oct 7 15:16:51 2020 -0700 Fix bug in hypre overset solver (#1442) ## Summary ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/PETSc/AMReX_PETSc.cpp commit 3972c6d582f024c4a4d108ff02e66d117e6db1e6 Author: Max Katz Date: Wed Oct 7 15:33:35 2020 -0400 Allow tagging value to vary by level (#1441) ## Summary The new error tagging scheme from #1166 is modified to allow the threshold value to vary by level. Using the example there, if we did: ``` amr.refinement_indicators = flame_tracer lo_temp amr.refine.flame_tracer.max_level = 3 amr.refine.flame_tracer.value_greater = 1.e-6 1.e-5 amr.refine.flame_tracer.field_name = Y(H) amr.refine.lo_temp.max_level = 2 amr.refine.lo_temp.value_less = 1000. amr.refine.lo_temp.field_name = temp ``` Then level 0 would be tagged for refinement for Y(H) >= 1.e-6, while level 1 and all higher levels would be tagged for refinement for Y(H) >= 1.e-5. (If a value for every level is not provided, we assume the last value holds for all remaining levels, similar to how we treat quantities like amr.n_error_buf.) ## Additional background This will help for cases where a field varies substantially on the domain and you want to have better control over refinement so that you don't have too many zones. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp commit 92399587f20875a98655ccbd06a083965cfc9c5c Author: Weiqun Zhang Date: Tue Oct 6 13:30:52 2020 -0700 Hypre with overset (#1439) * Hypre with overset Add overset mask support for hypre. * Keep bottom_verbose restricted to amrex side. Hypre side's verbosity will be controlled by parmparse parameters. Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_Hypre.cpp Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Tests/LinearSolvers/CellOverset/MyTest.H Tests/LinearSolvers/CellOverset/MyTest.cpp commit 6a20c352cf54ed9a1393676b1bf19c57941db2af Author: Axel Huebl Date: Mon Oct 5 16:17:43 2020 -0700 GNUmake: -Titan (OLCF) (#1440) R.I.P. Titan. Btw, only found because a colleague of mine could not compile WarpX. Their hostname is called -titan. Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit 9992d844099c3df81d9499519ffeb7f93a201dc6 Author: Shreyas Ananthan Date: Sun Oct 4 18:19:24 2020 -0600 Hypre IJ interface: Enable access to additional solvers and preconditioners available in Hypre (#1437) This PR extends the AMReX _hypre_ IJ interface to allow access to additional solvers and preconditioners available in _hypre_. - Refactored `HypreABecLap3` and `HypreNodeLap` (IJ matrix/vector ParCSR interfaces) and created a new IJ interface class that provides a unified way to access Hypre ParCSR solvers. - In addition to BoomerAMG, adds support for 7 other [ParCSR solvers](https://hypre.readthedocs.io/en/latest/ch-solvers.html) (GMRES (4 variants), PCG, BiCGSTAB, and Hybrid). Also adds support for BoomerAMG and Euclid as preconditioners for solvers that can use a preconditioner. - Adds support to parse user options to configure hypre library via AMReX ParmParse interface and input file, using a custom namespace (default is `hypre`) - Adds `MLMG::setHypreOptionsNamespace` that allow applications to customize the namespace for different types of linear solvers (e.g., `mac_proj.hypre.