commit c8cdfa61f443ecde21debc3ad52665bf48eded8b Author: Andrew Myers Date: Sat Dec 19 11:32:26 2020 -0800 Revert tiling change in #1655 (#1664) Src/Particle/AMReX_ParticleContainerI.H commit 875ebff912b5d3f4edc0e6ad7a983a56374a9e69 Author: Axel Huebl Date: Fri Dec 18 20:36:28 2020 -0800 CI: CUDA 9.1 & 11.0 (#1653) Add a recent CUDA 11 release for cutting-edge coverage, too. .github/workflows/dependencies/dependencies_nvcc11.sh .github/workflows/dependencies/dependencies_nvcc9.sh .github/workflows/linux.yml commit 25d06e82804e3e1b4f3f1b0cea4e9979ed37373c Author: Axel Huebl Date: Fri Dec 18 15:29:58 2020 -0800 CI: Lib + Plotfiles Install (#1662) Test the new install of plotfile tools works in CI. .github/workflows/linux.yml commit cc61f30f4266407225bd8c7354565931b275a07f Author: mic84 Date: Fri Dec 18 13:56:37 2020 -0800 CMake: streamline install of plotfile tools (#1661) CMakeLists.txt Docs/sphinx_documentation/source/BuildingAMReX.rst Src/CMakeLists.txt Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXInstallHelpers.cmake Tools/Plotfile/CMakeLists.txt commit 1c291c0b6bb52be354b77b2abb646216dcb62ff7 Author: kngott <30483578+kngott@users.noreply.github.com> Date: Fri Dec 18 10:39:38 2020 -0800 Add new clean commands to docs. (#1660) Docs/sphinx_documentation/source/BuildingAMReX.rst commit 3b897e1b7e04222ae718c00010085908f7693b41 Author: mic84 Date: Thu Dec 17 17:14:36 2020 -0800 Fix how fextract handle the tolerance options (#1659) Tools/Plotfile/fextract.cpp commit de1701ed9d6cbdf61a8c4c3015c1cf5a9b00f7f5 Author: jmsexton03 Date: Thu Dec 17 17:00:36 2020 -0500 AssignDensity and AssignCellDensitySingleLevel copy -> ParallelCopy (#1633) Src/AmrCore/AMReX_AmrParticles.H Src/Particle/AMReX_ParticleContainerI.H commit 07a21fd454317b7b434240d548765c4c4ae4464a Author: Andrew Myers Date: Thu Dec 17 13:29:53 2020 -0800 Have AssignCellDensitySingleLevel respect gpu launch guards. (#1655) Src/Particle/AMReX_ParticleContainerI.H commit e21901db47b85704e29b3800fcad7318b9544e76 Author: Weiqun Zhang Date: Thu Dec 17 12:27:08 2020 -0800 Clean up GPU pragma and CUDA Fortran (#1656) Clean up the remaining GPU pragma and CUDA Fortran. This also fixes a bug in #1650 for codes using the Fortran 77 version of filcc. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/GPU_Chapter.rst Src/Base/AMReX_BCUtil.cpp Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_PhysBCFunct.H Src/Base/AMReX_PhysBCFunct.cpp Src/Base/AMReX_filcc_mod.F90 Src/Base/AMReX_fort_mod.F90 Src/Base/Make.package Tools/GNUMake/Make.defs Tools/GNUMake/comps/hpcsdk.mak Tools/GNUMake/comps/ibm.mak Tools/GNUMake/comps/pgi.mak Tutorials/GPU/Launch/Readme.md commit 63db5cbe23e2ddc419e6895ec69610388519ca9d Author: mic84 Date: Thu Dec 17 11:14:35 2020 -0800 Add features to fextract (#1657) Add the following features to `fextract`: - user option to set the number of decimals in the output of reals - user option to set a cut-off tolerance: anything smaller than the tolerance gets printed as zero - user option to disable the job summary report at the end of the output The job summary report is disabled by default in contrast with the current behaviour. Tools/Plotfile/fextract.cpp commit 1f8891e49f23397b5633b34dae4437619d3b46a8 Author: Andrew Myers Date: Thu Dec 17 10:40:55 2020 -0800 Fix parallel add bug when both USE_GPU and USE_OMP are defined. (#1654) We cannot assume copies are thread safe any time AMREX_USE_GPU is defined, whether or not USE_OMP is also on. Note that this only would have been an issue when compiling with OMP but running on only one thread. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX Src/Base/AMReX_FabArrayBase.cpp commit e63397244cd19a67d1e76fdedce73667d41e8f2c Author: Andy Nonaka Date: Thu Dec 17 10:38:23 2020 -0800 Write a plotfile to ASCII (#1638) This utility reads in a single-level plotfile, copies it to a MultiFab with a single box, then writes out all the data in 'i j k comp ' format to the screen The user can modify this cpp file to write out on certain components, coordinates, row/column formatting, etc. Usage: ./WritePlotfileToASCII2d.gnu.MPI.ex infile=inputFileName Docs/sphinx_documentation/source/Post_Processing.rst Docs/sphinx_documentation/source/Post_Processing_Chapter.rst Docs/sphinx_documentation/source/index.rst Tools/Postprocessing/C_Src/GNUmakefile Tools/Postprocessing/C_Src/WritePlotfileToASCII.cpp commit 7b24139e3214b417b54f8d6238131dc484bc6e66 Author: mic84 Date: Wed Dec 16 12:31:01 2020 -0800 CMake: AMRDATA needs Fortran (#1645) Tools/CMake/AMReXOptions.cmake commit 75373313e44803c3732096a2dee413b52ed2d62e Author: mic84 Date: Wed Dec 16 12:30:36 2020 -0800 CMake: fix few quirks with GPU backends variables (#1561) Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake commit b851ce78a2f3f26d53eb101aa0a0266284f44bfb Author: kngott <30483578+kngott@users.noreply.github.com> Date: Wed Dec 16 12:23:39 2020 -0800 FaceLinear: Avoid accessing face outside of coarse box. (#1651) ## Summary Fix an index out-of-bound trigger in FaceLinear interpolater. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H commit 08e27d80f3cb0151566b05f964fb4fc30296f56c Author: Axel Huebl Date: Wed Dec 16 12:09:48 2020 -0800 CI: fix macOS (#1652) * CI: fix macOS brew did again some updates and things fail. let's revert our current work-around. * Brew: Add gfortran not pulled automatically anymore .github/workflows/dependencies/dependencies_mac.sh commit d7b6b97354c02f2e37761f8afae0fca36e7a4550 Author: Candace Gilet Date: Wed Dec 16 14:13:52 2020 -0500 Fix NodalProjector::getGradPhi for multilevel case (#1642) ## Summary Two bug fixes for multilevel in NodalProjector: 1. Ensure NodalProjector::getGradPhi() returns the correct result. Previously could return \sum{sigma grad phi} / (2**dim sigma_coarse). 2. Average down the complete projected velocity at the end. For alpha!=1, this ensures the projected velocity is averaged down correctly. This could change the results of multilevel regression tests. If the application uses the affected outputs from NodalProjector as is, then changes are expected. If the application subsequently averages down the outputs, then changes greater than roundoff level are not expected. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 70b7ab9999522a51666027ae3f7f3dc527541290 Author: Max Katz Date: Wed Dec 16 09:10:42 2020 -0800 Remove GPU pragma support from GNU make build system (#1650) This is no longer used. Tools/F_scripts/gpu_fortran.py Tools/F_scripts/write_cuda_headers.py Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/sites/Make.llnl commit affcccf3c900d1d29349aaeb0c5f21c9dd63e637 Author: Max Katz Date: Wed Dec 16 09:08:45 2020 -0800 Remove GPU pragma macros from ArrayLim (#1649) These are no longer used. Src/Base/AMReX_ArrayLim.H commit 78eae95a21df8fa1d0e2a7c86b0437dd68e8aa7a Author: Max Katz Date: Wed Dec 16 09:06:31 2020 -0800 Remove custom MFIter reduction code (#1648) This is no longer in use by any of the astro codes. Src/Base/AMReX_BLFort.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit 2f9277a64058c91a5d0153b77bd2476bd1f85690 Author: Axel Huebl Date: Tue Dec 15 17:06:11 2020 -0800 CMake: Fix Typos in SYCL Warnings (#1647) Just a typo and a missing space in a warning. Tools/CMake/AMReXOptions.cmake commit 006569febeae1329fc0a3e0c2fb17c77e42539c7 Author: Weiqun Zhang Date: Tue Dec 15 16:30:54 2020 -0800 DPCPP MPI wrapper (#1646) ## Summary Replace mpiicpx with mpiicpc because mpiicpc is shipped by Intel, whereas mpiicpx is a script that is only available on JLSE testbeds. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Co-authored-by: Michele Rosso Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake Tools/GNUMake/Make.defs Tutorials/GPU/CNS/Source/CNS_bcfill.cpp Tutorials/GPU/CNS/Source/hydro/CNS_hydro_K.H commit cd2e06e47cb586d0b8e2ef3eb0c534548634ad97 Author: Weiqun Zhang Date: Tue Dec 15 10:44:19 2020 -0800 Remove support for old versions of DPCPP (#1644) Src/Base/AMReX_Extension.H Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_RandomEngine.H Src/Base/AMReX_Scan.H commit cd31efb8aee323dafddb6040c0613667cfb5382c Author: Weiqun Zhang Date: Tue Dec 15 09:14:51 2020 -0800 HIP and DPCPP support in configure (#1640) Tools/GNUMake/comps/hip.mak Tools/libamrex/configure.py Tools/libamrex/mkconfig.py commit 66e52d44979d14859e007d769fb1b550ec591cc7 Author: Weiqun Zhang Date: Mon Dec 14 15:45:16 2020 -0800 Fused FillBoundary (#1570) ## Summary Add a new FillBoundary function for a Vector of FabArrays. There are two implementations in this PR. The one being used simply calls `FillBoundary_nowait` and `FillBoundary_finish`. The other version aggregates the MPI messages from multiple FabArrays into one. Tests on summit shows the latter is slower. But I don't want to lose the code, so it is kept for future investigation. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Array4.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_Box.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp commit 6a212600436d83fcfb656ce72d6c810c1b7c1cdd Author: kngott <30483578+kngott@users.noreply.github.com> Date: Mon Dec 14 13:51:13 2020 -0800 Divergence Free Stencil and API (#1483) API for multi-dimensional, face-based, coarse-to-fine interpolation (InterpFromCoarse and FillPatchTwoLevels). Includes a variation of Vanella et. al. stencil (doi:10.1016/j.jcp.2010.05.003, section 3.2), but instead of interpolating from a far coarse cell to solve interior closure problem (highly biased result dependent on chosen coarse cell), this uses least squares with initial guess equal to the average of fine face values across the cell. (Mathematica notebooks of solution are available). Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_1d.F90 Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_INTERP_1D.F90 Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_Interpolater.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_GpuLaunch.H Tests/DivFreePatch/GNUmakefile Tests/DivFreePatch/Make.package Tests/DivFreePatch/inputs Tests/DivFreePatch/main.cpp commit 78d6f64fadc606d65e30d4da5f218e941bc7ea59 Author: Weiqun Zhang Date: Mon Dec 14 10:19:51 2020 -0800 Constant coefficient MacProjector (#1636) Optimization of constant coefficient MAC projection by using constant coefficient Poisson solver. Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit 4794c1d1c2e1fd83bc5ac6be8ff71c4893861128 Author: Weiqun Zhang Date: Mon Dec 14 08:39:07 2020 -0800 Option to allow multiple active MFIters (#1631) Add MFIter::allowMultipleMFIters(bool) so that users can disable assertion on the number of active MFIters. This is needed by FLASH because multiple MFIters (one for each level) are built in its iterator. Docs/sphinx_documentation/source/Basics.rst Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp Src/F_Interfaces/Base/AMReX_multifab_fi.cpp Src/F_Interfaces/Base/AMReX_multifab_mod.F90 commit 8e73aa00126634cd83d4ca9c46fbac20da924e4e Author: drangara <69211175+drangara@users.noreply.github.com> Date: Mon Dec 14 11:37:42 2020 -0500 Convert AMReX_eb_to_pvd.F90 to cpp (#1627) ## Summary Convert the fortran file that writes the EB surface for paraview visualization to cpp ## Additional background Tested with the visualization files produced by `mfix/benchamrk/07-hopper` and `mfix/tutorials/clr/prototype`. Paraview visualization seems to look "good". When comparing the actual data from the files I do see some differences b/w the fortran and cpp version, but I believe the differences can be attributed to the precision of datatypes used in the fortran vs cpp code. These changes should not affect the solution results, just affects visualization. Src/EB/AMReX_EBToPVD.H Src/EB/AMReX_EBToPVD.cpp Src/EB/AMReX_WriteEBSurface.cpp Src/EB/AMReX_WriteEB_F.H Src/EB/AMReX_eb_to_pvd.F90 Src/EB/CMakeLists.txt Src/EB/Make.package commit c0a931f0ca46d62210e5b8f2e92c6952cfd60685 Author: Weiqun Zhang Date: Sat Dec 12 14:42:57 2020 -0800 Fix EB bug in #1629 (#1635) For EB nodal solver with constant sigma, the domain ghost cells of sigma should be set to zero. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit f6f883bfedf2ecf3900c7f79f1f4df3ec92cca76 Author: Weiqun Zhang Date: Sat Dec 12 10:45:05 2020 -0800 Nodal solver with constant coefficient (#1629) * Add specialization for constant coefficient case in the nodal solver. * Update NodalProjector. * Fix some warnings in 1D. Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 85a5f3be91eb9d58845f96c2907af8ee65661f87 Author: Andrew Myers Date: Sat Dec 12 09:44:58 2020 -0800 fix bug in pshifting neighbors introduced in #1590 (#1632) Src/Particle/AMReX_ParticleCommunication.H commit 52d32c6a07cd14498129ec3f03ae69a06a27573e Author: asalmgren Date: Fri Dec 11 21:10:38 2020 -0800 When we create a new level of refinement, we need to set the "level_s… (#1630) …teps" of that new level appropriately. ## Summary ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Amr/AMReX_Amr.cpp commit 389a0cc49f018ea86df0067bc8182336426fe414 Author: Andrew Myers Date: Fri Dec 11 16:24:59 2020 -0800 Expose the index of the neighboring particle to users of the iterator. (#1628) The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Particle/AMReX_NeighborList.H commit 73509be24826d16680289e5555cdb8db1e2217e5 Author: Weiqun Zhang Date: Thu Dec 10 10:02:10 2020 -0800 Documentation on avoiding continue and return inside GPU launch macros. (#1617) Docs/sphinx_documentation/source/GPU.rst commit 728db75fa70fcf10fb7fcb53210ba988ce24f1e4 Author: Weiqun Zhang Date: Thu Dec 10 10:01:16 2020 -0800 Implement overset mask for MLPoisson too (#1604) Src/LinearSolvers/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.cpp Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H Src/LinearSolvers/MLMG/Make.package commit 8db1187e6faa4d6628d47ddb28dcc23200516451 Author: Axel Huebl Date: Thu Dec 10 09:47:03 2020 -0800 Warning in SP: large ICC float constant (#1626) Fix an ICC warning for a too-large float constant when compiling for single precision in WarpX. Src/Amr/AMReX_Amr.cpp commit cd88adc2e64456116fcaf0dcce4669921bdb2296 Author: Weiqun Zhang Date: Wed Dec 9 18:57:46 2020 -0800 Fuse nodal jacobi smoother for GPU (#1625) ## Summary Fuse nodal jacobi smoother kernels by switch the order of the sweep and MFIter loops. For Amr-Wind's ABL test with `amr.n_cell = 256 256 64` and `amr.max_grid_size = 32` on one V100, the nodal project is about 2x faster. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit b846969f308e0402e188b95756b478261a31a8d2 Author: Matt Larsen Date: Wed Dec 9 16:26:11 2020 -0800 don't use geom.size() as the number of levels (#1624) ## Summary The conduit adapter was using the `geom.size()` to determine the number of levels, instead of believing `n_levels`. There was a case where the max levels was greater than the actual refinement level. In this case, geoms was the size of the max_levels in the input deck, but mfs was the size of the active levels. resolves #1623 The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 923752e81d08a87d8edd7aaf01e92359125e4f86 Author: OscarAntepara <52221614+OscarAntepara@users.noreply.github.com> Date: Wed Dec 9 08:15:17 2020 -0800 Adding eb slopes limiter (#1594) * Adding eb slope limiters * Adding comments and making 3d adjustments * Fixing face centroids computation in the z-direction and correcting the comments Src/EB/AMReX_EB_slopes_K.H commit ffa5387dd6d873987386cd9ebbea3bab9592c9fd Author: Andrew Myers Date: Tue Dec 8 13:36:03 2020 -0800 Support for SoA data for neighbor particles. (#1590) Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_Particles.H Tests/Particles/NeighborParticles/MDParticleContainer.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/NeighborParticles/main.cpp Tests/Particles/Redistribute/main.cpp commit a28b02f1d194a0bce76cf784c6dc98cad1339015 Author: Axel Huebl Date: Tue Dec 8 11:13:42 2020 -0800 CMake: Verify DPC++ Compiler ID and at best warn (#1619) * CMake: less stringed MPI+DPCPP Check `find_package(MPI)` is capable to find Intel's MPI with a dpcpp CXX compiler. I would relax this check. Also, `CMAKE_CXX_COMPILER` can be an absolute path, e.g. `-DCMAKE_CXX_COMPILER=$(which dpcpp)`, which is not covered by the original syntax. * DPCPP Compiler Check: Only Warn on ID * Update recommendation * keep leading \n (consistency) Tools/CMake/AMReXOptions.cmake commit e0896cbd9e0432f642d484675e2a4f090ce93e33 Author: Houjun Date: Tue Dec 8 10:53:32 2020 -0800 Fix a bug that causes errors when writing HDF5 plotfile and particle (#1621) * Add HDF5 optimizations * Fix an issue that causes incorrect HDF5 data write with Tutorials/Basic/HeatEquation_EX1_C example code * Minor HDF5 tuning parameter change Co-authored-by: Houjun Tang Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_ParticleHDF5.H commit 82ca8fa508cad15c3f63c6233c8b028da5b4e347 Author: Michael Zingale Date: Tue Dec 8 12:34:11 2020 -0500 the A64fx Cray 10.0.1 compilers need the old-style options (#1620) ## Summary The default cray compilers on the Ookami A64fx machine use the old-style options for cray compilers. This logic allows us to build there. ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/comps/cray.mak commit 91a48a24777dabe03b1ff32bfce05536ce806200 Author: Weiqun Zhang Date: Mon Dec 7 20:00:29 2020 -0800 Fortran Inteface: Porting more MultiFab functions (#1609) Port IntVect ghost cell version of MultiFab::Add, Subtruct, Multiply, Divide, Saxpy, and LinComb to Fortran Interface. Src/F_Interfaces/Base/AMReX_multifab_fi.cpp Src/F_Interfaces/Base/AMReX_multifab_mod.F90 commit 975135ebdec9e0d96f9225a0196fade8ccbd2196 Author: Andrew Myers Date: Mon Dec 7 14:46:19 2020 -0800 [CI] turn off CLEAN option to keep existing files in the deploy target. (#1618) .github/workflows/docs.yml commit 8d45dd95dde0b6cb579c41b13ed3b0b69add297b Author: Axel Huebl Date: Mon Dec 7 13:40:33 2020 -0800 Conduit: Fix unused variables (#1615) ## Summary Removes four (4) unused variable warnings. cc @cyrush @mclarsen ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 774ec481be4fdf28bd80f5a381af9b1c97d5b007 Author: Axel Huebl Date: Mon Dec 7 13:32:59 2020 -0800 DPC++: Re-Enable with Work-around. (#1613) My former colleagues shared a work-around that we can use to overwrite the compiler identification: https://gitlab.kitware.com/cmake/cmake/-/issues/21551#note_869580 This regression will also be fixed in CMake 3.19.2 .github/workflows/linux.yml commit ef6ebdbe62366e78d542fa13739e6d0557a3cbf4 Author: Andrew Myers Date: Mon Dec 7 11:48:19 2020 -0800 deploy to docs to seperate 'build' directory. (#1614) .github/workflows/docs.yml commit 2bbaf1bd81b0b8fee3dbd1bc95f2fc5c22009ac3 Author: Andrew Myers Date: Mon Dec 7 10:55:56 2020 -0800 fix and simplify script that builds the documentation (#1612) build_docs.sh commit 6fb23a385bf2fa0bc2e3b64a24e4940b4c0ea16d Author: Andrew Myers Date: Mon Dec 7 10:32:11 2020 -0800 need to specify the target directory on the website repository (#1611) .github/workflows/docs.yml commit 0a402eef6c0f7bdcb609e786d3140a991e99b7b3 Author: Andrew Myers Date: Mon Dec 7 08:59:32 2020 -0800 Migrate building and deploying the documentation to Github CI. (#1603) .github/workflows/dependencies/documentation.sh .github/workflows/docs.yml .travis.yml Docs/sphinx_tutorials/Makefile build_and_deploy.sh build_docs.sh deploy_key.enc commit 81a56bd628b92745c2baef58d8e7b044306cc34c Author: Weiqun Zhang Date: Sun Dec 6 18:46:48 2020 -0800 Add check for plotfile tools (#1610) .github/workflows/linux.yml Tools/Plotfile/fboxinfo.cpp Tools/Plotfile/fextract.cpp Tools/Plotfile/fextrema.cpp commit 65615e97ef62a2cac302ecbe30482d35f0806542 Author: Weiqun Zhang Date: Sun Dec 6 16:05:16 2020 -0800 Fix error in #1599 (#1608) The proposed change fixes an error in #1599. Tools/Plotfile/fsnapshot.cpp commit 88236accbcc263cbf3d4b103189bc7b522e06dfd Author: Andrew Myers Date: Sun Dec 6 14:33:50 2020 -0800 Remove alternative operators from AMReX (#1599) Operators like `and`, `or`, and `not` are allowed in the C++ standard; however, they cause problems on Windows and with our clang Windows CI. This PR removes them from AMReX in favor of `&&`, `||`, and `!`, and also configures the CI to throw a compilation error if they are re-introduced into the codebase. Application codes are still free to use these if they want. Co-authored-by: Weiqun Zhang .github/workflows/linux.yml .github/workflows/macos.yml Src/AmrCore/AMReX_AmrCore.cpp Src/AmrCore/AMReX_AmrParGDB.H Src/AmrCore/AMReX_Cluster.cpp Src/AmrCore/AMReX_FillPatchUtil_I.H Src/Base/AMReX_Array4.H Src/Base/AMReX_AsyncOut.cpp Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BlockMutex.cpp Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxArray.cpp Src/Base/AMReX_CuptiTrace.cpp Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FArrayBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_FilCC_C.cpp Src/Base/AMReX_FileSystem.cpp Src/Base/AMReX_Geometry.cpp Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_IArrayBox.cpp Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_PCI.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Base/AMReX_PlotFileDataImpl.cpp Src/Base/AMReX_Reduce.H Src/Base/AMReX_Slopes_K.H Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Vector.H Src/Base/AMReX_VisMF.cpp Src/Boundary/AMReX_InterpBndryData_2D_K.H Src/Boundary/AMReX_InterpBndryData_3D_K.H Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EBAmrUtil.cpp Src/EB/AMReX_EBFluxRegister_2D_C.H Src/EB/AMReX_EBFluxRegister_3D_C.H Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EB_slopes_K.H Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/AMReX_EB_utils.cpp Src/EB/AMReX_WriteEBSurface.cpp Src/EB/AMReX_algoim_K.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_NeighborParticlesI.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleCommunication.cpp Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_WriteBinaryParticleData.H Tests/LinearSolvers/CellOverset/MyTest.cpp Tests/LinearSolvers/EBTensor/MyTest.cpp Tests/LinearSolvers/NodalOverset/MyTest.cpp Tests/LinearSolvers/TensorOverset/MyTest.cpp Tools/Plotfile/fcompare.cpp Tools/Plotfile/fsnapshot.cpp Tools/Postprocessing/C_Src/particle_compare.cpp Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.cpp Tutorials/LinearSolvers/ABecLaplacian_C/MyTest.cpp Tutorials/Particles/CellSortedParticles/CellSortedPC.cpp Tutorials/Particles/NeighborList/MDParticleContainer.cpp commit ad7974de5b3e68fe2f8fa7c114ffaa460ad166a2 Author: hsitaram Date: Sun Dec 6 14:08:33 2020 -0700 Eb from stl (#1592) ## Summary This is a work-in-progress PR that adds a preliminary set of tools and helper functions for using triangulated STL files in amrex based applications. Coupling with AMReX's EB framework will be added in later pull requests. This PR is done early so as to provide a software framework for incremental changes. As of now, I have added one helper function that provides a nodal multifab with a marker value indicating inside or outside the STL geometry. I guess it can be used for particle-wall collisions. I have also added a test case in the EB folder that generates a nodal multifab with the blanked values for an airfoil. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Co-authored-by: Hariswaran Sitaraman Co-authored-by: Hariswaran Sitaraman Src/EB/AMReX_EB_STL_utils.H Src/EB/AMReX_EB_STL_utils.cpp Src/EB/AMReX_EB_triGeomOps_K.H Src/EB/CMakeLists.txt Src/EB/Make.package Tutorials/EB/STLtest/GNUmakefile Tutorials/EB/STLtest/Make.package Tutorials/EB/STLtest/airfoil.stl Tutorials/EB/STLtest/inputs Tutorials/EB/STLtest/main.cpp commit 8dc48ad8385cd395310567bf72a36470236e0ce2 Author: Weiqun Zhang Date: Fri Dec 4 17:18:32 2020 -0800 DPCPP: run stencil rap on device (#1606) An earlier compiler bug has been fixed. The stencil rap kernel can run on GPU now. Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit 0490136d291cc3b354558f091391277f16a46c5a Author: Weiqun Zhang Date: Fri Dec 4 17:11:29 2020 -0800 Remove DPCPP hacks (#1605) Because the Intel graphics driver has increased the kernel parameters to 2K by default, these hacks are no longer needed. Src/Base/AMReX_GpuLaunchFunctsG.H Src/EB/AMReX_EB2_3D_C.cpp Src/EB/AMReX_EB2_Level.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.cpp Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.cpp Src/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp commit 13d67fd18e736e96bfe4e67a6387750380900384 Author: Axel Huebl Date: Fri Dec 4 10:25:55 2020 -0800 CI: Disable DPC++ oneAPI 2020-11 (#1602) Someone patched in a dedicated CMake compiler ID for Intel DPC++ but did not add full compiler support. Release: 2021.1.1 Ref.: https://gitlab.kitware.com/cmake/cmake/-/issues/21551 .github/workflows/linux.yml commit a168f5927cee0d87ff839b23128c8fc878622313 Author: mic84 Date: Thu Dec 3 16:19:42 2020 -0800 CMake: bring back check on CUDA computing capabilities (#1600) Tools/CMake/AMReX_SetupCUDA.cmake commit 2a070723e249e3e4cf2befd61a576330fc280d80 Author: Andrew Myers Date: Thu Dec 3 14:05:14 2020 -0800 move the memcpy wrapper into amrex::Gpu (#1598) Src/Base/AMReX_GpuUtility.H commit 5b436831f47158a50738dcd64a6fe579c9ef8861 Author: Andrew Myers Date: Thu Dec 3 12:35:20 2020 -0800 Add wrapper for memcpy. (#1597) `std::memcpy` does not seem to work in device code in our HIP CI. This adds a wrapper that reverts to the global namespace `memcpy` in HIP device code. The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_GpuUtility.H commit ef665e2a2fcc245162767b3613aba48bbc59f958 Author: Axel Huebl Date: Thu Dec 3 11:16:38 2020 -0800 Fix: Windows and/or includes (iso646) (#1593) ## Summary Add missing includes to `AMReX_Extension.H`, which pulls the `` include on Windows for support for `and`/`or`. ## Additional background Seen with Clang 11.0.0 on x64 Windows with Visual Studio 2017 on conda-forge during a WarpX build. See also #947 ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Extension.H Src/Base/AMReX_Vector.H Src/Particle/AMReX_ParticleTile.H commit 5c1addce382f86f0701e6bddc209522cfe70c52d Author: Axel Huebl Date: Thu Dec 3 10:33:42 2020 -0800 CI: Tutorial with OMP (#1591) Making sure the OpenMP Tutorials build as well. .github/workflows/linux.yml Tutorials/Particles/ElectromagneticPIC/CMakeLists.txt commit 0d000b5397b5f6ef94924d85b65846c5811faf86 Author: Weiqun Zhang Date: Thu Dec 3 10:23:15 2020 -0800 AMReX_Config.H (#1566) Based on discussions with @sayerhs, @ax3l, and @mic84, We have decided to make the following changes to the build systems. All headers in Src/ will include AMReX_Config.H, which will be generated by the build systems (both gnu make and cmake). The file will include all the macros we currently pass as compiler options. The purpose of this is to solve some inconsistency issues in the current systems. We support both super build and using AMReX as a library. Moreover, we do not want to force libamrex users to use the same build tool that is used to build the library. In this PR, we are put all the macro definitions in AMReX_Config.H that is included by every AMReX header. Co-authored-by: Axel Huebl Co-authored-by: Michele Rosso Src/Amr/AMReX_Amr.H Src/Amr/AMReX_AmrLevel.H Src/Amr/AMReX_AuxBoundaryData.H Src/Amr/AMReX_Derive.H Src/Amr/AMReX_Extrapolater.H Src/Amr/AMReX_LevelBld.H Src/Amr/AMReX_PROB_AMR_F.H Src/Amr/AMReX_StateData.H Src/Amr/AMReX_StateDescriptor.H Src/Amr/AMReX_extrapolater_1D_K.H Src/Amr/AMReX_extrapolater_2D_K.H Src/Amr/AMReX_extrapolater_3D_K.H Src/Amr/AMReX_extrapolater_K.H Src/AmrCore/AMReX_AmrCore.H Src/AmrCore/AMReX_AmrMesh.H Src/AmrCore/AMReX_AmrParGDB.H Src/AmrCore/AMReX_AmrParticles.H Src/AmrCore/AMReX_Cluster.H Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_FLUXREG_F.H Src/AmrCore/AMReX_FillPatchUtil.H Src/AmrCore/AMReX_FillPatchUtil_F.H Src/AmrCore/AMReX_FillPatchUtil_I.H Src/AmrCore/AMReX_FluxReg_1D_C.H Src/AmrCore/AMReX_FluxReg_2D_C.H Src/AmrCore/AMReX_FluxReg_3D_C.H Src/AmrCore/AMReX_FluxReg_C.H Src/AmrCore/AMReX_FluxRegister.H Src/AmrCore/AMReX_INTERP_F.H Src/AmrCore/AMReX_Interp_1D_C.H Src/AmrCore/AMReX_Interp_2D_C.H Src/AmrCore/AMReX_Interp_3D_C.H Src/AmrCore/AMReX_Interp_C.H Src/AmrCore/AMReX_Interpolater.H Src/AmrCore/AMReX_TagBox.H Src/Base/AMReX.H Src/Base/AMReX_Algorithm.H Src/Base/AMReX_Arena.H Src/Base/AMReX_Array.H Src/Base/AMReX_Array4.H Src/Base/AMReX_ArrayLim.H Src/Base/AMReX_AsyncOut.H Src/Base/AMReX_BArena.H Src/Base/AMReX_BCRec.H Src/Base/AMReX_BCUtil.H Src/Base/AMReX_BC_TYPES.H Src/Base/AMReX_BLBackTrace.H Src/Base/AMReX_BLFort.H Src/Base/AMReX_BLProfiler.H Src/Base/AMReX_BLassert.H Src/Base/AMReX_BackgroundThread.H Src/Base/AMReX_BaseFab.H Src/Base/AMReX_BaseFabUtility.H Src/Base/AMReX_BaseUmap.H Src/Base/AMReX_BaseUmap_f.H Src/Base/AMReX_BlockMutex.H Src/Base/AMReX_Box.H Src/Base/AMReX_BoxArray.H Src/Base/AMReX_BoxDomain.H Src/Base/AMReX_BoxIterator.H Src/Base/AMReX_BoxList.H Src/Base/AMReX_CArena.H Src/Base/AMReX_CONSTANTS.H Src/Base/AMReX_COORDSYS_1D_C.H Src/Base/AMReX_COORDSYS_2D_C.H Src/Base/AMReX_COORDSYS_3D_C.H Src/Base/AMReX_COORDSYS_C.H Src/Base/AMReX_CoordSys.H Src/Base/AMReX_CudaGraph.H Src/Base/AMReX_CuptiTrace.H Src/Base/AMReX_DArena.H Src/Base/AMReX_Dim3.H Src/Base/AMReX_DistributionMapping.H Src/Base/AMReX_EArena.H Src/Base/AMReX_Exception.H Src/Base/AMReX_Extension.H Src/Base/AMReX_FACopyDescriptor.H Src/Base/AMReX_FArrayBox.H Src/Base/AMReX_FPC.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayUtility.H Src/Base/AMReX_FabConv.H Src/Base/AMReX_FabFactory.H Src/Base/AMReX_FilCC_1D_C.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_FilCC_3D_C.H Src/Base/AMReX_FilCC_C.H Src/Base/AMReX_FilND_C.H Src/Base/AMReX_FileSystem.H Src/Base/AMReX_ForkJoin.H Src/Base/AMReX_Functional.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_Gpu.H Src/Base/AMReX_GpuAllocators.H Src/Base/AMReX_GpuAssert.H Src/Base/AMReX_GpuAsyncArray.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuComplex.H Src/Base/AMReX_GpuContainers.H Src/Base/AMReX_GpuControl.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuElixir.H Src/Base/AMReX_GpuError.H Src/Base/AMReX_GpuFuse.H Src/Base/AMReX_GpuKernelInfo.H Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunchFunctsC.H Src/Base/AMReX_GpuLaunchFunctsG.H Src/Base/AMReX_GpuLaunchGlobal.H Src/Base/AMReX_GpuLaunchMacrosC.H Src/Base/AMReX_GpuLaunchMacrosG.H Src/Base/AMReX_GpuMemory.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuQualifiers.H Src/Base/AMReX_GpuRange.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_GpuTypes.H Src/Base/AMReX_GpuUtility.H Src/Base/AMReX_IArrayBox.H Src/Base/AMReX_INT.H Src/Base/AMReX_IndexSequence.H Src/Base/AMReX_IndexType.H Src/Base/AMReX_IntConv.H Src/Base/AMReX_IntVect.H Src/Base/AMReX_LayoutData.H Src/Base/AMReX_Lazy.H Src/Base/AMReX_Loop.H Src/Base/AMReX_MFCopyDescriptor.H Src/Base/AMReX_MFIter.H Src/Base/AMReX_Machine.H Src/Base/AMReX_Machine.cpp Src/Base/AMReX_MakeType.H Src/Base/AMReX_Math.H Src/Base/AMReX_MemPool.H Src/Base/AMReX_MemProfiler.H Src/Base/AMReX_MultiFab.H Src/Base/AMReX_MultiFabUtil.H Src/Base/AMReX_MultiFabUtil_1D_C.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_3D_C.H Src/Base/AMReX_MultiFabUtil_C.H Src/Base/AMReX_MultiFabUtil_nd_C.H Src/Base/AMReX_NFiles.H Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/AMReX_OpenMP.H Src/Base/AMReX_Orientation.H Src/Base/AMReX_PODVector.H Src/Base/AMReX_ParallelContext.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelReduce.H Src/Base/AMReX_ParmParse.H Src/Base/AMReX_Partition.H Src/Base/AMReX_Periodicity.H Src/Base/AMReX_PhysBCFunct.H Src/Base/AMReX_PlotFileDataImpl.H Src/Base/AMReX_PlotFileUtil.H Src/Base/AMReX_Print.H Src/Base/AMReX_REAL.H Src/Base/AMReX_Random.H Src/Base/AMReX_RandomEngine.H Src/Base/AMReX_RealBox.H Src/Base/AMReX_RealVect.H Src/Base/AMReX_Reduce.H Src/Base/AMReX_SPACE.H Src/Base/AMReX_Scan.H Src/Base/AMReX_Slopes_K.H Src/Base/AMReX_ThirdPartyProfiling.H Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_Tuple.H Src/Base/AMReX_TypeTraits.H Src/Base/AMReX_Utility.H Src/Base/AMReX_Vector.H Src/Base/AMReX_VectorIO.H Src/Base/AMReX_VisMF.H Src/Base/AMReX_ccse-mpi.H Src/Base/AMReX_filcc_f.H Src/Base/AMReX_iMultiFab.H Src/Base/AMReX_parstream.H Src/Boundary/AMReX_BndryData.H Src/Boundary/AMReX_BndryRegister.H Src/Boundary/AMReX_BoundCond.H Src/Boundary/AMReX_FabSet.H Src/Boundary/AMReX_InterpBndryData.H Src/Boundary/AMReX_InterpBndryData_1D_K.H Src/Boundary/AMReX_InterpBndryData_2D_K.H Src/Boundary/AMReX_InterpBndryData_3D_K.H Src/Boundary/AMReX_InterpBndryData_K.H Src/Boundary/AMReX_LOUtil_K.H Src/Boundary/AMReX_LO_BCTYPES.H Src/Boundary/AMReX_MacBndry.H Src/Boundary/AMReX_Mask.H Src/Boundary/AMReX_MultiMask.H Src/Boundary/AMReX_YAFluxRegister.H Src/Boundary/AMReX_YAFluxRegister_1D_K.H Src/Boundary/AMReX_YAFluxRegister_2D_K.H Src/Boundary/AMReX_YAFluxRegister_3D_K.H Src/Boundary/AMReX_YAFluxRegister_K.H Src/CMakeLists.txt Src/EB/AMReX_EB2.H Src/EB/AMReX_EB2_2D_C.H Src/EB/AMReX_EB2_3D_C.H Src/EB/AMReX_EB2_C.H Src/EB/AMReX_EB2_GeometryShop.H Src/EB/AMReX_EB2_Graph.H Src/EB/AMReX_EB2_IF.H Src/EB/AMReX_EB2_IF_AllRegular.H Src/EB/AMReX_EB2_IF_Base.H Src/EB/AMReX_EB2_IF_Box.H Src/EB/AMReX_EB2_IF_Complement.H Src/EB/AMReX_EB2_IF_Cylinder.H Src/EB/AMReX_EB2_IF_Difference.H Src/EB/AMReX_EB2_IF_Ellipsoid.H Src/EB/AMReX_EB2_IF_Extrusion.H Src/EB/AMReX_EB2_IF_Intersection.H Src/EB/AMReX_EB2_IF_Lathe.H Src/EB/AMReX_EB2_IF_Plane.H Src/EB/AMReX_EB2_IF_Polynomial.H Src/EB/AMReX_EB2_IF_Rotation.H Src/EB/AMReX_EB2_IF_Scale.H Src/EB/AMReX_EB2_IF_Sphere.H Src/EB/AMReX_EB2_IF_Spline.H Src/EB/AMReX_EB2_IF_Torus.H Src/EB/AMReX_EB2_IF_Translation.H Src/EB/AMReX_EB2_IF_Union.H Src/EB/AMReX_EB2_Level.H Src/EB/AMReX_EB2_MultiGFab.H Src/EB/AMReX_EBAmrUtil.H Src/EB/AMReX_EBCellFlag.H Src/EB/AMReX_EBDataCollection.H Src/EB/AMReX_EBFArrayBox.H Src/EB/AMReX_EBFabFactory.H Src/EB/AMReX_EBFluxRegister.H Src/EB/AMReX_EBFluxRegister_2D_C.H Src/EB/AMReX_EBFluxRegister_3D_C.H Src/EB/AMReX_EBFluxRegister_C.H Src/EB/AMReX_EBInterpolater.H Src/EB/AMReX_EBMultiFabUtil.H Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H Src/EB/AMReX_EBMultiFabUtil_C.H Src/EB/AMReX_EBSupport.H Src/EB/AMReX_EB_slopes_K.H Src/EB/AMReX_EB_utils.H Src/EB/AMReX_MultiCutFab.H Src/EB/AMReX_WriteEBSurface.H Src/EB/AMReX_WriteEB_F.H Src/EB/AMReX_algoim.H Src/EB/AMReX_algoim_K.H Src/EB/AMReX_distFcnElement.H Src/Extern/Conduit/AMReX_Conduit_Blueprint.H Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_Habec_K.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_HypreABecLap.H Src/Extern/HYPRE/AMReX_HypreABecLap2.H Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/Extern/HYPRE/CMakeLists.txt Src/Extern/PETSc/AMReX_PETSc.H Src/Extern/PETSc/CMakeLists.txt Src/Extern/ProfParser/AMReX_AVGDOWN_F.H Src/Extern/ProfParser/AMReX_BLProfStats.H Src/Extern/ProfParser/AMReX_BLProfUtilities.H Src/Extern/ProfParser/AMReX_BLWritePlotFile.H Src/Extern/ProfParser/AMReX_CommProfStats.H Src/Extern/ProfParser/AMReX_RegionsProfStats.H Src/Extern/ProfParser/CMakeLists.txt Src/Extern/SENSEI/AMReX_AmrDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrInSituBridge.H Src/Extern/SENSEI/AMReX_AmrMeshDataAdaptor.H Src/Extern/SENSEI/AMReX_AmrMeshInSituBridge.H Src/Extern/SENSEI/AMReX_InSituBridge.H Src/Extern/SENSEI/AMReX_InSituUtils.H Src/Extern/SENSEI/CMakeLists.txt Src/Extern/SWFFT/Dfft.H Src/Extern/amrdata/AMReX_AmrData.H Src/Extern/amrdata/AMReX_AmrvisConstants.H Src/Extern/amrdata/AMReX_DataServices.H Src/Extern/amrdata/AMReX_XYPlotDataList.H Src/Extern/hpgmg/BL_HPGMG.H Src/F_Interfaces/AmrCore/AMReX_FAmrCore.H Src/F_Interfaces/AmrCore/AMReX_FlashFluxRegister.H Src/F_Interfaces/Base/AMReX_FPhysBC.H Src/F_Interfaces/CMakeLists.txt Src/LinearSolvers/MLMG/AMReX_MLABecLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLALap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLALap_K.H Src/LinearSolvers/MLMG/AMReX_MLALaplacian.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellLinOp.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLEBTensor_K.H Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLLinOp_F.H Src/LinearSolvers/MLMG/AMReX_MLLinOp_K.H Src/LinearSolvers/MLMG/AMReX_MLMG.H Src/LinearSolvers/MLMG/AMReX_MLMGBndry.H Src/LinearSolvers/MLMG/AMReX_MLMG_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLMG_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLPoisson.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLPoisson_K.H Src/LinearSolvers/MLMG/AMReX_MLTensorOp.H Src/LinearSolvers/MLMG/AMReX_MLTensor_1D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_3D_K.H Src/LinearSolvers/MLMG/AMReX_MLTensor_K.H Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_NodalProjector.H Src/Particle/AMReX_ArrayOfStructs.H Src/Particle/AMReX_BinIterator.H Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_NeighborList.H Src/Particle/AMReX_NeighborParticles.H Src/Particle/AMReX_NeighborParticlesCPUImpl.H Src/Particle/AMReX_NeighborParticlesGPUImpl.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_ParIter.H Src/Particle/AMReX_Particle.H Src/Particle/AMReX_ParticleBufferMap.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleHDF5.H Src/Particle/AMReX_ParticleIO.H Src/Particle/AMReX_ParticleInit.H Src/Particle/AMReX_ParticleLocator.H Src/Particle/AMReX_ParticleMPIUtil.H Src/Particle/AMReX_ParticleMesh.H Src/Particle/AMReX_ParticleReduce.H Src/Particle/AMReX_ParticleTile.H Src/Particle/AMReX_ParticleTransformation.H Src/Particle/AMReX_ParticleUtil.H Src/Particle/AMReX_Particle_mod_K.H Src/Particle/AMReX_Particles.H Src/Particle/AMReX_SparseBins.H Src/Particle/AMReX_StructOfArrays.H Src/Particle/AMReX_TracerParticle_mod_K.H Src/Particle/AMReX_TracerParticles.H Src/Particle/AMReX_WriteBinaryParticleData.H Src/Particle/CMakeLists.txt Src/SDC/AMReX_SDCstruct.H Tools/CMake/AMReXGenerateConfigHeader.cmake Tools/CMake/AMReXInstallHelpers.cmake Tools/CMake/AMReXSetDefines.cmake Tools/CMake/AMReX_Config.H.in Tools/CMake/AMReX_Config.cmake Tools/CMake/AMReX_Defines.cmake Tools/CMake/AMReX_Utils.cmake Tools/CMake/modify_installed_headers.cmake Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/libamrex/mkconfig.py Tutorials/Amr/Advection_AmrLevel/Source/AmrLevelAdv.H commit 5fb87ea35fa521e9ce97dc9d36f21eda5e2343be Author: Shreyas Ananthan Date: Thu Dec 3 09:19:23 2020 -0700 Restore reproducibility of solutions with MacProjector reuse (#1596) Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit ae4cc9a1488719e284409b078887866fe65709d9 Author: mic84 Date: Wed Dec 2 15:12:33 2020 -0800 CMake: output missing components list when AMReX is not found (#1572) Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXInstallHelpers.cmake commit 75c0896594cd38d883d7907466657eac3ca1a188 Author: Axel Huebl Date: Wed Dec 2 15:09:17 2020 -0800 CI: Windows + Clang (#1588) Add Clang as Windows target. This compiler offers potentially a route to modern OpenMP. (MSVC: 2.0 standard, LLVM/Clang: 3.1 standard) .github/workflows/windows.yml commit 91bdc795d598b8bbf58d640d572f27dfa5ae31a5 Author: Shreyas Ananthan Date: Wed Dec 2 14:44:56 2020 -0700 MacProjector: Update setDomainBC API (#1589) Allow `setDomainBC` to be called before `setUMAC` has been called. Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit 820a30d7966426a96e24e3963b37f3789eec258f Author: Andrew Myers Date: Wed Dec 2 13:26:03 2020 -0800 Fix InitNRandomPerCell for single precision particles (#1582) * due to roundoff errors, we need to redraw these numbers sometimes when single precision particles are used. * missing ; Src/Particle/AMReX_ParticleInit.H commit dc0ec74c4d120509f9754fb7d43af6d275d9c1a4 Author: Shreyas Ananthan Date: Wed Dec 2 13:35:05 2020 -0700 DPCPP: Update CMake options to allow building with MPI+DPCPP (#1586) Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXSYCL.cmake commit 89220ed73d9fe74618386f8b1bfafd30760d6fde Author: Andrew Myers Date: Tue Dec 1 10:31:23 2020 -0800 Reimplement NumParticlesAtLevel (#1581) This removes a workaround that had this function fall back to CPUs for CUDA (but not HIP or DPC++). Additionally, this re-implements the function in terms of ReduceTuple to avoid multiple memcpy operations in the case of multiple boxes. Src/Particle/AMReX_ParticleContainerI.H Tests/Particles/ParticleReduce/main.cpp commit bbcce736ea9497cdb4722e8b4f4c2ee0d7c193ca Author: Robert Maynard Date: Tue Dec 1 13:24:16 2020 -0500 eval_genex correctly replaces the matching compiler version (#1580) Previously it would place the resulting output in the variable `1` instead of `_in`. Additionally it would try to add an extra `>`. Tools/CMake/AMReXGenexHelpers.cmake commit 9a523cf95dc3046081b0830cf435a3455929728d Author: Robert Maynard Date: Tue Dec 1 13:20:19 2020 -0500 Add msvc cuda support (#1573) ## Summary This set of changes has allowed me to build AMReX on windows using CUDA 11.X and VS2019. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_ErrorList.cpp Src/Base/AMReX.H Src/Base/AMReX.cpp Src/Base/AMReX_BLBackTrace.cpp Src/Base/AMReX_Extension.H Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_MultiFab.H Tools/CMake/AMReX_Config.cmake commit 1cb99250ecd4da445cfc8afe325664030c49c4f0 Author: Weiqun Zhang Date: Tue Dec 1 10:12:38 2020 -0800 Update CHANGES (#1584) CHANGES commit 94ca530cb43c497cf22dfbfbd13dfd087f7d910d Author: Andrew Myers Date: Mon Nov 30 21:55:26 2020 -0500 Fix bug affecting async IO + single precision particles (#1583) The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Particle/AMReX_WriteBinaryParticleData.H commit d9af9ab7f134b04bc5e3cc57362e82a64fecef0a Author: Weiqun Zhang Date: Mon Nov 30 12:49:50 2020 -0800 Fix warnings for DPCPP (#1579) Fix unused variable. Do not pass `-fsycl-unnamed-lambda` to `mpiicpx` for linking to get rid of warning on unknown argument. Src/Base/AMReX_FBI.H Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tutorials/GPU/CNS/Source/CNS.H commit 07afa8ec8f59403ac54c2b2c566c1c88b1863112 Author: Weiqun Zhang Date: Mon Nov 30 10:25:56 2020 -0800 Template specialization of MPI communication functions for char (#1569) * Add template specialization of MPI communication functions for char. Select a proper data type according to the size. * Use the new functions in FabArray and Particle communication. And this also fixes a bug in Particle communication. (The counts should be divided by the size of data type.) * Due to a gcc bug on explicit specialization in a namespace, workaround was implemented in AMReX_ParallelDescriptor.H and .cpp. Src/Base/AMReX_Arena.H Src/Base/AMReX_Arena.cpp Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_ParallelDescriptor.H Src/Base/AMReX_ParallelDescriptor.cpp Src/Particle/AMReX_ParticleCommunication.H commit 50c40c46382a83cdc98fd59e337c5037aba07c1e Author: Weiqun Zhang Date: Mon Nov 30 10:16:16 2020 -0800 EB2::PolyIF (#1567) The existing PolynomialIF contains a dynamic size vector. This makes it hard to support it on GPU. We have used an undocumented feature to get this run on GPU. However, this code can no longer compile, because the GNU make system now makes cross execution space calls an error. So we disable GPU support for PolynomialIF and add a new PolyIF class with fixed size array and therefore GPU support. Src/EB/AMReX_EB2_IF_Polynomial.H commit 26e4d30f346cb89921502e409da1b933a60f9c81 Author: Weiqun Zhang Date: Mon Nov 30 10:11:37 2020 -0800 Add .gitattributes (#1578) Make github detect *.H as C++. This file is borrowed from Castro. .gitattributes commit a77968674f3d9ec1d4298d99a506dd1951f88c67 Author: Shreyas Ananthan Date: Mon Nov 30 09:17:48 2020 -0700 Tools/Plotfile: Fix build errors in single precision mode (#1577) This PR fixes one build error and one warning observed when compiling `Tools/Plotfile` utilities with `-DAMREX_USE_FLOAT` Tools/Plotfile/fcompare.cpp Tools/Plotfile/fextract.cpp commit 82eceae29419ceec24450926af1f5445ac425d00 Author: Shreyas Ananthan Date: Mon Nov 30 09:14:54 2020 -0700 Arena: Add accessor to query Arena attributes through info object (#1575) ## Summary This PR adds an accessor to the `Arena` class that can be used to query the attributes of the Arena (e.g., `device_use_managed_memory` flag) from the application codes. ## Additional background Currently, there is no easy way to determine if `amrex::The_Arena()` is using managed memory. Applications would have to use a combination of `AMREX_USE_HIP` compile-time definition and `amrex.the_arena_is_managed` ParmParse variable to determine this. Testing individual pointers with `amrex::Gpu::isManaged()` is possible only on CUDA/HIP but not DPC++. Also it is costly compared to just querying the `Arena` object. Allowing read-only access to `ArenaInfo` object will allow a wide variety of checks in application code. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [X] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Arena.H commit acdb7eac1d5a1744e87c0caafd1c43cc5e1171e9 Author: Luca Fedeli Date: Mon Nov 30 16:58:32 2020 +0100 Fix some warnings related to double-->float conversions (#1571) I tried to compile `WarpX` in single precision and I noticed several warnings related to casts from `double` to `float` in `AMReX`. This PR should fix some of these warnings. I plan to continue in the next days with other PRs, if you agree. Src/AmrCore/AMReX_AmrMesh.H Src/Base/AMReX_CoordSys.H Src/Base/AMReX_FilCC_2D_C.H Src/Base/AMReX_Geometry.H Src/Base/AMReX_MultiFabUtil_2D_C.H Src/Base/AMReX_MultiFabUtil_nd_C.H commit abdab63376660238396626ee064130a5c1826024 Author: Shreyas Ananthan Date: Sun Nov 29 14:04:44 2020 -0700 Fix assertion check for MacProjection (#1576) Fixes bug introduced in #1574 Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit abeb357db57261fe8a8d811e161da0c6c2814d92 Author: Shreyas Ananthan Date: Sun Nov 29 09:53:42 2020 -0700 Allow MacProjection reuse (#1574) This commit modifies the MacProjector interface to allow reuse of the MacProjector. The changes to the API are backwards compatible. - Add a new constructor that does not require specification of beta, umac etc. - Add `initProjector` to explicitly initialize the projection instance at app level - Add methods to set MAC velocities, divU etc. Src/LinearSolvers/Projections/AMReX_MacProjector.H Src/LinearSolvers/Projections/AMReX_MacProjector.cpp commit cc8f323c426f3957619c93477af2b9e32e3b49c2 Author: Axel Huebl Date: Mon Nov 23 22:27:06 2020 -0800 CI: Make DPC++ Pretty (#1568) * CI: Make DPC++ Pretty Just combing the show pony. * oneAPI Activation: still needs set +e .github/workflows/linux.yml commit f068164b513c12828dac069510d64f8b4912028e Author: mic84 Date: Mon Nov 23 14:48:58 2020 -0800 Remove SUNDIALS support (#1559) ## Summary SUNDIALS support in AMReX consisted of only few Fortran interfaces to SUNDIALS functions for the convenience of app codes. Since there are no AMReX functionalities depending on SUNDIALS, support for SUNDIALS will be removed. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/SUNDIALS.rst Docs/sphinx_documentation/source/SUNDIALS_CVODE.rst Docs/sphinx_documentation/source/SUNDIALS_top.rst Docs/sphinx_tutorials/source/SUNDIALS_Tutorial.rst Src/CMakeLists.txt Src/Extern/CVODE/Make.package Src/Extern/CVODE/cvode_interface.f90 Src/Extern/CVODE/fnvector_serial.f90 Src/Extern/CVODE/integrator_stats.f90 Src/Extern/CVODE/sundials_fdlsmat.f90 Src/Extern/SUNDIALS/CMakeLists.txt Src/Extern/SUNDIALS/Make.package Src/Extern/SUNDIALS/arkode_interface.f90 Src/Extern/SUNDIALS/cvode_interface.f90 Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/FindSUNDIALS.cmake Tools/GNUMake/Make.defs Tools/GNUMake/packages/Make.cvode Tools/GNUMake/packages/Make.sundials Tutorials/CMakeLists.txt Tutorials/CVODE/EX1_F/GNUmakefile Tutorials/CVODE/EX1_F/Make.package Tutorials/CVODE/EX1_F/inputs Tutorials/CVODE/EX1_F/integrate_ode.f90 Tutorials/CVODE/EX1_F/main.cpp Tutorials/CVODE/EX1_F/myfunc_F.H Tutorials/CVODE/EX1_F/ode_mod.f90 Tutorials/CVODE/EX2_F/GNUmakefile Tutorials/CVODE/EX2_F/Make.package Tutorials/CVODE/EX2_F/inputs Tutorials/CVODE/EX2_F/integrate_ode_no_jac.f90 Tutorials/CVODE/EX2_F/integrate_ode_with_jac.f90 Tutorials/CVODE/EX2_F/main.cpp Tutorials/CVODE/EX2_F/myfunc_F.H Tutorials/CVODE/EX2_F/ode_mod.f90 Tutorials/SUNDIALS/EX-CUSOLVER/GNUmakefile Tutorials/SUNDIALS/EX-CUSOLVER/Make.CVODE Tutorials/SUNDIALS/EX-CUSOLVER/Make.package Tutorials/SUNDIALS/EX-CUSOLVER/README.md Tutorials/SUNDIALS/EX-CUSOLVER/extern_probin.template Tutorials/SUNDIALS/EX-CUSOLVER/inputs Tutorials/SUNDIALS/EX-CUSOLVER/inputs_128 Tutorials/SUNDIALS/EX-CUSOLVER/inputs_256 Tutorials/SUNDIALS/EX-CUSOLVER/inputs_32 Tutorials/SUNDIALS/EX-CUSOLVER/inputs_64 Tutorials/SUNDIALS/EX-CUSOLVER/main.cpp Tutorials/SUNDIALS/EX-CUSOLVER/make_cuda.sh Tutorials/SUNDIALS/EX-CUSOLVER/make_cuda_cusolver.sh Tutorials/SUNDIALS/EX-CUSOLVER/make_serial.sh Tutorials/SUNDIALS/EX-CUSOLVER/react_cuda.cpp Tutorials/SUNDIALS/EX-CUSOLVER/react_cuda_cusolver.cpp Tutorials/SUNDIALS/EX-CUSOLVER/react_serial.cpp Tutorials/SUNDIALS/EX-CUSOLVER/react_system.F90 Tutorials/SUNDIALS/EX-CUSOLVER/react_utils.F90 Tutorials/SUNDIALS/EX-CUSOLVER/test_react.H Tutorials/SUNDIALS/EX-CUSOLVER/test_react_F.H Tutorials/SUNDIALS/EX1_C/CMakeLists.txt Tutorials/SUNDIALS/EX1_C/GNUmakefile Tutorials/SUNDIALS/EX1_C/Make.package Tutorials/SUNDIALS/EX1_C/SetIC.f90 Tutorials/SUNDIALS/EX1_C/inputs Tutorials/SUNDIALS/EX1_C/inputs_2box Tutorials/SUNDIALS/EX1_C/inputs_non_vectorized Tutorials/SUNDIALS/EX1_C/main.cpp Tutorials/SUNDIALS/EX1_C/myfunc_F.H Tutorials/SUNDIALS/EX1_CUDA/GNUmakefile Tutorials/SUNDIALS/EX1_CUDA/Make.package Tutorials/SUNDIALS/EX1_CUDA/SetIC.f90 Tutorials/SUNDIALS/EX1_CUDA/inputs Tutorials/SUNDIALS/EX1_CUDA/main.cpp Tutorials/SUNDIALS/EX1_CUDA/myfunc_F.H Tutorials/SUNDIALS/EX1_CUDA/ode_mod.f90 Tutorials/SUNDIALS/EX1_F/GNUmakefile Tutorials/SUNDIALS/EX1_F/Make.package Tutorials/SUNDIALS/EX1_F/inputs Tutorials/SUNDIALS/EX1_F/integrate_ode.f90 Tutorials/SUNDIALS/EX1_F/main.cpp Tutorials/SUNDIALS/EX1_F/myfunc_F.H Tutorials/SUNDIALS/EX1_F/ode_mod.f90 Tutorials/SUNDIALS/EX2_F/GNUmakefile Tutorials/SUNDIALS/EX2_F/Make.package Tutorials/SUNDIALS/EX2_F/inputs Tutorials/SUNDIALS/EX2_F/integrate_ode_no_jac.f90 Tutorials/SUNDIALS/EX2_F/integrate_ode_with_jac.f90 Tutorials/SUNDIALS/EX2_F/main.cpp Tutorials/SUNDIALS/EX2_F/myfunc_F.H Tutorials/SUNDIALS/EX2_F/ode_mod.f90 commit e1ff59c8d9207a135f25284380a1ba8fc15abb1f Author: Roberto Porcu <53792251+rporcu@users.noreply.github.com> Date: Sun Nov 22 23:19:06 2020 -0500 allow MLEBABecLap to solve for multiple components (#1557) ## Summary MLLinOp::getNComp() is returning 1 and right now MLEBABecLap is working only with MultiFabs having 1 component. ## Additional background This PR has the goal to generalize MLEBABecLap solver for a generic N number of components. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Co-authored-by: Weiqun Zhang Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.H Src/LinearSolvers/MLMG/AMReX_MLEBABecLap.cpp commit 1f8163c397e1aa33fd633219166b87d4633a0b46 Author: Weiqun Zhang Date: Sun Nov 22 19:40:00 2020 -0800 DPCPP: MPI (#1556) Allow MPI for dpcpp build in gnu make assuming mpiicpx is used. Tools/GNUMake/Make.defs Tools/GNUMake/Make.machines Tools/GNUMake/comps/dpcpp.mak Tools/GNUMake/sites/Make.alcf commit a85fd8d7971f9448b8a00e796a392bbe84ae0f97 Author: Axel Huebl Date: Fri Nov 20 10:28:25 2020 -0800 CMake: OpenMP for Fortran (#1563) * CMake: OpenMP for Fortran Fix missing activation. * Update Tools/CMake/AMReXParallelBackends.cmake Co-authored-by: mic84 * CMakeL OpenMP Fortran fix Config, too Co-authored-by: mic84 Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXParallelBackends.cmake commit 7a0fea34be764441014e2ba7dac44fecbdbbabfb Author: Weiqun Zhang Date: Thu Nov 19 11:01:07 2020 -0800 Non-local boundary conditions (#1544) * Non-local boundary conditions * Rotate90 fills the lo-x and lo-y boundary regions by rotating the data around (x=0,y=0) by 90 degrees in either direction. It also fills the corner of lo-x and lo-y boundary region by rotating the data by 180 degrees. * Rotate180 fills the lo-x boundary by rotating the data around (x=0,y=L_y/2) by 180 degrees. * Fill the polar boundaries of the spherical coordinates (theta, phi, r). The lo-x boundary is filled with f(-x,y) = f(x,mod(y+pi,2*pi)), and the hi-x boundary is filled with f(pi+x,y) = f(pi-x,mod(y+pi,2*pi)). * GPU support for NonLocalBC Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArrayBase.H Src/Base/AMReX_FabArrayBase.cpp Src/Base/AMReX_NonLocalBC.H Src/Base/AMReX_NonLocalBCImpl.H Src/Base/CMakeLists.txt Src/Base/Make.package commit b5e08d92541f24e3ee3cb8d46f5b49c0147538bd Author: Weiqun Zhang Date: Thu Nov 19 10:28:53 2020 -0800 Remove deprecated MFUtil::convert function. One could use amrex::cast for FabArray casting. (#1523) Src/Base/AMReX_MultiFabUtilI.H commit 907baaede901d52601f120d606224bccd04d9611 Author: Andrew Myers Date: Thu Nov 19 13:24:16 2020 -0500 If TheZeroVector() is passed in to SortParticlesByBin, do nothing instead of crashing. (#1564) Src/Particle/AMReX_ParticleContainerI.H Src/Particle/AMReX_Particles.H commit 6038a3102fc55db861d69b242eecc35119675b96 Author: Weiqun Zhang Date: Wed Nov 18 09:49:08 2020 -0800 Scan size type (#1521) Make the size type a template parameter to support the scan of more than INT_MAX elements. Src/Base/AMReX_Scan.H commit ff92f1c129ea149eead13dc264eabf15a705e74d Author: mic84 Date: Wed Nov 18 09:14:51 2020 -0800 CMake: AMReX_MPI options depends on SYCL, not HIP (#1555) Tools/CMake/AMReXOptions.cmake commit 7cf969f1b656f9ab17975c90952360fb400dd712 Author: Axel Huebl Date: Wed Nov 18 09:14:25 2020 -0800 CMake: Fix Single-Precision Switch (#1550) * CMake: Fix Single-Precision Switch Since the latest refactoring, single-recision builds were not activated. This fixes it. * AMReX_Config.H: add AMREX_SINGLE_PRECISION_PARTICLES Src/Particle/CMakeLists.txt Tools/CMake/AMReX_Config.H.in Tools/CMake/AMReX_Defines.cmake commit 14c3003f8f4dda9ef9db0c04ba613e378677358a Author: Weiqun Zhang Date: Wed Nov 18 09:03:38 2020 -0800 Refactor FabArray's communication functions (#1553) Remove duplication and allow for more flexibility for extension. Src/AmrCore/AMReX_TagBox.cpp Src/Base/AMReX_FBI.H Src/Base/AMReX_FabArray.H Src/Base/AMReX_FabArrayCommI.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_MultiFabUtil.cpp Src/Base/AMReX_iMultiFab.cpp Src/Boundary/AMReX_YAFluxRegister.cpp commit 079f599f74057ea93d8c60b82760b8747b21849e Author: Weiqun Zhang Date: Tue Nov 17 18:40:51 2020 -0800 DPCPP: Fortran (#1554) Switch from gfortran to ifx that comes with oneapi in GNU make. Tools/GNUMake/comps/dpcpp.mak commit e38742e322f6fbdf0e0fb090271d229fc2c834b9 Author: mic84 Date: Tue Nov 17 13:26:24 2020 -0800 CMake: improvements to handling of buildInfo (#1552) Tools/CMake/AMReXBuildInfo.cmake commit 477df17dd96607c385fc9410c4fe7746f121b58d Author: Weiqun Zhang Date: Tue Nov 17 11:19:19 2020 -0800 Fix BaseFab::maxabs (#1546) ## Summary For the host version, call host function LoopOnCpu instead of host device function Loop because the function we pass to it is a host function. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_BaseFab.H commit cd6f4d0bdbc389f672e3d6a3b8d286729caca3b6 Author: Andrew Myers Date: Tue Nov 17 13:33:00 2020 -0500 Fix bug in the 'intarray' method of StructOfArrays (#1548) Thanks to @rporcu for finding this. The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Particle/AMReX_StructOfArrays.H commit b329cfe0bc4036aae7b69dc94b7bb34f8ab040c9 Author: Axel Huebl Date: Mon Nov 16 21:07:35 2020 -0800 CMake: generate_buildinfo target (#1545) This commit generates a static library `buildInfo::` for the passed user target. This target is then automatically linked. Also, the generated `.cpp` file is now scoped properly, avoiding collisions. This solves the following corner cases: - workflows with OBJECT libraries, that cannot be passed as targets since they have no `PRE_BUILD` custom command support - workflows with multiple targets that are not linked together (those call `generate_buildinfo` multiple times) - workflows with multiple targets that are linked together (those call `generate_buildinfo` once and re-use the target) Tools/CMake/AMReXBuildInfo.cmake commit 5076d33c583d88e0fb78214470715c2d09c083cd Author: Shreyas Ananthan Date: Mon Nov 16 10:00:02 2020 -0700 fcompare: fix logic error when comparing both rel/abs tolerances (#1542) PR #1537 introduced the option to allow tolerance checks on both absolute and relative tolerances. However, the check used `and` instead of `or` to allow tests to pass when either absolute or relative error was below user-specified tolerance. ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/Plotfile/fcompare.cpp commit 95b97799f2e611d19a4b685f725bfa3e5e12dc7f Author: Weiqun Zhang Date: Sun Nov 15 17:11:12 2020 -0800 nvcc --Werror cross-execution-space-call (#1540) * nvcc --Werror cross-execution-space-call * GNU make: Turn on the flag by default. Use `GPU_ERROR_CROSS_EXECUTION_SPACE_CALL=FALSE` to turn it off. * CMake: Add AMReX_ERROR_CROSS_EXECUTION_SPACE_CALL, which if off by default (just like AMReX_ERROR_CAPTURE_THIS). * update cmake documentation Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReX_SetupCUDA.cmake Tools/GNUMake/Make.defs Tools/GNUMake/comps/nvcc.mak commit aef49304278572eb3c4a5db75462c8c213d35221 Author: Max Katz Date: Sun Nov 15 11:22:29 2020 -0800 Don't throw C++ std error when not config checking (#1543) Tools/GNUMake/comps/gnu.mak Tools/GNUMake/comps/nvcc.mak commit c80007df2f90e373f3d947fa7f621ab902378f3c Author: Shreyas Ananthan Date: Thu Nov 12 17:36:06 2020 -0700 fcompare: Add option for an absolute tolerance check (#1537) This PR adds a command line option to `fcompare` that takes in a user-specified tolerance for absolute error and adds logic to compare both absolute and relative errors against user-specified tolerances. Since `-a` is already used as a short-form for `--allow_diff_grids`, I have only used `--abs_tol` as the option without a short version when parsing command line options. Tools/Plotfile/fcompare.cpp commit cdb236ee435635f8007d2be97c0f6f1df13ecba0 Author: Weiqun Zhang Date: Thu Nov 12 08:50:35 2020 -0800 Remove call to cudaDeviceSetCacheConfig (#1536) ## Summary No longer set CUDA device cache configuration to prefer L1 cache. This does not appear to affect any kernels in a negative way. With more shared memory available, reduction functions are faster in some cases. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_GpuDevice.cpp commit f55a950e0977a4b062bba6ff9787ba84e9f7da92 Author: Weiqun Zhang Date: Thu Nov 12 08:11:02 2020 -0800 Fix multi-component nodal solver tutorial (#1535) ## Summary `MultiFab::setVal` should not be called inside `MFIter`. ## Additional background This was caught by #1530. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tutorials/LinearSolvers/MultiComponent/MCNodalLinOp.cpp commit a01697fda4a2f38a1e7e3c849bc52baca2c2999d Author: Axel Huebl Date: Tue Nov 10 18:18:03 2020 -0800 Ascent: Check Bounds Particle Reals (#1515) Access of particle variable names with a bound-check, so users definitely pass the right amount of names. Src/Extern/Conduit/AMReX_Conduit_Blueprint_ParticlesI.H commit d9f60b4cb934852cb87653fbc84df22c98cfe0e0 Author: Andrew Myers Date: Tue Nov 10 09:02:30 2020 -0800 Allow for particle containers constructed with the same AmrCore... (#1532) Src/AmrCore/AMReX_AmrParGDB.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_Particles.H commit 10c2a5e5949213d607305f771edab433285166d7 Author: asalmgren Date: Mon Nov 9 17:58:38 2020 -0800 Update the docs to include hypre.adjust_singular_matrix (#1534) ## Summary adds documentation of hypre.adjust_singular_matrix ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [X] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/LinearSolvers.rst commit 59b8bf1cc1ebf6dac76602a24a4ce696080b070c Author: Weiqun Zhang Date: Mon Nov 9 15:59:56 2020 -0800 DPC++: fix scan (#1533) * Use atomics instead of volatile to read status written to global memory by other blocks * Workaround `Random()` bug in the scan test * Limit the memory usage in the scan test Src/Base/AMReX_Scan.H Tutorials/GPU/ParallelScan/main.cpp commit 15ab2368225b81777e2810e0e02b1293364bb575 Author: Weiqun Zhang Date: Mon Nov 9 14:02:21 2020 -0800 Fix #1530 (#1531) `FillPatchIterator` is derived from `MFIter`. We need to reset `MFIter::depth` so that the ctor of `FillPatchIterator` can start `MFIter`. Src/Amr/AMReX_AmrLevel.cpp commit b94e1d2fca2bffd2b0e826eebfca7d63290efd40 Author: Weiqun Zhang Date: Mon Nov 9 10:30:06 2020 -0800 Assert no nested MFIters (#1530) ## Summary Add assertion to catch nested MFIters (e.g., MultiFab functions are called inside MFIter). ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_MFIter.H Src/Base/AMReX_MFIter.cpp commit ad7c4eac3475fb7ddfc2abe8c74400d0274f8398 Author: Weiqun Zhang Date: Mon Nov 9 10:29:06 2020 -0800 Add anonymous namespace back to parameters in AMReX_Amr.cpp (#1529) Src/Amr/AMReX_Amr.H Src/Amr/AMReX_Amr.cpp commit eb29a4f6b40e6458323db681795c1a12e1403c1c Author: Weiqun Zhang Date: Mon Nov 9 07:51:46 2020 -0800 Free MPI_Datatype defined by AMReX (#1527) ## Summary This fixes a one-time memory leak in MPI_Datatypes defined by AMReX as reported in #1525. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_ParallelDescriptor.cpp commit 0fd351b118b695d9c439c33a0f1ff9c6f3f00aeb Author: Weiqun Zhang Date: Sun Nov 8 10:27:36 2020 -0800 Update OLCF makefile (#1528) ## Summary * Remove summitdev and peak. * Environment variable OLCF_MODULEPATH_ROOT is used in addition to host name to detect OLCF machines. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit 6cc5104e57548e575717051b04361ffed0fee9c2 Author: Weiqun Zhang Date: Sun Nov 8 10:26:56 2020 -0800 Remove getGridSize that is no longer used. (#1524) Src/Base/AMReX_GpuLaunch.H Src/Base/AMReX_GpuLaunch.cpp Src/Base/CMakeLists.txt Src/Base/Make.package commit 86abbe0221c1c3c53e69d740cdc24190ce1376c5 Author: Weiqun Zhang Date: Sat Nov 7 14:57:59 2020 -0800 Support CUDA arch < 6.0 (#1512) ## Summary * Implement atomicAdd for CUDA arch < 6.0. * Remove the CUDA arch check in CMake. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_BaseFab.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_GpuDevice.cpp Tools/CMake/AMReX_SetupCUDA.cmake commit ec4424eb66366c5b798639a100b30c6c01938f79 Author: Axel Huebl Date: Sat Nov 7 14:56:34 2020 -0800 Change Spelling: Env AMREX_CUDA_ARCH (#1522) ## Summary Make the _environment variable_ that sets a default CUDA architecture all-caps, as this is way more common in Unix. Typical Values: `7.0` or `Volta` (i.e. for V100) ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/GPU.rst Tools/CMake/AMReX_SetupCUDA.cmake commit 1b00f34ef1e170d4e2b398045e9dd45b4ba8ec34 Author: Weiqun Zhang Date: Fri Nov 6 18:18:27 2020 -0800 make clean & make cleanconfig (#1520) ## Summary `make clean` now does what `make realclean` does. `make realclean` is kept. `make cleanconfig` is introduced to do what `make clean` used to do. For most users, `make clean` is the one that should be used. Also add some tests in makefiles to avoid error messages in case when `make clean` is run on a makefile with the default not suitable for the system. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.defs Tools/GNUMake/Make.rules Tools/GNUMake/README.md Tools/GNUMake/comps/hip.mak Tools/GNUMake/comps/nvcc.mak Tools/GNUMake/sites/Make.unknown commit d9ac9a6c844a5055cdd2b332d101096da3ae5d72 Author: mic84 Date: Fri Nov 6 13:47:58 2020 -0800 CMake: fix options name in Test directory (#1514) Tests/CMakeLists.txt Tests/HDF5Benchmark/CMakeLists.txt commit 6dc26f704a386a03120f71395f00186049499264 Author: mic84 Date: Fri Nov 6 13:47:15 2020 -0800 CMake: add component check for Fortran (#1511) Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/CMake/AMReXConfig.cmake.in commit 9f2822bdcdd5604eea85a6206413684fb0137ea4 Author: mic84 Date: Fri Nov 6 13:44:41 2020 -0800 CMake: fix some quirks with options output (#1516) Tools/CMake/AMReXOptions.cmake commit 79d991ca8518943155e312a9a7566380e563bc93 Author: Shreyas Ananthan Date: Fri Nov 6 12:41:36 2020 -0700 hypre: Adjust singular matrices before hypre solves (#1519) Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreIJIface.H Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp commit fe6058b946fb4334b74c42fedde919836f82066d Author: Shreyas Ananthan Date: Fri Nov 6 12:04:55 2020 -0700 hypre: Update nodal Laplacian interface to accept custom options (#1518) HypreNodeLap class creates the hypre IJ instance in its constructor. Therefore, the custom option namespace specified by user must be part of the constructor arguments. Src/Extern/HYPRE/AMReX_HypreNodeLap.H Src/Extern/HYPRE/AMReX_HypreNodeLap.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp commit 7b7c1c8e6addc152fb9c0059577799bfe8dc6d93 Author: Weiqun Zhang Date: Thu Nov 5 16:27:28 2020 -0800 DPC++: sync after htod_memcpy on null stream (#1517) ## Summary DPC++ does not have the concept of null stream. Adding a sync after htod_memcpy on the "null" stream will help eliminate potential bugs (e.g., htod_memcpy is called before MFIter that uses non-null stream). ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_GpuDevice.H Src/Base/AMReX_Reduce.H commit 8035b01d86a7f8f4542bec6aa167dc864c8b3209 Author: Shreyas Ananthan Date: Thu Nov 5 15:06:36 2020 -0700 CMake: Fix ENABLE_HYPRE to AMReX_HYPRE (#1513) Fix the variable being checked to enable HYPRE. ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/CMake/AMReXThirdPartyLibraries.cmake commit 9aa75b1844e4ecc6c860a2d5d37b57bccedf73ed Author: Max Katz Date: Thu Nov 5 08:59:20 2020 -0800 Fix some build issues at LLNL (#1510) Tools/GNUMake/sites/Make.llnl commit 7ca532e5b299df1d7e9137e61f9271e4341c6189 Author: Andrew Myers Date: Thu Nov 5 08:54:57 2020 -0800 Add timer to neighbor list construction (#1509) Src/Particle/AMReX_NeighborList.H commit 86c6b3d00ae8d8e232b345b7060d9188fc838136 Author: mic84 Date: Wed Nov 4 19:39:39 2020 -0800 CMake: overhaul options (#1490) * Doc: move ENABLE_CUDA_FASTMATH to the GPU options section * CMake: change all options to include AMReX_ namespace * CMake: remove ENABLE_ACC option * CMake: make AMReX_SPACEDIM a multi-valued string * Doc: add option AMReX_ENABLE_TESTS * CMake: rename AMReX_Options.cmake to AMReXOptions.cmake * CMake: refactor GPU-related options * CMake: MPI must be turned off if DPCPP is enabled * CMake: set via precision via selection of value * CMake: forgot to update Particles/CMakeLists.txt * CMake: append prefix to TL_PROFILE * Update Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Co-authored-by: Axel Huebl * Update Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Co-authored-by: Axel Huebl * Update Docs/sphinx_documentation/source/BuildingAMReX.rst Co-authored-by: Axel Huebl * CMake: components in Config file shall not have namespace prefix * Revert "CMake: components in Config file shall not have namespace prefix" This reverts commit 924cf15f2a926467f3924a76f80595c0d1e53a7a. * Update Tools/CMake/AMReX_SetupCUDA.cmake Co-authored-by: Axel Huebl * CMake: update Doc * Update Tools/CMake/AMReX_SetupCUDA.cmake Co-authored-by: Axel Huebl Co-authored-by: Axel Huebl .github/workflows/linux.yml .github/workflows/macos.yml .github/workflows/windows.yml CHANGES CMakeLists.txt Docs/sphinx_documentation/source/AMReX_Profiling_Tools.rst Docs/sphinx_documentation/source/Basics.rst Docs/sphinx_documentation/source/BuildingAMReX.rst Docs/sphinx_documentation/source/GPU.rst Docs/sphinx_documentation/source/Visualization.rst Src/Amr/CMakeLists.txt Src/AmrCore/CMakeLists.txt Src/Base/CMakeLists.txt Src/Boundary/CMakeLists.txt Src/CMakeLists.txt Src/EB/CMakeLists.txt Src/Extern/HYPRE/CMakeLists.txt Src/Extern/PETSc/CMakeLists.txt Src/Extern/ProfParser/CMakeLists.txt Src/Extern/amrdata/CMakeLists.txt Src/F_Interfaces/CMakeLists.txt Src/LinearSolvers/CMakeLists.txt Src/Particle/CMakeLists.txt Tests/CMakeLists.txt Tools/CMake/AMReXConfig.cmake.in Tools/CMake/AMReXInstallHelpers.cmake Tools/CMake/AMReXOptions.cmake Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXSYCL.cmake Tools/CMake/AMReXThirdPartyLibraries.cmake Tools/CMake/AMReX_Config.cmake Tools/CMake/AMReX_Defines.cmake Tools/CMake/AMReX_Options.cmake Tools/CMake/AMReX_SetupCUDA.cmake Tools/Plotfile/CMakeLists.txt Tutorials/Amr/Advection_AmrCore/CMakeLists.txt Tutorials/Amr/Advection_AmrLevel/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_C/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_CF/CMakeLists.txt Tutorials/Basic/HeatEquation_EX1_F/CMakeLists.txt Tutorials/Basic/HeatEquation_EX2_CF/CMakeLists.txt Tutorials/Basic/HeatEquation_EX3_C/CMakeLists.txt Tutorials/Basic/HelloWorld_F/CMakeLists.txt Tutorials/Basic/main_C/CMakeLists.txt Tutorials/Basic/main_F/CMakeLists.txt Tutorials/Blueprint/AssignMultiLevelDensity/CMakeLists.txt Tutorials/Blueprint/CellSortedParticles/CMakeLists.txt Tutorials/Blueprint/HeatEquation_EX1_C/CMakeLists.txt Tutorials/CMakeLists.txt Tutorials/EB/CNS/CMakeLists.txt Tutorials/EB/GeometryGeneration/CMakeLists.txt Tutorials/ForkJoin/MLMG/CMakeLists.txt Tutorials/ForkJoin/Simple/MyTest.cpp Tutorials/FortranInterface/Advection_F/CMakeLists.txt Tutorials/FortranInterface/Advection_octree_F/CMakeLists.txt Tutorials/FortranInterface/Advection_octree_F2/CMakeLists.txt Tutorials/GPU/CNS/CMakeLists.txt Tutorials/GPU/EBCNS/CMakeLists.txt Tutorials/GPU/Launch/CMakeLists.txt Tutorials/LinearSolvers/ABecLaplacian_C/CMakeLists.txt Tutorials/LinearSolvers/ABecLaplacian_F/CMakeLists.txt Tutorials/LinearSolvers/MAC_Projection_EB/CMakeLists.txt Tutorials/LinearSolvers/Nodal_Projection_EB/CMakeLists.txt Tutorials/Particles/CellSortedParticles/CMakeLists.txt Tutorials/Particles/ElectromagneticPIC/CMakeLists.txt Tutorials/Particles/ElectrostaticPIC/CMakeLists.txt Tutorials/Particles/NeighborList/CMakeLists.txt Tutorials/SUNDIALS/EX1_C/CMakeLists.txt commit 22e6f93f2f66d24e364f35226b3f0d990182e316 Author: Weiqun Zhang Date: Wed Nov 4 17:00:42 2020 -0800 New Reduce::Sum function taking lambda function (#1508) ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Reduce.H commit ae9d0c696dad43b4477517728bf0f2ef978165db Author: Weiqun Zhang Date: Wed Nov 4 14:50:18 2020 -0800 GNU Make: AMREX_CUDA_ARCH (#1507) ## Summary Option to use AMREX_CUDA_ARCH in addition to CUDA_ARCH to set CUDA arch. If AMREX_CUDA is set, it will be used over CUDA_ARCH. ## Additional background The CMake system is also planning to support AMREX_CUDA_ARCH. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [x] are described in the proposed changes to the AMReX documentation, if appropriate Docs/sphinx_documentation/source/BuildingAMReX.rst Tools/GNUMake/Make.defs commit 1ab6d82e749b039b0417fa6c539c04d464ba95ac Author: Andrew Myers Date: Wed Nov 4 10:54:39 2020 -0800 Fix bug in new assertion (#1506) Src/Particle/AMReX_ParticleContainerI.H commit db41d3d75449b162ebe4113656da49536149e165 Author: Max Katz Date: Wed Nov 4 09:23:41 2020 -0800 Add option for device sync at beginning and end of TinyProfiler region (#1505) ## Summary Now when setting `tiny_profiler.device_synchronize_on_region_end = 1` in the inputs file, we will synchronize before calling nvtxRangePop() and nvtxRangePush(), which means that TINY_PROFILE regions will include the full kernel time, rather than just the kernel launch time. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_TinyProfiler.H Src/Base/AMReX_TinyProfiler.cpp commit 81fdb9a6254fee22daf3c32843c619e08d527bbc Author: Max Katz Date: Wed Nov 4 07:50:41 2020 -0800 Guard MFIter reduction macros against USE_GPU_PRAGMA (#1504) Src/Base/AMReX_BLFort.H commit 461ca5b42dfeb3ac24ab0aa344d3fc00129cc8b4 Author: Weiqun Zhang Date: Tue Nov 3 15:39:57 2020 -0800 GNU make: adjust multiple target rules (#1500) ## Summary Remove `/` from the object file name so that it works for multiple targets in different directories. This should not affect its current usage. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.rules commit 3ed4b614a91e8b49d3e6f9b434493cf8f1c9464a Author: mic84 Date: Tue Nov 3 14:09:34 2020 -0800 CMake; bump up hypre minimi required version (#1503) Tools/CMake/AMReXThirdPartyLibraries.cmake commit f4c041f181dd1bb5743483e4b86f19cbb6e40fb2 Author: Andrew Myers Date: Tue Nov 3 13:54:38 2020 -0800 assert if aos.size() != soa.size() (#1502) Src/Particle/AMReX_ParticleContainerI.H commit b5a506997ce5db41bc46d6d64a238053fa2ea1ea Author: Weiqun Zhang Date: Tue Nov 3 13:53:23 2020 -0800 Fix warnings in AMReX_Machine.cpp (#1499) Src/Base/AMReX_GpuDevice.cpp Src/Base/AMReX_Machine.H Src/Base/AMReX_Machine.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp commit e61266427dc11ffdfd2bad6577ac204e19ad5e88 Author: Andrew Myers Date: Tue Nov 3 13:52:40 2020 -0800 Use Long type for pindex to avoid potential overflow. (#1501) Src/Particle/AMReX_ParticleContainerI.H commit 7117a8f9006ad2d647c3e6412aee3ec3047d6def Author: Weiqun Zhang Date: Tue Nov 3 09:01:25 2020 -0800 Update GNU make for Cori GPU node (#1497) ## Summary Set CUDA_ARCH to 80 for A100 on Cori. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.nersc commit 24cacefc388e9811f52c1ebbceeb06cca90ba9b6 Author: Axel Huebl Date: Tue Nov 3 08:07:50 2020 -0800 Lin. Solvers (MLMG): Include Order (#1496) ## Summary Include own headers before stdlib headers to catch missing includes early. This avoid problems with less common compilers and environments. Carved out of #1485 ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/LinearSolvers/MLMG/AMReX_MLCGSolver.H Src/LinearSolvers/MLMG/AMReX_MLCGSolver.cpp Src/LinearSolvers/MLMG/AMReX_MLLinOp.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp commit ee3076a8306269bf6d18ffee3f4d4944c660241d Author: Max Katz Date: Tue Nov 3 08:06:58 2020 -0800 GNUMake: Merge Ascent and Peak with Summit (#1493) ## Summary These are all systems with the same architecture so we can share the build setup. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/GNUMake/sites/Make.olcf commit 43f112fe4b9deda918f3d3961d2979e124d0645d Author: Weiqun Zhang Date: Mon Nov 2 13:00:42 2020 -0800 Update CHANGES for AMReX 20.11 (#1494) CHANGES commit db23e834e9cdc24884b689e0bdbd285ba6124562 Author: Shreyas Ananthan Date: Mon Nov 2 11:00:19 2020 -0700 HYPRE: Fix issues with hypre IJ interface (#1495) This commit fixes two issues with the hypre IJ interface - Fixes a memory leak in HypreIJIface as it was calling parse_inputs in constructor - Sets default number of iterations and tolerance for BoomerAMG when used as a preconditioner Src/Extern/HYPRE/AMReX_HypreIJIface.cpp commit f29a0c9d1b8e6356f8007a0bcb9cfc1000f2a0ac Author: Weiqun Zhang Date: Fri Oct 30 15:45:23 2020 -0700 Change default hypre interface to ij for non-EB (#1492) Src/LinearSolvers/MLMG/AMReX_MLMG.H commit 93ffe6391b8a18c749610cabe05c785b6325378d Author: Weiqun Zhang Date: Fri Oct 30 10:39:39 2020 -0700 Fix DEBUG build for DPC++ (#1491) This fixes a bug introduced in #1489. Src/Base/AMReX.cpp commit 82c7a83ff6dd43be71d5960ad422f4db88d929e7 Author: Weiqun Zhang Date: Fri Oct 30 08:45:24 2020 -0700 Changes to make hipcc --save-temps work. (#1489) ## Summary It does not like AMREX_GPU_DEVICE_EXTERNAL. Also changed is the device version of `Error`, `Abort` and `Warning`. If `NDEBUG` is defined, they do not do anything. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX.H Src/Base/AMReX.cpp Tools/GNUMake/Make.defs commit 69c496f8e8e65f7fe4e9cd5afd4e9e89fa4ad9ba Author: Weiqun Zhang Date: Thu Oct 29 12:32:24 2020 -0700 Fix GPU race condition for EB (#1487) A GPU race condition was introduced in #1451 trying to fix faces and cells next to covered cells that were converted from small cut cells. It is fixed by launching a separate kernel for each direction. Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.cpp commit 147149885e7ca02dcfdc88f1f4953bf26dc10dc8 Author: Axel Huebl Date: Thu Oct 29 11:30:41 2020 -0700 Ascent: Use Default Name for Mesh Ghost (#1488) ## Summary Use the default name for ghost cells in Ascent. The current name is non-default and causes issues at domain-boundaries during volume rendering (unless one overwrites the `ghost_field_name` keyword in the options). Thanks to @mclarsen and @cyrush for debugging and hinting this with me. ## Additional background Seen with WarpX. Pre-PR: ![replay_000400(9)](https://user-images.githubusercontent.com/1353258/97613195-9eed1c80-19d5-11eb-9934-cafeffc51458.png) Post-PR: ![replay_000400(10)](https://user-images.githubusercontent.com/1353258/97614230-eb852780-19d6-11eb-8dcd-4ca4d86d2e21.png) ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/Conduit/AMReX_Conduit_Blueprint.cpp commit 32052a90a11da0dec54c353e0d6e29758e52b9a1 Author: Axel Huebl Date: Wed Oct 28 16:08:03 2020 -0700 CI: Check More Warnings (#1486) ## Summary Checks for more warnings that are likely to uncover nifty bugs: `-Wshadow -Woverloaded-virtual -Wunreachable-code` Also checks for superfluous `;` with `-Wextra-semi` (only a flag in Clang). ## Additional background Does not yet add `-Wall -Wextra -Wpedantic` because there are quite a few places where we need to add casts to avoid signed-unsigned-comparisons, etc. WarpX' CI covers currently: * GNU: `-Wall -Wextra -Wpedantic -Wshadow -Woverloaded-virtual -Wunreachable-code` * Clang: `-Wall -Wextra -Wpedantic -Wshadow -Woverloaded-virtual -Wextra-semi -Wunreachable-code` Even with super-builds, we current include AMReX with `-isystem` and build AMReX with default compiler flags; this enables us to suppress most AMReX build warnings besides the ones that drop-through from macros defined in headers, which atm. are all fixed for the code sections we use. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate .github/workflows/linux.yml .github/workflows/macos.yml Src/Particle/AMReX_NeighborParticles.H Tutorials/Basic/PrefixSum_MultiFab/main.cpp Tutorials/Particles/CellSortedParticles/cell_sorted_F.H Tutorials/Particles/ElectromagneticPIC/Exec/OpenMP/em_pic_F.H Tutorials/Particles/ElectrostaticPIC/ElectrostaticParticleContainer.cpp Tutorials/Particles/NeighborList/MDParticleContainer.cpp commit da1063cc5c16dcf519e74d630f898e7b23711c76 Author: Weiqun Zhang Date: Wed Oct 28 14:03:39 2020 -0700 HIP GNU Make: need to override COMP for regression tests (#1467) Tools/GNUMake/Make.defs commit ee020a50235341408297ff7018c536486eb30754 Author: Axel Huebl Date: Wed Oct 28 13:38:34 2020 -0700 CI: -Werror for host compilers (#1484) ## Summary Add more warning coverage in CI for CMake builds. ## Additional background ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate .github/workflows/linux.yml .github/workflows/macos.yml commit 0186e2f103f800280d8edfc5dffcd9280f132f42 Author: Weiqun Zhang Date: Mon Oct 26 11:18:29 2020 -0700 Fix warning in FabSet for single precision (#1481) * Fix warning in FabSet for single precision * Constify constants Co-authored-by: Axel Huebl Src/Boundary/AMReX_FabSet.cpp commit 6f07fe4e84a4dd57a0e89c2d2cb84c9953fda85d Author: Weiqun Zhang Date: Fri Oct 23 16:23:17 2020 -0700 Workaround for Mac CI (#1482) .github/workflows/dependencies/dependencies_mac.sh commit 61ef93db305fccfe81404686d6878f1f805fee02 Author: Axel Huebl Date: Fri Oct 23 13:29:11 2020 -0700 CMake 3.18+: CUDA Arch Policy (OLD) (#1480) Keep the old policy for now to avoid setting the code generation flags twice. We should be able to transition this well, but I need to first find out how device LTO generation flags are handled here, if at all, and how to restore the default-detection of the local GPU architecture, e.g. on Summits head-nodes. CMakeLists.txt commit f4d73901d209caa80951b4842f0e46848104a98f Author: asalmgren Date: Fri Oct 23 10:13:49 2020 -0700 More fixes to documenting hypre (previous text got lost). (#1477) Docs/sphinx_documentation/source/LinearSolvers.rst commit f42ab5658751c51d6792d7dd7f1999c4c3310143 Author: Luca Fedeli Date: Thu Oct 22 17:23:11 2020 +0200 Add fill method to GpuArray (#1472) ## Summary I would like to propose to add a `fill` method to `GpuArray`. This PR implements just that. ## Additional background I am wondering if it would be better to replace `i < N` with `i < amrex::max(N,std::size_t{1})`, in order to deal with the case `N=0`. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [X] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Array.H commit 0cfd7d349279040c1d9b9283f970b333296594d9 Author: asalmgren Date: Thu Oct 22 08:21:53 2020 -0700 Update the docs to reflect the new hypre options (#1473) Docs/sphinx_documentation/source/LinearSolvers.rst commit 6071cf447613beba8753fb0a5409dbe4e299111a Author: Weiqun Zhang Date: Tue Oct 20 18:10:01 2020 -0700 Gpu::Atomic::AddNoRet (#1469) Add Gpu::Atomic::AddNoRet to use HIP's atomicAddNoRet for float, which is much faster than atomicAdd for float that is currently implemented with CAS. Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuAtomic.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_MultiFab.cpp Src/Base/AMReX_TypeTraits.H Src/EB/AMReX_EB_utils.cpp Src/Particle/AMReX_DenseBins.H Src/Particle/AMReX_Particle_mod_K.H Tests/Particles/NeighborParticles/MDParticleContainer.cpp Tests/Particles/ParticleMesh/main.cpp Tutorials/Particles/ElectromagneticPIC/Exec/CUDA/em_pic_K.H commit a004707203dcbd3e28c5e637c623d569301c64cd Author: Andy Nonaka Date: Tue Oct 20 18:01:29 2020 -0700 Enforce singular solvable (#1471) ## Summary Option to make the code NOT enforce solvability for singular problems. This is accomplished by calling linop.setEnforceSingularSolvable(false) The default behavior is true, so no results will change in existing codes ## Additional background ## Checklist The proposed changes: - [x] add new capabilities to AMReX Src/LinearSolvers/MLMG/AMReX_MLLinOp.H Src/LinearSolvers/MLMG/AMReX_MLMG.cpp commit 8350a1641044aa222d922e85bd082be5734e2a6a Author: mic84 Date: Tue Oct 20 17:15:07 2020 -0700 CMake: improve HDF5 support (#1468) * CMake: look only for the C component of HDF5 * CMake: use only C-related variables from FindHDF5 * CMake: add Tests/HDF5Benchmark to CTest * CMake: fix typo * Update Tools/CMake/AMReXThirdPartyLibraries.cmake * CMake: use hdf5 imported target if available Co-authored-by: Axel Huebl Tests/CMakeLists.txt Tests/HDF5Benchmark/CMakeLists.txt Tools/CMake/AMReXThirdPartyLibraries.cmake commit c0710a2bc4725aa8e7cab57690b27c73285436ae Author: Shreyas Ananthan Date: Sun Oct 18 14:16:56 2020 -0600 Use rocrand from HIP APT repository (#1466) .github/workflows/dependencies/dependencies_hip.sh commit 4f0052023f6c68ca2898ed99552e641a2e4dcc8f Author: Shreyas Ananthan Date: Sun Oct 18 14:16:03 2020 -0600 Address warnings in CUDA builds (#1465) Src/Base/AMReX.cpp Src/Base/AMReX_FBI.H Tools/Plotfile/fsnapshot.cpp commit aef02b9a4745005856a45417632b964a760dadb0 Author: mic84 Date: Sun Oct 18 11:03:49 2020 -0700 CMake: FindPETSc must not overwrite CMAKE_Fortran_FLAGS (#1464) ## Summary FindPETSc used to set ```CMAKE_Fortran_FLAGS``` to the list of PETSc include directories. This caused problems with the compilation of Fortran files (see Issue #1463 ). This PR fixes this. ## Additional background ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Tools/CMake/FindPETSc.cmake commit f64fb79d0f8a04a785e7c21f49dd81330ce55692 Author: Axel Huebl Date: Fri Oct 16 18:56:55 2020 -0700 CI: rocRAND v1.8.2 (#1462) Building the development branch is a bit unstable. .github/workflows/dependencies/dependencies_hip.sh commit 0cea78bf2f4f24568a1d71d20e8b2614fe19012c Author: mic84 Date: Fri Oct 16 17:33:42 2020 -0700 CMake: some cleanup (#1460) * CMake: provide function to retrieve AMReX version * CMake: remove unused helper functions CMakeLists.txt Tools/CMake/AMReX_Utils.cmake commit 931cf5993b8ad2d5fda1a853ebabb36dbe2ae5c2 Author: Weiqun Zhang Date: Fri Oct 16 17:30:50 2020 -0700 DPCPP: Scan (#1461) ## Summary Need to use global address space when doing atomic inc on virtual block id. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Scan.H commit 31ca23418f488fe0b309ef3fc3e99c522ced4921 Author: mic84 Date: Fri Oct 16 14:53:58 2020 -0700 [WIP] CMake: add support for dpc++ beta10 (#1459) * CMake: retrieve dpcpp version * CMake: modify dpc++ flags to account for beta10 version Tools/CMake/AMReXSYCL.cmake commit d036a6df1c6f8f7bb458c7e5253a91f4b6f96b83 Author: Max Katz Date: Fri Oct 16 14:21:28 2020 -0700 Fix potential out of bounds access in #1441 (#1458) ## Summary The user may have given us a vector that has too many values, so we just ignore the remainder. ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_ErrorList.H commit 18b293dad9199deb329efecddd56ccce709371f5 Author: Weiqun Zhang Date: Fri Oct 16 10:12:19 2020 -0700 DPCPP beta10 (#1456) ## Summary * Update for DPCPP beta10 * Fix some warnings * Remove some device versions of Random() for HIP and DPC++ ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Base/AMReX_Extension.H Src/Base/AMReX_FBI.H Src/Base/AMReX_GpuPrint.H Src/Base/AMReX_GpuReduce.H Src/Base/AMReX_Random.H Src/Base/AMReX_Random.cpp Src/Base/AMReX_RandomEngine.H Src/Base/AMReX_Scan.H Src/Particle/AMReX_BinIterator.H Src/Particle/AMReX_ParticleCommunication.H Src/Particle/AMReX_ParticleContainerI.H Tools/GNUMake/comps/dpcpp.mak commit 15643f4b0ab28dc6b3277c5c44c18ac1f80e1345 Author: mic84 Date: Thu Oct 15 16:21:13 2020 -0700 CMake: prevent in-source builds (#1453) * CMake: prevent in-source builds * CMake: make error message more detailed CMakeLists.txt commit c53b1bbe88bed6b731bae4e11fd9ed0883a6b814 Author: Andy Nonaka Date: Thu Oct 15 14:29:41 2020 -0700 change char * to char const* so SWFFT code compiles with USE_CUDA=TRUE (#1454) Src/Extern/SWFFT/distribution.c commit 48b0a20210b9825c1c403e1e0b8a741ba913dffa Author: Shreyas Ananthan Date: Wed Oct 14 15:00:12 2020 -0600 Make HypreABecLap3::getSolution public for CUDA builds (#1452) Src/Extern/HYPRE/AMReX_HypreABecLap3.H commit 5843b838c1a86ea7a7978d0b6edd9f5e0c03d24a Author: asalmgren Date: Wed Oct 14 09:10:35 2020 -0700 Fix when small_volfrac is used -- we need to adjust the data in cut c… (#1451) Fix when small_volfrac is used -- we need to adjust the data in cut cell neighbors as well as regular neighbors This addresses issue #1450 ## Checklist The proposed changes: - [X] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/EB/AMReX_EB2_2D_C.cpp Src/EB/AMReX_EB2_3D_C.cpp commit 79f3a94fd50b6e9bcac6f99d0b03ffb47e9a5a87 Author: Andrew Myers Date: Wed Oct 14 00:39:23 2020 -0700 Allow users to set a Geometry object for ParticleContainer independently of the AmrCore / AmrLevel object. (#1446) Src/AmrCore/AMReX_AmrParGDB.H Src/Particle/AMReX_ParGDB.H Src/Particle/AMReX_Particles.H commit ff17a9295b54a36fa78c4f3c2350c3b3b5dd1338 Author: Houjun Date: Tue Oct 13 12:14:50 2020 -0700 HDF5 plotfile write bug fix and performance improvement (#1448) * Add HDF5 optimizations * Fix an issue that causes incorrect HDF5 data write with Tutorials/Basic/HeatEquation_EX1_C example code Co-authored-by: Houjun Tang Src/Base/AMReX_PlotFileUtil.cpp Src/Particle/AMReX_ParticleHDF5.H Tests/HDF5Benchmark/GNUmakefile commit 6649d227425fdc8ecbed301c4383c8dc74800323 Author: mic84 Date: Tue Oct 13 11:52:09 2020 -0700 CMake: add HIP support (#1316) * CMake: first basic attempt at HIP support * Update Tools/CMake/AMReXParallelBackends.cmake Co-authored-by: Axel Huebl * CI: Add HIP-Clang * fix typo: GCC 9.3 on focal * Fix sources: set two lists * fix target properties: append two lists * CMake: some changes to debug issue with generated files * CI: only run HIP check for now * CMake: fix typo * CMake: let's try this * CMake: other small modifications * CMake: attemp #1 * CMake: attemp #2 * CI: build just the bare minimum for now * CMake: attempt #3 * CMake: missing defines for HIP builds * CMake: defines were still missing :-P * CMake: AMREX_HIP_PLATFORM is actually the HIP_COMPILER * CMake: let's see if this works * Install rocRAND * CMake: re-factor and update defines for HIP compilation * CMake: list against hiprand too * CMake: find and link to hiprand * CMake: add -DNDEBUG to any type of build when HIP is on * CMake: we must manually add rocrand too * CMake: let's see if it works with Fortran enabled * Revert "CMake: let's see if it works with Fortran enabled" This reverts commit 065071539fd166f3bee994c2621d86c773180e3d. * CMake: try this * CMake: this should work too * Revert "Revert "CMake: let's see if it works with Fortran enabled"" This reverts commit 6a92adcc4ddfb06b4130f10d29cc49b298a66e73. * CMake: no fortran for now * CI: turn on linear solvers in HIP check * CI: turn on particles in HIP checks * Revert "CI: turn on linear solvers in HIP check" This reverts commit c723560a7b401266d6259af340130b1065e6be59. The reason is that Linear Solvers tutorials are not HIP-aware yet. * CMake: hipify tutorials * CMake: temporary fix * Revert "CMake: temporary fix" This reverts commit bd6724ed73f7afc82a4dc60c60d4390c6a83808d. * CMake: fix HIP compilation for Tutorials * CMake: some cleanup * CMake: add HIP options to pass in architecture and extra flags * CMake: oops * CMake: try to enable linear solvers * CMake: update ROCm version number in CI check name * CMake: enable Fortran * CMake: disable tutorials for the time being * CMake: commit custom FindHIP.cmake to debug the issue * CMake: trying this * CMake: now it should use the local FindHIP.cmake * CMake: now custom FindHIP.cmake should be able to find helper files * CMake: let's see if we really need this * CMake: let's try this * CMake: temporary fix * Revert "CMake: disable tutorials for the time being" This reverts commit 885b9113b72f54bca9eaf74f4e5b6b90b17032ff. * CMake: Tutorials/Particles/CellSortedParticles do not work with HIP * CMake: remove local FindHIP.cmake. * CMake: HIP arch flags are now PUBLIC and inheritable * CMake: no use for setup_target_for_hip_compilation anymore * CMake: remove unnecessary options for HIP * CMake: check that HIP_COMPILER is the same as CMAKE_CXX_COMPILER * CMake: better this * CI: uncomment all checks * CMake: fix visibility of arch flags for HIP * Doc: add subsection on HIP+CMake * Update Docs/sphinx_documentation/source/GPU.rst Co-authored-by: Shreyas Ananthan * CMake: fix incorrect configuration option passed to cmake * CI: target 'tutorials' no longer exists Co-authored-by: Axel Huebl Co-authored-by: Shreyas Ananthan .github/workflows/dependencies/dependencies_hip.sh .github/workflows/linux.yml Docs/sphinx_documentation/source/GPU.rst Src/CMakeLists.txt Tools/CMake/AMReXParallelBackends.cmake Tools/CMake/AMReXTargetHelpers.cmake Tools/CMake/AMReX_Defines.cmake Tools/CMake/AMReX_Options.cmake Tutorials/CMakeLists.txt Tutorials/Particles/CellSortedParticles/CMakeLists.txt commit 6dc6ce8f55a5ef19f349c267b7bb9fc4bb616b07 Author: Candace Gilet Date: Tue Oct 13 09:48:50 2020 -0400 volume multifab must be defined before calling GetVolume(volume) (#1449) Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 5e6d555d07d3862b5b88b6c38559a59918541194 Author: mic84 Date: Mon Oct 12 17:38:22 2020 -0700 NodalProjector: use volume-weighted average down (#1444) Src/LinearSolvers/Projections/AMReX_NodalProjector.cpp commit 95ddfdb2c1434b1705d84174047f06cd3004174d Author: Weiqun Zhang Date: Wed Oct 7 19:02:08 2020 -0700 Fix EB interpolation from cell centers to faces (#1443) ##Summary There were race conditions that made it fail on AMD GPUs. ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/EB/AMReX_EBMultiFabUtil.cpp Src/EB/AMReX_EBMultiFabUtil_2D_C.H Src/EB/AMReX_EBMultiFabUtil_3D_C.H commit 2c7b5004fb98d92bf185ecc73b6429dfd7ed8b6f Author: Weiqun Zhang Date: Wed Oct 7 15:16:51 2020 -0700 Fix bug in hypre overset solver (#1442) ## Summary ## Additional background ## Checklist The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [ ] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/PETSc/AMReX_PETSc.cpp commit 3972c6d582f024c4a4d108ff02e66d117e6db1e6 Author: Max Katz Date: Wed Oct 7 15:33:35 2020 -0400 Allow tagging value to vary by level (#1441) ## Summary The new error tagging scheme from #1166 is modified to allow the threshold value to vary by level. Using the example there, if we did: ``` amr.refinement_indicators = flame_tracer lo_temp amr.refine.flame_tracer.max_level = 3 amr.refine.flame_tracer.value_greater = 1.e-6 1.e-5 amr.refine.flame_tracer.field_name = Y(H) amr.refine.lo_temp.max_level = 2 amr.refine.lo_temp.value_less = 1000. amr.refine.lo_temp.field_name = temp ``` Then level 0 would be tagged for refinement for Y(H) >= 1.e-6, while level 1 and all higher levels would be tagged for refinement for Y(H) >= 1.e-5. (If a value for every level is not provided, we assume the last value holds for all remaining levels, similar to how we treat quantities like amr.n_error_buf.) ## Additional background This will help for cases where a field varies substantially on the domain and you want to have better control over refinement so that you don't have too many zones. ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] are described in the proposed changes to the AMReX documentation, if appropriate Src/AmrCore/AMReX_ErrorList.H Src/AmrCore/AMReX_ErrorList.cpp commit 92399587f20875a98655ccbd06a083965cfc9c5c Author: Weiqun Zhang Date: Tue Oct 6 13:30:52 2020 -0700 Hypre with overset (#1439) * Hypre with overset Add overset mask support for hypre. * Keep bottom_verbose restricted to amrex side. Hypre side's verbosity will be controlled by parmparse parameters. Src/Extern/HYPRE/AMReX_Habec_2D_K.H Src/Extern/HYPRE/AMReX_Habec_3D_K.H Src/Extern/HYPRE/AMReX_Hypre.H Src/Extern/HYPRE/AMReX_Hypre.cpp Src/Extern/HYPRE/AMReX_HypreABecLap.cpp Src/Extern/HYPRE/AMReX_HypreABecLap3.H Src/Extern/HYPRE/AMReX_HypreABecLap3.cpp Src/Extern/HYPRE/AMReX_HypreIJIface.cpp Src/LinearSolvers/MLMG/AMReX_MLABecLaplacian.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.H Src/LinearSolvers/MLMG/AMReX_MLCellABecLap.cpp Src/LinearSolvers/MLMG/AMReX_MLNodeLinOp.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_2D_K.H Src/LinearSolvers/MLMG/AMReX_MLNodeTensorLap_3D_K.H Tests/LinearSolvers/CellOverset/MyTest.H Tests/LinearSolvers/CellOverset/MyTest.cpp commit 6a20c352cf54ed9a1393676b1bf19c57941db2af Author: Axel Huebl Date: Mon Oct 5 16:17:43 2020 -0700 GNUmake: -Titan (OLCF) (#1440) R.I.P. Titan. Btw, only found because a colleague of mine could not compile WarpX. Their hostname is called -titan. Tools/GNUMake/Make.machines Tools/GNUMake/sites/Make.olcf commit 9992d844099c3df81d9499519ffeb7f93a201dc6 Author: Shreyas Ananthan Date: Sun Oct 4 18:19:24 2020 -0600 Hypre IJ interface: Enable access to additional solvers and preconditioners available in Hypre (#1437) This PR extends the AMReX _hypre_ IJ interface to allow access to additional solvers and preconditioners available in _hypre_. - Refactored `HypreABecLap3` and `HypreNodeLap` (IJ matrix/vector ParCSR interfaces) and created a new IJ interface class that provides a unified way to access Hypre ParCSR solvers. - In addition to BoomerAMG, adds support for 7 other [ParCSR solvers](https://hypre.readthedocs.io/en/latest/ch-solvers.html) (GMRES (4 variants), PCG, BiCGSTAB, and Hybrid). Also adds support for BoomerAMG and Euclid as preconditioners for solvers that can use a preconditioner. - Adds support to parse user options to configure hypre library via AMReX ParmParse interface and input file, using a custom namespace (default is `hypre`) - Adds `MLMG::setHypreOptionsNamespace` that allow applications to customize the namespace for different types of linear solvers (e.g., `mac_proj.hypre.